English - 4

Yesterday the LibreSSL project released the first portable version that works on Linux. LibreSSL is a fork of OpenSSL and was created by the OpenBSD team in the aftermath of the Heartbleed bug.

Yesterday and today I played around with it on Gentoo Linux. I was able to replace my system's OpenSSL completely with LibreSSL and with few exceptions was able to successfully rebuild all packages using OpenSSL.

After getting this running on my own system I installed it on a test server. The Webpage tlsfun.de runs on that server. The functionality changes are limited, the only thing visible from the outside is the support for the experimental, not yet standardized ChaCha20-Poly1305 cipher suites, which is a nice thing.

A warning ahead: This is experimental, in no way stable or supported and if you try any of this you do it at your own risk. Please report any bugs you have with my overlay to me or leave a comment and don't disturb anyone else (from Gentoo or LibreSSL) with it. If you want to try it, you can get a portage overlay in a subversion repository. You can check it out with this command:
svn co https://svn.hboeck.de/libressl-overlay/
git clone https://github.com/gentoo/libressl.git

This is what I had to do to get things running:

LibreSSL itself

First of all the Gentoo tree contains a lot of packages that directly depend on openssl, so I couldn't just replace that. The correct solution to handle such issues would be to create a virtual package and change all packages depending directly on openssl to depend on the virtual. This is already discussed in the appropriate Gentoo bug, but this would mean patching hundreds of packages so I skipped it and worked around it by leaving a fake openssl package in place that itself depends on libressl.

LibreSSL deprecates some APIs from OpenSSL. The first thing that stopped me was that various programs use the functions RAND_egd() and RAND_egd_bytes(). I didn't know until yesterday what egd is. It stands for Entropy Gathering Daemon and is a tool written in perl meant to replace the functionality of /dev/(u)random on non-Linux-systems. The LibreSSL-developers consider it insecure and after having read what it is I have to agree. However, the removal of those functions causes many packages not to build, upon them wget, python and ruby. My workaround was to add two dummy functions that just return -1, which is the error code if the Entropy Gathering Daemon is not available. So the API still behaves like expected. I also posted the patch upstream, but the LibreSSL devs don't like it. So on the long term it's probably better to fix applications to stop trying to use egd, but for now these dummy functions make it easier for me to build my system.

The second issue popping up was that the libcrypto.so from libressl contains an undefined main() function symbol which causes linking problems with a couple of applications (subversion, xorg-server, hexchat). According to upstream this undefined symbol is intended and most likely these are bugs in the applications having linking problems. However, for now it was easier for me to patch the symbol out instead of fixing all the apps. Like the egd issue on the long term fixing the applications is better.

The third issue was that LibreSSL doesn't ship pkg-config (.pc) files, some apps use them to get the correct compilation flags. I grabbed the ones from openssl and adjusted them accordingly.

OpenSSH

This was the most interesting issue from all of them.

To understand this you have to understand how both LibreSSL and OpenSSH are developed. They are both from OpenBSD and they use some functions that are only available there. To allow them to be built on other systems they release portable versions which ship the missing OpenBSD-only-functions. One of them is arc4random().

Both LibreSSL and OpenSSH ship their compatibility version of arc4random(). The one from OpenSSH calls RAND_bytes(), which is a function from OpenSSL. The RAND_bytes() function from LibreSSL however calls arc4random(). Due to the linking order OpenSSH uses its own arc4random(). So what we have here is a nice recursion. arc4random() and RAND_bytes() try to call each other. The result is a segfault.

I fixed it by using the LibreSSL arc4random.c file for OpenSSH. I had to copy another function called arc4random_stir() from OpenSSH's arc4random.c and the header file thread_private.h. Surprisingly, this seems to work flawlessly.

Net-SSLeay

This package contains the perl bindings for openssl. The problem is a check for the openssl version string that expected the name OpenSSL and a version number with three numbers and a letter (like 1.0.1h). LibreSSL prints the version 2.0. I just hardcoded the OpenSSL version numer, which is not a real fix, but it works for now.

SpamAssassin

SpamAssassin's code for spamc requires SSLv2 functions to be available. SSLv2 is heavily insecure and should not be used at all and therefore the LibreSSL devs have removed all SSLv2 function calls. Luckily, Debian had a patch to remove SSLv2 that I could use.

libesmtp / gwenhywfar

Some DES-related functions (DES is the old Data Encryption Standard) in OpenSSL are available in two forms: With uppercase DES_ and with lowercase des_. I can only guess that the des_ variants are for backwards compatibliity with some very old versions of OpenSSL. According to the docs the DES_ variants should be used. LibreSSL has removed the des_ variants.

For gwenhywfar I wrote a small patch and sent it upstream. For libesmtp all the code was in ntlm. After reading that ntlm is an ancient, proprietary Microsoft authentication protocol I decided that I don't need that anyway so I just added --disable-ntlm to the ebuild.

Dovecot

In Dovecot two issues popped up. LibreSSL removed the SSL Compression functionality (which is good, because since the CRIME attack we know it's not secure). Dovecot's configure script checks for it, but the check doesn't work. It checks for a function that LibreSSL keeps as a stub. For now I just disabled the check in the configure script. The solution is probably to remove all remaining stub functions. The configure script could probably also be changed to work in any case.

The second issue was that the Dovecot code has some #ifdef clauses that check the openssl version number for the ECDH auto functionality that has been added in OpenSSL 1.0.2 beta versions. As the LibreSSL version number 2.0 is higher than 1.0.2 it thinks it is newer and tries to enable it, but the code is not present in LibreSSL. I changed the #ifdefs to check for the actual functionality by checking a constant defined by the ECDH auto code.

Apache httpd

The Apache http compilation complained about a missing ENGINE_CTRL_CHIL_SET_FORKCHECK. I have no idea what it does, but I found a patch to fix the issue, so I didn't investigate it further.

Further reading:
Someone else tried to get things running on Sabotage Linux.

Update: I've abandoned my own libressl overlay, a LibreSSL overlay by various Gentoo developers is now maintained at GitHub.

Today I had a little twitter conversation which made me think about the responsibilities a science journalist has. It all started with a quote from Ivan Oransky (who is the editor of Retraction Watch) who said reporting on a study without reading it is 'journalist malpractice'. The source of this is another person who probably just heard him saying that, so I'm not sure what his exact words were.



Admittedly my first thought was: "He is right, too many journalists report about things they don't understand." My second thought was: "If he is right then I am probably guilty of 'journalist malpractice'." So I gave it a second thought and I probably won't agree with the statement any more.

I had a quick look at articles I wrote in the past and I have identified the last ten ones that more or less were coverages of a scientific piece of work. I have marked the ones I actually read with a [Y] and the ones I didn't read with a [N]. I've linked the appropriate scientific works and my articles (all in German). I must admit that I defined "read" widely, meaning that I haven't neccesarrily read the whole study/article in detail, I sometimes have just tried to parse the important parts for me.

1. [X] Supposedly successful Turing Test taz, 2014-06-13
2. [N]A quasi-polynomial algorithm for discrete logarithm in finite fields of small characteristic, Golem.de, 2014-05-17
3. [N] Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children (Cochrane-Review on Tamiflu), Neues Deutschland, 2014-04-26)
4. [N] 20 Years of SSL/TLS Research: An Analysis of the Internet's Security Foundation, Golem.de, 2014-04-17
5. [N] DRAFT FIPS 202 - SHA-3 Standard: Permutation-Based Hash and Extendable-Output Functions, Golem, 2014-04-05
6. [Y] Using Frankencerts for Automated Adversarial Testing of Certificate Validation in SSL/TLS Implementations, Golem.de, 2014-04-04
7. [Y] On the Practical Exploitability of Dual EC in TLS Implementations, Golem.de, 2014-04-01
8. [Y] Publishers withdraw more than 120 gibberish papers, Golem.de, 2014-02-27
9. [Y] Completeness of Reporting of Patient-Relevant Clinical Trial Outcomes: Comparison of Unpublished Clinical Study Reports with Publicly Available Data, taz, 2013-10-18
10. [Y] Factoring RSA keys from certified smart cards: Coppersmith in the wild, Golem.de, 2013-09-17

Now the first thing that comes to mind is that I seem to have become lazier recently in reading studies. I hope this isn't the case and I hoestly think this is mostly coincidence. Now let's get into some details: The first example (the Turing Test) is interesting because it seems there is no scientific publication at all, just a press release. This probably tells you something about the quality of that "research", but while I read the press release I haven't even bothered to check if there is a scientific publication I could read.

The second example becomes interesting. I understand enough to know what a "quasi-polynomial algorithm for discrete logarithm in finite fields of small characteristic" actually is and I think I also understand what it means, but there's just no way I could understand the paper itself. This is complex mathematics. I seriously doubt that any journalist who covered this work actually read it. If there is I'd like to meet that person. I'm also very sure that the people who wrote the press release overselling this research have neither read this paper nor understood its implications.

I think this example gets to the point why I would disagree with the very general statement that a journalist should've read every scientific piece he writes about: It's sometimes so specialized that it's basically impossible. And I don't think this is an out of the line example. Just think about the Higgs Boson: Certainly this is something we want journalists to write about. But I'm pretty sure there are very few - if any - journalists who are able to read the scientific publications that are the basis of this discovery.

Some quick notes on the others: Number 4 was part of a 200-page-thesis and the press release was already pretty detailed and technically, I think it was legitimate to not read the original source in that case. Number 5 is somewhat similar to 2, because it is about an algorithm that includes complex math. Number 8 is not really a scientific paper, it is merely a news item on the Nature webpage. In the above list, the only case where I think maybe I should've read the scientific paper and I didn't is the Cochrane-Review on Tamiflu.

Conclusion: Don't get me wrong. I certainly welcome the idea that science journalists should have a look into the original scientific papers they write about more often - and this doesn't exclude myself. However, as shown above I doubt that this works in all cases.

I recently held a workshop about cryptography for web developers at the company Internations. I am publishing the slides here.

Part 1: Crypto and Web [PDF] [LaTeX], [Slideshare]
Part 2: How broken is TLS? [PDF] [LaTeX], [Slideshare]
Part 3: Don't do this yourself [PDF] [LaTeX], [Slideshare]
Part 4: Hashing, Tokens, Randomness [PDF] [LaTeX], [Slideshare]
Part 5: Don't believe the Crypto Hype [PDF] [LaTeX] [Slideshare]

Part 2 is the same talk I recently have at the Easterhegg conference about TLS.

I recently switched my personal web page and my blog to deliver content exclusively encrypted via HTTPS. I want to take this opportunity to give some facts about enabling TLS encryption by default and problems you may face.

First of all the non-problems: Enabling HTTPS by default is almost never a significant performance problem. If people tell me that they can not possibly enable HTTPS due to performance reasons the first thing I ask is if they believe this or if they have real benchmark data showing this. If you don't believe me on that, I can quote Adam Langley from Google here: "In January this year (2010), Gmail switched to using HTTPS for everything by default. Previously it had been introduced as an option, but now all of our users use HTTPS to secure their email between their browsers and Google, all the time. In order to do this we had to deploy no additional machines and no special hardware. On our production frontend machines, SSL/TLS accounts for less than 1% of the CPU load, less than 10KB of memory per connection and less than 2% of network overhead."

Enabling HTTPS may cause a number of compatibility issues you may not instantly think about. First of all, we know that IPs in the IPv4 space are limited and expensive these days, so many people probably can't afford having a distinct IP for their web page. The solution to that is a TLS extension called SNI (Server Name Indication) which allows to have different certificates for different domain names on the same IP. It works in all major browsers and has been working for quite some time. The only major browser you'll face these days that doesn't support SNI is the Android 2.x browser.

There are some subtle issues with SNI. One is that browsers have fallback modes if they cannot connect via TLS and that may lead to a connection downgrade to SSLv3. And that ancient protocol doesn't support extensions and thus no SNI. So you may have irregular certificate errors if you are on a bad connection. A solution to that on the server side is to just disable SSLv3. It will make SNI much more reliable.

I don't really have a clear picture how many browsers will fail with SNI. There are probably a number of embedded devices out there like smart TVs with browsers or things alike that have problems. If you have any experiences feel free to post them in the comments.

The first issue I only noticed after I switched to HTTPS: I had an application called RSS Graffiti set up to automatically post all articles I write to a facebook fan page. After changing to HTTPS only it silently stopped working. Re-adding my feed didn't work. I now found a similar service called dlvr.it that I now use to post my RSS feed to facebook. I can only assume that this is a glimpse of a much bigger problem: There are probably tons of applications and online services out there not prepared for an encrypted Internet. If we want more people to deploy encryption by default we need to find these issues, document them and hopefully put enough pressure on their developers to fix them.

Another yet unfixed issue is the Yandex Bot. Yandex is a search engine and although you may never have heard of it it's probably one of the few companies in this area that can claim to be a serious competitor to Google. The reason you may not know it is that it's mostly operating in Russian language. Depending on who your page visitors are this may matter more or less.

The Yandex Bot speaks SSL but according to the Qualys SSL test it only supports the ancient SSLv3. So you have a choice between three possibilities: Don't enable HTTPS by default, enable HTTPS with a shitty configuration supporting ancient technology that will cause trouble for SNI or enable HTTPS with a sane configuration and get no traffic from the leading Russian search engine. None of them sounds very good to me.

Another issue is third party content. For security reasons today's browsers block all active HTTP content (CSS, JavaScript etc.) on HTTPS webpages. This isn't much of a problem for me, but it's a problem for webpages that rely on advertising because from what I hear most advertisement providers don't support HTTPS yet (Google being a laudable exception here). This is the main reason you won't see many news webpages enforcing HTTPS. However, I still have passive third party HTTP content on my blog. That's why you'll probably see a yellow warning sign in front of the URL in some browsers.

A number of people seem to be confused how to correctly install certificate chains for TLS servers. This happens quite often on HTTPS sites and to avoid having to explain things again and again I thought I'd write up something so I can refer to it. A few days ago flattr.com had a missing certificate chain (fixed now after I reported it) and various pages from the Chaos Computer Club have no certificate chain (not the main page, but several subdomains like events.ccc.de and frab.cccv.de). I've tried countless times to tell someone, but the problem persists. Maybe someone in charge will read this and fix it.

Web browsers ship a list of certificate authorities (CAs) that are allowed to issue certificates for HTTPS websites. The whole system is inherently problematic, but right now that's not the point I want to talk about. Most of the time, people don't get their certificate from one of the root CAs but instead from a subordinate CA. Every CA is allowed to have unlimited numbers of sub CAs.

The correct way of delivering a certificate issued by a sub CA is to deliver both the host certificate and the certificate of the sub CA. This is neccesarry so the browser can check the complete chain from the root to the host. For example if you buy your certificate from RapidSSL then the RapidSSL cert is not in the browser. However, the RapidSSL certificate is signed by GeoTrust and that is in your browser. So if your HTTPS website delivers both its own certificate by RapidSSL and the RapidSSL certificate, the browser can validate the whole chain.

However, and here comes the tricky part: If you forget to deliver the chain certificate you often won't notice. The reason is that browsers cache chain certificates. In our example above if a user first visits a website with a certificate from RapidSSL and the correct chain the browser will already know the RapidSSL certificate. If the user then surfs to a page where the chain is missing the browser will still consider the certificate as valid. Such certificates with missing chain have been called transvalid, I think the term was first used by the EFF for their SSL Observatory.

Now the CCC uses certificates from CAcert.org. Two more issues pop up here that make things even more complicated. First of all, the root certificate of CAcert is not in browsers, users have to manually import it. But CAcert offers both their root (Class 1) and sub (Class 3) certificate on the same webpage and doesn't really tell users that they usually only have to import the root. So everyone who imports both certificates will see transvalid CAcert certificates as valid. The second issue that pops up is that browsers sometimes do weird things when it comes to certificate error messages. I have no idea why exactly this is happening, but if you have the CAcert root installed and use Chromium to surf to a page with a transvalid CAcert certificate, it'll warn you about a weak signature algorithm. This doesn't make any sense, I can only assume that it has something to do with the fact that the CAcert root is self-signed with MD5 (which isn't a security issue, because self-signatures don't really have any meaning, they're just there because X.509 doesn't allow certificates without a signature).

So how can you check if you have a transvalid certificate? One way is to use a fresh browser installation without anything cached. If you then surf to a page with a transvalid certificate, you'll get an error message (however, as we've just seen, not neccessarily a meaningful one). An easier way is to use the SSL Test from Qualys. It has a line "Chain issues" and if it shows "None" you're fine. If it shows "Incomplete" then your certificate is most likely transvalid. If it shows anything else you have other things to look after (a common issues is that people unneccesarily send the root certificate, which doesn't cause issues but may make things slower). The Qualys test test will tell you all kinds of other things about your TLS configuration. If it tells you something is insecure you should probably look after that, too.

Last weekend I was at the Easterhegg in Stuttgart, an event organized by the Chaos Computer Club. I had a talk with the title "How broken is TLS?"

This was quite a lucky topic. I submitted the talk back in January, so I had no idea that the Heartbleed bug would turn up and raise the interest that much. However, it also made me rework large parts of the talk, because after Heartbleed I though I had to get a much broader view on the issues. The slides are here as PDF, here as LaTeX and here on Slideshare.

There's also a video recording here (media.ccc.de) and also on Youtube.

I also had a short lightning talk with some thoughs on paperless life, however it's only in German. Slides are here (PDF), here (LaTeX) and here (Slideshare). (It seems there is no video recording, if it appears later I'll add the link.)

If you don't know who I am and what I do, let me quickly introduce myself:

I live in Berlin and work as a freelance journalist. I sometimes write for the newspapers taz, the online version of the Zeit and I'm a regular author at the IT news magazine Golem.de. Currently, my main topics are IT security and cryptography. I cover those topics both for general interest media and for experts. I also write about some completely different topics like climate change, energy politics, science and problems in medicine and whatever I happen to find interesting. I maintain an extensive list of articles I wrote in the past on my website and just recently added an English version with Google Translate links to my articles.

A notable article I wrote lately was a large piece on the security of various encryption algorithms after the Snowden revelations which got around 900.000 visits. In the past days I covered the Heartbleed story extensively and am happy to say that I wrote the first article in German language that appeared on Google News.

I'm quite happy with my job right now. Especially my cooperation with Golem.de is going very well. I have enough topics to write about, have some new opportunities in sight and earn enough money from my work to pay my expenses However, all my current employers publish exclusively in German. I sometimes cover topics where I'd wish that I could target an international audience and where I'd like to publish in English language.

If you are working for any kind of media in English language and you think my work may be interesting for you: Please get in touch with me. Of course if you work for any kind of media in German language and think the same you may also get in touch with me.

I'm aware that this is difficult. Anyone who decides to cooperate with me on this needs to be aware: I'm no native speaker. I think my English language skills are decent, but they are far from perfect. My work probably requires more spell checking and editing than others.

I recently stepped upon a webpage where I wanted to extract an image. However, after saving the page with my browser I couldn't find any JPG or PNG file. After looking into this, I saw some CSS code that looked like this:

background-image:url("data:image/jpeg;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgAQAAAABbAUdZAAAAE0lEQVR4AWNgYPj/n4oElU1jAADtvT/BfzVwSgAAAABJRU5ErkJggg==";

What this does is that it embeds a base64 encoded image file into the CSS layout. I found some tools to create such images, but I found none to extract them. It isn't very hard to extract such an image, I wrote a small bash script that will do and that I'd like to share:

#!/bin/sh
n=1
for i in `grep -ho "base64,[A-Za-z0-9+/=]*" $@|sed -e "s:base64,::g"`; do
echo $i | base64 -d > file_$n
n=`expr $n + 1`
done

Save this as css2base64 and pass HTML or CSS files on the command line (e. g. css2base64 test.html test.css).

Hope this helps others. If this script is copyrightable at all (which I doubt), I hereby release it (like the other content of my blog) as CC0 / Public Domain.

tl;dr A very short key exchange crashes Chromium/Chrome. Other browsers accept parameters for a Diffie Hellman key exchange that are completely nonsense. In combination with recently found TLS problems this could be a security risk.

People who tried to access the webpage https://demo.cmrg.net/ recently with a current version of the Chrome browser or its free pendant Chromium have experienced that it causes a crash in the Browser. On Tuesday this was noted on the oss-security mailing list. The news spread quickly and gave this test page some attention. But the page was originally not set up to crash browsers. According to a thread on LWN.net it was set up in November 2013 to test extremely short parameters for a key exchange with Diffie Hellman. Diffie Hellman can be used in the TLS protocol to establish a connection with perfect forward secrecy.

For a key exchange with Diffie Hellman a server needs two parameters, those are transmitted to the client on a connection attempt: A large prime and a so-called generator. The size of the prime defines the security of the algorithm. Usually, primes with 1024 bit are used today, although this is not very secure. Mostly the Apache web server is responsible for this, because before the very latest version 2.4.7 it was not able to use longer primes for key exchanges.

The test page mentioned above tries a connection with 16 bit - extremely short - and it seems it has caught a serious bug in chromium. We had a look how other browsers handle short or nonsense key exchange parameters.

Mozilla Firefox rejects connections with very short primes like 256 bit or shorter, but connections with 512 and 768 bit were possible. This is completely insecure today. When the Chromium crash is prevented with a patch that is available it has the same behavior. Both browsers use the NSS library that blocks connections with very short primes.

The test with the Internet Explorer was a bit difficult because usually the Microsoft browser doesn't support Diffie Hellman key exchanges. It is only possible if the server certificate uses a DSA key with a length of 1024 bit. DSA keys for TLS connections are extremely rare, most certificate authorities only support RSA keys and certificates with 1024 bit usually aren't issued at all today. But we found that CAcert, a free certificate authority that is not included in mainstream browsers, still allows DSA certificates with 1024 bit. The Internet Explorer allowed only connections with primes of 512 bit or larger. Interestingly, Microsoft's browser also rejects connections with 2048 and 4096 bit. So it seems Microsoft doesn't accept too much security. But in practice this is mostly irrelevant, with common RSA certificates the Internet Explorer only allows key exchange with elliptic curves.

Opera is stricter than other browsers with short primes. Connections below 1024 bit produce a warning and the user is asked if he really wants to connect. Other browsers should probably also reject such short primes. There are no legitimate reasons for a key exchange with less than 1024 bit.

The behavior of Safari on MacOS and Konqueror on Linux was interesting. Both browsers accepted almost any kind of nonsense parameters. Very short primes like 17 were accepted. Even with 15 as a "prime" a connection was possible.

No browser checks if the transmitted prime is really a prime. A test connection with 1024 bit which used a prime parameter that was non-prime was possible with all browsers. The reason is probably that testing a prime is not trivial. To test large primes the Miller-Rabin test is used. It doesn't provide a strict mathematical proof for primality, only a very high probability, but in practice this is good enough. A Miller-Rabin test with 1024 bit is very fast, but with 4096 bit it can take seconds on slow CPUs. For a HTTPS connection an often unacceptable delay.

At first it seems that it is irrelevant if browsers accept insecure parameters for a key exchange. Usually this does not happen. The only way this could happen is a malicious server, but that would mean that the server itself is not trustworthy. The transmitted data is not secure anyway in this case because the server could send it to third parties completely unencrypted.

But in connection with client certificates insecure parameters can be a problem. Some days ago a research team found some possibilities for attacks against the TLS protocol. In these attacks a malicious server could pretend to another server that it has the certificate of a user connecting to the malicious server. The authors of this so-called Triple Handshake attack mention one variant that uses insecure Diffie Hellman parameters. Client certificates are rarely used, so in most scenarios this does not matter. The authors suggest that TLS could use standardized parameters for a Diffie Hellman key exchange. Then a server could check quickly if the parameters are known - and would be sure that they are real primes. Future research may show if insecure parameters matter in other scenarios.

The crash problems in Chromium show that in the past software wasn't very well tested with nonsense parameters in cryptographic protocols. Similar tests for other protocols could reveal further problems.

The mentioned tests for browsers are available at the URL https://dh.tlsfun.de/.

This text is mostly a translation of a German article I wrote for the online magazine Golem.de.

I finished my Asia trip a few days ago. As you could obviously see, my motivation for blogging decreased during the trip. I'll finish with a quick summary of what I did.

I entered Laos from China, but I didn't spend a lot of time in Laos. The main reason was that I was quite frustrated with the weather. It was a comparatively cold winter in southern China and Laos and the buildings there are not really isolated at all and heating usually doesn't exist. While the days were all sunny and nice, the nights were sometimes quite tough. In Laos, usually the only mode of transport are buses and minibuses. I crossed the border at Houay Xai and quickly moved on to Bangkok by bus and train.

Travelling in Laos and Thailand was quite a different experience when compared to Kazakhstan and China. For the first half of my trip, I mostly felt like "the stranger going to places rarely visited by strangers". In China, even at touristy places there were mostly domestic tourists. Laos and Thailand are flooded with western tourists, so I was more like "the western guy going to places everyone else is going". Honestly, I felt much more comfortable with the first role. Malaysia was somewhat in-between. The most important thing I was looking for in this part of Asia was mostly nature and rainforests.

For the whole travelling with buses and trains, I was surprised that it was often much easier than expected. Maybe irrational, but when I planned this I often felt "travelling with a bus/train in a place I barely know anything about just must me difficult". If you have any questions about overland travelling in any of the countries I visited, feel free to ask me. But basically, it usually comes down to "write your preferred train, time, class and destination on a piece of paper and the people at the ticket counter will understand even without knowing your language". The only real obstacle I faced at all was that in China there are some train routes that are booked out early.

If you know me, you know that I try to avoid flying. But it was clear that doing this trip without would be close to impossible. So I flew back from Kuala Lumpur in Malaysia earlier this week.

While I've seen a lot and experienced a lot, at the end I was at a point where I really didn't want to continue any more. I have a lot of respect and get inspiration from people people who consider themselves digital nomads, permanent travelers or something alike and I though a lot about that during travelling and in the months before. I'll probably write some more about that at a later point, as I find it quite desirable to organize life in a way to be less dependent on a fixed living spot. But for me, this has limits and I know where they are.

After crossing Yunnan province, I finally left China to go further south into Laos. I travelled by bus from Jinghong to Medan and from there to Mohan, which is the chinese border town. From there, I just walked across the border. In Laos, EU-citizens can get a visa-on-arrival, which was easy. I just had to fill in a form and got my visa about 10 minutes later.

Right behind the border in Laos is the small town Boten. There's a quite interesting story behind this place. A couple of years ago, this was a place of casinos for chinese gamblers. In China itself casinos are forbidden, so this gained quite some popularity. A couple of luxury Hotels and other facilities for the chinese gamblers emerged. However, some conflicts were arising. The casinos sometimes held chinese gamblers unable to pay their gambling debt as hostages. Chinese authorities were unhappy with this and pressured the laotian authorities to shut the place down, which finally happened in 2011.

So today Boten is mostly a ghost town of abandoned hotels and casinos. The town is split. On one side of the street is what probably was Boten before the casino boom and were most of today's inhabitants live. Some simple houses, some shops and also a small and simple hotel where I stayed for one night. Although this is in Laos, everything is still very chinese. Everyone pays and sells in chinese Yuan, which came quite handy for me, as I didn't have any laotian Kip at that time.

On the other side of the street is the ghost town. It's not completely empty. One large hotel is still welcoming customers, some shops are still operating and also a couple of people live there. But there are a number of former luxury buildings in different states of decay. Probably the most weird thing was a former luxury hotel that's now used as a cowshed.

It was definitely one of the more interesting places on my trip.

Pictures from Boten

It made the news last year that in China, a copy of the austrian historic town Hallstatt was built. Not far from Guangzhou, and as I was there anyway on my way to Hongkong, I thought that's interesting and I'd pay it a visit.

It was a bit tricky to get there. I found only limited information online. The place is near Luoyang, which is part of an area called Boluo, near the town Huizhou. Getting to Huizhou was mostly hassle-free, there are many trains every day from Guangzhou and it takes about two hours. How to come to Luoyang/Boluo is another story. I had two bus numbers I found in some Internet forums, but I couldn't find them at the train station. So either the information was wrong or the buses with that numbers are departing from another place in Huizhou. I also had some GPS coordinates I found online of the copycat Hallstatt, but it turned out later that they were not very accurate.

Checking the map, the place I wanted to visit was only about 10 kilometers away, so I took a taxi. About 15 minutes later and 100 Yuan poorer I arrived in Luoyang. It was already late in the evening, so I decided to take one of the few hotels there. The next day, equipped with the GPS coordinates I went to the austrian village in China. Starting from Luoyang, there's a street with a stadium that's mostly still in construction. I found out that the coordinates I had were about two kilometers off. But after asking some workers, I finally arrived. (correct coordinates for the entrance are lat 23.18112, lon 114.32656, I'll also add something to Openstreetmap)

What is interesting to know is that this place isn't primarily a tourist destination. It's a housing development, people are supposed to live here. However, I think nobody lives here yet, because except for the town's center, everything is still in construction. There are a number of similar projects in China. Tianducheng in eastern China, which rebuilts parts of Paris, has gotten quite a lot of media attention lately as an example for the Chinese housing bubble. Almost nobody lives there and it's mostly a ghost town. As the chinese Hallstatt isn't yet ready to house anyone, the future will tell if it'll fall to the same fate.

There were a number of things that made this place weird. The most obvious thing: You usually don't see tropical plants in an austrian town. I have no idea why they had a (fake/nonworking) red, british-style phone booth. Maybe for a chinese Britain isn't that far from Austria after all. Although this place isn't primarily meant as a tourist destination, it certainly attracts a lot of them. It was quite crowded. I haven't seen anyone who looked like being from Austria though.

After staying here for a while and having a coffee, I went back to Luoyang. There is a bus station in front of the entrance (line 6, but I don't know where it's heading), but no bus came for a while, so I walked back. Grabbing my luggage from my Hotel, I wanted to take a taxi again, but luckily, when I arrived at the main street a bus with Huizhou written on it just stopped in front of me (to be precies, it had 惠州 written on it, but I manage to recognize chinese characters well enough to remember the towns where I've been lately). So my way back was much cheaper and the bus - without a line number - brought me directly to the train station where I headed back to Guangzhou.

I'd like to finish with a quick comment. When reading western news commenting about this case and others, they're often quick in condemning this kind of "copycat" culture. I think one needs to be careful with that. The importance of the "original" is something that is deeply rooted in cultural norms and traditions. Sticking to the original is something that's probably more important in European cultures than in China. But I don't think either of them is right or wrong, it's just a different approach to culture. After all, it doesn't hurt anybody if someone is rebuilding Austrian villages in China. Also, there is almost certainly no legal issue at stake here. While it may be debatable if a town's layout can be covered by copyright, Hallstatt is a historic city. So if there is any copyright on it, it's already expired.

Pictures from Luoyang
Pictures from Chinese Hallstatt

Some Links:
Blog entry about Hallstatt / China on thechinachronicle.com
Blog entry about Hallstatt / China on liongrass.hk

The well-known way of getting from Europe to China overland is the transsiberian railway. However, I noted that the route through Kazakhstan I took is the quickest way to get to China by train and bus. I thought I'd write that up:

• Take train EC 43 from Berlin Hauptbahnhof (09:37 on Monday, Berlin time) to Warsaw Wschodnia, change to train D 10SZ (15:28). If you - like me - feel that the time to switch trains is a bit risky in case of delaqys, you can take the earlier Beriln-Warsaw-train EC 41 (06:37 until 12:08). There's also a direct Berlin-Moscow-train D 50472 (Berlin 04:28, Moskwa Belorusskaja 10:33), but it's often sold out early.
• Spend some time in Moscow and then take the Metro Line 5 (Circular Line) to Komsol'skaya (Metro of Kazasky railway station)
• Take train 090У from Moscow Kazansky (18:48 on Tuesday, Moscow time) to Petropavl/Petropavlovsk in Kazakhstan (09:46 on Thursday, Astana/Almaty time). Note that this only works every second Tuesday - you may choose other days where this train goes, but then other options may not work.
• Take train 152T from Petropavl (13:48 on Thursday, Astana/Almaty time) to Almaty (22:28 on Friday, Astana/Almaty time).
• Take bus number 100 to Sayran bus station and hope that they'll sell you a ticket late in the evening for the bus next morning. Find a place to sleep (but not very long).
• Take bus from Astana Sayran bus station (07:00 on Friday, Astana/Almaty time) to Yining (approx. 21:00 on Friday, Beijing time). You're in China.

With the transsiberian, you can leave Berlin on Monday (same options as above until Moscow) and take the D4ZJ direct train from Moscow to Beijing. You will enter China in Erlian on the next Monday at 00:47. So this makes almost 7 days vs. about 4 and a half days.

I wouldn't recommend anyone doing that. Better spend some time on the way and see some places in Russia or Kazakhstan. Also it should be noted that one obvious reason for being faster is that you'll enter China at a place much further in the west. And getting to the main part of china (the western part is much less inhabited than the eastern part and all big cities are in the east) can be somewhat troublesome. Still, I thought it might be of interest to document the fastest overland way from Europe to China.

I always assumed the starting point Berlin, obviously because I live there, adapting that to other starting places should be trivial. For example you can usually easily (and for a comparatively cheap price) reach Berlin by Eurolines bus in a day from other major european cities like Paris or London.

After another train ride, I arrived at Xi'an. This was releaving for me, because as I already wrote, I originally wanted to go directly from Urumqi to Xi'an, which wasn't possible. I finally passed that part now. The longer journey gave me the experience of now knowing various transport modes and train classes common in China.

Between Dunhuang and Lanzhou, I took what is called the "hard seat" class for a 14 hour ride (not overnight). I can quote Wikivoyage on that:
"Traveling in a seat (hard or soft-class) means you will share the car space with lots of locals. You will most likely encounter smokers, loud noise, and constant activity in the aisle while you try to sleep. *Do not* travel hard class if you are uncomfortable with these settings."

While I certainly was an interesting experience, it is not exactly one I'd like to repeat. It was challenging and I was quite happy when I finally arrived.

With my arrival in Xi'an, finally I noticed that I approach the warmer zones of China. While not really "summer-warm", I apprechiated not having to wear winter clothes all the time. Appart from that, Xi'an was quite different from the other cities I've visited before. It is in many small ways much more like a western city (and, to be not mistaken, in many ways this is a good thing - better hygiene, less dangerous traffic, no smoking in non-smoking zones). And regarding my last blog post, yes, coffee is usually available, although often expensive.

Xi'an is also home of one of Chinas most popular tourist attractions, the so-called terracotta warriors and - not that well known - the Chinese pyramids. I didn't know that there are pyramids in China, so found that worth seeing. Unlike pyramids in other places of the world, the chinese pyramids are not buildings, they are artificial hills. The biggest one, near the terracotta warriors, is the mausoleum of Chinas first emperor Qin Shi Huang.

My initial plan was to visit the pyramid and then decide if I still had time and motivation to see the terracotta warriors. I took the bus to the terracotta warriors and walked the roughly two kilometers to the pyramid. Turned out my planning was not really how one was supposed to do things. Going to the pyramid is only possible with an entrance ticket for the terracotta warrior museum - and you cannot buy it at the entrance of the pyramid area. There's a free shuttle service which I then took to get the ticket and drive back.

The pyramid doesn't look that spectacular and there's hardly a spot where you can actually see it's a pyramid. It once was much bigger, but during the centuries, the earth got compressed and it became smaller. I walked around a lot, the area around is a nice park.

As it is common amongst tourist destinations probably everywhere in the world, there is a huge amount of people who want to sell you things - from terracotta warrior replica in all sizes (including ones in original size and made mostly out of the same materials than the original ones) to the various tour guides. I refused all these offers and preferred to find my own way. One more thing notable: This isn't mainly a tourist spot for foreign tourist. The vast majority were domestic tourists.

As for accomodation, I stayed in the so-called "capsule hostel" in Xi'an. I read about these capsule hotels before (they're especially common in Japan), so I was curious. It is kind of a compromise between a dorm bed and a real hotel room. You get a small private capsule made out of plastic that contains your bed. They even have a television in every capsule (I fail to understand why no matter where in the world you are, a television in every hotel room seems to be standard). It was quite okay, although I would have one suggestion for an improvement: There capsule can not be locked, so you have to put your valuable private stuff into a separate locker every time you leave.

While writing this, I'm in the highspeed train from Xi'an to Guangzhou (taking only 8 hours). I've been to Guangzhou before, and I'm not going there not mainly because I want to see something there. I have a double-entry 2 x 30 days visa and from Guangzhou, I intend to make a quick hop to Hong Kong. Although part of China, in visa issues Hong Kong is like going to another country. So by going to Hong Kong, I have another 30 days to spend in China. Arriving in Guangzhou, I can also finally leave any traces of the northern hemisphere winter behind myself. It has comfortable temperatures all year round.

Pictures from Xian
Pictures from Guangzhou

I like drinking coffee and I usually drink quite a lot of it. And I'm pretty used to the fact that usually, you can get coffee just about everywhere. Well, at least so I thought.

When travelling, I usually have my own coffee making equipment, which consists of a couple of coffee filters, a collapsable filter holder, some coffee powder and some coffee creamer powder (which doesn't make a very delicious coffee and I prefer real milk or soy milk if possible, but it's a reasonable compromise when travelling).

I was running out of coffee powder shortly after I entered China. I really hadn't expected that it would be a problem. After all, the biggest international coffee chains, Starbucks and Costa Coffee, are flooding eastern and southern China with their restaurants and Chinese chains and independent coffee houses are trying to keep up with that. Well, the problem is that I entered China in the far northwest. And the Chinese love for coffee hasn't made it there yet.

I could hardly believe this, but between Yining and Dunhuang, I didn't find any coffee. None at all. Not in the form of coffee powder and not in brewed form. Yeah, supermarkets have a large collection of soluble instant coffee, most of it Nescafe and also some Chinese brands. But I refuse to call that stuff coffee. Most of it contains more sugar than anything else. When you get coffee in restaurants at all, you can bet it's also instant coffee. I checked where the next Starbucks is: In Xi'an, thousands of kilometers away. I checked for the next Costa Coffee: Also in Xi'an. (I'm on my way to Xi'an now - no, not just because of the coffee, it was my plan anyway.)

After all, I bought some non-sugared instant coffee. No, I wasn't happy with it, but it seemed it was the best I could get.

Lesson learned: There's just no coffee in northwest China.