Hanno's blog

Update: When I wrote this blog post it was an open question for me whether using Address Sanitizer in production is a good idea. A recent analysis posted on the oss-security mailing list explains in detail why using Asan in its current form is almost certainly not a good idea. Having any suid binary built with Asan enables a local root exploit - and there are various other issues. Therefore using Gentoo with Address Sanitizer is only recommended for developing and debugging purposes.

Address Sanitizer is a remarkable feature that is part of the gcc and clang compilers. It can be used to find many typical C bugs - invalid memory reads and writes, use after free errors etc. - while running applications. It has found countless bugs in many software packages. I'm often surprised that many people in the free software community seem to be unaware of this powerful tool.

Address Sanitizer is mainly intended to be a debugging tool. It is usually used to test single applications, often in combination with fuzzing. But as Address Sanitizer can prevent many typical C security bugs - why not use it in production? It doesn't come for free. Address Sanitizer takes significantly more memory and slows down applications by 50 - 100 %. But for some security sensitive applications this may be a reasonable trade-off. The Tor project is already experimenting with this with its Hardened Tor Browser.

One project I've been working on in the past months is to allow a Gentoo system to be compiled with Address Sanitizer. Today I'm publishing this and want to allow others to test it. I have created a page in the Gentoo Wiki that should become the central documentation hub for this project. I published an overlay with several fixes and quirks on Github.

I see this work as part of my Fuzzing Project. (I'm posting it here because the Gentoo category of my personal blog gets indexed by Planet Gentoo.)

I am not sure if using Gentoo with Address Sanitizer is reasonable for a production system. One thing that makes me uneasy in suggesting this for high security requirements is that it's currently incompatible with Grsecurity. But just creating this project already caused me to find a whole number of bugs in several applications. Some notable examples include Coreutils/shred, Bash ([2], [3]), man-db, Pidgin-OTR, Courier, Syslog-NG, Screen, Claws-Mail ([2], [3]), ProFTPD ([2], [3]) ICU, TCL ([2]), Dovecot. I think it was worth the effort.

I will present this work in a talk at FOSDEM in Brussels this Saturday, 14:00, in the Security Devroom.

While I got along well with my Thinkpad T61 laptop, for quite some time I had the plan to get a new one soon. It wasn't an easy decision and I looked in detail at the models available in recent months. I finally decided to buy one of Lenovo's Thinkpad X1 Carbon laptops in its 2014 edition. The X1 Carbon was introduced in 2012, however a completely new variant which is very different from the first one was released early 2014. To distinguish it from other models it is the 20A7 model.

Judging from the first days of use I think I made the right decision. I hadn't seen the device before I bought it because it seems rarely shops keep this device in stock. I assume this is due to the relatively high price.

I was a bit worried because Lenovo made some unusual decisions for the keyboard, however having used it for a few days I don't feel that it has any severe downsides. The most unusual thing about it is that it doesn't have normal F1-F12 keys, instead it has what Lenovo calls an adaptive keyboard: A touch sensitive line which can display different kinds of keys. The idea is that different applications can have their own set of special keys there. However, just letting them display the normal F-keys works well and not having "real" keys there doesn't feel like a big disadvantage. Beside that Lenovo removed the Caps lock and placed Pos1/End there, which is a bit unusual but also nothing I worried about. I also hadn't seen any pictures of the German keyboard before I bought the device. The ^/°-key is not where it's used to be (small downside), but the </>/| key is where it belongs(big plus, many laptop vendors get that wrong).

Good things:
* Lightweight, Ultrabook, no unnecessary stuff like CD/DVD drive
* High resolution (2560x1440)
* Hardware is up-to-date (Haswell chipset)

Downsides:
* Due to ultrabook / integrated design easy changing battery, ram or HD
* No SD card reader
* Have some trouble getting used to the touchpad (however there are lots of possibilities to configure it, I assume by playing with it that'll get better)

It used to be the case that people wrote docs how to get all the hardware in a laptop running on Linux which I did my previous laptops. These days this usually boils down to "run a recent Linux distribution with the latest kernels and xorg packages and most things will be fine". However I thought having a central place where I collect relevant information would be nice so I created one again. As usual I'm running Gentoo Linux.

For people who plan to run Linux without a dual boot it may be worth mentioning that there seem to be troublesome errors in earlier versions of the BIOS and the SSD firmware. You may want to update them before removing Windows. On my device they were already up-to-date.

Yesterday the LibreSSL project released the first portable version that works on Linux. LibreSSL is a fork of OpenSSL and was created by the OpenBSD team in the aftermath of the Heartbleed bug.

Yesterday and today I played around with it on Gentoo Linux. I was able to replace my system's OpenSSL completely with LibreSSL and with few exceptions was able to successfully rebuild all packages using OpenSSL.

After getting this running on my own system I installed it on a test server. The Webpage tlsfun.de runs on that server. The functionality changes are limited, the only thing visible from the outside is the support for the experimental, not yet standardized ChaCha20-Poly1305 cipher suites, which is a nice thing.

A warning ahead: This is experimental, in no way stable or supported and if you try any of this you do it at your own risk. Please report any bugs you have with my overlay to me or leave a comment and don't disturb anyone else (from Gentoo or LibreSSL) with it. If you want to try it, you can get a portage overlay in a subversion repository. You can check it out with this command:
svn co https://svn.hboeck.de/libressl-overlay/
git clone https://github.com/gentoo/libressl.git

This is what I had to do to get things running:

LibreSSL itself

First of all the Gentoo tree contains a lot of packages that directly depend on openssl, so I couldn't just replace that. The correct solution to handle such issues would be to create a virtual package and change all packages depending directly on openssl to depend on the virtual. This is already discussed in the appropriate Gentoo bug, but this would mean patching hundreds of packages so I skipped it and worked around it by leaving a fake openssl package in place that itself depends on libressl.

LibreSSL deprecates some APIs from OpenSSL. The first thing that stopped me was that various programs use the functions RAND_egd() and RAND_egd_bytes(). I didn't know until yesterday what egd is. It stands for Entropy Gathering Daemon and is a tool written in perl meant to replace the functionality of /dev/(u)random on non-Linux-systems. The LibreSSL-developers consider it insecure and after having read what it is I have to agree. However, the removal of those functions causes many packages not to build, upon them wget, python and ruby. My workaround was to add two dummy functions that just return -1, which is the error code if the Entropy Gathering Daemon is not available. So the API still behaves like expected. I also posted the patch upstream, but the LibreSSL devs don't like it. So on the long term it's probably better to fix applications to stop trying to use egd, but for now these dummy functions make it easier for me to build my system.

The second issue popping up was that the libcrypto.so from libressl contains an undefined main() function symbol which causes linking problems with a couple of applications (subversion, xorg-server, hexchat). According to upstream this undefined symbol is intended and most likely these are bugs in the applications having linking problems. However, for now it was easier for me to patch the symbol out instead of fixing all the apps. Like the egd issue on the long term fixing the applications is better.

The third issue was that LibreSSL doesn't ship pkg-config (.pc) files, some apps use them to get the correct compilation flags. I grabbed the ones from openssl and adjusted them accordingly.

OpenSSH

This was the most interesting issue from all of them.

To understand this you have to understand how both LibreSSL and OpenSSH are developed. They are both from OpenBSD and they use some functions that are only available there. To allow them to be built on other systems they release portable versions which ship the missing OpenBSD-only-functions. One of them is arc4random().

Both LibreSSL and OpenSSH ship their compatibility version of arc4random(). The one from OpenSSH calls RAND_bytes(), which is a function from OpenSSL. The RAND_bytes() function from LibreSSL however calls arc4random(). Due to the linking order OpenSSH uses its own arc4random(). So what we have here is a nice recursion. arc4random() and RAND_bytes() try to call each other. The result is a segfault.

I fixed it by using the LibreSSL arc4random.c file for OpenSSH. I had to copy another function called arc4random_stir() from OpenSSH's arc4random.c and the header file thread_private.h. Surprisingly, this seems to work flawlessly.

Net-SSLeay

This package contains the perl bindings for openssl. The problem is a check for the openssl version string that expected the name OpenSSL and a version number with three numbers and a letter (like 1.0.1h). LibreSSL prints the version 2.0. I just hardcoded the OpenSSL version numer, which is not a real fix, but it works for now.

SpamAssassin

SpamAssassin's code for spamc requires SSLv2 functions to be available. SSLv2 is heavily insecure and should not be used at all and therefore the LibreSSL devs have removed all SSLv2 function calls. Luckily, Debian had a patch to remove SSLv2 that I could use.

libesmtp / gwenhywfar

Some DES-related functions (DES is the old Data Encryption Standard) in OpenSSL are available in two forms: With uppercase DES_ and with lowercase des_. I can only guess that the des_ variants are for backwards compatibliity with some very old versions of OpenSSL. According to the docs the DES_ variants should be used. LibreSSL has removed the des_ variants.

For gwenhywfar I wrote a small patch and sent it upstream. For libesmtp all the code was in ntlm. After reading that ntlm is an ancient, proprietary Microsoft authentication protocol I decided that I don't need that anyway so I just added --disable-ntlm to the ebuild.

Dovecot

In Dovecot two issues popped up. LibreSSL removed the SSL Compression functionality (which is good, because since the CRIME attack we know it's not secure). Dovecot's configure script checks for it, but the check doesn't work. It checks for a function that LibreSSL keeps as a stub. For now I just disabled the check in the configure script. The solution is probably to remove all remaining stub functions. The configure script could probably also be changed to work in any case.

The second issue was that the Dovecot code has some #ifdef clauses that check the openssl version number for the ECDH auto functionality that has been added in OpenSSL 1.0.2 beta versions. As the LibreSSL version number 2.0 is higher than 1.0.2 it thinks it is newer and tries to enable it, but the code is not present in LibreSSL. I changed the #ifdefs to check for the actual functionality by checking a constant defined by the ECDH auto code.

Apache httpd

The Apache http compilation complained about a missing ENGINE_CTRL_CHIL_SET_FORKCHECK. I have no idea what it does, but I found a patch to fix the issue, so I didn't investigate it further.

Further reading:
Someone else tried to get things running on Sabotage Linux.

Update: I've abandoned my own libressl overlay, a LibreSSL overlay by various Gentoo developers is now maintained at GitHub.

I recently noted that I have never blogged about this nice little device I now own for a couple of years. I originally bought the Toshiba AC100 before a two-month-long trip through Russia and China.

I was looking for a possibility to have a basic laptop, but without much weight. The AC100 is an ARM-based laptop which originally ships with the Android operating system. It weights less than 800 gramms and thus is lighter than the usual subnotebooks. According to my knowledge, it's not produced any more, but it can still be bought on ebay.

The nice thing is: You can install Linux on it and thus it will give you the possibility to run an almost full desktop-system. Though a warning ahead: While basic things work, it is quite a hacky business and you should expect to see problems. If you aren't prepared to solve them, this is probably not the solution for you.

Originally I was running Gentoo Linux on it (and it did well on my two-month trip), but now I'm running Ubuntu. The reason is that it was just too hard to get anything fixed if it didn't work. I rarely could find help anywhere, I assume there are only a handful of people that ever tried installing Gentoo on this Device. Ubuntu up until version 12.10 has reasonable support.

The great thing is: This is probably one of the lightest solutions to have a desktop/laptop-like machine with a real keyboard. Perfect for travelling. As it's running Linux, you can have access to a large number of standard applications. With lightweight apps like Abiword or Claws-Mail you can use basic applications.

The limitations are the Browser and Video. You can run Chromium or Firefox, but the device clearly shows its limits. Expect to wait longer sometimes, don't open too many tabs - and I always have to remember to never try to open Chromium and Firefox at the same time, as this makes the system mostly unusable. Obviously there's no Flash and nothing else that's only available in binary form, because ARM Linux is such a niche OS that nobody will provide binary apps for it.

Videos work, but limited. There's no xv support in the free driver. That means if you want to upscale a video to fullscreen, this has to be done in software and that usually means you cannot play videos fast enough. There's a binary graphics driver by Nvidia (the internals of the device are based on the Nvidia Tegra chipset), but I haven't had much success with it.

Tobias Scherbaum already blogged this, but only in german, so I'm writing this again for the Planet Gentoo readers.

A german webpage called jugendschutzprogramm.de provides filters for webpages potentially dangerous for children. Now some people noticed that this page considers quite a lot dangerous.

Both gentoo.de and gentoo.org are considered only suitable for people over 14. So if you ever thought about installing Gentoo on the PC of a kid, think again what you might do to that kid.

Beside, my blog is even more dangerous: It's blocked by default.

The page is supported by a couple of companies providing pornographic content. Interesting enough, it's also supported by a big german Newspaper (BILD) that regularly has pornographic images on their frontpage. However, their page is considered harmless.

But what's really frightening is that jugendschutzprogramm.de is part of ICRA, an international system by big content and internet providers. It's even supported by the european union.

Update: Page has XSS, maybe someone wants to play with it?

<form action="http://jugendschutzprogramm.de/webmaster/label-generator.php" method="post">
<input name="URL" value='"><script>alert(1)</script>' type="text">
<input name="submit" type="submit">
</form>

Today I asked myself if I can ping an IDN host.

My default ping (iputils on linux) couldn't do it, but I found some patches out there, e.g. from Fedora. Thanks to SpanKY, we now also have IDN-enabled ping in Gentoo (he used a modified patch).

After some more »out of memory«-messages by josm, I thought it's time to look out for alternatives.

For the openstreetmap-project, the two main editors are josm (java) and potlach (flash). I think using java probably wasn't a very wise decision (I still wonder how an app can get »out of memory« after loading about 5 MB of images, do they create a pixel class and store every pixel in an object?) and I don't like flash either.

There's another project called merkaartor and today I had a look. My first feeling is that it's promising. It has good performance, it does nice live-rendering and it supports the basic features (adding nodes and ways, up/downloading stuff to the osm-system).

Sure, comparing to the large list of plugin features josm has, it's limited. Maybe I'll try to hack in some bits I'm missing at the moment.

In my continuing effort to improve gentoo for geo-related stuff, I've just added a merkaartor package to portage.

I know you've been waiting far too long for that. Now that Compiz and Compiz Fusion 0.6 are out, I've added them to portage.

The background: Compiz and Beryl, the two famous 3D-composite/windowmanagers for Linux, have merged forces. Main Compiz still resides in the package x11-wm/compiz, many additional plugins and tools are fetched in by the x11-wm/compiz-fusion metapackage.

The ebuilds are all based on the xeffects overlay, with some cleanup by me.

Happy window-wobbling!

I recently added some stuff to gentoo for openstreetmap and gps-related work.

For one, the java openstreetmap editor josm now has ebuilds. josm and josm-plugins, the first only installs the program itself plus language packs, the second installs most of the josm-plugins available. They can be enabled within the configuration.
I was a bit unsure how to handle it, as I first thought about adding some basic plugins to the josm-package itself. But as the opinions on what »basic« plugins are seem to differ a lot, I decided to do it this way.
Another new package is gebabbel, a gui-frontend for gpsbabel. gpsbabel is a commandline-tool that implements various proprietary gps coordinate formats and allows access to many gps-devices (e. g. garmin). Beside it can be used to filter and convert gps-tracks.

More to come. Probably also interesting stuff in portage is gpsdrive (which has some osm-stuff in svn, but not yet in a release), marble (world-view-tool for kde, osm-support is planned within the next months). Other stuff not yet in portage, I plan to make packages in the future: tiles@home, qlandkarte, mapnik and probably everything it takes to run an osm-server.

A bit offtopic, as gentoo doesn't run on mobiles (yet): Mobile Trail Explorer is the only free (as in freedom) software I found for J2ME-mobiles to create gps-tracks. It's a bit alpha, lacking some features and unstable, but it's free, so I hope it'll become better soon. It's a cheap way to get gps-tracks, assuming that you already have a java/bluetooth-mobile and you can get a gps-mouse starting at about 50 €.

If you have more suggestions for gps/osm-related stuff, feel free to open requests on the gentoo bugzilla and add me to the cc.

I'm happy to announce that I mentored Christian Hoffmann to become a new Gentoo Developer.

Christian did some PHP-security work for Gentoo recently, which is very important due to the high amount of security issues php had recently. Welcome on board and continue your good work.

I've just committed some compiz-related updates to Gentoo. First we now have version 0.3.6, the most interesting news is probably that it now has a working kde-window-decorator. gnome/kde-stuff is now only enabled on use-flags, so if you wanna continue to use gconf, you'll have to build compiz with the gnome-flag.
compiz-start tries to autodetect a running kde and then run the kde-window-decorator. If compiz-start fails for you, please report it, because I plan to deprecate all the compiz-aiglx/xgl/nvidia-scripts.

Beside that we now have compiz-settings in the tree, which is a simple configuration-tool for compiz and saves you from using gconf manually.

I just did some large updates to my »fun with x«-overlay after some experiences from the weekend where I installed it on various other people's machines, so I thought it's time to post some up-to-date information.
A few days ago I got a bunch of new patches from Kristian Høgsberg that should be much less hacky than the previous ones. You need to re-compile xorg-server and compiz together to use aiglx with compiz.
The compiz-ebuild has no longer a gnome and kde useflag, because the kde-window-decorator is not working at the moment and it doesn't make much sense to build compiz without any window decorations. Also, compiz now comes with two startscripts (compiz-aiglx and compiz-xgl) that basically just run the decorator and compiz with all default plugins. I noticed that the autodetection hack (whether it's running xgl or aiglx) doesn't really work, so the script also has all neccessary parameters. The patch is still in, but I'd like to have some better solution for that in the future.
In the main dir, I placed a sample package.keywords for people using the stable (no ~arch) tree of gentoo.

I've -*-keyworded the metacity-ebuild (because upstream isn't working at the moment on the libcm/metacity-stuff and compared to compiz it's boring anyway) and the compiz-quinnstorm-ebuild (because I don't work on it currently). You can still use them though if you add them to your package.keywords.

Probably one of the more interesting news: I have now commit-access to coffee's overlay, which means we work together to merge improvements forth and back. For the common question which overlay to take, I could say that mine is more polished, just contains the basic things to run xgl/aiglx and compiz and nothing more and is probably more stable, while coffee's contains more stuff (e. g. the now split up quinnstorm stuff).

Beside that, mesa is going to have a new release within days, which will make things much easier (and probably let us merge some stuff into main portage soon).

To get the fun, just
svn co https://svn.hboeck.de/xgl-overlay/

I've committed some updates to my xgl/aiglx-overlay. First of all, it now uses a git-ebuild for the xserver, because there have been some improvements (implementation of GLX_MESA_copy_sub_buffer) I wasn't able to backport easily. Then I've added an experimental patch to compiz autodetecting AIGLX, which removes the need for indirect and strict-binding parameters. Some other no longer needed patches removed. Get it with:
svn co http://svn.hboeck.de/xgl-overlay

Or, if you already have it, cd to it and
svn update

To use it properly, you'll need some entries in your /etc/portage/package.unmask (I've put a package.unmask.sample in the overlay root dir) for proper operation of the overlay:
x11-base/xorg-server
media-libs/glitz
x11-libs/cairo
dev-python/pycairo
(no longer needed)

Enjoy!

Short note for my english visitors (and planet gentoo readers), I'm here at the Libre Software Meeting (Rencontres Mondiales du Logiciel Libre, RMLL) in Nancy, a french free software event.
There's a gentoo booth with kernelsense and dams.

First set of rmll-pictures

Another update from the X-funky-and-cool-front: With some Patches from Kristian Høgsberg, I got compiz running with aiglx.

For those who don't know: Compiz is a combined window/composite manager that was released together with xgl earlier this year. aiglx is another approach to get 3D-accelerated Desktops on Linux/Xorg.

I've merged the appropriate (very experimental stuff, big fat warning!) gentoo packages into my overlay. For all non-gentooers: You are on your own, but you can grab Kristian's patches here. Update: Kristian told me that Fedora Rawhide users are also lucky.

You can get it with
svn co http://svn.hboeck.de/xgl-overlay
(It's still called xgl-overlay, although it should probably be named »various-funky-x-stuff« or so, but I was too lazy to rename it)

To install it, check it out like above, put the path into your PORTDIR_OVERLAY-var in make.conf. Then re-merge libdrm, mesa, xorg-server and compiz (all ~x86). You can also merge experimental libcm/metacity-packages (metacity with 3d-effects), therefore add xcomposite to your USE-flags and merge metacity-2.15.5.

To start compiz on aiglx (that means »normal« X):
LIBGL_ALWAYS_INDIRECT=1 compiz --replace --strict-binding move resize minimize place decoration wobbly cube rotate scale switcher zoom &
gnome-window-decorator
(replace plugins with anything you want, the LIBGL-var and strict-binding are required for running in aiglx, this should be automatically detected soon)

For metacity use:
USE_WOBBLY=1 metacity --replace

As before, there is still a package for xgl, so you now can try both.