Update: I got some nice hints in the comments. cpufreqd also includes this functionality and is probably the much more advanced solution. Also, I got a hint to linux-PHC, which allows undervolting a CPU and thus also saves energy.
I recently quite often had the problem that my system suddenly was shutting down. The reason was that when my processor got beyond 100 °C, my kernel decided that it's better to do so. I don't really know what caused this, but anyway, I needed a solution.
So i hacked together overheatd. A very effective way of cooling down a CPU is reducing its speed / frequency. Pretty much any modern CPU can do that and on Linux this can be controlled via the cpufreq interface. I wrote a little daemon that simply checks every 5 seconds (adjustable) if the temperature is over a certain treshold (90 °C default, also adjustable) and if yes, it sets cpufreq to the powersave governor (which means lowest speed possible). When the temperature is below or at 90 °C again, it's set back to the (default) ondemand governor. It also works for more than one CPU (I have a dual core), though it's very likely that it has bugs as soon as one goes beyond 10 CPUs - but I have no way to test this. Feel free to report bugs.
This could be made more sophisticated (not going to the lowest frequency but step by step to lower frequencies), but it does its job quite well for now. It might be a good idea to support something like this directly in the kernel (I wonder why that isn't the case already - it's pretty obvious), but that would probably involve a skilled kernel-hacker.