walkah: security

There have been reports that Harvard recently had a Joomla! based website compromised, and the database contents have been made available via BitTorrent. Of interest - the compromise was apparently via the usage of an insecure password. From the Torrent Freak article:

A file included with the release labeled password.txt carries a message:

Thomas gatton….stupid people, you don’t use a secure password

While it's not entirely clear whether it was an insecure system password or an insecure Joomla! password used - it does highlight an important aspect of security.

Ensuring that you write secure code is only (a small) part of the security problem. With our recent Drupal 6.0 release, we have tried to incorporate several changes to help our users be more secure:

• Password strength checker: when selecting a password now in Drupal, users are advised when their passwords are "weak". Encouraging tougher to crack/guess passwords particularly for admin and privileged users.
• OpenID support: Even a strong (hard to guess / crack) password can be compromised by a clever attacker if you consistently log in without SSL (i.e. when you're at that internet cafe). Also, remembering several (hundreds!) of complicated, strong passwords can be daunting and frequently leads to poor password choices. By including OpenID authentication support, Drupal users and administrators no longer have to remember passwords to every site they administer. They can use their OpenID - which in turn can implement stronger authentication methods to limit potential vulnerabilities. Development Seed has a great article on how they use OpenID to avoid sharing passwords for admin accounts.
• Update module: One of the biggest security challenges is keeping you site up to date. Drupal sites tend to be a combination of Drupal core and several (10 - 50) contributed modules - keeping them all up to date is a complicated task. It's also a crucial security precaution.

The point being: writing secure code is one thing, but there is a much trickier, critical task in educating users and administrators. It's something we're working towards within the Drupal Security Team and within the community in general. We're not done yet, and welcome your feedback and suggestions!

So, faithful readers, as you may have noticed if you tried to visit this site - I got "hacked". All of the sites hosted here had their index.php files replaced with a defacement message reminding us, amongst other things, that "Hack is not a crime". Since lots of folks have asked - specifically if it was drupal related - and since the information might be generally useful for the internets, I've decided to post a brief recap here.

First off: in a move I'd never seen before, the guys removed *all* log files from the system which makes figuring out exactly what happened pretty darned tricky. In fact, they had removed anything with 'log' in the name - things like logwatch and logrotate binaries were also removed. So, what I say here is largely speculation based on the few remaining traces I found left behind. As such, here is my theory (perhaps the cracking team will be nice enough to post comments with clarifications ;):

The only real clue how they got in was the following in /tmp:

drwxr-xr-x 3 www-data www-data 4.0K Apr 28 16:24 .sc/

That might not mean a whole lot - except that the datestamp is right and the directory is owned by www-data : the uid of my apache process. This strongly suggests a web-based exploit. Since pretty much all that runs on my server is Drupal it looks like we've got a problem...

I had an old Drupal 4.4 site still running on this server (sudden-thoughts.com - down until further notice). Drupal 4.4 is susceptible to the (fairly) infamous XML-RPC hole. I had manually removed xmlrpc.php from this site, however, it was back - it looks like in a brain cramp moment by yours truly I had cvs up'ed the directory which brought the file back (as part of my routine drupal site maintenance on my server).

Now, that part is speculation, but a likely guess. Without my apache logs I can't even accurately guess which site was targetted... but 'statistics' module is enabled on most sites and showed nothing suspicious in accesslog or watchdog... thus suggesting xmlrpc as a likely candidate. All other sites run Drupal 4.7 or 5 - with all security updates applied. So, if it wasn't that site, then there is a new remote-execution vulnerability in drupal that we've not yet heard about on the Drupal security team. Possible, but unlikely given that the other glaring hole was available.

Remote code execution is a serious problem, but usually containable from a web application if you run apache as a non-privileged user (as i do). So, how then were they able to overwrite all the index.php files?

My server runs Debian Etch (the latest release) with all security updates applied. So, again, unless they were using 2 un-resolved exploits - it leaves one likely option : the kernel. I was running an old, custom compiled 2.4 kernel - I mean *really* old, from 2003. Said kernel is vulnerable to things like a ptrace exploit for privilege escalation. My theory is that something like this was used to gain root access. From there, they were able to overwrite all index.php files. They also installed the "shv5" rootkit - which modifies a bunch of system binaries (ls, ifconfig, nestat, etc) - detected by both rkhunter and chkrootkit. More information on that available via google.

So, that's my theory and I'm sticking to it. So, is Drupal insecure? No. Not if you're good about running recent, maintained versions and keeping an eye on security announcements. The problem here was more accurately due to lackadaisical administration on my part - both with the drupal version and the stale kernel. I know better ... chalk it up as a "shoemaker's son" scenario.

Thanks to Steven who was the first to notify me (via SMS) that something was up. Of course, other speculations are welcome in the comments :)

I noticed via Scott Kveton's blog this morning that myOpenID got a bit of a refresh this morning. Along with a (very pleasant) new visual re-design, they've officially launched a feature which, imo, is worth noting. Specifically, myOpenID now supports certificate-based authentication:

Phishing is always on the minds of members of the OpenID community and we’re excited to announce the release of our client-side certificate functionality. Client-side certificates leverage the tried-and-true technology known as transport layer security (also known as TLS). This essentially the same technology you probably know of as SSL that is used to secure millions of transactions on the web every single day. With the click of a mouse you can configure your own unique certificate right in your web browser (yep, Firefox, IE, Safari and even Opera). This then gives you the ability to authenticate quickly and very securely from your machine without the use of a password.

As I mentioned in my presentation last month, this is an important aspect of OpenID - namely the choice, and option for stronger, more secure methods of authentication and making that choice user-centric.

Kudos to JanRain - keep up the great work!