Strong password encryption - what to do if you've been ripped

(and what do to make sure you aren't, if you haven't been)

If you're reading this page, you've probably found that a large number of your customers' passwords have been compromised. Perhaps Strongbox notified you of this problem, or perhaps you don't have Strongbox yet and found out about the problem when you recieved a huge hosting bill. If a great number of your user's passwords have been compromised, a malevolent hacker, known as a cracker, has probably gotten a copy of your password file or database. Though the passwords in such files are normally encrypted, the encryption most commonly used is so weak that it can be cracked in minutes, if not seconds. This is called "ripping" the password file and is possible because the old DES encryption used just can't stand up to modern computers with one or more processor cores each capable of performing billions of calculations per second. Our password generator page has some general information about password file ripping and how it can be effectively prevented by using strong encryption of the passwords in your password file. Upgraded encryption will prevent a cracker from decrypting your password file in the future, but what other possible problems could there be instead and how can we prevent the cracker from getting the password file in the first place? What about the passwords that the cracker already has?

Is it really ripped?

First, let's log in the the admin area of your credit card processor's site, CCbill.com or whoever you use, and see if it looks like the password file was actually ripped or if the compromised passwords were perhaps never legitimate members in the first place. In CCBill, click on "Reports" at the top of the screen and follow the navigation links to get to your user list. Look to see if the users that have been compromised are even listed. If not, the cracker may have added them directly to the password file using a bug in some PHP or CGI script on your server, or even by gaining FTP access. In that case, you'll want to beef up your CGI and PHP security and also change your FTP password, being sure to choose a good, secure password that is not based on an English word. If the users are listed, look for a pattern. Did all of the "compromised" users join in one day, or over a short span of time? If so, these are likely to be fraudulent joins. The cracker may have joined several times using stolen credit card information. Also look at the details for several of the users. If a pattern emerges, such as many of the joins coming from a suspicious country like Russia or Romania, again suspect fraud. These may not be legitimate users who had their passwords ripped. In case of probable fraud, contact your credit card processor regarding what measures to take so that you aren't hit with unexpected fees for chargebacks and the the like, or even worse, have your account locked.

How did the cracker get the password file?

If they appear to be legitimate users, we can proceed to see if we can tell how the cracker got the password file. Essentially the same procedure will be used the users aren't listed at all with the processor, to see how they were added and prevent this in the future. If the passwords were ripped, the cracker probably got the password file by exploiting a security issue with a PHP or CGI script. Any script on the server may have a security hole the attacker might use, but particularly susceptible are scripts such as guestbooks and forums that allow an visitor to add something to one of your pages. The attacker will add PHP or SSI code which causes the server to display the password file rather than a plain message. Also susceptible are scripts that can send email. The attacker will submit a specially crafted form entry to cause the server to email the password file to them.

Assigning new passwords

If your password file has been ripped, the cracker now has the passwords for probably half of your users. If you don't have more than about 50 users, you will probably want to issue each user a new password and email it to them. Don't change their user name, just their password. If you have more than 50 or so users, you have three choices. You could manually assign each user a new password to replace the one the cracker may have, but that will take a long time. Your processor may be able to get you a list of users and their email addresses, in which case your favorite programmer can write a simple script to generate new passwords and email them to users. Some webmasters also choose not to issue new passwords to all of the users, even though a great many of them are compromised. Instead, they use Strongbox or some other method to watch for individual passwords which appear to be compromised and change only those passwords. All of these options are inconvenient, but unfortunately that's the case when your password file is ripped. On your next site, you'll know to use our tools to assign good passwords and encrypt them using strong encryption so that it's very unlikely the password file will get ripped again.

The last modified date and the logs - an investigator's best friends

Using FTP or SSH, look at the last modified date of the password file and take a note of it. Then, look at the last line of the file and use the Strongbox reports to see if the last user added appears in your processor's list of users. If not, the date on the password file will often indicate exactly when the user was added by the cracker. Send this exact date and time to your system administrator and ask them to "grep" ALL of the Apache logs for scripts run or files uploaded within a few seconds of that time. This will often tell you exactly which script the attacker exploited to add this user. Note that the script used by the cracker will not necesarily be located on the same site that the users were added to or ripped from, so it is important to check the logs for all sites. Also ask your system administrator to grep the FTP log in case the cracker was able to download or upload the password file via FTP. This will normally find the problem, if your server is set up properly.

If your host or administrator does not have logs going back at least several days, ask them to fix that. Log files should be rotated or truncated once per day or so, not completely deleted. That way you can always look at logs from a few days ago. Proper logs are essential for security and can be very useful for marketing and legal purposes, so if your host or admin will not set this up correctly consider replacing them with a true professional. These essential logs are a basic requirement for any professionally managed web server. If the offending script is found, you'll need to quickly remove or upgrade it to a version that does not have the vulnerability. Beware, though, that many systems admistrators have a tendency to randomly pick something to blame without having any real knowledge of what really happened and little or no evidence to support their conclusion. Therefore don't take your administrator's conclusion as the final word. Their opinion is only as good as their ability to clearly explain and to show in the logs exactly what happened, when, and how. Ask then to send you the log entries showing exactly what happened and to explain how they decided to blame that particular script or issue. If they cannot fully support their conclusion, treat it as only one possibility and continue investigating fully. Even if ONE problem is conclusively found, there may be other problems as well, with other scripts, or PHP and CGI configuration problems that allowed the initial script to be exploited and will allow other scripts to be exploited in the future, so keep reading.

Essential steps to securing your scripts

Regardless of whether or not some problem script or configuration was found by looking at the logs and other forensic evidence your system admin may have found, now is the time to secure your whole server. Just because ONE problem was found and fixed doesn't mean that there are not other problems a cracker may find, or even holes that a cracker created for themselves after initially getting in through the first hole. Most security breaches on web sites are related to PHP or CGI scripts - either the scripts themselves have holes, the permissions are incorrectly set, or the server configuration allows them to run insecurely. Often, it is a combination of these factors - a poor configuration allows a bug in a script to be exploited, for example. Resist the temptation to assume that becuse you are accustomed to using FTP to work on your site, the cracker must have also used FTP. PHP script exploits are over one hundred times more common than FTP breaches. For reasons discussed below, the PHP language is particularly vulnerable, both due to holes in PHP itself, which the programmer can't avoid, and PHP language misfeatures which encourage poorly written, insecure script code.

The first and often most effective way to secure your PHP and CGI scripts is to get rid of them. A script that has been deleted can't be exploited, so go through your site, especially the cgi-bin directory, and get rid of any old, unused scripts. You may find some scripts that you can't readily identify. Your server admin, or better yet, your favorite programmer, can often help you identify the function and origin of a particular script. If you're not sure, you can temporarily rename a script, then wait some time, several days or a week, perhaps, to see if the function of your site is impaired by having the script no longer accessible. Only when your sure that the script isn't needed for the site would you want to totally delete it. You will want to carefully download any questionable scripts to your local computer and save them in a safe place before deleting them from the server. That way you still have a copy if it turns out to be something you needed. Also you can provide a copy to your system admin or security consultant for forensic analysis if needed. Before deleting any file or directory which appears that it may have been used by a hacker, take note of any dates for the file. The last modified time will be visible via FTP. If you have SSH access you can note the creation time, last modified time, and last access time. These can be very helpful when investigating a breach because they can be compared to the logs, as explained above. If you have SSH access, you can save a copy of the script with all of this information and then delete the directory using these commands:
tar -czvf thescript_suspicious.tar.gz thescript
rm -rf thescript

After removing any unused or suspicious scripts, you'll want to check the permissions (chmod) on the remaining scripts. Check the installation documentation for each script for exact details, but a few simple rules normally apply:

Also check with the vendor or group who develops the scripts you use to see if there are any security related upgrades or patches. Not all upgrades are good for security, so if the CHANGELOG doesn't show any security fixes and the script already does what you want, don't automtically upgrade. As an example, in a typical year Microsoft updates will add 1,200 NEW security problems that you wouldn't have if you didn't update your Microsoft software. As they say, if it aint broke, don't fix it. If it is broke, though ...

Essential steps to securing your server

Steps to secure a typical server running Apache, PHP, and MySQL could fill a book. In fact, Apache security fills 500 pages of Maximum Apache Security, available from your local bookstore or sites like Amazon.com. Because this is such a broad and complex topic, it is best to employ a qualified administrator to harden your server, but we'll touch on a few highlights you can check or mention to your admin. You want someone who is truly qualified, one who has read these 500 page books. If you insist on doing it yourself, here's your reading list to do the job right:
Apache security reading list
This is the point where I would put in contact information for a qualified professional, but I don't know any other than myself, so if you think you're qualified contact us and after asking a few questions to check your knowledge we can mention you here. Some of the web hosts we list may be able to assist you with good security if you host with them. Swiftwill is the most secure we know of. Their caution / paranoia can be inconvenient or even downright annoying for webmasters who don't place much emphasis on security, but if you take security seriously you'll definitely want to consider hosting with Swiftwill. phatservers, chilihost, and atcihosting are knowledgeable as well as having great customer support. If you have a site that is producing a lot of revenue, you may want to click on the reading list link above and contact the authors of some of those books to see if they will help you with your security or refer you to someone who will. Webmasters occasionally email us asking us to harden their server. While we are able to do a basic hardening job, we're generally quite busy with Strongbox and our other regular work, so taking the time out to harden a server means we have to charge the webmaster a significant fee. Unfortunately you can't just hire someone knowledgeable and ask them to make your server secure. You'll need to specifically mention at least a few key things that must be included, discussed below.

Root Kits
All of the below steps that you can take assume that the system itself hasn't been completely breached. If the whole system has been taken over by a cracker using something called a "root kit" then you can't trust anything the server tells you. For example, below we will look at what PHP files are on the server, and look at the permissions (chmod) on certain files. We can't do this if the cracker has set up the system to LIE to us about what files exist, what their permissions, are, etc. In order to be sure that you can trust the system, ask your admin to boot the system from a trusted CD or floppy and run chkrootkit or a similar tool. Remember, chkrootkit must be run from a CD or floppy. If not, the system itself my lie to chkrootkit because the cracker is in control of the system. If you find that your server is infected with a root kit, you'll need to back up your web site, then re-install the operating system from scratch. A qualified professional will need to examine all of the files that make up your web site and clear them before putting them on the clean system to avoid contaminating it.

PHP is the biggest security risk on most servers, as the language was never designed with security in mind. The language, orginally cross between a blog and a CMS written in Perl, was designed to be super easy to use, not to be secure or powerful enough for general programming. As PHP has become popular so quickly a major challenge with new versions has been to try to make it more secure. Security has impoved somewhat, but still with the current version as of early 2007 every PHP script ever written allows any visitor to your site to upload anything they want to your server. Just as we said about scripts, the best way to avoid PHP security issues is simply to get rid of PHP. If your site doesn't use any PHP, or at least doesn't need PHP for significant functions, consider removing PHP from the server entirely. That will eliminate many security concerns. Many sites today do depend on PHP, so you may need to make your PHP installation as secure as possible. This again is a major undertaking to really do right, complex enough to fill 528 pages in the book Pro PHP Security, available from Amazon. Fortunately, major security improvements are possible in PHP in under an hour by reading just a few pages and following the steps outlined. Securing PHP - Step by Step will guide you or your admin through the basics. Even if you think your admin already knows the basics, send them that link to use as a checklist or reference, to be sure nothing listed is missed. In case that link becomes outdated or is removed, the amazing Google - search securing PHP will always find the information we need. There are dozens of good pages about securing PHP instantly avaiable via the Google link, so I won't rehash the information here. I'll just comment that some of the security settings may seem to "break" some of your PHP scripts. This is an illusion, in my opinion. In fact, the scripts were already "broken" in that they have security holes. The changed PHP settings simply let you know what the holes were there, and refused to run the scripts that had obvious security holes. Contact the developers of the scripts about fixes or upgrades for the script to eliminate the security holes so that they will run OK with your new more secure configuration. As a last resort, you can often use a .htaccess file or <Directory> section in your Apache configuration to set the PHP configuration to allow the risky code only for the script that uses it.

CGI script security is largely a matter of making sure that permissions (chmod) are correct as listed above and getting your scripts only from reliable sources who are knowledgeable about security and pay attention to security in the way their script is written. Unless you are a programmer and know how to audit the code itself, one of your best defenses may be to ask the developer about what kinds of security concerns could come up with a script of that general type and what they've done to mitigate those risks. You may not fully understand their answer, but you can generally get a feel for whether or not the developer has thought about the potential risks and dealt with them. If they answer that security isn't a real concern or that there aren't significant security concerns that they had to address, that may be bad news. If the developer can't tell you about the security concerns associated with that general type of script it may indicate that they've never thought about what the risks may be. It's best to ask these questions via telephone or instant messaging rather than email so you can tell if the developer is at ease discussing the subject, answering questions quickly rather than hemming and hawing. It's much easier to BS you via email, when the developer can take the time to carefully think of ways to confuse you with jargon or put you at ease with answers that don't really address the subject. If the script includes a form that the user can fill out or some other way for the user to affect how the script runs, ask the developer about code injection attacks. Again the full answer may sound like a foreign language, but the developer should be knowledgeable enough to give a very basic explanation in plain English (or French, or German, or ... ) and by listening carefully to the technical bits you can usually determine whether or not the developer knows what they are talking about - even if you don't know quite what they are talking about. Any script with a form of any type should include server side validation of the content of the form to avoid injection attacks.

Looking at the very top line of the CGI script, you should see a line that looks like this: #!/usr/bin/perl or sometimes: #!/usr/local/bin/perl If the line includes "-T" at the end that's great news. That runs Perl in taint mode, which prohibits the script from doing certain things that are obvious security risks. If you see "use strict;" near the top of the file that's also good news because it suggests that the developer uses high quality code. Higher quality code is less likely to have bugs, including security bugs. There are a LOT of excellent scripts available which use neither taint mode nor Strict, so just because you don't see these doesn't mean the script is not well written and secure. These are just two indications that tend to reflect high quality if they are there. You can add -T to the top of a script that doesn't have it and see if it runs. Just change the first line to read: #!/usr/bin/perl -T If it still runs, great, the script is more secure than most. If not, you can check the error log for more details about possible security holes if you're geeky enough to understand the messages in the log. If not, you can ask your system admin or the developer of the script to trying running in taint mode "-T" and take care of any potential security issues that are revealed.

Lastly, trying chmodding an existing script 700 and see if it runs. If so, you're server is running with a setting called "CGI SuExec" that is extremely problematic for security. The setting is designed for mass hosting environments where hundreds of webmasters share a single server. The idea of SuExec is that it helps protect you from being attacked by another webmaster who has a site on your same server. It does that, but at a huge cost. It also gives any visitor to your site full rights to delete your files, change them, and upload there own files to your site. On a dedicated server, running SuExec is pure insanity because it creates serious security issues with no benefit at all. On a shared server with just a few other webasters, it's only slightly less insane. Only if you feel that the other webmasters hosting sites on the server are more likely to attack you than a cracker attacking using more traditional methods would you consider running CGI SuExec. If you're on a server with hundreds of other webmasters, some of which are likely to be hackers, you'll have to decide whether it makes more sense to run the risk of SuExec or to simply switch to another server. With dedicated servers available for under one hundred dollars and good shared accounts running about twenty five dollars, I'd never run SuExec. Unfortunately at least one of the popular control panels is set to use SuExe by default, presumably either because it was designed for mass hosting environments with hundreds of webmasters per server, or because the person who chose that setting as the default simply didn't understand the setting. This exception to this is if SuExec is set to run CGI scripts as a different user ID than the one you use for FTP. That setting may make good sense. With that type of setting, CGI scripts will NOT run when chmod 700. If your web host or admin is unfamiliar with the risk or simply doesn't know how to disable SuExec when using the control panel you're using and insists on using SuExec, insist that they set SuExec to use a different user ID that your FTP user ID. Turning off SuExec after you've been using it may cause a script to stop functioning and you'll need to correct the permissions (chmod) on some files to make the script work again. The proper permissions are listed above and the security improvement is worth the time it takes to correct the permissions.

More information about CGI security can be found in W3C's FAQ.

Securing Apache itself is of course a complex topic as well that varies depending on which version of Apache you're using but the Apache project has some good tips for recent versions:
Apache 1.3 security tips Apache 2.0 security tips Apache 2.2 security tips

ScannerX is an automated online tool to scan your server for many vulnerabilities in different parts of the system, primarily related to bugs in the different services that your server may run, such as the mail server, the DNS server, and MySQL. Your system admin should take care of these types of issues, but busy admins don't always have the time to stay on top of these things, so you may want to scan your own server.

Now that you've secured your scripts and your server, don't forget to upgrade the encryption on that password file and make sure users are using strong passwords. Our password generator page discusses these two topics in further detail, but the bottom line is if you send us $30 we'll take care of it for you.