The Value of Procedures and Fun with PCI and ISPs

While working with a prospect recently I was asked to help get a server up to speed to pass a PCI scan. I asked to see the report to get an understanding of how much work there would need to be done. In the initial report there were not too many things gone wrong, and basically what was in use was an SSL powered site on Linux and Apache. It was pretty straightforward stuff:

SSLversion 2 was enabled, medium to weak ciphers were in use. Typical things you would find on a server that's not been hardened.

So we discussed the situation, the work needed to be done, the time frame involved and we went on our way. For reference the so-called "Total Risk" calculation was not very high - it was something around 25 or so. The way this vendor had reported its overall risk of the server was to give each vulnerability a weight from one to 10, and then simply add up all the potential vulnerabilities.

Come to find out the client reran the PCI scan and now instead of having a "Total Risk" of 40 it was 108. Here's a mock of what that would that would have looked like (i.e. not the real report).

The main problem came in the fact that now instead of simply editing some Apache configuration files to make it use strong ciphers and strong versions of SSL, testing the new setup and documenting it, now the SSH Daemon and Apache were flagged to be upgraded as a resolution.

Again, really not a problem and really not a major reason for concern, except for the fact the correct method of updating the server packages was unknown. I did not want to move forward without knowing that, and in any case, any sort of validated backup and rollback procedures for any and all changes to a production instance.

Like anything else the best practice before diving headfirst into project is to understand the scope of the project, clearly define what needs to be done, and have a good ballpark of the time needed to take place.

Since there are many ways of updating packages on Linux my first task was to understand really how this server was should be properly updated (Most Shared and Dedicated ISPs used some sort of easy to use frontend to make this easy - others don't!). Recently I updated all of my binaries on my dedicated host and it only took a few minutes using apt-get. Apt-get dutifully stopped all the necessary services, updated what it needed and all was well. Of course I had a full image backup ready and waiting to go should anything happen.

My concern at this point for the client was that if I were to simply "make it happen" by downloading some software off the Internet, there may be some other problems introduced later for the client as you'll see.

A quick call to the ISP revealed "Plesk is crucial to all of the running services on the server. Specifically no one should go ahead and try to upgrade any services or software without using the Plesk control panel. Applications will break horribly if so." Ah, good to know, right?

To put it another way if I were to have jumped straight into the project, asking for root access and added RPMs or apt-get'd the box, it may actually worked - short term.

But the problem would have come long term, maybe a few weeks/months or longer when the client went to upgrade his system all sorts of version incompatibilities would have shown up and mayhem would have ensued. Bad Times!

Without an understanding of the overall potential risks involved, it's never wise to rush in. I used to work with a systems administrator who had the following tag line in his e-mail: "There are old administrators and there are bold administrators but there are no old, bold administrators."

Good advice kids!