Replacing a server – part 1: the audit

Windows Servers - Data Center by jaxmac used under Creative Comments Attribution-Non-Commercial-No Derivative Works license

For a small digital agency, running an off-site server is as important as it is unglamorous. You don’t get any of the desirable super-tech of running a cluster but all of the headaches of running a constantly used, high-availability external computer. My workplace’s existing dedicated server (which I championed, configured and maintain) is used to provide web hosting to a variety of clients – both large and small – and for the past three and half years has provided a flawless service. Upgrading is not to be taken lightly and the reasons for doing so must always result in a better service to clients – whether that’s decreased work load for you or improved site responsiveness. For me it boiled down to entropy: three and a half years is a long time for hardware to run and it will eventually fail and make my day/week/month hell on toast.

First step on this crazy adventure: audit.

Cleaning house

Audit is a filthy word round most parts and conjures up images of bespectacled pencil-pushers or greasy tax collectors. Despite this, documenting what you have is the first step to getting something better. When I begun this process however I found that, like a house, over time a server accumulates clutter: old domains, long since defunct sites, errant processes; automation only goes so far before a cleaner has to step in.

Spending a day archiving and removing cruft is tantamount to dusting the shelves and throwing away old books and furniture before moving house – it reduces the effort required later in the process. My removals included:

• Domain name end-points – for ones which had either expired or the persons / companies had moved on
• E-mail accounts – accounts for expired domains are useless, just as accounts for long since lapsed campaigns are
• Test folders – a separate test environment means accumulation of in-progress sites was inevitable. I found a year without modification is a good metric for when to cull
• Errant services – automated / scheduled processes such as a log-parsers; awstats was set to run on Apache’s log files – no longer necessary when every site we host uses Google Analytics
• Old databases – very few of these but the odd one sometimes slips through

After archiving, it was time for the document itself.

In-depth

Putting as much information into the audit as possible is immeasurably important for a server move. Anything and everything to do with the hardware, software, services, clients, accounts, databases, configuration files, contacts to name but a few. Doing this creates a single tome to refer to rather than having to scrabble around for details if, in the worst case, things go awry (as oft do).

A non-exhaustive list of areas I documented included:

• Hardware – what is the server running on, processor speed, RAM size (latency, bad sectors), hard drive set up (RAID, size, throughput)
• Operating system – flavour, version, kernel version, partitions (including flags), upgrade system, custom repositories
• Core services – what the server does and doesn’t offer
• Key applications – what is running all the time, version numbers, patch levels and dependencies
• Scheduled tasks – what runs when and its purpose, could be as simple as logwatch, or perhaps as specialised as a point-in-time backup script
• Domain end-points – even if you don’t run your own DNS server, your web server needs to know what points where
• Domain names – nameservers, which company manages them, who to speak with to get domains repointed / altered
• Clients – who has a stake with what on the server, includes contact numbers for content providers, technical liaison, who manages the domain, who pays the bills. List associated domain names, e-mail addresses etc. alongside
• Databases – not just MySQL/Postgres, include details of any that the mail server may build, any embedded ones (SQLite), password caches
• E-mail addresses – service type (POP3, IMAP, SSL), quotas, usernames, passwords, retention policies
• Config files – any modified or customised configuration should be included in full (probably in an appendix) with the location of the file and related service
• Firewall rules – a simple iptables output may be all that’s necessary
• Open ports – anything that (x)inetd may respond to, non-standard port definitions, port knocking
• Shell accounts – usernames, home directories (both location and contents)

Security

With the document created, there now exists an all-access pass to your server and will be coveted by anyone with nefarious purposes in mind. How you secure this document is entirely up to you: whether you print out one copy and encrypt the digital version, set up a trusted third party to hold on to it or go for public-private key encryption and hand it out to trusted members of your company.

Also be sure to note that the document is not just beneficial for you, but also for anyone who may have to pick up where you left off. If things go awry mid-move, say a sudden onset of kidnapping befalls you, colleagues and contractors will benefit enormously from the document. Who knows, they may even chip in for your ransom. Locking the document away where only you can access it may not be the best move.

In theory the utility of the document is time limited as you’re aiming to get your new server ready soon which would make the existing one obsolete; the key message really is: don't underestimate how sensitive a server audit document actually is.

Conclusion

Creating a server audit document serves two primary purposes:

1. A document that wholly encapsulates your server providing a single reference point for any query you or anyone may have
2. It gives you a better understanding of the server as a whole

This contributes towards the maxim that I find so important: the more you know and the better prepared you are, the easier it’s going to be.