Linux System Administrator’s Schedule

My recent post on benchmarking Linux system security generated a question about my procedures for maintaining Linux system health. Aside from some specific procedures based on special needs of some clients, I have noted a few odd tasks and then broken down my routine in a daily, weekly and monthly schedules.

There are a few things I only do on occasion such as quarterly user audits, house cleaning on administrative storage directories (downloads, patches, odds and ends) and quarterly backup restore tests. The latter every administrator should do at least twice annually – literally testing a restore from tape, disc or online storage to insure your backup processes are working properly.

One final quarterly tasks is flushing old Apache access logs, where applicable, off of the backup server as some of them chew up gigabyte after gigabyte of space.

As some background, a majority of the Linux servers I manage are not all in my own data center or from one client by spread all over in different environments. I still have managed to maintain a limited number of operating system builds and configurations so I do not create chaos as a system administrator. I also spend most of my time hardening a Linux box via iptables, setting up verbose logging and tightening apps like Apache, SSH, mail servers, ftp and others at the outset of its deployment.

My daily routine is pretty straightforward and includes cursory log reviews for iptables, system logs, mail, web, ftp and ssh. I generally query all of the mail servers for what is in queue (usually garbage spam but every once in a while a legitimate lost message). In qmail this is very easy via qmailctl on the command line and a handy Perl script that shows mail headers in the queue.

Aside from the expected daily tasks that run in cron on a majority of production Linux servers, I have some custom cron jobs I check in on that handle hot online backups via rsync, some mail tasks, scripts specific to clients, bandwidth use measurements and a Tripwire log.

Weekly I set aside some time to again check some system and client logs to be sure all is running smoothly, numerous automated data transfers and a glance at my own database of SSL certs, domain name and hosting contract dates. It is this weekly window when I also take time to investigate deeper into any flaky log entries like penetration attempts and other possible malicous attack attempts. If anything substantive comes from the research I report to the corresponding ISP the source IP originates from.

I also set aside a weekly slot to run down the various security and vulnerability emails received and RSS feeds or web sites I monitor (aside from of course the red alert emails that I respond to on demand for patching).

Finally, on a monthly basis I insure the log rotation occurs across the boxes I manage and take a closer look at backup logs. Finally I usually run some utilities monitoring disk space, review ‘top’ and ‘mytop’ logging done to file at various times throughout the previous month and perhaps runs rootkit checks.

Of course this is not completely all inclusive as there are proprietary client specific tasks, but reflects that keeping a Linux server(s) healthy entails building it right, knowing what is going on and keeping a system current.

If one has the flexibility, it is ideal to have at least one development Linux server so you can run quick tests of patches, updates and so on for your environments (I dupe a production box to an internal Linux server running a 10.5.x.x non-public IP and test for my own needs). This allows you to react quickly to emergency updates and still test so no production services are broken by it.

Perhaps most important, my final monthly activity is sending out invoices! ;>)

Frequently Asked Questions (FAQs) about Linux System Administration