Checking UNIX Server Performance

Today, fast load times are crucial to the success of your Website. They improve the user experience, allowing your visitors to take full advantage of your site’s functionality, and ultimately help you gain more traffic. But how can you achieve these fast load times? Good server performance is the answer.

Many Webmasters today don’t bother — or know how — to check a server’s performance. Virtual (or shared) Web hosts might check their servers’ performance now and then; however, if you’re running on a dedicated server, you really need to check its performance on regular basis. If a potential problem isn’t taken care of quickly, it could soon bog down the entire server.

Getting Started

There are many different protocols and other ways to connect to your server, and among the most popular command line variants are Telnet and SSH. Any information you transfer via Telnet is sent in plain text, so anyone running a Network packet sniffer could gain access to any sensitive details you might send.
In order to accommodate the need for security and privacy Secure Socket Shell (SSH) was developed, which is basically a secure version of Telnet that encrypts any information you enter prior to sending the data across the Web. As the SSH read me file technically phrases it,

“SSH (Secure Shell) is a program to log into another computer over a network, to execute commands in a remote machine, and to move files from one machine to another. It provides strong authentication and secure communications over insecure channels. It is intended as a replacement for rlogin, rsh, and rcp.”

For more info and help on SSH head to the SSH homepage.

In order to access your server, you’ll need a program capable of using SSH protocols. One of my preferred Windows programs is Putty, which has a nice, streamlined command prompt interface. Once you start Putty or another SSH program of your choice, you’ll need to connect to your Web server before you can run any commands. Some programs will provide a GUI (graphical user interface) through which you can enter your info, while others will provide a command line interface. If you’re left with a plain command line, obviously you’ll need to enter a command in order to connect.

ssh hostname -l username

You’ll need to substitute the hostname with the domain name or IP address of the site you’re connecting to. The -l parameter tells the program to not log you in as root, but under some other username. Submit the command and you’ll be prompted for your password. Note that unlike Windows, with which you’re probably familiar, you won’t see any asterisks as you enter your password. After you’re authenticated, you’ll be logged in.

If you ever forget what username you’re logged in as, you can run this simple UNIX command:

whoami

Once you’re logged in, you are exposed to the world of command line commands of your operating system. Explore all you wish, but try to find out what a command does before you run it. You can obtain a description of what any UNIX command does using the man command, followed by the command you’re interested in learning about. For example:

man top

This will give you an explanation of the top command, which we’ll get into a bit later. Don’t run any commands you aren’t sure about — it just takes one command to wipe your hard drive clean, so be careful!

Server Specifications

What your server can do is greatly affected by the hardware inside it. CPU(s) and RAM are the two main players. How do you know what your server’s capable of? Well, one of the most useful UNIX commands regarding performance is top. It displays the most important data about what’s going on in your server. Here’s a typical example of what you might get as a part of top query result:

7:06pm up 81 days, 7:47, 1 user, load average: 2.90, 2.25, 2.14
214 processes: 209 sleeping, 5 running, 0 zombie, 0 stopped
CPU0 states: 66.3% user, 14.4% system, 27.1% nice, 18.4% idle
CPU1 states: 71.1% user, 14.4% system, 37.1% nice, 13.2% idle
Mem: 3867832K av, 3853780K used, 14052K free, 0K shrd, 100388K buff
Swap: 530104K av, 45980K used, 484124K free 3228192K cached

Now, let’s go through this information step by step.

7:06pm up 81 days, 7:47, 1 user, load average: 2.90, 2.25, 2.14

The first number is the server time, followed by how many days and hours ago the server was last rebooted. The next figure shows how many users are connected to the machine (through protocols such as SSH). Lastly, come the server load averages. The first number indicates the average number of processes running over the past minute. The second indicates the average over the last 5 minutes. The third shows the average over the past 15 minutes. The next line displays information about process, but we’ll come back to that in a moment. The lines following these first two provide CPU usage statistics (in this example, the server has two CPUs):

CPU0 states: 66.3% user, 14.4% system, 27.1% nice, 18.4% idle
CPU1 states: 71.1% user, 14.4% system, 37.1% nice, 13.2% idle

The foremost CPU is identified by its number (starting at 0). Processor execution modes fall into two categories – non-privileged mode (user), and privileged mode (system). Privileged mode is often called “kernel” (system) mode; non-privileged mode may be referred to as “user” mode. If executed in non-privileged mode, a process will be able to access only its own memory, whereas in privileged mode, access is available to all of the kernel’s data structures, as well as the underlying hardware. The kernel executes processes in non-privileged mode to prevent user processes from accessing data structures or hardware registers that may affect other processes or the operating environment.

On a UNIX system, each process runs according to what’s called a scheduling priority level. A process called “scheduler” distributes CPU time to processes according to their priority, or level. Processes with higher priority level (-20 being the highest) get to run ahead of those with a lower priority. Idle percentage displays the amount of CPU power that isn’t in use. The following two lines give figures about memory and swap file usage:

Mem: 3867832K av, 3853780K used, 14052K free, 0K shrd, 100388K buff
Swap: 530104K av, 45980K used, 484124K free 3228192K cached

The first number represents the total amount available, while the second displays amounts used in kilobytes. The third is the amount free, again in kilobytes, and lastly are the buffer and cache stats. The buffer is the space to which a system writes, before writing to disk, and cache optimizes the performance of your programs.

Processes

In the process of analyzing the beginning of a typical top result, we skipped a line about processes, and a few other figures. All of these statistics refer to things called processes. Processes are the programs being executed, or run, on a Web server. Let’s start off with the 2nd line of a top result we omitted before.

214 processes: 209 sleeping, 5 running, 0 zombies, 0 stopped

The line starts with a count of total processes. Afterwards are explanations of the categories into which each of those processes falls. There are four default categories for processes: sleeping, running, zombie, and stopped. The majority of the processes fall into the sleeping category – they’re processes waiting in a queue to be worked on, and once their turn comes, they become running processes. Few processes are running — those are the processes being worked on right now.

During the life of a sleeping process, child processes can be born. Normally, a main process would kill its child processes before it ended, however, if that doesn’t occur, the child process will keep on going without its parent. When this occurs, this “lost child” is classified as a zombie process. A high number of zombies is one thing a system admin dreads, though fortunately those processes are terminated by the system at predefined intervals. Finally, stopped processes are those that are paused, typically because they’re waiting for a resource.

As you would have noticed, at the bottom of the top result was a big list – this is a more thorough listing of the processes included in the second line of a top result. Here’s an example of a part of the list.

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
10685 ze-card 19 19 984 736 724 R N 93.3 0.0 5671m webalizer
32072 nobody 9 0 16932 15M 12156 S 6.0 0.3 0:01 httpd
32196 nobody 9 0 16224 14M 12144 S 3.5 0.3 0:01 httpd
1868 nobody 9 0 16284 14M 12620 S 3.1 0.3 0:00 httpd
2136 nobody 9 0 16080 14M 12164 S 2.5 0.3 0:00 httpd
32205 nobody 9 0 16300 14M 12136 S 2.3 0.3 0:00 httpd
32231 nobody 9 0 16316 14M 12172 S 2.3 0.3 0:00 httpd
32124 nobody 9 0 16620 14M 12184 S 1.9 0.3 0:01 httpd

On top of the list are short descriptions of each column, and in order to understand the listings it’s a good idea to know what each column shows. The first column shows the unique identifier of a task, called a “process id”. The “User” column displays the owner of the task. “PRI” is short for priority, which symbolizes how important a task is. “NI” stands for Nice, also a queue-related factor, where negative values are given a higher priority.

The size of the task’s code, data, and stack space (in kilobytes) are next. “RSS” is the total physical memory utilization of the process. “SHARE” represents shared memory. “STAT” is an abbreviation of the word “state”, and refers to the condition of a process. Here, “S” stands for sleeping, “D” for uninterruptible sleep, “R” for running, “Z” for zombies, “T” for stopped or traced, “N” for a process with positive nice value, “W” for a swapped out process. %CPU and %MEM indicate the share of the total CPU power or memory used since the last screen update. “TIME” shows the total amount of CPU time that the process has used in its lifetime (i.e. the amount of time it has spent in the ‘running’ state). And lastly, “COMMAND” displays the command the task is running.

Users and Accounts

One question Webmasters hosted on a virtual or semi-dedicated account might ask themselves is how many people are on the server besides them – as no one wants to be on an overloaded server. Another UNIX command comes to the rescue:

wc -l </etc/passwd

The command is an acronym for “word count”. The -l parameter tells the program to count lines, while the remainder of the line tells it what to count. In this case, the command tells the program to count the file with the list of all accounts on the system. Keep in mind that not only user accounts are counted — all system accounts will be counted, too. By default, UNIX will have about 15 system accounts. In addition to that, your server system administrator most likely added more system users to accommodate for many features on your server. Typically, on a virtual hosting server you can expect to find around 40 users who are not hosting clients. If you ever want to view a full list of the users, there’s a command which you can utilize.

more /etc/passwd

The more command will call the passwd file, which essentially contains a list of every account on the system. You can look at what other sites or users are on your server, count all the actual hosting accounts (based on their username), or complete a range of other queries. Also, if you ever want to know the list of everyone who is logged in to the machine at the time, you can run the “who” command.

Additional Utilities

One more useful utility available on some servers is System Account Report, or “SAR” for short. The command is composed like this:

sar parameter count timeinterval

Parameter is a dash followed by a letter, which tells SAR what to report on. Following this are commands of how many times to check, and how many seconds to wait between checks. Usually the results aren't hard to decipher, especially with the help of the man command; but some common result abbreviations are "TPS" and "BPS". "TPS" stands for transfers per second, in kilobytes. "BPS" is an acronym for blocks per second; one block is half of a kilobyte, or 512 bytes.

Here are some useful parameters:

• -b (I/O and transfer rate statistics)
• -c (Process creation activity)
• -d (Disk activity)

There are plenty of other ways to check up on performance, but this covers the most relevant ones. And don’t forget to use the man command if ever you’re not sure about something. Have fun exploring!

Further Reading (Books)

• “Modern Operating Systems” by Andrew S. Tanenbaum
  
• “Essential System Administration” by A. Eleen Frisch
  
• “TCP/IP Illustrated, Vol. 1” by Richard Stevens

Frequently Asked Questions (FAQs) about Unix Server Performance