The Furtive Penguin

Six Commands To Run Daily on A Linux Server To Monitor Performance and Security

Having acquainted themselves with Linux on the Desktop many people have opted to run an internet or home intranet server on their distribution of choice. Running a home server on a LAMP stack can seem an intimidating prospect. Installation from disk is easy enough and there are a wide range of distro's to choose from but there is also much to learn before your installation will do your bidding. Supposing that you persevere and succeed in hosting a few internet sites on your server, How do you then monitor its performance and secure it against intruders? It is not my intention to add to the plethora of excellent installation and setup guides which already exist in various places on the web. The purpose of this article is to introduce a number of elementary procedures which, if practiced regularly, should ensure healthy system performance and a reasonable degree of security. (The prescribed commands are highlighted in red)

Update and Upgrade

You should probably check for updates on a daily basis and if they are available, upgrade immediately. Platitudes, platitudes! But in all seriousness this is probably the single most important thing to do if you want to remain secure. In the world of open source the 'many eyes' pouring over the code ensure that the good guys spot possible exploits first. To benefit from this constant scrutiny you must keep up to date. Of course if you are running an 'Ubuntu' server this is as simple as:-

1. apt-get update

  apt-get upgrade

Security and Performance Monitoring

The next step in your daily security routine should be security and performance monitoring. The netstat command will display your incoming and outgoing network connections. If used with the appropriate options it will tell you which services are running and on which ports. Here is my preferred combination:-

2. netstat -pltun

And here is some sample output:-

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 4715/mysqld

tcp6 0 0 :::80 :::* LISTEN 5144/apache2

tcp6 0 0 :::21 :::* LISTEN 4847/sshd

udp 0 0 0.0.0.0:68 0.0.0.0:* 4209/dhclient

As you can see this server is running Apache, SSH and MySQL. If this is what you expected to see then all is well. If on the other hand some unidentified service is running on a non-standard port you may have a problem. The function of the p,l,t,u and n options are explained in the netstat man pages which can be found here. It may be the case that a different combination of options are better suited to your needs. It is well worth the trouble to acquaint yourself intimately with this powerful and versatile tool. If you have any suspicious processes running on your machine you should investigate them using 'lsof' e.g:-

lsof -c dhclient

Of course there is nothing suspicious about the dhclient process in this case but nonetheless 'lsof -c' will provide us with a list of all the open files that the process is using. We are then in a position to investigate further by checking for permissions on individual files etc. There are many options for lsof. It is one of the most critical tools to master on a unix/linux system. Ideally you should study the man page but failing that here are two 'lsof' resources, one short and sweet and another which is much more detailed.

3. cat /var/log/auth.log

This command will present you with a list of all recent login attempts made on your server. This is particularly important if you are running SSH. If you find that continuous login attempts are being made with a variety of usernames then it is likely that you are being targetted by an automated script. Read this article for further details. If this is the case you should adopt one or more of these remedial measures immediately:-

a.If possible deny remote logins and use SSH on your intranet only. To achieve this you simply need to disable portforwarding on your router.

b.Consider running SSH on a nonstandard port. This involves a few changes to the configuration files.

c.Abandon password logins and switch to pkcrypto. This will defeat any password based login attempts, automated or otherwise.

d.Install and configure the excellent Denyhosts script. ( not necessary if you resort to a. or c. above )

4. ps; sleep 2; ps

This command will check that your server is not spawning an excessive number of processes. The output should be somewhat similar to this:-

PID TTY TIME CMD

26327 pts/3 00:00:00 bash

26351 pts/3 00:00:00 ps

PID TTY TIME CMD

26327 pts/3 00:00:00 bash

26353 pts/3 00:00:00 ps

Note that the PID of the second 'ps' command (26353) is two numbers higher than the PID of the first command (26351). This is as it should be. If the second PID number is consistently much higher than the first ( assuming that you repeat this test a number of times ) then you have a problem. PID numbers are assigned in sequence so that if the second number is 10, 20 or a 100 times greater than the first it follows that a great many processes are being spawned in a short period of time. If your server is not especially busy then this is problematic. Extensive troubleshooting may be necessary in order to resolve the issue.

For a much more in-depth analysis of server performance consult the man page for the 'vmstat' command. This command should be left to run for an extended period of time and this is definitely not something that needs to be done on a daily basis.. Analysis of the results will reveal much about your servers' current performance.

Check For Rootkits

But what if you have been duped? If a rootkit has been installed on your box then the output from all of the above commands is likely to be bogus. Rootkits install their own version of the very sytem binaries which you would use to detect them. They are obviously doctored in order to conceal the nefarious activities of the hacker who installed them. In order to guard against this possibility you should run daily rootkit checks. Chkrootkit, which is available from the Debian/Ubuntu repositories, runs a battery of tests which will detect the presence of known rootkits on your system. Since it relies on a number of system binaries in order to do its job it is wise to back these up to an independent medium immediately after installing your server. CD is best....I don't think they all fit on a floppy. You should then run chkrootkit using the '-p' flag to specify the path to your "known-good" binaries. A sample command would be:-

/path/to/chkrootkit -p /mnt/cdrom

The binaries you need to back up are as follows:-

id, cut, ps, find, head, awk, ls, netstat, egrep, uname, sed, strings

Assuming that you have not backed up and are using the installed system binaries just run:-

5. chkrootkit

Another tool which does roughly the same job ( plus a few extras ) is Rootkit Hunter. Rkhunter also has a much more pleasing interface, though still command-line based. The project page for Rkhunter can be found here. Its worth running both because it never hurts to doublecheck. The command to run is:-

6. rkhunter -c

So...we have updated, monitored logins, connections and processes and checked for rootkits. I am not suggesting that the daily half-dozen listed above will preserve your server from all ills for ever and ever more. It is much more likely however, that you will have a trouble free experience if you stick to the above regimen.

(I know that all this material is available elsewhere on the web. My presumption in presenting it here is partly justified by the fact that I know of few other sites where it is all gathered in one place. Hope this helps!........If anyone wishes to suggest any other security or performance related commands which could usefully be run on a regular basis please post in the comments section below and I will add them to the list.)