Server administration course: Part 1 – Managing a Linux Server

Server deployment needs

As a company we are daily deploying and updating servers for a wide range of projects. We have servers running on a wide range of cloud platforms, including Amazon, Rackspace and Digital Ocean. We have simple single server projects and auto scaling projects running on 5-10 servers as the load varies. It’s also important for us to keep up to date with best practices and emerging tech so that we can offer our clients the best solutions.

Given all the possiblities it’s important that our staff are able to work effectively with all the different solutions we use. We have recently started an internal training program for our server experts to refresh and deepen their knowlege.

This course

The course will cover every level of the server stack, from administration of linux servers to managing a cluster of Amazon virtual machines.

In this, the first installment, we cover the basics of managing a linux server. We assume you have probably been here before, but maybe missed some details and need a reminder of a few things. We will start at the beginning.

Navigating the system

The key commands for navigating your new machine:

* cd will help you move from one directory to the next, remember that you can refer to the current directory as . and the parent directory as ..
* ls to see a list of files in a directory. It has useful flags such as l which will show more details such as permissions and modification times and a which will show you hidden files

ubuntu@my-server:/$ ls -la
 total 96
 drwxr-xr-x 23 root root 4096 Apr 30 06:58 .
 drwxr-xr-x 23 root root 4096 Apr 30 06:58 ..
 drwxr-xr-x 2 root root 4096 Apr 9 06:49 bin
 drwxr-xr-x 3 root root 4096 May 6 06:41 boot
 drwxr-xr-x 13 root root 3900 Apr 30 07:00 dev
 drwxr-xr-x 100 root root 4096 May 1 06:51 etc
 drwxr-xr-x 3 root root 4096 Mar 28 2014 home
 lrwxrwxrwx 1 root root 33 Apr 30 06:58 initrd.img -> boot/initrd.img-3.13.0-51-generic
 lrwxrwxrwx 1 root root 33 Apr 9 06:50 initrd.img.old -> boot/initrd.img-3.13.0-49-generic
 drwxr-xr-x 21 root root 4096 May 23 2014 lib
 drwxr-xr-x 2 root root 4096 Feb 27 2015 lib64
 drwx------ 2 root root 16384 Feb 12 2014 lost+found
 drwxr-xr-x 2 root root 4096 Feb 12 2014 media
 drwxr-xr-x 2 root root 4096 Oct 13 2013 mnt
 drwxr-xr-x 2 root root 4096 Feb 12 2014 opt
 dr-xr-xr-x 93 root root 0 Feb 23 2015 proc
 drwx------ 3 root root 4096 Jun 19 2014 root
 drwxr-xr-x 22 root root 880 Oct 5 08:20 run
 drwxr-xr-x 2 root root 12288 Apr 27 06:55 sbin
 drwxr-xr-x 2 root root 4096 Feb 12 2014 srv
 dr-xr-xr-x 13 root root 0 Feb 23 2015 sys
 drwxrwxrwt 3 root root 4096 Oct 5 08:17 tmp
 drwxr-xr-x 10 root root 4096 Feb 12 2014 usr
 drwxr-xr-x 14 root root 4096 Mar 28 2014 var
 lrwxrwxrwx 1 root root 30 Apr 30 06:58 vmlinuz -> boot/vmlinuz-3.13.0-51-generic
 lrwxrwxrwx 1 root root 30 Apr 9 06:50 vmlinuz.old -> boot/vmlinuz-3.13.0-49-generic
 drwxr-xr-x 4 root root 4096 Mar 28 2014 vol

Here we see an example of this, including some symbolic links.

We can also use this oppurtunity to have a quick look at the top directory layout:

– bin contains some basic programs (many of the ones we are talking about now)
– dev contains physical (or virtual) devices represented as files
– etc contains configuration files
– home has each users home folder
– media is where ubuntu will auto mount external drives
– proc interesting information about running processes
– srv server files
– lib libraries and external files for programs
– usr all of this again, for installed programs
– var log files, amongst others

* cat and less are quick ways to see the contents of a file
* tail is great for checking the end of a log file, and you can use -n 100 to see a bit more. -f will keep outputting more as the file is added to, great for checking out an issue as it happens.
* nano is a simple way to edit a file
* vi lets face it, you will come across vi even if you don’t want to, [esc] will get you to the command mode, :q will help you get out, :w if you want to save your changes
* touch make a new file, or change modification time on an existing file.
* chown, chgrp and chmod, read “ch” as change and these make sense
– Change a whole directory tree with -R
– chmod can make relative alterations, +r or g+w etc
* which to find a program

 ubuntu@my-server:/$ which which
 /usr/bin/which

Getting up and running

We’ve started up a new machine, what are some of the first things we need to do?

NTP

If you are planning on keeping a server running for any length of time, making sure that the clock stays accurate is surprisingly important. I have seens servers with a clock 10 minutes out of sync after just a couple of months of running. You’ll be surprised to learn that even virtual servers suffer from clock drift. Luckily this is quickly solved:

`apt-get install ntp`

Automatic updates

This is an issue with many sides to it, and depending on your workload you will have different needs. Many of the servers we set up need to be largely left to their own devices once they are up and running. To keep up to date with the latest security updates is especially important as all software has bugs.

Enabling security updates on Ubuntu/Debian is really easy:

`dpkg-reconfigure --priority=low unattended-upgrades`

Check out /etc/apt/apt.conf.d/50unattended-upgrades for more options once you have enabled it. I often enable updates, not just security, automatic removal of old packages, and for low traffic servers automatic reboot can often be enabled.

Creating users

Depending on how we set up our server we might be logged in as root at first. This isn’t good practice for everyday use. Its too easy to mess up, even for the most experienced sys admins.

We are going to use the adduser command to make a new user. Then we will do adduser my-user sudo to enable sudo for our new users. Now we have a new default user account and we can forget the root login that we had.

Talking of sudo, here are some useful options:

* -u another-user you can run commands as any user, not just root
* -s starts a new shell as another user (root by default), great when you need to do a bunch of stuff
* -H run the command with the other users home environment

Installing software

There are many ways to install software, but the most common is using the system package manager. This should generally be the prefered way as packages are built for and tested together on your system. Unfortunately the package systems are often slow to update, so if you need to run the latest version of something that is developing quickly you will need to look to other sources, sometimes compiling the code yourself.

On Debian, including Ubuntu, systems apt-get is the main way to interact with the package manager. On Redhat Linux, including Fedora, Centos, and Amazon Linux systems yum is the equivilent.

Sites such as http://packages.ubuntu.com/ can help find the package you need.

What’s happening

Now we can find our way around a bit, some useful programs to help us understand what is happening on our server. Maybe we need to find what is making our server feel slow, or where all that memory has gone.

top is a great starting place, showing a list of running processes, that we can order by cpu or memory usage.

This might be a typical output on a single server site:

ubuntu@my-server:~# top
 top - 08:38:47 up 16 days, 16:57, 1 user, load average: 1.05, 1.10, 1.17
 Tasks: 127 total, 2 running, 125 sleeping, 0 stopped, 0 zombie
 Cpu(s): 5.8%us, 5.0%sy, 0.0%ni, 88.8%id, 0.4%wa, 0.0%hi, 0.0%si, 0.0%st
 Mem: 4109264k total, 3570540k used, 538724k free, 280996k buffers
 Swap: 8385920k total, 580k used, 8385340k free, 2627788k cached
 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 20975 www-data 20 0 299m 19m 4944 S 10 0.5 0:04.12 apache2
 20544 www-data 20 0 306m 26m 6240 S 10 0.7 0:26.37 apache2
 20545 www-data 20 0 306m 25m 6372 S 10 0.6 0:27.17 apache2
 21066 www-data 20 0 303m 23m 4560 S 10 0.6 0:01.81 apache2
 20543 www-data 20 0 313m 34m 5240 S 10 0.9 0:23.27 apache2
 20985 www-data 20 0 304m 24m 4776 S 10 0.6 0:02.45 apache2
 21133 root 20 0 219m 28m 8864 S 9 0.7 0:00.28 php
 20440 www-data 20 0 327m 47m 6676 R 6 1.2 0:32.67 apache2
 553 mysql 20 0 351m 126m 8068 S 1 3.1 1507:27 mysqld

Things to notice are load averages and free memory, often signs of an overloaded system. Then start looking at the specific programs that are running and using these resources.

* Tap c to see full commands used to start a program
* > and < will change the sort column, x will highlight the currently selected column

That’s the end of this initial lesson, but keep your eyes peeled for the second one, coming soon.