Netflix shows how they get a high level look at Linux in 60 seconds

In 60 seconds you can get a high level idea of system resource usage and running processes by running the following ten commands. Look for errors and saturation metrics, as they are both easy to interpret, and then resource utilization. Saturation is where a resource has more load than it can handle, and can be exposed either as the length of a request queue, or time spent waiting.

uptime
dmesg | tail
vmstat 1
mpstat -P ALL 1
pidstat 1
iostat -xz 1
free -m
sar -n DEV 1
sar -n TCP,ETCP 1
top

Some of these

commands

  require the sysstat package installed. The metrics these commands expose will help you complete some of the USE Method: a methodology for locating performance bottlenecks. This involves checking utilization, saturation, and error metrics for all resources (CPUs, memory, disks, e.t.c.). Also pay attention to when you have checked and exonerated a resource, as by process of elimination this narrows the targets to study, and directs any follow on investigation. 

The Netflix Tech Blog: Linux Performance Analysis in 60,000 Milliseconds http://techblog.netflix.com/2015/11/linux-performance-analysis-in-60s.html

Lots of good info here, though I suspect many sys admins already run through most of this once they land on a box.

CALI’s Looking For a Sys Admin, Here’s A Brief History

Usually I might not be too keen to lose some of my job responsibilities, but in this case I couldn’t be happier. CALI is adding a systems administrator to wrangle all our servers, more than 20 at last count, on a full time basis. Since I started working at CALI 9 years ago my time has been split between web/database/cool project development and administering CALI’s servers and systems.

Back in 2003 that meant riding herd on an aging Windows NT server, a Win2K server handling some video streaming, and a couple of dark servers whose futures where not yet set. Of course the servers where in Chicago and I was in Atlanta. Things changed rapidly. The dark servers where brought online running Linux and our production web and storage systems where built out on the LAMP stack. Within a couple of years I added 3 servers at Emory in Atlanta to handle the increased demand for CALI services and resources online.

It wasn’t very long before we were struggling with large spikes in demand that were taxing our servers and we needed a better solution. Simply increasing the amount of hardware we owned wasn’t really an option since we were borrowing space and bandwidth from the law schools at Kent and Emory. At just the right time, Amazon Web Services came along and CALI jumped into the cloud.

Moving our web infrastructure to the AWS cloud gave us tremendous flexibility at a reasonable cost. After some trial and error I was able to configure a load-balanced web cluster that could be scaled up and down as demand for CALI resources and services flowed over the course of an academic year. Using the cloud meant that I could provision some services on their own servers so that things like Apache Solr and Asterisk could stand alone. As a result of the move to the cloud, by the beginning of 2011 I found myself administering 15 to 20 servers in the cloud alone (exact numbers depended on the time of year) plus another half dozen physical servers in 2 geographically dispersed locations.

All that sounds like a full time job itself, but that was only half the job. While all that infrastructure was being built out I was also developing 3 different versions of the CALI website, the Classcaster phone-to-blog system, a couple of iterations of eLangdell, the Free Law Reporter, and dealing with various other projects. Working on these development projects is what I really enjoy, but they often get pushed aside since I need to keep the servers running as a priority.

Now CALI is hiring a systems administrator to take over (or clean up) the running of our infrastructure. I’m looking forward to handing the keys of the cloud over to someone else so I can focus on all of the great projects that are in the pipeline. When can you start?

Details on the CALI sys admin job, which is located in our Chicago office, are at http://cca.li/6J.