Recommendations for advanced linux system administration training?

March 22, 2013 10:48 AM

I've recently come up against several vexing linux system administration issues that have our company outsourcing solutions to other people. I'm looking for advanced linux system administration courses that go beyond installing packages and installing/configuring/starting and stopping services.

My routine for the past 15 years has been:

- A problem comes up on a linux server.
- Google the error message.
- Implement whatever fixes Google returns that don't look too sketchy.
- If that doesn't work, go to the next Google result.

When I was just learning linux system administration, I was fresh out of college and had plenty of time to pursue learning what I loved in an environment where "mission critical" wasn't a phrase I had to worry about. I could iterate through the above scenario all day.

Now, having to iterate through the above scenario only to solve the problem 50% of the time is tiresome. Having to constantly go to clients without a solid explanation of the problem makes my company look incompetent and is professionally and personally deflating. I want to be better at this, but I don't have the time or energy to dedicate to getting better at it when I'm working 50+ hours a week with clients.

Thankfully, my company provides each employee a professional development budget. I'd like to find a training program that takes me away from my client duties and out of the office and gives me time to focus specifically on my linux system admin skills.

I've taken several entry level classes. I'm very comfortable on the command line and can do package management across several common distros. The training courses I've found have been remedial to what I can already do. I'm looking to learn strategies for dealing with the unknown.

Things I'd like to to learn:
- Techniques and tools for debugging services and issues you're not familiar with.
- Understanding standard system services logs (what services write to which log files and when).
- Understand linux filesystem management (mounting and unmounting file systems, NFS).

Traveling for training is, oddly enough, preferable as I can physically extricate myself from the office and the stresses found there. Virtual training is good too.

posted by anonymous to computers & internet (6 answers total)

This may be an unorthodox answer, but my troubleshooting skills and understanding of the GNU/Linux ecosystem increased exponentially over the month or so that I spent setting up Gentoo on a test server years and years ago - from scratch, the hard way - all the way to having a stable, optimized installation running a variety of applications compiled by hand. No reading, testing or training I've engaged in before or since has come close.
posted by eschatfische at 11:21 AM on March 22 [2 favorites]

Red Hat has various training courses for many levels of prior ability, including multiple classes in the categories of core sysadmin skills and advanced sysadmin skills. The main page is here.

Full disclosure: I work for RH, but not in that part of the company. I'd normally avoid pepsi-blue here, but it seems like a fairly direct answer.
posted by atbash at 11:32 AM on March 22

I have delivered Linux Professional Institute (LPI) courses, it is a good syllabus and will take you through the things you ought to understand.

Self-taught system administrators are like self-taught drivers. So I long as I don't have to share the road with them I have no problems. The trouble with your piecemeal approach is you may not appreciate the side effects of what you are doing and may introduce problems which lurk for a while then bring servers down in the middle of the night. Anyway, how are you going to Google for it if the network is down? Google can't teach you a disciplined and methodical approach to fault diagnosis (neither can LPI, a problem with all these certifications).

The following books should be on your shelf, "Linux Administration Handbook" by Nemeth et al, "The Theory and Practice of System and Network Administration" by Limoncelli et al and "Linux Quick Fix Notebook" by Harrison.
posted by epo at 11:47 AM on March 22

I second the Gentoo suggestion. Consider it a sort of 'guided' linux from scratch - which is a great, somewhat accelerated way to learn what-does-what-and-why because YOU're the one setting it up.
posted by destructive cactus at 12:31 PM on March 22

Another good place to firm up your toolset is with the command line toolset. Knowing a little bit about the shell utilities will help a great deal when you're poring through log files looking for the needle. You needn't be an expert, but at least a passing acquaintance with the likes of sed, awk, cut, uniq, wc, sort and some of the others will server you very well. Basic shell-scripting is also a very useful skill to have, and if you want to make it a bit more challenging, stay with the Bourne shell (/bin/sh) rather than BASH. Generally, you want to become something of a toolsmith.

Having a box (or VM) of your own to pound away on is probably the best way to keep your chops up, and any old x86 bit of hardware will do.

More detailed troubleshooting tools: strace (for peering into running processes) and tcpdump (for spelunking into network traffic). Between those two, the shell stuff listed above, and a decent central logging facility (you are logging everything to a syslog server, yes?), you've got a fair bit of what you need to troubleshoot almost anything.
posted by jquinby at 12:59 PM on March 22

What are your company's core competencies? What are your own core competencies?

I ask, because the best answer may be to come up with a more formal arrangement with your outside SA resources and your clients about response and resolution times. When there is a problem, you see if you can fix it yourself within X amount of time. If you can't, you bring in the big guns.

This doesn't preclude additional training on your part, and, you might even work something out with your outside resources for training, and/or consulting on non-time-critical issues in a way that lets you learn from them.
posted by Good Brain at 4:04 PM on March 22