[UbuntuFilter] Hate Computers, Love Computer Science
July 4, 2011 9:04 PM Subscribe
I have recently become the defacto sysadmin for a number of Ubuntu machines, and it has become painfully obvious that I don't really know what I'm doing. Can you recommend resources for me to better understand how Linux works? Please note that I am not at all interested in this for its own sake, but only because I need to know this to be able to do the fun parts of my job.
(FWIW, I'm a grad student and these machines are on my robot, so there's not really anybody else to go to for help.)
I am comfortable using Ubuntu to develop code, and have no problem installing software. Even though it's painful, I'm proficient at using google to sort out that level of error messages. However, I'm struggling more than I should with tasks that require me to mess with setting up networking and device drivers and the like. I don't know enough to know where to start, or what the side effects of changes are likely to be ... it seems to be a Bad Thing to have sudo privileges and be mucking about with the BIOS and the kernel without really knowing what's what.
Examples of what I've struggled with:
* getting a FireWire camera to work. Seems to involve problems with the kernel and permissions.
* networking. Ugh. Getting one computer to forward packets between other computers on the local network and the rest of the Internet. Debugging a flaky wireless connection with no apparent pattern to the flakiness.
* it took me way too long to figure out how to get the machines to boot without a keyboard attached.
So - what resources have helped you to understand how Linux works under the hood? I'm not looking for super-detailed tomes. I just want a basic understanding so I won't be so baffled by the next problem that comes up. Readability is a big plus.
(FWIW, I'm a grad student and these machines are on my robot, so there's not really anybody else to go to for help.)
I am comfortable using Ubuntu to develop code, and have no problem installing software. Even though it's painful, I'm proficient at using google to sort out that level of error messages. However, I'm struggling more than I should with tasks that require me to mess with setting up networking and device drivers and the like. I don't know enough to know where to start, or what the side effects of changes are likely to be ... it seems to be a Bad Thing to have sudo privileges and be mucking about with the BIOS and the kernel without really knowing what's what.
Examples of what I've struggled with:
* getting a FireWire camera to work. Seems to involve problems with the kernel and permissions.
* networking. Ugh. Getting one computer to forward packets between other computers on the local network and the rest of the Internet. Debugging a flaky wireless connection with no apparent pattern to the flakiness.
* it took me way too long to figure out how to get the machines to boot without a keyboard attached.
So - what resources have helped you to understand how Linux works under the hood? I'm not looking for super-detailed tomes. I just want a basic understanding so I won't be so baffled by the next problem that comes up. Readability is a big plus.
I would start sifting archives or asking questions on ServerFault (sysadmin version of StackOverflow Q&A site).
posted by xtine at 11:23 PM on July 4, 2011 [2 favorites]
posted by xtine at 11:23 PM on July 4, 2011 [2 favorites]
My Linux rules of thumb:
1. Device problems: try the most recent release of your distro of choice. If that doesn't help, try installing the most recent available kernel into the most recent release of your distro of choice. If that doesn't help either, put something else on the front burner for a couple months and try again later.
2. Everything else: if you can't figure it out on your own, even when wearing the googles, ask for help. Right here is as good a place as any.
The "not knowing what the side effects of changes are going to be" thing is one of the major reasons I switched from the Red Hat family, which tends toward Everything And The Kitchen Sink in a default installation, to Debian, whose default installation is nicely minimal; Ubuntu is somewhere in between.
It's much, much easier to gain a justified confidence that you know what's going on if most of what's running on the machine is stuff you chose to put there.
There are many, many ways to do the Internet routing thing, and which one you choose will depend a lot on how you want your network to function. If this is still a problem for you, can you describe your setup in a bit more detail?
posted by flabdablet at 1:19 AM on July 5, 2011
1. Device problems: try the most recent release of your distro of choice. If that doesn't help, try installing the most recent available kernel into the most recent release of your distro of choice. If that doesn't help either, put something else on the front burner for a couple months and try again later.
2. Everything else: if you can't figure it out on your own, even when wearing the googles, ask for help. Right here is as good a place as any.
The "not knowing what the side effects of changes are going to be" thing is one of the major reasons I switched from the Red Hat family, which tends toward Everything And The Kitchen Sink in a default installation, to Debian, whose default installation is nicely minimal; Ubuntu is somewhere in between.
It's much, much easier to gain a justified confidence that you know what's going on if most of what's running on the machine is stuff you chose to put there.
There are many, many ways to do the Internet routing thing, and which one you choose will depend a lot on how you want your network to function. If this is still a problem for you, can you describe your setup in a bit more detail?
posted by flabdablet at 1:19 AM on July 5, 2011
I would suggest a tutor, or someone you can interact with and bounce questions and ideas off of and get an immediate response. I don't think book learnin' is going to help you here, you need guidance from someone with experience.
Online forums can be helpful for answering very specific questions, but gaining general knowledge of the underpinnings linux can get lost in the tranlsation very quickly because nobody knows what you know or don't know.
You are on a college campus? Put up some flyers in the CS building asking for a Ubuntu tutor/mentor. something along the lines of "help me build my robot" would probably catch some talent. you could put the same on craigslist or any other local/campus want ad site.
posted by j03 at 2:39 AM on July 5, 2011
Online forums can be helpful for answering very specific questions, but gaining general knowledge of the underpinnings linux can get lost in the tranlsation very quickly because nobody knows what you know or don't know.
You are on a college campus? Put up some flyers in the CS building asking for a Ubuntu tutor/mentor. something along the lines of "help me build my robot" would probably catch some talent. you could put the same on craigslist or any other local/campus want ad site.
posted by j03 at 2:39 AM on July 5, 2011
I think j03 has it - the kind of things you're talking about are pretty simple for a competent unix sysadmin, but the amount of time you'll spend reading (and understanding) the relevant docs for forwarding/routing, NAT, v4l, etc, you're probably better off throwing a couple of $ or some beers at someone who does that stuff for fun (and will probably be happy to be involved in a robotics project)...
posted by russm at 3:16 AM on July 5, 2011 [2 favorites]
posted by russm at 3:16 AM on July 5, 2011 [2 favorites]
Three letters: IRC. But don't just sit in the huge channels like #ubuntu, those are full of cruft. Find a channel that's smaller and more trustable. Maybe your local LUG? Then just help other people out when you have a few minutes of down-time, and you can expect their help in return.
It doesn't always work, sometimes everyone's busy, or you have a very unique problem. But half the times I've looked like a genius at work, it was thanks to simply asking around on FreeNode.
posted by vasi at 3:26 AM on July 5, 2011 [1 favorite]
It doesn't always work, sometimes everyone's busy, or you have a very unique problem. But half the times I've looked like a genius at work, it was thanks to simply asking around on FreeNode.
posted by vasi at 3:26 AM on July 5, 2011 [1 favorite]
Response by poster: I should've guessed that most people learned this stuff on an as-needed basis. That's what I've been doing up to now, but it just seems like until you gain massive amounts of experience the resulting knowledge is fragmented and disorganized. I figure that some pretty smart people have designed and refined this operating system, so there should be some basic principles behind how it works. I guess I was hoping for a framework that would help me to actually understand the steps I'm taking when I follow advice from a message board, and ideally, help me understand what other problems that particular fix could apply to.
For example, with logs - I've previously been told to check dmesg, as well as a few log files, but I don't really know where to look in different circumstances. What winds up where, and why?
While Linux in a Nutshell looks great for reference, it's definitely not something I can sit down and read cover-to-cover :)
posted by Metasyntactic at 7:08 AM on July 5, 2011
For example, with logs - I've previously been told to check dmesg, as well as a few log files, but I don't really know where to look in different circumstances. What winds up where, and why?
While Linux in a Nutshell looks great for reference, it's definitely not something I can sit down and read cover-to-cover :)
posted by Metasyntactic at 7:08 AM on July 5, 2011
+1 on IRC, forums, and mailing lists. IRC and mailing lists are perhaps the subtlest one. Quite well hidden until you start participating, but absolute gold.
+1 on all the answers here, actually.
Here something I really can emphatically recommend; It's quite involved but teaches you A LOT of stuff:
I learned almost everything I know about Linux by setting up Linux from Scratch. Totally recommended. It's not really a 'distribution', but rather a book. You download all the core packages and compile them yourself and fit together a working Linux system, indeed from scratch. At least try starting the setup and see if it's interesting enough to finish. It gives an excellent high-level overview along with low-level intimacy.
Then I can warmly recommend that you assign sacrificial goat status to one machine. On that one, anything goes. Root access, BIOS wrangling, anything. You can get out of any mess anyway - the most extreme recovery procedure would be a BIOS reset which involves opening the case, setting one jumper, waiting 10 seconds, and setting it back. No problem.
And also, I really like to install Ubuntu from USB sticks instead of CD. You simply format a stick in FAT32, download the ISO of whatever distro you want to install, and use unetbootin to clone the installation CD contents onto the stick and make it bootable. Then you plug it into the machine you're installing to, reboot it, and select the USB stick in the boot menu, which is usually brought up by pressing F12 or F10 at the same time you press DEL or similar to enter the setup. This has the advantage that it is much faster and more responsive, so installing is much less of a chore. Very nice for backdoor access into a machine that you may have hosed :)
And ... Feel free to PM me in the future if you have any questions and don't know where to go!
posted by krilli at 7:22 AM on July 5, 2011 [3 favorites]
+1 on all the answers here, actually.
Here something I really can emphatically recommend; It's quite involved but teaches you A LOT of stuff:
I learned almost everything I know about Linux by setting up Linux from Scratch. Totally recommended. It's not really a 'distribution', but rather a book. You download all the core packages and compile them yourself and fit together a working Linux system, indeed from scratch. At least try starting the setup and see if it's interesting enough to finish. It gives an excellent high-level overview along with low-level intimacy.
Then I can warmly recommend that you assign sacrificial goat status to one machine. On that one, anything goes. Root access, BIOS wrangling, anything. You can get out of any mess anyway - the most extreme recovery procedure would be a BIOS reset which involves opening the case, setting one jumper, waiting 10 seconds, and setting it back. No problem.
And also, I really like to install Ubuntu from USB sticks instead of CD. You simply format a stick in FAT32, download the ISO of whatever distro you want to install, and use unetbootin to clone the installation CD contents onto the stick and make it bootable. Then you plug it into the machine you're installing to, reboot it, and select the USB stick in the boot menu, which is usually brought up by pressing F12 or F10 at the same time you press DEL or similar to enter the setup. This has the advantage that it is much faster and more responsive, so installing is much less of a chore. Very nice for backdoor access into a machine that you may have hosed :)
And ... Feel free to PM me in the future if you have any questions and don't know where to go!
posted by krilli at 7:22 AM on July 5, 2011 [3 favorites]
Best answer: but it just seems like until you gain massive amounts of experience the resulting knowledge is fragmented and disorganized
the primary skill of a sysadmin isn't knowing things, it's knowing how to find things out - the specific facts you pick up are useful the next time you need them ("how do I enable SNAT on Linux using raw iptables?") but will certainly be fragmented in terms of All There Is To Know About Unix. the real skill is getting more comfortable and familiar with the documentation, and constructing google queries that will return useful results, and asking questions in IRC that will attract answers. this is the skillset that you don't have, don't want, and don't need.
I figure that some pretty smart people have designed and refined this operating system, so there should be some basic principles behind how it works
there are, but they tend to be at a lower level than "find the right control panel". your firewire camera issue sounds like it might need tweaking of your udev rules (or perhaps the group membership of the account that's trying to access the video stream). your networking issues will almost certainly be fixed by disabling NetworkManager (aka "network mangler") and configuring things in /etc/network/interfaces except that way you lose the nice pointy-clicky interface. the keyboard thing? that's presumably a BIOS issue and not Linux's problem.
posted by russm at 7:28 AM on July 5, 2011 [1 favorite]
the primary skill of a sysadmin isn't knowing things, it's knowing how to find things out - the specific facts you pick up are useful the next time you need them ("how do I enable SNAT on Linux using raw iptables?") but will certainly be fragmented in terms of All There Is To Know About Unix. the real skill is getting more comfortable and familiar with the documentation, and constructing google queries that will return useful results, and asking questions in IRC that will attract answers. this is the skillset that you don't have, don't want, and don't need.
I figure that some pretty smart people have designed and refined this operating system, so there should be some basic principles behind how it works
there are, but they tend to be at a lower level than "find the right control panel". your firewire camera issue sounds like it might need tweaking of your udev rules (or perhaps the group membership of the account that's trying to access the video stream). your networking issues will almost certainly be fixed by disabling NetworkManager (aka "network mangler") and configuring things in /etc/network/interfaces except that way you lose the nice pointy-clicky interface. the keyboard thing? that's presumably a BIOS issue and not Linux's problem.
posted by russm at 7:28 AM on July 5, 2011 [1 favorite]
Best answer: Your source of information should be:
- syslog. This is a program that handles logging messages from everywhere and dispatches them to log files. Check the documentation for syslog and figure out where its configuration file is, so you know where to find your logs.
- dmesg: prints a buffer system kernel messages, but some of them may be going to syslog and flushed away from the kernel buffers. See syslog above.
- log files for individual programs that don't use syslog: for each program this is different, but check the documentation. As a rule of thumb, if you do not understand what a program is doing chances are it has a hidden configuration option called "debug logging" or "verbose logging" that will increase the amount of information going to the log files. You can enable that, restart and see the app's logs.
- networking: here you really need to read some documentation first. You need some basic theoretical knowledge about, in the following order: network addressing, network interfaces, routing tables, filtering. Not only this order makes it more easy to understand the concepts, the corresponding software entities are also configured in this order in a running system.
- once you understand how your network "should" be, if anything goes wrong then you can use "tcpdump" or "etherreal". This allows to look at individual network streams as an observer (ie from the outside) and figure out what is going on.
- keep bookmarks to all the "good" documentation sites you find on your way. Finding information is an important skill, but being able to come back to a doc that helped you is key. You can't expect yourself to remember all the technical details all the time. On this topic the command "man" is your friend.
posted by knz at 9:25 AM on July 5, 2011
- syslog. This is a program that handles logging messages from everywhere and dispatches them to log files. Check the documentation for syslog and figure out where its configuration file is, so you know where to find your logs.
- dmesg: prints a buffer system kernel messages, but some of them may be going to syslog and flushed away from the kernel buffers. See syslog above.
- log files for individual programs that don't use syslog: for each program this is different, but check the documentation. As a rule of thumb, if you do not understand what a program is doing chances are it has a hidden configuration option called "debug logging" or "verbose logging" that will increase the amount of information going to the log files. You can enable that, restart and see the app's logs.
- networking: here you really need to read some documentation first. You need some basic theoretical knowledge about, in the following order: network addressing, network interfaces, routing tables, filtering. Not only this order makes it more easy to understand the concepts, the corresponding software entities are also configured in this order in a running system.
- once you understand how your network "should" be, if anything goes wrong then you can use "tcpdump" or "etherreal". This allows to look at individual network streams as an observer (ie from the outside) and figure out what is going on.
- keep bookmarks to all the "good" documentation sites you find on your way. Finding information is an important skill, but being able to come back to a doc that helped you is key. You can't expect yourself to remember all the technical details all the time. On this topic the command "man" is your friend.
posted by knz at 9:25 AM on July 5, 2011
My bookshelf at work has two books I've found invaluable: Operating Systems by Tanenbaum. Aka the Minix book. Computer Networks by Tanenbaum. These two books cover a lot of IT infrastructure. I acquired these books as part of my CS grad student cirriculum, and it's been a good choice. They probably won't solve your problems, but they'll start you down the right trail via citations. I don't own it yet, but Limoncelli's The Practice of System and Network Administration is a book I'd like to spend some time reading, some day.
Your problem, I suspect, is that while the above books are timeless, debugging firewire udev camera rules are not. Linux is an ever evolving beast, and what worked in 2008 is liable to be replaced with something different. Your best bet really, isn't a book or website. It'll be the IRC channel for wplug.
posted by pwnguin at 4:42 PM on July 5, 2011 [2 favorites]
Your problem, I suspect, is that while the above books are timeless, debugging firewire udev camera rules are not. Linux is an ever evolving beast, and what worked in 2008 is liable to be replaced with something different. Your best bet really, isn't a book or website. It'll be the IRC channel for wplug.
posted by pwnguin at 4:42 PM on July 5, 2011 [2 favorites]
Response by poster: Huge thanks to everybody who answered, especially those who offered help beyond the scope of my question! (I didn't want to provide any more technical details of what I can't get to work in this thread, as it would have seemed like piggybacking on my own thread. Next week, if I'm still stuck =). )
Rather than getting want I thought I wanted, y'all have pointed me to the resources that I actually need. AskMeFi is awesome for that.
posted by Metasyntactic at 4:54 PM on July 5, 2011
Rather than getting want I thought I wanted, y'all have pointed me to the resources that I actually need. AskMeFi is awesome for that.
posted by Metasyntactic at 4:54 PM on July 5, 2011
« Older I left my heart in Northern Cal, but just for the... | Help me identify a forgotten hero Newer »
This thread is closed to new comments.
That said, you really could do with a nice chunky reference book, and O'Reilly's Linux In A Nutshell is a decent place to start for Linux-specific stuff. You might also benefit from Essential System Administration, which is less concerned with specifics than fundamental concepts, especially for things like ownership/permissions and networking.
A sysadmin lives and dies by the logs. Learn where the important logfiles arefor the things you want to set up and control (dmesg may elucidate the camera issues), learn how to search through them for relevant messages (grep is your friend) and then take those messages either to the Google or to a forum.
posted by holgate at 10:16 PM on July 4, 2011 [1 favorite]