Network administrator advice
December 28, 2009 6:27 AM   Subscribe

I landed my first Network/System Administrator job! Looking for pointers and suggestions.

I have been in school for the past 2 years and on a paid internship for the past 9 months. I received my A+ Cert a few months ago and will be taking Net+ in two weeks. I have a good overview of how servers operate and I'm pretty good on desktop and laptop hardware.

I will be working for a small company - 20 desktop/laptop and 15 servers. Pretty much 100% Windows environment, a few Mac desktops but I'm not sure if they are tied into the domain.

After about 2 months it will be a 1 man show. For all the System/Network Administrators who have a good bit of experience, what pointers can you give to a new IT guy? What are some of the things you wish you knew when you started?
posted by bsexton to Computers & Internet (25 answers total) 23 users marked this as a favorite
 
What do you have in place or will you have in place for monitoring?

Key: backups, monitoring (hardware and software) and automation, unless you want to go crazy.
posted by mckenney at 6:44 AM on December 28, 2009


Response by poster: @mckenney not sure what is in place there now. I've used Spiceworks for inventory/light monitoring.

Do you have any suggestions for monitoring?
posted by bsexton at 6:48 AM on December 28, 2009


backups as said before. asset management as well. you'll spend the first month or so just getting used to everything. use this time to find out what's breaking all the time - what needs fixed and get it fixed. learn to be proactive instead of reactive. get phone numbers for the important vendors and keep them with you (ISPs - hardware service and etc). get something to help you go through the log files from a central point every morning.
posted by jaythebull at 7:07 AM on December 28, 2009


Response by poster: These are great thanks so much!
posted by bsexton at 7:07 AM on December 28, 2009


The book network warrior, while cisco LAN focused, has a great section on this.
posted by anti social order at 7:54 AM on December 28, 2009


Before the other guy leaves, have a frank, nonjudgmental discussion about all the gotchas/exceptions floating around:

- People who have special remote access privileges
- People who have admin access to important machines
- Unlicensed software
- Machines that aren't part of your main network
- Any software services your employees may have subscriptions to (even one-offs)
posted by mkultra at 7:54 AM on December 28, 2009


Best answer: Ignore what everyone else said about 'implement this' and 'install that'. Yes, Backups Are Important. Don't get me started on monitoring -- if you MeFiMail me, I can give you horror stories about a group that I collaborate with that doesn't have any monitoring infrastructure in place. You're not even at the place in your job where you can make a budget. How can you possibly write a budget if you don't have a thorough understanding of your craft and, more importantly, your local installation? Everyone's got good suggestions, but as a notice sysadmin, you need to have a foundation down before you do anything, or else you (and those that follow you) will regret it. Don't fall into the trap of becoming a firefighter. People will want you to come right in and start fixing stuff the last guy never fixed. You'll get to it, in time. Make sure you have a solid foundation, then make things better. Never try to hit a moving target.

In my area, reputation is everything, the world is small, and word travels fast. Do things right, because you only get one chance at this.

There is only one thing for you to do at first, and that is to read these Tom Limoncelli books:

* The Practice of System and Network Administration, Second Edition
This book is the BIBLE of System Administration, written by some really, really smart people. Tom Limoncelli knows his stuff, and if you ever get the chance, go see one of his talks. Read that book cover to cover before you even think about implementing ANYTHING. There are right ways and wrong ways for a sysadmin to do stuff, and most times the wrong way is the way that shows itself first. Don't fall in to that trap. I have been a professional unix sysadmin for 12 years now, and I read that book every few months. Each time I read it I find ways to improve my infrastructure and make my life better.

* Time Management for System Administrators
Since you say you are going to be a one person shop, this book is a requirement. It's too easy to fall into the trap of firefighting and damage control, and you will never really get anything done. Manage your time with intelligence and care, and you will be able to achieve all the milestones that you set for yourself.

OK, so you've done that. Now what? DOCUMENT (oh, how I miss the BLINK tag!). Unless you're burning down the building and starting over, document your current environment. Do it in a wiki, if possible. If not, a text file is fine, but a wiki is better. Spend your first week or two just looking and documenting. There's a very good chance that since it's a small shop, documentation is probably non-existent. Document some more. Get your predecessor to help you. Document all your current processes. Cut and paste your scripts into your wiki. Draw diagrams. Inventory every computer, every database, as much as you can. Know what you are responsible for. Interview your users, determine what they do, and how they use IT to do it. Don't ask them how you can improve anything, that's still way ahead of you.

If you've done these things, I would guess that you would have a solid foundation to be able to figure out what needs to be done, what is working, what is broken, and you can begin to formulate your near, middle, and long term project plans. THEN, and only then, should you think about adding or changing infrastructure.

Oh, and join these groups:
The League of Professional System Administrators
The Usenix SIG for System Administrators

The people in these groups can be freakin' geniuses, and you will be able to bounce all your crazy ideas off them before going to your colleagues.

Go to the appropriate conferences for your area. For me, it's the USENIX Lisa conference. Awesome, awesome conference. This is training that you can really use. I think if you join USENIX and SAGE, you get access to all the classes. They are an invaluable resource to me.


Remember, ALWAYS BE DOCUMENTING!
posted by Geckwoistmeinauto at 8:07 AM on December 28, 2009 [8 favorites]


As far as monitoring, I don't know about the Windoze world, but two systems that have worked great for me in the past are Zenoss and Nagios. I have monitored all sorts of systems with Zenoss and Nagios, and for 15 servers, you can use Nagios to monitor your entire infrastructure from network devices to printers to desktops, probably set it up in a day, and it won't cost you a dime. Paying for monitoring software is a ripoff.
posted by Geckwoistmeinauto at 8:10 AM on December 28, 2009


Once you get settled you may want to think about beginning the process of virtualization. Both VMware Server and Microsoft Hyper-V are able to take physical machines and turn them into virtual devices. This process has allowed me to retire 3 old servers and have them all run on one newer box.

Of course you have to be careful with resource utilization and running a virtualized domain controller is still tricky but being able to simplify the rack was a huge improvement where I am.
posted by talkingmuffin at 8:11 AM on December 28, 2009


Antivirus and antimalware! What do they have now? How up to date are the software and the defs? How do they keep them up to date?

Belarc Advisor is a good inventory and reporting tool. You can try the personal version for free, but you are required to pay for the commercial licence.

Be sure to build an emergency boot disk for every flavour of machine in your environment in case anything goes wrong.

How does your company organise all the cables, fasteners, and other spare parts that inevitably accumulate? Life will be easier if the spare parts are out of the way when you don't need them, but readily accessible when you do.

Similarly, how do they store the installation disks now? Software is expensive and theft-prone. Good IT departments organise their software and licences like a well-ordered library in a secure location.

Does your company have accounts with the local computer stores? If the CEO needs some more RAM before he leaves for a flight, is there anyplace where you can pick some up quickly?

Does your company use Active Directory? When it works, it's a great tool for central administration. When it doesn't work, it's a pain in the ass. Careful planning, and design helps a lot in an AD environment.

Now that you have an A+ and a Net+, the next step is to get an MCSA, MCSE, and MCDST certification. They require a lot of studying, but they make you more valuable in the long run. If you really want the big bucks, take a look at the CISSP certification.
posted by Multicellular Exothermic at 8:18 AM on December 28, 2009


Study the Bastard Operator from Hell archives for ideas and examples of proper techniques for interacting with your users. He's been full of great advice lately, but the how-to guides and articles he published early on had a certain quality to them that can only be described by someone better with words than I.
posted by fore at 8:47 AM on December 28, 2009 [1 favorite]


Everything you do will take approximately 1.5 times as long as you think it will. It doesn't matter if you take this into account in your estimate.

Don't be afraid to spend your company's money when appropriate. My biggest failures as a new sys admin mostly had to do with learning or doing things the hard way because I didn't feel qualified to tell people they should spend 50K on something to make it easy. Ask for what you think is a good idea - let it be your boss's job to say you can't afford it.
posted by dreadpiratesully at 9:01 AM on December 28, 2009


Users are idiots.
It's a fact.
They'll completely ignore your hard-written documentation, click the wrong thing even though you've told them 5 times that week not to do it. Be completely incapable of figuring out that "Second Floor Printer" is, in fact, the printer on the second floor.

This is not a value judgment. I have no doubt my mechanic thinks I'm a gibbering fool. The roofing guy probably thinks any half-way competent male should be able to figure out how much roofing shingles they need and don't even get me started on the plumber.

Point is, after a while, it'll get to you, so don't become "that guy"
Don't be dismissive of your user's problems. Continue to educate them, even though it has proved futile in the past. Don't ditch their phone calls, ignore their email, or dodge them in the hall.
The more visible and proactive you can be, the better your relationship (and thus, your day) will be. In short, treat them like you want your mechanic/plumber/contractor to treat you.

And whatever you do, never, ever, say yes to "I have a computer at home acting funny. Can you take a look at it?".
posted by madajb at 9:20 AM on December 28, 2009


On the subject of backups: don't just test the backup process, test the restore process as well
posted by nomad at 9:40 AM on December 28, 2009


Use Python on as your language to write most, if not all, of your administration scripts. Its great for both Windows and Mac (and other platforms that might surely pop up).
posted by cowmix at 10:19 AM on December 28, 2009


Best answer: Look, all of the advice above is good, but don't go charging in to re-invent the wheel. Unless you're building an organization from scratch you won't be doing most of what is articulated above because there's either a process in place or no budget to redo things the way you think they should be done (and quite honestly, you don't have the experience yet to articulate new ways of doing things if there's already a working process in place).

Your job is to be a sponge and to take on as much knowledge as you possibly can in as short a time period as possible. That means you will sit there, ask questions and call out documentation deficiencies as you go so that you're not caught out in the wind and you can articulate where the gaps are before you're handed the mess (that will inevitably exist) on your own.

Start with the basics:

Do you have a solid and accurate inventory list that is in a maintainable format.
Do you have support contracts for the critical pieces of infrastructure.
Do you have licenses for all the software in use on the supported machines.
Does the business have their critical business processes articulated, and do you have a list of the owners and decision makers around those processes who will need to sign off on changes that are required?

Start with those 4 questions and go from there. Based on the last one prioritize everything else you might do (backups, upgrades, automation, etc). It will be a mess, and I'm not trying to be rude but you don't know what you don't know, consequently your main focus should be to get as much written down as possible so you have a sense of where the gaps are before there's an outage and those gaps become your problem with no recourse to the business.

You basically have a years worth of experience, what you have is an opportunity to get a lot more, but be willing to be flexible and learn what the priorities are from a business perspective and learn how to articulate risks to those priorities if the business doesn't fund their support appropriately.
posted by iamabot at 11:17 AM on December 28, 2009 [2 favorites]


Use a wiki to collect everything you may need more than once: server inventory, shell commands, path locations, calls, anything and everything.

Monitor servers remotely. Nagios is free, others work well.

Help users remotely. Team Viewer is free.

Automate sysadmin tasks, using bash or python or whatever. Write what each one does in the wiki.

Use version control for your scripts: cvs, svn, github, whatever.

Be nice, and never be a know-it-all like Geckwoistmeinauto. There are way too many loud mouth, know everything, my way or the highway sysadmins. Know that you don't know everything, and always be open to new ideas and new ways of doing things.

Go to conferences if you can, it's fun to network and see how other people do things.

Don't freak out if servers fail, because they will fail regardless of if you are calm or scattered. Go slow. Be methodic. If you sweep things under the rug it will only come back to haunt you later.

Offsite backups.

Be nice to users.

Good luck.
posted by four panels at 12:10 PM on December 28, 2009 [1 favorite]


I'll add:
1) Set up ticket tracking software, it helps you organize your time and makes you look more professional

2) Use Ghost, etc to make a standard system image you can use on all machines

3) Charge a set amount for helping coworkers with their personal machines. Otherwise helping them may use up a ton of your time.
posted by miyabo at 12:24 PM on December 28, 2009


Response by poster: @four panels I use evernote now for all note taking, scripts and documentation. Are there any advantages to creating a wiki instead of using evernote?

These responses are great. Thank you all!
posted by bsexton at 12:40 PM on December 28, 2009


After about 2 months it will be a 1 man show.

Figure out a backup plan if you're hospitalized or sick or need training or whatever. It can be outside or inside help, but you can't possibly work forever with no time off.
posted by anti social order at 12:58 PM on December 28, 2009


Here are a bunch of random thoughts:

As with any job, the first few months are where everyone around you will form an opinion of you. Have a good attitude, work hard, deliver on your promises. Build trust and friendships with your coworkers (customers) and management. Your good reputation will help to get you the next job.

Some people to like frame everything as marketing versus IT, my group versus your group, etc. Don't let yourself fall into that. Remember that you all work for the same company, and the success of one group over another can -- and will -- be detrimental to all. Also, you never know when there will be a restructuring and you'll end up reporting to some random person you didn't expect. Be professional and friendly to everyone. It will pay off.

Stay at the same company for a couple of years, especially for this first job. Not only is it good for your resume, you will also get to see how your decisions turn out. People who tend to do short stints at each company may make decisions based on the short term which will come back to haunt others later.

As many others have already said, make backups and test the restores regularly. Also, invest in UPSs if your company doesn't already have them. They're fairly cheep and a total lifesaver. The idea is to build a net for yourself because a stitch in time saves 9 (thousand? million?). If you're using Active Directory, invest in something that does online backups of just AD and allows you to restore individual objects online (take a look at Quest AD tools as they've saved me many times). Check your websites, scripts, and (non-wiki) documentation into source control. Have extra hardware on hand. Be paranoid, because no one thing can be trusted.

Deploy vmware or equivalent, especially for Windows servers. Besides consolidating hardware, you also get a virtual IP KVM and the ability to take snapshots. Snapshots are the best thing to happen to Windows.

Learn to write scripts if you don't already know how. If you want to focus on Microsoft products, PowerShell may be good to learn, otherwise I'd go for Perl or Python.

Lastly, don't be an IT nazi. Always treat your customers as adults (even when they don't deserve it). Your job is to enable others to do their jobs, not necessarily to make your own job easier.

And yeah, make sure someone else can cover for you. You need to be able to take time off without constantly worrying about work. Maybe this person is your boss, or possible any random person who's more computer literate than the rest.

Good luck!
posted by DrumsIntheDeep at 1:54 PM on December 28, 2009


as for the advantage of using a wiki:

the software is free, doesn't use much in the way of resources, and can easily be hosted in-house giving you total control over the information.

Wikis are nicer than source control or older document management systems because they're very quick and easy for doing minor changes (no checkout, edit, checkin). They make collaboration easier, and they generally track version changes for you.

Lastly, your documentation can be easily available to anyone in your company, so you could make a bunch of how-to pages for your customers (how do I connect to a printer?).

Your customers probably won't search the wiki and find their own solutions, but you can easily send them the url to answer their questions when they ask you. :)
posted by DrumsIntheDeep at 2:00 PM on December 28, 2009


Hardware:
Buy IBM. Nobody ever got fired for buying IBM.

Actually, buy pretty much any of the top-tier vendors (Dell, IBM, HP). The key here is to not buy crap. People will say the hardware is the same blah blah blah and it probably is -- I doubt there's much difference between an IBM server and a whitebox server. What you're really paying for, though, is the support contract. And, as a one-man-band, you need the support contract. Get the best one you can, that way you're not futzing around trying to find replacement parts when the fileserver has fallen over.

This goes equally for your laptops and desktops -- standardise on a brand, and a model if you possibly can. This will have benefits not only in reduced costs from buying in bulk, but from simplifying your support process.

Network:
- Buy Cisco. See above.
- Have a real firewall. Your ADSL modem doesn't count

Backups:
- Have them. Test them. Restore from them regularly (always something different!) and have someone confirm the file you've restored is valid.
- Have more than one, cos they will fail. EG Volume shadow copy + nightly tape backups + rsync (better, rsnapshot) of vital data to a backup server (encrypted in transit and storage!) gives you three options for getting back the CEO's porn. And he wants it. NOW.

Disaster Recovery:
- Shit happens.
- Develop a Disaster Recovery plan. This is not "backups".

Document:
- What it says on the tin. You need documentation. Preferably centralised doco (eg the aformentioned wiki) so you can get to it from anywhere, and you aren't reliant on having your laptop (cos it just got run over by the bus that just crashed into your building which is on fire and you need to get the business back up and running NOW NOW NOW)
- Wiki's also allow multiple people to edit, and read, them. So when your company has grown and you're the boss of four people, you can just.. point them to the wiki. See also: user education. Depending on how self-reliant your users will be*, you can write how-to docs and just point them to them.

Misc
- Monitoring is good. Nagios and the like will require you to figure out some linux -- this is also good. In fact, making your desktop a linux box is not a bad idea, as it will force you to get hands-on with it. One of the best things that ever happened to me (as a Windows admin) was having a linux desktop for two years -- the mindset is completely different, and you can do things with a linux box you never would've thought of with a Windows box.
- Srsly. Download Ubuntu (or, my preference, Centos), and play with it. Install Nagios. Play with rsync/rsnapshot to take regular incremental backups. Build a firewall (although I wouldn't put it into production unless you have someone who knows what they are doing audit it first).
- Ask questions.
- Listen to the answers.

* Protip: 'professional' individuals (eg lawyers, accountants) will not be self-sufficient. You will be their bitch, expected to do everything up to and including clicking the mouse for them. Engineers tend to be the opposite end of the spectrum.
posted by coriolisdave at 2:02 PM on December 28, 2009


Again, just want to re-iterate things because there are a lot of ideas being tossed about in this thread that are good ideas given specific requirements. You haven't articulated anywhere near enough detail for any of the suggestions around specific technologies to be applicable and so just reading through I can't see much value in them because specific technology/vendor evaluation requires way way more data than you've given.

Beyond my answers above, I wish I learned how to work with the business earlier and that IT isn't the end all be all, in 95% of the world out there IT is a cost center, not a revenue center. Knowing how to be a cost center within a business is really key to making sure you run efficiently and are funded in a manner that allows you to succeed.

Beyond specific technology recommendations and what I wish I knew 15 years ago...

Vendors:
Make sure you talk to your vendors and know them. Get your vendor contacts to give you their support organizational structure so you can escalate when you need to. Make a calender for when support contracts and managed services are up for renewal. When things are almost to within 3 months of renewal start asking for quotes for the next renewal period, get competitive quotes. Use vendors against each other for better pricing.

Monitoring:
Whatever monitoring you do, make sure you're tracking impacts identified via monitoring to costs to the business. Drive monitoring from the perspective of business processes or application availability. Monitoring is an iterative process, it requires a lot of care and feeding to be effective. Anyone who tells you otherwise doesn't know what they are talking about.

Operations:
Define maintenance windows and stick to them (for stability and your own sanity). Try to avoid changing more than one thing in an application stack or business flow in the same window. Make people accountable for the changes they want to make to the infrastructure.

DTRTFT:
Do Things Right The First Time - not always possible, but over time you will encounter an aversion in the business or yourself to DTRTFT, and you may not understand this now, but you will eventually. It _always_ costs more in terms of money, downtime and loss of sleep when you don't DTRTFT..._always_. So take the hit, DTRTFT, and when you're forced to not DTRTFT, make sure you have articulated your objections to the powers that be in clear, coherent and unemotional language.

Downtime:
Everything needs down time. Including you. Be prepared to walk away and find another job if you're not getting it or be prepared to suck as much experience out of the place as you can and when you're not getting any more knowledge, training or advancement, leave. Don't get emotional about it. It's your career and future and if you're being overworked and underpaid the business certainly isn't going to feel bad about it and fix it for you. Know your limits and boundaries for life vs work and be rational about it. Never make a decision about your career at the end of an outage or at the end of a bad meeting. Sleep on it, take a day off and figure out what you should do on a full belly and a good nights sleep.

Career Advancement:
Use your companies performance reviews to your advantage. Make sure your goals are clearly articulated and the path to the next level (raise/title/responsibility) is clearly articulated in your performance reviews. Bosses change, a lot of the time they just disappear over night. Make sure that what the new boss has to work off of (your performance reviews) are accurate and contain your feedback. Make it an HR issue if you have to. Accuracy is important if you plan on sticking around for more than 2 years.

No:
Say no with a smile and always make an effort to meet people half way, you'll avoid a lot of foot stomping from people and at the end of the day get your way more often. Even in instances where you don't get your way, you'll be closer to DTRTFT and you'll build a reputation for being someone who can be worked with and who understands the business needs. when you're forced to do something by the business, again with DTRTFT make sure you articulate objections and consequences in clear, coherent and unemotional language. Especially when it comes to security issues.
posted by iamabot at 4:35 PM on December 28, 2009 [1 favorite]


I just like to second iamabot's "do things right the first time" (part of my recommendation for top tier!), but also to mention that "clearly articulating" your objections should be done in writing. If there's a conversation, where your boss tells you to do something the not-right way? Send him an email followup, immediately, with your recommendation and his request that you do it otherwise.

That way you might (might!) have CYA.
posted by coriolisdave at 1:25 PM on December 29, 2009


« Older Upside down mortgage?   |   recovering traffic accident damages when lawyers... Newer »
This thread is closed to new comments.