The other parts of programming

April 27, 2005 2:59 PM

Programmers: where did you learn all the aspects of programming besides actually writing code?

Finding information on writing code is easy enough, but I've lost track of how many times I've taken somebody else's tutorial code, for instance, and found myself unable to compile it successfully. How do I grok the various compiler and linker options? How do multiple source files interact? What's a makefile? More generally, how do I set up my workspace and directory structures to streamline my workflow?

I'm not so much asking for answers to these specific questions, though I'll accept any wisdom you've got lying around. I'm just baffled at how most info I've found seems to assume the reader already knows all this stuff. Incidentally, I'm running win2k and have both VC++ 6 and Bloodshed Dev-C++.

posted by squidlarkin to computers & internet (14 comments total)

One thing is learning the tools themselves; you mention makefiles, the format and workings of which can be studied and learned independently. But the most time-consuming aspect is often the process, especially the conventions, and the glue between all the tools you need to use.

It's easy to learn how to compile a C program; it's harder to know how you use automake, autoconf, make, GCC, the linker, etc. together to produce a finished package complete with documentation/man pages, changelog, and so on. How do you arrange and structure the source code, for example? Learning C is dead easy; C++ less so, but in the end it's just a language; but learning to develop efficiently in that language, picking up well-established techniques and patterns, takes time.

This problem is not limited to Unix programming, which is oriented around C/C++ and tons of conventions which often exist because the languages are so low-level; compared to systems like Java or Ruby where you never need to deal with linking, and cross-platform support often comes gratis. (One reason you may have problems compiling tutorials is precisely because you don't have the right dependencies: the right include files installed, or perhaps not the platform the code was designed for.)

Many of those conventions are documented; there's a GNU ChangeLog format, the man format is documented, there's the Unix Programming Frequently Asked Questions, there's the O'Reilly books, and so on. And some you just pick up by reading other people's code or studying their projects. You can read the GNU Makefile manual from cover to cover; but you might need to read a few real-world makefiles to "get" it.

You mention compiler and link options specifically -- and I think that's a bitch for everyone, except perhaps the original compiler developers. VC++ is laughably simple compared to the GCC toolchain, which has lots of options, lots of front ends, lots of back-end support for different platforms and formats, and invokes many subtools (cpp, ld, libtool). Sometimes these fail, and the pipeline doesn't track the context in a way that lets the tools know what went wrong -- more often than not you will get an error which just doesn't describe the actual problem.

In the end, even learning these tools is covered somewhere. The bad part is when there isn't any documentation, or where the documentation is so scattered that the ramp-up time destroys every last smidgen of creativity you have. For example, writing a GCC front end -- which involves generating parse trees and using a barrage of undocumented macros and APIs, where the only way to learn is to study the existing code.

Incidentally, autoconf and automake are designed to automate the compiler stuff. They will generate scripts that construct the makefile for you, based on a number of heuristics, such as what compiler is installed, what the OS is, what libraries and include files are available, etc. You've seen configure scripts, right? That was made with autoconf.

A caveat, though: Even with these tools you will need to know how the pieces fit together; so while they will be useful later on, for now I recommend sticking with manual compilation (with makefiles, natch), until you feel confident enough to delegate the process to the automatic tools. Of course, on small, personal projects, a manual makefile will do fine.

If I haven't answered any of your questions, it's because I can't possibly tell you everything you need to know in a single reply. The answers are out there, if you look. For example, the Wikipedia entry on makefiles.

When I first read your main question, I assumed you were talking about the development process itself. Test-driven development, for example, or how to coordinate a project between multiple developers, all the Mythical Man-Month stuff of managing large projects. For me, writing code is cheap and easy; what's hard is doing it in the minimum amount of time and making the technical decisions that are right for the product, as opposed to being made because they involve cool technology.
posted by gentle at 3:32 PM on April 27, 2005

three letters: IRC

If they aren't willing to help you, they'll at least tell you where to get the answer. If you don't have one already, install an IRC client. Then connect to a net such as freenode (good for open source and developers) or undernet (where you'll find me) and join a relevant channel such as #c #java or #vb and if you don't find the information in the help files in the topic or auto-join script, then someone in the channel will be glad to help. Just remember to be polite and sometimes it can take a while before anyone answers.
posted by furtive at 3:48 PM on April 27, 2005

The internet is your friend, but for a basic introduction to programming and comparitive languages, then the following two books are invaluable.

Code Complete
The Pragmatic Programmer

For specifics like "How do I make a makefile?" and "How dd COM objects work?", you'll need to either search t'internet or buy a book on that specific subject.

Note: Although Makefiles may seem quite basic to the art of programming, they encompass a huge subject. Don't assume that all programmers grok them. I don't (I don't really need them).
posted by seanyboy at 3:56 PM on April 27, 2005

Iterative error and improvement. The stuff you listed is hard to learn all in one go. And then it all changes in the next release of the tool anyway. Skim through the man page of whatever tool your using every once and a while, and experiment with useful-looking stuff.
posted by grouse at 4:20 PM on April 27, 2005

Though I'm not sure how relevant these answers will be to you squidlarkin as they are Unix centric.

"How do multiple source files interact?"

Depends on the programing language. In Java, assuming your CLASSPATH variable is properly setup the java "compiler" will handle all dependencies vis a vis compiling needed files. Now take note, I said assuming. setting up a proper CLASSPATH is always a pain in the ass.

In C it gets more complicated. C header files define variable, functions and other symbols. So when in your .c file you say "#include blah.h" you're telling the compiler "look, I know there are a bunch of function names and variables you don't recognize but if you look in blah.h file you'll see their types and such". Now the next question is, where does the compiler look for blah.h? Well, by default most compilers check the directory that your .c file is in. You can specify other locations with command line switches (typically -I path).

C code, .c files, gets compiled into object files, .o files. The next step if for the linker to put all the object files into a library or into an executable.

"What's a makefile?"

So above I described a two things. Compiling source code into object code. Linking object code into a library or executable. Those are just two steps. They both have a lot of command line options (like -I path) and generally there are a lot of them between your source code and a finished executable. Keeping track of all that is just a pain. Sure, you could write scripts to automate the task but every new coding project you'd have to redo em. Makefiles are a generic way of handling all that messiness automagically. That's what makefiles are.

"How do I set up my workspace and directory structures to streamline my workflow?"

This I can't really help with. Windows is very different in this respect. Or so I've heard.

Some suggestions:
http://en.wikipedia.org/wiki/Object_code
http://en.wikipedia.org/wiki/Linker

I know they seem very technical and look like they have little to do with writing source code but the more you understand how source code goes from text to executable the clearer you'll get on why things are the way they are. Plus any extra knowledge helps the debugging process. Trust me.
posted by cm at 5:19 PM on April 27, 2005

Oh, I can't say enough good things about How to be a Programmer (PDF, HTML). Only 50 pages and covers everything nicely.
posted by furtive at 5:49 PM on April 27, 2005

it sounds like you're looking at unix projects on a windows machine. this is not a good combination. you might look at cygwin or mingw as possible solutions, but if you're starting out programming it's probably better to look for windows-specific projects that mention MS Visual tools.
posted by andrew cooke at 5:50 PM on April 27, 2005

If you weren't a windows only programmer, I would say purchase every W. Richard Steven's book and read them cover to cover. They will teach you the internals of unix and TCP/IP.
posted by about_time at 6:42 PM on April 27, 2005

You need a trustworthy friend. You also need to locate online groups of people using the technologies you want to use (be it via IRC, Usenet, web-based forums or whatever). Seriously, we all stand not just on the shoulders of giants, but on the shoulders of people who are just a wee bit taller. I personally learned most of what I treasure now from programmers I worked with or studied with who were helpful to me. Almost all people who are good at things like to talk about it. People who guard their foo jealously are not people you want to be like. (The converse is not true. Many talkative people, maybe even your helpful correspondents here, may only have a weak grip on the matter at hand. However, learning how to sort us out is something that you can only master by experience also.) Anyway, be humble and stupid and people will help you.

Cultivate extreme determination and google-fu. If things don't work, why not? Doggedly pursue loose ends and oddities until things bloody well do work. None of the dead ends or side tracks will be wasted in the long run.

Try and develop an aesthetic sense. Does what you wrote feel ugly? Would you be happy to show it to someone else? Is it easy to reuse on another project? If not, how would you change it? Why don't you change it? In fact, how about doing it that way in the first place?

Consider picking up a scripting language (Python, Ruby, Perl, Javascript, whatever) in addition to C++. C++ has an extremely steep learning curve, I think. If that's not manly enough, try Java; even plain C might do. It would help you a lot to gain proficiency in an environment where you can learn the real basics of data structures, algorithms and modelling fairly easily. Then you can apply it in the cruel, nasty and above all huge world of C++.

Note to Unixoids - I hear they have make in Windows land too.
posted by i_am_joe's_spleen at 2:21 AM on April 28, 2005

Another recommendation for Code Complete. Marvellous book. Also I found this dictionary of terms invaluable.
posted by tommyc at 4:26 AM on April 28, 2005

On the larger questions of how to organize a project, how to use objects, and so on, questions of personal taste come into it. There are many very different ways of doing things, many of which work well enough. Particularly in C++ there are many ways of doing things. Take any advice you find in the many books on the subject as good examples to learn from, but remember there's no definitive answer on many of these things.

If you're making a career of it, you'll probably have to deal with many different types of projects. You can (and should) develop your own style, but learning to work with others is something you get only with years of experience.
posted by sfenders at 5:00 AM on April 28, 2005

I hear they have make in Windows land too.

yes, i know. i was mislead by the discussion of autconf etc which appeared in an answer somewhere. sorry.

if you're using window's make then read window's help. both within VC and on msdn. in general, windows tools are pretty well documented by microsoft, although it can take a bit of poking around to find.
posted by andrew cooke at 11:13 AM on April 28, 2005

Re makefiles: this is a huge subject.

I learned makefiles by reading the GNU make manual from cover to cover. I did this with a specific complex build problem in mind. You can read other guides, but you really have to learn-by-doing to ingrain it into your memory.

As others have said, it really helps to know the basics of a scripting language, e.g. perl. I recommend studying Bourne-shell derivatives (ksh, bash) as well, since you will (should) be calling those from GNU make to run your commands.

I then had questions I couldn't answer, specifically about GNU make, not about the compiler. I asked these questions on the [e-mail] mailing list, giving small toy makefile examples to illustrate the problems. I started corresponding with Paul Smith, who later took over as GNU make maintainer. Here are some helpful sites:

GNU make page

Recursive make considered harmful

Cascade recompilation problem.

At the moment I can do any build procedure I want to in GNU make, with one exception -- I can't do a distributed parallel build on a linux cluster. But with machines as fast as they are these days, compiling 10K fortran routines doesn't take as long as it used to.

One problem you're going to find on windows, though, is that you are isolated from the mechanics of the build. You're never going to understand the mechanics if you never open the hood.
posted by Araucaria at 11:41 AM on April 28, 2005

Another vote for Code Complete and The Pragmatic Programmer.

But also... my advice: you can learn an awful lot from working examples.

Go to sourceforge.net or tigris.org and download a few open source projects that seem like they're trying to accomplish something that you yourself have tried, or are trying to do (so that you have a somewhat common frame of reference). After you've downloaded and decompressed the source files, explore the project structure.

Read the makefiles. Look at the directory structure. Open the code files and look at how they're structured. Read whatever comments exist.

Join the support message boards or developer IRC threads for those projects and read the conversations for a while to try and get a sense of how the project evolved(/is evolving).

If you are strictly a Windows/"Visual ???" developer, you may have a more difficult time finding an appropriate open source example (I honestly don't know though).
posted by C.Batt at 12:43 PM on April 28, 2005