I have no IDeaL what I'm doing (IDL programming for absolute dum-dums)
April 25, 2014 4:22 PM Subscribe
So, I recently got selected to do undergraduate physics research this summer. Great! Awesome! Yay me!
However, my research advisor has asked me to use something called IDL to analyze tomographic images of compounds and turn them into 3D images. This is good news, because IDL is used a lot by astronomers to analyze astronomical data, and I want to continue my education in astronomy. But, I've never used IDL ever before. I'm not even sure exactly what it is (a programming language?).
My question is this: how do I get up to speed so that I can hit the ground running with my research?
Difficulty: Research begins in two weeks.
Details:
I have a smidge of programming experience, if you consider one semester of Intro to C++ experience. So, assume I have no programming experience. I find that makes things easier. I have looked at some of the resources available on the web. NASA seems to have wealth of tools and links that I can go through, but there doesn't seem to be a lot of stuff for beginners like there is for other programming languages (environments?). So far I've only found a couple of beginner courses, and they're not free, they're pretty sparse, or they appear daunting. Most of the stuff looks like it was written by professors who were tired of answering the same question from every student about IDL.
I'm a visual learner (which is possibly why I might not be so great at learning programming, too much text), so videos, visually appealing stuff, and lots and lots of examples are what work best for me.
Bonus points if there's something out there that relates directly to the kind of image manipulation I'm trying to do.
I have a smidge of programming experience, if you consider one semester of Intro to C++ experience. So, assume I have no programming experience. I find that makes things easier. I have looked at some of the resources available on the web. NASA seems to have wealth of tools and links that I can go through, but there doesn't seem to be a lot of stuff for beginners like there is for other programming languages (environments?). So far I've only found a couple of beginner courses, and they're not free, they're pretty sparse, or they appear daunting. Most of the stuff looks like it was written by professors who were tired of answering the same question from every student about IDL.
I'm a visual learner (which is possibly why I might not be so great at learning programming, too much text), so videos, visually appealing stuff, and lots and lots of examples are what work best for me.
Bonus points if there's something out there that relates directly to the kind of image manipulation I'm trying to do.
Response by poster: Thanks BrashTech. I'll have access to the school's computers and IDL licenses, and I'm sure I can get some time in on them before the end of the semester. I'm sure my adviser isn't expecting me to know anything at all. Still, I'd like to get a head start.
posted by runcibleshaw at 5:03 PM on April 25, 2014
posted by runcibleshaw at 5:03 PM on April 25, 2014
Have you asked your advisor whether there's anything they would recommend? Speaking as a physics professor myself, I would be more than happy to point my summer research students towards reading and/or tutorials before the summer began.
posted by Johnny Assay at 6:40 PM on April 25, 2014 [1 favorite]
posted by Johnny Assay at 6:40 PM on April 25, 2014 [1 favorite]
Search github for IDL files and try to get something to work. But really, you'll pick it up when you need to learn it. You've got two weeks before you actually need to begin this work and nobody expects you to be an IDL professional right now but you. I think you should go to your public library and find a relaxing book!
posted by oceanjesse at 6:10 AM on April 26, 2014
posted by oceanjesse at 6:10 AM on April 26, 2014
Since you're an undergraduate, I'd say chances are good that your work will initially be writing general logic around snippets of core code given to you by professors, so you'll want to understand how to glue code together and the best way to call/reuse/modify it for your purposes.
I'd start learning Python (when dealing with IDL and MATLAB it's always helpful to have knowledge of less domain specific languages to avoid their more crufty elements) and getting a feel for how code is organized (modules, functions, etc) and how data moves from one part of the code to another (variables, scope, pass-by-reference/value).
posted by RonButNotStupid at 7:26 AM on April 26, 2014
I'd start learning Python (when dealing with IDL and MATLAB it's always helpful to have knowledge of less domain specific languages to avoid their more crufty elements) and getting a feel for how code is organized (modules, functions, etc) and how data moves from one part of the code to another (variables, scope, pass-by-reference/value).
posted by RonButNotStupid at 7:26 AM on April 26, 2014
If you want to be an overachiever, you can document what you learn as you go online as a tutorial for others to follow in your footsteps, since there aren't many existing resources.
posted by TheAdamist at 8:09 AM on April 26, 2014
posted by TheAdamist at 8:09 AM on April 26, 2014
Best answer: I am a professional astronomer, and I have used IDL quite a bit, though not in many years. First, some bad news. IDL is used in astronomy almost exclusively by senior people who cannot be bothered to change to something newer, or deep within data processing pipelines where it would be too difficult to do all the requisite quality control to replace them. It is not at all worth learning on its own: it is proprietary, expensive, and frankly, terrible at what it does. I don't think it is worth the time or effort to write up public documentation on anything about IDL. The reason you're having trouble finding any resources is because it is very much on its way out.
However, as an undergraduate, you don't have a choice in what tool you use, and your goal for your project is likely to learn something about the process of research. So, if your adviser uses IDL, then you will need to use it for the summer. The most salient feature of IDL is that it is an array programming language: it understands N-dimensional arrays as basic building blocks of the language, and can perform linear algebra on these arrays without requiring you to write explicit loops over their indices. As a concrete example: you can add two 2D arrays image_a and image_b by doing
image_c = image_a + image_b
where as in a non-array programming language, you'd have to do something like
for (i = 0; i < n_i; ++i)
for (j = 0; j < n_j; ++j)
image_c[j,i] = image_a[j, i] + image_bb[j, i];
The good news is that you can learn how to use array programming languages OTHER than IDL before you start. This will prepare you for the kinds of operations you will likely be doing in your 3D image reconstruction project.
Luckily, one of the most widely used (and rapidly growing) array programming languages in physics, and especially astrophysics, is Python and its scientific add-on packages. Python is open-source, free, and easy to learn. If you want to get a jump on your summer project, what I would suggest is for you to download python, numpy (which provides fast array objects in python), ipython (an interactive python computing environment that allows web-based notebooks), and matplotlib (a python plotting library). Follow the lectures here:
https://github.com/jrjohansson/scientific-python-lectures
and you will learn a great wealth of techniques and concepts that will likely ease your path this summer. Beyond this summer, knowing python and its scientific ecosystem will be directly useful in a large number of fields.
posted by q9f9A at 9:20 AM on April 26, 2014 [3 favorites]
However, as an undergraduate, you don't have a choice in what tool you use, and your goal for your project is likely to learn something about the process of research. So, if your adviser uses IDL, then you will need to use it for the summer. The most salient feature of IDL is that it is an array programming language: it understands N-dimensional arrays as basic building blocks of the language, and can perform linear algebra on these arrays without requiring you to write explicit loops over their indices. As a concrete example: you can add two 2D arrays image_a and image_b by doing
image_c = image_a + image_b
where as in a non-array programming language, you'd have to do something like
for (i = 0; i < n_i; ++i)
for (j = 0; j < n_j; ++j)
image_c[j,i] = image_a[j, i] + image_bb[j, i];
The good news is that you can learn how to use array programming languages OTHER than IDL before you start. This will prepare you for the kinds of operations you will likely be doing in your 3D image reconstruction project.
Luckily, one of the most widely used (and rapidly growing) array programming languages in physics, and especially astrophysics, is Python and its scientific add-on packages. Python is open-source, free, and easy to learn. If you want to get a jump on your summer project, what I would suggest is for you to download python, numpy (which provides fast array objects in python), ipython (an interactive python computing environment that allows web-based notebooks), and matplotlib (a python plotting library). Follow the lectures here:
https://github.com/jrjohansson/scientific-python-lectures
and you will learn a great wealth of techniques and concepts that will likely ease your path this summer. Beyond this summer, knowing python and its scientific ecosystem will be directly useful in a large number of fields.
posted by q9f9A at 9:20 AM on April 26, 2014 [3 favorites]
Best answer: As a working scientist who grew up using IDL, and still turns to it fairly often thanks to many tens of thousands of lines of legacy code, I have to agree with what others have said. If you were free to pick a language which will be useful in the future, python is the way to go. Don't plan on spending your life in IDL.
But, spending a year or two working in IDL is actually a pretty good way to learn general programming techniques, and you won't be wasting your time.
As mentioned previously, there aren't many visually interesting introductions to IDL, and most documentation is aging.
Do ask your advisor for suggestions. Chances are you're not the first person to ask, and it's likely they've documents or physical books to recommend. Also, they may wind up giving you a quick starter-project to work towards. (For me, trying to learn a new programming language without a specific task at hand seems unbearably dull.)
As far as specific resources go:
The built-in help pages that you can launch from IDL itself ("? [enter]") are actually pretty good, especially if you've seen even a small amount of programming before.
Coyote's guide to IDL programming is a great resources - and it's definitely worth learning to use his "cg" graphics routines, which will save you many headaches. There's not a whole lot of brand-new introductory material, but he does offer detailed recommendations several books that your school library can order through inter library loan.
Carl Heiles has some nice tutorials which I give to new students. They're definitely not multimedia rich, but they're sound and well written: Basics from an undergrad radio astronomy lab (start with idltut1.ps). There are also much more advanced material at his home page.
And, when you get to the point where you're looking for individual routines to do specific tasks, it's worth checking out the GSFC IDL Astronomy user's library (astrolib).
But, if reading documentation isn't your thing, I recommend picking a quick starter project and trying to do it. Consider asking your advisor for suggestions, or making one up yourself. For example, writting a monte-carlo routine that solves a coin-flipping or sock-drawer-selection problem from the first chapter of a statistics book and plotting the results will teach you things worth knowing.
Congrats on the getting selected, and best of luck!
posted by eotvos at 11:02 AM on April 26, 2014
But, spending a year or two working in IDL is actually a pretty good way to learn general programming techniques, and you won't be wasting your time.
As mentioned previously, there aren't many visually interesting introductions to IDL, and most documentation is aging.
Do ask your advisor for suggestions. Chances are you're not the first person to ask, and it's likely they've documents or physical books to recommend. Also, they may wind up giving you a quick starter-project to work towards. (For me, trying to learn a new programming language without a specific task at hand seems unbearably dull.)
As far as specific resources go:
The built-in help pages that you can launch from IDL itself ("? [enter]") are actually pretty good, especially if you've seen even a small amount of programming before.
Coyote's guide to IDL programming is a great resources - and it's definitely worth learning to use his "cg" graphics routines, which will save you many headaches. There's not a whole lot of brand-new introductory material, but he does offer detailed recommendations several books that your school library can order through inter library loan.
Carl Heiles has some nice tutorials which I give to new students. They're definitely not multimedia rich, but they're sound and well written: Basics from an undergrad radio astronomy lab (start with idltut1.ps). There are also much more advanced material at his home page.
And, when you get to the point where you're looking for individual routines to do specific tasks, it's worth checking out the GSFC IDL Astronomy user's library (astrolib).
But, if reading documentation isn't your thing, I recommend picking a quick starter project and trying to do it. Consider asking your advisor for suggestions, or making one up yourself. For example, writting a monte-carlo routine that solves a coin-flipping or sock-drawer-selection problem from the first chapter of a statistics book and plotting the results will teach you things worth knowing.
Congrats on the getting selected, and best of luck!
posted by eotvos at 11:02 AM on April 26, 2014
Response by poster: Thanks guys. Some good advice up there. Some more background: my actual research advisor doesn't actually know much about IDL other than that she used it in the past to turn 2D slices into 3D images. I think I can get some help from the professor who is doing astronomy research, but he's going to be pretty busy with his own research and students this summer.
It's a little disheartening to know some of the drawbacks of IDL. I'm starting to feel like I'm getting put in the dusty file room looking at microfiche rather than doing more current materials research. q9f9A, do you think I should try to push toward using a more modern open-source language like python for this project? I know that the goal of what I'm doing is a little vague, even to me, and that might be a hard question to answer. I don't think my advisor cares how I do the task, just that it gets done. I think she suggested me using IDL to analyze her data because she knows it's used in astronomy and that's my main area of interest, not materials. But, it sounds like IDL might be on the way out and/or gone by the time I'm doing my graduate studies.
posted by runcibleshaw at 12:20 PM on April 26, 2014
It's a little disheartening to know some of the drawbacks of IDL. I'm starting to feel like I'm getting put in the dusty file room looking at microfiche rather than doing more current materials research. q9f9A, do you think I should try to push toward using a more modern open-source language like python for this project? I know that the goal of what I'm doing is a little vague, even to me, and that might be a hard question to answer. I don't think my advisor cares how I do the task, just that it gets done. I think she suggested me using IDL to analyze her data because she knows it's used in astronomy and that's my main area of interest, not materials. But, it sounds like IDL might be on the way out and/or gone by the time I'm doing my graduate studies.
posted by runcibleshaw at 12:20 PM on April 26, 2014
Best answer: runcibleshaw: Oh man. If IDL is only a suggestion, then I would strongly recommend doing your summer research in python. If your advisor is not an IDL user herself, and there is no (or very little) existing IDL code for your task, then that dramatically changes the situation. The community using python for science is at least an order of magnitude larger than that using IDL, and much younger. You can absolutely do the kind of work it sounds like you are going to be doing. Just googling "python materials science" yielded this: http://pymatgen.org/. I have no idea if this will actually be useful to you, but it just shows there are people using python--and developing large complete packages--for every corner of science. I'd strongly encourage you to work through those ipython notebooks I posted a link to above. They're not only a great way to learn about python, they're a nice introduction to computational science in general.
Even if by some bizarre stroke of fate you end up in grad school working on an enormous IDL data reduction pipeline for some crazy, cutting edge instrument (legacy code makes this less implausible than I'd care to think), as eotvos says, it's the concepts that are important, and you'll learn them no matter which language you use.
Also, I can't believe I swapped those indices in that code example above. Now everyone will know how slow my code is.
posted by q9f9A at 2:19 PM on April 26, 2014 [1 favorite]
Even if by some bizarre stroke of fate you end up in grad school working on an enormous IDL data reduction pipeline for some crazy, cutting edge instrument (legacy code makes this less implausible than I'd care to think), as eotvos says, it's the concepts that are important, and you'll learn them no matter which language you use.
Also, I can't believe I swapped those indices in that code example above. Now everyone will know how slow my code is.
posted by q9f9A at 2:19 PM on April 26, 2014 [1 favorite]
Response by poster: Thanks again. I think this has given me a good place to start, and a little context about IDL in the greater code biosphere.
posted by runcibleshaw at 2:26 PM on April 26, 2014
posted by runcibleshaw at 2:26 PM on April 26, 2014
Another professional astronomer, here.
I have used both IDL and python (and perl and C and supermongo and Fortran!) I wouldn't fuss too much about which language you learn right now. By the time that you get into graduate school, a different language will likely be in vogue. The computational landscape will constantly shift under your feet.
I agree that the thing you should learn is how to program. Start with any language that you find convenient right now. It definitely won't be wasted effort. Once you know one, a new language is easy to learn, as long as you have time.
Just get a good recommendation, and put some programming experience on your resume.
You may also wish to check out astrobetter.com. They have a whole IDL vs python discussion there.
Good luck!
posted by pizzazz at 10:16 PM on April 26, 2014
I have used both IDL and python (and perl and C and supermongo and Fortran!) I wouldn't fuss too much about which language you learn right now. By the time that you get into graduate school, a different language will likely be in vogue. The computational landscape will constantly shift under your feet.
I agree that the thing you should learn is how to program. Start with any language that you find convenient right now. It definitely won't be wasted effort. Once you know one, a new language is easy to learn, as long as you have time.
Just get a good recommendation, and put some programming experience on your resume.
You may also wish to check out astrobetter.com. They have a whole IDL vs python discussion there.
Good luck!
posted by pizzazz at 10:16 PM on April 26, 2014
This thread is closed to new comments.
If you want to play around with it, there is a very limited demo mode, but if you don't want to be booted out every 7 minutes, you would need an IDL license. A student license is around $75 for a year, but most likely your professor is planning to let you use a license on one of their machines.
The user pool is not particularly large, and mostly of the old-fogey-scientist persuasion so I don't think you're going to find a lot snazzy videos or even graphically-appealing websites. Mostly it's going to be Web-1.0 style text-based tutorials, and they mostly start from a specific configuration that they expect you to work with (which varies quite a bit based on platform and personal preference.)
Sorry to be discouraging—but your boss is going to know about these hurdles. When I teach IDL to undergrads, I usually sit them down and walk them through doing simple mathematical operations, using print statements and making a Hello World procedure, and then using tv to display images, rather than expecting them to be able to pick it up on their own.
posted by BrashTech at 4:36 PM on April 25, 2014 [2 favorites]