Jumping ship from academia to industry
March 8, 2015 10:43 AM Subscribe
I'm have a PhD in computer science, and after an unsuccessful location-constrained academic job search, have decided to finally try industry. If you made a similar shift, I'd love to hear your experiences. What was the career transition like? What sorts of jobs should I be applying for?
I specialize in machine learning, which I know is all the rage in software nowadays, though what I do is kind of theoretical/niche. I'm applying to "data scientist" positions at major software companies as well as startups.
My question is for people who happen to be data scientists (or similar) coming from academia. What do you do on a day to day basis? Do you find the work challenging and fulfilling? To what extent do you keep in touch with academic research?
I'm open to other sorts of tech jobs too, but having no industry experience whatsoever, I'm not sure what's out there. I enjoy programming, but C is not my forte, so heavyweight back-end jobs wouldn't be a good fit. I love teaching (and have quite a bit of experience there). I prefer to spend time on big picture problem solving rather than bug-chasing. Don't have kids at the moment, so long hours are ok, as long as the work is enjoyable. I like interacting with people, but I also need quiet time to think.
(Looking for jobs in the Boston area.)
I specialize in machine learning, which I know is all the rage in software nowadays, though what I do is kind of theoretical/niche. I'm applying to "data scientist" positions at major software companies as well as startups.
My question is for people who happen to be data scientists (or similar) coming from academia. What do you do on a day to day basis? Do you find the work challenging and fulfilling? To what extent do you keep in touch with academic research?
I'm open to other sorts of tech jobs too, but having no industry experience whatsoever, I'm not sure what's out there. I enjoy programming, but C is not my forte, so heavyweight back-end jobs wouldn't be a good fit. I love teaching (and have quite a bit of experience there). I prefer to spend time on big picture problem solving rather than bug-chasing. Don't have kids at the moment, so long hours are ok, as long as the work is enjoyable. I like interacting with people, but I also need quiet time to think.
(Looking for jobs in the Boston area.)
This is me, my PhD was in environmental science and I've been a data scientist for 1.5 years now in the energy sector. Data science is a likely landing spot for you especially given your specialization.
There is a spectrum of roles which ranges from pure engineering to pure modelling. On the engineering end people are setting up databases and data pipelines and writing operational software to implement algorithms at scale, and the skillset is pure software engineering. On the modelling end people are experimenting with data and models and developing prototype algorithms - more science than engineering. Most roles fall somewhere in between. Larger companies like Twitter or LinkedIn will have a higher degree of specialization and so you are more likely to find more purely scientific roles. At a startup you will need to get your hands dirty, by which I mean writing the infrastructure code that allows you to push data around, as opposed to just pure experimentation.
I work in a midsize company and our team writes algorithms and models but also does a lot of data cleaning, munging, plumbing, etc, which is the annoying tedious part of the job. We also handle analysis requests from others in the company. My day to day will vary; sometimes I'll be developing features and trying out machine learning algorithms for a specific problem; sometimes I'll be writing code to let me automate those tasks and do them at a bigger scale; sometimes I'll be evaluating and debugging larger-scale production runs; and lately I am spending a lot of time working with the engineering team to evaluate new technologies that we might want to use.
The work is different from academia. It is applied and immediate. You have to get your best effort out the door in a constrained time, rather than spend as much time as you need to find an optimal solution. You get used to the idea of "good enough". It can be a bit unsatisfying at times, but the converse is that your work makes an immediate impact on the team and the business. I found it very refreshing to have my work so valued and relevant to the real world, after a long period of what felt like toiling in obscurity. And I find it more interesting and challenging than my research was. I also have incredibly rich data to play with which I never could have gotten my hands on in academia. However I don't get to pick my projects -- they are determined by the needs of the business.
Our team tries to keep up with academic research, but it's not a major priority as we tend to have a lot of projects on the go which demand most of our time. Likewise we try to get the odd publication or conference paper out but it is relatively rare, and only when it aligns with an opportunity to promote our company within our industry. These are the realities of industry, at least for us; we're constrained, we can't do everything we would like. I do feel though that my research brain is getting exercised and I am learning all the time, so it is a pretty satisfying career. I think before leaping into industry that I was afraid my work would be a monotonous grind and that I would just die of boredom and lack of stimulation, but that's not been true at all -- if anything I am more engaged than I was in academia, and the practical relevance of the work turns out to count for a lot.
Happy to follow up if you have more questions.
posted by PercussivePaul at 11:33 AM on March 8, 2015 [1 favorite]
There is a spectrum of roles which ranges from pure engineering to pure modelling. On the engineering end people are setting up databases and data pipelines and writing operational software to implement algorithms at scale, and the skillset is pure software engineering. On the modelling end people are experimenting with data and models and developing prototype algorithms - more science than engineering. Most roles fall somewhere in between. Larger companies like Twitter or LinkedIn will have a higher degree of specialization and so you are more likely to find more purely scientific roles. At a startup you will need to get your hands dirty, by which I mean writing the infrastructure code that allows you to push data around, as opposed to just pure experimentation.
I work in a midsize company and our team writes algorithms and models but also does a lot of data cleaning, munging, plumbing, etc, which is the annoying tedious part of the job. We also handle analysis requests from others in the company. My day to day will vary; sometimes I'll be developing features and trying out machine learning algorithms for a specific problem; sometimes I'll be writing code to let me automate those tasks and do them at a bigger scale; sometimes I'll be evaluating and debugging larger-scale production runs; and lately I am spending a lot of time working with the engineering team to evaluate new technologies that we might want to use.
The work is different from academia. It is applied and immediate. You have to get your best effort out the door in a constrained time, rather than spend as much time as you need to find an optimal solution. You get used to the idea of "good enough". It can be a bit unsatisfying at times, but the converse is that your work makes an immediate impact on the team and the business. I found it very refreshing to have my work so valued and relevant to the real world, after a long period of what felt like toiling in obscurity. And I find it more interesting and challenging than my research was. I also have incredibly rich data to play with which I never could have gotten my hands on in academia. However I don't get to pick my projects -- they are determined by the needs of the business.
Our team tries to keep up with academic research, but it's not a major priority as we tend to have a lot of projects on the go which demand most of our time. Likewise we try to get the odd publication or conference paper out but it is relatively rare, and only when it aligns with an opportunity to promote our company within our industry. These are the realities of industry, at least for us; we're constrained, we can't do everything we would like. I do feel though that my research brain is getting exercised and I am learning all the time, so it is a pretty satisfying career. I think before leaping into industry that I was afraid my work would be a monotonous grind and that I would just die of boredom and lack of stimulation, but that's not been true at all -- if anything I am more engaged than I was in academia, and the practical relevance of the work turns out to count for a lot.
Happy to follow up if you have more questions.
posted by PercussivePaul at 11:33 AM on March 8, 2015 [1 favorite]
There are definitely a lot of PhD-level "data scientists" in industry; I know several, though I am not myself a data scientist or a PhD.
I would actually research the backgrounds of data scientists at prominent startups/companies (Palantir, Facebook, Google, Twitter, etc.) and reach out to those who have PhDs. At least some of them will be receptive to giving you advice about the transition from academia to industry.
It is a hot field, with a lot of hype right now. But it's also a field for which qualified applicants are lacking so if you can sell yourself well you will get a job.
posted by dfriedman at 12:30 PM on March 8, 2015
I would actually research the backgrounds of data scientists at prominent startups/companies (Palantir, Facebook, Google, Twitter, etc.) and reach out to those who have PhDs. At least some of them will be receptive to giving you advice about the transition from academia to industry.
It is a hot field, with a lot of hype right now. But it's also a field for which qualified applicants are lacking so if you can sell yourself well you will get a job.
posted by dfriedman at 12:30 PM on March 8, 2015
I have had quite a bit of visibility into the hiring process at a few major tech companies and I've seen a lot of PhDs apply with the stance that they'll ONLY take positions in the research division. That's of course their prerogative but it massively limits the likelihood that they'll be hired. Those research positions are very limited and coveted. I know plenty of PhDs who are quite happy working in more standard roles. Obviously only you can decide this for yourself, but think hard before zeroing in on "research scientist" positions and the like.
posted by primethyme at 12:53 PM on March 8, 2015 [1 favorite]
posted by primethyme at 12:53 PM on March 8, 2015 [1 favorite]
I dropped out of a CS PhD program, and I now work at a giant tech company with many PhDs. Most of them aren't really paid to do research, but get to do some as a small part of their work (maybe 20%). It's a relaxed, academic-y environment and many people from academia are happy to stay there basically forever -- in fact we just had a guy retire after 20 years in the same job.
The worry when hiring someone with a PhD is that they won't be willing to do the hard, boring work like maintaining the stupid 10-year-old Perl scripts that your company needs in order to operate, or making yet another PowerPoint to justify a project to management. Another worry is that they'll just jump ship to a better job in a year. So do whatever you can to allay those fears.
I'm sure you'll be fine though, if you apply to 10 places you'll get at least one or two good offers.
My only real advice would be to go to lots of meetups and conferences in order to build out an industry network. Maybe you'll get your first job by applying, but you should get your second job because some manager or executive knows you well and thinks you'd be a perfect fit for a position that's opening up. You can't get there just by hanging out and doing your own personal tasks, you have to actively put yourself out there and make it happen.
posted by miyabo at 1:42 PM on March 8, 2015 [2 favorites]
The worry when hiring someone with a PhD is that they won't be willing to do the hard, boring work like maintaining the stupid 10-year-old Perl scripts that your company needs in order to operate, or making yet another PowerPoint to justify a project to management. Another worry is that they'll just jump ship to a better job in a year. So do whatever you can to allay those fears.
I'm sure you'll be fine though, if you apply to 10 places you'll get at least one or two good offers.
My only real advice would be to go to lots of meetups and conferences in order to build out an industry network. Maybe you'll get your first job by applying, but you should get your second job because some manager or executive knows you well and thinks you'd be a perfect fit for a position that's opening up. You can't get there just by hanging out and doing your own personal tasks, you have to actively put yourself out there and make it happen.
posted by miyabo at 1:42 PM on March 8, 2015 [2 favorites]
If keeping up with academic research is a priority for you, I'd think about which environments already value having employees in R&D roles who are plugged in to current research. Seeing as you're in Boston, you're near a lot of life science and biotech companies that are likely seeking data scientists in addition to the tech and software companies you mentioned, but sussing out which ones treat their data scientists like IT support and which ones see them as an integral part of the research function would likely require you to apply, interview, and ask a bunch of pointed questions.
posted by deludingmyself at 1:46 PM on March 8, 2015
posted by deludingmyself at 1:46 PM on March 8, 2015
Oh, and our official pay grades say that someone with a PhD is in the same slot as someone with 6-8 years of industry experience. Be prepared for the fact that someone who is younger than you, and has no advanced degree, but has done really excellent work could easily be senior to you.
posted by miyabo at 1:53 PM on March 8, 2015 [1 favorite]
posted by miyabo at 1:53 PM on March 8, 2015 [1 favorite]
The worry when hiring someone with a PhD is that they won't be willing to do the hard, boring work like maintaining the stupid 10-year-old Perl scripts that your company needs in order to operate, or making yet another PowerPoint to justify a project to management. Another worry is that they'll just jump ship to a better job in a year. So do whatever you can to allay those fears.I feel really strongly that this is the result of years of confusion about what doing a PhD is actually like. Research is EXACTLY this kind of thing for YEARS AT A TIME. Everyone working in research knows how terrible research tooling + code practices are and should never be horrified by whatever obscene legacy perl tooling you have to support since they've seen worse. Grant-writing and administration is at least as tedious and formulaic as fighting for resources inside a company, so that should be no great surprise either.
There is an ambient risk with any candidate making a big career switch like this that they will jump ship in a year when they realize they don't like the work much. But I don't think PhDs are any more apt to do that than anyone else. If anything, PhDs have spent years working towards a distant goal and putting up with poor pay and lonely working conditions and should have an easier time handling that sort of thing than an egotistic fresh-out-of-Stanford CS undergrad who thinks they're owed the world. When I found the position I'm in now, I was/am so grateful to have found an organization that values all the things I can do that I feel a lot more loyal than people who have been in industry for a while.
posted by heresiarch at 2:21 PM on March 8, 2015 [9 favorites]
A friend of mine did the Insight Data Fellows program, and raves about it. He did the Silicon Valley program, but it looks like the NYC program is set up to work with employers in Boston. I'm not sure exactly your limitations on location, but if you could swing a few weeks in NYC this could be a cool opportunity.
posted by rainbowbrite at 3:22 PM on March 8, 2015 [1 favorite]
posted by rainbowbrite at 3:22 PM on March 8, 2015 [1 favorite]
You might find some good advice from Cathy O'Neil, who was coincidentally featured over on the blue today.
posted by sockermom at 6:41 PM on March 8, 2015
posted by sockermom at 6:41 PM on March 8, 2015
This thread is closed to new comments.
There is massive confusion about what a data scientist does and what is reasonable preparation for the job. Broadly speaking, there are two categories of data science jobs. There are data scientists working on data products (e.g. Netflix recommendation systems, FB newsfeed optimization, some kinds of financial tooling, fancy demand modeling, etc) and there are data scientists working on data decision-making work (e.g. should we build feature X or Y? what impact is feature Z having on our ecosystem? how should we measure the impact of our transactional emails?). You should think deeply about which of those you are most interested in. This blog posts nicely captures how data decision-making teams (which is what I do) think about this distinction.
The mania around machine learning is very, very real, but a lot of times companies think they have data product problems when really they have data decision-making problems. If a company does not yet have a lot of data scientists, they are unlikely to understand the difference and might lead you astray.
Regardless of which kind of work you'd like to do, some general advice about making the transition. I've seen probably 50 job applications from people with phds at this point, and they are among the least effective job applicants because they make the same mistakes over and over again:
- Write a cover letter! I know this is probably obvious, but the vast majority of applicants don't do it. If you've worked in a specific role for a long time you can maybe get away with not writing one, but as a candidate transitioning from what is essentially one career to another, you must translate your experience actively into the field you're transitioning into. This doesn't mean writing about "synergies" or "leverage" necessarily, it just means picking out the stuff you've done in your research work that will be meaningful for them.
- Don't write off teaching or publishing histories - at least on my team we value both of those a lot, since we have to be internal advocates across a lot of teams for good data-thinking practices and raw technical talent is worth less to us than communication and mentoring. But don't assume those will speak for themselves either; you have to explain when and why those skills will be useful.
- Figure out which kind of job it is and write the cover letter appropriately. If they clearly demonstrate they want machine learning expertise, it might make sense to get into more detail about your research. Otherwise, you'll want to situate machine learning as one tool among many that you can use effectively.
- Recognize that you are a high risk candidate and act accordingly. You have a lot of experience in an abstract sense, but you're like a new grad in the sense that since you haven't done exactly what you're applying to do, the organization has to look at proxies to judge whether you'll enjoy it + be good at it. Anything you can do to decrease that sense of risk will be in your favor.
- Do not talk about how you're excited about how much data a company has. All companies have lots of data.
LMK if you have more questions! I was on the other side of this ~10 months ago and am happy to help. I'm SF-based now so I'm not super aware of Boston-based opportunities. I looked there for a while and didn't find anything that was a great fit for me, but YMMV. I didn't have a strong machine learning background at all (I was at the Media Lab) so I was probably looking for slightly different roles than you are.Good luck!
posted by heresiarch at 11:21 AM on March 8, 2015 [14 favorites]