Must automate, ASAP.

August 29, 2014 1:01 PM

Dear Metafilter, I need to automate some tasks at work, and I'd like to make it happen quickly.

These tasks have been digitized, which saves a lot of time, but digitization is not the same thing as automation. The work in question has absolutely not been automated; in fact, it requires a ton of hand-holding. However, in the meantime, my workload keeps increasing, which means I have less time to dedicate to these repetitive tasks that consume my attention, but must be performed on a regular schedule.

So I want to learn whatever I need to learn to make these processes run on their own, start to finish.

Whatever I create to accomplish this goal needs to interface with my OS, with other programs, and with various webapps. (E.g., locate this file on this SFTP server, place it in this directory, rename it with these naming conventions, open this webapp, log in with these credentials, load the file, run an error report, and so on.)

So my question is, can you recommend any quick-to-learn, quick-to-implement tools or programming languages, or anything else, that would enable me to fully automate some digital, but still time- and labor-intensive, tasks?

Thanks in advance, all.

posted by schooley to Technology (13 answers total)

AutoIt if you are on Windows.
posted by meta87 at 1:12 PM on August 29, 2014

I think Python may be what you want. There are other ways to go about what you're suggesting, but if you have the fortitude and aptitude, Python covers all your bases. It has a huge, friendly user community and a ton of modules to get specific stuff done (like launching applications, transferring files via SFTP, web stuff, etc.). It'll run in pretty much any environment.

You give the impression that you don't already know any scripting languages and you say you want this to be quick -- it won't be quick. But if you stick with it it'll be very successful and rewarding. Good luck!
posted by mindsound at 1:13 PM on August 29, 2014 [2 favorites]

To expand on the Python bit, you can get into an SFTP server using the paramiko module (plus its dependencies). I have a process doing that right now.

Touching web applications might be harder and there's a bunch of ways to do it, everything from grunt-level pushing HTTP requests like POST and GET around using urllib (or urllib2 or Requests) to more screen-level automation with something like Selenium.

Calling various executables is typically done with a module like subprocess and with sufficient perseverance, you can perform magical feats with this module. I have another process also doing that now.
posted by adipocere at 1:22 PM on August 29, 2014

Python is good but you'll definitely need to learn some programming skills if you don't have them already. This isn't going to be a weekend project.

If you go with Python, I'd recommend against urllib2 for automating web stuff. It's pretty outdated and the API is horrendously designed. requests seems to be the popular alternative.
posted by zixyer at 1:27 PM on August 29, 2014 [1 favorite]

I do a lot of this exact kind of thing, and I always use Ruby or Python. Using a full-fledged scripting language like these lets you build automation in a flexible, elegant way, and it's easy to get libraries for tasks like SFTP transfers or web automation. If you don't know either, I'd probably lean towards looking at Ruby - it has had a lot of beginners learning it over the last 5-10 years so there are tons of introductory resources available. Ruby also has the fantastic web automation library Mechanize, Python's version of which is a little dated. You could definitely get the job done with any modern scripting language, though.

If you're not up for learning a whole language, you can also find single-purpose tools that will do a lot of what you ask. WinSCP can do an automated SFTP transfer, Bulk Rename Utility may be able to do the moves/renames you want, and it looks like there are several web browser automation/macro tools out there. You'd end up with something a little more rickety and less flexible, but you could probably cobble it together and start getting value out of it quicker.
posted by pocams at 1:36 PM on August 29, 2014

Prediction : if you have the aptitude and application to automate the jobs in Python, you will get such a buzz you will be thinking career change.
posted by smugly rowan at 1:41 PM on August 29, 2014 [6 favorites]

I've done a LOT of automation with various tools. What you might end up seeing is that you can get 95% of the way there, but there's one step in the middle that requires handholding.

Start thinking "user experience" with you being the user. You're designing a workflow. If step 5 of 10 needs handholding can you design it to batch a whole bunch of 1-4 and 6-10, so that you only have to deal with it once?

Another caution: if you have a bunch of workflows in various steps then to really know whats going on when something fails you have to keep all those steps in your head. This will be tricky so design around it. You're the Sorcerer's Apprentice, control your brooms.

If you're on OS X, Automator CAN help with somethings but I find it woefully limited.

Do any of your web apps have APIs?

Seconding Smugly Rowan.
posted by Brainy at 1:47 PM on August 29, 2014 [1 favorite]

Best answer: Catharsis: I just finished a huge project similar to what you're describing - cataloging and processing mountains of reference footage from a film set.

While some or all of these steps could probably be automated, it sounds like it's an issue of communication between you and your supervisors about your workload. Starting a dialogue about how your time should be spent is probably worth doing, if you haven't already.

Stuff like this is classic shitwork for an intern or a junior and your boss will love that you're handing him a great pitch to his or her bosses for hiring more (cheap) staff.

With that in mind, I would probably try to communicate any ideas you have about automation to your supervisors since spending time on unapproved work may not go over well.

If you decide to move ahead with the project - and I also reccomend python - here's a few tips:

- Automate each step separately as a discrete single-step process. That way if one step fails you can manually intervene without having to start from the beginning. THEN write code to automate each step.

- Document and comment your code very very thoroughly.

- Your code should produce detailed logs that describe exactly what it's done.

- Error checking, exception catching and unit testing are your new best friends.

- ALWAYS include the option of a 'dry run', especially if you're moving files around.

Good luck! And if you succeed, this will definitely propel you towards a new focus with your work. Code-enhanced laziness is very very addictive.
posted by misterdaniel at 2:28 PM on August 29, 2014 [2 favorites]

Response by poster: Brainy, they do, but I'm afraid I have what could generously be called a bird's-eye understanding of APIs, so I'm afraid I can't tell you much else (sorry for sounding like the functional-not-technical employee that I am).

Misterdaniel, thanks. Fortunately, my supervisors agree with me that automation is a great idea, so I think they'd be delighted to hear about some "code-enhanced laziness." :)

Smugly Rowan, I'm looking forward to it.

These are awesome, super helpful answers. Thank you!
posted by schooley at 4:55 PM on August 29, 2014

If you're doing automation on Windows, learning some PowerShell scripting can help, especially when working with Microsoft technologies.
posted by Aleyn at 5:09 PM on August 29, 2014

Bash scripting can also help with routine tasks, you could set things like cron to take care of your business.
posted by oceanjesse at 5:46 PM on August 29, 2014

Quick debugging and pain reduction tip: if you're modifying files, make sure your automation is leaving the originals alone. You want to be able to safely re-run a step without fear or worry.
posted by migurski at 8:19 PM on August 29, 2014

Also: stackoverflow will be your new best friend (but she will have a few quirks you need to get used to).
posted by smugly rowan at 12:49 AM on August 30, 2014