Which Online Automation Tools Can Batch Download Files?

June 19, 2015 12:17 PM

CheckVist (you needn't be a user to answer this!) is a collaborative too for creating/editing outline docs. But you can't revert bad edits (there's not even multiple undo), nor does it ever back up. So I'm building a script to download all 100 of my CheckVist outlines to computer. I'll make a Quickey or Applescript to call up each page in a browser, trigger CheckVist's keyboard shortcut to initiate export/download of the OPML file (the universal outliner format), and then repeat with the next page....times 100. CheckVist has nothing to help me, but perhaps third party tools, e.g. IFTTT? What's available for this sort of batch work?
posted by Quisp Lover to Computers & Internet (8 answers total)

Response by poster: Note: I do need to download to my computer, not elsewhere. Because in addition to backing up, I'll sometimes need to do a batch text search/replace with my text editor to update templated text common to all files (it'd take all eternity to update one-outline-at-a-time online). Then, I suppose, reverse the downloading process to upload (CheckVist allows you to import over existing files with a keyboard shortcut).
posted by Quisp Lover at 12:20 PM on June 19, 2015

Are the URLs to those files generally pretty static? That is, could you generate a list of them all in the form of:

http://some.host.name/path/to/file1
http://some.host.name/path/to/file2.something

...and so on?
posted by jquinby at 12:20 PM on June 19, 2015 [1 favorite]

Response by poster: Static URLs, yes. But it'd be a lot better, and more prudent, to download in OPML format rather than HTML. Per my follow-up, I'd also like to re-upload/import them all on occasions where I need to globally change certain text common to all. In OPML format, there's no noise and everything's fully importable/exportable anywhere.

OTOH, if there's a way to convert HTML to OPML, this could actually work! Much easier to batch process the conversion locally than to klunkily manipulate all those web pages! I'll start looking around...
posted by Quisp Lover at 12:52 PM on June 19, 2015

Best answer: They have an API that will return JSON, which would be much easier than trying to scrape the web pages. You could use some combination of Curl and JSONHelper with AppleScript to download/upload and manipulate the JSON (I'm not familiar with AppleScript but it shouldn't be too difficult as JSON is structured). It also does XML which might map closely to the OPML.
posted by tallus at 2:34 PM on June 19, 2015

Response by poster: I'd love to rig that up, but it's just a bit beyond my tech skills, alas. I've downloaded JSONHelper, though, and will keep trying to get into it....

thanks!
posted by Quisp Lover at 7:17 PM on June 19, 2015

Best answer: If you add "/export.opml?export_notes=true" to the note URL, you get the OPML as HTML in the browser... maybe this nudges a step closer...

So depending on how many notes you have, you need a list of the links ala
- https://checkvist.com/checklists/
- https://checkvist.com/checklists/

..and then you can append the appropriate text using a tool of your choice (Sublime text here).

Now you have a list of URLs to feed to curl, wget, etc.

The only obviously annoying step is getting the base URLs. If you can stomach a lot of right-clicking, then have at it. If not, perhaps others can help.

The markup on the list of lists is pretty clear:

...

posted by Jack Karaoke at 5:29 AM on June 21, 2015

Best answer: Err, I don't know how to post code here. On your "my lists" page, the list item has a class assigned which includes the note URL number.
posted by Jack Karaoke at 5:39 AM on June 21, 2015

Response by poster: Thanks, Jack. Great pointers!
posted by Quisp Lover at 4:06 PM on June 21, 2015