Fetch, Website, Fetch! How do I get a website to fetch USGS Data?

April 12, 2017 11:28 AM

I'm trying to setup a web page on my Wordpress site to fetch river flow data from the USGS and drop it into a table. It seems like our friends at the USGS want to make this process easy for me, but my webdev skills are lacking.

I think this may be straightfoward for someone with the right know-how, but maybe not.

I run a little website. It serves, in part, as a guide to river surfing waves in Colorado.

For a particular wave, say River Run Park, I have an individual wave page where I'm usually able to display flow data from the USGS, like this. The chart comes from the main USGS flow page, here. Sometimes flow data is hosted by the State of Colorado, like this, but I gather that's a little tougher to gather. It seems like the USGS wants to made automated data retrieval easy, so that's nice of them.

What I'd like to do is put together a chart, on a single page, that scrapes the current flow data from the USGS and assigns a label based on low/medium/high values assigned by me for each wave. So, along these lines:

Wave / Value / Level
Union / 150 cfs / Low
Beaver / 500 cfs / Medium

Sites like American Whitewater manage to do it for a great many sites; I'd be looking to put together a single page with 10-15 gauges on it.

So, easy to do/hard to do? Is there some piece of code I can drop in and modify for each gauge, or is it more complicated than that? And if it's ultimately something I would need to hire a freelancer for, any guess at how long something like this would take to put together?

posted by craven_morhead to Computers & Internet (5 answers total)

Doing a scrape for a single piece of data per page is pretty straightforward -- find a piece of text that occurs right before the data you want, then find a piece of text that occurs right after the data you want. You use explode to split the page into smaller chunks, and then you might need to do some cleanup to make sure you got the data you want. Do note that these functions don't do any testing to see if the scrape failed or any kind of caching; you might consider doing some of that depending on your site's volume.

<?php
echo get_cfm_colorado('http://www.dwr.state.co.us/surfacewater/data/detail_graph.aspx?ID=PLACHACO&MTYPE=DISCHRG');
echo '<br>';
echo get_cfm_usgs('https://waterdata.usgs.gov/nwis/uv?site_no=06710247');

function get_cfm_colorado($page_url){
$page = file_get_contents($page_url);
$page = explode('Most Recent Value:',$page);
$page = explode('cfs',$page[1]);
$cfm = strip_tags($page[0]);
return($cfm);
}

function get_cfm_usgs($page_url){
$page = file_get_contents($page_url);
$page = explode('Most recent instantaneous value: ',$page);
$page = explode(' ',$page[1]);
$cfm = trim($page[0]);
return($cfm);
}
?>


Since you're putting this into a wordpress, adding a shortcode to your wordpress theme's functions file would probably be the best way to go.
posted by gregr at 12:31 PM on April 12, 2017 [1 favorite]

I've usually worked with Python tools to access USGS NWIS data, but you can do the direct JSON requests via the automated retrieval link in your post --- there's a JSON example at the bottom, and, probably more helpful is their little REST URL generation tool.

You can use jQuery or something similar to then grab and parse the JSON file (example for the gage you linked) for the discharge value. It's buried pretty far in there, but it's all in there.
posted by yukonho at 12:42 PM on April 12, 2017

There are pretty much two ways of doing it. One way (suggested by gregr) is to write something in php that works with wordpress, either as a plugin or as (gregr's better suggestion of) shortcode in the theme. That would fetch the data server-side and render your chart and serve it as html to the browser.

The other way, suggested by yukonho, would be to write something in js/jquery that does the fetch in-browser and renders it directly there.

Personally I'd favor the former solution, and if I were a php developer I'd totally jump on it. It's not fundamentally difficult at all, though, and would just be a couple hours of work for a dedicated wordpress dev.

Since we're local, if you're a slack user I'd recommend jumping on the Denver Devs slack team. There's a #help-wordpress channel on there, and that'd be a good place to look either for a full-on freelancer or for someone to answer questions and provide guidance if you feel like working through it yourself. I'd be happy to provide an introduction if you'd like, dm.
posted by 7segment at 1:21 PM on April 12, 2017 [1 favorite]

If you're looking to do it in Python there's a helpful (i.e. I understood it) JSON demo as part of newcoder.io's dataviz tutorial - with bonus extra maps!
posted by A Robot Ninja at 5:38 AM on April 13, 2017

Response by poster: Awesome, thanks for the tips all.
posted by craven_morhead at 11:44 AM on April 13, 2017