Opening the World of Open Mapping?
August 21, 2012 9:20 AM Subscribe
Please give me a thorough, basic summary of open mapping (open source mapping) so I can add pretty, complex maps to websites without making it a full-time job.
I've begun exploring open-source mapping software and I'm finding it overwhelming.
I haven't found a clear, thorough overview of the tools, so I'm turning to AskMe. I've read Sebastian Delmont's post about StreetEasy's switch to open mapping (excellent, but overwhelming) and Switch to OSM at OpenStreetmap (helpful, but light on details).
I'm interested in mapping a few different cities, but let's take San Francisco as an example.
I'd like to be able to:
I'd like to balance being as independent as possible (not relying on other services) with making this as easy on myself as possible (I'd much rather use existing tools than roll my own).
For #1, can I just pull up what I need in OpenStreetMap, take a screenshot, and use that?
For interactive things, do I need a tile generator?
How do I decide between OpenLayers and Leaflet? (I DO need one of them, right?)
Does it make sense to use open options for some of this and just use Google Maps for other parts?
Can you recommend open data sources that focus more on geographic and political boundaries rather than streets?
Is this whole idea way too resource-intensive for shared VPS hosting? (I would be surprised if I ever got more than 1000 page views a day.)
I am a big Drupal user, so anything that already interacts with Drupal would be great. (I know there are OpenLayers and Leaflet modules, but please share any info you have.)
TL;DR: I'm fairly technically savvy. What's the quickest, easiest way for me to add maps (ideally, open source/CC maps) to websites and movies with the most options and fewest restrictions?
Thank you!
I've begun exploring open-source mapping software and I'm finding it overwhelming.
I haven't found a clear, thorough overview of the tools, so I'm turning to AskMe. I've read Sebastian Delmont's post about StreetEasy's switch to open mapping (excellent, but overwhelming) and Switch to OSM at OpenStreetmap (helpful, but light on details).
I'm interested in mapping a few different cities, but let's take San Francisco as an example.
I'd like to be able to:
- take screen shots of maps to add to websites
- create custom "shapes" (think neighborhoods or regions) and let website visitors select them to learn more - for example, let's say I wanted to create a handful of regions showing the sunny parts of the city, the foggy parts of the city, and the rainy parts of the city; and as users clicked a month (Jan, Feb, March), the regions would update to show which regions were sunny during that month
- map a LOT of points - for example, let's say I wanted to show which restaurants have burgers and which have tuna melts (and if you can tell me about marker aggregators, that would be great)
- shade streets to indicate status - for example, let's say I want to color each block I've been down, and I want to use pink for blocks I've walked on, green if I jogged, and blue if I biked - and ideally, I could do each side of the street separately: blue on the north side, green on the south side
- maybe even do fly-overs - is that sort of thing only available using Google Earth, and therefore not legally embeddable in a movie?
I'd like to balance being as independent as possible (not relying on other services) with making this as easy on myself as possible (I'd much rather use existing tools than roll my own).
For #1, can I just pull up what I need in OpenStreetMap, take a screenshot, and use that?
For interactive things, do I need a tile generator?
How do I decide between OpenLayers and Leaflet? (I DO need one of them, right?)
Does it make sense to use open options for some of this and just use Google Maps for other parts?
Can you recommend open data sources that focus more on geographic and political boundaries rather than streets?
Is this whole idea way too resource-intensive for shared VPS hosting? (I would be surprised if I ever got more than 1000 page views a day.)
I am a big Drupal user, so anything that already interacts with Drupal would be great. (I know there are OpenLayers and Leaflet modules, but please share any info you have.)
TL;DR: I'm fairly technically savvy. What's the quickest, easiest way for me to add maps (ideally, open source/CC maps) to websites and movies with the most options and fewest restrictions?
Thank you!
Best answer: I've spent the last year or two learning about all this stuff, there's a lot to learn. Answering all these questions would take a book. And fair question, is there a good book? I don't know of one, maybe someone can recommend one. But here's some scattered responses:
You should become familiar with Quantum GIS, PostGIS, GeoJSON, and shp2pgsql. These are essential parts of the open GIS tech stack for working with geographic data.
GeoDjango is a pretty good full-stack for doing geo webapps. I've only used it a bit, but it may be a good place to start your project.
If you want to generate your own tiles, TileMill from MapBox is absolutely terrific. It's a good way to learn about custom map-making; good docs, fun app. That being said, for web stuff I prefer taking someone else's base map tiles and then drawing my own data on top of them in Javascript.
Leaflet is a modern, lightweight alternative to OpenLayers. I prefer it. OpenLayers has a lot more functionality, but it's also heavier. Other Javascript map API options: Polymaps, Modest Maps JS.
Google's map API ecosystem is entirely separate. I avoid it, both because it's not free and because they have various restrictions on how you can use the data. OTOH they have the best data and their Javascript library is good. I'm not aware of any alternative to Google Earth.
posted by Nelson at 10:13 AM on August 21, 2012 [2 favorites]
You should become familiar with Quantum GIS, PostGIS, GeoJSON, and shp2pgsql. These are essential parts of the open GIS tech stack for working with geographic data.
GeoDjango is a pretty good full-stack for doing geo webapps. I've only used it a bit, but it may be a good place to start your project.
If you want to generate your own tiles, TileMill from MapBox is absolutely terrific. It's a good way to learn about custom map-making; good docs, fun app. That being said, for web stuff I prefer taking someone else's base map tiles and then drawing my own data on top of them in Javascript.
Leaflet is a modern, lightweight alternative to OpenLayers. I prefer it. OpenLayers has a lot more functionality, but it's also heavier. Other Javascript map API options: Polymaps, Modest Maps JS.
Google's map API ecosystem is entirely separate. I avoid it, both because it's not free and because they have various restrictions on how you can use the data. OTOH they have the best data and their Javascript library is good. I'm not aware of any alternative to Google Earth.
posted by Nelson at 10:13 AM on August 21, 2012 [2 favorites]
Best answer: I don't know much more than you, myself. But you might be interested in Cartaro, a "geospatial CMS" that seems to be built on Drupal.
posted by The Biggest Dreamer at 10:34 AM on August 21, 2012
posted by The Biggest Dreamer at 10:34 AM on August 21, 2012
The difficulty with taking screenshots of maps is that the bottom left pixel of your image doesn't know it's the intersection of Haight & Baker, and the top right is unaware of its status as Hayes & Webster. There's a lot of work to georeferencing an image, and at least as much making sure that the georeferencing is maintained through your workflow.
I know people who use and like MapBox.
posted by scruss at 10:53 AM on August 21, 2012
I know people who use and like MapBox.
posted by scruss at 10:53 AM on August 21, 2012
I'll be the first to advocate for server-side map customization at ArcGIS Online - a significantly different approach. You build your own map service using a combination of their data and your own. Then you can easily use any client (like ESRI's own JavaScript client, or OpenLayers, or whatever). Here is a web map service for marine geophysics created there, and here is a client application that displays it.
So yeah, while marine geophysics is a pretty complex subject matter area, making the map service and the viewer aren't hard at all.
There are plenty of examples, too.
posted by j_curiouser at 12:12 PM on August 21, 2012
So yeah, while marine geophysics is a pretty complex subject matter area, making the map service and the viewer aren't hard at all.
There are plenty of examples, too.
posted by j_curiouser at 12:12 PM on August 21, 2012
seconding scruss, I'll also dissuade you from using screen shots as basemaps in any capacity. There are really way too many issues with that to begin listing here.
posted by j_curiouser at 12:15 PM on August 21, 2012
posted by j_curiouser at 12:15 PM on August 21, 2012
Response by poster: Just to clarify - probably the only time I would use screen shots of maps would be in a movie or presentation (where there's no interactivity and the map is either static or briefly scrolled, by me, while making the movie).
For anything interactive, I would definitely prefer to have "slippy tiles", so users could zoom in and out and do all those nifty map-like things.
Thanks - I appreciate the input so far, and would love to hear any and all other thoughts!
posted by kristi at 2:42 PM on August 21, 2012
For anything interactive, I would definitely prefer to have "slippy tiles", so users could zoom in and out and do all those nifty map-like things.
Thanks - I appreciate the input so far, and would love to hear any and all other thoughts!
posted by kristi at 2:42 PM on August 21, 2012
Best answer: Just a quick addition, 'cause I think some of the people up-thread have given you advice that's correct, but is pretty darned daunting and you can start simpler.
I was just browsing InciWeb trying to plan some camping around forest fires. I was struggling with trying to figure out where things were from the provided "terrain" and "hybrid" maps, and then and realized that their fire perimeter maps were drawn with KML and I could easily copy and paste that KML URL to the map viewer of my choice.
And, indeed, I plugged the KML into the standard Google Maps viewer and was able to see the fires relative to things that made sense to me.
I then said "but what does it look like on Open Street Map tiles?", and a few lines of code later I'd embedded it in an OpenLayers map on my server.
So I'm going to make an ill-advised argument for "Put all the data in KML files and use whatever as off-the-shelf as possible", at least for starters. Taking your points in order:
> 1. take screen shots of maps to add to websites
As I think has been clarified, you really mean grab screen shots for presentations and add actual maps to web sites. That seems like a licensing issue with your base map tiles and the data that renders them, and you can make the call there.
I'd go so far as to say you should publish the data and also display it on a slippy map on the web site. Mashups are fun and awesome and we should encourage them.
> 2. create custom "shapes" (think neighborhoods or regions) and let website visitors select them to learn more ...
KML popups can (usually do) have HTML data embedded in them, so you can do some "click here to learn a little more" (as in that fire link above), but as you start getting further into your requirements, yeah, you're gonna have to have some dynamic JavaScript querying. But as a simple on-the-way you could just use KML and a layer for each one. Yes, they could select the "January" and "June" weather layer simultaneously, and that's kinda ugly. Oh well.
> 3. map a LOT of points
What's "a LOT"? this page has 2219 selectable points, it can slow a little bit, for some reason my mind wants to say "8k points is a reasonable limit", with Firefox on a relatively modern laptop, or start thinking about ways to dynamically page and refresh data as users slippy around. This also varies depending on browser.
> 4. shade streets to indicate status
That KML track on the page above was grabbed from my GPS using gpsbabel and hand-edited a little bit (I think I nuked everything to the start and end points in Emacs). If you zoom in far enough you can see that there are lines, and in the KML you can specify line characteristics, color, alpha, etc. I don't have a good public-facing example right now, but it's easy enough to suck in the tracks you've got and spit out a different color (or decimate the points).
But as I said, when I do that with 70k points (for example, and I have) things start to dog a bit and at some point it starts to make sense to be aware of zoom levels and view regions and adapt.
That, of course, assumes that your street status is pulled from your tracks and you're not trying to say "Find me this street segment and color it X". That's harder to do, and there I'd start by trying to set up an OpenStreetMap instance of your own. Good luck.
> 5. maybe even do fly-overs - is that sort of thing only available using Google Earth, and therefore not legally embeddable in a movie?
With KML, you can tell people to get their own damned Google Earth. Or fly it live if you're doing presentations. For instance, click on the KML file of that little sailboat trip I linked to up page, "open in Google Earth", and there you go: Beautiful Petaluma, virtually.
Yes, this oversimplifies the hell out of things. The GIS nerds are screaming "but... but... projections! and datums! and..." and they're all right, but if you just want to start hacking on stuff, put all of your coordinates in WGS84 (probably the default of what comes off your GPS device anyway), write your data to KML (or serve it out of a database to KML), it'll probably take you an hour per JavaScript framework to figure out how to put a map on a web page with some base layers that displays that (or several) KML files (remember you'll probably need a proxy for remote KML files to get past JavaScript security issues), and when that doesn't work for you, then you can start worrying about optimization.
And when you come across "shapefiles" or local aerial imagery that's georeferenced to a local projection or whatever else, or discover that your data set really is too large to let the web browser and JavaScript handle it all, then you can learn about using QGIS or running your own tile server or bla bla bla.
posted by straw at 4:00 PM on August 21, 2012
I was just browsing InciWeb trying to plan some camping around forest fires. I was struggling with trying to figure out where things were from the provided "terrain" and "hybrid" maps, and then and realized that their fire perimeter maps were drawn with KML and I could easily copy and paste that KML URL to the map viewer of my choice.
And, indeed, I plugged the KML into the standard Google Maps viewer and was able to see the fires relative to things that made sense to me.
I then said "but what does it look like on Open Street Map tiles?", and a few lines of code later I'd embedded it in an OpenLayers map on my server.
So I'm going to make an ill-advised argument for "Put all the data in KML files and use whatever as off-the-shelf as possible", at least for starters. Taking your points in order:
> 1. take screen shots of maps to add to websites
As I think has been clarified, you really mean grab screen shots for presentations and add actual maps to web sites. That seems like a licensing issue with your base map tiles and the data that renders them, and you can make the call there.
I'd go so far as to say you should publish the data and also display it on a slippy map on the web site. Mashups are fun and awesome and we should encourage them.
> 2. create custom "shapes" (think neighborhoods or regions) and let website visitors select them to learn more ...
KML popups can (usually do) have HTML data embedded in them, so you can do some "click here to learn a little more" (as in that fire link above), but as you start getting further into your requirements, yeah, you're gonna have to have some dynamic JavaScript querying. But as a simple on-the-way you could just use KML and a layer for each one. Yes, they could select the "January" and "June" weather layer simultaneously, and that's kinda ugly. Oh well.
> 3. map a LOT of points
What's "a LOT"? this page has 2219 selectable points, it can slow a little bit, for some reason my mind wants to say "8k points is a reasonable limit", with Firefox on a relatively modern laptop, or start thinking about ways to dynamically page and refresh data as users slippy around. This also varies depending on browser.
> 4. shade streets to indicate status
That KML track on the page above was grabbed from my GPS using gpsbabel and hand-edited a little bit (I think I nuked everything to the start and end points in Emacs). If you zoom in far enough you can see that there are lines, and in the KML you can specify line characteristics, color, alpha, etc. I don't have a good public-facing example right now, but it's easy enough to suck in the tracks you've got and spit out a different color (or decimate the points).
But as I said, when I do that with 70k points (for example, and I have) things start to dog a bit and at some point it starts to make sense to be aware of zoom levels and view regions and adapt.
That, of course, assumes that your street status is pulled from your tracks and you're not trying to say "Find me this street segment and color it X". That's harder to do, and there I'd start by trying to set up an OpenStreetMap instance of your own. Good luck.
> 5. maybe even do fly-overs - is that sort of thing only available using Google Earth, and therefore not legally embeddable in a movie?
With KML, you can tell people to get their own damned Google Earth. Or fly it live if you're doing presentations. For instance, click on the KML file of that little sailboat trip I linked to up page, "open in Google Earth", and there you go: Beautiful Petaluma, virtually.
Yes, this oversimplifies the hell out of things. The GIS nerds are screaming "but... but... projections! and datums! and..." and they're all right, but if you just want to start hacking on stuff, put all of your coordinates in WGS84 (probably the default of what comes off your GPS device anyway), write your data to KML (or serve it out of a database to KML), it'll probably take you an hour per JavaScript framework to figure out how to put a map on a web page with some base layers that displays that (or several) KML files (remember you'll probably need a proxy for remote KML files to get past JavaScript security issues), and when that doesn't work for you, then you can start worrying about optimization.
And when you come across "shapefiles" or local aerial imagery that's georeferenced to a local projection or whatever else, or discover that your data set really is too large to let the web browser and JavaScript handle it all, then you can learn about using QGIS or running your own tile server or bla bla bla.
posted by straw at 4:00 PM on August 21, 2012
Best answer: Oh, and 'cause I just discovered it, for your #5: Marble is an open-source Google-Earth like thingie that does indeed display KML files, but the imagery is pretty darned low res, so though it might be good for an animated "They went from London to LA" flyover, it's not gonna work for "They went from San Francisco to Berkeley".
And it's been several years, but i have looked into using VTP for doing fly-arounds in real-world datasets.
posted by straw at 4:29 PM on August 21, 2012
And it's been several years, but i have looked into using VTP for doing fly-arounds in real-world datasets.
posted by straw at 4:29 PM on August 21, 2012
And I am apparently an idiot: Here's the USGS active fire perimeters as retrieved this afternoon on an Open Layers map on my server, with layers for Open Street Map (cached indefinitely on my server) and all the Google types.
posted by straw at 4:42 PM on August 21, 2012
posted by straw at 4:42 PM on August 21, 2012
And while we're using a wildfire example, here's a map service that uses the same data as straw in conjunction with many other elements. And here's the web client.
posted by j_curiouser at 8:33 PM on August 21, 2012 [1 favorite]
posted by j_curiouser at 8:33 PM on August 21, 2012 [1 favorite]
Response by poster: These are all great! Thanks for all the terrific perspectives.
Special thanks to straw for all the great info - especially the link to VTP! I had no idea that existed.
Thank you all so much!
posted by kristi at 1:21 PM on August 24, 2012
Special thanks to straw for all the great info - especially the link to VTP! I had no idea that existed.
Thank you all so much!
posted by kristi at 1:21 PM on August 24, 2012
This thread is closed to new comments.
For data, I've just been rendering into the subset of KML that OpenLayers and Google Maps like as a layer (or several). I haven't been using it for filled regions, but for points and lines it's been great.
If your data is of the "render it out" sort, I'd seriously try writing a KML that has your regions and your weather data inside a clickable point, add that as an OpenLayers layer that uses whatever base layers you want, and see if that meets your needs.
posted by straw at 9:38 AM on August 21, 2012