A plan for future-proof tile URLs

[migurski]

We are starting to see versioned/dated URLs in sharedstreets/sharedstreets-builder#10 (comment). @kpwebb and I talked by phone to see if there was a sensible URL structure that SharedStreets could commit to for tiles going forward. It would provide immutability so that tile consumers could be reassured of stable identifiers, links to the source OpenStreetMap data so OSM IDs could be unambiguously referenced, and an index format so that clients could choose when to move forward. As with OSM Planet, latest would be a special date to indicate the newest copy of the data.

Here’s what a sample JSON index with URI templates might look like:

[
  {
    "url_template": "https://tiles.sharedstreets.io/planet-latest/{z}-{x}-{y}.{layer}.pbf",
    "planet_url": "http://planet.openstreetmap.org/pbf/planet-180319.osm.pbf",
    "planet_date": "latest",
    "version": "0.2.0"
  },
  {
    "url_template": "https://tiles.sharedstreets.io/planet-180319/{z}-{x}-{y}.{layer}.pbf",
    "planet_url": "http://planet.openstreetmap.org/pbf/planet-180319.osm.pbf",
    "planet_date": "2018-03-19",
    "version": "0.2.0"
  },
  {
    "url_template": "https://tiles.sharedstreets.io/planet-180312/{z}-{x}-{y}.{layer}.pbf",
    "planet_url": "http://planet.openstreetmap.org/pbf/planet-180312.osm.pbf",
    "planet_date": "2018-03-12",
    "version": "0.1.0"
  }
]

We also talked a bit about different build levels for tiles based on road class. Right now, the SharedStreets tiles include all roads down to small service alleys and parking lots. A tile URL might include a minimum road class to slice the data:

• All roads: https://tiles.sharedstreets.io/planet-latest/{z}-{x}-{y}.{layer}.pbf
• Roads at and above a specific class: https://tiles.sharedstreets.io/planet-latest/{z}-{x}-{y}.{layer}.{roadClass}.pbf
  • Residential streets and larger: https://tiles.sharedstreets.io/planet-latest/{z}-{x}-{y}.{layer}.6.pbf
  • …
  • Motorways only: https://tiles.sharedstreets.io/planet-latest/{z}-{x}-{y}.{layer}.0.pbf

Feedback on this proposal welcome!

/cc @hassoncs

[DenisCarriere]

@migurski 👍 Looks rock solid!

I really like the fact that you could extract only certain layers, that would benefit low bandwidth connectivity and only deliver the data that's needed.

It could be interesting to play with.

👍 I like the URI Template sample

[migurski]

Thanks @DenisCarriere! I’m excited to see this move forward, and it’ll be possible to do a lot of cool stuff wth an internally-consistent planet. URLs are currently a blocker for one PR: sharedstreets/sharedstreets-python#7

[kpwebb]

@migurski and @DenisCarriere this looks great from my side. I can start working on the layer implementation now -- we can already generate these using the builder app.

Two additional ideas:

1. what do you all think about adding a tile source to the URL schema to allow for tiles from OSM and other data providers (e.g. public authorities, proprietary maps willing to share their references, etc.). Something like this:

https://tiles.sharedstreets.io/{source}/{tileset-version}/{z}-{x}-{y}.{layer}.0.pbf

So OSM would look like:

https://tiles.sharedstreets.io/osm/planet-latest/{z}-{x}-{y}.{layer}.0.pbf

2. I'm also curious about adding a really simple header line to each tile that says where it comes from. Could be as simple as:

{ source: "osm/planet-latest" last_modified: {timestamp} }

I think everything else is in the file name but could make the z/x/y +layer part of the header to improve legibility.

[DenisCarriere]

👍 I like including {source} to easily expand to other data sources.

I'm thinking maybe the {roadClass} might a bit too much customization, that's going to be a lot of maintenance to host new datasets if we also have support an additional 6+ road classes pbf datasets.

I'd stick with the essentials at first, and if we see the need to include {roadClass} because the file sizes are getting too big, then we can include it later on.

URL Template

https://tiles.sharedstreets.io/{source}/{tileset-version}/{z}-{x}-{y}.{layer}.pbf

I'm also curious about adding a really simple header line to each tile that says where it comes from. Could be as simple as:

👍 Sounds like a good idea, would that be stored in the PBF? Or stored as a *.json file?

We could host two files per tile:

• *.pbf (tile data)
• *.json (metadata)

[josiekre]

I've been following along as you carry out this discussion. I thought I would throw my two cents in.

I think @migurski's proposal to handle {roadClass} layers is elegant. My development team struggles to handle displaying, for example, volume data on all streets when zoomed out on a map. We have brainstormed in the past and thought splitting road classes into different layers would solve the problem so that data-driven styling is snappier no matter the zoom.

I think it would make more sense from a user perspective too. When zoomed out far, I see only traffic volumes (traffic counts) on motorways and trunks, for example. As I zoom in, I see traffic volumes on local roads.

This was actually what caught my attention and had me following this issue.

[migurski]

+1 to {source}, looks good.

I’ll say a bit more about {roadClass} to explain where the idea is coming from. Right now in our Remix work, we're finding that tiny non-roads like parking lot aisles, alleys, and driveways are interfering with our use of the graph. We’ve looked at filtering them out, but it’s not always possible to do this correctly near tile boundaries. It’d be easier to have multiple levels of road class cutoff, allowing us to pick and show all driveable roads.

[kpwebb]

Just to second @josiekre and @migurski's comments about {roadClass} We need to be able to build complete SharedStreets references for various levels in the road hierarchy for a variety of reasons, and because of the tile boundaries it's not easy to do filtering on the client. We can automatically generate these alternative tiles during the build stage.

A few clarifying questions:

1. @migurski can you share a bit more about how you imagine the {layer} attribute working? I get the {roadClass} (driven by the OSM tag classification) but not sure I understand how the layer attribute works

2. How should we handle segments with multiple OSM classifications within the segment? Right now anything that has two or more mergeable classifications we're combining them and labeling it as unclassified. It's a bit of a hack but perhaps we can get away with it because it's just an edge case.

[highway=primary]     [highway=secondary]
===================*======================
         [roadClass=unclassified]

As we start removing roads with excluded classes and merging things that were previously real intersections, I suspect that this will pop up frequently:

              [excluded road]                 
                    | 
[highway=primary]   |  [highway=secondary]
====================*=====================
        [roadClass=unclassified]

Should we change the underlying "unclassified" concept to something more robust? Perhaps we categorize as the highest class road in the segment but flag that theres more than one in the segment?

3. As an FYI, we also have this "form of way" concept that we're including in the references. This borrows from OpenLR and is included primarily for compatibility with the OpenLR spec, however, I'm curious if it is useful as a layer of abstraction that sits on top of the OSM tags?

[migurski]

{layer} comes directly from the current tile URLs with one of the four possible tile name extensions: “reference“, “intersection”, “geometry“, and “metadata”.

The way I was imagining classification, it would happen prior to conversion from OSM via a filter. In the example above, there might be a tile level with the highway=primary included, and the highway=secondary excluded. That might lead to a missed intersection, which I’m unsure is important. It’s one of the many cases where some graph awareness is necessary to merge different SharedStreets data sources together.

[kpwebb]

@migurski doh! of course! sorry for being dense about the layer thing.

This approach to creating the roadClass tiles works great - the builder already roadClass filter ways during the OSM ingest and we can just iterate through the list of possible filter options:

https://github.com/sharedstreets/sharedstreets-builder/blob/master/src/main/java/io/sharedstreets/tools/builder/osm/OSMDataStream.java#L244

I guess my main question is if the you're thinking of the roadClass filter as being something like highway=primary or greater (how it currently works) or if it should filter exclusively filter out one class per tile?

If it's, "{roadClass} or greater" I'm curious to get your take on how we should merge connecting segments of different classes (per point 2 above)?

Currently we merge different classes together, but we could switch to keeping each class discreet. We're merging aggressively, and creating mixed roadClass references to prevent poorly mapped OSM ways getting split up when they should be a contiguous reference.

I don't have a strong opinion about how do this, and am curious to get other takes on how to handle

[migurski]

I’m thinking of it as highway=primary or greater. If you got residential streets, you’d also get primary roads and motorways in the same tile. The reverse might not be true to traverse just graphs of highways and major roads.

Skipping a few levels for brevity, it could look like this:

• Minor roads, major roads, and motorways: https://tiles.sharedstreets.io/planet-latest/{z}-{x}-{y}.{layer}.6.pbf
• Major roads and motorways: https://tiles.sharedstreets.io/planet-latest/{z}-{x}-{y}.{layer}.3.pbf
• Motorways only: https://tiles.sharedstreets.io/planet-latest/{z}-{x}-{y}.{layer}.0.pbf

[josiekre]

+1 @migurski

cc: @gregmacfarlane (my colleague who might want to weigh in)

[kpwebb]

@migurski @josiekre Fantastic -- this works for me!

Any thoughts on the question about combining different roadClass segments together per:

              [excluded road]                 
                    | 
[highway=primary]   |  [highway=secondary]
====================*=====================
               [roadClass=??]

[migurski]

I’d choose either the higher or the lower value for this case — unsure if that would be roadClass=2 (primary) or roadClass=3 (secondary). Probably the latter, since it’s the most limiting class along the segment.

So,

              [excluded road]                 
                    | 
[highway=primary]   |  [highway=secondary]
====================*=====================
               [roadClass=3]

[kpwebb]

@migurski Thanks for all the feedback -- I've taken a stab at implementing this as described above. For testing purposes I'm producing tiles for each classification breakpoint.

Just from experimenting with SF data it looks like a lot of streets are "highway=unclassified" which means that they only show up in level 6 or below.

There are also a lot of surprises in the higher levels (e.g. Market is mapped a tertiary route through most of SOMA...), so we may end up just sticking with level 6 as the primary layer until we can use this is drive improvements in the OSM QA process.

Screenshot of level 6 tiles for SF:

I'm attaching a zip of all the tiles (level 0-7) for SF. Once you have a chance to take a look, let me know and I'll kick off the process for a global tile build.

sf_test_tiles.zip

[josiekre]

@gregmacfarlane and I will be in the same workspace next week and plan to dedicate an hour or two to dive into SharedStreets repos/documentation for CityCast. We'll be brainstorming then about this topic, and the SF tiles will be helpful.

Greg and I both agree that the OSM data with regards to road class are still messy enough that maybe this discussion is putting the cart before the horse, but it's true that developing this could improve the OSM QA process in a really good way.