OpenStreetMap (OSM) has been hosting raster tiles of its dataset for much of its 20-year history. These maps have had their rules and styles defined ahead of their rendering and present the end user with static PNGs. Below is a screenshot of the area around the Burj Khalifa in Dubai.
This week, OSM began hosting vector tiles in Mapbox Vector Tiles (MVT) format.
This allows for the end user to adjust the style and rendering rules as well as extract the underlying information within each of the tiles. Imagery should appear much sharper and switching the language of the labels should become possible.
Below is the same area in Dubai but rendered with OSM's new vector tiles.
The raster imagery OSM produced used larger iconography for points of interest (POI) but I suspect it is only a matter of time before the OSM community releases some spectacular and fine-tuned styles for the new vector tiles.
This is the current iteration of the tiles rendering with a style sheet that I'll discuss in further detail below.
The main OSM website still serves raster tiles but this web demo of the new vector tiles should work on most phones and desktop devices.
In this post, I'll walk through visualising and analysing OSM's vector tiles.
My Workstation
I'm using a 6 GHz Intel Core i9-14900K CPU. It has 8 performance cores and 16 efficiency cores with a total of 32 threads and 32 MB of L2 cache. It has a liquid cooler attached and is housed in a spacious, full-sized, Cooler Master HAF 700 computer case. I've come across videos on YouTube where people have managed to overclock the i9-14900KF to 9.1 GHz.
The system has 96 GB of DDR5 RAM clocked at 6,000 MT/s and a 5th-generation, Crucial T700 4 TB NVMe M.2 SSD which can read at speeds up to 12,400 MB/s. There is a heatsink on the SSD to help keep its temperature down. This is my system's C drive.
The system is powered by a 1,200-watt, fully modular, Corsair Power Supply and is sat on an ASRock Z790 Pro RS Motherboard.
I'm running Ubuntu 22 LTS via Microsoft's Ubuntu for Windows on Windows 11 Pro. In case you're wondering why I don't run a Linux-based desktop as my primary work environment, I'm still using an Nvidia GTX 1080 GPU which has better driver support on Windows and I use ArcGIS Pro from time to time which only supports Windows natively.
Installing Prerequisites
I'll use Python and a few other tools to help visualise OSM's data in this post.
$ sudo apt update
$ sudo apt install \
jq \
python3-pip \
python3-virtualenv
I'll set up a Python Virtual Environment and install some dependencies.
$ python3 -m venv ~/.osm_mvt
$ source ~/.osm_mvt/bin/activate
$ pip install \
"leafmap[maplibre]" \
mapbox_vector_tile \
morecantile \
notebook
I'll be using a tiles2columns utility that I've been working on in this post.
$ git clone https://github.com/marklit/tiles2columns ~/tiles2columns
$ python -m pip install -r ~/tiles2columns/requirements.txt
I'll use DuckDB, along with its H3, JSON, Lindel, Parquet and Spatial extensions, in this post.
$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.1.3/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb
INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;
I'll set up DuckDB to load every installed extension each time it launches.
$ vi ~/.duckdbrc
.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;
The maps in this post were rendered with QGIS version 3.40. QGIS is a desktop application that runs on Windows, macOS and Linux. The application has grown in popularity in recent years and has ~15M application launches from users all around the world each month.
OSM's Vector Tiles in QGIS
You can view OSM's vector tiles, along with an example style sheet in QGIS by clicking the Layer Menu -> Add Layer -> Add Vector Tile Layer.
Create a new connection and set its Style URL to:
https://pnorman.github.io/tilekiln-shortbread-demo/colorful.json
Then set the Source URL to:
https://vector.openstreetmap.org/shortbread_v1/{z}/{x}/{y}.mvt
Below is what the connection UI should look like. Click OK and then Add to add OSM's vector tiles to your project.
Below is a rendering of Dubai with the connection that was set up above.
The style sheet is optional and without it, QGIS can still render the vector data, albeit with colours it'll pick randomly.
The only issue I've come across is the rendering of iconography seems to be off. Other mapping tools can render the sprites for this style just fine but they appear as blurry black icons in QGIS for some reason.
OSM's Vector Tiles in Leafmap
I'll create a configuration folder for Jupyter Notebook and set a password.
$ mkdir -p ~/.jupyter
$ jupyter-notebook password
The following will launch Jupyter Notebook. Note it'll run under your user's permissions and expose any files in its current directory or below to anyone who knows the password you set. I'll run the server from an empty folder as a safety precaution.
$ mkdir -p ~/empty
$ cd ~/empty
$ jupyter-notebook \
--no-browser \
--ip=127.0.0.1 \
--NotebookApp.iopub_data_rate_limit=100000000
I'll open http://127.0.0.1:8888/ and type in the password I set above. I'll click the "New" button in the top right and pick "Python 3 (ipykernel)" to create a new notebook to work in.
Below I'll render the area around the Burj Khalifa again. The POI icons render properly.
import leafmap.maplibregl as leafmap
m = leafmap.Map(style='https://pnorman.github.io/tilekiln-shortbread-demo/colorful.json')
m
Analysis-Ready Data
Below I'll download a zoom-level 14 tile near the Burj Khalifa.
I'll first get its tile x and y values from its approximate longitude and latitude.
$ echo "[55.27, 25.2]" \
| morecantile tiles 14
[10707, 7006, 14]
I'll then use those values to build the tile URL and download its contents.
$ wget https://vector.openstreetmap.org/shortbread_v1/14/10707/7006.mvt
The following will convert the 114 KB MVT file into a 1.4 MB JSON file.
$ python3
import json
import math
import mapbox_vector_tile
# From https://gis.stackexchange.com/questions/475398/
def pixel2deg(xtile, ytile, zoom, xpixel, ypixel, extent=4096):
n = 2.0 ** zoom
xtile = xtile + (xpixel / extent)
ytile = ytile + ((extent - ypixel) / extent)
lon_deg = (xtile / n) * 360.0 - 180.0
lat_rad = math.atan(math.sinh(math.pi * (1 - 2 * ytile / n)))
lat_deg = math.degrees(lat_rad)
return (lon_deg, lat_deg)
open('7006.mvt.json', 'w').write(
json.dumps(
mapbox_vector_tile.decode(
tile=open('7006.mvt', 'rb').read(),
transformer=lambda x, y: pixel2deg(10707, 7006, 14, x, y))))
Below are the top-level keys for the tile.
$ jq -S keys 7006.mvt.json
[
"addresses",
"bridges",
"buildings",
"ferries",
"land",
"ocean",
"pier_lines",
"pier_polygons",
"place_labels",
"pois",
"public_transport",
"sites",
"street_labels",
"street_polygons",
"streets",
"streets_labels_points",
"water_polygons",
"water_polygons_labels"
]
I'll extract the POIs into their own JSON file and open it in DuckDB.
$ jq .pois 7006.mvt.json \
> 7006.pois.mvt.json
$ ~/duckdb
Below are the records where a cuisine value is present.
.maxrows 10
SELECT geom,
amenity,
cuisine
FROM ST_READ('7006.pois.mvt.json')
WHERE cuisine IS NOT NULL;
┌───────────────────────────────────────────────┬────────────┬─────────────┐
│ geom │ amenity │ cuisine │
│ geometry │ varchar │ varchar │
├───────────────────────────────────────────────┼────────────┼─────────────┤
│ POINT (55.27600407600403 25.19474315833422) │ restaurant │ french │
│ POINT (55.27602016925812 25.194947029527537) │ fast_food │ greek │
│ POINT (55.27712523937225 25.194786845047236) │ restaurant │ american │
│ POINT (55.27717888355255 25.194961591742572) │ restaurant │ lebanese │
│ POINT (55.27799963951111 25.19481596951387) │ restaurant │ lebanese │
│ · │ · │ · │
│ · │ · │ · │
│ · │ · │ · │
│ POINT (55.265355706214905 25.189505493297244) │ cafe │ coffee_shop │
│ POINT (55.265318155288696 25.189209381318772) │ restaurant │ asian │
│ POINT (55.265103578567505 25.189209381318772) │ restaurant │ persian │
│ POINT (55.263118743896484 25.18939384460301) │ fast_food │ chicken │
│ POINT (55.263070464134216 25.189340447365243) │ restaurant │ pizza │
├───────────────────────────────────────────────┴────────────┴─────────────┤
│ 67 rows (10 shown) 3 columns │
└──────────────────────────────────────────────────────────────────────────┘
The null_percentage and approx_unique fields below give you an idea of how well-populated this tile's dataset is.
.maxrows 100
SUMMARIZE FROM ST_READ('7006.pois.mvt.json');
┌─────────────────────────┬─────────────┬─────────────────────────────────────────────┬──────────────────────────────────────────────┬───────────────┬───────┬───────┬───────┬───────┬───────┬───────┬─────────────────┐
│ column_name │ column_type │ min │ max │ approx_unique │ avg │ std │ q25 │ q50 │ q75 │ count │ null_percentage │
│ varchar │ varchar │ varchar │ varchar │ int64 │ int32 │ int32 │ int32 │ int32 │ int32 │ int64 │ decimal(9,2) │
├─────────────────────────┼─────────────┼─────────────────────────────────────────────┼──────────────────────────────────────────────┼───────────────┼───────┼───────┼───────┼───────┼───────┼───────┼─────────────────┤
│ emergency │ VARCHAR │ fire_hydrant │ fire_hydrant │ 1 │ │ │ │ │ │ 474 │ 99.16 │
│ name │ VARCHAR │ % Arabica │ نافورة دبي │ 258 │ │ │ │ │ │ 474 │ 29.96 │
│ name_de │ VARCHAR │ % Arabica │ نافورة دبي │ 264 │ │ │ │ │ │ 474 │ 29.96 │
│ name_en │ VARCHAR │ % Arabica │ مول دبي حمامات +تواليت │ 245 │ │ │ │ │ │ 474 │ 29.96 │
│ amenity │ VARCHAR │ atm │ waste_basket │ 27 │ │ │ │ │ │ 474 │ 31.86 │
│ atm │ BOOLEAN │ false │ true │ 2 │ │ │ │ │ │ 474 │ 97.68 │
│ historic │ VARCHAR │ memorial │ monument │ 2 │ │ │ │ │ │ 474 │ 99.37 │
│ housename │ VARCHAR │ Level 3, Souk al Bahar │ Level 3, Souk al Bahar │ 1 │ │ │ │ │ │ 474 │ 99.79 │
│ leisure │ VARCHAR │ ice_rink │ swimming_pool │ 3 │ │ │ │ │ │ 474 │ 96.41 │
│ man_made │ VARCHAR │ surveillance │ tower │ 2 │ │ │ │ │ │ 474 │ 98.52 │
│ office │ VARCHAR │ diplomatic │ diplomatic │ 1 │ │ │ │ │ │ 474 │ 99.58 │
│ recycling:glass_bottles │ BOOLEAN │ false │ false │ 1 │ │ │ │ │ │ 474 │ 99.79 │
│ recycling:paper │ BOOLEAN │ false │ false │ 1 │ │ │ │ │ │ 474 │ 99.79 │
│ recycling:clothes │ BOOLEAN │ false │ false │ 1 │ │ │ │ │ │ 474 │ 99.79 │
│ recycling:scrap_metal │ BOOLEAN │ false │ false │ 1 │ │ │ │ │ │ 474 │ 99.79 │
│ shop │ VARCHAR │ bakery │ travel_agency │ 19 │ │ │ │ │ │ 474 │ 83.76 │
│ tourism │ VARCHAR │ hotel │ viewpoint │ 5 │ │ │ │ │ │ 474 │ 91.35 │
│ housenumber │ VARCHAR │ 13 │ roof top sofitel downtown │ 18 │ │ │ │ │ │ 474 │ 95.99 │
│ cuisine │ VARCHAR │ american │ turkish;lebanese;kebab │ 39 │ │ │ │ │ │ 474 │ 85.86 │
│ religion │ VARCHAR │ muslim │ muslim │ 1 │ │ │ │ │ │ 474 │ 99.16 │
│ denomination │ VARCHAR │ sunni │ sunni │ 1 │ │ │ │ │ │ 474 │ 99.79 │
│ tower:type │ VARCHAR │ defensive │ defensive │ 1 │ │ │ │ │ │ 474 │ 99.37 │
│ geom │ GEOMETRY │ POINT (55.2777099609375 25.196383826354648) │ POINT (55.27044653892517 25.187825897783735) │ 511 │ │ │ │ │ │ 474 │ 0.00 │
├─────────────────────────┴─────────────┴─────────────────────────────────────────────┴──────────────────────────────────────────────┴───────────────┴───────┴───────┴───────┴───────┴───────┴───────┴─────────────────┤
│ 23 rows 12 columns │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
tiles2columns
Earlier this year I built a tool called osm_split that would take GeoFabrik-generated PBF files of OSM data and convert them into named GeoPackage files.
These PBF files will inevitably be a few days old by the time they can be processed whereas OSM's new vector tiles are updated shortly after their underlying data is edited.
For this post, I've built a Python-based utility called tiles2columns that can download OSM tiles for a given bounding box and convert them into either GeoPackage or Parquet files.
Being able to package up OSM data as soon as it's edited into named, easy-to-use files is a big step forward for consuming OSM data.
Below I'll install tiles2columns.
$ python3 -m venv ~/.tiles2columns
$ source ~/.tiles2columns/bin/activate
$ git clone https://github.com/marklit/tiles2columns \
~/tiles2columns
$ python -m pip install -r \
~/tiles2columns/requirements.txt
I'll walk through extracting vector data from OSM for an area around Northern Dubai.
$ mkdir -p ~/dubai
$ cd ~/dubai
$ python3 ~/tiles2columns/main.py \
bbox \
55.2112 25.2745 \
55.34279 25.17104
The above downloaded 42 tiles and produced a few MBs of GeoPackage files.
$ ls -lhS *.gpkg
6.8M .. streets.gpkg
6.2M .. buildings.gpkg
1.4M .. pois.gpkg
1.4M .. street_labels.gpkg
744K .. land.gpkg
372K .. sites.gpkg
248K .. addresses.gpkg
244K .. water_polygons.gpkg
216K .. ocean.gpkg
196K .. pier_polygons.gpkg
164K .. public_transport.gpkg
132K .. pier_lines.gpkg
124K .. streets_labels_points.gpkg
112K .. ferries.gpkg
104K .. bridges.gpkg
104K .. place_labels.gpkg
104K .. street_polygons.gpkg
104K .. water_lines.gpkg
96K .. dam_polygons.gpkg
96K .. water_lines_labels.gpkg
96K .. water_polygons_labels.gpkg
These can be dropped into a QGIS project.
They're broken up by the key used in OSM's MVT files and should be easier to identify, style and analyse independently of one another.
The properties of each piece of geometry live in their own columns.
$ ~/duckdb
.maxrows 20
SELECT geom,
name
FROM ST_READ('pois.gpkg')
WHERE name IS NOT NULL;
┌───────────────────────────────────────────────┬─────────────────────────────────────────────┐
│ geom │ name │
│ geometry │ varchar │
├───────────────────────────────────────────────┼─────────────────────────────────────────────┤
│ POINT (55.34712553024292 25.24918890107886) │ Kiku │
│ POINT (55.34385323524475 25.248873526657135) │ Fujiya Restaurant │
│ POINT (55.34536063671112 25.24823792341191) │ Mövenpick Grand Al Bustan Dubai │
│ POINT (55.3471577167511 25.248616374599663) │ Sukhothai │
│ POINT (55.34707725048065 25.248858970894826) │ Méridien Village Terrace │
│ POINT (55.347533226013184 25.24869400546682) │ M's Beef Bistro │
│ POINT (55.347495675086975 25.248757080509858) │ M's Seafood Bistro │
│ POINT (55.34720063209534 25.248645486180656) │ Le Méridien Dubai Hotel & Conference Centre │
│ POINT (55.34764587879181 25.248529039814816) │ Long Yin │
│ POINT (55.34729182720184 25.24847081659004) │ Le Méridien │
│ · │ · │
│ · │ · │
│ · │ · │
│ POINT (55.34357964992523 25.250887056969173) │ Grand Mercure Dubai Airport │
│ POINT (55.343536734580994 25.250236908661204) │ Perfect Nail │
│ POINT (55.344228744506836 25.25049890914417) │ ibis Styles Dubai Airport │
│ POINT (55.343976616859436 25.250867649607375) │ Grand Mercure Dubai Airport │
│ POINT (55.3454464673996 25.251726422399933) │ Ashrafi │
│ POINT (55.34471154212952 25.25214852883526) │ Bin Hindi Outlet │
│ POINT (55.34568250179291 25.251959308890466) │ Restaurant │
│ POINT (55.349024534225464 25.25042127943035) │ Emirates NBD ATM │
│ POINT (55.344985127449036 25.257708552546877) │ The Draft House Sports Bar and Canteen │
│ POINT (55.344722270965576 25.25779102916772) │ McDonald's │
├───────────────────────────────────────────────┴─────────────────────────────────────────────┤
│ 6329 rows (20 shown) 2 columns │
└─────────────────────────────────────────────────────────────────────────────────────────────┘
Below I've rendered the POIs with name labels in QGIS.
