Tech Blog

12K+ JPEGs from NASA's Artemis II Mission

2026-05-08T20:00:00+03:00

In April, NASA sent four astronauts on a flyby mission around the Moon. They've since published 12,217 JPEGs that were captured during the 9-day journey.

In this post, I'll attempt to categorise the imagery and analyse the accompanying metadata.

My Workstation

I'm using a 5.7 GHz AMD Ryzen 9 9950X CPU. It has 16 cores and 32 threads and 1.2 MB of L1, 16 MB of L2 and 64 MB of L3 cache. It has a liquid cooler attached and is housed in a spacious, full-sized Cooler Master HAF 700 computer case.

The system has 96 GB of DDR5 RAM clocked at 4,800 MT/s and a 5th-generation, Crucial T700 4 TB NVMe M.2 SSD which can read at speeds up to 12,400 MB/s. There is a heatsink on the SSD to help keep its temperature down. This is my system's C drive.

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

I'm running Ubuntu 24 LTS via Microsoft's Ubuntu for Windows on Windows 11 Pro. In case you're wondering why I don't run a Linux-based desktop as my primary work environment, I'm still using an Nvidia GTX 1080 GPU which has better driver support on Windows and ArcGIS Pro only supports Windows natively.

Installing Prerequisites

I'll use Python 3.12.3 and a few other tools to help analyse the data in this post.

$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable
$ sudo apt update
$ sudo apt install \
    jq \
    libimage-exiftool-perl \
    python3-pip \
    python3.12-venv

I'll be using JSON Convert (jc) to convert the output of various CLI tools into JSON.

$ wget https://github.com/kellyjonbrazil/jc/releases/download/v1.25.2/jc_1.25.2-1_amd64.deb
$ sudo dpkg -i jc_1.25.2-1_amd64.deb

I'll set up a Python Virtual Environment and install a few dependencies needed to run one of OpenAI's models on NASA's imagery.

$ python3 -m venv ~/.apple_depth
$ source ~/.apple_depth/bin/activate
$ python3 -m pip install \
    'pillow==10.4.0' \
    rich \
    transformers

I'll be using a Python-based image contact sheet generator in this post.

$ git clone https://github.com/cobanov/contact-sheet \
    ~/contact-sheet

$ pip install -r \
    ~/contact-sheet/requirements.txt

I'll use DuckDB, along with its H3, JSON, Lindel, Parquet and Spatial extensions in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.5.1/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

Downloading 12K+ JPEGs

I'll create a folder for the contents of this post.

$ mkdir -p ~/artemis2
$ cd ~/artemis2

NASA lists and serves the images on a Perl-based portal. I had to copy the list of image IDs and paste them into a text file.

The file contains 12,217 lines.

$ wc -l manifest.txt # 12217

Below are a few example lines.

$ sort -R manifest.txt | head

ART002-E-20055
ART002-E-29232
ART002-E-11714
ART002-E-24812
ART002-E-23347
ART002-E-29751
ART002-E-11580
ART002-E-26846
ART002-E-20928
ART002-E-10963

I used Python to generate a BASH file containing wget commands to download each image.

$ python3

from pathlib import Path


with open('run.bash', 'w') as f:
    for line in open('manifest.txt'):
        id_ = int(line.split('-')[2])

        if not Path("%d.JPG" % id_).is_file():
            f.write("wget -c -O \"%d.JPG\" \"https://eol.jsc.nasa.gov/DatabaseImages/ESC/large/ART002/ART002-E-%d.JPG\"\n" % (id_, id_))

The following will run 16 concurrent downloads from the above BASH script.

$ cat run.bash \
    | xargs -n1 \
            -P16 \
            -I% \
            bash -c "%"

Some of the downloads failed for one reason or another. I ran the following to remove any 0-byte files. Following that, I regenerated the BASH script and re-ran it to try and see if any more images could be downloaded.

$ find . \
       -type f \
       -size 0 \
       -print0 \
    | xargs \
        -I{} \
        -0 \
        rm {}

I was able to download a total of 11,362 JPEGs totalling 14 GB altogether. 885 images are still outstanding.

$ ls *.JPG | wc -l # 11362
$ du -hsc *.JPG | tail -n1 # 14 GB

Extracting EXIF Metadata

Below, I've extracted the EXIF metadata from each image into its own JSON-formatted line.

$ touch metadata.json; rm metadata.json

$ for FILENAME in *.JPG; do
      exiftool $FILENAME \
        | jc --kv \
        | jq -Sc . \
        >> metadata.json
  done

I usually import JSON files directly into DuckDB but the JSONL files have at least 7 or 8 different schemas. DuckDB will only import these as a single JSON map column instead of as a few-hundred-column table.

To get around this, I cherry-picked columns with keywords in their names and produced a resulting JSONL file with far fewer columns.

$ python3

import json


keywords = (
    'aperture',
    'exposure',
    'focal',
    'shutter',
    'camera',
    'lens',
    'date',
    'time',
)

with open('metadata.cleaned.json', 'w') as f:
    for line in open('metadata.json'):
        f.write(
            json.dumps(
                {k.lower().replace(' ', '_'): v
                 for k, v in json.loads(line).items()
                 if any(True
                        for keyword in keywords
                        if keyword.lower() in k.lower() or
                           k.lower() == 'file name')},
                sort_keys=True,
                default=str) + '\n')

Below is a breakdown of unique values and NULL coverage across each column. I've excluded columns with fewer than two unique values and a few with very long names that I'm not interested in.

$ ~/duckdb

.maxrows 100

SELECT   column_name,
         null_percentage,
         approx_unique,
         min[:30],
         max[:30]
FROM     (SUMMARIZE
          FROM 'metadata.cleaned.json')
WHERE    approx_unique > 1
AND      column_name NOT LIKE '%param%'
AND      column_name NOT LIKE '%perspective_model_scale_factor%'
ORDER BY LOWER(column_name);

┌──────────────────────────────────┬─────────────────┬───────────────┬────────────────────────────────┬────────────────────────────────┐
│           column_name            │ null_percentage │ approx_unique │            min[:30]            │            max[:30]            │
│             varchar              │  decimal(9,2)   │     int64     │            varchar             │            varchar             │
├──────────────────────────────────┼─────────────────┼───────────────┼────────────────────────────────┼────────────────────────────────┤
│ aperture                         │           11.34 │            22 │ 1.2                            │ 9.0                            │
│ aperture_value                   │           11.34 │            22 │ 1.2                            │ 9.0                            │
│ camera_model_name                │           11.34 │             2 │ NIKON D5                       │ NIKON Z 9                      │
│ camera_profile                   │           11.34 │             2 │ Adobe Standard                 │ Camera Standard                │
│ camera_profile_digest            │           11.34 │             4 │ 69725A6B5714B6777D9E68B8C0DC18 │ EF370D7B22F72601AB8DD41D8D45B0 │
│ camera_profiles_aperture_value   │           75.55 │            11 │ 2                              │ 8.918863                       │
│ camera_profiles_focal_length     │           75.55 │            18 │ 125                            │ 98                             │
│ camera_profiles_lens             │           75.55 │             3 │ 14-24mm f/2.8G                 │ VR 80-400mm f/4.5-5.6G         │
│ camera_profiles_lens_pretty_name │           75.55 │             3 │ 14-24mm f/2.8G                 │ VR 80-400mm f/4.5-5.6G         │
│ create_date                      │            0.00 │         11150 │ 2025:09:19 15:24:02.62         │ 2026:04:13 08:03:42-05:00      │
│ date/time_created                │            0.00 │          5515 │ 2025:09:19 15:24:02            │ 2026:04:13 08:03:42-05:00      │
│ date/time_original               │            0.00 │         11150 │ 2025:09:19 15:24:02.62         │ 2026:04:13 08:03:42-05:00      │
│ date_created                     │            0.00 │         11459 │ 2025:09:19 15:24:02.62         │ 2026:04:13 08:03:42-05:00      │
│ digital_creation_date            │            0.00 │            12 │ 2025:09:19                     │ 2026:04:13                     │
│ digital_creation_date/time       │            0.00 │          5166 │ 2025:09:19 15:24:02            │ 2026:04:13 08:03:42-05:00      │
│ digital_creation_time            │            0.00 │          3863 │ 00:04:19                       │ 23:59:18+00:00                 │
│ exposure_compensation            │           11.34 │            17 │ +1                             │ 0                              │
│ exposure_mode                    │           11.34 │             2 │ Auto                           │ Manual                         │
│ exposure_program                 │           11.34 │             2 │ Manual                         │ Program AE                     │
│ exposure_time                    │           11.34 │            53 │ 0.3                            │ 8                              │
│ file_access_date/time            │            0.00 │          1701 │ 2026:05:07 20:12:53+03:00      │ 2026:05:08 11:48:34+03:00      │
│ file_inode_change_date/time      │            0.00 │          3308 │ 2026:05:07 18:46:56+03:00      │ 2026:05:08 11:48:34+03:00      │
│ file_modification_date/time      │            0.00 │          2177 │ 2026:04:22 20:29:35+03:00      │ 2026:04:22 21:30:07+03:00      │
│ file_name                        │            0.00 │         11567 │ 10000.JPG                      │ 9999.JPG                       │
│ focal_length                     │           11.34 │            61 │ 100.0 mm (35 mm equivalent: 10 │ 98.0 mm (35 mm equivalent: 98. │
│ focal_length_in_35mm_format      │           11.34 │            50 │ 100 mm                         │ 98 mm                          │
│ focal_plane_x_resolution         │           11.34 │             2 │ 1552.056122                    │ 2301.324615                    │
│ focal_plane_y_resolution         │           11.34 │             2 │ 1552.056122                    │ 2301.324615                    │
│ hyperfocal_distance              │           11.34 │           410 │ 0.47 m                         │ 99.95 m                        │
│ lens                             │           11.34 │             5 │ 0.0 mm f/0.0                   │ VR 80-400mm f/4.5-5.6G         │
│ lens_id                          │           11.34 │             6 │ 0.0mm f/0.0                    │ VR 80-400mm f/4.5-5.6G         │
│ lens_info                        │           11.34 │             3 │ 0mm f/0                        │ 80-400mm f/4.5-5.6             │
│ lens_model                       │           11.34 │             5 │ 0.0 mm f/0.0                   │ VR 80-400mm f/4.5-5.6G         │
│ lens_profile_digest              │           78.59 │            76 │ 0084F848238B2AD219A96E222F8145 │ FC43CB862288DC032EBB3109258565 │
│ lens_profile_enable              │            0.00 │             2 │ 0                              │ 1                              │
│ max_aperture_value               │           35.79 │            10 │ 1.0                            │ 5.7                            │
│ metadata_date                    │            0.00 │          7217 │ 2026:04:03 04:25:44-05:00      │ 2026:04:21 17:31:25-05:00      │
│ modify_date                      │            0.00 │          7217 │ 2026:04:03 04:25:44-05:00      │ 2026:04:21 17:31:25-05:00      │
│ offset_time_digitized            │           64.22 │             2 │ +00:00                         │ -05:00                         │
│ recommended_exposure_index       │           11.34 │            23 │ 1000                           │ 8000                           │
│ shutter_speed                    │           11.34 │            53 │ 0.3                            │ 8                              │
│ shutter_speed_value              │           11.34 │            53 │ 0.3                            │ 8                              │
│ sub_sec_time_digitized           │           11.34 │            90 │ 00                             │ 99                             │
│ sub_sec_time_original            │           11.34 │            90 │ 00                             │ 99                             │
│ time_created                     │            0.00 │          3547 │ 00:04:19                       │ 23:59:18                       │
└──────────────────────────────────┴─────────────────┴───────────────┴────────────────────────────────┴────────────────────────────────┘

Image Classification

Below, I'll run each JPEG through a zero-shot image classifier OpenAI released a few years ago. I've given the model five classification labels to choose from.

I randomised the order of the images to assist with spot-checking the results during the hour it took to run.

$ python3

import json
from   glob import glob
import random

from PIL           import Image
from rich.progress import track
from transformers  import pipeline


detector  = pipeline(model='openai/clip-vit-large-patch14',
                     task='zero-shot-image-classification')
labels    = ['earth', 'moon', 'stars', 'blank', 'glare']

filenames = list(glob('*.JPG'))
random.shuffle(filenames)

with open('classifications.json', 'w') as f:
    for filename in track(filenames):
        resp = detector(Image.open(filename),
                        candidate_labels=labels)
        top_res = sorted(resp,
                         key=lambda x: x['score'],
                         reverse=True)[0]
        resp = {'filename': filename,
                'label':    top_res['label'],
                'score':    top_res['score']}
        f.write(json.dumps(resp, sort_keys=True) + '\n')

Joining Datasets

I'll import the camera EXIF metadata along with the classifications above into DuckDB and join them into a single table.

$ ~/duckdb nasa.duckdb

CREATE OR REPLACE TABLE classifications AS
    FROM 'classifications.json';

CREATE OR REPLACE TABLE camera_settings AS
    FROM 'metadata.cleaned.json';

CREATE OR REPLACE TABLE imagery AS
    FROM classifications c
    JOIN camera_settings s ON c.filename = s.file_name;

Contact Sheets

Below are the number of imges for each label.

$ ~/duckdb nasa.duckdb

SELECT   COUNT(*),
         label
FROM     imagery
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬─────────┐
│ count_star() │  label  │
│    int64     │ varchar │
├──────────────┼─────────┤
│         7028 │ moon    │
│         1490 │ stars   │
│         1250 │ earth   │
│         1030 │ blank   │
│          564 │ glare   │
└──────────────┴─────────┘

I'll generate a contact sheet for each classification label.

$ for LABEL in blank earth moon stars glare; do
      echo "SELECT filename
            FROM   classifications
            WHERE  label = '$LABEL'" \
          | ~/duckdb -list -noheader nasa.duckdb \
          > "${LABEL}.txt"
      python3 ~/contact-sheet/contact_sheet.py \
              --file_list "${LABEL}.txt" \
              "thumbnails.${LABEL}.jpeg"
  done

These are thumbnails of the images that the above model labelled as the Earth.

These are thumbnails of the images that the above model labelled as the Moon.

These are thumbnails of the images that the above model labelled as stars.

These are thumbnails of the images that the above model labelled as glare.

These are thumbnails of the images that the above model labelled as being blank.

Below are the image counts for each score bins for each label.

$ ~/duckdb nasa.duckdb

WITH a AS (
    SELECT   label,
             score_bin: ROUND(score, 1),
             num_pics:  COUNT(*)
    FROM     imagery
    GROUP BY 1, 2
)
PIVOT    a
ON       label,
USING    SUM(num_pics)
GROUP BY score_bin
ORDER BY score_bin;

┌───────────┬────────┬────────┬────────┬────────┬────────┐
│ score_bin │ blank  │ earth  │ glare  │  moon  │ stars  │
│  double   │ int128 │ int128 │ int128 │ int128 │ int128 │
├───────────┼────────┼────────┼────────┼────────┼────────┤
│       0.2 │   NULL │   NULL │   NULL │   NULL │      1 │
│       0.3 │     76 │     12 │     22 │    115 │    101 │
│       0.4 │    188 │     19 │    131 │    237 │    538 │
│       0.5 │    157 │     43 │    145 │    201 │    480 │
│       0.6 │    596 │     48 │     91 │    145 │     63 │
│       0.7 │     13 │     35 │     75 │    115 │     20 │
│       0.8 │   NULL │     44 │     52 │    108 │     42 │
│       0.9 │   NULL │    115 │     37 │    239 │    154 │
│       1.0 │   NULL │    934 │     11 │   5868 │     91 │
└───────────┴────────┴────────┴────────┴────────┴────────┘

Spain at Night

The following is 192.JPG. Spain can be seen in the bottom left of the Earth's surface with its southern coast pointing upward. The Sun appears to illuminate the Earth from behind and the city lights all around Spain's coast and Madrid are clearly visible.

This is the metadata for this image.

$ echo "SELECT COLUMNS(c -> c NOT LIKE '%profiles%')
        FROM   imagery
        WHERE  file_name = '192.JPG'
        LIMIT  1" \
    | ~/duckdb -json nasa.duckdb 2>&1 \
    | grep '{.*' \
    | tail -c+6 \
    | jq -S .

[
  {
    "aperture": "4.0",
    "aperture_value": "4.0",
    "camera_model_name": "NIKON D5",
    "camera_profile": "Adobe Standard",
    "camera_profile_digest": "DC0173EBB7ECE22257A40AD42B5C9460",
    "create_date": "2026:04:03 00:27:39.26",
    "date/time_created": "2026:04:03 00:27:39",
    "date/time_original": "2026:04:03 00:27:39.26",
    "date_created": "2026:04:03 00:27:39.26",
    "digital_creation_date": "2026:04:03",
    "digital_creation_date/time": "2026:04:03 00:27:39",
    "digital_creation_time": "00:27:39",
    "exposure_2012": "0.00",
    "exposure_compensation": "+1",
    "exposure_mode": "Manual",
    "exposure_program": "Manual",
    "exposure_time": "1/4",
    "file_access_date/time": "2026:05:08 10:57:40+03:00",
    "file_inode_change_date/time": "2026:05:07 18:47:47+03:00",
    "file_modification_date/time": "2026:04:22 20:29:40+03:00",
    "file_name": "192.JPG",
    "filename": "192.JPG",
    "focal_length": "22.0 mm (35 mm equivalent: 22.0 mm)",
    "focal_length_in_35mm_format": "22 mm",
    "focal_plane_resolution_unit": "cm",
    "focal_plane_x_resolution": "1552.056122",
    "focal_plane_y_resolution": "1552.056122",
    "hyperfocal_distance": "4.03 m",
    "label": "earth",
    "lens": "14.0-24.0 mm f/2.8",
    "lens_id": "AF-S Zoom-Nikkor 14-24mm f/2.8G ED",
    "lens_info": "14-24mm f/2.8",
    "lens_manual_distortion_amount": "0",
    "lens_model": "14.0-24.0 mm f/2.8",
    "lens_profile_digest": null,
    "lens_profile_distortion_scale": null,
    "lens_profile_enable": "0",
    "lens_profile_is_embedded": null,
    "lens_profile_name": null,
    "lens_profile_setup": null,
    "lens_profile_vignetting_scale": null,
    "look_parameters_camera_profile": "Adobe Standard",
    "max_aperture_value": "2.8",
    "metadata_date": "2026:04:03 06:54:26-05:00",
    "modify_date": "2026:04:03 06:54:26-05:00",
    "offset_time": "-05:00",
    "offset_time_digitized": null,
    "offset_time_original": null,
    "profile_date_time": "1998:02:09 06:49:00",
    "recommended_exposure_index": "51200",
    "score": 0.9894962906837463,
    "shutter_speed": "1/4",
    "shutter_speed_value": "1/4",
    "sub_sec_time_digitized": "26",
    "sub_sec_time_original": "26",
    "time_created": "00:27:39"
  }
]

My guess from the sea of timestamps above is that this image was taken at 2026-04-03 00:27:39 UTC which would be 02:27:39 AM local time on the Spanish mainland.

The exposure time was only 1/4th of a second which normally isn't enough for any nighttime photography but the Nikon D5 camera this image was taken with supports an ISO of 51200. This level will make any dark scene very grainy but it ended up being a good trade-off in this case.

Settings and Timestamps

Below are the image counts per day by camera model.

$ ~/duckdb nasa.duckdb

WITH a AS (
    SELECT   camera_model_name,
             created_at: REPLACE(create_date[:10], ':', '-')::DATE,
             num_pics: COUNT(*)
    FROM     imagery
    GROUP BY 1, 2
)
PIVOT    a
ON       camera_model_name
USING    SUM(num_pics)
GROUP BY created_at
ORDER BY 1;

┌────────────┬──────────┬───────────┐
│ created_at │ NIKON D5 │ NIKON Z 9 │
│    date    │  int128  │  int128   │
├────────────┼──────────┼───────────┤
│ 2025-09-19 │        1 │      NULL │
│ 2026-04-02 │      629 │      NULL │
│ 2026-04-03 │      315 │        17 │
│ 2026-04-04 │       77 │      NULL │
│ 2026-04-05 │      160 │       149 │
│ 2026-04-06 │     4606 │      1689 │
│ 2026-04-07 │     1430 │       871 │
│ 2026-04-08 │       37 │        17 │
│ 2026-04-09 │       32 │        35 │
│ 2026-04-10 │        9 │      NULL │
│ 2026-04-13 │     NULL │      NULL │
└────────────┴──────────┴───────────┘

These are the lens counts.

WITH a AS (
    SELECT   camera_model_name,
             lens,
             num_pics: COUNT(*)
    FROM     imagery
    WHERE    camera_model_name IS NOT NULL
    GROUP BY 1, 2
)
PIVOT    a
ON       camera_model_name
USING    SUM(num_pics)
GROUP BY lens
ORDER BY 2 DESC;

┌─────────────────────────┬──────────┬───────────┐
│          lens           │ NIKON D5 │ NIKON Z 9 │
│         varchar         │  int128  │  int128   │
├─────────────────────────┼──────────┼───────────┤
│ 80.0-400.0 mm f/4.5-5.6 │     6987 │      NULL │
│ 14.0-24.0 mm f/2.8      │      246 │      NULL │
│ 35.0 mm f/2.0           │       62 │      NULL │
│ 0.0 mm f/0.0            │        1 │      NULL │
│ VR 80-400mm f/4.5-5.6G  │     NULL │       704 │
│ 35mm f/2D               │     NULL │      1959 │
│ 14-24mm f/2.8G          │     NULL │       115 │
└─────────────────────────┴──────────┴───────────┘

These are the lens counts for each type of image taken.

WITH a AS (
    SELECT   label,
             lens,
             num_pics: COUNT(*)
    FROM     imagery
    WHERE    camera_model_name IS NOT NULL
    GROUP BY 1, 2
)
PIVOT    a
ON       label
USING    SUM(num_pics)
GROUP BY lens
ORDER BY 2 DESC;

┌─────────────────────────┬────────┬────────┬────────┬────────┬────────┐
│          lens           │ blank  │ earth  │ glare  │  moon  │ stars  │
│         varchar         │ int128 │ int128 │ int128 │ int128 │ int128 │
├─────────────────────────┼────────┼────────┼────────┼────────┼────────┤
│ 80.0-400.0 mm f/4.5-5.6 │    665 │   1061 │    370 │   4784 │    107 │
│ 35mm f/2D               │    186 │     43 │    181 │   1308 │    241 │
│ 14.0-24.0 mm f/2.8      │     26 │     73 │      1 │     71 │     75 │
│ 35.0 mm f/2.0           │      8 │     11 │   NULL │     34 │      9 │
│ 0.0 mm f/0.0            │      1 │   NULL │   NULL │   NULL │   NULL │
│ 14-24mm f/2.8G          │      1 │   NULL │      8 │     10 │     96 │
│ VR 80-400mm f/4.5-5.6G  │   NULL │     30 │   NULL │    674 │   NULL │
└─────────────────────────┴────────┴────────┴────────┴────────┴────────┘

Below is a breakdown of exposure and aperture settings across the images. The numeric columns are the aperture setting. I'm in the process of figuring out how to get DuckDB to sort numerically without needing to name them individually.

WITH a AS (
    SELECT   exposure_time,
             aperture,
             num_pics: COUNT(*)
    FROM     imagery
    WHERE    camera_model_name IS NOT NULL
    GROUP BY 1, 2
    HAVING   COUNT(*) > 100
)
PIVOT    a
ON       aperture
USING    SUM(num_pics)
GROUP BY exposure_time
ORDER BY IF(exposure_time LIKE '%/%',
            1 / SPLIT_PART(exposure_time, '/', 2)::FLOAT,
            exposure_time::FLOAT);

┌───────────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┐
│ exposure_time │  10.0  │  11.0  │  14.0  │  16.0  │  2.0   │  4.5   │  5.6   │  7.1   │  8.0   │
│    varchar    │ int128 │ int128 │ int128 │ int128 │ int128 │ int128 │ int128 │ int128 │ int128 │
├───────────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
│ 1/4000        │   NULL │   NULL │   NULL │    125 │   NULL │   NULL │   NULL │   NULL │   NULL │
│ 1/2000        │   NULL │   1227 │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │
│ 1/1600        │    233 │   NULL │    118 │    196 │   NULL │   NULL │   NULL │   NULL │   NULL │
│ 1/1000        │   NULL │    118 │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │   3272 │
│ 1/800         │    118 │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │    233 │   NULL │
│ 1/640         │    191 │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │
│ 1/500         │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │   1101 │   NULL │   NULL │
│ 1/400         │   NULL │   NULL │   NULL │    123 │   NULL │   NULL │   NULL │   NULL │   NULL │
│ 1/200         │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │    379 │   NULL │   NULL │
│ 1/80          │   NULL │   NULL │   NULL │   NULL │   NULL │    110 │   NULL │   NULL │   NULL │
│ 1/25          │   NULL │   NULL │   NULL │   NULL │   NULL │    128 │    174 │   NULL │   NULL │
│ 5             │   NULL │   NULL │   NULL │   NULL │    115 │   NULL │   NULL │   NULL │   NULL │
└───────────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┘

Below are the top five settings that were used across each type of image.

WITH b AS (
    WITH a AS (
        SELECT   label,
                 exposure_time,
                 aperture,
                 num_pics: COUNT(*)
        FROM     imagery
        GROUP BY 1, 2, 3
        ORDER BY 4 DESC
    )
    SELECT *,
           ROW_NUMBER() OVER (PARTITION BY label
                              ORDER BY     num_pics DESC) AS rn
    FROM   a
)
SELECT   * EXCLUDE(rn)
FROM     b
WHERE    rn < 6
ORDER BY label, rn;

┌─────────┬───────────────┬──────────┬──────────┐
│  label  │ exposure_time │ aperture │ num_pics │
│ varchar │    varchar    │ varchar  │  int64   │
├─────────┼───────────────┼──────────┼──────────┤
│ blank   │ 1/1600        │ 16.0     │      179 │
│ blank   │ 1/640         │ 10.0     │      162 │
│ blank   │ 1/200         │ 5.6      │      147 │
│ blank   │ NULL          │ NULL     │      143 │
│ blank   │ 1/12800       │ 5.6      │       72 │
│ earth   │ 1/1600        │ 14.0     │      114 │
│ earth   │ 1/1000        │ 11.0     │      114 │
│ earth   │ 1/800         │ 10.0     │      114 │
│ earth   │ 1/400         │ 16.0     │      108 │
│ earth   │ 1/1000        │ 8.0      │       99 │
│ glare   │ 1/200         │ 5.6      │      103 │
│ glare   │ 1/25          │ 5.6      │       83 │
│ glare   │ 1/25          │ 4.5      │       59 │
│ glare   │ 0.3           │ 5.6      │       58 │
│ glare   │ 1/80          │ 5.6      │       53 │
│ moon    │ 1/1000        │ 8.0      │     3164 │
│ moon    │ 1/2000        │ 11.0     │     1140 │
│ moon    │ 1/500         │ 5.6      │     1075 │
│ moon    │ 1/800         │ 7.1      │      170 │
│ moon    │ 1/1600        │ 10.0     │      161 │
│ stars   │ NULL          │ NULL     │      962 │
│ stars   │ 5             │ 2.0      │      115 │
│ stars   │ 3             │ 8.0      │       81 │
│ stars   │ 1/6           │ 4.5      │       48 │
│ stars   │ 1/200         │ 5.6      │       21 │
└─────────┴───────────────┴──────────┴──────────┘

These are the ISO levels that were used broken down by label. Note, 1,288 images used another field to record this information.

WITH a AS (
    SELECT   iso: recommended_exposure_index,
             label,
             num_pics: COUNT(*)
    FROM     imagery
    WHERE    recommended_exposure_index IS NOT NULL
    GROUP BY 1, 2
)
PIVOT    a
ON       label
USING    SUM(num_pics)
GROUP BY iso
ORDER BY iso::INT;

┌─────────┬────────┬────────┬────────┬────────┬────────┐
│   iso   │ blank  │ earth  │ glare  │  moon  │ stars  │
│ varchar │ int128 │ int128 │ int128 │ int128 │ int128 │
├─────────┼────────┼────────┼────────┼────────┼────────┤
│ 200     │   NULL │   NULL │   NULL │   NULL │      1 │
│ 400     │     89 │    949 │      8 │   6185 │     34 │
│ 500     │      4 │     80 │   NULL │     88 │   NULL │
│ 640     │   NULL │      6 │   NULL │   NULL │   NULL │
│ 800     │      1 │   NULL │      8 │      2 │     83 │
│ 1000    │      2 │   NULL │   NULL │      1 │   NULL │
│ 1250    │      3 │     19 │   NULL │      8 │   NULL │
│ 1600    │     15 │   NULL │     19 │    101 │      3 │
│ 2000    │      4 │   NULL │   NULL │      8 │      2 │
│ 2500    │   NULL │      3 │   NULL │   NULL │   NULL │
│ 3200    │   NULL │   NULL │   NULL │   NULL │      1 │
│ 4000    │      8 │      2 │   NULL │     26 │      8 │
│ 5000    │   NULL │   NULL │   NULL │      2 │      2 │
│ 6400    │    747 │     54 │    524 │    350 │     76 │
│ 8000    │   NULL │      3 │   NULL │      1 │      1 │
│ 12800   │     14 │     13 │      1 │     34 │    182 │
│ 16000   │   NULL │   NULL │   NULL │   NULL │      1 │
│ 20000   │   NULL │   NULL │   NULL │      1 │     14 │
│ 25600   │   NULL │   NULL │   NULL │   NULL │      4 │
│ 32000   │   NULL │   NULL │   NULL │   NULL │      1 │
│ 40000   │   NULL │      1 │   NULL │     39 │     31 │
│ 51200   │   NULL │     88 │   NULL │     35 │     84 │
└─────────┴────────┴────────┴────────┴────────┴────────┘

There were gaps in every day where there weren't any photos taken in this set of images they published. The bulk of the photos were taken from 4PM UTC on the 6th till 2AM on the 7th.

WITH a AS (
    SELECT   day:      SPLIT_PART(SPLIT_PART(create_date, ' ', 1), ':', 3),
             hour:     SPLIT_PART(SPLIT_PART(create_date, ' ', 2), ':', 1),
             num_pics: COUNT(*)
    FROM     imagery
    WHERE    create_date LIKE '2026:04%'
    GROUP BY 1, 2
    ORDER BY 1, 2
)
PIVOT    a
ON       day
USING    SUM(num_pics)
GROUP BY hour
ORDER BY hour;

┌─────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┐
│  hour   │   02   │   03   │   04   │   05   │   06   │   07   │   08   │   09   │   10   │   13   │
│ varchar │ int128 │ int128 │ int128 │ int128 │ int128 │ int128 │ int128 │ int128 │ int128 │ int128 │
├─────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
│ 00      │   NULL │    171 │   NULL │   NULL │   NULL │   1579 │      2 │      6 │      3 │   NULL │
│ 01      │   NULL │      1 │   NULL │     12 │     13 │    472 │   NULL │      1 │      2 │   NULL │
│ 02      │   NULL │      1 │     30 │   NULL │     32 │   NULL │   NULL │   NULL │   NULL │   NULL │
│ 03      │   NULL │   NULL │   NULL │     24 │    126 │   NULL │      6 │     27 │      1 │   NULL │
│ 04      │      5 │     20 │     30 │    168 │     93 │   NULL │      1 │     20 │   NULL │   NULL │
│ 05      │   NULL │   NULL │     17 │      2 │     23 │   NULL │   NULL │      1 │      3 │   NULL │
│ 06      │   NULL │   NULL │   NULL │     55 │   NULL │   NULL │      2 │   NULL │   NULL │   NULL │
│ 07      │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │      9 │   NULL │   NULL │    681 │
│ 08      │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │    540 │
│ 11      │     15 │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │
│ 12      │    254 │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │
│ 14      │   NULL │     13 │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │   NULL │
│ 15      │   NULL │   NULL │   NULL │   NULL │     10 │   NULL │   NULL │      3 │   NULL │   NULL │
│ 16      │   NULL │   NULL │   NULL │     48 │    624 │   NULL │      3 │      4 │      1 │   NULL │
│ 17      │   NULL │   NULL │   NULL │   NULL │    389 │      1 │     12 │   NULL │   NULL │   NULL │
│ 18      │     97 │   NULL │   NULL │   NULL │    148 │     16 │      6 │   NULL │   NULL │   NULL │
│ 19      │     63 │      6 │   NULL │   NULL │    738 │     61 │      2 │      1 │   NULL │   NULL │
│ 20      │     58 │   NULL │   NULL │   NULL │    980 │      2 │      5 │   NULL │   NULL │   NULL │
│ 21      │     92 │   NULL │   NULL │   NULL │   1834 │    156 │     19 │   NULL │   NULL │   NULL │
│ 22      │   NULL │   NULL │   NULL │   NULL │    830 │     30 │      6 │      3 │   NULL │   NULL │
│ 23      │     45 │    150 │   NULL │   NULL │    455 │   NULL │      1 │      1 │   NULL │   NULL │
└─────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┘

The vast majority of the photos of the Moon were taken on April 6th.

WITH a AS (
    SELECT   day: SPLIT_PART(SPLIT_PART(create_date, ' ', 1), ':', 3),
             label,
             num_pics: COUNT(*)
    FROM     imagery
    WHERE    create_date LIKE '2026:04%'
    GROUP BY 1, 2
    ORDER BY 1, 2
)
PIVOT    a
ON       label
USING    SUM(num_pics)
GROUP BY day
ORDER BY day;

┌─────────┬────────┬────────┬────────┬────────┬────────┐
│   day   │ blank  │ earth  │ glare  │  moon  │ stars  │
│ varchar │ int128 │ int128 │ int128 │ int128 │ int128 │
├─────────┼────────┼────────┼────────┼────────┼────────┤
│ 02      │     32 │    577 │   NULL │      9 │     11 │
│ 03      │      7 │    212 │      4 │     77 │     62 │
│ 04      │      3 │     42 │   NULL │     32 │   NULL │
│ 05      │     10 │     41 │   NULL │    258 │   NULL │
│ 06      │     43 │    289 │     11 │   5853 │     99 │
│ 07      │    776 │     56 │    545 │    626 │    314 │
│ 08      │      7 │   NULL │      4 │     30 │     33 │
│ 09      │      6 │     31 │   NULL │     17 │     13 │
│ 10      │      2 │      1 │   NULL │      5 │      2 │
│ 13      │    143 │      1 │   NULL │    121 │    956 │
└─────────┴────────┴────────┴────────┴────────┴────────┘

The following is the most common subject for each hour of the mission.

WITH c AS (
    WITH b AS (
        WITH a AS (
            SELECT   day:      SPLIT_PART(SPLIT_PART(create_date, ' ', 1), ':', 3),
                     hour:     SPLIT_PART(SPLIT_PART(create_date, ' ', 2), ':', 1),
                     label,
                     num_pics: COUNT(*)
            FROM     imagery
            WHERE    create_date LIKE '2026:04%'
            GROUP BY 1, 2, 3
            ORDER BY 1, 2
        )
        SELECT *,
               ROW_NUMBER() OVER (PARTITION BY day, hour
                                  ORDER BY     num_pics DESC) AS rn
        FROM   a
    )
    FROM     b
    WHERE    rn = 1
    ORDER BY num_pics DESC
)
PIVOT    c
ON       day
USING    MAX(label)
GROUP BY hour
ORDER BY hour;

┌─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐
│  hour   │   02    │   03    │   04    │   05    │   06    │   07    │   08    │   09    │   10    │   13    │
│ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │
├─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ 00      │ NULL    │ earth   │ NULL    │ NULL    │ NULL    │ blank   │ stars   │ moon    │ moon    │ NULL    │
│ 01      │ NULL    │ stars   │ NULL    │ earth   │ moon    │ glare   │ NULL    │ moon    │ moon    │ NULL    │
│ 02      │ NULL    │ moon    │ moon    │ NULL    │ moon    │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │
│ 03      │ NULL    │ NULL    │ NULL    │ moon    │ moon    │ NULL    │ moon    │ earth   │ blank   │ NULL    │
│ 04      │ blank   │ earth   │ earth   │ moon    │ stars   │ NULL    │ stars   │ stars   │ NULL    │ NULL    │
│ 05      │ NULL    │ NULL    │ earth   │ blank   │ moon    │ NULL    │ NULL    │ moon    │ moon    │ NULL    │
│ 06      │ NULL    │ NULL    │ NULL    │ earth   │ NULL    │ NULL    │ stars   │ NULL    │ NULL    │ NULL    │
│ 07      │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ stars   │ NULL    │ NULL    │ stars   │
│ 08      │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ stars   │
│ 11      │ earth   │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │
│ 12      │ earth   │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │
│ 14      │ NULL    │ earth   │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │
│ 15      │ NULL    │ NULL    │ NULL    │ NULL    │ moon    │ NULL    │ NULL    │ blank   │ NULL    │ NULL    │
│ 16      │ NULL    │ NULL    │ NULL    │ moon    │ moon    │ NULL    │ stars   │ stars   │ earth   │ NULL    │
│ 17      │ NULL    │ NULL    │ NULL    │ NULL    │ moon    │ stars   │ stars   │ NULL    │ NULL    │ NULL    │
│ 18      │ earth   │ NULL    │ NULL    │ NULL    │ moon    │ moon    │ blank   │ NULL    │ NULL    │ NULL    │
│ 19      │ earth   │ stars   │ NULL    │ NULL    │ moon    │ moon    │ blank   │ moon    │ NULL    │ NULL    │
│ 20      │ earth   │ NULL    │ NULL    │ NULL    │ moon    │ stars   │ moon    │ NULL    │ NULL    │ NULL    │
│ 21      │ earth   │ NULL    │ NULL    │ NULL    │ moon    │ stars   │ moon    │ NULL    │ NULL    │ NULL    │
│ 22      │ NULL    │ NULL    │ NULL    │ NULL    │ moon    │ stars   │ stars   │ blank   │ NULL    │ NULL    │
│ 23      │ earth   │ earth   │ NULL    │ NULL    │ earth   │ NULL    │ moon    │ moon    │ NULL    │ NULL    │
└─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘

Photographers

After the initial publication of this post, I was shown how to import the camera EXIF JSON file as a 336-column table.

$ ~/duckdb nasa.duckdb

CREATE OR REPLACE TABLE exif_settings AS
    FROM READ_JSON('metadata.json',
                   map_inference_threshold=-1);

While inspecting the values of each of the fields, I came across a description field which appears to note the astronaut that took each photo.

SELECT   COUNT(*),
         Description
FROM     exif_settings
GROUP BY 2
ORDER BY 1 DESC
LIMIT    20;

┌──────────────┬───────────────────────────────────────────────────────────────────────────────────────────────┐
│ count_star() │                                          Description                                          │
│    int64     │                                            varchar                                            │
├──────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤
│         2298 │ FD06_fd6 Lunar Flyby Wiseman                                                                  │
│         1754 │ FD10_Lunar flyby Koch SN 1015 long lens                                                       │
│         1630 │ FD07_FD6 Lunar flyby imagry - D5 short lens 1022 Koch, wiseman, glover -wrong SN these are Z9 │
│         1220 │ FD07_DockCam-WW-Imaging                                                                       │
│          658 │ FD02_Returned_1003_D5_015_Glover                                                              │
│          467 │ FD06_Returned_0025_Z9_019_Wiseman                                                             │
│          377 │ Hansen lunar flyby first shift onlybother shifts on other cards                               │
│          299 │ FD06_Returned_1013_D5_015_Wiseman                                                             │
│          294 │ FD06_Returned_1008_D5_015_Glover                                                              │
│          250 │ FD06_Returned_1015_D5_015_Koch                                                                │
│          186 │ FD07_Returned_0007_Z9_019                                                                     │
│          150 │ FD05_Returned_0021_Z9_019_Koch                                                                │
│          144 │ FD06_Lunar Flyby Glover 1008                                                                  │
│          137 │ FD06_Returned_1013_D5_017_Wiseman                                                             │
│          131 │ FD02_Returned_1004_D5_017                                                                     │
│          128 │ FD04_Returned_0022_Z9_019_Glover                                                              │
│          116 │ FD08_Lunar flyby Koch SN 1015 long lens prelim                                                │
│           83 │ FD03_Returned_1021_D5_015_Koch                                                                │
│           71 │ FD06_Returned_1014_D5_015_Koch                                                                │
│           65 │ FD05_Returned_1013_D5_015_Wiseman                                                             │
└──────────────┴───────────────────────────────────────────────────────────────────────────────────────────────┘

Not all images can be attributed to any one astronaut but most can.

CREATE OR REPLACE TABLE photographers AS
    SELECT file_name: "File Name",
           photographer:
                CASE WHEN Description ILIKE '%Koch%'    THEN 'Koch'
                     WHEN Description ILIKE '%Glover%'  THEN 'Glover'
                     WHEN Description ILIKE '%Hansen%'  THEN 'Hansen'
                     WHEN Description ILIKE '%Wiseman%' THEN 'Wiseman'
                END
    FROM   exif_settings;

SELECT   COUNT(*),
         photographer
FROM     photographers
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬──────────────┐
│ count_star() │ photographer │
│    int64     │   varchar    │
├──────────────┼──────────────┤
│         4347 │ Koch         │
│         3324 │ Wiseman      │
│         1956 │ NULL         │
│         1353 │ Glover       │
│          382 │ Hansen       │
└──────────────┴──────────────┘

Below, I'll join all three tables together.

CREATE OR REPLACE TABLE imagery AS
    FROM classifications c
    JOIN camera_settings s ON c.filename = s.file_name
    JOIN photographers   p ON c.filename = p.file_name;

This is the lowest number of images by subject captured by each of the astronauts. Note, 1,956 images in this set aren't attributed to any one photographer.

WITH a AS (
    SELECT   photographer,
             label,
             num_pics: COUNT(*)
    FROM     imagery
    WHERE    create_date LIKE '2026:04%'
    AND      photographer IS NOT NULL
    GROUP BY 1, 2
    ORDER BY 1, 2
)
PIVOT    a
ON       label
USING    SUM(num_pics)
GROUP BY photographer
ORDER BY photographer;

┌──────────────┬────────┬────────┬────────┬────────┬────────┐
│ photographer │ blank  │ earth  │ glare  │  moon  │ stars  │
│   varchar    │ int128 │ int128 │ int128 │ int128 │ int128 │
├──────────────┼────────┼────────┼────────┼────────┼────────┤
│ Glover       │     55 │    838 │      3 │    421 │     36 │
│ Hansen       │      2 │      1 │   NULL │    379 │   NULL │
│ Koch         │    196 │    185 │    186 │   3631 │    148 │
│ Wiseman      │    607 │     54 │    369 │   2219 │     75 │
└──────────────┴────────┴────────┴────────┴────────┴────────┘

These are the minimum number of photos each astronaut took during each day of the mission.

WITH a AS (
    SELECT   day: SPLIT_PART(SPLIT_PART(create_date, ' ', 1), ':', 3),
             photographer,
             num_pics: COUNT(*)
    FROM     imagery
    WHERE    create_date LIKE '2026:04%'
    AND      photographer IS NOT NULL
    GROUP BY 1, 2
    ORDER BY 1, 2
)
PIVOT    a
ON       photographer
USING    SUM(num_pics)
GROUP BY day
ORDER BY day;

┌─────────┬────────┬────────┬────────┬─────────┐
│   day   │ Glover │ Hansen │  Koch  │ Wiseman │
│ varchar │ int128 │ int128 │ int128 │ int128  │
├─────────┼────────┼────────┼────────┼─────────┤
│ 02      │    611 │   NULL │   NULL │    NULL │
│ 03      │     47 │   NULL │    137 │      19 │
│ 04      │   NULL │   NULL │     15 │      29 │
│ 05      │    140 │   NULL │     73 │      53 │
│ 06      │    537 │    377 │   3358 │    1879 │
│ 07      │      2 │   NULL │    750 │    1334 │
│ 08      │     16 │      5 │     13 │    NULL │
│ 09      │   NULL │   NULL │   NULL │      10 │
└─────────┴────────┴────────┴────────┴─────────┘

Jeremy Hansen was the only one not to shoot with the Nikon Z9.

WITH a AS (
    SELECT   camera_model_name,
             photographer,
             num_pics: COUNT(*)
    FROM     imagery
    WHERE    create_date LIKE '2026:04%'
    AND      camera_model_name IS NOT NULL
    GROUP BY 1, 2
    ORDER BY 1, 2
)
PIVOT    a
ON       photographer
USING    SUM(num_pics)
GROUP BY camera_model_name
ORDER BY camera_model_name;

┌───────────────────┬────────┬────────┬────────┬─────────┐
│ camera_model_name │ Glover │ Hansen │  Koch  │ Wiseman │
│      varchar      │ int128 │ int128 │ int128 │ int128  │
├───────────────────┼────────┼────────┼────────┼─────────┤
│ NIKON D5          │   1225 │    382 │   2480 │    2834 │
│ NIKON Z 9         │    128 │   NULL │   1866 │     490 │
└───────────────────┴────────┴────────┴────────┴─────────┘

Christina Koch used the broadest range of lenses.

WITH a AS (
    SELECT   lens,
             photographer,
             num_pics: COUNT(*)
    FROM     imagery
    WHERE    create_date LIKE '2026:04%'
    AND      photographer IS NOT NULL
    GROUP BY 1, 2
    ORDER BY 1, 2
)
PIVOT    a
ON       photographer
USING    SUM(num_pics)
GROUP BY lens
ORDER BY lens;

┌─────────────────────────┬────────┬────────┬────────┬─────────┐
│          lens           │ Glover │ Hansen │  Koch  │ Wiseman │
│         varchar         │ int128 │ int128 │ int128 │ int128  │
├─────────────────────────┼────────┼────────┼────────┼─────────┤
│ 14-24mm f/2.8G          │   NULL │   NULL │    102 │    NULL │
│ 14.0-24.0 mm f/2.8      │     17 │      5 │     12 │       3 │
│ 35.0 mm f/2.0           │      9 │   NULL │      7 │       3 │
│ 35mm f/2D               │     42 │   NULL │   1704 │      16 │
│ 80.0-400.0 mm f/4.5-5.6 │   1199 │    377 │   2461 │    2828 │
│ VR 80-400mm f/4.5-5.6G  │     86 │   NULL │     60 │     474 │
└─────────────────────────┴────────┴────────┴────────┴─────────┘

The following is the most prolific photographer for each hour of the mission.

WITH c AS (
    WITH b AS (
        WITH a AS (
            SELECT   day:      SPLIT_PART(SPLIT_PART(create_date, ' ', 1), ':', 3),
                     hour:     SPLIT_PART(SPLIT_PART(create_date, ' ', 2), ':', 1),
                     photographer,
                     num_pics: COUNT(*)
            FROM     imagery
            WHERE    create_date LIKE '2026:04%'
            AND      photographer IS NOT NULL
            GROUP BY 1, 2, 3
            ORDER BY 1, 2
        )
        SELECT *,
               ROW_NUMBER() OVER (PARTITION BY day, hour
                                  ORDER BY     num_pics DESC) AS rn
        FROM   a
    )
    FROM     b
    WHERE    rn = 1
    ORDER BY num_pics DESC
)
PIVOT    c
ON       day
USING    MAX(photographer)
GROUP BY hour
ORDER BY hour;

┌─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐
│  hour   │   02    │   03    │   04    │   05    │   06    │   07    │   08    │   09    │
│ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │
├─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ 00      │ NULL    │ Glover  │ NULL    │ NULL    │ NULL    │ Wiseman │ Koch    │ NULL    │
│ 01      │ NULL    │ NULL    │ NULL    │ Glover  │ Wiseman │ Wiseman │ NULL    │ NULL    │
│ 02      │ NULL    │ NULL    │ Wiseman │ NULL    │ Glover  │ NULL    │ NULL    │ NULL    │
│ 03      │ NULL    │ NULL    │ NULL    │ Koch    │ Koch    │ NULL    │ Koch    │ NULL    │
│ 04      │ NULL    │ Wiseman │ Koch    │ Glover  │ Koch    │ NULL    │ Koch    │ Wiseman │
│ 05      │ NULL    │ NULL    │ NULL    │ Koch    │ Glover  │ NULL    │ NULL    │ NULL    │
│ 06      │ NULL    │ NULL    │ NULL    │ Wiseman │ NULL    │ NULL    │ NULL    │ NULL    │
│ 07      │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ Koch    │ NULL    │
│ 11      │ Glover  │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │
│ 12      │ Glover  │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │ NULL    │
│ 15      │ NULL    │ NULL    │ NULL    │ NULL    │ Koch    │ NULL    │ NULL    │ NULL    │
│ 16      │ NULL    │ NULL    │ NULL    │ Koch    │ Wiseman │ NULL    │ Glover  │ NULL    │
│ 17      │ NULL    │ NULL    │ NULL    │ NULL    │ Wiseman │ NULL    │ Glover  │ NULL    │
│ 18      │ Glover  │ NULL    │ NULL    │ NULL    │ Hansen  │ NULL    │ Glover  │ NULL    │
│ 19      │ Glover  │ Koch    │ NULL    │ NULL    │ Wiseman │ Koch    │ Glover  │ NULL    │
│ 20      │ Glover  │ NULL    │ NULL    │ NULL    │ Koch    │ Glover  │ Hansen  │ NULL    │
│ 21      │ Glover  │ NULL    │ NULL    │ NULL    │ Koch    │ NULL    │ Hansen  │ NULL    │
│ 22      │ NULL    │ NULL    │ NULL    │ NULL    │ Koch    │ NULL    │ Glover  │ NULL    │
│ 23      │ Glover  │ Koch    │ NULL    │ NULL    │ Glover  │ NULL    │ NULL    │ NULL    │
└─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘

e-Bike Fleet Monitoring

2026-05-06T20:00:00+03:00

Bird, Lime and Neuron are three e-Bike rental vendors operating in Edmonton, Canada. They all provide JSON-formatted API feeds of their fleets' current locations and range left in their batteries.

In this post, I'll collect their feeds over the course of a day and analyse them.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use jq to help analyse the data in this post.

$ sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable
$ sudo apt update
$ sudo apt install \
    jq

I'll use DuckDB, along with its H3, JSON, Lindel, Parquet and Spatial extensions in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.5.1/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

The maps in this post were rendered with QGIS version 4.0.1. QGIS is a desktop application that runs on Windows, macOS and Linux. The application has grown in popularity in recent years and has ~15M application launches from users all around the world each month.

I used QGIS' HCMGIS plugin to add a satellite imagery basemap from Esri to this post as well as Trajectools for conducting trajectory analysis. The vector basemap is from OpenStreetMap's vector tiles service, which I reviewed in late 2024.

Edmonton's e-Bike Vendors

The first feed I'll analyse in this post is from Bird. Below are their bike locations across Edmonton as of Tuesday, May 5th, 2026 at 7AM. It was 0C at this time but it warmed to 10C by the afternoon.

The second vendor is Lime. Below, their bike locations have been laid out in green on top of Bird's.

The third vendor is Neuron. Below, their bike locations have been laid out in orange on top of Bird's and Lime's.

Below is a heatmap showing which vendor has the most bikes in any one hexagon as of 7AM yesterday.

Fleet Snapshots

I collected JSON files from each of the three vendors for 12 of the 13 hours between 10AM and 10PM local time in Edmonton yesterday. The only gap was the 2PM download which failed for all three for some reason.

$ mkdir -p ~/ebikes_edmonton
$ cd ~/ebikes_edmonton

$ while :
  do
      DATE=`date --date="NOW" +"%Y.%m.%d.%H.%M.%S"`

      wget -O lime.$DATE.json   https://data.lime.bike/api/partners/v2/gbfs/edmonton/free_bike_status.json
      wget -O bird.$DATE.json   https://mds.bird.co/gbfs/v2/public/edmonton/free_bike_status.json
      wget -O neuron.$DATE.json https://mds.neuron-mobility.com/yeg/gbfs/2/en/free_bike_status

      echo "Sleeping for an hour.."
      sleep 3600
  done

Lime serves their responses via Cloudflare, Bird are from AWS Cloudfront and Neuron via Google Cloud.

The JSON filenames are prefixed with the name of the vendor, followed by the local time here in Estonia.

The JSON files are between 142 KB and 340 KB each.

Data Fluency

The feeds are very similar but do contain some differences. Neuron doesn't have a "last_reported" field but it's the only feed with a "station_id" field. This field was blank in every response I got from them. Below is an example record:

$ jq -S '.data.bikes[0]' \
    neuron.2026.05.06.11.22.20.json

{
  "bike_id": "e4c7de34-6c4b-3e7c-a147-c82dc07e12a5",
  "current_range_meters": 39500.0,
  "is_disabled": false,
  "is_reserved": false,
  "lat": 53.539043,
  "lon": -113.49915,
  "station_id": null,
  "vehicle_type_id": "e8fbc1f0-3504-3dff-852a-9e6dab5e204d"
}

Bird is the only feed with a "current_fuel_percent" field. Below is an example record.

$ jq -S '.data.bikes[0]' \
    bird.2026.05.05.19.55.10.json

{
  "bike_id": "c67cce09-ba6d-4114-86e2-0416f5d80ed4",
  "current_fuel_percent": 0.6363636,
  "current_range_meters": 21616.0,
  "is_disabled": false,
  "is_reserved": false,
  "last_reported": 1777999880,
  "lat": 53.5559835,
  "lon": -113.48228766666668,
  "vehicle_type_id": "bae2102b-56ba-42ba-9097-720e5990b4b2"
}

Lime has a textual "vehicle_type" field; the others use UUIDs. Below is an example record.

$ jq -S '.data.bikes[0]' \
    lime.2026.05.05.19.55.10.json

{
  "bike_id": "36a377ae-7bf9-42a9-902d-65efc0e99e9a",
  "current_range_meters": 38705,
  "is_disabled": false,
  "is_reserved": false,
  "last_reported": 1778000064,
  "lat": 53.522562,
  "lon": -113.51187,
  "vehicle_type": "scooter",
  "vehicle_type_id": "2"
}

Analysis-Ready Data

Below, I'll import each JSON file into vendor-specific tables in DuckDB.

$ touch bikes.duckdb || rm bikes.duckdb

$ for FILENAME in *.json; do
      VENDOR=`echo $FILENAME | cut -d. -f1`

      echo "CREATE OR REPLACE TABLE loading AS
              WITH a AS (
                  SELECT UNNEST(data.bikes) b
                  FROM   '$FILENAME'
              )
              SELECT b.*
              FROM   a;

            ALTER TABLE loading
            ADD COLUMN  source TEXT;

            UPDATE loading
            SET    source = '$FILENAME';

            CREATE TABLE IF NOT EXISTS $VENDOR AS
              FROM  loading
              LIMIT 0;

            INSERT INTO $VENDOR
            FROM        loading;

            DROP TABLE loading;
            " | ~/duckdb bikes.duckdb
  done

I'll merge the compatible fields from all three vendor tables into a single "bikes" table.

$ ~/duckdb bikes.duckdb

CREATE OR REPLACE TABLE bikes AS
    SELECT vendor: 'Bird',
           bike_id,
           geometry: ST_POINT(lon, lat),
           is_disabled,
           is_reserved,
           vehicle_type_id: vehicle_type_id::TEXT,
           vehicle_type: 'Unknown',
           current_range_meters,
           source,
           downloaded_at:
            MAKE_TIMESTAMPTZ(
                SPLIT(source, '.')[2]::BIGINT,
                SPLIT(source, '.')[3]::BIGINT,
                SPLIT(source, '.')[4]::BIGINT,
                SPLIT(source, '.')[5]::BIGINT,
                SPLIT(source, '.')[6]::BIGINT,
                SPLIT(source, '.')[7]::BIGINT,
                'Europe/Tallinn')
    FROM bird

    UNION ALL BY NAME

    SELECT vendor: 'Lime',
           bike_id,
           geometry: ST_POINT(lon, lat),
           is_disabled,
           is_reserved,
           vehicle_type_id: vehicle_type_id::TEXT,
           vehicle_type,
           current_range_meters,
           source,
           downloaded_at:
            MAKE_TIMESTAMPTZ(
                SPLIT(source, '.')[2]::BIGINT,
                SPLIT(source, '.')[3]::BIGINT,
                SPLIT(source, '.')[4]::BIGINT,
                SPLIT(source, '.')[5]::BIGINT,
                SPLIT(source, '.')[6]::BIGINT,
                SPLIT(source, '.')[7]::BIGINT,
                'Europe/Tallinn')
    FROM lime

    UNION ALL BY NAME

    SELECT vendor: 'Neuron',
           bike_id,
           geometry: ST_POINT(lon, lat),
           is_disabled,
           is_reserved,
           vehicle_type_id: vehicle_type_id::TEXT,
           vehicle_type: 'Unknown',
           current_range_meters,
           source,
           downloaded_at:
            MAKE_TIMESTAMPTZ(
                SPLIT(source, '.')[2]::BIGINT,
                SPLIT(source, '.')[3]::BIGINT,
                SPLIT(source, '.')[4]::BIGINT,
                SPLIT(source, '.')[5]::BIGINT,
                SPLIT(source, '.')[6]::BIGINT,
                SPLIT(source, '.')[7]::BIGINT,
                'Europe/Tallinn')
    FROM neuron;

The above produced a 50,972-row, 10-column table. The following is an example record.

$ echo "FROM  bikes
        LIMIT 1" \
    | ~/duckdb -json bikes.duckdb \
    | jq -S .

[
  {
    "bike_id": "c67cce09-ba6d-4114-86e2-0416f5d80ed4",
    "current_range_meters": 21616.0,
    "downloaded_at": "2026-05-05 19:55:10+03",
    "geometry": "POINT (-113.48228766666668 53.5559835)",
    "is_disabled": "false",
    "is_reserved": "false",
    "source": "bird.2026.05.05.19.55.10.json",
    "vehicle_type": "Unknown",
    "vehicle_type_id": "bae2102b-56ba-42ba-9097-720e5990b4b2",
    "vendor": "Bird"
  }
]

e-Bike Availability

Below, I'll count the number of unique bike IDs per vendor per hour. The timestamp field is the local time in Edmonton.

$ ~/duckdb bikes.duckdb

WITH a AS (
    SELECT   date_hour: (TIMEZONE('Canada/Mountain', downloaded_at)::DATE || ' ' ||
                         HOUR(TIMEZONE('Canada/Mountain', downloaded_at)) || ':00:00')::TIMESTAMP,
             vendor,
             num_recs:  COUNT(DISTINCT bike_id)
    FROM     bikes
    GROUP BY 1, 2
)
PIVOT    a
ON       vendor
USING    SUM(num_recs)
GROUP BY date_hour
ORDER BY date_hour;

┌─────────────────────┬────────┬────────┬────────┐
│      date_hour      │  Bird  │  Lime  │ Neuron │
│      timestamp      │ int128 │ int128 │ int128 │
├─────────────────────┼────────┼────────┼────────┤
│ 2026-05-05 10:00:00 │   1110 │   1409 │    626 │
│ 2026-05-05 11:00:00 │   1127 │   1395 │    637 │
│ 2026-05-05 12:00:00 │   1153 │   1392 │    637 │
│ 2026-05-05 13:00:00 │   1149 │   1386 │    645 │
│ 2026-05-05 15:00:00 │   1153 │   1392 │    651 │
│ 2026-05-05 16:00:00 │   1157 │   1354 │    647 │
│ 2026-05-05 17:00:00 │   1196 │   1359 │    634 │
│ 2026-05-05 18:00:00 │   1215 │   1368 │    633 │
│ 2026-05-05 19:00:00 │   1205 │   1380 │    641 │
│ 2026-05-05 20:00:00 │   1195 │   1366 │    636 │
│ 2026-05-05 21:00:00 │   1208 │   1394 │    643 │
│ 2026-05-05 22:00:00 │   1222 │   1410 │    650 │
└─────────────────────┴────────┴────────┴────────┘

Rotating Bike IDs

Judging by the unique bike ID counts, Bird and Lime appear to rotate the IDs for their bikes at least every hour. Neuron appears to only use one ID per bike over the period I was observing their feed.

$ ~/duckdb bikes.duckdb

SELECT   vendor,
         num_bikes: COUNT(DISTINCT bike_id)
FROM     bikes
GROUP BY 1;

┌─────────┬───────────┐
│ vendor  │ num_bikes │
│ varchar │   int64   │
├─────────┼───────────┤
│ Bird    │     14090 │
│ Lime    │     16605 │
│ Neuron  │       657 │
└─────────┴───────────┘

Bike Types

Each vendor has two types of bikes on offer in Edmonton. Below are their type identifiers.

$ ~/duckdb bikes.duckdb

SELECT   DISTINCT vendor,
                  vehicle_type,
                  vehicle_type_id
FROM     bikes
GROUP BY 1, 2, 3
ORDER BY 1;

┌─────────┬──────────────┬──────────────────────────────────────┐
│ vendor  │ vehicle_type │           vehicle_type_id            │
│ varchar │   varchar    │               varchar                │
├─────────┼──────────────┼──────────────────────────────────────┤
│ Bird    │ Unknown      │ bae2102b-56ba-42ba-9097-720e5990b4b2 │
│ Bird    │ Unknown      │ 2ea3c8b2-ed07-4c53-b87e-638c08471309 │
│ Lime    │ e-bike       │ 3                                    │
│ Lime    │ scooter      │ 2                                    │
│ Neuron  │ Unknown      │ 9e8e37ae-b444-3d57-9e9a-58ce02e9f2fd │
│ Neuron  │ Unknown      │ e8fbc1f0-3504-3dff-852a-9e6dab5e204d │
└─────────┴──────────────┴──────────────────────────────────────┘

There aren't a lot of bike capabilities listed within the feeds I monitored. I searched for "bae2102b-56ba-42ba-9097-720e5990b4b2" which brought up this Japanese-language site with the following JSON snippet for two of Bird's bikes:

[
    {
        "vehicle_type_id": "bae2102b-56ba-42ba-9097-720e5990b4b2",
        "form_factor": "scooter",
        "propulsion_type": "electric",
        "max_range_meters": 24000
    },
    {
        "vehicle_type_id": "2ea3c8b2-ed07-4c53-b87e-638c08471309",
        "form_factor": "bicycle",
        "propulsion_type": "electric_assist",
        "max_range_meters": 60000
    }
]

There is further work to collect the metadata around each vendor's fleet capabilities as it could uncover some interesting trends.

Bike Ranges

Below are the number of bikes available, binned by their range and rounded up to the nearest 10 KM. Lime was the only vendor with 60KM+ bikes available at the time of this observation.

$ ~/duckdb bikes.duckdb

WITH a AS (
    SELECT   range_km: (current_range_meters/10000)::INT*10,
             vendor,
             num_recs: COUNT(DISTINCT bike_id)
    FROM     bikes
    WHERE    downloaded_at::TEXT LIKE '2026-05-05 20%'
    GROUP BY 1, 2
)
PIVOT    a
ON       vendor
USING    SUM(num_recs)
GROUP BY range_km
ORDER BY range_km;

┌──────────┬────────┬────────┬────────┐
│ range_km │  Bird  │  Lime  │ Neuron │
│  int32   │ int128 │ int128 │ int128 │
├──────────┼────────┼────────┼────────┤
│        0 │     12 │    125 │     21 │
│       10 │    286 │    284 │     65 │
│       20 │    358 │    227 │    149 │
│       30 │    383 │    257 │    146 │
│       40 │     86 │    329 │    121 │
│       50 │      2 │     45 │    135 │
│       60 │   NULL │     69 │   NULL │
│       70 │   NULL │     57 │   NULL │
│       80 │   NULL │      2 │   NULL │
└──────────┴────────┴────────┴────────┘

Bike Footprints

Below, I'll calculate each Neuron's bikes' bounding box for all of its journeys over the day I monitored their feed. Note, I only have snapshots of their location every hour and a bike will only show up if it's available so there is room for error in these measurements.

$ ~/duckdb bikes.duckdb

CREATE OR REPLACE TABLE neuron_extents AS
    WITH a AS (
        SELECT   bike_id,
                 geometry: {'min_x': MIN(ST_X(ST_CENTROID(geometry))),
                            'min_y': MIN(ST_Y(ST_CENTROID(geometry))),
                            'max_x': MAX(ST_X(ST_CENTROID(geometry))),
                            'max_y': MAX(ST_Y(ST_CENTROID(geometry)))}::BOX_2D::GEOMETRY,
        FROM     bikes
        WHERE    vendor = 'Neuron'
        GROUP BY 1
        ORDER BY 2 DESC
    )
    SELECT bike_id,
           area_size: ST_AREA(geometry)
    FROM   a;

Of Neuron's 657 bikes, only 76 hadn't moved at all throughout the day.

SELECT   no_movement: area_size=0.0,
         COUNT(*)
FROM     neuron_extents
GROUP BY 1;

┌─────────────┬──────────────┐
│ no_movement │ count_star() │
│   boolean   │    int64     │
├─────────────┼──────────────┤
│ false       │          581 │
│ true        │           76 │
└─────────────┴──────────────┘

I'll build a trajectory map of their 20 most-travelled bikes for the period I was observing their feed.

COPY (
   SELECT   * EXCLUDE (geometry, bike_id),
            bike_id:  bike_id::TEXT,
            bbox:     {'xmin': ST_XMIN(ST_EXTENT(geometry)),
                       'ymin': ST_YMIN(ST_EXTENT(geometry)),
                       'xmax': ST_XMAX(ST_EXTENT(geometry)),
                       'ymax': ST_YMAX(ST_EXTENT(geometry))},
            geometry: ST_ASWKB(geometry)
   FROM     bikes
   WHERE    bike_id IN (
                SELECT   bike_id
                FROM     neuron_extents
                ORDER BY area_size DESC
                LIMIT    20)
   ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geometry)),
                            ST_X(ST_CENTROID(geometry))]::double[2])
) TO 'Neuron.parquet' (
   FORMAT            'PARQUET',
   CODEC             'ZSTD',
   COMPRESSION_LEVEL 22,
   ROW_GROUP_SIZE    15000);

The maps in the screenshots below are projected with EPSG:3771. After adding any Parquet file into QGIS, select its layer properties and then under source, change its assigned coordinate system (CRS) to EPSG:4326.

I'll search for and select "Create trajectories" in the Processing Toolbox.

I've set the input point layer to the "Neuron" one that contains the data from the Parquet file I generated above. I've also set the Trajectory ID field to "bike_id" and the timestamp field to "downloaded_at".

After hitting run, the following was generated. The extent of the map is ~11.4 KM tall by ~7 KM wide. The start of each bike's journey is in blue and follows a gradient till they reach red at the end.

I did notice that some bikes moved short distances hour-by-hour during certain time windows. This could be down to GPS drift or people moving the bike out of the way.

Trajectools supports loitering analysis which could go some way to determine how many Neuron bikes made significant journeys. Below I'll export all of Neuron's journeys to a Parquet file.

$ ~/duckdb bikes.duckdb

COPY (
   SELECT   * EXCLUDE (geometry, bike_id),
            bike_id:  bike_id::TEXT,
            bbox:     {'xmin': ST_XMIN(ST_EXTENT(geometry)),
                       'ymin': ST_YMIN(ST_EXTENT(geometry)),
                       'xmax': ST_XMAX(ST_EXTENT(geometry)),
                       'ymax': ST_YMAX(ST_EXTENT(geometry))},
            geometry: ST_ASWKB(geometry)
   FROM     bikes
   WHERE    vendor = 'Neuron'
   ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geometry)),
                            ST_X(ST_CENTROID(geometry))]::double[2])
) TO 'Neuron.all_day.parquet' (
   FORMAT            'PARQUET',
   CODEC             'ZSTD',
   COMPRESSION_LEVEL 22,
   ROW_GROUP_SIZE    15000);

I've imported that into QGIS and run Trajectools' "Split trajectories at stops" tool with the following parameters.

I exported a GeoPackage (GPKG) file of the resulting analysis. It appears only 160 Neuron bikes moved more than 1 KM throughout the day.

$ ~/duckdb bikes.duckdb

SELECT num_bikes: COUNT(DISTINCT(bike_id))
FROM   ST_READ('Neuron.traj.gpkg')
WHERE  length_km > 1;

┌───────────┐
│ num_bikes │
│   int64   │
├───────────┤
│       160 │
└───────────┘

Changing Availability

Below, I'll generate a heatmap of the number of bikes available across all vendors throughout the day I observed their feeds.

$ ~/duckdb bikes.duckdb

CREATE OR REPLACE TABLE h3_stats AS
    SELECT   date_hour: (TIMEZONE('Canada/Mountain', downloaded_at)::DATE || ' ' ||
                         HOUR(TIMEZONE('Canada/Mountain', downloaded_at)) || ':00:00')::TIMESTAMP,
             hexagon:   H3_LATLNG_TO_CELL(
                          ST_Y(ST_CENTROID(geometry)),
                          ST_X(ST_CENTROID(geometry)),
                          7),
             num_recs:  COUNT(DISTINCT bike_id)
    FROM     bikes
    GROUP BY 1, 2;

COPY (
    SELECT geometry: ST_ASWKB(H3_CELL_TO_BOUNDARY_WKT(hexagon)::geometry),
           date_hour,
           num_recs
    FROM   h3_stats
) TO 'bikes_per_hour.h3_4.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

Regardless of the time of day, there is a fairly consistent availability in just about every hexagon.

10K+ Satellites in Space

2026-05-04T20:00:00+03:00

The General Catalog of Artificial Space Objects (GCAT) is a dataset collection of space launch sites, vehicles, satellites and other related items. It's compiled and maintained by Jonathan C. McDowell, who uses its data for his space reports. 854 such reports have been published to date.

In this post, I'll convert a handful of the GCAT datasets to Parquet and do some analysis on them.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use jq to help analyse the data in this post.

$ sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable
$ sudo apt update
$ sudo apt install \
    jq

I'll use DuckDB, along with its H3, JSON, Lindel, Parquet and Spatial extensions in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.5.1/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

I used QGIS' HCMGIS plugin to add basemaps from Esri to this post.

Organizations

I'll start by downloading the organisations dataset.

$ wget https://planet4589.org/space/gcat/tsv/tables/orgs.tsv

The first row contains the field names but starts with a hash. The second row is a timestamp of when this file was last updated.

$ head -n5 orgs.tsv | cut -c-30

#Code   UCode   StateCode       Type    Cla
# Updated 2026 Apr 24 1739:51
EARTH   EARTH   EARTH   AP      C       -       -       Ear
LUNA    LUNA    LUNA    AP      C       -       -       Luna    L
SSYS    SSYS    SSYS    AP      C       -       -       Solar

I'll exclude the header row as well as the timestamp. This will create a file that's easier for DuckDB to import.

$ head -n1 orgs.tsv | tail -c+2 > working.csv
$ tail -n+3 orgs.tsv >> working.csv

I'll clean up the values, casting them to stronger types where I can and producing a parquet file with their contents.

$ ~/duckdb

COPY(
    SELECT   Class,
             Code,
             EName: IF(LENGTH(TRIM(EName)) < 2, NULL, EName),
             Error,
             Location: IF(Location = '-', NULL, Location),
             Name,
             Parent: IF(LENGTH(TRIM(Parent)) < 2, NULL, Parent),
             ShortEName: IF(LENGTH(TRIM(ShortEName)) < 2, NULL, ShortEName),
             ShortName: IF(LENGTH(TRIM(ShortName)) < 4, NULL, ShortName),
             StateCode,
             TStart: IF(LENGTH(TRIM(TStart)) < 4, NULL, TStart[:4]::INT),
             TStop: IF(LENGTH(TRIM(TStop)) < 4, NULL, TStop[:4]::INT),
             Type: IF(LENGTH(TRIM(Type)) < 4, NULL, Type),
             UCode,
             UName,
             geometry: IF(LENGTH(Latitude)>1,
                          ST_POINT(Longitude::FLOAT, Latitude::FLOAT),
                          NULL),
             bbox:      {'xmin': ST_XMIN(ST_EXTENT(geometry)),
                         'ymin': ST_YMIN(ST_EXTENT(geometry)),
                         'xmax': ST_XMAX(ST_EXTENT(geometry)),
                         'ymax': ST_YMAX(ST_EXTENT(geometry))},
    FROM     'working.csv'
) TO 'orgs.parquet' (
  FORMAT 'PARQUET',
  CODEC  'ZSTD',
  COMPRESSION_LEVEL 22,
  ROW_GROUP_SIZE 15000);

There are 4,013 records in this dataset.

SELECT COUNT(*)
FROM   'orgs.parquet'; -- 4013

Below is a breakdown of unique values and NULL coverage across each column.

SELECT   column_name,
         column_type[:30],
         null_percentage,
         approx_unique,
         min[:30],
         max[:30]
FROM     (SUMMARIZE
          FROM 'orgs.parquet')
ORDER BY LOWER(column_name);

┌─────────────┬────────────────────────────────┬─────────────────┬───────────────┬────────────────────────────────┬───────────────────────────────────────────────┐
│ column_name │        column_type[:30]        │ null_percentage │ approx_unique │            min[:30]            │                   max[:30]                    │
│   varchar   │            varchar             │  decimal(9,2)   │     int64     │            varchar             │                    varchar                    │
├─────────────┼────────────────────────────────┼─────────────────┼───────────────┼────────────────────────────────┼───────────────────────────────────────────────┤
│ bbox        │ STRUCT(xmin DOUBLE, ymin DOUBL │            0.00 │          1455 │ {'xmin': -213.25, 'ymin': 49.2 │ {'xmin': NULL, 'ymin': NULL, '                │
│ Class       │ VARCHAR                        │            0.00 │             4 │ A                              │ D                                             │
│ Code        │ VARCHAR                        │            0.00 │          3675 │ 10ADS                          │ ZZWYZ                                         │
│ EName       │ VARCHAR                        │           66.43 │          1468 │ 2nd Academy of 7th Machine Bui │ [Same as CALT?]                               │
│ Error       │ VARCHAR                        │            0.00 │            13 │ 0.0000                         │ 0.5000                                        │
│ geometry    │ GEOMETRY('OGC:CRS84')          │            0.07 │          1716 │ POINT (2.25 48.900001525878906 │ POINT (-122.02880096435547 37.                │
│ Location    │ VARCHAR                        │            0.17 │          1657 │ 's-Gravenhage                  │ ar-Riyad, Al Arabiyah                         │
│ Name        │ VARCHAR                        │            0.00 │          4553 │          -            -        │ zhongguo zairen hangtian gongc                │
│ Parent      │ VARCHAR                        │           67.95 │           429 │ 21AT                           │ ZZB                                           │
│ ShortEName  │ VARCHAR                        │           89.48 │           393 │ 5th Academy                    │ iSpace                                        │
│ ShortName   │ VARCHAR                        │           11.06 │          2707 │ 061 Jidi                       │ ti Space                                      │
│ StateCode   │ VARCHAR                        │            0.00 │           182 │ AAT                            │ ZW                                            │
│ TStart      │ INTEGER                        │            7.43 │           303 │ 700                            │ 2025                                          │
│ TStop       │ INTEGER                        │           66.58 │            86 │ 1660                           │ 2025                                          │
│ Type        │ VARCHAR                        │           66.98 │            89 │ E/LS                           │ PL/S                                          │
│ UCode       │ VARCHAR                        │            0.00 │          3003 │ 10ADS                          │ ZZWYZ                                         │
│ UName       │ VARCHAR                        │            0.00 │          4238 │          -            -        │ ＮＥＣ東芝スペースシステム株式会社 府中事業所 │
└─────────────┴────────────────────────────────┴─────────────────┴───────────────┴────────────────────────────────┴───────────────────────────────────────────────┘

CREATE OR REPLACE TABLE h3_2_stats AS
    SELECT   h3_2: H3_LATLNG_TO_CELL(
                        bbox.ymin,
                        bbox.xmin,
                        2),
             COUNT(*) num_recs
    FROM     'orgs.parquet'
    GROUP BY 1;

COPY (
    SELECT geometry: ST_ASWKB(H3_CELL_TO_BOUNDARY_WKT(h3_2)::geometry),
           num_recs
    FROM   h3_2_stats
    WHERE  ST_XMIN(H3_CELL_TO_BOUNDARY_WKT(h3_2)::geometry) BETWEEN -179 AND 179
    AND    ST_XMAX(H3_CELL_TO_BOUNDARY_WKT(h3_2)::geometry) BETWEEN -179 AND 179
    AND    ST_YMIN(H3_CELL_TO_BOUNDARY_WKT(h3_2)::geometry) BETWEEN -88  AND  88
) TO 'orgs.h3_2.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

Below is a heatmap of the 4K organisation's locations.

Below are a few of the organisations in Florida.

Below are some more in and around Moscow.

Launch Platforms

Below, I'll download and prepare the platforms dataset for DuckDB.

$ wget https://planet4589.org/space/gcat/tsv/tables/platforms.tsv

$ head -n1 platforms.tsv | tail -c+2 > working.csv
$ tail -n+3 platforms.tsv >> working.csv

I'll clean up the values, casting them to stronger types where I can and producing a parquet file with their contents.

$ ~/duckdb

COPY(
    SELECT   Class,
             Code,
             "Group": IF("Group" = '-', NULL, "Group"),
             Location: IF(Location = '-', NULL, Location),
             Name,
             Parent,
             ShortEName: IF(ShortEName = '-', NULL, ShortEName),
             ShortName,
             StateCode,
             TStart: IF(TStart = '-', NULL, TStart[:4]::INT),
             TStop: IF(TStop = '-', NULL, TStop[:4]::INT),
             Type,
             UCode,
             UName,
             VClass: IF(VClass = '-', NULL, VClass),
             VClassID: IF(VClassID = '-', NULL, VClassID),
             VID: IF(VID = '-', NULL, VID)
    FROM     'working.csv'
) TO 'platforms.parquet' (
  FORMAT 'PARQUET',
  CODEC  'ZSTD',
  COMPRESSION_LEVEL 22,
  ROW_GROUP_SIZE 15000);

There are 384 records in this dataset.

SELECT COUNT(*)
FROM   'platforms.parquet'; -- 384

Below is a breakdown of unique values and NULL coverage across each column.

SELECT   column_name,
         column_type[:30],
         null_percentage,
         approx_unique,
         min[:30],
         max[:30]
FROM     (SUMMARIZE
          FROM 'platforms.parquet')
ORDER BY LOWER(column_name);

┌─────────────┬──────────────────┬─────────────────┬───────────────┬────────────────────────────────┬────────────────────┐
│ column_name │ column_type[:30] │ null_percentage │ approx_unique │            min[:30]            │      max[:30]      │
│   varchar   │     varchar      │  decimal(9,2)   │     int64     │            varchar             │      varchar       │
├─────────────┼──────────────────┼─────────────────┼───────────────┼────────────────────────────────┼────────────────────┤
│ Class       │ VARCHAR          │            0.00 │             3 │ B                              │ D                  │
│ Code        │ VARCHAR          │            0.00 │           388 │ 824YONG                        │ ZULUV              │
│ Group       │ VARCHAR          │            2.34 │            35 │ AEGIS                          │ USSHIPS            │
│ Location    │ VARCHAR          │           79.95 │            27 │ Beale AFB                      │ White Sands        │
│ Name        │ VARCHAR          │            0.00 │           364 │ 8.24 Yongung (Gorae class sub) │ Yu. M. Shokal'skiy │
│ Parent      │ VARCHAR          │            0.00 │            51 │ AANII                          │ VORB               │
│ ShortEName  │ VARCHAR          │          100.00 │             0 │ NULL                           │ NULL               │
│ ShortName   │ VARCHAR          │            0.00 │           363 │ AIR   B-1B                     │ White Knight 1     │
│ StateCode   │ VARCHAR          │            0.00 │            11 │ CN                             │ US                 │
│ TStart      │ INTEGER          │           46.61 │            68 │ 1944                           │ 2024               │
│ TStop       │ INTEGER          │           72.92 │            42 │ 1958                           │ 2020               │
│ Type        │ VARCHAR          │            0.00 │             4 │ AIR                            │ SUB                │
│ UCode       │ VARCHAR          │            0.00 │           408 │ 824YOG                         │ ZULUV              │
│ UName       │ VARCHAR          │            0.00 │           375 │ 629                            │ 长城-406           │
│ VClass      │ VARCHAR          │            7.29 │            83 │ 224 A.I. Voeykov               │ Wind               │
│ VClassID    │ VARCHAR          │           14.58 │           307 │ 07-2                           │ WYG-279            │
│ VID         │ VARCHAR          │           94.79 │            18 │ IMO5010244                     │ NASA 218           │
└─────────────┴──────────────────┴─────────────────┴───────────────┴────────────────────────────────┴────────────────────┘

Below is an example record for the McDonnell Douglas F-15 Eagle.

$ echo "FROM  'platforms.parquet'
        WHERE Code = 'F-15A-08x'" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "Class": "D",
    "Code": "F-15A-08x",
    "Group": "USAIRF",
    "Location": "Edwards AFB",
    "Name": "F-15 Eagle, ASAT program (084 or 086)",
    "Parent": "USAF",
    "ShortEName": null,
    "ShortName": "AIR   F-15",
    "StateCode": "US",
    "TStart": null,
    "TStop": null,
    "Type": "AIR",
    "UCode": "F-15A-08x",
    "UName": "F-15 Eagle, ASAT program (084 or 086)",
    "VClass": "F-15A",
    "VClassID": "76-0084/0086",
    "VID": null
  }
]

This is a breakdown of the 23 groups in this dataset.

$ ~/duckdb

SELECT   COUNT(*),
         "group": SPLIT_PART("Group", '.', 1)
FROM     'platforms.parquet'
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬────────────┐
│ count_star() │   group    │
│    int64     │  varchar   │
├──────────────┼────────────┤
│          134 │ SUSUB      │
│           41 │ POLARIS    │
│           34 │ AEGIS      │
│           26 │ USAIRB     │
│           26 │ USAIRF     │
│           18 │ TRIDENT    │
│           14 │ USSHIPS    │
│           13 │ SOVSHIPS   │
│           11 │ SNLE       │
│           10 │ OTHERSHIPS │
│            9 │ NULL       │
│            8 │ USMSHIPS   │
│            8 │ UKSUB      │
│            7 │ USAIRC     │
│            4 │ USAIRT     │
│            4 │ ASDS       │
│            3 │ KRSUB      │
│            3 │ ROCKOON    │
│            3 │ INAV       │
│            3 │ CNSUB      │
│            2 │ KPSUB      │
│            2 │ INSUB      │
│            1 │ SUAIR      │
└──────────────┴────────────┘

Below is a list of the Trident group of submarines found in this dataset.

SELECT   Code,
         Name,
         TStart,
         TStop,
         VClass
FROM     'platforms.parquet'
WHERE    "Group" = 'TRIDENT'
ORDER BY TStart;

┌──────────┬──────────────────────┬────────┬───────┬─────────┐
│   Code   │         Name         │ TStart │ TStop │ VClass  │
│ varchar  │       varchar        │ int32  │ int32 │ varchar │
├──────────┼──────────────────────┼────────┼───────┼─────────┤
│ SSBN-726 │ USS Ohio             │   1979 │  2003 │ Ohio    │
│ SSBN-727 │ USS Michigan         │   1980 │  2007 │ Ohio    │
│ SSBN-728 │ USS Florida          │   1981 │  2003 │ Ohio    │
│ SSBN-729 │ USS Georgia          │   1982 │  2003 │ Ohio    │
│ SSBN-730 │ USS Henry M. Jackson │   1983 │  NULL │ Ohio    │
│ SSBN-731 │ USS Alabama          │   1984 │  NULL │ Ohio    │
│ SSBN-732 │ USS Alaska           │   1985 │  NULL │ Ohio    │
│ SSBN-733 │ USS Nevada           │   1985 │  NULL │ Ohio    │
│ SSBN-734 │ USS Tennessee        │   1986 │  NULL │ Ohio    │
│ SSBN-735 │ USS Pennsylvania     │   1988 │  NULL │ Ohio    │
│ SSBN-736 │ USS West Virginia    │   1989 │  NULL │ Ohio    │
│ SSBN-737 │ USS Kentucky         │   1990 │  NULL │ Ohio    │
│ SSBN-738 │ USS Maryland         │   1991 │  NULL │ Ohio    │
│ SSBN-739 │ USS Nebraska         │   1992 │  NULL │ Ohio    │
│ SSBN-740 │ USS Rhode Island     │   1993 │  NULL │ Ohio    │
│ SSBN-741 │ USS Maine            │   1994 │  NULL │ Ohio    │
│ SSBN-742 │ USS Wyoming          │   1995 │  NULL │ Ohio    │
│ SSBN-743 │ USS Louisiana        │   1996 │  NULL │ Ohio    │
└──────────┴──────────────────────┴────────┴───────┴─────────┘

Launch Sites

Below, I'll download and prepare the launch sites dataset for DuckDB.

$ wget https://planet4589.org/space/gcat/tsv/tables/sites.tsv

$ head -n1 sites.tsv | tail -c+2 > working.csv
$ tail -n+3 sites.tsv >> working.csv

I'll clean up the values, casting them to stronger types where I can and producing a parquet file with their contents.

$ ~/duckdb

COPY(
    SELECT   EName: IF(LENGTH(TRIM(EName)) < 2, NULL, EName),
             Error: Error::FLOAT,
             Location: IF(Location = '-', NULL, Location),
             Name,
             Parent: IF(LENGTH(TRIM(Parent)) < 2, NULL, Parent),
             ShortEName: IF(LENGTH(TRIM(ShortEName)) < 2, NULL, ShortEName),
             ShortName: IF(LENGTH(TRIM(ShortName)) < 4, NULL, ShortName),
             Site,
             TStart: IF(LENGTH(TRIM(TStart)) < 4, NULL, TStart[:4]::INT),
             TStop: IF(LENGTH(TRIM(TStop)) < 4, NULL, TStop[:4]::INT),
             UCode,
             UName: IF(LENGTH(TRIM(UName)) < 2, NULL, UName),
             geometry: IF(LENGTH(Latitude)>1,
                          ST_POINT(Longitude::FLOAT, Latitude::FLOAT),
                          NULL),
             bbox:      {'xmin': ST_XMIN(ST_EXTENT(geometry)),
                         'ymin': ST_YMIN(ST_EXTENT(geometry)),
                         'xmax': ST_XMAX(ST_EXTENT(geometry)),
                         'ymax': ST_YMAX(ST_EXTENT(geometry))},
    FROM     'working.csv'
) TO 'sites.parquet' (
  FORMAT 'PARQUET',
  CODEC  'ZSTD',
  COMPRESSION_LEVEL 22,
  ROW_GROUP_SIZE 15000);

There are 684 records in this dataset.

SELECT COUNT(*)
FROM   'sites.parquet'; -- 684

Below is a breakdown of unique values and NULL coverage across each column.

SELECT   column_name,
         column_type[:30],
         null_percentage,
         approx_unique,
         min[:30],
         max[:30]
FROM     (SUMMARIZE
          FROM 'sites.parquet')
ORDER BY LOWER(column_name);

┌─────────────┬────────────────────────────────┬─────────────────┬───────────────┬────────────────────────────────┬────────────────────────────────┐
│ column_name │        column_type[:30]        │ null_percentage │ approx_unique │            min[:30]            │            max[:30]            │
│   varchar   │            varchar             │  decimal(9,2)   │     int64     │            varchar             │            varchar             │
├─────────────┼────────────────────────────────┼─────────────────┼───────────────┼────────────────────────────────┼────────────────────────────────┤
│ bbox        │ STRUCT(xmin DOUBLE, ymin DOUBL │            0.00 │           445 │ {'xmin': -177.5, 'ymin': 24.79 │ {'xmin': NULL, 'ymin': NULL, ' │
│ EName       │ VARCHAR                        │           99.29 │             2 │ - -                            │ Taiyuan Space Center           │
│ Error       │ FLOAT                          │            0.00 │            30 │ 0.0                            │ 9.0                            │
│ geometry    │ GEOMETRY('OGC:CRS84')          │            3.25 │           618 │ POINT (0 68)                   │ POINT (-3.0829999446868896 30. │
│ Location    │ VARCHAR                        │           52.33 │           135 │ Abbot Point, Queensland        │ t'Harde, Nederland             │
│ Name        │ VARCHAR                        │            0.00 │           772 │ 1-y Gosudarstvenniy Ispitateln │ Zubov Expedition 9,1985        │
│ Parent      │ VARCHAR                        │            2.40 │           111 │ AADC                           │ ZWO                            │
│ ShortEName  │ VARCHAR                        │           98.02 │             9 │ Baykonur                       │ Vandenberg                     │
│ ShortName   │ VARCHAR                        │            3.54 │           386 │ AGB-3                          │ t'Harde                        │
│ Site        │ VARCHAR                        │            0.00 │           694 │ A51                            │ ZINGST                         │
│ TStart      │ INTEGER                        │            9.19 │            86 │ 1878                           │ 2026                           │
│ TStop       │ INTEGER                        │           37.34 │            75 │ 1944                           │ 2024                           │
│ UCode       │ VARCHAR                        │            0.00 │           694 │ A51                            │ ZINGST                         │
│ UName       │ VARCHAR                        │            0.57 │           794 │ 1-y Gosudarstvenniy Ispitateln │ 海口                           │
└─────────────┴────────────────────────────────┴─────────────────┴───────────────┴────────────────────────────────┴────────────────────────────────┘

Below is an example record for Vandenberg Space Force Base.

$ echo "FROM  'sites.parquet'
        WHERE Site = 'VSFB'" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "EName": null,
    "Error": 0.019999999552965164,
    "Location": null,
    "Name": "Vandenberg Space Force Base, California",
    "Parent": "USSF",
    "ShortEName": "Vandenberg",
    "ShortName": "Vandenberg",
    "Site": "VSFB",
    "TStart": 2021,
    "TStop": null,
    "UCode": "V",
    "UName": "Vandenberg Space Force Base, California",
    "bbox": {
      "xmax": -120.62000274658203,
      "xmin": -120.62000274658203,
      "ymax": 34.75,
      "ymin": 34.75
    },
    "geometry": "POINT (-120.62000274658203 34.75)"
  }
]

Below is a heatmap of the 684 launch sites.

$ ~/duckdb

CREATE OR REPLACE TABLE h3_2_stats AS
    SELECT   h3_2: H3_LATLNG_TO_CELL(
                        bbox.ymin,
                        bbox.xmin,
                        2),
             COUNT(*) num_recs
    FROM     'sites.parquet'
    GROUP BY 1;

COPY (
    SELECT geometry: ST_ASWKB(H3_CELL_TO_BOUNDARY_WKT(h3_2)::geometry),
           num_recs
    FROM   h3_2_stats
    WHERE  ST_XMIN(H3_CELL_TO_BOUNDARY_WKT(h3_2)::geometry) BETWEEN -179 AND 179
    AND    ST_XMAX(H3_CELL_TO_BOUNDARY_WKT(h3_2)::geometry) BETWEEN -179 AND 179
    AND    ST_YMIN(H3_CELL_TO_BOUNDARY_WKT(h3_2)::geometry) BETWEEN -88  AND  88
) TO 'sites.h3_2.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

These are some of the sites in Southern California and Nevada.

Launch Vehicles

Below, I'll download and prepare the launch vehicles dataset for DuckDB.

$ wget https://planet4589.org/space/gcat/tsv/tables/lv.tsv

$ head -n1 lv.tsv | tail -c+2 > working.csv
$ tail -n+3 lv.tsv >> working.csv

I'll clean up the values, casting them to stronger types where I can and producing a parquet file with their contents.

$ ~/duckdb

COPY(
    SELECT Apogee: IF(Apogee='-', NULL, Apogee::INT),
           Class,
           Diameter: IF(Diameter='-', NULL, Diameter::FLOAT),
           GTO_Capacity: IF(GTO_Capacity='-', NULL, GTO_Capacity::INT),
           Launch_Mass: IF(Launch_Mass='-', NULL, Launch_Mass::FLOAT),
           Length: IF(Length='-', NULL, Length::FLOAT),
           LEO_Capacity: IF(LEO_Capacity='-', NULL, LEO_Capacity::INT),
           LV_Alias: IF(LV_Alias='-', NULL, TRIM(LV_Alias)),
           LV_Family: TRIM(LV_Family),
           LV_Manufacturer: IF(LV_Manufacturer='-', NULL, TRIM(LV_Manufacturer)),
           LV_Max_Stage: LV_Max_Stage::INT,
           LV_Min_Stage: LV_Min_Stage::INT,
           LV_Name: TRIM(LV_Name),
           LV_Variant: TRIM(LV_Variant),
           Range: IF(Range='-', NULL, Range::INT),
           TO_Thrust: IF(TO_Thrust='-', NULL, TO_Thrust::INT),
    FROM     'working.csv'
) TO 'lv.parquet' (
  FORMAT 'PARQUET',
  CODEC  'ZSTD',
  COMPRESSION_LEVEL 22,
  ROW_GROUP_SIZE 15000);

There are 1,820 records in this dataset.

SELECT COUNT(*)
FROM   'lv.parquet'; -- 1820

Below is a breakdown of unique values and NULL coverage across each column.

SELECT   column_name,
         column_type[:30],
         null_percentage,
         approx_unique,
         min[:30],
         max[:30]
FROM     (SUMMARIZE
          FROM 'lv.parquet')
ORDER BY LOWER(column_name);

┌─────────────────┬──────────────────┬─────────────────┬───────────────┬──────────────────────┬──────────┐
│   column_name   │ column_type[:30] │ null_percentage │ approx_unique │       min[:30]       │ max[:30] │
│     varchar     │     varchar      │  decimal(9,2)   │     int64     │       varchar        │ varchar  │
├─────────────────┼──────────────────┼─────────────────┼───────────────┼──────────────────────┼──────────┤
│ Apogee          │ INTEGER          │            0.33 │           169 │ 0                    │ 400000   │
│ Class           │ VARCHAR          │            0.00 │            15 │ 0                    │ Y        │
│ Diameter        │ FLOAT            │            0.16 │           160 │ 0.0                  │ 14.0     │
│ GTO_Capacity    │ INTEGER          │            2.47 │           100 │ 0                    │ 22200    │
│ Launch_Mass     │ FLOAT            │            3.13 │           393 │ 0.0                  │ 7000.0   │
│ Length          │ FLOAT            │            0.99 │           304 │ 0.0                  │ 124.4    │
│ LEO_Capacity    │ INTEGER          │            7.69 │           152 │ 0                    │ 200000   │
│ LV_Alias        │ VARCHAR          │           98.74 │            18 │ ARPA Taurus          │ Zenit-3F │
│ LV_Family       │ VARCHAR          │            0.00 │           468 │ 10KS2500             │ Zulfiqar │
│ LV_Manufacturer │ VARCHAR          │            0.16 │           320 │ ABL                  │ ZKYT     │
│ LV_Max_Stage    │ INTEGER          │            0.00 │             6 │ 1                    │ 6        │
│ LV_Min_Stage    │ INTEGER          │            0.00 │             3 │ -1                   │ 1        │
│ LV_Name         │ VARCHAR          │            0.00 │          1369 │ 1Ya2TA               │ eMRBM-T1 │
│ LV_Variant      │ VARCHAR          │            0.00 │           117 │ (2)                  │ d        │
│ Range           │ INTEGER          │           85.82 │            66 │ 20                   │ 16000    │
│ TO_Thrust       │ INTEGER          │           25.82 │           356 │ 0                    │ 103000   │
└─────────────────┴──────────────────┴─────────────────┴───────────────┴──────────────────────┴──────────┘

Below is an example record for Starship V4.

$ echo "FROM  'lv.parquet'
        WHERE LV_Name = 'Starship V4'" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "Apogee": 200,
    "Class": "O",
    "Diameter": 9.0,
    "GTO_Capacity": 0,
    "LEO_Capacity": 200000,
    "LV_Alias": null,
    "LV_Family": "Starship",
    "LV_Manufacturer": "SPX",
    "LV_Max_Stage": 2,
    "LV_Min_Stage": 1,
    "LV_Name": "Starship V4",
    "LV_Variant": "-",
    "Launch_Mass": 7000.0,
    "Length": 124.4000015258789,
    "Range": null,
    "TO_Thrust": 103000
  }
]

Below are the top 20 launch vehicles by launch mass.

$ ~/duckdb

SELECT   Launch_Mass,
         Length,
         TO_Thrust,
         LV_Name
FROM     'lv.parquet'
ORDER BY Launch_Mass DESC
LIMIT    20;

┌─────────────┬────────┬───────────┬─────────────────────┐
│ Launch_Mass │ Length │ TO_Thrust │       LV_Name       │
│    float    │ float  │   int32   │       varchar       │
├─────────────┼────────┼───────────┼─────────────────────┤
│      7000.0 │  124.4 │    103000 │ Starship V4         │
│      5500.0 │  123.0 │     80800 │ Starship V3         │
│      5200.0 │  121.4 │     80800 │ Starship V2         │
│      5020.0 │  121.3 │     72500 │ Starship V1.0       │
│      2913.0 │  110.6 │     34030 │ Saturn V            │
│      2822.0 │  104.8 │     34030 │ Saturn V            │
│      2788.0 │  105.3 │     45300 │ N-1 11A52           │
│      2608.0 │   98.1 │     39000 │ SLS Block 1         │
│      2524.0 │   58.7 │     35139 │ Energiya            │
│      2524.0 │   58.7 │     35139 │ Energiya/Buran      │
│      2040.0 │   38.1 │     28200 │ Space Shuttle       │
│      1470.0 │   50.0 │     12000 │ Starship V1.0 Ship  │
│      1400.0 │   69.2 │     17000 │ Falcon Heavy        │
│      1200.0 │  122.0 │     25622 │ New Glenn 9x4       │
│      1050.0 │   98.0 │     17219 │ New Glenn 7x2       │
│      1050.0 │   98.0 │     19928 │ New Glenn 7x2       │
│       939.0 │   62.2 │     15000 │ Titan 401B/Centaur  │
│       925.0 │   54.0 │     15000 │ Titan 402B/IUS      │
│       910.8 │   67.0 │     17688 │ Vulcan Centaur VC6S │
│       910.0 │   54.0 │     14200 │ Titan 402A/IUS      │
└─────────────┴────────┴───────────┴─────────────────────┘

Launches

Below, I'll download and prepare the launches dataset for DuckDB.

$ wget https://planet4589.org/space/gcat/tsv/launch/launch.tsv

$ head -n1 launch.tsv | tail -c+2 > working.csv
$ tail -n+3 launch.tsv >> working.csv

I'll clean up the values, casting them to stronger types where I can and producing a parquet file with their contents.

$ ~/duckdb

COPY(
    SELECT Agency,
           Apogee: IF(TRIM(Apogee)='-', NULL, Apogee::INT),
           Ascent_Pad: IF(TRIM(Ascent_Pad)='-', NULL, TRIM(Ascent_Pad)),
           Ascent_Site: IF(TRIM(Ascent_Site)='-', NULL, TRIM(Ascent_Site)),
           Category: IF(TRIM(Category)='-', NULL, TRIM(Category)),
           Cite: IF(TRIM(Cite)='-', NULL, TRIM(Cite)),
           Dest: IF(TRIM(Dest)='-', NULL, TRIM(Dest)),
           FailCode: IF(TRIM(FailCode)='-', NULL, TRIM(FailCode)),
           Flight,
           Flight_ID,
           FlightCode,
           "Group": IF(TRIM("Group")='-', NULL, TRIM("Group")),
           Launch_Year: Launch_Date[:4]::INT,
           Launch_Date,
           Launch_JD,
           Launch_Pad: IF(TRIM(Launch_Pad)='-', NULL, TRIM(Launch_Pad)),
           Launch_Site,
           Launch_Tag,
           LaunchCode,
           LTCite: IF(TRIM(LTCite)='-', NULL, TRIM(LTCite)),
           LV_Type,
           Mission: IF(TRIM(Mission)='-', NULL, TRIM(Mission)),
           Notes: IF(TRIM(Notes)='-', NULL, TRIM(Notes)),
           OrbPay: OrbPay::FLOAT,
           Platform: IF(TRIM(Platform)='-', NULL, TRIM(Platform)),
           Range: IF(TRIM(Range)='-', NULL, Range::INT),
           Variant: IF(TRIM(Variant)='-', NULL, TRIM(Variant)),
    FROM     'working.csv'
) TO 'launch.parquet' (
  FORMAT 'PARQUET',
  CODEC  'ZSTD',
  COMPRESSION_LEVEL 22,
  ROW_GROUP_SIZE 15000);

There are 75,811 records in this dataset.

SELECT COUNT(*)
FROM   'launch.parquet'; -- 75811

Below is a breakdown of unique values and NULL coverage across each column.

SELECT   column_name,
         column_type[:30],
         null_percentage,
         approx_unique,
         min[:30],
         max[:30]
FROM     (SUMMARIZE
          FROM 'launch.parquet')
ORDER BY LOWER(column_name);

┌─────────────┬──────────────────┬─────────────────┬───────────────┬──────────────────────┬──────────────────────────┐
│ column_name │ column_type[:30] │ null_percentage │ approx_unique │       min[:30]       │         max[:30]         │
│   varchar   │     varchar      │  decimal(9,2)   │     int64     │       varchar        │         varchar          │
├─────────────┼──────────────────┼─────────────────┼───────────────┼──────────────────────┼──────────────────────────┤
│ Agency      │ VARCHAR          │            0.00 │           745 │ 10ADS                │ ZSU                      │
│ Apogee      │ INTEGER          │            1.28 │          2929 │ 0                    │ 225000                   │
│ Ascent_Pad  │ VARCHAR          │           99.01 │            66 │ A                    │ Yellow Sea               │
│ Ascent_Site │ VARCHAR          │           98.99 │            37 │ A51                  │ WTR                      │
│ Category    │ VARCHAR          │            0.04 │           955 │ -            /0:30   │ usaAeron                 │
│ Cite        │ VARCHAR          │           86.87 │          1448 │ (MexicoImpact)       │ zenite.nu                │
│ Dest        │ VARCHAR          │           91.68 │           319 │ (ASC BOA)            │ Zhytomyr                 │
│ FailCode    │ VARCHAR          │           96.36 │            88 │ AD                   │ WX                       │
│ Flight      │ VARCHAR          │            0.00 │         22029 │ "Scores of missiles" │ vs MKO camps             │
│ Flight_ID   │ VARCHAR          │            0.00 │         25955 │ #117 R-112           │ xxx/B1103.1  F625        │
│ FlightCode  │ VARCHAR          │            0.00 │          2417 │ (100th)              │ silo                     │
│ Group       │ VARCHAR          │           64.08 │          2348 │ 321SMW/              │ ZARM/Guerlebeck          │
│ Launch_Date │ VARCHAR          │            0.00 │         66313 │ 1942 Aug 16 1115     │ 2026 Mar 31              │
│ Launch_JD   │ VARCHAR          │            0.00 │         67539 │ 2430523.95           │ 2461154.77               │
│ Launch_Pad  │ VARCHAR          │           66.96 │          1038 │ 200NM                │ lC9A                     │
│ Launch_Site │ VARCHAR          │            0.00 │           589 │ A51                  │ ZINGST                   │
│ Launch_Tag  │ VARCHAR          │            0.00 │         79286 │ 1942-A01             │ 2026-S99                 │
│ Launch_Year │ INTEGER          │            0.00 │            89 │ 1942                 │ 2026                     │
│ LaunchCode  │ VARCHAR          │            0.00 │            52 │ !                    │ YS                       │
│ LTCite      │ VARCHAR          │            0.01 │          3415 │ 1STRAD               │ zenite.nu/orbita         │
│ LV_Type     │ VARCHAR          │            0.00 │          1340 │ 1Ya2TA               │ eMRBM-T1                 │
│ Mission     │ VARCHAR          │           82.35 │          8502 │ 'Ofeq-1              │ ziyuan yihao 02C weixing │
│ Notes       │ VARCHAR          │           98.37 │           860 │ (ex-U2)              │ www.wsmr.army.mil        │
│ OrbPay      │ FLOAT            │            0.00 │          2774 │ 0.0                  │ 132.025                  │
│ Platform    │ VARCHAR          │           91.80 │           359 │ 824YONG              │ ZULUV?                   │
│ Range       │ INTEGER          │           97.65 │           463 │ 0                    │ 14560                    │
│ Variant     │ VARCHAR          │           95.04 │           113 │ (2)                  │ d                        │
└─────────────┴──────────────────┴─────────────────┴───────────────┴──────────────────────┴──────────────────────────┘

Below is the record for the Starlink Group 6-88 mission.

$ echo "FROM  'launch.parquet'
        WHERE Mission = 'Starlink Group 6-88'" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "Agency": "SPX",
    "Apogee": null,
    "Ascent_Pad": null,
    "Ascent_Site": null,
    "Category": "Sat LEO D",
    "Cite": null,
    "Dest": null,
    "FailCode": null,
    "Flight": "Starlink-345",
    "FlightCode": "V2MO 6-88",
    "Flight_ID": "585/B1101.1  F585",
    "Group": "C",
    "LTCite": "nasaspaceflight.com",
    "LV_Type": "Falcon 9",
    "LaunchCode": "OS",
    "Launch_Date": "2026 Jan  4 0648:10",
    "Launch_JD": "2461044.78 ",
    "Launch_Pad": "LC40",
    "Launch_Site": "CC",
    "Launch_Tag": "2026-002 ",
    "Launch_Year": 2026,
    "Mission": "Starlink Group 6-88",
    "Notes": null,
    "OrbPay": 16.100000381469727,
    "Platform": null,
    "Range": null,
    "Variant": "FT5"
  }
]

Below are the category counts for launches with a 2026 tag prefix.

$ ~/duckdb

SELECT   Category,
         COUNT(*)
FROM     'launch.parquet'
WHERE    Launch_Tag LIKE '2026%'
GROUP BY 1
ORDER BY 2 DESC;

┌────────────────────┬──────────────┐
│      Category      │ count_star() │
│      varchar       │    int64     │
├────────────────────┼──────────────┤
│ Weapon             │          215 │
│ Sat LEO D          │           50 │
│ Sat LEO O          │           12 │
│ Test               │           10 │
│ Sat SSO O          │            5 │
│ Auroral            │            4 │
│ Sat SSO L          │            4 │
│ Sat SSO D          │            4 │
│ Sat LEO OO         │            4 │
│ Sat SSO OO         │            3 │
│ Sat GTO D          │            3 │
│ Tech               │            2 │
│ Interceptor Weapon │            2 │
│ Hypersonic         │            2 │
│ Sat ISS D          │            2 │
│ Target?            │            2 │
│ Sat DSO E          │            1 │
│ Sat GEO SO         │            1 │
│ Sat SSO SO         │            1 │
│ Op Test            │            1 │
│ Sat GEO O          │            1 │
│ Sat ISS O          │            1 │
│ Aeron              │            1 │
│ Sat SSO SD         │            1 │
│ Sat GTO O          │            1 │
│ Sat LEO P          │            1 │
│ Tourist Flight     │            1 │
│ Sat MOL SO         │            1 │
└────────────────────┴──────────────┘

Below are the Starlink launches for this year.

SELECT   Agency,
         LV_Type,
         Launch_Tag,
         Mission
FROM     'launch.parquet'
WHERE    Launch_Tag LIKE '2026%'
AND      Flight LIKE 'Starlink%'
ORDER BY Launch_Tag DESC;

┌─────────┬──────────┬────────────┬──────────────────────┐
│ Agency  │ LV_Type  │ Launch_Tag │       Mission        │
│ varchar │ varchar  │  varchar   │       varchar        │
├─────────┼──────────┼────────────┼──────────────────────┤
│ SPX     │ Falcon 9 │ 2026-089   │ Starlink Group 17-14 │
│ SPX     │ Falcon 9 │ 2026-086   │ Starlink Group 17-22 │
│ SPX     │ Falcon 9 │ 2026-082   │ Starlink Group 17-27 │
│ SPX     │ Falcon 9 │ 2026-081   │ Starlink Group 10-24 │
│ SPX     │ Falcon 9 │ 2026-077   │ Starlink Group 17-21 │
│ SPX     │ Falcon 9 │ 2026-073   │ Starlink Group 17-35 │
│ SPX     │ Falcon 9 │ 2026-070   │ Starlink Group 10-58 │
│ SPX     │ Falcon 9 │ 2026-068   │ Starlink Group 10-44 │
│ SPX     │ Falcon 9 │ 2026-063   │ Starlink Group 17-17 │
│ SPX     │ Falcon 9 │ 2026-059   │ Starlink Group 10-62 │
│ SPX     │ Falcon 9 │ 2026-057   │ Starlink Group 17-15 │
│ SPX     │ Falcon 9 │ 2026-055   │ Starlink Group 10-33 │
│ SPX     │ Falcon 9 │ 2026-054   │ Starlink Group 10-46 │
│ SPX     │ Falcon 9 │ 2026-053   │ Starlink Group 17-24 │
│ SPX     │ Falcon 9 │ 2026-050   │ Starlink Group 10-48 │
│ SPX     │ Falcon 9 │ 2026-049   │ Starlink Group 17-31 │
│ SPX     │ Falcon 9 │ 2026-044   │ Starlink Group 17-18 │
│ SPX     │ Falcon 9 │ 2026-042   │ Starlink Group 10-40 │
│ SPX     │ Falcon 9 │ 2026-041   │ Starlink Group 10-41 │
│ SPX     │ Falcon 9 │ 2026-040   │ Starlink Group 17-23 │
│ SPX     │ Falcon 9 │ 2026-039   │ Starlink Group 6-108 │
│ SPX     │ Falcon 9 │ 2026-038   │ Starlink Group 17-26 │
│ SPX     │ Falcon 9 │ 2026-037   │ Starlink Group 6-110 │
│ SPX     │ Falcon 9 │ 2026-036   │ Starlink Group 6-104 │
│ SPX     │ Falcon 9 │ 2026-035   │ Starlink Group 17-25 │
│ SPX     │ Falcon 9 │ 2026-034   │ Starlink Group 10-36 │
│ SPX     │ Falcon 9 │ 2026-033   │ Starlink Group 6-103 │
│ SPX     │ Falcon 9 │ 2026-032   │ Starlink Group 17-13 │
│ SPX     │ Falcon 9 │ 2026-026   │ Starlink Group 17-34 │
│ SPX     │ Falcon 9 │ 2026-025   │ Starlink Group 17-33 │
│ SPX     │ Falcon 9 │ 2026-022   │ Starlink Group 17-32 │
│ SPX     │ Falcon 9 │ 2026-020   │ Starlink Group 6-101 │
│ SPX     │ Falcon 9 │ 2026-018   │ Starlink Group 17-19 │
│ SPX     │ Falcon 9 │ 2026-016   │ Starlink Group 17-20 │
│ SPX     │ Falcon 9 │ 2026-014   │ Starlink Group 17-30 │
│ SPX     │ Falcon 9 │ 2026-012   │ Starlink Group 6-100 │
│ SPX     │ Falcon 9 │ 2026-008   │ Starlink Group 6-98  │
│ SPX     │ Falcon 9 │ 2026-005   │ Starlink Group 6-97  │
│ SPX     │ Falcon 9 │ 2026-003   │ Starlink Group 6-96  │
│ SPX     │ Falcon 9 │ 2026-002   │ Starlink Group 6-88  │
└─────────┴──────────┴────────────┴──────────────────────┘

These are the top 20 agency-category launch pairs for this decade.

SELECT   Agency,
         Category,
         COUNT(*)
FROM     'launch.parquet'
WHERE    Launch_Tag LIKE '202%'
GROUP BY 1, 2
ORDER BY 3 DESC
LIMIT    20;

┌─────────┬─────────────┬──────────────┐
│ Agency  │  Category   │ count_star() │
│ varchar │   varchar   │    int64     │
├─────────┼─────────────┼──────────────┤
│ SPX     │ Sat LEO D   │          338 │
│ IR      │ Weapon      │          284 │
│ KPA     │ Test        │           83 │
│ SPX     │ Sat LEO D 4 │           63 │
│ CALT    │ Sat GTO O   │           61 │
│ SAST    │ Sat LEO O   │           38 │
│ SAST    │ Sat SSO L   │           37 │
│ SPX     │ Sat SSO D   │           37 │
│ SPX     │ Sat ISS D   │           36 │
│ FKA     │ Sat ISS O   │           34 │
│ RLABN   │ Sat LEO OO  │           29 │
│ SAST    │ Sat SSO O   │           26 │
│ RLABN   │ Sat SSO OO  │           25 │
│ ANSAL   │ Weapon      │           24 │
│ MDA     │ Target      │           24 │
│ SAST    │ Sat SSO D   │           24 │
│ USNSSP  │ OT          │           21 │
│ PLA2AC  │ Test        │           20 │
│ CALT    │ Sat LEO O   │           20 │
│ JRG     │ OT          │           20 │
└─────────┴─────────────┴──────────────┘

Satellites

Below, I'll download and prepare the objects dataset, which contains satellites, for DuckDB.

$ wget https://planet4589.org/space/gcat/tsv/cat/satcat.tsv

$ head -n1 satcat.tsv | tail -c+2 > working.csv
$ tail -n+3 satcat.tsv >> working.csv

I'll clean up the values, casting them to stronger types where I can and producing a parquet file with their contents.

$ ~/duckdb

COPY(
    SELECT AltNames: IF(TRIM(AltNames)='-', NULL, TRIM(AltNames)),
           Apogee: IF(TRIM(Apogee) = '-' OR TRIM(Apogee) = 'Inf', NULL, Apogee::INT),
           Bus: IF(TRIM(Bus)='-', NULL, TRIM(Bus)),
           DDate: IF(TRIM(DDate)='-', NULL, TRIM(DDate)),
           Dest: IF(TRIM(Dest)='-', NULL, TRIM(Dest)),
           Diameter: IF(TRIM(Diameter)='-', NULL, Diameter::FLOAT),
           DryMass: IF(TRIM(DryMass)='-', NULL, TRIM(DryMass)),
           Inc: IF(TRIM(Inc)='-', NULL, Inc::FLOAT),
           JCAT: TRIM(JCAT),
           Launch_Tag: IF(TRIM(Launch_Tag)='-', NULL, TRIM(Launch_Tag)),
           LDate: IF(TRIM(LDate)='-', NULL, TRIM(LDate)),
           Length: IF(TRIM(Length)='-', NULL, Length::FLOAT),
           Manufacturer: IF(TRIM(Manufacturer)='-', NULL, TRIM(Manufacturer)),
           Mass: IF(TRIM(Mass)='-', NULL, Mass::FLOAT),
           Motor: IF(TRIM(Motor)='-', NULL, TRIM(Motor)),
           Name: TRIM(Name),
           ODate: IF(TRIM(ODate)='-', NULL, TRIM(ODate)),
           OpOrbit: IF(TRIM(OpOrbit)='-', NULL, TRIM(OpOrbit)),
           OQUAL: IF(TRIM(OQUAL)='-', NULL, TRIM(OQUAL)),
           Owner: IF(TRIM(Owner)='-', NULL, TRIM(Owner)),
           Parent: IF(TRIM(Parent)='-', NULL, TRIM(Parent)),
           Perigee: IF(TRIM(Perigee)='-', NULL, TRIM(Perigee)),
           Piece: IF(TRIM(Piece)='-', NULL, TRIM(Piece)),
           PLName: IF(TRIM(PLName)='-', NULL, TRIM(PLName)),
           "Primary": IF(TRIM("Primary")='-', NULL, TRIM("Primary")),
           Satcat: IF(TRIM(Satcat)='-', NULL, TRIM(Satcat)),
           SDate: IF(TRIM(SDate)='-', NULL, TRIM(SDate)),
           SDate_Year: IF(TRIM(SDate)='-', NULL, SDate[:4]::INT),
           Shape: IF(TRIM(Shape)='-', NULL, TRIM(Shape)),
           Span: IF(TRIM(Span)='-', NULL, Span::FLOAT),
           State: IF(TRIM(State)='-', NULL, TRIM(State)),
           Status: IF(TRIM(Status)='-', NULL, TRIM(Status)),
           TotMass: IF(TRIM(TotMass)='-', NULL, TotMass::FLOAT),
           Type: IF(TRIM(Type)='-', NULL, TRIM(Type)),
           Coarse_type:
               CASE WHEN Type[1] = 'P' THEN 'Payload (for orbital attempt)'
                    WHEN Type[1] = 'C' THEN 'Component'
                    WHEN Type[1] = 'R' THEN 'Launch vehicle stage'
                    WHEN Type[1] = 'D' THEN 'Fragmentation debris'
                    WHEN Type[1] = 'S' THEN 'Suborbital payload (e.g. sounding rocket payload or missile reentry vehicle)'
                    WHEN Type[1] = 'X' THEN 'Catalog entry that has been deleted (used in auxcat etc.)'
                    WHEN Type[1] = 'Z' THEN 'Spurious catalog entry (was in SATCAT, perhaps in TLEs, but there was no real object)'
               END,

           Attach_flag:
               CASE WHEN Type[3] = 'A' THEN 'Permanently attached component or payload'
                    WHEN Type[3] = 'F' THEN 'Stuck attached by mistake'
                    WHEN Type[3] = 'S' THEN 'Expected to separate in future'
                    WHEN Type[3] = 'T' THEN 'Never flew free but transferred'
                    WHEN Type[3] = 'I' THEN 'Internal'
               END,

           Subtype_flag:
               CASE WHEN Type[4] = 'A' THEN 'Payload adapter, support structures, interfaces'
                    WHEN Type[4] = 'B' THEN 'Battery explosion debris'
                    WHEN Type[4] = 'C' THEN 'Passive calibration satellites, test objects or chaff'
                    WHEN Type[4] = 'D' THEN 'Dummy satellite'
                    WHEN Type[4] = 'E' THEN 'Spacesuit on tethered spacewalk'
                    WHEN Type[4] = 'F' THEN 'Fairings and other covers'
                    WHEN Type[4] = 'G' THEN 'General, miscellaneous debris'
                    WHEN Type[4] = 'H' THEN 'Human spaceflight related'
                    WHEN Type[4] = 'I' THEN 'Impact (accidental collision)'
                    WHEN Type[4] = 'J' THEN 'Anomalous debris (insulation, soft material, ablated material)'
                    WHEN Type[4] = 'K' THEN 'Possible solid motor slag'
                    WHEN Type[4] = 'L' THEN 'Separated from vehicle after landing (rovers, etc)'
                    WHEN Type[4] = 'M' THEN 'Jettisoned motor or tank'
                    WHEN Type[4] = 'N' THEN 'Nuclear reactor core or coolant blob'
                    WHEN Type[4] = 'O' THEN 'Unknown debris released at orbit insertion'
                    WHEN Type[4] = 'P' THEN 'Propulsion related, residual-propellant breakup'
                    WHEN Type[4] = 'Q' THEN 'Aerodynamic breakup at low perigee'
                    WHEN Type[4] = 'R' THEN 'Reentry vehicle'
                    WHEN Type[4] = 'S' THEN 'Subsatellite or subpayload'
                    WHEN Type[4] = 'T' THEN 'Ejected section of payload'
                    WHEN Type[4] = 'U' THEN 'Untethered EVA'
                    WHEN Type[4] = 'V' THEN 'Ejection mechanism (deploy canister, clamp band)'
                    WHEN Type[4] = 'W' THEN 'Weapons test, ASAT debris'
                    WHEN Type[4] = 'X' THEN 'Debris of unknown nature'
                    WHEN Type[4] = 'Y' THEN 'Despin (yo-yo) device'
                    WHEN Type[4] = 'Z' THEN 'Breakup debris from on-board destruct device'
               END,

           Orbit_flag:
               CASE WHEN Type[5] = 'D' THEN 'Deep Space or escape'
                    WHEN Type[5] = 'E' THEN 'Destroyed in pad explosion'
                    WHEN Type[5] = 'F' THEN 'Failed to reach orbit'
                    WHEN Type[5] = 'L' THEN 'Active on planet surface during this phase'
                    WHEN Type[5] = 'M' THEN 'Missing from SATCAT by mistake (EXPRESS, IXV)'
                    WHEN Type[5] = 'O' THEN 'Orbital-Energy but Non-Orbit'
                    WHEN Type[5] = 'P' THEN 'Partial orbit - reached legit orbit but deorbited after less than 1 rev'
                    WHEN Type[5] = 'R' THEN 'Reentry orbit: objects that were attached and separated in post-deorbit-burn suborbital trajectory'
                    WHEN Type[5] = 'S' THEN 'Near-Orbit (marginally suborbital)'
                    WHEN Type[5] = 'T' THEN 'Transient orbit - separated just (perhaps seconds) before deorbit'
                    WHEN Type[5] = 'V' THEN 'Escape energy but not deep space'
                    WHEN Type[5] = 'X' THEN 'Extra catalog entry for extraterrestrially launched object'
                    WHEN Type[5] = 'Z' THEN 'Launch from extraterrestrial object, recataloged with new D series number'
               END,

           Status_or_failure_flag:
               CASE WHEN Type[8] = '*' THEN 'This object was the one that failed during launch.'
                    WHEN Type[8] = 'A' THEN 'Satellite ascending: orbit raising to op orbit (or lowering from high drift orbit)'
                    WHEN Type[8] = 'D' THEN 'Satellite in plane drift orbit'
                    WHEN Type[8] = 'F' THEN 'Satellite failed early in mission, before reaching operational orbit (”screened”)'
                    WHEN Type[8] = 'G' THEN 'Satellite retired to a graveyard orbit of some kind'
                    WHEN Type[8] = 'L' THEN 'Satellite removed far from operational constellation by lowering or raising orbit'
                    WHEN Type[8] = 'M' THEN 'Satellite failed in operational orbit and is undergoing uncontrolled decay - non manueverable'
                    WHEN Type[8] = 'O' THEN 'Satellite is active in operational orbit'
                    WHEN Type[8] = 'R' THEN 'Satellite active orbit lowering to reentry'
                    WHEN Type[8] = 'S' THEN 'Satellite was used for special tests outside of main constellation'
                    WHEN Type[8] = 'T' THEN 'Satellite removed slighly (and possibly temporarily) from operational constellation by lowering or raising orbit a little'
                    WHEN Type[8] = 'U' THEN 'Satellite apparently malfunctioning, held in intermediate orbit for debugging?'
               END,

           ID_flag:
               CASE WHEN Type[9] = '?' THEN 'The association of this satellite with this catalog number is a guess because JSpOC has not yet assigned a name. ID may change in future.'
                    WHEN Type[9] = '+' THEN 'Starlink filing with FCC reports this satellite is out of service but could still maneuver, at the time of filing. Starlink filing with FCC reports this satellite is out of service and can not maneuver.'
                    WHEN Type[9] = 'm' THEN 'Multiple objects. This entry is a placeholder for a known debris event where no debris has yet been cataloged by DoD.'
                    WHEN Type[9] = 'C' THEN 'US government orbital data for this object are secret.'
                    WHEN Type[9] = 'c' THEN 'Older US government orbital data for this object were secret, but current data is public.'
                    WHEN Type[9] = 'U' THEN 'This object is a cargo item on ISS (or other station) and has been assigned to a likely cargo launch but the actual launch for this object is not known.'
                    WHEN Type[9] = 'D' THEN 'This object is a cargo item on ISS (or other station). The return date is uncertain and I’ve made a guess at which ship it went down on.'
                    WHEN Type[9] = 'X' THEN 'We don’t know which launch this object is from, so the launch date is unknown (and other parameters may also be unknown).'
                    WHEN Type[9] = 's' THEN 'Disagreement between TLE and SupTLE data (true for S45139)'
               END

    FROM     READ_CSV('working.csv', sample_size=-1)
) TO 'satcat.parquet' (
  FORMAT 'PARQUET',
  CODEC  'ZSTD',
  COMPRESSION_LEVEL 22,
  ROW_GROUP_SIZE 15000);

There are 68,834 records in this dataset.

SELECT COUNT(*)
FROM   'satcat.parquet'; -- 68834

Below is a breakdown of unique values and NULL coverage across each column.

SELECT   column_name,
         column_type[:30],
         null_percentage,
         approx_unique,
         min[:30],
         max[:30]
FROM     (SUMMARIZE
          FROM 'satcat.parquet')
ORDER BY LOWER(column_name);

┌────────────────────────┬──────────────────┬─────────────────┬───────────────┬────────────────────────────────┬────────────────────────────────┐
│      column_name       │ column_type[:30] │ null_percentage │ approx_unique │            min[:30]            │            max[:30]            │
│        varchar         │     varchar      │  decimal(9,2)   │     int64     │            varchar             │            varchar             │
├────────────────────────┼──────────────────┼─────────────────┼───────────────┼────────────────────────────────┼────────────────────────────────┤
│ AltNames               │ VARCHAR          │           93.49 │          6148 │ 02                             │ r7411F633 No. 670-51-102:RA    │
│ Apogee                 │ INTEGER          │            0.16 │          7251 │ -4788696                       │ 3971603                        │
│ Attach_flag            │ VARCHAR          │           99.46 │             4 │ Expected to separate in future │ Stuck attached by mistake      │
│ Bus                    │ VARCHAR          │           46.24 │          2057 │ 'Ofeq                          │ sPCM                           │
│ Coarse_type            │ VARCHAR          │            0.09 │             5 │ Component                      │ Spurious catalog entry (was in │
│ DDate                  │ VARCHAR          │           47.80 │         16604 │ 1957 Dec  1 1000?              │ 2205 Jun  6                    │
│ Dest                   │ VARCHAR          │           94.84 │          3075 │ 0.5W 43.8S                     │ Zimbabwe                       │
│ Diameter               │ FLOAT            │            0.09 │           120 │ 0.0                            │ 41.0                           │
│ DryMass                │ VARCHAR          │            0.09 │          1661 │ 0                              │ 9996                           │
│ ID_flag                │ VARCHAR          │           97.22 │             6 │ Disagreement between TLE and S │ US government orbital data for │
│ Inc                    │ FLOAT            │            0.09 │          6895 │ 0.0                            │ 150.97                         │
│ JCAT                   │ VARCHAR          │            0.00 │         81214 │ S00001                         │ S68834                         │
│ Launch_Tag             │ VARCHAR          │            0.09 │          6171 │ 1957 ALP                       │ 2026-091                       │
│ LDate                  │ VARCHAR          │            0.09 │          4848 │ 1957 Nov  3                    │ 2026 Mar 30                    │
│ Length                 │ FLOAT            │            0.09 │           221 │ 0.0                            │ 41.0                           │
│ Manufacturer           │ VARCHAR          │           34.40 │          1718 │ AACH                           │ ZZQY                           │
│ Mass                   │ FLOAT            │            0.09 │          1971 │ 0.0                            │ 104420.0                       │
│ Motor                  │ VARCHAR          │           95.87 │           341 │ 0                              │ YF-85 x 4                      │
│ Name                   │ VARCHAR          │            0.00 │         29692 │ 'Ofeq-1                        │ uHETsat                        │
│ ODate                  │ VARCHAR          │            0.16 │         15034 │ 1957 Nov  3                    │ 2026 Mar 31                    │
│ OpOrbit                │ VARCHAR          │            0.11 │            27 │ CLO                            │ VHEO                           │
│ OQUAL                  │ VARCHAR          │           99.47 │            18 │ - s                            │ UN                             │
│ Orbit_flag             │ VARCHAR          │           99.22 │             6 │ Deep Space or escape           │ Partial orbit - reached legit  │
│ Owner                  │ VARCHAR          │            0.09 │          1761 │ 21AT                           │ ZZWYZ?                         │
│ Parent                 │ VARCHAR          │            0.11 │         21722 │ A00002                         │ S68824                         │
│ Perigee                │ VARCHAR          │            0.09 │          3903 │ -110                           │ 9999                           │
│ Piece                  │ VARCHAR          │            0.09 │         79387 │ 1957 ALP 1                     │ 2026-091D                      │
│ PLName                 │ VARCHAR          │           44.05 │         28956 │ 'Ofeq-1                        │ was [UNID128]                  │
│ Primary                │ VARCHAR          │            0.09 │             1 │ Earth                          │ Earth                          │
│ Satcat                 │ VARCHAR          │            0.09 │         69528 │ 00001                          │ NNA                            │
│ SDate                  │ VARCHAR          │            0.09 │         17672 │ 1957 Nov  3 0235               │ 2026 Mar 30 2216:37            │
│ SDate_Year             │ INTEGER          │            0.09 │            69 │ 1957                           │ 2026                           │
│ Shape                  │ VARCHAR          │           46.51 │           543 │ 2 Box + Tether                 │ Unk                            │
│ Span                   │ FLOAT            │            0.09 │           329 │ 0.0                            │ 19695.0                        │
│ State                  │ VARCHAR          │            0.09 │           123 │ AM                             │ ZW                             │
│ Status                 │ VARCHAR          │            0.09 │            19 │ AO                             │ TFR                            │
│ Status_or_failure_flag │ VARCHAR          │           79.67 │            11 │ Satellite active orbit lowerin │ This object was the one that f │
│ Subtype_flag           │ VARCHAR          │           45.86 │            26 │ Aerodynamic breakup at low per │ Weapons test, ASAT debris      │
│ TotMass                │ FLOAT            │            0.09 │          2064 │ 0.0                            │ 117440.0                       │
│ Type                   │ VARCHAR          │            0.09 │           404 │ C   X                          │ Z  X    c                      │
└────────────────────────┴──────────────────┴─────────────────┴───────────────┴────────────────────────────────┴────────────────────────────────┘

Below are the most represented states in this dataset since 2022.

SELECT   State,
         COUNT(*)
FROM     'satcat.parquet'
WHERE    SDate_Year >= 2022
GROUP BY 1
ORDER BY 2 DESC
LIMIT    20;

┌─────────┬──────────────┐
│  State  │ count_star() │
│ varchar │    int64     │
├─────────┼──────────────┤
│ US      │        11731 │
│ CN      │         3040 │
│ RU      │          513 │
│ UK      │          313 │
│ J       │          148 │
│ NZ      │          126 │
│ I       │           84 │
│ F       │           81 │
│ IN      │           63 │
│ KR      │           55 │
│ E       │           55 │
│ L       │           49 │
│ CA      │           47 │
│ D       │           47 │
│ TR      │           45 │
│ AU      │           35 │
│ FI      │           33 │
│ UY      │           32 │
│ IR      │           29 │
│ IL      │           29 │
└─────────┴──────────────┘

This is the record for Starlink 36609.

$ echo "FROM 'satcat.parquet'
        WHERE Name = 'Starlink 36609'
        LIMIT 1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "AltNames": null,
    "Apogee": 260,
    "Attach_flag": null,
    "Bus": "Starlink V2MO",
    "Coarse_type": "Payload (for orbital attempt)",
    "DDate": null,
    "Dest": null,
    "Diameter": 4.099999904632568,
    "DryMass": "530",
    "ID_flag": null,
    "Inc": 97.27999877929688,
    "JCAT": "S67582",
    "LDate": "2026 Jan 25",
    "Launch_Tag": "2026-016",
    "Length": 0.30000001192092896,
    "Manufacturer": "SPXS",
    "Mass": 575.0,
    "Motor": null,
    "Name": "Starlink 36609",
    "ODate": "2026 Jan 26",
    "OQUAL": null,
    "OpOrbit": "LLEO/S",
    "Orbit_flag": null,
    "Owner": "SPXS",
    "PLName": "Starlink Group 17-20-22",
    "Parent": "A11741",
    "Perigee": "247",
    "Piece": "2026-016X",
    "Primary": "Earth",
    "SDate": "2026 Jan 25 1832:44",
    "SDate_Year": 2026,
    "Satcat": "67582",
    "Shape": "Box + pan",
    "Span": 29.0,
    "State": "US",
    "Status": "O",
    "Status_or_failure_flag": "Satellite is active in operational orbit",
    "Subtype_flag": null,
    "TotMass": 575.0,
    "Type": "P      O"
  }
]

Below are the counts for each "start year" for Starlink satellites.

$ ~/duckdb

SELECT   SDate_Year,
         COUNT(*)
FROM     'satcat.parquet'
WHERE    Name LIKE 'Starlink%'
GROUP BY 1
ORDER BY 1;

┌────────────┬──────────────┐
│ SDate_Year │ count_star() │
│   int32    │    int64     │
├────────────┼──────────────┤
│       2019 │          128 │
│       2020 │          890 │
│       2021 │         1063 │
│       2022 │         1826 │
│       2023 │         2065 │
│       2024 │         1964 │
│       2025 │         3169 │
│       2026 │         1076 │
└────────────┴──────────────┘

Below is a breakdown of unique values and NULL coverage across each column for Starlink satellites.

SELECT   column_name,
         column_type[:30],
         null_percentage,
         approx_unique,
         min[:30],
         max[:30]
FROM     (SUMMARIZE
          FROM 'satcat.parquet'
          WHERE Name LIKE 'Starlink%')
ORDER BY LOWER(column_name);

┌────────────────────────┬──────────────────┬─────────────────┬───────────────┬────────────────────────────────┬────────────────────────────────┐
│      column_name       │ column_type[:30] │ null_percentage │ approx_unique │            min[:30]            │            max[:30]            │
│        varchar         │     varchar      │  decimal(9,2)   │     int64     │            varchar             │            varchar             │
├────────────────────────┼──────────────────┼─────────────────┼───────────────┼────────────────────────────────┼────────────────────────────────┤
│ AltNames               │ VARCHAR          │           99.98 │             3 │ 6                              │ VisorSat                       │
│ Apogee                 │ INTEGER          │            0.00 │           170 │ 232                            │ 588                            │
│ Attach_flag            │ VARCHAR          │          100.00 │             0 │ NULL                           │ NULL                           │
│ Bus                    │ VARCHAR          │            2.92 │             4 │ Starlink                       │ Starlink V2MO                  │
│ Coarse_type            │ VARCHAR          │            0.00 │             2 │ Component                      │ Payload (for orbital attempt)  │
│ DDate                  │ VARCHAR          │           84.60 │          1582 │ 2020 Apr  2 1100?              │ 2026 Mar 31 0937               │
│ Dest                   │ VARCHAR          │           92.28 │          1024 │ 0.5W 43.8S                     │ 9W 13S                         │
│ Diameter               │ FLOAT            │            0.00 │             6 │ 0.02                           │ 4.1                            │
│ DryMass                │ VARCHAR          │            0.00 │            16 │ 100                            │ 910                            │
│ ID_flag                │ VARCHAR          │           97.95 │             2 │ Starlink filing with FCC repor │ The association of this satell │
│ Inc                    │ FLOAT            │            0.00 │            37 │ 42.98                          │ 97.68                          │
│ JCAT                   │ VARCHAR          │            0.00 │         10039 │ S44235                         │ S68825                         │
│ Launch_Tag             │ VARCHAR          │            0.00 │           423 │ 2019-029                       │ 2026-089                       │
│ LDate                  │ VARCHAR          │            0.00 │           368 │ 2019 May 24                    │ 2026 Mar 30                    │
│ Length                 │ FLOAT            │            0.00 │             4 │ 0.2                            │ 7.0                            │
│ Manufacturer           │ VARCHAR          │            0.00 │             1 │ SPXS                           │ SPXS                           │
│ Mass                   │ FLOAT            │            0.00 │            15 │ 5.0                            │ 960.0                          │
│ Motor                  │ VARCHAR          │          100.00 │             0 │ NULL                           │ NULL                           │
│ Name                   │ VARCHAR          │            0.00 │         10070 │ Starlink 1007                  │ Starlink rideshare adapter     │
│ ODate                  │ VARCHAR          │            0.00 │           557 │ 2019 May 26                    │ 2026 Mar 31                    │
│ OpOrbit                │ VARCHAR          │            0.00 │             2 │ LLEO/I                         │ LLEO/S                         │
│ OQUAL                  │ VARCHAR          │           99.95 │             5 │ - s                            │ -x                             │
│ Orbit_flag             │ VARCHAR          │          100.00 │             0 │ NULL                           │ NULL                           │
│ Owner                  │ VARCHAR          │            0.00 │             1 │ SPXS                           │ SPXS                           │
│ Parent                 │ VARCHAR          │            0.00 │           368 │ A09431                         │ S67673                         │
│ Perigee                │ VARCHAR          │            0.00 │           189 │ 136                            │ 566                            │
│ Piece                  │ VARCHAR          │            0.00 │         12093 │ 2019-029A                      │ 2026-089Z                      │
│ PLName                 │ VARCHAR          │            2.87 │         12746 │ Starlink Group 10-1-01         │ adapter?                       │
│ Primary                │ VARCHAR          │            0.00 │             1 │ Earth                          │ Earth                          │
│ Satcat                 │ VARCHAR          │            0.00 │          9708 │ 44235                          │ NNA                            │
│ SDate                  │ VARCHAR          │            0.00 │           471 │ 2019 May 24 0332               │ 2026 Mar 30 2216:37            │
│ SDate_Year             │ INTEGER          │            0.00 │             9 │ 2019                           │ 2026                           │
│ Shape                  │ VARCHAR          │            0.00 │             4 │ Box + pan                      │ Plate                          │
│ Span                   │ FLOAT            │            0.00 │             5 │ 2.0                            │ 29.0                           │
│ State                  │ VARCHAR          │            0.00 │             1 │ US                             │ US                             │
│ Status                 │ VARCHAR          │            0.00 │             2 │ O                              │ R                              │
│ Status_or_failure_flag │ VARCHAR          │            2.92 │             9 │ Satellite active orbit lowerin │ Satellite removed slighly (and │
│ Subtype_flag           │ VARCHAR          │           97.08 │             1 │ Payload adapter, support struc │ Payload adapter, support struc │
│ TotMass                │ FLOAT            │            0.00 │            15 │ 5.0                            │ 960.0                          │
│ Type                   │ VARCHAR          │            0.00 │            63 │ C  A                           │ P      T r                     │
└────────────────────────┴──────────────────┴─────────────────┴───────────────┴────────────────────────────────┴────────────────────────────────┘

Below are the status counts for Starlink satellites.

SELECT   COUNT(*),
         Status_or_failure_flag
FROM     'satcat.parquet'
WHERE    Name LIKE 'Starlink%'
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ count_star() │                                                  Status_or_failure_flag                                                   │
│    int64     │                                                          varchar                                                          │
├──────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│         8629 │ Satellite is active in operational orbit                                                                                  │
│         1558 │ Satellite active orbit lowering to reentry                                                                                │
│          730 │ Satellite removed far from operational constellation by lowering or raising orbit                                         │
│          356 │ NULL                                                                                                                      │
│          292 │ Satellite ascending: orbit raising to op orbit (or lowering from high drift orbit)                                        │
│          227 │ Satellite in plane drift orbit                                                                                            │
│          158 │ Satellite failed in operational orbit and is undergoing uncontrolled decay - non manueverable                             │
│          131 │ Satellite failed early in mission, before reaching operational orbit (”screened”)                                         │
│          100 │ Satellite removed slighly (and possibly temporarily) from operational constellation by lowering or raising orbit a little │
└──────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Starlink satellites have a much more documented status than usual. Of the 68K+ records, almost 55K of them don't have any status or failure flag to speak of.

SELECT   COUNT(*),
         Status_or_failure_flag
FROM     'satcat.parquet'
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ count_star() │                                                  Status_or_failure_flag                                                   │
│    int64     │                                                          varchar                                                          │
├──────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│        54843 │ NULL                                                                                                                      │
│        10168 │ Satellite is active in operational orbit                                                                                  │
│         1634 │ Satellite active orbit lowering to reentry                                                                                │
│          735 │ Satellite removed far from operational constellation by lowering or raising orbit                                         │
│          392 │ Satellite in plane drift orbit                                                                                            │
│          344 │ Satellite ascending: orbit raising to op orbit (or lowering from high drift orbit)                                        │
│          238 │ Satellite failed in operational orbit and is undergoing uncontrolled decay - non manueverable                             │
│          165 │ Satellite retired to a graveyard orbit of some kind                                                                       │
│          144 │ Satellite failed early in mission, before reaching operational orbit (”screened”)                                         │
│          136 │ Satellite removed slighly (and possibly temporarily) from operational constellation by lowering or raising orbit a little │
│           33 │ This object was the one that failed during launch.                                                                        │
│            2 │ Satellite was used for special tests outside of main constellation                                                        │
└──────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Below are the Starlink bus counts.

SELECT   Bus,
         COUNT(*)
FROM     'satcat.parquet'
WHERE    Name LIKE 'Starlink%'
GROUP BY 1;

┌───────────────┬──────────────┐
│      Bus      │ count_star() │
│    varchar    │    int64     │
├───────────────┼──────────────┤
│ Starlink V2MD │          663 │
│ NULL          │          356 │
│ Starlink V2M  │         2760 │
│ Starlink V2MO │         3722 │
│ Starlink      │         4680 │
└───────────────┴──────────────┘

Below are the Starlink operational orbit counts.

SELECT   OpOrbit,
         COUNT(*)
FROM     'satcat.parquet'
WHERE    Name LIKE 'Starlink%'
GROUP BY 1;

┌─────────┬──────────────┐
│ OpOrbit │ count_star() │
│ varchar │    int64     │
├─────────┼──────────────┤
│ LLEO/I  │        11110 │
│ LLEO/S  │         1071 │
└─────────┴──────────────┘

The Type column in this dataset is where a lot of interesting data is baked away. I still have work to do on decoding each of the 12-character strings. If you're interested in working on this, page 34 onward of GCAT's documentation describes the Type column in detail.

Reverse Geocoding with Overture Maps

2026-04-28T20:00:00+03:00

I'm working on a reverse geocoder that uses Overture Maps' datasets. The prototype in this post can fetch the 2-letter ISO country code for any given point within a country's boundaries, return the nearest address and list countries within a given polygon.

My aim is to avoid paying API fees or making any network requests outside of the initial downloads of Overture's data.

In this post, I'll walk through this prototype and reverse geocode some satellite footprints.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use Python 3.12.3 and jq to help analyse the data in this post.

$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt update
$ sudo apt install \
    jq \
    python3-pip \
    python3.12-venv

Below, I'll set up a Python Virtual Environment and install some dependencies.

$ python3 -m venv ~/.overture_rcm
$ source ~/.overture_rcm/bin/activate
$ python3 -m pip install \
    awscli \
    duckdb \
    rich

I'll use DuckDB, along with its H3, JSON, Lindel, Parquet and Spatial extensions in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.5.1/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

I used QGIS' HCMGIS plugin to add basemaps from Bing and Esri to this post.

Overture's Divisions and Addresses

I did a deep dive comparing Overture's divisions with eleven other boundary datasets in December. Overture is tied in third place for most ISO country codes at 272 and has well-laid-out boundaries along coastlines.

These are their boundaries for Iceland. It's coloured by the boundary type with light blue for their land boundaries.

Most competing datasets did a poor job of hugging coastlines. Below are several examples for Bahrain.

I'll get this month's divisions dataset. The following downloaded 5.2 GB worth of Parquet files.

$ aws s3 --no-sign-request sync \
    s3://overturemaps-us-west-2/release/2026-04-15.0/theme=divisions/type=division_area/ \
    ~/division_area_2026-04-15.0

Overture's address dataset doesn't cover the globe but does have coverage for 39 countries as of this month's release. The following downloaded 21 GB worth of data.

$ aws s3 --no-sign-request sync \
    s3://overturemaps-us-west-2/release/2026-04-15.0/theme=addresses/type=address/ \
    ~/address_2026-04-15.0

Overture removes releases after 60 days. Use the following command to find out what the latest release is and replace the release string in the above URLs before trying to download anything.

$ aws s3 --no-sign-request ls \
    s3://overturemaps-us-west-2/release/

PRE 2026-03-18.0/
PRE 2026-04-15.0/

Reverse Geocoding in Python

I've put together three reverse geocoding functions.

The first takes a longitude and latitude pair and returns the nearest division's two-letter ISO country code. The list returned will contain an element that has a two-letter code. I'll likely refactor this to just return either a string in the future.

$ python3

import json

import duckdb
from   rich.progress import track


divisions = '/home/mark/division_area_2026-04-15.0/*.parquet'

con = duckdb.connect(database=':memory:')
con.sql('INSTALL spatial; LOAD spatial')

def nearest_iso2(lon:float, lat:float):
    fields = (
        'bbox',
        'class',
        'country',
        'division_id',
        'id',
        'is_land',
        'is_territorial',
        'names',
        'region',
        'sources',
        'subtype',
        'version')

    sql = '''SELECT %s
             FROM  '%s'
             ORDER BY ST_DISTANCE(
                        ST_POINT(bbox.xmin, bbox.ymin),
                        ST_POINT(%f, %f))
             LIMIT 1'''

    res = list(con.sql(sql % (','.join(fields),
                              divisions,
                              lon,
                              lat)).fetchall())

    return [{fields[num]: val
             for num, val in enumerate(rec)}
            for rec in res]

Below is an example result from a point in Tallinn, Estonia.

nearest_iso2(24.75294, 59.43412)

[{'bbox': {'xmin': 24.752965927124023,
   'xmax': 24.754074096679688,
   'ymin': 59.432708740234375,
   'ymax': 59.433204650878906},
  'class': 'land',
  'country': 'EE',
  'division_id': '18916d55-9bf3-4758-9691-83a462ec8204',
  'id': '89ee8bb5-8912-4d39-a8b9-487193cc9653',
  'is_land': True,
  'is_territorial': True,
  'names': {'primary': 'Islandi väljak', 'common': None, 'rules': None},
  'region': 'EE-37',
  'sources': [{'property': '',
    'dataset': 'OpenStreetMap',
    'license': 'ODbL-1.0',
    'record_id': 'w43735045@7',
    'update_time': '2024-02-14T15:13:23Z',
    'confidence': None,
    'between': None}],
  'subtype': 'microhood',
  'version': 1}]

The following will return a list of any country whose borders intersect a given polygon.

def countries_in_polygon(polygon:str):
    fields = (
        'id',
        'bbox',
        'country',
        'version',
        'sources',
        'subtype',
        'class',
        'names',
        'is_land',
        'is_territorial',
        'region',
        'division_id')

    sql = '''SELECT %s
             FROM '%s'
             WHERE country IN (
                SELECT DISTINCT country
                 FROM  '%s'
                 WHERE ST_Intersects(ST_POINT(bbox.xmin, bbox.ymin),
                                     '%s'::GEOMETRY)
                 LIMIT 100
             )
             AND subtype = 'country'
             AND class   = 'land'
             '''

    res = list(con.sql(sql % (','.join(fields),
                              divisions,
                              divisions,
                              polygon)).fetchall())

    return [{fields[num]: val
             for num, val in enumerate(rec)}
            for rec in res]

I've generated an example polygon as shown below.

The following are the resulting matches.

res = countries_in_polygon('POLYGON ((41.32741049318353 12.40313010412833, 44.00517608732055 13.780566115995342, 44.79343608905786 12.53895263826337, 42.486616378091334 10.631257051879393, 41.32741049318353 12.40313010412833))')

print([(x['country'],
        x['names']['common']['en'])
       for x in res])

[('YE', 'Yemen'), ('ER', 'Eritrea'), ('DJ', 'Djibouti'), ('ET', 'Ethiopia')]

I'll need to use polygons for intersecting testing in the future as Somalia is missing from the results.

Below is a function that will return the nearest address to a given point. If the country is known ahead of time, this can be included as a predicate and will help minimise the search space. I've also added ranges around the longitude and latitude to help speed up the query. This is unlikely to be the fastest way to do a lookup and is still a work in progress.

addresses = '/home/mark/address_2026-04-15.0/*.parquet'


def nearest_address(lon:float, lat:float, countries:list=None):
    where = '''WHERE bbox.xmin > %f AND bbox.xmin < %f
               AND   bbox.ymin > %f AND bbox.ymin < %f''' % (
                    lon - 1, lon + 1,
                    lat - 1, lat + 1)

    if countries:
        where = where + ' AND country IN (%s)' % \
                        ','.join("'%s'" % x.upper()
                                 for x  in countries)

    fields = (
        'id',
        'street',
        'number',
        'unit',
        'postcode',
        'postal_city',
        'address_levels',
        'country',
        'sources',
        'version',
        'bbox',)

    sql = '''SELECT %s
             FROM  '%s'
             %s
             ORDER BY ST_DISTANCE(
                        ST_POINT(bbox.xmin, bbox.ymin),
                        ST_POINT(%f, %f))
             LIMIT 1'''

    res = list(con.sql(sql % (','.join(fields),
                              addresses,
                              where,
                              lon,
                              lat)).fetchall())

    return [{fields[num]: val
             for num, val in enumerate(rec)}
            for rec in res]

Below is an example from a point in Tallinn, Estonia.

nearest_address(24.75294, 59.43412, ['EE'])

[{'id': 'd054bf1b-05d7-4ebd-8e65-69aca6ea41fb',
  'street': 'Estonia puiestee',
  'number': '7',
  'unit': None,
  'postcode': None,
  'postal_city': None,
  'address_levels': [{'value': 'Harju maakond'},
   {'value': 'Tallinn'},
   {'value': 'Kesklinna linnaosa'}],
  'country': 'EE',
  'sources': [{'property': '',
    'dataset': 'OpenAddresses/MAA-AMET',
    'license': None,
    'record_id': None,
    'update_time': None,
    'confidence': None,
    'between': None}],
  'version': 1,
  'bbox': {'xmin': 24.752660751342773,
   'xmax': 24.75266456604004,
   'ymin': 59.434288024902344,
   'ymax': 59.434295654296875}}]

Canada's RADARSAT

Canada has been running a satellite programme called RADARSAT since the 1990s. The first constellation was launched in 1995, the second in 2007 and the third (RCM) in 2019.

The three satellites in the RCM constellation capture Synthetic Aperture Radar (SAR) imagery as well as AIS tracking data from ships. SAR allows for imagery to be taken day or night and can see through clouds and through some types of camouflage.

Below is a screenshot of a 96 GB, Cloud-Optimised GeoTIFF mosaic of RCM's imagery of Canada.

Below, I've zoomed into Calgary.

RADARSAT's Acquisition Plans

Natural Resources Canada publishes its RCM imagery capture plans every two weeks.

Below I've downloaded their plan for March 11th - 25th. Note, they don't version these files so the file you'd download will likely differ from the one I downloaded a few weeks ago.

$ wget https://ftp.maps.canada.ca/pub/csa_asc/Space-technology_Technologie-spatiale/radarsat_constellation_mission_plan/radarsat_constellation_mission_planned.shp.zip

$ unzip -j \
    -d radarsat_constellation_mission_planned \
    radarsat_constellation_mission_planned.shp.zip

I've cleaned up and converted their dataset into a Parquet file. There isn't a primary key in this dataset so I've added a UUID column. This will be used to join against other datasets later on in this post.

$ ~/duckdb

COPY(
    SELECT   baq:       BAQ,
             beam_id:   BEAMID,
             beam_type: BEAMTYPE_E,
             ccd:       CCD_E::BOOL,
             ccd_exact: EXACTCCD_E::BOOL,
             pol_type:  POLTYPE_E,
             prod_type: PRODTYPE_E,
             radar_md:  RADARMD,
             rx_pol:    RXPOL,
             sat_id:    SATID,
             tx_pol:    TXPOL,
             start_at:  UTC_STRT::TIMESTAMP,
             end_at:    UTC_END::TIMESTAMP,
             geometry:  geom,
             bbox:      {'xmin': ST_XMIN(ST_EXTENT(geom)),
                         'ymin': ST_YMIN(ST_EXTENT(geom)),
                         'xmax': ST_XMAX(ST_EXTENT(geom)),
                         'ymax': ST_YMAX(ST_EXTENT(geom))},
             uuid:      UUID()
    FROM     ST_READ('radarsat_constellation_mission_planned/radarsat_constellation_mission_planned.shx')
    ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geom)),
                             ST_X(ST_CENTROID(geom))]::double[2])
) TO 'planned.parquet' (
  FORMAT 'PARQUET',
  CODEC  'ZSTD',
  COMPRESSION_LEVEL 22,
  ROW_GROUP_SIZE 15000);

The above produced a 3.1 MB Parquet file containing 4,911 rows.

Below is a breakdown of unique values and NULL coverage across each column.

SELECT   column_name,
         column_type[:30],
         null_percentage,
         approx_unique,
         min[:30],
         max[:30]
FROM     (SUMMARIZE
          FROM 'planned.parquet')
ORDER BY LOWER(column_name);

┌─────────────┬────────────────────────────────┬─────────────────┬───────────────┬────────────────────────────────┬────────────────────────────────┐
│ column_name │        column_type[:30]        │ null_percentage │ approx_unique │            min[:30]            │            max[:30]            │
│   varchar   │            varchar             │  decimal(9,2)   │     int64     │            varchar             │            varchar             │
├─────────────┼────────────────────────────────┼─────────────────┼───────────────┼────────────────────────────────┼────────────────────────────────┤
│ baq         │ VARCHAR                        │            0.00 │             3 │ 1 bit                          │ 3 bit                          │
│ bbox        │ STRUCT(xmin DOUBLE, ymin DOUBL │            0.00 │          5222 │ {'xmin': -180.0, 'ymin': -16.8 │ {'xmin': 177.82678, 'ymin': -1 │
│ beam_id     │ VARCHAR                        │            0.00 │           133 │ 16M10                          │ SCSDA                          │
│ beam_type   │ VARCHAR                        │            0.00 │            10 │ High Resolution 5m             │ Very High Resolution 3m        │
│ ccd         │ BOOLEAN                        │            0.00 │             2 │ false                          │ true                           │
│ ccd_exact   │ BOOLEAN                        │            0.00 │             2 │ false                          │ true                           │
│ end_at      │ TIMESTAMP                      │            0.00 │          4648 │ 2026-03-11 03:11:52            │ 2026-03-25 23:55:24            │
│ geometry    │ GEOMETRY('EPSG:4326')          │            0.00 │          4866 │ POLYGON ((-47.79754 44.06532,  │ MULTIPOLYGON (((-165.64809 74. │
│ pol_type    │ VARCHAR                        │            0.00 │             5 │ Compact Polarization           │ Single Polarization            │
│ prod_type   │ VARCHAR                        │            0.00 │             6 │ GRD - 16bit                    │ null - 32bit                   │
│ radar_md    │ VARCHAR                        │            0.00 │             3 │ ScanSAR                        │ Stripmap Continuous            │
│ rx_pol      │ VARCHAR                        │            0.00 │             3 │ H                              │ V                              │
│ sat_id      │ VARCHAR                        │            0.00 │             3 │ RCM-1                          │ RCM-3                          │
│ start_at    │ TIMESTAMP                      │            0.00 │          4306 │ 2026-03-11 03:09:27            │ 2026-03-25 23:55:17            │
│ tx_pol      │ VARCHAR                        │            0.00 │             4 │ C                              │ V                              │
│ uuid        │ UUID                           │            0.00 │          4543 │ 0017484c-9210-4479-935c-dec8bf │ fffe46dc-6588-429f-9787-eacafa │
└─────────────┴────────────────────────────────┴─────────────────┴───────────────┴────────────────────────────────┴────────────────────────────────┘

This is a rendering of the above dataset's footprints.

Below is an animation of the imagery capture plan. I've coloured every footprint by its date and used the temporal functionality in QGIS to render them by the hour they're captured.

Footprint Enrichment

I'll take every footprint in the RCM capture plan and run them through the reverse geocoding functions. The following produced a 72 MB, 4,911-line JSON file.

sql = '''SELECT uuid,
                geometry::TEXT,
                ST_X(ST_CENTROID(geometry)),
                ST_Y(ST_CENTROID(geometry)),
         FROM   'planned.parquet'
      '''

with open('planned.enriched.json', 'w') as f:
    for (uuid_, wkt, lon, lat) in track(list(con.sql(sql).fetchall())):
        nearest_iso2_ = nearest_iso2(lon, lat)
        countries_in_polygon_ = countries_in_polygon(wkt)

        f.write(json.dumps({
                    'uuid':                 uuid_,
                    'nearest_iso2':         nearest_iso2_,
                    'countries_in_polygon': countries_in_polygon_,
                    'nearest_address':
                        nearest_address(lon,
                                        lat,
                                        [x['country']
                                         for x in countries_in_polygon_])
                    },
                    sort_keys=True,
                    default=str) + '\n')

The above took a few hours to complete. I was hoping it would take a few minutes. I'm going to have to look into optimising this further in order to reach that performance target.

I'll create an enriched Parquet file that joins the above lookups with the acquisition plan dataset.

$ ~/duckdb

COPY(
    SELECT   a.*,
             b.* EXCLUDE(uuid),
             nearest_country: b.nearest_iso2[1].country
    FROM     'planned.parquet' a
    JOIN     'planned.enriched.json' b ON a.uuid = b.uuid
) TO 'planned.enriched.parquet' (
  FORMAT 'PARQUET',
  CODEC  'ZSTD',
  COMPRESSION_LEVEL 22,
  ROW_GROUP_SIZE 15000);

The above produced a 3.7 MB Parquet file.

Below is an example record. I've truncated the names and sources lists as they're quite lengthy.

$ echo "FROM 'planned.enriched.parquet'
        LIMIT 1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "baq": "3 bit",
    "bbox": {
      "xmax": -112.45445,
      "xmin": -116.8121,
      "ymax": 43.57581,
      "ymin": 42.57175
    },
    "beam_id": "SCLNB",
    "beam_type": "Low Noise",
    "ccd": "false",
    "ccd_exact": "false",
    "countries_in_polygon": [
      {
        "bbox": {
          "xmax": 179.7726287841797,
          "xmin": -179.14785766601562,
          "ymax": 71.38683319091797,
          "ymin": 18.91116714477539
        },
        "class": "land",
        "country": "US",
        "division_id": "f39eb4af-5206-481b-b19e-bd784ded3f05",
        "id": "79d2f02b-59b9-4981-8c88-43f07431788b",
        "is_land": true,
        "is_territorial": false,
        "names": {
          "common": {
            "en": "United States",
            ...
          },
          "primary": "United States of America",
          "rules": [...]
        },
        "region": null,
        "sources": [...],
        "subtype": "country",
        "version": 4
      }
    ],
    "end_at": "2026-03-11 13:46:18",
    "geometry": "POLYGON ((-112.45445 43.04819, -112.46727 43.00002, -112.48336 42.93853, -112.49843 42.87689, -112.51336 42.81523, -112.52955 42.75376, -112.54715 42.69249, -112.5631 42.63098, -112.57883 42.57175, -113.98181 42.76447, -115.39313 42.9399, -116.8121 43.09782, -116.80517 43.15753, -116.79738 43.2194, -116.78634 43.28094, -116.77776 43.34273, -116.76946 43.40456, -116.76037 43.46629, -116.74823 43.52771, -116.7382 43.57581, -115.30216 43.41781, -113.87399 43.24186, -112.45445 43.04819))",
    "nearest_address": [
      {
        "address_levels": [
          {
            "value": "ID"
          },
          {
            "value": null
          }
        ],
        "bbox": {
          "xmax": -114.61167907714844,
          "xmin": -114.6116943359375,
          "ymax": 43.03421401977539,
          "ymin": 43.03420639038086
        },
        "country": "US",
        "id": "fa6e6168-4ed2-4207-b268-7629f86f5e6e",
        "number": "2414",
        "postal_city": null,
        "postcode": "83314",
        "sources": [
          {
            "between": null,
            "confidence": null,
            "dataset": "NAD",
            "license": null,
            "property": "",
            "record_id": null,
            "update_time": null
          }
        ],
        "street": "East 1150 South",
        "unit": null,
        "version": 1
      }
    ],
    "nearest_country": "US",
    "nearest_iso2": [
      {
        "bbox": {
          "xmax": -114.69801330566406,
          "xmin": -114.7276382446289,
          "ymax": 42.94951629638672,
          "ymin": 42.91918182373047
        },
        "class": "land",
        "country": "US",
        "division_id": "2f9add0b-97fb-45cd-844e-017882674973",
        "id": "39c72352-3cdc-4bec-8221-67d5e347a2ca",
        "is_land": true,
        "is_territorial": true,
        "names": {
          "common": null,
          "primary": "Gooding",
          "rules": null
        },
        "region": "US-ID",
        "sources": [
          {
            "between": null,
            "confidence": null,
            "dataset": "OpenStreetMap",
            "license": "ODbL-1.0",
            "property": "",
            "record_id": "r121376@4",
            "update_time": "2020-12-11 20:02:43"
          }
        ],
        "subtype": "locality",
        "version": 1
      }
    ],
    "pol_type": "Dual Co/Cross Polarization",
    "prod_type": "GRD - 16bit",
    "radar_md": "ScanSAR",
    "rx_pol": "H+V",
    "sat_id": "RCM-3",
    "start_at": "2026-03-11 13:46:08",
    "tx_pol": "H",
    "uuid": "03e63006-456d-4fc3-92eb-40b0ee1535c6"
  }
]

Below are some of the planned footprints between Greenland and Scandinavia. I've labelled each footprint with the nearest ISO2 country code in Overture's dataset. This image was taken while the above enrichment job was still running so there aren't as many footprints as in the final dataset.

A number of the footprints were over open oceans and wouldn't intersect with any country's boundaries. Below is a breakdown of the footprints with at least one country intersecting its footprint.

SELECT   has_intersection: LENGTH(countries_in_polygon) > 0,
         COUNT(*)
FROM     'planned.enriched.parquet'
GROUP BY 1;

┌──────────────────┬──────────────┐
│ has_intersection │ count_star() │
│     boolean      │    int64     │
├──────────────────┼──────────────┤
│ true             │         1794 │
│ false            │         3117 │
└──────────────────┴──────────────┘

If a footprint's nearest country isn't supported by Overture's address dataset and/or there are no addresses +/- 1 degree of latitude or longitude from the point given, then no address will be returned. Below is the breakdown of how many footprints have a nearest address.

SELECT   has_address: LENGTH(nearest_address) > 0,
         COUNT(*)
FROM     'planned.enriched.parquet'
GROUP BY 1;

┌─────────────┬──────────────┐
│ has_address │ count_star() │
│   boolean   │    int64     │
├─────────────┼──────────────┤
│ false       │         4032 │
│ true        │          879 │
└─────────────┴──────────────┘

Below are the footprint's most common nearest countries.

CREATE OR REPLACE TABLE country_names AS
    WITH b AS (
        WITH a AS (
            SELECT   country,
                     names.common.en,
                     area: ST_AREA(geometry)
            FROM     '/home/mark/division_area_2026-04-15.0/*.parquet'
            WHERE    names.common.en IS NOT NULL
        )
        SELECT *,
               ROW_NUMBER() OVER (PARTITION BY country
                                  ORDER BY     area DESC) AS rn
        FROM   a
    )
    FROM     b
    WHERE    rn = 1
    ORDER BY area DESC;

SELECT    num_footprints: COUNT(*),
          nearest_country,
          country_name: SPLIT_PART(en, '/', 1)
FROM      'planned.enriched.parquet'
LEFT JOIN country_names ON (nearest_country = country)
GROUP BY  2, 3
ORDER BY  1 DESC
LIMIT     20;

┌────────────────┬─────────────────┬──────────────────────────────────────────┐
│ num_footprints │ nearest_country │               country_name               │
│     int64      │     varchar     │                 varchar                  │
├────────────────┼─────────────────┼──────────────────────────────────────────┤
│           1868 │ CA              │ Canada                                   │
│            903 │ RU              │ Russia                                   │
│            589 │ US              │ United States                            │
│            405 │ GL              │ Greenland                                │
│            181 │ XJ              │ Jan Mayen                                │
│            102 │ SJ              │ Svalbard                                 │
│             98 │ IS              │ Iceland                                  │
│             69 │ FM              │ Federated States of Micronesia           │
│             61 │ CL              │ Chile                                    │
│             59 │ EC              │ Ecuador                                  │
│             47 │ NO              │ Norway                                   │
│             46 │ JP              │ Japan                                    │
│             45 │ AU              │ Australia                                │
│             42 │ PG              │ Papua New Guinea                         │
│             32 │ CO              │ Colombia                                 │
│             32 │ GS              │ South Georgia And South Sandwich Islands │
│             30 │ NC              │ New Caledonia                            │
│             29 │ SB              │ Solomon Islands                          │
│             27 │ CP              │ Clipperton Island                        │
│             24 │ PH              │ Philippines                              │
└────────────────┴─────────────────┴──────────────────────────────────────────┘

These are all of the 4,911 footprints.

3.4M Solar Panels

2026-04-22T20:00:00+03:00

In October, I reviewed the Ground-Mounted Solar Energy in the United States (GM-SEUS) dataset. This dataset attempted to outline the majority of solar farm arrays and panels across the US. Version 1 of this dataset contained 2.9M panels.

On Monday, version 2 of this dataset was released and now contains more than 3.4M panels. In addition to the panels and arrays being refreshed, there is a new rooftop array dataset.

In this post, I'll review v2 of the GM-SEUS dataset.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use GDAL 3.9.3 to help analyse the data in this post.

$ sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable
$ sudo apt update
$ sudo apt install \
    gdal-bin

I'll use DuckDB, along with its H3, JSON, Lindel, Parquet and Spatial extensions in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.5.1/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

I used QGIS' HCMGIS plugin to add basemaps from Esri to this post.

Analysis-Ready Datasets

The following will download a 3.4 GB ZIP file. I'll extract any GeoPackage (GPKG) file from it.

$ wget -O GMSEUS_v2.zip \
    'https://zenodo.org/records/19581821/files/GMSEUS.zip?download=1'
$ unzip -j GMSEUS_v2.zip "*.gpkg"

The following is the projection used by the GPKG files in this dataset.

$ gdalsrsinfo \
    -o proj4 \
    GMSEUS_RooftopArrays_2025_v2_0.gpkg

+proj=aea +lat_0=23 +lon_0=-96 +lat_1=29.5 +lat_2=45.5 +x_0=0 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs

The following will convert the rooftop arrays into Parquet. I needed to use DuckDB v1.4.4 for the following as v1.5.1 was raising exceptions.

$ ~/duckdb

COPY (
   WITH a AS (
       SELECT Source,
              grndCvr,
              modType,
              mount,
              nativeID,
              roofArrID,
              area: IF(area::TEXT='-9999.0', NULL, area::DOUBLE),
              azimuth: IF(azimuth::TEXT='-9999.0', NULL, azimuth::DOUBLE),
              capMWAC: IF(capMWAC::TEXT='-9999.0', NULL, capMWAC::DOUBLE),
              capMWDC: IF(capMWDC::TEXT='-9999.0', NULL, capMWDC::DOUBLE),
              tilt: IF(tilt::TEXT='-9999.0', NULL, tilt::DOUBLE),
              instYr: CASE WHEN instYr::INT = -9999 THEN NULL ELSE instYr     END,
              ST_FLIPCOORDINATES(
                  ST_TRANSFORM(
                      geom,
                      '+proj=aea +lat_0=23 +lon_0=-96 +lat_1=29.5 +lat_2=45.5 +x_0=0 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs',
                      'EPSG:4326')) geometry
       FROM ST_READ('GMSEUS_RooftopArrays_2025_v2_0.gpkg')
   )
   SELECT   * EXCLUDE (geometry),
            {'xmin': ST_XMIN(ST_EXTENT(geometry)),
             'ymin': ST_YMIN(ST_EXTENT(geometry)),
             'xmax': ST_XMAX(ST_EXTENT(geometry)),
             'ymax': ST_YMAX(ST_EXTENT(geometry))} AS bbox,
             ST_ASWKB(geometry) geometry
   FROM     a
   ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geometry)),
                            ST_X(ST_CENTROID(geometry))]::double[2])
) TO 'GMSEUS_RooftopArrays_2025_v2_0.parquet' (
   FORMAT            'PARQUET',
   CODEC             'ZSTD',
   COMPRESSION_LEVEL 22,
   ROW_GROUP_SIZE    15000);

There are 5,822 records in this dataset.

SELECT COUNT(*)
FROM   'GMSEUS_RooftopArrays_2025_v2_0.parquet';

5,822

Below is a breakdown of unique values and NULL coverage across each column.

SELECT   column_name,
         column_type,
         null_percentage,
         approx_unique,
         min,
         max
FROM     (SUMMARIZE
          FROM READ_PARQUET('GMSEUS_RooftopArrays_2025_v2_0.parquet'))
WHERE    column_name != 'geometry'
AND      column_name != 'bbox'
ORDER BY LOWER(column_name);

┌─────────────┬─────────────┬─────────────────┬───────────────┬────────────┬────────────────────┐
│ column_name │ column_type │ null_percentage │ approx_unique │    min     │        max         │
│   varchar   │   varchar   │  decimal(9,2)   │     int64     │  varchar   │      varchar       │
├─────────────┼─────────────┼─────────────────┼───────────────┼────────────┼────────────────────┤
│ area        │ DOUBLE      │            2.77 │          5180 │ 15.0       │ 487111.0           │
│ azimuth     │ DOUBLE      │           89.63 │           156 │ 0.0        │ 530.02323408881    │
│ capMWAC     │ DOUBLE      │           89.52 │            60 │ 0.2        │ 74.9               │
│ capMWDC     │ DOUBLE      │           87.12 │           166 │ 0.00448    │ 99.7               │
│ grndCvr     │ VARCHAR     │           97.61 │             2 │ impervious │ vegetation         │
│ instYr      │ BIGINT      │           72.43 │            23 │ 2003       │ 2025               │
│ modType     │ VARCHAR     │            0.00 │             2 │ c-si       │ thin-film          │
│ mount       │ VARCHAR     │           87.53 │             5 │ dual_axis  │ unknown            │
│ nativeID    │ VARCHAR     │            0.00 │          4540 │ 1          │ Xebec 1 solar farm │
│ roofArrID   │ BIGINT      │            0.00 │          5830 │ 1          │ 5822               │
│ Source      │ VARCHAR     │            0.00 │            15 │ CCVPV      │ gspt               │
│ tilt        │ DOUBLE      │           90.64 │            31 │ 0.0        │ 52.0               │
├─────────────┴─────────────┴─────────────────┴───────────────┴────────────┴────────────────────┤
│ 12 rows                                                                             6 columns │
└───────────────────────────────────────────────────────────────────────────────────────────────┘

The following will convert the panels into Parquet.

COPY (
   WITH a AS (
       SELECT Source,
              arrayID: arrayID::INT,
              panelID: panelID::INT,
              pnlSource,
              rowArea,
              rowAzimuth,
              rowLength,
              rowMount,
              rowSpace: IF(rowSpace::TEXT='-9999.0', NULL, rowSpace::DOUBLE),
              rowWidth,
              ST_FLIPCOORDINATES(
                  ST_TRANSFORM(
                      geom,
                      '+proj=aea +lat_0=23 +lon_0=-96 +lat_1=29.5 +lat_2=45.5 +x_0=0 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs',
                      'EPSG:4326')) geometry
       FROM ST_READ('GMSEUS_Panels_Final_2025_v2_0.gpkg')
   )
   SELECT   * EXCLUDE (geometry),
            {'xmin': ST_XMIN(ST_EXTENT(geometry)),
             'ymin': ST_YMIN(ST_EXTENT(geometry)),
             'xmax': ST_XMAX(ST_EXTENT(geometry)),
             'ymax': ST_YMAX(ST_EXTENT(geometry))} AS bbox,
             ST_ASWKB(geometry) geometry
   FROM     a
   ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geometry)),
                            ST_X(ST_CENTROID(geometry))]::double[2])
) TO 'GMSEUS_Panels_Final_2025_v2_0.parquet' (
   FORMAT            'PARQUET',
   CODEC             'ZSTD',
   COMPRESSION_LEVEL 22,
   ROW_GROUP_SIZE    15000);

There are 3,429,157 records in this dataset.

SELECT COUNT(*)
FROM   'GMSEUS_Panels_Final_2025_v2_0.parquet';

3,429,157

Below is a breakdown of unique values and NULL coverage across each column.

SELECT   column_name,
         column_type,
         null_percentage,
         approx_unique,
         min,
         max
FROM     (SUMMARIZE
          FROM READ_PARQUET('GMSEUS_Panels_Final_2025_v2_0.parquet'))
WHERE    column_name != 'geometry'
AND      column_name != 'bbox'
ORDER BY LOWER(column_name);

┌─────────────┬─────────────┬─────────────────┬───────────────┬───────────┬─────────────┐
│ column_name │ column_type │ null_percentage │ approx_unique │    min    │     max     │
│   varchar   │   varchar   │  decimal(9,2)   │     int64     │  varchar  │   varchar   │
├─────────────┼─────────────┼─────────────────┼───────────────┼───────────┼─────────────┤
│ arrayID     │ INTEGER     │            0.03 │         12653 │ 1         │ 18980       │
│ panelID     │ INTEGER     │            0.00 │       3323765 │ 1         │ 3429157     │
│ pnlSource   │ VARCHAR     │            0.00 │             5 │ CCVPV     │ OSM         │
│ rowArea     │ DOUBLE      │            0.00 │        100105 │ 15.01     │ 9982.68     │
│ rowAzimuth  │ DOUBLE      │            0.00 │         22029 │ 90.0      │ 540.0       │
│ rowLength   │ DOUBLE      │            0.00 │         25531 │ 3.96      │ 737.38      │
│ rowMount    │ VARCHAR     │            0.00 │             3 │ dual_axis │ single_axis │
│ rowSpace    │ DOUBLE      │            1.27 │          1836 │ 0.01      │ 20.0        │
│ rowWidth    │ DOUBLE      │            0.00 │          2258 │ 0.45      │ 135.33      │
│ Source      │ VARCHAR     │            0.00 │            12 │ CCVPV     │ USPVDB      │
├─────────────┴─────────────┴─────────────────┴───────────────┴───────────┴─────────────┤
│ 10 rows                                                                     6 columns │
└───────────────────────────────────────────────────────────────────────────────────────┘

The following will convert the arrays and panels datasets into Parquet.

COPY (
   WITH a AS (
       SELECT COUNTYFP,
              GCR1,
              GCR2,
              STATEFP,
              Source,
              arrayID,
              avgAzimuth: IF(avgAzimuth::TEXT='-9999.0', NULL, avgAzimuth::DOUBLE),
              avgLength: IF(avgLength::TEXT='-9999.0', NULL, avgLength::DOUBLE),
              avgSpace: IF(avgSpace::TEXT='-9999.0', NULL, avgSpace::DOUBLE),
              avgWidth: IF(avgWidth::TEXT='-9999.0', NULL, avgWidth::DOUBLE),
              capMWAC,
              capMWACest,
              capMWDC,
              capMWDCest,
              effInit: IF(effInit::TEXT='-9999.0', NULL, effInit::DOUBLE),
              grndCvr,
              instYr: CASE WHEN instYr::INT = -9999 THEN NULL ELSE instYr END,
              instYrEst: CASE WHEN instYrEst::INT = -9999 THEN NULL ELSE instYrEst END,
              modType,
              mount,
              nativeID: IF(nativeID::TEXT='unknown', NULL, nativeID),
              newBound,
              numRow,
              tilt: IF(tilt::TEXT='-9999.0', NULL, tilt::DOUBLE),
              tiltEst: CASE WHEN tiltEst::INT = -9999 THEN NULL ELSE tiltEst END,
              totArea,
              totRowArea,
              ST_FLIPCOORDINATES(
                  ST_TRANSFORM(
                      geom,
                      '+proj=aea +lat_0=23 +lon_0=-96 +lat_1=29.5 +lat_2=45.5 +x_0=0 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs',
                      'EPSG:4326')) geometry
       FROM ST_READ('GMSEUS_Arrays_Final_2025_v2_0.gpkg')
   )
   SELECT   * EXCLUDE (geometry),
            {'xmin': ST_XMIN(ST_EXTENT(geometry)),
             'ymin': ST_YMIN(ST_EXTENT(geometry)),
             'xmax': ST_XMAX(ST_EXTENT(geometry)),
             'ymax': ST_YMAX(ST_EXTENT(geometry))} AS bbox,
             ST_ASWKB(geometry) geometry
   FROM     a
   ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geometry)),
                            ST_X(ST_CENTROID(geometry))]::double[2])
) TO 'GMSEUS_Arrays_Final_2025_v2_0.parquet' (
   FORMAT            'PARQUET',
   CODEC             'ZSTD',
   COMPRESSION_LEVEL 22,
   ROW_GROUP_SIZE    15000);

There are 18,980 records in this dataset.

SELECT COUNT(*)
FROM   'GMSEUS_Arrays_Final_2025_v2_0.parquet';

18,980

Below is a breakdown of unique values and NULL coverage across each column.

SELECT   column_name,
         column_type,
         null_percentage,
         approx_unique,
         min,
         max
FROM     (SUMMARIZE
          FROM READ_PARQUET('GMSEUS_Arrays_Final_2025_v2_0.parquet'))
WHERE    column_name != 'geometry'
AND      column_name != 'bbox'
ORDER BY LOWER(column_name);

┌─────────────┬─────────────┬─────────────────┬───────────────┬────────────┬────────────┐
│ column_name │ column_type │ null_percentage │ approx_unique │    min     │    max     │
│   varchar   │   varchar   │  decimal(9,2)   │     int64     │  varchar   │  varchar   │
├─────────────┼─────────────┼─────────────────┼───────────────┼────────────┼────────────┤
│ arrayID     │ BIGINT      │            0.00 │         16914 │ 1          │ 18980      │
│ avgAzimuth  │ DOUBLE      │           32.88 │          5865 │ 90.22      │ 540.0      │
│ avgLength   │ DOUBLE      │           32.88 │         12674 │ 4.02       │ 360.128    │
│ avgSpace    │ DOUBLE      │           32.88 │          7965 │ 0.024      │ 20.0       │
│ avgWidth    │ DOUBLE      │           32.88 │          6528 │ 0.863      │ 83.13      │
│ capMWAC     │ DOUBLE      │            0.00 │          4182 │ 0.003      │ 1128.931   │
│ capMWACest  │ DOUBLE      │            0.00 │          6620 │ 0.003      │ 1352.693   │
│ capMWDC     │ DOUBLE      │            0.00 │          7203 │ 0.003      │ 1467.61    │
│ capMWDCest  │ DOUBLE      │            0.00 │          8883 │ 0.004      │ 1758.501   │
│ COUNTYFP    │ VARCHAR     │            0.00 │           237 │ 001        │ 840        │
│ effInit     │ DOUBLE      │            0.07 │            12 │ 0.09       │ 0.21       │
│ GCR1        │ DOUBLE      │            0.00 │          5096 │ 0.1        │ 1.0        │
│ GCR2        │ DOUBLE      │            0.00 │          5913 │ 0.0943     │ 0.9883     │
│ grndCvr     │ VARCHAR     │            0.00 │             3 │ impervious │ vegetation │
│ instYr      │ BIGINT      │            0.00 │            26 │ 1985       │ 2025       │
│ instYrEst   │ BIGINT      │            0.32 │            23 │ 2003       │ 2025       │
│ modType     │ VARCHAR     │            0.00 │             3 │ c-si       │ thin-film  │
│ mount       │ VARCHAR     │            0.00 │             9 │ dual_axis  │ unknown    │
│ nativeID    │ VARCHAR     │            0.23 │         14342 │ 1          │ York Solar │
│ newBound    │ BIGINT      │            0.00 │             2 │ 0          │ 1          │
│ numRow      │ INTEGER     │            0.00 │          1177 │ 0          │ 90145      │
│ Source      │ VARCHAR     │            0.00 │            10 │ CCVPV      │ USPVDB     │
│ STATEFP     │ VARCHAR     │            0.00 │            49 │ 01         │ 56         │
│ tilt        │ DOUBLE      │           46.39 │            63 │ 0.0        │ 90.0       │
│ tiltEst     │ BIGINT      │           46.39 │            30 │ 10         │ 43         │
│ totArea     │ INTEGER     │            0.00 │         14669 │ 30         │ 19603313   │
│ totRowArea  │ INTEGER     │            0.00 │         13415 │ 30         │ 8537538    │
├─────────────┴─────────────┴─────────────────┴───────────────┴────────────┴────────────┤
│ 27 rows                                                                     6 columns │
└───────────────────────────────────────────────────────────────────────────────────────┘

Rooftop Solar Arrays

Below is a heatmap of the rooftop solar arrays dataset.

CREATE OR REPLACE TABLE h3_4_stats AS
    SELECT   h3_4: H3_LATLNG_TO_CELL(
                        bbox.ymin,
                        bbox.xmin,
                        4),
             COUNT(*) num_recs
    FROM     'GMSEUS_RooftopArrays_2025_v2_0.parquet'
    GROUP BY 1;

COPY (
    SELECT geometry: ST_ASWKB(H3_CELL_TO_BOUNDARY_WKT(h3_4)::geometry),
           num_recs
    FROM   h3_4_stats
) TO 'GMSEUS_RooftopArrays_2025_v2_0.h3_4.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

These are the source counts for this dataset.

SELECT   COUNT(*),
         Source
FROM     'GMSEUS_RooftopArrays_2025_v2_0.parquet'
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬────────────────────────────┐
│ count_star() │           Source           │
│    int64     │          varchar           │
├──────────────┼────────────────────────────┤
│         2175 │ OSM                        │
│         1835 │ CECSFC                     │
│         1024 │ TZSAM                      │
│          485 │ USPVDB                     │
│           93 │ GRW                        │
│           54 │ GMSEUSdigArraysPanels_v2_0 │
│           46 │ gspt                       │
│           43 │ SAM                        │
│           24 │ GMSEUSgeoref_v2_0          │
│           16 │ CCVPV                      │
│           15 │ GPPDB                      │
│           10 │ CWSD                       │
│            2 │ InSPIRE                    │
├──────────────┴────────────────────────────┤
│ 13 rows                         2 columns │
└───────────────────────────────────────────┘

Below are the most-represented source per hexagon.

CREATE OR REPLACE TABLE h3_3s AS
    WITH b AS (
        WITH a AS (
            SELECT   h3_3: H3_LATLNG_TO_CELL(
                                bbox.ymin,
                                bbox.xmin,
                                3) ,
                     Source,
                     num_recs: COUNT(*)
            FROM     'GMSEUS_RooftopArrays_2025_v2_0.parquet'
            GROUP BY 1, 2
        )
        SELECT *,
               ROW_NUMBER() OVER (PARTITION BY h3_3
                                  ORDER BY     num_recs DESC) AS rn
        FROM   a
    )
    FROM     b
    WHERE    rn = 1
    ORDER BY num_recs DESC;

COPY (
    SELECT geom: H3_CELL_TO_BOUNDARY_WKT(h3_3)::GEOMETRY,
           Source
    FROM   h3_3s
) TO 'GMSEUS_RooftopArrays_2025_v2_0.Source.h3_3.parquet' (
    FORMAT 'PARQUET',
    CODEC  'ZSTD',
    COMPRESSION_LEVEL 22,
    ROW_GROUP_SIZE 15000);

This is the breakdown of mount and mod types.

WITH a AS (
    SELECT   modType,
             mount,
             cnt: COUNT(*)
    FROM     'GMSEUS_RooftopArrays_2025_v2_0.parquet'
    GROUP BY 1, 2
)
PIVOT    a
ON       modType
USING    SUM(cnt)
GROUP BY mount
ORDER BY mount;

┌─────────────┬────────┬───────────┐
│    mount    │  c-si  │ thin-film │
│   varchar   │ int128 │  int128   │
├─────────────┼────────┼───────────┤
│ dual_axis   │     33 │      NULL │
│ fixed_axis  │    381 │         2 │
│ mixed       │      2 │      NULL │
│ single_axis │    210 │      NULL │
│ unknown     │     98 │      NULL │
│ NULL        │   5096 │      NULL │
└─────────────┴────────┴───────────┘

These are the area stats broken down by year of installation.

SELECT   instYr,
         count: COUNT(*),
         avg:   AVG(area)::INT,
         mean:  MEAN(area)::INT,
         max:   MAX(area)::INT
FROM     'GMSEUS_RooftopArrays_2025_v2_0.parquet'
WHERE    instYr IS NOT NULL
GROUP BY 1
ORDER BY 1;

┌────────┬───────┬───────┬───────┬────────┐
│ instYr │ count │  avg  │ mean  │  max   │
│ int64  │ int64 │ int32 │ int32 │ int32  │
├────────┼───────┼───────┼───────┼────────┤
│   2003 │    23 │  2630 │  2630 │  18124 │
│   2005 │     1 │ 20342 │ 20342 │  20342 │
│   2006 │     1 │ 37587 │ 37587 │  37587 │
│   2007 │     6 │ 16848 │ 16848 │  27067 │
│   2008 │     4 │ 24989 │ 24989 │  30107 │
│   2009 │    11 │  7877 │  7877 │  30229 │
│   2010 │    24 │ 15834 │ 15834 │  43854 │
│   2011 │    46 │ 41511 │ 41511 │ 487111 │
│   2012 │    54 │ 13371 │ 13371 │  38923 │
│   2013 │    47 │ 14476 │ 14476 │ 151620 │
│   2014 │    24 │ 29289 │ 29289 │  65995 │
│   2015 │    37 │ 34130 │ 34130 │ 218080 │
│   2016 │    47 │ 21094 │ 21094 │  46138 │
│   2017 │   105 │ 20882 │ 20882 │ 315564 │
│   2018 │   225 │ 13584 │ 13584 │ 152636 │
│   2019 │   102 │ 21613 │ 21613 │ 111497 │
│   2020 │   143 │ 16052 │ 16052 │  70018 │
│   2021 │   120 │ 16169 │ 16169 │ 101955 │
│   2022 │   141 │ 17343 │ 17343 │ 185698 │
│   2023 │   164 │ 18865 │ 18865 │ 137787 │
│   2024 │   132 │ 14743 │ 14743 │  63190 │
│   2025 │   148 │ 12363 │ 12363 │ 135270 │
├────────┴───────┴───────┴───────┴────────┤
│ 22 rows                       5 columns │
└─────────────────────────────────────────┘

Rooftop Arrays Footprints

The map below shows Los Angeles to Long Beach. The detections are coloured by their source.

Some sources conservatively outline the building footprint.

Other sources have more organic shapes.

The "gspt" source uses vague circles for its detections. Below, there are four warehouses with roof panels but only two large circles are shown for their detections.

There are plenty of rooftop arrays that weren't detected all over LA. Given that this dataset only contains ~5K records, its coverage leaves a fair amount of room for improvement.

Arrays & Panels

Below is a heatmap of the arrays dataset.

CREATE OR REPLACE TABLE h3_4_stats AS
    SELECT   h3_4: H3_LATLNG_TO_CELL(
                        bbox.ymin,
                        bbox.xmin,
                        4),
             COUNT(*) num_recs
    FROM     'GMSEUS_Arrays_Final_2025_v2_0.parquet'
    GROUP BY 1;

COPY (
    SELECT geometry: ST_ASWKB(H3_CELL_TO_BOUNDARY_WKT(h3_4)::geometry),
           num_recs
    FROM   h3_4_stats
) TO 'GMSEUS_Arrays_Final_2025_v2_0.h3_4.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

These are the record counts for each source.

SELECT   COUNT(*),
         Source
FROM     'GMSEUS_Arrays_Final_2025_v2_0.parquet'
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬────────────────────────────┐
│ count_star() │           Source           │
│    int64     │          varchar           │
├──────────────┼────────────────────────────┤
│         5222 │ OSM                        │
│         4024 │ USPVDB                     │
│         3278 │ TZSAM                      │
│         2288 │ CECSFC                     │
│         1697 │ GMSEUSgeoref_v2_0          │
│         1291 │ GMSEUSdigArraysPanels_v2_0 │
│          957 │ GRW                        │
│          155 │ CCVPV                      │
│           68 │ CWSD                       │
└──────────────┴────────────────────────────┘

Below are the most-represented source per hexagon.

CREATE OR REPLACE TABLE h3_3s AS
    WITH b AS (
        WITH a AS (
            SELECT   h3_3: H3_LATLNG_TO_CELL(
                                bbox.ymin,
                                bbox.xmin,
                                3) ,
                     Source,
                     num_recs: COUNT(*)
            FROM     'GMSEUS_Arrays_Final_2025_v2_0.parquet'
            GROUP BY 1, 2
        )
        SELECT *,
               ROW_NUMBER() OVER (PARTITION BY h3_3
                                  ORDER BY     num_recs DESC) AS rn
        FROM   a
    )
    FROM     b
    WHERE    rn = 1
    ORDER BY num_recs DESC;

COPY (
    SELECT geom: H3_CELL_TO_BOUNDARY_WKT(h3_3)::GEOMETRY,
           Source
    FROM   h3_3s
) TO 'GMSEUS_Arrays_Final_2025_v2_0.Source.h3_3.parquet' (
    FORMAT 'PARQUET',
    CODEC  'ZSTD',
    COMPRESSION_LEVEL 22,
    ROW_GROUP_SIZE 15000);

Solar Farms in close proximity can have different sources attributed to their detection.

Not all solar farms are picked up within the arrays dataset. There are also array datasets that haven't marked out the panels themselves. The following is an example from -118.355, 34.837. The panels are coloured purple.

These are the array capacity stats broken down by year of installation.

SELECT   instYr,

         count:  COUNT(*),

         ACavg:  AVG(capMWAC)::INT,
         DCavg:  AVG(capMWDC)::INT,

         ACmean: MEAN(capMWAC)::INT,
         DCmean: MEAN(capMWDC)::INT,

         ACmax:  MAX(capMWAC)::INT,
         DCmax:  MAX(capMWDC)::INT
FROM     'GMSEUS_Arrays_Final_2025_v2_0.parquet'
WHERE    instYr IS NOT NULL
GROUP BY 1
ORDER BY 1;

┌────────┬───────┬───────┬───────┬────────┬────────┬───────┬───────┐
│ instYr │ count │ ACavg │ DCavg │ ACmean │ DCmean │ ACmax │ DCmax │
│ int64  │ int64 │ int32 │ int32 │ int32  │ int32  │ int32 │ int32 │
├────────┼───────┼───────┼───────┼────────┼────────┼───────┼───────┤
│   1985 │     1 │    14 │    17 │     14 │     17 │    14 │    17 │
│   1990 │     2 │    88 │    90 │     88 │     90 │    92 │    92 │
│   2002 │     1 │     3 │     3 │      3 │      3 │     3 │     3 │
│   2003 │     7 │     2 │     2 │      2 │      2 │     2 │     3 │
│   2005 │     4 │     0 │     0 │      0 │      0 │     0 │     1 │
│   2006 │    64 │     3 │     4 │      3 │      4 │   100 │   125 │
│   2007 │   101 │     2 │     3 │      2 │      3 │    68 │    68 │
│   2008 │   195 │     2 │     3 │      2 │      3 │   226 │   294 │
│   2009 │   268 │     2 │     2 │      2 │      2 │   150 │   195 │
│   2010 │   401 │     2 │     2 │      2 │      2 │   150 │   195 │
│   2011 │   544 │     3 │     3 │      3 │      3 │   175 │   225 │
│   2012 │   777 │     3 │     3 │      3 │      3 │   331 │   430 │
│   2013 │   793 │     4 │     5 │      4 │      5 │   280 │   313 │
│   2014 │   913 │     6 │     7 │      6 │      7 │   586 │   752 │
│   2015 │   910 │     5 │     7 │      5 │      7 │   314 │   398 │
│   2016 │  1096 │     9 │    12 │      9 │     12 │   300 │   350 │
│   2017 │  1765 │     5 │     6 │      5 │      6 │   350 │   455 │
│   2018 │  1941 │     5 │     6 │      5 │      6 │   352 │   458 │
│   2019 │  1531 │     6 │     8 │      6 │      8 │   300 │   390 │
│   2020 │  1673 │    11 │    15 │     11 │     15 │   638 │   829 │
│   2021 │  1705 │    19 │    24 │     19 │     24 │   504 │   656 │
│   2022 │  1389 │    16 │    21 │     16 │     21 │   868 │  1128 │
│   2023 │  2017 │    34 │    44 │     34 │     44 │  1095 │  1423 │
│   2024 │   730 │    37 │    44 │     37 │     44 │   868 │  1128 │
│   2025 │   152 │    18 │    23 │     18 │     23 │  1129 │  1468 │
├────────┴───────┴───────┴───────┴────────┴────────┴───────┴───────┤
│ 25 rows                                                8 columns │
└──────────────────────────────────────────────────────────────────┘

Below is a gradient of the azimuth field in a solar farm in California (-115.47, 35.57). Though these detections are in the panels dataset, someone on Hacker News was kind enough to point out that these are mirrors and are a part of the Ivanpah Solar Power Facility.

Here is the same visualisation at another park.

The desert is full of these microchip-looking patterns.

French Mobile Network Datasets

2026-04-18T12:00:00+03:00

France runs a public data platform with 74K datasets and APIs containing and/or representing 387K files from 6K organisations. "My Mobile Network" is a group of datasets hosted on this platform. It contains six datasets describing the structure and performance of France's mobile networks.

These datasets include the locations, operators and capabilities of mobile sites, theoretical coverage of these networks, performance measurements against popular websites and several other metrics.

In this post, I'll walk through converting a few of these datasets into Parquet and examining their features.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use GDAL 3.9.3, Python 3.12.3 and a few other tools to help analyse the data in this post.

$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable
$ sudo apt update
$ sudo apt install \
    gdal-bin \
    jq \
    python3-pip \
    python3.12-venv

Below, I'll set up a Python Virtual Environment and install some dependencies.

$ python3 -m venv ~/.france-mobile
$ source ~/.france-mobile/bin/activate
$ python3 -m pip install \
    duckdb \
    py7zr \
    rich \
    shpyx

I'll use DuckDB, along with its H3, JSON, Lindel, Parquet and Spatial extensions in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.5.1/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

The maps in this post were rendered with QGIS version 4.0.0. QGIS is a desktop application that runs on Windows, macOS and Linux. The application has grown in popularity in recent years and has ~15M application launches from users all around the world each month.

I used QGIS' HCMGIS plugin to add basemaps from Esri to this post.

Metropolitan, Overseas & Departments of France

Metropolitan and Overseas France will be mentioned a lot in these datasets and files are often partitioned by these geographies.

Below is Metropolitan France, the hexagon-shaped landmass found in Europe.

I've coloured each department above by its number with the brightest colours being the highest values. The polygons and their labels were sourced from this gist. There isn't a strong geographic relationship between a department number and its location.

Below are ten of the eleven inhabited French overseas territories. Apologies for Saint Barthélemy's label being obscured by Saint Martin's.

Below is the 11th, New Caledonia, with its proximity to Australia and New Zealand.

Mobile Service Sites

The "Mobile Service Sites" dataset contains every French mobile provider's cell sites' capabilities for every quarter going back to 2017. Below, I've downloaded the latest quarter's CSV file. I'll import it into DuckDB.

$ ~/duckdb

CREATE OR REPLACE TABLE a AS
    FROM READ_CSV('2025_T4_sites_Metropole.csv',
                  decimal_separator=',');

The CSV file contains 123K rows.

SELECT COUNT(*) FROM a; -- 123068

These are the unique value counts and NULL % for each of the columns. The dataset contains the date when 5G was first activated at any site that supports it, several columns describing the geography and each site's network generation support.

SELECT   column_name,
         column_type[:30],
         null_percentage,
         approx_unique,
         min[:30],
         max[:30]
FROM     (SUMMARIZE
          FROM a)
ORDER BY LOWER(column_name);

┌──────────────────────────────┬──────────────────────────┬─────────────────┬───────────────┬─────────────────────────────┬────────────────────────┐
│         column_name          │     column_type[:30]     │ null_percentage │ approx_unique │          min[:30]           │        max[:30]        │
│           varchar            │         varchar          │  decimal(9,2)   │     int64     │           varchar           │        varchar         │
├──────────────────────────────┼──────────────────────────┼─────────────────┼───────────────┼─────────────────────────────┼────────────────────────┤
│ code_op                      │ BIGINT                   │            0.00 │             3 │ 20801                       │ 20820                  │
│ date_ouverturecommerciale_5g │ TIMESTAMP WITH TIME ZONE │           41.42 │          1636 │ 2020-10-09 03:00:00+03      │ 2025-12-31 02:00:00+02 │
│ id_site_partage              │ VARCHAR                  │           78.48 │          5065 │ 2019_LOT1_Site_ZG_03_003_S5 │ ZPZ95405               │
│ id_station_anfr              │ VARCHAR                  │            0.47 │         89043 │ 00008962D1                  │ 0952750744             │
│ insee_com                    │ VARCHAR                  │            0.00 │         20391 │ 01001                       │ 95690                  │
│ insee_dep                    │ VARCHAR                  │            0.00 │            90 │ 01                          │ 95                     │
│ latitude                     │ DOUBLE                   │            0.00 │         69205 │ 41.36445                    │ 51.08034               │
│ longitude                    │ DOUBLE                   │            0.00 │         66599 │ -5.08888                    │ 9.55028                │
│ mes_4g_trim                  │ BIGINT                   │            0.00 │             2 │ 0                           │ 1                      │
│ nom_com                      │ VARCHAR                  │            0.00 │         25727 │ Aast                        │ Œuilly                 │
│ nom_dep                      │ VARCHAR                  │            0.00 │            89 │ Ain                         │ Yvelines               │
│ nom_op                       │ VARCHAR                  │            0.00 │             4 │ Bouygues Telecom            │ SFR                    │
│ nom_reg                      │ VARCHAR                  │            0.00 │            15 │ Auvergne-Rhône-Alpes        │ Île-de-France          │
│ num_site                     │ VARCHAR                  │            0.00 │        128897 │ 00000001A1                  │ TA0005                 │
│ site_2g                      │ BIGINT                   │            0.00 │             2 │ 0                           │ 1                      │
│ site_3g                      │ BIGINT                   │            0.00 │             2 │ 0                           │ 1                      │
│ site_4g                      │ BIGINT                   │            0.00 │             2 │ 0                           │ 1                      │
│ site_5g                      │ BIGINT                   │            0.00 │             2 │ 0                           │ 1                      │
│ site_5g_1800_m_hz            │ BIGINT                   │            0.00 │             1 │ 0                           │ 0                      │
│ site_5g_2100_m_hz            │ BIGINT                   │            0.00 │             2 │ 0                           │ 1                      │
│ site_5g_3500_m_hz            │ BIGINT                   │            0.00 │             2 │ 0                           │ 1                      │
│ site_5g_700_m_hz             │ BIGINT                   │            0.00 │             2 │ 0                           │ 1                      │
│ site_5g_800_m_hz             │ BIGINT                   │            0.00 │             1 │ 0                           │ 0                      │
│ site_capa_240mbps            │ BIGINT                   │            0.00 │             2 │ 0                           │ 1                      │
│ site_DCC                     │ BIGINT                   │            0.00 │             2 │ 0                           │ 1                      │
│ site_strategique             │ BIGINT                   │            0.00 │             2 │ 0                           │ 1                      │
│ site_ZB                      │ BIGINT                   │            0.00 │             2 │ 0                           │ 1                      │
│ x                            │ DOUBLE                   │            0.00 │         83320 │ 102979.0                    │ 1240681.0              │
│ y                            │ DOUBLE                   │            0.00 │         92010 │ 6050020.0                   │ 7109525.0              │
└──────────────────────────────┴──────────────────────────┴─────────────────┴───────────────┴─────────────────────────────┴────────────────────────┘

Below are the number of sites for each network operator. Note, this file is only for Metropolitan France, not its overseas territories.

SELECT   COUNT(*),
         nom_op
FROM     a
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬──────────────────┐
│ count_star() │      nom_op      │
│    int64     │     varchar      │
├──────────────┼──────────────────┤
│        32367 │ Orange           │
│        30286 │ Free Mobile      │
│        30255 │ Bouygues Telecom │
│        30160 │ SFR              │
└──────────────┴──────────────────┘

Below is the number of sites supporting each network generation broken down by operator.

SELECT   nom_op,
         "2g":          COUNT(*) FILTER(site_2g),
         "3g":          COUNT(*) FILTER(site_3g),
         "4g":          COUNT(*) FILTER(site_4g),
         "5g":          COUNT(*) FILTER(site_5g),
         "5g_1800":     COUNT(*) FILTER(site_5g_1800_m_hz),
         "5g_2100":     COUNT(*) FILTER(site_5g_2100_m_hz),
         "5g_3500":     COUNT(*) FILTER(site_5g_3500_m_hz),
         "5g_700":      COUNT(*) FILTER(site_5g_700_m_hz),
         "5g_800":      COUNT(*) FILTER(site_5g_800_m_hz),
         "capa_240":    COUNT(*) FILTER(site_capa_240mbps),
         "DCC":         COUNT(*) FILTER(site_DCC),
         "strategique": COUNT(*) FILTER(site_strategique),
         "ZB":          COUNT(*) FILTER(site_ZB)
FROM     a
GROUP BY 1;

┌──────────────────┬───────┬───────┬───────┬───────┬─────────┬─────────┬─────────┬────────┬────────┬──────────┬───────┬─────────────┬───────┐
│      nom_op      │  2g   │  3g   │  4g   │  5g   │ 5g_1800 │ 5g_2100 │ 5g_3500 │ 5g_700 │ 5g_800 │ capa_240 │  DCC  │ strategique │  ZB   │
│     varchar      │ int64 │ int64 │ int64 │ int64 │  int64  │  int64  │  int64  │ int64  │ int64  │  int64   │ int64 │    int64    │ int64 │
├──────────────────┼───────┼───────┼───────┼───────┼─────────┼─────────┼─────────┼────────┼────────┼──────────┼───────┼─────────────┼───────┤
│ SFR              │ 21990 │ 29199 │ 30144 │ 16089 │       0 │   11578 │   10507 │      0 │      0 │    27983 │  3925 │          36 │  2647 │
│ Bouygues Telecom │ 19556 │ 28590 │ 30222 │ 17636 │       0 │   17377 │   10513 │      0 │      0 │    27838 │  3929 │          36 │  2647 │
│ Orange           │ 15531 │ 29649 │ 32327 │ 15875 │       0 │    2232 │   13508 │  12548 │      0 │    31123 │  3936 │          36 │  2648 │
│ Free Mobile      │     0 │  6654 │ 30266 │ 22499 │       0 │       0 │   10501 │  21080 │      0 │    30130 │  3961 │          36 │  2647 │
└──────────────────┴───────┴───────┴───────┴───────┴─────────┴─────────┴─────────┴────────┴────────┴──────────┴───────┴─────────────┴───────┘

I'll export this table to a Parquet file.

COPY (
    SELECT   * EXCLUDE(latitude, longitude),
             geometry: ST_POINT(longitude, latitude)
    FROM     a
    ORDER BY HILBERT_ENCODE([latitude,
                             longitude]::double[2])
) TO '2025_T4_sites_Metropole.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

Below are the sites around Nice Airport along with the network generations they support.

Theoretical Coverage Maps

The theoretical coverage dataset has polygons showing each brand's coverage for each quarter going back to 2017. These are broken down by network generation and separated by voice and data.

Below, I'll download this dataset.

$ mkdir -p ~/coverage
$ cd ~/coverage
$ wget -r \
       --no-parent \
       -A 7z \
       https://data.arcep.fr/mobile/couvertures_theoriques/

The resulting set of 7Zip files is 46 GB.

$ du -hs data.arcep.fr/ # 46 GB

These 7Zip files can contain either a GeoDatabase set of files for the older datasets or a GeoPackage (GPKG) file in case of the newer ones. Below, I'll figure out how many contain a GPKG file.

$ python3

from   collections import Counter
import pathlib

import py7zr
from   rich.progress import track


has_gpkg = {}

for x in track(list(pathlib.Path(".").glob("**/*.7z"))):
    try:
        with py7zr.SevenZipFile(x, 'r') as archive:
            has_gpkg[str(x)] = any([True
                                    for y in archive.getnames()
                                    if y.lower().endswith('.gpkg')])
    except Exception as e:
        print(x, e)

Three 7Zip files weren't readable by the above. I'll exclude them from this analysis.

data.arcep.fr/mobile/couvertures_theoriques/2023_T2/Outremer/973_Guyane/2023_T2_couv_973_ORCA_2G_voix.gpkg.7z not a 7z file
data.arcep.fr/mobile/couvertures_theoriques/2023_T2/Outremer/973_Guyane/2023_T2_couv_973_ORCA_3G_data.gpkg.7z not a 7z file
data.arcep.fr/mobile/couvertures_theoriques/2023_T2/Outremer/973_Guyane/2023_T2_couv_973_ORCA_4G_data.gpkg.7z not a 7z file

Of the remaining files, 1,150 contain GPKG files with the remaining containing legacy GeoDatabases.

Counter(has_gpkg.values()).most_common()

[(True, 1150), (False, 1002)]

A variety of projections are used across the GPKG files.

$ gdalsrsinfo \
    -o proj4 \
    2022_T3_couv_971_OMT_4G_data.gpkg

+proj=utm +zone=20 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs

$ gdalsrsinfo \
    -o proj4 \
    2022_T3_couv_Metropole_BOUY_4G_data.gpkg

+proj=lcc +lat_0=46.5 +lon_0=3 +lat_1=49 +lat_2=44 +x_0=700000 +y_0=6600000 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs

Neither GeoDatabases nor GPKG files are suitable for distribution and analysis at this scale. Both are row-based formats meant to be easy to edit but at the expense of being very slow to query in comparison to any column-based format.

I'll run the following to convert the GPKG files into Parquet using EPSG:4326 for their projection.

$ python3

import pathlib

import duckdb
import py7zr
from   rich.progress import track
from   shpyx         import run as execute


con = duckdb.connect(database=':memory:')
con.sql('INSTALL spatial; LOAD spatial')
con.sql('INSTALL parquet; LOAD parquet')

sql = """
COPY(SELECT operateur,
            date,
            techno,
            usage,
            niveau,
            dept,
            geometry: ST_FLIPCOORDINATES(ST_TRANSFORM(geom, '%s', 'EPSG:4326'))
     FROM   ST_READ('%s'))
TO '%s' (FORMAT 'PARQUET')
"""

for x in track(list(pathlib.Path(".").glob("**/*.7z"))):
    try:
        with py7zr.SevenZipFile(x, 'r') as archive:
            selective_files = [y
                               for y in archive.getnames()
                               if y.lower().endswith('.gpkg')]

            if selective_files:
                archive.extract(targets=selective_files)

                for y in selective_files:
                    srs = str(execute('gdalsrsinfo -o proj4 %s' % y).stdout).strip()

                    con.sql(sql % (srs, y, str(y) + '.parquet'))
    except Exception as e:
        print(x, e)

The GPKG filenames contain a truncated mobile phone operator brand while their "operateur" column contains their numeric "Public Land Mobile Network" (PLMN) identifier. I'll build a table of these that'll be used in a join later on.

$ ~/duckdb spill.duckdb

CREATE OR REPLACE TABLE brands AS
    SELECT DISTINCT brand: SPLIT_PART(filename, '_', 5),
                    operateur
    FROM   READ_PARQUET('*.gpkg.parquet', filename=true);

Below are the brand and numeric code pairs.

FROM brands;

┌─────────┬───────────┐
│  brand  │ operateur │
│ varchar │   int64   │
├─────────┼───────────┤
│ FREE    │     20815 │
│ ORCA    │     34001 │
│ TELC    │     64703 │
│ SRR     │     64710 │
│ UTS     │     34012 │
│ ZEOP    │     64700 │
│ DIGIC   │     34020 │
│ OF      │     64700 │
│ OMT     │     34002 │
│ FREE    │     20801 │
│ FRCA    │     34020 │
│ ZEOP    │     34001 │
│ ZEOP    │     64704 │
│ DAUPH   │     34008 │
│ OF      │     20801 │
│ TELC    │     64702 │
│ BOUY    │     20820 │
│ FRCA    │     34004 │
│ SFR0    │     20810 │
└─────────┴───────────┘

I'll then merge the 1K+ Parquet files into one for each quarter of each year.

$ for QUARTER in `ls 20*.gpkg.parquet | cut -c1-7 | sort | uniq`; do
      echo $QUARTER
      ~/duckdb \
          -c "SET memory_limit='64GB';" \
          -c "COPY (
                  SELECT   a.*,
                           b.brand,
                           bbox: {'xmin': ST_XMIN(ST_EXTENT(a.geometry)),
                                  'ymin': ST_YMIN(ST_EXTENT(a.geometry)),
                                  'xmax': ST_XMAX(ST_EXTENT(a.geometry)),
                                  'ymax': ST_YMAX(ST_EXTENT(a.geometry))},
                  FROM     '$QUARTER*.gpkg.parquet' a
                  JOIN     brands b ON a.operateur = b.operateur
                  WHERE    ST_X(ST_CENTROID(a.geometry)) IS NOT NULL
                  ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(a.geometry)),
                                           ST_X(ST_CENTROID(a.geometry))]::double[2])
              ) TO 'theoretical_cover.$QUARTER.parquet' (
                      FORMAT 'PARQUET',
                      CODEC  'ZSTD',
                      COMPRESSION_LEVEL 22,
                      ROW_GROUP_SIZE 15000);" \
          spill.duckdb
  done

Below are the resulting Parquet files.

$ du -hcs theoretical_cover.202*.parquet

345M    theoretical_cover.2022_T3.parquet
4.6G    theoretical_cover.2022_T4.parquet
229M    theoretical_cover.2023_T1.parquet
4.0G    theoretical_cover.2023_T2.parquet
238M    theoretical_cover.2023_T3.parquet
3.6G    theoretical_cover.2023_T4.parquet
226M    theoretical_cover.2024_T1.parquet
3.6G    theoretical_cover.2024_T2.parquet
219M    theoretical_cover.2024_T3.parquet
4.8G    theoretical_cover.2024_T4.parquet
1.6G    theoretical_cover.2025_T1.parquet
5.0G    theoretical_cover.2025_T2.parquet
1.6G    theoretical_cover.2025_T3.parquet
5.6G    theoretical_cover.2025_T4.parquet
36G     total

The GPKG format usage only began in the third quarter of 2022 so this Parquet-formatted dataset won't contain data before that.

Below is a breakdown of unique values and NULL coverage across each column for the 2025 data.

$ ~/duckdb

SELECT   column_name,
         column_type[:30],
         null_percentage,
         approx_unique,
         min[:30],
         max[:30]
FROM     (SUMMARIZE
          FROM 'theoretical_cover.2025*.parquet')
ORDER BY LOWER(column_name);

┌─────────────┬────────────────────────────────┬─────────────────┬───────────────┬────────────────────────────────┬────────────────────────────────┐
│ column_name │        column_type[:30]        │ null_percentage │ approx_unique │            min[:30]            │            max[:30]            │
│   varchar   │            varchar             │  decimal(9,2)   │     int64     │            varchar             │            varchar             │
├─────────────┼────────────────────────────────┼─────────────────┼───────────────┼────────────────────────────────┼────────────────────────────────┤
│ bbox        │ STRUCT(xmin DOUBLE, ymin DOUBL │            0.00 │          9828 │ {'xmin': -63.1531358, 'ymin':  │ {'xmin': 55.276453316344025, ' │
│ brand       │ VARCHAR                        │            0.00 │            15 │ BOUY                           │ ZEOP                           │
│ date        │ VARCHAR                        │            0.00 │             4 │ 2025-03-31                     │ 2025-12-31                     │
│ dept        │ VARCHAR                        │            0.00 │           105 │ 01                             │ 978                            │
│ geometry    │ GEOMETRY('OGC:CRS84')          │            0.00 │         10597 │ POLYGON ((2.333099900266716 48 │ MULTIPOLYGON (((3.389827530127 │
│ niveau      │ VARCHAR                        │            0.00 │             4 │                                │ TBC                            │
│ operateur   │ BIGINT                         │            0.00 │            13 │ 20801                          │ 64710                          │
│ techno      │ VARCHAR                        │            0.00 │             5 │ 2G                             │ 5G                             │
│ usage       │ VARCHAR                        │            0.00 │             1 │ data                           │ voix                           │
└─────────────┴────────────────────────────────┴─────────────────┴───────────────┴────────────────────────────────┴────────────────────────────────┘

Coverage is broken down by mobile network generation as well as data and voice (French: voix) support. These are the row counts for each.

Note: data coverage isn't collected for older generations and voice coverage isn't collected for newer generations.

WITH a AS (
    SELECT   techno,
             usage,
             cnt: COUNT(*)
    FROM     'theoretical_cover.*.parquet'
    GROUP BY 1, 2
)
PIVOT a
ON       usage
USING    SUM(cnt)
GROUP BY techno
ORDER BY techno;

┌─────────┬────────┬────────┐
│ techno  │  data  │  voix  │
│ varchar │ int128 │ int128 │
├─────────┼────────┼────────┤
│ 2G      │   NULL │ 861398 │
│ 2G3G    │   NULL │ 924737 │
│ 3G      │  51345 │   NULL │
│ 4G      │ 154727 │   NULL │
│ 5G      │    480 │   NULL │
└─────────┴────────┴────────┘

Below are the changes in 4G data coverage area broken down by brand and quarter of reporting.

.maxrows 100

SELECT   brand,
         date,
         area: ROUND(SUM(ST_AREA(geometry)), 4)
FROM     'theoretical_cover.2025*.parquet'
WHERE    techno = '4G'
AND      usage  = 'data'
GROUP BY 1, 2
ORDER BY 1, 2;

┌─────────┬────────────┬──────────┐
│  brand  │    date    │   area   │
│ varchar │  varchar   │  double  │
├─────────┼────────────┼──────────┤
│ BOUY    │ 2025-03-31 │  61.8234 │
│ BOUY    │ 2025-06-30 │   61.868 │
│ BOUY    │ 2025-09-30 │  61.9096 │
│ BOUY    │ 2025-12-31 │  61.8935 │
│ DAUPH   │ 2025-03-31 │   0.0051 │
│ DAUPH   │ 2025-06-30 │   0.0052 │
│ DAUPH   │ 2025-09-30 │   0.0052 │
│ DAUPH   │ 2025-12-31 │   0.0052 │
│ DIGIC   │ 2025-03-31 │   0.4977 │
│ DIGIC   │ 2025-06-30 │   0.4079 │
│ DIGIC   │ 2025-09-30 │    0.407 │
│ DIGIC   │ 2025-12-31 │    0.411 │
│ FRCA    │ 2025-03-31 │    0.796 │
│ FRCA    │ 2025-06-30 │   0.7558 │
│ FRCA    │ 2025-09-30 │   0.7559 │
│ FRCA    │ 2025-12-31 │   0.7599 │
│ FREE    │ 2025-03-31 │ 121.0205 │
│ FREE    │ 2025-06-30 │ 121.1464 │
│ FREE    │ 2025-09-30 │ 121.5269 │
│ FREE    │ 2025-12-31 │ 124.2319 │
│ OF      │ 2025-03-31 │  62.0904 │
│ OF      │ 2025-06-30 │  62.1185 │
│ OF      │ 2025-09-30 │  62.4091 │
│ OF      │ 2025-12-31 │   62.372 │
│ OMT     │ 2025-03-31 │   0.5476 │
│ OMT     │ 2025-06-30 │   0.5496 │
│ OMT     │ 2025-09-30 │   0.5528 │
│ OMT     │ 2025-12-31 │   0.5584 │
│ ORCA    │ 2025-03-31 │   0.5986 │
│ ORCA    │ 2025-06-30 │   0.5867 │
│ ORCA    │ 2025-09-30 │   0.5867 │
│ ORCA    │ 2025-12-31 │   0.5861 │
│ SFR0    │ 2025-03-31 │  62.3481 │
│ SFR0    │ 2025-06-30 │  62.4215 │
│ SFR0    │ 2025-09-30 │  62.4297 │
│ SFR0    │ 2025-12-31 │  62.4926 │
│ SRR     │ 2025-03-31 │   0.2295 │
│ SRR     │ 2025-06-30 │   0.2287 │
│ SRR     │ 2025-09-30 │   0.2287 │
│ SRR     │ 2025-12-31 │   0.2285 │
│ TELC    │ 2025-03-31 │   0.1628 │
│ TELC    │ 2025-06-30 │   0.1704 │
│ TELC    │ 2025-09-30 │   0.1731 │
│ TELC    │ 2025-12-31 │   0.1731 │
│ UTS     │ 2025-03-31 │   0.0026 │
│ UTS     │ 2025-09-30 │   0.0052 │
│ ZEOP    │ 2025-03-31 │   1.0214 │
│ ZEOP    │ 2025-06-30 │    1.307 │
│ ZEOP    │ 2025-09-30 │   1.0065 │
│ ZEOP    │ 2025-12-31 │   1.0079 │
└─────────┴────────────┴──────────┘

I'll extract Free's 4G data coverage for the end of 2025.

COPY (
    FROM  'theoretical_cover.2025*.parquet'
    WHERE techno = '4G'
    AND   usage  = 'data'
    AND   date   = '2025-12-31'
    AND   brand  = 'FREE'
) TO 'tc.free.parquet' (
      FORMAT 'PARQUET',
      CODEC  'ZSTD',
      COMPRESSION_LEVEL 22,
      ROW_GROUP_SIZE 15000);

Most of France is well-covered.

When zooming in on the Alps, you can see gaps where there isn't any coverage.

Ookla & Speedchecker

Since 2022, crowd-sourced speed measurements have been included in these datasets. The last quarter of 2025 has measurements from both Ookla and Speedchecker.

I'll first import Ookla's data into a DuckDB table.

$ ~/duckdb

CREATE OR REPLACE TABLE a AS
    SELECT * EXCLUDE(date_time_start,
                     latitude_start,
                     longitude_start,
                     protocole),
            started_at: STRPTIME(date_time_start, '%d/%m/%Y %H:%M:%S.%g'),
            geom:       ST_POINT(longitude_start, latitude_start)
    FROM '2025_T4_crowd_Ookla.csv';

There are almost ~430K rows in this table.

SELECT COUNT(*) FROM a; -- 428851

Below are the field names, data types, percentages of NULLs per column, number of unique values and minimum and maximum values for each column.

Several columns only contain NULLs so I've excluded them from this query.

SELECT   column_name,
         column_type[:30],
         null_percentage,
         approx_unique,
         min[:30],
         max[:30]
FROM     (SUMMARIZE
          FROM a)
WHERE    null_percentage < 100
ORDER BY LOWER(column_name);

┌─────────────┬──────────────────┬─────────────────┬───────────────┬─────────────────────┬─────────────────────┐
│ column_name │ column_type[:30] │ null_percentage │ approx_unique │      min[:30]       │      max[:30]       │
│   varchar   │     varchar      │  decimal(9,2)   │     int64     │       varchar       │       varchar       │
├─────────────┼──────────────────┼─────────────────┼───────────────┼─────────────────────┼─────────────────────┤
│ bitrate_dl  │ DOUBLE           │            0.00 │        265709 │ 0.001               │ 1810.234            │
│ geom        │ GEOMETRY         │            0.00 │         50624 │ POINT (0 44.58)     │ POINT (-1.81 48.34) │
│ mcc_start   │ BIGINT           │            0.00 │             3 │ 208                 │ 647                 │
│ mnc_start   │ VARCHAR          │            0.00 │             6 │ 01                  │ 20                  │
│ operator    │ VARCHAR          │            0.00 │             5 │ Bouygues            │ SRR                 │
│ started_at  │ TIMESTAMP        │            0.00 │        425630 │ 2025-10-01 02:00:22 │ 2026-01-01 00:59:34 │
└─────────────┴──────────────────┴─────────────────┴───────────────┴─────────────────────┴─────────────────────┘

Below are the number of measurements for each month broken down by operator.

WITH a AS (
    SELECT   operator,
             yyyy_mm: STRFTIME(started_at, '%Y-%m'),
             cnt:     COUNT(*)
    FROM     a
    GROUP BY 1, 2
)
PIVOT    a
ON       operator
USING    SUM(cnt)
GROUP BY yyyy_mm
ORDER BY yyyy_mm;

┌─────────┬──────────┬────────┬────────┬────────┬────────┐
│ yyyy_mm │ Bouygues │  Free  │ Orange │  SFR   │  SRR   │
│ varchar │  int128  │ int128 │ int128 │ int128 │ int128 │
├─────────┼──────────┼────────┼────────┼────────┼────────┤
│ 2025-10 │    32928 │  42820 │  51075 │  13862 │      3 │
│ 2025-11 │    32571 │  40649 │  49487 │  15334 │      6 │
│ 2025-12 │    37163 │  44794 │  52850 │  15148 │     11 │
│ 2026-01 │       31 │     47 │     50 │     22 │   NULL │
└─────────┴──────────┴────────┴────────┴────────┴────────┘

I'll create a heatmap of the highest measurements seen for each operator.

CREATE OR REPLACE TABLE h3_stats AS
    SELECT   h3_3: H3_LATLNG_TO_CELL(
                       ST_Y(geom),
                       ST_X(geom),
                       6),
             operator,
             bitrate_dl: MAX(bitrate_dl)
    FROM     a
    GROUP BY 1, 2;

COPY (
    SELECT geometry: ST_ASWKB(H3_CELL_TO_BOUNDARY_WKT(h3_3)::geometry),
           operator,
           bitrate_dl
    FROM   h3_stats
    WHERE  ST_XMIN(H3_CELL_TO_BOUNDARY_WKT(h3_3)::geometry) BETWEEN -179 AND 179
    AND    ST_XMAX(H3_CELL_TO_BOUNDARY_WKT(h3_3)::geometry) BETWEEN -179 AND 179
) TO 'ookla.h3_3_stats.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

The brightest hexagons below show where Orange's highest speeds were seen in this dataset.

Speedchecker's measurements are further broken down by device and also include an upload speed measurement. Below, I'll import their results from the end of last year into a DuckDB table.

$ ~/duckdb

CREATE OR REPLACE TABLE a AS
    SELECT * EXCLUDE(latitude_start,
                     longitude_start),
           geom: ST_POINT(longitude_start, latitude_start)
    FROM   '2025_T4_crowd_Speedchecker.csv';

This dataset has over 51K records.

SELECT COUNT(*) FROM a; -- 51594

Below are the field names, data types, percentages of NULLs per column, number of unique values and minimum and maximum values for each column.

Several columns only contain NULLs so I've excluded them from this query.

SELECT   column_name,
         column_type[:30],
         null_percentage,
         approx_unique,
         min[:30],
         max[:30]
FROM     (SUMMARIZE
          FROM a)
WHERE    null_percentage < 100
ORDER BY LOWER(column_name);

┌─────────────┬──────────────────┬─────────────────┬───────────────┬─────────────────┬────────────────────────────┐
│ column_name │ column_type[:30] │ null_percentage │ approx_unique │    min[:30]     │          max[:30]          │
│   varchar   │     varchar      │  decimal(9,2)   │     int64     │     varchar     │          varchar           │
├─────────────┼──────────────────┼─────────────────┼───────────────┼─────────────────┼────────────────────────────┤
│ bitrate_dl  │ DOUBLE           │            0.02 │         48321 │ 0.01            │ 1491.45                    │
│ bitrate_ul  │ DOUBLE           │           31.64 │         18354 │ 0.008           │ 249.076                    │
│ geom        │ GEOMETRY         │            0.00 │         45497 │ POINT (2 48.31) │ POINT (1.5921199 49.04092) │
│ mcc_start   │ BIGINT           │            0.00 │             1 │ 208             │ 208                        │
│ mnc_start   │ BIGINT           │            0.00 │             4 │ 1               │ 20                         │
│ operator    │ VARCHAR          │            0.00 │             4 │ Bouygues        │ SFR                        │
│ protocole   │ VARCHAR          │            0.00 │             1 │ DOWNLOAD        │ DOWNLOAD                   │
│ terminal    │ VARCHAR          │            5.19 │          1346 │ 2112123AG       │ vivo Y76 5G                │
└─────────────┴──────────────────┴─────────────────┴───────────────┴─────────────────┴────────────────────────────┘

Below are the fastest download speeds seen on each network along with the device the speed was seen with.

WITH b AS (
    WITH a AS (
        SELECT   operator,
                 terminal,
                 bitrate_dl
        FROM     a
        GROUP BY 1, 2, 3
        ORDER BY 3 DESC
    )
    SELECT *,
           ROW_NUMBER() OVER (PARTITION BY operator
                              ORDER BY     bitrate_dl DESC) AS rn
    FROM   a
)
SELECT   operator,
         terminal,
         bitrate_dl
FROM     b
WHERE    rn = 1
ORDER BY bitrate_dl DESC;

┌──────────┬────────────────────┬────────────┐
│ operator │      terminal      │ bitrate_dl │
│ varchar  │      varchar       │   double   │
├──────────┼────────────────────┼────────────┤
│ Orange   │ Galaxy S23         │    1491.45 │
│ SFR      │ Galaxy Z Flip5     │   1457.584 │
│ Bouygues │ Samsung Galaxy A54 │     1002.3 │
│ Free     │ Samsung Galaxy A54 │     895.83 │
└──────────┴────────────────────┴────────────┘

Quality of Service

The Quality of Service files are broken down by year, separated by voice and data and also separated by measurements taken while stationary or while moving. I'll load the stationary data measurements for Metropolitan France into DuckDB.

$ ~/duckdb

CREATE OR REPLACE TABLE a AS
    SELECT * EXCLUDE(bitrate_dl,
                     bitrate_ul,
                     acess_duration,
                     date_start,
                     hour_start,
                     latitude_start,
                     longitude_start,
                     loaded_in_less_10_secondes,
                     loaded_in_less_5_secondes,
                     quality_correct,
                     quality_perfect,
                     upload_ok,
                     situation),
           bitrate_dl: REPLACE(bitrate_dl, ',', '.')::DOUBLE,
           bitrate_ul: REPLACE(bitrate_ul, ',', '.')::DOUBLE,
           acess_duration: REPLACE(acess_duration, ',', '.')::DOUBLE,
           started_at: CONCAT(date_start, ' ', hour_start)::TIMESTAMP,
           geom: ST_POINT(longitude_start, latitude_start),
           loaded_in_less_10_secondes: loaded_in_less_10_secondes::BOOLEAN,
           loaded_in_less_5_secondes: loaded_in_less_5_secondes::BOOLEAN,
           quality_correct: quality_correct::BOOLEAN,
           quality_perfect: quality_perfect::BOOLEAN,
           upload_ok: upload_ok::BOOLEAN,
           situation: LOWER(situation)
    FROM READ_CSV('2025_QoS_Metropole_data_habitations.csv',
                  delim=';',
                  ignore_errors=true);

This table has over 273K rows.

SELECT COUNT(*) FROM a; -- 273710

Below are the field names, data types, percentages of NULLs per column, number of unique values and minimum and maximum values for each column.

Several columns only contain NULLs so I've excluded them from this query.

SELECT   column_name,
         column_type[:30],
         null_percentage,
         approx_unique,
         min[:30],
         max[:30]
FROM     (SUMMARIZE
          FROM a)
WHERE    null_percentage < 100
ORDER BY LOWER(column_name);

┌────────────────────────────┬──────────────────┬─────────────────┬───────────────┬────────────────────────────────┬────────────────────────────────┐
│        column_name         │ column_type[:30] │ null_percentage │ approx_unique │            min[:30]            │            max[:30]            │
│          varchar           │     varchar      │  decimal(9,2)   │     int64     │            varchar             │            varchar             │
├────────────────────────────┼──────────────────┼─────────────────┼───────────────┼────────────────────────────────┼────────────────────────────────┤
│ acess_duration             │ DOUBLE           │            5.17 │          2252 │ 0.0                            │ 66.02                          │
│ bitrate_dl                 │ DOUBLE           │           90.57 │         14847 │ 0.0                            │ 993.9109005                    │
│ bitrate_ul                 │ DOUBLE           │           95.29 │         10844 │ 0.0                            │ 155.9220818                    │
│ geom                       │ GEOMETRY         │            0.00 │         73125 │ POINT (2.125 48.6238)          │ POINT (5.73514 45.08009)       │
│ insee_com                  │ BIGINT           │            0.00 │          1245 │ 1043                           │ 95678                          │
│ loaded_in_less_10_secondes │ BOOLEAN          │           28.53 │             2 │ false                          │ true                           │
│ loaded_in_less_5_secondes  │ BOOLEAN          │           28.53 │             2 │ false                          │ true                           │
│ mcc_start                  │ BIGINT           │           11.49 │             3 │ 0                              │ 214                            │
│ mnc_start                  │ BIGINT           │           11.49 │            11 │ 0                              │ 88                             │
│ operator                   │ VARCHAR          │            0.00 │             4 │ Bouygues                       │ SFR                            │
│ protocole                  │ VARCHAR          │            0.00 │             6 │ DOWNLOAD                       │ WEB                            │
│ quality_correct            │ BOOLEAN          │           95.22 │             2 │ false                          │ true                           │
│ quality_perfect            │ BOOLEAN          │           95.22 │             2 │ false                          │ true                           │
│ Result_1                   │ VARCHAR          │            0.00 │             5 │ Drop                           │ Traffic Fail                   │
│ rscp                       │ DOUBLE           │           98.03 │          2485 │ -121.0                         │ -38.89                         │
│ rsrp                       │ DOUBLE           │            4.38 │          6304 │ -154.33                        │ -53.67                         │
│ rsrq                       │ DOUBLE           │            4.23 │          2044 │ -30.0                          │ -1.16                          │
│ situation                  │ VARCHAR          │            0.00 │             3 │ incar                          │ outdoor                        │
│ started_at                 │ TIMESTAMP        │            0.00 │        161212 │ 2025-05-26 08:20:12            │ 2025-08-29 18:09:16            │
│ techno_start               │ VARCHAR          │            0.00 │             1 │ 5G                             │ 5G                             │
│ terminal                   │ VARCHAR          │            0.00 │             3 │ SM-F731B                       │ SM-S921B                       │
│ territory                  │ VARCHAR          │            0.00 │             1 │ METROPOLE                      │ METROPOLE                      │
│ transfert_file_size        │ BIGINT           │           81.14 │             3 │ 1                              │ 250                            │
│ upload_ok                  │ BOOLEAN          │           95.28 │             2 │ false                          │ true                           │
│ url                        │ VARCHAR          │            9.67 │            32 │ https://fr.linkedin.com        │ https://x.com/?locale=fr       │
│ zone                       │ VARCHAR          │            0.00 │             4 │ Zones denses                   │ Zones touristiques             │
│ zone_name                  │ VARCHAR          │            0.00 │          1175 │ Abbaye du Mont-Saint-Michel Mo │ Zoo parc de Beauval Saint-Aign │
└────────────────────────────┴──────────────────┴─────────────────┴───────────────┴────────────────────────────────┴────────────────────────────────┘

These are the measurement counts for each operator for each month.

WITH a AS (
    SELECT   operator,
             yyyy_mm: STRFTIME(started_at, '%Y-%m'),
             cnt:     COUNT(*)
    FROM     a
    GROUP BY 1, 2
)
PIVOT    a
ON       operator
USING    SUM(cnt)
GROUP BY yyyy_mm
ORDER BY yyyy_mm;

┌─────────┬──────────┬────────┬────────┬────────┐
│ yyyy_mm │ Bouygues │  Free  │ Orange │  SFR   │
│ varchar │  int128  │ int128 │ int128 │ int128 │
├─────────┼──────────┼────────┼────────┼────────┤
│ 2025-05 │      949 │    939 │    950 │    948 │
│ 2025-06 │    19349 │  19182 │  19346 │  19226 │
│ 2025-07 │    29692 │  29898 │  30012 │  29998 │
│ 2025-08 │    18259 │  18245 │  18374 │  18343 │
└─────────┴──────────┴────────┴────────┴────────┘

There are three different types of situations that tests can fall under.

SELECT   situation,
         count(*)
FROM     a
GROUP BY 1;

┌───────────┬──────────────┐
│ situation │ count_star() │
│  varchar  │    int64     │
├───────────┼──────────────┤
│ indoor    │       104322 │
│ outdoor   │       104231 │
│ incar     │        65157 │
└───────────┴──────────────┘

These are the test result counts for each operator.

WITH a AS (
    SELECT   operator,
             Result_1,
             cnt:     COUNT(*)
    FROM     a
    GROUP BY 1, 2
)
PIVOT    a
ON       operator
USING    SUM(cnt)
GROUP BY Result_1
ORDER BY Result_1;

┌──────────────┬──────────┬────────┬────────┬────────┐
│   Result_1   │ Bouygues │  Free  │ Orange │  SFR   │
│   varchar    │  int128  │ int128 │ int128 │ int128 │
├──────────────┼──────────┼────────┼────────┼────────┤
│ Drop         │      445 │    802 │    479 │    399 │
│ Setup Fail   │       12 │      9 │      8 │      9 │
│ Success      │    56651 │  54544 │  59420 │  56466 │
│ TimeOut      │     3988 │   4868 │   2841 │   4270 │
│ Traffic Fail │     7153 │   8041 │   5934 │   7371 │
└──────────────┴──────────┴────────┴────────┴────────┘

Below are the top 30 operator-url pair counts where there was anything other than a successful test.

SELECT   operator,
         url,
         COUNT(*)
FROM     a
WHERE    Result_1 != 'Success'
AND      url IS NOT NULL
GROUP BY 1, 2
ORDER BY 3 DESC
LIMIT    30;

┌──────────┬──────────────────────────────────────────────────┬──────────────┐
│ operator │                       url                        │ count_star() │
│ varchar  │                     varchar                      │    int64     │
├──────────┼──────────────────────────────────────────────────┼──────────────┤
│ Free     │ https://test-debit7-v6.afdtech.com               │         3146 │
│ SFR      │ https://test-debit7-v6.afdtech.com               │         2977 │
│ Bouygues │ https://test-debit7-v6.afdtech.com               │         2952 │
│ Orange   │ https://test-debit7-v6.afdtech.com               │         2643 │
│ Free     │ https://test-debit7-v6.afdtech.com/250Mo.bin     │         2364 │
│ Free     │ https://test-debit8-v6.afdtech.com/250Mo.bin     │         2168 │
│ SFR      │ https://test-debit7-v6.afdtech.com/250Mo.bin     │         2031 │
│ SFR      │ https://test-debit8-v6.afdtech.com/250Mo.bin     │         1980 │
│ Bouygues │ https://test-debit7-v6.afdtech.com/250Mo.bin     │         1933 │
│ Bouygues │ https://test-debit8-v6.afdtech.com/250Mo.bin     │         1877 │
│ Orange   │ https://test-debit7-v6.afdtech.com/250Mo.bin     │         1492 │
│ Orange   │ https://test-debit8-v6.afdtech.com/250Mo.bin     │         1492 │
│ Free     │ https://www.messervices.etudiant.gouv.fr/envole/ │          298 │
│ Bouygues │ https://www.messervices.etudiant.gouv.fr/envole/ │          289 │
│ Free     │ https://www.marmiton.org/recettes                │          273 │
│ Free     │ https://www.francetravail.fr/accueil/            │          249 │
│ Free     │ https://recherche.lefigaro.fr/recherche/         │          247 │
│ Free     │ https://www.index-education.com/fr/              │          243 │
│ Free     │ https://www.doctolib.fr/                         │          243 │
│ Free     │ https://www.tiktok.com/                          │          241 │
│ SFR      │ https://www.messervices.etudiant.gouv.fr/envole/ │          233 │
│ Orange   │ https://www.messervices.etudiant.gouv.fr/envole/ │          231 │
│ Free     │ https://redditinc.com/                           │          231 │
│ Bouygues │ https://redditinc.com/                           │          226 │
│ SFR      │ https://www.marmiton.org/recettes                │          222 │
│ Free     │ https://www.ubereats.com/fr                      │          217 │
│ Bouygues │ https://www.doctolib.fr/                         │          216 │
│ Bouygues │ https://www.francetravail.fr/accueil/            │          205 │
│ SFR      │ https://www.index-education.com/fr/              │          203 │
│ SFR      │ https://www.tiktok.com/                          │          200 │
└──────────┴──────────────────────────────────────────────────┴──────────────┘

Canadian Wind Farms

2026-03-12T12:00:00+02:00

In November, Natural Resources Canada refreshed their Canadian Wind Turbine Database. It documents the location 7,841 wind turbines, their capabilities and associated project data across Canada.

In this post, I'll walk through converting this dataset into Parquet and examining some of its features.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use jq to help analyse the data in this post.

$ sudo apt update
$ sudo apt install \
    jq

I'll use DuckDB, along with its H3, JSON, Lindel, Parquet and Spatial extensions in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.5.0/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

Most of the maps in this post were rendered with QGIS version 4.0.0. QGIS is a desktop application that runs on Windows, macOS and Linux. The application has grown in popularity in recent years and has ~15M application launches from users all around the world each month.

I used QGIS' HCMGIS plugin to add basemaps from Esri to this post.

Downloading the Dataset

I'll download the dataset and convert it into Parquet.

$ mkdir -p ~/canada_wind
$ cd ~/canada_wind

$ wget https://ftp.cartes.canada.ca/pub/nrcan_rncan/Wind-energy_Energie-eolienne/wind_turbines_database/wind_turbine_database_en.gdb.zip
$ unzip -j wind_turbine_database_en.gdb.zip

$ ~/duckdb

COPY (
    SELECT * EXCLUDE(SHAPE,
                     Longitude,
                     Latitude,
                     Turbine_Rated_Capacity_kW,
                     Total_Project_Capacity_MW,
                     Number_of_Turbines_in_Project,
                     Turbine_Number,
                     Rotor_Diameter_m,
                     Commissioning,
                     Hub_Height_m),
           commissioning:             SPLIT(Commissioning, '/')[1]::INT,
           Turbine_Rated_Capacity_kW: SPLIT(SPLIT(Turbine_Rated_Capacity_kW, '/')[1], '-')[1]::FLOAT,
           Hub_Height_m:              SPLIT(Hub_Height_m, '-')[1]::FLOAT,
           Number_of_Turbines_in_Project:
                                      Number_of_Turbines_in_Project::INT,
           Rotor_Diameter_m:          Rotor_Diameter_m::FLOAT,
           Total_Project_Capacity_MW: Total_Project_Capacity_MW::FLOAT,
           Turbine_Number:            Turbine_Number::INT,

           geometry:                  ST_POINT(Longitude,
                                               Latitude),
           bbox:                      {'xmin': ST_XMIN(ST_EXTENT(geometry)),
                                       'ymin': ST_YMIN(ST_EXTENT(geometry)),
                                       'xmax': ST_XMAX(ST_EXTENT(geometry)),
                                       'ymax': ST_YMAX(ST_EXTENT(geometry))}
    FROM     ST_READ('a00000009.gdbtable')
    ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geometry)),
                             ST_X(ST_CENTROID(geometry))]::double[2])
) TO 'wind.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

The above produced a 316 KB Parquet file containing 7,841 rows.

Data Fluency

The following is an example record from this dataset.

$ echo "FROM 'wind.parquet'
        LIMIT 1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "Hub_Height_m": 50.0,
    "Manufacturer": "Vestas",
    "Model": "V47-660",
    "Notes": "",
    "Number_of_Turbines_in_Project": 16,
    "Project_Name": "North Cape Wind Farm Phase",
    "Province_Territory": "Prince Edward Island",
    "Rotor_Diameter_m": 47.0,
    "Total_Project_Capacity_MW": 10.5600004196167,
    "Turbine_Identifier": "NCW7",
    "Turbine_Number": 7,
    "Turbine_Rated_Capacity_kW": 660.0,
    "bbox": {
      "xmax": -63.9950147,
      "xmin": -63.9950147,
      "ymax": 47.0485158,
      "ymin": 47.0485158
    },
    "commissioning": 2001,
    "geometry": "POINT (-63.9950147 47.0485158)"
  }
]

Below are the field names, data types, percentages of NULLs per column, number of unique values and minimum and maximum values for each column.

$ ~/duckdb

SELECT   column_name,
         column_type[:30],
         null_percentage,
         approx_unique,
         min[:30],
         max[:30]
FROM     (SUMMARIZE
          FROM 'wind.parquet')
ORDER BY LOWER(column_name);

┌───────────────────────────────┬────────────────────────────────┬─────────────────┬───────────────┬────────────────────────────────┬────────────────────────────────┐
│          column_name          │        column_type[:30]        │ null_percentage │ approx_unique │            min[:30]            │            max[:30]            │
│            varchar            │            varchar             │  decimal(9,2)   │     int64     │            varchar             │            varchar             │
├───────────────────────────────┼────────────────────────────────┼─────────────────┼───────────────┼────────────────────────────────┼────────────────────────────────┤
│ bbox                          │ STRUCT(xmin DOUBLE, ymin DOUBL │            0.00 │          8837 │ {'xmin': -138.841952, 'ymin':  │ {'xmin': -52.970072, 'ymin': 4 │
│ commissioning                 │ INTEGER                        │            0.00 │            30 │ 1993                           │ 2025                           │
│ geometry                      │ GEOMETRY('OGC:CRS84')          │            0.00 │          7466 │ POINT (-82.29153133 42.2526119 │ POINT (-107.177635 50.31832261 │
│ Hub_Height_m                  │ FLOAT                          │            0.00 │            62 │ 15.2                           │ 150.0                          │
│ Manufacturer                  │ VARCHAR                        │            0.00 │            35 │ Acciona Wind Power             │ Windmatic                      │
│ Model                         │ VARCHAR                        │            0.00 │           109 │ 25xc                           │ WM15S                          │
│ Notes                         │ VARCHAR                        │            0.00 │            79 │                                │ Vertical Axis Hybrid           │
│ Number_of_Turbines_in_Project │ INTEGER                        │            0.00 │            71 │ 1                              │ 175                            │
│ Project_Name                  │ VARCHAR                        │            0.00 │           294 │ Adelaide Wind Energy Centre    │ Zurich                         │
│ Province_Territory            │ VARCHAR                        │            0.00 │            12 │ Alberta                        │ Yukon                          │
│ Rotor_Diameter_m              │ FLOAT                          │            0.00 │            59 │ 1.93                           │ 170.0                          │
│ Total_Project_Capacity_MW     │ FLOAT                          │            0.00 │           195 │ 0.0                            │ 494.0                          │
│ Turbine_Identifier            │ VARCHAR                        │            0.00 │          8495 │ AAV1                           │ ZUR1                           │
│ Turbine_Number                │ INTEGER                        │            0.00 │           184 │ 1                              │ 175                            │
│ Turbine_Rated_Capacity_kW     │ FLOAT                          │            0.00 │            66 │ 0.0                            │ 6600.0                         │
└───────────────────────────────┴────────────────────────────────┴─────────────────┴───────────────┴────────────────────────────────┴────────────────────────────────┘

Turbines

Below I'll generate a heatmap of turbine installation locations.

$ ~/duckdb

CREATE OR REPLACE TABLE h3_4_stats AS
    SELECT   h3_4: H3_LATLNG_TO_CELL(
                        bbox.ymin,
                        bbox.xmin,
                        4),
             COUNT(*) num_turbines
    FROM     'wind.parquet'
    GROUP BY 1;

COPY (
    SELECT geometry: ST_ASWKB(H3_CELL_TO_BOUNDARY_WKT(h3_4)::geometry),
           num_turbines
    FROM   h3_4_stats
) TO 'turbines.h3_4.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

There are 34 unique manufactures listed in this dataset. These are the ones with the most and least number of turbines installed.

$ ~/duckdb

SELECT   num_turbines: COUNT(*),
         Manufacturer
FROM     'wind.parquet'
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬──────────────────────────┐
│ num_turbines │       Manufacturer       │
│    int64     │         varchar          │
├──────────────┼──────────────────────────┤
│         2173 │ Vestas                   │
│         1726 │ General Electric         │
│         1247 │ Siemens                  │
│         1092 │ Enercon                  │
│          643 │ Senvion                  │
│          491 │ Siemens-Gamesa           │
│          132 │ NEG Micon                │
│          105 │ Nordex                   │
│           74 │ Acciona Wind Power       │
│           50 │ Goldwind                 │
│           16 │ Ghrepower                │
│           15 │ Suzlon                   │
│           12 │ Seaforth                 │
│            9 │ Vensys                   │
│            8 │ Gamesa                   │
│            7 │ Endurance Wind Power     │
│            6 │ Windmatic                │
│            5 │ DeWind                   │
│            5 │ EWT                      │
│            4 │ Samsung Renewable Energy │
│            3 │ Northwind                │
│            3 │ Turbowinds               │
│            2 │ Bonus                    │
│            2 │ America Wind Energy      │
│            2 │ Lagerwey                 │
│            1 │ Pfleiderer               │
│            1 │ Bergey                   │
│            1 │ Tacke                    │
│            1 │ Leitwind                 │
│            1 │ Enercon                  │
│            1 │ Neg Micon                │
│            1 │ Wind Energy Solutions    │
│            1 │ Southwest Windpower      │
│            1 │ Darrieus-Savonius        │
└──────────────┴──────────────────────────┘

Below are the number of turbines by manufacturer commissioned by decade.

WITH a AS (
    SELECT   num_turbines: COUNT(*),
             decade:       (Commissioning / 10)::INT * 10,
             manufacturer: Manufacturer
    FROM     'wind.parquet'
    GROUP BY 2, 3
    ORDER BY 1 DESC
)
PIVOT    a
ON       decade
USING    SUM(num_turbines)
GROUP BY manufacturer
ORDER BY manufacturer;

┌──────────────────────────┬────────┬────────┬────────┬────────┐
│       manufacturer       │  1990  │  2000  │  2010  │  2020  │
│         varchar          │ int128 │ int128 │ int128 │ int128 │
├──────────────────────────┼────────┼────────┼────────┼────────┤
│ Acciona Wind Power       │   NULL │   NULL │     40 │     34 │
│ America Wind Energy      │   NULL │   NULL │      2 │   NULL │
│ Bergey                   │   NULL │   NULL │      1 │   NULL │
│ Bonus                    │      2 │   NULL │   NULL │   NULL │
│ Darrieus-Savonius        │   NULL │   NULL │   NULL │      1 │
│ DeWind                   │   NULL │   NULL │      5 │   NULL │
│ EWT                      │   NULL │   NULL │   NULL │      5 │
│ Endurance Wind Power     │   NULL │   NULL │      3 │      4 │
│ Enercon                  │   NULL │      3 │    796 │    293 │
│ Enercon                  │   NULL │   NULL │   NULL │      1 │
│ Gamesa                   │   NULL │   NULL │      8 │   NULL │
│ General Electric         │   NULL │     20 │   1460 │    246 │
│ Ghrepower                │   NULL │   NULL │      2 │     14 │
│ Goldwind                 │   NULL │   NULL │   NULL │     50 │
│ Lagerwey                 │   NULL │      2 │   NULL │   NULL │
│ Leitwind                 │   NULL │   NULL │      1 │   NULL │
│ NEG Micon                │   NULL │    132 │   NULL │   NULL │
│ Neg Micon                │   NULL │      1 │   NULL │   NULL │
│ Nordex                   │   NULL │     20 │   NULL │     85 │
│ Northwind                │   NULL │   NULL │      3 │   NULL │
│ Pfleiderer               │   NULL │   NULL │      1 │   NULL │
│ Samsung Renewable Energy │   NULL │   NULL │      4 │   NULL │
│ Seaforth                 │   NULL │   NULL │     12 │   NULL │
│ Senvion                  │   NULL │   NULL │    488 │    155 │
│ Siemens                  │   NULL │   NULL │    710 │    537 │
│ Siemens-Gamesa           │   NULL │   NULL │   NULL │    491 │
│ Southwest Windpower      │   NULL │   NULL │      1 │   NULL │
│ Suzlon                   │   NULL │   NULL │     15 │   NULL │
│ Tacke                    │   NULL │      1 │   NULL │   NULL │
│ Turbowinds               │   NULL │      3 │   NULL │   NULL │
│ Vensys                   │   NULL │      1 │      8 │   NULL │
│ Vestas                   │   NULL │    421 │   1239 │    513 │
│ Wind Energy Solutions    │   NULL │   NULL │      1 │   NULL │
│ Windmatic                │   NULL │      6 │   NULL │   NULL │
└──────────────────────────┴────────┴────────┴────────┴────────┘

Below I'll plot out the top manufacturer per hexagon among the overall top ten.

CREATE OR REPLACE TABLE top_manufacturers AS
    SELECT   num_turbines: COUNT(*),
             manufacturer: Manufacturer
    FROM     'wind.parquet'
    GROUP BY 2
    ORDER BY 1 DESC
    LIMIT    10;

CREATE OR REPLACE TABLE h3_3s AS
    WITH b AS (
        WITH a AS (
            SELECT   h3_3: H3_LATLNG_TO_CELL(
                                bbox.ymin,
                                bbox.xmin,
                                3) ,
                     manufacturer: Manufacturer,
                     num_recs: COUNT(*)
            FROM     'wind.parquet'
            WHERE    Manufacturer IN (
                SELECT DISTINCT manufacturer
                FROM   top_manufacturers
            )
            GROUP BY 1, 2
        )
        SELECT *,
               ROW_NUMBER() OVER (PARTITION BY h3_3
                                  ORDER BY     num_recs DESC) AS rn
        FROM   a
    )
    FROM     b
    WHERE    rn = 1
    ORDER BY num_recs DESC;

COPY (
    SELECT geom: H3_CELL_TO_BOUNDARY_WKT(h3_3)::GEOMETRY,
           manufacturer
    FROM   h3_3s
) TO 'turbine.top_manufacturer.h3_3.parquet' (
    FORMAT 'PARQUET',
    CODEC  'ZSTD',
    COMPRESSION_LEVEL 22,
    ROW_GROUP_SIZE 15000);

Below are the provinces and territories ranked by their total GW of installed capacity along with the number of turbines and distinct manufacturers represented.

SELECT   Province_Territory,
         gw:                ROUND(SUM(Turbine_Rated_Capacity_kW / 1000 ** 2), 3),
         num_turbines:      COUNT(*),
         num_manufacturers: COUNT(DISTINCT Manufacturer)
FROM     'wind.parquet'
GROUP BY 1
ORDER BY 2 DESC
LIMIT    20;

┌───────────────────────────┬────────┬──────────────┬───────────────────┐
│    Province_Territory     │   gw   │ num_turbines │ num_manufacturers │
│          varchar          │ double │    int64     │       int64       │
├───────────────────────────┼────────┼──────────────┼───────────────────┤
│ Alberta                   │   5.77 │         1778 │                10 │
│ Ontario                   │   5.55 │         2712 │                14 │
│ Quebec                    │  3.933 │         2005 │                 6 │
│ Saskatchewan              │  0.821 │          277 │                 6 │
│ British Columbia          │  0.747 │          300 │                 5 │
│ Nova Scotia               │  0.622 │          347 │                11 │
│ New Brunswick             │  0.422 │          146 │                 4 │
│ Manitoba                  │  0.258 │          133 │                 2 │
│ Prince Edward Island      │  0.201 │          104 │                 3 │
│ Newfoundland and Labrador │  0.055 │           27 │                 3 │
│ Northwest Territories     │  0.013 │            5 │                 1 │
│ Yukon                     │  0.005 │            7 │                 3 │
└───────────────────────────┴────────┴──────────────┴───────────────────┘

Below shows the ever-increasing capacity of turbines in MW over the decades.

WITH a AS (
    SELECT   decade:       (commissioning / 10)::INT * 10,
             mw:           CEIL(Turbine_Rated_Capacity_kW / 1000),
             num_turbines: COUNT(*)
    FROM     'wind.parquet'
    GROUP BY 1, 2
)
PIVOT    a
ON       decade
USING    SUM(num_turbines)
GROUP BY mw
ORDER BY mw;

┌───────┬────────┬────────┬────────┬────────┐
│  mw   │  1990  │  2000  │  2010  │  2020  │
│ float │ int128 │ int128 │ int128 │ int128 │
├───────┼────────┼────────┼────────┼────────┤
│   0.0 │      1 │     24 │      3 │   NULL │
│   1.0 │      1 │    358 │     37 │     26 │
│   2.0 │   NULL │    226 │   3160 │    469 │
│   3.0 │   NULL │      2 │   1600 │    500 │
│   4.0 │   NULL │   NULL │   NULL │    569 │
│   5.0 │   NULL │   NULL │   NULL │    529 │
│   6.0 │   NULL │   NULL │   NULL │    269 │
│   7.0 │   NULL │   NULL │   NULL │     67 │
└───────┴────────┴────────┴────────┴────────┘

Project Footprints

Each turbine record contains a project name. Below I'll generate the footprints for each project.

I'll group the projects by hexagon and name to generate their geographical footprints. This helps avoid country-wide overlaps that would result from grouping by name alone.

$ ~/duckdb

COPY (
    SELECT   h3_3: H3_LATLNG_TO_CELL(bbox.ymin,
                                     bbox.xmin,
                                     2),
             project_name: Project_Name,
             geometry: {
                'min_x': MIN(ST_X(geometry)),
                'min_y': MIN(ST_Y(geometry)),
                'max_x': MAX(ST_X(geometry)),
                'max_y': MAX(ST_Y(geometry))}::BOX_2D::GEOMETRY
    FROM     READ_PARQUET('wind.parquet')
    GROUP BY 1, 2
) TO 'projects.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

A total of 327 distinct projects were returned. Below are a few of these projects in southern Alberta.

Google Street View in 2026

2026-02-26T20:00:00+02:00

Last year, I came across a dataset documenting Google's global Street View coverage. Each point in this dataset includes the year and month of that point's last capture.

In this post, I'll convert this dataset into Parquet and examine its geospatial patterns.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use DuckDB v1.4.3, along with its H3, JSON, Lindel, Parquet and Spatial extensions, in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.4.3/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

The maps in this post were rendered using QGIS version 3.44. QGIS is a desktop application that runs on Windows, macOS and Linux. The application has grown in popularity in recent years and has ~15M application launches from users all around the world each month.

I used QGIS' HCMGIS plugin to add basemaps from Esri to the maps in this post.

Downloading Emily's JSON Files

The following will download 131 JSON files which are 647 MB uncompressed. These files were last refreshed on December 4th.

$ mkdir -p ~/emily_biz
$ cd ~/emily_biz

$ wget -r -A json https://geo.emily.bz/coverage-dates

Below is an example record from one of the JSON files.

$ jq -S \
    .customCoordinates[0] \
    geo.emily.bz/coverage-dates/aland.json

{
  "extra": {
    "tags": [
      "2009-08"
    ]
  },
  "lat": 60.023421733271704,
  "lng": 20.58331925203021
}

Producing Parquet

Below, I'll create a table in DuckDB and import the data from the JSON files.

$ ~/duckdb street_view.duckdb

CREATE OR REPLACE TABLE street_view (
    geometry   GEOMETRY,
    updated_at DATE);

$ for FILENAME in geo.emily.bz/coverage-dates/*.json; do
      echo $FILENAME
      echo "
          INSERT INTO street_view
              WITH a AS (
                  SELECT UNNEST(customCoordinates) a
                  FROM   READ_JSON('$FILENAME'))
              SELECT geometry:   ST_POINT(a.lng, a.lat),
                     updated_at: (a.extra.tags[-1] || '-01')::DATE
              FROM   a
              WHERE  a.extra.tags[-1] LIKE '2%'" \
          | ~/duckdb street_view.duckdb
  done

I'll then export this table as a spatially-sorted, ZStandard-compressed Parquet file.

$ ~/duckdb street_view.duckdb

COPY (
    FROM     street_view
    ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geometry)),
                             ST_X(ST_CENTROID(geometry))]::double[2])
) TO 'street_view.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

The resulting Parquet file is 85 MB and contains 7,163,407 rows.

Data for Bosnia and Herzegovina, Cyprus, Namibia, Paraguay and Vietnam are missing in this release. Hopefully, they will be available after the next refresh.

Data Fluency

Below are the point counts rounded up to the nearest thousand and broken down by year.

$ ~/duckdb

SELECT   year:  YEAR(updated_at),
         count: CEIL((COUNT(*) / 1000))::INT,
FROM     'street_view.parquet'
GROUP BY 1
ORDER BY 1;

┌───────┬───────┐
│ year  │ count │
│ int64 │ int32 │
├───────┼───────┤
│  2003 │     1 │
│  2006 │     1 │
│  2007 │    28 │
│  2008 │   251 │
│  2009 │   659 │
│  2010 │   344 │
│  2011 │   619 │
│  2012 │   792 │
│  2013 │   622 │
│  2014 │   474 │
│  2015 │   661 │
│  2016 │   529 │
│  2017 │    70 │
│  2018 │   142 │
│  2019 │   195 │
│  2020 │    50 │
│  2021 │   267 │
│  2022 │   507 │
│  2023 │   588 │
│  2024 │   290 │
│  2025 │    83 │
├───────┴───────┤
│    21 rows    │
└───────────────┘

Below is the coverage across Europe. The darker colours are points that were updated closer to 2007 and the brighter colours closer to December of last year.

The following shows India and Southeast Asia's coverage.

The following shows the coverage across Australia and New Zealand.

This is the coverage for North America.

This is the coverage for Latin America and the Caribbean.

Data Crew's Route Optimiser Framework

2026-02-25T10:00:00+02:00

Data Crew is a consultancy that was started last year in Argentina. They've recently released a 1,419-line, Python-based Route Optimiser Framework.

It supports both the Chinese Postman (CPP) and Travelling Salesman (TSP) problems. The framework is meant to be easy to expand and there are plans to add support for Vehicle Routing (VRP), Capacitated VRP (CVRP) and VRP with Time Windows (VRPTW) at a later date.

In this post, I'll examine two example solvers from Data Crew's Route Optimiser Framework.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use Python 3.12.3 and jq to help analyse the data in this post.

$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt update
$ sudo apt install \
    jq \
    python3-pip \
    python3.12-venv

Below, I'll set up a Python Virtual Environment and install some dependencies.

$ python3 -m venv ~/.route-optimizer
$ source ~/.route-optimizer/bin/activate
$ python3 -m pip install \
    contextily \
    folium \
    geopandas \
    matplotlib \
    networkx \
    openpyxl \
    osmnx \
    pandas \
    scikit-learn \
    shapely

I'll clone the repo that contains the framework as well as some example scenarios that I'll walk through below.

$ git clone https://github.com/Data-Crew/route-optimizer \
    ~/route-optimizer

I'll use DuckDB v1.4.3, along with its H3, JSON, Lindel, Parquet and Spatial extensions, in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.4.3/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

Parking Enforcement

The first example is a parking enforcement solver. An inspector needs the shortest path along a circuit that visits every edge at least once.

Below is the configuration file that this solver will call upon. It isolates a bounding box around La Plata, Argentina, a planned city with diagonal avenues, which is to the southeast of Buenos Aires.

This bounding box will be used to isolate the OpenStreetMap (OSM) road network for this area via the OSMnx library.

There are also three zones defined within this area, each with its own unique parking restrictions.

$ cd ~/route-optimizer
$ cat zone_config.json

{
  "configuration": {
    "city": "La Plata, Argentina",
    "bbox": {
      "description": "Bounding box (west, south, east, north)",
      "west": -57.9605,
      "south": -34.9210,
      "east": -57.9455,
      "north": -34.9095
    },
    "start_point": {
      "description": "Start and return point (lat, lon)",
      "name": "Corner of 12th and 50th",
      "latitude": -34.91719,
      "longitude": -57.95067
    }
  },
  "zones": [
    {
      "name": "courthouse",
      "description": "Courthouse Zone",
      "polygon": [
        [-57.957, -34.915],
        [-57.949, -34.915],
        [-57.949, -34.911],
        [-57.957, -34.911]
      ],
      "schedule": {
        "start": "08:00",
        "end": "14:00"
      },
      "weekdays": [0, 1, 2, 3, 4],
      "weekday_names": ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"],
      "prohibited_streets": [],
      "color": "pink"
    },
    {
      "name": "downtown",
      "description": "Downtown Zone",
      "polygon": [
        [-57.955, -34.9145],
        [-57.9475, -34.9145],
        [-57.9475, -34.9185],
        [-57.955, -34.9185]
      ],
      "schedule": {
        "start": "08:00",
        "end": "20:00"
      },
      "weekdays": [0, 1, 2, 3, 4, 5],
      "weekday_names": ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"],
      "prohibited_streets": [],
      "color": "gold"
    },
    {
      "name": "12th_street",
      "description": "12th Street Zone",
      "polygon": [
        [-57.955, -34.9125],
        [-57.952, -34.9125],
        [-57.952, -34.9195],
        [-57.955, -34.9195]
      ],
      "schedule": {
        "start": "08:00",
        "end": "20:00"
      },
      "weekdays": [0, 1, 2, 3, 4, 5],
      "weekday_names": ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"],
      "prohibited_streets": [],
      "color": "violet"
    }
  ],
  "route_types": {
    "full": {
      "name": "Full Route",
      "description": "All zones (Monday to Friday 8am-8pm)",
      "zones": ["courthouse", "downtown", "12th_street"]
    },
    "no_courthouse": {
      "name": "No Courthouse",
      "description": "For enforcement officers working after 2pm",
      "zones": ["downtown", "12th_street"]
    },
    "saturday": {
      "name": "Saturday",
      "description": "Weekend route",
      "zones": ["downtown", "12th_street"]
    }
  }
}

Below, I'll run the solver. This first deliverable is a map of the zone within the bounding box.

$ python3 parking_enforcement.py

======================================================================
 ROUTE OPTIMIZER - PARKING ENFORCEMENT (CPP)
======================================================================
📥 Downloading street network...
✅ Network downloaded: 208 nodes, 369 edges
📍 Start node: 249417335

🏷️  Labeling zones...
   • courthouse: 39 nodes
   • downtown: 34 nodes
   • 12th_street: 27 nodes

📊 Zone summary:
zone
outside        135
courthouse      29
12th_street     27
downtown        17
Name: count, dtype: int64

🗺️  Generating visualization...
💾 Map saved to: output/cpp_zones.png

Then, the full route is calculated and results in a solution that is 11.06 KM long.

======================================================================
 CALCULATING OPTIMAL ROUTES
======================================================================

📍 CASE 1: Full Route

============================================================
🚀 SOLVING ROUTE: FULL (CPP)
============================================================

🔍 Route type 'full':
   • Zones: courthouse, downtown, 12th_street
   • Nodes: 73
   • Edges: 123

🔄 Solving Chinese Postman Problem (CPP)...
   ⚠️  Graph not strongly connected. Extracting main component...
   🔄 Eulerizing graph...
      • 13 unbalanced nodes
      ✅ Eulerized: 139 edges
   🔍 Finding Eulerian circuit...
   ⚠️  Could not create perfect Eulerian circuit
   📍 Using approximation...
      📍 Expanded 11 path segments with intermediate nodes
   ✅ Route found: 119 nodes, 11.09 km

✅ Route calculated (CPP):
   • Nodes visited: 119
   • Total distance: 11.09 km

The next calculation handles the "no courthouse" custom restrictions and produces a solution with a distance of 5.39 KM.

📍 CASE 2: No Courthouse (after 2pm)

============================================================
🚀 SOLVING ROUTE: NO_COURTHOUSE (CPP)
============================================================

🔍 Route type 'no_courthouse':
   • Zones: downtown, 12th_street
   • Nodes: 44
   • Edges: 69

🔄 Solving Chinese Postman Problem (CPP)...
   ⚠️  Graph not strongly connected. Extracting main component...
   🔄 Eulerizing graph...
      • 10 unbalanced nodes
      ✅ Eulerized: 79 edges
   🔍 Finding Eulerian circuit...
   ⚠️  Could not create perfect Eulerian circuit
   📍 Using approximation...
      📍 Expanded 4 path segments with intermediate nodes
   ✅ Route found: 59 nodes, 5.39 km

✅ Route calculated (CPP):
   • Nodes visited: 59
   • Total distance: 5.39 km

The next calculation handles the Saturday route and produces a solution with a distance of 5.39 KM.

📍 CASE 3: Saturday Route

============================================================
🚀 SOLVING ROUTE: SATURDAY (CPP)
============================================================

🔍 Route type 'saturday':
   • Zones: downtown, 12th_street
   • Nodes: 44
   • Edges: 69

🔄 Solving Chinese Postman Problem (CPP)...
   ⚠️  Graph not strongly connected. Extracting main component...
   🔄 Eulerizing graph...
      • 10 unbalanced nodes
      ✅ Eulerized: 79 edges
   🔍 Finding Eulerian circuit...
   ⚠️  Could not create perfect Eulerian circuit
   📍 Using approximation...
      📍 Expanded 4 path segments with intermediate nodes
   ✅ Route found: 59 nodes, 5.39 km

✅ Route calculated (CPP):
   • Nodes visited: 59
   • Total distance: 5.39 km

The results of the three above cases are then saved to an Excel file.

📊 Exporting results...
✅ Results exported to: output/cpp_routes.xlsx

📋 Summary:
   Route_Type  Node_Count  Distance_KM  Distance_Meters
         full         119        11.09            11093
no_courthouse          59         5.39             5392
     saturday          59         5.39             5392

Below is the summary sheet from that file.

$ echo "FROM ST_READ('output/cpp_routes.xlsx')" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "Distance_KM": 11.09,
    "Distance_Meters": 11093,
    "Node_Count": 119,
    "Route_Type": "full"
  },
  {
    "Distance_KM": 5.39,
    "Distance_Meters": 5392,
    "Node_Count": 59,
    "Route_Type": "no_courthouse"
  },
  {
    "Distance_KM": 5.39,
    "Distance_Meters": 5392,
    "Node_Count": 59,
    "Route_Type": "saturday"
  }
]

Each case has its own sheet with its sequence list. Below is the sheet for the Saturday route.

$ ~/duckdb

.maxrows 60

SELECT sequence: Sequence::INT,
       node_id:  Node_ID::BIGINT,
       geom:     ST_POINT(Longitude, Latitude),
       zone:     Zone
FROM   READ_XLSX('output/cpp_routes.xlsx',
                 sheet='saturday');

┌──────────┬────────────┬─────────────────────────────────┬─────────────┐
│ sequence │  node_id   │              geom               │    zone     │
│  int32   │   int64    │            geometry             │   varchar   │
├──────────┼────────────┼─────────────────────────────────┼─────────────┤
│        1 │  249417335 │ POINT (-57.9510606 -34.91725)   │ downtown    │
│        2 │  249417330 │ POINT (-57.9517689 -34.9167244) │ downtown    │
│        3 │  249417040 │ POINT (-57.9527766 -34.9176466) │ 12th_street │
│        4 │  249416478 │ POINT (-57.9538079 -34.9185892) │ 12th_street │
│        5 │ 6127585995 │ POINT (-57.9546872 -34.9193893) │ 12th_street │
│        6 │ 6127586000 │ POINT (-57.9547393 -34.918265)  │ 12th_street │
│        7 │  249416467 │ POINT (-57.9545168 -34.9180625) │ 12th_street │
│        8 │  249416459 │ POINT (-57.9547554 -34.9178853) │ 12th_street │
│        9 │  249418061 │ POINT (-57.954798 -34.9169539)  │ 12th_street │
│       10 │  249417004 │ POINT (-57.9548565 -34.9161046) │ 12th_street │
│       11 │  249418057 │ POINT (-57.9548928 -34.9155776) │ 12th_street │
│       12 │  249417305 │ POINT (-57.9541766 -34.9149376) │ 12th_street │
│       13 │  249417664 │ POINT (-57.9531609 -34.9140082) │ 12th_street │
│       14 │  249417665 │ POINT (-57.9522478 -34.9146926) │ 12th_street │
│       15 │  249417312 │ POINT (-57.9532914 -34.9155946) │ 12th_street │
│       16 │  249417026 │ POINT (-57.9543003 -34.9165164) │ 12th_street │
│       17 │  249417033 │ POINT (-57.9534809 -34.9171231) │ 12th_street │
│       18 │  249417320 │ POINT (-57.9524713 -34.9162031) │ 12th_street │
│       19 │  249417672 │ POINT (-57.9514609 -34.9152824) │ downtown    │
│       20 │  249417675 │ POINT (-57.9507639 -34.9158047) │ downtown    │
│       21 │  249417679 │ POINT (-57.9500587 -34.9163333) │ downtown    │
│       22 │ 1868768411 │ POINT (-57.948899 -34.9171887)  │ downtown    │
│       23 │  249417681 │ POINT (-57.9488464 -34.9172257) │ downtown    │
│       24 │ 2761285280 │ POINT (-57.9478281 -34.9163417) │ downtown    │
│       25 │ 1868768407 │ POINT (-57.9477386 -34.9162624) │ downtown    │
│       26 │ 2714068787 │ POINT (-57.9478392 -34.9161947) │ downtown    │
│       27 │ 2761285279 │ POINT (-57.9479159 -34.9162708) │ downtown    │
│       28 │ 2761285280 │ POINT (-57.9478281 -34.9163417) │ downtown    │
│       29 │ 1868768407 │ POINT (-57.9477386 -34.9162624) │ downtown    │
│       30 │ 2714068787 │ POINT (-57.9478392 -34.9161947) │ downtown    │
│       31 │  249417867 │ POINT (-57.9496921 -34.9147895) │ downtown    │
│       32 │ 2761285275 │ POINT (-57.949781 -34.9148569)  │ downtown    │
│       33 │ 2761285277 │ POINT (-57.9490549 -34.9154107) │ downtown    │
│       34 │ 2761285279 │ POINT (-57.9479159 -34.9162708) │ downtown    │
│       35 │ 1868768411 │ POINT (-57.948899 -34.9171887)  │ downtown    │
│       36 │ 1868768422 │ POINT (-57.9499061 -34.9181067) │ downtown    │
│       37 │  249417335 │ POINT (-57.9510606 -34.91725)   │ downtown    │
│       38 │  249417330 │ POINT (-57.9517689 -34.9167244) │ downtown    │
│       39 │  249417320 │ POINT (-57.9524713 -34.9162031) │ 12th_street │
│       40 │  249417312 │ POINT (-57.9532914 -34.9155946) │ 12th_street │
│       41 │  249417026 │ POINT (-57.9543003 -34.9165164) │ 12th_street │
│       42 │  249418061 │ POINT (-57.954798 -34.9169539)  │ 12th_street │
│       43 │ 6127586001 │ POINT (-57.954951 -34.9170952)  │ 12th_street │
│       44 │ 6127585996 │ POINT (-57.9549152 -34.9177665) │ 12th_street │
│       45 │ 2760842529 │ POINT (-57.9548801 -34.9183932) │ 12th_street │
│       46 │ 6127586000 │ POINT (-57.9547393 -34.918265)  │ 12th_street │
│       47 │  249416467 │ POINT (-57.9545168 -34.9180625) │ 12th_street │
│       48 │  249417033 │ POINT (-57.9534809 -34.9171231) │ 12th_street │
│       49 │  249417040 │ POINT (-57.9527766 -34.9176466) │ 12th_street │
│       50 │  249417046 │ POINT (-57.9520687 -34.9181727) │ 12th_street │
│       51 │  249416487 │ POINT (-57.9530977 -34.9191169) │ 12th_street │
│       52 │  249416478 │ POINT (-57.9538079 -34.9185892) │ 12th_street │
│       53 │  249416467 │ POINT (-57.9545168 -34.9180625) │ 12th_street │
│       54 │  249417033 │ POINT (-57.9534809 -34.9171231) │ 12th_street │
│       55 │  249417320 │ POINT (-57.9524713 -34.9162031) │ 12th_street │
│       56 │  249417672 │ POINT (-57.9514609 -34.9152824) │ downtown    │
│       57 │  249417675 │ POINT (-57.9507639 -34.9158047) │ downtown    │
│       58 │  249417679 │ POINT (-57.9500587 -34.9163333) │ downtown    │
│       59 │  249417335 │ POINT (-57.9510606 -34.91725)   │ downtown    │
├──────────┴────────────┴─────────────────────────────────┴─────────────┤
│ 59 rows                                                     4 columns │
└───────────────────────────────────────────────────────────────────────┘

Images of each case, along with interactive maps, are then generated.

======================================================================
 GENERATING ROUTE VISUALIZATIONS
======================================================================

🔍 Route type 'full':
   • Zones: courthouse, downtown, 12th_street
   • Nodes: 73
   • Edges: 123

🔍 Route type 'no_courthouse':
   • Zones: downtown, 12th_street
   • Nodes: 44
   • Edges: 69

🔍 Route type 'saturday':
   • Zones: downtown, 12th_street
   • Nodes: 44
   • Edges: 69

📸 Generating static images...

🗺️  Visualizing route: CPP: Full Route (All Zones)
💾 Route saved to: output/cpp_route_full.png

🗺️  Visualizing route: CPP: No Courthouse Route (After 2pm)
💾 Route saved to: output/cpp_route_no_courthouse.png

🗺️  Visualizing route: CPP: Saturday Route
💾 Route saved to: output/cpp_route_saturday.png

🌐 Generating interactive maps (open in browser)...

🗺️  Generating interactive map: CPP: Full Route (All Zones)
💾 Interactive map saved to: output/cpp_route_full.html
   Open in browser and use the playback controls!

🗺️  Generating interactive map: CPP: No Courthouse Route (After 2pm)
💾 Interactive map saved to: output/cpp_route_no_courthouse.html
   Open in browser and use the playback controls!

🗺️  Generating interactive map: CPP: Saturday Route
💾 Interactive map saved to: output/cpp_route_saturday.html
   Open in browser and use the playback controls!

======================================================================
 FINAL SUMMARY - CPP (Chinese Postman Problem)
======================================================================

✅ Full Route:      11.09 km (119 nodes)
✅ No Courthouse:   5.39 km (59 nodes)
✅ Saturday:        5.39 km (59 nodes)

📁 Output files (all prefixed with 'cpp_'):

   📸 Static images (with street basemap):
   • cpp_zones.png              - Map of enforcement zones
   • cpp_route_full.png         - Full route
   • cpp_route_no_courthouse.png - Afternoon route
   • cpp_route_saturday.png     - Weekend route

   🌐 Interactive maps (RECOMMENDED - open in browser):
   • cpp_route_full.html        - Full route with playback
   • cpp_route_no_courthouse.html - Afternoon route
   • cpp_route_saturday.html    - Weekend route

   📊 Data:
   • cpp_routes.xlsx            - Route coordinates

💡 TIP: Open the .html files for interactive step-by-step playback!

======================================================================

Below is a capture of the interactive map of the full route. It has been sped up 9x.

Below is the static image generated of the route with no courthouse.

Below is the static image generated of the Saturday route.

Delivery Routing

Below is the second example solver. It finds the shortest possible distance a delivery driver would need to travel in order to deliver ten packages.

$ python3 delivery_route.py

======================================================================
 ROUTE OPTIMIZER - DELIVERY ROUTE (TSP)
======================================================================

📦 Scenario: A delivery driver has 10 packages to deliver.
   Goal: Find the shortest route visiting all addresses.

📥 Downloading street network...
✅ Network downloaded: 208 nodes, 369 edges
📍 Start node: 249417335

🏷️  Labeling zones...
   • delivery_area: 79 nodes

📊 Zone summary:
zone
outside          129
delivery_area     79
Name: count, dtype: int64

📍 Delivery addresses (10 packages):
   📦 Package 1: Node 249417664 (-34.91401, -57.95316)
   📦 Package 2: Node 249417672 (-34.91528, -57.95146)
   📦 Package 3: Node 249417679 (-34.91633, -57.95006)
   📦 Package 4: Node 249417046 (-34.91817, -57.95207)
   📦 Package 5: Node 4771931492 (-34.91434, -57.95498)
   📦 Package 6: Node 249416993 (-34.91586, -57.95518)
   📦 Package 7: Node 249417033 (-34.91712, -57.95348)
   📦 Package 8: Node 249417862 (-34.91428, -57.95036)
   📦 Package 9: Node 249417312 (-34.91559, -57.95329)
   📦 Package 10: Node 249417335 (-34.91725, -57.95106)

   ℹ️  10 addresses → 10 unique stops

======================================================================
 CALCULATING OPTIMAL ROUTE (TSP)
======================================================================

============================================================
🚀 SOLVING ROUTE: FULL (TSP)
============================================================

🔍 Route type 'full':
   • Zones: delivery_area
   • Nodes: 79
   • Edges: 135

🔄 Solving Traveling Salesman Problem (TSP)...
   📍 Nodes to visit: 10
   ⚠️  Graph not strongly connected. Extracting main component...
   📊 Computing distance matrix...
   🔍 Finding optimal tour...
      📍 Expanded 8 path segments with intermediate nodes
   ✅ Route found: 10 stops, 37 total nodes, 3.60 km

✅ Route calculated (TSP):
   • Nodes visited: 37
   • Total distance: 3.60 km

📊 Route summary:
   • Packages: 10
   • Unique stops: 10
   • Route nodes: 37 (including path between stops)
   • Total distance: 3.60 km

An Excel file of the results, along with images and an interactive map are then generated from the above solution.

======================================================================
 GENERATING VISUALIZATIONS
======================================================================

🗺️  Generating visualization...
💾 Map saved to: output/tsp_zone.png

🔍 Route type 'full':
   • Zones: delivery_area
   • Nodes: 79
   • Edges: 135

🗺️  Visualizing route: TSP: Delivery Route (10 packages)
💾 Route saved to: output/tsp_route.png

🗺️  Generating interactive map: TSP: Delivery Route
💾 Interactive map saved to: output/tsp_route.html
   Open in browser and use the playback controls!

📊 Exporting results...
✅ Results exported to: output/tsp_routes.xlsx

📋 Summary:
    Route_Type  Node_Count  Distance_KM  Distance_Meters
delivery_route          37          3.6             3598

======================================================================
 FINAL SUMMARY - TSP (Traveling Salesman Problem)
======================================================================

✅ Delivery route optimized!
   • Packages: 10
   • Distance: 3.60 km
   • Estimated time: ~7 min (at 30 km/h avg)

📁 Output files (all prefixed with 'tsp_'):

   📸 Static images:
   • tsp_zone.png   - Delivery area map
   • tsp_route.png  - Optimized route with basemap

   🌐 Interactive map:
   • tsp_route.html - Route with step-by-step playback

   📊 Data:
   • tsp_routes.xlsx - Route coordinates
======================================================================

Below is a capture of the interactive map. It has been sped up 9x.

Below is the route summary sheet.

$ echo "FROM ST_READ('output/tsp_routes.xlsx')" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "Distance_KM": 3.6,
    "Distance_Meters": 3598,
    "Node_Count": 37,
    "Route_Type": "delivery_route"
  }
]

Below is the sequence list.

SELECT sequence: Sequence::INT,
       node_id:  Node_ID::BIGINT,
       geom:     ST_POINT(Longitude, Latitude),
       zone:     Zone
FROM   READ_XLSX('output/tsp_routes.xlsx',
                 sheet='delivery_route');

┌──────────┬────────────┬─────────────────────────────────┬───────────────┐
│ sequence │  node_id   │              geom               │     zone      │
│  int32   │   int64    │            geometry             │    varchar    │
├──────────┼────────────┼─────────────────────────────────┼───────────────┤
│        1 │  249417335 │ POINT (-57.9510606 -34.91725)   │ delivery_area │
│        2 │  249417330 │ POINT (-57.9517689 -34.9167244) │ delivery_area │
│        3 │  249417320 │ POINT (-57.9524713 -34.9162031) │ delivery_area │
│        4 │  249417672 │ POINT (-57.9514609 -34.9152824) │ delivery_area │
│        5 │ 2761285274 │ POINT (-57.9504434 -34.9143554) │ delivery_area │
│        6 │  249417862 │ POINT (-57.950361 -34.9142803)  │ delivery_area │
│        7 │  249417857 │ POINT (-57.9511569 -34.913688)  │ delivery_area │
│        8 │  249417851 │ POINT (-57.9520626 -34.9130139) │ delivery_area │
│        9 │  249417849 │ POINT (-57.9529888 -34.9123245) │ delivery_area │
│       10 │ 2761285268 │ POINT (-57.9530811 -34.9123952) │ delivery_area │
│       11 │  249417663 │ POINT (-57.9540895 -34.9133121) │ delivery_area │
│       12 │  249417664 │ POINT (-57.9531609 -34.9140082) │ delivery_area │
│       13 │  249417665 │ POINT (-57.9522478 -34.9146926) │ delivery_area │
│       14 │  249417312 │ POINT (-57.9532914 -34.9155946) │ delivery_area │
│       15 │  249417026 │ POINT (-57.9543003 -34.9165164) │ delivery_area │
│       16 │  249417033 │ POINT (-57.9534809 -34.9171231) │ delivery_area │
│       17 │  249417320 │ POINT (-57.9524713 -34.9162031) │ delivery_area │
│       18 │  249417312 │ POINT (-57.9532914 -34.9155946) │ delivery_area │
│       19 │  249417305 │ POINT (-57.9541766 -34.9149376) │ delivery_area │
│       20 │ 4771931492 │ POINT (-57.9549762 -34.9143443) │ delivery_area │
│       21 │ 6040327689 │ POINT (-57.9551144 -34.9142417) │ delivery_area │
│       22 │ 2760842520 │ POINT (-57.9550254 -34.9157142) │ delivery_area │
│       23 │ 2760842522 │ POINT (-57.9550061 -34.9159896) │ delivery_area │
│       24 │  249417004 │ POINT (-57.9548565 -34.9161046) │ delivery_area │
│       25 │  249417026 │ POINT (-57.9543003 -34.9165164) │ delivery_area │
│       26 │  249417033 │ POINT (-57.9534809 -34.9171231) │ delivery_area │
│       27 │  249417040 │ POINT (-57.9527766 -34.9176466) │ delivery_area │
│       28 │  249417046 │ POINT (-57.9520687 -34.9181727) │ delivery_area │
│       29 │  249416487 │ POINT (-57.9530977 -34.9191169) │ delivery_area │
│       30 │  249416478 │ POINT (-57.9538079 -34.9185892) │ delivery_area │
│       31 │  249416467 │ POINT (-57.9545168 -34.9180625) │ delivery_area │
│       32 │  249417033 │ POINT (-57.9534809 -34.9171231) │ delivery_area │
│       33 │  249417320 │ POINT (-57.9524713 -34.9162031) │ delivery_area │
│       34 │  249417672 │ POINT (-57.9514609 -34.9152824) │ delivery_area │
│       35 │  249417675 │ POINT (-57.9507639 -34.9158047) │ delivery_area │
│       36 │  249417679 │ POINT (-57.9500587 -34.9163333) │ delivery_area │
│       37 │  249417335 │ POINT (-57.9510606 -34.91725)   │ delivery_area │
├──────────┴────────────┴─────────────────────────────────┴───────────────┤
│ 37 rows                                                       4 columns │
└─────────────────────────────────────────────────────────────────────────┘

72M Points of Interest

2026-02-02T20:00:00+02:00

Overture Map's publish a Places dataset along with each of their monthly releases. Places can be thought of as points of interest (POIs). As of January, they have over 72 million of these POIs covering the world.

Below is an example from Viru Street in Tallinn, Estonia. Each point is labelled with its primary name and basic category in brackets. The point colour is based on the POI's basic category.

Overture has made several improvements to the Places dataset in recent months. In September, they added operating status and confidence properties. In October, they added a basic category property and in December, a taxonomy property was added.

In this post, I'll examine their latest Places release.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use Python 3.12.3 and a few other tools to help analyse the data in this post.

$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt update
$ sudo apt install \
    jq \
    python3-pip \
    python3.12-venv

Below, I'll set up a Python Virtual Environment.

$ python3 -m venv ~/.pois
$ source ~/.pois/bin/activate

I'll use a Parquet debugging tool I've been working on to see how much space each column takes up in one of the Parquet files.

$ git clone https://github.com/marklit/pqview \
    ~/pqview

$ python3 -m pip install \
          -r ~/pqview/requirements.txt

I'll also install the latest AWS CLI.

$ python3 -m pip install awscli

I'll use DuckDB v1.4.3, along with its H3, JSON, Lindel, Parquet and Spatial extensions, in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.4.3/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

I used QGIS' Tile+ and HCMGIS plugins to add basemaps from Bing and Esri to the maps in this post.

Downloading Overture's Places

As of last September, Overture will remove releases after 60 days. The Places URL used in this post will stop working sometime in March.

Below lists Overture's available releases. Use the latest release version in the S3 URL below to download this dataset.

$ curl https://labs.overturemaps.org/data/releases.json

{
    "latest": "2026-01-21.0",
    "releases": [
        "2026-01-21.0",
        "2025-12-17.0"
    ]
}

$ aws s3 --no-sign-request sync \
    s3://overturemaps-us-west-2/release/2026-01-21.0/theme=places/type=place/ \
    ~/places

The above downloaded 8 Parquet files with a total disk footprint of 7.2 GB.

Data Fluency

This dataset contains 72,444,739 records.

$ ~/duckdb

SELECT COUNT(*)
FROM   'places/part*.parquet';

┌─────────────────┐
│  count_star()   │
│      int64      │
├─────────────────┤
│    72444739     │
│ (72.44 million) │
└─────────────────┘

Below is a heatmap showing where the POIs are most heavily concentrated.

CREATE OR REPLACE TABLE h3_3_stats AS
    SELECT   h3_3: H3_LATLNG_TO_CELL(
                       bbox.ymin,
                       bbox.xmin,
                       3),
             num_pois: COUNT(*)
    FROM     'places/part*.parquet'
    GROUP BY 1;

COPY (
    SELECT geometry: ST_ASWKB(H3_CELL_TO_BOUNDARY_WKT(h3_3)::geometry),
           num_pois
    FROM   h3_3_stats
    WHERE  ST_XMIN(geometry::geometry) BETWEEN -179 AND 179
    AND    ST_XMAX(geometry::geometry) BETWEEN -179 AND 179
) TO 'num_pois.h3_3_stats.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

Below is an example record from this dataset.

$ echo "SELECT * EXCLUDE(addresses,
                         bbox,
                         brand,
                         categories,
                         names,
                         taxonomy,
                         sources),
               addresses:  addresses::JSON,
               bbox:       bbox::JSON,
               brand:      brand::JSON,
               categories: categories::JSON,
               names:      names::JSON,
               taxonomy:   taxonomy::JSON,
               sources:    sources::JSON
        FROM   'places/part*.parquet'
        WHERE  brand.names.primary = 'Starbucks'
        LIMIT  1" \
       | ~/duckdb -json \
       | jq -S .

[
  {
    "addresses": [
      {
        "country": "CL",
        "freeform": "Juan Soler Manfredini 131, local 302 y 302B, Mall Paseo Costanera Puerto Montt",
        "locality": "Puerto Montt",
        "postcode": "5504750",
        "region": null
      }
    ],
    "basic_category": "cafe",
    "bbox": {
      "xmax": -72.9016342163086,
      "xmin": -72.90164184570312,
      "ymax": -41.48642349243164,
      "ymin": -41.48643112182617
    },
    "brand": {
      "names": {
        "common": null,
        "primary": "Starbucks",
        "rules": null
      },
      "wikidata": null
    },
    "categories": {
      "alternate": null,
      "primary": "cafe"
    },
    "confidence": 0.32347911067676516,
    "emails": null,
    "geometry": "POINT (-72.90164 -41.48643)",
    "id": "72da3584-2530-4bac-b062-124393ecdb0c",
    "names": {
      "common": null,
      "primary": "Starbucks Chile",
      "rules": null
    },
    "operating_status": "open",
    "phones": "[+56232621864]",
    "socials": "['https://www.facebook.com/299687346558972']",
    "sources": [
      {
        "between": null,
        "confidence": 0.3234791106767652,
        "dataset": "meta",
        "license": "CDLA-Permissive-2.0",
        "property": "",
        "record_id": "299687346558972",
        "update_time": "2025-12-01T08:00:00.000Z"
      },
      {
        "between": null,
        "confidence": null,
        "dataset": "Overture",
        "license": "CDLA-Permissive-2.0",
        "property": "/properties/confidence",
        "record_id": null,
        "update_time": "2026-01-13T21:57:48Z"
      }
    ],
    "taxonomy": {
      "alternates": null,
      "hierarchy": [
        "food_and_drink",
        "casual_eatery",
        "cafe"
      ],
      "primary": "cafe"
    },
    "theme": "places",
    "type": "place",
    "version": 5,
    "websites": null
  }
]

Below is a breakdown of how much space each column takes up relative to the others. This dataset only uses point geometry so its geometry field footprint is much smaller than what you'd find in datasets with linestrings and polygons.

The columns using up more space tend to have fewer NULL values and overall better coverage. The exception to this would be something like the basic category column which has ~400 unique values and compresses very well.

$ python3 ~/pqview/main.py \
    types \
    --html \
    places/part-00000-faac29ab-3031-41b7-836a-26e1193347c0-c000.zstd.parquet \
    > places.types.html

Below are the column types, NULL-value ratios, number of unique values and the lowest and highest value from each field.

$ ~/duckdb

SELECT   column_name,
         column_type[:30],
         null_percentage,
         approx_unique,
         min[:30],
         max[:30]
FROM     (SUMMARIZE
          FROM 'places/part*.parquet')
ORDER BY 1;

┌──────────────────┬────────────────────────────────┬─────────────────┬───────────────┬───────────────────────────────────┬────────────────────────────────────────┐
│   column_name    │        column_type[:30]        │ null_percentage │ approx_unique │             min[:30]              │                max[:30]                │
│     varchar      │            varchar             │  decimal(9,2)   │     int64     │              varchar              │                varchar                 │
├──────────────────┼────────────────────────────────┼─────────────────┼───────────────┼───────────────────────────────────┼────────────────────────────────────────┤
│ addresses        │ STRUCT(freeform VARCHAR, local │            0.00 │      45527236 │ [{'freeform': '', 'locality':     │ [{'freeform': NULL, 'locality'         │
│ basic_category   │ VARCHAR                        │            0.00 │           405 │ accommodation                     │ zoo                                    │
│ bbox             │ STRUCT(xmin FLOAT, xmax FLOAT, │            0.00 │      68675209 │ {'xmin': -180.0, 'xmax': -180.    │ {'xmin': 180.0, 'xmax': 180.0,         │
│ brand            │ STRUCT(wikidata VARCHAR, "name │           93.56 │         89854 │ {'wikidata': Q100146251, 'name    │ {'wikidata': NULL, 'names': {'         │
│ categories       │ STRUCT("primary" VARCHAR, alte │            0.00 │       2625563 │ {'primary': 3d_printing_servic    │ {'primary': zoo, 'alternate':          │
│ confidence       │ DOUBLE                         │            0.00 │        222625 │ 0.0                               │ 1.0                                    │
│ emails           │ VARCHAR[]                      │           97.32 │       1494892 │ ['']                              │ [合羽090-6252-0042kakaricho@char       │
│ geometry         │ GEOMETRY                       │            0.00 │      65426272 │ POINT (2 28)                      │ POINT (-6.240234374999999 53.3         │
│ id               │ VARCHAR                        │            0.00 │      67893339 │ 00000057-00a2-439b-9581-cf9e24    │ ffffffe2-6947-4705-ad9b-936fab         │
│ names            │ STRUCT("primary" VARCHAR, comm │            0.00 │      64313486 │ {'primary': \b음성 A, 'common': N │ {'primary': 𩻸魚堀溪觀魚步道, 'common' │
│ operating_status │ VARCHAR                        │            0.00 │             3 │ closed                            │ temporarily closed                     │
│ phones           │ VARCHAR[]                      │           16.58 │      54450308 │ ['']                              │ [‎+34 951 51 92 20]                     │
│ socials          │ VARCHAR[]                      │           15.59 │      63756430 │ [' https://www.facebook.com/Jo    │ [www.motrio.com]                       │
│ sources          │ STRUCT(property VARCHAR, datas │            0.00 │      65629291 │ [{'property': '', 'dataset': A    │ [{'property': '', 'dataset': m         │
│ taxonomy         │ STRUCT("primary" VARCHAR, hier │            0.00 │          1718 │ {'primary': 3d_printing_servic    │ {'primary': zoo, 'hierarchy':          │
│ version          │ INTEGER                        │            0.00 │             9 │ 1                                 │ 8                                      │
│ websites         │ VARCHAR[]                      │           39.07 │      39341765 │ ['']                              │ ['้http://www.krungsri.com']            │
├──────────────────┴────────────────────────────────┴─────────────────┴───────────────┴───────────────────────────────────┴────────────────────────────────────────┤
│ 17 rows                                                                                                                                                6 columns │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Sources

Overture has 8 outside sources they collect data from and combine to build this dataset.

$ ~/duckdb

SELECT COUNT(*),
       a.dataset
FROM (
    SELECT UNNEST(sources) a
    FROM   'places/part*.parquet'
)
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬──────────────┐
│ count_star() │   dataset    │
│    int64     │   varchar    │
├──────────────┼──────────────┤
│     72444739 │ Overture     │
│     58958274 │ meta         │
│      6003562 │ Foursquare   │
│      5644707 │ Microsoft    │
│      1554249 │ AllThePlaces │
│       148042 │ DAC          │
│       127862 │ PinMeTo      │
│         4844 │ RenderSEO    │
│         3199 │ Krick        │
└──────────────┴──────────────┘

In researching the team behind Places, I came across this LinkedIn post from Dana Bauer at Overture where she was looking for someone to join TomTom, one of the Overture Foundation's partner firms, to help her team work on this dataset.

Operating Status

I suspect the operating status data is in its early stages. A lot of businesses collapse in the first year and I expect in future releases that the closed count will grow from where it is now.

$ ~/duckdb

SELECT   COUNT(*),
         operating_status
FROM     'places/part*.parquet'
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬────────────────────┐
│ count_star() │  operating_status  │
│    int64     │      varchar       │
├──────────────┼────────────────────┤
│     72443934 │ open               │
│          785 │ closed             │
│           20 │ temporarily closed │
└──────────────┴────────────────────┘

Version

Records contain version counts. Version 5 is the most common in this dataset at the moment.

$ ~/duckdb

CREATE OR REPLACE TABLE h3_5s AS
    WITH b AS (
        WITH a AS (
            SELECT   H3_LATLNG_TO_CELL(bbox.ymin,
                                       bbox.xmin,
                                       5) h3_5,
                     version,
                     COUNT(*) num_recs
            FROM     'places/part*.parquet'
            GROUP BY 1, 2
        )
        SELECT *,
               ROW_NUMBER() OVER (PARTITION BY h3_5
                                  ORDER BY     num_recs DESC) AS rn
        FROM   a
    )
    FROM     b
    WHERE    rn = 1
    ORDER BY num_recs DESC;

COPY (
    SELECT geometry: H3_CELL_TO_BOUNDARY_WKT(h3_5)::GEOMETRY,
           version
    FROM   h3_5s
    WHERE  ST_XMIN(geometry::geometry) BETWEEN -179 AND 179
    AND    ST_XMAX(geometry::geometry) BETWEEN -179 AND 179
) TO 'version.h3_5_stats.parquet' (
    FORMAT 'PARQUET',
    CODEC  'ZSTD',
    COMPRESSION_LEVEL 22,
    ROW_GROUP_SIZE 15000);

SELECT   version,
         COUNT(*)
FROM     'version.h3_5_stats.parquet'
GROUP BY 1
ORDER BY 1;

┌─────────┬──────────────┐
│ version │ count_star() │
│  int32  │    int64     │
├─────────┼──────────────┤
│       1 │          645 │
│       2 │        12272 │
│       3 │        28228 │
│       4 │        31962 │
│       5 │       137824 │
│       6 │        80749 │
│       7 │         7962 │
│       8 │          412 │
└─────────┴──────────────┘

Brand Names

Of the 72M+ records, ~4M contain a brand name. Below are the most common ones.

$ ~/duckdb

SELECT   COUNT(*),
         brand.names.primary
FROM     'places/part*.parquet'
GROUP BY 2
ORDER BY 1 DESC
LIMIT    25;

┌──────────────┬─────────────────────────────┐
│ count_star() │           primary           │
│    int64     │           varchar           │
├──────────────┼─────────────────────────────┤
│     68001883 │ NULL                        │
│        57163 │ Citibank                    │
│        56977 │ Wildberries                 │
│        45430 │ Western Union               │
│        42517 │ Shell                       │
│        42036 │ McDonald's                  │
│        33750 │ Amazon Locker               │
│        33117 │ サントリー                  │
│        31789 │ Subway                      │
│        30119 │ Starbucks                   │
│        26846 │ 7-Eleven                    │
│        26509 │ LibertyX Bitcoin ATM        │
│        20088 │ KFC                         │
│        19844 │ Dollar General              │
│        18152 │ Burger King                 │
│        16320 │ Pizza Hut                   │
│        15583 │ InPost                      │
│        15035 │ Domino's Pizza              │
│        14169 │ Indian Oil Corporation Ltd. │
│        13487 │ Enterprise                  │
│        13422 │ OXXO                        │
│        12769 │ Lidl                        │
│        12719 │ bp                          │
│        12366 │ Honda                       │
│        12251 │ TotalEnergies               │
├──────────────┴─────────────────────────────┤
│ 25 rows                          2 columns │
└────────────────────────────────────────────┘

There are 44K McDonald's locations in the world so to have 42K is pretty good coverage.

Taxonomy & Hierarchy

Below are the most common values for the new taxonomy.hierarchy field that was added late last year.

$ ~/duckdb

SELECT   COUNT(*),
         taxonomy.hierarchy
FROM     'places/part*.parquet'
GROUP BY 2
ORDER BY 1 DESC
LIMIT    25;

┌──────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ count_star() │                                                          hierarchy                                                          │
│    int64     │                                                          varchar[]                                                          │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│      2011362 │ [food_and_drink, restaurant]                                                                                                │
│      1393271 │ [lifestyle_services, beauty_service, beauty_salon]                                                                          │
│      1339177 │ [services_and_business, professional_service]                                                                               │
│      1302723 │ [lodging, hotel]                                                                                                            │
│      1141151 │ [shopping]                                                                                                                  │
│      1027893 │ [shopping, food_and_beverage_store, grocery_store]                                                                          │
│      1023589 │ [shopping, fashion_and_apparel_store, clothing_store]                                                                       │
│      1015254 │ [cultural_and_historic, architectural_landmark]                                                                             │
│       926424 │ [cultural_and_historic, religious_organization, place_of_worship, christian_place_of_worshop]                               │
│       895567 │ [travel_and_transportation, automotive_and_ground_transport, automotive, automotive_services_and_repair, automotive_repair] │
│       808932 │ [services_and_business, professional_service, event_planning]                                                               │
│       730509 │ [food_and_drink, bar]                                                                                                       │
│       715677 │ [education, school]                                                                                                         │
│       711235 │ [food_and_drink, casual_eatery, cafe]                                                                                       │
│       706174 │ [services_and_business, real_estate]                                                                                        │
│       703978 │ [community_and_government, community_service]                                                                               │
│       686114 │ [food_and_drink, beverage_shop, coffee_shop]                                                                                │
│       652453 │ [travel_and_transportation, automotive_and_ground_transport, fueling_station, gas_station]                                  │
│       631680 │ [services_and_business, real_estate, real_estate_agent]                                                                     │
│       589904 │ [sports_and_recreation, sports_and_recreation_venue, gym]                                                                   │
│       578091 │ [health_care, dentist]                                                                                                      │
│       558413 │ [health_care, hospital]                                                                                                     │
│       553915 │ [lifestyle_services, beauty_service, hair_salon]                                                                            │
│       546472 │ [shopping, specialty_store, pharmacy]                                                                                       │
│       535707 │ [food_and_drink, casual_eatery, bakery]                                                                                     │
├──────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 25 rows                                                                                                                          2 columns │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

There are 13 top-level categories.

SELECT   COUNT(*),
         taxonomy.hierarchy[1]
FROM     'places/part*.parquet'
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬───────────────────────────┐
│ count_star() │   taxonomy.hierarchy[1]   │
│    int64     │          varchar          │
├──────────────┼───────────────────────────┤
│     17110369 │ services_and_business     │
│     13743976 │ shopping                  │
│     10518826 │ food_and_drink            │
│      4657642 │ health_care               │
│      4416975 │ travel_and_transportation │
│      4198338 │ lifestyle_services        │
│      3460090 │ education                 │
│      3177980 │ community_and_government  │
│      3166644 │ cultural_and_historic     │
│      2874162 │ sports_and_recreation     │
│      2595527 │ lodging                   │
│      1521803 │ arts_and_entertainment    │
│      1002407 │ geographic_entities       │
├──────────────┴───────────────────────────┤
│ 13 rows                        2 columns │
└──────────────────────────────────────────┘

There can be anywhere between one and six levels for any given category.

SELECT   COUNT(*),
         LENGTH(taxonomy.hierarchy)
FROM     'places/part*.parquet'
GROUP BY 2
ORDER BY 2;

┌──────────────┬────────────────────────────┐
│ count_star() │ length(taxonomy.hierarchy) │
│    int64     │           int64            │
├──────────────┼────────────────────────────┤
│      3512828 │                          1 │
│     20617150 │                          2 │
│     31707834 │                          3 │
│     11822674 │                          4 │
│      3608857 │                          5 │
│      1175396 │                          6 │
└──────────────┴────────────────────────────┘

Below is a starburst chart of the first and second levels broken down by record count.

$ echo "SELECT   COUNT(*),
                 taxonomy.hierarchy[1],
                 taxonomy.hierarchy[2]
        FROM     'places/part*.parquet'
        GROUP BY 2, 3
        ORDER BY 1 DESC" \
    | ~/duckdb -csv -noheader \
    > ~/hierarchy.csv

$ cd ~/pqview
$ python3

from main import render_sunburst


stats = [{'top':    rec[1],
          'second': 'Other' if rec[2] == 'NULL' else rec[2],
          'value':  int(rec[0])}
         for rec in [line.strip().split(',')
                     for line in open('/home/mark/hierarchy.csv')]]

html = render_sunburst(stats,
                       parent_key='top',
                       child_key='second',
                       group_under=500_000)

open('places.starburst.html', 'w').write(html)

Maritime POIs

There are a large number of POIs that are over open ocean and seas in this post. I'm not sure how many of these have been placed in error.

Marine Regions has GeoPackage files delineating maritime boundaries. Seven of the nine files contain polygons that I'll use to determine the waters a given POI is in.

Below are the GeoPackage file sizes.

$ cd ~
$ ls -lh ~/marineregions_org/*.gpkg

..  69M .. eez_12nm_v4.gpkg
..  60M .. eez_24nm_v4.gpkg
..  77M .. eez_internal_waters_v4.gpkg
..  16M .. eez_boundaries_v12.gpkg
.. 157M .. eez_v12.gpkg
..  39M .. eez_archipelagic_waters_v4.gpkg
.. 2.1M .. ecs_boundaries_v01.gpkg
.. 7.0M .. ecs_v01.gpkg
.. 8.8M .. High_Seas_v1.gpkg

Below is a rendering of this dataset's geometry and names for the Baltic Sea.

I'll import the GeoPackage files into a DuckDB table.

$ echo "CREATE OR REPLACE TABLE marineregions (
            name VARCHAR,
            geom GEOMETRY);" \
    | ~/duckdb places.duckdb

$ for FILENAME in ~/marineregions_org/High_Seas_v1.gpkg; do
    echo "INSERT INTO marineregions
            SELECT name,
                   geom
            FROM   ST_READ('$FILENAME');" \
        | ~/duckdb places.duckdb
  done

$ for FILENAME in ~/marineregions_org/{ecs_v01,eez_12nm_v4,eez_24nm_v4,eez_archipelagic_waters_v4,eez_internal_waters_v4,eez_v12}*.gpkg; do
    echo $FILENAME
    echo "INSERT INTO marineregions
            SELECT name: GEONAME,
                   geom
            FROM   ST_READ('$FILENAME');" \
        | ~/duckdb places.duckdb
  done

There are 1,006 records in the Marine Regions table in DuckDB. I'll group Overture's 72M+ records into ~914K hexagons. This will give me a quick and rough estimate of how many POIs could be over open oceans and seas.

$ ~/duckdb places.duckdb

CREATE OR REPLACE TABLE h3_stats AS
    SELECT   hexagon:
                H3_LATLNG_TO_CELL(
                    bbox.ymin,
                    bbox.xmin,
                    6) ,
             num_pois: COUNT(*)
    FROM     'places/part*.parquet'
    GROUP BY 1;

SELECT SUM(num_pois)
FROM   h3_stats a
JOIN   marineregions b
    ON ST_Overlaps(H3_CELL_TO_BOUNDARY_WKT(a.hexagon)::GEOMETRY,
                   b.geom)
WHERE  b.name IS NOT NULL;

┌─────────────────┐
│  sum(num_pois)  │
│     int128      │
├─────────────────┤
│    29590753     │
│ (29.59 million) │
└─────────────────┘

The above shows over 29M+ POIs over open oceans and seas. This number won't be super exact. Some hexagons could cover both bodies of water along with a crowded coastline. Also, a cafe on a beach might have its position located in the neighbouring body of water due to inaccuracies in GPS. Nonetheless, 29M shouldn't be too far from an accurate figure.

For reference, this is what the 1,006 Marine Regions polygons look like rendered on a map. They cover just about every major body of water but also have a wide buffer with Antarctica.

COPY (
    SELECT geom,
           name
    FROM   marineregions
) TO 'marineregions.parquet' (
    FORMAT 'PARQUET',
    CODEC  'ZSTD',
    COMPRESSION_LEVEL 22,
    ROW_GROUP_SIZE 15000);

Confidence

Below are the most common confidence values across this dataset. The brighter the hexagon the higher the confidence.

$ ~/duckdb

SELECT   COUNT(*),
         confidence: (confidence*10)::INT / 10
FROM     'places/part*.parquet'
GROUP BY 2
ORDER BY 2;

┌──────────────┬────────────┐
│ count_star() │ confidence │
│    int64     │   double   │
├──────────────┼────────────┤
│         4083 │        0.0 │
│         4149 │        0.1 │
│         3624 │        0.2 │
│     11248354 │        0.3 │
│      1492960 │        0.4 │
│         7243 │        0.5 │
│     11360452 │        0.6 │
│         7405 │        0.7 │
│     10562751 │        0.8 │
│      1522626 │        0.9 │
│     36231092 │        1.0 │
├──────────────┴────────────┤
│ 11 rows         2 columns │
└───────────────────────────┘

Below, I'll build a map of the most common confidence value for each H3 zoom-level 5 hexagon.

CREATE OR REPLACE TABLE h3_5s AS
    WITH b AS (
        WITH a AS (
            SELECT   H3_LATLNG_TO_CELL(bbox.ymin,
                                       bbox.xmin,
                                       5) h3_5,
                     confidence: (confidence*10)::INT / 10,
                     COUNT(*) num_recs
            FROM     'places/part*.parquet'
            GROUP BY 1, 2
        )
        SELECT *,
               ROW_NUMBER() OVER (PARTITION BY h3_5
                                  ORDER BY     num_recs DESC) AS rn
        FROM   a
    )
    FROM     b
    WHERE    rn = 1
    ORDER BY num_recs DESC;

COPY (
    SELECT geometry: H3_CELL_TO_BOUNDARY_WKT(h3_5)::GEOMETRY,
           confidence
    FROM   h3_5s
    WHERE  ST_XMIN(geometry::geometry) BETWEEN -179 AND 179
    AND    ST_XMAX(geometry::geometry) BETWEEN -179 AND 179
) TO 'confidence.h3_5_stats.parquet' (
    FORMAT 'PARQUET',
    CODEC  'ZSTD',
    COMPRESSION_LEVEL 22,
    ROW_GROUP_SIZE 15000);

Below, I'll use larger hexagons to make generalising the confidence scores over open oceans easier.

CREATE OR REPLACE TABLE h3_3s AS
    WITH b AS (
        WITH a AS (
            SELECT   H3_LATLNG_TO_CELL(bbox.ymin,
                                       bbox.xmin,
                                       3) h3_3,
                     confidence: (confidence*10)::INT / 10,
                     COUNT(*) num_recs
            FROM     'places/part*.parquet'
            GROUP BY 1, 2
        )
        SELECT *,
               ROW_NUMBER() OVER (PARTITION BY h3_3
                                  ORDER BY     num_recs DESC) AS rn
        FROM   a
    )
    FROM     b
    WHERE    rn = 1
    ORDER BY num_recs DESC;

COPY (
    SELECT geometry: H3_CELL_TO_BOUNDARY_WKT(h3_3)::GEOMETRY,
           confidence
    FROM   h3_3s
    WHERE  ST_XMIN(geometry::geometry) BETWEEN -179 AND 179
    AND    ST_XMAX(geometry::geometry) BETWEEN -179 AND 179
) TO 'confidence.h3_3_stats.parquet' (
    FORMAT 'PARQUET',
    CODEC  'ZSTD',
    COMPRESSION_LEVEL 22,
    ROW_GROUP_SIZE 15000);

The most common confidence value over the oceans and seas is 0.6. This matches most of African's landmass as well as that of the Middle East's and China's.

Microsoft's 2026 Global ML Building Footprints

2026-01-30T20:00:00+02:00

Since 2014, Microsoft has published GlobalMLBuildingFootprints, a building footprint and height database covering much of the world. It's constructed from running machine learning algorithms against satellite and other aerial imagery. The imagery has been sourced from providers such as Maxar (now re-branded as Vantor), Airbus, Vexcel and IGN France.

The dataset is made up of 1.4B records and has a disk footprint of 113 GB across 30,344 GZIP-compressed, CSV files.

I previously wrote about this dataset in October 2024 but since then, there have been six documented updates and the entire dataset was re-built and re-published two weeks ago.

In this post, I'll examine their latest release.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use DuckDB v1.4.3, along with its H3, JSON, Lindel, Parquet and Spatial extensions, in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.4.3/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

I used QGIS' Tile+ and HCMGIS plugins to add basemaps from Bing and Esri to the maps in this post.

Thousands of CSVs

Below, I'll download the manifest of the 30,344 CSV files that make up this dataset.

$ mkdir -p ~/ms_buildings_2026
$ cd ~/ms_buildings_2026

$ wget https://minedbuildings.z5.web.core.windows.net/global-buildings/dataset-links.csv

Below is an example record from the manifest.

$ echo "FROM     'dataset-links.csv'
        ORDER BY UploadDate DESC
        LIMIT    1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "Location": "Abyei",
    "QuadKey": "122320113",
    "Size": "74.6KB",
    "UploadDate": "2026-01-15",
    "Url": "https://minedbuildings.z5.web.core.windows.net/global-buildings/2026-01-09/global-buildings.geojsonl/RegionName=Abyei/quadkey=122320113/part-00108-66ec874b-f074-4991-9da4-67716003c6cd.c000.csv.gz"
  }
]

I'll import this manifest into DuckDB and clean up its fields.

$ ~/duckdb

CREATE OR REPLACE TABLE mlb AS
    SELECT *,
           region: SPLIT_PART(SPLIT_PART(Url, '/', 7), '=', 2),
           num_bytes:
                CASE
                WHEN Size LIKE '%KB' THEN ((REPLACE(Size, 'KB', '')::FLOAT) * 1024)::BIGINT
                WHEN Size LIKE '%MB' THEN ((REPLACE(Size, 'MB', '')::FLOAT) * 1024 ** 2)::BIGINT
                WHEN Size LIKE '%B'  THEN ((REPLACE(Size, 'B',  ''))::FLOAT)::BIGINT
                ELSE Size::BIGINT END,
           unique_key: UUIDv4()
    FROM 'dataset-links.csv';

The manifest describes the dataset as having a footprint of 113 GB across 30,344 files, all with a publication date of 2026-01-15.

SELECT gb: SUM(num_bytes) / 1024 ** 3
FROM   mlb;

┌────────────────────┐
│         gb         │
│       double       │
├────────────────────┤
│ 113.94016721565276 │
└────────────────────┘

SELECT   UploadDate,
         COUNT(*)
FROM     mlb
GROUP BY 1;

┌────────────┬──────────────┐
│ UploadDate │ count_star() │
│    date    │    int64     │
├────────────┼──────────────┤
│ 2026-01-15 │    30344     │
└────────────┴──────────────┘

I'll build a BASH script that will download each CSV file into its own unique filename.

COPY (
    SELECT 'wget -c -O source/'
            || unique_key
            || '.csv.gz "'
            || Url
            || '"'
    FROM   mlb
) TO 'manifest.bash' (
        HEADER False,
        QUOTE '');

The following will download the 30,344 GZIP-compressed CSV files with four concurrent downloads at any one time.

$ mkdir -p source

$ cat manifest.bash \
    | xargs -P4 \
            -I% \
            bash -c "%"

CSV to Parquet

This dataset suffers from a small-file problem. The largest files are 100s of MB while the smallest are 100s of bytes.

$ ls -lS source/*.csv.gz | head

306627976 ... source/cdcae05e-e44b-4c1c-a571-0c315cc99cad.csv.gz
202647620 ... source/cee4b164-fca8-42f6-995d-f78df92a9d6f.csv.gz
190078342 ... source/5d4c71dc-58d2-4be9-8873-02ff15387b97.csv.gz
186396633 ... source/4c477651-366c-469f-86fd-321597e75f40.csv.gz
178589273 ... source/5683e3c5-07cf-4527-b412-743f55dfcf56.csv.gz
177762037 ... source/e2edc6da-368f-41bf-bf81-911b1393b896.csv.gz
176717777 ... source/f6dd29c9-be49-4b45-9b25-138281668829.csv.gz
173960772 ... source/7524bfe1-c314-40bb-b9c0-35418ba01f8c.csv.gz
172921655 ... source/5dd97452-8e58-4be8-ba1f-a8a13c339669.csv.gz
167773711 ... source/de07c8a4-a4f2-4c1c-9de1-fb1c107402bc.csv.gz

$ ls -lS source/*.csv.gz | tail

182 ... source/670f7c64-553a-481e-920e-9acbbae98a9d.csv.gz
182 ... source/a8e9994c-a016-481a-9062-a1209aaec5ef.csv.gz
181 ... source/d5709426-efab-404c-bb0c-3f0849283fc8.csv.gz
178 ... source/271643a9-5700-44d4-a82e-9d089bb5b10d.csv.gz
177 ... source/1ae077ff-615b-472f-9bad-2418b55aab17.csv.gz
177 ... source/dcc3f082-9da2-45ea-917f-ceea0e841e28.csv.gz
172 ... source/1f9f324e-9ad4-4af9-94f3-f00fbb9715a7.csv.gz
170 ... source/41afa18a-d3c0-49f3-85d9-5d34a7e24f5f.csv.gz
168 ... source/4e1ca57b-3aac-49ad-ad7b-ef6c3dcacb0f.csv.gz
163 ... source/9c03ce06-df35-4ddc-9ef5-c59fb71b3ef6.csv.gz

While these files were downloading, I converted each of them into Parquet, cleaning up their height and confidence values in the process. The confidence values were binned to the nearest 10th.

$ mkdir -p parquet

$ while :
      do
      for FILENAME in source/*.csv.gz; do
          echo $FILENAME

          BASENAME=`basename $FILENAME | cut -d. -f1`
          OUTPUT="parquet/$BASENAME.parquet"

          if [ ! -f $OUTPUT ]; then
              gunzip -c $FILENAME > working.csv

              echo "CREATE OR REPLACE TABLE buildings AS
                        SELECT height:     a.properties.height::FLOAT,
                               confidence: a.properties.confidence::FLOAT,
                               geometry:   ST_GEOMFROMGEOJSON(a.geometry)
                        FROM (SELECT a: column0::JSON
                              FROM   READ_CSV('working.csv',
                                              header=false,
                                              columns={'column0': 'VARCHAR'}));

                    COPY(
                        SELECT   height:     IF(height='-1.0',     NULL, height::FLOAT),
                                 confidence: IF(confidence='-1.0', NULL, (confidence::FLOAT * 10)::INT / 10),
                                 geometry,
                                 bbox:       {'xmin': ST_XMIN(ST_EXTENT(geometry)),
                                              'ymin': ST_YMIN(ST_EXTENT(geometry)),
                                              'xmax': ST_XMAX(ST_EXTENT(geometry)),
                                              'ymax': ST_YMAX(ST_EXTENT(geometry))}
                        FROM     buildings
                        WHERE    ST_Y(ST_CENTROID(geometry)) IS NOT NULL
                        ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geometry)),
                                                 ST_X(ST_CENTROID(geometry))]::double[2])
                  ) TO '$OUTPUT' (
                      FORMAT 'PARQUET',
                      CODEC  'ZSTD',
                      COMPRESSION_LEVEL 22,
                      ROW_GROUP_SIZE 15000);
                  " | ~/duckdb
          fi
      done

      echo "Finished a pass of the manifest file. Sleeping for 5 minutes.."
      sleep 300
  done

I then merged the 30,344 resulting Parquet files into 16 larger files.

$ mkdir -p merged

$ for PREFIX in `ls parquet/ | cut -c1 | sort | uniq`; do
      echo 'Merging Prefix:', $PREFIX, \
           'file(s):',        `ls parquet/$PREFIX* | wc -l`

      echo "COPY(
                FROM     'parquet/$PREFIX*.parquet'
                WHERE    ST_Y(ST_CENTROID(geometry)) IS NOT NULL
                ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geometry)),
                                         ST_X(ST_CENTROID(geometry))]::double[2])
            ) TO 'merged/$PREFIX.parquet' (
                FORMAT 'PARQUET',
                CODEC  'ZSTD',
                COMPRESSION_LEVEL 22,
                ROW_GROUP_SIZE 15000);" \
        | ~/duckdb
  done

The above maxed out my CPU and my RAM's high water mark usage hit ~56 GB.

The above produced 16 Parquet files with a combined 1,409,848,189 records and a disk footprint of 109 GB.

$ ~/duckdb

SELECT COUNT(*)
FROM   'merged/*.parquet';

┌────────────────┐
│  count_star()  │
│     int64      │
├────────────────┤
│   1409848189   │
│ (1.41 billion) │
└────────────────┘

Below is an example record of the resulting dataset.

$ echo "SELECT   * EXCLUDE(bbox),
                 bbox: bbox::JSON
        FROM     'merged/*.parquet'
        WHERE    height     IS NOT NULL
        AND      confidence IS NOT NULL
        LIMIT    1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "bbox": {
      "xmax": 0.0000429153442383,
      "xmin": 0.0,
      "ymax": 45.01591632361135,
      "ymin": 45.015844272478645
    },
    "confidence": 0.9,
    "geometry": "POLYGON ((0 45.015844272478645, 0.000042915344238 45.015844272478645, 0.000042915344238 45.01591632361135, 0 45.01591632361135, 0 45.015844272478645))",
    "height": 1.0493422
  }
]

The above only managed to shave a few GB off of the deliverable Microsoft published. Floating-point values are the toughest data type for compression codecs to recognise patterns within and thus often produce poor compression ratios. This dataset is entirely made up of floating-point values so even with sorting it was hard to improve on the GZIP-compressed CSV disk footprint.

The bounding boxes do add some weight to the dataset but will help avoid the need to read the entire dataset when looking for specific geographies. In previous posts, I've only needed a few MB of bandwidth to pull data for entire cities out of 100 GB+, S3-hosted, Parquet-based, spatially-sorted and ZStandard-compressed datasets.

The statistics collected every 15K rows means some queries can be answered by looking at a few MB of metadata rather than the underlying GBs of data.

Turning 30K+ files into 16 means the file system overheads will be greatly reduced. Also, fewer files and arguably larger row-group sizes offer a good cost optimisation for request-metered CDNs like Cloudflare.

Buildings Heatmap

Below is a heatmap of the buildings across the Globe.

$ ~/duckdb

CREATE OR REPLACE TABLE h3_3_stats AS
    SELECT   h3_3: H3_LATLNG_TO_CELL(
                       bbox.ymin,
                       bbox.xmin,
                       3),
             num_locations: COUNT(*)
    FROM     'merged/*.parquet'
    GROUP BY 1;

COPY (
    SELECT geometry: ST_ASWKB(H3_CELL_TO_BOUNDARY_WKT(h3_3)::geometry),
           num_locations
    FROM   h3_3_stats
    WHERE  ST_XMIN(geometry::geometry) BETWEEN -179 AND 179
    AND    ST_XMAX(geometry::geometry) BETWEEN -179 AND 179
) TO 'merged.h3_3_stats.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

Building Heights

225M buildings in this dataset contain height information.

$ ~/duckdb

SELECT   COUNT(*),
         height IS NULL
FROM     'merged/*.parquet'
GROUP BY 2;

┌──────────────┬──────────────────┐
│ count_star() │ (height IS NULL) │
│    int64     │     boolean      │
├──────────────┼──────────────────┤
│    225418753 │ false            │
│   1184429436 │ true             │
└──────────────┴──────────────────┘

The brightest hexagons have the greatest ratio of buildings with height information.

CREATE OR REPLACE TABLE h3_3_stats AS
      SELECT   h3_3: H3_LATLNG_TO_CELL(
                             bbox.ymin,
                             bbox.xmin,
                             3),
               null_ratio: ROUND(100 /
                                 COUNT(*) *
                                 COUNT(*) FILTER (height IS NULL), 1)
      FROM     'merged/*.parquet'
      GROUP BY 1;

COPY (
    SELECT geometry: ST_ASWKB(H3_CELL_TO_BOUNDARY_WKT(h3_3)::geometry),
           null_ratio
    FROM   h3_3_stats
    WHERE  ST_XMIN(geometry::geometry) BETWEEN -179 AND 179
    AND    ST_XMAX(geometry::geometry) BETWEEN -179 AND 179
) TO 'height_null_ratio.h3_3_stats.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

Below are the hexagon counts by null ratio, binned to the nearest 10th.

SELECT   null_ratio_bin: round(null_ratio / 10) * 10,
         COUNT(*)
FROM     h3_3_stats
GROUP BY null_ratio_bin
ORDER BY 1;

┌────────────────┬──────────────┐
│ null_ratio_bin │ count_star() │
│     double     │    int64     │
├────────────────┼──────────────┤
│            0.0 │          318 │
│           10.0 │          372 │
│           20.0 │          292 │
│           30.0 │          260 │
│           40.0 │          202 │
│           50.0 │          158 │
│           60.0 │          131 │
│           70.0 │          187 │
│           80.0 │          180 │
│           90.0 │          116 │
│          100.0 │         6996 │
├────────────────┴──────────────┤
│ 11 rows             2 columns │
└───────────────────────────────┘

Confidence Levels

334M Buildings in this dataset have a confidence level attached to their record.

$ ~/duckdb

SELECT   COUNT(*),
         confidence IS NULL
FROM     'merged/*.parquet'
GROUP BY 2;

┌──────────────┬──────────────────────┐
│ count_star() │ (confidence IS NULL) │
│    int64     │       boolean        │
├──────────────┼──────────────────────┤
│   1065597051 │ true                 │
│    344251138 │ false                │
└──────────────┴──────────────────────┘

Below are the most common confidence values for locations with them broken down by hexagons at zoom-level 5.

CREATE OR REPLACE TABLE h3_5s AS
    WITH b AS (
        WITH a AS (
            SELECT   H3_LATLNG_TO_CELL(bbox.ymin,
                                       bbox.xmin,
                                       5) h3_5,
                     confidence,
                     COUNT(*) num_recs
            FROM     'merged/*.parquet'
            GROUP BY 1, 2
        )
        SELECT *,
               ROW_NUMBER() OVER (PARTITION BY h3_5
                                  ORDER BY     num_recs DESC) AS rn
        FROM   a
    )
    FROM     b
    WHERE    rn = 1
    ORDER BY num_recs DESC;

COPY (
    SELECT geometry: H3_CELL_TO_BOUNDARY_WKT(h3_5)::GEOMETRY,
           confidence
    FROM   h3_5s
    WHERE  ST_XMIN(geometry::geometry) BETWEEN -179 AND 179
    AND    ST_XMAX(geometry::geometry) BETWEEN -179 AND 179
) TO 'confidence.h3_5_stats.parquet' (
    FORMAT 'PARQUET',
    CODEC  'ZSTD',
    COMPRESSION_LEVEL 22,
    ROW_GROUP_SIZE 15000);

Below are the hexagon counts by confidence value. In almost all cases, where there is a confidence level, it's above 90%.

SELECT   confidence,
         COUNT(*)
FROM     h3_5s
GROUP BY confidence
ORDER BY 1;

┌────────────┬──────────────┐
│ confidence │ count_star() │
│   double   │    int64     │
├────────────┼──────────────┤
│        0.2 │            1 │
│        0.4 │            1 │
│        0.5 │            6 │
│        0.6 │            6 │
│        0.7 │           10 │
│        0.8 │           52 │
│        0.9 │         8748 │
│        1.0 │        67702 │
│       NULL │       233423 │
└────────────┴──────────────┘

It's a shame the confidence level couldn't be applied more stringently or allow for more nuance beyond most records having no confidence level at all.

Detection Edge Cases

Below, I'll go over some of the strange detection edge cases I've come across in this dataset. This shouldn't suggest these problems are widespread, just that when there is an issue with a building's footprint being detected, these are some of the ways they're manifested.

Below, at least six apartment buildings in Albania are detected as a single building. I don't know what imagery Microsoft used for this detection but the latest imagery on Bing has these buildings captured off-nadir in a way that lines up well with the detection. These buildings aren't very tall by global standards and even they suffer from the sides of the buildings being included in the building footprint.

Some of the pieces of furniture laid out on the Albanian beach below are detected as buildings.

Larger buildings might not be completely detected and may be left with unnatural shapes.

There can be footprints that overlap one another.

Below, there are maritime vessels being mistaken for buildings.

The Omani hydrocarbon-maritime infrastructure below was mistaken for buildings.

Some of the wind turbines in this off-shore Wind Farm in Vietnam were mistaken for buildings.

Some of the rocks along this shoreline in Thailand were mistaken for buildings.

This highway in Algeria was mistaken for a building.

I suspect Microsoft have a shape regularisation algorithm they run their detections through in order to create more plausible building shapes. It appears to not support curved buildings, like this one in France, very well.

Millions of Locations for Thousands of Brands

2025-12-08T20:00:00+02:00

All The Places has a set of spiders and scrapers that extract location information from thousands of brands' websites.

Their GitHub repo is made up of 145K lines of Python with commits going back ten years. There have been a total of 162 contributors to date.

Their crawlers run weekly and the collected data is published shortly afterwards.

In this post, I'll examine their latest release.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use Python 3.12.3 and a few other tools to help analyse the data in this post.

$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt update
$ sudo apt install \
    jq \
    python3-pip \
    python3.12-venv

I'll set up a Python Virtual Environment.

$ python3 -m venv ~/.atp
$ source ~/.atp/bin/activate

I'll use DuckDB v1.4.1, along with its H3, JSON, Lindel, Parquet and Spatial extensions, in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.4.1/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

I used QGIS' Tile+ plugin to add basemaps from Bing to the maps in this post.

Thousands of GeoJSON Files

The following will download a 1.4 GB ZIP file that contains 4,367 GeoJSON files totalling 14 GB when uncompressed.

$ mkdir -p ~/atp
$ cd ~/atp

$ wget -c https://alltheplaces-data.openaddresses.io/runs/2025-11-29-13-32-39/output.zip
$ unzip -j output.zip

The GeoJSON files are built by individual Python scripts which often can be country- or region-specific depending on the brand's online structure. Below are the files for Nando's.

$ ls -l nandos*

122771 .. nandos.geojson
 16497 .. nandos_ae.geojson
  6822 .. nandos_bh.geojson
 16634 .. nandos_bw.geojson
555223 .. nandos_gb_ie.geojson
 17450 .. nandos_in.geojson
  3904 .. nandos_mu.geojson
 39668 .. nandos_my.geojson
  3663 .. nandos_om.geojson
 10247 .. nandos_qa.geojson
  3740 .. nandos_sg.geojson
     0 .. nandos_us.geojson
308489 .. nandos_za.geojson
 14091 .. nandos_zm.geojson
 15739 .. nandos_zw.geojson

Roughly 500 of the 1,200+ Nando's locations are in either the UK or the Republic of Ireland. Below is an example record from their gb_ie file.

$ echo "FROM  ST_READ('nandos_gb_ie.geojson')
        LIMIT 1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "@source_uri": "https://www.nandos.co.uk/restaurants/yate",
    "@spider": "nandos_gb_ie",
    "addr:city": "Yate, Gloucestershire",
    "addr:country": "GB",
    "addr:postcode": "BS37 4AS",
    "addr:state": "England",
    "addr:street_address": "Unit R2, Yate Shopping Centre, Link Rd",
    "amenity": "restaurant",
    "branch": "Yate",
    "brand": "Nando's",
    "brand:wikidata": "Q3472954",
    "contact:facebook": "https://www.facebook.com/Nandos.UnitedKingdom",
    "contact:twitter": "NandosUK",
    "cuisine": "chicken;portuguese",
    "geom": "POINT (-2.408309 51.539583)",
    "id": "K8QmVBoxykViF1_30Vo-VcOrrto=",
    "image": "https://www.nandos.co.uk/replatform/restaurants/img/default-restaurant-hero-background.png",
    "name": "Nando's",
    "nsi_id": "nandos-32ebee",
    "opening_hours": "Mo-Su 11:30-22:00",
    "payment:cash": "yes",
    "payment:credit_cards": "yes",
    "payment:debit_cards": "yes",
    "phone": "+44 1454 312504",
    "ref": "https://www.nandos.co.uk/restaurants/yate#restaurant",
    "website": "https://www.nandos.co.uk/restaurants/yate"
  }
]

Unique Brands

Below, I tried to get a rough idea of how many brands are represented in this release.

There were a few hundred brands with zero-byte GeoJSON files that I excluded. Many files ended with a single 2-digit country identifier; others, like the above Nando's example, had more than one. For the sake of this post's timebox, I accepted that margin of error.

$ python3

from   glob import glob
import os


names = [x.split('.')[0]
         for x in glob('*.geojson')
         if os.path.getsize(x)]
brands = set()

for name in names:
    # WIP: nandos_gb_ie won't get handled properly.
    if len(name.split('_')[-1]) == 2:
        brands.add('_'.join(name.split('_')[:1]))
    else:
        brands.add(name)

Without the zero-byte filter, there are 3,070 unique brands, but with the filter, there are 2,716.

There were over 900 open issues in the GitHub repository at the time of this writing. I didn't uncover many common themes between issues outside of Cloudflare becoming more popular among brands, while being unfriendly with their crawlers.

Perhaps it's possible that those empty GeoJSON files could have their datasets repopulated at some point down the line.

The Nando's GeoJSON file for the US in this release was zero bytes in size. The site itself is hosted directly from AWS. I suspect either the site has had a refresh or there was a network failure when the crawler ran.

I'm sure the crawler has functioned properly in the past. Publishing the last successful crawl's dataset instead of the latest, and potentially empty, dataset would be a good stopgap.

Location Metadata

Below, I'll count how many times each field name has been used across this release. There were seven GeoJSON files that Python was unable to parse, so they've been excluded.

$ python3

from   glob import glob
import json
import os


key_counts = {}

for filename in glob('*.geojson'):
    if os.path.getsize(filename) < 1:
        continue

    try:
        for key in json.loads(open(filename).read())\
                        ['features'][0]['properties'].keys():
            if key not in key_counts:
                key_counts[key] = 1
            else:
                key_counts[key] = key_counts[key] + 1
    except json.JSONDecodeError:
        print('Invalid JSON:', filename)

A total of 712 unique field names were seen in this release.

len(key_counts) # 712

Below are the top 25 field names.

sorted(key_counts, key=lambda x: x[1])[:25]

['name',
 'camera:type',
 'man_made',
 'name:en',
 'name:zh-Hans',
 'name:zh-Hant',
 'name:bg',
 'name:mk',
 'name:sq',
 'cash_out',
 'name:ja',
 'natural',
 'taxon:en',
 'gambling',
 'takeaway',
 'map',
 'camera:direction',
 'payment:american_express',
 'payment:apple_pay',
 'payment:cash',
 'payment:discover_card',
 'payment:mastercard',
 'payment:visa',
 'fax',
 'name:sr']

GeoJSON to Parquet

There already is a Parquet distribution, but it has been down since at least October. I'll run 24 concurrent processes that will convert the GeoJSON files into Parquet.

$ ls *.geojson \
    | xargs -P24 \
            -I% \
            bash -c "
    BASENAME=\`echo \"%\" | cut -f1 -d.\`
    echo \"Building \$BASENAME.\"
    echo \"COPY (FROM ST_READ('%'))
           TO '\$BASENAME.parquet' (
               FORMAT 'PARQUET',
               CODEC  'ZSTD',
               COMPRESSION_LEVEL 22,
               ROW_GROUP_SIZE 15000);\" \
        | ~/duckdb"

The following will merge the Parquet files into a single file.

$ ~/duckdb

COPY(
    SELECT   name,
             geometry: geom,
             bbox:     {'xmin': ST_XMIN(ST_EXTENT(geom)),
                        'ymin': ST_YMIN(ST_EXTENT(geom)),
                        'xmax': ST_XMAX(ST_EXTENT(geom)),
                        'ymax': ST_YMAX(ST_EXTENT(geom))},
             * EXCLUDE(name, geom)
    FROM     READ_PARQUET('*.parquet',
                          union_by_name=True,
                          filename=True)
    WHERE    ST_Y(ST_CENTROID(geom)) IS NOT NULL
    ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geom)),
                             ST_X(ST_CENTROID(geom))]::double[2])
) TO '~/atp.parquet' (
    FORMAT 'PARQUET',
    CODEC  'ZSTD',
    COMPRESSION_LEVEL 22,
    ROW_GROUP_SIZE 15000);

The resulting Parquet file is 1.4 GB and contains 19,059,494 records.

Locations Heatmap

Below is a heatmap of the locations in this release. The brighter hexagons have more locations.

$ ~/duckdb

CREATE OR REPLACE TABLE h3_3_stats AS
    SELECT   h3_3: H3_LATLNG_TO_CELL(
                       bbox.ymin,
                       bbox.xmin,
                       3),
             num_locations: COUNT(*)
    FROM     '~/atp.parquet'
    GROUP BY 1;

COPY (
    SELECT geometry: ST_ASWKB(H3_CELL_TO_BOUNDARY_WKT(h3_3)::geometry),
           num_locations
    FROM   h3_3_stats
    WHERE  ST_XMIN(geometry::geometry) BETWEEN -179 AND 179
    AND    ST_XMAX(geometry::geometry) BETWEEN -179 AND 179
) TO '~/atp.h3_3_stats.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

There are a lot of locations located in bodies of water and all across Antarctica.

These are the most common brand names.

SELECT   COUNT(*),
         name
FROM     '~/atp.parquet'
WHERE    LENGTH(name)
GROUP BY 2
ORDER BY 1 DESC
LIMIT    40;

┌──────────────┬──────────────────────────────────┐
│ count_star() │               name               │
│    int64     │             varchar              │
├──────────────┼──────────────────────────────────┤
│        68227 │ Wildberries                      │
│        55171 │ サントリー                       │
│        39731 │ Shell                            │
│        26767 │ 7-Eleven                         │
│        24920 │ Coinstar                         │
│        23271 │ McDonald's                       │
│        21495 │ Starbucks                        │
│        20504 │ Dollar General                   │
│        20380 │ Punjab National Bank             │
│        19519 │ FedEx Drop Box                   │
│        13760 │ Walgreens                        │
│        13660 │ HDFC Bank ATM                    │
│        13346 │ Burger King                      │
│        13050 │ Circle K                         │
│        13028 │ Chevron                          │
│        13006 │ KFC                              │
│        12670 │ FedEx OnSite                     │
│        12279 │ Żabka                            │
│        12074 │ InPost                           │
│        11652 │ TotalEnergies                    │
│        10993 │ HDFC Bank                        │
│        10772 │ Aldi                             │
│        10612 │ Şok                              │
│         9993 │ BP                               │
│         9993 │ Lidl                             │
│         9435 │ Dunkin'                          │
│         9292 │ Dollar Tree                      │
│         9080 │ Bitcoin Depot                    │
│         8568 │ ICICI Bank                       │
│         8424 │ Taco Bell                        │
│         8395 │ 星巴克                           │
│         8193 │ Bank of Baroda                   │
│         7751 │ Verizon                          │
│         7700 │ CVS Pharmacy                     │
│         7677 │ Subway                           │
│         7577 │ T-Mobile                         │
│         7403 │ AutoZone                         │
│         7367 │ H&R Block Tax Preparation Office │
│         7339 │ Family Dollar                    │
│         7318 │ Blue Rhino                       │
├──────────────┴──────────────────────────────────┤
│ 40 rows                               2 columns │
└─────────────────────────────────────────────────┘

There are 2.36M unique strings in the name column. I was expecting this number to be much close to the ~3-4K brands there are scrapers for.

SELECT COUNT(DISTINCT name)
FROM   '~/atp.parquet'; -- 2,358,505

I'll build a bounding box that surrounds each brand's locations in each country they're represented in.

COPY (
    SELECT   name,
             country: "addr:country",
             geometry: {
                'min_x': MIN(ST_X(ST_CENTROID(geometry))),
                'min_y': MIN(ST_Y(ST_CENTROID(geometry))),
                'max_x': MAX(ST_X(ST_CENTROID(geometry))),
                'max_y': MAX(ST_Y(ST_CENTROID(geometry)))}::BOX_2D::GEOMETRY
    FROM     '~/atp.parquet'
    GROUP BY 1, 2
) TO '~/atp.extent_per_country.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

Below are the countries with Nando's locations in this release. Note, the US locations are missing.

Below are the Starbucks locations. I've been to several Starbucks locations in the UAE and Thailand that aren't present in this release. After I made this map, I found an additional 8,395 Starbucks locations with the the Chinese name "星巴克".

For some reason, only 11 countries have at least one McDonalds restaurant in this release.

SELECT COUNT(DISTINCT "addr:country")
FROM   '~/atp.parquet'
WHERE  name LIKE '%McDonalds%'; -- 11

There are 44K McDonald's restaurants in the world but this release only lists 23,271 of them.

Spot Checking

Despite wide IKEA coverage, Estonia's Flagship store is missing.

There is a Romanian Library just off the coast of one of Estonia's Islands.

A Finnish fast food chain placed its restaurant outside of an Estonian shopping centre rather than inside.

Better Boundaries

2025-12-02T20:00:00+02:00

In this post, I'll look at several level 0 administrative boundary datasets.

In some cases, their geometry is very accurate and takes a good deal of effort to find errors. In other cases, they've used course outlines and I'll share examples of these.

These datasets also vary in how many countries they outline in the world as well as the amount of metadata they store on each of them. I'll collect these values and show an example record from each of these datasets.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use Python 3.12.3 and a few other tools to help analyse the data in this post.

$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt update
$ sudo apt install \
    jq \
    python3-pip \
    python3.12-venv

I'll set up a Python Virtual Environment and a JavaScript interpreter for Python.

$ python3 -m venv ~/.boundaries
$ source ~/.boundaries/bin/activate
$ python -m pip install \
    awscli

I'll use DuckDB v1.4.1, along with its H3, JSON, Lindel, Parquet and Spatial extensions, in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.4.1/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

I used QGIS' Tile+ plugin to add basemaps from Bing to the maps in this post.

OpenStreetMap

OpenStreetMap (OSM) now has a Parquet dump of its dataset refreshed weekly and hosted via Cloudflare.

$ mkdir ~/osm
$ cd ~/osm

Below is the boundaries Parquet file. It's 1.8 GB and contains 796,070 records.

$ wget -c https://data.openstreetmap.us/layercake/boundaries.parquet

Below is an example record.

$ echo "SELECT * EXCLUDE(geometry,
                         bbox,
                         tags),
               bbox::JSON bbox,
               tags::JSON tags
        FROM   'boundaries.parquet'
        LIMIT  1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "bbox": {
      "xmax": -175.5414276123047,
      "xmin": -177.24468994140625,
      "ymax": -43.3108024597168,
      "ymin": -44.63499450683594
    },
    "id": 2647558,
    "tags": {
      "ISO3166-1:alpha2": null,
      "ISO3166-1:alpha3": null,
      "ISO3166-2": "NZ-CIT",
      "admin_level": "4",
      "alt_name": null,
      "alt_names": {},
      "border_type": null,
      "boundary": "administrative",
      "claimed_by": null,
      "controlled_by": null,
      "disputed_by": null,
      "int_name": null,
      "name": [
        "Chatham Islands"
      ],
      "names": {
        "cs": [
          "Chathamské ostrovy"
        ],
        "en": [
          "Chatham Islands"
        ],
        "ja": [
          "チャタム諸島"
        ],
        "mi": [
          "Wharekauri"
        ],
        "mk": [
          "Четемски Острови"
        ],
        "rrm": [
          "Rēkohu"
        ],
        "ru": [
          "Архипелаг Чатем"
        ]
      },
      "official_name": null,
      "official_names": {},
      "place": null,
      "recognized_by": null,
      "type": null,
      "wikidata": "Q115459",
      "wikipedia": "en:Chatham Islands"
    },
    "type": "relation"
  }
]

Below are the counts of the different admin levels contained within this dataset. OSM has had 20M volunteers over the years so normalisation of its attributes is an ongoing challenge.

$ ~/duckdb

SELECT   COUNT(*),
         tags.admin_level
FROM     'boundaries.parquet'
GROUP BY 2
ORDER BY 2;

┌──────────────┬─────────────────────────┐
│ count_star() │       admin_level       │
│    int64     │         varchar         │
├──────────────┼─────────────────────────┤
│           12 │ 1                       │
│       234135 │ 10                      │
│            1 │ 10-                     │
│         8883 │ 11                      │
│          352 │ 12                      │
│            1 │ 13                      │
│           14 │ 14                      │
│            1 │ 17                      │
│            1 │ 18                      │
│            1 │ 1o                      │
│          816 │ 2                       │
│          201 │ 3                       │
│         4519 │ 4                       │
│         6538 │ 5                       │
│        50190 │ 6                       │
│        35348 │ 7                       │
│       252801 │ 8                       │
│       175427 │ 9                       │
│            1 │ Bairro                  │
│            1 │ Cabecera Municipal      │
│            1 │ Cadastramento de Bairro │
│            1 │ Desa                    │
│            1 │ Eclesiastico            │
│            1 │ El Tinto                │
│            1 │ Municipalité            │
│            1 │ Neighborhood            │
│            1 │ RW/8                    │
│            5 │ Village                 │
│            4 │ neighborhood            │
│            1 │ nieghborhood            │
│            1 │ suburb                  │
│            1 │ هجرة الجيبان            │
│        26808 │ NULL                    │
├──────────────┴─────────────────────────┤
│ 33 rows                      2 columns │
└────────────────────────────────────────┘

Below are the border type counts.

SELECT   COUNT(*),
         tags.border_type
FROM     'boundaries.parquet'
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬──────────────────────────┐
│ count_star() │       border_type        │
│    int64     │         varchar          │
├──────────────┼──────────────────────────┤
│       749336 │ NULL                     │
│        10072 │ city                     │
│         7517 │ town                     │
│         6304 │ township                 │
│         4252 │ village                  │
│         3827 │ district                 │
│         3312 │ county                   │
│         3262 │ freguesia                │
│         1472 │ municipi                 │
│         1220 │ borough                  │
│         1014 │ municipality             │
│          886 │ barangay                 │
│          487 │ região_autónoma          │
│          383 │ territorial              │
│          309 │ município                │
│          277 │ suburb                   │
│          238 │ neighbourhood            │
│          194 │ arrondissement           │
│          158 │ baseline                 │
│          126 │ region                   │
│            · │   ·                      │
│            · │   ·                      │
│            · │   ·                      │
│            1 │ hamlet.CDP               │
│            1 │ aboriginal_lands         │
│            1 │ rural_municipality       │
│            1 │ muncipality              │
│            1 │ Village                  │
│            1 │ reserve                  │
│            1 │ mrc                      │
│            1 │ unincorporated_community │
│            1 │ commune                  │
│            1 │ homeowners_association   │
│            1 │ indian_reservation       │
│            1 │ county;town              │
│            1 │ adminstrative            │
│            1 │ m                        │
│            1 │ International            │
│            1 │ 禁止水域                 │
│            1 │ Consejo Comunal          │
│            1 │ 10                       │
│            1 │ first_nation             │
│            1 │ municipio                │
├──────────────┴──────────────────────────┤
│ 113 rows (40 shown)           2 columns │
└─────────────────────────────────────────┘

Below are the place counts.

SELECT   COUNT(*),
         tags.place
FROM     'boundaries.parquet'
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬─────────────────────────┐
│ count_star() │          place          │
│    int64     │         varchar         │
├──────────────┼─────────────────────────┤
│       617500 │ NULL                    │
│        60313 │ locality                │
│        29191 │ hamlet                  │
│        20311 │ village                 │
│        15782 │ neighbourhood           │
│        13065 │ municipality            │
│         9763 │ suburb                  │
│         6429 │ town                    │
│         4363 │ quarter                 │
│         3786 │ civil_parish            │
│         3191 │ city                    │
│         2262 │ district                │
│         1666 │ county                  │
│         1553 │ islet                   │
│         1001 │ isolated_dwelling       │
│          990 │ subdistrict             │
│          939 │ plot                    │
│          441 │ subdivision             │
│          409 │ island                  │
│          388 │ province                │
│            · │   ·                     │
│            · │   ·                     │
│            · │   ·                     │
│            3 │ payam                   │
│            3 │ neighborhood            │
│            3 │ municipal_federation    │
│            2 │ country                 │
│            2 │ *                       │
│            2 │ sub                     │
│            2 │ league                  │
│            2 │ township                │
│            1 │ census                  │
│            1 │ qah22                   │
│            1 │ districts               │
│            1 │ parish                  │
│            1 │ governorate             │
│            1 │ historic_urban_district │
│            1 │ place                   │
│            1 │ Quarter                 │
│            1 │ industrial              │
│            1 │ border_type             │
│            1 │ natural                 │
│            1 │ block_number            │
├──────────────┴─────────────────────────┤
│ 66 rows (40 shown)           2 columns │
└────────────────────────────────────────┘

This is Iceland's data, coloured by the boundary field.

This is the place field's data around Reykjavík.

There are 247 unique country codes in this dataset.

$ ~/duckdb

SELECT COUNT(DISTINCT tags."ISO3166-1:alpha3")
FROM   'boundaries.parquet'; -- 247

The geometry is a little over 8 GB in WKT format.

SELECT SUM(LENGTH(geometry::TEXT)) / 1024 ** 3
FROM   'boundaries.parquet'; -- 8.111921741627157

Overture

Overture release a new dataset monthly. They also remove old releases after 60 days so the URL below will likely need to be changed by the time you read this.

The following will list which releases are available for download.

$ curl https://labs.overturemaps.org/data/releases.json

{
    "latest": "2025-11-19.0",
    "releases": [
        "2025-11-19.0",
        "2025-10-22.0"
    ]
}

I'll download October's release. It's kept in four Parquet files totalling 3.2 GB and containing 1,052,542 records.

$ mkdir ~/overture
$ cd ~/overture

$ aws s3 \
    --no-sign-request \
    sync \
    s3://overturemaps-us-west-2/release/2025-11-19.0/theme=divisions/type=division_area/ \
    ./

Below is an example record.

$ echo "SELECT * EXCLUDE(geometry,
                         bbox,
                         names,
                         sources),
               bbox:    bbox::JSON,
               names:   names::JSON,
               sources: sources::JSON,
        FROM   'part-*-c000.zstd.parquet'
        WHERE  country = 'EE'
        LIMIT  1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "bbox": {
      "xmax": 27.38501739501953,
      "xmin": 27.367538452148438,
      "ymax": 57.655853271484375,
      "ymin": 57.64270782470703
    },
    "class": "land",
    "country": "EE",
    "division_id": "9a1adadd-7f92-4cf9-8deb-5004a3b0c9ac",
    "id": "45cafb8b-3a64-493c-8ad5-d038d2642a9e",
    "is_land": true,
    "is_territorial": true,
    "names": {
      "common": {
        "ru": "Коорла"
      },
      "primary": "Koorla küla",
      "rules": [
        {
          "between": null,
          "language": null,
          "perspectives": null,
          "side": null,
          "value": "Koorla",
          "variant": "alternate"
        }
      ]
    },
    "region": "EE-87",
    "sources": [
      {
        "between": null,
        "confidence": null,
        "dataset": "OpenStreetMap",
        "license": "ODbL-1.0",
        "property": "",
        "record_id": "r353065@4",
        "update_time": "2024-02-20T18:32:20Z"
      },
      {
        "between": null,
        "confidence": null,
        "dataset": "OpenStreetMap",
        "license": "ODbL-1.0",
        "property": "/properties/names/common/ru",
        "record_id": "n1973426847@6",
        "update_time": "2023-08-22T03:53:36Z"
      }
    ],
    "subtype": "locality",
    "version": 1
  }
]

Below are the number of records per class type.

SELECT   COUNT(*),
         class
FROM     'part-*-c000.zstd.parquet'
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬──────────┐
│ count_star() │  class   │
│    int64     │ varchar  │
├──────────────┼──────────┤
│      1050537 │ land     │
│         2005 │ maritime │
└──────────────┴──────────┘

Below are the number of records by is_land's value.

SELECT   COUNT(*),
         is_land
FROM     'part-*-c000.zstd.parquet'
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬─────────┐
│ count_star() │ is_land │
│    int64     │ boolean │
├──────────────┼─────────┤
│      1050537 │ true    │
│         2005 │ false   │
└──────────────┴─────────┘

Below are the number of records by is_territorial's value.

SELECT   COUNT(*),
         is_territorial
FROM     'part-*-c000.zstd.parquet'
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬────────────────┐
│ count_star() │ is_territorial │
│    int64     │    boolean     │
├──────────────┼────────────────┤
│      1050537 │ true           │
│         2005 │ false          │
└──────────────┴────────────────┘

Below are the number of records by subtype.

SELECT   COUNT(*),
         subtype
FROM     'part-*-c000.zstd.parquet'
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬──────────────┐
│ count_star() │   subtype    │
│    int64     │   varchar    │
├──────────────┼──────────────┤
│       560682 │ locality     │
│       303353 │ neighborhood │
│        78566 │ microhood    │
│        43856 │ macrohood    │
│        39786 │ county       │
│        21109 │ localadmin   │
│         4707 │ region       │
│          378 │ country      │
│          105 │ dependency   │
└──────────────┴──────────────┘

There are 272 unique country codes in this dataset.

$ ~/duckdb

SELECT COUNT(DISTINCT country)
FROM   'part-*-c000.zstd.parquet'; -- 272

The geometry is a little over 8 GB in WKT format.

SELECT SUM(LENGTH(geometry::TEXT)) / 1024 ** 3
FROM   'part-*-c000.zstd.parquet'; -- 8.173572240397334

UN COD

The UN Common Operational Datasets (COD) is 9 GB across four Parquet files and contains 541,172 records.

$ mkdir ~/un_cod
$ cd ~/un_cod

$ wget -c https://data.fieldmaps.io/edge-matched/humanitarian/intl/adm1_polygons.parquet
$ wget -c https://data.fieldmaps.io/edge-matched/humanitarian/intl/adm2_polygons.parquet
$ wget -c https://data.fieldmaps.io/edge-matched/humanitarian/intl/adm3_polygons.parquet
$ wget -c https://data.fieldmaps.io/edge-matched/humanitarian/intl/adm4_polygons.parquet

Below is an example record.

$ echo "SELECT * EXCLUDE(geometry,
                         geometry_bbox),
               bbox: geometry_bbox::JSON
        FROM   'adm*_polygons.parquet'
        WHERE  iso_2 = 'EE'
        LIMIT  1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "adm0_id": "EST-20250729",
    "adm0_name": "Estonia",
    "adm0_name1": "Estonia",
    "adm0_name2": null,
    "adm0_src": "EST",
    "adm1_id": "EST-20230119-01",
    "adm1_name": "Harju maakond",
    "adm1_name1": null,
    "adm1_name2": null,
    "adm1_src": "21999144B19319879871883",
    "bbox": {
      "xmax": 25.955127716064453,
      "xmin": 23.730022430419922,
      "ymax": 59.700199127197266,
      "ymin": 58.989349365234375
    },
    "fid": 937,
    "iso_2": "EE",
    "iso_3": "EST",
    "iso_3_grp": "EST",
    "iso_cd": 233,
    "region1_cd": 150,
    "region1_nm": "Europe",
    "region2_cd": 154,
    "region2_nm": "Northern Europe",
    "region3_cd": 154,
    "region3_nm": "Northern Europe",
    "src_date": "2020-01-01",
    "src_grp": "geoBoundaries",
    "src_lang": "en",
    "src_lang1": null,
    "src_lang2": null,
    "src_lic": "Open Data Commons Open Database License 1.0",
    "src_lvl": 3,
    "src_name": "OpenStreetMap, OSM Boundaries",
    "src_name1": null,
    "src_update": "2023-01-19",
    "src_url": "https://www.geoboundaries.org/api/current/gbOpen/EST/ALL/",
    "status_cd": 1,
    "status_nm": "State",
    "wld_date": "2025-02-24",
    "wld_notes": null,
    "wld_update": "2025-07-29",
    "wld_view": "intl"
  }
]

There are 285 unique country codes in this dataset.

$ ~/duckdb

SELECT COUNT(DISTINCT iso_3)
FROM   'adm*_polygons.parquet'; -- 285

The geometry is a little over 16 GB in WKT format.

SELECT SUM(LENGTH(geometry::TEXT)) / 1024 ** 3
FROM   'adm*_polygons.parquet'; -- 16.433973343111575

Who's On First

Who's On First (WOF) stores its data as GeoJSON and partitions by country into 272 separate repositories on GitHub. Below, I'll convert their data for Bahrain into a single Parquet file.

$ mkdir ~/wof
$ cd ~/wof

$ git clone https://github.com/whosonfirst-data/whosonfirst-data-admin-bh/
$ cd whosonfirst-data-admin-bh/data

$ ~/duckdb wof.duckdb

CREATE OR REPLACE TABLE wof (
    id         BIGINT,
    "type"     VARCHAR,
    properties JSON,
    bbox       double[],
    geometry   geometry);

$ for FILENAME in `find . | grep geojson`; do
      echo $FILENAME
      echo "INSERT INTO wof
              SELECT id,
                     type,
                     properties,
                     bbox,
                     ST_GeomFromGeoJSON(geometry)
              FROM READ_JSON('$FILENAME');" \
          | ~/duckdb wof.duckdb
  done

The following produced a 112 KB Parquet file.

$ ~/duckdb wof.duckdb

COPY (
    FROM     wof
    ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geometry)),
                             ST_X(ST_CENTROID(geometry))]::double[2])
) TO 'wof.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

I haven't studied their data structure in depth but the above produced records with either a linestring or point geometry.

Below is an example record. The preferred names list was lengthy so I removed it for readability.

$ echo "SELECT   * EXCLUDE(geometry),
        FROM     'wof.parquet'
        ORDER BY LENGTH(properties.\"wof:hierarchy\")
        LIMIT    1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "bbox": "[50.310047, 25.55625, 50.82486, 26.331343]",
    "id": 85632167,
    "properties": {
      "edtf:cessation": "uuuu",
      "edtf:inception": "uuuu",
      "geom:area": 0.070331,
      "geom:area_square_m": 781290976.825043,
      "geom:bbox": "50.310047,25.55625,50.82486,26.331343",
      "geom:latitude": 26.051802,
      "geom:longitude": 50.564827,
      "gn:population": 738004,
      "iso:country": "BH",
      "itu:country_code": [
        "973"
      ],
      "label:eng_x_preferred_placetype": [
        "country"
      ],
      "label:eng_x_preferred_shortcode": [
        "BH"
      ],
      "lbl:latitude": 26.0845,
      "lbl:longitude": 50.550672,
      "lbl:max_zoom": 8.0,
      "lbl:min_zoom": 4.0,
      "mps:latitude": 26.0845,
      "mps:longitude": 50.550672,
      "mz:hierarchy_label": 1,
      "mz:is_current": 1,
      "mz:max_zoom": 9.0,
      "mz:min_zoom": 4.0,
      "ne:abbrev": "Bahr.",
      "ne:abbrev_len": 5,
      "ne:adm0_a3": "BHR",
      "ne:adm0_a3_ar": "BHR",
      "ne:adm0_a3_bd": "BHR",
      "ne:adm0_a3_br": "BHR",
      "ne:adm0_a3_cn": "BHR",
      "ne:adm0_a3_de": "BHR",
      "ne:adm0_a3_eg": "BHR",
      "ne:adm0_a3_es": "BHR",
      "ne:adm0_a3_fr": "BHR",
      "ne:adm0_a3_gb": "BHR",
      "ne:adm0_a3_gr": "BHR",
      "ne:adm0_a3_id": "BHR",
      "ne:adm0_a3_il": "BHR",
      "ne:adm0_a3_in": "BHR",
      "ne:adm0_a3_is": "BHR",
      "ne:adm0_a3_it": "BHR",
      "ne:adm0_a3_jp": "BHR",
      "ne:adm0_a3_ko": "BHR",
      "ne:adm0_a3_ma": "BHR",
      "ne:adm0_a3_nl": "BHR",
      "ne:adm0_a3_np": "BHR",
      "ne:adm0_a3_pk": "BHR",
      "ne:adm0_a3_pl": "BHR",
      "ne:adm0_a3_ps": "BHR",
      "ne:adm0_a3_pt": "BHR",
      "ne:adm0_a3_ru": "BHR",
      "ne:adm0_a3_sa": "BHR",
      "ne:adm0_a3_se": "BHR",
      "ne:adm0_a3_tr": "BHR",
      "ne:adm0_a3_tw": "BHR",
      "ne:adm0_a3_ua": "BHR",
      "ne:adm0_a3_un": -99,
      "ne:adm0_a3_us": "BHR",
      "ne:adm0_a3_vn": "BHR",
      "ne:adm0_a3_wb": -99,
      "ne:adm0_dif": 0,
      "ne:admin": "Bahrain",
      "ne:brk_a3": "BHR",
      "ne:brk_diff": 0,
      "ne:brk_group": null,
      "ne:brk_name": "Bahrain",
      "ne:continent": "Asia",
      "ne:economy": "6. Developing region",
      "ne:featurecla": "Admin-0 country",
      "ne:fips_10": "BA",
      "ne:fips_10_": "BA",
      "ne:formal_en": "Kingdom of Bahrain",
      "ne:formal_fr": null,
      "ne:gdp_md": 38574,
      "ne:gdp_md_est": 26820,
      "ne:gdp_year": 2019,
      "ne:geou_dif": 0,
      "ne:geounit": "Bahrain",
      "ne:gu_a3": "BHR",
      "ne:homepart": 1,
      "ne:income_grp": "2. High income: nonOECD",
      "ne:iso_a2": "BH",
      "ne:iso_a2_eh": "BH",
      "ne:iso_a3": "BHR",
      "ne:iso_a3_eh": "BHR",
      "ne:iso_n3": "048",
      "ne:iso_n3_eh": "048",
      "ne:labelrank": 4,
      "ne:lastcensus": 2010,
      "ne:level": 2,
      "ne:long_len": 7,
      "ne:mapcolor13": 9,
      "ne:mapcolor7": 1,
      "ne:mapcolor8": 1,
      "ne:mapcolor9": 1,
      "ne:max_label": 9,
      "ne:min_label": 4,
      "ne:min_zoom": 0,
      "ne:name_alt": null,
      "ne:name_ar": "البحرين",
      "ne:name_bn": "বাহরাইন",
      "ne:name_ciawf": "Bahrain",
      "ne:name_de": "Bahrain",
      "ne:name_el": "Μπαχρέιν",
      "ne:name_en": "Bahrain",
      "ne:name_es": "Baréin",
      "ne:name_fr": "Bahreïn",
      "ne:name_he": "בחריין",
      "ne:name_hi": "बहरीन",
      "ne:name_hu": "Bahrein",
      "ne:name_id": "Bahrain",
      "ne:name_it": "Bahrein",
      "ne:name_ja": "バーレーン",
      "ne:name_ko": "바레인",
      "ne:name_len": 7,
      "ne:name_long": "Bahrain",
      "ne:name_nl": "Bahrein",
      "ne:name_pl": "Bahrajn",
      "ne:name_pt": "Bahrein",
      "ne:name_ru": "Бахрейн",
      "ne:name_sort": "Bahrain",
      "ne:name_sv": "Bahrain",
      "ne:name_tr": "Bahreyn",
      "ne:name_uk": "Бахрейн",
      "ne:name_ur": "بحرین",
      "ne:name_vi": "Bahrain",
      "ne:name_zh": "巴林",
      "ne:note_adm0": null,
      "ne:note_brk": null,
      "ne:pop_est": 1641172,
      "ne:pop_rank": 12,
      "ne:pop_year": 2019,
      "ne:postal": "BH",
      "ne:region_un": "Asia",
      "ne:region_wb": "Middle East & North Africa",
      "ne:scalerank": 5,
      "ne:sov_a3": "BHR",
      "ne:sovereignt": "Bahrain",
      "ne:su_a3": "BHR",
      "ne:su_dif": 0,
      "ne:subregion": "Western Asia",
      "ne:subunit": "Bahrain",
      "ne:tiny": 2,
      "ne:type": "Sovereign country",
      "ne:un_a3": "048",
      "ne:wb_a2": "BH",
      "ne:wb_a3": "BHR",
      "ne:wikidataid": "Q398",
      "ne:wikipedia": "-99",
      "ne:woe_id": 23424753,
      "ne:woe_id_eh": 23424753,
      "ne:woe_note": "Exact WOE match as country",
      "qs:a0": "Bahrain",
      "qs:a0_alt": "Bahrain",
      "qs:adm0": "Bahrain",
      "qs:adm0_a3": "BHR",
      "qs:level": "adm0",
      "qs:scale": 200000,
      "qs:source": "US State Department, with Natural Earth mods",
      "src:geom": "whosonfirst",
      "src:geom_alt": [
        "naturalearth",
        "quattroshapes"
      ],
      "src:geom_via": "us-dshiu",
      "src:lbl_centroid": "mapshaper",
      "src:population": "naturalearth",
      "src:population_date": "2019",
      "statoids:dial": "973",
      "statoids:ds": "BRN",
      "statoids:fifa": "BHR",
      "statoids:gaul": "21",
      "statoids:gec": "BA",
      "statoids:independent": "Yes",
      "statoids:ioc": "BRN",
      "statoids:iso_a2": "BH",
      "statoids:iso_a3": "BHR",
      "statoids:iso_num": "48",
      "statoids:itu": "BHR",
      "statoids:marc": "ba",
      "statoids:name": "Bahrain",
      "statoids:wmo": "BN",
      "wd:latitude": 26.0675,
      "wd:longitude": 50.551111,
      "wd:population": 1332171,
      "wd:wordcount": 15434,
      "wof:belongsto": [
        102191569
      ],
      "wof:breaches": [],
      "wof:capital": [
        421190647
      ],
      "wof:concordances": {
        "dbp:id": "Bahrain",
        "digitalenvoy:country_code": 48,
        "fb:id": "en.bahrain",
        "fifa:id": "BHR",
        "fips:code": "BA",
        "gaul:id": "21",
        "gn:id": 290291,
        "gp:id": 23424753,
        "hasc:id": "BH",
        "icao:code": "A9C",
        "ioc:id": "BRN",
        "iso:code": "BH",
        "itu:id": "BHR",
        "m49:code": "048",
        "marc:id": "ba",
        "mzb:id": "bahrain",
        "ne:adm0_a3": "BHR",
        "ne:id": 1159320413,
        "nyt:id": "59420554464253622721",
        "qs_pg:id": 1079189,
        "uncrt:id": "BRN",
        "wd:id": "Q398",
        "wk:page": "Bahrain",
        "wmo:id": "BN"
      },
      "wof:concordances_official": "iso:code",
      "wof:country": "BH",
      "wof:country_alpha3": "BHR",
      "wof:geom_alt": [
        "naturalearth-display-terrestrial-zoom6",
        "naturalearth",
        "quattroshapes"
      ],
      "wof:geomhash": "3036998dbe467babc2f25c2329dc054c",
      "wof:hierarchy": [
        {
          "continent_id": 102191569,
          "country_id": 85632167
        }
      ],
      "wof:id": 85632167,
      "wof:lang_x_official": [
        "ara"
      ],
      "wof:lang_x_spoken": [
        "ara"
      ],
      "wof:lastmodified": 1695881186,
      "wof:name": "Bahrain",
      "wof:parent_id": 102191569,
      "wof:placetype": "country",
      "wof:population": 1641172,
      "wof:population_rank": 18,
      "wof:repo": "whosonfirst-data-admin-bh",
      "wof:shortcode": "BH",
      "wof:superseded_by": [],
      "wof:supersedes": [],
      "wof:tags": []
    },
    "type": "Feature"
  }
]

In the North of Bahrain, there are several areas of land that are outside of their boundaries.

The Estonian repository contains 12,120 GeoJSON files and suffers from a small file problem. Converting these into a single, 11 MB Parquet file took a long time.

USGS

The US Geological Survey (USGS) has a 1.1 GB Parquet file containing 318 records on level 0 administrative boundaries.

$ mkdir ~/usgs
$ cd ~/usgs

$ wget -c https://data.fieldmaps.io/adm0/usgs/intl/adm0_polygons.parquet

Below is an example record.

$ echo "SELECT * EXCLUDE(geometry,
                         geometry_bbox),
               bbox: geometry_bbox::JSON
        FROM   'adm0_polygons.parquet'
        WHERE  iso_2 = 'EE'
        LIMIT  1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "adm0_id": "EST-20250729",
    "adm0_name": "Estonia",
    "adm0_name1": "Estonia",
    "adm0_name2": null,
    "adm0_src": "EST",
    "bbox": {
      "xmax": 28.20932388305664,
      "xmin": 21.76363754272461,
      "ymax": 59.821800231933594,
      "ymin": 57.50931167602539
    },
    "iso_2": "EE",
    "iso_3": "EST",
    "iso_3_grp": "EST",
    "iso_cd": 233,
    "region1_cd": 150,
    "region1_nm": "Europe",
    "region2_cd": 154,
    "region2_nm": "Northern Europe",
    "region3_cd": 154,
    "region3_nm": "Northern Europe",
    "status_cd": 1,
    "status_nm": "State",
    "wld_date": "2025-02-24",
    "wld_land": "usgs",
    "wld_notes": null,
    "wld_update": "2025-07-29",
    "wld_view": "intl"
  }
]

There are 318 unique country codes in this dataset.

$ ~/duckdb

SELECT COUNT(DISTINCT adm0_src)
FROM   'adm0_polygons.parquet'; -- 318

The geometry is a little over 1.7 GB in WKT format.

SELECT SUM(LENGTH(geometry::TEXT)) / 1024 ** 3
FROM   'adm0_polygons.parquet'; -- 1.7173228040337563

The border around Northern Bahrain cuts corners and appears to be offset somewhat.

This contrasts OSM which draws the external boundaries over their maritime borders. This ensures any point along their coastlines will be seen as inside the country when reverse geocoded.

World Bank

The World Bank has a GeoPackage (GPKG) file that is 60 MB and contains 264 records. I needed to use their JS-based UI to download the GPKG file.

$ mkdir ~/world_bank
$ cd ~/world_bank

$ ~/duckdb world_bank.duckdb

CREATE OR REPLACE TABLE wb AS
    FROM ST_READ('World Bank Official Boundaries - Admin 0.gpkg');

Below is an example record.

$ echo "SELECT * EXCLUDE(geom),
        FROM   wb
        WHERE  ISO_A2 = 'EE'
        LIMIT  1" \
    | ~/duckdb -json world_bank.duckdb \
    | jq -S .

[
  {
    "GAUL_0": 78,
    "HASC_0": "EE",
    "ISO_A2": "EE",
    "ISO_A3": "EST",
    "NAM_0": "Estonia",
    "SOVEREIGN": "EST",
    "WB_A3": "EST",
    "WB_REGION": "ECA",
    "WB_STATUS": "Member State"
  }
]

There are 244 unique country codes in this dataset.

$ ~/duckdb -json world_bank.duckdb

SELECT COUNT(DISTINCT ISO_A3)
FROM   wb; -- 244

The geometry is a little over 142 MB in WKT format.

SELECT SUM(LENGTH(geom::TEXT)) / 1024 ** 2
FROM   wb; -- 142.03707027435303

Their Northern Border for Bahrain cuts a lot of corners.

GADM

GADM version 4.1 is distributed as a 1.4 GB ZIP containing a 2.6 GB GPKG file with 356,508 records.

$ mkdir ~/gadm
$ cd ~/gadm

$ wget -c https://geodata.ucdavis.edu/gadm/gadm4.1/gadm_410-gdb.zip
$ unzip -j gadm_410-gdb.zip

Below is an example record.

$ ~/duckdb gadm.duckdb

CREATE OR REPLACE TABLE gadm AS
    FROM ST_READ('gadm_410.gpkg');

$ echo "SELECT * EXCLUDE(geom),
        FROM   gadm
        WHERE  COUNTRY = 'Estonia'
        LIMIT  1" \
    | ~/duckdb -json gadm.duckdb \
    | jq -S .

[
  {
    "CC_1": "",
    "CC_2": "",
    "CC_3": "",
    "CC_4": "",
    "CC_5": "",
    "CONTINENT": "Europe",
    "COUNTRY": "Estonia",
    "DISPUTEDBY": "",
    "ENGTYPE_1": "County",
    "ENGTYPE_2": "Parish",
    "ENGTYPE_3": "Town",
    "ENGTYPE_4": "",
    "ENGTYPE_5": "",
    "GID_0": "EST",
    "GID_1": "EST.1_1",
    "GID_2": "EST.1.1_1",
    "GID_3": "EST.1.1.1_1",
    "GID_4": "",
    "GID_5": "",
    "GOVERNEDBY": "",
    "HASC_1": "EE.HA",
    "HASC_2": "EE.HA.AE",
    "HASC_3": "",
    "ISO_1": "",
    "NAME_0": "Estonia",
    "NAME_1": "Harju",
    "NAME_2": "Aegviidu",
    "NAME_3": "Aegviidu",
    "NAME_4": "",
    "NAME_5": "",
    "NL_NAME_1": "",
    "NL_NAME_2": "",
    "NL_NAME_3": "",
    "REGION": "",
    "SOVEREIGN": "Estonia",
    "SUBCONT": "",
    "TYPE_1": "Maakond",
    "TYPE_2": "Vald",
    "TYPE_3": "Alev",
    "TYPE_4": "",
    "TYPE_5": "",
    "UID": 49772,
    "VALIDFR_1": "Unknown",
    "VALIDFR_2": "2002",
    "VALIDFR_3": "Unknown",
    "VALIDFR_4": "",
    "VARNAME_0": "",
    "VARNAME_1": "Harjumaa|Harju maakond",
    "VARNAME_2": "",
    "VARNAME_3": "",
    "VARNAME_4": "",
    "VARREGION": ""
  }

There are 263 unique country codes in this dataset.

$ ~/duckdb -json gadm.duckdb

SELECT COUNT(DISTINCT GID_0)
FROM   gadm; -- 263

The geometry is a little over 5.4 GB in WKT format.

SELECT SUM(LENGTH(geom::TEXT)) / 1024 ** 3
FROM   gadm; -- 5.4283982161432505

Their Northern Border for Bahrain cuts a lot of corners.

GAUL

The Food and Agriculture Organization (FAO) of the United Nations has published FAO GAUL 2024, a 310 MB ZIP file containing 433 MB in Shapefiles across 3,110 records.

$ mkdir ~/gaul
$ cd ~/gaul

$ wget -c https://storage.googleapis.com/fao-maps-catalog-data/boundaries/GAUL_2024_L1.zip
$ unzip -j GAUL_2024_L1.zip

Below is an example record.

$ ~/duckdb gaul.duckdb

CREATE OR REPLACE TABLE gaul AS
    FROM ST_READ('GAUL_2024_L1.shx');

$ echo "SELECT * EXCLUDE(geom),
        FROM   gaul
        WHERE  iso3_code = 'EST'
        LIMIT  1" \
    | ~/duckdb -json gaul.duckdb \
    | jq -S .

[
  {
    "continent": "Europe",
    "disp_en": "Tartu maakond, Estonia",
    "gaul0_code": 298,
    "gaul0_name": "Estonia",
    "gaul1_code": 3419,
    "gaul1_name": "Tartu maakond",
    "iso3_code": "EST",
    "map_code": "EST"
  }
]

There are 265 unique country codes in this dataset.

$ ~/duckdb gaul.duckdb

SELECT COUNT(DISTINCT iso3_code)
FROM   gaul; -- 265

There is a little over one GB of geometry in WKT format in this dataset.

SELECT SUM(LENGTH(geom::TEXT)) / 1024 ** 2
FROM   gaul; -- 1030.7613172531128

Their Northern Border for Bahrain cuts a lot of corners.

geoBoundaries

The geoBoundaries project has a 155 MB GPKG file containing 218 records.

$ mkdir ~/geoboundaries
$ cd ~/geoboundaries

$ wget -c https://github.com/wmgeolab/geoBoundaries/raw/main/releaseData/CGAZ/geoBoundariesCGAZ_ADM0.gpkg

Below is an example record.

$ ~/duckdb geoBoundaries.duckdb

CREATE OR REPLACE TABLE gb AS
    FROM ST_READ('geoBoundariesCGAZ_ADM0.gpkg');

$ echo "SELECT * EXCLUDE(geom),
        FROM   gb
        WHERE  shapeGroup = 'EST'
        LIMIT  1" \
    | ~/duckdb -json geoBoundaries.duckdb \
    | jq -S .

[
  {
    "id": null,
    "shapeGroup": "EST",
    "shapeName": "Estonia",
    "shapeType": "ADM0"
  }
]

There are 218 unique country records in this dataset.

$ ~/duckdb geoBoundaries.duckdb

SELECT COUNT(DISTINCT shapeGroup)
FROM   gb; -- 218

The geometry is a little over 367 MB in WKT format.

SELECT SUM(LENGTH(geom::TEXT)) / 1024 ** 2
FROM   gb; -- 367.0248956680298

Their Northern Border for Bahrain cuts a few corners.

GeoNames

For GeoNames, I needed to download two ZIP files totalling 398 MB. These contained 1.7 GB of CSV data. I then joined these together to create a 249-record dataset of the world's level 0 administrative boundaries.

$ mkdir ~/geonames
$ cd ~/geonames

$ wget -c https://download.geonames.org/export/dump/allCountries.zip
$ unzip -j allCountries.zip

$ ~/duckdb geonames.duckdb

CREATE OR REPLACE TABLE geonames (
     geonameid         BIGINT,
     name              VARCHAR,
     asciiname         VARCHAR,
     alternatenames    VARCHAR,
     latitude          DOUBLE,
     longitude         DOUBLE,
     feature_class     VARCHAR,
     feature_code      VARCHAR,
     country_code      VARCHAR,
     cc2               VARCHAR,
     admin1_code       VARCHAR,
     admin2_code       VARCHAR,
     admin3_code       VARCHAR,
     admin4_code       VARCHAR,
     population        BIGINT,
     elevation         BIGINT,
     dem               BIGINT,
     timezone          VARCHAR,
     modification_date DATE);

INSERT INTO geonames
FROM        READ_CSV('allCountries.txt');

$ wget https://download.geonames.org/export/dump/shapes_all_low.zip
$ unzip -j shapes_all_low.zip

$ ~/duckdb geonames.duckdb

CREATE OR REPLACE TABLE geonames_boundaries AS
    SELECT    a.geoNameId,
              b.*,
              geometry: ST_GeomFromGeoJSON(geoJSON)
    FROM      READ_CSV('shapes_all_low.txt') a
    LEFT JOIN geonames b ON a.geoNameId = b.geonameid;

Below is an example record.

$ echo "SELECT * EXCLUDE(geometry, alternatenames),
               alternate_names: SPLIT(alternatenames, ',')::JSON
        FROM   geonames_boundaries
        WHERE  country_code = 'EE'
        LIMIT  1" \
    | ~/duckdb -json geonames.duckdb \
    | jq -S .

[
  {
    "admin1_code": "00",
    "admin2_code": null,
    "admin3_code": null,
    "admin4_code": null,
    "alternate_names": [
      "'Esitonia",
      "An Eastoin",
      "E-xto-ni-a",
      "E-xto-ni-a (Estonia)",
      "E-xtô-ni-a",
      "E-xtô-ni-a (Estonia)",
      "EE",
      "Eastland",
      "Eesti",
      "Eesti Noukogude Sotsialistlik Vabariik",
      "Eesti Nõukogude Sotsialistlik Vabariik",
      "Eesti Vabariik",
      "Ehstoni",
      "Ehstonija",
      "Eistland",
      "Esetoni",
      "Esitoni",
      "Esitonia",
      "Esitoniya",
      "Esland",
      "Esthonia",
      "Estija",
      "Estiska",
      "Estlan",
      "Estland",
      "Estlanda",
      "Estlandi",
      "Estlánda",
      "Estlân",
      "Estoni",
      "Estonia",
      "Estonia nutome",
      "Estonian Soviet Socialist Republic",
      "Estonie",
      "Estonii",
      "Estonija",
      "Estonio",
      "Estoniya",
      "Estoniýa",
      "Estonja",
      "Estonnie",
      "Estonskaya Sovetskaya Sotsialisticheskaya Respublika",
      "Estonsko",
      "Estonujo",
      "Estonya",
      "Estonïi",
      "Estooniya",
      "Estounii",
      "Estteeana",
      "Estònia",
      "Estónia",
      "Estónsko",
      "Estóńskô",
      "Estônia",
      "Esztorszag",
      "Igaunija",
      "Istunya",
      "Lestiyaen",
      "Lestiyän",
      "Republic of Estonia",
      "Viro",
      "ai sha ni ya",
      "an Eastoin",
      "an Eastóin",
      "astwny",
      "astwnya",
      "aysٹwniya",
      "aysٽwnya",
      "eseutonia",
      "estoni'a",
      "estoniya",
      "estʼoneti",
      "esutonia",
      "esutonia gong he guo",
      "gugrxesti",
      "i-Estonia",
      "isatoni'a",
      "istoniya",
      "l-Estonja",
      "pra thes xe s to neiy",
      "xe s to neiy",
      "yېstwnyyە",
      "Èstonie",
      "Éstonia",
      "Észtország",
      "Ēastland",
      "Ɛstonia",
      "ʻEsitōnia",
      "ʻstlʼand",
      "ʼstwnyh",
      "ʿesitoniya",
      "Εσθονία",
      "Естония",
      "Естонија",
      "Естонія",
      "Эстони",
      "Эстония",
      "Эстонія",
      "Էստոնիա",
      "אסטוניה",
      "עסטלאַנד",
      "أستونيا",
      "إستونيا",
      "ئېستونىيە",
      "استونی",
      "استونیا",
      "اسٹونیا",
      "ايسٽونيا",
      "ایسٹونِیا",
      "اېسټونيا",
      "ܐܣܬܘܢܝܐ",
      "इस्टोनिया",
      "एस्टोनिया",
      "ইষ্টোনিয়া",
      "এস্তোনিয়া",
      "এস্তোনিয়া",
      "ਇਸਟੋਨੀਆ",
      "એસ્ટોનિયા",
      "ଏସ୍ତୋନିଆ",
      "எஸ்டோனியா",
      "ఎస్టోనియా",
      "ಎಸ್ಟೋನಿಯಾ",
      "എസ്റ്റോണിയ‍",
      "එස්තෝනියාව",
      "ประเทศเอสโตเนีย",
      "เอสโตเนีย",
      "ເອສໂຕເນຍ",
      "ཨས་ཊ་ན་ཡ",
      "ཨིསྟོ་ནི་ཡ།",
      "ཨེས་ཊོ་ནི་ཡ",
      "အက်စတိုးနီးယား",
      "ესტონეთი",
      "ኤስቶኒያ",
      "អេសតន",
      "អេស្តូនី",
      "エストニア",
      "エストニア共和国",
      "愛沙尼亞",
      "爱沙尼亚",
      "에스토니아"
    ],
    "asciiname": "Republic of Estonia",
    "cc2": null,
    "country_code": "EE",
    "dem": 99,
    "elevation": null,
    "feature_class": "A",
    "feature_code": "PCLI",
    "geoNameId": 453733,
    "geonameid_1": 453733,
    "latitude": 59.0,
    "longitude": 26.0,
    "modification_date": "2024-09-05",
    "name": "Republic of Estonia",
    "population": 1320884,
    "timezone": "Europe/Tallinn"
  }
]

There are 249 unique country codes in this dataset.

$ ~/duckdb geonames.duckdb

SELECT COUNT(DISTINCT country_code)
FROM   geonames_boundaries; -- 249

The geometry is a little over 4.2 GB in WKT format.

SELECT SUM(LENGTH(geometry::TEXT)) / 1024 ** 2
FROM   geonames_boundaries; -- 4.253528594970703

I'll convert this dataset into Parquet.

COPY (
    FROM     geonames_boundaries
    ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geometry)),
                             ST_X(ST_CENTROID(geometry))]::double[2])
) TO 'geonames_boundaries.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

Their Northern Border for Bahrain cuts a lot of corners.

GISCO

The European Commission has a GISCO dataset with level 0 administrative boundaries. There are several resolutions, years, formats and projections available.

$ mkdir ~/gisco
$ cd ~/gisco

I used their form to download the 1M resolution GPKG file for 2024. It was 23 MB and contained 263 records.

$ ~/duckdb eu.duckdb

CREATE OR REPLACE TABLE eu AS
    FROM ST_READ('CNTR_RG_01M_2024_4326.gpkg');

Below is an example record.

$ echo "SELECT * EXCLUDE(geom),
        FROM   eu
        WHERE  ISO3_CODE = 'EST'
        LIMIT  1" \
    | ~/duckdb -json eu.duckdb \
    | jq -S .

[
  {
    "CAPT": "Tallinn",
    "CC_STAT": "F",
    "CNTR_ID": "EE",
    "CNTR_NAME": "Eesti",
    "EFTA_STAT": "F",
    "EU_STAT": "T",
    "ISO3_CODE": "EST",
    "NAME_ENGL": "Estonia",
    "NAME_FREN": "Estonie",
    "NAME_GERM": "Estland",
    "SVRG_UN": "UN Member State"
  }
]

There are 260 unique country codes in this dataset.

$ ~/duckdb eu.duckdb

SELECT COUNT(DISTINCT CNTR_ID)
FROM   eu; -- 260

There is a little over 1.5 GB of geometry in WKT format in this dataset.

SELECT SUM(LENGTH(geom::TEXT)) / 1024 ** 2
FROM   eu; -- 1.5088157653808594

Their Northern Border for Bahrain cuts a lot of corners.

Natural Earth

Natural Earth v5.1.2 comes in 10M, 50M and 110M resolutions. I'll use the 10M resolution for this post.

$ mkdir ~/natural_earth
$ cd ~/natural_earth

There are 33 localisations of the level-0 administrative boundaries dataset.

$ aws s3 \
    ls \
    --no-sign-request \
    s3://naturalearth/5.1.2/10m_cultural/ \
    | grep 'ne\_10m\_admin\_0\_countries\_[a-z][a-z][a-z]\.zip' \
    | cut -c32-

ne_10m_admin_0_countries_arg.zip
ne_10m_admin_0_countries_bdg.zip
ne_10m_admin_0_countries_bra.zip
ne_10m_admin_0_countries_chn.zip
ne_10m_admin_0_countries_deu.zip
ne_10m_admin_0_countries_egy.zip
ne_10m_admin_0_countries_esp.zip
ne_10m_admin_0_countries_fra.zip
ne_10m_admin_0_countries_gbr.zip
ne_10m_admin_0_countries_grc.zip
ne_10m_admin_0_countries_idn.zip
ne_10m_admin_0_countries_ind.zip
ne_10m_admin_0_countries_iso.zip
ne_10m_admin_0_countries_isr.zip
ne_10m_admin_0_countries_ita.zip
ne_10m_admin_0_countries_jpn.zip
ne_10m_admin_0_countries_kor.zip
ne_10m_admin_0_countries_mar.zip
ne_10m_admin_0_countries_nep.zip
ne_10m_admin_0_countries_nld.zip
ne_10m_admin_0_countries_pak.zip
ne_10m_admin_0_countries_pol.zip
ne_10m_admin_0_countries_prt.zip
ne_10m_admin_0_countries_pse.zip
ne_10m_admin_0_countries_rus.zip
ne_10m_admin_0_countries_sau.zip
ne_10m_admin_0_countries_swe.zip
ne_10m_admin_0_countries_tlc.zip
ne_10m_admin_0_countries_tur.zip
ne_10m_admin_0_countries_twn.zip
ne_10m_admin_0_countries_ukr.zip
ne_10m_admin_0_countries_usa.zip
ne_10m_admin_0_countries_vnm.zip

I'll use the German localisation. The following downloaded a 4.7 MB ZIP file containing 9.3 MB worth of shapefiles.

$ aws s3 \
    --no-sign-request \
    cp \
    s3://naturalearth/5.1.2/10m_cultural/ne_10m_admin_0_countries_deu.zip \
    ./
$ unzip -j ne_10m_admin_0_countries_deu.zip

$ ~/duckdb ne.duckdb

CREATE OR REPLACE TABLE ne AS
    FROM ST_READ('ne_10m_admin_0_countries_deu.shx');

Below is an example record.

$ echo "SELECT * EXCLUDE(geom),
        FROM   ne
        WHERE  ADM0_A3 = 'EST'
        LIMIT  1" \
    | ~/duckdb -json ne.duckdb \
    | jq -S .

[
  {
    "ABBREV": "Est.",
    "ABBREV_LEN": 4,
    "ADM0_A3": "EST",
    "ADM0_A3_AR": "EST",
    "ADM0_A3_BD": "EST",
    "ADM0_A3_BR": "EST",
    "ADM0_A3_CN": "EST",
    "ADM0_A3_DE": "EST",
    "ADM0_A3_EG": "EST",
    "ADM0_A3_ES": "EST",
    "ADM0_A3_FR": "EST",
    "ADM0_A3_GB": "EST",
    "ADM0_A3_GR": "EST",
    "ADM0_A3_ID": "EST",
    "ADM0_A3_IL": "EST",
    "ADM0_A3_IN": "EST",
    "ADM0_A3_IT": "EST",
    "ADM0_A3_JP": "EST",
    "ADM0_A3_KO": "EST",
    "ADM0_A3_MA": "EST",
    "ADM0_A3_NL": "EST",
    "ADM0_A3_NP": "EST",
    "ADM0_A3_PK": "EST",
    "ADM0_A3_PL": "EST",
    "ADM0_A3_PS": "EST",
    "ADM0_A3_PT": "EST",
    "ADM0_A3_RU": "EST",
    "ADM0_A3_SA": "EST",
    "ADM0_A3_SE": "EST",
    "ADM0_A3_TR": "EST",
    "ADM0_A3_TW": "EST",
    "ADM0_A3_UA": "EST",
    "ADM0_A3_UN": -99,
    "ADM0_A3_US": "EST",
    "ADM0_A3_VN": "EST",
    "ADM0_A3_WB": -99,
    "ADM0_DIF": 0,
    "ADM0_DIFF": null,
    "ADM0_ISO": "EST",
    "ADM0_TLC": "EST",
    "ADMIN": "Estonia",
    "BRK_A3": "EST",
    "BRK_DIFF": 0,
    "BRK_GROUP": null,
    "BRK_NAME": "Estonia",
    "CONTINENT": "Europe",
    "ECONOMY": "2. Developed region: nonG7",
    "FCLASS_AR": null,
    "FCLASS_BD": null,
    "FCLASS_BR": null,
    "FCLASS_CN": null,
    "FCLASS_DE": null,
    "FCLASS_EG": null,
    "FCLASS_ES": null,
    "FCLASS_FR": null,
    "FCLASS_GB": null,
    "FCLASS_GR": null,
    "FCLASS_ID": null,
    "FCLASS_IL": null,
    "FCLASS_IN": null,
    "FCLASS_ISO": "Admin-0 country",
    "FCLASS_IT": null,
    "FCLASS_JP": null,
    "FCLASS_KO": null,
    "FCLASS_MA": null,
    "FCLASS_NL": null,
    "FCLASS_NP": null,
    "FCLASS_PK": null,
    "FCLASS_PL": null,
    "FCLASS_PS": null,
    "FCLASS_PT": null,
    "FCLASS_RU": null,
    "FCLASS_SA": null,
    "FCLASS_SE": null,
    "FCLASS_TLC": "Admin-0 country",
    "FCLASS_TR": null,
    "FCLASS_TW": null,
    "FCLASS_UA": null,
    "FCLASS_US": null,
    "FCLASS_VN": null,
    "FIPS_10": "EN",
    "FORMAL_EN": "Republic of Estonia",
    "FORMAL_FR": null,
    "GDP_MD": 31471,
    "GDP_YEAR": 2019,
    "GEOUNIT": "Estonia",
    "GEOU_DIF": 0,
    "GU_A3": "EST",
    "HOMEPART": 1,
    "INCOME_GRP": "1. High income: OECD",
    "ISO_A2": "EE",
    "ISO_A2_EH": "EE",
    "ISO_A3": "EST",
    "ISO_A3_EH": "EST",
    "ISO_N3": "233",
    "ISO_N3_EH": "233",
    "LABELRANK": 6,
    "LABEL_X": 25.867126,
    "LABEL_Y": 58.724865,
    "LEVEL": 2,
    "LONG_LEN": 7,
    "MAPCOLOR13": 10,
    "MAPCOLOR7": 3,
    "MAPCOLOR8": 2,
    "MAPCOLOR9": 1,
    "MAX_LABEL": 8.0,
    "MIN_LABEL": 3.0,
    "MIN_ZOOM": 0.0,
    "NAME": "Estonia",
    "NAME_ALT": null,
    "NAME_AR": "إستونيا",
    "NAME_BN": "এস্তোনিয়া",
    "NAME_CIAWF": "Estonia",
    "NAME_DE": "Estland",
    "NAME_EL": "Εσθονία",
    "NAME_EN": "Estonia",
    "NAME_ES": "Estonia",
    "NAME_FA": "استونی",
    "NAME_FR": "Estonie",
    "NAME_HE": "אסטוניה",
    "NAME_HI": "एस्टोनिया",
    "NAME_HU": "Észtország",
    "NAME_ID": "Estonia",
    "NAME_IT": "Estonia",
    "NAME_JA": "エストニア",
    "NAME_KO": "에스토니아",
    "NAME_LEN": 7,
    "NAME_LONG": "Estonia",
    "NAME_NL": "Estland",
    "NAME_PL": "Estonia",
    "NAME_PT": "Estónia",
    "NAME_RU": "Эстония",
    "NAME_SORT": "Estonia",
    "NAME_SV": "Estland",
    "NAME_TR": "Estonya",
    "NAME_UK": "Естонія",
    "NAME_UR": "استونیا",
    "NAME_VI": "Estonia",
    "NAME_ZH": "爱沙尼亚",
    "NAME_ZHT": "愛沙尼亞",
    "NE_ID": 1159320615,
    "NOTE_ADM0": null,
    "NOTE_BRK": null,
    "POP_EST": 1326590.0,
    "POP_RANK": 12,
    "POP_YEAR": 2019,
    "POSTAL": "EST",
    "REGION_UN": "Europe",
    "REGION_WB": "Europe & Central Asia",
    "SOVEREIGNT": "Estonia",
    "SOV_A3": "EST",
    "SUBREGION": "Northern Europe",
    "SUBUNIT": "Estonia",
    "SU_A3": "EST",
    "SU_DIF": 0,
    "TINY": -99,
    "TLC": "1",
    "TLC_DIFF": null,
    "TYPE": "Sovereign country",
    "UN_A3": "233",
    "WB_A2": "EE",
    "WB_A3": "EST",
    "WIKIDATAID": "Q191",
    "WOE_ID": 23424805,
    "WOE_ID_EH": 23424805,
    "WOE_NOTE": "Exact WOE match as country",
    "featurecla": "Admin-0 country",
    "scalerank": 0
  }
]

There are 250 unique country codes in this dataset.

$ ~/duckdb ne.duckdb

SELECT COUNT(DISTINCT ADM0_A3_DE)
FROM   ne; -- 250

The geometry is a little over 19 MB in WKT format.

SELECT SUM(LENGTH(geom::TEXT)) / 1024 ** 2
FROM   ne; -- 19.976508140563965

Their Northern Border for Bahrain cuts a lot of corners.

Scoreboard

During this review, I also checked the borders around Monaco and Estonia. Below is a summary of the accuracy of the borders around these three countries, the size of their geometry in WKT format and the number of unique country codes.

The following shows the overlap between the datasets that cut corners around Bahrain's Northern coast. I excluded WOF as it made the map too complex.

American Data Centers

2025-11-18T20:00:00+02:00

In September, Business Insider published a video on the locations, ownership and power and water consumption of America's Data Centers. They collected diesel generator permits from every state in the US and mapped the applicant's company name back to their parent company. This revealed 1,240 sites along with a large amount of metadata for each location.

They published an interactive map with the underlying dataset embedded as JavaScript.

In this post, I'll explore the Business Insider's dataset.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use Python 3.12.3 and a few other tools to help analyse the data in this post.

$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt update
$ sudo apt install \
    jq \
    python3-pip \
    python3.12-venv

I'll set up a Python Virtual Environment and a JavaScript interpreter for Python.

$ python3 -m venv ~/.dcs
$ source ~/.dcs/bin/activate
$ python -m pip install \
    duckdb \
    esprima \
    levenshtein

I'll use DuckDB v1.4.1, along with its H3, JSON, Lindel, Parquet and Spatial extensions, in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.4.1/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

I used QGIS' Tile+ plugin to add basemaps from Esri and Bing to the maps in this post.

Analysis-Ready Data

I'll download the JavaScript used for Business Insider's interactive map.

$ mkdir -p ~/business_insider
$ cd ~/business_insider

$ wget https://tbimedia.s3.us-east-1.amazonaws.com/bistudios/_00/dev_edit/graphics/2025/09/2025-09-datacenters-map-table/index.js

I'll use Python to convert the dataset into line-delimited JSON.

$ python3

import json

import esprima


resp = esprima.parseScript(
            open('index.js', 'r')
                .read()
                .split('features:Q0},')[-1]
                .split(',I9=[];')[0])


def get_key(prop):
    if prop.key.value:
        return prop.key.value.lower()

    if prop.key.name:
        return prop.key.name.lower()

    return 'unknown'


dataset = [
    {get_key(prop): prop.value.value
     for prop in element.properties}
    for element in resp.body[0].expression.right.elements]

with open('data_centres.json', 'w') as f:
    for rec in dataset:
        f.write(json.dumps(rec, sort_keys=True) + '\n')

I'll use DuckDB to convert the JSON into a Parquet file. This will make analysis quicker than any of the above formats.

$ ~/duckdb

COPY (
    SELECT   * EXCLUDE(long, lat),
             ST_POINT(long::FLOAT, lat::FLOAT) geometry
    FROM     'data_centres.json'
    ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geometry)),
                             ST_X(ST_CENTROID(geometry))]::double[2])
) TO 'data_centres.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

The above produced a 234 KB, 1,240-row, 99-column Parquet file.

Data Fluency

Below is an example record from this dataset. It's of a hyperscaler-level site owned by Apple in Arizona.

$ echo "FROM  'data_centres.parquet'
        WHERE address LIKE '%Mesa AZ 85212'
        AND   brand = 'Apple'
        LIMIT 1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "# facilities note": "",
    "# of tier 2": "51",
    "# of tier 3": "",
    "# of tier 4": "",
    "address": "3740 S Signal Butte Rd, Mesa AZ 85212",
    "annual water consumption (gallons)": "-",
    "aquifer name": "-",
    "avert region": "Southwest",
    "brand": "Apple",
    "brand source": "https://www.sec.gov/Archives/edgar/data/1394954/000144530514004770/a8-kapplesettlementagreeme.htm",
    "case status desc": "-",
    "city": "Mesa",
    "co tpy": "-",
    "co2e tpy": "-",
    "company": "Platypus Development LLC",
    "county": "Maricopa",
    "daily water consumption (gallons)": "-",
    "data center construct year": "",
    "estimate power consumption in kw/hr (calculated 30%)": "55864",
    "estimate power consumption in kw/hr (calculated 50%)": "93106",
    "estimate power consumption in kw/hr (calculated 60%)": "111,727",
    "estimated at 2n": "46553",
    "estimated at 2n (in twh)": "0.408",
    "estimated at n +1(in twh)": "0.652",
    "estimated at n+1": "74484.8",
    "estimated data center electricity use in megawatt-hours at 50% capacity": "93.11",
    "estimated data center electricity use in terrawatt-hours a year at 50% capacity": "0.82",
    "first permit issue year": "2020",
    "generator type": "Caterpillar",
    "geometry": "POINT (-111.60407257080078 33.34719467163086)",
    "large facilites": "Possible hyperscaler",
    "latest permit issue year": "2022",
    "link to records": "Apple - Platypus Development LLC.docx",
    "major basin name": "North America, Colorado",
    "minor basin name": "Middle Gila",
    "new permit or existing update?": "Existing update",
    "nox tpy": "90",
    "number of buildings": "",
    "number of epa enforcements": "-",
    "page": "5",
    "pm tpy": "-",
    "pm10 tpy": "-",
    "pm2.5 tpy": "-",
    "population within 1-mile radius - 10,000": "fewer than 10,000",
    "population within 1-mile radius - 5,000": "More than 5,000",
    "primary law": "-",
    "private equity or asset manager?": "",
    "private equity or asset manager? (source)": "",
    "rate capacity original": "249715 HP",
    "reporter": "Narimes",
    "reporter notes": "eight emergency generators rated at 3604 HP, two rated at 2206 HP, one rated at 762 HP, one rated at 1141 HP, fifty seven rated at 5646 HP",
    "size capacity over 100 mw": "-",
    "size category at 30%": "Possible hyperscaler",
    "size category at 50% capacity": "Possible hyperscaler",
    "size category at 50% capacity 'large' vs 'small'": "large-scale",
    "size category at 60%": "Possible hyperscaler",
    "sox tpy": "-",
    "state": "AZ",
    "state environmental justice concern": "no",
    "state percentile for ej index for diesel particulate matter": "32",
    "state percentile for ej index for drinking water non-compliance": "68",
    "state percentile for ej index for hazardous waste proximity": "39",
    "state percentile for ej index for lead paint indicator": "48",
    "state percentile for ej index for nitrogen dioxide (no2)": "27",
    "state percentile for ej index for ozone": "52",
    "state percentile for ej index for particulate matter": "35",
    "state percentile for ej index for rmp proximity": "44",
    "state percentile for ej index for superfund proximity": "68",
    "state percentile for ej index for toxic releases to air": "55",
    "state percentile for ej index for traffic proximity and volume": "33",
    "state percentile for ej index for underground storage tanks (ust) indicator": "36",
    "state percentile for ej index for wastewater discharge indicator": "32",
    "statename": "Arizona",
    "total generator rate capacity kw": "186212",
    "total penalties assessed": "-",
    "total population within 1 mile of site": "9,026",
    "unique id": "37",
    "unknown": "60518.9",
    "us environmental justice concern": "no",
    "us percentile for ej index for diesel particulate matter": "45",
    "us percentile for ej index for drinking water non-compliance": "79",
    "us percentile for ej index for hazardous waste proximity": "33",
    "us percentile for ej index for lead paint indicator": "15",
    "us percentile for ej index for nitrogen dioxide (no2)": "47",
    "us percentile for ej index for ozone": "62",
    "us percentile for ej index for particulate matter": "25",
    "us percentile for ej index for rmp proximity": "41",
    "us percentile for ej index for superfund proximity": "66",
    "us percentile for ej index for toxic releases to air": "51",
    "us percentile for ej index for traffic proximity and volume": "42",
    "us percentile for ej index for underground storage tanks (ust) indicator": "38",
    "us percentile for ej index for wastewater discharge indicator": "50",
    "vocs tpy": "-",
    "water notes": "-",
    "water record link": "-",
    "water requested?": "denied",
    "water stress": "Extremely High (>80%)",
    "zip": "85212"
  }
]

Below are the field names, data types, percentages of NULLs per column, number of unique values and minimum and maximum values for each column.

$ ~/duckdb

.maxrows 500

SELECT   column_name,
         column_type,
         null_percentage,
         approx_unique,
         min[:30],
         max[:30]
FROM     (SUMMARIZE
          FROM   'data_centres.parquet')
ORDER BY 1;

┌─────────────────────────────────────────────────────────────────────────────────┬─────────────┬─────────────────┬───────────────┬────────────────────────────────┬────────────────────────────────┐
│                                   column_name                                   │ column_type │ null_percentage │ approx_unique │            min[:30]            │            max[:30]            │
│                                     varchar                                     │   varchar   │  decimal(9,2)   │     int64     │            varchar             │            varchar             │
├─────────────────────────────────────────────────────────────────────────────────┼─────────────┼─────────────────┼───────────────┼────────────────────────────────┼────────────────────────────────┤
│ # facilities note                                                               │ VARCHAR     │            0.00 │             9 │                                │ 1 of 8 facilities              │
│ # of tier 2                                                                     │ VARCHAR     │            0.00 │            12 │                                │ 8                              │
│ # of tier 3                                                                     │ VARCHAR     │            0.00 │             2 │                                │ 1                              │
│ # of tier 4                                                                     │ VARCHAR     │            0.00 │             2 │                                │ 414                            │
│ address                                                                         │ VARCHAR     │            0.40 │          1360 │ 0 Williams Road, Palmetto, Geo │ intersection of Morse Rd and B │
│ annual water consumption (gallons)                                              │ VARCHAR     │            0.00 │            67 │                                │ aggregate                      │
│ aquifer name                                                                    │ VARCHAR     │            0.00 │             6 │ -                              │ Northern Great Plains / Interi │
│ avert region                                                                    │ VARCHAR     │            0.00 │            14 │ California                     │ Texas                          │
│ brand                                                                           │ VARCHAR     │            0.00 │           345 │ 11:11 Systems                  │ unWired Broadband              │
│ brand source                                                                    │ VARCHAR     │            0.24 │           620 │                                │ zColo = Databank               │
│ case status desc                                                                │ VARCHAR     │            0.00 │             5 │                                │ Resolved                       │
│ city                                                                            │ VARCHAR     │            0.00 │           376 │                                │ Wood Dale                      │
│ co tpy                                                                          │ VARCHAR     │            0.00 │           604 │                                │ 99.9                           │
│ co2e tpy                                                                        │ VARCHAR     │            0.00 │            38 │                                │ 9772.31                        │
│ company                                                                         │ VARCHAR     │            0.08 │           742 │ 1000 Coit Road, L.P.           │ zColo, LLC                     │
│ county                                                                          │ VARCHAR     │            0.00 │           198 │ Accomack                       │ Yolo                           │
│ daily water consumption (gallons)                                               │ VARCHAR     │            0.00 │            10 │                                │ aggregate water                │
│ data center construct year                                                      │ VARCHAR     │            0.00 │             5 │                                │ in progress                    │
│ estimate power consumption in kw/hr (calculated 30%)                            │ VARCHAR     │            0.00 │           630 │ 100                            │ redacted                       │
│ estimate power consumption in kw/hr (calculated 50%)                            │ VARCHAR     │            0.00 │           801 │ 100                            │ redacted                       │
│ estimate power consumption in kw/hr (calculated 60%)                            │ VARCHAR     │            0.00 │           778 │ 1,048                          │ redacted                       │
│ estimated at 2n                                                                 │ VARCHAR     │            0.00 │           930 │ 100                            │ redacted                       │
│ estimated at 2n (in twh)                                                        │ VARCHAR     │            0.00 │           335 │ 0                              │ redacted                       │
│ estimated at n +1(in twh)                                                       │ VARCHAR     │            0.00 │           388 │ 0                              │ redacted                       │
│ estimated at n+1                                                                │ VARCHAR     │            0.00 │           881 │ 100                            │ redacted                       │
│ estimated data center electricity use in megawatt-hours at 50% capacity         │ VARCHAR     │            0.00 │           820 │ 0.02                           │ redacted                       │
│ estimated data center electricity use in terrawatt-hours a year at 50% capacity │ VARCHAR     │            0.00 │           130 │ 0                              │ redacted                       │
│ first permit issue year                                                         │ VARCHAR     │            0.00 │            38 │ 1976                           │ 2024                           │
│ generator type                                                                  │ VARCHAR     │            0.08 │           134 │                                │ Waukesha; Doosan               │
│ geometry                                                                        │ GEOMETRY    │            0.00 │          1165 │ POINT (-112.328125 33.57888793 │ POINT (-121.95311737060547 37. │
│ large facilites                                                                 │ VARCHAR     │            0.00 │             5 │ Possible hyperscaler           │ redacted                       │
│ latest permit issue year                                                        │ VARCHAR     │            0.00 │            34 │ 1977                           │ 2025                           │
│ link to records                                                                 │ VARCHAR     │            0.00 │           876 │                                │ wsdc0101.pdf wsdc0102.pdf wsdc │
│ major basin name                                                                │ VARCHAR     │            0.00 │            11 │                                │ United States, North Atlantic  │
│ minor basin name                                                                │ VARCHAR     │            0.00 │           184 │                                │ Wheeler Lake                   │
│ new permit or existing update?                                                  │ VARCHAR     │            0.00 │             2 │ Existing update                │ New                            │
│ nox tpy                                                                         │ VARCHAR     │            0.00 │           712 │ -                              │ 99.97                          │
│ number of buildings                                                             │ VARCHAR     │            0.00 │            10 │                                │ 8                              │
│ number of epa enforcements                                                      │ VARCHAR     │            0.00 │             5 │                                │ 3                              │
│ page                                                                            │ VARCHAR     │            0.00 │           209 │                                │ rows 9481-9484                 │
│ pm tpy                                                                          │ VARCHAR     │            0.00 │           280 │                                │ 9.9                            │
│ pm10 tpy                                                                        │ VARCHAR     │            0.00 │           345 │                                │ 9.97                           │
│ pm2.5 tpy                                                                       │ VARCHAR     │            0.00 │           344 │                                │ 9.9                            │
│ population within 1-mile radius - 10,000                                        │ VARCHAR     │            0.00 │             3 │ -                              │ fewer than 10,000              │
│ population within 1-mile radius - 5,000                                         │ VARCHAR     │            0.00 │             3 │                                │ fewer than 5,000               │
│ primary law                                                                     │ VARCHAR     │            0.00 │             4 │                                │ CWA                            │
│ private equity or asset manager?                                                │ VARCHAR     │            0.00 │            20 │                                │ pg 1 PI32426_PCP210001.pdf     │
│ private equity or asset manager? (source)                                       │ VARCHAR     │            0.00 │            29 │                                │ https://www.streamdatacenters. │
│ rate capacity original                                                          │ VARCHAR     │            0.00 │           453 │                                │ [combined]                     │
│ reporter                                                                        │ VARCHAR     │            0.00 │            12 │                                │ Yuheng/Rosemarie               │
│ reporter notes                                                                  │ VARCHAR     │           16.69 │          1142 │                                │ two operating scenarios: 1) fo │
│ size capacity over 100 mw                                                       │ VARCHAR     │            0.00 │             3 │                                │ over 100 MW                    │
│ size category at 30%                                                            │ VARCHAR     │            0.00 │             5 │ Possible hyperscaler           │ redacted                       │
│ size category at 50% capacity                                                   │ VARCHAR     │            0.00 │             5 │ Possible hyperscaler           │ redacted                       │
│ size category at 50% capacity 'large' vs 'small'                                │ VARCHAR     │            0.00 │             5 │ large-scale                    │ small-scale                    │
│ size category at 60%                                                            │ VARCHAR     │            0.00 │             5 │ Possible hyperscaler           │ redacted                       │
│ sox tpy                                                                         │ VARCHAR     │            0.00 │           243 │                                │ 9.907                          │
│ state                                                                           │ VARCHAR     │            0.00 │            50 │ AL                             │ WY                             │
│ state environmental justice concern                                             │ VARCHAR     │            0.00 │             4 │                                │ yes                            │
│ state percentile for ej index for diesel particulate matter                     │ VARCHAR     │            0.00 │            85 │                                │ 99                             │
│ state percentile for ej index for drinking water non-compliance                 │ VARCHAR     │            0.00 │            41 │                                │ 99                             │
│ state percentile for ej index for hazardous waste proximity                     │ VARCHAR     │            0.00 │            84 │                                │ 99                             │
│ state percentile for ej index for lead paint indicator                          │ VARCHAR     │            0.00 │            92 │                                │ 98                             │
│ state percentile for ej index for nitrogen dioxide (no2)                        │ VARCHAR     │            0.00 │            88 │                                │ 98                             │
│ state percentile for ej index for ozone                                         │ VARCHAR     │            0.00 │            92 │                                │ 99                             │
│ state percentile for ej index for particulate matter                            │ VARCHAR     │            0.00 │            90 │                                │ 99                             │
│ state percentile for ej index for rmp proximity                                 │ VARCHAR     │            0.00 │            84 │                                │ 99                             │
│ state percentile for ej index for superfund proximity                           │ VARCHAR     │            0.00 │            59 │                                │ 99                             │
│ state percentile for ej index for toxic releases to air                         │ VARCHAR     │            0.00 │            92 │                                │ 98                             │
│ state percentile for ej index for traffic proximity and volume                  │ VARCHAR     │            0.00 │            92 │                                │ 99                             │
│ state percentile for ej index for underground storage tanks (ust) indicator     │ VARCHAR     │            0.00 │            81 │                                │ 98                             │
│ state percentile for ej index for wastewater discharge indicator                │ VARCHAR     │            0.00 │            92 │                                │ 99                             │
│ statename                                                                       │ VARCHAR     │            0.00 │            49 │ Alabama                        │ Wyoming                        │
│ total generator rate capacity kw                                                │ VARCHAR     │            0.00 │           825 │ 100                            │ redacted                       │
│ total penalties assessed                                                        │ VARCHAR     │            0.00 │            38 │                                │ -                              │
│ total population within 1 mile of site                                          │ VARCHAR     │            0.00 │          1166 │ -                              │ 994                            │
│ unique id                                                                       │ VARCHAR     │            0.00 │          1103 │ 10                             │ 999                            │
│ unknown                                                                         │ VARCHAR     │            0.00 │           923 │ 1006.53                        │ redacted                       │
│ us environmental justice concern                                                │ VARCHAR     │            0.00 │             4 │                                │ yes                            │
│ us percentile for ej index for diesel particulate matter                        │ VARCHAR     │            0.00 │            92 │                                │ 99                             │
│ us percentile for ej index for drinking water non-compliance                    │ VARCHAR     │            0.00 │            27 │                                │ 99                             │
│ us percentile for ej index for hazardous waste proximity                        │ VARCHAR     │            0.00 │            78 │                                │ 99                             │
│ us percentile for ej index for lead paint indicator                             │ VARCHAR     │            0.00 │            78 │                                │ 97                             │
│ us percentile for ej index for nitrogen dioxide (no2)                           │ VARCHAR     │            0.00 │            92 │                                │ 99                             │
│ us percentile for ej index for ozone                                            │ VARCHAR     │            0.00 │            92 │                                │ 99                             │
│ us percentile for ej index for particulate matter                               │ VARCHAR     │            0.00 │            88 │ -                              │ 99                             │
│ us percentile for ej index for rmp proximity                                    │ VARCHAR     │            0.00 │            69 │                                │ 99                             │
│ us percentile for ej index for superfund proximity                              │ VARCHAR     │            0.00 │            39 │                                │ 98                             │
│ us percentile for ej index for toxic releases to air                            │ VARCHAR     │            0.00 │            92 │                                │ 99                             │
│ us percentile for ej index for traffic proximity and volume                     │ VARCHAR     │            0.00 │            91 │                                │ 99                             │
│ us percentile for ej index for underground storage tanks (ust) indicator        │ VARCHAR     │            0.00 │            69 │                                │ 99                             │
│ us percentile for ej index for wastewater discharge indicator                   │ VARCHAR     │            0.00 │            91 │                                │ 98                             │
│ vocs tpy                                                                        │ VARCHAR     │            0.00 │           382 │                                │ 9.94                           │
│ water notes                                                                     │ VARCHAR     │            0.08 │            51 │                                │ usage is from Jan 2023-Dec 202 │
│ water record link                                                               │ VARCHAR     │            0.00 │            47 │                                │ https://www.denverpost.com/202 │
│ water requested?                                                                │ VARCHAR     │            0.00 │             7 │                                │ yes                            │
│ water stress                                                                    │ VARCHAR     │            0.00 │             6 │ Arid and Low Water Use         │ Medium - High (20-40%)         │
│ zip                                                                             │ VARCHAR     │            0.00 │           552 │ -                              │ 99019                          │
├─────────────────────────────────────────────────────────────────────────────────┴─────────────┴─────────────────┴───────────────┴────────────────────────────────┴────────────────────────────────┤
│ 98 rows                                                                                                                                                                                 6 columns │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

A lot could be done to group these fields into dictionaries and normalise their values. If this post proves popular, I might revisit this.

Diesel Generator Permits

Virginia has published a list of 188 diesel generator permits they've issued to data centers in their state.

In some cases, the well-known trading name of the applicant will be stated. The following is a quote from a permit issued to AWS.

Attached is a permit to construct and operate emergency diesel engine generator sets (gen-sets) at Amazon Data Services’ data centers (IAD-51, IAD-56, IAD-88, IAD-89, IAD-192, and IAD214), in accordance with the provisions of the Commonwealth of Virginia State Air Pollution Control Board (Board’s) Regulations for the Control and Abatement of Air Pollution (Regulations). This permit document combines the terms and conditions from, and supersedes your permit document dated October 12, 2023.

Below is a manifest of diesel generators they're using at their site.

Not all states make permits accessible via google searches. If anyone is looking to continue this research, the BBC listed the top jurisdictions and their market share in and outside of the US where data centers are located. This can help narrow the search space.

14% Northern Virginia
 6% Oregon
 4% Iowa
 3% Dallas, Texas
 2% Arizona
 2% Nebraska
 2% Illinois

6% Beijing
4% Dublin
2% Shanghai

Data Center Owners

Below are the most and least common brands represented in this dataset.

$ ~/duckdb

SELECT   COUNT(*),
         brand
FROM     'data_centres.parquet'
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬──────────────────────────────────────────────────────────────┐
│ count_star() │                            brand                             │
│    int64     │                           varchar                            │
├──────────────┼──────────────────────────────────────────────────────────────┤
│          177 │ Amazon                                                       │
│           70 │ Digital Realty                                               │
│           53 │ Equinix                                                      │
│           47 │ Google                                                       │
│           44 │ Microsoft                                                    │
│           34 │ QTS                                                          │
│           33 │ DataBank                                                     │
│           33 │ Centersquare                                                 │
│           32 │ Meta                                                         │
│           31 │ CyrusOne                                                     │
│           31 │ Lumen Technologies                                           │
│           23 │ Verizon                                                      │
│           19 │ TierPoint                                                    │
│           18 │ Flexential                                                   │
│           17 │ CoreSite                                                     │
│           16 │ Iron Mountain                                                │
│           15 │ NTT                                                          │
│           14 │ Stack Infrastructure                                         │
│           13 │ Cogent                                                       │
│           13 │ EdgeConneX                                                   │
│            · │     ·                                                        │
│            · │     ·                                                        │
│            · │     ·                                                        │
│            1 │ SOMO village                                                 │
│            1 │ Datacate                                                     │
│            1 │ PC Solutions                                                 │
│            1 │ Showers Development, LLC                                     │
│            1 │ US Signal                                                    │
│            1 │ Reliance Industries                                          │
│            1 │ Inova                                                        │
│            1 │ George Washington University                                 │
│            1 │ The Vanguard Group                                           │
│            1 │ University of Minnesota - Minnesota Supercomputing Institute │
│            1 │ WW Grainger                                                  │
│            1 │ MasterCard                                                   │
│            1 │ PSCU Financial Services                                      │
│            1 │ SRI Ten 706 Wilshire LLC                                     │
│            1 │ Rosegate LLC                                                 │
│            1 │ Stanford University Data Center                              │
│            1 │ California Legislative Counsel                               │
│            1 │ Adobe                                                        │
│            1 │ American Honda Motor Company                                 │
│            1 │ NeuStar                                                      │
├──────────────┴──────────────────────────────────────────────────────────────┤
│ 305 rows (40 shown)                                               2 columns │
└─────────────────────────────────────────────────────────────────────────────┘

Locations Heatmap

Below is a heatmap of the data center locations. The brighter hexagons have more sites.

$ ~/duckdb

CREATE OR REPLACE TABLE h3_3_stats AS
    SELECT   H3_LATLNG_TO_CELL(
                ST_Y(ST_CENTROID(geometry)),
                ST_X(ST_CENTROID(geometry)), 3) AS h3_3,
             COUNT(*) num_buildings
    FROM     'data_centres.parquet'
    GROUP BY 1;

COPY (
    SELECT ST_ASWKB(H3_CELL_TO_BOUNDARY_WKT(h3_3)::geometry) geometry,
           num_buildings
    FROM   h3_3_stats
    WHERE  ST_XMIN(geometry::geometry) BETWEEN -179 AND 179
    AND    ST_XMAX(geometry::geometry) BETWEEN -179 AND 179
) TO 'h3_3_stats.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

In many cases, a single record points to a single building.

But in other cases, the point rests on the parcel of land that more than one facility is located on.

The metadata does mention when there is more than one building being referred to in the record but it would be nice to see this dataset turned into a building footprint-specific dataset at some point.

Hyperscalers

Each location has some indication if its a hyperscaler location or not.

SELECT   "large facilites",
         COUNT(*)
FROM     'data_centres.parquet'
GROUP BY 1;

┌──────────────────────┬──────────────┐
│   large facilites    │ count_star() │
│       varchar        │    int64     │
├──────────────────────┼──────────────┤
│ Possible hyperscaler │          156 │
│ multiple permited    │          165 │
│ no value             │           19 │
│ not a hyperscaler    │          880 │
│ redacted             │           20 │
└──────────────────────┴──────────────┘

These are the number of hyperscaler sites by brand. It's interesting to see Apple only has two locations.

SELECT   brand,
         COUNT(*)
FROM     'data_centres.parquet'
WHERE    "large facilites" = 'Possible hyperscaler'
GROUP BY 1
ORDER BY 2 DESC;

┌─────────────────────────────┬──────────────┐
│            brand            │ count_star() │
│           varchar           │    int64     │
├─────────────────────────────┼──────────────┤
│ Amazon                      │           45 │
│ Microsoft                   │           21 │
│ Google                      │           12 │
│ QTS                         │           12 │
│ Aligned Data Centers        │            9 │
│ Digital Realty              │            9 │
│ Meta                        │            8 │
│ CyrusOne                    │            6 │
│ Vantage Data Centers        │            5 │
│ NTT                         │            5 │
│ CloudHQ                     │            4 │
│ Stack Infrastructure        │            4 │
│ Compass Datacenters         │            3 │
│ Apple                       │            2 │
│ Yondr                       │            1 │
│ Stream Data Centers         │            1 │
│ Edged Energy                │            1 │
│ Sabey Data Centers          │            1 │
│ Corscale Data Centers       │            1 │
│ Iron Mountain               │            1 │
│ CoreSite                    │            1 │
│ Skybox                      │            1 │
│ Cologix                     │            1 │
│ Equinix                     │            1 │
│ US National Security Agency │            1 │
├─────────────────────────────┴──────────────┤
│ 25 rows                          2 columns │
└────────────────────────────────────────────┘

Below are the individual hyperscaler locations along with the year of their first permit.

OpenStreetMap Data

I looked for metadata in OpenStreetMap (OSM) for data centers so I could get an idea for how it compares against Business Insider's dataset. I was only able to location ~900 building footprints with OSM's Layercake dataset, which is updated weekly.

$ ~/duckdb

SELECT   COUNT(*),
         tags.building,
         tags."building:use"
FROM     'https://data.openstreetmap.us/layercake/buildings.parquet'
where    tags.building       ILIKE '%data%'
OR       tags."building:use" ILIKE '%data%'
GROUP BY 2, 3
ORDER BY 1 DESC;

┌──────────────┬────────────────────────────────────────────────────────────┬──────────────────┐
│ count_star() │                          building                          │   building:use   │
│    int64     │                          varchar                           │     varchar      │
├──────────────┼────────────────────────────────────────────────────────────┼──────────────────┤
│          901 │ data_center                                                │ NULL             │
│            8 │ datacenter                                                 │ NULL             │
│            8 │ data_centre                                                │ NULL             │
│            3 │ data                                                       │ NULL             │
│            3 │ industrial                                                 │ data_center      │
│            2 │ data center                                                │ NULL             │
│            1 │ Structure added, not on data, appears on satellite imagery │ NULL             │
│            1 │ sa_data_yaye                                               │ NULL             │
│            1 │ data_center                                                │ industrial       │
│            1 │ Ciber DataClic                                             │ NULL             │
│            1 │ office                                                     │ data_center      │
│            1 │ yes                                                        │ NDATANG K KORUNG │
│            1 │ industrial                                                 │ datacenter       │
│            1 │ apartments                                                 │ data_center      │
├──────────────┴────────────────────────────────────────────────────────────┴──────────────────┤
│ 14 rows                                                                            3 columns │
└──────────────────────────────────────────────────────────────────────────────────────────────┘

Below are the data centers in OSM for the Bay Area.

Below are Business Insider's.

I noted that telecom: data_center or variations of that attribute are used from time to time in OSM. Below is one example for an Apple data center.

This attribute isn't making its way into Layercake. I've raised a ticket so hopefully at some point, that might help fill in the gaps in OSM's DC coverage.

$ echo "SELECT * EXCLUDE(tags),
               tags: tags::JSON
        FROM   'https://data.openstreetmap.us/layercake/buildings.parquet'
        WHERE  id = 300974499
        LIMIT  1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "bbox": "{'xmin': -111.60577, 'ymin': 33.34491, 'xmax': -111.60252, 'ymax': 33.349155}",
    "geometry": "MULTIPOLYGON (((-111.6057634 33.3462637, -111.6057589 33.3458225, -111.6054351 33.3458248, -111.6054259 33.3449094, -111.6027724 33.344928, -111.60278 33.3456873, -111.6025203 33.3456891, -111.6025379 33.3474469, -111.6025464 33.3482889, -111.6028057 33.3482871, -111.6028144 33.3491573, -111.6054686 33.3491387, -111.6054397 33.346266, -111.6057634 33.3462637)))",
    "id": 300974499,
    "tags": {
      "access": null,
      "addr:city": "Mesa",
      "addr:housenumber": "3740",
      "addr:postcode": "85212",
      "addr:street": "South Signal Butte Road",
      "building": "industrial",
      "building:colour": null,
      "building:flats": null,
      "building:levels": null,
      "building:material": null,
      "building:part": null,
      "building:use": "data_center",
      "height": null,
      "name": "Apple Data Center",
      "roof:colour": null,
      "roof:height": null,
      "roof:levels": null,
      "roof:material": null,
      "roof:orientation": null,
      "roof:shape": null,
      "start_date": "2012",
      "website": null,
      "wheelchair": null,
      "wikidata": null,
      "wikipedia": null
    },
    "type": "way"
  }
]

Diesel Generators

I wanted to do some analysis of the manufacturers of the diesel generators. Around 1/3rd of the records either don't contain the manufacturer(s) or my normalisation script below wasn't able to work on those records properly.

$ python3

from collections import Counter
import re

import duckdb
from   Levenshtein import levenshtein_cpp


brands = [
    'Allis-Chalmers',
    'Alzeta',
    'Baudouin',
    'Baylor',
    'Blue Star',
    'Caterpillar',
    'Clarke',
    'Cummins',
    'Detroit Diesel',
    'Deutz',
    'Doosan',
    'Generac',
    'Hino',
    'Hitec',
    'John Deere',
    'Katolight',
    'Kohler',
    'Mercedes-Benz',
    'Mitsubishi',
    'MTU',
    'Olympian',
    'Onan',
    'Perkins',
    'Rolls Royce',
    'Solar Saturn',
    'Turbinen-Union',
    'Volvo',
    'Waukesha',]

duckdb.sql('INSTALL parquet')
duckdb.sql('LOAD parquet')

sql = '''SELECT "generator type"
         FROM   'data_centres.parquet';'''

resp = []

for raw_val in duckdb.sql(sql).fetchall():
    if raw_val is None or raw_val[0] is None:
        continue

    raw_vals = re.split('[^a-zA-Z]', raw_val[0])

    brand_set = set()

    for brand in brands:
        for brand_part in re.split('[^a-zA-Z]', brand):
            for raw_val in raw_vals:
                if float(levenshtein_cpp.seqratio(brand, raw_val)) > 0.7:
                    brand_set.add(brand)

    resp.append(sorted(brand_set))

Counter(','.join(list(x)) for x in resp).most_common()

[('', 417),
 ('Caterpillar', 364),
 ('Cummins', 182),
 ('Caterpillar,Cummins', 69),
 ('MTU', 54),
 ('Kohler', 21),
 ('Caterpillar,MTU', 16),
 ('Caterpillar,Kohler', 9),
 ('Generac', 9),
 ('Caterpillar,Cummins,MTU', 7),
 ('Cummins,Kohler', 7),
 ('Volvo', 7),
 ('Mitsubishi', 6),
 ('Cummins,MTU', 6),
 ('Waukesha', 5),
 ('Kohler,MTU', 4),
 ('Clarke', 4),
 ('Caterpillar,Clarke', 3),
 ('Caterpillar,Cummins,Kohler', 3),
 ('Cummins,Generac', 3),
 ('Katolight', 2),
 ('Cummins,Onan', 2),
 ('Caterpillar,Onan', 2),
 ('Onan', 2),
 ('Cummins,Katolight', 2),
 ('Deutz', 2),
 ('Mercedes-Benz', 2),
 ('Clarke,Cummins', 2),
 ('Caterpillar,MTU,Mitsubishi', 2),
 ('Doosan,Waukesha', 2),
 ('Caterpillar,Katolight,MTU', 1),
 ('Caterpillar,Generac', 1),
 ('Caterpillar,Clarke,MTU', 1),
 ('Caterpillar,Clarke,Volvo', 1),
 ('Caterpillar,Kohler,MTU', 1),
 ('Clarke,Cummins,MTU', 1),
 ('Caterpillar,Cummins,MTU,Olympian', 1),
 ('Baylor', 1),
 ('Caterpillar,Cummins,Hitec,Kohler', 1),
 ('Allis-Chalmers,Caterpillar,Cummins', 1),
 ('Alzeta,Caterpillar', 1),
 ('Allis-Chalmers', 1),
 ('Caterpillar,Perkins', 1),
 ('Caterpillar,Cummins,Generac,Hino', 1),
 ('Caterpillar,Cummins,Generac', 1),
 ('Caterpillar,Cummins,Generac,MTU', 1),
 ('Cummins,Mitsubishi', 1),
 ('Cummins,Perkins', 1),
 ('Caterpillar,Olympian,Perkins', 1),
 ('Caterpillar,Olympian', 1),
 ('Kohler,Turbinen-Union', 1),
 ('Caterpillar,Waukesha', 1),
 ('Baudouin', 1)]

I suspect running AI and OCR on the underlying permit PDFs would yield a richer dataset where model numbers, power ratings, etc.. could be extracted and used to model power usage during blackouts, etc...

I haven't seen the underlying PDFs published anywhere yet but hopefully they will be in the near future.

American Wind Farms

2025-11-06T09:00:00+02:00

In May, the US Geological Survey (USGS) published version 8.1 of the U.S. Wind Turbine Database (USWTDB). The first version was published back in 2018 and it documents the location 76K+ wind turbines, their capabilities and associated project data across the US and its territories.

Below is a heatmap of their locations.

In this post, I'll walk through converting this dataset into Parquet and examining some of its features.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use GDAL 3.9.3 and a few other tools to help analyse the data in this post.

$ sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable
$ sudo apt update
$ sudo apt install \
    gdal-bin \
    jq

I'll use DuckDB, along with its H3, JSON, Lindel, Parquet and Spatial extensions in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.4.1/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

Most of the maps in this post were rendered with QGIS version 3.44. QGIS is a desktop application that runs on Windows, macOS and Linux. The application has grown in popularity in recent years and has ~15M application launches from users all around the world each month.

I used QGIS' Tile+ plugin to add basemaps from NASA's Blue Marble and Bing for spatial context.

I also used the following GeoJSON files for counties and states.

Downloading the Dataset

I'll download the dataset and convert it into Parquet.

The original field names contained a lot of acronyms and truncated names. I've renamed and restructured these for easier human consumption. I've taken all the values representing missing values and converted them into NULLs. I've also converted any magic numbers into English strings.

$ mkdir -p ~/usgs_wind
$ cd ~/usgs_wind

$ wget https://energy.usgs.gov/uswtdb/assets/data/uswtdbSHP.zip
$ unzip uswtdbSHP.zip

$ ~/duckdb

COPY (
    SELECT
        case_id: case_id::INT,
        eia_id:  IF(eia_id::TEXT='-9999', NULL, eia_id::INT),
        faa: {
            asn: IF(faa_asn='missing', NULL, faa_asn),
            ors: IF(faa_ors='missing', NULL, faa_ors),
        },
        project: {
            avg_turbine_capacity: IF(p_cap::TEXT='-9999.0', NULL, p_cap::DOUBLE),
            name:                 p_name,
            num_turbines:         p_tnum::INT,
            online_year:          IF(p_year::TEXT='-9999', NULL, p_year::INT),
        },
        geometry: geom,
        bbox:     {'xmin': ST_XMIN(ST_EXTENT(geom)),
                   'ymin': ST_YMIN(ST_EXTENT(geom)),
                   'xmax': ST_XMAX(ST_EXTENT(geom)),
                   'ymax': ST_YMAX(ST_EXTENT(geom))},
        location: {
            country: t_county,
            fips:    IF(t_fips::VARCHAR='NA', NULL, t_fips::VARCHAR),
            state:   t_state,
        },
        turbine: {
            capacity:          IF(t_cap::TEXT='-9999', NULL, t_cap::DOUBLE),
            attribute_confidence:
                CASE WHEN t_conf_atr = 1 THEN 'Low'
                     WHEN t_conf_atr = 2 THEN 'Medium'
                     WHEN t_conf_atr = 3 THEN 'High'
                END,
            location_confidence:
                CASE WHEN t_conf_loc = 1 THEN 'Low'
                     WHEN t_conf_loc = 2 THEN 'Medium'
                     WHEN t_conf_loc = 3 THEN 'High'
                END,
            hub_height:        IF(t_hh::TEXT='-9999.0', NULL, t_hh::DOUBLE),
            imagery: {
                captured_at:   t_img_date,
                provider:      t_img_src,
            },
            manufacturer:      IF(t_manu='missing', NULL, t_manu),
            model:             IF(t_model='missing', NULL, t_model),
            is_offshore:       t_offshore=1,
            rated_capacity: IF(t_rd::TEXT='-9999.0', NULL, t_rd::DOUBLE),
            retrofitted_year:  IF(t_retro_yr::TEXT='-9999', NULL, t_retro_yr::INT),
            is_retrofitted:    t_retrofit=1,
            rsa:               IF(t_rsa::TEXT='-9999.0', NULL, t_rsa::DOUBLE),
            ttlh:              IF(t_ttlh::TEXT='-9999.0', NULL, t_ttlh::DOUBLE),
        },
        usgs_pr_id:            IF(usgs_pr_id::TEXT='-9999', NULL, usgs_pr_id::INT),
    FROM ST_READ('uswtdb_V8_1_20250522.shx')
    ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geom)),
                             ST_X(ST_CENTROID(geom))]::double[2])
) TO 'uswtdb_V8_1_20250522.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

Data Fluency

The following is an example record from this dataset.

$ echo "SELECT * EXCLUDE(bbox, faa, project, location, turbine),
               bbox::JSON     AS bbox,
               faa::JSON      AS faa,
               project::JSON  AS project,
               location::JSON AS location,
               turbine::JSON  AS turbine
        FROM   'uswtdb_V8_1_20250522.parquet'
        LIMIT  1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "bbox": {
      "xmax": 144.72265600000003,
      "xmin": 144.72265600000003,
      "ymax": 13.389381000000071,
      "ymin": 13.389381000000071
    },
    "case_id": 3063607,
    "eia_id": null,
    "faa": {
      "asn": "2013-WTW-2712-OE",
      "ors": null
    },
    "geometry": "POINT (144.72265600000003 13.389381000000071)",
    "location": {
      "country": "Guam",
      "fips": "66010",
      "state": "GU"
    },
    "project": {
      "avg_turbine_capacity": 0.275,
      "name": "Guam Power Authority Wind Turbine",
      "num_turbines": 1,
      "online_year": 2016
    },
    "turbine": {
      "attribute_confidence": "High",
      "capacity": 275.0,
      "hub_height": 55.0,
      "imagery": {
        "captured_at": "2017-08-10",
        "provider": "Maxar"
      },
      "is_offshore": false,
      "is_retrofitted": false,
      "location_confidence": "High",
      "manufacturer": "Vergnet",
      "model": "GEV MP-C",
      "rated_capacity": 32.0,
      "retrofitted_year": null,
      "rsa": 804.25,
      "ttlh": 71.0
    },
    "usgs_pr_id": null
  }
]

Below are the field names, data types, percentages of NULLs per column, number of unique values and minimum and maximum values for each column.

$ ~/duckdb

SELECT   column_name,
         column_type,
         null_percentage,
         approx_unique,
         min,
         max
FROM     (SUMMARIZE
          SELECT case_id,
                 eia_id,
                 faa.*,
                 location.*,
                 project.*,
                 turbine.* EXCLUDE(imagery),
                 turbine.imagery.captured_at,
                 turbine.imagery.provider,
                 usgs_pr_id,
          FROM   'uswtdb_V8_1_20250522.parquet')
ORDER BY 1;

┌──────────────────────┬─────────────┬─────────────────┬───────────────┬──────────────────┬─────────────────────┐
│     column_name      │ column_type │ null_percentage │ approx_unique │       min        │         max         │
│       varchar        │   varchar   │  decimal(9,2)   │     int64     │     varchar      │       varchar       │
├──────────────────────┼─────────────┼─────────────────┼───────────────┼──────────────────┼─────────────────────┤
│ asn                  │ VARCHAR     │            5.37 │         50703 │ 1987-AGL-900-OE  │ 2025-WTW-669-OE     │
│ attribute_confidence │ VARCHAR     │            0.00 │             3 │ High             │ Medium              │
│ avg_turbine_capacity │ DOUBLE      │            3.86 │           762 │ 0.05             │ 1055.6              │
│ capacity             │ DOUBLE      │            5.25 │           111 │ 50.0             │ 13000.0             │
│ captured_at          │ DATE        │            0.00 │          1669 │ 1970-01-01       │ 2025-03-25          │
│ case_id              │ INTEGER     │            0.00 │         78762 │ 3000001          │ 3140627             │
│ country              │ VARCHAR     │            0.00 │           566 │ Adair County     │ Zapata County       │
│ eia_id               │ INTEGER     │            5.17 │          1359 │ 90               │ 68318               │
│ fips                 │ VARCHAR     │            0.08 │           637 │ 02013            │ 72133               │
│ hub_height           │ DOUBLE      │            5.99 │           111 │ 19.0             │ 136.0               │
│ is_offshore          │ BOOLEAN     │            0.00 │             2 │ false            │ true                │
│ is_retrofitted       │ BOOLEAN     │            0.00 │             2 │ false            │ true                │
│ location_confidence  │ VARCHAR     │            0.00 │             3 │ High             │ Medium              │
│ manufacturer         │ VARCHAR     │            5.48 │            60 │ AAER             │ Zond                │
│ model                │ VARCHAR     │            5.65 │           318 │ 1.5-70.5         │ Zond                │
│ name                 │ VARCHAR     │            0.00 │          1806 │ 25 Mile Creek    │ unknown Yuma County │
│ num_turbines         │ INTEGER     │            0.00 │           192 │ 1                │ 713                 │
│ online_year          │ INTEGER     │            1.20 │            42 │ 1982             │ 2025                │
│ ors                  │ VARCHAR     │            5.91 │         72659 │ 02-000669        │ 56-064307           │
│ provider             │ VARCHAR     │            0.00 │             5 │ Bing Maps Aerial │ Planet              │
│ rated_capacity       │ DOUBLE      │            5.88 │           113 │ 13.4             │ 200.0               │
│ retrofitted_year     │ INTEGER     │           89.39 │             9 │ 2015             │ 2023                │
│ rsa                  │ DOUBLE      │            5.88 │           121 │ 141.03           │ 31415.93            │
│ state                │ VARCHAR     │            0.00 │            55 │ AK               │ WY                  │
│ ttlh                 │ DOUBLE      │            5.99 │           235 │ 31.0             │ 210.9               │
│ usgs_pr_id           │ INTEGER     │           51.44 │         36701 │ 1                │ 49135               │
├──────────────────────┴─────────────┴─────────────────┴───────────────┴──────────────────┴─────────────────────┤
│ 26 rows                                                                                             6 columns │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Turbines

Below I'll generate a heatmap of turbine installation locations.

$ ~/duckdb

CREATE OR REPLACE TABLE h3_4_stats AS
    SELECT   h3_4: H3_LATLNG_TO_CELL(
                        bbox.ymin,
                        bbox.xmin,
                        4),
             COUNT(*) num_turbines
    FROM     'uswtdb_V8_1_20250522.parquet'
    GROUP BY 1;

COPY (
    SELECT ST_ASWKB(H3_CELL_TO_BOUNDARY_WKT(h3_4)::geometry) geometry,
           num_turbines
    FROM   h3_4_stats
    WHERE  ST_XMIN(geometry::geometry) BETWEEN -179 AND 179
    AND    ST_XMAX(geometry::geometry) BETWEEN -179 AND 179
) TO 'turbines.h3_4.gpkg'
  WITH (FORMAT GDAL,
        DRIVER 'GPKG',
        LAYER_CREATION_OPTIONS 'WRITE_BBOX=YES');

The following was needed for ArcGIS Pro to recognise the projection of the above hexagons properly.

$ ogr2ogr \
    -f GPKG \
    -a_srs EPSG:4326 \
    turbines.h3_4.4326.gpkg \
    turbines.h3_4.gpkg

There are 63 unique manufactures listed in this dataset. These are the ones with the most and least number of turbines installed.

$ ~/duckdb

SELECT   num_turbines: COUNT(*),
         turbine.manufacturer
FROM     'uswtdb_V8_1_20250522.parquet'
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬──────────────────────────────────────────────────────────────────────────┐
│ num_turbines │                               manufacturer                               │
│    int64     │                                 varchar                                  │
├──────────────┼──────────────────────────────────────────────────────────────────────────┤
│        34613 │ GE Wind                                                                  │
│        17649 │ Vestas                                                                   │
│         4689 │ Siemens                                                                  │
│         4167 │ NULL                                                                     │
│         2999 │ Gamesa                                                                   │
│         2765 │ Siemens Gamesa Renewable Energy                                          │
│         2214 │ Mitsubishi                                                               │
│         1917 │ Nordex                                                                   │
│         1190 │ Suzlon                                                                   │
│          758 │ Acciona                                                                  │
│            · │    ·                                                                     │
│            · │    ·                                                                     │
│            · │    ·                                                                     │
│            1 │ Wind Energy Solutions                                                    │
│            1 │ Bergey Energy                                                            │
│            1 │ Leitner Poma                                                             │
│            1 │ Norwin                                                                   │
│            1 │ DWT                                                                      │
│            1 │ Unison                                                                   │
│            1 │ Silver Eagle                                                             │
│            1 │ Changzhou Railcar Propulsion Engineering Research and Development Center │
│            1 │ Wincon                                                                   │
│            1 │ Renewtech LLC                                                            │
├──────────────┴──────────────────────────────────────────────────────────────────────────┤
│ 64 rows (20 shown)                                                            2 columns │
└─────────────────────────────────────────────────────────────────────────────────────────┘

Below are the number of turbines by manufacturer where at least 40 were installed in any one decade.

.maxrows 40

WITH a AS (
    SELECT   num_turbines: COUNT(*),
             decade:       (project.online_year / 10)::INT * 10,
             manufacturer: turbine.manufacturer
    FROM     'uswtdb_V8_1_20250522.parquet'
    WHERE    turbine.manufacturer IS NOT NULL
    GROUP BY 2, 3
    HAVING   num_turbines > 40
    ORDER BY 1 DESC
)
PIVOT    a
ON       decade
USING    SUM(num_turbines)
GROUP BY manufacturer
ORDER BY manufacturer;

┌─────────────────────────────────┬────────┬────────┬────────┬────────┬────────┐
│          manufacturer           │  1980  │  1990  │  2000  │  2010  │  2020  │
│             varchar             │ int128 │ int128 │ int128 │ int128 │ int128 │
├─────────────────────────────────┼────────┼────────┼────────┼────────┼────────┤
│ Acciona                         │   NULL │   NULL │   NULL │    603 │    155 │
│ Bonus                           │    152 │     55 │     97 │   NULL │   NULL │
│ Clipper                         │   NULL │   NULL │   NULL │    198 │   NULL │
│ Enron                           │   NULL │   NULL │    395 │   NULL │   NULL │
│ GE Wind                         │   NULL │   NULL │   1397 │  15234 │  17950 │
│ Gamesa                          │   NULL │   NULL │    386 │   1842 │    771 │
│ Goldwind                        │   NULL │   NULL │   NULL │    106 │     88 │
│ Goldwind Americas               │   NULL │   NULL │   NULL │   NULL │     48 │
│ Micon                           │   NULL │    198 │   NULL │   NULL │   NULL │
│ Mitsubishi                      │   NULL │   NULL │    396 │   1818 │   NULL │
│ NEG Micon                       │   NULL │   NULL │    440 │   NULL │   NULL │
│ Nordex                          │   NULL │   NULL │     41 │    265 │   1611 │
│ Nordtank                        │    207 │   NULL │   NULL │   NULL │   NULL │
│ Northern Power Systems          │   NULL │   NULL │   NULL │     92 │   NULL │
│ REpower                         │   NULL │   NULL │   NULL │    559 │   NULL │
│ Siemens                         │   NULL │   NULL │   NULL │   3300 │   1389 │
│ Siemens Gamesa Renewable Energy │   NULL │   NULL │    314 │    450 │   2001 │
│ Suzlon                          │   NULL │   NULL │   NULL │   1172 │   NULL │
│ Vestas                          │     48 │   NULL │   1532 │   4787 │  11265 │
│ Zond                            │   NULL │   NULL │     83 │   NULL │   NULL │
├─────────────────────────────────┴────────┴────────┴────────┴────────┴────────┤
│ 20 rows                                                            6 columns │
└──────────────────────────────────────────────────────────────────────────────┘

Below I'll plot out the top manufacturer per hexagon in the lower 48 states among the overall top ten.

CREATE OR REPLACE TABLE top_manufacturers AS
    SELECT   num_turbines: COUNT(*),
             manufacturer: turbine.manufacturer
    FROM     'uswtdb_V8_1_20250522.parquet'
    WHERE    turbine.manufacturer IS NOT NULL
    GROUP BY 2
    ORDER BY 1 DESC
    LIMIT    10;

CREATE OR REPLACE TABLE h3_3s AS
    WITH b AS (
        WITH a AS (
            SELECT   H3_LATLNG_TO_CELL(bbox.ymin,
                                       bbox.xmin,
                                       3) h3_3,
                     manufacturer: turbine.manufacturer,
                     COUNT(*) num_recs
            FROM     'uswtdb_V8_1_20250522.parquet'
            WHERE    turbine.manufacturer IN (
                SELECT DISTINCT manufacturer
                FROM   top_manufacturers
            )
            GROUP BY 1, 2
        )
        SELECT *,
               ROW_NUMBER() OVER (PARTITION BY h3_3
                                  ORDER BY     num_recs DESC) AS rn
        FROM   a
    )
    FROM     b
    WHERE    rn = 1
    ORDER BY num_recs DESC;

COPY (
    SELECT geom: H3_CELL_TO_BOUNDARY_WKT(h3_3)::GEOMETRY,
           manufacturer
    FROM   h3_3s
    WHERE  ST_XMIN(geom::GEOMETRY) BETWEEN -179 AND 179
    AND    ST_XMAX(geom::GEOMETRY) BETWEEN -179 AND 179
) TO 'turbine.top_manufacturer.h3_3.parquet' (
    FORMAT 'PARQUET',
    CODEC  'ZSTD',
    COMPRESSION_LEVEL 22,
    ROW_GROUP_SIZE 15000);

Only 67 turbines in this dataset are listed as being offshore.

SELECT   COUNT(*),
         turbine.is_offshore
FROM     'uswtdb_V8_1_20250522.parquet'
GROUP BY 2;

┌──────────────┬─────────────┐
│ count_star() │ is_offshore │
│    int64     │   boolean   │
├──────────────┼─────────────┤
│        75984 │ false       │
│           67 │ true        │
└──────────────┴─────────────┘

Below are the top 20 states by their total GW of installed capacity, number of turbines and distinct manufacturers represented.

SELECT   location.state,
         gw:                ROUND(SUM(turbine.capacity) / 1000 ** 2, 2),
         num_turbines:      COUNT(*),
         num_manufacturers: COUNT(DISTINCT turbine.manufacturer)
FROM     'uswtdb_V8_1_20250522.parquet'
GROUP BY 1
ORDER BY 2 DESC
LIMIT    20;

┌─────────┬────────┬──────────────┬───────────────────┐
│  state  │   gw   │ num_turbines │ num_manufacturers │
│ varchar │ double │    int64     │       int64       │
├─────────┼────────┼──────────────┼───────────────────┤
│ TX      │  42.77 │        19415 │                22 │
│ IA      │  13.18 │         6472 │                22 │
│ OK      │  12.72 │         5597 │                13 │
│ KS      │    9.1 │         4415 │                11 │
│ IL      │   7.92 │         3837 │                16 │
│ CA      │   6.05 │         5510 │                18 │
│ CO      │   5.39 │         2908 │                12 │
│ MN      │   4.83 │         2736 │                19 │
│ ND      │   4.48 │         2169 │                 9 │
│ NM      │   4.33 │         2305 │                 7 │
│ OR      │    4.1 │         2173 │                 9 │
│ MI      │   3.73 │         1715 │                 9 │
│ WY      │   3.65 │         1713 │                 6 │
│ IN      │   3.62 │         1651 │                12 │
│ SD      │   3.62 │         1503 │                 9 │
│ NE      │   3.42 │         1504 │                 6 │
│ WA      │    3.4 │         1825 │                 9 │
│ NY      │   2.74 │         1429 │                13 │
│ MO      │   2.44 │         1108 │                 5 │
│ MT      │   1.89 │         1039 │                 7 │
├─────────┴────────┴──────────────┴───────────────────┤
│ 20 rows                                   4 columns │
└─────────────────────────────────────────────────────┘

Below is the ever-increasing capacity of turbines in MW over the decades.

WITH a AS (
    SELECT   decade:       (project.online_year / 10)::INT * 10,
             mw:           CEIL(turbine.capacity / 1000)::INT,
             num_turbines: COUNT(*)
    FROM     'uswtdb_V8_1_20250522.parquet'
    WHERE    turbine.capacity IS NOT NULL
    GROUP BY 1, 2
)
PIVOT    a
ON       decade
USING    SUM(num_turbines)
GROUP BY mw
ORDER BY mw;

┌───────┬────────┬────────┬────────┬────────┬────────┐
│  mw   │  1980  │  1990  │  2000  │  2010  │  2020  │
│ int32 │ int128 │ int128 │ int128 │ int128 │ int128 │
├───────┼────────┼────────┼────────┼────────┼────────┤
│     1 │    419 │    434 │   2750 │   1514 │     45 │
│     2 │   NULL │   NULL │   2359 │  20972 │  10223 │
│     3 │   NULL │   NULL │      1 │   7969 │  18733 │
│     4 │   NULL │   NULL │   NULL │    207 │   3588 │
│     5 │   NULL │   NULL │     38 │     19 │   2514 │
│     6 │   NULL │   NULL │   NULL │   NULL │    215 │
│    11 │   NULL │   NULL │   NULL │   NULL │     12 │
│    13 │   NULL │   NULL │   NULL │   NULL │      9 │
└───────┴────────┴────────┴────────┴────────┴────────┘

Of the 76K turbines in this dataset, 67K have a high confidence rating for both their location and attributes.

WITH a AS (
    SELECT   attribute:    turbine.attribute_confidence,
             location:     turbine.location_confidence,
             num_turbines: COUNT(*)
    FROM     'uswtdb_V8_1_20250522.parquet'
    GROUP BY 1, 2
    ORDER BY 3 DESC
)
PIVOT    a
ON       attribute,
USING    SUM(num_turbines)
GROUP BY location
ORDER BY location;

┌──────────┬────────┬────────┬────────┐
│ location │  High  │  Low   │ Medium │
│ varchar  │ int128 │ int128 │ int128 │
├──────────┼────────┼────────┼────────┤
│ High     │  67060 │   2924 │   4472 │
│ Low      │    137 │   1256 │    141 │
│ Medium   │      1 │     60 │   NULL │
└──────────┴────────┴────────┴────────┘

Project Footprints

Each turbine record contains denormalised project information. Below I'll generate the footprints for each project.

I'll group the projects by level-3 hexagons and name to generate their geographical footprints. This helps avoid country-wide overlaps that would result from grouping by name alone.

$ ~/duckdb

COPY (
    SELECT   h3_3: H3_LATLNG_TO_CELL(bbox.ymin,
                                     bbox.xmin,
                                     3),
             project_name: project.name,
             geometry: {
                'min_x': MIN(ST_X(geometry)),
                'min_y': MIN(ST_Y(geometry)),
                'max_x': MAX(ST_X(geometry)),
                'max_y': MAX(ST_Y(geometry))}::BOX_2D::GEOMETRY
    FROM     READ_PARQUET('uswtdb_V8_1_20250522.parquet')
    GROUP BY 1, 2
) TO 'projects.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

A total of 2,008 distinct projects were returned. Below are a few of these projects in Texas.

2.7 Billion Buildings

2025-11-03T09:00:00+02:00

On Monday last week, a paper was published announcing the OpenBuildingMap (OBM) dataset. With varying degrees of coverage, it contains building footprints, heights, 8 categories of usage and floorspace for 2.7B buildings across the globe. Below is a heatmap of its building footprints.

Existing data from OpenStreetMap (OSM), Google's Open Buildings, Microsoft Global ML Building Footprints and the Global Human Settlement Characteristics Layer were used to build this dataset. OBM's paper states the authors didn't use the CLSM dataset for East Asia due to uncertain licensing regarding the imagery that dataset was built from.

Efforts were made to clean up the data including removing buildings located over water bodies and excluding height information from any building claiming to be taller that the world's tallest tower, the Burj Khalifa in Dubai.

OBM is broken up into 1,270 BZip2-compressed GeoPackage (GPKG) files. I downloaded these and converted them into ZStandard-compressed, spatial-sorted Parquet files. These are now being hosted on AWS S3 thanks to the kind generosity of Source Cooperative and Taylor Geospatial Engine.

GPKG files are great if you're going to edit the data but for analysis, surgical downloading and archiving, it's really hard to beat Parquet. Parquet stores data as columns and has statistics for groups of rows in each column, often every 10-100K rows. This means some analytical queries only need a few MBs of bandwidth to analyse GBs or TBs of data stored remotely on AWS S3 or HTTPS-based CDNs like Cloudflare's.

GPKG files are based on SQLite and are row-oriented so compressing them rarely reduces the file size by more than a 5:1 ratio. With columnar formats like Parquet, 15:1 reductions are the norm. Decompressing the BZip2 files in this post maxed out around 10MB/s on my system. ZStandard runs circles around this.

In this post, I'll walk through converting this dataset into Parquet and examining some of its features.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use GDAL 3.9.3 and a few other tools to help analyse the data in this post.

$ sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable
$ sudo apt update
$ sudo apt install \
    gdal-bin \
    jq

I'll use DuckDB, along with its H3, JSON, Lindel, Parquet and Spatial extensions in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.4.1/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

The maps in this post were rendered with QGIS version 3.44. QGIS is a desktop application that runs on Windows, macOS and Linux. The application has grown in popularity in recent years and has ~15M application launches from users all around the world each month.

The dark heatmaps in this post are mostly made up of vector data from Natural Earth and Overture.

Downloading the Buildings

In QGIS, click the "Plugins" Menu and then the "Manage and Install Plugins" item. Click the "All" filter in the top left of the dialog and then search for "QuickMapServices". Click to install or upgrade the plugin in the bottom right of the dialog.

Click the "Web" Menu, then "QuickMapServices" and then the "Settings" item. Click the "More Services" tab at the top of the dialog. Click the "Get contributed pack" button.

In the "Web" menu under "QuickMapServices" you should now see the list of several basemap providers.

Select "OSM" and then "OSM Standard" to add a world map to your scene.

Under the "Plugins" menu, select "Python Console". Paste in the following line of Python. It'll ensure you've got the latest version of DuckDB installed in QGIS' Python Environment.

import pip; pip.main(['install', '--upgrade', 'duckdb'])

In QGIS, click the "Plugins" Menu and then the "Manage and Install Plugins" item. Click the "All" filter in the top left of the dialog and then search for "GeoParquet Downloader". Click to install or upgrade the plugin in the bottom right of the dialog.

Zoom into a city of interest somewhere in the world. It's important to make sure you're not looking at an area larger than a major city as the viewport will set the boundaries for how much data will be downloaded.

If you can see the whole earth, GeoParquet Downloader will end up downloading 194 GB of data. Downloading only a city's worth of data will likely only need a few MB of data.

In your toolbar, click on the GeoParquet Downloader icon. It's the one with blue, rotated rectangles.

Click the "Custom URL" option and paste the following URL in.

s3://us-west-2.opendata.source.coop/tge-labs/openbuildingmap/*.parquet

Hit "OK". You'll be prompted to save a Parquet file onto your computer. Once you've done so, the building data for your map's footprint should appear shortly afterword.

Once the buildings have loaded, select the layer styling for the downloaded buildings layer. The combo box at the top of the styling panel should be switched from "Single Symbol" to "2.5D".

I've set the roof colour to #b13327, wall colour to #4f090b, angle to 230 degrees and added a drop shadow effect to the layer.

In the "Height" field, change it to the following expression:

if(string_to_array(height, ':')[0] = 'H', string_to_array(height, ':')[1], 1) / 50000

You should now see a 2.5D representation of the buildings you downloaded.

There is an "Identify Features" icon in the toolbar.

This tool lets you select any building and bring up its metadata.

The Parquet file you saved can also be read via DuckDB and even dropped into Esri's ArcGIS Pro 3.5 or newer if you have it installed on your machine.

Downloading OpenBuildingMap

Below I'll show how I turned the GPKG files into Parquet. These steps don't need to be repeated as the resulting files are now on S3.

OBM has a website where you can see the footprints of their GPKG files and download them individually. I found a GeoJSON-formatted manifest file used by the site. It contains the URLs for all of their BZ2-compressed GPKG files.

$ mkdir -p ~/obm
$ cd ~/obm

$ curl -S 'https://umap.openstreetmap.de/de/datalayer/81684/ac9788a9-c215-417d-a580-cb545acd78b9/' \
    > manifest.json

Below is an example record from this manifest.

$ echo "SELECT a.properties::JSON
        FROM (SELECT UNNEST(features) a
              FROM   READ_JSON('manifest.json'))
        LIMIT 1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "CAST(a.properties AS \"JSON\")": {
      "URL": "https://datapub.gfz.de/download/10.5880.GFZ.LKUT.2025.002-Caweb/2025-002_Oostwegel-et-al_data/building.002202.gpkg.bz2",
      "completeness_index": 92.93,
      "fid": 1,
      "filename": "building.002202.gpkg",
      "license": "ODbL",
      "number_of_buildings": 127,
      "percentage_known_floorspace": 0.0,
      "percentage_known_height": 3.94,
      "percentage_known_occupancy": 47.24,
      "percentage_source_google": 0.0,
      "percentage_source_microsoft": 0.0,
      "percentage_source_openstreetmap": 100.0,
      "quadkey": "002202"
    }
  }
]

I'll build a list of URLs and use wget to download them with a concurrency of four.

$ echo "SELECT a.properties.URL
        FROM (SELECT UNNEST(features) a
              FROM   READ_JSON('manifest.json'))" \
    | ~/duckdb -csv \
               -noheader \
    > manifest.txt

$ cat manifest.txt \
    | xargs -P4 \
            -I% \
            wget -c "%"

GPKG to Parquet

The following converted 278 GB worth of BZip2 files containing ~722 GB worth of GPKG data into 194 GB of Parquet containing 2,693,211,269 records. Even with 24 concurrent processes running, it took the better part of a day to complete.

$ ls building.*.gpkg.bz2 \
    | xargs -P24 \
            -I% \
            bash -c "
    BASENAME=\`echo \"%\" | cut -f2 -d.\`
    PQ_FILE=\"building.\$BASENAME.parquet\"

    if [ ! -f \$PQ_FILE ]; then
        echo \"Building \$PQ_FILE.\"

        bzip2 -dk \"%\" --stdout > \"working.\$BASENAME.gpkg\"

        echo \"COPY (
                   SELECT   * EXCLUDE (geom,
                                       source_id),
                            geometry: geom,
                            bbox:     {'xmin': ST_XMIN(ST_EXTENT(geom)),
                                       'ymin': ST_YMIN(ST_EXTENT(geom)),
                                       'xmax': ST_XMAX(ST_EXTENT(geom)),
                                       'ymax': ST_YMAX(ST_EXTENT(geom))},
                            source:   CASE WHEN source_id = 0 THEN 'OSM'
                                           WHEN source_id = 1 THEN 'Google'
                                           WHEN source_id = 2 THEN 'Microsoft'
                                      END
                   FROM     ST_READ('working.\$BASENAME.gpkg')
                   ORDER BY HILBERT_ENCODE([
                                ST_Y(ST_CENTROID(geom)),
                                ST_X(ST_CENTROID(geom))]::double[2])
                ) TO '\$PQ_FILE' (
                    FORMAT            'PARQUET',
                    CODEC             'ZSTD',
                    COMPRESSION_LEVEL 22,
                    ROW_GROUP_SIZE    15000);\" \
                | ~/duckdb

        rm \"working.\$BASENAME.gpkg\"
    else
        echo \"\$PQ_FILE already exists, skipping..\"
    fi"

Below is what my system's CPU usage looked like while the above was running.

Data Fluency

The following is an example record from this dataset.

$ echo "SELECT * EXCLUDE(bbox),
               bbox::JSON AS bbox
        FROM   'building.*.parquet'
        WHERE  floorspace IS NOT NULL
        LIMIT  1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "bbox": {
      "xmax": -160.03150641447675,
      "xmin": -160.03228425509127,
      "ymax": 70.63789626901166,
      "ymin": 70.63765269090544
    },
    "floorspace": "719.5123100951314",
    "geometry": "POLYGON ((-160.03228425509127 70.63772552964403, -160.03180145746848 70.63789626901166, -160.03150641447675 70.63781446163671, -160.0315654230751 70.6377735579492, -160.0318229151406 70.63767574113916, -160.0319946603366 70.63765269090544, -160.03228425509127 70.63772552964403))",
    "height": "H:2",
    "id": 500236864,
    "last_update": "2024-09-25 17:23:12.186+03",
    "occupancy": "RES3",
    "quadkey": "002213322233003100",
    "relation_id": null,
    "source": "OSM"
  }
]

Below are the field names, data types, percentages of NULLs per column, number of unique values and minimum and maximum values for each column.

$ ~/duckdb

SELECT   column_name,
         column_type,
         null_percentage,
         approx_unique,
         min,
         max
FROM     (SUMMARIZE
          FROM 'building.*.parquet')
WHERE    column_name != 'geometry'
AND      column_name != 'bbox'
ORDER BY 1;

┌─────────────┬──────────────────────────┬─────────────────┬───────────────┬────────────────────────────┬───────────────────────────┐
│ column_name │       column_type        │ null_percentage │ approx_unique │            min             │            max            │
│   varchar   │         varchar          │  decimal(9,2)   │     int64     │          varchar           │          varchar          │
├─────────────┼──────────────────────────┼─────────────────┼───────────────┼────────────────────────────┼───────────────────────────┤
│ floorspace  │ VARCHAR                  │           98.81 │      31981048 │ -4.910890285223722         │ 9999.999575164324         │
│ height      │ VARCHAR                  │           24.78 │         31689 │ H:1                        │ HHT:99.95                 │
│ id          │ BIGINT                   │            0.00 │    1985306599 │ -17787084                  │ 2304182979                │
│ last_update │ TIMESTAMP WITH TIME ZONE │            0.00 │      18555352 │ 2024-09-25 17:23:12.185+03 │ 2024-10-09 15:08:17.53+03 │
│ occupancy   │ VARCHAR                  │            0.00 │            42 │ AGR                        │ UNK                       │
│ quadkey     │ VARCHAR                  │            0.00 │     354530921 │ 002202022321313300         │ 331301200330323102        │
│ relation_id │ VARCHAR                  │           99.99 │         57371 │ -10000436.0                │ 998911012.0               │
│ source      │ VARCHAR                  │            0.00 │             3 │ Google                     │ OSM                       │
└─────────────┴──────────────────────────┴─────────────────┴───────────────┴────────────────────────────┴───────────────────────────┘

I'll generate a heatmap of the building footprints in this dataset.

$ ~/duckdb obm.duckdb

CREATE OR REPLACE TABLE h3_4_stats AS
    SELECT   H3_LATLNG_TO_CELL(
                bbox.ymin,
                bbox.xmin, 4) AS h3_4,
             COUNT(*) num_buildings
    FROM     'building.*.parquet'
    GROUP BY 1;

COPY (
    SELECT ST_ASWKB(H3_CELL_TO_BOUNDARY_WKT(h3_4)::geometry) geometry,
           num_buildings
    FROM   h3_4_stats
    WHERE  ST_XMIN(geometry::geometry) BETWEEN -179 AND 179
    AND    ST_XMAX(geometry::geometry) BETWEEN -179 AND 179
) TO 'h3_4_stats.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

There are very few Chinese buildings in this dataset.

Below is the TUM dataset which used satellite imagery of China to help detect as many building footprints as possible. The heatmap below shows China's South East has a building density on par with most of its neighbouring countries.

Building Uses

There are eight different building use classifications in this dataset. The uses and their acronyms are residential (RES), commercial and public (COM), mixed use (MIX), industrial (IND), agricultural (AGR), assembly (ASS), government (GOV) and education (EDU). UNK stands for unknown and accounts for ~60% of the buildings in this dataset.

After the 3-letter building use acronym there can be suffixes that provide more detail, such as RES2, which means the residential building is an apartment building. The "Building occupancy type" section of the paper goes into more detail.

Below are the building counts by use and suffix.

WITH a AS (
    SELECT   num_recs: COUNT(*),
             use:      occupancy[:3],
             use_ext:  occupancy[4:]
    FROM     'building.*.parquet'
    GROUP BY 2, 3
    ORDER BY 2, 3
)
PIVOT    a
ON       use
USING    SUM(num_recs)
GROUP BY use_ext
ORDER BY TRY_CAST(REGEXP_REPLACE(use_ext, '[A-Z]', '') AS INT);

┌─────────┬──────────┬─────────┬─────────┬─────────┬─────────┬──────────┬─────────┬───────────┬────────────┐
│ use_ext │   AGR    │   ASS   │   COM   │   EDU   │   GOV   │   IND    │   MIX   │    RES    │    UNK     │
│ varchar │  int128  │ int128  │ int128  │ int128  │ int128  │  int128  │ int128  │  int128   │   int128   │
├─────────┼──────────┼─────────┼─────────┼─────────┼─────────┼──────────┼─────────┼───────────┼────────────┤
│ 1       │   785006 │ 1982076 │ 4053587 │   93893 │  516140 │   934295 │ 4072344 │  64826183 │       NULL │
│ 2A      │     NULL │    NULL │    NULL │    NULL │    NULL │     NULL │    NULL │    120885 │       NULL │
│ 2       │   189285 │  182617 │  161023 │ 5513273 │  190597 │    15282 │    9524 │   8874805 │       NULL │
│ 3       │  1199083 │   22159 │  422796 │ 1813089 │    NULL │     NULL │    NULL │  16615755 │       NULL │
│ 4       │     NULL │  106813 │ 1774918 │    8714 │    NULL │     NULL │ 1833277 │     32207 │       NULL │
│ 5       │     NULL │    NULL │  643386 │    NULL │    NULL │     NULL │  724050 │      NULL │       NULL │
│ 6       │     NULL │    NULL │  200419 │    NULL │    NULL │     NULL │    NULL │      NULL │       NULL │
│ 7       │     NULL │    NULL │ 1641089 │    NULL │    NULL │     NULL │    NULL │      NULL │       NULL │
│ 8       │     NULL │    NULL │   18887 │    NULL │    NULL │     NULL │    NULL │      NULL │       NULL │
│ 9       │     NULL │    NULL │   99485 │    NULL │    NULL │     NULL │    NULL │      NULL │       NULL │
│ 10      │     NULL │    NULL │  122578 │    NULL │    NULL │     NULL │    NULL │      NULL │       NULL │
│ 11      │     NULL │    NULL │ 1662269 │    NULL │    NULL │     NULL │    NULL │      NULL │       NULL │
│         │ 38664645 │     222 │ 4062284 │   53091 │ 2786516 │ 22215578 │    NULL │ 861941989 │ 1642025155 │
├─────────┴──────────┴─────────┴─────────┴─────────┴─────────┴──────────┴─────────┴───────────┴────────────┤
│ 13 rows                                                                                       10 columns │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Below are the building use categories for each building in Tallinn's Old Town.

Building Sources

Google is the source for almost 60% of the building footprints. OSM accounts for 23% and Microsoft for 18%.

SELECT   COUNT(*),
         source
FROM     'building.*.parquet'
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬───────────┐
│ count_star() │  source   │
│    int64     │  varchar  │
├──────────────┼───────────┤
│   1597737974 │ Google    │
│    612804756 │ OSM       │
│    482668539 │ Microsoft │
└──────────────┴───────────┘

OSM had 657M buildings in its dataset a few weeks ago so its interesting to see some ~45M of its buildings haven't made it into OBM.

Below I've highlighted the most common source for each H3 hexagon at zoom level 5.

CREATE OR REPLACE TABLE h3_5s AS
    WITH b AS (
        WITH a AS (
            SELECT   H3_LATLNG_TO_CELL(bbox.ymin,
                                       bbox.xmin,
                                       5) h3_5,
                     source,
                     COUNT(*) num_recs
            FROM     'building.*.parquet'
            GROUP BY 1, 2
        )
        SELECT *,
               ROW_NUMBER() OVER (PARTITION BY h3_5
                                  ORDER BY     num_recs DESC) AS rn
        FROM   a
    )
    FROM     b
    WHERE    rn = 1
    ORDER BY num_recs DESC;

COPY (
    SELECT geom: H3_CELL_TO_BOUNDARY_WKT(h3_5)::GEOMETRY,
           source
    FROM   h3_5s
    WHERE  ST_XMIN(geom::GEOMETRY) BETWEEN -179 AND 179
    AND    ST_XMAX(geom::GEOMETRY) BETWEEN -179 AND 179
) TO 'obm.top_sources.parquet' (
    FORMAT 'PARQUET',
    CODEC  'ZSTD',
    COMPRESSION_LEVEL 22,
    ROW_GROUP_SIZE 15000);

Microsoft is red, Google is green and OSM is yellow.

For spatial context, you'd only need a dozen or so of these zoom level 5 hexagons to cover Dubai's urban areas.

Building Heights

Both building height and/or the number of stories are contained within the height field. In some cases, ranges instead of absolute values are used. The field has the ability to express how many stories below ground a building extends as well. The "Building height" section in the paper goes into detail.

Below are the building counts by the initial height prefix and if the multiple measurements delimiter is present.

SELECT   prefix:        SPLIT(height, ':')[1],
         multi:         height LIKE '%+%',
         num_buildings: COUNT(*)
FROM     'building.*.parquet'
GROUP BY 1, 2
ORDER BY 3 DESC;

┌─────────┬─────────┬───────────────┐
│ prefix  │  multi  │ num_buildings │
│ varchar │ boolean │     int64     │
├─────────┼─────────┼───────────────┤
│ HBET    │ false   │    1879363563 │
│ NULL    │ NULL    │     667500586 │
│ HHT     │ false   │     114324028 │
│ H       │ false   │      27654116 │
│ H       │ true    │       4360694 │
│ HBEX    │ false   │          8036 │
│ HBEX    │ true    │           246 │
└─────────┴─────────┴───────────────┘

~25% of the buildings in this dataset don't have any height data.

Below I've colour-coded the buildings in Tallinn's Old Town by their height prefix. The 'H' records in red contain the number of stories, the 'HBET' records in orange contain a range of stories, the 'HHT' records in light green contain the height in meters and the blue buildings have no height information.

Below are an example of combination values, delimited by a plus sign.

.maxrows 10

SELECT   height,
         COUNT(*)
FROM     'building.*.parquet'
WHERE    height LIKE '%+%'
GROUP BY 1
ORDER BY 2 DESC;

┌───────────────────────┬──────────────┐
│        height         │ count_star() │
│        varchar        │    int64     │
├───────────────────────┼──────────────┤
│ H:1+HHT:3.50          │       903295 │
│ H:2+HHT:7.00          │       395789 │
│ H:1+HHT:3.00          │       261288 │
│ H:2+HHT:6.00          │       136564 │
│ H:1+HHT:4.00          │       115217 │
│ H:2+HBEX:1            │        67691 │
│ H:1+HBEX:1            │        59680 │
│ H:1+HHT:5.00          │        51128 │
│ H:1+HHT:2.80          │        50296 │
│ H:2+HHT:8.00          │        48202 │
│ H:1+HHT:3.30          │        48074 │
│ H:1+HHT:3.40          │        47996 │
│ H:1+HHT:3.60          │        47382 │
│ H:3+HBEX:1            │        46986 │
│ H:1+HHT:3.20          │        43824 │
│ H:1+HHT:3.70          │        43422 │
│ H:1+HHT:3.80          │        40676 │
│ H:1+HHT:3.10          │        39174 │
│ H:3+HHT:9.00          │        38437 │
│ H:1+HHT:6.00          │        37171 │
│      ·                │            · │
│      ·                │            · │
│      ·                │            · │
│ H:2+HBEX:1+HHT:7.40   │            1 │
│ H:33+HHT:10.00        │            1 │
│ H:17+HBEX:1+HHT:49.60 │            1 │
│ H:15+HHT:4.80         │            1 │
│ H:15+HHT:71.50        │            1 │
│ H:1+HHT:15.51         │            1 │
│ H:42+HHT:136.55       │            1 │
│ H:60+HHT:239.12       │            1 │
│ H:53+HHT:229.00       │            1 │
│ H:39+HHT:98.00        │            1 │
│ H:8+HHT:2.90          │            1 │
│ H:35+HHT:82.00        │            1 │
│ H:7+HHT:12.20         │            1 │
│ H:4+HBEX:1+HHT:13.50  │            1 │
│ H:25+HHT:96.36        │            1 │
│ H:3+HBEX:1+HHT:10.55  │            1 │
│ H:32+HHT:142.00       │            1 │
│ H:43+HHT:147.70       │            1 │
│ H:11+HHT:35.09        │            1 │
│ H:65+HHT:258.00       │            1 │
├───────────────────────┴──────────────┤
│ 22423 rows (40 shown)      2 columns │
└──────────────────────────────────────┘

It would be nice to have separate columns for height in meters and stories both going above and below ground. I'll have to analyse this column further to figure out how complex that task would be.

Canadian Coverage

I'll download a rough outline of Canada's provinces and territories in GeoJSON format. I'll then convert it into GPKG as GPKG files require less syntax to work with in DuckDB.

$ wget -c https://gist.github.com/Thiago4breu/6ba01976161aa0be65e0a289412dc54c/raw/8ec57d8317a2abe5bae18e5fd86f777fab649f84/canada-provinces.geojson
$ ogr2ogr \
    -f GPKG \
    canada-provinces.gpkg \
    canada-provinces.geojson

$ ~/duckdb

CREATE OR REPLACE TABLE canada AS
    FROM ST_READ('canada-provinces.gpkg');

SELECT    COUNT(*)
FROM      'building.*.parquet' b
LEFT JOIN canada c ON ST_COVEREDBY(b.geometry, c.geom)
WHERE     c.name IS NOT NULL;

The above returned a count of 13,239,095 Canadian buildings in OBM.

For comparison, the ODB dataset I reviewed last month contains 14.4M buildings in Canada, PSC has 13.7M, TUM has 13.4M and the Layercake Project, which produces weekly exports of OSM data into Parquet and hosts via Cloudflare's CDN, has 7.8M Canadian buildings.

Below on top of Google's Satellite Map in yellow is OBM and in purple is Overture's October building footprints for South East Calgary. There are several areas where OBM has gaps in its coverage.

With that said, cities like Calgary are growing fast and new neighbourhoods can be waiting a while before they appear in any public datasets.

I spotted a few cases where a OBM building footprint will cover two or three properties.

This Mall, just north of Calgary's Airport, has ODM building footprints on top of one another.

For reference, Overture's October release contains 13,453,441 buildings for Canada.

SELECT    COUNT(*)
FROM      's3://overturemaps-us-west-2/release/2025-10-22.0/theme=buildings/type=building/part-000*.parquet' b
LEFT JOIN canada c ON ST_COVEREDBY(ST_POINT(bbox.xmin,
                                            bbox.ymin),
                                   c.geom)
WHERE     c.name IS NOT NULL;

Note: Overture has 236 Parquet files for its building footprints dataset. If I scan all of those files, the query above will exhaust my system's RAM capacity. So to get around this, the URL glob limits the query to just the first 100 Parquet files. The Parquet files are spatially sorted so Canada's data should usually live in the lower file numbers.

Australian Coastline Imagery

2025-10-19T20:00:00+03:00

Digital Earth Australia's Tidal Composites dataset is a cloud-free mosaic of Australia’s coasts, estuaries and reefs at low and high tide. These were built off of Sentinel-2 satellite imagery, covering 2016 till 2023 and total ~2.6 TB in GeoTIFF format. These are being hosted on AWS S3.

The Sentinel-2 constellation is made up of three satellites that capture 10-meter imagery across 13-spectral bands. The European Space Agency launched in the first satellite in 2015, the second in 2017 and the third in September of last year.

Below is a high-tide, blue-band image taken of Hervey Bay, Queensland in 2023. It's coloured with a rainbow colour ramp.

In this post, I'll explore the Tidal Composites dataset.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use Python 3.12.3, GDAL 3.9.3 and a few other tools to help analyse the data in this post.

$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable
$ sudo apt update
$ sudo apt install \
    gdal-bin \
    jq \
    python3-pip \
    python3.12-venv

I'll set up a Python Virtual Environment and install the latest version of the AWS CLI toolkit.

$ python3 -m venv ~/.aus
$ source ~/.aus/bin/activate
$ python -m pip install awscli

I'll use DuckDB v1.4.1, along with its H3, JSON, Lindel, Parquet and Spatial extensions, in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.4.1/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

I used QGIS' Tile+ plugin to add basemaps from Bing to the maps in this post.

Analysis-Ready Metadata

I'll download the list of files and their respective metadata from S3. The following produced a 325,557-line, 140 MB JSONL file.

$ mkdir -p ~/dea
$ cd ~/dea

$ aws --no-sign-request \
      --output json \
      s3api \
      list-objects \
      --bucket dea-public-data \
      --max-items=1000000000 \
      --prefix=derivative/ga_s2_tidal_composites_cyear_3 \
    | jq -c '.Contents[]' \
    > dea.s3.json

I'll build a DuckDB table from the JSONL file to help analyse the file listing.

$ ~/duckdb dea.duckdb

CREATE OR REPLACE TABLE s3 AS
    SELECT *,
           LOWER(SPLIT(SPLIT(SPLIT(Key, '/')[7], '_')[-1], '.')[-1]) AS format,
           REPLACE(SPLIT(Key, '/')[4], 'x', '')::INT AS x,
           REPLACE(SPLIT(Key, '/')[5], 'y', '')::INT AS y,
           SPLIT(SPLIT(SPLIT(SPLIT(Key, '/')[7], '_')[-1], '.')[1], '-')[1] AS tide,
           REPLACE(REPLACE(SPLIT(SPLIT(SPLIT(Key, '/')[7], '_')[-1], '.')[1], 'high-', ''), 'low-', '') AS band,
           SPLIT(SPLIT(Key, '/')[6], '-')[1]::INT AS year
    FROM   READ_JSON('dea.s3.json')
    WHERE  SPLIT(Key, '/')[6] LIKE '20%--P1Y'
    AND    SPLIT(SPLIT(SPLIT(SPLIT(Key, '/')[7], '_')[-1], '.')[1], '-')[1] != 'qa';

Below is an example record from the table created above.

$ echo "SELECT * EXCLUDE(Owner),
               Owner::JSON AS Owner
        FROM   s3
        WHERE  format = 'tif'
        LIMIT  1" \
    | ~/duckdb -json dea.duckdb \
    | jq -S .

[
  {
    "ChecksumAlgorithm": "[CRC32]",
    "ChecksumType": "COMPOSITE",
    "ETag": "\"4953d4e03bd4795567419b9a02b3fc53-2\"",
    "Key": "derivative/ga_s2_tidal_composites_cyear_3/1-0-0/x078/y124/2016--P1Y/ga_s2_tidal_composites_cyear_3_x078y124_2016--P1Y_final_high-blue.tif",
    "LastModified": "2025-05-06T08:42:43.000Z",
    "Owner": {
      "DisplayName": null,
      "ID": "aa1c69e19883ec24c399e75c3b2ee0812828dccf66574abf4ad7a91767cdb3fa"
    },
    "Size": 11535601,
    "StorageClass": "INTELLIGENT_TIERING",
    "band": "blue",
    "format": "tif",
    "tide": "high",
    "x": 78,
    "y": 124,
    "year": 2016
  }
]

Cloud-Optimised GeoTIFFs

There's ~2.6 TB of content in this dataset with the vast majority being made up of GeoTIFF images. Below are the number of MBs for each file format broken down by year.

$ ~/duckdb dea.duckdb

PIVOT    s3
ON       format
USING    ROUND(SUM(Size) / 1024 ** 2)::INT
GROUP BY year
ORDER BY year;

┌───────┬───────┬───────┬───────┬───────┬────────┬───────┐
│ year  │  jpg  │ json  │  png  │ sha1  │  tif   │ yaml  │
│ int32 │ int32 │ int32 │ int32 │ int32 │ int32  │ int32 │
├───────┼───────┼───────┼───────┼───────┼────────┼───────┤
│  2016 │    12 │    21 │    29 │     3 │ 254858 │    11 │
│  2017 │    17 │    36 │    45 │     4 │ 358245 │    23 │
│  2018 │    17 │    42 │    48 │     4 │ 354476 │    29 │
│  2019 │    17 │    46 │    50 │     4 │ 353264 │    33 │
│  2020 │    16 │    46 │    50 │     4 │ 354057 │    33 │
│  2021 │    16 │    44 │    49 │     4 │ 354838 │    31 │
│  2022 │    16 │    43 │    49 │     4 │ 353765 │    30 │
│  2023 │    16 │    43 │    49 │     4 │ 351874 │    30 │
└───────┴───────┴───────┴───────┴───────┴────────┴───────┘

I'll download a tile's worth of GeoTIFFs and other metadata.

$ aws s3 sync \
    s3://dea-public-data/derivative/ga_s2_tidal_composites_cyear_3/1-0-0/x202/y125/2023--P1Y/ \
    ./ \
    --no-sign-request

The GeoTIFFs are Cloud-Optimised GeoTIFF containers. They contain Tiled Multi-Resolution TIFFs / Tiled Pyramid TIFFs. This means there are several versions of the same image at different resolutions within the TIFF file. The largest image within any one GeoTIFF is 3200 x 3200-pixels.

$ gdalinfo -json \
    ga_s2_tidal_composites_cyear_3_x202y125_2023--P1Y_final_high-blue.tif \
    | jq -S '.size'

These files are structured so it's easy to only read a portion of a file for any one resolution you're interested in. A file might be 10s or 100s of MB but a JavaScript-based Web Application might only need to download ~2 MB of data from that file in order to render its lowest resolution.

Below you can see there are 1600x1600, 800x800, etc.. versions of the image within a single GeoTIFF file.

$ gdalinfo -json \
    ga_s2_tidal_composites_cyear_3_x202y125_2023--P1Y_final_high-blue.tif \
    | jq -S '.bands[0].overviews'

[
  {
    "size": [
      1600,
      1600
    ]
  },
  {
    "size": [
      800,
      800
    ]
  },
  {
    "size": [
      400,
      400
    ]
  },
  {
    "size": [
      200,
      200
    ]
  },
  {
    "size": [
      100,
      100
    ]
  }
]

Below is a single GeoTIFF rendered on the grid used for this dataset. It's using a rainbow colour ramp.

Multispectral Imagery

A single GeoTIFF can be 10 - 15 MB and will represent one of the many bands supported by Sentinel-2. Below are the number of GeoTIFFs broken down by band and year.

WITH a AS (
    SELECT   band,
             year,
             COUNT(DISTINCT Key) AS num_keys
    FROM     s3
    WHERE    format = 'tif'
    GROUP BY 1, 2
)
PIVOT    a
ON       year
USING    SUM(num_keys)
GROUP BY band
ORDER BY band;

┌─────────────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┐
│      band       │  2016  │  2017  │  2018  │  2019  │  2020  │  2021  │  2022  │  2023  │
│     varchar     │ int128 │ int128 │ int128 │ int128 │ int128 │ int128 │ int128 │ int128 │
├─────────────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
│ blue            │   1940 │   2728 │   2722 │   2722 │   2722 │   2722 │   2722 │   2712 │
│ coastal-aerosol │   1940 │   2728 │   2722 │   2722 │   2722 │   2722 │   2722 │   2712 │
│ green           │   1940 │   2728 │   2722 │   2722 │   2722 │   2722 │   2722 │   2712 │
│ nir-1           │   1940 │   2728 │   2722 │   2722 │   2722 │   2722 │   2722 │   2712 │
│ nir-2           │   1940 │   2728 │   2722 │   2722 │   2722 │   2722 │   2722 │   2712 │
│ red             │   1940 │   2728 │   2722 │   2722 │   2722 │   2722 │   2722 │   2712 │
│ red-edge-1      │   1940 │   2728 │   2722 │   2722 │   2722 │   2722 │   2722 │   2712 │
│ red-edge-2      │   1940 │   2728 │   2722 │   2722 │   2722 │   2722 │   2722 │   2712 │
│ red-edge-3      │   1940 │   2728 │   2722 │   2722 │   2722 │   2722 │   2722 │   2712 │
│ swir-2          │   1940 │   2728 │   2722 │   2722 │   2722 │   2722 │   2722 │   2712 │
│ swir-3          │   1940 │   2728 │   2722 │   2722 │   2722 │   2722 │   2722 │   2712 │
├─────────────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┤
│ 11 rows                                                                       9 columns │
└─────────────────────────────────────────────────────────────────────────────────────────┘

The GeoTIFFs are compressed using DEFLATE.

$ gdalinfo \
    -json \
    ga_s2_tidal_composites_cyear_3_x202y125_2023--P1Y_final_high-blue.tif \
    | jq -S '.metadata'

{
  "": {
    "AREA_OR_POINT": "Area"
  },
  "IMAGE_STRUCTURE": {
    "COMPRESSION": "DEFLATE",
    "INTERLEAVE": "BAND",
    "LAYOUT": "COG",
    "PREDICTOR": "2"
  }
}

This means imagery over open ocean will often compress better than imagery captured over land.

Below I'll download the grid system used by this dataset.

$ wget -c https://data.dea.ga.gov.au/derivative/ga_summary_grid_sentinel_c3.geojson

I'll then build a heatmap showing how many MBs of GeoTIFFs have been collected for each tile.

$ ~/duckdb dea.duckdb

CREATE OR REPLACE TABLE grid AS
    SELECT geometry,
           ix AS x,
           iy AS y
    FROM (
        SELECT ST_GEOMFROMGEOJSON(a.geometry) AS geometry,
               UNNEST(a.properties)
        FROM (SELECT UNNEST(features) a
              FROM   READ_JSON('ga_summary_grid_sentinel_c3.geojson')));

COPY (
    SELECT   ROUND(SUM(a.Size) / 1024 ** 2)::INT AS MB,
             ST_ASWKB(b.geometry) geometry
    FROM     s3 a
    JOIN     grid b ON a.x = b.x AND a.y = b.y
    WHERE    format = 'tif'
    GROUP BY 2
) TO 'grid_stats.gpkg'
  WITH (FORMAT GDAL,
        DRIVER 'GPKG',
        LAYER_CREATION_OPTIONS 'WRITE_BBOX=YES');

Below you can see the lighter tiles have ~2.5 - 3.4 GB of GeoTIFFs while the darker tiles have less than 1.8 GB.

Along Queensland's coastline you can see the divide between imagery taken over land and the ocean.

With the exception of 2016, any given band has ~26 - 35 GB worth of GeoTIFFs for any given year.

WITH a AS (
    FROM  s3
    WHERE format = 'tif'
)
PIVOT    a
ON       year
USING    CEIL(SUM(Size) / 1024 ** 3)::INT
GROUP BY band
ORDER BY band;

┌─────────────────┬───────┬───────┬───────┬───────┬───────┬───────┬───────┬───────┐
│      band       │ 2016  │ 2017  │ 2018  │ 2019  │ 2020  │ 2021  │ 2022  │ 2023  │
│     varchar     │ int32 │ int32 │ int32 │ int32 │ int32 │ int32 │ int32 │ int32 │
├─────────────────┼───────┼───────┼───────┼───────┼───────┼───────┼───────┼───────┤
│ blue            │    24 │    34 │    34 │    33 │    34 │    34 │    33 │    33 │
│ coastal-aerosol │    19 │    27 │    26 │    26 │    26 │    26 │    26 │    26 │
│ green           │    25 │    34 │    34 │    34 │    34 │    34 │    34 │    34 │
│ nir-1           │    25 │    36 │    35 │    35 │    35 │    35 │    35 │    35 │
│ nir-2           │    23 │    33 │    32 │    32 │    32 │    32 │    32 │    32 │
│ red             │    25 │    35 │    34 │    34 │    34 │    34 │    34 │    34 │
│ red-edge-1      │    22 │    31 │    31 │    31 │    31 │    31 │    31 │    31 │
│ red-edge-2      │    23 │    32 │    32 │    32 │    32 │    32 │    32 │    32 │
│ red-edge-3      │    23 │    32 │    32 │    32 │    32 │    32 │    32 │    32 │
│ swir-2          │    22 │    31 │    31 │    31 │    31 │    31 │    31 │    31 │
│ swir-3          │    22 │    30 │    30 │    30 │    30 │    30 │    30 │    30 │
├─────────────────┴───────┴───────┴───────┴───────┴───────┴───────┴───────┴───────┤
│ 11 rows                                                               9 columns │
└─────────────────────────────────────────────────────────────────────────────────┘

Urban Imagery

Imagery can often but not always fully cover Australia's coastal population centres. Below much of Sydney has imagery but the settlements further west aren't covered by this dataset.

The following used OpenStreetMap's (OSM) vector basemap.

Awesome Spectral Indices

The Awesome Spectral Indices project lists hundreds of chemical and naturally-occurring phenomenon that can be detected from multi- and hyperspectral imagery.

The bands in the GeoTIFFs can be paired with known algorithms that can identify things that are either invisible to the naked eye or hard to confirm with RGB-based analysis alone.

Below are a few of the soil-specific indices listed in the project. Signatures that are compatible with Sentinel-2 are tagged as such.

There is a package called spyndex which provides Python bindings for the Awesome Spectral Indices project as well.

Web Viewer

The Tidal Composites dataset also has a web viewer set up.

Below is a screenshot demonstrating false-colour tidal composites where a slider can compare data between different years.

Below low tide lines are being compared between two different years.

American Solar Farms

2025-10-13T20:00:00+03:00

Last week, Jake Stid, a postdoctoral research associate at Michigan State University, announced Ground-Mounted Solar Energy in the United States (GM-SEUS). This is a 15K-array, 2.9M-panel dataset of utility and commercial-grade solar farms across the lower 48 states plus the District of Columbia. This dataset was constructed by a team of researchers including alumni from NOAA, NASA and the USGS.

Below is a heatmap of the assets catalogued in this dataset.

GM-SEUS is broken up into two datasets, one for arrays and another panels. Below you can see a solar farm with the array outlined in red and the panels covered purple.

In this post, I'll explore GM-SEUS's Solar Farm dataset.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use GDAL 3.9.3 and a few other tools to help analyse the data in this post.

$ sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable
$ sudo apt update
$ sudo apt install \
    gdal-bin \
    jq

I'll use DuckDB v1.4.1, along with its H3, JSON, Lindel, Parquet and Spatial extensions, in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.4.1/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

The maps in this post were mostly rendered with QGIS version 3.44. QGIS is a desktop application that runs on Windows, macOS and Linux. The application has grown in popularity in recent years and has ~15M application launches from users all around the world each month.

I used QGIS' Tile+ plugin to add basemaps from Esri to the maps in this post.

Analysis-Ready Data

I'll download a dataset containing the US CENSUS State codes. This will let me map the state ID in the arrays dataset to their state name.

$ wget https://gist.github.com/a8dx/2340f9527af64f8ef8439366de981168/raw/81d876daea10eab5c2675811c39bcd18a79a9212/US_State_Bounding_Boxes.csv

I'll download the ZIP file of deliverables for GM-SEUS.

$ wget -O GMSEUS_v1_0.zip \
    'https://zenodo.org/records/14827819/files/GMSEUS_v1_0.zip?download=1'
$ unzip GMSEUS_v1_0.zip

I'll extract the projection used. This proj4 string will be used to below to re-project the data into EPSG:4326.

$ gdalsrsinfo \
    -o proj4 \
    GMSEUS_v1_0/GPKG/GMSEUS_Arrays_Final.gpkg

+proj=aea +lat_0=37.5 +lon_0=-96 +lat_1=29.5 +lat_2=45.5 +x_0=0 +y_0=0 +datum=NAD83 +units=m +no_defs

I'll use DuckDB to clean up the values and produce both a geometry field and a bounding box for each feature in this dataset. This will make working with this dataset remotely, such as from AWS S3, much easier.

$ ~/duckdb

This following produced a ZStandard-compressed, spatially-sorted Parquet file of the arrays dataset. I dropped the Z dimension as it was unused. The unknown values have been turned into NULLs. The original GPKG file was 108 MB and the resulting Parquet file is 37 MB.

COPY (
   WITH a AS (
       SELECT * EXCLUDE(geom),
              ST_FORCE2D(
                ST_FLIPCOORDINATES(
                  ST_TRANSFORM(
                      geom,
                      '+proj=aea +lat_0=37.5 +lon_0=-96 +lat_1=29.5 +lat_2=45.5 +x_0=0 +y_0=0 +datum=NAD83 +units=m +no_defs',
                      'EPSG:4326'))) geometry
       FROM   ST_READ('GMSEUS_v1_0/GPKG/GMSEUS_Arrays_Final.gpkg')
   )
   SELECT   a.* EXCLUDE (geometry,
                         tilt,
                         tiltEst,
                         instYr,
                         instYrLT,
                         effInit,
                         avgAzimuth,
                         avgLength,
                         avgSpace,
                         avgWidth),
            {'xmin': ST_XMIN(ST_EXTENT(geometry)),
             'ymin': ST_YMIN(ST_EXTENT(geometry)),
             'xmax': ST_XMAX(ST_EXTENT(geometry)),
             'ymax': ST_YMAX(ST_EXTENT(geometry))} AS bbox,
             ST_ASWKB(geometry) geometry,
             CASE WHEN instYr::INT     = -9999 THEN NULL ELSE instYr::INT   END AS instYr,
             CASE WHEN instYrLT::INT   = -9999 THEN NULL ELSE instYrLT::INT END AS instYrLT,
             CASE WHEN numRow::INT     = -9999 THEN NULL ELSE numRow::INT   END AS numRow,
             CASE WHEN tilt::INT       = -9999 THEN NULL ELSE tilt::INT     END AS tilt,
             CASE WHEN tiltEst::INT    = -9999 THEN NULL ELSE tiltEst::INT  END AS tiltEst,
             CASE WHEN effInit::INT    = -9999 THEN NULL ELSE effInit       END AS effInit,
             CASE WHEN avgAzimuth::INT = -9999 THEN NULL ELSE avgAzimuth    END AS avgAzimuth,
             CASE WHEN avgLength::INT  = -9999 THEN NULL ELSE avgLength     END AS avgLength,
             CASE WHEN avgSpace::INT   = -9999 THEN NULL ELSE avgSpace      END AS avgSpace,
             CASE WHEN avgWidth::INT   = -9999 THEN NULL ELSE avgWidth      END AS avgWidth,
             b.NAME state_name
   FROM     a
   JOIN     'US_State_Bounding_Boxes.csv' b ON a.STATEFP = b.STATEFP
   ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geometry)),
                            ST_X(ST_CENTROID(geometry))]::double[2])
) TO 'arrays.parquet' (
   FORMAT            'PARQUET',
   CODEC             'ZSTD',
   COMPRESSION_LEVEL 22,
   ROW_GROUP_SIZE    15000);

The original GPKG file for the panels dataset was 1.1 GB and the resulting Parquet file is 334 MB.

COPY (
   WITH a AS (
       SELECT * EXCLUDE(geom),
              ST_FORCE2D(
                ST_FLIPCOORDINATES(
                  ST_TRANSFORM(
                      geom,
                      '+proj=aea +lat_0=37.5 +lon_0=-96 +lat_1=29.5 +lat_2=45.5 +x_0=0 +y_0=0 +datum=NAD83 +units=m +no_defs',
                      'EPSG:4326'))) geometry
       FROM   ST_READ('GMSEUS_v1_0/GPKG/GMSEUS_Panels_Final.gpkg')
   )
   SELECT   * EXCLUDE (geometry,
                       rowSpace),
            {'xmin': ST_XMIN(ST_EXTENT(geometry)),
             'ymin': ST_YMIN(ST_EXTENT(geometry)),
             'xmax': ST_XMAX(ST_EXTENT(geometry)),
             'ymax': ST_YMAX(ST_EXTENT(geometry))} AS bbox,
             ST_ASWKB(geometry) geometry,
             CASE WHEN rowSpace::INT = -9999 THEN NULL ELSE rowSpace END AS rowSpace
   FROM     a
   ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geometry)),
                            ST_X(ST_CENTROID(geometry))]::double[2])
) TO 'panels.parquet' (
   FORMAT            'PARQUET',
   CODEC             'ZSTD',
   COMPRESSION_LEVEL 22,
   ROW_GROUP_SIZE    15000);

Solar Arrays

The arrays Parquet file has 15,017 rows. Below is an example record.

$ echo "SELECT * EXCLUDE(bbox,
                         geometry),
               bbox::JSON bbox
        FROM   'arrays.parquet'
        LIMIT  1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "COUNTYFP": "019",
    "GCR1": 0.6996,
    "GCR2": 0.614,
    "STATEFP": "45",
    "Source": "OSM",
    "arrayID": 2807,
    "avgAzimuth": 170.63,
    "avgLength": 47.76166666666666,
    "avgSpace": 3.003333333333333,
    "avgWidth": 4.776666666666666,
    "bbox": {
      "xmax": -79.97229830431786,
      "xmin": -79.97325770533094,
      "ymax": 32.87833627192598,
      "ymin": 32.87808294640646
    },
    "capMW": 0.246,
    "capMWest": 0.246,
    "effInit": 0.197963503102977,
    "instYr": 2021,
    "instYrLT": 2021,
    "latitude": 32.87818725544087,
    "longitude": -79.97276617375104,
    "modType": "c-si",
    "mount": "fixed_axis",
    "nativeID": "9324",
    "newBound": 1,
    "numRow": 6.0,
    "numRow_1": 6,
    "state_name": "South Carolina",
    "tilt": 30,
    "tiltEst": 30,
    "totArea": 1779.0,
    "totRowArea": 1244.93,
    "version": "v1.0"
  }
]

Below are the field names, data types, percentages of NULLs per column, number of unique values and minimum and maximum values for each column.

$ ~/duckdb

SELECT   column_name,
         column_type,
         null_percentage,
         approx_unique,
         min,
         max
FROM     (SUMMARIZE
          FROM READ_PARQUET('arrays.parquet'))
WHERE    column_name != 'geometry'
AND      column_name != 'bbox'
ORDER BY 1;

┌─────────────┬─────────────┬─────────────────┬───────────────┬─────────────────────┬────────────────────┐
│ column_name │ column_type │ null_percentage │ approx_unique │         min         │        max         │
│   varchar   │   varchar   │  decimal(9,2)   │     int64     │       varchar       │      varchar       │
├─────────────┼─────────────┼─────────────────┼───────────────┼─────────────────────┼────────────────────┤
│ COUNTYFP    │ VARCHAR     │            0.00 │           235 │ 001                 │ 810                │
│ GCR1        │ DOUBLE      │            0.00 │          5057 │ 0.1047              │ 1.0                │
│ GCR2        │ DOUBLE      │            0.00 │          5013 │ 0.1245              │ 0.988              │
│ STATEFP     │ VARCHAR     │            0.00 │            49 │ 01                  │ 56                 │
│ Source      │ VARCHAR     │            0.00 │             6 │ CCVPV               │ USPVDB             │
│ arrayID     │ BIGINT      │            0.00 │         13155 │ 1                   │ 15017              │
│ avgAzimuth  │ DOUBLE      │           32.84 │          4295 │ 25.0                │ 269.27             │
│ avgLength   │ DOUBLE      │           39.79 │          8358 │ 4.02                │ 449.5004           │
│ avgSpace    │ DOUBLE      │           39.79 │          8623 │ 0.024               │ 20.0               │
│ avgWidth    │ DOUBLE      │           39.79 │          9185 │ 0.67625             │ 29.80222222222222  │
│ capMW       │ DOUBLE      │            0.00 │          5280 │ 0.001250225184651   │ 1051.703           │
│ capMWest    │ DOUBLE      │            0.00 │          7863 │ 0.004               │ 3170.1             │
│ effInit     │ DOUBLE      │            0.49 │            39 │ 0.132210289727273   │ 0.205484167047619  │
│ instYr      │ INTEGER     │            0.00 │            24 │ 1985                │ 2024               │
│ instYrLT    │ INTEGER     │            0.24 │            17 │ 2009                │ 2023               │
│ latitude    │ DOUBLE      │            0.00 │         16986 │ 25.53796582594631   │ 48.99547137225406  │
│ longitude   │ DOUBLE      │            0.00 │         15656 │ -124.10440474967092 │ -67.15066374183608 │
│ modType     │ VARCHAR     │            0.00 │             3 │ c-si                │ thin-film          │
│ mount       │ VARCHAR     │            0.00 │            10 │ dual_axis           │ unknown            │
│ nativeID    │ VARCHAR     │            0.00 │         15141 │ 1                   │ York Solar         │
│ newBound    │ BIGINT      │            0.00 │             2 │ 0                   │ 1                  │
│ numRow      │ DOUBLE      │            0.00 │          1461 │ 0.0                 │ 56782.0            │
│ numRow_1    │ INTEGER     │            0.00 │          1117 │ 0                   │ 56782              │
│ state_name  │ VARCHAR     │            0.00 │            57 │ Alabama             │ Wyoming            │
│ tilt        │ INTEGER     │           55.46 │            47 │ 0                   │ 83                 │
│ tiltEst     │ INTEGER     │           55.46 │            30 │ 10                  │ 43                 │
│ totArea     │ DOUBLE      │            0.00 │         13182 │ 54.0                │ 13735113.0         │
│ totRowArea  │ DOUBLE      │            0.00 │         15396 │ 44.97               │ 7223924.662        │
│ version     │ VARCHAR     │            0.00 │             1 │ v1.0                │ v1.0               │
├─────────────┴─────────────┴─────────────────┴───────────────┴─────────────────────┴────────────────────┤
│ 29 rows                                                                                      6 columns │
└────────────────────────────────────────────────────────────────────────────────────────────────────────┘

I'll generate a heatmap of the asset locations in this dataset.

CREATE OR REPLACE TABLE h3_4_stats AS
    SELECT   H3_LATLNG_TO_CELL(
                bbox.ymin,
                bbox.xmin, 4) AS h3_4,
             COUNT(*) num_buildings
    FROM     READ_PARQUET('arrays.parquet')
    WHERE    bbox.xmin BETWEEN -178.5 AND 178.5
    GROUP BY 1;

COPY (
    SELECT ST_ASWKB(H3_CELL_TO_BOUNDARY_WKT(h3_4)::geometry) geometry,
           num_buildings
    FROM   h3_4_stats
) TO 'h3_4_stats.gpkg'
  WITH (FORMAT GDAL,
        DRIVER 'GPKG',
        LAYER_CREATION_OPTIONS 'WRITE_BBOX=YES');

Normally I would produce a Parquet file as even with 10s of thousands of records it'll generate in seconds versus a minute or so with GPKG. But ArcGIS Pro 3.5 didn't want to open the Parquet file I generated. QGIS 3.44 was fine with it but I wanted to use Esri's Nova basemap for the rendering below.

ArcGIS Pro 3.6 should be released sometime in the next few weeks so I'll re-examine this issue when it's out.

Below is the relationship between the sources of data and the installation year.

PIVOT    'arrays.parquet'
ON       Source
USING    COUNT(*)
GROUP BY instYr
ORDER BY instYr;

┌────────┬───────┬───────┬───────────────┬───────┬───────┬────────┐
│ instYr │ CCVPV │ CWSD  │ GMSEUSgeorect │  OSM  │  SAM  │ USPVDB │
│ int32  │ int64 │ int64 │     int64     │ int64 │ int64 │ int64  │
├────────┼───────┼───────┼───────────────┼───────┼───────┼────────┤
│   1985 │     0 │     0 │             0 │     0 │     0 │      1 │
│   1986 │     0 │     0 │             0 │     1 │     0 │      0 │
│   2002 │     0 │     0 │             0 │     0 │     0 │      1 │
│   2005 │     0 │     0 │             0 │    26 │     0 │      0 │
│   2006 │     0 │     0 │             0 │     2 │     0 │      1 │
│   2007 │     0 │     0 │             0 │    44 │     0 │      5 │
│   2008 │     0 │     0 │             0 │    58 │     1 │     11 │
│   2009 │     5 │     0 │             0 │    10 │     5 │     19 │
│   2010 │    20 │     0 │             0 │    71 │    20 │     37 │
│   2011 │    24 │     0 │             2 │   193 │    30 │    102 │
│   2012 │    59 │     0 │             2 │   267 │    88 │    157 │
│   2013 │    83 │     0 │             3 │   259 │    82 │    209 │
│   2014 │   102 │     0 │             1 │   335 │   119 │    291 │
│   2015 │   107 │     3 │             0 │   532 │   125 │    320 │
│   2016 │   145 │     1 │             2 │   564 │   170 │    412 │
│   2017 │   135 │     0 │             1 │   661 │   167 │    476 │
│   2018 │    66 │    34 │             4 │   644 │   210 │    414 │
│   2019 │    28 │    39 │             6 │   467 │   178 │    453 │
│   2020 │    10 │    75 │             1 │   437 │   186 │    496 │
│   2021 │     5 │    33 │             6 │   406 │   241 │    446 │
│   2022 │     1 │   173 │             3 │   231 │   354 │    166 │
│   2023 │     0 │     0 │             3 │   176 │   722 │    134 │
│   2024 │     0 │     0 │             0 │    31 │  1571 │      0 │
├────────┴───────┴───────┴───────────────┴───────┴───────┴────────┤
│ 23 rows                                               7 columns │
└─────────────────────────────────────────────────────────────────┘

Below is the relationship between the mount and mod type.

PIVOT    'arrays.parquet'
ON       modType
USING    COUNT(*)
GROUP BY mount
ORDER BY mount;

┌─────────────┬───────┬───────┬───────────┐
│    mount    │ c-si  │  csp  │ thin-film │
│   varchar   │ int64 │ int64 │   int64   │
├─────────────┼───────┼───────┼───────────┤
│ dual_axis   │   301 │    18 │         1 │
│ fixed_axis  │  6057 │    32 │       208 │
│ mixed       │     2 │     0 │         0 │
│ mixed_df    │   189 │     7 │         0 │
│ mixed_dfs   │    94 │     0 │         0 │
│ mixed_ds    │    38 │     1 │         0 │
│ mixed_fs    │    60 │     0 │         1 │
│ single_axis │  2876 │    11 │       231 │
│ unknown     │  4885 │     5 │         0 │
└─────────────┴───────┴───────┴───────────┘

Below are the array capacity counts rounded to the neared 100 MW and broken down by source.

WITH a AS (
    SELECT   Source,
             ROUND(capMW / 100) * 100 AS capacity,
             COUNT(*) num_recs
    FROM     'arrays.parquet'
    GROUP BY 1, 2
)
PIVOT    a
ON       Source
USING    SUM(num_recs)
GROUP BY capacity
ORDER BY capacity;

┌──────────┬────────┬────────┬───────────────┬────────┬────────┬────────┐
│ capacity │ CCVPV  │  CWSD  │ GMSEUSgeorect │  OSM   │  SAM   │ USPVDB │
│  double  │ int128 │ int128 │    int128     │ int128 │ int128 │ int128 │
├──────────┼────────┼────────┼───────────────┼────────┼────────┼────────┤
│      0.0 │    790 │    356 │            33 │   5295 │   4022 │   3669 │
│    100.0 │   NULL │      2 │          NULL │     67 │    143 │    350 │
│    200.0 │   NULL │   NULL │             1 │     22 │     49 │     73 │
│    300.0 │   NULL │   NULL │          NULL │     17 │     21 │     49 │
│    400.0 │   NULL │   NULL │          NULL │      6 │     13 │      7 │
│    500.0 │   NULL │   NULL │          NULL │      4 │     11 │      2 │
│    600.0 │   NULL │   NULL │          NULL │      2 │      3 │   NULL │
│    700.0 │   NULL │   NULL │          NULL │      1 │      3 │   NULL │
│    800.0 │   NULL │   NULL │          NULL │   NULL │      2 │      1 │
│    900.0 │   NULL │   NULL │          NULL │   NULL │      1 │   NULL │
│   1000.0 │   NULL │   NULL │          NULL │   NULL │      1 │   NULL │
│   1100.0 │   NULL │   NULL │          NULL │      1 │   NULL │   NULL │
├──────────┴────────┴────────┴───────────────┴────────┴────────┴────────┤
│ 12 rows                                                     7 columns │
└───────────────────────────────────────────────────────────────────────┘

Solar Panels

The panels Parquet file has 2,917,782 rows. Below is an example record.

$ echo "SELECT * EXCLUDE(bbox,
                         geometry),
               bbox::JSON bbox
        FROM   'panels.parquet'
        LIMIT  1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "Source": "gmseus",
    "arrayID": 2807.0,
    "bbox": {
      "xmax": -79.97312295800064,
      "xmin": -79.97325770533483,
      "ymax": 32.87833627193374,
      "ymin": 32.87830393275682
    },
    "panelID": 2620732,
    "rowArea": 29.1,
    "rowAzimuth": 174.62,
    "rowLength": 12.77,
    "rowMount": "fixed_axis",
    "rowSpace": 8.42,
    "rowWidth": 3.0,
    "version": "v1.0"
  }
]

Below are the field names, data types, percentages of NULLs per column, number of unique values and minimum and maximum values for each column.

$ ~/duckdb

SELECT   column_name,
         column_type,
         null_percentage,
         approx_unique,
         min,
         max
FROM     (SUMMARIZE
          FROM READ_PARQUET('panels.parquet'))
WHERE    column_name != 'geometry'
AND      column_name != 'bbox'
ORDER BY 1;

┌─────────────┬─────────────┬─────────────────┬───────────────┬───────────────┬─────────────┐
│ column_name │ column_type │ null_percentage │ approx_unique │      min      │     max     │
│   varchar   │   varchar   │  decimal(9,2)   │     int64     │    varchar    │   varchar   │
├─────────────┼─────────────┼─────────────────┼───────────────┼───────────────┼─────────────┤
│ Source      │ VARCHAR     │            0.00 │             3 │ CCVPV         │ gmseus      │
│ arrayID     │ DOUBLE      │            0.08 │          9451 │ 1.0           │ 15017.0     │
│ panelID     │ BIGINT      │            0.00 │       2703164 │ 1             │ 2917782     │
│ rowArea     │ DOUBLE      │            0.00 │         88974 │ 15.01         │ 1999.76     │
│ rowAzimuth  │ DOUBLE      │            0.00 │         14901 │ 90.0          │ 270.0       │
│ rowLength   │ DOUBLE      │            0.00 │         26759 │ 4.02          │ 530.05      │
│ rowMount    │ VARCHAR     │            0.00 │             3 │ dual_axis     │ single_axis │
│ rowSpace    │ DOUBLE      │            0.17 │         13376 │ 7.4765186e-08 │ 20.0        │
│ rowWidth    │ DOUBLE      │            0.00 │          1863 │ 0.45          │ 102.14      │
│ version     │ VARCHAR     │            0.00 │             1 │ v1.0          │ v1.0        │
├─────────────┴─────────────┴─────────────────┴───────────────┴───────────────┴─────────────┤
│ 10 rows                                                                         6 columns │
└───────────────────────────────────────────────────────────────────────────────────────────┘

Below is the relationship between the sources of data and the row mount.

PIVOT    'panels.parquet'
ON       Source
USING    COUNT(*)
GROUP BY rowMount
ORDER BY rowMount;

┌─────────────┬────────┬────────┬─────────┐
│  rowMount   │ CCVPV  │  OSM   │ gmseus  │
│   varchar   │ int64  │ int64  │  int64  │
├─────────────┼────────┼────────┼─────────┤
│ dual_axis   │     44 │   5975 │   80225 │
│ fixed_axis  │  13344 │ 118639 │  163512 │
│ single_axis │ 189699 │ 743371 │ 1602973 │
└─────────────┴────────┴────────┴─────────┘

Of the 15,017 arrays in this dataset, only 5,358 have any panels in them.

.maxrows 20

SELECT   a.arrayID,
         COUNT(DISTINCT b.panelID)
FROM     READ_PARQUET('arrays.parquet') a
JOIN     READ_PARQUET('panels.parquet') b
ON       ST_COVERS(a.geometry, b.geometry)
GROUP BY a.arrayID
ORDER BY 2 DESC;

┌─────────┬───────────────────────────┐
│ arrayID │ count(DISTINCT b.panelID) │
│  int64  │           int64           │
├─────────┼───────────────────────────┤
│   11958 │                     56762 │
│   14225 │                     51140 │
│   12162 │                     43741 │
│   12433 │                     37304 │
│   14461 │                     31898 │
│   13229 │                     30093 │
│    6589 │                     27080 │
│   13329 │                     25120 │
│   12597 │                     24054 │
│   12224 │                     23449 │
│      ·  │                         · │
│      ·  │                         · │
│      ·  │                         · │
│    1792 │                         1 │
│    2286 │                         1 │
│     863 │                         1 │
│    1816 │                         1 │
│    8997 │                         1 │
│   12358 │                         1 │
│    6564 │                         1 │
│    3845 │                         1 │
│    6574 │                         1 │
│     991 │                         1 │
├─────────┴───────────────────────────┤
│ 5358 rows (20 shown)      2 columns │
└─────────────────────────────────────┘

Below is a solar farm in Nevada where some arrays have panels and others do not.

I was interested in seeing the solar farm with 56K panels. Below are its coordinates.

SELECT ST_CENTROID(geometry)
FROM   'arrays.parquet'
WHERE  arrayID = 11958;

┌────────────────────────────────────────────────┐
│             st_centroid(geometry)              │
│                    geometry                    │
├────────────────────────────────────────────────┤
│ POINT (-115.34248808114013 35.611919498003175) │
└────────────────────────────────────────────────┘

Even this has arrays without marked panels.

I'm looking forward to v2 of this dataset with better panel detection. It'll be great to get a good approximation of how many are deployed in the US.

A Global Mining Dataset

2025-10-06T20:00:00+03:00

The International Council on Mining and Metals (ICMM) is a mining industry group that was founded in London in 2001. In September, they released a Global Mining Dataset containing over 8,000 mines and other related assets around the globe.

Below is a heatmap of the assets they catalogued in this dataset.

In this post, I'll explore ICMM's mining dataset.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use Python 3.12.3 and a few other tools to help analyse the data in this post.

$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt update
$ sudo apt install \
    jq \
    python3-pip \
    python3.12-venv

I'll set up a Python Virtual Environment and install an OSM vector tile data extraction utility I wrote last year.

$ python3 -m venv ~/.osm
$ source ~/.osm/bin/activate

$ git clone https://github.com/marklit/tiles2columns \
    ~/tiles2columns
$ python -m pip install \
         -r ~/tiles2columns/requirements.txt

I'll use DuckDB v1.3.0, along with its H3, JSON, Lindel, Parquet and Spatial extensions, in this post. Normally I try and use the latest release of DuckDB but v1.4.0 has an issue where it's Parquet files aren't readable by many of the tools I use at the moment.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.3.0/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

I used QGIS' Tile+ plugin to add basemaps from Esri to one of the maps in this post.

The dark, non-satellite map of ICMM's asset locations above is mostly made up of vector data from Natural Earth and Overture.

Analysis-Ready Data

I'll download ICMM's Excel file for this dataset.

$ wget -O global-mining-dataset.xlsx \
    'https://www.icmm.com/website/data/2025/global-mining-dataset.xlsx?cb=117612'

$ ~/duckdb

CREATE OR REPLACE TABLE icmm AS
    SELECT * EXCLUDE(Latitude,
                     Longitude,
                     "Mine Name",
                     "Primary Commodity",
                     "Secondary Commodity",
                     "Other Commodities"),
           ST_POINT(Longitude, Latitude) AS geometry,
           {'xmin': ST_XMIN(ST_EXTENT(geometry)),
            'ymin': ST_YMIN(ST_EXTENT(geometry)),
            'xmax': ST_XMAX(ST_EXTENT(geometry)),
            'ymax': ST_YMAX(ST_EXTENT(geometry))} AS bbox,
           TRIM("Mine Name")           AS "Mine Name",
           TRIM("Primary Commodity")   AS "Primary Commodity",
           TRIM("Secondary Commodity") AS "Secondary Commodity"
    FROM READ_XLSX('global-mining-dataset.xlsx',
                   sheet='External',
                   ignore_errors=true);

This following produced a ZStandard-compressed, spatially-sorted Parquet file of ICMM's dataset.

COPY (
    SELECT   * EXCLUDE (geometry),
             ST_ASWKB(geometry) geometry
    FROM     icmm
    WHERE    ST_X(ST_CENTROID(geometry)) IS NOT NULL
    ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geometry)),
                             ST_X(ST_CENTROID(geometry))]::double[2])
) TO 'icmm.parquet' (
   FORMAT            'PARQUET',
   CODEC             'ZSTD',
   COMPRESSION_LEVEL 22,
   ROW_GROUP_SIZE    15000);

Data Fluency

The above Parquet file is 508 KB and contains 8,508 rows. Below is an example record.

$ echo "SELECT * EXCLUDE(bbox),
               bbox::JSON bbox
        FROM   READ_PARQUET('icmm.parquet')
        WHERE  ICMMID = 'ICMM08147'" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "Asset Type": "Mine",
    "Confidence Factor": "High",
    "Country or Region": "Canada",
    "Group Names": "Diavik;Diavik",
    "ICMMID": "ICMM08147",
    "Mine Name": "Diavik",
    "Primary Commodity": "diamond",
    "Secondary Commodity": null,
    "bbox": {
      "xmax": -110.2743898,
      "xmin": -110.2743898,
      "ymax": 64.49236014,
      "ymin": 64.49236014
    },
    "geometry": "POINT (-110.2743898 64.49236014)"
  }
]

Below are the field names, data types, percentages of NULLs per column, number of unique values and minimum and maximum values for each column.

SELECT   column_name,
         column_type,
         null_percentage,
         approx_unique,
         min,
         max
FROM     (SUMMARIZE
          FROM icmm)
WHERE    column_name != 'geometry'
AND      column_name != 'bbox'
ORDER BY 1;

┌─────────────────────┬─────────────┬─────────────────┬───────────────┬───────────────────────┬────────────────────┐
│     column_name     │ column_type │ null_percentage │ approx_unique │          min          │        max         │
│       varchar       │   varchar   │  decimal(9,2)   │     int64     │        varchar        │      varchar       │
├─────────────────────┼─────────────┼─────────────────┼───────────────┼───────────────────────┼────────────────────┤
│ Asset Type          │ VARCHAR     │            0.00 │            17 │ Mine                  │ Steel Plant        │
│ Confidence Factor   │ VARCHAR     │            0.00 │             3 │ High                  │ Very Low           │
│ Country or Region   │ VARCHAR     │            0.00 │           129 │ Afghanistan           │ Zimbabwe           │
│ Group Names         │ VARCHAR     │           50.25 │          3639 │ # 5 Mine;No. 5 Coal…  │ ÄŒSA Coal Mine;CSA │
│ ICMMID              │ VARCHAR     │            0.00 │          7920 │ ICMM00026             │ ICMM21432          │
│ Mine Name           │ VARCHAR     │            0.00 │          8754 │ # 5 Mine              │ ÄŒSA Coal Mine     │
│ Primary Commodity   │ VARCHAR     │            0.00 │            43 │ alumina               │ zircon             │
│ Secondary Commodity │ VARCHAR     │           72.57 │            60 │ alumina               │ zircon             │
└─────────────────────┴─────────────┴─────────────────┴───────────────┴───────────────────────┴────────────────────┘

Commodities

Below is the number of assets per primary community type broken down by confidence factor.

.maxrows 100

PIVOT    READ_PARQUET('icmm.parquet')
ON       "Confidence Factor"
USING    COUNT(*)
GROUP BY "Primary Commodity"
ORDER BY "Primary Commodity";

┌────────────────────────┬───────┬──────────┬──────────┐
│   Primary Commodity    │ High  │ Moderate │ Very Low │
│        varchar         │ int64 │  int64   │  int64   │
├────────────────────────┼───────┼──────────┼──────────┤
│ alumina                │    64 │        5 │        0 │
│ aluminium              │    91 │        9 │        0 │
│ antimony               │     0 │        1 │        5 │
│ barium                 │     0 │        0 │      134 │
│ bauxite                │    17 │        5 │        0 │
│ borates                │     0 │        1 │        0 │
│ boron                  │     0 │        0 │        2 │
│ chromite               │     2 │       27 │        4 │
│ chromium               │     3 │        9 │       46 │
│ coal                   │   508 │      729 │        0 │
│ cobalt                 │     3 │        0 │       12 │
│ copper                 │   336 │      212 │      353 │
│ diamond                │     5 │        0 │        0 │
│ ferrochrome            │     0 │        2 │        0 │
│ ferromanganese         │     3 │        2 │        0 │
│ ferronickel            │     8 │        1 │        0 │
│ ferroniobium           │     0 │        1 │        0 │
│ ferrosilicon manganese │     0 │        1 │        0 │
│ fluorspar              │     0 │        1 │        0 │
│ gold                   │   379 │      261 │      237 │
│ heavy mineral sands    │     3 │        0 │        0 │
│ ilmenite               │     1 │        3 │        0 │
│ iron ore               │   299 │      312 │        0 │
│ lanthanides            │     0 │        1 │        0 │
│ lead                   │    31 │       46 │       33 │
│ lithium                │    34 │        7 │       24 │
│ manganese              │     5 │        1 │        0 │
│ mercury                │     0 │        0 │        2 │
│ metallurgical coal     │   366 │      525 │        0 │
│ molybdenum             │     2 │        1 │        0 │
│ nickel                 │    52 │       25 │       71 │
│ niobium                │     1 │        0 │        0 │
│ palladium              │     1 │        0 │        0 │
│ phosphate              │     2 │        5 │        0 │
│ platinum               │    24 │        2 │       59 │
│ potash                 │     1 │        0 │        0 │
│ silver                 │    49 │       18 │      179 │
│ steel                  │     8 │       75 │        0 │
│ thermal coal           │   675 │     1382 │        0 │
│ tin                    │     1 │       20 │      277 │
│ titanium               │     1 │        0 │        0 │
│ tungsten               │     0 │       11 │       94 │
│ uranium                │     1 │        3 │        3 │
│ vanadium               │     1 │        0 │        0 │
│ zinc                   │   183 │       18 │       89 │
│ zircon                 │     0 │        1 │        0 │
├────────────────────────┴───────┴──────────┴──────────┤
│ 46 rows                                    4 columns │
└──────────────────────────────────────────────────────┘

Below are the most common primary and secondary commodity pairs.

SELECT   "Primary Commodity",
         "Secondary Commodity",
         COUNT(*)
FROM     READ_PARQUET('icmm.parquet')
WHERE    "Secondary Commodity" IS NOT NULL
GROUP BY 1, 2
ORDER BY 3 DESC
LIMIT    50;

┌────────────────────┬─────────────────────┬──────────────┐
│ Primary Commodity  │ Secondary Commodity │ count_star() │
│      varchar       │       varchar       │    int64     │
├────────────────────┼─────────────────────┼──────────────┤
│ gold               │ silver              │          398 │
│ metallurgical coal │ thermal coal        │          396 │
│ copper             │ gold                │          361 │
│ zinc               │ lead                │          115 │
│ copper             │ silver              │           53 │
│ lead               │ zinc                │           52 │
│ gold               │ copper              │           48 │
│ copper             │ cobalt              │           44 │
│ copper             │ lead                │           40 │
│ nickel             │ cobalt              │           40 │
│ copper             │ molybdenum          │           40 │
│ silver             │ gold                │           36 │
│ nickel             │ copper              │           35 │
│ copper             │ zinc                │           28 │
│ gold               │ platinum            │           28 │
│ gold               │ lead                │           27 │
│ copper             │ nickel              │           27 │
│ silver             │ lead                │           23 │
│ lead               │ silver              │           22 │
│ zinc               │ copper              │           21 │
│ tin                │ tungsten            │           20 │
│ chromite           │ chromium            │           20 │
│ zinc               │ silver              │           20 │
│ platinum           │ palladium           │           19 │
│ alumina            │ aluminium           │           19 │
│ iron ore           │ manganese           │           17 │
│ silver             │ zinc                │           13 │
│ lithium            │ tantalum            │           11 │
│ tin                │ zinc                │           11 │
│ lead               │ copper              │           10 │
│ silver             │ copper              │           10 │
│ gold               │ palladium           │            9 │
│ steel              │ iron                │            8 │
│ gold               │ antimony            │            8 │
│ ferronickel        │ nickel              │            8 │
│ copper             │ platinum            │            8 │
│ alumina            │ bauxite             │            8 │
│ gold               │ uranium             │            7 │
│ zinc               │ cadmium             │            7 │
│ iron ore           │ copper              │            6 │
│ iron ore           │ gold                │            6 │
│ gold               │ zinc                │            6 │
│ gold               │ arsenic             │            6 │
│ lithium            │ potash              │            5 │
│ tin                │ silver              │            5 │
│ chromium           │ chromite            │            5 │
│ gold               │ iron ore            │            5 │
│ gold               │ nickel              │            5 │
│ zinc               │ gold                │            5 │
│ antimony           │ gold                │            4 │
├────────────────────┴─────────────────────┴──────────────┤
│ 50 rows                                       3 columns │
└─────────────────────────────────────────────────────────┘

Assets

This is the number of asset types in this dataset broken down by confidence factor. Some assets are the combination of two or more types.

PIVOT    READ_PARQUET('icmm.parquet')
ON       "Confidence Factor"
USING    COUNT(*)
GROUP BY "Asset Type"
ORDER BY "Asset Type";

┌────────────────────────┬───────┬──────────┬──────────┐
│       Asset Type       │ High  │ Moderate │ Very Low │
│        varchar         │ int64 │  int64   │  int64   │
├────────────────────────┼───────┼──────────┼──────────┤
│ Mine                   │  2702 │     3518 │     1576 │
│ Mine;Plant             │    68 │       48 │        7 │
│ Mine;Refinery          │    13 │        5 │        0 │
│ Mine;Refinery;Plant    │     0 │        1 │        0 │
│ Mine;Smelter           │    22 │       21 │        1 │
│ Mine;Smelter;Plant     │     3 │        3 │        0 │
│ Mine;Smelter;Refinery  │    18 │       11 │        0 │
│ Plant                  │     9 │       11 │       31 │
│ Refinery               │    59 │       16 │        1 │
│ Refinery; Steel Plant  │     1 │        0 │        0 │
│ Refinery;Plant         │     3 │        1 │        1 │
│ Smelter                │   143 │       64 │        7 │
│ Smelter;Plant          │     3 │        1 │        0 │
│ Smelter;Refinery       │   106 │        3 │        0 │
│ Smelter;Refinery;Plant │     5 │        2 │        0 │
│ Smelter;Steel Plant    │     2 │        0 │        0 │
│ Steel Plant            │     3 │       18 │        0 │
├────────────────────────┴───────┴──────────┴──────────┤
│ 17 rows                                    4 columns │
└──────────────────────────────────────────────────────┘

Countries

Below are the most and least represented countries in this dataset.

.maxrows 25

SELECT   COUNT(*),
         "Country or Region"
FROM     READ_PARQUET('icmm.parquet')
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬─────────────────────┐
│ count_star() │  Country or Region  │
│    int64     │       varchar       │
├──────────────┼─────────────────────┤
│         1839 │ China               │
│         1627 │ United States       │
│          588 │ Australia           │
│          473 │ Indonesia           │
│          424 │ India               │
│          328 │ Russia              │
│          272 │ South Africa        │
│          252 │ Brazil              │
│          206 │ Peru                │
│          197 │ Canada              │
│          146 │ Bolivia             │
│          144 │ Chile               │
│          136 │ Mexico              │
│            · │  ·                  │
│            · │  ·                  │
│            · │  ·                  │
│            2 │ Fiji                │
│            1 │ Rwanda              │
│            1 │ Trinidad and Tobago │
│            1 │ Gabon               │
│            1 │ Bangladesh          │
│            1 │ Bahrain             │
│            1 │ Nepal               │
│            1 │ Luxembourg          │
│            1 │ Taiwan              │
│            1 │ Niger               │
│            1 │ Qatar               │
│            1 │ Greenland           │
├──────────────┴─────────────────────┤
│ 129 rows (25 shown)      2 columns │
└────────────────────────────────────┘

I'll generate a heatmap of the asset locations in this dataset.

CREATE OR REPLACE TABLE h3_3_stats AS
    SELECT   H3_LATLNG_TO_CELL(
                bbox.ymin,
                bbox.xmin, 3) AS h3_3,
             COUNT(*) num_buildings
    FROM     READ_PARQUET('icmm.parquet')
    WHERE    bbox.xmin BETWEEN -178.25 AND 178.25
    GROUP BY 1;

COPY (
    SELECT ST_ASWKB(H3_CELL_TO_BOUNDARY_WKT(h3_3)::geometry) geometry,
           num_buildings
    FROM   h3_3_stats
) TO 'h3_3_stats.gpkg'
  WITH (FORMAT GDAL,
        DRIVER 'GPKG',
        LAYER_CREATION_OPTIONS 'WRITE_BBOX=YES');

Diavik Diamond Mine

I'll download OpenStreetMap's (OSM) data for Canada's Diavik Diamond Mine. This is using the tiles2columns utility I built last year following OSM's addition of vector-based tiles to their service.

$ mkdir -p ~/diavik
$ cd ~/diavik

$ python3 ~/tiles2columns/main.py \
    centroid \
    -110.29299 64.48937 \
    --distance 0.1

The above extracted 1.2 MB worth of GeoPackage (GPKG) files. They're grouped by feature type.

$ du -hsc *

96K     buildings.gpkg
96K     dam_polygons.gpkg
124K    land.gpkg
96K     public_transport.gpkg
96K     street_labels.gpkg
128K    streets.gpkg
148K    water_lines.gpkg
404K    water_polygons.gpkg
1.2M    total

Umbra are a satellite manufacturer and constellation operator. They produce Synthetic Aperture Radar (SAR) satellites that can see through clouds and capture imagery of the earth day or night.

They operate an open feed with some of the imagery they capture. One of the locations they publish imagery of regularly is the Diavik Diamond Mine.

I've placed four images they captured on August 26th and September 5th, 17th and 26th on top of one another. Each image has a slightly different footprint so the four images in combination cover a wider area than any one image. Given how mines can change from day to day I've constrained the overlapping of imagery to a few weeks.

The image below has Esri's World Imagery basemap tinted blue. Tinted red is Umbra's imagery and the vector data on top is from OSM. I've used EPSG:3978 for the projection as this is in Northern Canada.

Canada's 14M Buildings

2025-10-03T20:00:00+03:00

In April, Statistics Canada released a refreshed version of their Open Database of Buildings (ODB) dataset for Canada. This is one of Canada's most comprehensive building datasets. Below is a heatmap of its building footprints.

Statscan pulled buildings data from 530 datasets across 107 government sources. When I initially began to spot-check the data, even newly constructed neighbourhoods in South East Calgary were covered. Below is Overture's September release in purple and ODB's buildings in yellow.

In this post, I'll explore Statistics Canada's ODB dataset.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use GDAL 3.9.3 and a few other tools to help analyse the data in this post.

$ sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable
$ sudo apt update
$ sudo apt install \
    gdal-bin \
    jq

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.3.0/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

I used QGIS' Tile+ plugin to add basemaps from Google and OpenStreetMap (OSM) to the maps throughout this post.

Analysis-Ready Data

Statscan broke up the dataset into Zipped, GeoPackage (GPKG) files by province / territory with some of these broken across multiple files. Below I'll build a manifest of these URLs and download them with four concurrent threads.

$ mkdir -p ~/odb
$ cd ~/odb

$ vi urls.txt

https://www150.statcan.gc.ca/pub/34-26-0001/2018001/zip/ODB_v3_NL.zip
https://www150.statcan.gc.ca/pub/34-26-0001/2018001/zip/ODB_v3_PE.zip
https://www150.statcan.gc.ca/pub/34-26-0001/2018001/zip/ODB_v3_NS.zip
https://www150.statcan.gc.ca/pub/34-26-0001/2018001/zip/ODB_v3_NB.zip
https://www150.statcan.gc.ca/pub/34-26-0001/2018001/zip/ODB_v3_QC_1.zip
https://www150.statcan.gc.ca/pub/34-26-0001/2018001/zip/ODB_v3_QC_2.zip
https://www150.statcan.gc.ca/pub/34-26-0001/2018001/zip/ODB_v3_ON_1.zip
https://www150.statcan.gc.ca/pub/34-26-0001/2018001/zip/ODB_v3_ON_2.zip
https://www150.statcan.gc.ca/pub/34-26-0001/2018001/zip/ODB_v3_ON_3.zip
https://www150.statcan.gc.ca/pub/34-26-0001/2018001/zip/ODB_v3_MB.zip
https://www150.statcan.gc.ca/pub/34-26-0001/2018001/zip/ODB_v3_SK.zip
https://www150.statcan.gc.ca/pub/34-26-0001/2018001/zip/ODB_v3_AB.zip
https://www150.statcan.gc.ca/pub/34-26-0001/2018001/zip/ODB_v3_BC.zip
https://www150.statcan.gc.ca/pub/34-26-0001/2018001/zip/ODB_v3_YT.zip
https://www150.statcan.gc.ca/pub/34-26-0001/2018001/zip/ODB_v3_NT.zip

$ cat urls.txt \
    | xargs -n1 \
            -P4 \
            -I% \
            wget -c "%"

I'll extract the GPKG files from each of the ZIPs.

$ find . \
    -name "*.zip" \
    -type f \
    -exec unzip {} "*.gpkg" \;

Below is an example record from one of the GPKG files. Note ".." is being used for NULL values.

$ echo "FROM  ST_READ('ODB_v3_AB.gpkg')
        LIMIT 1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "address": "..",
    "csdname": "Clearwater County",
    "csduid": "4809002",
    "dataset": "Building Footprints",
    "floors": "..",
    "geom": "MULTIPOLYGON (((4615587.032158795 2032592.4950364884, 4615575.2249902915 2032585.4147511278, 4615570.227409255 2032593.7528553815, 4615582.04455081 2032600.8323487062, 4615587.032158795 2032592.4950364884)))",
    "height": "..",
    "id": "c9ffb5f954f942b3a98bad6b1932360c",
    "name": "Residence 2",
    "prov_terr": "AB",
    "source": "Government of Canada",
    "source_id": "1.0",
    "sq_ft": "..",
    "type": "..",
    "units": "..",
    "year_built": ".."
  }
]

I'll extract the projection Statscan used. This proj4 string will be used to below to re-project the data into EPSG:4326.

$ gdalsrsinfo \
    -o proj4 \
    ODB_v3_AB.gpkg

+proj=lcc +lat_0=63.390675 +lon_0=-91.8666666666667 +lat_1=49 +lat_2=77 +x_0=6200000 +y_0=3000000 +datum=NAD83 +units=m +no_defs

I'll convert the GPKG files into spatially-sorted, ZStandard-compressed Parquet format with an EPSG:4326 projection. I've also made a clear source field and added bounding boxes to each piece of geometry.

This format will load without issue in QGIS 3.44 and ArcGIS Pro 3.5. The bounding boxes will help optimise bandwidth usage when querying this data on remote servers, like AWS S3.

$ for FILENAME in *.gpkg; do
     echo $FILENAME

     BASENAME=`basename $FILENAME | cut -d. -f1`

     echo "COPY (
               WITH a AS (
                   SELECT * EXCLUDE(geom),
                          ST_FLIPCOORDINATES(
                              ST_TRANSFORM(
                                  geom,
                                  '+proj=lcc +lat_0=63.390675 +lon_0=-91.8666666666667 +lat_1=49 +lat_2=77 +x_0=6200000 +y_0=3000000 +datum=NAD83 +units=m +no_defs',
                                  'EPSG:4326')) geometry
                   FROM ST_READ('$FILENAME')
               )
               SELECT   * EXCLUDE (address,
                                   floors,
                                   height,
                                   name,
                                   sq_ft,
                                   type,
                                   units,
                                   year_built,
                                   geometry),
                        {'xmin': ST_XMIN(ST_EXTENT(geometry)),
                         'ymin': ST_YMIN(ST_EXTENT(geometry)),
                         'xmax': ST_XMAX(ST_EXTENT(geometry)),
                         'ymax': ST_YMAX(ST_EXTENT(geometry))} AS bbox,
                         ST_ASWKB(geometry) geometry,
                         CASE WHEN address = '..'    THEN NULL ELSE address    END AS address,
                         CASE WHEN floors = '..'     THEN NULL ELSE floors     END AS floors,
                         CASE WHEN height = '..'     THEN NULL ELSE height     END AS height,
                         CASE WHEN name = '..'       THEN NULL ELSE name       END AS name,
                         CASE WHEN sq_ft = '..'      THEN NULL ELSE sq_ft      END AS sq_ft,
                         CASE WHEN type = '..'       THEN NULL ELSE type       END AS type,
                         CASE WHEN units = '..'      THEN NULL ELSE units      END AS units,
                         CASE WHEN year_built = '..' THEN NULL ELSE year_built END AS year_built
               FROM     a
               ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geometry)),
                                        ST_X(ST_CENTROID(geometry))]::double[2])
           ) TO '$BASENAME.parquet' (
               FORMAT            'PARQUET',
               CODEC             'ZSTD',
               COMPRESSION_LEVEL 22,
               ROW_GROUP_SIZE    15000);
           " | ~/duckdb
 done

The above turned 2.5 GB of ZIP files containing 6.2 GB of GPKG files into 1.8 GB of Parquet.

$ du -hsc *.parquet

164M    ODB_v3_AB.parquet
188M    ODB_v3_BC.parquet
90M     ODB_v3_MB.parquet
75M     ODB_v3_NB.parquet
21M     ODB_v3_NL.parquet
62M     ODB_v3_NS.parquet
1.4M    ODB_v3_NT.parquet
261M    ODB_v3_ON_1.parquet
259M    ODB_v3_ON_2.parquet
211M    ODB_v3_ON_3.parquet
11M     ODB_v3_PE.parquet
249M    ODB_v3_QC_1.parquet
223M    ODB_v3_QC_2.parquet
35M     ODB_v3_SK.parquet
1.5M    ODB_v3_YT.parquet
1.8G    total

Heatmap

Below is a heatmap of this dataset.

CREATE OR REPLACE TABLE h3_4_stats AS
    SELECT   H3_LATLNG_TO_CELL(
                bbox.ymin,
                bbox.xmin, 4) AS h3_4,
             COUNT(*) num_buildings
    FROM     READ_PARQUET('ODB_v3*.parquet')
    WHERE    bbox.xmin BETWEEN -178.5 AND 178.5
    GROUP BY 1;

COPY (
    SELECT ST_ASWKB(H3_CELL_TO_BOUNDARY_WKT(h3_4)::geometry) geometry,
           num_buildings
    FROM   h3_4_stats
) TO 'h3_4_stats.gpkg'
  WITH (FORMAT GDAL,
        DRIVER 'GPKG',
        LAYER_CREATION_OPTIONS 'WRITE_BBOX=YES');

The following was needed for ArcGIS Pro to recognise the projection of the above hexagons properly.

$ ogr2ogr \
    -f GPKG \
    -a_srs EPSG:4326 \
    h3_4_stats.4326.gpkg \
    h3_4_stats.gpkg

Data Fluency

Below are the field names, data types, percentages of NULLs per column, number of unique values and minimum and maximum values for each column.

$ ~/duckdb

SELECT   column_name,
         column_type,
         null_percentage,
         approx_unique,
         min,
         max
FROM     (SUMMARIZE
          FROM READ_PARQUET('ODB_v3*.parquet'))
WHERE    column_name != 'geometry'
AND      column_name != 'bbox'
ORDER BY 1;

┌─────────────┬─────────────┬─────────────────┬───────────────┬──────────────────────────────────────────┬────────────────────────────────────────┐
│ column_name │ column_type │ null_percentage │ approx_unique │                   min                    │                  max                   │
│   varchar   │   varchar   │  decimal(9,2)   │     int64     │                 varchar                  │                varchar                 │
├─────────────┼─────────────┼─────────────────┼───────────────┼──────────────────────────────────────────┼────────────────────────────────────────┤
│ address     │ VARCHAR     │           69.19 │       4840881 │                                          │ à définir                              │
│ csdname     │ VARCHAR     │            0.00 │          3099 │ Abbotsford                               │ qathet E                               │
│ csduid      │ VARCHAR     │            0.00 │          3465 │ 1001124                                  │ 6106097                                │
│ dataset     │ VARCHAR     │            0.00 │           966 │ 2022 Voting Location Building Footprint  │ Édifices municipaux; Lieux Publics     │
│ floors      │ VARCHAR     │           97.34 │            35 │ 1.0                                      │ 9.0                                    │
│ height      │ VARCHAR     │           91.59 │        139106 │ -0.10836829                              │ 99.99                                  │
│ id          │ VARCHAR     │            0.00 │      14730146 │ 0000014f86fe39839c5f1c118e058bfb         │ ffffff6e9491f9ecb4f4444f7457e84b       │
│ name        │ VARCHAR     │           98.73 │         29024 │                                          │ �cole secondaire de Par-en-Bas         │
│ prov_terr   │ VARCHAR     │            0.00 │            11 │ AB                                       │ YT                                     │
│ source      │ VARCHAR     │            0.00 │            99 │ Cape Breton Regional Municipality (CBRM) │ Ville de Sherbrooke                    │
│ source_id   │ VARCHAR     │            0.00 │       8539298 │                                          │ {FFFFFB51-F6CE-4DF6-93AA-FBFDDF22FEEB} │
│ sq_ft       │ VARCHAR     │           99.05 │         41985 │ 10.60419022                              │ 9996.0                                 │
│ type        │ VARCHAR     │           85.22 │           989 │ 117                                      │ Église                                 │
│ units       │ VARCHAR     │           98.42 │           176 │ 1.0                                      │ 99.0                                   │
│ year_built  │ VARCHAR     │           98.39 │           235 │ 1750                                     │ c                                      │
├─────────────┴─────────────┴─────────────────┴───────────────┴──────────────────────────────────────────┴────────────────────────────────────────┤
│ 15 rows                                                                                                                               6 columns │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

The NULL percentages on the metadata fields seems very high. Having an address for each building is unique as other datasets like Overture don't link buildings and addresses to one another. It's a shame there's ~30% coverage here.

Some buildings have an address made up of several concatenated addresses. The example below looks to list every unit within a building with its full first line address.

$ ~/duckdb

SELECT   address
FROM     'ODB_v3_*.parquet'
ORDER BY LENGTH(address) DESC
LIMIT    1;

S1001-330 Phillip St; S1002-330 Phillip St; S1003-330 Phillip St; S1004-330 Phillip St; S1005-330 Phillip St; S1006-330 Phillip St; S1007-330...

Sources

Below are a breakdown of sources in this dataset.

$ ~/duckdb

SELECT   source,
         COUNT(*)
FROM     'ODB_v3_*.parquet'
GROUP BY 1
ORDER BY 2 DESC;

┌───────────────────────────────────────┬──────────────┐
│                source                 │ count_star() │
│                varchar                │    int64     │
├───────────────────────────────────────┼──────────────┤
│ Government of Canada                  │      3797173 │
│ Government of Québec                  │      2448716 │
│ Government of New Brunswick           │       572033 │
│ City of Toronto                       │       531571 │
│ City of Calgary                       │       488028 │
│ City of Ottawa                        │       385184 │
│ City of Edmonton                      │       365567 │
│ Regional Municipality of York         │       320664 │
│ Ville de Québec                       │       279793 │
│ Regional Municipality of Durham       │       263487 │
│ City of Mississauga                   │       225262 │
│ Ville de Montréal                     │       220311 │
│ City of Brampton                      │       215959 │
│ City of Hamilton                      │       200234 │
│ City of London                        │       191992 │
│ County of Simcoe                      │       191123 │
│ Niagara Region                        │       168182 │
│ Halifax Regional Municipality         │       163638 │
│ City of Vancouver                     │       153988 │
│ City of Surrey                        │       137180 │
│       ·                               │           ·  │
│       ·                               │           ·  │
│       ·                               │           ·  │
│ Regional District of Central Okanagan │         9248 │
│ City of Waterloo                      │         9017 │
│ District of Summerland                │         8899 │
│ Town of Orangeville                   │         8528 │
│ City of Welland                       │         8273 │
│ City of Oshawa                        │         7722 │
│ District of Squamish                  │         7031 │
│ Resort Municipality of Whistler       │         6500 │
│ Town of Canmore                       │         6365 │
│ Town of Truro                         │         5893 │
│ City of White Rock                    │         5347 │
│ City of Grand Forks                   │         3000 │
│ Town of Gibsons                       │         2581 │
│ Town of Banff                         │         2045 │
│ City of Pickering                     │         1416 │
│ City of Port Moody                    │         1304 │
│ Ville de Sherbrooke                   │          590 │
│ Ville de Shawinigan                   │          424 │
│ City of Markham                       │           81 │
│ City of Maple Ridge                   │           10 │
├───────────────────────────────────────┴──────────────┤
│ 107 rows (40 shown)                        2 columns │
└──────────────────────────────────────────────────────┘

Building Use

12.2M buildings don't have a classification for their usage. Oddly, Calgary seems to have good coverage of this so it might hit-or-miss based on municipality supplying the data.

SELECT   type,
         COUNT(*)
FROM     'ODB_v3_*.parquet'
GROUP BY 1
ORDER BY 2 DESC;

┌─────────────────────────────────────────┬──────────────┐
│                  type                   │ count_star() │
│                 varchar                 │    int64     │
├─────────────────────────────────────────┼──────────────┤
│ NULL                                    │     12286872 │
│ Residential                             │       616944 │
│ General                                 │       218243 │
│ Résidence                               │       142671 │
│ Single Family Dwelling                  │       134333 │
│ Garage - Annexe - Remise                │       127242 │
│ Residential Garage                      │       119061 │
│ Shed                                    │        75565 │
│ Residence                               │        75498 │
│ Résidentielle                           │        74947 │
│ Single / Semi / Duplex                  │        55742 │
│ Detached House                          │        45730 │
│ Single-Family Home                      │        42663 │
│ Detached                                │        36366 │
│ Résidentiel                             │        31142 │
│ Commercial                              │        27585 │
│ Résidentiel et commercial               │        24130 │
│ Garage                                  │        16461 │
│ General/Residential                     │        15758 │
│ Accessory                               │        15452 │
│     ·                                   │            · │
│     ·                                   │            · │
│     ·                                   │            · │
│ Arts Program Facility                   │            1 │
│ Ontario Provincial Police               │            1 │
│ Hospital//Hôpital                       │            1 │
│ Private High School                     │            1 │
│ Education / Fitness / Recreation        │            1 │
│ Auditorium / Concert Hall               │            1 │
│ Multi Recreation Facility               │            1 │
│ Performance Space - Outdoor Venue       │            1 │
│ Cafe / Bakery / Restaurant              │            1 │
│ Farmers Market                          │            1 │
│ Hanger                                  │            1 │
│ Grocery                                 │            1 │
│ Education / Museum / Historic Sites     │            1 │
│ Youth Services                          │            1 │
│ Golf Club                               │            1 │
│ ECOLE ACADIENNE                         │            1 │
│ Lighthouse property//Propriété du phare │            1 │
│ Public Secondary School                 │            1 │
│ LEASE                                   │            1 │
│ Bowling Alley                           │            1 │
├─────────────────────────────────────────┴──────────────┤
│ 968 rows (40 shown)                          2 columns │
└────────────────────────────────────────────────────────┘

Below is South Calgary.

Building Heights

13.2M buildings don't have any height data and there are a lot of values which seem implausible. There are other sources for building heights in Canada that, with some processing, could probably do a good job in this area.

SELECT   height,
         COUNT(*)
FROM     'ODB_v3_*.parquet'
GROUP BY 1
ORDER BY 1;

┌─────────────┬──────────────┐
│   height    │ count_star() │
│   varchar   │    int64     │
├─────────────┼──────────────┤
│ -0.10836829 │            1 │
│ -0.12246253 │            1 │
│ -0.16561381 │            1 │
│ -0.21919132 │            1 │
│ -0.377      │            2 │
│ -0.50739307 │            1 │
│ 0.00344336  │            1 │
│ 0.0106756   │            1 │
│ 0.01068541  │            1 │
│ 0.01268867  │            1 │
│ 0.01376321  │            1 │
│ 0.01511616  │            1 │
│ 0.016097    │            1 │
│ 0.02035425  │            1 │
│ 0.02102936  │            1 │
│ 0.02107398  │            1 │
│ 0.02117555  │            1 │
│ 0.02222852  │            1 │
│ 0.02618704  │            1 │
│ 0.02664106  │            1 │
│   ·         │            · │
│   ·         │            · │
│   ·         │            · │
│ 99.79       │            2 │
│ 99.8        │            6 │
│ 99.81       │            2 │
│ 99.82       │            1 │
│ 99.84       │            2 │
│ 99.85       │            1 │
│ 99.86       │            8 │
│ 99.87       │            2 │
│ 99.88       │            2 │
│ 99.9        │            9 │
│ 99.91       │            5 │
│ 99.92       │            5 │
│ 99.93       │            3 │
│ 99.94       │            4 │
│ 99.95       │            2 │
│ 99.96       │            3 │
│ 99.97       │            3 │
│ 99.98       │            5 │
│ 99.99       │            2 │
│ NULL        │     13204713 │
├─────────────┴──────────────┤
│   139513 rows (40 shown)   │
└────────────────────────────┘

Year of Construction

It would be nice to see integers in this field but I can understand parts of Canada were settled at a time when record-keeping wasn't amazing or good records might have been lost at some point.

SELECT   year_built,
         COUNT(*)
FROM     'ODB_v3_*.parquet'
GROUP BY 1
ORDER BY 1;

┌──────────────────┬──────────────┐
│    year_built    │ count_star() │
│     varchar      │    int64     │
├──────────────────┼──────────────┤
│ 1750             │            1 │
│ 1758             │            1 │
│ 1784             │            2 │
│ 1785             │            1 │
│ 1786             │            2 │
│ 1787             │            1 │
│ 1791             │            1 │
│ 1797             │            1 │
│ 1800             │           41 │
│ 1807             │            2 │
│ 1810             │            1 │
│ 1812             │            1 │
│ 1814             │            2 │
│ 1816             │            2 │
│ 1818             │            1 │
│ 1819             │            1 │
│ 1820             │            4 │
│ 1824             │            3 │
│ 1825             │            3 │
│ 1826             │            1 │
│  ·               │            · │
│  ·               │            · │
│  ·               │            · │
│ C 1880           │            2 │
│ C 1885           │            1 │
│ C 1890           │            2 │
│ C 1900           │            1 │
│ C 1910           │            4 │
│ C 1920           │            1 │
│ C 1930           │            1 │
│ C 1940           │            1 │
│ CIRCA 1870       │            1 │
│ CIRCA 1900       │            1 │
│ CIRCA 1930       │            1 │
│ Mid-19th century │            2 │
│ PRIOR 1956       │         5437 │
│ PRIOR 1962       │           31 │
│ PRIOR 1966       │            1 │
│ PRIOR 1969       │           20 │
│ PRIOR 1978       │            3 │
│ PRIOR 1989       │            6 │
│ c                │            1 │
│ NULL             │     14186018 │
├──────────────────┴──────────────┤
│ 247 rows (40 shown)   2 columns │
└─────────────────────────────────┘

Calgary's years of construction are entirely unknown.

SELECT   year_built,
         COUNT(*)
FROM     'ODB_v3_*.parquet'
WHERE    ST_X(ST_CENTROID(geometry)) BETWEEN -114.3461 AND -113.8326
AND      ST_Y(ST_CENTROID(geometry)) BETWEEN   50.8334 AND   51.2422
GROUP BY 1
ORDER BY 1;

┌────────────┬──────────────┐
│ year_built │ count_star() │
│  varchar   │    int64     │
├────────────┼──────────────┤
│ NULL       │    496793    │
└────────────┴──────────────┘

Building Floor Counts

382K buildings have a floor count.

SELECT   floors,
         COUNT(*)
FROM     'ODB_v3_*.parquet'
GROUP BY 1
ORDER BY 1;

┌─────────┬──────────────┐
│ floors  │ count_star() │
│ varchar │    int64     │
├─────────┼──────────────┤
│ 1.0     │       212594 │
│ 1.5     │          877 │
│ 10.0    │           44 │
│ 11.0    │           38 │
│ 12.0    │           47 │
│ 13.0    │           15 │
│ 14.0    │           24 │
│ 15.0    │           16 │
│ 16.0    │           13 │
│ 17.0    │           13 │
│ 18.0    │           15 │
│ 19.0    │           16 │
│ 2.0     │       161415 │
│ 20.0    │            8 │
│ 21.0    │            8 │
│ 22.0    │            5 │
│ 23.0    │            2 │
│ 24.0    │            3 │
│ 25.0    │            7 │
│ 26.0    │            3 │
│ 27.0    │            1 │
│ 28.0    │            2 │
│ 3.0     │         6402 │
│ 31.0    │            1 │
│ 32.0    │            1 │
│ 34.0    │            1 │
│ 39.0    │            1 │
│ 4.0     │          642 │
│ 45.0    │            1 │
│ 5.0     │          156 │
│ 6.0     │          240 │
│ 7.0     │           78 │
│ 8.0     │           79 │
│ 9.0     │           60 │
│ NULL    │     14034601 │
├─────────┴──────────────┤
│ 35 rows      2 columns │
└────────────────────────┘

Footprint Coverage

This dataset contains 14,417,429 buildings.

SELECT COUNT(*)
FROM   'ODB_v3_*.parquet';

14,417,429

I'll download OSM's buildings from September 23rd. This file was produced by the Layercake project.

$ wget -c https://data.openstreetmap.us/layercake/buildings.parquet

I'll download a rough outline of Canada's provinces in GeoJSON format. I'll then convert it into GPKG as GPKG files require less syntax to work with in DuckDB.

$ wget -c https://gist.github.com/Thiago4breu/6ba01976161aa0be65e0a289412dc54c/raw/8ec57d8317a2abe5bae18e5fd86f777fab649f84/canada-provinces.geojson
$ ogr2ogr \
    -f GPKG \
    canada-provinces.gpkg \
    canada-provinces.geojson

I'll turn the provinces dataset into a DuckDB table and count how many buildings OSM has that are covered by any of the provinces.

$ ~/duckdb

CREATE OR REPLACE TABLE canada AS
    FROM ST_READ('canada-provinces.gpkg');

SELECT    COUNT(*)
FROM      'buildings.parquet' b
LEFT JOIN canada c ON ST_CoveredBy(b.geometry, c.geom)
WHERE     c.name IS NOT NULL;

The above returned a count of 7,849,223 buildings.

The PSC dataset I reviewed the other week contains 13.7M buildings which is ~305K more than the TUM dataset I also reviewed a few weeks ago.

So far, ODB has the largest building count but I noticed Dease Lake, a remote community in Northern BC, is absent from this dataset. The community has been mapped out in OSM since at least February 2023.

It looks like no one has the perfect dataset. The highest coverage will come from mixing and matching these datasets.

Provincial Boundaries

I looked along Alberta's borders with BC and Saskatchewan and the provincial attributions to each of the buildings look very accurate.

The following settlement at -110 W, 50.956 N sits along the Alberta-Saskatchewan border. The buildings in Alberta are in red and the ones in Saskatchewan are yellow.

Analysis-Ready OpenStreetMap

2025-09-26T20:00:00+03:00

The Layercake project produces Parquet files of buildings, highways and settlements of OpenStreetMap (OSM) data. These are updated on a weekly basis. They're served from Cloudflare and I was able to max out my 560 Mbps line downloading them.

They also support bounding box queries. A typical ~500K-inhabitant city might only need a few seconds and MBs for its buildings and roads to download from Layercake's multi-GB datasets.

The scripts used to generate the Parquet files are open source and there is a call to have contributors put together more ETL jobs of other features types found on OSM.

Below is a heatmap of their settlements dataset.

In this post, I'll explore Layercake's initial three deliverables.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use Python 3.12.3, GDAL 3.9.3 and a few other tools to help analyse the data in this post.

$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable
$ sudo apt update
$ sudo apt install \
    gdal-bin \
    jq \
    python3-pip \
    python3.12-venv

I'll be using JSON Convert (jc) to convert the output of various CLI tools into JSON.

$ wget https://github.com/kellyjonbrazil/jc/releases/download/v1.25.2/jc_1.25.2-1_amd64.deb
$ sudo dpkg -i jc_1.25.2-1_amd64.deb

I'll use a Parquet debugging tool I've been working on to see how much space each column takes up in one of the Parquet files.

$ git clone https://github.com/marklit/pqview \
    ~/pqview

$ python3 -m venv ~/.pqview
$ source ~/.pqview/bin/activate
$ python3 -m pip install \
          -r ~/pqview/requirements.txt

I'll use DuckDB, along with its H3, JSON, Lindel, Parquet and Spatial extensions, in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.3.2/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

Buildings, Highways & Settlements

There are three datasets at the time of this writing.

The Buildings dataset is 39 GB and contains 657,586,639 rows. Below is an example record.

$ echo "SELECT * EXCLUDE(bbox, tags),
               bbox::JSON bbox,
               tags::JSON tags
        FROM READ_PARQUET('https://data.openstreetmap.us/layercake/buildings.parquet')
        LIMIT 1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "bbox": {
      "xmax": -179.96388244628906,
      "xmin": -179.96853637695312,
      "ymax": -77.95700073242188,
      "ymin": -77.95773315429688
    },
    "geometry": "MULTIPOLYGON (((-179.9685336 -77.9573186, -179.9669886 -77.9577349, -179.9646068 -77.9576499, -179.9638772 -77.9571306, -179.9663234 -77.9570007, -179.9685336 -77.9573186)))",
    "id": 1417351285,
    "tags": {
      "access": null,
      "addr:city": null,
      "addr:housenumber": null,
      "addr:postcode": null,
      "addr:street": null,
      "building": "yes",
      "building:colour": null,
      "building:flats": null,
      "building:levels": null,
      "building:material": null,
      "building:part": null,
      "building:use": null,
      "height": null,
      "name": "St. Paul's Burgers",
      "roof:colour": null,
      "roof:height": null,
      "roof:levels": null,
      "roof:material": null,
      "roof:orientation": null,
      "roof:shape": null,
      "start_date": null,
      "website": null,
      "wheelchair": null,
      "wikidata": null,
      "wikipedia": null
    },
    "type": "way"
  }
]

The Highways dataset is 25 GB and contains 281,406,480 rows. Below is an example record.

$ echo "SELECT * EXCLUDE(bbox, tags),
               bbox::JSON bbox,
               tags::JSON tags
        FROM READ_PARQUET('https://data.openstreetmap.us/layercake/highways.parquet')
        LIMIT 1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "bbox": {
      "xmax": -176.4818115234375,
      "xmin": -180.0,
      "ymax": -82.42407989501953,
      "ymin": -83.1589584350586
    },
    "geometry": "LINESTRING (-179.9999994 -82.42408, -179.4974012 -82.5336833, -178.994803 -82.6417098, -178.4922048 -82.7481817, -177.9896066 -82.8531212, -177.4870083 -82.9565499, -176.9844101 -83.0584892, -176.4818119 -83.1589603)",
    "id": 1159749847,
    "tags": {
      "access": null,
      "bicycle": null,
      "bridge": null,
      "bus": null,
      "construction": null,
      "covered": null,
      "crossing": null,
      "cycleway": null,
      "cycleway:left": null,
      "cycleway:right": null,
      "foot": null,
      "footway": null,
      "hgv": null,
      "highway": "secondary",
      "lanes": null,
      "layer": null,
      "lit": null,
      "maxspeed": null,
      "motor_vehicle": null,
      "motorcycle": null,
      "name": "South Pole Traverse",
      "oneway": null,
      "ref": null,
      "service": null,
      "sidewalk": "no",
      "smoothness": "very_bad",
      "surface": "ice",
      "toll": null,
      "tracktype": null,
      "tunnel": null,
      "wheelchair": null,
      "width": null
    },
    "type": "way"
  }
]

The Settlements dataset is 164 MB and contains 4,399,103 rows. Below is an example record.

$ echo "SELECT * EXCLUDE(bbox, tags),
               bbox::JSON bbox,
               tags::JSON tags
        FROM READ_PARQUET('https://data.openstreetmap.us/layercake/settlements.parquet')
        LIMIT 1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "bbox": {
      "xmax": -139.27279663085938,
      "xmin": -139.27279663085938,
      "ymax": -89.99800109863281,
      "ymin": -89.99800109863281
    },
    "geometry": "POINT (-139.2728 -89.998)",
    "id": 5314074411,
    "tags": {
      "alt_name": "Amundsen-Scott Base",
      "alt_names": {},
      "name": "Amundsen–Scott South Pole",
      "names": {
        "en": "Amundsen–Scott South Pole Station",
        "etymology:wikidata": "Q926;Q102527",
        "hu": "Amundsen-Scott kutatóállomás",
        "ko": "아문센-스콧 남극점 기지",
        "mk": "Станица Амундсен-Скот",
        "zh": "阿蒙森-斯科特南极站"
      },
      "official_name": "Amundsen–Scott South Pole Station",
      "official_names": {},
      "place": "village",
      "population": 49,
      "wikidata": "Q243307",
      "wikipedia": "en:Amundsen–Scott South Pole Station"
    },
    "type": "node"
  }
]

The following gets the publication date of the Buildings file. It was published on Tuesday the 23rd of September when I ran this.

$ curl -sI https://data.openstreetmap.us/layercake/buildings.parquet \
    | jc --kv \
    | jq -S .

{
  "HTTP/2 200": "",
  "accept-ranges": "bytes",
  "cf-cache-status": "DYNAMIC",
  "cf-ray": "98513d4c8e71542e-TLL",
  "content-length": "41545837871",
  "content-type": "application/octet-stream",
  "date": "Fri, 26 Sep 2025 08:02:30 GMT",
  "etag": "4f1bc544e027d4e5281072931b28a1f8-7925",
  "last-modified": "Tue, 23 Sep 2025 01:07:00 GMT",
  "nel": "{\"report_to\":\"cf-nel\",\"success_fraction\":0.0,\"max_age\":604800}",
  "report-to": "{\"group\":\"cf-nel\",\"max_age\":604800,\"endpoints\":[{\"url\":\"https://a.nel.cloudflare.com/report/v4?s=QfwN2UyRDYsN%2B%2BN4SEPc7ppTRIoxP0HUIEn%2FPuPw2WPvkTMBb4p0ydtbf9K4xmxfxadRO%2BR2SnDGAybZWUvJhAAsOh2RhkcKLiqGfStxxW3AlOs%3D\"}]}",
  "server": "cloudflare",
  "vary": "Accept-Encoding"
}

Downloading Buildings

Click the "Web" Menu, then "QuickMapServices" and then the "Settings" item. Click the "More Services" tab at the top of the dialog. Click the "Get contributed pack" button.

In the "Web" menu under "QuickMapServices" you should now see the list of several basemap providers.

Select "OSM" and then "OSM Standard" to add a world map to your scene.

Under the "Plugins" menu, select "Python Console". Paste in the following line of Python. It'll ensure you've got the latest version of DuckDB installed in QGIS' Python Environment.

import pip; pip.main(['install', '--upgrade', 'duckdb'])

If you can see the whole earth, GeoParquet Downloader will end up downloading 39 GB of data. Downloading only a city's worth of data will likely only need a few MB of data.

In your toolbar, click on the GeoParquet Downloader icon. It's the one with blue, rotated rectangles.

Click the "Custom URL" option and paste the following URL in.

https://data.openstreetmap.us/layercake/buildings.parquet

Hit "OK". You'll be prompted to save a Parquet file onto your computer. Once you've done so, the GBA data for the area in your map should appear shortly afterword.

Once the buildings have loaded, select the layer styling for the downloaded buildings layer. The combo box at the top of the styling panel should be switched from "Single Symbol" to "2.5D".

I've set the roof colour to #b13327, wall colour to #4f090b, angle to 230 degrees and added a drop shadow effect to the layer.

In the height field, change it to the following expression:

min("tags.height", 1) / 30000

You should now see a 2.5D representation of the buildings you downloaded.

There is an "Identify Features" icon in the toolbar.

This tool lets you select any building and bring up its metadata.

The Parquet file you saved can also be read via DuckDB and even dropped into Esri's ArcGIS Pro 3.5 or newer if you have it installed on your machine.

Bounding Boxes

Each of Layercake's deliverables is spatially sorted and in columns, rather than rows. These columns provide easier-to-spot patterns for ZStandard when it compresses the underlying data.

There are also row-groups where there are minimum and maximum values cached for each column every 122K rows in the Buildings files.

There are bounding box columns around each piece of geometry. These are much smaller than the geometry itself. Since they are spatially sorted, the row-group statistics helps tools like QGIS' GeoParquet Plugin and DuckDB skip over the parts of the 39 GB file that aren't relevant to any given query.

Below is a pie chart showing how much space each column takes up in the Buildings file. The bbox fields are much smaller than the geometry field.

$ python3 ~/pqview/main.py \
    types --html \
    buildings.parquet \
    > buildings.html

The DuckDB and the GeoParquet Downloader Plugin for QGIS can use these bounding boxes to download data quicker than it would if it had used the geometry field alone.

The following counted the number of instances of each value in the type field in the 39 GB Buildings Parquet file and only needed 2.2 MB of bandwidth.

$ ~/duckdb

EXPLAIN ANALYZE
    SELECT   COUNT(*),
             type
    FROM     READ_PARQUET('https://data.openstreetmap.us/layercake/buildings.parquet')
    GROUP BY 2
    ORDER BY 1 DESC;

┌─────────────────────────────────────┐
│┌───────────────────────────────────┐│
││    Query Profiling Information    ││
│└───────────────────────────────────┘│
└─────────────────────────────────────┘
EXPLAIN ANALYZE     SELECT   COUNT(*),              type     FROM     READ_PARQUET('https://data.openstreetmap.us/layercake/buildings.parquet')     GROUP BY 2     ORDER BY 1 DESC;
┌─────────────────────────────────────┐
│┌───────────────────────────────────┐│
││         HTTPFS HTTP Stats         ││
││                                   ││
││            in: 2.2 MiB            ││
││            out: 0 bytes           ││
││              #HEAD: 1             ││
││             #GET: 5372            ││
││              #PUT: 0              ││
││              #POST: 0             ││
││             #DELETE: 0            ││
│└───────────────────────────────────┘│
└─────────────────────────────────────┘
┌────────────────────────────────────────────────┐
│┌──────────────────────────────────────────────┐│
││              Total Time: 194.39s             ││
│└──────────────────────────────────────────────┘│
└────────────────────────────────────────────────┘
┌───────────────────────────┐
│           QUERY           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│      EXPLAIN_ANALYZE      │
│    ────────────────────   │
│           0 Rows          │
│          (0.00s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│          ORDER_BY         │
│    ────────────────────   │
│     count_star() DESC     │
│                           │
│           2 Rows          │
│          (0.00s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         PROJECTION        │
│    ────────────────────   │
│        count_star()       │
│             1             │
│                           │
│           2 Rows          │
│          (0.00s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│       HASH_GROUP_BY       │
│    ────────────────────   │
│         Groups: #0        │
│                           │
│        Aggregates:        │
│        count_star()       │
│                           │
│           2 Rows          │
│          (0.88s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         PROJECTION        │
│    ────────────────────   │
│            type           │
│                           │
│       657586639 Rows      │
│          (0.04s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         TABLE_SCAN        │
│    ────────────────────   │
│         Function:         │
│        READ_PARQUET       │
│                           │
│     Projections: type     │
│    Total Files Read: 1    │
│                           │
│       657586639 Rows      │
│         (4502.25s)        │
└───────────────────────────┘

Only needing to look at columns of interest, rather than rows, along with being able to use statistics in some cases, rather than underlying data has a huge impact on bandwidth usage and query times.

Clustering Values

Parquet files are well-optimised to examine value distributions in individual columns. Below you can see how diverse the value distributions are in each of the columns for the building dataset.

I need to look into techniques around clustering these values and ideally turning them in numeric rather than free-text form where possible.

$ wget -c https://data.openstreetmap.us/layercake/buildings.parquet
$ ~/duckdb

SELECT   column_name,
         column_type,
         null_percentage,
         approx_unique,
         min,
         max
FROM     (SUMMARIZE
          SELECT * EXCLUDE(geometry, bbox, tags),
                 tags.*
          FROM   'buildings.parquet')
ORDER BY 1;

┌───────────────────┬─────────────┬─────────────────┬───────────────┬─────────────────────────────────────┬──────────────────────────────────────────┐
│    column_name    │ column_type │ null_percentage │ approx_unique │                 min                 │                   max                    │
│      varchar      │   varchar   │  decimal(9,2)   │     int64     │               varchar               │                 varchar                  │
├───────────────────┼─────────────┼─────────────────┼───────────────┼─────────────────────────────────────┼──────────────────────────────────────────┤
│ access            │ VARCHAR     │           99.98 │            64 │ agricultural                        │ 関係者のみｑ                             │
│ addr:city         │ VARCHAR     │           90.41 │        214287 │                                     │ ｈ                                       │
│ addr:housenumber  │ VARCHAR     │           87.26 │        903255 │                                     │ ﾌ９８－１                                │
│ addr:postcode     │ VARCHAR     │           91.48 │        874940 │  040315                             │ ９９８−００５４                          │
│ addr:street       │ VARCHAR     │           87.51 │       2009837 │                                     │ 𫙮魚坑路                                 │
│ building          │ VARCHAR     │            0.00 │          8183 │ #HKUSTChangeMaker2025               │ 홈플러스_천안점                          │
│ building:colour   │ VARCHAR     │           99.85 │         73829 │ #                                   │ 어두운회색                               │
│ building:flats    │ VARCHAR     │           99.64 │          2144 │ #OFC;2;3;4                          │ ８                                       │
│ building:levels   │ VARCHAR     │           94.76 │          1956 │                                     │ ９                                       │
│ building:material │ VARCHAR     │           99.69 │          2577 │ "cement_block"                      │ 鐵皮屋                                   │
│ building:part     │ VARCHAR     │           99.99 │           196 │ *                                   │ 金球                                     │
│ building:use      │ VARCHAR     │           99.89 │           960 │ 0                                   │ বাসা এবং অফিস                                │
│ height            │ VARCHAR     │           97.09 │        142743 │ "                                   │ ８                                       │
│ id                │ BIGINT      │            0.00 │     605442738 │ 114                                 │ 1445313890                               │
│ name              │ VARCHAR     │           98.80 │       4616374 │  Lucky Plaza                        │ 𫙮魚隧道西側機房                         │
│ roof:colour       │ VARCHAR     │           99.72 │        127591 │  #B62121                            │ ｖ                                       │
│ roof:height       │ VARCHAR     │           99.95 │          1449 │ #6D7367                             │ ß.3                                      │
│ roof:levels       │ VARCHAR     │           99.31 │           247 │ !                                   │ ß.5                                      │
│ roof:material     │ VARCHAR     │           99.72 │          1016 │ "Deutsches Haus August Fiege"       │ ഓട്                                       │
│ roof:orientation  │ VARCHAR     │           99.85 │           227 │ #333333                             │ west                                     │
│ roof:shape        │ VARCHAR     │           98.78 │           757 │ "Berliner Dach"                     │ ｈ                                       │
│ start_date        │ VARCHAR     │           97.56 │         34325 │ "first recorded 1569"               │ 高雄市                                   │
│ type              │ VARCHAR     │            0.00 │             2 │ relation                            │ way                                      │
│ website           │ VARCHAR     │           99.86 │        756480 │ 'https:/msc-altenbach.de            │ 홈페이지 http://www.aritaum.com/         │
│ wheelchair        │ VARCHAR     │           99.94 │            66 │ 0                                   │ 海老山南第一公園公衆トイレ               │
│ wikidata          │ VARCHAR     │           99.91 │        657923 │ * (preencher se houver ID Wikidata) │ https://www.wikidata.org/wiki/Q136376633 │
│ wikipedia         │ VARCHAR     │           99.97 │        202600 │ 4735910                             │ 打铁关站                                 │
├───────────────────┴─────────────┴─────────────────┴───────────────┴─────────────────────────────────────┴──────────────────────────────────────────┤
│ 27 rows                                                                                                                                  6 columns │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Canada's 13M Buildings

2025-09-24T20:00:00+03:00

On September 12th, Public Safety Canada (PSC) released Canada Structures. This dataset contains 13,760,754 Canadian building footprints; many with heights and metadata describing their usage.

They sourced data from OpenStreetMap (OSM), Microsoft Building Footprints (MSB), Open Data Base of Buildings (ODB) and height data was collected from Natural Resources Canada.

The dataset was assembled in April 2024. The ODB data, which is a collection of data from municipalities of Canada, was from 2019, not the newer 2025 release.

I'm not certain when they collected their data from OSM but I found records from February 2023 that didn't make the cut into this release.

The MSB data was likely version 1.1 which was published in June of 2019.

Below is a heatmap of the building footprints.

In this post, I'll explore Canada Structures' version 1.0.0.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use GDAL 3.9.3 and a few other tools to help analyse the data in this post.

$ sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable
$ sudo apt update
$ sudo apt install \
    gdal-bin \
    jq

I'll use DuckDB, along with its H3, JSON, Lindel, Parquet and Spatial extensions, in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.3.2/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

I used QGIS' Tile+ plugin to add basemaps from Google and OpenStreetMap (OSM) to the maps throughout this post.

Analysis-Ready Data

PSC broke up the dataset into GeoPackage (GPKG) files by province / territory. Below I'll build a manifest of these URLs and download them with four concurrent threads.

$ mkdir -p ~/psc
$ cd ~/psc

$ vi urls.txt

https://open.canada.ca/data/dataset/3829eee9-f898-4643-9ad8-f48575b8873d/resource/909a44d6-1040-408d-896e-7362d2165688/download/ab_structures_en.gpkg
https://open.canada.ca/data/dataset/3829eee9-f898-4643-9ad8-f48575b8873d/resource/d858fd57-2c9a-4978-bc50-79aef286bccf/download/mb_structures_en.gpkg
https://open.canada.ca/data/dataset/3829eee9-f898-4643-9ad8-f48575b8873d/resource/f7fdee7d-09dd-4eb6-88fd-2234ad701309/download/on_structures_en.gpkg
https://open.canada.ca/data/dataset/3829eee9-f898-4643-9ad8-f48575b8873d/resource/71db1a0f-0099-4fbe-ad9a-f51626506e8a/download/qc_structures_en.gpkg
https://open.canada.ca/data/dataset/3829eee9-f898-4643-9ad8-f48575b8873d/resource/5490a8a0-753a-41f7-8ab5-d3870ee0ea69/download/bc_structures_en.gpkg
https://open.canada.ca/data/dataset/3829eee9-f898-4643-9ad8-f48575b8873d/resource/b903c565-8fdc-453e-9927-8b1a33829d07/download/ns_structures_en.gpkg
https://open.canada.ca/data/dataset/3829eee9-f898-4643-9ad8-f48575b8873d/resource/f4a95ecb-5b58-4021-a591-ca1d8b2bbae4/download/sk_structures_en.gpkg
https://open.canada.ca/data/dataset/3829eee9-f898-4643-9ad8-f48575b8873d/resource/610fb5b5-e12c-41a1-aa28-415b51c24333/download/nt_structures_en.gpkg
https://open.canada.ca/data/dataset/3829eee9-f898-4643-9ad8-f48575b8873d/resource/d892ce68-bcd8-463b-a2fa-0439ef75c2b8/download/pe_structures_en.gpkg
https://open.canada.ca/data/dataset/3829eee9-f898-4643-9ad8-f48575b8873d/resource/680f47b9-9e23-4888-9950-10e561ae0556/download/nl_structures_en.gpkg
https://open.canada.ca/data/dataset/3829eee9-f898-4643-9ad8-f48575b8873d/resource/fcd7fe5b-be3e-4499-8b6e-11ce5ca391cc/download/nu_structures_en.gpkg
https://open.canada.ca/data/dataset/3829eee9-f898-4643-9ad8-f48575b8873d/resource/15737be6-24f3-4764-b40c-6f23f8dd44e0/download/yk_structures_en.gpkg
https://open.canada.ca/data/dataset/3829eee9-f898-4643-9ad8-f48575b8873d/resource/6ff6252a-50f4-43b6-b6d0-7d6d904a3e68/download/nb_structures_en.gpkg

$ cat urls.txt \
    | xargs -n1 \
            -P4 \
            -I% \
            wget -c "%"

I'll extract the projection PSC used. This proj4 string will be used to below to re-project the data into EPSG:4326.

$ gdalsrsinfo \
    -o proj4 \
    ab_structures_en.gpkg

+proj=lcc +lat_0=63.390675 +lon_0=-91.8666666666667 +lat_1=49 +lat_2=77 +x_0=6200000 +y_0=3000000 +ellps=GRS80 +units=m +no_defs

This format will load without issue in QGIS 3.44 and ArcGIS Pro 3.5. The bounding boxes will help optimise bandwidth usage when querying this data on remote servers, like AWS S3.

$ for FILENAME in *.gpkg; do
     echo $FILENAME

     BASENAME=`basename $FILENAME | cut -d. -f1`

     echo "COPY (
               WITH a AS (
                   SELECT * EXCLUDE(geom),
                          ST_FLIPCOORDINATES(
                              ST_TRANSFORM(
                                  geom,
                                  '+proj=lcc +lat_0=63.390675  +lon_0=-91.8666666666667 +lat_1=49  +lat_2=77 +x_0=6200000 +y_0=3000000  +ellps=GRS80 +units=m +no_defs  +type=crs',
                                  'EPSG:4326')) geometry
                   FROM ST_READ('$FILENAME')
               )
               SELECT   *,
                        {'xmin': ST_XMIN(ST_EXTENT(geometry)),
                         'ymin': ST_YMIN(ST_EXTENT(geometry)),
                         'xmax': ST_XMAX(ST_EXTENT(geometry)),
                         'ymax': ST_YMAX(ST_EXTENT(geometry))} AS bbox,
                         CASE WHEN MSB IS NOT NULL THEN 'MSB'
                              WHEN ODB IS NOT NULL THEN 'ODB'
                              WHEN OSM IS NOT NULL THEN 'OSM'
                         END AS source
               FROM     a
               ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geometry)),
                                        ST_X(ST_CENTROID(geometry))]::double[2])
           ) TO '$BASENAME.parquet' (
               FORMAT            'PARQUET',
               CODEC             'ZSTD',
               COMPRESSION_LEVEL 22,
               ROW_GROUP_SIZE    15000);
           " | ~/duckdb
 done

The above turned 3.9 GB of GPKG files into 1.5 GB of Parquet.

$ du -hsc *structures_en.parquet

197M    ab_structures_en.parquet
169M    bc_structures_en.parquet
69M     mb_structures_en.parquet
40M     nb_structures_en.parquet
26M     nl_structures_en.parquet
51M     ns_structures_en.parquet
2.9M    nt_structures_en.parquet
1.1M    nu_structures_en.parquet
536M    on_structures_en.parquet
7.6M    pe_structures_en.parquet
276M    qc_structures_en.parquet
71M     sk_structures_en.parquet
1.6M    yk_structures_en.parquet
1.5G    total

Heatmap

Below is a heatmap of this dataset.

CREATE OR REPLACE TABLE h3_4_stats AS
    SELECT   H3_LATLNG_TO_CELL(
                bbox.ymin,
                bbox.xmin, 4) AS h3_4,
             COUNT(*) num_buildings
    FROM     READ_PARQUET('*_structures_en.parquet')
    WHERE    bbox.xmin BETWEEN -178.5 AND 178.5
    GROUP BY 1;

COPY (
    SELECT ST_ASWKB(H3_CELL_TO_BOUNDARY_WKT(h3_4)::geometry) geometry,
           num_buildings
    FROM   h3_4_stats
) TO 'h3_4_stats.gpkg'
  WITH (FORMAT GDAL,
        DRIVER 'GPKG',
        LAYER_CREATION_OPTIONS 'WRITE_BBOX=YES');

The following was needed for ArcGIS Pro to recognise the projection of the above hexagons properly.

$ ogr2ogr \
    -f GPKG \
    -a_srs EPSG:4326 \
    h3_4_stats.4326.gpkg \
    h3_4_stats.gpkg

Data Fluency

Below are the field names, data types, percentages of NULLs per column, number of unique values and minimum and maximum values for each column.

$ ~/duckdb

SELECT   column_name,
         column_type,
         null_percentage,
         approx_unique,
         min,
         max
FROM     (SUMMARIZE
          FROM READ_PARQUET('*_structures_en.parquet'))
WHERE    column_name != 'geometry'
AND      column_name != 'bbox'
ORDER BY 1;

┌─────────────┬─────────────┬─────────────────┬───────────────┬───────────────────┬────────────────────┐
│ column_name │ column_type │ null_percentage │ approx_unique │        min        │        max         │
│   varchar   │   varchar   │  decimal(9,2)   │     int64     │      varchar      │      varchar       │
├─────────────┼─────────────┼─────────────────┼───────────────┼───────────────────┼────────────────────┤
│ Area        │ DOUBLE      │            0.00 │      14644202 │ 0.00022589575436  │ 2788966.1142695914 │
│ CS_ID       │ BIGINT      │            0.00 │      12278694 │ 1                 │ 13763995           │
│ Height      │ DOUBLE      │           13.72 │       8158934 │ 0.0               │ 270.43516061452516 │
│ LC_Name     │ VARCHAR     │           96.04 │         19469 │ "Sifton Cemetery" │ Ïles aux Chats     │
│ MSB         │ VARCHAR     │           50.69 │             1 │ 1                 │ 1                  │
│ ODB         │ VARCHAR     │           67.74 │             1 │ 1                 │ 1                  │
│ ODB_ID      │ VARCHAR     │           67.74 │       4186001 │ 12010010000001    │ 61060970000014     │
│ ODB_Source  │ VARCHAR     │           67.74 │            66 │ Airdrie           │ York Region        │
│ OSM         │ VARCHAR     │           56.89 │             1 │ 1                 │ 1                  │
│ OSM_ID      │ VARCHAR     │           56.89 │       6684280 │ 1000016583        │ j                  │
│ OSM_LC      │ VARCHAR     │           32.15 │            16 │ allotments        │ vineyard           │
│ OSM_Name    │ VARCHAR     │           99.10 │         95599 │ "8-Building"      │ étable             │
│ OSM_Type    │ VARCHAR     │           82.44 │           358 │ Administratif     │ yurt               │
│ Perimeter   │ DOUBLE      │            0.00 │      13164420 │ 0.107992344195257 │ 16624.67147343245  │
│ Province    │ VARCHAR     │            0.00 │            12 │ AB                │ YK                 │
│ source      │ VARCHAR     │            0.00 │             3 │ MSB               │ OSM                │
├─────────────┴─────────────┴─────────────────┴───────────────┴───────────────────┴────────────────────┤
│ 16 rows                                                                                    6 columns │
└──────────────────────────────────────────────────────────────────────────────────────────────────────┘

Sources

This is the breakdown of records per source per province. There is a clear preference for MSB with ODB often completely absent.

$ ~/duckdb

WITH a AS (
    SELECT   Province,
             source,
             COUNT(*) cnt
    FROM     '*_structures_en.parquet'
    GROUP BY 1, 2)
PIVOT    a
ON       source
USING    SUM(cnt)
GROUP BY Province
ORDER BY 2 DESC;

┌──────────┬─────────┬─────────┬────────┐
│ Province │   MSB   │   ODB   │  OSM   │
│ varchar  │ int128  │ int128  │ int128 │
├──────────┼─────────┼─────────┼────────┤
│ QC       │ 1798578 │  491156 │ 504603 │
│ ON       │ 1333101 │ 2465651 │ 934046 │
│ AB       │ 1082368 │  495617 │ 332508 │
│ BC       │  688603 │  556590 │ 290223 │
│ MB       │  638831 │    NULL │  82979 │
│ SK       │  509567 │   98226 │ 130160 │
│ NL       │  255350 │    NULL │  32917 │
│ NB       │  216369 │   95832 │  91412 │
│ NS       │  168198 │  230801 │ 108674 │
│ PE       │   77119 │    NULL │   4567 │
│ YK       │   11563 │    NULL │   4461 │
│ NT       │    3192 │    5835 │  10589 │
│ NU       │    2882 │    NULL │   8186 │
├──────────┴─────────┴─────────┴────────┤
│ 13 rows                     4 columns │
└───────────────────────────────────────┘

The following is an example record which has been attributed to OSM.

$ echo "FROM  'ab_structures_en.parquet'
        WHERE source = 'OSM'
        LIMIT 1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "Area": 1951.60849850855,
    "CS_ID": 7889231,
    "Height": 0.0,
    "LC_Name": null,
    "MSB": null,
    "ODB": null,
    "ODB_ID": null,
    "ODB_Source": null,
    "OSM": "1",
    "OSM_ID": "385156020",
    "OSM_LC": "forest",
    "OSM_Name": null,
    "OSM_Type": "industrial",
    "Perimeter": 283.5263470280223,
    "Province": "AB",
    "bbox": "{'xmin': -110.42687370425128, 'ymin': 55.06546590051143, 'xmax': -110.4250928025303, 'ymax': 55.0657240008682}",
    "geometry": "POLYGON ((-110.42687370425128 55.06546590051143, -110.42687370423948 55.0657240008682, -110.4250928025303 55.065687100742686, -110.42509280279089 55.06555190107596, -110.42655190404591 55.06558880094082, -110.42655190337649 55.06546590070232, -110.42687370425128 55.06546590051143))",
    "source": "OSM"
  }
]

As the following shows, records which are not attributed to OSM can still contain OSM metadata.

$ echo "FROM  'ab_structures_en.parquet'
        WHERE source = 'MSB'
        LIMIT 1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "Area": 197.982910306995,
    "CS_ID": 8653450,
    "Height": 0.0,
    "LC_Name": "Cold Lake Air Weapons Range",
    "MSB": "1",
    "ODB": null,
    "ODB_ID": null,
    "ODB_Source": null,
    "OSM": null,
    "OSM_ID": null,
    "OSM_LC": "military",
    "OSM_Name": null,
    "OSM_Type": null,
    "Perimeter": 64.80059892006688,
    "Province": "AB",
    "bbox": "{'xmin': -110.59742700235742, 'ymin': 55.01551100079211, 'xmax': -110.59729400280793, 'ymax': 55.01572700092107}",
    "geometry": "POLYGON ((-110.59742700235742 55.01551100079211, -110.59742300421074 55.01572700092107, -110.59729400280793 55.015726000806524, -110.59729800362015 55.0155110012002, -110.59742700235742 55.01551100079211))",
    "source": "MSB"
  }
]

When ODB is used as a soruce, the ODB_Source column names the municipality the data came from. I'm a bit surprised to only see 64 unique values here.

.maxrows 20

SELECT   COUNT(*),
         ODB_Source
FROM     '*_structures_en.parquet'
WHERE    source  = 'ODB'
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬─────────────────┐
│ count_star() │   ODB_Source    │
│    int64     │     varchar     │
├──────────────┼─────────────────┤
│       407142 │ Toronto         │
│       321334 │ Edmonton        │
│       320418 │ Ottawa          │
│       299972 │ York Region     │
│       253642 │ Durham          │
│       231396 │ Quebec          │
│       191918 │ Hamilton        │
│       144092 │ Halifax         │
│       133103 │ Surrey          │
│       130356 │ Brampton        │
│          ·   │    ·            │
│          ·   │    ·            │
│          ·   │    ·            │
│        10807 │ Cochrane        │
│         9027 │ New Westminster │
│         6839 │ Chestermere     │
│         6638 │ Squamish        │
│         5835 │ Yellowknife     │
│         5556 │ Canmore         │
│         5289 │ Whistler        │
│         5117 │ White Rock      │
│         1910 │ Banff           │
│          409 │ Sherbrooke      │
├──────────────┴─────────────────┤
│ 64 rows (20 shown)   2 columns │
└────────────────────────────────┘

Building Use

There are two fields, OSM_LC and OSM_Type, which can describe a building's use and surrounding area. Below are the record counts for the 21 unique values for OSM_LC. It's good to see most records are non-NULL.

SELECT   COUNT(*),
         OSM_LC
FROM     '*_structures_en.parquet'
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬───────────────────┐
│ count_star() │      OSM_LC       │
│    int64     │      varchar      │
├──────────────┼───────────────────┤
│      7907072 │ residential       │
│      4423474 │ NULL              │
│       917211 │ forest            │
│       177327 │ industrial        │
│        94802 │ retail            │
│        79847 │ commercial        │
│        49464 │ farmyard          │
│        29279 │ park              │
│        27556 │ farmland          │
│        24495 │ nature_reserve    │
│         9042 │ military          │
│         4942 │ quarry            │
│         4847 │ grass             │
│         2934 │ cemetery          │
│         2921 │ recreation_ground │
│         2249 │ meadow            │
│         1787 │ scrub             │
│         1222 │ orchard           │
│          131 │ vineyard          │
│          108 │ allotments        │
│           44 │ heath             │
├──────────────┴───────────────────┤
│ 21 rows                2 columns │
└──────────────────────────────────┘

The OSM_Type has much more of a long tail and could probably do with some clustering. Also, out of the ~13M records in this dataset, this column is empty for 11.3M of them. There are also some values here, like industrial, which overlap with OSM_LC.

SELECT   COUNT(*),
         OSM_Type
FROM     '*_structures_en.parquet'
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬──────────────────────┐
│ count_star() │       OSM_Type       │
│    int64     │       varchar        │
├──────────────┼──────────────────────┤
│     11344842 │ NULL                 │
│       864092 │ detached             │
│       766544 │ house                │
│       140230 │ residential          │
│       122762 │ garage               │
│       112894 │ terrace              │
│        67584 │ shed                 │
│        63904 │ apartments           │
│        41522 │ commercial           │
│        35143 │ industrial           │
│        32287 │ semidetached_house   │
│        30928 │ retail               │
│        25959 │ semi                 │
│        17495 │ farm_auxiliary       │
│        14288 │ static_caravan       │
│        14078 │ school               │
│         9024 │ roof                 │
│         8401 │ barn                 │
│         5980 │ cabin                │
│         5887 │ church               │
│            · │   ·                  │
│            · │   ·                  │
│            · │   ·                  │
│            1 │ residential;yes      │
│            1 │ air-supported        │
│            1 │ Control Tower        │
│            1 │ Banquet_Hall         │
│            1 │ hydro                │
│            1 │ outside              │
│            1 │ G                    │
│            1 │ changing_rooms       │
│            1 │ walkway              │
│            1 │ sse                  │
│            1 │ tourism              │
│            1 │ balcony              │
│            1 │ ramada_studio        │
│            1 │ military_armoury     │
│            1 │ Sports,_exercise_are │
│            1 │ Résidence_Maria-Gor  │
│            1 │ Hospital and Nursing │
│            1 │ recreational         │
│            1 │ social_facility      │
│            1 │ school;church        │
├──────────────┴──────────────────────┤
│ 401 rows (40 shown)       2 columns │
└─────────────────────────────────────┘

Below is the number of records with the above two fields populated broken down by source.

WITH a AS (
    SELECT   COUNT(*) cnt,
             source,
             OSM_LC   IS NOT NULL AS has_lc,
             OSM_Type IS NOT NULL AS has_type
    FROM     '*_structures_en.parquet'
    GROUP BY 2, 3, 4
)
PIVOT    a
ON       has_lc, has_type
USING    SUM(cnt)
GROUP BY source
ORDER BY 1;

┌─────────┬─────────────┬────────────┬────────────┬───────────┐
│ source  │ false_false │ false_true │ true_false │ true_true │
│ varchar │   int128    │   int128   │   int128   │  int128   │
├─────────┼─────────────┼────────────┼────────────┼───────────┤
│ MSB     │     3014756 │      22326 │    3676601 │     72038 │
│ ODB     │      552400 │     158812 │    2518475 │   1210021 │
│ OSM     │      490810 │     184370 │    1091800 │    768345 │
└─────────┴─────────────┴────────────┴────────────┴───────────┘

Building Heights

Not all buildings have heights. Below is the number of buildings that do broken down by source.

WITH a AS (
    SELECT   COUNT(*) cnt,
             source,
             Height > 0 AS has_height
    FROM     '*_structures_en.parquet'
    GROUP BY 2, 3
)
PIVOT    a
ON       has_height
USING    SUM(cnt)
GROUP BY source
ORDER BY 1;

┌─────────┬─────────┬─────────┐
│ source  │  false  │  true   │
│ varchar │ int128  │ int128  │
├─────────┼─────────┼─────────┤
│ MSB     │ 2078923 │ 3488475 │
│ ODB     │  145537 │ 4019747 │
│ OSM     │  491994 │ 1648099 │
└─────────┴─────────┴─────────┘

Building heights can look good at a distance. Below is Downtown Calgary.

But in neighbourhoods with uniform housing, the heights can vary wildly. Below is Douglasdale in Calgary. The labels are each home's height, rounded to the nearest tenth.

Footprint Coverage

Buildings constructed after 2023 (or possibly even earlier) won't appear in this dataset. The image below is from South East Calgary. In yellow is the PSC's dataset with Overture's August release in purple.

This is one of the purple areas on June 25th.

The TUM dataset I reviewed recently contains 305,208 fewer Canadian buildings than PSC's dataset. I couldn't find anywhere in Calgary where TUM data was the only footprint present.

Some properties that are detached from one another are showing as being attached in this dataset. PSC's buildings are outlined in yellow, Overture in purple and TUM in red.

This is the same area on Google Streetview. As you can see, the houses are detached from one another.

MSB footprints can take on a life of their own. Below is a screen shot from Northern Alberta.

These are the sources of each footprint in PSC's dataset. It is a bit of a shame OSM wasn't picked as a first choice with ODB filling in the gaps.

I noticed Dease Lake, a remote community in Northern BC, has many of its homes, which have been on OSM since at least February 2023, missing.

Below are PSC's footprints in yellow on top of OSM's vector tiles for the area.

The following is one such example missing from PSC's dataset.

Provincial Boundaries

I looked along Alberta's borders with BC and SK and the provincial attributions to each of the buildings look very accurate.

The World's 2.75B Buildings

2025-09-06T20:00:00+03:00

The United Nations (UN) believes there are 4B buildings on Earth. This week, a dataset called "GlobalBuildingAtlas" (GBA) was published by researchers at the Technical University of Munich (TUM) that attempts to estimate this number at being closer to 2.75B.

GBA is broken up into two datasets. The first is a 1.1 TB, level-of-detail 1 (LoD1) dataset made up of 922 uncompressed GeoJSON files. I spent the last few days downloading these and converting them into 210 GB of Parquet. These are now being hosted on AWS S3 thanks to the kind generosity of Source Cooperative and Taylor Geospatial Engine.

Below is a heatmap of this dataset.

The above basemap is mostly made up of vector data from Natural Earth and Overture.

The second "Height" dataset is made up of 35 TB of raster height maps. Given the LoD1 files already contain building heights and downloading 35 TB off of TUM's servers will likely take months, I'll just show an individual raster in this post rather than run any dataset-wide operations on this dataset.

Parts of these datasets were developed by running open source (code, not imagery or weights) deep learning models on Planet Labs' satellite imagery. Planet Labs operate several constellations totalling hundreds of satellites in low earth orbit. They capture images of the entire Earth's landmasses every day.

In this post, I'll walk through loading the LoD1 data from S3 into QGIS, describe the ETL process I ran to build the Parquet version of this dataset and discuss some observations of the footprints themselves.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use Python 3.12.3 and a few other tools to help analyse the data in this post.

$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt update
$ sudo apt install \
    exiftool \
    jq \
    python3-pip \
    python3.12-venv

I'll be using JSON Convert (jc) to convert the output of various CLI tools into JSON.

$ wget https://github.com/kellyjonbrazil/jc/releases/download/v1.25.2/jc_1.25.2-1_amd64.deb
$ sudo dpkg -i jc_1.25.2-1_amd64.deb

I'll use a Parquet debugging tool I've been working on to see how much space each column takes up in one of the Parquet files.

$ git clone https://github.com/marklit/pqview \
    ~/pqview

$ python3 -m venv ~/.pqview
$ source ~/.pqview/bin/activate
$ python3 -m pip install \
          -r ~/pqview/requirements.txt

I'll use DuckDB, along with its H3, JSON, Lindel, Parquet and Spatial extensions, in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.3.2/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

Downloading the Buildings

Click the "Web" Menu, then "QuickMapServices" and then the "Settings" item. Click the "More Services" tab at the top of the dialog. Click the "Get contributed pack" button.

In the "Web" menu under "QuickMapServices" you should now see the list of several basemap providers.

Select "OSM" and then "OSM Standard" to add a world map to your scene.

Under the "Plugins" menu, select "Python Console". Paste in the following line of Python. It'll ensure you've got the latest version of DuckDB installed in QGIS' Python Environment.

import pip; pip.main(['install', '--upgrade', 'duckdb'])

If you can see the whole earth, GeoParquet Downloader will end up downloading 210 GB of data. Downloading only a city's worth of data will likely only need a few MB of data.

In your toolbar, click on the GeoParquet Downloader icon. It's the one with blue, rotated rectangles.

Click the "Custom URL" option and paste the following URL in.

s3://us-west-2.opendata.source.coop/tge-labs/globalbuildingatlas-lod1/*.parquet

Hit "OK". You'll be prompted to save a Parquet file onto your computer. Once you've done so, the GBA data for the area in your map should appear shortly afterword.

Once the buildings have loaded, select the layer styling for the downloaded buildings layer. The combo box at the top of the styling panel should be switched from "Single Symbol" to "2.5D".

In the "Height" field, change it to the following expression:

min("height", 1) / 50000

You should now see a 2.5D representation of the buildings you downloaded.

There is an "Identify Features" icon in the toolbar.

This tool lets you select any building and bring up its metadata.

The Parquet file you saved can also be read via DuckDB and even dropped into Esri's ArcGIS Pro 3.5 or newer if you have it installed on your machine.

The LoD1 URLs

Below I'll discuss how I turned the 1.1 TB of GeoJSON into Parquet. These steps don't need to be repeated as the resulting files are now on S3.

I first set up a working folder on a partition that had at least 600 GB of free space.

$ mkdir -p ~/tum/stats
$ cd ~/tum

I built a 922-line list of URLs for the LoD1 dataset. They're available from this gist and I have them saved as manifest.lod.txt. Below are a few lines from this file.

$ head -n5 manifest.lod.txt

https://dataserv.ub.tum.de/s/m1782307/download?path=%2FLoD1%2Fafrica&files=e000_n10_e005_n05.geojson
https://dataserv.ub.tum.de/s/m1782307/download?path=%2FLoD1%2Fafrica&files=e000_n15_e005_n10.geojson
https://dataserv.ub.tum.de/s/m1782307/download?path=%2FLoD1%2Fafrica&files=e000_n20_e005_n15.geojson
https://dataserv.ub.tum.de/s/m1782307/download?path=%2FLoD1%2Fafrica&files=e000_n25_e005_n20.geojson
https://dataserv.ub.tum.de/s/m1782307/download?path=%2FLoD1%2Fafrica&files=e000_n30_e005_n25.geojson

I collected file size data for these URLs from TUM's server.

$ touch lods1.json && rm lods1.json

$ for URL in `cat manifest.txt`; do
    echo $URL
    curl -sI "$URL" | jc --kv >> lods1.json
  done

I then used Python to clean up some of the output and make it easier to analyse.

$ python3

import json
from   urllib.parse import unquote


urls  = [x.strip()     for x in open('manifest.lod.txt', 'r').readlines()]
resps = [json.loads(x) for x in open('lods1.json', 'r').readlines()]

for num, resp in enumerate(resps):
    resp['content_length'] = int(resp['content-length'])
    resp['http_code']      = [int(x.split(' ')[-1])
                              for x in resp.keys()
                              if x.startswith('HTTP/2')][0]

    resp['filename'] = resp['content-disposition']\
                            .split('filename=')[-1]\
                            .strip('"')
    resp['url']      = [x for x in urls if resp['filename'] in x][0]
    resp['folder']   = unquote(resp['url']
                                .split('path=')[-1]
                                .split('&')[0])

    resps[num] = resp

open('lods1.enriched.json', 'w')\
    .write('\n'.join(json.dumps(x) for x in resps))

There were 922 HTTP 200 responses from the above crawl. One for every URL in the manifest.

$ ~/duckdb

SELECT   http_code,
         COUNT(*)
FROM     'lods1.enriched.json'
GROUP BY 1;

┌───────────┬──────────────┐
│ http_code │ count_star() │
│   int64   │    int64     │
├───────────┼──────────────┤
│    200    │     922      │
└───────────┴──────────────┘

This is the number of GB each region of the world contains on TUM's servers. Note, this is all uncompressed GeoJSON.

SELECT   folder,
         (SUM(content_length) / 1024 ** 3)::INT AS GB
FROM     'lods1.enriched.json'
GROUP BY 1
ORDER BY 2 DESC;

┌────────────────────┬───────┐
│       folder       │  GB   │
│      varchar       │ int32 │
├────────────────────┼───────┤
│ /LoD1/asiawest     │   388 │
│ /LoD1/africa       │   294 │
│ /LoD1/europe       │   202 │
│ /LoD1/northamerica │   147 │
│ /LoD1/southamerica │   140 │
│ /LoD1/asiaeast     │   108 │
│ /LoD1/oceania      │    67 │
└────────────────────┴───────┘

These are the number of GeoJSON files that make up each region of the world.

SELECT   folder,
         COUNT(*) num_urls
FROM     'lods1.enriched.json'
GROUP BY 1
ORDER BY 2 DESC;

┌────────────────────┬──────────┐
│       folder       │ num_urls │
│      varchar       │  int64   │
├────────────────────┼──────────┤
│ /LoD1/northamerica │      179 │
│ /LoD1/africa       │      176 │
│ /LoD1/asiawest     │      130 │
│ /LoD1/asiaeast     │      128 │
│ /LoD1/oceania      │      114 │
│ /LoD1/southamerica │      101 │
│ /LoD1/europe       │       94 │
└────────────────────┴──────────┘

These are the smallest and largest files in the LoD1 dataset.

SELECT   content_length,
         folder,
         filename
FROM     'lods1.enriched.json'
ORDER BY content_length;

┌────────────────┬────────────────────┬───────────────────────────┐
│ content_length │       folder       │         filename          │
│     int64      │      varchar       │          varchar          │
├────────────────┼────────────────────┼───────────────────────────┤
│            674 │ /LoD1/asiaeast     │ e135_n25_e140_n20.geojson │
│            677 │ /LoD1/asiaeast     │ e170_n75_e175_n70.geojson │
│           2137 │ /LoD1/northamerica │ w050_n60_w045_n55.geojson │
│           3437 │ /LoD1/asiawest     │ e095_n70_e100_n65.geojson │
│           5746 │ /LoD1/northamerica │ w105_n65_w100_n60.geojson │
│           6913 │ /LoD1/northamerica │ w145_n60_w140_n55.geojson │
│          11094 │ /LoD1/southamerica │ w030_s20_w025_s25.geojson │
│          12586 │ /LoD1/northamerica │ w125_n75_w120_n70.geojson │
│          14778 │ /LoD1/northamerica │ w075_n60_w070_n55.geojson │
│          15223 │ /LoD1/oceania      │ e165_n05_e170_n00.geojson │
│          18611 │ /LoD1/asiaeast     │ e150_n75_e155_n70.geojson │
│          20510 │ /LoD1/oceania      │ e115_s15_e120_s20.geojson │
│          23356 │ /LoD1/asiaeast     │ e115_n75_e120_n70.geojson │
│          27109 │ /LoD1/northamerica │ w150_n60_w145_n55.geojson │
│          29734 │ /LoD1/asiawest     │ e085_n75_e090_n70.geojson │
│          31258 │ /LoD1/oceania      │ e065_s45_e070_s50.geojson │
│          32752 │ /LoD1/asiaeast     │ e160_n55_e165_n50.geojson │
│          32891 │ /LoD1/northamerica │ w170_n20_w165_n15.geojson │
│          33177 │ /LoD1/europe       │ e055_n75_e060_n70.geojson │
│          34303 │ /LoD1/northamerica │ w065_n65_w060_n60.geojson │
│            ·   │      ·             │             ·             │
│            ·   │      ·             │             ·             │
│            ·   │      ·             │             ·             │
│    12115126130 │ /LoD1/europe       │ e005_n50_e010_n45.geojson │
│    12663234796 │ /LoD1/asiawest     │ e090_n25_e095_n20.geojson │
│    12907517141 │ /LoD1/africa       │ e030_n05_e035_n00.geojson │
│    13682576090 │ /LoD1/northamerica │ w100_n20_w095_n15.geojson │
│    13778082922 │ /LoD1/asiawest     │ e075_n20_e080_n15.geojson │
│    13858487472 │ /LoD1/asiawest     │ e070_n30_e075_n25.geojson │
│    14316795070 │ /LoD1/asiawest     │ e075_n25_e080_n20.geojson │
│    14440822865 │ /LoD1/asiawest     │ e105_n15_e110_n10.geojson │
│    14459258848 │ /LoD1/southamerica │ w050_s20_w045_s25.geojson │
│    14558343630 │ /LoD1/asiawest     │ e080_n25_e085_n20.geojson │
│    15756826807 │ /LoD1/europe       │ e005_n55_e010_n50.geojson │
│    15986484228 │ /LoD1/oceania      │ e105_s05_e110_s10.geojson │
│    16830813690 │ /LoD1/asiawest     │ e070_n25_e075_n20.geojson │
│    18731290715 │ /LoD1/oceania      │ e110_s05_e115_s10.geojson │
│    22094633389 │ /LoD1/asiawest     │ e085_n30_e090_n25.geojson │
│    24040590951 │ /LoD1/asiawest     │ e070_n35_e075_n30.geojson │
│    24784041081 │ /LoD1/asiawest     │ e075_n15_e080_n10.geojson │
│    27418761453 │ /LoD1/asiawest     │ e085_n25_e090_n20.geojson │
│    31113223192 │ /LoD1/asiawest     │ e080_n30_e085_n25.geojson │
│    31830016696 │ /LoD1/asiawest     │ e075_n30_e080_n25.geojson │
├────────────────┴────────────────────┴───────────────────────────┤
│ 922 rows (40 shown)                                   3 columns │
└─────────────────────────────────────────────────────────────────┘

Downloading GeoJSON

One day after I started downloading the GeoJSON files, the URLs were changed on TUM's server. I had to run the following script to produce a new list of URLs to download.

$ echo "SELECT 'https://dataserv.ub.tum.de/public.php/dav/files/m1782307/LoD1/'
                    || REPLACE(folder, '/LoD1/', '')
                    || '/'
                    || filename AS url
         FROM   'lods1.enriched.json'" \
    | ~/duckdb -csv -noheader \
    > manifest.lod.dav.txt

Below is the download script I used. Once a file was downloaded, it was compressed to preserve space on my machine. The files are downloaded in a random order and after each pass of the 922 URLs, any missing files not downloaded will be retried. I experienced several outages of TUMs server during this process so the retries came in handy.

$ while :
  do
      for LINE in `sort -R manifest.lod.dav.txt`; do
          GEOJSON=`echo "$LINE" | cut -d/ -f10`
          ZIPFILE="$GEOJSON.zip"

          if [ ! -f $ZIPFILE ]; then
              echo $LINE
              wget -c -O $GEOJSON "$LINE"

              LINE_COUNT=$(wc -l < $GEOJSON)

              if [ "$LINE_COUNT" -eq 0 ]; then
                  rm "$GEOJSON"
                  echo "$GEOJSON has zero lines. Sleeping for 3 minutes.."
                  sleep 180
              else
                  zip -9v "$ZIPFILE" "$GEOJSON"
                  rm "$GEOJSON"
              fi
          else
              echo "$ZIPFILE already exists, skipping.."
          fi
      done

      echo "Finished a pass of the manifest file. Sleeping for 5 minutes.."
      sleep 300
  done

The above took the better part of a week to complete. The 1.1 TB of uncompressed GeoJSON was compressed into 357 GB worth of ZIP files.

GeoJSON to Parquet

The following GeoJSON to Parquet script ran concurrently to the above download script. It extracts the GeoJSON file inside of any unprocessed ZIP file, converts its projection from EPSG:3857 to EPSG:4326, creates a bounding box and spatially sorts the dataset into a ZStandard-compressed Parquet file.

This script runs on a loop and looks for new files process during each run. The GeoJSON to Parquet process is pretty quick on my machine so this script spent a lot of time waiting for new ZIPs.

$ while :
  do
      for ZIPFILE in *.geojson.zip; do
          BASENAME=`echo "$ZIPFILE" | cut -f1 -d.`
          PQ_FILE="$BASENAME.parquet"

          if [ ! -f $PQ_FILE ]; then
              echo $BASENAME

              mkdir -p working
              rm working/* || true 2>/dev/null
              unzip $ZIPFILE -d working/
              FIRST=`ls working/*.geojson | head -n1`

              echo "COPY (
                        SELECT   * EXCLUDE (geom),
                                 ST_FLIPCOORDINATES(
                                      ST_TRANSFORM(geom,
                                                   'EPSG:3857',
                                                   'EPSG:4326')) geometry,
                                 {'xmin': ST_XMIN(ST_EXTENT(ST_FLIPCOORDINATES(
                                              ST_TRANSFORM(geom,
                                                           'EPSG:3857',
                                                           'EPSG:4326')))),
                                  'ymin': ST_YMIN(ST_EXTENT(ST_FLIPCOORDINATES(
                                              ST_TRANSFORM(geom,
                                                           'EPSG:3857',
                                                           'EPSG:4326')))),
                                  'xmax': ST_XMAX(ST_EXTENT(ST_FLIPCOORDINATES(
                                              ST_TRANSFORM(geom,
                                                           'EPSG:3857',
                                                           'EPSG:4326')))),
                                  'ymax': ST_YMAX(ST_EXTENT(ST_FLIPCOORDINATES(
                                              ST_TRANSFORM(geom,
                                                           'EPSG:3857',
                                                           'EPSG:4326'))))} AS bbox
                        FROM     ST_READ('$FIRST')
                        ORDER BY HILBERT_ENCODE([
                              ST_Y(ST_CENTROID(ST_FLIPCOORDINATES(
                                      ST_TRANSFORM(geom,
                                                   'EPSG:3857',
                                                   'EPSG:4326')))),
                              ST_X(ST_CENTROID(ST_FLIPCOORDINATES(
                                      ST_TRANSFORM(geom,
                                                   'EPSG:3857',
                                                   'EPSG:4326'))))]::double[2])
                    ) TO '$BASENAME.parquet' (
                        FORMAT            'PARQUET',
                        CODEC             'ZSTD',
                        COMPRESSION_LEVEL 22,
                        ROW_GROUP_SIZE    15000);" | ~/duckdb
          else
              echo "$PQ_FILE already exists, skipping.."
          fi
      done

      echo "Sleeping for 5 minutes.."
      sleep 300
  done

The above turned the 357 GB worth of ZIPs into 210 GB of Parquet.

The base filenames of the Parquet files match their GeoJSON counterparts. These GeoJSON files used a e115_n45_e120_n40.geojson convention where the filename indicated where the contents were located.

Below I've rendered the footprints with their filenames formatted for clarity. This was run while the download script was still going so there are footprints missing in this screen shot. The basemap used is CARTO Dark.

$ ~/duckdb

COPY (
    WITH a AS (
        SELECT ST_XMIN(ST_EXTENT(geometry)) AS xmin,
               ST_YMIN(ST_EXTENT(geometry)) AS ymin,
               ST_XMAX(ST_EXTENT(geometry)) AS xmax,
               ST_YMAX(ST_EXTENT(geometry)) AS ymax,
               filename
        FROM   READ_PARQUET('*.parquet', filename=true)
    )
    SELECT   filename,
             ST_ASWKB({'min_x': MIN(xmin),
                       'min_y': MIN(ymin),
                       'max_x': MAX(xmax),
                       'max_y': MAX(ymax)}::BOX_2D::GEOMETRY) AS geometry,
             COUNT(*) num_buildings
    FROM     a
    GROUP BY filename
    ORDER BY filename
) TO 'stats/bbox.parquet' (
        FORMAT            'PARQUET',
        CODEC             'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE    15000);

LoD1 Records

There are 2,747,186,815 records in this dataset with each record representing one building.

$ ~/duckdb

SELECT COUNT(*)
FROM   READ_PARQUET('*.parquet');

2747186815

For comparison, Overture's August release contains 2,539,170,484 buildings (~208M fewer).

$ ~/duckdb

SELECT COUNT(*)
FROM   READ_PARQUET('s3://overturemaps-us-west-2/release/2025-08-20.0/theme=buildings/type=building/*.parquet',
                    hive_partitioning=1);

2539170484

Below is an example LoD1 record.

$ echo "SELECT * EXCLUDE(bbox),
               bbox::JSON AS bbox
        FROM  READ_PARQUET('e000_n10_e005_n05.parquet')
        LIMIT 1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "bbox": {
      "xmax": 0.00008150001546320158,
      "xmin": 0.0,
      "ymax": 7.993254849391249,
      "ymin": 7.993157996915336
    },
    "geometry": "POLYGON ((0.000052522233915 7.993254849391249, 0.000081500015463 7.993157996915336, 0 7.993181313497494, 0.000052522233915 7.993254849391249))",
    "height": 6.524652481079102,
    "id": "4538",
    "region": "GHA",
    "source": "ours2",
    "var": 0.7772102355957031
  }
]

Below I'll generate a heatmap of the records and render them in QGIS.

$ ~/duckdb stats/stats.duckdb

CREATE OR REPLACE TABLE h3_4_stats AS
    SELECT   H3_LATLNG_TO_CELL(
                bbox.ymin,
                bbox.xmin, 4) AS h3_4,
             COUNT(*) num_buildings
    FROM     READ_PARQUET('*.parquet')
    WHERE    bbox.xmin BETWEEN -178.5 AND 178.5
    GROUP BY 1;

COPY (
    SELECT ST_ASWKB(H3_CELL_TO_BOUNDARY_WKT(h3_4)::geometry) geometry,
           num_buildings
    FROM   h3_4_stats
) TO 'stats/h3_4_stats.parquet' (
        FORMAT 'PARQUET',
        CODEC  'ZSTD',
        COMPRESSION_LEVEL 22,
        ROW_GROUP_SIZE 15000);

Bounding Boxes

In the above rendering I've used the bounding box values instead of the underlying geometry. This produces results quicker. This is due to the values taking up less space and not needing as many transformations as the underlying geometry would.

Below is a pie chart showing how much space each column takes up in one of the Parquet files. The bbox.xmin and bbox.ymin fields are much smaller than the geometry field.

$ python3 ~/pqview/main.py \
    types --html \
    e000_n10_e005_n05.parquet \
    > stats/e000_n10_e005_n05.html

The GeoParquet Downloader Plugin for QGIS also uses these bounding boxes to download data quicker than it would if it had used the geometry field alone.

Data Sources

Below are the sources of data in GBA.

$ ~/duckdb

SELECT   COUNT(*),
         source
FROM     READ_PARQUET('*.parquet')
GROUP BY 2
ORDER BY 1 DESC;

┌──────────────┬──────────┐
│ count_star() │  source  │
│    int64     │ varchar  │
├──────────────┼──────────┤
│   1619713039 │ google   │
│    490593924 │ osm      │
│    432475081 │ ms       │
│    135815381 │ ours2    │
│     68589390 │ 3dglobfp │
└──────────────┴──────────┘

Google's 2023 Open Buildings data is the most used. The OSM data is from 2025 but I'm not sure which month and day. Microsoft Building Footprints is from 2024. The 135M "ours2" source is their Planet Labs', deep learning-derived dataset. The "3dglobfp" is from a paper published last year that shares three authors with the GBA paper published this year.

Adding "ours2" and 3dglobfp together makes ~204M which isn't far off from the 208M more buildings they have compared to Overture's release from last month.

And even though "ours2" and 3dglobfp are only 7% of this dataset's footprint sources, their geographical spread is vast. Below, the orange hexagons are where "ours2" is the most common source and yellow are where 3dglobfp are the most common source.

$ ~/duckdb sources.duckdb

CREATE OR REPLACE TABLE h3_5s AS
    WITH b AS (
        WITH a AS (
            SELECT   H3_LATLNG_TO_CELL(bbox.ymin,
                                       bbox.xmin,
                                       5) h3_5,
                     source,
                     COUNT(*) num_recs
            FROM     READ_PARQUET('*.parquet')
            WHERE    bbox.xmin BETWEEN -175 AND 175
            GROUP BY 1, 2
        )
        SELECT *,
               ROW_NUMBER() OVER (PARTITION BY h3_5
                                  ORDER BY     num_recs DESC) AS rn
        FROM   a
    )
    FROM     b
    WHERE    rn = 1
    ORDER BY num_recs DESC;

COPY (
    SELECT H3_CELL_TO_BOUNDARY_WKT(h3_5)::GEOMETRY geom,
           source
    FROM   h3_5s
) TO 'stats/gba.top_sources.parquet' (
    FORMAT 'PARQUET',
    CODEC  'ZSTD',
    COMPRESSION_LEVEL 22,
    ROW_GROUP_SIZE 15000);

Below I've added Google in green, Microsoft in red and OSM in blue.

I'll build a heatmap showing how many sources any one hexagon has for its building footprints.

CREATE OR REPLACE TABLE h3_5_sources AS
    SELECT   H3_LATLNG_TO_CELL(bbox.ymin,
                               bbox.xmin,
                               5) h3_5,
             COUNT(DISTINCT source) num_sources
    FROM     READ_PARQUET('*.parquet')
    WHERE    bbox.xmin BETWEEN -175 AND 175
    GROUP BY 1;

Most hexagons only have one or two sources. Even three is pretty rare.

SELECT   COUNT(*),
         num_sources
FROM     h3_5_sources
GROUP BY 2
ORDER BY 2;

┌──────────────┬─────────────┐
│ count_star() │ num_sources │
│    int64     │    int64    │
├──────────────┼─────────────┤
│       128349 │           1 │
│       240610 │           2 │
│        28151 │           3 │
│         1623 │           4 │
│            1 │           5 │
└──────────────┴─────────────┘

Below is a heatmap where the brightest hexagons have the largest number of sources.

COPY (
    SELECT H3_CELL_TO_BOUNDARY_WKT(h3_5)::geometry geom,
           num_sources
    FROM   h3_5_sources
) TO 'stats/gba.distinct_sources.parquet' (
    FORMAT 'PARQUET',
    CODEC  'ZSTD',
    COMPRESSION_LEVEL 22,
    ROW_GROUP_SIZE 15000);

For reference, you'd only need a dozen or so of these hexagons to cover Dubai's urban areas.

Buildings in Beijing

Last year, I wrote a blog post about a team of researchers that built an AI-detected building dataset from Google Map's Satellite Imagery. The dataset uses the acronym CLSM. It did a lot to fill in the many missing areas of China that aren't mapped on OpenStreetMap (OSM). This dataset ended up being the bulk of Overture's building dataset for China earlier this year.

In some areas, CLSM did a great job of accurately identifying building footprints but there were other areas where the model wasn't as accurate.

Below in yellow is CLSM's data for a neighbourhood in Beijing. In purple is GBA's. As you can see, there has been a serious improvement in accuracy.

Still, GBA isn't without its faults. This is another neighbourhood in Beijing.

Below is the area around Beijing's Capital International Airport (PEK). Red is OSM, yellow is CLSM (via Overture) and purple is the AI-generated GBA building footprints that I believe were built exclusively from deep learning on Planet Labs' imagery.

Ultimately, map makers are left needing to do a pick-and-mix approach to cleaning up this data. That said, where GBA can't be the final deliverable, it can at least help speed up the process.

Lack of Timestamps

China is currently building the world's largest military command centre. It's located at 116.06753 East, 39.81821 North.

As of this writing, the satellite map images from Bing and Yandex show the village that used to be here a few years ago before construction began. Below is a rendering of GBA's building footprints for this area on top of a satellite image from Yandex.

The buildings shown aren't from GBA's "ours2" dataset but rather old OSM data. This is what OSM currently looks like.

Esri has a relatively recent image of this area. Below you can see GBA's data on top of it.

As GBA has now been published and there hasn't been any announcement of upcoming updates, any priority on conflicting footprints should go in favour of the latest OSM building footprints.

For specific areas with a lot of construction, it might be worth excluding any OSM-sourced buildings in GBA's dataset entirely and use Overture's latest feed instead. Overture prioritise OSM data that will be, at most, weeks old when available.

Planet Labs' Metadata

The timestamp issue is more than just old OSM data. Any of GBA's AI-generated building footprints fails to identify the date of capture of the imagery they ran inference on. It would have been great if they had included Planet Labs' metadata in each of their records.

Countries

Border towns can often get marked as being in another country in datasets I come across. I'm pleased to say I haven't found this issue in this dataset.

Below is Narva, Estonia's buildings in blue and Ivangorod, Russia's in red. All of the buildings appear to be correctly marked with the country they're located in.

Below are the building counts, rounded to the nearest million for the top 40 most prominent countries in this dataset.

$ ~/duckdb

SELECT   ROUND(COUNT(*) / 1_000_000)::INT AS num_buildings,
         region
FROM     READ_PARQUET('*.parquet')
GROUP BY 2
ORDER BY 1 DESC
LIMIT    40;

┌───────────────┬─────────┐
│ num_buildings │ region  │
│     int32     │ varchar │
├───────────────┼─────────┤
│           467 │ IND     │
│           165 │ CHN     │
│           151 │ USA     │
│           139 │ BRA     │
│           115 │ IDN     │
│            84 │ MEX     │
│            74 │ RUS     │
│            67 │ NGA     │
│            62 │ PAK     │
│            60 │ VNM     │
│            53 │ FRA     │
│            51 │ JPN     │
│            48 │ THA     │
│            43 │ DEU     │
│            43 │ ZAF     │
│            40 │ BGD     │
│            38 │ ETH     │
│            35 │ PHL     │
│            33 │ ARG     │
│            30 │ TZA     │
│            28 │ KEN     │
│            28 │ UKR     │
│            27 │ SDN     │
│            26 │ COD     │
│            25 │ EGY     │
│            23 │ GBR     │
│            22 │ POL     │
│            22 │ UGA     │
│            21 │ TUR     │
│            21 │ ITA     │
│            21 │ PER     │
│            18 │ COL     │
│            18 │ MOZ     │
│            16 │ BFA     │
│            16 │ DZA     │
│            14 │ VEN     │
│            14 │ CHL     │
│            13 │ IRN     │
│            13 │ GHA     │
│            13 │ ROU     │
├───────────────┴─────────┤
│ 40 rows       2 columns │
└─────────────────────────┘

The 151M Buildings for the US is a bit of a surprise for me. The ORNL dataset for last year had this number at 131M which included building usage and addresses.

The Height Dataset

35 of the 36 TB of data TUM published is made up of GeoTIFF height maps stored in ZIP files. Each ZIP file contains 100s of GeoTIFFs.

$ wget -c "https://dataserv.ub.tum.de/public.php/dav/files/m1782307/Height/africa/e000_n10_e005_n05.zip"

The above 96 GB ZIP file contains a total of 114.5 GB worth of GeoTIFFs.

$ unzip -l e000_n10_e005_n05.zip

Archive:  e000_n10_e005_n05.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
266161026  2024-04-19 18:16   0.0_9.2_0.2_9.0_sr_ss.tif
264664332  2024-04-19 22:05   1.2_7.4_1.4_7.2_sr_ss.tif
...
262812174  2024-04-20 04:38   3.6_8.4_3.8_8.2_sr_ss.tif
264402340  2024-04-20 07:33   4.8_9.0_5.0_8.8_sr_ss.tif
---------                     -------
114525128728                     434 files

Below is one of the height maps rendered in QGIS.

Below is the metadata on the above 254 MB GeoTIFF.

$ exiftool 0.2_8.8_0.4_8.6_sr_ss.tif \
    | jc --kv \
    | jq -S .

{
  "Bits Per Sample": "32",
  "Compression": "Uncompressed",
  "Directory": ".",
  "Exif Byte Order": "Little-endian (Intel, II)",
  "ExifTool Version Number": "12.76",
  "File Access Date/Time": "2025:09:02 19:17:14+03:00",
  "File Inode Change Date/Time": "2025:09:02 18:16:24+03:00",
  "File Modification Date/Time": "2024:04:19 18:52:54+03:00",
  "File Name": "0.2_8.8_0.4_8.6_sr_ss.tif",
  "File Permissions": "-rwxrwxrwx",
  "File Size": "266 MB",
  "File Type": "TIFF",
  "File Type Extension": "tif",
  "GDAL No Data": "-1",
  "GT Citation": "WGS 84 / UTM zone 31N",
  "GT Model Type": "Projected",
  "GT Raster Type": "Pixel Is Area",
  "Geo Tiff Version": "1.1.0",
  "Geog Angular Units": "Angular Degree",
  "Geog Citation": "WGS 84",
  "Image Height": "8172",
  "Image Size": "8133x8172",
  "Image Width": "8133",
  "MIME Type": "image/tiff",
  "Megapixels": "66.5",
  "Model Tie Point": "0 0 0 190683 975006 0",
  "Photometric Interpretation": "BlackIsZero",
  "Pixel Scale": "3 3 0",
  "Planar Configuration": "Chunky",
  "Proj Linear Units": "Linear Meter",
  "Projected CS Type": "WGS84 UTM zone 31N",
  "Rows Per Strip": "1",
  "Sample Format": "Float",
  "Samples Per Pixel": "1",
  "Strip Byte Counts": "(Binary data 49031 bytes, use -b option to extract)",
  "Strip Offsets": "(Binary data 78308 bytes, use -b option to extract)"
}

Alberta's Pipelines

2025-08-01T20:00:00+03:00

Alberta produces 85% of Canada's oil helping it rank 5th among the world's top oil exporters.

There are 173K active pipelines in Alberta, 2,884 of which ship crude oil. Below is a map of Alberta's active pipelines.

For context, I shifted Alberta's borders 140 degrees East to compare its landmass to Europe's. Alberta's Northern border sits at the same latitude as Helsinki.

Alberta's Energy Regulator (AER) hosts a large number of datasets describing the province's pipelines and other energy-related infrastructure.

In this post, I'll explore a few of their open datasets.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use GDAL 3.9.3 and a few other tools to help analyse the data in this post.

$ sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable
$ sudo apt update
$ sudo apt install \
    gdal-bin \
    jq

I'll use DuckDB, along with its H3, JSON, Lindel, Parquet and Spatial extensions, in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.3.0/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

I used QGIS' Tile+ plugin to add basemaps from Esri and CARTO to the maps throughout this post.

Analysis-Ready Data

I'll download AER's Pipelines and Facilities datasets. I'll convert their projections to EPSG:4326 and store them in spatially-sorted, ZStandard-compressed Parquet format. This format will load without issue in QGIS 3.44 and ArcGIS Pro 3.5.

$ wget https://www.aer.ca/data/pipeline/Pipelines_SHP.zip
$ unzip Pipelines_SHP.zip
$ ~/duckdb pipes.duckdb

COPY (
    SELECT   * EXCLUDE(geom),
             ST_ASWKB(
                ST_TRANSFORM(geom,
                             'EPSG:4269',
                             'EPSG:4326')) geom,
             {'xmin': ST_XMIN(ST_EXTENT(geom)),
              'ymin': ST_YMIN(ST_EXTENT(geom)),
              'xmax': ST_XMAX(ST_EXTENT(geom)),
              'ymax': ST_YMAX(ST_EXTENT(geom))} AS bbox
    FROM     ST_READ('Pipelines_SHP/Pipelines_GCS_NAD83.shx')
    ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geom)),
                             ST_X(ST_CENTROID(geom))]::double[2])
) TO 'pipelines.parquet' (
    FORMAT            'PARQUET',
    CODEC             'ZSTD',
    COMPRESSION_LEVEL 22,
    ROW_GROUP_SIZE    15000);

The above produced a 323,259-record, 47 MB Parquet file. Below is an example record.

$ echo "FROM READ_PARQUET('pipelines.parquet')
        LIMIT 1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "BA_CODE": "A5D4",
    "BIDIRE_IND": null,
    "CLASS_LOC": null,
    "COMP_NAME": "Cenovus Energy Inc.",
    "EXT_COAT": null,
    "FLD_CTR_NM": "Bonnyville",
    "FROM_FAC": "Blind End",
    "FROM_LOC": "10-07-071-02W4",
    "GEOM_SRCE": "Mapping",
    "H2S_CONT": "0",
    "H2S_R_LEVL": null,
    "H2S_R_VOLM": null,
    "HDD_IND": null,
    "INT_COAT": "Uncoated",
    "IS_NEB": "N",
    "JOINTMETHD": "Welded",
    "LAST_OCCYR": "2007",
    "LICAPPDATE": "2007-01-04",
    "LICENCE_NO": "26186",
    "LIC_LI_NO": "26186-477",
    "LINER_GRD": null,
    "LINER_TYPE": null,
    "LINE_NO": "477",
    "OBJECTID": 42504337,
    "ORG_ISSUED": "2006-01-04",
    "ORIGLIN_NO": "477",
    "ORIGPSPPID": null,
    "ORIG_LICNO": "26186",
    "OUT_DIAMET": "168.3",
    "PERMT_APPR": "2013-06-04",
    "PERMT_EXPI": "2007-01-04",
    "PIPE_ENVIR": "Creek Crossing",
    "PIPE_GRADE": "3592",
    "PIPE_MAOP": "0",
    "PIPE_TYPE": "Z245.1",
    "PIPTECHSTD": null,
    "PIP_MATERL": "Steel",
    "PLLICSEGID": "245633",
    "PL_SPEC_ID": "19",
    "SEG_LENGTH": "1.76",
    "SEG_STATUS": "Abandoned",
    "SHAPE_LEN": "1873.2395712004013",
    "STRESSLEVL": "0",
    "SUBSTANCE1": "Natural Gas",
    "SUBSTANCE2": null,
    "SUBSTANCE3": null,
    "TEMPSURFPL": null,
    "TO_FAC": "Blind End",
    "TO_LOC": "07-12-071-03W4",
    "WALL_THICK": "3.2",
    "bbox": "{'xmin': -110.32211225441641, 'ymin': 55.132266882976076, 'xmax': -110.30557263240699, 'ymax': 55.13400688790718}",
    "geom": "LINESTRING (-110.30557263240699 55.13400688790718, -110.31362641340809 55.13333017544028, -110.32211225441641 55.132266882976076)"
  }

I'll generate a heatmap of the pipelines.

$ ~/duckdb pipes.duckdb

CREATE OR REPLACE TABLE heatmap AS
    SELECT UNNEST(
               H3_POLYGON_WKT_TO_CELLS_EXPERIMENTAL_STRING(
                    ST_BUFFER(geom, 0.1),
                    'CONTAINMENT_OVERLAPPING',
                    5)) AS h3_5
    FROM   READ_PARQUET('pipelines.parquet');

COPY (
    SELECT   ST_ASWKB(h3_cell_to_boundary_wkt(h3_5)::GEOMETRY) geom,
             COUNT(*) AS num_recs
    FROM     heatmap
    GROUP BY 1
    ORDER BY 2 DESC
) TO 'pipelines.h3_5.parquet' (
    FORMAT            'PARQUET',
    CODEC             'ZSTD',
    COMPRESSION_LEVEL 22,
    ROW_GROUP_SIZE    15000);

I'll do the same for the facilities dataset below.

$ https://www.aer.ca/data/codes/ST102-SHP.zip
$ unzip ST102-SHP.zip
$ ~/duckdb

COPY (
    SELECT   * EXCLUDE(geom),
             ST_ASWKB(
                ST_TRANSFORM(geom,
                             'EPSG:4269',
                             'EPSG:4326')) geom,
             {'xmin': ST_XMIN(ST_EXTENT(geom)),
              'ymin': ST_YMIN(ST_EXTENT(geom)),
              'xmax': ST_XMAX(ST_EXTENT(geom)),
              'ymax': ST_YMAX(ST_EXTENT(geom))} AS bbox
    FROM     ST_READ('ST102_Facility_GCS_NAD83.shx')
    ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geom)),
                             ST_X(ST_CENTROID(geom))]::double[2])
) TO 'facilities.parquet' (
    FORMAT            'PARQUET',
    CODEC             'ZSTD',
    COMPRESSION_LEVEL 22,
    ROW_GROUP_SIZE    15000);

The above produced a 123,532-record, 4.8 MB Parquet file. Below is an example record.

$ echo "FROM READ_PARQUET('facilities.parquet')
        LIMIT 1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "EDCT_DESCR": "Compressor Stn <0.01",
    "EDCT_TYPE": "040",
    "FAC_ID": "ABCS0030747",
    "FAC_NAME": "Encana Corporation",
    "FAC_STATUS": "Abandoned",
    "FAC_SUB_TY": "Compressor Station",
    "LICENSEE": "Cenovus Energy Inc.",
    "LIC_BA_ID": "A5D4",
    "LIC_NUMBER": "30747",
    "LIC_TYPE": "F",
    "LOC_SOURCE": "Facility Licence",
    "OPERATOR": "Cenovus Energy Inc.",
    "OP_BA_ID": "A5D4",
    "SUB_CODE": "601",
    "bbox": "{'xmin': -110.32407899999998, 'ymin': 55.13219800000007, 'xmax': -110.32407899999998, 'ymax': 55.13219800000007}",
    "geom": "POINT (-110.32407899999998 55.13219800000007)"
  }
]

Below is a rendering of the two datasets overlapping one another.

This is another rendering showing pipelines connected to various batteries.

Some clear patterns emerge in this dataset, like pipelines largely avoiding major population centres.

The facilities heatmap closely matches the footprint of energy resources in the province.

$ ~/duckdb

COPY (
    SELECT   ST_ASWKB(h3_cell_to_boundary_wkt(
                  h3_latlng_to_cell(ST_Y(geom),
                                    ST_X(geom),
                                    6))::GEOMETRY) geom,
             COUNT(*) as num_recs
    FROM     READ_PARQUET('facilities.parquet')
    GROUP BY 1
) TO 'facilities.h3_6.parquet' (
    FORMAT            'PARQUET',
    CODEC             'ZSTD',
    COMPRESSION_LEVEL 22,
    ROW_GROUP_SIZE    15000);

Pipelines

The metadata for these datasets is pretty rich and the values are well-normalised.

Below are the most common materials used to construct the pipelines.

$ ~/duckdb

SELECT   COUNT(*),
         PIP_MATERL
FROM     READ_PARQUET('pipelines.parquet')
GROUP BY 2
ORDER BY 1 DESC
LIMIT    10

┌──────────────┬──────────────────────┐
│ count_star() │      PIP_MATERL      │
│    int64     │       varchar        │
├──────────────┼──────────────────────┤
│       258867 │ Steel                │
│        36175 │ Polyethylene         │
│        13052 │ Fibreglass           │
│        12302 │ Composite            │
│         1983 │ Aluminum             │
│          364 │ Polyvinyl Chloride   │
│          194 │ Asbestos Cement      │
│          155 │ Unknown              │
│          105 │ Stainless Steel      │
│           45 │ Poly (not certified) │
├──────────────┴──────────────────────┤
│ 10 rows                   2 columns │
└─────────────────────────────────────┘

As mentioned earlier, there are 173K pipelines currently operating.

SELECT   COUNT(*),
         SEG_STATUS
FROM     READ_PARQUET('pipelines.parquet')
GROUP BY 2
ORDER BY 1 DESC
LIMIT    10

┌──────────────┬──────────────┐
│ count_star() │  SEG_STATUS  │
│    int64     │   varchar    │
├──────────────┼──────────────┤
│       173188 │ Operating    │
│        97653 │ Abandoned    │
│        48234 │ Discontinued │
│         2147 │ Removed      │
│         2037 │ Permitted    │
└──────────────┴──────────────┘

Pipelines can transport more than one substance in some cases. Below is the most common primary substances being transported.

SELECT   COUNT(*),
         SUBSTANCE1
FROM     READ_PARQUET('pipelines.parquet')
GROUP BY 2
ORDER BY 1 DESC
LIMIT    10

┌──────────────┬───────────────────┐
│ count_star() │    SUBSTANCE1     │
│    int64     │      varchar      │
├──────────────┼───────────────────┤
│       184727 │ Natural Gas       │
│        76939 │ Oil-Well Effluent │
│        18950 │ Salt Water        │
│        13547 │ Fuel Gas          │
│        11285 │ Sour Natural Gas  │
│         5498 │ Crude Oil         │
│         4842 │ Fresh Water       │
│         1819 │ HVP Products      │
│         1573 │ LVP Products      │
│         1178 │ Steam             │
├──────────────┴───────────────────┤
│ 10 rows                2 columns │
└──────────────────────────────────┘

Below are the top firm and primary substance pairs.

SELECT   COUNT(*),
         COMP_NAME,
         SUBSTANCE1
FROM     READ_PARQUET('pipelines.parquet')
GROUP BY 2, 3
ORDER BY 1 DESC
LIMIT    20;

┌──────────────┬───────────────────────────────────────┬───────────────────┐
│ count_star() │               COMP_NAME               │    SUBSTANCE1     │
│    int64     │                varchar                │      varchar      │
├──────────────┼───────────────────────────────────────┼───────────────────┤
│        38126 │ Canadian Natural Resources Limited    │ Natural Gas       │
│        11280 │ Ipc Canada Ltd.                       │ Natural Gas       │
│        11266 │ Canadian Natural Resources Limited    │ Oil-Well Effluent │
│        11121 │ Ember Resources Inc.                  │ Natural Gas       │
│        10317 │ Torxen Energy Ltd.                    │ Natural Gas       │
│         7736 │ Cenovus Energy Inc.                   │ Natural Gas       │
│         6744 │ Canlin Energy Corporation             │ Natural Gas       │
│         6028 │ Pine Cliff Energy Ltd.                │ Natural Gas       │
│         5618 │ ATCO Gas And Pipelines Ltd.           │ Natural Gas       │
│         5202 │ Lynx Energy ULC                       │ Natural Gas       │
│         4815 │ Tourmaline Oil Corp.                  │ Natural Gas       │
│         3683 │ Canadian Natural Resources Limited    │ Fuel Gas          │
│         3278 │ TAQA North Ltd.                       │ Natural Gas       │
│         3260 │ Canadian Natural Resources Limited    │ Salt Water        │
│         2961 │ Cardinal Energy Ltd.                  │ Oil-Well Effluent │
│         2896 │ Conifer Energy Inc.                   │ Oil-Well Effluent │
│         2709 │ Cenovus Energy Inc.                   │ Oil-Well Effluent │
│         2687 │ InPlay Oil Corp.                      │ Oil-Well Effluent │
│         2686 │ Sequoia Resources Corp.               │ Natural Gas       │
│         2658 │ Peyto Exploration & Development Corp. │ Natural Gas       │
├──────────────┴───────────────────────────────────────┴───────────────────┤
│ 20 rows                                                        3 columns │
└──────────────────────────────────────────────────────────────────────────┘

Below are the top 20 firms ranked by number of primary substances transported.

SELECT   COMP_NAME,
         COUNT(DISTINCT SUBSTANCE1)
FROM     READ_PARQUET('pipelines.parquet')
GROUP BY 1
ORDER BY 2 DESC
LIMIT    20;

┌───────────────────────────────────────┬────────────────────────────┐
│               COMP_NAME               │ count(DISTINCT SUBSTANCE1) │
│                varchar                │           int64            │
├───────────────────────────────────────┼────────────────────────────┤
│ Canadian Natural Resources Limited    │                         21 │
│ Keyera Energy Ltd.                    │                         17 │
│ Suncor Energy Inc.                    │                         17 │
│ Cenovus Energy Inc.                   │                         16 │
│ Strathcona Resources Ltd.             │                         14 │
│ Tourmaline Oil Corp.                  │                         14 │
│ Shell Canada Limited                  │                         14 │
│ Imperial Oil Resources Limited        │                         14 │
│ Pembina Pipeline Corporation          │                         13 │
│ ConocoPhillips Canada Resources Corp. │                         13 │
│ PGI Processing ULC                    │                         13 │
│ Conifer Energy Inc.                   │                         13 │
│ Canadian Natural Upgrading Limited    │                         12 │
│ Plains Midstream Canada ULC           │                         12 │
│ ARC Resources Ltd.                    │                         12 │
│ Canlin Energy Corporation             │                         12 │
│ Baytex Energy Ltd.                    │                         11 │
│ Secure Waste Infrastructure Corp.     │                         11 │
│ Ovintiv Canada ULC                    │                         11 │
│ Loyal Energy (canada) Operating Ltd.  │                         11 │
├───────────────────────────────────────┴────────────────────────────┤
│ 20 rows                                                  2 columns │
└────────────────────────────────────────────────────────────────────┘

Planet Labs' Hyperspectral Imagery

2025-07-30T20:00:00+03:00

Last August, Planet Labs launched a hyperspectral satellite called Tanager-1. It uses a Dyson imaging spectrometer to capture 426 bands across the 380 - 2500nm Visible to Short Wavelength Infrared (VSWIR) spectral range.

Satellite imagery that appears in Google Maps often is only contains three bands of light (red, green and blue). Having more bands means things that are invisible to the naked eye become visible and measurable. The Awesome Spectral Indices project lists hundreds of chemical and naturally-occurring phenomenon that can be detected from hyperspectral imagery.

A few weeks after Tanager's launch, Will Marshall, Planet Labs' CEO published locations of methane leaks they detected over Texas, South Africa and Pakistan.

A few weeks ago, Planet Labs announced an open data feed of their Tanager-1-collected imagery. As of this writing there are 52 images totalling 25 GB in HDF5 format. These images were collected between February and June and have a ground sampling distance (GSD) of 32.58 to 39.12 meters.

In this post, I'll explore Planet Labs' open Hyperspectral Imagery.

My Workstation

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

Installing Prerequisites

I'll use GDAL 3.9.3, Python 3.12.3 and a few other tools to help analyse the data in this post.

$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable
$ sudo apt update
$ sudo apt install \
    gdal-bin \
    jq \
    python3-pip \
    python3.12-venv

I'll set up a Python Virtual Environment and install a few dependencies.

$ python3 -m venv ~/.tanager
$ source ~/.tanager/bin/activate

$ python3 -m pip install \
    geocoder \
    hypercoast \
    jupyterlab \
    pystac \
    pyvista \
    rich \
    shapely \
    skyfield

I'll use DuckDB, along with its H3, JSON, Lindel, Parquet and Spatial extensions, in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.3.0/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

The maps in this post were rendered with QGIS version 3.42. QGIS is a desktop application that runs on Windows, macOS and Linux. The application has grown in popularity in recent years and has ~15M application launches from users all around the world each month.

Tanager-1

Below is an image of Tanager-1 published by Planet Labs prior to its launch.

Below is an image SpaceX published of Tanager-1, along with 115 other payloads being readied for launch.

Below is the moment SpaceX deployed Tanager-1 into low earth orbit.

Planet Labs operate an Ephemerides service which gives live status and trajectory information for the satellites in their constellations.

Below is the operation status of Tanager-1 at the time of writing.

$ curl -sL http://ephemerides.planet-labs.com/operational_status.txt \
    | grep -i tanager

The following lists Tanager-1's hardware ID, name, NORAD ID and current status.

4001    Tanager-4001            60507           Operational

Below is Tanager-1's trajectory information at the time of writing, expressed as a two-line element set.

$ curl -s https://ephemerides.planet-labs.com/planet_mc.tle \
    | grep -A2 TANAGER

0 TANAGER 4001 4001
1 60507U PLANET   25204.11440972  .00000000  00000+0  34603-3 0    00
2 60507 097.4024 288.9070 0009426 241.7275 267.6297 15.43677241    08

I'll use Skyfield to estimate Tanager-1's current position over the Earth.

$ python3

from skyfield.api import load, wgs84, EarthSatellite


name  = 'TANAGER 4001 4001'
line1 = '1 60507U PLANET   25204.11440972  .00000000  00000+0  34603-3 0    00'
line2 = '2 60507 097.4024 288.9070 0009426 241.7275 267.6297 15.43677241    08'

ts        = load.timescale()
satellite = EarthSatellite(line1, line2, name, ts)
lat, lon  = wgs84.latlon_of(satellite.at(ts.now()))
print(lon.degrees, lat.degrees)

When I ran the above it reported Tanager-1's position as 72.577, -26.391 which lined up with what n2yo estimated.

Open Data Feed

I'll collect a list of Tanager imagery available from Planet Labs' STAC service. I'll use Mapbox to get the address details of each image as well.

$ mkdir -p ~/hyper
$ cd ~/hyper

$ python3

import json

import geocoder
from   pystac           import Catalog
from   rich.progress    import track
from   shapely.geometry import shape


mapbox_key = '...' # WIP: Replace with your key

root = Catalog.from_file(href='https://www.planet.com/data/stac/tanager-core-imagery/catalog.json')

try:
    recs = [json.loads(l)
            for l in open('enriched.json', 'r').readlines()]
except FileNotFoundError:
    recs = []

seen = set()
found_new = False

for item in track(list(root.get_items(recursive=True))):
    if item.assets['basic_radiance_hdf5'].href in seen:
        continue

    if any([True for x in recs if x['id'] == item.id]):
        continue

    seen.add(item.assets['basic_radiance_hdf5'].href)

    centroid_ = shape(item.geometry).centroid
    resp = geocoder.mapbox([centroid_.y, centroid_.x],
                           key=mapbox_key,
                           method='reverse')

    found_new = True
    recs.append({
        'properties':          item.properties,
        'geom':                shape(item.geometry).wkt,
        'id':                  item.id,
        'bbox':                item.bbox,
        'basic_radiance_hdf5': item.assets['basic_radiance_hdf5'].href,
        'thumbnail':           item.assets['thumbnail'].href,
        'mapbox':              resp.current_result.__dict__ if resp.ok else {},
        'collection_id':       item.collection_id})

if found_new:
    with open('enriched.json', 'w') as f:
        for rec in recs:
            f.write(json.dumps(rec, sort_keys=True) + '\n')

The above produced a 52-line JSONL file. Below is an example record.

$ head -n1 enriched.json | jq -S .

{
  "basic_radiance_hdf5": "https://storage.googleapis.com/open-cogs/planet-stac/release1-basic-radiance/20250511_074311_00_4001_basic_radiance.h5",
  "bbox": [
    53.632003821175445,
    23.990194665823015,
    53.89426128660131,
    24.19200197892316
  ],
  "collection_id": "coastal-water-bodies",
  "geom": "POLYGON ((53.67273222436389 24.19200197892316, 53.632003821175445 24.013441053575885, 53.856983165437306 23.990194665823015, 53.89426128660131 24.16789764336515, 53.67273222436389 24.19200197892316))",
  "id": "20250511_074311_00_4001",
  "mapbox": {
    "_geometry": {
      "coordinates": [
        54.377401,
        24.453835
      ],
      "type": "Point"
    },
    "east": 56.014935,
    "eastnorth": [
      56.014935,
      25.247884
    ],
    "fieldnames": [
      "accuracy",
      "address",
      "bbox",
      "city",
      "confidence",
      "country",
      "housenumber",
      "lat",
      "lng",
      "ok",
      "postal",
      "quality",
      "raw",
      "state",
      "status",
      "street"
    ],
    "json": {
      "address": "Abu Dhabi, United Arab Emirates",
      "bbox": {
        "northeast": [
          25.247884,
          56.014935
        ],
        "southwest": [
          22.631514,
          51.42123
        ]
      },
      "confidence": 1,
      "country": "United Arab Emirates",
      "lat": 24.453835,
      "lng": 54.377401,
      "ok": true,
      "quality": 1,
      "raw": {
        "bbox": [
          51.42123,
          22.631514,
          56.014935,
          25.247884
        ],
        "center": [
          54.377401,
          24.453835
        ],
        "context": [
          {
            "id": "country.8707",
            "mapbox_id": "dXJuOm1ieHBsYzpJZ00",
            "short_code": "ae",
            "text": "United Arab Emirates",
            "wikidata": "Q878"
          }
        ],
        "country": "United Arab Emirates",
        "geometry": {
          "coordinates": [
            54.377401,
            24.453835
          ],
          "type": "Point"
        },
        "id": "region.33795",
        "place_name": "Abu Dhabi, United Arab Emirates",
        "place_type": [
          "region"
        ],
        "properties": {
          "mapbox_id": "dXJuOm1ieHBsYzpoQU0",
          "short_code": "AE-AZ",
          "wikidata": "Q187712"
        },
        "relevance": 1,
        "text": "Abu Dhabi",
        "type": "Feature"
      },
      "status": "OK"
    },
    "north": 25.247884,
    "northeast": [
      25.247884,
      56.014935
    ],
    "northwest": [
      25.247884,
      51.42123
    ],
    "raw": {
      "bbox": [
        51.42123,
        22.631514,
        56.014935,
        25.247884
      ],
      "center": [
        54.377401,
        24.453835
      ],
      "context": [
        {
          "id": "country.8707",
          "mapbox_id": "dXJuOm1ieHBsYzpJZ00",
          "short_code": "ae",
          "text": "United Arab Emirates",
          "wikidata": "Q878"
        }
      ],
      "country": "United Arab Emirates",
      "geometry": {
        "coordinates": [
          54.377401,
          24.453835
        ],
        "type": "Point"
      },
      "id": "region.33795",
      "place_name": "Abu Dhabi, United Arab Emirates",
      "place_type": [
        "region"
      ],
      "properties": {
        "mapbox_id": "dXJuOm1ieHBsYzpoQU0",
        "short_code": "AE-AZ",
        "wikidata": "Q187712"
      },
      "relevance": 1,
      "text": "Abu Dhabi",
      "type": "Feature"
    },
    "south": 22.631514,
    "southeast": [
      22.631514,
      56.014935
    ],
    "southwest": [
      22.631514,
      51.42123
    ],
    "west": 51.42123,
    "westsouth": [
      51.42123,
      22.631514
    ]
  },
  "properties": {
    "constellation": "Tanager",
    "datetime": "2025-05-11T07:43:11Z",
    "description": "Basic (georeferenced, unprojected) core imagery data products from Tanager-1",
    "gsd": 34.77,
    "instruments": [
      "4001"
    ],
    "license": "CC-BY-SA-4.0",
    "platform": "Planet",
    "title": "TanagerScene 20250511_074311_00_4001 Basic Core Imagery",
    "view:azimuth": 209.1,
    "view:off_nadir": 25.8,
    "view:sun_azimuth": 122.8,
    "view:sun_elevation": 79.2
  },
  "thumbnail": "https://storage.googleapis.com/open-cogs/planet-stac/release1-basic-radiance/20250511_074311_00_4001_thumb.png"
}

The following will download 37 MB of PNG-formatted thumbnails.

$ jq .thumbnail enriched.json \
    | xargs -P4 \
            -I% \
            wget -qc %

The following will download 25 GB of HDF5-formatted imagery.

$ jq .basic_radiance_hdf5 enriched.json \
    | xargs -P4 \
            -I% \
            wget -qc %

Below is a screenshot of the thumbnails.

I'll generate a Parquet file pin-pointing the location of each image.

$ echo "COPY(
            SELECT id,
                   ST_ASWKB(ST_CENTROID(geom::geometry)) AS geom
            FROM   'enriched.json'
        ) TO 'footprints.parquet' (
            FORMAT            'PARQUET',
            CODEC             'ZSTD',
            COMPRESSION_LEVEL 22,
            ROW_GROUP_SIZE    15000);" \
    | ~/duckdb

The following was rendered in ArcGIS Pro 3.5.

Below I've listed the ID prefix of each image along with the country's footprint it was taken in. The one image missing from this list was taken close to the North Pole.

$ ~/duckdb

.maxrows 100

SELECT   SPLIT(
              SPLIT(basic_radiance_hdf5, '/')[-1],
              '_4001_')[1] AS id_prefix,
         mapbox.json.country
FROM     'enriched.json'
WHERE    mapbox.json.country IS NOT NULL
ORDER BY 2;

┌────────────────────┬────────────────────────┐
│     id_prefix      │        country         │
│      varchar       │        varchar         │
├────────────────────┼────────────────────────┤
│ 20250601_104901_58 │ Algeria                │
│ 20250224_145149_32 │ Argentina              │
│ 20250510_005001_00 │ Australia              │
│ 20250606_030446_90 │ Australia              │
│ 20250608_014315_58 │ Australia              │
│ 20250606_152952_74 │ Bolivia                │
│ 20250221_102048_16 │ Bosnia and Herzegovina │
│ 20250409_150657_00 │ Brazil                 │
│ 20250501_143138_87 │ Brazil                 │
│ 20250419_173128_87 │ Canada                 │
│ 20250520_171055_87 │ Canada                 │
│ 20250515_202305_00 │ Canada                 │
│ 20250606_181248_58 │ Canada                 │
│ 20250411_163421_16 │ Canada                 │
│ 20250510_112042_16 │ Denmark                │
│ 20250406_170447_47 │ Ecuador                │
│ 20250606_090504_75 │ Egypt                  │
│ 20250305_053421_32 │ India                  │
│ 20250311_061550_53 │ India                  │
│ 20250321_054913_40 │ India                  │
│ 20250302_030003_92 │ Indonesia              │
│ 20250504_030711_32 │ Indonesia              │
│ 20250606_103541_58 │ Italy                  │
│ 20250430_101724_87 │ Italy                  │
│ 20250503_020525_16 │ Japan                  │
│ 20250503_112050_32 │ Netherlands            │
│ 20250503_112042_00 │ Netherlands            │
│ 20250503_112046_16 │ Netherlands            │
│ 20250301_143913_32 │ Paraguay               │
│ 20250606_085958_90 │ Russia                 │
│ 20250511_011730_87 │ Russia                 │
│ 20250515_080954_16 │ Saudi Arabia           │
│ 20250516_121248_87 │ Senegal                │
│ 20250504_092952_87 │ South Africa           │
│ 20250504_025815_32 │ South Korea            │
│ 20250514_115302_15 │ Spain                  │
│ 20250509_090323_87 │ Sudan                  │
│ 20250608_091605_90 │ Türkiye                │
│ 20250511_074311_00 │ United Arab Emirates   │
│ 20250516_181930_35 │ United States          │
│ 20250528_180922_74 │ United States          │
│ 20250514_223907_74 │ United States          │
│ 20250521_194038_10 │ United States          │
│ 20250423_193451_57 │ United States          │
│ 20250501_185910_74 │ United States          │
│ 20250205_175845_00 │ United States          │
│ 20250516_164837_16 │ United States          │
│ 20250514_193937_64 │ United States          │
│ 20250503_064345_16 │ Uzbekistan             │
│ 20250225_153627_16 │ Venezuela              │
│ 20250608_074757_58 │ Yemen                  │
├────────────────────┴────────────────────────┤
│ 51 rows                           2 columns │
└─────────────────────────────────────────────┘

Imagery Metadata

Each image contains a large amount of metadata. Below is the non-band and non-STAC metadata for one of their images.

$ gdalinfo -json \
    20250225_153627_16_4001_basic_radiance.tif \
    > 20250225_153627_16_4001_basic_radiance.json
$ jq 'del(.bands)' \
    20250225_153627_16_4001_basic_radiance.json \
    | jq -S 'del(.stac)'

{
  "cornerCoordinates": {
    "center": [
      303.5,
      250.5
    ],
    "lowerLeft": [
      0.0,
      501.0
    ],
    "lowerRight": [
      607.0,
      501.0
    ],
    "upperLeft": [
      0.0,
      0.0
    ],
    "upperRight": [
      607.0,
      0.0
    ]
  },
  "description": "20250225_153627_16_4001_basic_radiance.tif",
  "driverLongName": "GeoTIFF",
  "driverShortName": "GTiff",
  "extent": {
    "coordinates": [
      []
    ],
    "type": "Polygon"
  },
  "files": [
    "20250225_153627_16_4001_basic_radiance.tif"
  ],
  "metadata": {
    "": {
      "HDFEOS_INFORMATION_HDFEOSVersion": "HDFEOS_5.1.15",
      "HDFEOS_SWATHS_HYP_Geolocation_Fields_Planet_Ortho_Framing": "{\"cols\": 778, \"epsg_code\": 32619, \"geotransform\": [355530.0, 30.0, 0.0, 1299210.0, 0.0, -30.0], \"rows\": 657}",
      "HDFEOS_SWATHS_HYP_created_at": "2025-07-12T14:33:39.649432+00:00",
      "HDFEOS_SWATHS_HYP_strip_id": "20250225_153623_00_4001_strip"
    },
    "IMAGE_STRUCTURE": {
      "INTERLEAVE": "PIXEL"
    }
  },
  "size": [
    607,
    501
  ]
}

The metadata documents each of the 426 bands collected in each image.

$ jq '.bands[]|.metadata."".wavelengths' \
    20250225_153627_16_4001_basic_radiance.json \
    | wc -l # 426

Below is the metadata for an individual band within one of their images.

$ jq '.bands[0]' 20250225_153627_16_4001_basic_radiance.json

{
  "band": 1,
  "block": [
    607,
    1
  ],
  "type": "Float32",
  "colorInterpretation": "Gray",
  "metadata": {
    "": {
      "applied_radiometric_coefficient": "0.24117877",
      "applied_radiometric_coefficients_units": "W/(m^2 sr um)",
      "fwhm": "5.3899999",
      "fwhm_units": "nm",
      "Unit": "W/(m^2 sr um)",
      "wavelengths": "376.44",
      "wavelengths_units": "nm"
    }
  }
}

Below are the first 10 wavelengths in one of their images.

$ jq '.bands[]|.metadata."".wavelengths' \
    20250225_153627_16_4001_basic_radiance.json \
    | head

"376.44"
"381.41"
"386.38"
"391.35001"
"396.32001"
"401.29001"
"406.26001"
"411.23001"
"416.20999"
"421.17999"

The GSD appears to be unique for each of the 52 images in this dataset.

$ jq -S .properties.gsd enriched.json | sort

32.58
32.62
32.64
...
38.98
39.1
39.12

According to their documentation, their Tanager-1 imagery goes through the following processing steps before publication.

Dark Subtraction
Pedestal Correction
Flat Field Correction
Bad Pixel Correction
Optical Scatter Correction
Optical Ghost Correction
Absolute Radiometric Calibration
Order Sorting Filter (OSF) Seam Correction
Visual Product Processing
Orthorectification
Atmospheric Correction

They state they use ISOFIT in the Atmospheric Correction stage. The two biggest contributors to this project both work at NASA’s Jet Propulsion Laboratory.

QGIS

QGIS can open Planet Labs' HDF5 files as they are but won't georeference them properly. Below you can see each field as its own layer with the toa_radiance field on top. I've adjusted the red, green and blue bands assignments from what QGIS selected by default.

Each image comes with a cloud mask layer as well. This helps exclude cloud-covered areas from any calculations or processing you'd run the image through.

The following will convert an HDF5 file from Planet Labs into a georeferenced GeoTIFF. It will position itself automatically when dropped into a QGIS workbook.

$ gdalwarp \
    -t_srs "EPSG:4326" \
    HDF5:"20250225_153627_16_4001_basic_radiance.h5"://HDFEOS/SWATHS/HYP/Data_Fields/toa_radiance  \
    test.tif

Note, only the toa_radiance data will be copied into the above GeoTIFF. I'll discuss this more later on but I'm working on a script that can make Planet Labs' HDF5s more analysis-ready.

ArcGIS Pro

I tried to import an HDF5 file into ArcGIS Pro 3.5. It sat at the following screen for some time.

After a while, it began reporting XML below the progress bar.

After 30+ minutes, only the beta_cirrus_mask layer was imported.

Very few tools were available for this layer.

I then tried to import the GeoTIFF I had produced earlier. ArcGIS Pro 3.5 sat here for an hour before I gave up.

I did a review of Wyvern's Hyperspectral Imagery a while back. Their GeoTIFFs can be dropped into ArcGIS Pro 3.5, have ideal RGB bands selected and rendered in under a minute on my machine. Below is one such image.

I need to figure out how to convert Planet Labs' HDF5s into this sort of analysis-ready GeoTIFF.

HyperCoast

Over the past year, Qiusheng Wu of the University of Tennessee has been working on HyperCoast, a python package for visualising and analysing hyperspectral imagery.

I'll launch Jupyter Lab from the same folder containing Planet Labs' HDF5 files. Note, I had to run the following from a dedicated Ubuntu Desktop 24 VM as I'm battling a networking issue with WSL.

$ cd ~/hyper

$ jupyter lab \
    --no-browser \
    --ip=0.0.0.0 \
    --NotebookApp.iopub_data_rate_limit=100000000

I'll create a new Python 3 notebook and run the following to render one of Planet Labs' images.

import hypercoast


dataset = hypercoast.read_tanager('20250608_074757_58_4001_basic_radiance.h5')
m = hypercoast.Map()
m.add_tanager(dataset,
              bands=[100, 60, 50],
              vmin=0,
              vmax=120,
              layer_name='Tanager')
m.add('spectral')
m

Below I've marked two points on Tanager-1's image of Venezuela. The point on the left is in the sea and its spectral signature is in green. The point on the right has its spectral signature drawn in red.

HyperCoast can render hyperspectral imagery as a 3D cube. Below is a cube for the above image of Venezuela.

gridded = hypercoast.grid_tanager(dataset,
                                  row_range=(200, 400),
                                  col_range=(200, 400))
gridded = (gridded / 100).clip(0, 1)

cube = hypercoast.image_cube(
    gridded,
    variable='toa_radiance',
    cmap='jet',
    clim=(0, 1),
    rgb_wavelengths=[1000, 600, 500],
    title='Radiance * 100',
)
cube.show()

The 3D cubes can also be sliced, including with an interactive threshold, as shown below.

Awesome Spectral Indices

The Awesome Spectral Indices project lists hundreds of chemical and naturally-occurring phenomenon that can be detected from hyperspectral imagery.

The spectral signature analysis above is useful as it can be paired with known algorithms that can identify things in hyperspectral imagery that are either invisible to the naked eye or hard to confirm with RGB imagery alone.

Below are a few of the soil-specific indices listed in the project.

There is a package called spyndex which provides Python bindings for the project as well.