Supported portals

Supported portals

Ceres currently harvests 300+ portals into a single synchronized catalog of 2M+ datasets — national portals, EU aggregators, US federal agencies, statistics offices, and city open data sites. Ten harvest paths are shipped today, and every one of them shares the same sync machinery: incremental sync where the source supports it, content-hash delta detection, bounded streaming that keeps memory flat on multi-million-dataset catalogs, and stale dataset marking. Most families stream page by page; SDMX is the exception, because the standard defines no pagination for structure queries, so its catalog arrives in one bounded request and is emitted in fixed-size chunks.

Type	Selector	Serves	Example portals
CKAN	--type ckan	CKAN action API portals	dati.comune.milano.it, catalog.data.gov, dati.gov.it
DCAT udata REST	--type dcat (default profile)	udata-flavored DCAT-AP portals	data.gouv.fr, data.public.lu
DCAT SPARQL	--type dcat --profile sparql	SPARQL-backed DCAT-AP catalogs	data.europa.eu
Project Open Data	--type dcat --profile static_json	Static DCAT-US data.json catalogs	data.va.gov, census.gov, justice.gov
Socrata	--type socrata	Socrata Discovery API catalogs	data.cityofnewyork.us, data.wa.gov
OpenDataSoft	--type opendatasoft	OpenDataSoft Explore API v2.1 catalogs	opendata.paris.fr, data.economie.gouv.fr
ArcGIS Hub	--type arcgis	ArcGIS Hub Search API catalogs	opendata.dc.gov, opendata.gis.utah.gov
OGC Records	--type ogc_records	CSW 2.0.2 / GeoNetwork catalogues	EMODnet, British Geological Survey
STAC	--type stac	Collection-level STAC APIs	Copernicus Data Space, Canada DataCube
SDMX	--type sdmx	Dataflow-level SDMX REST services	Eurostat, OECD, ILOSTAT, ISTAT, ECB

Every client preserves the complete source metadata payload, so downstream features like resource-schema extraction keep working no matter which portal a dataset came from.

Quick examples

# CKAN
ceres harvest https://dati.comune.milano.it --metadata-only

# DCAT udata REST
ceres harvest https://data.public.lu --type dcat --metadata-only

# SPARQL-backed DCAT (e.g. the EU aggregator)
ceres harvest https://data.europa.eu --type dcat --profile sparql --metadata-only

# Static Project Open Data / DCAT-US data.json
ceres harvest https://www.data.va.gov/data.json --type dcat --profile static_json --metadata-only

# Socrata Discovery API
ceres harvest https://data.cityofnewyork.us --type socrata --metadata-only

# OpenDataSoft Explore API
ceres harvest https://opendata.paris.fr --type opendatasoft --metadata-only

# ArcGIS Hub Search API
ceres harvest https://opendata.dc.gov --type arcgis --metadata-only

# OGC CSW 2.0.2
ceres harvest https://emodnet.ec.europa.eu/geonetwork/emodnet/eng/csw --type ogc_records --metadata-only

# STAC Collections (never individual Items)
ceres harvest https://stac.dataspace.copernicus.eu/v1/ --type stac --metadata-only

# SDMX dataflows (never individual observations)
ceres harvest https://sdmx.oecd.org/public/rest --type sdmx --metadata-only

Or configure them once in portals.toml and harvest in batch:

[[portals]]
name = "milano"
url = "https://dati.comune.milano.it"
type = "ckan"
language = "it"

[[portals]]
name = "nyc"
url = "https://data.cityofnewyork.us"
type = "socrata"
language = "en"

[[portals]]
name = "europa"
url = "https://data.europa.eu"
type = "dcat"
profile = "sparql"
language = "en"

See examples/portals.toml for a larger, curated configuration set.

Portal-specific notes

CKAN

• Modified-since filtering for incremental syncs
• Adaptive page sizing: on timeouts the page size is quartered (1000 → 250 → 62 → 15 → 10) and the per-request timeout grows, which handles portals that choke on large responses
• catalog.data.gov relocated its CKAN API to api.gsa.gov and requires DATA_GOV_API_KEY (free key at api.data.gov/signup)

DCAT udata REST

• Streams paginated JSON-LD catalog pages
• Resolves multilingual fields according to the configured portal language
• Waits out 429 rate limits with an escalating per-page cooldown instead of stopping with partial results

DCAT SPARQL

• Pages through the catalog with LIMIT/OFFSET queries and deduplicates by dataset URI
• Localized title/description selection: requested language → sibling language → English → untagged
• Endpoint defaults to {url}/sparql; override with sparql_endpoint

Project Open Data (data.json)

• Handles static DCAT-US catalogs published as one JSON document (US federal agencies)
• Hard 256 MiB response limit by default; override with CERES_STATIC_JSON_MAX_BYTES
• Incremental runs filter the catalog locally using modified

Socrata Discovery API

• Harvests the paginated /api/catalog/v1 endpoint, scoped to the portal domain and limited to dataset assets
• Uses updatedAt ordering for incremental synchronization
• HTML descriptions are converted to plain text for search and embedding; the original HTML stays in the raw metadata
• No credentials required for public reads; set SOCRATA_APP_TOKEN for higher rate limits

OpenDataSoft Explore API

• Harvests the paginated /api/explore/v2.1/catalog/datasets endpoint (100 datasets per page, the API maximum)
• Normalizes title, description, themes, keywords, license, publisher, and modified from metas.default; the complete catalog entry, including fields schema hints, stays in the raw metadata
• Catalogs deeper than the API’s 10,000-row pagination window (e.g. the data.opendatasoft.com federation hub) are walked with a keyset cursor on modified
• Incremental sync filters server-side with an ODSQL where=modified >= date'...' clause
• No credentials required for public reads; set ODS_API_KEY for higher quotas

ArcGIS Hub Search API

• Harvests the paginated /api/search/v1/collections/dataset/items endpoint (100 items per page, the API maximum)
• Normalizes title, description (falling back to the item snippet), categories, tags, license, publisher, and modified; the complete catalog feature, including geometry and service metadata, stays in the raw metadata
• Catalog entries mix hosted services (Feature/Image/Map Services) and file items (CSV, shapefile): the normalized dataset URL is always the Hub landing page — a service endpoint is a queryable API, not a file download
• Catalogs deeper than the API’s 10,000-row result window are walked with a keyset cursor on modified; incremental sync filters server-side with filter=modified >= <epoch millis>
• Hub sites with an empty catalogV2 item scope are rejected because their search endpoint returns global ArcGIS content rather than datasets belonging to that portal
• No credentials required for public reads

OGC Records / CSW

• Discovers GetRecords and GetRecordById bindings from CSW 2.0.2 GetCapabilities
• Streams bounded record windows and preserves each complete source XML record
• Resolves localized titles and abstracts, spatial/temporal extents, contacts, constraints, and online resources
• Classifies datasets, series, services, and maps instead of treating every catalog record as a downloadable dataset
• Asks for the ISO 19115 (gmd:MD_Metadata) profile and falls back per catalogue to the mandatory Dublin Core (csw:Record) profile when the service rejects it, which cannot be decided from GetCapabilities — Italy’s RNDT advertises ISO and refuses it. Dublin Core records carry their links as dct:references, read as online resources so the profile still contributes resource depth

STAC

• Discovers the Collections endpoint from the landing page rel=data link and follows rel=next pagination
• Stores one Ceres series record per STAC Collection, preserving the complete Collection JSON (extent, providers, assets, summaries, and extension fields)
• Never follows Collection items links; item/scene-level indexing is intentionally out of scope
• Supports STAC 1.0 and 1.1 APIs; public reads require no credentials unless the catalog itself is private

SDMX

• Reads dataflow/all/all/latest?detail=full&references=none and stores one Ceres series record per SDMX dataflow — the unit these services name, document, and serve data for
• Never queries the observation cubes behind a dataflow, mirroring the STAC decision to stop at Collections: one Eurostat dataflow alone holds millions of observations
• Parses SDMX-ML structure messages by element local name, so both 2.1 and 3.0 services work; SDMX-JSON is deliberately not requested, since Eurostat and the ECB reject it outright and every probed service serves SDMX-ML
• Resolves multilingual names and descriptions against the configured language, falling back to English and then to any published translation
• Preserves every annotation, which is where services record update timestamps, observation counts, DOIs, and links to reference metadata
• Identity is AGENCY:ID without the version, so a dataflow bumped from 1.0 to 1.1 keeps its row instead of retiring and reappearing
• Exposes each dataflow’s /data/{flowRef} query as its one normalized resource; the referenced data structure definition is kept as a reference and never followed
• Structure queries carry no standard modified-since filter, so every sync is a full sync; public reads require no credentials

Coverage validation set (v0.6.0)

The v0.6.0 milestone is proven by a small, reproducible set of portals — one per harvest profile — rather than by code paths alone. Every entry below is present in examples/portals.toml (shipped enabled = false) so you can reproduce a metadata-only validation without guessing the type/profile settings:

# Preview (no DB writes), then a real metadata-only harvest
ceres harvest --config examples/portals.toml --portal ann-arbor --metadata-only --dry-run
ceres harvest --config examples/portals.toml --portal ann-arbor --metadata-only

Profile	Selector	portals.toml name	Approx. datasets	Language	Credentials	Use as
CKAN	--type ckan	ann-arbor	~95	en	none	CI smoke
DCAT udata REST	--type dcat	luxembourg	~2,490	fr	none	Manual smoke
DCAT SPARQL	--type dcat --profile sparql	slovakia	~11,680	sk	none	Manual smoke
Project Open Data	--type dcat --profile static_json	us-national-archives	~84	en	none	CI smoke
Socrata	--type socrata	new-brunswick	~310	en	optional SOCRATA_APP_TOKEN	Manual smoke
OpenDataSoft	--type opendatasoft	paris	~490	fr	optional ODS_API_KEY	Manual smoke
ArcGIS Hub	--type arcgis	washington-dc	~1,500	en	none	Manual smoke
OGC CSW	--type ogc_records	copernicus-marine-csw	~310	en	none	Manual smoke
STAC	--type stac	canada-datacube-stac	~50 collections	en	none	CI smoke
SDMX	--type sdmx	norges-bank-sdmx	~25 dataflows	en	none	CI smoke
SPARQL (scale)	--type dcat --profile sparql	eu-open-data	~2M	en	none	Scale only

Notes:

• Use as classifies each portal for validation: CI smoke portals are small, fast, and reliable enough for automated checks; Manual smoke portals are moderate and best run by hand; Scale only portals (e.g. data.europa.eu) prove throughput but should never be the sole proof that a client works.
• Counts are approximate and drift as portals publish; treat them as an order of magnitude, not an assertion.
• No entry requires embedding credentials. SOCRATA_APP_TOKEN and ODS_API_KEY only raise public rate limits and can be omitted.
• Each profile has a matching opt-in live smoke test (most with a CERES_*_SMOKE_URL override; STAC uses CERES_STAC_*_URL, SDMX uses CERES_SDMX_*_URL, and the OGC CSW smokes hard-code their endpoints), documented in Harvesting → Opt-in live smoke tests.

The published index

The catalog Ceres maintains is published as the Ceres Open Data Index on Hugging Face: curated Parquet snapshots with versioned manifests, SHA-256 checksums, coverage/quality reports, and snapshot-to-snapshot changelogs. Snapshot schema v2 exposes normalized resources in both all.parquet and every per-portal subset as {name, format, media_type, url, field_count} structs, so consumers do not need to understand each portal’s raw metadata shape. The versioned quality report also tracks resource and field-schema completeness globally and per portal, with resource-count format and media-type distributions from the same export pass.

ceres export --format parquet --output DIR writes one snapshot directory:

• all.parquet is the canonical complete index. data/<portal>.parquet files are convenience subsets that repeat its rows; never add their counts to the total.
• resources is a non-null list. An empty list means the portal exposed no usable resource detail; the optional strings in each struct stay null when a facet is absent.
• metadata.json is the manifest: snapshot ID, UTC generation time, Ceres version and commit, portal-config checksum, curation counts, per-portal inclusion status, and a SHA-256 checksum for every Parquet file. Verify the checksums before using a copied or mirrored snapshot. A library caller that supplies no build metadata records the commit as unknown.
• reports.json and report.md carry coverage, field completeness, and curation outcomes, including how many rows each noise rule removed. Completeness rates are over exported rows, after curation.
• identity.parquet and changelog.json support snapshot-to-snapshot diffs. “Changed” means the title or description changed; see Delta Detection for what that misses.

A snapshot is dated by its export. It is not a claim that every portal was harvested on that date: the maintainer’s scheduled harvest refreshes only the portals enabled in examples/portals.toml, and the rest of the catalog was last harvested by hand.

Status

Ceres is in maintenance mode after v0.7.0, which shipped first-class resource metadata and the SDMX client. Fixes and security updates continue to land; new portal families are not planned.

Want a portal that none of the current clients cover? The client layer is trait-based and designed for extension — see Contributing or open an issue.