Supported portals

Supported portals

Ceres currently harvests 300+ portals into a single synchronized catalog of 2M+ datasets — national portals, EU aggregators, US federal agencies, and city open data sites. Nine harvest paths are shipped today, and every one of them shares the same sync machinery: incremental sync, content-hash delta detection, streaming page-by-page processing, and stale dataset marking.

Type	Selector	Serves	Example portals
CKAN	--type ckan	CKAN action API portals	dati.comune.milano.it, catalog.data.gov, dati.gov.it
DCAT udata REST	--type dcat (default profile)	udata-flavored DCAT-AP portals	data.gouv.fr, data.public.lu
DCAT SPARQL	--type dcat --profile sparql	SPARQL-backed DCAT-AP catalogs	data.europa.eu
Project Open Data	--type dcat --profile static_json	Static DCAT-US data.json catalogs	data.va.gov, census.gov, justice.gov
Socrata	--type socrata	Socrata Discovery API catalogs	data.cityofnewyork.us, data.wa.gov
OpenDataSoft	--type opendatasoft	OpenDataSoft Explore API v2.1 catalogs	opendata.paris.fr, data.economie.gouv.fr
ArcGIS Hub	--type arcgis	ArcGIS Hub Search API catalogs	opendata.dc.gov, opendata.gis.utah.gov
OGC Records	--type ogc_records	CSW 2.0.2 / GeoNetwork catalogues	EMODnet, British Geological Survey
STAC	--type stac	Collection-level STAC APIs	Copernicus Data Space, Canada DataCube

Every client preserves the complete source metadata payload, so downstream features like resource-schema extraction keep working no matter which portal a dataset came from.

Quick examples

# CKAN
ceres harvest https://dati.comune.milano.it --metadata-only

# DCAT udata REST
ceres harvest https://data.public.lu --type dcat --metadata-only

# SPARQL-backed DCAT (e.g. the EU aggregator)
ceres harvest https://data.europa.eu --type dcat --profile sparql --metadata-only

# Static Project Open Data / DCAT-US data.json
ceres harvest https://www.data.va.gov/data.json --type dcat --profile static_json --metadata-only

# Socrata Discovery API
ceres harvest https://data.cityofnewyork.us --type socrata --metadata-only

# OpenDataSoft Explore API
ceres harvest https://opendata.paris.fr --type opendatasoft --metadata-only

# ArcGIS Hub Search API
ceres harvest https://opendata.dc.gov --type arcgis --metadata-only

# OGC CSW 2.0.2
ceres harvest https://emodnet.ec.europa.eu/geonetwork/emodnet/eng/csw --type ogc_records --metadata-only

# STAC Collections (never individual Items)
ceres harvest https://stac.dataspace.copernicus.eu/v1/ --type stac --metadata-only

Or configure them once in portals.toml and harvest in batch:

[[portals]]
name = "milano"
url = "https://dati.comune.milano.it"
type = "ckan"
language = "it"

[[portals]]
name = "nyc"
url = "https://data.cityofnewyork.us"
type = "socrata"
language = "en"

[[portals]]
name = "europa"
url = "https://data.europa.eu"
type = "dcat"
profile = "sparql"
language = "en"

See examples/portals.toml for a larger, curated configuration set.

Portal-specific notes

CKAN

• Modified-since filtering for incremental syncs
• Adaptive page sizing: on timeouts the page size is quartered (1000 → 250 → 62 → 15 → 10) and the per-request timeout grows, which handles portals that choke on large responses
• catalog.data.gov relocated its CKAN API to api.gsa.gov and requires DATA_GOV_API_KEY (free key at api.data.gov/signup)

DCAT udata REST

• Streams paginated JSON-LD catalog pages
• Resolves multilingual fields according to the configured portal language
• Waits out 429 rate limits with an escalating per-page cooldown instead of stopping with partial results

DCAT SPARQL

• Pages through the catalog with LIMIT/OFFSET queries and deduplicates by dataset URI
• Localized title/description selection: requested language → sibling language → English → untagged
• Endpoint defaults to {url}/sparql; override with sparql_endpoint

Project Open Data (data.json)

• Handles static DCAT-US catalogs published as one JSON document (US federal agencies)
• Hard 256 MiB response limit by default; override with CERES_STATIC_JSON_MAX_BYTES
• Incremental runs filter the catalog locally using modified

Socrata Discovery API

• Harvests the paginated /api/catalog/v1 endpoint, scoped to the portal domain and limited to dataset assets
• Uses updatedAt ordering for incremental synchronization
• HTML descriptions are converted to plain text for search and embedding; the original HTML stays in the raw metadata
• No credentials required for public reads; set SOCRATA_APP_TOKEN for higher rate limits

OpenDataSoft Explore API

• Harvests the paginated /api/explore/v2.1/catalog/datasets endpoint (100 datasets per page, the API maximum)
• Normalizes title, description, themes, keywords, license, publisher, and modified from metas.default; the complete catalog entry, including fields schema hints, stays in the raw metadata
• Catalogs deeper than the API’s 10,000-row pagination window (e.g. the data.opendatasoft.com federation hub) are walked with a keyset cursor on modified
• Incremental sync filters server-side with an ODSQL where=modified >= date'...' clause
• No credentials required for public reads; set ODS_API_KEY for higher quotas

ArcGIS Hub Search API

• Harvests the paginated /api/search/v1/collections/dataset/items endpoint (100 items per page, the API maximum)
• Normalizes title, description (falling back to the item snippet), categories, tags, license, publisher, and modified; the complete catalog feature, including geometry and service metadata, stays in the raw metadata
• Catalog entries mix hosted services (Feature/Image/Map Services) and file items (CSV, shapefile): the normalized dataset URL is always the Hub landing page — a service endpoint is a queryable API, not a file download
• Catalogs deeper than the API’s 10,000-row result window are walked with a keyset cursor on modified; incremental sync filters server-side with filter=modified >= <epoch millis>
• Hub sites with an empty catalogV2 item scope are rejected because their search endpoint returns global ArcGIS content rather than datasets belonging to that portal
• No credentials required for public reads

OGC Records / CSW

• Discovers GetRecords and GetRecordById bindings from CSW 2.0.2 GetCapabilities
• Streams bounded record windows and preserves each complete source XML record
• Resolves localized titles and abstracts, spatial/temporal extents, contacts, constraints, and online resources
• Classifies datasets, series, services, and maps instead of treating every catalog record as a downloadable dataset

STAC

• Discovers the Collections endpoint from the landing page rel=data link and follows rel=next pagination
• Stores one Ceres series record per STAC Collection, preserving the complete Collection JSON (extent, providers, assets, summaries, and extension fields)
• Never follows Collection items links; item/scene-level indexing is intentionally out of scope
• Supports STAC 1.0 and 1.1 APIs; public reads require no credentials unless the catalog itself is private

Coverage validation set (v0.6.0)

The v0.6.0 milestone is proven by a small, reproducible set of portals — one per harvest profile — rather than by code paths alone. Every entry below is present in examples/portals.toml (shipped enabled = false) so you can reproduce a metadata-only validation without guessing the type/profile settings:

# Preview (no DB writes), then a real metadata-only harvest
ceres harvest --config examples/portals.toml --portal ann-arbor --metadata-only --dry-run
ceres harvest --config examples/portals.toml --portal ann-arbor --metadata-only

Profile	Selector	portals.toml name	Approx. datasets	Language	Credentials	Use as
CKAN	--type ckan	ann-arbor	~95	en	none	CI smoke
DCAT udata REST	--type dcat	luxembourg	~2,490	fr	none	Manual smoke
DCAT SPARQL	--type dcat --profile sparql	slovakia	~11,680	sk	none	Manual smoke
Project Open Data	--type dcat --profile static_json	us-national-archives	~84	en	none	CI smoke
Socrata	--type socrata	new-brunswick	~310	en	optional SOCRATA_APP_TOKEN	Manual smoke
OpenDataSoft	--type opendatasoft	paris	~490	fr	optional ODS_API_KEY	Manual smoke
ArcGIS Hub	--type arcgis	washington-dc	~1,500	en	none	Manual smoke
OGC CSW	--type ogc_records	copernicus-marine-csw	~310	en	none	Manual smoke
STAC	--type stac	canada-datacube-stac	~50 collections	en	none	CI smoke
SPARQL (scale)	--type dcat --profile sparql	eu-open-data	~2M	en	none	Scale only

Notes:

• Use as classifies each portal for validation: CI smoke portals are small, fast, and reliable enough for automated checks; Manual smoke portals are moderate and best run by hand; Scale only portals (e.g. data.europa.eu) prove throughput but should never be the sole proof that a client works.
• Counts are approximate and drift as portals publish; treat them as an order of magnitude, not an assertion.
• No entry requires embedding credentials. SOCRATA_APP_TOKEN and ODS_API_KEY only raise public rate limits and can be omitted.
• Each profile has a matching opt-in live smoke test (most with a CERES_*_SMOKE_URL override; STAC uses CERES_STAC_*_URL and the OGC CSW smokes hard-code their endpoints), documented in Harvesting → Opt-in live smoke tests.

The published index

The catalog Ceres maintains is published as the Ceres Open Data Index on Hugging Face: curated Parquet snapshots with versioned manifests, SHA-256 checksums, coverage/quality reports, and snapshot-to-snapshot changelogs.

What’s next

Coverage keeps expanding. The v0.6.0 milestone shipped the OGC CSW and collection-level STAC clients and the coverage validation set above; the v0.7.0 milestone turns to resource-level metadata depth — making distribution/resource metadata first-class in published snapshots and the API.

Want a portal that none of the current clients cover? The client layer is trait-based and designed for extension — see Contributing or open an issue.