Harvesting Is The Hard Part
Open data work usually breaks on synchronization, portal quirks, and stale records before search even starts.
Ceres
One synchronized catalog from 300+ open data portals. Harvest first — embed, search, and publish when you're ready.
Open data is scattered across thousands of portals speaking different APIs. Ceres harvests CKAN, DCAT, SPARQL, data.json, Socrata, OpenDataSoft, and ArcGIS Hub sources into one synchronized PostgreSQL catalog — then layers embeddings, semantic search, exports, and a public index on top, only when you want them.
Named after the Roman goddess of harvest and agriculture.
300+portals in sync
2M+datasets harvested
6portal API families
1.85M+datasets in the published index
Why Ceres
Section titled “Why Ceres”Portals Are Fragmented
CKAN, DCAT, SPARQL, static data.json, Socrata, OpenDataSoft, and ArcGIS Hub portals expose different capabilities, languages, and reliability profiles. Ceres speaks all of them.
Embedding Should Be Optional
Many teams need a trustworthy harvested catalog first, then decide later whether to add local or hosted embeddings.
Ceres is designed around that order of work: harvest first, embed later if useful, search once the catalog is ready.
The published Open Data Index is one downstream product of the pipeline: harvest and normalize first, then optionally embed, search, export, and visualize.
See It In Action
Section titled “See It In Action”Features and Scale
Section titled “Features and Scale”Harvest First
Stream metadata from CKAN, DCAT (udata REST, SPARQL, static data.json), Socrata, OpenDataSoft, and ArcGIS Hub portals into PostgreSQL, track sync history, detect stale datasets, and keep the catalog current even when embeddings are disabled.
Decoupled Pipeline
Harvesting and embedding are separate services. Run metadata-only syncs, backfill embeddings later, or switch provider without re-harvesting your sources.
Local Embeddings By Default
When you do want vectors, Ollama gives you a local zero-cost path. Gemini and OpenAI remain supported, but the project no longer assumes cloud embeddings are required.
Operations Layer
- CLI: harvest, embed, search, export, stats
- API: Axum server with Swagger UI
- Jobs: database-backed harvest queue with retries
- Exports: JSON, JSONL, CSV, and curated Parquet snapshots with manifests, checksums, and changelogs
Architecture
Section titled “Architecture”
Operational Model
Section titled “Operational Model”Step 1: Harvest
Use ceres harvest to populate the catalog, optionally in metadata-only mode. Incremental sync, delta detection, and stale marking reduce churn and keep memory bounded.
Step 2: Embed
Run ceres embed when you want vectors. Ollama is the preferred local option; hosted providers are still available for teams that want them.
Step 3: Search, Export, Publish
Search, API access, Hugging Face snapshots, and downstream analytics all build on the same harvested catalog.
Roadmap
Section titled “Roadmap”Now (v0.6.0)
Coverage foundations: OpenDataSoft Explore, ArcGIS Hub, Project Open Data data.json, OGC CSW 2.0.2, and collection-level STAC harvest paths — plus opt-in per-profile live smoke checks and a reproducible metadata-only coverage validation set.
Next (v0.7.0)
Resource-level metadata depth: first-class distribution and schema tracking across every harvest path, surfaced in published snapshots and the API.
Shipped (v0.5.0)
A trustable published index: versioned snapshot manifests, integrity checksums, coverage and quality reports, duplicate/source-alias semantics, and snapshot changelogs.