Dataset Discovery

OpenData can automatically discover and configure datasets from URLs. Point it at a data source and it handles the rest.

From URL to dataset config

1. Submit a URL pointing to a data source
2. OpenData analyzes the URL and content
3. It selects the appropriate connector and parser
4. Schema is inferred from the data
5. A dataset configuration is generated

Supported Sources

Discovery works with various data source types:

PDF documents are processed using AI-powered extraction. The system renders each page as an image and uses a vision model to extract tabular data. Results are cached so repeated ingestion doesn't re-process pages.

Using the API

Discover (Preview)

The discover endpoint analyzes a URL and returns a suggested configuration:

POST /v1/discoveries
Content-Type: application/json
Authorization: Bearer YOUR_API_KEY

{
  "url": "https://api.census.gov/data/2022/acs/acs5?get=NAME,B01001_001E&for=state:*"
}

{
  "provider": "census",
  "name": "acs5-state-populations",
  "connector": "opendata/census",
  "source_url": "...",
  "schema": {
    "columns": [
      {"name": "NAME", "type": "string"},
      {"name": "B01001_001E", "type": "integer"},
      {"name": "state", "type": "string"}
    ]
  },
  "sample_data": [...]
}

Add (Create Dataset)

To actually create the dataset, use the add endpoint or accept the discovery result:

POST /v1/datasets
Content-Type: application/json
Authorization: Bearer YOUR_API_KEY

{
  "url": "https://api.census.gov/data/2022/acs/acs5?get=NAME,B01001_001E&for=state:*",
  "provider": "census",
  "name": "State Populations 2022"
}

CLI Usage

The OpenData CLI gives you a few commands for working with datasets:

discover

Preview what would be created without actually ingesting anything. This is useful for checking that OpenData correctly identifies the schema, connector, and sample data before committing.

opendata discover https://api.census.gov/data/2022/acs/acs5?get=NAME,B01001_001E&for=state:*

add

Runs discovery AND creates the dataset AND triggers ingestion in one step. If you already know the URL is good, this is the fastest path from URL to queryable data.

opendata add https://api.census.gov/data/2022/acs/acs5?get=NAME,B01001_001E&for=state:*

status

Shows the status of all datasets (pending, ready, error). Handy for checking whether ingestion has finished.

opendata status

sync

Syncs YAML-defined datasets to the database. If you've edited a dataset.yaml file directly, run this to pick up the changes.

opendata sync

Schema Inference

During discovery, OpenData analyzes the data to infer column types:

• String - Text values
• Integer - Whole numbers
• Float - Decimal numbers
• Boolean - True/false values
• Date - Date values (parsed from various formats)
• Datetime - Timestamp values

Customizing Discovery

You can override the inferred configuration:

POST /v1/datasets
Content-Type: application/json

{
  "url": "https://example.com/data.csv",
  "provider": "custom",
  "name": "My Dataset",
  "description": "Custom description",
  "connector_config": {
    "delimiter": ";",
    "encoding": "utf-8"
  }
}

Troubleshooting

Discovery fails with "Unsupported URL"

The URL format isn't recognized. Try:

• Use a direct link to a CSV or JSON file
• Use a known API endpoint (BLS, Census, FRED)
• Check that the URL is publicly accessible

Schema inference is wrong

Override the schema in the create request or edit the dataset.yaml after creation.

Data is not updating

Trigger a re-sync: POST /v1/datasets/{provider}/{dataset}/sync

Related

• API Reference - Discovery endpoints
• Search Guide - Find existing datasets