Ingestor

Local cache curator

Pulls deltas from each catalog dataset into a local SQLite cache. Enables cross-dataset SQL JOINs that Socrata's SoQL can't express. Surfaces source pill (cache | live | cache-fallback) on every tool envelope so the user/judge can see provenance.

What this agent actually does, step by step.

1. 01

   Reads catalog from app/lib/catalog.ts

2. 02

   For each dataset, paginated SODA pulls (limit 1000, $offset) since last_ingested

3. 03

   UPSERTs into data/cache.db keyed on dataset primary key

4. 04

   Updates data/cache_meta.json with timestamp + row count + schema hash

5. 05

   Writes back to repo via github-actions[bot] commit (dev mode also supports manual run)

6. 06

   Tool calls in app/lib/socrata.ts try cache first; fall through to live with cache-fallback pill

Inputs & outputs.

Where this agent lives in the codebase.