fix(ingestion): make the pipeline actually run against live feeds
First end-to-end run against Postgres and the real government APIs, which surfaced four defects that static checks could not. - Agmarknet pointed at 9ef84268-…, "Current Daily Price": a today-only snapshot holding one arrival_date, in which none of the four pilot districts appear. Switched to 35985678-…, the historical resource. - Filter fields were lowercase. This resource matches exactly on the capitalised name; the lowercase form is tokenised, so "Madhya Pradesh" also matched Andhra, Himachal and Uttar Pradesh — wrong-state prices written under a pilot district, with freshness looking healthy. - Rows arrive unordered and the lookback filter is client side, so a page was mostly years-old data and every commodity wrote zero weeks. Added sort[Arrival_Date]=desc. - ExternalFeedConfig injected a RestClient.Builder bean Boot does not expose, so the context never built and the app could not start. Also: Open-Meteo meters by request cost, so the 30-year baseline call exhausted the minutely allowance and 429'd two districts out of their climate data. Rate-limited calls now retry with backoff, while genuine failures still fail once and are reported. FODDER moves to MANUAL — Agmarknet carries Green Fodder, but every recent row is a single market in Ambala, Haryana. Result: 16,000 rows fetched, 202 market weeks and 52 climate weeks written, 0 failed calls; 142 scored, 33 trip-wires fired. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -70,13 +70,25 @@ udyam.ingestion.http.read-timeout-ms=30000
|
||||
# Register at https://data.gov.in for a key; without one the run records SKIPPED.
|
||||
udyam.ingestion.datagov.api-key=${DATAGOV_API_KEY:}
|
||||
udyam.ingestion.datagov.base-url=https://api.data.gov.in/resource
|
||||
# "Variety-wise Daily Market Prices Data of Commodity" (Agmarknet)
|
||||
udyam.ingestion.datagov.resource-id=${DATAGOV_RESOURCE_ID:9ef84268-d588-465a-a308-a864a43d0070}
|
||||
# "Variety-wise Daily Market Prices Data of Commodity" — the historical Agmarknet
|
||||
# resource, ~81M rows, current to yesterday. NOT 9ef84268-…, which is "Current
|
||||
# Daily Price": a today-only snapshot carrying a single arrival_date, from which
|
||||
# no week-over-week history can be built and in which none of the four pilot
|
||||
# districts appear.
|
||||
udyam.ingestion.datagov.resource-id=${DATAGOV_RESOURCE_ID:35985678-0d79-46b4-9ed6-6f13308a1d24}
|
||||
udyam.ingestion.datagov.page-limit=1000
|
||||
# Filter names must match the resource's own field names; they differ by resource version.
|
||||
udyam.ingestion.datagov.field.state=state
|
||||
udyam.ingestion.datagov.field.district=district
|
||||
udyam.ingestion.datagov.field.commodity=commodity
|
||||
# Filter names must match the resource's own field names; they differ by resource
|
||||
# version. This resource capitalises them, and — unlike the current-price
|
||||
# resource — matches exactly on the bare name. Appending ".keyword" here returns
|
||||
# zero rows; omitting the capital returns rows from every state whose name shares
|
||||
# a token, so "Madhya Pradesh" would silently pull Andhra, Himachal and Uttar.
|
||||
udyam.ingestion.datagov.field.state=State
|
||||
udyam.ingestion.datagov.field.district=District
|
||||
udyam.ingestion.datagov.field.commodity=Commodity
|
||||
# Rows come back in arbitrary order and the lookback filter is applied client
|
||||
# side, so without a descending sort a 1000-row page is mostly years-old data and
|
||||
# every commodity reports zero weeks written.
|
||||
udyam.ingestion.datagov.sort-field=Arrival_Date
|
||||
udyam.ingestion.mandi.lookback-weeks=12
|
||||
|
||||
# Climate — Open-Meteo reanalysis archive. No key. The archive trails real time,
|
||||
@@ -88,6 +100,11 @@ udyam.ingestion.climate.lag-days=5
|
||||
udyam.ingestion.climate.heat-threshold-c=40
|
||||
udyam.ingestion.climate.baseline-from=1995-01-01
|
||||
udyam.ingestion.climate.baseline-to=2024-12-31
|
||||
# Open-Meteo meters by request cost, so the 30-year baseline call alone can trip
|
||||
# the minutely limit and 429 the district calls queued behind it. A rate-limited
|
||||
# call waits and retries; anything else fails once and is reported.
|
||||
udyam.ingestion.climate.retry-attempts=3
|
||||
udyam.ingestion.climate.retry-backoff-ms=20000
|
||||
|
||||
# ── Feature store (ARCHITECTURE Phase C) ────────────────────
|
||||
# Window sizes for the derivations. A level is not a feature; every one of these
|
||||
|
||||
Reference in New Issue
Block a user