fix(ingestion): make the pipeline actually run against live feeds

First end-to-end run against Postgres and the real government APIs, which
surfaced four defects that static checks could not.

- Agmarknet pointed at 9ef84268-…, "Current Daily Price": a today-only
  snapshot holding one arrival_date, in which none of the four pilot
  districts appear. Switched to 35985678-…, the historical resource.
- Filter fields were lowercase. This resource matches exactly on the
  capitalised name; the lowercase form is tokenised, so "Madhya Pradesh"
  also matched Andhra, Himachal and Uttar Pradesh — wrong-state prices
  written under a pilot district, with freshness looking healthy.
- Rows arrive unordered and the lookback filter is client side, so a page
  was mostly years-old data and every commodity wrote zero weeks. Added
  sort[Arrival_Date]=desc.
- ExternalFeedConfig injected a RestClient.Builder bean Boot does not
  expose, so the context never built and the app could not start.

Also: Open-Meteo meters by request cost, so the 30-year baseline call
exhausted the minutely allowance and 429'd two districts out of their
climate data. Rate-limited calls now retry with backoff, while genuine
failures still fail once and are reported.

FODDER moves to MANUAL — Agmarknet carries Green Fodder, but every recent
row is a single market in Ambala, Haryana.

Result: 16,000 rows fetched, 202 market weeks and 52 climate weeks written,
0 failed calls; 142 scored, 33 trip-wires fired.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
LincPay
2026-08-06 14:01:25 +05:30
co-authored by Claude Opus 5
parent 1b9b02f2c7
commit 20879b5904
5 changed files with 121 additions and 20 deletions
+23 -6
View File
@@ -70,13 +70,25 @@ udyam.ingestion.http.read-timeout-ms=30000
# Register at https://data.gov.in for a key; without one the run records SKIPPED.
udyam.ingestion.datagov.api-key=${DATAGOV_API_KEY:}
udyam.ingestion.datagov.base-url=https://api.data.gov.in/resource
# "Variety-wise Daily Market Prices Data of Commodity" (Agmarknet)
udyam.ingestion.datagov.resource-id=${DATAGOV_RESOURCE_ID:9ef84268-d588-465a-a308-a864a43d0070}
# "Variety-wise Daily Market Prices Data of Commodity" — the historical Agmarknet
# resource, ~81M rows, current to yesterday. NOT 9ef84268-…, which is "Current
# Daily Price": a today-only snapshot carrying a single arrival_date, from which
# no week-over-week history can be built and in which none of the four pilot
# districts appear.
udyam.ingestion.datagov.resource-id=${DATAGOV_RESOURCE_ID:35985678-0d79-46b4-9ed6-6f13308a1d24}
udyam.ingestion.datagov.page-limit=1000
# Filter names must match the resource's own field names; they differ by resource version.
udyam.ingestion.datagov.field.state=state
udyam.ingestion.datagov.field.district=district
udyam.ingestion.datagov.field.commodity=commodity
# Filter names must match the resource's own field names; they differ by resource
# version. This resource capitalises them, and — unlike the current-price
# resource — matches exactly on the bare name. Appending ".keyword" here returns
# zero rows; omitting the capital returns rows from every state whose name shares
# a token, so "Madhya Pradesh" would silently pull Andhra, Himachal and Uttar.
udyam.ingestion.datagov.field.state=State
udyam.ingestion.datagov.field.district=District
udyam.ingestion.datagov.field.commodity=Commodity
# Rows come back in arbitrary order and the lookback filter is applied client
# side, so without a descending sort a 1000-row page is mostly years-old data and
# every commodity reports zero weeks written.
udyam.ingestion.datagov.sort-field=Arrival_Date
udyam.ingestion.mandi.lookback-weeks=12
# Climate — Open-Meteo reanalysis archive. No key. The archive trails real time,
@@ -88,6 +100,11 @@ udyam.ingestion.climate.lag-days=5
udyam.ingestion.climate.heat-threshold-c=40
udyam.ingestion.climate.baseline-from=1995-01-01
udyam.ingestion.climate.baseline-to=2024-12-31
# Open-Meteo meters by request cost, so the 30-year baseline call alone can trip
# the minutely limit and 429 the district calls queued behind it. A rate-limited
# call waits and retries; anything else fails once and is reported.
udyam.ingestion.climate.retry-attempts=3
udyam.ingestion.climate.retry-backoff-ms=20000
# ── Feature store (ARCHITECTURE Phase C) ────────────────────
# Window sizes for the derivations. A level is not a feature; every one of these