fix(ingestion): make the pipeline actually run against live feeds

First end-to-end run against Postgres and the real government APIs, which
surfaced four defects that static checks could not.

- Agmarknet pointed at 9ef84268-…, "Current Daily Price": a today-only
  snapshot holding one arrival_date, in which none of the four pilot
  districts appear. Switched to 35985678-…, the historical resource.
- Filter fields were lowercase. This resource matches exactly on the
  capitalised name; the lowercase form is tokenised, so "Madhya Pradesh"
  also matched Andhra, Himachal and Uttar Pradesh — wrong-state prices
  written under a pilot district, with freshness looking healthy.
- Rows arrive unordered and the lookback filter is client side, so a page
  was mostly years-old data and every commodity wrote zero weeks. Added
  sort[Arrival_Date]=desc.
- ExternalFeedConfig injected a RestClient.Builder bean Boot does not
  expose, so the context never built and the app could not start.

Also: Open-Meteo meters by request cost, so the 30-year baseline call
exhausted the minutely allowance and 429'd two districts out of their
climate data. Rate-limited calls now retry with backoff, while genuine
failures still fail once and are reported.

FODDER moves to MANUAL — Agmarknet carries Green Fodder, but every recent
row is a single market in Ambala, Haryana.

Result: 16,000 rows fetched, 202 market weeks and 52 climate weeks written,
0 failed calls; 142 scored, 33 trip-wires fired.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
LincPay
2026-08-06 14:01:25 +05:30
co-authored by Claude Opus 5
parent 1b9b02f2c7
commit 20879b5904
5 changed files with 121 additions and 20 deletions
@@ -7,6 +7,8 @@ import org.springframework.http.client.SimpleClientHttpRequestFactory;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.web.client.RestClient;
import java.time.Duration;
/**
* The HTTP client used for external feeds, and the scheduler that drives them.
*
@@ -24,11 +26,18 @@ public class ExternalFeedConfig {
@Value("${udyam.ingestion.http.read-timeout-ms:30000}")
private int readTimeoutMs;
/**
* Built from {@link RestClient#builder()} rather than an injected
* auto-configured {@code RestClient.Builder}, which Boot does not always
* expose as a bean. It also keeps this client free of any global customiser
* meant for the request path — the feed client's timeouts are deliberately
* its own.
*/
@Bean("externalFeedClient")
public RestClient externalFeedClient(RestClient.Builder builder) {
public RestClient externalFeedClient() {
SimpleClientHttpRequestFactory factory = new SimpleClientHttpRequestFactory();
factory.setConnectTimeout(connectTimeoutMs);
factory.setReadTimeout(readTimeoutMs);
return builder.requestFactory(factory).build();
factory.setConnectTimeout(Duration.ofMillis(connectTimeoutMs));
factory.setReadTimeout(Duration.ofMillis(readTimeoutMs));
return RestClient.builder().requestFactory(factory).build();
}
}