fix(ingestion): make the pipeline actually run against live feeds
First end-to-end run against Postgres and the real government APIs, which surfaced four defects that static checks could not. - Agmarknet pointed at 9ef84268-…, "Current Daily Price": a today-only snapshot holding one arrival_date, in which none of the four pilot districts appear. Switched to 35985678-…, the historical resource. - Filter fields were lowercase. This resource matches exactly on the capitalised name; the lowercase form is tokenised, so "Madhya Pradesh" also matched Andhra, Himachal and Uttar Pradesh — wrong-state prices written under a pilot district, with freshness looking healthy. - Rows arrive unordered and the lookback filter is client side, so a page was mostly years-old data and every commodity wrote zero weeks. Added sort[Arrival_Date]=desc. - ExternalFeedConfig injected a RestClient.Builder bean Boot does not expose, so the context never built and the app could not start. Also: Open-Meteo meters by request cost, so the 30-year baseline call exhausted the minutely allowance and 429'd two districts out of their climate data. Rate-limited calls now retry with backoff, while genuine failures still fail once and are reported. FODDER moves to MANUAL — Agmarknet carries Green Fodder, but every recent row is a single market in Ambala, Haryana. Result: 16,000 rows fetched, 202 market weeks and 52 climate weeks written, 0 failed calls; 142 scored, 33 trip-wires fired. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -147,7 +147,11 @@ public class UdyamDataSeeder implements CommandLineRunner {
|
||||
commodity("GRAM", "Gram", "चना", "AGMARKNET", "Bengal Gram(Gram)(Whole)", "QUINTAL"),
|
||||
commodity("EGG", "Egg", "अंडा", "NECC", "Bhopal", "HUNDRED"),
|
||||
commodity("MILK", "Milk", "दूध", "DAIRY_FEDERATION", null, "LITRE"),
|
||||
commodity("FODDER", "Green fodder", "हरा चारा", "AGMARKNET", "Green Fodder", "QUINTAL")));
|
||||
// Agmarknet carries Green Fodder, but every recent row is one market
|
||||
// in Ambala, Haryana — nothing for the pilot districts. Left as a
|
||||
// manual-entry commodity rather than a nightly call that can only
|
||||
// ever return an empty page.
|
||||
commodity("FODDER", "Green fodder", "हरा चारा", "MANUAL", null, "QUINTAL")));
|
||||
log.info("Seeded {} commodities", commodityRepository.count());
|
||||
}
|
||||
|
||||
@@ -283,7 +287,10 @@ public class UdyamDataSeeder implements CommandLineRunner {
|
||||
private static final BigDecimal PINNED_SHORTFALL = new BigDecimal("12400");
|
||||
|
||||
private void seedPortfolio() {
|
||||
if (enterpriseRepository.count() > 0) return;
|
||||
if (enterpriseRepository.count() > 0) {
|
||||
backfillPortfolio();
|
||||
return;
|
||||
}
|
||||
|
||||
Map<String, Sector> sectors = new HashMap<>();
|
||||
sectorRepository.findAll().forEach(s -> sectors.put(s.getCode(), s));
|
||||
@@ -403,6 +410,24 @@ public class UdyamDataSeeder implements CommandLineRunner {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* A step added to the seeder after a database was first populated never
|
||||
* reaches it — the portfolio guard short-circuits every per-enterprise step
|
||||
* along with the enterprises themselves. Loan records arrived with Phase C,
|
||||
* so a database seeded before then holds none and {@code emi_delay} resolves
|
||||
* {@code NO_CBS} forever, which reads as "the bank sent us nothing" rather
|
||||
* than "this seeder never ran". Each step below carries its own
|
||||
* per-enterprise guard, so this is safe on every boot.
|
||||
*/
|
||||
private void backfillPortfolio() {
|
||||
long before = loanRecordRepository.count();
|
||||
enterpriseRepository.findAll().forEach(this::seedLoans);
|
||||
long added = loanRecordRepository.count() - before;
|
||||
if (added > 0) {
|
||||
log.info("Backfilled {} loan records onto an already-seeded portfolio", added);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The institutional side of the record — what a nightly CBS extract would
|
||||
* carry. DPD tracks the band, because in the demo the band is the ground
|
||||
|
||||
Reference in New Issue
Block a user