feat(ingestion): per-district commodity baskets, and a probe that verifies them
Commodities 7 -> 21 and a district_commodity table holding a per-district basket of 60 pairs, replacing the district x commodity cross product that would have been 252 mostly-impossible calls a night. District, not zone: Kolar and Belagavi are both SOUTH and trade almost disjoint baskets. Only the four original cereals had verified Agmarknet spellings, so the rest carry sourceVerified = false until POST /admin/ingestion/probe confirms them. The probe separates a wrong spelling from an absent local market by cross-referencing which districts and commodities returned rows anywhere — a distinction that matters because both fail silently and identically. Three district names were costing us every price in their district: Nasik -> Nashik, Ahmadnagar -> Ahilyanagar, Belgaum -> Belagavi. The last is the nastiest: Belgaum still answers with 1000 rows, but they stop in May 2026 while Belagavi is current, so a retired name serving a frozen archive reads as a healthy feed. Only the arrival dates give it away. data.gov.in meters by quota per window, so the limit is reached by the run and not by any one call: a 62-pair run straight after a 62-pair probe returned 429 on all 62 and recorded the whole ingestion FAILED with zero rows. DataGovClient now retries with a 45s backoff, long because the window is minutes not seconds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,64 @@
|
||||
package com.lincfusion.udyamlinc.master;
|
||||
|
||||
import jakarta.persistence.*;
|
||||
import lombok.*;
|
||||
|
||||
/**
|
||||
* What a district actually trades — the basket ingestion pulls for it.
|
||||
*
|
||||
* <p>Without this table ingestion is a cross product: every active district
|
||||
* against every active commodity. That is affordable at four cereals and four
|
||||
* districts and stops being affordable the moment fruit and vegetables are
|
||||
* tracked across six states, because most of the pairs it generates cannot
|
||||
* exist. Kolar has no soyabean arrivals, Kamrup no grapes, Nashik no chana.
|
||||
*
|
||||
* <p>The cost is not really the wasted calls, it is what they do to the
|
||||
* freshness screen. Agmarknet answers a filter that matches nothing with
|
||||
* HTTP 200 and an empty record list, so a structurally impossible pair and a
|
||||
* genuinely broken filter produce byte-identical responses. Under a cross
|
||||
* product the majority of pairs return empty on a healthy night, and a
|
||||
* commodity whose {@code sourceCode} is misspelt disappears into that crowd.
|
||||
* Pulling only what a district trades makes an empty result mean something
|
||||
* again: either arrivals genuinely stopped, or something is wrong.
|
||||
*
|
||||
* <p>Zone is deliberately not the grain. Kolar and Belagavi are both SOUTH and
|
||||
* trade almost disjoint baskets — tomato and mango against onion and maize.
|
||||
*/
|
||||
@Entity
|
||||
@Table(name = "district_commodity", indexes = {
|
||||
@Index(name = "ix_dc_district", columnList = "districtId"),
|
||||
@Index(name = "ix_dc_pair", columnList = "districtId,commodityId", unique = true),
|
||||
})
|
||||
@Getter
|
||||
@Setter
|
||||
@NoArgsConstructor
|
||||
@AllArgsConstructor
|
||||
@Builder
|
||||
public class DistrictCommodity {
|
||||
|
||||
@Id
|
||||
@GeneratedValue(strategy = GenerationType.IDENTITY)
|
||||
private Long id;
|
||||
|
||||
@Column(nullable = false)
|
||||
private Long districtId;
|
||||
|
||||
@Column(nullable = false)
|
||||
private Long commodityId;
|
||||
|
||||
/**
|
||||
* True where this district is a reference market for the commodity — Nashik
|
||||
* for onion, Kolar for tomato. Not used to decide what to pull; it marks the
|
||||
* rows whose absence is worth a louder complaint, because a national onion
|
||||
* signal that has lost Nashik has lost the market that sets the price.
|
||||
*/
|
||||
private Boolean benchmark;
|
||||
|
||||
private Boolean active;
|
||||
|
||||
@PrePersist
|
||||
void onCreate() {
|
||||
if (active == null) active = Boolean.TRUE;
|
||||
if (benchmark == null) benchmark = Boolean.FALSE;
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user