LincPayandClaude Opus 5
ec057a8124
Commodities 7 -> 21 and a district_commodity table holding a per-district basket of 60 pairs, replacing the district x commodity cross product that would have been 252 mostly-impossible calls a night. District, not zone: Kolar and Belagavi are both SOUTH and trade almost disjoint baskets. Only the four original cereals had verified Agmarknet spellings, so the rest carry sourceVerified = false until POST /admin/ingestion/probe confirms them. The probe separates a wrong spelling from an absent local market by cross-referencing which districts and commodities returned rows anywhere — a distinction that matters because both fail silently and identically. Three district names were costing us every price in their district: Nasik -> Nashik, Ahmadnagar -> Ahilyanagar, Belgaum -> Belagavi. The last is the nastiest: Belgaum still answers with 1000 rows, but they stop in May 2026 while Belagavi is current, so a retired name serving a frozen archive reads as a healthy feed. Only the arrival dates give it away. data.gov.in meters by quota per window, so the limit is reached by the run and not by any one call: a 62-pair run straight after a 62-pair probe returned 429 on all 62 and recorded the whole ingestion FAILED with zero rows. DataGovClient now retries with a 45s backoff, long because the window is minutes not seconds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
65 lines
2.3 KiB
Java
65 lines
2.3 KiB
Java
package com.lincfusion.udyamlinc.master;
|
|
|
|
import jakarta.persistence.*;
|
|
import lombok.*;
|
|
|
|
/**
|
|
* What a district actually trades — the basket ingestion pulls for it.
|
|
*
|
|
* <p>Without this table ingestion is a cross product: every active district
|
|
* against every active commodity. That is affordable at four cereals and four
|
|
* districts and stops being affordable the moment fruit and vegetables are
|
|
* tracked across six states, because most of the pairs it generates cannot
|
|
* exist. Kolar has no soyabean arrivals, Kamrup no grapes, Nashik no chana.
|
|
*
|
|
* <p>The cost is not really the wasted calls, it is what they do to the
|
|
* freshness screen. Agmarknet answers a filter that matches nothing with
|
|
* HTTP 200 and an empty record list, so a structurally impossible pair and a
|
|
* genuinely broken filter produce byte-identical responses. Under a cross
|
|
* product the majority of pairs return empty on a healthy night, and a
|
|
* commodity whose {@code sourceCode} is misspelt disappears into that crowd.
|
|
* Pulling only what a district trades makes an empty result mean something
|
|
* again: either arrivals genuinely stopped, or something is wrong.
|
|
*
|
|
* <p>Zone is deliberately not the grain. Kolar and Belagavi are both SOUTH and
|
|
* trade almost disjoint baskets — tomato and mango against onion and maize.
|
|
*/
|
|
@Entity
|
|
@Table(name = "district_commodity", indexes = {
|
|
@Index(name = "ix_dc_district", columnList = "districtId"),
|
|
@Index(name = "ix_dc_pair", columnList = "districtId,commodityId", unique = true),
|
|
})
|
|
@Getter
|
|
@Setter
|
|
@NoArgsConstructor
|
|
@AllArgsConstructor
|
|
@Builder
|
|
public class DistrictCommodity {
|
|
|
|
@Id
|
|
@GeneratedValue(strategy = GenerationType.IDENTITY)
|
|
private Long id;
|
|
|
|
@Column(nullable = false)
|
|
private Long districtId;
|
|
|
|
@Column(nullable = false)
|
|
private Long commodityId;
|
|
|
|
/**
|
|
* True where this district is a reference market for the commodity — Nashik
|
|
* for onion, Kolar for tomato. Not used to decide what to pull; it marks the
|
|
* rows whose absence is worth a louder complaint, because a national onion
|
|
* signal that has lost Nashik has lost the market that sets the price.
|
|
*/
|
|
private Boolean benchmark;
|
|
|
|
private Boolean active;
|
|
|
|
@PrePersist
|
|
void onCreate() {
|
|
if (active == null) active = Boolean.TRUE;
|
|
if (benchmark == null) benchmark = Boolean.FALSE;
|
|
}
|
|
}
|