feat(manual-entry): bulk and single entry for the feeds no API serves
Egg, milk and fodder have no usable API for the pilot districts — NECC publishes HTML, dairy rates exist only as annual state averages, and every recent Agmarknet fodder row is one market in Ambala. This is how a bank gets that data in. Upload stages a PREVIEW batch and writes nothing. It reports what would land, what was rejected and why, and which rows would overwrite an existing value. Only commit touches market_signal. Effective dating: effective_date is the date the fact describes and comes from the file; observedAt is when we learned it and is stamped at commit. The uploader cannot reach observedAt — a March rate uploaded in August must stay invisible to a March scoring run, or the model trains on data that did not exist yet. Rejections name themselves: UNKNOWN_DISTRICT, UNKNOWN_COMMODITY, FUTURE_DATE, BAD_DATE, BAD_NUMBER, UNIT_MISMATCH, PRICE_ORDER, DUPLICATE_IN_FILE, MISSING_REQUIRED. A price far from last week warns but is accepted — a genuine price shock is the event this system exists to catch, and rejecting it would be backwards. Rows are written with source=MANUAL, which the unique index keeps separate from a feed's own row for the same week, so manual data never silently replaces a government feed. Also distinguishes INSUFFICIENT_HISTORY from NO_SIGNAL in the price derivations. A newly onboarded manual feed spends its first weeks below the four-week minimum, and reporting NO_SIGNAL there reads as a broken integration rather than a series still filling up. Result: margin_squeeze and output_price_change compute for the first time — 57 of 142 enterprises, exactly the DAIRY (30) and POULTRY (27) sectors that have an output commodity. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -139,10 +139,12 @@ public class FeatureEngineeringService {
|
||||
BigDecimal weightedSum = BigDecimal.ZERO;
|
||||
int totalWeight = 0;
|
||||
boolean anyHistory = false;
|
||||
boolean anyRows = false;
|
||||
|
||||
for (SectorCommodity link : commodities) {
|
||||
List<MarketSignal> series = marketSignalRepository.findTrailing(
|
||||
districtId, link.getCommodityId(), asOfDate, observedAsOf);
|
||||
if (!series.isEmpty()) anyRows = true;
|
||||
BigDecimal change = changeVsTrailingMean(series);
|
||||
if (change == null) continue;
|
||||
anyHistory = true;
|
||||
@@ -152,7 +154,14 @@ public class FeatureEngineeringService {
|
||||
}
|
||||
|
||||
if (!anyHistory || totalWeight == 0) {
|
||||
return Computed.missing(FeatureCatalog.NO_SIGNAL);
|
||||
// "Nobody has ever sent us a price" and "the series has started but is
|
||||
// still too short to mean anything" are different facts, and only the
|
||||
// second one gets better on its own. A newly onboarded manual feed
|
||||
// reporting NO_SIGNAL reads as a broken integration for the weeks it
|
||||
// spends filling up.
|
||||
return Computed.missing(anyRows
|
||||
? FeatureCatalog.INSUFFICIENT_HISTORY
|
||||
: FeatureCatalog.NO_SIGNAL);
|
||||
}
|
||||
return Computed.of(weightedSum.divide(BigDecimal.valueOf(totalWeight), 4, RoundingMode.HALF_UP), staleness);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user