feat(manual-entry): bulk and single entry for the feeds no API serves

Egg, milk and fodder have no usable API for the pilot districts — NECC
publishes HTML, dairy rates exist only as annual state averages, and every
recent Agmarknet fodder row is one market in Ambala. This is how a bank
gets that data in.

Upload stages a PREVIEW batch and writes nothing. It reports what would
land, what was rejected and why, and which rows would overwrite an
existing value. Only commit touches market_signal.

Effective dating: effective_date is the date the fact describes and comes
from the file; observedAt is when we learned it and is stamped at commit.
The uploader cannot reach observedAt — a March rate uploaded in August
must stay invisible to a March scoring run, or the model trains on data
that did not exist yet.

Rejections name themselves: UNKNOWN_DISTRICT, UNKNOWN_COMMODITY,
FUTURE_DATE, BAD_DATE, BAD_NUMBER, UNIT_MISMATCH, PRICE_ORDER,
DUPLICATE_IN_FILE, MISSING_REQUIRED. A price far from last week warns but
is accepted — a genuine price shock is the event this system exists to
catch, and rejecting it would be backwards.

Rows are written with source=MANUAL, which the unique index keeps separate
from a feed's own row for the same week, so manual data never silently
replaces a government feed.

Also distinguishes INSUFFICIENT_HISTORY from NO_SIGNAL in the price
derivations. A newly onboarded manual feed spends its first weeks below
the four-week minimum, and reporting NO_SIGNAL there reads as a broken
integration rather than a series still filling up.

Result: margin_squeeze and output_price_change compute for the first time
— 57 of 142 enterprises, exactly the DAIRY (30) and POULTRY (27) sectors
that have an output commodity.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
LincPay
2026-08-06 14:12:45 +05:30
co-authored by Claude Opus 5
parent 20879b5904
commit 5401496a64
11 changed files with 943 additions and 1 deletions
@@ -139,10 +139,12 @@ public class FeatureEngineeringService {
BigDecimal weightedSum = BigDecimal.ZERO;
int totalWeight = 0;
boolean anyHistory = false;
boolean anyRows = false;
for (SectorCommodity link : commodities) {
List<MarketSignal> series = marketSignalRepository.findTrailing(
districtId, link.getCommodityId(), asOfDate, observedAsOf);
if (!series.isEmpty()) anyRows = true;
BigDecimal change = changeVsTrailingMean(series);
if (change == null) continue;
anyHistory = true;
@@ -152,7 +154,14 @@ public class FeatureEngineeringService {
}
if (!anyHistory || totalWeight == 0) {
return Computed.missing(FeatureCatalog.NO_SIGNAL);
// "Nobody has ever sent us a price" and "the series has started but is
// still too short to mean anything" are different facts, and only the
// second one gets better on its own. A newly onboarded manual feed
// reporting NO_SIGNAL reads as a broken integration for the weeks it
// spends filling up.
return Computed.missing(anyRows
? FeatureCatalog.INSUFFICIENT_HISTORY
: FeatureCatalog.NO_SIGNAL);
}
return Computed.of(weightedSum.divide(BigDecimal.valueOf(totalWeight), 4, RoundingMode.HALF_UP), staleness);
}