Multimodal inspection / Guide 88
Retail Shelf Video Audits Must Separate Empty From Unseen
Design retail shelf video audits that validate shelf identity, planogram revisions, timing, and visibility before turning apparent gaps into human-reviewed tasks.
Illustrative engineering guide. Examples are not client deployments, production-ready integrations, or evidence of measured results. Validate the design with your domain and security owners.
A hypothetical store associate walks a phone camera along a shelf. A shopping basket blocks two facings, a promotional display covers another, and the assistant reports three out-of-stock products. The clip also shows yesterday’s shelf label arrangement because the store changed its planogram that morning. What appears to be an inventory finding is actually a combination of occlusion and stale location mapping.
Build an advisory shelf audit that reports visible occupied space, visible empty space, and unknown space separately. A shelf gap is not proof that the store has no inventory; stock may be elsewhere. Product recognition is also distinct from stock accounting. The audit should prepare human-reviewed verification tasks rather than directly order replenishment or alter inventory quantities.
Register the Shelf, Not Just the Camera
Require store, aisle, bay, shelf level, planogram revision, capture session, source video digest, and acquisition interval. Keep presentation timestamps for each extracted frame and preserve the transformation between full frames and crops. A screenshot without a bay identity may be visually useful but cannot support a location-specific action.
Use image coordinates in pixels from the upper-left corner. Define shelf-local coordinates in millimeters from the left upright, with horizontal distance increasing right and height increasing upward. Product-facing intervals belong to that physical shelf frame. Store the scale method and registration residual so reviewers can tell whether a candidate gap might actually belong to the neighboring slot.
OpenCV’s calibration tutorial describes distortion correction and the role of known object points. Those methods support shelf registration, but a hand-held oblique view still requires appropriate pose estimation and checks. Never infer millimeter gap width directly from an uncalibrated thumbnail simply because the package size seems familiar.
Preserve the Difference Between Absence and Occlusion
A candidate detector can propose package regions and visible empty regions. A specialist registration stage maps them to the approved planogram and verifies whether each slot was actually observable. Negative evidence is conditional on visibility: no package detection behind a basket is not an empty-slot observation. Repeated frames of that same basket do not improve the missing coverage.
The W3C provenance overview provides a useful vocabulary for linking video, frame extraction, slot association, and reviewer correction. Keep the original planogram revision in the result even after a new revision is published. Reprocessing under a new map should produce a new audit version, not silently change the meaning of an earlier finding.
Location uncertainty and product-classification uncertainty are different. NIST’s uncertainty guidance discusses measurement components and covariance. A common shelf-registration offset affects many slots together; averaging detections across those slots will not remove it. Classification confidence is not a physical registration bound and should not be added to one.
Test Visibility Before Creating a Gap Candidate
This self-contained Python fixture tests the HARNESS using synthetic structured observations. It does not evaluate a trained vision model, camera, tracker, or real inspection performance. Slot coverage, registration, and freshness limits are illustrative workflow policy values, not retail standards. The code performs no real media or engine API calls.
from math import isfinite
base = dict(store="S3", bay="B7", planogram=4, source="synthetic-video",
frame="shelf-left-mm", captured_s=100, registration_mm=4,
slots={"A": (True, False), "B": (False, False), "C": (True, True)})
def audit(m, now_s=130):
if (m["store"], m["bay"], m["planogram"], m["frame"]) != (
"S3", "B7", 4, "shelf-left-mm"):
return "abstain:map"
if not m["source"] or not 0 <= now_s - m["captured_s"] <= 60:
return "abstain:freshness"
u = m["registration_mm"]
if not isfinite(u) or not 0 <= u <= 10:
return "abstain:registration"
if set(m["slots"]) != {"A", "B", "C"}:
return "abstain:manifest"
gaps, unknown = [], []
for slot, (visible, occupied) in m["slots"].items():
if not visible:
unknown.append(slot)
elif not occupied:
gaps.append(slot)
return dict(review_gaps=gaps, unknown=unknown,
coverage=(3 - len(unknown)) / 3)
result = audit(base)
assert result["review_gaps"] == ["A"]
assert result["unknown"] == ["B"] and result["coverage"] == 2/3
assert audit(dict(base, planogram=3)) == "abstain:map"
assert audit(base, now_s=200) == "abstain:freshness"
assert audit(dict(base, registration_mm=20)) == "abstain:registration"
print("shelf audit harness: five assertions passed")Slot B remains unknown even though the candidate stage reported no occupancy. The fixture assumes trusted, already derived visibility flags; a production integration must validate how those flags were obtained. It also treats slots equally for a simple denominator. If the audit scope is shelf length rather than slot count, compute coverage from physical intervals without mixing the two units.
Keep the Task Narrow and Recoverable
Use a deterministic join between verified slot observations and the current planogram. A language model can write a concise task such as “check visible gap at bay B7, slot A,” but should not upgrade it to “order twelve units.” Additional specialist components are justified for independent barcode evidence or verified shelf geometry, not for repeated opinions about identical video frames.
The broader retail replenishment workflow addresses inventory freshness and ordering authority. The warehouse pallet verification exercise applies a related identity and coverage gate to storage positions. Keeping these boundaries separate prevents a visual audit from acquiring unintended stock-adjustment permissions.
Authorize store and tenant access server-side. Shelf labels, QR codes, and OCR text may contain arbitrary instructions; treat them as evidence only. Avoid worker identity and productivity judgments from background video. Minimize unnecessary personal imagery in review exports and retain originals only under an approved access and retention policy.
The fixture omits authentication, trusted planogram storage, real visibility estimation, persistent review state, approvals, transactions, and task-provider semantics. Approved verification tasks need idempotency keys containing store, bay, session, slot, and policy revision. After a timeout, reconcile the authoritative task before retrying. A processed clip does not prove that a task was created or a shelf was replenished.
Acceptance Tests and Evaluation
- Place an opaque basket over a slot; require unknown rather than empty even when no package is detected.
- Reuse a superseded planogram; block slot-specific task creation until the mapping is reviewed.
- Repeat a single clear frame; unique slot coverage must not increase beyond the slots actually shown.
- Supply uncertain registration across adjacent facings; abstain from naming a unique slot.
- Process a stale clip after a restock event; preserve historical evidence without issuing a current-state conclusion.
- Replay an approved task after an uncertain response; reconcile one verification request rather than creating duplicates.
Evaluate using human-adjudicated shelf sessions, including busy aisles and partial captures. Measure false empty labels divided by all adjudicated nonempty or unobservable slots, correctly localized review tasks divided by all proposed tasks, and observed slots divided by the planned slot inventory. Report unknown slots separately and retain them in scope totals rather than evaluating only easy views.
Compare against a baseline of manually reviewed clips without deterministic visibility accounting. Split evaluation by store visit and bay rather than random neighboring frames, since repeated observations are correlated. Begin in shadow mode on one bay format, measure task usefulness and reviewer correction time, and permit only approved verification tickets. AI systems engineering should make “unseen” a first-class result before attempting any automated replenishment handoff.