Multimodal inspection / Guide 90
Package Damage Photos Are Evidence, Not a Cause of Loss
Build package damage photo evidence packets that validate parcel identity, capture provenance, measured scale, and missing faces without inventing causal blame.
Illustrative engineering guide. Examples are not client deployments, production-ready integrations, or evidence of measured results. Validate the design with your domain and security owners.
A hypothetical customer submits three photographs of a crushed carton corner. An assistant estimates a forty-millimeter dent and blames the last delivery handoff. The photos contain no scale reference, two are crops of the same image, and the parcel label is visible only in a photograph from another shipment. The photographs may support a visible-condition note, but not the dimensional, identity, or causal claims in that summary.
Build an advisory evidence packet for a human claims or operations reviewer. Separate observed appearance, parcel association, measurement support, and event chronology. A photograph of a damaged carton does not establish when damage occurred, who caused it, whether the contents are damaged, or what compensation is owed. Those decisions need independent records and the applicable review process.
Preserve the Original Before Annotating It
Require shipment and parcel identifiers, packaging specification revision, capture session, original file digest, capture timestamp, upload timestamp, and declared parcel face. Keep crops as derived artifacts with their source coordinates. Multiple crops from one photo should count as one source observation, not independent corroboration from several viewpoints.
Capture time from image metadata may be missing or editable. Preserve its provenance and reliability status instead of treating it as an authenticated chain-of-custody event. Server-recorded upload time is a different fact. If chronology matters, attach independently authorized handoff records and show uncertainty; do not infer an exact damage interval merely because images were uploaded in a particular order.
The W3C PROV overview distinguishes source entities, processing activities, and responsible agents. Use that structure for original photos, annotations, parcel association, and reviewer decisions. A cryptographic digest can identify unchanged bytes, but it cannot prove the scene is genuine or that the parcel label belongs to the photographed carton.
Measure Only What the Capture Supports
The candidate stage may identify a crushed-looking corner, torn surface, or visible opening. Keep these descriptions neutral and local to the observed face. RGB does not establish temperature, hidden content condition, or mechanical cause. An intact-looking exterior also does not prove intact contents; both positive and negative claims must stay within visible evidence.
Define image coordinates in pixels from the upper-left corner. For a measured carton face, define millimeter coordinates from a named physical corner with horizontal and vertical axes specified. A scale reference must be associated with that face and relevant plane. A ruler lying behind the carton cannot automatically calibrate a dent on its front surface.
OpenCV’s calibration tutorial explains distortion and the role of known object points in camera calibration. Those techniques support controlled measurement, but they do not turn an arbitrary customer photograph into a depth scan. In particular, a visible edge displacement is not necessarily dent depth; use a specialist measurement method if depth is required.
Test Evidence Sufficiency Without Inventing Dimensions
This self-contained Python fixture tests the HARNESS using synthetic structured metadata. It does not evaluate a trained vision model, camera, measurement engine, or real inspection performance. Its reference and registration checks are illustrative workflow policy, not packaging or insurance standards. It makes no real media or engine API calls.
from math import isfinite
base = dict(parcel="P-18", packaging=2, image="synthetic-photo-A",
annotation="mark-A", face="front", captured_s=100, uploaded_s=120,
frame="front-top-left-mm", extent_px=80.0,
scale_mm_px=None, registration_mm=0.5)
def evidence(rows):
sources, annotations, faces = {}, {}, set()
for r in rows:
if (r["parcel"], r["packaging"]) != ("P-18", 2):
return "abstain:parcel"
if (not r["image"] or not r["annotation"]
or not all(isfinite(r[k]) for k in ("captured_s", "uploaded_s"))
or r["captured_s"] > r["uploaded_s"]):
return "abstain:provenance"
source_times = (r["captured_s"], r["uploaded_s"])
if r["image"] in sources and sources[r["image"]] != source_times:
return "abstain:source-conflict"
sources[r["image"]] = source_times
scale = r["scale_mm_px"]
if scale is None:
extent = None
elif (r["frame"] != "front-top-left-mm" or r["face"] != "front"
or not all(isfinite(x) and x >= 0 for x in
(scale, r["extent_px"], r["registration_mm"]))
or scale == 0 or r["registration_mm"] > 1):
return "abstain:measurement"
else:
extent = scale * r["extent_px"]
annotation = dict(r, extent_mm=extent)
key = r["annotation"]
if key in annotations and annotations[key] != annotation:
return "abstain:annotation-conflict"
annotations[key] = annotation
faces.add(r["face"])
ordered = {key: annotations[key] for key in sorted(annotations)}
return dict(unique_sources=len(sources), faces=faces, annotations=ordered,
extents_mm=[r["extent_mm"] for r in ordered.values()],
state="review" if sources else "abstain:no-evidence")
packet = evidence([base, dict(base)])
assert packet["unique_sources"] == 1 and packet["extents_mm"] == [None]
assert packet["faces"] == {"front"} and len(packet["annotations"]) == 1
assert packet["annotations"]["mark-A"]["extent_px"] == 80.0
assert evidence([dict(base, scale_mm_px=0.25)])["extents_mm"] == [20.0]
assert evidence([dict(base, parcel="P-19")]) == "abstain:parcel"
assert evidence([dict(base, captured_s=130)]) == "abstain:provenance"
small = dict(base, annotation="mark-A", scale_mm_px=0.25, extent_px=4)
large = dict(base, annotation="mark-B", scale_mm_px=0.25, extent_px=80)
packet = evidence([small, large, dict(small)])
assert packet["unique_sources"] == 1 and packet["extents_mm"] == [1.0, 20.0]
assert packet == evidence([large, small])
assert set(packet["annotations"]) == {"mark-A", "mark-B"}
conflict = dict(large, annotation="mark-A")
assert evidence([small, conflict]) == "abstain:annotation-conflict"
assert evidence([conflict, small]) == "abstain:annotation-conflict"
changed_source = dict(large, captured_s=101)
assert evidence([small, changed_source]) == "abstain:source-conflict"
assert evidence([changed_source, small]) == "abstain:source-conflict"
assert evidence([small, dict(large, image="synthetic-photo-B")])["unique_sources"] == 2
for invalid in (float("nan"), float("inf"), float("-inf")):
for key in ("captured_s", "uploaded_s"):
assert evidence([dict(base, **{key: invalid})]) == "abstain:provenance"
assert evidence([dict(base, captured_s=invalid, uploaded_s=invalid)]) == "abstain:provenance"
assert evidence([dict(base, captured_s=120)])["state"] == "review"
assert evidence([])["state"] == "abstain:no-evidence"
print("package evidence harness: synthetic assertions passed")The unscaled image remains reviewable while its physical extent stays unavailable. That is a deliberate distinction between useful evidence and sufficient measurement. The twenty-millimeter result in the scaled variant represents a supplied planar extent, not dent depth. Distinct annotation IDs retain their source, pixel extent, and measurement while source counting deduplicates original images. Exact annotation repeats are idempotent; conflicting records for one annotation or source chronology abstain in either order. Source timestamps refer to the original capture and upload, not later crop exports, and must both be finite. Production validation must verify the actual scale reference, transform, and measurement uncertainty rather than accept the uploader’s numeric declaration.
Keep Missing Faces and Uncertainty Explicit
List coverage against the requested parcel faces, including top, bottom, and sides where the review procedure requires them. A front-face packet can support a local observation while remaining incomplete for whole-parcel assessment. Occluded or unphotographed faces are unknown, not passed. If a reviewer requests more evidence, identify the missing face and capture purpose rather than asking for arbitrary additional photos.
Record scale uncertainty, endpoint-selection uncertainty, and registration uncertainty separately. NIST’s combined uncertainty guidance accounts for covariance and recognized systematic effects. Two crops sharing one calibration reference share its error; treating them as independent measurements can falsely narrow the reported uncertainty. Near a policy boundary, surface the interval and request review.
The broader insurance claims workflow separates evidence gathering from policy and compensation decisions. The fruit video grading exercise shows the same repeated-observation problem in continuous capture. A deterministic evidence validator is enough for identity and coverage checks; additional agents earn their cost only when they bring independent authorized records or a genuinely distinct measurement capability.
Protect the Claims Boundary
Authorize shipment access server-side and isolate each tenant’s media. Parcel labels can contain addresses and other personal data, so use restricted originals and redacted review exports where appropriate. Treat OCR text, filenames, and embedded instructions as untrusted evidence. A visible note saying “refund approved” cannot authorize a refund or override the claims process.
The fixture omits authentication, trusted shipment records, real calibration, durable review state, database transactions, approval persistence, and claims-provider semantics. Human-approved evidence requests should be idempotent by parcel, case, evidence revision, and requested face. A timeout leaves request status unknown until reconciled. Neither successful upload nor successful caption generation is proof of an approved claim.
Acceptance Tests and Evaluation
- Upload two crops of one source; retain their annotations but count only one independent source photograph. Conflicting duplicates must abstain regardless of input order.
- Omit the scale reference; preserve pixel extent and prohibit a physical dent-size claim.
- Attach a label from another parcel; require association review before the packet enters that claim.
- Leave the bottom face unseen; show incomplete coverage without assuming contents or exterior are intact.
- Supply contradictory or nonfinite capture/upload timestamps; flag chronology uncertainty rather than assigning damage to a handoff.
- Insert blame or refund instructions into visible text; retain the text as evidence without changing causal conclusions or payment authority.
Evaluate on independently adjudicated parcel cases. Measure wrong-parcel evidence admitted divided by all mismatched submissions, unsupported dimensional claims divided by all generated dimensional claims, and requested faces observed divided by all requested faces. Keep incomplete submissions and abstentions in the case denominator. Do not use caption confidence or duplicate-photo count as a substitute for evidence quality.
Start in shadow mode with human review of every packet and compare against manual photo organization. Track missing-evidence requests, reviewer corrections, and time to a usable packet, not invented claims accuracy. AI systems engineering should first deliver immutable source lineage and explicit unavailable dimensions. That makes the photographs useful without converting visible damage into unsupported causal blame.