Multimodal inspection / Guide 91
Automotive Assembly Video Needs Evidence, Not Completion Scores
Build automotive assembly video verification around clip provenance, occlusion, tool receipts, and measured coverage before a quality reviewer approves action.
Illustrative engineering guide. Examples are not client deployments, production-ready integrations, or evidence of measured results. Validate the design with your domain and security owners.
A hypothetical door assembly video shows a driver approaching three fasteners. The narration says all three were tightened. One fastener was behind a hand, however, and the controller receipt belongs to the previous vehicle. A convincing description of the motion has become a false completion record. The useful engineering problem is not generating a better caption. It is binding each required operation to the correct product, evidence interval, and independent measurement.
Treat video as a source of candidate events: tool approach, connector placement, or a latch changing position. None independently proves torque, electrical continuity, or hidden engagement. An operator should receive a packet saying which operation is supported, which is contradicted, and which remains unobserved. The system must never turn assembly footage into worker identity, attendance, skill, or productivity judgments.
Preserve the Cycle Boundary
Start with the station controller’s vehicle token, work instruction revision, camera identifier, recording digest, and capture interval. Vehicle tokens should be scoped operational identifiers rather than broadly exposed customer records. Bind the clip to a station cycle before interpreting any action. A filename containing the expected serial number is only a label, not trusted identity.
Store original presentation timestamps and the mapping used for extracted frames. FFprobe documents frame and stream inspection, including timecode extraction and imperfect seek boundaries. Consequently, requesting a timestamp is not proof that the returned frame represents that exact instant. Preserve actual decoded timestamps, stream time base, dropped intervals, and extraction version. Repeated adjacent frames are correlated observations, not additional independent confirmations.
For image overlays, define pixels from the top left, with horizontal coordinates increasing right and vertical coordinates increasing down. For spatial measurements, use a separately registered station frame in millimeters. OpenCV’s calibration tutorial distinguishes image points from object points and explains distortion. A convenient pixel distance does not become a physical fastener separation without that mapping and its uncertainty.
Match Evidence Before Measuring Success
A specialist adapter should retrieve authenticated tool records containing cycle identity, operation identity, units, calibration revision, and controller timestamp. A process engineer defines the acceptable torque interval for the actual joint. This article’s numerical ranges are illustrative policy values, not automotive standards. Video can corroborate location and sequence; the receipt supplies the torque observation, still subject to tool calibration and correct socket engagement.
Synchronize controller time to camera time using a documented offset and bounded clock uncertainty. Do not silently choose whichever nearby receipt makes the video interpretation succeed. If multiple receipts fit, abstain. If the clock bound exceeds the matching window, return a synchronization problem rather than an assembly defect. These are different corrective actions with different owners.
If a depth sensor supplies tool coordinates, its pose registration is another dependency. Open3D’s ICP tutorial describes refinement from an initial alignment and reports registration metrics. A low residual is useful diagnostic evidence, not proof that two repeated fixtures were matched correctly. Check independent landmarks and record the transform direction explicitly.
A Deterministic Cycle Fixture
This self-contained Python example tests the HARNESS on synthetic structured metadata and measurements. It does not test a trained vision model, camera, torque instrument, or actual inspection performance. Every record represents a pre-extracted operation candidate; no media is decoded and no machine command is issued.
from math import isfinite
base = dict(vehicle="V17", revision="door-r4", visible=True,
camera_ms=1200, tool_ms=1220, clock_bound_ms=15,
torque_nm=8.0, torque_bound_nm=0.2, receipt_vehicle="V17")
def inspect(row):
if row["vehicle"] != row["receipt_vehicle"]:
return "wrong-cycle"
if row["revision"] != "door-r4" or not row["visible"]:
return "unknown"
values = [row[k] for k in ("camera_ms", "tool_ms", "clock_bound_ms",
"torque_nm", "torque_bound_nm")]
if not all(isfinite(v) for v in values):
return "unknown"
if min(row["clock_bound_ms"], row["torque_bound_nm"]) < 0:
return "unknown"
if abs(row["camera_ms"] - row["tool_ms"]) + row["clock_bound_ms"] > 80:
return "unknown"
low = row["torque_nm"] - row["torque_bound_nm"]
high = row["torque_nm"] + row["torque_bound_nm"]
return "supported-for-review" if 7 <= low <= high <= 9 else "joint-review"
assert inspect(base) == "supported-for-review"
assert inspect(dict(base, receipt_vehicle="V16")) == "wrong-cycle"
assert inspect(dict(base, visible=False)) == "unknown"
assert inspect(dict(base, clock_bound_ms=100)) == "unknown"
assert inspect(dict(base, torque_nm=8.9)) == "joint-review"
assert inspect(dict(base, torque_nm=float("nan"))) == "unknown"
print("six synthetic cycle checks passed")The uncertainty interval must fit inside the illustrative torque band. This is deliberately stricter than accepting a point estimate near the boundary. The result is still only supported evidence for review, not a release decision. Production also needs schema validation for missing fields, signed evidence origins, socket identity, tool health, persistent state, and a validated method for assigning uncertainty bounds.
Review Without Surveillance
Define coverage against the instruction’s required operations, not against the number of frames processed. Three supported joints out of four required joints means one unresolved joint even if the remaining video contains thousands of frames. Retain explicit reasons: occluded, outside capture, missing receipt, ambiguous correspondence, or stale instruction. A reviewer can request a focused reinspection rather than replay an entire shift.
Restrict retrieval server side to the plant, station, and authorized quality role. Mask unrelated people where practical, set short footage retention, and audit evidence access. Text on screens, badges, and labels is untrusted content, never an instruction to bypass a gate. A language model may summarize findings but cannot approve a joint or update a worker record.
An approved quality ticket should use vehicle, operation, instruction revision, and evidence digest as its deduplication identity. A timeout during ticket creation means unknown delivery; reconcile with the ticket store before retrying. New footage creates a new evidence version instead of overwriting the reviewer rationale. This complements maintenance workflow ownership while keeping inspection separate from repair execution.
Acceptance Tests and Evaluation
- Replay a valid clip with a previous vehicle’s receipt; reject the association even when torque is in range.
- Hide one required joint behind a hand; retain that joint as unknown and exclude it from supported coverage.
- Shift the camera clock beyond the matching budget; request synchronization review rather than alleging poor assembly.
- Duplicate extracted frames; operation coverage and evidence confidence must not increase automatically.
- Replace the instruction revision midcycle; invalidate the old completeness decision and require reconciliation.
- Retry an approved ticket after a simulated timeout; recover its identity without creating a duplicate.
Establish a baseline using blinded quality reviewers on complete cycles, with disagreements adjudicated by a process specialist. Report correctly escalated defective operations divided by all adjudicated defective operations, plus falsely supported operations divided by all supported operations. Publish unknown operations divided by all required operations and review minutes per cycle. Split evaluation by vehicle family and recording session, never random neighboring frames.
Begin with one station and one joint family in shadow mode. Proposed promotion criteria should include acceptable unresolved coverage and zero unauthorized release actions, not a headline caption accuracy. Furniture assembly validation explores a related hidden-fastener problem without industrial tool receipts. For implementation, AI systems engineering should begin with a cycle manifest and reviewer packet, not a fleet of autonomous agents.