Real-Time Visual Anomaly Detection for Steel Surface Inspection

On a hot-strip rolling line, steel can move at 10–20 m/s. A surface defect — a sliver, scab, roll mark, or scale pit — may appear for a few hundred milliseconds before it is wound into a coil that ships to a customer. Manual inspection cannot keep that pace, and post-hoc lab sampling finds problems only after thousands of metres are already defective. This article walks through the architecture and implementation of a system that inspects every frame of the strip in real time and flags defects to the operator within the line's control-loop latency budget.

Design constraints

The architecture is driven by four hard constraints that shape every downstream decision:

• Latency budget. End-to-end (photon to operator alert) must stay under ~80 ms so the HMI flags the defect against a position encoder while the affected section is still identifiable.
• Throughput. A 2k-wide line scan at 5–10 kHz line rate generates 300–600 MB/s per camera; multiple cameras cover the full width and both faces.
• Label scarcity. Defects are rare and long-tailed. A purely supervised classifier starves on the rarest, most costly defect classes.
• Plant reality. Variable lighting, scale, water, and vibration mean the model must be robust to nuisance variation that is not a defect.

System architecture

The system splits into a hot path (deterministic, on the edge, in the latency budget) and a cold path (training and analytics, off the line). The anomaly model handles the "is this normal?" question without needing labels for every defect; a lightweight classifier then names the defect only when the anomaly score is high.

Why anomaly detection first

We treat the normal product surface as the modelled distribution and score deviation from it, rather than trying to enumerate every defect class up front. PatchCore is a strong fit: it builds a memory bank of patch-level features from a pretrained backbone over defect-free images only, then scores a test patch by its distance to the nearest neighbour in that bank. This gives pixel-level anomaly maps, needs no defect labels to bootstrap, and catches novel defects the team has never labelled.

Building the PatchCore memory bank (offline, defect-free images)

import torch, timm
from sklearn.neighbors import NearestNeighbors

backbone = timm.create_model("wide_resnet50_2", features_only=True,
                             out_indices=(2, 3), pretrained=True).eval().cuda()

def patch_features(x):                       # x: [B,3,H,W] normalized
    feats = backbone(x)                      # multi-scale feature maps
    feats = [torch.nn.functional.adaptive_avg_pool2d(f, f.shape[-2:]) for f in feats]
    f = torch.cat([torch.nn.functional.interpolate(f, size=feats[0].shape[-2:],
                   mode="bilinear") for f in feats], dim=1)
    return f.permute(0, 2, 3, 1).reshape(-1, f.shape[1])   # [N_patches, C]

bank = torch.cat([patch_features(b.cuda()) for b in normal_loader])  # all-normal
bank = coreset_subsample(bank, ratio=0.01)   # greedy coreset keeps it small
nn = NearestNeighbors(n_neighbors=1).fit(bank.cpu().numpy())

At inference, the anomaly score of a tile is the max patch distance to the bank; the per-patch distances reshape into a heat map that localises the defect for the operator.

The hot path: serving inside 80 ms

The edge node (an industrial GPU box, e.g. an RTX A4000 or Jetson AGX class device per camera group) runs a GStreamer pipeline that pulls frames from the grabber, applies flat-field correction to neutralise lighting non-uniformity, and tiles the wide strip into fixed crops. Each tile flows through a single fused engine.

For determinism we export the backbone to TensorRT (FP16) and keep the nearest-neighbour lookup on-GPU with a FAISS index, so the whole tile path avoids host round-trips.

Per-tile inference loop on the edge

import tensorrt as trt, faiss, numpy as np

trt_engine = load_engine("patchcore_wrn50.plan")   # FP16, fixed batch
index = faiss.read_index("bank.faiss"); faiss.index_cpu_to_all_gpus(index)

def score_tile(tile):                    # tile: preprocessed HxWx3 uint8
    feats = trt_engine.infer(tile)       # [N_patches, C], on GPU
    d, _ = index.search(feats, 1)        # nearest-neighbour distance
    heat = d.reshape(patch_h, patch_w)
    return float(heat.max()), heat       # image-level score, localisation

score, heat = score_tile(tile)
if score > TAU:                          # gate: only classify suspicious tiles
    defect = classifier.infer(crop_around(tile, heat))   # MobileNetV3 head
    publish(defect, encoder_position(), heat)            # to HMI + Kafka

The cold path: closing the loop

Every high-score tile, its heat map, and the operator's adjudication (true defect / nuisance / class correction) land in the data lake. This is the training signal that the hot path can't generate on its own. Three jobs run on it:

1. Active labelling. Operators confirm or correct flagged tiles; uncertain, high-impact samples are prioritised so labelling effort goes where it moves the metric.
2. Memory-bank refresh. As the product mix and surface finish drift seasonally, the PatchCore bank is refit on recent confirmed-normal images so "normal" tracks reality.
3. Classifier fine-tuning. The MobileNetV3 head is retrained as new confirmed defects accumulate, with class weighting for the long tail.

Monitoring & drift

Two failure modes matter in production. Score drift (the normal-tile score distribution shifting upward) signals lighting or surface changes and triggers a bank refresh. Coverage gaps (a defect class the lab finds that the model never flagged) feed straight back into active labelling. We track image-level precision/recall against lab sampling, alert latency P99, and the false-alarm rate that operators actually experience — because an inspection system operators learn to ignore is worse than none.

What this delivers

The result is defect capture at the moment of formation rather than shifts later in the lab: less scrap, fewer defective coils reaching customers, and a continuously improving model whose "normal" stays anchored to current production. The same edge-plus-anomaly pattern transfers directly to cement clinker, billet, and casting inspection — anywhere defects are rare, costly, and visible.