GPU Vision Pipeline

Background

The system photographs produce moving at speed along a conveyor using up to 8 industrial cameras, then runs GPU-based YOLOv8 inference and machine vision analysis in real time. After a year of accumulated features, initialization, process construction, CPU assignment, and configuration management were all tangled into the main process. Contention between processes made latency unstable. Equipment vendors differed from site to site, so each protocol had to be handled individually. Machine vision algorithms for more than 8 kinds of produce needed a structure that could grow.

System architecture

• Execution pipeline: a real-time pipeline split into libs/{camera, inference, socket, sync, transfer, db}. main.py starts processes with torch.multiprocessing.set_start_method('spawn') and decouples about ten processes through a Queue-based producer-consumer design.
• Vision algorithm library: a standalone Python package with its own pyproject.toml. On top of the BaseProcessor (abc.ABC) abstraction sit 6 produce runners, mv/ handlers for 8 produce types, and the grading, inference, and io_schemas subpackages.
• Equipment protocol plugins: Senders for 9 vendors (aiofarm, aiofarm_weight, gj_prusen, multi_tp_rail, prusen_wandong, snp, tp, tp_rail, wh) inherit a common base and implement only the vendor-specific differences.
• Arduino trigger: Mega 2560 firmware takes the photoelectric sensor signal and generates multi-camera triggers with millisecond precision. 7 firmware variants, one per produce type.

How it was built

Pipeline modularization. Capture, inference, MV, and transfer had been entangled in the main process. Each moved into its own subpackage under libs/ so every stage owns only its own responsibility. FlowControls centralizes data flow routing, and backpressure plus timeouts at each Queue boundary keep one stalled process from deadlocking the whole pipeline.

CPU affinity tuning. psutil detects physical cores and assigns dedicated cores per process. Separating the inference, MV, transfer, and capture processes so they do not compete for the same core reduced latency variance.

Equipment Sender plugins. Equipment vendor protocols differ per site, so dynamic loading was required. One base Sender exists, and each vendor file overrides only the transfer format and sequence differences. Onboarding a new vendor means adding one Sender file.

Produce framework. Machine vision algorithms for different produce were mixed together, which made adding a new type expensive. Defining BaseProcessor (abc.ABC) fixed PredictYoloV8, MVCalculators, and MVInput/MVOutput as the standard interface. A produce developer inherits BaseProcessor in {produce}_run.py and implements only process_single_image(). I designed the library architecture; teammates developed the individual produce machine vision algorithms independently.

Packaging as a standalone library. The vision algorithm library was split out as a standalone package with a pyproject.toml. The execution pipeline pulls it in via pip install, so produce algorithm changes never land in the pipeline repository.

Per-produce Arduino firmware. The Mega 2560 firmware that takes conveyor photoelectric sensor signals and emits camera trigger signals is maintained as 7 produce variants. Firmware ships through per-site branches.

Results

• Real-time pipeline modularized into 6 libs/ subpackages (camera / inference / socket / sync / transfer / db).
• 9 equipment vendor protocols unified behind Sender plugins; a new vendor is one added file.
• Vision algorithm library split into a standalone package, with teammates independently building 6 produce runners and 8 produce mv handlers on the BaseProcessor abstraction.
• libjpeg-turbo JPEG encoding cut network bandwidth by 90%, and a multi-GPU setup scaled inference throughput linearly.
• Arduino Mega 2560 camera triggering in 7 produce variants, at millisecond precision.
• torch.multiprocessing(spawn) plus Queue backpressure fully decoupled the processes and prevented deadlock.