Sorting System Backend
The central Django backend for on-premises AI sorting systems at 20+ sites nationwide
Background
The existing AI produce grading system ran as a PyQt local application plus scattered scripts. Every on-site configuration change required a field visit, and remote monitoring was impossible. The initial Flask Blueprint backend accumulated pressure as the team grew: role separation, hand-built admin pages, and schema migrations driven by AI model changes. Dashboard performance, caching, and real-time communication all had to be solved while handling 30+ sorting requests per second.
System architecture
- 8 Django apps: authentication, admin settings, dashboard, monitoring, PLC communication, server core, sorting configuration, sorting.
- 5 Celery queues across 3 worker pools: the queues are
default / io / maintenance / image_processing / plc_polling. Workers are split into general, I/O, and PLC-only pools to prevent protocol conflicts. - Three-tier fallback cache: worker-level → Redis → DB. Configuration lookups go through a DTO-based cache manager handling warming, invalidation, and thread-safe initialization.
- 7 Docker Compose services: PostgreSQL, Redis, Django, three Celery workers (priority / general / plc), and Celery Beat.
How it was built
Rebuilt on Django as 8 apps. Domain apps were split out over roughly two months through the v3.5 release. The apps cover JWT authentication, RBAC permissions, and configuration profiles (versioning, import/export, soft delete). Django Admin removed the need for a separate admin UI, and DRF automated the API documentation.
30+ requests per second. To keep grade computation, PLC polling, and image uploads from blocking the API, Celery was split into 5 queues and 3 worker pools. The PLC polling queue is bound to a single dedicated worker (concurrency:1), which eliminates Modbus protocol conflicts at the source. The default queue uses a lock-free Redis distributed counter, and 50-row batch inserts with a 1-second timeout cut DB transaction overhead.
Dashboard speed restored through partitioning. PostgreSQL RANGE partitioning (psqlextra) was introduced for sorting data accumulating tens of thousands of rows per day. Celery Beat rolls partitions daily at 00:05 and purges data older than 30 days. Partition relationships are queried through pg_inherits, and SAVEPOINT-based partial rollback keeps a failure on one partition from affecting the others. Composite indexes were added to the 7 core models.
Three-tier fallback cache. Reading configuration from the DB on every sorting request became a three-tier path: worker-level (5-second TTL) → Redis → DB. A DTO-based cache manager warms the cache at server start and propagates invalidation events over Pub/Sub when configuration changes.
Dual real-time channels. Django Channels with Redis Pub/Sub pushes equipment status, temperature, and sorting data over WebSocket. The desktop client consumes both adaptive polling and the WebSocket channel, lowering network load while idle and still reflecting state changes immediately.
Results
- Dashboard load time down 85%, graph API response time improved 90% (measured at commit
663e3a0). - 5 Celery queues across 3 worker pools handle 30+ sorting requests per second without blocking.
- Composite indexes on the 7 core models, RANGE partitioning, and automatic partition rolling keep query speed steady as data accumulates.
- A DTO-based caching framework across 4 domains (Vision / Weight / Discharge / Farm) reduced DB load from configuration lookups.
- v3.5.2 shipped to 20+ sites nationwide via 7 Docker Compose services and per-site release branches.
- Dual real-time channels over Django Channels and Redis Pub/Sub push equipment status, temperature, and sorting data to clients.