Monitoring, Data Collection, and CI/CD Operations Infrastructure
Remote monitoring of 20+ field sites, automated anomaly data collection, standardized CI/CD and deployment, GitHub-Slack-Notion webhook integration
Background
On-premises equipment deployed to more than 20 sites nationwide could not be inspected remotely, so failures were only discovered after the fact. Anomaly data needed for AI model improvement was collected by hand and frequently missed. Manual deployment and the absence of issue tracking meant recurring human error at release time. On the development workflow side, GitHub pull requests were not reflected in Slack or Notion automatically, so review requests got buried.
System architecture
- Monitoring metrics: Prometheus collects hardware status, camera temperature and FPS, and CPU/GPU utilization, and Grafana visualizes them. Threshold breaches trigger a Slack alert.
- Webhook as a standalone service: built as its own Django app (integrations + accounts). Runs on 2 replicas through three Kubernetes manifests: Django Deployment, Service, and Celery Deployment.
- Automated anomaly collection: FastAPI with Socket.IO lets on-premises equipment push unusual data to the central server.
- Deployment standard: Docker Compose with standardized resource limits, logging, and health checks, plus Jira issue tracking.
How it was built
Monitoring infrastructure. Built Prometheus + Grafana dashboards collecting hardware status, camera temperature and FPS, and CPU/GPU utilization metrics. Threshold breaches alert through a Slack webhook, and Sentry error tracking plus structured logging (JSON format) surface failure signals early.
Automated data collection. Set up a FastAPI + Socket.IO collection server so equipment detects anomalies locally and sends them to the central server. This removed the manual collection step and minimized the delay between an event occurring on the device and being stored.
CI/CD and deployment standardization. Added lint and automated test pipelines to GitHub Actions, and standardized resource limits, logging, and health checks for Docker Compose based on-premises deployment. Set up issue and release tracking in Jira.
Development workflow webhook. Built a separate Django app that receives GitHub PR events and converts them into Slack alerts and Notion tasks across three platforms. GitHub payloads are authenticated with HMAC-SHA256 signature verification, and Celery handles conversion asynchronously so responses are not delayed. Cross-platform user mapping by email and name means alerts reach the right reviewer even when their accounts differ per platform. Deployed to production on Kubernetes with 2 replicas.
Results
- Equipment at 20+ sites nationwide is monitored remotely in real time, moving failure response from reactive to preemptive.
- Automated anomaly collection improved the quality of AI model training data.
- GitHub Actions lint and tests plus standardized Docker Compose deployment reduced release human error, and Jira established issue management.
- GitHub PR events flow automatically into Slack and Notion, with HMAC-SHA256 verification, async Celery processing, and 2 replicas on Kubernetes.
- Three Kubernetes manifests make up the webhook service deployment pipeline.