Edge-First Security at Scale: Lessons from orchestrating security across 35,000+ POS Devices

How do you enforce real-time security on 30,000 unorchestrated POS devices that go offline for days? With no k8s, no fleet orchestration, and unreliable connectivity — we had to rethink cloud-native assumptions and build a resilient edge-first architecture.

Technical Highlights:

• Devices required persistent runtime security, even while disconnected.

• Cloud-based EDR models failed due to their reliance on constant connectivity and orchestration.

• KubeArmor enabled local runtime policy enforcement directly on the devices.

• gRPC wasn’t a failure, but scaling connection management across 30,000+ devices added unnecessary complexity.

• RabbitMQ handled bidirectional, event-driven messaging with built-in reliability and no custom connection logic.

• Streams and replayability were key for delayed policy delivery, audits, and recovery.

• Policies and telemetry flowed asynchronously between the central control plane and devices.

• Local cache persisted on disk to work within edge limits (0.5CPU, 500Mi RAM).

• We reduced RabbitMQ from 20 to 4 nodes using clustering, tuning, and stream optimizations.

• Open-source RabbitMQ plugins enabled customization without vendor lock-in.

1. Cloud-based EDRs are insufficient for disconnected edge environments — devices need local, independent runtime enforcement.

2. KubeArmor enables localized runtime security, enforcing policies without cloud or orchestration dependencies.

3. gRPC adds high operational complexity at scale; maintaining 30,000+ persistent connections is not practical.

4. RabbitMQ provides built-in connection management, reliable delivery, bidirectional communication, and supports event-driven design.

5. Replayable streams and disk-based local caching are essential for handling policy sync, audit trails, and device restarts in low-resource edge environments.