Deterministic Vision: Offloading Real-Time Camera Pipelines to FreeRTOS on Heterogeneous SoCs

Modern automotive DMS/OMS demand deterministic, real-time camera processing under
safety-critical constraints, a requirement that goes beyond real-time performance. While RT-Linux can deliver real-time performance, the ISO 26262 compliance requires predictable resource isolation, deterministic scheduling guarantees, and functional safety certification. This work consolidates CSIRX capture, sensor drivers, and VPAC control into a FreeRTOS-based safety-critical firmware on AM62A's Device Manager R5F core, eliminating A53 real-time burden. Linux-based capture remains viable for general-purpose and industrial applications that do not require safety-critical deterministic requirements.

AM62A is a heterogeneous SoC featuring A53 (Linux/applications), C7x NPU (AI/ML),
VPAC accelerators (VISS → LDC → MSC), and two R5F cores: MCU R5F (AUTOSAR/safety) and DM R5F (Device Manager). Rather than competing for MCU R5F reserved for safety, the entire vision stack consisting of CSIRX camera, sensor drivers, VPAC control, 2A algorithms consolidates onto DM R5F under FreeRTOS. This: (1) frees A53 from real-time duties; (2) makes capture HLOS-agnostic; (3) enables concurrent execution of camera capture, VPAC, Device Manager, EthFW (with gPTP stack) on one safety-critical core.

TIOVX (TI OpenVx) middleware layer controls the overall graph execution, node/kernel
dispatch on different target cores over RPMsg-based communication in the present use case. The R5F controls VPAC via memory-mapped registers and interrupt-driven completion, ensuring deterministic processing. DMR5 runs all services at ~40% peak load, leaving 60% headroom. Future work targets On-The-Fly CSI-RX to VPAC path to eliminate DDR roundtrips.

Key Takeaways from this talk are:

1. Bridges the gap - between general-purpose Linux and dedicated safety systems

2. Solves A53 Overloading as mixing camera capture with application logic on main processor causes unacceptable latency due to IPC overhead

3. Proper utilization of DM R5F core by consolidating different tasks and functionalities on one single core.

4. 40% Load, 60% Headroom - all services (capture + VPAC + DM + EthFW + gPTP) run at only ~40% peak, leaving room for future workloads

5. HLOS-Agnostic Capture - A53 running Linux is completely decoupled from real-time camera duties