Multi scan radar object classification on RadarScenes [P]

Hello all, I built a radar object classifier on RadarScenes, extending a prior single-scan classifier to accumulate observations over a tracked object's history instead of classifying each scan in isolation. A single RadarScenes object instance contains only about 2.9 radar points on average, very sparse. A single scan also can't capture temporal characteristics: RCS and micro-Doppler both vary continuously as an object moves. Pedestrians produce characteristic micro-Doppler from limb motion; different object classes show different RCS fluctuation patterns as aspect angle and scattering geometry change scan to scan. Accumulating observations gives both higher point density and provides temporal dynamics. Multi-scan baseline DeepReflecs encoder (Ulrich, Glaser & Timm, RadarConf 2021), PointNet style, per point shared weights, on single scans across car, large_vehicle, two_wheeler, pedestrian, pedestrian_group: 0.7370 macro F1. Using RadarScenes' persistent `track_id`, I build a causal, N=20, per track sliding-window buffer: - x_seq/y_seq: Global, odometry-corrected coordinates recentered per scan on the object centroid. Unlike x_cc/y_cc (car-frame coordinates that accumulate over time to form a trajectory). - Cross sensor buffer: whichever of the 4 sensors currently observe the track push to the same buffer. - Stride 1, causal: every new scan updates the buffer and produces a prediction. No future context, real time streaming compatible. - Each scan is encoded once by a frozen per scan encoder and cached - Fusion concatenates the causal GRU's hidden state (order aware) with an order-invariant pooled embedding (all N scans' points as one set, no sequence structure) through a small trained mlp head. Results Model Macro F1 Delta Single scan 0.7370 (baseline) 20 scan point pooling 0.8613 +0.1243 Causal GRU 0.8895 +0.0282 over pooling GRU + pooled embedding (fusion) 0.8897 +0.0002 over GRU, noise Pooling alone, no sequence model, no notion of scan order at all, recovers +0.1243 macro F1. The GRU adds a real but much smaller +0.0282 on top. Fusion adds nothing measurable beyond the GRU. Ablation Llarger GRUs, a Transformer, a state space model, point level self attention, all trained on the exact same frozen per scan embeddings, land inside a 0.86 to 0.89 band, a 0.03 spread. End to end fine tuning of the frozen encoder makes things slightly worse (about -0.002 to -0.003), not better. Conclusion So, in this setup, most of the improvement comes from accumulating more radar returns rather than from the choice of sequence model. Once the per-scan embedding is fixed, none of the sequence-models tested changes macro F1 by more than ~0.003 beyond the pooled representation. That does not necessarily mean that temporal ordering contains no useful information. Rather, it suggests that the current per-scan representation and sequence models are not extracting much additional discriminative signal. For this reason, the results suggest that improving the point-level/per-scan representation may be a more promising direction than further searching over sequence architectures. Full report, every ablation, confusion matrix, coordinate frame reasoning: https://github.com/brunopinto900/radar-ml-autonomous-driving/blob/track-accumulation/final_report.md Thank you.