Semi-supervised video anomaly detection methods face two critical challenges: (1) Strong generalization blurs the boundary between normal and abnormal patterns. Although existing approaches attempt to alleviate this issue using memory modules, their rigid prototype-matching process limits adaptability to diverse scenarios; (2) Relying solely on low-level appearance and motion cues makes it difficult to perceive high-level semantic anomalies in complex scenes. To address these limitations, we propose SIB-VAD, a novel framework based on adaptive information bottleneck filtering and semantic-aware enhancement. We propose the Sparse Feature Filtering Module (SFFM) to replace traditional memory modules. It compresses normal features directly into a low-dimensional manifold based on the information bottleneck principle and uses an adaptive routing mechanism to dynamically select the most suitable normal bottleneck subspace. Trained only on normal data, SFFMs only learn normal low-dimensional manifolds, while abnormal features deviate and are effectively filtered. Unlike memory modules, SFFM directly removes abnormal information and adaptively handles scene variations. To improve semantic awareness, we further design a multimodal prediction framework that jointly models appearance, motion, and semantics. Through multimodal consistency constraints and joint error computation, it achieves more robust VAD performance. Experimental results validate the effectiveness of our feature filtering paradigm based on semantics-aware information bottleneck.
Overview of SIB-VAD. Firstly, TMFE extracts descriptions from the input clips and encodes and fuses semantic, appearance, and motion features. Secondly, MJAD performs joint decoding of the fused features to predict the next frame and semantic features. SFFM filters abnormal information to increase the prediction error when anomalies occur. Finally, anomaly scores are calculated based on frame prediction errors and semantic errors.
Comparison between the previous and our proposed framework. (a) Existing memory modules filter abnormal information through inflexible prototype matching. (b) Our proposed sparse feature filtering paradigm achieves better VAD performance compared to memory-based methods.
@misc{li2025memoryout,
title={Video Anomaly Detection with Semantics-Aware Information Bottleneck},
author={Li, Juntong and Dang, Lingwei and Xiao, Qingxin and Shang, Shishuo and Cheng, Jiajia and Wu, Haomin and Hao, Yun and Wu, Qingyao},
year={2025},
eprint={2506.02535},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2506.02535v3},
}