Transformer-Based Anomaly Detection in Satellite IoT: A Hybrid Approach with Rotary Embedding and Kernelized Attention

Abstract

Satellite-based Internet of Things (IoT) systems provide crucial connectivity for remote applications; however, they face challenges such as limited bandwidth and noisy data. To address these issues, we propose a novel hybrid Transformer architecture for anomaly detection in satellite IoT imagery. Our model integrates Rotary Positional Encoding to capture spatial relationships and Kernelized Attention for efficient non-linear feature extraction. We compare proposed hybrid approach with Rotary-only and Kernelized-only Transformers, demonstrating that the hybrid model achieves superior performance with an accuracy of 95.62% and a ROC-AUC of 99.44%. The results highlight the potential of our method to enhance the reliability of satellite-based IoT systems, particularly in challenging environments.