A comprehensive survey on big data privacy and Hadoop security: Insights into encryption mechanisms and emerging trends

Abstract

Big data has transformed analytics and data processing in many different industries, but securing security and privacy in distributed systems like Hadoop is still rather complex. This article gives a deep analysis of the symmetric, asymmetric, and hybrid encryption techniques applied in Hadoop to preserve massive amounts of data. We critically analyze earlier research, underlining its advantages, flaws, and important trade-offs, specifically with reference to scalability, computing expense, and implementation complexity. Additionally, we analyze new improvements like blockchain integration and post-quantum encryption, analyzing their potential to increase Hadoop security. We find weaknesses in existing techniques via a comparative study and provide a hybrid encryption system aimed at secure and efficient data processing in Hadoop settings. Researchers and practitioners searching for scalable, privacy-preserving big data platform solutions should use this paper as a reference. • A comprehensive survey of privacy-preserving techniques in Hadoop big data environments . • Comparative analysis of symmetric, asymmetric, homomorphic, and post-quantum encryption . • Proposed a hybrid encryption framework integrating blockchain for secure HDFS storage. • Discussed real-world Hadoop applications and emerging trends like federated learning . • Identified key future research directions , including post-quantum cryptography and AI-driven security.