Challenges and Research Progress of Big Data Management System Evaluation Benchmark

Abstract

This paper presents a systematic review and prospective analysis of benchmarking methodologies for big data management systems. We begin by articulating the principal challenges inherent in benchmarking such systems, focusing on critical dimensions including data scale and complexity, data security and privacy preservation, data quality assurance, and consistency maintenance. Subsequently, we synthesize current research advancements in benchmarking big data management systems, which encompass dataset-centric evaluation methods, task-driven performance assessments, multi-faceted performance metrics, and user-experience-oriented analysis. Building upon this foundation, the paper proposes key directions for future research, emphasizing the development of multidimensional evaluation frameworks, the formulation of standardized benchmarking protocols, and the creation of automated, scalable evaluation tools. These contributions are intended to provide both theoretical insights and practical guidance for the design, implementation, and continuous evolution of effective benchmarks for big data management systems. The ultimate objective is to advance the state of the art in evaluating system performance, robustness, and usability, thereby facilitating the development of more reliable, efficient, and secure big data solutions.