Schema validation and evaluation framework for extracted schemas in JSON databases

Abstract

The increasing use of schemaless data systems has intensified the need for reliable methods to assess the quality of extracted schemas intended for downstream tasks such as data integration, query optimisation, and interoperability. Although numerous schema inference techniques have been proposed, the field still lacks standardised and method-independent criteria for evaluating the validity and accuracy of inferred schemas. This paper introduces the Schema Validation and Evaluation Framework (SVEF), a systematic evaluation model for assessing extracted schemas across six complementary dimensions that capture essential structural and semantic properties: Data Type Accuracy, Required and Optional Fields, Multiple Type Support, Collection Structure Consistency, Entity Relationships, and Temporal Evolution Detection. Each dimension is defined through formal, data-driven metrics that quantify the degree to which an inferred schema reflects characteristics observed in the underlying dataset. In the present study, the framework is instantiated and evaluated for schemaless document-oriented data represented in JSON or JSON-like form. SVEF is evaluated using controlled benchmark datasets with curated ground-truth schemas and is applied to three representative schema extraction approaches. The results show that, while existing methods achieve strong performance in basic type reconstruction, substantial differences remain in modelling conditional fields, complex collection structures, and schema evolution over time. SVEF provides a consistent and interpretable basis for comparing schema extraction strategies and supports more rigorous empirical analysis of their behaviour in dynamic document-oriented data environments.