Trustworthy DataOps for the Medallion Lakehouse

Abstract

Modern data platforms increasingly rely on DataOps to orchestrate large-scale, automated data pipelines. However, most existing approaches prioritise speed and agility while treating trust spanning data quality, compliance, and lineage as a secondary concern, which is particularly problematic in regulated environments. This paper introduces a Trustworthy DataOps framework for a Medallion Lakehouse architecture, operated under a Data Mesh model, where trust becomes a first-class, measurable property of the pipeline. The proposal embeds Trust Data Gates between the Bronze, Silver and Gold layers, governed by a composite Trust Data Score (TDS) that aggregates three dimensions: data quality, policy and regulatory compliance, and lineage/observability. Promotion thresholds and circuit-breaker mechanisms prevent untrusted data from reaching analytical and ML workloads, while quarantine paths ensure controlled recovery from failures. The framework is operationalised through a continuous lifecycle (Discover, Instrument, Enforce, Observe, Improve) and set of trust-aligned KPIs that monitor the health of both data and processes across layers. Together, these mechanisms enable measurable trust, shift-left compliance and improved auditability, turning Medallion Lakehouse data products into explainable, governable and recoverable assets rather than opaque pipeline outputs.