The Monday 9:05am Dashboard Meltdown: Data Quality Monitoring That Stops the Blast Radius
Build quality gates that catch bad data before executives, finance, and RevOps do—using tests, SLOs, and alerting that actually match how analytics breaks in the real world.
The goal isn’t to “monitor everything.” The goal is to stop bad data from getting promoted to the tables your business trusts.Back to all posts
Key takeaways
- Treat datasets like products: define **freshness, completeness, and correctness SLOs** (and page on breaches).
- Put checks at **choke points**: ingestion, transformation, and publishing—not just on dashboards.
- Start with **deterministic tests** (nulls, uniqueness, referential integrity, schema) before fancy anomaly detection.
- Wire monitoring into the tools you already run: `dbt`, `Great Expectations`/`Soda`, `Airflow`/`Dagster`, `Prometheus`/`Grafana`.
- Make failures actionable: every alert needs an **owner, runbook, and rollback/backfill plan**.
Implementation checklist
- Pick 10–20 critical metrics/tables and define freshness/volume/error budget targets
- Add `dbt` tests for `not_null`, `unique`, accepted values, and relationship constraints
- Add schema drift detection on raw/bronze tables (columns, types, nullability)
- Implement a publish gate so “gold” tables only update if checks pass
- Route alerts to Slack + PagerDuty with a clear owner and severity model
- Add a runbook: how to quarantine data, re-run jobs, and backfill safely
- Track outcomes: broken dashboards/week, incident count, MTTR, and stakeholder time saved
Questions we hear from teams
- Should we start with Great Expectations/Soda, or with dbt tests?
- Start with `dbt` tests for deterministic constraints (nulls, uniqueness, relationships, accepted values). They’re cheap to run and easy to operationalize. Add Great Expectations or `Soda Core` where you need richer checks like distribution shifts, row-count bounds, or complex business rules.
- How do we prevent bad data without blocking the entire pipeline?
- Use a **publish gate**: build and validate into staging tables/partitions, then promote via an atomic swap (or partition promotion). If validation fails, you keep serving the last known-good “gold” table while you investigate.
- What’s the first metric to monitor if we can only pick one?
- Freshness on your tier-1 facts (e.g., `time since last successful load`). Freshness breaches are the most common trigger for downstream failures and the easiest to page on with low false positives.
Ready to modernize your codebase?
Let GitPlumbers help you transform AI-generated chaos into clean, scalable applications.