Data Lineage Tracking

By Michael Samuel Naeem · July 20, 2025 · 5 min read

The question that justifies lineage investment: "If I change this column, what breaks?" Without an answer, schema migrations become roulette. Lineage turns them into a checklist — every downstream dashboard, export, and model flagged before merge.

Table vs column lineage

Table lineage — raw.orders → stg_orders → fct_orders → Looker explore. Enough for ownership, freshness alerts, and coarse impact analysis.

Column lineage — raw.orders.discount_code → stg_orders.promo_id → fct_orders.net_revenue (via gmv - discount). Required for PII tagging propagation, GDPR deletion requests, and precise migration scoping.

Most teams start table-level; add column-level where compliance or complex transforms demand it.

Capture mechanisms

OpenLineage standardizes event emission:

# Simplified OpenLineage emit from a Python job
from openlineage.client import OpenLineageClient
from openlineage.client.run import Run, Job, Dataset

client = OpenLineageClient(url="http://marquez:5000")
client.emit(
    Run(
        runId=str(uuid.uuid4()),
        job=Job(namespace="prod", name="daily_revenue_etl"),
        inputs=[Dataset(namespace="snowflake", name="analytics.stg_orders")],
        outputs=[Dataset(namespace="snowflake", name="analytics.fct_revenue")],
    )
)

Orchestrators (Airflow with the OpenLineage provider, Dagster's asset graph) emit these automatically when configured.

Impact analysis workflow

When deprecating legacy_user_id:

1. Query lineage for downstream nodes within N hops
2. Notify owners from catalog metadata
3. Block schema PR until acks or migration tickets exist
4. Run regression tests on affected dbt models

# dbt exposure links dashboards to models — feeds lineage
exposures:
  - name: executive_revenue_dashboard
    type: dashboard
    maturity: high
    depends_on:
      - ref('fct_daily_revenue')
    owner:
      email: [email protected]

Exposures close the gap between warehouse tables and business consumers.

Lineage quality problems

Missing ad-hoc queries. Analysts create CREATE TABLE AS SELECT outside dbt — invisible until you mine warehouse audit logs.

Broken parser coverage. Exotic SQL, multi-statement scripts, and dynamic SQL defeat static analysis. Tag those jobs manual-review.

Environment mixing. Dev tables linked to prod dashboards poison graphs. Namespace by environment strictly.

Stale edges. Deleted models linger until garbage collection. Schedule lineage reconciliation jobs.

Treat lineage confidence as a metric — % edges verified in last 7 days.

Privacy and compliance use cases

Lineage answers "where does email propagate?" for DSAR deletion — trace columns tagged pii:email through transforms, identify all materialized copies, queue deletion or anonymization jobs.

Audit trails pair with Iceberg/Delta time travel: lineage says what depends on what; snapshots say what values looked like when.

Organizational adoption

Embed lineage in PR bots: "This change affects 4 models and 2 Looker explores." Don't make engineers open a separate portal for every question.

Start with production dbt project + top Airflow DAGs. Expand to Spark and BI when table-level coverage exceeds 80% of critical assets.

Lineage in CI/CD workflows

Embed lineage checks in the development workflow, not as a separate portal:

PR bot integration:

# GitHub Action: on dbt model change, query lineage API
- name: Lineage impact check
  run: |
    AFFECTED=$(datahub get --urn "urn:li:dataset:(urn:li:dataPlatform:dbt,${{ env.MODEL_NAME }},PROD)" --downstream)
    echo "Downstream impact: $AFFECTED"
    # Post as PR comment

When an engineer changes stg_orders, the PR automatically lists affected marts, dashboards, and ML features. Reviewers see blast radius before approving.

Schema change gate:

Block column renames/drops until downstream owners acknowledge:

1. Engineer proposes schema change in PR
2. Lineage API returns downstream nodes
3. Bot creates ack tickets for each downstream owner
4. Merge blocked until all acks received or migration plan documented

OpenLineage in practice

Standardize event emission across orchestrators:

# Airflow with OpenLineage provider
from airflow.providers.openlineage.plugins.listener import OpenLineageListener

# Dagster emits automatically when configured
# Spark: spark.openlineage.* config properties

Events flow to Marquez or DataHub:

Airflow/Dagster/Spark → OpenLineage events → Marquez → DataHub ingestion

Start with table-level input/output datasets. Add column-level facets when compliance requires it.

Lineage for ML feature stores

ML pipelines often escape traditional lineage:

raw.events → dbt stg_events → fct_user_features → feature store → model v3

Register feature store tables as dbt exposures or OpenLineage datasets. When fct_user_features.churn_score changes, lineage should show which model versions and serving endpoints are affected.

Without this, data scientists discover feature schema changes when predictions silently degrade.

Failure modes

• Ad-hoc SQL invisible — analysts CREATE TABLE AS SELECT outside dbt; mine warehouse audit logs to discover
• Stale lineage graph — deleted models linger as orphaned nodes; schedule reconciliation
• Environment mixing — dev tables linked to prod dashboards; namespace strictly by environment
• Manual diagrams as source of truth — outdated within weeks; automate or don't bother
• Column lineage without confidence score — parser-guessed edges treated as verified; tag confidence level

Production checklist

• dbt manifest.json ingested into lineage catalog
• OpenLineage events emitted from orchestrator (Airflow/Dagster)
• dbt exposures link models to dashboards and ML features
• PR bot shows downstream impact on model changes
• Schema change gate requires downstream owner ack
• Lineage freshness metric tracked (% edges verified in last 7 days)
• Ad-hoc query audit supplements automated capture

Column-level lineage via SQL parsing (sqlglot, DataHub SQL parser) adds precision but increases maintenance — enable for PII-tagged columns first, expand based on compliance audit requirements.

Run lineage impact analysis before every column rename in production — downstream dashboards break silently long before the warehouse team notices schema drift.

Resources

• OpenLineage project
• Marquez — OpenLineage reference implementation
• DataHub lineage features
• dbt exposures documentation
• sqllineage — Python SQL lineage parser