How can I automate data lineage for ETL pipelines without heavy maintenance?

[Jacob_G]

I’ve been trying to build a reliable data lineage map for our ETL pipelines, but the sheer number of transformations and dependencies between our Snowflake tables and dbt models makes it feel impossible to track manually. I’m not even sure where to start with automating this, or what tools would actually capture everything without creating a huge maintenance burden.

[ScottPL]

We burned a lot of cycles trying to map everything by hand, then realized we needed a living map instead of a static diagram. We started by pulling dbt's dependencies into a doc and watching which Snowflake objects got touched, little by little.

[MilaKM]

We hooked OpenLineage into our dbt runs and the Snowflake side via the API, and we did get an auto graph of which models touch which tables. It helped surface big gaps, but macros and ad hoc SQL still weren't captured, and maintenance grew as models moved.

[OliviaOM]

We tried DataHub as a metadata sink and used its UI to connect dbt models to Snowflake tables. It was decent for searching and creds inventory, but turning that into a clean dependency map took custom scripts and still felt brittle.

[LarryL]

Is the real hurdle maybe naming conventions and scope rather than tool coverage—the things people call lineage that aren’t actually dependencies?

[Brandon89]

We kept it lightweight: rely on dbt docs for core dependencies, pull a nightly snapshot of Snowflake objects, and store a simple map in a small metadata db. It didn’t capture every edge, but it surprised us by catching the big wins without drowning in maintenance.

[CharlesS]

We sometimes drift toward dashboards, but the team ends up chasing noisy edges. Sometimes I think we should just track the critical ETL paths and pair that with release notes and tests.