[Feature Request] Ability to support Spark based DBT Models where all references will be <tablename>.<column name>. They throw error in the code

[Sathish-Metcash]

📝 Feature Description

Support DBT Models without database / catalog ( Lakehouse Pattern).
i.e. Support 2 part references . instead of always requiring 3 part naming expected by the code.

🤔 Problem Statement

To support DBT models which are built with Lakehouse Adapter ( FabricSpark) which are of dialect 'spark' . Spark which is well supported by 'sqlglot' libraries.

The issue is current code expects minimum 3 part naming <db/catalog/schema>.

.
Without giving 3 part it throws error like 'NoneType' has no attribute 'meta'.

Even after providing default value for database , The Table Linking code for lineage seems to searching for table Catalog.Schema.Table pattern and in our case it is looking for table in DBT manifest for "..". The leading "." is forcing the lineage to be created as '_HARDCODED__REF' even though it already exists in the manifest.json as ,

🚀 Proposed Solution

By applying the 2 fixes the python source code, this can be easily supported.

1. To check if DB is null while parsing the schema from manifest.json ( created by DBT docs), default to "default" to avoid the 'NoneType' runtime error
2. When linking tables , if the catalog is not provided , remove the leading "." when looking up tables to make sure lineage links to correct tables . Without this fix , it will link to HARDCODED table without any column information

📋 Alternatives

🙋 Contribution Interest

• I’d like to help implement this feature

✅ Additional Context

We appreciate dbt-colibri, its looks quite promising for our use case.

[b-ned]

Hi @Sathish-Metcash ,

Thanks for raising this issue!

We've seen a similar issue for clickhouse, where the table referrence in SQL has 2 parts (schema.table) versus 3 parts in most other dialects.

I think we can fix this in a similar way, as we did in #83,
Could you give an example of what the
compiled sql, and the source manifest node would look like? With that we can create a proper solution.
(for clickhouse these would be the snippets from manifest: https://github.com/b-ned/dbt-colibri/pull/83/files#diff-8f5275d82a7922f8599f1c9e4bf101a3817175ee80683f32595916fa95ce817dR728)

Thanks!