Support Delta constraints

[allisonwang-db]

Description

This PR adds support for Delta Constraints with dbt table models and incremental models. It supports two types of constraints: model level Check constraint and column level not_null constraint.

Users can specify model-level and column-level constraints under the meta field of the model config. For example:

# schema.yml
models:
  - name: my_model
    meta:
      constraints:
        - name: id_greater_than_zero
          condition: id > 0	
    columns:
      - name: id
      - name: name
        meta:
          constraint: not_null

Constraints specified will be created if the persist_constraints config is enabled (default: false):

-- my_model.sql
{{ config(
    materailized='table',
    persist_constraints=True
) }}
...

Note, Delta constraints are only available in Databricks Runtime 7.4 and above. So DBR 7.3 LTS won't support this feature.

Checklist

• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

[allisonwang-db]

cc @jtcohen6

[ueshin]

I'm just wondering if we need to enable them? I'm feeling it's ok to enable automatically if there is such configs in schema.yml.
I also feel a bit weird that the config named persist_docs enables the constraints.

[jtcohen6]

I'm just wondering if we need to enable them? I'm feeling it's ok to enable automatically if there is such configs in schema.yml.

Fair, I think there's good arguments to be made in either direction. From my perspective, meta is usually for user's own metadata / annotation purposes, with no functional effect in dbt, until/unless they opt into that via an extension macro/package or custom config.

I also feel a bit weird that the config named persist_docs enables the constraints.

Agreed. Even if the mechanism to add these constraints is very very similar to persist_docs, I'd rather see this enabled via another config. Users (and therefore adapter plugins) can define whichever configs they like. All that would require is another macro call within the materialization, right alongside the calls to persist_docs:

dbt-databricks/dbt/include/databricks/macros/materializations/table.sql

Lines 24 to 26 in 5d44dc6

	{% do persist_docs(target_relation, model) %}

dbt-databricks/dbt/include/databricks/macros/materializations/incremental/incremental.sql

Lines 43 to 45 in 5d44dc6

	{% do persist_docs(target_relation, model) %}

[allisonwang-db]

Make sense! I will use a different config for constraints.

[ueshin]

We might also need it in snapshot.sql?

dbt-databricks/dbt/include/databricks/macros/materializations/snapshot.sql

Lines 96 to 98 in 6dd2a8f

	{% do persist_docs(target_relation, model) %}

[ueshin]

LGTM.

Note, Delta constraints are only available in Databricks Runtime 7.4 and above. So DBR 7.3 LTS won't support this feature.

If it doesn't support this feature with DBR 7.3 LTS, we should mention it in README to say:

• The dbt-databricks adapter has been tested against Databricks SQL and Databricks runtime releases 9.1 LTS and later.

or

• Delta constraints feature only works with Databricks runtime releases 9.1 LTS and later.

@superdupershant Which would you recommend, or other option?

[ueshin]

Thanks! merging.