[CT-1442] [Feature] Enable to access disabled resources with the `graph` macro

[yu-iskw]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

As far as I know, there are no macros to get information about disabled models. If we can take advantage of get information about disabled resources even in Jinja2 macros, that would be helpful.

For instance, I would like to implement a custom macro to drop tables which were created by dbt but are no longer used.
We don't manage the entire lifecycle of relations with dbt, because dbt 1.3 or later doesn't manage non-maintained models. We have to manually drop tables/views of disabled models.

Expected behavior

If information about disabled models with the graph macro, we can implement the desired feature to drop them as the code block below.

{% macro %}
  {% set disabled_models = graph.disabled.values() | selectattr("resource_type", "equals", "model") %}

  {% for disable_model in disabled_models %}
    {# implementation to drop a relation #}
  {% endfor %}
{% endmacro %}

Describe alternatives you've considered

As the manifest artifact contains the disabled object, we can alternatively implement desired features with a manifest JSON file. But, we can't refer the artifacts in macros.

• https://schemas.getdbt.com/dbt/manifest/v7/index.html

Besides, there is a discussion to automatically maintain schemas. The feature partially satisfies what I want to do as dropping tables/views of not-maintained models.

• [CT-426] [Feature] Manage full schema content with dbt by dropping tables/views not maintained in models #4957

Who will this benefit?

The feature would be useful to take advantage of disabled resources in macros.

For example. according to my research, brooklyn-data/dbt_artifacts enables us to persist information from the graph macro into a warehouse. I am guessing there are some dbt users to persist metadata of disabled resources as well. If disable resources get accessible with macros, we can persist them.

• https://github.com/brooklyn-data/dbt_artifacts/blob/main/macros/upload_models.sql

Are you interested in contributing this feature?

I would love to contribute this feature.

Anything else?

No response

[jtcohen6]

@yu-iskw Thanks for the thoughtful proposal, as always!

Schema management

I am intrigued that the primary use case you have in mind here is the same one we've been discussing over in #4957 / #5392.

There are limitations for each approach:

• If we take the "manage schemas" approach, you have to manually specify and configure the schemas to be managed. If you configure a custom schema for a one-off model, run that model, then disable it or remove it from your project, it's likely to keep hanging around the warehouse. (Upside: There are other ways in which users have wanted to configure schemas, and have been missing the place to do it: Configuring + managing `schemas` #5781.)
• If we take this approach, users would need to explicitly disable all models any time they rename — for all time, or for at least one "rinse" cycle. (Let's not call it vacuum.)

My personal preference leans toward the "schema management" approach, rather than adding significance to the act of disabling models. Same goes for being table to track them in metadata, within the revamped dbt_artifacts. Yes, they appear in manifest.json — but should they, really? Could it be enough to know, via longitudinal analysis, that there used to be an enabled model by a certain name, and no longer?

---

What is enabled: false, really?

This gets at a larger philosophical question that @lostmygithubaccount and I have been having over the past few weeks. We now have three separate ways to define a model inside a dbt project, and not have it run. What are the distinctions between each of these?

1. Ignoring a model. New in v1.3 — you can define a .dbtignore file that completely and utterly ignores reading the file in the first place, as if it didn't exist. This uses .gitignore-style syntax, so it's best if you want to ignore a whole suite of files based on their subdirectory name or file extension, yet still have them live within your project paths.
2. Disabling a model. dbt knows about it a little bit more, because it still reads the file—but you cannot ref a disabled model. You can use dbt logic (Jinja, vars, target, etc) to conditionally enable/disable it, with surgical precision. E.g. You have a package where per-adapter implementations of certain models are different enough to justify different model files, such as in our legacy snowplow package. Should that stretch to including metadata on the disabled model within the Jinja context variable, though? It's quite literally not in the graph! I really tend to think of disabling a model as if it were deleted, but could return at any moment.
3. Excluding a model. dbt knows everything about this model! It's a full member of your project and a full node in the DAG. You just don't want to run it right now (or perhaps not in the general case, via a default yaml selector).

Since enabled: false is between the other two extremes, it's the fuzziest to define. We should seek to develop (and document) a clear narrative for when & why someone should set enabled: false, in contrast to the other two options.

---

Metadata on disabled models

One concern: I know the way we store disabled nodes isn't the most sensible today, and could be liable to change in the future. For instance, while unique_id must be unique among all enabled nodes, there can be multiple disabled nodes with the same unique_id, so each entry in the disabled dict is actually an array (unlike in nodes). It's possible to see things like:

-- models/subfolder_one/my_disabled_model.sql
select 1 as id

-- models/subfolder_two/my_disabled_model.sql
select 1 as id

Yielding manifest entries like:

  "disabled": {
    "model.test.my_disabled_model": [
      {
        "resource_type": "model",
        "unique_id": "model.test.my_disabled_model",
        "path": "subfolder_one/my_disabled_model.sql",
        "raw_code": "{{ config(enabled = false) }}\nselect 1 as id",
        ...
      },
      {
        "resource_type": "model",
        "unique_id": "model.test.my_disabled_model",
        "path": "subfolder_two/my_disabled_model.sql",
        "raw_code": "{{ config(enabled = false) }}\n\nselect 1 as id",
        ...
      },
    ]
  },

---

Having written all that, I'm leaning sorta against adding a disabled context variable in parallel to the graph — but very curious to hear thoughts from folks who might disagree!

[lostmygithubaccount]

closing as not planned -- please let us know if you disagree!

[yu-iskw]

@jtcohen6 Thank you for the detailed explanation. Good to know the three philosophical aspects of disable: false. Now I agree with the 'manage schemas ' approach.