Skip to content

Commit 380a8e9

Browse files
McKnight-42gshankjtcohen6emmyoop
authored andcommitted
Add Grant SQL Macros (dbt-labs#5369)
* init push or ct-660 work * changes to default versions of get_show_grant_sql and get_grant_sql * completing init default versions of all macros being called for look over and collaboration * minor update to should_revoke * post pairing push up (does have log statements to make sure we remove) * minor spacing changes * minor changes, and removal of logs so people can have clean grab of code * minor changes to how get_revoke_sql works * init attempt at applying apply_grants to all materialzations * name change from recipents -> grantee * minor changes * working on making a context to handle the diff gathering between grant_config and curreent_grants to see what needs to be revoked, I know if we assign a role, and a model becomes dependent on it we can't drop the role now still not seeing the diff appear in log * removing logs from most materializations to better track diff of grants generation logs * starting to build out postgres get_show_grant_sql getting empty query errors hopefully will clear up as we add the other postgres versions of macros and isn't a psycopg2 issue as indicated by searching * 6/27 eod update looking into diff_grants variable not getting passed into get_revoke_sql * changes to loop cases * changes after pairing meeting * adding apply_grants to create_or_replace_view.sql * models are building but testing out small issues around revoke statement never building * postgrest must fixes from jeremy's feedback * postgres minor change to standarize_grants_dict * updating after pairing with dough and jeremey incorporating the new version of should revoke logic. * adding ref of diff_of_two_dicts to base keys ref * change of method type for standardize_grants_dict * minor update trying to fix unit test * changes based on morning feedback * change log message in default_apply_grants macro * CT-808 grant adapter tests (dbt-labs#5447) * Create initial test for grants Co-authored-by: Jeremy Cohen <[email protected]> * rename grant[privilege] -> grant_config[privilege] * postgres macro rename to copy_grants * CT-808 more grant adapter tests (dbt-labs#5452) * Add tests for invalid user and invalid privilege * Add more grant tests * Macro touch ups * Many more tests * Allow easily replacing privilege names * Keep adding tests * Refactor macros to return lists, fix test * Code checks * Keep tweaking tests * Revert cool grantees join bc Snowflake isnt happy * Use Postgres/BQ as standard for standardize_grants_dict * Code checks * add missing replace * small replace tweak, add additional dict diffs * All tests passing on BQ * Add type cast to test_snapshot_grants * Refactor for DRYer apply_grants macros Co-authored-by: Jeremy Cohen <[email protected]> Co-authored-by: Emily Rockman <[email protected]> * update to main, create changelog, whitespace fixes Co-authored-by: Gerda Shank <[email protected]> Co-authored-by: Jeremy Cohen <[email protected]> Co-authored-by: Emily Rockman <[email protected]>
1 parent aa64bc2 commit 380a8e9

25 files changed

Lines changed: 924 additions & 21 deletions

File tree

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
kind: Features
2+
body: Allow users to define grants as a reasonable default in the dbt_project.yml
3+
or within each model sql or yml file combined.
4+
time: 2022-07-11T11:15:14.695386-05:00
5+
custom:
6+
Author: McKnight-42
7+
Issue: "5263"
8+
PR: "5369"
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
kind: Under the Hood
2+
body: Add tests for SQL grants
3+
time: 2022-07-06T21:50:01.498562-04:00
4+
custom:
5+
Author: gshank
6+
Issue: "5437"
7+
PR: "5447"

.github/workflows/main.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -125,6 +125,9 @@ jobs:
125125
TOXENV: integration
126126
PYTEST_ADDOPTS: "-v --color=yes -n4 --csv integration_results.csv"
127127
DBT_INVOCATION_ENV: github-actions
128+
DBT_TEST_USER_1: dbt_test_user_1
129+
DBT_TEST_USER_2: dbt_test_user_2
130+
DBT_TEST_USER_3: dbt_test_user_3
128131

129132
steps:
130133
- name: Check out the repository

.github/workflows/structured-logging-schema-check.yml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,11 @@ jobs:
3030
LOG_DIR: "/home/runner/work/dbt-core/dbt-core/logs"
3131
# tells integration tests to output into json format
3232
DBT_LOG_FORMAT: "json"
33+
# Additional test users
34+
DBT_TEST_USER_1: dbt_test_user_1
35+
DBT_TEST_USER_2: dbt_test_user_2
36+
DBT_TEST_USER_3: dbt_test_user_3
37+
3338
steps:
3439
- name: checkout dev
3540
uses: actions/checkout@v2

core/dbt/adapters/base/impl.py

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -159,6 +159,7 @@ class BaseAdapter(metaclass=AdapterMeta):
159159
- convert_datetime_type
160160
- convert_date_type
161161
- convert_time_type
162+
- standardize_grants_dict
162163
163164
Macros:
164165
- get_catalog
@@ -538,6 +539,33 @@ def list_relations_without_caching(self, schema_relation: BaseRelation) -> List[
538539
"`list_relations_without_caching` is not implemented for this " "adapter!"
539540
)
540541

542+
###
543+
# Methods about grants
544+
###
545+
@available
546+
def standardize_grants_dict(self, grants_table: agate.Table) -> dict:
547+
"""Translate the result of `show grants` (or equivalent) to match the
548+
grants which a user would configure in their project.
549+
550+
Ideally, the SQL to show grants should also be filtering:
551+
filter OUT any grants TO the current user/role (e.g. OWNERSHIP).
552+
If that's not possible in SQL, it can be done in this method instead.
553+
554+
:param grants_table: An agate table containing the query result of
555+
the SQL returned by get_show_grant_sql
556+
:return: A standardized dictionary matching the `grants` config
557+
:rtype: dict
558+
"""
559+
grants_dict: Dict[str, List[str]] = {}
560+
for row in grants_table:
561+
grantee = row["grantee"]
562+
privilege = row["privilege_type"]
563+
if privilege in grants_dict.keys():
564+
grants_dict[privilege].append(grantee)
565+
else:
566+
grants_dict.update({privilege: [grantee]})
567+
return grants_dict
568+
541569
###
542570
# Provided methods about relations
543571
###

core/dbt/context/base.py

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
import json
22
import os
3-
from typing import Any, Dict, NoReturn, Optional, Mapping, Iterable, Set
3+
from typing import Any, Dict, NoReturn, Optional, Mapping, Iterable, Set, List
44

55
from dbt import flags
66
from dbt import tracking
@@ -657,6 +657,35 @@ def print(msg: str) -> str:
657657
print(msg)
658658
return ""
659659

660+
@contextmember
661+
@staticmethod
662+
def diff_of_two_dicts(
663+
dict_a: Dict[str, List[str]], dict_b: Dict[str, List[str]]
664+
) -> Dict[str, List[str]]:
665+
"""
666+
Given two dictionaries of type Dict[str, List[str]]:
667+
dict_a = {'key_x': ['value_1', 'VALUE_2'], 'KEY_Y': ['value_3']}
668+
dict_b = {'key_x': ['value_1'], 'key_z': ['value_4']}
669+
Return the same dictionary representation of dict_a MINUS dict_b,
670+
performing a case-insensitive comparison between the strings in each.
671+
All keys returned will be in the original case of dict_a.
672+
returns {'key_x': ['VALUE_2'], 'KEY_Y': ['value_3']}
673+
"""
674+
675+
dict_diff = {}
676+
dict_b_lowered = {k.casefold(): [x.casefold() for x in v] for k, v in dict_b.items()}
677+
for k in dict_a:
678+
if k.casefold() in dict_b_lowered.keys():
679+
diff = []
680+
for v in dict_a[k]:
681+
if v.casefold() not in dict_b_lowered[k.casefold()]:
682+
diff.append(v)
683+
if diff:
684+
dict_diff.update({k: diff})
685+
else:
686+
dict_diff.update({k: dict_a[k]})
687+
return dict_diff
688+
660689

661690
def generate_base_context(cli_vars: Dict[str, Any]) -> Dict[str, Any]:
662691
ctx = BaseContext(cli_vars)
Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
{# ------- BOOLEAN MACROS --------- #}
2+
3+
{#
4+
-- COPY GRANTS
5+
-- When a relational object (view or table) is replaced in this database,
6+
-- do previous grants carry over to the new object? This may depend on:
7+
-- whether we use alter-rename-swap versus CREATE OR REPLACE
8+
-- user-supplied configuration (e.g. copy_grants on Snowflake)
9+
-- By default, play it safe, assume TRUE: that grants ARE copied over.
10+
-- This means dbt will first "show" current grants and then calculate diffs.
11+
-- It may require an additional query than is strictly necessary,
12+
-- but better safe than sorry.
13+
#}
14+
15+
{% macro copy_grants() %}
16+
{{ return(adapter.dispatch('copy_grants', 'dbt')()) }}
17+
{% endmacro %}
18+
19+
{% macro default__copy_grants() %}
20+
{{ return(True) }}
21+
{% endmacro %}
22+
23+
24+
{#
25+
-- SUPPORT MULTIPLE GRANTEES PER DCL STATEMENT
26+
-- Does this database support 'grant {privilege} to {grantee_1}, {grantee_2}, ...'
27+
-- Or must these be separate statements:
28+
-- `grant {privilege} to {grantee_1}`;
29+
-- `grant {privilege} to {grantee_2}`;
30+
-- By default, pick the former, because it's what we prefer when available.
31+
#}
32+
33+
{% macro support_multiple_grantees_per_dcl_statement() %}
34+
{{ return(adapter.dispatch('support_multiple_grantees_per_dcl_statement', 'dbt')()) }}
35+
{% endmacro %}
36+
37+
{%- macro default__support_multiple_grantees_per_dcl_statement() -%}
38+
{{ return(True) }}
39+
{%- endmacro -%}
40+
41+
42+
{% macro should_revoke(existing_relation, full_refresh_mode=True) %}
43+
44+
{% if not existing_relation %}
45+
{#-- The table doesn't already exist, so no grants to copy over --#}
46+
{{ return(False) }}
47+
{% elif full_refresh_mode %}
48+
{#-- The object is being REPLACED -- whether grants are copied over depends on the value of user config --#}
49+
{{ return(copy_grants()) }}
50+
{% else %}
51+
{#-- The table is being merged/upserted/inserted -- grants will be carried over --#}
52+
{{ return(True) }}
53+
{% endif %}
54+
55+
{% endmacro %}
56+
57+
{# ------- DCL STATEMENT TEMPLATES --------- #}
58+
59+
{% macro get_show_grant_sql(relation) %}
60+
{{ return(adapter.dispatch("get_show_grant_sql", "dbt")(relation)) }}
61+
{% endmacro %}
62+
63+
{% macro default__get_show_grant_sql(relation) %}
64+
show grants on {{ relation }}
65+
{% endmacro %}
66+
67+
68+
{% macro get_grant_sql(relation, privilege, grantees) %}
69+
{{ return(adapter.dispatch('get_grant_sql', 'dbt')(relation, privilege, grantees)) }}
70+
{% endmacro %}
71+
72+
{%- macro default__get_grant_sql(relation, privilege, grantees) -%}
73+
grant {{ privilege }} on {{ relation }} to {{ grantees | join(', ') }}
74+
{%- endmacro -%}
75+
76+
77+
{% macro get_revoke_sql(relation, privilege, grantees) %}
78+
{{ return(adapter.dispatch('get_revoke_sql', 'dbt')(relation, privilege, grantees)) }}
79+
{% endmacro %}
80+
81+
{%- macro default__get_revoke_sql(relation, privilege, grantees) -%}
82+
revoke {{ privilege }} on {{ relation }} from {{ grantees | join(', ') }}
83+
{%- endmacro -%}
84+
85+
86+
{# ------- RUNTIME APPLICATION --------- #}
87+
88+
{% macro get_dcl_statement_list(relation, grant_config, get_dcl_macro) %}
89+
{{ return(adapter.dispatch('get_dcl_statement_list', 'dbt')(relation, grant_config, get_dcl_macro)) }}
90+
{% endmacro %}
91+
92+
{%- macro default__get_dcl_statement_list(relation, grant_config, get_dcl_macro) -%}
93+
{#
94+
-- Unpack grant_config into specific privileges and the set of users who need them granted/revoked.
95+
-- Depending on whether this database supports multiple grantees per statement, pass in the list of
96+
-- all grantees per privilege, or (if not) template one statement per privilege-grantee pair.
97+
-- `get_dcl_macro` will be either `get_grant_sql` or `get_revoke_sql`
98+
#}
99+
{%- set dcl_statements = [] -%}
100+
{%- for privilege, grantees in grant_config.items() %}
101+
{%- if support_multiple_grantees_per_dcl_statement() and grantees -%}
102+
{%- set dcl = get_dcl_macro(relation, privilege, grantees) -%}
103+
{%- do dcl_statements.append(dcl) -%}
104+
{%- else -%}
105+
{%- for grantee in grantees -%}
106+
{% set dcl = get_dcl_macro(relation, privilege, [grantee]) %}
107+
{%- do dcl_statements.append(dcl) -%}
108+
{% endfor -%}
109+
{%- endif -%}
110+
{%- endfor -%}
111+
{{ return(dcl_statements) }}
112+
{%- endmacro %}
113+
114+
115+
{% macro call_dcl_statements(dcl_statement_list) %}
116+
{{ return(adapter.dispatch("call_dcl_statements", "dbt")(dcl_statement_list)) }}
117+
{% endmacro %}
118+
119+
{% macro default__call_dcl_statements(dcl_statement_list) %}
120+
{#
121+
-- By default, supply all grant + revoke statements in a single semicolon-separated block,
122+
-- so that they're all processed together.
123+
124+
-- Some databases do not support this. Those adapters will need to override this macro
125+
-- to run each statement individually.
126+
#}
127+
{% call statement('grants') %}
128+
{% for dcl_statement in dcl_statement_list %}
129+
{{ dcl_statement }};
130+
{% endfor %}
131+
{% endcall %}
132+
{% endmacro %}
133+
134+
135+
{% macro apply_grants(relation, grant_config, should_revoke) %}
136+
{{ return(adapter.dispatch("apply_grants", "dbt")(relation, grant_config, should_revoke)) }}
137+
{% endmacro %}
138+
139+
{% macro default__apply_grants(relation, grant_config, should_revoke=True) %}
140+
{#-- If grant_config is {} or None, this is a no-op --#}
141+
{% if grant_config %}
142+
{% if should_revoke %}
143+
{#-- We think previous grants may have carried over --#}
144+
{#-- Show current grants and calculate diffs --#}
145+
{% set current_grants_table = run_query(get_show_grant_sql(relation)) %}
146+
{% set current_grants_dict = adapter.standardize_grants_dict(current_grants_table) %}
147+
{% set needs_granting = diff_of_two_dicts(grant_config, current_grants_dict) %}
148+
{% set needs_revoking = diff_of_two_dicts(current_grants_dict, grant_config) %}
149+
{% if not (needs_granting or needs_revoking) %}
150+
{{ log('On ' ~ relation ~': All grants are in place, no revocation or granting needed.')}}
151+
{% endif %}
152+
{% else %}
153+
{#-- We don't think there's any chance of previous grants having carried over. --#}
154+
{#-- Jump straight to granting what the user has configured. --#}
155+
{% set needs_revoking = {} %}
156+
{% set needs_granting = grant_config %}
157+
{% endif %}
158+
{% if needs_granting or needs_revoking %}
159+
{% set revoke_statement_list = get_dcl_statement_list(relation, needs_revoking, get_revoke_sql) %}
160+
{% set grant_statement_list = get_dcl_statement_list(relation, needs_granting, get_grant_sql) %}
161+
{% set dcl_statement_list = revoke_statement_list + grant_statement_list %}
162+
{% if dcl_statement_list %}
163+
{{ call_dcl_statements(dcl_statement_list) }}
164+
{% endif %}
165+
{% endif %}
166+
{% endif %}
167+
{% endmacro %}

core/dbt/include/global_project/macros/materializations/models/incremental/incremental.sql

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@
2020
-- BEGIN, in a separate transaction
2121
{%- set preexisting_intermediate_relation = load_cached_relation(intermediate_relation)-%}
2222
{%- set preexisting_backup_relation = load_cached_relation(backup_relation) -%}
23+
-- grab current tables grants config for comparision later on
24+
{% set grant_config = config.get('grants') %}
2325
{{ drop_relation_if_exists(preexisting_intermediate_relation) }}
2426
{{ drop_relation_if_exists(preexisting_backup_relation) }}
2527

@@ -59,6 +61,9 @@
5961
{% do to_drop.append(backup_relation) %}
6062
{% endif %}
6163

64+
{% set should_revoke = should_revoke(existing_relation, full_refresh_mode) %}
65+
{% do apply_grants(target_relation, grant_config, should_revoke=should_revoke) %}
66+
6267
{% do persist_docs(target_relation, model) %}
6368

6469
{% if existing_relation is none or existing_relation.is_view or should_full_refresh() %}

core/dbt/include/global_project/macros/materializations/models/table/table.sql

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,8 @@
1414
{%- set backup_relation = make_backup_relation(target_relation, backup_relation_type) -%}
1515
-- as above, the backup_relation should not already exist
1616
{%- set preexisting_backup_relation = load_cached_relation(backup_relation) -%}
17+
-- grab current tables grants config for comparision later on
18+
{% set grant_config = config.get('grants') %}
1719

1820
-- drop the temp relations if they exist already in the database
1921
{{ drop_relation_if_exists(preexisting_intermediate_relation) }}
@@ -40,6 +42,9 @@
4042

4143
{{ run_hooks(post_hooks, inside_transaction=True) }}
4244

45+
{% set should_revoke = should_revoke(existing_relation, full_refresh_mode=True) %}
46+
{% do apply_grants(target_relation, grant_config, should_revoke=should_revoke) %}
47+
4348
{% do persist_docs(target_relation, model) %}
4449

4550
-- `COMMIT` happens here

core/dbt/include/global_project/macros/materializations/models/view/create_or_replace_view.sql

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,12 +13,12 @@
1313
{%- set identifier = model['alias'] -%}
1414

1515
{%- set old_relation = adapter.get_relation(database=database, schema=schema, identifier=identifier) -%}
16-
1716
{%- set exists_as_view = (old_relation is not none and old_relation.is_view) -%}
1817

1918
{%- set target_relation = api.Relation.create(
2019
identifier=identifier, schema=schema, database=database,
2120
type='view') -%}
21+
{% set grant_config = config.get('grants') %}
2222

2323
{{ run_hooks(pre_hooks) }}
2424

@@ -34,6 +34,9 @@
3434
{{ get_create_view_as_sql(target_relation, sql) }}
3535
{%- endcall %}
3636

37+
{% set should_revoke = should_revoke(exists_as_view, full_refresh_mode=True) %}
38+
{% do apply_grants(target_relation, grant_config, should_revoke=True) %}
39+
3740
{{ run_hooks(post_hooks) }}
3841

3942
{{ return({'relations': [target_relation]}) }}

0 commit comments

Comments
 (0)