You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: website/docs/reference/warehouse-profiles/spark-profile.md
+27Lines changed: 27 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -21,6 +21,11 @@ dbt-spark can connect to Spark clusters by three different methods:
21
21
-`odbc` is the preferred method when connecting to Databricks. It supports connecting to a SQL Endpoint or an all-purpose interactive cluster.
22
22
-`http` is a more generic method for connecting to a managed service that provides an HTTP endpoint. Currently, this includes connections to a Databricks interactive cluster.
23
23
-`thrift` connects directly to the lead node of a cluster, either locally hosted / on premise or in the cloud (e.g. Amazon EMR).
24
+
-`session`*FOR ADVANCED USERS* use a pySpark session.
25
+
26
+
:::info session
27
+
The `session` connection method is intended for advanced users. This connection method is not supported by dbt cloud.
28
+
:::
24
29
25
30
### ODBC
26
31
@@ -112,6 +117,25 @@ Databricks interactive clusters can take several minutes to start up. You may
112
117
include the optional profile configs `connect_timeout` and `connect_retries`,
113
118
and dbt will periodically retry the connection.
114
119
120
+
### Session
121
+
122
+
Use the `session` method if you want to run `dbt` against a pySpark session.
123
+
124
+
<File name='~/.dbt/profiles.yml'>
125
+
126
+
```yaml
127
+
your_profile_name:
128
+
target: dev
129
+
outputs:
130
+
dev:
131
+
type: spark
132
+
method: session
133
+
schema: [database/schema name]
134
+
host: NA # not used, but required by `dbt-core`
135
+
```
136
+
137
+
</File>
138
+
115
139
## Installation and Distribution
116
140
117
141
dbt's adapter for Apache Spark and Databricks is managed in its own repository, [dbt-spark](https://github.com/dbt-labs/dbt-spark). To use it,
0 commit comments