Skip to content

Commit 5ed123c

Browse files
JCZuurmondjtcohen6
authored andcommitted
Add session method
Documentation updates that go along with dbt-labs/dbt-spark#279
1 parent ebd0003 commit 5ed123c

1 file changed

Lines changed: 27 additions & 0 deletions

File tree

website/docs/reference/warehouse-profiles/spark-profile.md

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,11 @@ dbt-spark can connect to Spark clusters by three different methods:
2121
- `odbc` is the preferred method when connecting to Databricks. It supports connecting to a SQL Endpoint or an all-purpose interactive cluster.
2222
- `http` is a more generic method for connecting to a managed service that provides an HTTP endpoint. Currently, this includes connections to a Databricks interactive cluster.
2323
- `thrift` connects directly to the lead node of a cluster, either locally hosted / on premise or in the cloud (e.g. Amazon EMR).
24+
- `session` *FOR ADVANCED USERS* use a pySpark session.
25+
26+
:::info session
27+
The `session` connection method is intended for advanced users. This connection method is not supported by dbt cloud.
28+
:::
2429

2530
### ODBC
2631

@@ -112,6 +117,25 @@ Databricks interactive clusters can take several minutes to start up. You may
112117
include the optional profile configs `connect_timeout` and `connect_retries`,
113118
and dbt will periodically retry the connection.
114119

120+
### Session
121+
122+
Use the `session` method if you want to run `dbt` against a pySpark session.
123+
124+
<File name='~/.dbt/profiles.yml'>
125+
126+
```yaml
127+
your_profile_name:
128+
target: dev
129+
outputs:
130+
dev:
131+
type: spark
132+
method: session
133+
schema: [database/schema name]
134+
host: NA # not used, but required by `dbt-core`
135+
```
136+
137+
</File>
138+
115139
## Installation and Distribution
116140

117141
dbt's adapter for Apache Spark and Databricks is managed in its own repository, [dbt-spark](https://github.com/dbt-labs/dbt-spark). To use it,
@@ -130,6 +154,9 @@ $ pip install "dbt-spark[ODBC]"
130154
131155
# thrift or http connections
132156
$ pip install "dbt-spark[PyHive]"
157+
158+
# session connections
159+
$ pip install "dbt-spark[session]"
133160
```
134161

135162
## Caveats

0 commit comments

Comments
 (0)