File tree Expand file tree Collapse file tree 1 file changed +1
-33
lines changed
Expand file tree Collapse file tree 1 file changed +1
-33
lines changed Original file line number Diff line number Diff line change 1- # My shiny new job
2-
3- Versatile Data Kit feature allows you to implement automated pull ingestion and batch data processing.
4-
5- ### Create the data job Files
6-
7- Data Job directory can contain any files, however there are some files that are treated in a specific way:
8-
9- * SQL files (.sql) - called SQL steps - are directly executed as queries against your configured database;
10- * Python files (.py) - called Python steps - are Python scripts that define run function that takes as argument the job_input object;
11- * config.ini is needed in order to configure the Job. This is the only file required to deploy a Data Job;
12- * requirements.txt is an optional file needed when your Python steps use external python libraries.
13-
14- Delete all files you do not need and replace them with your own.
15-
16- ### Data Job Code
17-
18- VDK supports having many Python and/or SQL steps in a single Data Job. Steps are executed in ascending alphabetical order based on file names.
19- Prefixing file names with numbers makes it easy to have meaningful file names while maintaining the steps' execution order.
20-
21- Run the Data Job from a Terminal:
22- * Make sure you have vdk installed. See Platform documentation on how to install it.
23- ```
24- vdk run <path to Data Job directory>
25- ```
26-
27- ### Deploy Data Job
28-
29- When a Job is ready to be deployed in a Versatile Data Kit runtime (cloud):
30- Run the command below and follow its instructions (you can see its options with ` vdk --help ` )
31- ``` python
32- vdk deploy
33- ```
1+ TODO: add explanations
You can’t perform that action at this time.
0 commit comments