An end to end data engineering project using Asana API data as an example to create a complete data pipeline in Google Cloud. In addition, both the loader and the dbt project can be configured to work locally in a PostgreSQL instance.
- Loader functionalities written in Node.js.
- Google Cloud Platform orchestration.
- Cloud Scheduler
- Cloud Pub/Sub
- Cloud Functions
- Cloud Storage
- BigQuery
- Data modeling using dbt.
- Analyses queries for visualization (showcased in Metabase).
- Reverse ETL example.
TBD...