From f12bd2a7538d0f1e2debac032fd7b7dcc94113ef Mon Sep 17 00:00:00 2001 From: Stewart Jingga Date: Tue, 28 Jun 2022 01:46:54 +0700 Subject: [PATCH 1/6] doc: add tour sections --- docs/docs/concepts/{overview.md => asset.mdx} | 42 ++-- docs/docs/concepts/overview.mdx | 24 +++ docs/docs/concepts/type.md | 3 + docs/docs/tour/1-my-first-asset.md | 97 ++++++++++ docs/docs/tour/2-querying-assets.mdx | 182 ++++++++++++++++++ docs/docs/tour/3-asset-lineage.mdx | 3 + docs/sidebars.js | 14 +- 7 files changed, 337 insertions(+), 28 deletions(-) rename docs/docs/concepts/{overview.md => asset.mdx} (67%) create mode 100644 docs/docs/concepts/overview.mdx create mode 100644 docs/docs/concepts/type.md create mode 100644 docs/docs/tour/1-my-first-asset.md create mode 100644 docs/docs/tour/2-querying-assets.mdx create mode 100644 docs/docs/tour/3-asset-lineage.mdx diff --git a/docs/docs/concepts/overview.md b/docs/docs/concepts/asset.mdx similarity index 67% rename from docs/docs/concepts/overview.md rename to docs/docs/concepts/asset.mdx index 899e95f7..26da255f 100644 --- a/docs/docs/concepts/overview.md +++ b/docs/docs/concepts/asset.mdx @@ -1,22 +1,13 @@ -# Overview +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; -Compass has three major concept when it comes to data ingestion: Asset, Type, and Service. +# Asset -Asset is essentially an arbitrary JSON object that represent a metadata of a specific service with a specific type. +In Compass, we call every metadata that you input as an Asset. All your tables, dashboards, topics, jobs are an example of assets. -Type defines a ‘type’ of an asset and it is pre-defined. There are currently 4 supported types in Compass: `table`, `job`, `dashboard`, and `topic`. - -Service defines the application name that the asset was coming from. For example: `biquery`, `postgres`, etc. If you wanted to push data for `bigquery` dataset\(s\) to Compass, you would need to first define the ‘`bigquery`’ service in compass. + + -Some features that compass has: -* Asset Tagging -* User -* Discussion -* Starring - -## Asset - -An Asset is a JSON document that describes a metadata. Asset has a schema: | Field | Required | Type | Description | |---|---|---|---| | id | false | string | compass' auto-generated uuid | @@ -29,7 +20,10 @@ An Asset is a JSON document that describes a metadata. Asset has a schema: | labels | false |json | labels of metadata, written in key-value string | | owners | false | []json | array of json, where each json contains `email` field | -```text + + + +```json { "urn": "topic/order-log", @@ -57,9 +51,13 @@ An Asset is a JSON document that describes a metadata. Asset has a schema: } ``` + + + + Every asset that is pushed SHOULD have the required fields: `urn`, `type`, `service`, `name`. The value of these fields MUST be string, if present. -Asset ingestion API \(/v1beta1/assets\) is using HTTP PATCH method. The behavioud would be similar with how PATCH works. It is possible to patch one field only in an asset by sending the updated field to the ingestion API. This also works for the data in dynamic `data` field. The combination of `urn`, `type`, `service` will be the identifier to patch an asset. +Asset ingestion API (`/v1beta1/assets`) is using HTTP PATCH method. The behavioud would be similar with how PATCH works. It is possible to patch one field only in an asset by sending the updated field to the ingestion API. This also works for the data in dynamic `data` field. The combination of `urn`, `type`, `service` will be the identifier to patch an asset. In case the `urn` does not exist, the asset ingestion PATCH API \(/v1beta1/assets\) will create a new asset. ## Lineage @@ -146,13 +144,3 @@ If there is an update to the `environment` in the asset labels, here is the asse ## Tagging an Asset Compass allows user to tag a specific asset. To tag a new asset, one needs to create a template of the tag. Tag's template defines a set of fields' tag that are applicable to tag each field in an asset. Once a template is created, each field in an asset is possible to be tagged by calling `/v1beta1/tags` API. More detail about [Tagging](../guides/tagging.md). - -## User -The current version of Compass does not have user management. Compass expect there is an external instance that manages user. Compass consumes user information from the configurable identity uuid header in every API call. The default name of the header is `Compass-User-UUID`. -Compass does not make any assumption of what kind of identity format that is being used. The `uuid` indicates that it could be in any form (e.g. email, UUIDv4, etc) as long as it is universally unique. -The current behaviour is, Compass will add a new user if the user information consumed from the header does not exist in Compass' database. More detail about [User](./user.md). -## Discussion -Compass supports discussion feature. User could drop comments in each discussion. Currently, there are three types of discussions `issues`, `open ended`, and `question and answer`. Depending on the type, the discussion could have multiple possible states. In the current version, all types only have two states: `open` and `closed`. A newly created discussion will always be assign an `open` state. More detail about [Discussion](../guides/discussion.md). - -## Starring -Compass allows a user to stars an asset. This bookmarking functionality is introduced to increase the speed of a user to get information. There is also an API to see which users star an asset (stargazers). More detail about [Starring](../guides/starring.md). \ No newline at end of file diff --git a/docs/docs/concepts/overview.mdx b/docs/docs/concepts/overview.mdx new file mode 100644 index 00000000..3052baea --- /dev/null +++ b/docs/docs/concepts/overview.mdx @@ -0,0 +1,24 @@ +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Overview + +Compass has three major concept when it comes to data ingestion: Asset, Type, and Service. + +Asset is essentially an arbitrary JSON object that represent a metadata of a specific service with a specific type. + +Type defines a ‘type’ of an asset and it is pre-defined. There are currently 4 supported types in Compass: `table`, `job`, `dashboard`, and `topic`. + +Service defines the application name that the asset was coming from. For example: `biquery`, `postgres`, etc. If you wanted to push data for `bigquery` dataset\(s\) to Compass, you would need to first define the ‘`bigquery`’ service in compass. + +Some features that compass has: +* [Asset Tagging](./asset#tagging-an-asset) +* [User](./user.md) +* [Discussion](../guides/discussion.md) +* [Starring](../guides/starring.md) + +## Discussion +Compass supports discussion feature. User could drop comments in each discussion. Currently, there are three types of discussions `issues`, `open ended`, and `question and answer`. Depending on the type, the discussion could have multiple possible states. In the current version, all types only have two states: `open` and `closed`. A newly created discussion will always be assign an `open` state. More detail about [Discussion](../guides/discussion.md). + +## Starring +Compass allows a user to stars an asset. This bookmarking functionality is introduced to increase the speed of a user to get information. There is also an API to see which users star an asset (stargazers). More detail about [Starring](../guides/starring.md). \ No newline at end of file diff --git a/docs/docs/concepts/type.md b/docs/docs/concepts/type.md new file mode 100644 index 00000000..19b772ac --- /dev/null +++ b/docs/docs/concepts/type.md @@ -0,0 +1,3 @@ +# Type + +TBD diff --git a/docs/docs/tour/1-my-first-asset.md b/docs/docs/tour/1-my-first-asset.md new file mode 100644 index 00000000..fbc873dc --- /dev/null +++ b/docs/docs/tour/1-my-first-asset.md @@ -0,0 +1,97 @@ +# 1. My First Asset + +Before starting the tour, make sure you have a running Compass instance. You can refer this [guide](../guides/installation). + +## 1.1 Introduction + +In Compass, we call every metadata that you input as an [Asset](../concepts/asset). All your tables, dashboards, topics, jobs are an example of assets. + +In this section, we will help you to build your first Asset and hopefully it will give your clear idea about what an Asset is in Compass. + +## 1.2 Hello, ~~World~~ Asset! + +Let's imagine we have a `postgres` instance that we keep referring to as our `main-postgres`. Inside it there is a database called `my-database` that has plenty of tables. One of the tables is named `orders`, and below is how you represent that `table` as an Compass' Asset. + +```json +{ + "urn": "main-postgres:my-database.orders", + "type": "table", + "service": "postgres", + "name": "orders", + "data": { + "database": "my-database", + "namespace": "main-postgres" + } +} +``` + +- **urn** is a unique name you assign to an asset. You need to make sure you don't have a duplicate urns across all of your assets because Compass treats `urn` as an identifier of your asset. For this example, we use the following format to make sure our urn is unique, `{NAMESPACE}:{DB_NAME}.{TABLE_NAME}`. (more info about URN generation can be found [here](../guides/urn-generation)) + +- **type** is your Asset's type. The value for type has to be recognizable by Compass. More info about Asset's Type can be found [here](../concepts/type). + +- **service** can be seen as the source of your asset. `service` can be anything, in this case since our `orders` table resides in `postgres`, we can just put `postgres` as the service. + +- **name** is the name of your asset, it does not have to be unique. We don't need to worry to get mixed up if there are other tables with the same name, `urn` will be the main identifier for your asset, that is why we need to make it unique across all of your assets. + +- **data** can hold your asset's extra details if there is any. In the example, we use it to store information of the **database name** and the **alias/namespace** that we use when referring the postgres instance. + +## 1.3 Sending your first asset to Compass + +Here is the asset that we built on previous section. + +```json +{ + "urn": "main-postgres:my-database.orders", + "type": "table", + "service": "postgres", + "name": "orders", + "data": { + "database": "my-database", + "namespace": "main-postgres" + } +} +``` +Let's send this into Compass so that it would be discoverable. + +As of now, Compass supports ingesting assets via `gRPC` and `http`. In this example, we will use `http` to send your first asset to Compass. +Compass exposes an API `[PATCH] /v1beta1/assets` to upload your asset. + +```bash +curl --location --request PATCH 'http://localhost:8080/v1beta1/assets' \ +--header 'Content-Type: application/json' \ +--header 'Compass-User-UUID: john.doe@example.com' \ +--data-raw '{ + "asset": { + "urn": "main-postgres:my-database.orders", + "type": "table", + "service": "postgres", + "name": "orders", + "data": { + "database": "my-database", + "namespace": "main-postgres" + } + } +} +' +``` + +There are a few things to notice here: +1. The HTTP method used is `PATCH`. This is because Compass does not have a dedicated `Create` or `Update` apis, it uses a single API to `Patch / Create` an asset instead. So when updating or patching your asset, you can use the same API. + +2. Compass requires `Compass-User-UUID` header to be in the request. More information about the identity header can be found [here](../concepts/user). To simplify this tour, let's just use `john.doe@example.com`. + +3. When sending our asset to Compass, we need to put our asset object inside an `asset` field as shown in the sample curl above. + +On a success insertion, your will receive below response: + +```json +{ "id": "cebeb793-8933-434c-b38f-beb6dbad91a5" } +``` + +**id** is an identifier of your asset. Unlike `urn` which is provided by you, `id` is auto generated by Compass. + +## Summary + +Now that you have successfully ingested your asset to Compass, we can now search and find it via Compass. + +In the next section, we will see how Compass can help you in searching and discovering your assets. diff --git a/docs/docs/tour/2-querying-assets.mdx b/docs/docs/tour/2-querying-assets.mdx new file mode 100644 index 00000000..a1a4fc25 --- /dev/null +++ b/docs/docs/tour/2-querying-assets.mdx @@ -0,0 +1,182 @@ +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# 2. Querying your Assets + +In this section, we will learn how we can find and search our assets using the following approaches: +- [Using ID](#21-using-id) +- [Using Search API](#23-using-search-api) + +## 2.1 Using ID + +Using the ID returned from when [you are uploading your asset](./1-my-first-asset.md#13-sending-your-first-asset-to-compass), you can easily find your asset like below + +```bash +curl --location --request GET 'http://localhost:8080/v1beta1/assets/cebeb793-8933-434c-b38f-beb6dbad91a5' \ +--header 'Content-Type: application/json' \ +--header 'Compass-User-UUID: john.doe@example.com' +``` + +It will return +```json +{ + "data": { + "id": "cebeb793-8933-434c-b38f-beb6dbad91a5", + "urn": "main-postgres:my-database.orders", + "type": "table", + "service": "postgres", + "name": "orders", + "description": "", + "data": { + "database": "my-database", + "namespace": "main-postgres" + }, + "labels": null, + "owners": [], + "version": "0.2", + "updated_by": { + "uuid": "john.doe@example.com" + }, + "changelog": [], + "created_at": "2021-03-22T22:45:11.160593Z", + "updated_at": "2021-03-22T22:45:11.160593Z" + } +} +``` + +## 2.2 Adding more assets + +Before we try other APIs let's first add **5 additional assets** to Compass. + + + + +```bash +curl --location --request PATCH 'http://localhost:8080/v1beta1/assets' \ +--header 'Content-Type: application/json' \ +--header 'Compass-User-UUID: john.doe@example.com' \ +--data-raw '{ + "asset": { + "urn": "main-postgres:my-database.products", + "type": "table", + "service": "postgres", + "name": "products", + "data": { + "database": "my-database", + "namespace": "main-postgres" + } + } +} +' +``` + + + + +```bash +curl --location --request PATCH 'http://localhost:8080/v1beta1/assets' \ +--header 'Content-Type: application/json' \ +--header 'Compass-User-UUID: john.doe@example.com' \ +--data-raw '{ + "asset": { + "urn": "main-postgres:temp-database.invoices", + "type": "table", + "service": "postgres", + "name": "invoices", + "data": { + "database": "temp-database", + "namespace": "main-postgres" + } + } +} +' +``` + + + + +```bash +curl --location --request PATCH 'http://localhost:8080/v1beta1/assets' \ +--header 'Content-Type: application/json' \ +--header 'Compass-User-UUID: john.doe@example.com' \ +--data-raw '{ + "asset": { + "urn": "userdb:identity.users", + "type": "table", + "service": "mysql", + "name": "users", + "data": { + "database": "identity", + "namespace": "userdb" + } + } +} +' +``` + + + + +```bash +curl --location --request PATCH 'http://localhost:8080/v1beta1/assets' \ +--header 'Content-Type: application/json' \ +--header 'Compass-User-UUID: john.doe@example.com' \ +--data-raw '{ + "asset": { + "urn": "mymetabase:collections/123", + "type": "dashboard", + "service": "metabase", + "name": "My Profit Dashboard", + "data": { + "collection_id": 123, + "charts": [ + "Income Chart", + "Outcome Chart" + ] + } + } +} +' +``` + + + + +## 2.3 Using Search API + +Search API is the preferred way when browsing through your assets in Compass. Let's see how powerful Compass is for discovering your assets. + +Now that we have added more assets to Compass [here](#22-adding-more-assets), let's try to search for our newly added `products` table. To use Search API, we just need to provide a query/text/term. + +Let's search for our `products` table using a typo query `"podcts"`. + +```bash +curl --location --request GET 'http://localhost:8080/v1beta1/search?text=podcts' \ +--header 'Compass-User-UUID: john.doe@example.com' +``` + +Search results: +```json +{ + "data": [ + { + "id": "7c0759f4-feec-4b5e-bf26-bf0d0b1236b1", + "urn": "main-postgres:my-database.products", + "type": "table", + "service": "postgres", + "name": "products", + "description": "" + } + ] +} +``` + +Compass Search API supports fuzzy search, so even when you give `"podcts"`, it will still be able to fetch your `products` table. + +## Summary + +Search API is a really powerful discovery tool that you can leverage when storing your assets. It has lots of feature like `fuzzy search` which we just see, you can also easily filter through asset's type, service and much more. + +Up to this point, you have learnt how to create assets, inserting assets and querying them. Using those features only you can start leveraging Compass to be your Metadata Discovery Service. + +Next we will see how you can use Compass to build a Lineage between your assets. diff --git a/docs/docs/tour/3-asset-lineage.mdx b/docs/docs/tour/3-asset-lineage.mdx new file mode 100644 index 00000000..42c6bf6b --- /dev/null +++ b/docs/docs/tour/3-asset-lineage.mdx @@ -0,0 +1,3 @@ +# 3. Asset Lineage + +TBD.. diff --git a/docs/sidebars.js b/docs/sidebars.js index 45781a4f..80fb30ab 100644 --- a/docs/sidebars.js +++ b/docs/sidebars.js @@ -1,11 +1,20 @@ module.exports = { docsSidebar: [ 'introduction', + 'guides/installation', + { + type: 'category', + label: 'Tour', + items: [ + "tour/my-first-asset", + "tour/querying-assets", + "tour/asset-lineage" + ] + }, { type: "category", label: "Guides", items: [ - "guides/installation", "guides/ingestion", "guides/querying", "guides/starring", @@ -18,6 +27,9 @@ module.exports = { label: "Concepts", items: [ "concepts/overview", + "concepts/asset", + "concepts/type", + "concepts/user", "concepts/architecture", "concepts/internals", ], From ebd8580292557ca44796e7dc8a3c55161d6a28ab Mon Sep 17 00:00:00 2001 From: Stewart Jingga Date: Mon, 5 Dec 2022 13:07:03 +0700 Subject: [PATCH 2/6] feat: add Tour section --- docs/docs/{guides => }/installation.md | 0 docs/docs/tour/1-my-first-asset.md | 5 +- docs/docs/tour/2-querying-assets.mdx | 6 +-- docs/docs/tour/3-asset-lineage.mdx | 71 ++++++++++++++++++++++++++ docs/sidebars.js | 2 +- 5 files changed, 77 insertions(+), 7 deletions(-) rename docs/docs/{guides => }/installation.md (100%) diff --git a/docs/docs/guides/installation.md b/docs/docs/installation.md similarity index 100% rename from docs/docs/guides/installation.md rename to docs/docs/installation.md diff --git a/docs/docs/tour/1-my-first-asset.md b/docs/docs/tour/1-my-first-asset.md index fbc873dc..572d18de 100644 --- a/docs/docs/tour/1-my-first-asset.md +++ b/docs/docs/tour/1-my-first-asset.md @@ -1,6 +1,6 @@ # 1. My First Asset -Before starting the tour, make sure you have a running Compass instance. You can refer this [guide](../guides/installation). +Before starting the tour, make sure you have a running Compass instance. You can refer this [installation guide](../installation). ## 1.1 Introduction @@ -71,8 +71,7 @@ curl --location --request PATCH 'http://localhost:8080/v1beta1/assets' \ "namespace": "main-postgres" } } -} -' +}' ``` There are a few things to notice here: diff --git a/docs/docs/tour/2-querying-assets.mdx b/docs/docs/tour/2-querying-assets.mdx index a1a4fc25..1a0e29d5 100644 --- a/docs/docs/tour/2-querying-assets.mdx +++ b/docs/docs/tour/2-querying-assets.mdx @@ -4,12 +4,12 @@ import TabItem from '@theme/TabItem'; # 2. Querying your Assets In this section, we will learn how we can find and search our assets using the following approaches: -- [Using ID](#21-using-id) +- [Using URN](#21-using-urn) - [Using Search API](#23-using-search-api) -## 2.1 Using ID +## 2.1 Using URN -Using the ID returned from when [you are uploading your asset](./1-my-first-asset.md#13-sending-your-first-asset-to-compass), you can easily find your asset like below +Using the URN returned from when [you are uploading your asset](./1-my-first-asset.md#13-sending-your-first-asset-to-compass), you can easily find your asset like below ```bash curl --location --request GET 'http://localhost:8080/v1beta1/assets/cebeb793-8933-434c-b38f-beb6dbad91a5' \ diff --git a/docs/docs/tour/3-asset-lineage.mdx b/docs/docs/tour/3-asset-lineage.mdx index 42c6bf6b..f3ba57e3 100644 --- a/docs/docs/tour/3-asset-lineage.mdx +++ b/docs/docs/tour/3-asset-lineage.mdx @@ -1,3 +1,74 @@ # 3. Asset Lineage TBD.. + +## 3.1 Inserting lineage +To have a lineage, we need at least two assets to be linked with each other. +Compass only needs one of the assets to define the lineage. + +Let's create a new dashboard asset (tableau) that uses data from the `main-postgres:my-database.orders` table asset that we created earlier in [first section](./1-my-first-asset.md#13-sending-your-first-asset-to-compass). +And while creating the asset, we can also define its lineage in the same request. + +```bash +curl --location --request PATCH 'http://localhost:8080/v1beta1/assets' \ +--header 'Content-Type: application/json' \ +--header 'Compass-User-UUID: john.doe@example.com' \ +--data-raw '{ + "asset": { + "urn": "tableau:my-dashboard.daily-orders", + "type": "dashboard", + "service": "tableau", + "name": "Daily Orders Dashboard", + "description": "This is a dashboard showing daily orders" + }, + "upstreams": [ + { + "urn": "main-postgres:my-database.orders" + } + ] +}' +``` + +We use `upstreams` and `downstreams` field to define lineage. +From the example above `main-postgres:my-database.orders` is the source of the dashboard, so we put it inside `upstreams` field. + +And just like that we just created a lineage :) + +## Querying Lineage + +Once lineage is defined, we can easily fetch them using a single asset URN to be the center of the lineage like below + +```bash +curl --location --request GET 'http://localhost:8080/v1beta1/lineage/tableau:my-dashboard.daily-orders' \ +--header 'Compass-User-UUID: john.doe@example.com' +``` + +And you will get this as the response + +```json + "data": [ + { + "source": "main-postgres:my-database.orders", + "target": "tableau:my-dashboard.daily-orders", + "prop": { + "root": "tableau:my-dashboard.daily-orders" + } + } + ] +} +``` + +You can also get the lineage using `main-postgres:my-database.orders`, even when you have only defined `upstreams` on `tableau:my-dashboard.daily-orders`. + +```bash +curl --location --request GET 'http://localhost:8080/v1beta1/lineage/main-postgres:my-database.orders' \ +--header 'Compass-User-UUID: john.doe@example.com' +``` + +## Summary + +This is merely a simple example of how Compass can help you manage and build your Data Lineage. +You just need to dump all of your assets along with its `upstreams` and `downstreams` as shown above and Compass will take care of the rest. + +And that's it, we have just covered all the basics usage of Compass and how you can use it to manage and maintain your metadata. +Aside from features introduced in this tour, Compass also has another features such as Asset starring and discussion, you can check those in the [Guides section](../guides/starring). diff --git a/docs/sidebars.js b/docs/sidebars.js index 80fb30ab..732ebf5e 100644 --- a/docs/sidebars.js +++ b/docs/sidebars.js @@ -1,7 +1,7 @@ module.exports = { docsSidebar: [ 'introduction', - 'guides/installation', + 'installation', { type: 'category', label: 'Tour', From a57fc528450148fbc2ddbf3e0ccaa22653e35bdd Mon Sep 17 00:00:00 2001 From: Stewart Jingga Date: Tue, 6 Dec 2022 15:18:05 +0700 Subject: [PATCH 3/6] doc: update docs --- docs/docs/concepts/overview.mdx | 19 ++++++++++++------- docs/docs/concepts/type.md | 13 ++++++++++++- docs/docs/installation.md | 4 ++-- docs/docs/tour/1-my-first-asset.md | 2 +- docs/docs/tour/2-querying-assets.mdx | 4 ++-- docs/docs/tour/3-asset-lineage.mdx | 4 ++-- 6 files changed, 31 insertions(+), 15 deletions(-) diff --git a/docs/docs/concepts/overview.mdx b/docs/docs/concepts/overview.mdx index 3052baea..cc17884d 100644 --- a/docs/docs/concepts/overview.mdx +++ b/docs/docs/concepts/overview.mdx @@ -1,17 +1,22 @@ -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; - # Overview Compass has three major concept when it comes to data ingestion: Asset, Type, and Service. -Asset is essentially an arbitrary JSON object that represent a metadata of a specific service with a specific type. +Asset is the main model that represents a metadata of a specific service with a specific type. -Type defines a ‘type’ of an asset and it is pre-defined. There are currently 4 supported types in Compass: `table`, `job`, `dashboard`, and `topic`. +Type defines a ‘type’ of an asset and it is pre-defined. Compass currently supports the following types: +1. `table` +2. `job` +3. `dashboard` +4. `topic` +5. `feature_table` +6. `model` (under development) -Service defines the application name that the asset was coming from. For example: `biquery`, `postgres`, etc. If you wanted to push data for `bigquery` dataset\(s\) to Compass, you would need to first define the ‘`bigquery`’ service in compass. +Service defines the application or source that maintains or generates the asset. Examples would be `biquery`, `postgres`, etc. Some features that compass has: +* [Discovery](../tour/2-querying-assets.mdx) +* [Lineage](../tour/3-asset-lineage.mdx) * [Asset Tagging](./asset#tagging-an-asset) * [User](./user.md) * [Discussion](../guides/discussion.md) @@ -21,4 +26,4 @@ Some features that compass has: Compass supports discussion feature. User could drop comments in each discussion. Currently, there are three types of discussions `issues`, `open ended`, and `question and answer`. Depending on the type, the discussion could have multiple possible states. In the current version, all types only have two states: `open` and `closed`. A newly created discussion will always be assign an `open` state. More detail about [Discussion](../guides/discussion.md). ## Starring -Compass allows a user to stars an asset. This bookmarking functionality is introduced to increase the speed of a user to get information. There is also an API to see which users star an asset (stargazers). More detail about [Starring](../guides/starring.md). \ No newline at end of file +Compass allows a user to stars an asset. This bookmarking functionality is introduced to increase the speed of a user to get information. There is also an API to see which users star an asset (stargazers). More detail about [Starring](../guides/starring.md). diff --git a/docs/docs/concepts/type.md b/docs/docs/concepts/type.md index 19b772ac..d06fc2f5 100644 --- a/docs/docs/concepts/type.md +++ b/docs/docs/concepts/type.md @@ -1,3 +1,14 @@ # Type -TBD +Each Asset will have a `Type` to represent the kind of metadata it represents. It is currently pre-defined by Compass, so no arbitrary types will be supported. + +Compass currently supports the following types: +1. `table` +2. `job` +3. `dashboard` +4. `topic` +5. `feature_table` +6. `model` (under development) + +Type will be extremely useful for categorizing your assets and it will be really helpful during discovery. +Check [this section on querying assets](../guides/querying#using-the-get-assets-api) on how to leverage `type` for your discovery. diff --git a/docs/docs/installation.md b/docs/docs/installation.md index 70123b01..8c071c14 100644 --- a/docs/docs/installation.md +++ b/docs/docs/installation.md @@ -76,6 +76,6 @@ time="2022-04-27T09:18:08Z" level=info msg="server started" ``` ## Required Header/Metadata in API -Compass has a concept of [User](../concepts/user.md). In the current version, all HTTP & gRPC APIs in Compass requires an identity header/metadata in the request. The header key is configurable but the default name is `Compass-User-UUID`. +Compass has a concept of [User](./concepts/user.md). In the current version, all HTTP & gRPC APIs in Compass requires an identity header/metadata in the request. The header key is configurable but the default name is `Compass-User-UUID`. -Compass APIs also expect an additional optional e-mail header. This is also configurable and the default name is `Compass-User-Email`. The purpose of having this optional e-mail header is described in the [User](../concepts/user.md) section. \ No newline at end of file +Compass APIs also expect an additional optional e-mail header. This is also configurable and the default name is `Compass-User-Email`. The purpose of having this optional e-mail header is described in the [User](./concepts/user.md) section. \ No newline at end of file diff --git a/docs/docs/tour/1-my-first-asset.md b/docs/docs/tour/1-my-first-asset.md index 572d18de..69333c9b 100644 --- a/docs/docs/tour/1-my-first-asset.md +++ b/docs/docs/tour/1-my-first-asset.md @@ -75,7 +75,7 @@ curl --location --request PATCH 'http://localhost:8080/v1beta1/assets' \ ``` There are a few things to notice here: -1. The HTTP method used is `PATCH`. This is because Compass does not have a dedicated `Create` or `Update` apis, it uses a single API to `Patch / Create` an asset instead. So when updating or patching your asset, you can use the same API. +1. The HTTP method used is `PATCH`. This is because Compass does not have a dedicated `Create` apis, it uses a single API to `Patch / Create` an asset instead. So when updating or patching your asset, you can use the same API. 2. Compass requires `Compass-User-UUID` header to be in the request. More information about the identity header can be found [here](../concepts/user). To simplify this tour, let's just use `john.doe@example.com`. diff --git a/docs/docs/tour/2-querying-assets.mdx b/docs/docs/tour/2-querying-assets.mdx index 1a0e29d5..9cdcd0c6 100644 --- a/docs/docs/tour/2-querying-assets.mdx +++ b/docs/docs/tour/2-querying-assets.mdx @@ -12,7 +12,7 @@ In this section, we will learn how we can find and search our assets using the f Using the URN returned from when [you are uploading your asset](./1-my-first-asset.md#13-sending-your-first-asset-to-compass), you can easily find your asset like below ```bash -curl --location --request GET 'http://localhost:8080/v1beta1/assets/cebeb793-8933-434c-b38f-beb6dbad91a5' \ +curl 'http://localhost:8080/v1beta1/assets/cebeb793-8933-434c-b38f-beb6dbad91a5' \ --header 'Content-Type: application/json' \ --header 'Compass-User-UUID: john.doe@example.com' ``` @@ -151,7 +151,7 @@ Now that we have added more assets to Compass [here](#22-adding-more-assets), le Let's search for our `products` table using a typo query `"podcts"`. ```bash -curl --location --request GET 'http://localhost:8080/v1beta1/search?text=podcts' \ +curl 'http://localhost:8080/v1beta1/search?text=podcts' \ --header 'Compass-User-UUID: john.doe@example.com' ``` diff --git a/docs/docs/tour/3-asset-lineage.mdx b/docs/docs/tour/3-asset-lineage.mdx index f3ba57e3..45fea05c 100644 --- a/docs/docs/tour/3-asset-lineage.mdx +++ b/docs/docs/tour/3-asset-lineage.mdx @@ -39,7 +39,7 @@ And just like that we just created a lineage :) Once lineage is defined, we can easily fetch them using a single asset URN to be the center of the lineage like below ```bash -curl --location --request GET 'http://localhost:8080/v1beta1/lineage/tableau:my-dashboard.daily-orders' \ +curl 'http://localhost:8080/v1beta1/lineage/tableau:my-dashboard.daily-orders' \ --header 'Compass-User-UUID: john.doe@example.com' ``` @@ -61,7 +61,7 @@ And you will get this as the response You can also get the lineage using `main-postgres:my-database.orders`, even when you have only defined `upstreams` on `tableau:my-dashboard.daily-orders`. ```bash -curl --location --request GET 'http://localhost:8080/v1beta1/lineage/main-postgres:my-database.orders' \ +curl 'http://localhost:8080/v1beta1/lineage/main-postgres:my-database.orders' \ --header 'Compass-User-UUID: john.doe@example.com' ``` From 33a0650357c898e477c5ef90e7a99d789e6ca7ef Mon Sep 17 00:00:00 2001 From: Stewart Jingga Date: Tue, 6 Dec 2022 15:21:23 +0700 Subject: [PATCH 4/6] doc: update docs --- docs/docs/tour/1-my-first-asset.md | 2 +- docs/docs/tour/2-querying-assets.mdx | 2 +- docs/docs/tour/3-asset-lineage.mdx | 9 ++++----- 3 files changed, 6 insertions(+), 7 deletions(-) diff --git a/docs/docs/tour/1-my-first-asset.md b/docs/docs/tour/1-my-first-asset.md index 69333c9b..0df1e4b7 100644 --- a/docs/docs/tour/1-my-first-asset.md +++ b/docs/docs/tour/1-my-first-asset.md @@ -89,7 +89,7 @@ On a success insertion, your will receive below response: **id** is an identifier of your asset. Unlike `urn` which is provided by you, `id` is auto generated by Compass. -## Summary +## Conclusion Now that you have successfully ingested your asset to Compass, we can now search and find it via Compass. diff --git a/docs/docs/tour/2-querying-assets.mdx b/docs/docs/tour/2-querying-assets.mdx index 9cdcd0c6..41cf2dfc 100644 --- a/docs/docs/tour/2-querying-assets.mdx +++ b/docs/docs/tour/2-querying-assets.mdx @@ -173,7 +173,7 @@ Search results: Compass Search API supports fuzzy search, so even when you give `"podcts"`, it will still be able to fetch your `products` table. -## Summary +## Conclusion Search API is a really powerful discovery tool that you can leverage when storing your assets. It has lots of feature like `fuzzy search` which we just see, you can also easily filter through asset's type, service and much more. diff --git a/docs/docs/tour/3-asset-lineage.mdx b/docs/docs/tour/3-asset-lineage.mdx index 45fea05c..e82435f7 100644 --- a/docs/docs/tour/3-asset-lineage.mdx +++ b/docs/docs/tour/3-asset-lineage.mdx @@ -1,7 +1,5 @@ # 3. Asset Lineage -TBD.. - ## 3.1 Inserting lineage To have a lineage, we need at least two assets to be linked with each other. Compass only needs one of the assets to define the lineage. @@ -46,6 +44,7 @@ curl 'http://localhost:8080/v1beta1/lineage/tableau:my-dashboard.daily-orders' \ And you will get this as the response ```json +{ "data": [ { "source": "main-postgres:my-database.orders", @@ -65,10 +64,10 @@ curl 'http://localhost:8080/v1beta1/lineage/main-postgres:my-database.orders' \ --header 'Compass-User-UUID: john.doe@example.com' ``` -## Summary +## Conclusion This is merely a simple example of how Compass can help you manage and build your Data Lineage. You just need to dump all of your assets along with its `upstreams` and `downstreams` as shown above and Compass will take care of the rest. -And that's it, we have just covered all the basics usage of Compass and how you can use it to manage and maintain your metadata. -Aside from features introduced in this tour, Compass also has another features such as Asset starring and discussion, you can check those in the [Guides section](../guides/starring). +And that's it, we have just covered all the basics of Compass and how you can use it to manage and maintain your metadata. +Aside from features introduced in this tour, Compass also has additional features such as Asset starring and discussion, you can check those in the [Guides section](../guides/starring). From 5acdb87e40d60fd16c693326a86ac040ac69a61d Mon Sep 17 00:00:00 2001 From: Stewart Jingga Date: Tue, 6 Dec 2022 15:28:49 +0700 Subject: [PATCH 5/6] doc: update docs --- docs/docs/tour/1-my-first-asset.md | 2 +- docs/docs/tour/2-querying-assets.mdx | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/docs/tour/1-my-first-asset.md b/docs/docs/tour/1-my-first-asset.md index 0df1e4b7..88b6b78d 100644 --- a/docs/docs/tour/1-my-first-asset.md +++ b/docs/docs/tour/1-my-first-asset.md @@ -87,7 +87,7 @@ On a success insertion, your will receive below response: { "id": "cebeb793-8933-434c-b38f-beb6dbad91a5" } ``` -**id** is an identifier of your asset. Unlike `urn` which is provided by you, `id` is auto generated by Compass. +**id** is an identifier of your asset. Unlike `urn` which is provided by you, `id` is auto generated by Compass if there was no asset found with the given URN. ## Conclusion diff --git a/docs/docs/tour/2-querying-assets.mdx b/docs/docs/tour/2-querying-assets.mdx index 41cf2dfc..119e4bf8 100644 --- a/docs/docs/tour/2-querying-assets.mdx +++ b/docs/docs/tour/2-querying-assets.mdx @@ -175,8 +175,8 @@ Compass Search API supports fuzzy search, so even when you give `"podcts"`, it w ## Conclusion -Search API is a really powerful discovery tool that you can leverage when storing your assets. It has lots of feature like `fuzzy search` which we just see, you can also easily filter through asset's type, service and much more. +Search API is a really powerful discovery tool that you can leverage when storing your assets. It has lots of feature like `fuzzy search` which we just saw, you can also easily filter through asset's type, service and much more. -Up to this point, you have learnt how to create assets, inserting assets and querying them. Using those features only you can start leveraging Compass to be your Metadata Discovery Service. +Up to this point, you have learnt how to create assets, inserting assets and querying them. Using these features, you can start leveraging Compass to be your Metadata Discovery Service. Next we will see how you can use Compass to build a Lineage between your assets. From c6d29fab1136c029eabd1ea2c3155c08d50ab056 Mon Sep 17 00:00:00 2001 From: Stewart Jingga Date: Tue, 20 Dec 2022 16:58:04 +0700 Subject: [PATCH 6/6] doc: update docs --- docs/docs/concepts/overview.mdx | 1 + docs/docs/concepts/type.md | 1 + docs/docs/tour/1-my-first-asset.md | 2 +- 3 files changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/docs/concepts/overview.mdx b/docs/docs/concepts/overview.mdx index cc17884d..e93fea08 100644 --- a/docs/docs/concepts/overview.mdx +++ b/docs/docs/concepts/overview.mdx @@ -11,6 +11,7 @@ Type defines a ‘type’ of an asset and it is pre-defined. Compass currently s 4. `topic` 5. `feature_table` 6. `model` (under development) +7. `application` (under development) Service defines the application or source that maintains or generates the asset. Examples would be `biquery`, `postgres`, etc. diff --git a/docs/docs/concepts/type.md b/docs/docs/concepts/type.md index d06fc2f5..567b7ff2 100644 --- a/docs/docs/concepts/type.md +++ b/docs/docs/concepts/type.md @@ -9,6 +9,7 @@ Compass currently supports the following types: 4. `topic` 5. `feature_table` 6. `model` (under development) +7. `application` (under development) Type will be extremely useful for categorizing your assets and it will be really helpful during discovery. Check [this section on querying assets](../guides/querying#using-the-get-assets-api) on how to leverage `type` for your discovery. diff --git a/docs/docs/tour/1-my-first-asset.md b/docs/docs/tour/1-my-first-asset.md index 88b6b78d..f8e1d993 100644 --- a/docs/docs/tour/1-my-first-asset.md +++ b/docs/docs/tour/1-my-first-asset.md @@ -75,7 +75,7 @@ curl --location --request PATCH 'http://localhost:8080/v1beta1/assets' \ ``` There are a few things to notice here: -1. The HTTP method used is `PATCH`. This is because Compass does not have a dedicated `Create` apis, it uses a single API to `Patch / Create` an asset instead. So when updating or patching your asset, you can use the same API. +1. The HTTP method used is `PATCH`. This is because Compass does not have a dedicated `Create` API, it uses a single API to `Patch / Create` an asset instead. So when updating or patching your asset, you can use the same API. 2. Compass requires `Compass-User-UUID` header to be in the request. More information about the identity header can be found [here](../concepts/user). To simplify this tour, let's just use `john.doe@example.com`.