diff --git a/docs/docs/concepts/overview.md b/docs/docs/concepts/asset.mdx similarity index 67% rename from docs/docs/concepts/overview.md rename to docs/docs/concepts/asset.mdx index 899e95f7..26da255f 100644 --- a/docs/docs/concepts/overview.md +++ b/docs/docs/concepts/asset.mdx @@ -1,22 +1,13 @@ -# Overview +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; -Compass has three major concept when it comes to data ingestion: Asset, Type, and Service. +# Asset -Asset is essentially an arbitrary JSON object that represent a metadata of a specific service with a specific type. +In Compass, we call every metadata that you input as an Asset. All your tables, dashboards, topics, jobs are an example of assets. -Type defines a ‘type’ of an asset and it is pre-defined. There are currently 4 supported types in Compass: `table`, `job`, `dashboard`, and `topic`. - -Service defines the application name that the asset was coming from. For example: `biquery`, `postgres`, etc. If you wanted to push data for `bigquery` dataset\(s\) to Compass, you would need to first define the ‘`bigquery`’ service in compass. + + -Some features that compass has: -* Asset Tagging -* User -* Discussion -* Starring - -## Asset - -An Asset is a JSON document that describes a metadata. Asset has a schema: | Field | Required | Type | Description | |---|---|---|---| | id | false | string | compass' auto-generated uuid | @@ -29,7 +20,10 @@ An Asset is a JSON document that describes a metadata. Asset has a schema: | labels | false |json | labels of metadata, written in key-value string | | owners | false | []json | array of json, where each json contains `email` field | -```text + + + +```json { "urn": "topic/order-log", @@ -57,9 +51,13 @@ An Asset is a JSON document that describes a metadata. Asset has a schema: } ``` + + + + Every asset that is pushed SHOULD have the required fields: `urn`, `type`, `service`, `name`. The value of these fields MUST be string, if present. -Asset ingestion API \(/v1beta1/assets\) is using HTTP PATCH method. The behavioud would be similar with how PATCH works. It is possible to patch one field only in an asset by sending the updated field to the ingestion API. This also works for the data in dynamic `data` field. The combination of `urn`, `type`, `service` will be the identifier to patch an asset. +Asset ingestion API (`/v1beta1/assets`) is using HTTP PATCH method. The behavioud would be similar with how PATCH works. It is possible to patch one field only in an asset by sending the updated field to the ingestion API. This also works for the data in dynamic `data` field. The combination of `urn`, `type`, `service` will be the identifier to patch an asset. In case the `urn` does not exist, the asset ingestion PATCH API \(/v1beta1/assets\) will create a new asset. ## Lineage @@ -146,13 +144,3 @@ If there is an update to the `environment` in the asset labels, here is the asse ## Tagging an Asset Compass allows user to tag a specific asset. To tag a new asset, one needs to create a template of the tag. Tag's template defines a set of fields' tag that are applicable to tag each field in an asset. Once a template is created, each field in an asset is possible to be tagged by calling `/v1beta1/tags` API. More detail about [Tagging](../guides/tagging.md). - -## User -The current version of Compass does not have user management. Compass expect there is an external instance that manages user. Compass consumes user information from the configurable identity uuid header in every API call. The default name of the header is `Compass-User-UUID`. -Compass does not make any assumption of what kind of identity format that is being used. The `uuid` indicates that it could be in any form (e.g. email, UUIDv4, etc) as long as it is universally unique. -The current behaviour is, Compass will add a new user if the user information consumed from the header does not exist in Compass' database. More detail about [User](./user.md). -## Discussion -Compass supports discussion feature. User could drop comments in each discussion. Currently, there are three types of discussions `issues`, `open ended`, and `question and answer`. Depending on the type, the discussion could have multiple possible states. In the current version, all types only have two states: `open` and `closed`. A newly created discussion will always be assign an `open` state. More detail about [Discussion](../guides/discussion.md). - -## Starring -Compass allows a user to stars an asset. This bookmarking functionality is introduced to increase the speed of a user to get information. There is also an API to see which users star an asset (stargazers). More detail about [Starring](../guides/starring.md). \ No newline at end of file diff --git a/docs/docs/concepts/overview.mdx b/docs/docs/concepts/overview.mdx new file mode 100644 index 00000000..e93fea08 --- /dev/null +++ b/docs/docs/concepts/overview.mdx @@ -0,0 +1,30 @@ +# Overview + +Compass has three major concept when it comes to data ingestion: Asset, Type, and Service. + +Asset is the main model that represents a metadata of a specific service with a specific type. + +Type defines a ‘type’ of an asset and it is pre-defined. Compass currently supports the following types: +1. `table` +2. `job` +3. `dashboard` +4. `topic` +5. `feature_table` +6. `model` (under development) +7. `application` (under development) + +Service defines the application or source that maintains or generates the asset. Examples would be `biquery`, `postgres`, etc. + +Some features that compass has: +* [Discovery](../tour/2-querying-assets.mdx) +* [Lineage](../tour/3-asset-lineage.mdx) +* [Asset Tagging](./asset#tagging-an-asset) +* [User](./user.md) +* [Discussion](../guides/discussion.md) +* [Starring](../guides/starring.md) + +## Discussion +Compass supports discussion feature. User could drop comments in each discussion. Currently, there are three types of discussions `issues`, `open ended`, and `question and answer`. Depending on the type, the discussion could have multiple possible states. In the current version, all types only have two states: `open` and `closed`. A newly created discussion will always be assign an `open` state. More detail about [Discussion](../guides/discussion.md). + +## Starring +Compass allows a user to stars an asset. This bookmarking functionality is introduced to increase the speed of a user to get information. There is also an API to see which users star an asset (stargazers). More detail about [Starring](../guides/starring.md). diff --git a/docs/docs/concepts/type.md b/docs/docs/concepts/type.md new file mode 100644 index 00000000..567b7ff2 --- /dev/null +++ b/docs/docs/concepts/type.md @@ -0,0 +1,15 @@ +# Type + +Each Asset will have a `Type` to represent the kind of metadata it represents. It is currently pre-defined by Compass, so no arbitrary types will be supported. + +Compass currently supports the following types: +1. `table` +2. `job` +3. `dashboard` +4. `topic` +5. `feature_table` +6. `model` (under development) +7. `application` (under development) + +Type will be extremely useful for categorizing your assets and it will be really helpful during discovery. +Check [this section on querying assets](../guides/querying#using-the-get-assets-api) on how to leverage `type` for your discovery. diff --git a/docs/docs/guides/installation.md b/docs/docs/installation.md similarity index 93% rename from docs/docs/guides/installation.md rename to docs/docs/installation.md index 70123b01..8c071c14 100644 --- a/docs/docs/guides/installation.md +++ b/docs/docs/installation.md @@ -76,6 +76,6 @@ time="2022-04-27T09:18:08Z" level=info msg="server started" ``` ## Required Header/Metadata in API -Compass has a concept of [User](../concepts/user.md). In the current version, all HTTP & gRPC APIs in Compass requires an identity header/metadata in the request. The header key is configurable but the default name is `Compass-User-UUID`. +Compass has a concept of [User](./concepts/user.md). In the current version, all HTTP & gRPC APIs in Compass requires an identity header/metadata in the request. The header key is configurable but the default name is `Compass-User-UUID`. -Compass APIs also expect an additional optional e-mail header. This is also configurable and the default name is `Compass-User-Email`. The purpose of having this optional e-mail header is described in the [User](../concepts/user.md) section. \ No newline at end of file +Compass APIs also expect an additional optional e-mail header. This is also configurable and the default name is `Compass-User-Email`. The purpose of having this optional e-mail header is described in the [User](./concepts/user.md) section. \ No newline at end of file diff --git a/docs/docs/tour/1-my-first-asset.md b/docs/docs/tour/1-my-first-asset.md new file mode 100644 index 00000000..f8e1d993 --- /dev/null +++ b/docs/docs/tour/1-my-first-asset.md @@ -0,0 +1,96 @@ +# 1. My First Asset + +Before starting the tour, make sure you have a running Compass instance. You can refer this [installation guide](../installation). + +## 1.1 Introduction + +In Compass, we call every metadata that you input as an [Asset](../concepts/asset). All your tables, dashboards, topics, jobs are an example of assets. + +In this section, we will help you to build your first Asset and hopefully it will give your clear idea about what an Asset is in Compass. + +## 1.2 Hello, ~~World~~ Asset! + +Let's imagine we have a `postgres` instance that we keep referring to as our `main-postgres`. Inside it there is a database called `my-database` that has plenty of tables. One of the tables is named `orders`, and below is how you represent that `table` as an Compass' Asset. + +```json +{ + "urn": "main-postgres:my-database.orders", + "type": "table", + "service": "postgres", + "name": "orders", + "data": { + "database": "my-database", + "namespace": "main-postgres" + } +} +``` + +- **urn** is a unique name you assign to an asset. You need to make sure you don't have a duplicate urns across all of your assets because Compass treats `urn` as an identifier of your asset. For this example, we use the following format to make sure our urn is unique, `{NAMESPACE}:{DB_NAME}.{TABLE_NAME}`. (more info about URN generation can be found [here](../guides/urn-generation)) + +- **type** is your Asset's type. The value for type has to be recognizable by Compass. More info about Asset's Type can be found [here](../concepts/type). + +- **service** can be seen as the source of your asset. `service` can be anything, in this case since our `orders` table resides in `postgres`, we can just put `postgres` as the service. + +- **name** is the name of your asset, it does not have to be unique. We don't need to worry to get mixed up if there are other tables with the same name, `urn` will be the main identifier for your asset, that is why we need to make it unique across all of your assets. + +- **data** can hold your asset's extra details if there is any. In the example, we use it to store information of the **database name** and the **alias/namespace** that we use when referring the postgres instance. + +## 1.3 Sending your first asset to Compass + +Here is the asset that we built on previous section. + +```json +{ + "urn": "main-postgres:my-database.orders", + "type": "table", + "service": "postgres", + "name": "orders", + "data": { + "database": "my-database", + "namespace": "main-postgres" + } +} +``` +Let's send this into Compass so that it would be discoverable. + +As of now, Compass supports ingesting assets via `gRPC` and `http`. In this example, we will use `http` to send your first asset to Compass. +Compass exposes an API `[PATCH] /v1beta1/assets` to upload your asset. + +```bash +curl --location --request PATCH 'http://localhost:8080/v1beta1/assets' \ +--header 'Content-Type: application/json' \ +--header 'Compass-User-UUID: john.doe@example.com' \ +--data-raw '{ + "asset": { + "urn": "main-postgres:my-database.orders", + "type": "table", + "service": "postgres", + "name": "orders", + "data": { + "database": "my-database", + "namespace": "main-postgres" + } + } +}' +``` + +There are a few things to notice here: +1. The HTTP method used is `PATCH`. This is because Compass does not have a dedicated `Create` API, it uses a single API to `Patch / Create` an asset instead. So when updating or patching your asset, you can use the same API. + +2. Compass requires `Compass-User-UUID` header to be in the request. More information about the identity header can be found [here](../concepts/user). To simplify this tour, let's just use `john.doe@example.com`. + +3. When sending our asset to Compass, we need to put our asset object inside an `asset` field as shown in the sample curl above. + +On a success insertion, your will receive below response: + +```json +{ "id": "cebeb793-8933-434c-b38f-beb6dbad91a5" } +``` + +**id** is an identifier of your asset. Unlike `urn` which is provided by you, `id` is auto generated by Compass if there was no asset found with the given URN. + +## Conclusion + +Now that you have successfully ingested your asset to Compass, we can now search and find it via Compass. + +In the next section, we will see how Compass can help you in searching and discovering your assets. diff --git a/docs/docs/tour/2-querying-assets.mdx b/docs/docs/tour/2-querying-assets.mdx new file mode 100644 index 00000000..119e4bf8 --- /dev/null +++ b/docs/docs/tour/2-querying-assets.mdx @@ -0,0 +1,182 @@ +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# 2. Querying your Assets + +In this section, we will learn how we can find and search our assets using the following approaches: +- [Using URN](#21-using-urn) +- [Using Search API](#23-using-search-api) + +## 2.1 Using URN + +Using the URN returned from when [you are uploading your asset](./1-my-first-asset.md#13-sending-your-first-asset-to-compass), you can easily find your asset like below + +```bash +curl 'http://localhost:8080/v1beta1/assets/cebeb793-8933-434c-b38f-beb6dbad91a5' \ +--header 'Content-Type: application/json' \ +--header 'Compass-User-UUID: john.doe@example.com' +``` + +It will return +```json +{ + "data": { + "id": "cebeb793-8933-434c-b38f-beb6dbad91a5", + "urn": "main-postgres:my-database.orders", + "type": "table", + "service": "postgres", + "name": "orders", + "description": "", + "data": { + "database": "my-database", + "namespace": "main-postgres" + }, + "labels": null, + "owners": [], + "version": "0.2", + "updated_by": { + "uuid": "john.doe@example.com" + }, + "changelog": [], + "created_at": "2021-03-22T22:45:11.160593Z", + "updated_at": "2021-03-22T22:45:11.160593Z" + } +} +``` + +## 2.2 Adding more assets + +Before we try other APIs let's first add **5 additional assets** to Compass. + + + + +```bash +curl --location --request PATCH 'http://localhost:8080/v1beta1/assets' \ +--header 'Content-Type: application/json' \ +--header 'Compass-User-UUID: john.doe@example.com' \ +--data-raw '{ + "asset": { + "urn": "main-postgres:my-database.products", + "type": "table", + "service": "postgres", + "name": "products", + "data": { + "database": "my-database", + "namespace": "main-postgres" + } + } +} +' +``` + + + + +```bash +curl --location --request PATCH 'http://localhost:8080/v1beta1/assets' \ +--header 'Content-Type: application/json' \ +--header 'Compass-User-UUID: john.doe@example.com' \ +--data-raw '{ + "asset": { + "urn": "main-postgres:temp-database.invoices", + "type": "table", + "service": "postgres", + "name": "invoices", + "data": { + "database": "temp-database", + "namespace": "main-postgres" + } + } +} +' +``` + + + + +```bash +curl --location --request PATCH 'http://localhost:8080/v1beta1/assets' \ +--header 'Content-Type: application/json' \ +--header 'Compass-User-UUID: john.doe@example.com' \ +--data-raw '{ + "asset": { + "urn": "userdb:identity.users", + "type": "table", + "service": "mysql", + "name": "users", + "data": { + "database": "identity", + "namespace": "userdb" + } + } +} +' +``` + + + + +```bash +curl --location --request PATCH 'http://localhost:8080/v1beta1/assets' \ +--header 'Content-Type: application/json' \ +--header 'Compass-User-UUID: john.doe@example.com' \ +--data-raw '{ + "asset": { + "urn": "mymetabase:collections/123", + "type": "dashboard", + "service": "metabase", + "name": "My Profit Dashboard", + "data": { + "collection_id": 123, + "charts": [ + "Income Chart", + "Outcome Chart" + ] + } + } +} +' +``` + + + + +## 2.3 Using Search API + +Search API is the preferred way when browsing through your assets in Compass. Let's see how powerful Compass is for discovering your assets. + +Now that we have added more assets to Compass [here](#22-adding-more-assets), let's try to search for our newly added `products` table. To use Search API, we just need to provide a query/text/term. + +Let's search for our `products` table using a typo query `"podcts"`. + +```bash +curl 'http://localhost:8080/v1beta1/search?text=podcts' \ +--header 'Compass-User-UUID: john.doe@example.com' +``` + +Search results: +```json +{ + "data": [ + { + "id": "7c0759f4-feec-4b5e-bf26-bf0d0b1236b1", + "urn": "main-postgres:my-database.products", + "type": "table", + "service": "postgres", + "name": "products", + "description": "" + } + ] +} +``` + +Compass Search API supports fuzzy search, so even when you give `"podcts"`, it will still be able to fetch your `products` table. + +## Conclusion + +Search API is a really powerful discovery tool that you can leverage when storing your assets. It has lots of feature like `fuzzy search` which we just saw, you can also easily filter through asset's type, service and much more. + +Up to this point, you have learnt how to create assets, inserting assets and querying them. Using these features, you can start leveraging Compass to be your Metadata Discovery Service. + +Next we will see how you can use Compass to build a Lineage between your assets. diff --git a/docs/docs/tour/3-asset-lineage.mdx b/docs/docs/tour/3-asset-lineage.mdx new file mode 100644 index 00000000..e82435f7 --- /dev/null +++ b/docs/docs/tour/3-asset-lineage.mdx @@ -0,0 +1,73 @@ +# 3. Asset Lineage + +## 3.1 Inserting lineage +To have a lineage, we need at least two assets to be linked with each other. +Compass only needs one of the assets to define the lineage. + +Let's create a new dashboard asset (tableau) that uses data from the `main-postgres:my-database.orders` table asset that we created earlier in [first section](./1-my-first-asset.md#13-sending-your-first-asset-to-compass). +And while creating the asset, we can also define its lineage in the same request. + +```bash +curl --location --request PATCH 'http://localhost:8080/v1beta1/assets' \ +--header 'Content-Type: application/json' \ +--header 'Compass-User-UUID: john.doe@example.com' \ +--data-raw '{ + "asset": { + "urn": "tableau:my-dashboard.daily-orders", + "type": "dashboard", + "service": "tableau", + "name": "Daily Orders Dashboard", + "description": "This is a dashboard showing daily orders" + }, + "upstreams": [ + { + "urn": "main-postgres:my-database.orders" + } + ] +}' +``` + +We use `upstreams` and `downstreams` field to define lineage. +From the example above `main-postgres:my-database.orders` is the source of the dashboard, so we put it inside `upstreams` field. + +And just like that we just created a lineage :) + +## Querying Lineage + +Once lineage is defined, we can easily fetch them using a single asset URN to be the center of the lineage like below + +```bash +curl 'http://localhost:8080/v1beta1/lineage/tableau:my-dashboard.daily-orders' \ +--header 'Compass-User-UUID: john.doe@example.com' +``` + +And you will get this as the response + +```json +{ + "data": [ + { + "source": "main-postgres:my-database.orders", + "target": "tableau:my-dashboard.daily-orders", + "prop": { + "root": "tableau:my-dashboard.daily-orders" + } + } + ] +} +``` + +You can also get the lineage using `main-postgres:my-database.orders`, even when you have only defined `upstreams` on `tableau:my-dashboard.daily-orders`. + +```bash +curl 'http://localhost:8080/v1beta1/lineage/main-postgres:my-database.orders' \ +--header 'Compass-User-UUID: john.doe@example.com' +``` + +## Conclusion + +This is merely a simple example of how Compass can help you manage and build your Data Lineage. +You just need to dump all of your assets along with its `upstreams` and `downstreams` as shown above and Compass will take care of the rest. + +And that's it, we have just covered all the basics of Compass and how you can use it to manage and maintain your metadata. +Aside from features introduced in this tour, Compass also has additional features such as Asset starring and discussion, you can check those in the [Guides section](../guides/starring). diff --git a/docs/sidebars.js b/docs/sidebars.js index 45781a4f..732ebf5e 100644 --- a/docs/sidebars.js +++ b/docs/sidebars.js @@ -1,11 +1,20 @@ module.exports = { docsSidebar: [ 'introduction', + 'installation', + { + type: 'category', + label: 'Tour', + items: [ + "tour/my-first-asset", + "tour/querying-assets", + "tour/asset-lineage" + ] + }, { type: "category", label: "Guides", items: [ - "guides/installation", "guides/ingestion", "guides/querying", "guides/starring", @@ -18,6 +27,9 @@ module.exports = { label: "Concepts", items: [ "concepts/overview", + "concepts/asset", + "concepts/type", + "concepts/user", "concepts/architecture", "concepts/internals", ],