From c4106e4dfbb069a11e18eb156927f8aea909c738 Mon Sep 17 00:00:00 2001 From: bkellam Date: Fri, 30 May 2025 12:06:29 -0700 Subject: [PATCH 01/16] wip on refactoring docs --- CHANGELOG.md | 14 +- docs/docs.json | 69 +++------ .../configuration/authentication.mdx | 10 +- .../configuration/declarative-config.mdx | 0 .../configuration/environment-variables.mdx | 10 +- .../configuration/tenancy.mdx | 2 +- .../configuration/transactional-emails.mdx | 0 docs/docs/connections/bitbucket-cloud.mdx | 1 + .../connections/bitbucket-data-center.mdx | 1 + docs/docs/connections/generic-git-host.mdx | 1 + docs/docs/connections/gerrit.mdx | 1 + docs/docs/connections/gitea.mdx | 5 +- docs/docs/connections/github.mdx | 5 +- docs/docs/connections/gitlab.mdx | 5 +- docs/docs/connections/local-repos.mdx | 1 + docs/docs/connections/overview.mdx | 4 +- docs/docs/connections/request-new.mdx | 1 + docs/docs/getting-started-selfhost.mdx | 8 - docs/docs/getting-started.mdx | 55 ------- docs/{self-hosting => docs}/license-key.mdx | 0 docs/docs/overview.mdx | 142 ++++++++++++++++-- docs/docs/quick-start-guide.mdx | 86 +++++++++++ docs/docs/search/code-navigation.mdx | 2 +- docs/docs/search/search-contexts.mdx | 6 +- .../upgrade/v2-to-v3-guide.mdx | 0 .../upgrade/v3-to-v4-guide.mdx | 12 +- docs/images/bitbucket.png | Bin 0 -> 7042 bytes docs/images/gerrit.png | Bin 0 -> 2828 bytes docs/images/git.png | Bin 0 -> 4628 bytes docs/images/gitea.png | Bin 0 -> 8730 bytes docs/images/github.png | Bin 0 -> 5510 bytes docs/images/gitlab.png | Bin 0 -> 10317 bytes docs/images/local.png | Bin 0 -> 1902 bytes docs/self-hosting/overview.mdx | 139 ----------------- docs/snippets/bitbucket-app-password.mdx | 4 +- docs/snippets/bitbucket-token.mdx | 4 +- .../app/[domain]/settings/license/page.tsx | 2 +- packages/web/src/lib/newsData.ts | 2 +- 38 files changed, 288 insertions(+), 304 deletions(-) rename docs/{self-hosting => docs}/configuration/authentication.mdx (88%) rename docs/{self-hosting => docs}/configuration/declarative-config.mdx (100%) rename docs/{self-hosting => docs}/configuration/environment-variables.mdx (91%) rename docs/{self-hosting => docs}/configuration/tenancy.mdx (90%) rename docs/{self-hosting => docs}/configuration/transactional-emails.mdx (100%) delete mode 100644 docs/docs/getting-started-selfhost.mdx delete mode 100644 docs/docs/getting-started.mdx rename docs/{self-hosting => docs}/license-key.mdx (100%) create mode 100644 docs/docs/quick-start-guide.mdx rename docs/{self-hosting => docs}/upgrade/v2-to-v3-guide.mdx (100%) rename docs/{self-hosting => docs}/upgrade/v3-to-v4-guide.mdx (87%) create mode 100644 docs/images/bitbucket.png create mode 100644 docs/images/gerrit.png create mode 100644 docs/images/git.png create mode 100644 docs/images/gitea.png create mode 100644 docs/images/github.png create mode 100644 docs/images/gitlab.png create mode 100644 docs/images/local.png delete mode 100644 docs/self-hosting/overview.mdx diff --git a/CHANGELOG.md b/CHANGELOG.md index 94e942625..0f0a1bac3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,7 +17,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [4.0.0] - 2025-05-28 -Sourcebot V4 introduces authentication, performance improvements and code navigation. Checkout the [migration guide](https://docs.sourcebot.dev/self-hosting/upgrade/v3-to-v4-guide) for information on upgrading your instance to v4. +Sourcebot V4 introduces authentication, performance improvements and code navigation. Checkout the [migration guide](https://docs.sourcebot.dev/docs/upgrade/v3-to-v4-guide) for information on upgrading your instance to v4. ### Changed - [**Breaking Change**] Authentication is now required by default. Notes: @@ -25,7 +25,7 @@ Sourcebot V4 introduces authentication, performance improvements and code naviga - The first user that logs into the instance is given the `owner` role. ([docs](https://docs.sourcebot.dev/docs/more/roles-and-permissions)). - Subsequent users can request to join the instance. The `owner` can approve / deny requests to join the instance via `Settings` > `Members` > `Pending Requests`. - If a user is approved to join the instance, they are given the `member` role. - - Additional login providers, including email links and SSO, can be configured with additional environment variables. ([docs](https://docs.sourcebot.dev/self-hosting/configuration/authentication)). + - Additional login providers, including email links and SSO, can be configured with additional environment variables. ([docs](https://docs.sourcebot.dev/docs/configuration/authentication)). - Clicking on a search result now takes you to the `/browse` view. Files can still be previewed by clicking the "Preview" button or holding `Cmd` / `Ctrl` when clicking on a search result. [#315](https://github.com/sourcebot-dev/sourcebot/pull/315) ### Added @@ -113,17 +113,17 @@ Sourcebot V4 introduces authentication, performance improvements and code naviga ## [3.0.0] - 2025-04-01 -Sourcebot v3 is here and brings a number of structural changes to the tool's foundation, including a SQL database, parallelized indexing, authentication support, multitenancy, and more. Checkout the [migration guide](https://docs.sourcebot.dev/self-hosting/upgrade/v2-to-v3-guide) for information on upgrading your instance to v3. +Sourcebot v3 is here and brings a number of structural changes to the tool's foundation, including a SQL database, parallelized indexing, authentication support, multitenancy, and more. Checkout the [migration guide](https://docs.sourcebot.dev/docs/upgrade/v2-to-v3-guide) for information on upgrading your instance to v3. ### Changed -- [**Breaking Change**] Changed the config schema such that connection objects are specified in the `connection` map, instead of the `repos` array. [See migration guide](https://docs.sourcebot.dev/self-hosting/upgrade/v2-to-v3-guide). +- [**Breaking Change**] Changed the config schema such that connection objects are specified in the `connection` map, instead of the `repos` array. [See migration guide](https://docs.sourcebot.dev/docs/upgrade/v2-to-v3-guide). - Updated the tool's color-palette in dark mode. ### Added -- Added parallelized repo indexing and connection syncing via Redis & BullMQ. See the [architecture overview](https://docs.sourcebot.dev/self-hosting/overview#architecture). +- Added parallelized repo indexing and connection syncing via Redis & BullMQ. See the [architecture overview](https://docs.sourcebot.dev/docs/overview#architecture). - Added repo indexing progress indicators in the navbar. -- Added authentication support via OAuth or email/password. For instructions on enabling, see [this doc](https://docs.sourcebot.dev/self-hosting/configuration/authentication). -- Added the following UI for managing your deployment when **[auth is enabled](https://docs.sourcebot.dev/self-hosting/configuration/authentication)**: +- Added authentication support via OAuth or email/password. For instructions on enabling, see [this doc](https://docs.sourcebot.dev/docs/configuration/authentication). +- Added the following UI for managing your deployment when **[auth is enabled](https://docs.sourcebot.dev/docs/configuration/authentication)**: - connection management: create and manage your JSON configs via a integrated web-editor. - secrets: import personal access tokens (PAT) into Sourcebot (AES-256 encrypted). Reference secrets in your connection config by name. - team & invite management: invite users to your instance to give them access. Configure team [roles & permissions](https://docs.sourcebot.dev/docs/more/roles-and-permissions). diff --git a/docs/docs.json b/docs/docs.json index 607deb9d8..4c099ccfd 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -1,6 +1,6 @@ { "$schema": "https://mintlify.com/docs.json", - "theme": "mint", + "theme": "willow", "name": "Sourcebot", "colors": { "primary": "#851EE7", @@ -15,23 +15,23 @@ "anchors": [ { "anchor": "Docs", - "icon": "book-open", + "icon": "books", "groups": [ { - "group": "General", + "group": "Getting Started", "pages": [ "docs/overview", - "docs/getting-started", - "docs/getting-started-selfhost" + "docs/quick-start-guide", + "docs/license-key" ] }, { - "group": "Connecting your code", + "group": "Configuration", "pages": [ - "docs/connections/overview", { - "group": "Supported platforms", + "group": "Connecting your code", "pages": [ + "docs/connections/overview", "docs/connections/github", "docs/connections/gitlab", "docs/connections/bitbucket-cloud", @@ -42,7 +42,10 @@ "docs/connections/local-repos", "docs/connections/request-new" ] - } + }, + "docs/configuration/environment-variables", + "docs/configuration/authentication", + "docs/configuration/transactional-emails" ] }, { @@ -68,39 +71,12 @@ "docs/more/roles-and-permissions", "docs/more/mcp-server" ] - } - ] - }, - { - "anchor": "Self Hosting", - "icon": "server", - "groups": [ - { - "group": "Getting Started", - "pages": [ - "self-hosting/overview", - "self-hosting/license-key" - ] - }, - { - "group": "Configuration", - "pages": [ - "self-hosting/configuration/environment-variables", - "self-hosting/configuration/authentication", - "self-hosting/configuration/transactional-emails", - "self-hosting/configuration/declarative-config" - ] - }, - { - "group": "Security", - "pages": [ - ] }, { "group": "Upgrade", "pages": [ - "self-hosting/upgrade/v3-to-v4-guide", - "self-hosting/upgrade/v2-to-v3-guide" + "docs/upgrade/v3-to-v4-guide", + "docs/upgrade/v2-to-v3-guide" ] } ] @@ -122,21 +98,18 @@ "dark": "/logo/dark.png" }, "navbar": { - "links": [ - { - "label": "GitHub", - "href": "https://github.com/sourcebot-dev/sourcebot" - } - ], "primary": { "type": "button", - "label": "Sourcebot Cloud", - "href": "https://app.sourcebot.dev" + "label": "GitHub", + "href": "https://github.com/sourcebot-dev/sourcebot" } }, "footer": { "socials": { - "github": "https://github.com/sourcebot-dev/sourcebot" + "github": "https://github.com/sourcebot-dev/sourcebot", + "twitter": "https://x.com/sourcebot_dev", + "discord": "https://discord.gg/Y6b78RqM", + "linkedin": "https://www.linkedin.com/company/sourcebot" } }, "integrations": { @@ -146,6 +119,6 @@ }, "appearance": { "default": "dark", - "strict": true + "strict": false } } \ No newline at end of file diff --git a/docs/self-hosting/configuration/authentication.mdx b/docs/docs/configuration/authentication.mdx similarity index 88% rename from docs/self-hosting/configuration/authentication.mdx rename to docs/docs/configuration/authentication.mdx index 0ec2a7203..3604f5f47 100644 --- a/docs/self-hosting/configuration/authentication.mdx +++ b/docs/docs/configuration/authentication.mdx @@ -7,7 +7,7 @@ sidebarTitle: Authentication Sourcebot has built-in authentication that gates access to your organization. OAuth, email codes, and email / password are supported. -The first account that's registered on a Sourcebot deployment is made the owner. All other users who register must be [approved](/self-hosting/configuration/authentication#approving-new-members) by the owner. +The first account that's registered on a Sourcebot deployment is made the owner. All other users who register must be [approved](/docs/configuration/authentication#approving-new-members) by the owner. ![Login Page](/images/login.png) @@ -16,10 +16,10 @@ The first account that's registered on a Sourcebot deployment is made the owner. All account registrations after the first account must be approved by the owner. The owner can see all join requests by going into **Settings -> Members**. -If you have an [enterprise license](/self-hosting/license-key), you can enable [AUTH_EE_ENABLE_JIT_PROVISIONING](/self-hosting/configuration/authentication#enterprise-authentication-providers) to +If you have an [enterprise license](/docs/license-key), you can enable [AUTH_EE_ENABLE_JIT_PROVISIONING](/docs/configuration/authentication#enterprise-authentication-providers) to have Sourcebot accounts automatically created and approved on registration. -You can setup emails to be sent when new join requests are created/approved by configurating [transactional emails](/self-hosting/configuration/transactional-emails) +You can setup emails to be sent when new join requests are created/approved by configurating [transactional emails](/docs/configuration/transactional-emails) # Authentication Providers To enable an authentication provider in Sourcebot, configure the required environment variables for the provider. Under the hood, Sourcebot uses Auth.js which supports [many providers](https://authjs.dev/getting-started/authentication/oauth). Submit a [feature request on GitHub](https://github.com/sourcebot-dev/sourcebot/discussions/categories/ideas) if you want us to add support for a specific provider. @@ -39,11 +39,11 @@ Email codes are 6 digit codes sent to a provided email. Email codes are enabled - `EMAIL_FROM_ADDRESS` -See [transactional emails](/self-hosting/configuration/transactional-emails) for more details. +See [transactional emails](/docs/configuration/transactional-emails) for more details. ## Enterprise Authentication Providers -The following authentication providers require an [enterprise license](/self-hosting/license-key) to be enabled. +The following authentication providers require an [enterprise license](/docs/license-key) to be enabled. By default, a new user registering using these providers must have their join request accepted by the owner of the organization to join. To allow a user to join automatically when they register for the first time, set the `AUTH_EE_ENABLE_JIT_PROVISIONING` environment variable to `true`. diff --git a/docs/self-hosting/configuration/declarative-config.mdx b/docs/docs/configuration/declarative-config.mdx similarity index 100% rename from docs/self-hosting/configuration/declarative-config.mdx rename to docs/docs/configuration/declarative-config.mdx diff --git a/docs/self-hosting/configuration/environment-variables.mdx b/docs/docs/configuration/environment-variables.mdx similarity index 91% rename from docs/self-hosting/configuration/environment-variables.mdx rename to docs/docs/configuration/environment-variables.mdx index 8f65fb5ed..ade59913f 100644 --- a/docs/self-hosting/configuration/environment-variables.mdx +++ b/docs/docs/configuration/environment-variables.mdx @@ -10,20 +10,20 @@ The following environment variables allow you to configure your Sourcebot deploy | Variable | Default | Description | | :------- | :------ | :---------- | -| `AUTH_CREDENTIALS_LOGIN_ENABLED` | `true` |

Enables/disables authentication with basic credentials. Username and passwords are stored encrypted at rest within the postgres database. Checkout the [auth docs](/self-hosting/configuration/authentication) for more info

| -| `AUTH_EMAIL_CODE_LOGIN_ENABLED` | `false` |

Enables/disables authentication with a login code that's sent to a users email. `SMTP_CONNECTION_URL` and `EMAIL_FROM_ADDRESS` must also be set. Checkout the [auth docs](/self-hosting/configuration/authentication) for more info

| +| `AUTH_CREDENTIALS_LOGIN_ENABLED` | `true` |

Enables/disables authentication with basic credentials. Username and passwords are stored encrypted at rest within the postgres database. Checkout the [auth docs](/docs/configuration/authentication) for more info

| +| `AUTH_EMAIL_CODE_LOGIN_ENABLED` | `false` |

Enables/disables authentication with a login code that's sent to a users email. `SMTP_CONNECTION_URL` and `EMAIL_FROM_ADDRESS` must also be set. Checkout the [auth docs](/docs/configuration/authentication) for more info

| | `AUTH_SECRET` | Automatically generated at startup if no value is provided. Generated using `openssl rand -base64 33` |

Used to validate login session cookies

| | `AUTH_URL` | - |

URL of your Sourcebot deployment, e.g., `https://example.com` or `http://localhost:3000`.

| -| `CONFIG_PATH` | `-` |

The container relative path to the declerative configuration file. See [this doc](/self-hosting/configuration/declarative-config) for more info.

| +| `CONFIG_PATH` | `-` |

The container relative path to the declerative configuration file. See [this doc](/docs/configuration/declarative-config) for more info.

| | `DATA_CACHE_DIR` | `$DATA_DIR/.sourcebot` |

The root data directory in which all data written to disk by Sourcebot will be located.

| | `DATA_DIR` | `/data` |

The directory within the container to store all persistent data. Typically, this directory will be volume mapped such that data is persisted across container restarts (e.g., `docker run -v $(pwd):/data`)

| | `DATABASE_DATA_DIR` | `$DATA_CACHE_DIR/db` |

The data directory for the default Postgres database.

| | `DATABASE_URL` | `postgresql://postgres@ localhost:5432/sourcebot` |

Connection string of your Postgres database. By default, a Postgres database is automatically provisioned at startup within the container.

If you'd like to use a non-default schema, you can provide it as a parameter in the database url

| -| `EMAIL_FROM_ADDRESS` | `-` |

The email address that transactional emails will be sent from. See [this doc](/self-hosting/configuration/transactional-emails) for more info.

| +| `EMAIL_FROM_ADDRESS` | `-` |

The email address that transactional emails will be sent from. See [this doc](/docs/configuration/transactional-emails) for more info.

| | `REDIS_DATA_DIR` | `$DATA_CACHE_DIR/redis` |

The data directory for the default Redis instance.

| | `REDIS_URL` | `redis://localhost:6379` |

Connection string of your Redis instance. By default, a Redis database is automatically provisioned at startup within the container.

| | `SHARD_MAX_MATCH_COUNT` | `10000` |

The maximum shard count per query

| -| `SMTP_CONNECTION_URL` | `-` |

The url to the SMTP service used for sending transactional emails. See [this doc](/self-hosting/configuration/transactional-emails) for more info.

| +| `SMTP_CONNECTION_URL` | `-` |

The url to the SMTP service used for sending transactional emails. See [this doc](/docs/configuration/transactional-emails) for more info.

| | `SOURCEBOT_ENCRYPTION_KEY` | Automatically generated at startup if no value is provided. Generated using `openssl rand -base64 24` |

Used to encrypt connection secrets and generate API keys.

| | `SOURCEBOT_LOG_LEVEL` | `info` |

The Sourcebot logging level. Valid values are `debug`, `info`, `warn`, `error`, in order of severity.

| | `SOURCEBOT_TELEMETRY_DISABLED` | `false` |

Enables/disables telemetry collection in Sourcebot. See [this doc](/self-hosting/security/telemetry) for more info.

| diff --git a/docs/self-hosting/configuration/tenancy.mdx b/docs/docs/configuration/tenancy.mdx similarity index 90% rename from docs/self-hosting/configuration/tenancy.mdx rename to docs/docs/configuration/tenancy.mdx index 7dd2cd90c..ddd98d98b 100644 --- a/docs/self-hosting/configuration/tenancy.mdx +++ b/docs/docs/configuration/tenancy.mdx @@ -4,7 +4,7 @@ sidebarTitle: Multi tenancy --- If you're switching from single-tenant mode, delete the Sourcebot cache (the `.sourcebot` folder) before starting. -[Authentication](/self-hosting/configuration/authentication) must be enabled to enable multi tenancy mode +[Authentication](/docs/configuration/authentication) must be enabled to enable multi tenancy mode Multi tenancy allows your Sourcebot deployment to have **multiple organizations**, each with their own set of members and repos. To enable multi tenancy mode, define an environment variable named `SOURCEBOT_TENANCY_MODE` and set its value to `multi`. When multi tenancy mode is enabled: diff --git a/docs/self-hosting/configuration/transactional-emails.mdx b/docs/docs/configuration/transactional-emails.mdx similarity index 100% rename from docs/self-hosting/configuration/transactional-emails.mdx rename to docs/docs/configuration/transactional-emails.mdx diff --git a/docs/docs/connections/bitbucket-cloud.mdx b/docs/docs/connections/bitbucket-cloud.mdx index 66636b022..62cc4442b 100644 --- a/docs/docs/connections/bitbucket-cloud.mdx +++ b/docs/docs/connections/bitbucket-cloud.mdx @@ -1,6 +1,7 @@ --- title: Linking code from Bitbucket Cloud sidebarTitle: Bitbucket Cloud +icon: Bitbucket --- import BitbucketToken from '/snippets/bitbucket-token.mdx'; diff --git a/docs/docs/connections/bitbucket-data-center.mdx b/docs/docs/connections/bitbucket-data-center.mdx index b80afbfdd..f6843600e 100644 --- a/docs/docs/connections/bitbucket-data-center.mdx +++ b/docs/docs/connections/bitbucket-data-center.mdx @@ -1,6 +1,7 @@ --- title: Linking code from Bitbucket Data Center sidebarTitle: Bitbucket Data Center +icon: Bitbucket --- import BitbucketToken from '/snippets/bitbucket-token.mdx'; diff --git a/docs/docs/connections/generic-git-host.mdx b/docs/docs/connections/generic-git-host.mdx index cee01aaab..df6202b2e 100644 --- a/docs/docs/connections/generic-git-host.mdx +++ b/docs/docs/connections/generic-git-host.mdx @@ -1,5 +1,6 @@ --- title: Other Git hosts +icon: git-alt --- import GenericGitHost from '/snippets/schemas/v3/genericGitHost.schema.mdx' diff --git a/docs/docs/connections/gerrit.mdx b/docs/docs/connections/gerrit.mdx index 29bb627b7..b72532598 100644 --- a/docs/docs/connections/gerrit.mdx +++ b/docs/docs/connections/gerrit.mdx @@ -1,6 +1,7 @@ --- title: Linking code from Gerrit sidebarTitle: Gerrit +icon: crow --- import GerritSchema from '/snippets/schemas/v3/gerrit.schema.mdx' diff --git a/docs/docs/connections/gitea.mdx b/docs/docs/connections/gitea.mdx index 810085a2c..0a85eaeac 100644 --- a/docs/docs/connections/gitea.mdx +++ b/docs/docs/connections/gitea.mdx @@ -1,6 +1,7 @@ --- title: Linking code from Gitea sidebarTitle: Gitea +icon: mug-tea --- import GiteaSchema from '/snippets/schemas/v3/gitea.schema.mdx' @@ -82,7 +83,7 @@ Next, provide the access token via the `token` property, either as an environmen - Environment variables are only supported in a [declarative config](/self-hosting/configuration/declarative-config) and cannot be used in the web UI. + Environment variables are only supported in a [declarative config](/docs/configuration/declarative-config) and cannot be used in the web UI. 1. Add the `token` property to your connection config: ```json @@ -107,7 +108,7 @@ Next, provide the access token via the `token` property, either as an environmen - Secrets are only supported when [authentication](/self-hosting/configuration/authentication) is enabled. + Secrets are only supported when [authentication](/docs/configuration/authentication) is enabled. 1. Navigate to **Secrets** in settings and create a new secret with your PAT: diff --git a/docs/docs/connections/github.mdx b/docs/docs/connections/github.mdx index 52bb2f3ec..525622331 100644 --- a/docs/docs/connections/github.mdx +++ b/docs/docs/connections/github.mdx @@ -1,6 +1,7 @@ --- title: Linking code from GitHub sidebarTitle: GitHub +icon: GitHub --- import GitHubSchema from '/snippets/schemas/v3/github.schema.mdx' @@ -111,7 +112,7 @@ Next, provide the PAT via the `token` property, either as an environment variabl - Environment variables are only supported in a [declarative config](/self-hosting/configuration/declarative-config) and cannot be used in the web UI. + Environment variables are only supported in a [declarative config](/docs/configuration/declarative-config) and cannot be used in the web UI. 1. Add the `token` property to your connection config: ```json @@ -136,7 +137,7 @@ Next, provide the PAT via the `token` property, either as an environment variabl - Secrets are only supported when [authentication](/self-hosting/configuration/authentication) is enabled. + Secrets are only supported when [authentication](/docs/configuration/authentication) is enabled. 1. Navigate to **Secrets** in settings and create a new secret with your PAT: diff --git a/docs/docs/connections/gitlab.mdx b/docs/docs/connections/gitlab.mdx index 4740bcc04..3ca971a58 100644 --- a/docs/docs/connections/gitlab.mdx +++ b/docs/docs/connections/gitlab.mdx @@ -1,6 +1,7 @@ --- title: Linking code from GitLab sidebarTitle: GitLab +icon: GitLab --- import GitLabSchema from '/snippets/schemas/v3/gitlab.schema.mdx' @@ -116,7 +117,7 @@ Next, provide the PAT via the `token` property, either as an environment variabl - Environment variables are only supported in a [declarative config](/self-hosting/configuration/declarative-config) and cannot be used in the web UI. + Environment variables are only supported in a [declarative config](/docs/configuration/declarative-config) and cannot be used in the web UI. 1. Add the `token` property to your connection config: ```json @@ -141,7 +142,7 @@ Next, provide the PAT via the `token` property, either as an environment variabl - Secrets are only supported when [authentication](/self-hosting/configuration/authentication) is enabled. + Secrets are only supported when [authentication](/docs/configuration/authentication) is enabled. 1. Navigate to **Secrets** in settings and create a new secret with your PAT: diff --git a/docs/docs/connections/local-repos.mdx b/docs/docs/connections/local-repos.mdx index ebed93b5e..23a205146 100644 --- a/docs/docs/connections/local-repos.mdx +++ b/docs/docs/connections/local-repos.mdx @@ -1,5 +1,6 @@ --- title: Local Git repositories +icon: folder --- import GenericGitHost from '/snippets/schemas/v3/genericGitHost.schema.mdx' diff --git a/docs/docs/connections/overview.mdx b/docs/docs/connections/overview.mdx index 65bac506e..fc5b41854 100644 --- a/docs/docs/connections/overview.mdx +++ b/docs/docs/connections/overview.mdx @@ -11,11 +11,11 @@ There are two ways to define connections: - This is only supported when self-hosting, and is the default mechanism to define connections. Connections are defined in a [JSON file](/self-hosting/configuration/declarative-config) + This is only supported when self-hosting, and is the default mechanism to define connections. Connections are defined in a [JSON file](/docs/configuration/declarative-config) and the path to the file is provided through the `CONFIG_PATH` environment variable - This is the only way to define connections when using Sourcebot Cloud, and can be configured when self-hosting by enabling [authentication](/self-hosting/configuration/authentications). + This is the only way to define connections when using Sourcebot Cloud, and can be configured when self-hosting by enabling [authentication](/docs/configuration/authentications). In this method, connections are defined and managed within the webapp: diff --git a/docs/docs/connections/request-new.mdx b/docs/docs/connections/request-new.mdx index dc42d9fcf..5308736ef 100644 --- a/docs/docs/connections/request-new.mdx +++ b/docs/docs/connections/request-new.mdx @@ -2,6 +2,7 @@ sidebarTitle: Request another host url: https://github.com/sourcebot-dev/sourcebot/discussions/categories/ideas title: Request another code host +icon: plus --- Is your code host not supported? Please open a [feature request](https://github.com/sourcebot-dev/sourcebot/discussions/categories/ideas). \ No newline at end of file diff --git a/docs/docs/getting-started-selfhost.mdx b/docs/docs/getting-started-selfhost.mdx deleted file mode 100644 index 17113ea69..000000000 --- a/docs/docs/getting-started-selfhost.mdx +++ /dev/null @@ -1,8 +0,0 @@ ---- -sidebarTitle: Quick start guide (self-host) -url: /self-hosting/overview ---- - -{/*This page acts as a navigation link*/} - -[Quick start guide (self-host)](/self-hosting/overview) \ No newline at end of file diff --git a/docs/docs/getting-started.mdx b/docs/docs/getting-started.mdx deleted file mode 100644 index 2d78b61c0..000000000 --- a/docs/docs/getting-started.mdx +++ /dev/null @@ -1,55 +0,0 @@ ---- -title: Cloud quick start guide -sidebarTitle: Quick start guide (cloud) ---- - -Looking for a self-hosted solution? Checkout our [self-hosting docs](/self-hosting/overview). - -This page will provide a quick walkthrough of how to get onboarded on Sourcebot, import your code, and start searching. - -{/*@todo: record a quick start guide - -*/} - - - - Head over to [app.sourcebot.dev](https://app.sourcebot.dev) and create an account. - - - - After logging in, you'll be asked to create an organization. You'll invite your team members to this organization later so they can also use Sourcebot. - - ![Org Creation](/images/org_create.png) - - - - After selecting a code host you want to connect to, you'll be presented with the connection creation page. This page has the following three inputs: - - Connection name (required): The name of the connection within Sourcebot - - Secret (optional): An [access token](/access-tokens/overview) that is used to fetch private repos - - Configuration: The JSON configuration schema that defines the repos/orgs to fetch. - - For a more detailed explanation of connections, check out the [Connections](/docs/connections/overview) page. - - The example below shows a connection named `sourcebot-org` that fetches all of the repos for the `sourcebot-dev` GitHub organization, but excludes the `sourcebot-dev/zoekt` repo - - This page won't let you continue with an invalid connection schema. If you're hitting errors, make sure the input you're providing is a valid JSON - ![Connection Create Example](/images/create_connection_example.png) - - - -### Search - -Once you create your organization's first connection successfully, you'll be redirected to your org's main search page. From here, you can use the search bar to search across all -of the repos you've indexed - -![Onboard Complete](/images/onboard_complete.png) - -Congrats, you've successfuly setup Sourcebot! Read on to learn more about the Sourcebot's capabilities. Checkout the [Connections](/docs/connections/overview) page to learn how to control which repos Sourcebot fetches \ No newline at end of file diff --git a/docs/self-hosting/license-key.mdx b/docs/docs/license-key.mdx similarity index 100% rename from docs/self-hosting/license-key.mdx rename to docs/docs/license-key.mdx diff --git a/docs/docs/overview.mdx b/docs/docs/overview.mdx index 04aa3bd9c..55a23bc13 100644 --- a/docs/docs/overview.mdx +++ b/docs/docs/overview.mdx @@ -2,19 +2,137 @@ title: "Overview" --- -Sourcebot is an **[open-source](https://github.com/sourcebot-dev/sourcebot) code search tool** that is purpose built to search multi-million line codebases in seconds. It integrates with [GitHub](/docs/connections/github), [GitLab](/docs/connections/gitlab), and [other platforms](/docs/connections). +import SupportedPlatforms from '/snippets/platform-support.mdx' -## Getting Started +Sourcebot is an [open-source](https://github.com/sourcebot-dev/sourcebot) code search tool that is purpose built to search multi-million line codebases in seconds. It integrates with [GitHub](/docs/connections/github), [GitLab](/docs/connections/gitlab), and [other platforms](/docs/connections). -There are two ways to get started using Sourcebot: +## Features +--- + +### Fast-indexed based search + +Search across millions of lines of code in seconds using Sourcebot's blazingly fast indexed search. Find exactly what you are looking for with regular expressions, search filters, boolean logic, and more. + + +- **Regex support:** Use regular expressions to find code with precision. +- **Query language:** Scope searches to specific files, repos, languages, symbol definitions and more using a rich [query language](/docs/search/syntax-reference). +- **Fast & scalable:** Sourcebot uses [trigram indexing](https://en.wikipedia.org/wiki/Trigram_search), allowing it to scale to massive codebases. +- **Syntax highlighting:** Syntax highlighting support for over [100+ languages](https://github.com/sourcebot-dev/sourcebot/blob/57724689303f351c279d37f45b6406f1d5d5d5ab/packages/web/src/lib/codemirrorLanguage.ts#L125). +- **Multi-repository:** Search across all of your repositories in a single search. +- **Search suggestions:** Get search suggestions as you craft your query. +- **Filter panel:** Filter results by repository or by language. + + + + +### Code Navigation + +[Code navigation](/docs/search/code-navigation) helps you jump between symbol definitions and references quickly when browsing source code in Sourcebot. + + +- **Hover popover:** Hovering over a symbol reveals the symbol's definition signature in a inline preview. +- **Go to definition:** Navigate to a symbol's definition(s). +- **Find references:** Get all references to a symbol. +- **Cross-repository:** Sourcebot can resolve references and definitions across repositories. + + + + + +### Multi code-host support + +Connect your code from multiple code-host platforms and search them all from a single interface. + + +- **Auto re-indexing:** Sourcebot will take care of keeping it's index upto date with the latest changes. + + + + + + + + + + + + + + + + + + + + + +### Authentication + +TODO + +### Light / Dark mode support + +TODO + +### Self-hosted + +TODO + +## Architecture + +Sourcebot is shipped as a single docker container that runs a collection of services using [supervisord](https://supervisord.org/): + +![architecture diagram](/images/architecture_diagram.png) + +{/*TODO: outline the different services, how Sourcebot communicates with code hosts, and the different*/} + +Sourcebot consists of the following components: +- **Web Server** : main Next.js web application serving the Sourcebot UI. +- **Backend Worker** : Node.js process that incrementally syncs with code hosts (e.g., GitHub, GitLab etc.) and asynchronously indexes configured repositories. +- **Zoekt** : the [open-source](https://github.com/sourcegraph/zoekt), trigram indexing code search engine that powers Sourcebot under the hood. +- **Postgres** : transactional database for storing business-logic data. +- **Redis Job Queue** : fast in-memory store. Used with [BullMQ](https://docs.bullmq.io/) for queuing asynchronous work. +- **`.sourcebot/` cache** : file-system cache where persistent data is written. + +You can use managed Redis / Postgres services that run outside of the Sourcebot container by providing the `REDIS_URL` and `DATABASE_URL` environment variables, respectively. See the [configuration](/self-hosting/configuration) for more configuration options. + +## Scalability + +One of our design philosophies for Sourcebot is to keep our infrastructure [radically simple](https://www.radicalsimpli.city/) while balancing scalability concerns. Depending on the number of repositories you have indexed and the instance you are running Sourcebot on, you may experience slow search times or other performance degradations. Our recommendation is to vertically scale your instance by increasing the number of CPU cores and memory. + +Sourcebot does not support horizontal scaling at this time, but it is on our roadmap. If this is something your team would be interested in, please contact us at [team@sourcebot.dev](mailto:team@sourcebot.dev). + + +## Telemetry +By default, Sourcebot collects anonymized usage data through [PostHog](https://posthog.com/) to help us improve the performance and reliability of our tool. We don't collect or transmit any information related to your codebase. In addition, all events are [sanitized](https://github.com/sourcebot-dev/sourcebot/blob/HEAD/packages/web/src/app/posthogProvider.tsx) to ensure that no sensitive details (ex. ip address, query info) leave your machine. + +The data we collect includes general usage statistics and metadata such as query performance (e.g., search duration, error rates) to monitor the application's health and functionality. This information helps us better understand how Sourcebot is used and where improvements can be made. + +If you'd like to disable all telemetry, you can do so by setting the environment variable `SOURCEBOT_TELEMETRY_DISABLED` to `true`: + +```bash +docker run \ + -e SOURCEBOT_TELEMETRY_DISABLED=true \ + /* additional args */ \ + ghcr.io/sourcebot-dev/sourcebot:latest +``` - - - Deploy Sourcebot on your own infrastructure. - - - Use Sourcebot on our managed infrastructure. - - +If you disabled telemetry correctly, you'll see the following log when starting Sourcebot: -We also have a [public demo](https://demo.sourcebot.dev) if you'd like to try Sourcebot out before registering. +```sh +Disabling telemetry since SOURCEBOT_TELEMETRY_DISABLED was set. +``` \ No newline at end of file diff --git a/docs/docs/quick-start-guide.mdx b/docs/docs/quick-start-guide.mdx new file mode 100644 index 000000000..621fa9ec7 --- /dev/null +++ b/docs/docs/quick-start-guide.mdx @@ -0,0 +1,86 @@ +--- +title: "Deployment guide" +--- + +## Quick Start Guide + +{/* + +*/} + + + + By default, Sourcebot requires a configuration file with a list of [code host connections](/docs/connections/overview) that specify what repositories should be **synced** (cloned and indexed). To get started, run the following command to create a starter `config.json`: + + ```bash + touch config.json + echo '{ + "$schema": "https://raw.githubusercontent.com/sourcebot-dev/sourcebot/main/schemas/v3/index.json", + "connections": { + // Comments are supported + "starter-connection": { + "type": "github", + "repos": [ + "sourcebot-dev/sourcebot" + ] + } + } + }' > config.json + ``` + + This config creates a single GitHub connection named `starter-connection` that specifies [Sourcebot](https://github.com/sourcebot-dev/sourcebot) as a repo to sync. + + + + If you're deploying Sourcebot behind a domain, you must set the [AUTH_URL](/docs/configuration/environment-variables) environment variable + Sourcebot is packaged as a [single Docker image](https://github.com/sourcebot-dev/sourcebot/pkgs/container/sourcebot). In the same directory as `config.json`, run the following command to start your instance: + + ``` bash + docker run \ + -p 3000:3000 \ + --pull=always \ + --rm \ + -v $(pwd):/data \ + -e CONFIG_PATH=/data/config.json \ + --name sourcebot \ + ghcr.io/sourcebot-dev/sourcebot:latest + ``` + + Navigate to `localhost:3000` to start searching the Sourcebot repo. + + + **This command**: + - pulls the latest version of the `sourcebot` docker image. + - mounts the working directory to `/data` in the container to allow Sourcebot to persist data across restarts, and to access the `config.json`. In your local directory, you should see a `.sourcebot` folder created that contains all persistent data. + - runs any pending database migrations. + - starts up all services, including the webserver exposed on port 3000. + - reads `config.json` and starts syncing. + + + Hit an issue? Please let us know on [GitHub discussions](https://github.com/sourcebot-dev/sourcebot/discussions/categories/support) or by [emailing us](mailto:team@sourcebot.dev). + + + + Sourcebot has built-in authentication which gates your instance. The first account which is registered on a fresh Sourcebot deployment is made owner. + + Registration is performed using basic credentials which are stored encrypted within your deployment. To setup more authentication providers + check out the [auth docs](/docs/configuration/authentication) + + + + + + Sourcebot supports indexing public & private code on the following code hosts: + + + + Missing your code host? [Submit a feature request on GitHub](https://github.com/sourcebot-dev/sourcebot/discussions/categories/ideas). + + \ No newline at end of file diff --git a/docs/docs/search/code-navigation.mdx b/docs/docs/search/code-navigation.mdx index daced8bab..a18c851ff 100644 --- a/docs/docs/search/code-navigation.mdx +++ b/docs/docs/search/code-navigation.mdx @@ -6,7 +6,7 @@ sidebarTitle: Code navigation import SearchContextSchema from '/snippets/schemas/v3/searchContext.schema.mdx' -This feature is only available in [Sourcebot cloud](app.sourcebot.dev) or with an active Enterprise license when [self-hosting](/self-hosting). Please add your [license key](/self-hosting/license-key) to activate it. +This feature is only available in [Sourcebot cloud](app.sourcebot.dev) or with an active Enterprise license when [self-hosting](/self-hosting). Please add your [license key](/docs/license-key) to activate it. **Code navigation** allows you to jump between symbol definition and references when viewing source files in Sourcebot. This feature is enabled **automatically** when a valid license key is present and works with all popular programming languages. diff --git a/docs/docs/search/search-contexts.mdx b/docs/docs/search/search-contexts.mdx index e6afca59e..6d222aa70 100644 --- a/docs/docs/search/search-contexts.mdx +++ b/docs/docs/search/search-contexts.mdx @@ -6,7 +6,7 @@ sidebarTitle: Search contexts import SearchContextSchema from '/snippets/schemas/v3/searchContext.schema.mdx' -This feature is only available when [self-hosting](/self-hosting) with an active Enterprise license. Please add your [license key](/self-hosting/license-key) to activate it. +This feature is only available when [self-hosting](/self-hosting) with an active Enterprise license. Please add your [license key](/docs/license-key) to activate it. A **search context** is a user-defined grouping of repositories that helps focus searches on specific areas of your codebase, like frontend, backend, or infrastructure code. Some example queries using search contexts: @@ -16,7 +16,7 @@ A **search context** is a user-defined grouping of repositories that helps focus - `( context:project1 or context:project2 ) logger\.debug` - search for debug log calls in project1 and project2 -Search contexts are defined in the `context` object inside of a [declarative config](/self-hosting/configuration/declarative-config). Repositories can be included / excluded from a search context by specifying the repo's URL in either the `include` array or `exclude` array. Glob patterns are supported. +Search contexts are defined in the `context` object inside of a [declarative config](/docs/configuration/declarative-config). Repositories can be included / excluded from a search context by specifying the repo's URL in either the `include` array or `exclude` array. Glob patterns are supported. ## Example @@ -41,7 +41,7 @@ shared/ ├─ ... ``` -To make searching easier, we can create three search contexts in our [config.json](/self-hosting/configuration/declarative-config): +To make searching easier, we can create three search contexts in our [config.json](/docs/configuration/declarative-config): - `web`: For all frontend-related code - `backend`: For backend services and shared APIs - `pipelines`: For all CI/CD configurations diff --git a/docs/self-hosting/upgrade/v2-to-v3-guide.mdx b/docs/docs/upgrade/v2-to-v3-guide.mdx similarity index 100% rename from docs/self-hosting/upgrade/v2-to-v3-guide.mdx rename to docs/docs/upgrade/v2-to-v3-guide.mdx diff --git a/docs/self-hosting/upgrade/v3-to-v4-guide.mdx b/docs/docs/upgrade/v3-to-v4-guide.mdx similarity index 87% rename from docs/self-hosting/upgrade/v3-to-v4-guide.mdx rename to docs/docs/upgrade/v3-to-v4-guide.mdx index 2c6d83e91..c51da3a9d 100644 --- a/docs/self-hosting/upgrade/v3-to-v4-guide.mdx +++ b/docs/docs/upgrade/v3-to-v4-guide.mdx @@ -8,7 +8,7 @@ This guide will walk you through upgrading your Sourcebot deployment from v3 to Please note that the following features are no longer supported in v4: - Multi-tenancy mode -- Unauthenticated access to a Sourcebot deployment - authentication is now built in by default. Unauthenticated access to a organization can be enabled with an unlimited seat [enterprise license](/self-hosting/license-key) +- Unauthenticated access to a Sourcebot deployment - authentication is now built in by default. Unauthenticated access to a organization can be enabled with an unlimited seat [enterprise license](/docs/license-key) ### If your deployment doesn't have authentication enabled @@ -22,12 +22,12 @@ Please note that the following features are no longer supported in v4: When you visit your new deployment you'll be presented with a sign-in page. Sourcebot now requires authentication, and all users must register and sign-in to the deployment. The first account that's registered will be made the owner. By default, you can register using basic credentials which will be stored encrypted within the postgres DB connected to Sourcebot. Check out - the [auth docs](/self-hosting/configuration/authentication) to setup additional auth providers. + the [auth docs](/docs/configuration/authentication) to setup additional auth providers. - Emails can be sent on organization join request/approval by configuring [transactional emails](/self-hosting/configuration/transactional-emails) + Emails can be sent on organization join request/approval by configuring [transactional emails](/docs/configuration/transactional-emails) @@ -36,7 +36,7 @@ Please note that the following features are no longer supported in v4: ![Pending Approval Page](/images/pending_approval.png) - The owner can view and approve join requests by navigating to **Settings -> Members**. Automatic provisioning of accounts is supported when using SSO/Oauth providers, check out the [auth docs](/self-hosting/configuration/authentication#enterprise-authentication-providers) for more info + The owner can view and approve join requests by navigating to **Settings -> Members**. Automatic provisioning of accounts is supported when using SSO/Oauth providers, check out the [auth docs](/docs/configuration/authentication#enterprise-authentication-providers) for more info @@ -47,11 +47,11 @@ Please note that the following features are no longer supported in v4: ### If your deployment has authentication enabled The only change that's required if your deployment has authentication enabled is to unset the `SOURCEBOT_AUTH_ENABLED` environment variable. New user registrations will now submit a request to join the organization which can be approved by the owner by -navigating to **Settings -> Members**. Emails can be sent on organization join request/approval by configuring [transactional emails](/self-hosting/configuration/transactional-emails) +navigating to **Settings -> Members**. Emails can be sent on organization join request/approval by configuring [transactional emails](/docs/configuration/transactional-emails) ### If your deployment uses multi-tenancy mode -Unfortunately, multi-tenancy mode is no longer officially supported in v4. To upgrade to v4, you'll need to unset the `SOURCEBOT_TENANCY_MODE` environment variable and wipe your Sourcebot cache. You can then follow the [instructions above](/self-hosting/upgrade/v3-to-v4-guide#if-your-deployment-doesnt-have-authentication-enabled) +Unfortunately, multi-tenancy mode is no longer officially supported in v4. To upgrade to v4, you'll need to unset the `SOURCEBOT_TENANCY_MODE` environment variable and wipe your Sourcebot cache. You can then follow the [instructions above](/docs/upgrade/v3-to-v4-guide#if-your-deployment-doesnt-have-authentication-enabled) to finish upgrading to v4 in single-tenant mode. ## Troubleshooting diff --git a/docs/images/bitbucket.png b/docs/images/bitbucket.png new file mode 100644 index 0000000000000000000000000000000000000000..259e5d1eddab043a2405303bfde4bcb371528fef GIT binary patch literal 7042 zcmdT}_g7QVlMf(*QVgIVMY?Y@(oOj>cx%bY@XFfA8F%KW;FrDW*4+4Rh^mMh1 zK_IH6(~JHrFcRJ@*8}`e`55bHQ2p)_hXFTdT+|KJL7>V+hGTnL;GW+>*Vq6A3c3b} zM}k14(`hRpkl#%Z2>%2Gx|0q9alFVz8z}(~&OW`bqXnXzem`MKQh z(~BygNcAN!Naw3(piQ?*%gZVW>Emo*1cA8P^|aJY1I9O|4B~{B1Nr_gZy+`~z4i2p zuUfKSee+H<`6jI%yKCFovdFrSF1=254asxf-tVnKq!%uq<+>{#Q$=ONHs!#TRoGBG7v^ZL%o!qV_vh&ah|x4UI!)Q061)p&^_yj_=mQ6yfMf=hW}0?hHKQ zzz(X4+v27IHuwMXBO(bolQlc7+DKZpz0sT`bYi5%$iq_iFySI<0dbP&vO9Bfa9PC~ zB1|25HDu)itSNZH#-EaQaT_Ig6~%ES=KXQ2ZS8?n9_Oag@{=-Du4YC1+g9IGnfedE zn%Q}M*GA+*i*Fpwp#1|@J?S*VE5d_=VI-*WJ<4SArxZ@sRT}sh87(BBX0h-?Z*>N8 z-d9Qc_0Rm^u=obVxTN<@3%^k<~YZ-I!gm5lKVUGRa|>fGr3&y77|d3{?wH z*x&f5yDZz8La(^t$A|)cwe*=GW~!W2AEe~T94DPd_|2_aHm`A`PxH1xC&`V|FE5{= z_o363qxn4X-tsHRevsa4dw^j1)lDtDA_4$)QollI2$0;EDH|8IJg?hTeh}I!n zO{bcZ9bM$^_@FMjJUx7|s7nui^dr;4chR|v&4M}-$A!yn%fICVrV>=MD4Gh`{GGj< zEA)M&U#+K(mcG<4=up2tc<)n*)j`yIfQ@&|7NKB= z-dat6bSr9!(mQ8TL9>@pH%&t7O#2{`7{B&cV1Awn*=uTJzkC$ue%!JP-lEmf2!G+u zk#P|H`>|&Nph9$UT)Vx_+ecVUVT5D7#i~STd0n@{PhQ7)lsa_w z5sXUrU`+C#4s169^s1VK)~y#`r|e~*n${=~RzSLW@(le%Du16VRZEw(`?Oel85dOx zHHlDE!b97G090)3{hfkp^`Y*tx-(3Sod-YI3F`%pXA-_p>I`@^rE%^Iy_JvR-crWw z21j+PcP4SpqNHWFLH-UZ~B(={he0#$Kfwpq$U;3Cu=JuSmvzR`LhdMup#&o z$nxlwb8@y<0o&BSW=NDs50A_4ppIlqJ44T%$D5WLjIZS2iRv}IbA~&c50aPgp)NEW zUiu}hgy#oN5>v+^WlKU8Jm120vamqf<-)t72e67U&;Bb1d=bPpE0_ z)`H;v6Hs%rOUcYptzo)x@mmId4)!U`Aw>c_319Bt!l&Y-x+xp?*!d^Vx7sK0Si$Jh ze0-mumEp1>lOtZ;luj>s`#oe8y`9lsi&j8;!nq@&r%WQ6y#*M+o;e#ZZm}OF^9(cV zJq?e|3ZmNYu17?K>A!6au(>^~9gC)cCKbK?t&?d~|EX%G-+%f9&n9V4)l%N5Iz!S4 zG$~Zvg#FoV9?3|nL8#WLs_=2`4LK?lcFpvxMAdwWeT#OruZ}! z#$^1QKB~@upfFubdgW+&L+f3Y()@Yg&>G9RLF+m!L}R+JVL<$iZeDrm^B{$63lj`n zvVE<+%Jl>+`ts*i<7-;_`d_t%+n(ec>w8xMS=~&@(~fxsaLegB__Qepliwg%FsjA= zhXG;g`yVfk3efDMwYIlPtK4B$s%Ute~TLqBH_~)kariY4Xv+O`s*7I((5MmYR)HDt>^j8o(GS7YYaPH_}xjo2iT;XIZ4cn z{32(KCLaI0cc(_*B=k~u`t1>eK3y)l+%Tf1ITB>gbo__=LcE|h$L5L(DOn0b%=IA8 z-fbYcM4_jTBEp7JEW2;xI$DYwfWu2V;|$IvfA8CzM_N<0WOg<%xIHbp{dGRU&|{-t z9-*Qfwj1{B%4DUAj)_899h|rHP%F4K(ei`SI=1xSig`K6-h);6&LG_?ZlbX!*ld(9 zXGA?_dg+dkd{bbQ=FP`*Z z;<_~a;GaG3&Iohi^FwVQE0jsX4K<4rFIj|VSxjwF{9dq*9qE}_{JU+hkoj6#|3YU0v+yDcc1re*p(r~-qmfRxbKv8A&k*OEZWofZ zv+z0+{=Wc>&A+?EUzWVv8jRFRC~vO5pZwEg>*O2F#J&#>w^A)1wG`tVcn1wvS&S)7 z|DxDe90Zsx#em>lLXj{p1KE>(=RL~u>;0e(W`5+Z^uxdY&sX#eT@ifQ4B}I!c8k1% z^Qvkn^EOW}Jk%ON9a+n5*|;tI=)uH+@FV0}aIvJ}FK^BioGeac>B9Xj0q^`N zhbEDJt@yA6!a8%FrI!5IKMOHcGB*j0ang5|#dWI_KZpSbY0rf%AnSeJncwl+Vb7Y; zMR^mfe3ZR#FQG|JnOki3x8zFFe*swGrhGoD7_rr$0YD_ofj75pJX=(k@GYcwhjuM> zT+@+Q9E2}Ys8u`^GcN(zZ_te(0bmdfpW+}h7*#4giOj{Qv*bEIy5+MB>jJEmlQP%H z-}#%@TwMCl3V=$L>wXJ*Wgq5q?f9^S`&lIN%z+MS%E$}{kEySiOHY+{HRIUG!aLaC z3Wg@-H}CFWz-|w8;zZRfj;wYQ%3?kmBYwdRJ@EC-qp*V#Z?laHZBZ((ro*5~fJdeg z#4!Efh`po5`;z)pE!P5;z0KLp;+nn(7mfz*HNXw6dvU-K?=$&HWa2AVcm<`aiMH&E zL#M)L6^=eC+QnwWB-KWg)Qky+?Gf_>YLf%vhOxJC%Qo$fpV7k)>nXL~CC7G1Kz?cE zr?{g7jqoV(M00BMGW4~f94>3^=}QykM>D3-BgaRD7n6!_3I5nR4QF(4*roAWVBH}YVxh&C#f3wPHVqY150G#otk0)D!r;w6r8iElFTM*_x-B-B?yTKcR7JdT;Rb4m7BsS;`OP~zls zAWqPLa7o;u3SCS(@weRnc}r3kuutG-KNJR8KE5mBY!$f6wTk<4(o4)*)EVw&%z6gk ztF)Mu-JQNiTAXP(RV!C~k6JBEDOs2k92K-$Kk+d|gz;Qr7MtXD%rLCc(K1R^^}7nl z|5{mP>U5Mbi~bJP?$PY?T3c>|)=k3f(@OI=f?9&)5q zCt3zaw`tjQ7YE%ly6ChgL}Ub-7$0b_;-2uML8~JW(IR9ux3eKm6l&k=iIB&YKjTa> zlb;dFzs!itCI8GpHP8d$mc=&(Wcv=%Rx&m%s5zrRWq3InNb*pGRiP#J$<(QLj~4rF zURkNE`P0`OGk-=r7AtW!4r??<~)$)tmf6J%i<`Q1Rs3{8QsJu zD8IHd>33AWK1#RxL2xG^dXneE{J@hi2YOY7m5%Q>ITd+F*O-||@*2y3J~jVByERL6 z?@-Xxm^VMJajRG_^7P*UGjYCe&-&&NY{TQOz=w#HJFN&t*vt*O)AfmrodHukfcL|W z+=6VxM?z+1=-N&jyx!(pi9-o3ECTPjS_Fz(?ZznslVbtD9|CkI0fc;RF{yGBc!qy` z8z+>%@=oY8BRgh9gp&DOYJ*KXAI#$VJtUWu`tc=GpZen zQP0F1&+`gmssfY&mRVrscmIm9&U#ftpPbGqXy7-PFDW}*Ytb&u^)KZDU|f*7R(e!O zszrTtA_|Fo1lcpJxBK)jWgqaw6v!<6&C(bIC{KcCTplQ@^FK>I9xhs13cfx3V>V+z z&~NZ-oZg2;w!Z6BE$j-)Ay1@=6xMWmSOj(Ca3=F{jxz3o*RbOVfB-|ki&Qfw9E(x3 zjtDb*v2zoB*Qi?d|H?g@07GsX*9FbtwKDN0snRiT)hW^F3=Qbju55blNxv3tsy;^2np|4#~Pwj1eBo52AMw0rEGzz_bbVAmiSx6R_9 z9{1NMEFAk&S!*1qIPxjZjC{y!E$YZ}y`-Ovj<3KOO~l;vME%>}C**L;Zdjnas40zI z%4nYrw=9&DjnqZiHT_utv?g~$n@IW#>&O?Ij2YGNumx|C!G|2T*0{3p9oVQ&Y6Ajy z#648V&kP~8F+r(Na%-BUOOpyF0dj@LwEMOBP*!`SPTl6VO_B4Ce3F*L*CG6X{r6 za?-!nKAUcq#_onK0gBz4Cnlj)$eFEWbQlpL5(lzB5^PJcxc|1Zt;`Xq1|H3XI$P1hE9lyVQD;Y54a!^-%^ww;Dx?S34Q8M)2O3(( zW66M+mZpOKwU%MSPlm(2WXDHk3v3&%gU-YcXj@k{QDD)ZBW;gyw;f*<2W4X9b#F70 zDz!{P-*o5HxFSkE(?1P~>08&~2PwD0PUr6>1={;z#h28o#~C3{p-G~CgTz~@TF>zYYdsq_j{Khq?Iqgb1h{) z@&F7h{kAv=2!Ed7XlsfKYDIZoA5d7y(M-yf_gToh!zz~dw#E%1RTI^@UQD`|Ds8E5 zgBm@;vZ;qxkelND^aWUNX%@mgiH0?(unWlmpJW(17hRMd+o05@0I$$C=Cfrh6Qnhr zCH!SDaFv;9{tir(?sHEdIH%%Y6|Q&iBb8&_cqGwiq}Qy0*!Wl zcTU^q-#V!4l6kz?BPJ9U1T5jup!|66>fX+4K@?>nF0C8yy=bK z@laCH0oWCQ)ej&U92N%su8wR3p7vcZ#AFIJFX>Lc24_@F>gmUgN@Q(S2AG5*myTGv z(=QGlede$ZAGicjs)lI@$m~zF&c3K#4>$lr4ppsJ*fs`l3PEnlk2g%6=E}`DH{M0% z@Bafp2_DcSH>SN6%q>zdss{8~(F7d7XLV6`^++049g?A}mH*k3(u>R&-_IrP zh#L4(X_ycD$F#TMiHC)mRiXb1Et zn&AVYz{fi(rd%Vbi2tL(%(=pkwhC|y(`KZl7DoB==-p1ry zl9dvZ z?XEJi>V54k9a}9b4Z57rxyoJ0a%@6dd0&#y{@Dy;4iCKZrT~z52J<~B{Xd7R>7qfO zQ+UZwbBvYN)XI$h7XTLi#1-0wO_YkLt$cC!Zy}sZ1{%Mv1%=sDH*AilncT>aJ61+z zgFaUmq~0ZQ8E68Tl>_JxUYdj^JL$i(l}hv450nC)H=2NNCCM`UhjzMQ`JqURI+B7g z-=97uivHMZYAg5vGc#((4H?G~Z;_!a{O$%*)oTEJZ~9^MVDx%lTAmUI#Li22Pd2&-7m`TK!fP9~y$R@bi9SQ0 zqk7Jpd?BGFD5`S|dLgQ{w`zV$bKwu=X3_Knic$42fkP*JfXLa2gX;8sq^a!VlyNa3 zcP>5vxY2p!$|QH8x$ymTezjr0}WaD(SoZ*whOjfh8RJ5u%O`F zb3z}iGr7DBed?u%Fdz+jtT}Ho{W9ekl#1=JJ*=MUz;OMOJpMae8t6-`W+B29xZISP zET3FR<}@H=1V`u(FXoFk6AM&T34u~)qWXBhE@5YdJC?d-y~k=01>|NVbu d!e#dPBGgCu606^k449*LBNN+Tnzz2kgH!*!AlI|>d_m_{Z!-&{tzUU$L{zthdFFte_>Y?z9yxiPKF)BslQOcserSFF{>Q7> z;_S9!SAqL<<1DhNtHnt%!??ufU9R)*N^Rq1;$;l6yTXsVyG>Wg%_(FkD)liEVK>`( z1?-|-it=5GqWQjL?#(be@FlRglH1a;r7i>Z(bBhRqRD4Fe_0&_foRDt zbLl8I&uYh_a?CQuz5Q zM=dL=F`=Q1-rMWYUCWoru#|Cj_DhftNyE<^Zd)`{81}XjEBF3Qu=>0pEK~5^Y? zXMTz1Sb?8PZ=fVAUOrE}|<&F`sY`P*MPC{K>qpVYwubtxj zp**I*-tWc^(0S7GtCV&6-4}5!%q6Zz2@Kl)C^4C&v~CjmOdJts9nTd((>^=j3H%M> zVkP+)Sl9+NKv!7kKUg{~4a*c?>J`zA?r(y@;Q$m$A^~!88|(i?U?JM83t@eVv9WCH zkRURaQfDsDbP)5pdXDn7KzPlTVV;b$(3$DXd2*e*vQlfNX`(BFoHU|*n>#bO*Pk$h z%att99ZNy@DE%qi6YjVigDJa3qJ;^a z9mX`dJc-A1Mn_Md0}Js~SAO63Jm`1~m}SVqDzvC6Y$$(5#$5mQ~(kd#(s2t;2WKp-|23HlRD!AQ*{_65aQ~!=gu{lX3>#|ky_%ucK$(6xn6@7OOB$Q&S^t(ouMZZB0c zUu^VHT{`X^RVb~aM^`d%kS+yUKaDInEkw|JP+m9Yy&o;>uL4v`uhT7UMcGWe9*;TDSg4~FRHQRja z!dA_~Momcx2<%o#xAs0l`T@16RXeL7O_O`poI zdh2C)O0?B(2XReIRgbl;uMwtJxq88t>;V<(+uac5Jr>;DurL9NkOCFc$X zx#MhbtXAShre^T?%}LpN8O+{u2^=ev4o>rKeC;hpScZ>FQArHroaOb{?C3s@uT^DgCWp!b2ZV#zHd1u)($IXPNSzhrQAu=~R6!qGQxAK*It z|CgykAn14vh(&cB!8eudi{@aCxHBrEf@#kTq|0u%cXPg)6y2gIzT!(qyVl z(A{hMc;%@g?3RWaD8q6+Gm#(GMx%%^CNa_r`H#vugWVv1%BQNj)VYAw%0F49>R0k) ztYW40P&H)A$>% z&~c!xtw=u0n;7YB5wnsUk59Wu-z`B!4>QER&IJ^_IRhTanknLM`r8E3A`$2G7UP|> TY0Vm2HdvY&SsE4?IEDQaR$f3r literal 0 HcmV?d00001 diff --git a/docs/images/git.png b/docs/images/git.png new file mode 100644 index 0000000000000000000000000000000000000000..f4e98b22859f7e307ddfec04cbeba1e0ddd7c07a GIT binary patch literal 4628 zcmd5=hgXx?whs^`>L5fA9Hp5k4AN8(1_&fbB27R+=|w40lr9iJnxX^{kO8p}K!y?# z1nE*UYUlzI6f`tRfY1Ua^pfO#@y^`4)_d;{xF_qZto=Ltl)caX?Q_nTc*)#QX#dgu zAP`8%*a&S20`WcKRmffdi6v@$1pfGfEe+4}ed|(+0R*t8zL`D4o>6}>CxR;& za&x8G@#=H@iU;hCy?u19i)sAjJNrU4k&$YB(Y4tdQuk+^T=#D^PJ553E}#rV@8|DJ z+2{T8+nbekthd)!(h+zlnXEZf{-zGD93J@j{CT&DmBqB0*uBbuccOxpTJqAatjX+1 z)0s?m?u6Dx?lC@KY5(Qrig^_{;Ipy~#+Q*M=xkeBWD0Amwo0lFN;ObugJb%_*j}$H zeYfgHi2g+=lq?M9M|A&kclto%M3&E1npNRRi;WO17y(bz%b~Vo3PkJew7_8PudV5E z$&Iim4Omw14~Ke0^z4s?CoywN#-+GeRKhdKqnr1Pn3b898_YW~sk<&IB{k6lL(FAc zGoUXiZZa-x$gz>27_lbq+xesW3f001d=Hu4AuV| z?@mj=$;!$~!}$5{V)4KKTZeiaCM)}ipD6+bgGmNx6iP%G%EzaVu>b9!arj;L0azUE zay7W-7jjd3e%lQVIsaI4N8ABC|`2f5-I5Pp6b3)EA zpyrpV;KNv}pBJq9mQ56NZO|ejUX$2Nsr0;fcc1wOnXD$uLKVP}Ygq?>*>G_%ttBD? zEuL2s<<5%kBy0~2Suyh9MT3gRpu^~=0(!#n#TL@?XQiP~g`x?fyU%ypF%xZ+{af#i z^^!j|j-<8;YhVHT>+LwFO~u&@SVHfXqdOmZn!kc|!H+xJwOP1xZY7&>@Aj+e0xHgpKLW ziPwwJfW$z&caYftBT7oB6x zD}lio9jXyQ5AVEPO(f`8)Q)OD1N{D6FZ|&uk#JZ(EpMPYe@e4=sGxNGkIE>YLqMR! z7hjaVypx>8F)UQ)zJ`~IKW^wMpTn9js5}9J&7}&uVOm*j^j*D+jRKtSP2i zGL!o5-S(^T4sNSBlTv4EAY*cWU%Yv8#{j&z z#@)H)=7K?m<2Q%(6V%({aw3!qSCJ3u@Wv;WRFm4b?`MZjSk0Uuhqb%qrDi&N9(o5p zZ)zc(b2Lpp4R8JV=32}Iax}){SqARF!c)}g@R~vOKWNCK7{_|AmCb`0IAgFVL@w=_ zq_K-_1wBP=D_5xc_sz43va&T(Qv=2lAL-!KYaum4eZ@N0zmrzX#C6o*b9CZmQ36)!p)I& z(D3;>d94oNpuUkLNtu4Fn9VLOM}vOH0bRaJf0JyK&6vn~u|}I*KT-VzzPuonV|%B} zg}9N_dU$MXZ6|e$8QfYPc8hAzv9rQBB3Xvye(tc2cI_eM0GU8_`}vNF(?IZzeg##t z5>C6=)_kUAn8)yl(2+f0rpTjb2=6`>m|7*ho8#H9=o6D;`S40;FQ#x>>;6gU%#fNa zvY=5=>#sV*P%xa5w+!@5wCY4& zjnK7hUY%6zM~>tkmsO`hYMsk!_xn0MVix61tX=cwGiKh$7rb2LNT-XdM>ugRF6PK zp0hff2OKVcD)a22CKUOf?SMZXr^EZ}-5`)&t)r2oQ#C8`zgk&O($}|it@Tk}-IKPB zJ+PV;nd|nJ9B6c#1+-}Qg{}e0>+S~(#hR%Zb4I6MfeS$@m$L4PHE<3$0ds>r{V7`k zXwHMjUW0sUkbc>nrR4vptiJ45$$&gWA#1ib4PVU@`1-h@h@ zl2E%>KN7m8acRyim5s|*`Y8Lp_XA9a;J%SEw2rxKdZYQN@k^9-%jU+$b1B@YYuZtx z&Vb9l<-a7V7>@d%q%eigxCBoWDbDPg_Xs}NcE(#?VL6R!(M#rF&P_KK-adJK0MZ+} z{23 z*K1tbV@6zIjjyp)Q^aIS+!3&euR0n_ZqN6`2g#>z`eqnqV@{B@be-g*Ci8n#y(l~C z>j#B)t!;EuteJyK!D7n{sT?eEqB$1GcM8zNFNE8}VM{KWKmPotHw4^`G{cH&bTRg} zU0rUshXr@r_kc4Xy$_8{Om8~P-OMl$h^eKMTQO>Ej^TwNd{_YXT3HsEa8df>zS^VUw$tyn2c&E5bbCVjjWh;7#MRRRutwfRZ23u@GG;g$C@}RbK~w| zJ10xU@w&v;I%m_0M57AhlIMarL)vwghzJ^u>Sk*NFQ#-(BpId6bP5xUR)VKa){6DO zJCb2VTVI?#nO^cvSudiLzMW+<*alIm6&OZoSHh)QI@Bep4LVmKtiz0SRgl)T+r zz<#@zd_&xi5@nH|oa|&vh+qcv8lXg=BEn!`Dg-bgm|Jt@+Ozjvzm8?Ap(l9XmZExpBDROijSxQsj*t+sNUHxrK6x zY&eyHCDqnh7a8f#+X0&4=O^u8F!BvtR3-NDqvGrF^8djBViZa)O z%zqqUR?u+^lt@+p1;{C)^68S?Mz57}<=}=y-LV9&9DO&oOMrLEro*O_DatS)CL_!1 z1IOE&2uVgUL($CB;lNRS3(Z2wGWN=9^Q`Tq==d zdH~?;Ca&hg{k02Fms96#8#Y~Uc6`_q433xcNMHLrv6|7am{+l%&yq8DI4+y1kyt~XD-%l>U&S-ag7O{>l82HKc-E0{yoT`FJAThb zN+kmQ6w8u%-j_!!=JdUqzGK(r)<6CFO!)7<_W|B)dGycPWY!jqsM^)aVjOP)>vlaE%+(dSdnUap%xR3(*51;lA-ipuJ@_Wpz8EGB8X__!@`}+Z>4UL`j z4dK_E9D8Mr;{6<_tt0k8oO-7&OFd6|qCBF_xl}D3u>9jMR9ImJCdrC)&wNr$Jorzl z#86B~bXe|J&E}cuUhJE$e2!wH)u?qBRxI!y_8R}Y6XaF+3Kj+{XGafDTIBNf>RB|` z>7Js6b7*_>0=x(r2n0vm!Q{@2;|)lOK$?JXNBcP%ZIHOQAk#SiKAdpTqD2cvtfxqP zY~%5#dw-WZa8`+J9ncz_d9Lqe`ZnT#NAaP^S05+7w~ryYn0!fMNN) zi!yV_`m8};g`(=RcFDE^+_4DF6mG-gzbVNFg#wgBqY*rr2?C}d2>_<^C<(|YfT%xZ zs4y>a07^WW@Bi-vwEy2uUTHX-An&0CY#9!REFeDjQVF{5s{}m|BNXs8=wR&@ikY-4Rf6A3_lgr+!8iOk}s3v=a-tvw<%xL z4SBcQk0v$#Iz*eCd)X2?@U^eP4)gD~Zcwu-VKsa1SB^?hUt~~H8shY0Kj-R@-8YX7 zP_YG~UDFauIt@+H;mSoTK_s%+8KmK2qNK2RdTVn_Bz^PCv4S_ihm`;Na)r$0OR?3y VMT(dNzQKS%#u#&S`T1*i{|EQm6fyt+ literal 0 HcmV?d00001 diff --git a/docs/images/gitea.png b/docs/images/gitea.png new file mode 100644 index 0000000000000000000000000000000000000000..77e228cbbe6fc578c831a6c7cb54bdd07730e298 GIT binary patch literal 8730 zcmeHt_gfQP&^9PiM34tW>7XDAQlujtmEHtG3nfVCp-Arm5k)by(2Ib8^w2}^y>|kH zs`MJ91pji@J!j6DGjq>9v-|CxwhGl_!XL4Rp2}B2R%70Q7Swe@$i99or)^1!@0R5Go;vpTG{ zz#4@Qc=7zXLP*wUChXgMlDK65k8YkV)`BrZmf(0ASPi1Y$ndFf-mGdk!r&WB-FpAM zNdiL-t%<(VI$L7YRqd$kk*j_&{I09!_%(b1{>$X3;@6Yqt=@pVE-uOb@g6mJj~%C-C{y(f;ugS{m92Fz)anF*!j(e zTp{5NTD8CL7vDn$pC+jCCIEPItX?EA77M@7=LUaA7!-mvQ+A7nRG*eEbVkY8$bkoC z#9UImvLmpKOO;Bkd?oMi+qcOb?eTn5Yf1GrQvcg6sTLJT5tYsbNw8w26P)25NEVNJ z;2IezfvQGUQV4VZ?o(W@5i(R0=AHvSWz3ms^SN!hGq4~EvAPbdp&<^XVZI#FX;4y9 zT3ltYQO|QO<_krN8S=&nb;J3^^iq(#(zNgLI#j3O>WB&pf{I|L4W;TZFl0??y{9e# zUg#baHRBK9RXeI*j~YcyDKnay(wp@zk$JZlebhUgSLFGhUg`n#qpO&bahk)%lu*}tnACx;b~evmM0-z^K%sa6JT7`v*7YdYaV6a-zyWLSARx>;0plk`~K!$ z209g`vDb1N)I)mJX#BMf%yWx4l!I9o zI1;vZqOvBL*`Iummo|?t|NCv;%xP5{-8Z*L+S9zOPrqu_Nv_&}hcGs4KcCoF(9wX= zb~S@5w`qw(Eg6ojm8HF+OdNBC0JjlV4`~r1g|B^oYn+ga9@J;B%Jc{%3gHV$fx2NQ z*`y@ox9;8V^_7DQc-~WBg{DD|!`_cxklNRGp zSjn#*=^q`~^5p%9N@1_zS0Xfp5AB^u+jxp3d>vyX&o93<8jWgTxlbJ0){$f9%6ObK z#X4b>t$`Mt8^Dm2appkgnCx2~)HpvUEV!o+9{bn2xX^G6>HPz_al7AqzhPsy z=+~W#XU8Py)YuhH?ZB=%Rt6SiLApIev!%?TP;4bhw=BvRkYu? zw>r1;lqfK2gwSx;cYu;E`+RfEAD|!ZzSxV> zkLS38L5m>iv85Dz@_UP~;_k{(*|ydk%$@&Omk~`B{QUi&!K|u&{obnD!$nHNh8ax; zB(v^8`G?`7#*LBuVH=ht=s3`e9|n11hu+!95`E1UeN9E+0c@ZJtrm% z!RiDPxVN%TX{Uv5`nip>y*u~XK98pl8le!f!k{d_B%&mof4&$pgEyt~apN?l7^dS8 zAbuUPVoZyEOCHK9v2fhT_!^(XBxFeX0m6lz4e2i3jEVcC?2l?M}8cIJ}4oN6&vN)8w{Pw`zVL$H~ z8`6pMvGoUrAtJ~8fR9A&f5X*_GF@d@irD-YfdcDF_P1wgISBT5q&wXzRu22~db;}+ zSXv&u%ajvVfMx&f8@cMbRFcMW|j*1EyEp z;@X)l0~)M^&2TMXSTzdi>nkwVjDB}I={vlGs?_Sqz?b`1%Oigyg`!4wygswlyR;lM zoXiE^8NHssh#pV!;cX8-@B=TKiW~>$elzRo6uTA>4ExJ4*BK7KVJ{&I!1FD$|9~KP zfw;52Et$8Q*k%+)lYL>;3O_Slwu|~1Y#69n$yHcq)oO8hM6oDgSHKn5XA!*c`Up=( z97;N@kh^%@I>Y#&2s$!2fDluDVTwDe+kiFF!%QW`_f2yKQuWmIUhlR%W<<{i-P{=$_>1>(K@zVKBun{1s2m<^<>dDM| zASfeOW(3kFI2@Jq^UL7Q`0+wqlSDal zUi$OMdj=p7&gM7q3uIq$MJdaJ^ts+_uhHTjQ-e)pF{w@|*5g@?)(q<-*+O}XR@KeZ zdPBBw)9MzTn6iIJO`2L&W?6j8gu83&3ysf3b7BpZw&RS^?Z*3*bw4VLp77((rv06< z81WO8S%c(#BX0yZ+Q}xEFE?+?G}WCK_1aqe`Q?O> zcu8QWJ10<;+VG+>H&A{_xwz*GVhJQ2`eY&|(~K!>PNgDoZ3HoMwk<2*0FZX3m2w=K zyT8+4q2bxxCjEdtZpy#s<8Y&84t^sF-lZ3L;qw4Cp( zwh64-(=46m3EZ3Zha`2sRk_S$7giG&->_8+rqWGXj`MWRooNQEOW><#5}?hsxkp(U zAe<>;`ES4OO>q9yfmE^4%8W#p(lf`kwt3I-LNL<0lcl-!`e_1y%_)W0>V!h&wk?N! z?kgXcITstsTWO!SJ~C$>{9u9WQSe>Zkxvg`D-il0MX4sozY*yJ=Xw$mv~T& zjL-&WYi-UW}uFb$e0xL}w=*%{jptHb+K$E7*Xq|FT0}`j!*792#u52)*3ax;9VR<+Zxa zCHAv<1mp1ogjqBJ6gawe8m=t6*@NTcDZ5tRZa2XoC$p!RoefGheJdw*YiMBQ1~}># z3Hg<%YlfN0r4>!8RF&-q2Sa_V&&Jx-Y2`=0&WS()%nfC$>nuc*qpnm3&sfnPx5hAXP$Ju_1( zX+PL929ugNUZCLtGR9@G`JNud$flG#TRmG0?$%Jp$VkG)2hr&~rdW3p4h5PJP9v~` z?#kwEYwLIs7e@?t(3g4$r=Ws>taGSU4&402R(KCEnN zN#({fgR}pLUHr*UprK>TuPUj)h~Ow_NNmJ00elSm3^^pdZVw|fQ{|cpaZ(?+n|IeJnx}naus{M2 zm3*$VaPqk%R5%N!8|pT6HMA#}B&KFpWgU59n>%$zaQV=q!oc~DV8N^ff$cD`09A0g zQ<4Q?_1DYu@-vC}-LyRn>lE1c-1Pxua5y7%l;MJeJXAbyh&^;VQAoYK$Nc5*`jdb( z`+*aGt|?J*DaE9dG@1|6rKb+=^c{T&P?*t^aa_*Ucx{gNU!%lzB3-vj zmW-~qh9lCjh{+sR^RlGGo6@Bxu=cUX7rCmlqU`I|&IsEn71E4+#>c`{lb{5l8M0R? z52sPH^P#w|TE3Jaf$g-Xlbq?huiV=ZO#7RK`N6f>hpbv4cS5jLrphRRd|ANfU|zjz z(5IZyoH=Upb6?B>agK}eziDN=c>NAzDOe^~B5jw>h`Oxb-9e?q9f2&nL~bk67)(oH zNrI+L!OCxYQ<1Bg8m+SS*a+)>AY|jS2R~u!JWX!WKE8H;U1bP$RpOrmKR%#$cc4px_`^tLAXsJJ|&?O@dC9aMM*Wj4+z8-E%K|*7p;~ z-)Oz0aNd-(1)gp{txwSph-Q?PIt=RAJRPRm1X_&wRO|hjCtDa9;Agkr{Mn$9NF{%& z{InbCJa#Fae>S%E{YK@!g_ugS6m<#qrj~U!OTtKwwYs&@aX|`o8&s>{rpEW=LPyDop3M2v3O28ks_V{g%LBh zPR^qlSyQ8?^Nhg@E_6THe)J1rLB+zEgXhj0Rol0jaM!?@utf*H-oN&1A@iXVsH8%<6X@_Ke8%~Aaq?ABe ziwL^mdgArfxdY=&bkN4LMd+Sv)w9j@L3Fv)Hq;xVyAwlu{;Ooy zn{Nh%_hL5D(mT7Qb>B_1;?Zn7s6Sj}M1Ir5snCRz4q{998$wV)Wv7u(Wl)L!3&8mN?RLqyY zgV4(!pAq|AhQ$T5pqI+5Tp}H+oYql$?=rebBs_pP?-#7(NkFP?zZ9w33$?cWwF9%C z9nHnNP2L2)>6qmC`W>$Rv`ymFg3=lV(DPHsTo1?oAu{ydFX6ev%s3t!u5TR6r{R5> zaW(Ihj0lBrsR+x$hGGu4qQ?_Hg^FAz-+Y=2U-cz~`1wz$nQ2q`A}nx}8jp8z{VO3z zc4`6VMuU=r2*ik5qiXHDdBT;)F1iq~x56$xVX6p?H*M9-uv8@V>hkWLOmoaMU%jTrNMgb5Bir}pC(02 zi8+qt=A{HN=#1Hl!%t38-Mob*0VqtizN=n!Z>y=h+{h18%w>G}AT{JBMa0eHaR3i- zsBNN$N_$-C^t2-Ah%5Am_r%3Nj^DQLv{Y-z+)~`Ay2se8OmaH6s^PI~alx|wb~ju9 zDqz+ivQ`~a7R+Oi2{wBt=3Q`#xOvK5=DE62s-->%=D(GN15t%F{tE(N?y*o zJuh2|gud~&IKcjD9~5uN>!c^^2v{{XG2IV{Y!-EQqc_yJ7uzU-d6y`_zRZIc2cP5$ zao3BXey`Du==s~-ls+l_Yxoa~ftBxT28wrewzM&B-?W|5*s5DvYR~SN%t@7*Pq?Oo zoc)5tfYsu+9#)Ha_m>bxwCobTtytF&|M9BOvcYs;dKx%vAGsWPUL}y=xn=>A7i7ry zY02=PI*iFBUKF?ol>a_}-5S)Q+-*Yr%X0Q4+se-5`UA|Bh?fEx?v@T|+ymSqc`wJD zA|xcQk8GJNK_WIBlj6_4r4YLp&}QuVDBJkPDj)mNPLd9Zb8P2*)M)(ZDavo(UA}X4 z)4wx#j#dq|Q_lbDikmZ@NRcGjZ8L9yk&d#|2hps6YMGv}Rj+&!G_>N37d&JDXXMVQQ^hw96i47GGs38;uIHwdulC0IF3-dg45z7nzJC_2 z>jY?G@3l6fG_+Ic=@A-|Bd4iuGU+GqK6vF*n`r1+e3xXjPo0W)GOYfcA>L-~TYT#; z%TZpc<5lh^{l<(Jy}nG<{?4tXn-`1nWv0bg)U<3=&)e?ZKTwH_>7{>h$w>EndCVEX zTQZZHxH3v!Pvb24dW z#Ve`=Wc;e-MuOHQMG~SK4%c?^JM-<8=0M$-txLaZI<@QD%j}S*Ce2m2dh)rdz4R0x z7nv?{f|!V*l%{6miEPYGEZ5?>CuYq)dQv3{H&Bp-JbMcsqA;vU%N6KJJ{z81Cg8w& z;8DP$U;i=_{__cye9*hb?-Dm#oR)Ea&|3$C8Tw|PR-fiQ%`@ftgUu<8bfk9-^n`TX zn8ue|?IQkvj8g7#+>%Ra6-otCzo8fdD9cZa2?&Yv8n8W^%ijX1h)>sX5Bu*GbH9D` z!OU~Oc+v9!W23nW{V{sFgFk)s&jG0uS^Y?YoIFEEwqCt9%7i9Oz#Bx15Dps9prQO* z8EI}v7gk&ys47}zsm{>!zSGyRx!#F%>APQqgvn{zTzK;bv{+*Nme1t{JSD3Z>teHcL@E3ju{nj-! z$bDMx5q0U2AZCNJ#KV+pZ#tiDbV|rp%X)ha1fD?&JYDr)MHYe7czq#V40&<#TSDY? z-bi44fj$`E*4+|wwG`Rd3ZF5W%*w?7cK_Q;)w4es)wrf!_u-daz7A;J2J=iI#B%5j z8zgEOxQdt}+qK)ee)VV@!+xWDsh)b))7Z1%WUS{^uInslOvi`~GA{14sCAUOlp(%F z(hd7|p}~c)FI~oXK~DYJ=l#!-FHRt2kLuR_xivxZ7y$dN} zIbPM#z+ifIavBa}cCAISv_?vcTEjt!>o@nC{8RD-T1*=wB<>*bE#@PRhnx<~$9u=B z7YCg{1Ax)A-0;Yfu-|jy)WOs{;Wbn&31|BKC)A(b{2;*h&gw!v+5k{6;tb86I&X6Y zfBP0;kWhP7SM~Wk*?LiK?;Gi&DZKIc_-nn}nCfVA;AY;;+&l#h#}>|f4%#f#&>1%T zblp(;`(30cmfm^pCQ>TTQ-8ion;1EpKgfvow(N`+@MIv+PM#zggv-9R%ew$UzJNmz zHZQDReHGVJVqy&F$j}ZUQml1pZ*RU`rn&13W2A8jnh)?TPIOJrU-*$(9OQ1~rFyVq zBpvm4t0aRn;lYeD!6AyXH~Q}4bhWEVz1T~?J-Vs&_hOmNs%cp~{k##3cY5n&I)l_1 zNjB?ylLLI|!u7UlaOhxukx61ywc_`}D1jb|xwknp&Em5|(EqA0K-UC4I;rQ}MQ)UA z^4BAh>-*QdR}w`A6}w*k6$kVpG+Am0lMgG0ZMqK5kwbHYA>Pgxt9Gd$_^_ zK=%{&Q4ygRz*ca*-s*g@ej1zb2F!755xg(w$Lk8~p-p1=EI zR&9c!duu`wk=l|6qh$+MEtN?N%+>nuwNlwFuoH~>cjc`|pDKjBq==PEG664(@$WS@ zenK41dWi)=JSj?Ja zbWK<5u$#=z{JQ4p)|k#Wp~54RGH-h4iH7|_fG*wRK^2q6Uo}0{$@swzy)#2X&lfo= zgKC7gF#Md6*-Y;fS>X@_tCWXT`!hg}*B@bvFeD*W@h@Ivw&9n^`wR+`QS_zZP%xHu z_wVvZK%zCoI=G}tTErV6H{aE(UD7*Vcl8B|gTJ2IrwFNMF%2`ZE7hQ>Q~nNi3{QnJ zlGFU>p0`#eTAIS~;y}#4bSuVe#;%8!jZi7@do274!(KYgT3HF;Jb~@G+G&pjN2@FR z(e`|sg`~V+%JxAC{TMt%Ui9C@cJJgXI?C&~*97kk3-9|`hKqKzixh=9hH*%nU1Avn zC=aIe90`cW|M=}g7Blgch-y7f4+qtZttrRJ&XJ{>+7%SS#GQxqn)MU)f3B~+dn6ez zvtz$yGqsOU%j*BtyVBFvDcA<+O24zR0nH#f$~fE8Wb$2u;WB zA%!u2J5K#eGdosnFihy#yT~t^(g42gSVX{4rbTLZWGX@KKqR!uPCbxWCy{ee08Crh zr|aHX)S)_RL_k$>YzF^j0ihbTvixbSP#@j%%-7W;mq&O9wL|H?MtM{QJ8|?f*mnxO zO+x;miGz{$LGb7e+p4k(tca|+A{Qi0D`4f;{X5v literal 0 HcmV?d00001 diff --git a/docs/images/github.png b/docs/images/github.png new file mode 100644 index 0000000000000000000000000000000000000000..21e5d95362abbd853eb7147560fd6c2497a31b41 GIT binary patch literal 5510 zcmd5=2Uk-|*A7J?cqxJk8k(2l^-`5~As}2t%8MW%5Re**NC`c(P(%{&Bwvg~XvgoVxsfj}T( zD@!v65QyjD?0ko_ zB`9<0=^)VVZ?_*H5K=mk|x(RW{2KX z-M-iV*H_h7e`*VUl^*#|_T^iQi%mf#A030Vor-}VUyqN<<~D^n6&LKzLx_7aCl}Jw-+br2+;~FU*E#K}~NW!3;o29Jw)$tvg z*2-nrO9mWzaJmSXUzak$NN2nRvVe73xbwu+alr?qn5>$bBh1=s`4~!I|{;eUjm}AdD@8zlO10dix}@1 z`T#;(^mbXyA;U&^+^<@YHH&A;;W2CZ8vRe?hAR6PYf38wD8)i9EW!`Afl$ongNSmt?8xAV~FX6PD7H$z`vZuluX(Iw^13O{y%jzJtX z_P1ReWXux)#?*Ty7yWI_C1+}1xp%I5wJl~e{Jf%A!T7(xIqYQ*A>FQKNX&(QDCI7~ zZ3KRm1!tg5QB?OsHvq-B5l)B;4zb>_$v_g9I9Ugz49(;`t>s1pYt+~xPfsdtSE&ca zc?>?LJ+GgYbtX2?d;c}53WFKD?KfJVP1Ur?bS|q7r90GuBpu0)vkpSU3>Irs+jJ+?h2%XXn064vPs0 zS);L;2eXVW9nV8wNjtL2MLscQIG$mBOiD4?SAapVAf0uV8;;pvo;SuF)Ea~e%=E(~ z?dTfAUxl+L!!&N$wUKQnIH5psg9XLr=_W=}M^%V=8t-bi@AVSZ6Gv@+zf_Cbl1?GP z0Z7C3HP#Rt!0TMS`Ybb3<40@=cBQ(wXD8(CrG^)&^COTo)XEKF(t9t2hcgjFV3pvP z(Z>=$aU4%+l76&@r}9xZb=P7`UG|ri4+>UclA~x$hhwR0sTPN(f!Mt1NGykxS?P$D zd3mEEq*or-iumbEyCnJZtvFH`Jcy;HlWv#*r^A=iVFCyHDm8_Fj0R8YZ0${z(RHq> zb+pV!_chLS(<%r_$o{2WxkzG4hXt}yrhWWU(iVa78Y10b928Lr_B2HGwA)eczURC? zGG`MS{EXdrTBK?&MvKLEV9~IJ&Uw`+tlss+pdzGUGmDI$pgzrg^1CM(#!^B8!lrkRbZZq03&<8#I=6TI(?h6sJ zbKWT-aQaBj%#gW*I~MnB_eVBGVl7Yx-5}XkFBOZO^d$VIKgZE zTl+m1tps#BsF`?-h~$}T@*&v)+8x!ktADBHM^6ee-!;BRtO;5pf=LJ0 zeA|NCo2D(|l<80U;*Sot>TC2$2u5xHmkwt!r<#V?D{RGnt(&W-~aS!}*Jb#q8f5kPzIAh+Ih4NW6-CqiNH(VD8yj z4QPP#`Fj!#;on#@<#4D_#eB`QAua>@Zc;*k;*jk1OxYjL2s59ns0l2|H}`fx8*k(u z6i$f--xsOw(AsEXxx??2Zgw1xUBl~1s|{(Fer`Dz(C&86n@p`I`1InXWdAnH-=0yh zl`qt559N#LdTlY=>ZN384R7w-H~MC8$`Ac6tg!r&UYvDpmzTY5NiO{Y9Bm zc*p`rB=QO=+Sf3(_ULwr(xVIU!6e6P1v3F>ZXV_g$O6>6=Dcu3t}MKCPFYoDVOar} zPem(qfI85Nv8D-wT{ns@w-Hi969#;`OB-WLii&tUtsaJb1an(8fKQZa98Si|>@`d`{B_&0b zV2va`u-}co7?FOpbU%Np!+)c4cZ#!HXcI*-vq5N%4NOYy{aJY{q-~82yzs zS4dqcRGpg&--yPCmh!Xj5DM75)@^KXNS$7}J1^WP);06kz~C9TrDiZ!m`z)xAFJ-} zRuI9Xv6=r4|3$Sz!m;-`Bw>P?3wL(_tV9O)I8MXqovZ7HtB zl|Y#gOZoAr13_kVI~Q{DYHjIYZnl0I41jy;9b)jgpMH;;4pqR7N92S(IH|k!(>I%A zYb7r(C~;LD1+zFdG8ory$z&m3J~C+uq-VeL^z^OH&166{ZQArg0_v0e7Kx?};#+TP zq~M&H0&1L5YIsAF6HOXUly#PoH?&?|&xxA zoWtKUy|>Se&;Qajhy%-$3Z#yT$%4}x^^@MP4Bi^J#1%{o2|6<(1y_v}^SGkr_&FXD6%UEl>wDn~hVOGzmTZB$5`ouK&blvrhxG3dqyA(WzTM?R1br64 z%~5Luzn<8WJDXM-#bFd>&0Yw_AE*i(W`BeOFy+s=HE6q6Y4%WrsZ0urmD?Bp*GCM& znw68)_)=65GE8nbH^Ih3K(qg<&t(De5+TN&e@AP|ahzbb;bQ+Z&(8=JhZQnDXXNNt zPJ|}AQ_Th8$#eSMsop43^vA+)3%fTBUL=f}%?xcZ5T=5gtVyt_;0$(5cSXjlu)7Ax z$FAD8SMTBMdJ++(MDe1Hu5bjaU`poVCeoetHmf;r8JI#l!+rNSN>cJodLdu7cNy|$ zsez)ngRq6U)_9{uB~t^L7>Ck=%h;$Q+BzDh9pW5oj*K8R9nTzFxtTYe4((RK=)v%a zNs+sa#yJN*r1N*k8Xs2)krM~%Vm^~nhf%h%33AI7+xY2DgYx;A_cz<84XWXc@73u$ z<}x0{ia^^T4{NsX*^C_IY>5H0)0LdENylpG1&XzX6eyNGK`OePQI>6gwY&FNIfqFz zlP*V*`Z)d421~BY_8A&B!N0hz<_jZZv({^(Hf^cixHRPL%5|e=xb?GDwfIjh$?^pVp`BTOMTKHFqLlxXn?NkjUJs6&{3@~dF3{WR!(W$r83fT4x_trR7EGME6EnW ze(pvim!uESc;`HPM@w!BU0rx)nC7oY`Px1u0q6r1%(hdRlszt{+VdaOm8f2vH12Re zrP4_>;1)3@GdPNjWxOAXH5I2v(&I^o--P=SUE^S22Ac%lE;Qy6*}=O zwT-r@g&VBBQpUsM}c`i!?f!1zwsXI{WPJk*fLqRau!ig``2L+kCk?+Mo+Q9ko*;88G+v;xpH z!sQ7#0x8{RYfLGxCd4pb)4QA)m}bC!4qW0gj9tHAaTHrsPdCWT?CMBfb{;`3n{?1X zv4wp6NbAT=*c;_N`YIR$wbi^?Srfjo6VrrCwb%FOgYIcjTr!E6alLrWU1nAvv#aj{ z+xz$;EE*(qPM?MK-$r?=4J1s9oSwIS=CuYXmGogw1AzSa15_jX? z*7ihBP47`McT5PI=nlVug|SV<$%0QjBZtzPl>{7g@wD|5!%LLf6#*9q3upRB#D|>- zmPsKp(LG5NVL^bbDkl@8$jNUZY&(PKhpV@b(UmNxjGD3|kOJ=#yAag}YfD+Td7 z=8s8SBpi7So(^mt1DfX)c8*egv;dK4lcAe{q6h^Cr zjjVBv?B9u9<3)$-7)d=`{Ty|XXf;;sz0rb8c=LgW5FrUMJJ3{L`>Qs?ueEZUu< zyT%DyOQC;as0c$>&{LWZg53@^iE(2Y)bwra7Gkn`dk!eYs5NvhbTRH^d97ZDG)2p` z;dR@ab~Ds@XP>meXeC`T>lansQQiDCn*mzq??AluJVW~nvH6(sM%vTunv5xD8uI)H z9#?FbwyN%)#B!`s3TZ1NjU+q1MmX-BX^3U3|5E~U&3NwNYiP%K%vdinLYq9xG!A+8 z64|i@jrA?=-&mZ=r$e<-ieldv6@WUR@*S<*Mn0_c{f>1JR9tX#YngEt2tJdm1|%0M zq`FVnW==`CjVi>XK8iOXF>|kUU^|Dz-N}XOJ7+pH+W5bK(`~l0GEUCpJ->|KJNK{@ z#fXbxRb}WHgN{Y`kb*WwI#3;CarZWwdl!-ujtm$%6&s%2O~(8#=@4Q3e(omt+pCQ8 zQ=&DapF&dWTEj@$)gvwWo*BN;hs(b(K(W`F%KxyI{motem~MajaxO=ilc20(yNRv{>F#aD+XMszcU6?-bqELufBbvMZ{d4B zwuublUxc1IN^c2g`(B3N8zhc$>T(1Gm9dmp)};70o2{~rIspNg6F>bk0m0=zvMmAv z?^gr_J5~e);wc0K^zYM~v?cKaw`|mu7%3HTn0_sYhe1O$BS{~kiWd>J2n zCz+Rux&qlYDa(C6GHSoPLi|SvROIFK{1B*l!yo27L7NAQxOAukdXV73k8y&OkEwFc z$ioOp$Ov-(uyKOKXb7BvwWn_Wwb&(dLlp7V?{Hp2)YOjs0H3kL4+Z*+euyEJ2Z3^2 zqH^SC*|6WwXzmbE(A?>LG&py(NcQ95lP7?N<#dU~kp$JpgXzfm!jM92CK`el?El2Y zCLR6sIaJgKaODfvUL8Y{YCYz`7X{X} z{!woEHUH*0k6+tj#oCg`(oA+Jb1em`d1CpxXTHjQ{Mnzf*IIe~b6f@Rdil}r+#ou5 z8F3+|8coptP{7M`(<2oU!z=509Xe!A$yE!pc60|$nOQO--+kM0T;HwJa=~l=!^_T= zML)%kGw6#Bl#vd0Dw4|BEopLZ+z3DTFga$ofI6i+O+4f6tKlY}X6$+`P&e^0VHnra zZ-_n0!Q}O@hE3-URAN&LJc-&yqifcDzi%$4z#5$qFbmaKg7%4`;}5Z%gAdnomfTd! zo6H3ZD`p4Ny0MtI3*Ap;UkRrmKfC$lCKLs;07^m1XXrhBWy1SxMKw+s*h%6E4h5L#`X2 zk#y{OL?rqa%MNFn=*4jrJs!#ql}R{szU>opD+P^SVUQ>v6jcU%egSk_m`94ojFs|j zO58+fSO$r_o&1GrF`w2-BOx!=0-2ma)vJcpXS-iv?NzIrBx_(t;t4}pfw3LnCqyKZ z&hg`hSo5 zO5`RCi-xMq6@m&@}fB~bH( zwl8Dqb;M8!gOmy5y+!B-=q27Lx_umtsl=x zHqHWZ-|Oz=r<-wx{1j@NHelXEZRLHYpm}D*2lQ3t9`&UaX~CK`MCfP~h>Cjqq?8tV zwW$5(f{>8kw`kALP_@#w_E2J_OY|DedL5(ik*(3fLZYx+A92Fy(3HFicmO_12Df~j zox}zt^XBB?`D&?k)*y#U^LV<|U7S;WJc#o%UIsBqn3$XBuGCqJ@t}D+*krZ%)=lmW zB8?&vC_BkY@0gZ*Msg5_ksxM~X?N{>A%*uYh&5%DO;x8l6&k0UXHKr6$W5Ikyi9i1 zDb2S-3#FNluj*U*JJe*sfV2*mofz-@n~C<3|o&-TH@oLxc&Hg+AyK?dCk z#A03X1ae{NAw2#4V4ucRG_w6eLEq9k>hC?1?qAz}kYU?g9&o_(K0OTz8m#-6&K|!tZL=R7Y0kD6(e3LSJaQe{K!^Mca7)wkkFQn zauet*hN7^y`}l_l4PQc0@Zq{6n5ZAontLrSga?L;gjzoe*cT=j88#I*tid$#hJ`b)AvqTFRsbPxPEBoQCaNoY!FHr68RLl{^3t`xW!z zr190N+Px{tdu-omJ-hY-$V3eD`1n2uyZNH_KtnqGc$uw=e2rrVn^uyXntZl4(4-R@ z_W#Dxb_ZSqna1gO>c1eK$kGtQ*n%xhpmtoo_~@%pfVAire0_8(8LdS@^C!2y&Tk?5 zC3or7T^6LL$zJAB4;9|nU3WlxE`rYX0DesOKWtFv)$bmkpLg!VM6TAR;_>Wq8(PFc zL_+(pUzd;hwD#3jQzin>Adc(f7Gc!2*czlP zBBh(Ve(L9uq9Bw3D>9h#7XS5P$$I&@z=NO%MUfnZgP6s8*l;!)HhqW1m;r}oFY~v2 zXOLufi!oVvQV&(%;B}__TDt2^ox1sMkuVX7yrAZ>CD?|5o=PAUH^#q1|NM#3-whMB)pD`VaAC(efwr>8!fuoi*DS{`GG zCr~=pxv|=`G9)nX7zTV6W106P-g=q(DTH53i8dB(5x~NVI3-Y;kpF!n3x= z-Occh=eVR;7{pxMFoPIes!8mj`$vX=Iu|EfLndLBk3UH~jImWJ!W5W|0^baZ0NBuq zc!j-jb1LNdwP&7AFl#M-&ch`8$=83Vi3=(tL)Zhn14FC;w)faxG$l_D!#-?(cR7FQ z))c!6hzepiRC|GJ`j<-R?eAMFLkBKd+y3frqot`y36WzFQCu!J;L*pI zkjlm#j7n@V>W#fVgPiFv0{m(Piw4X1QORu;1AS%rq^&$ED+o);eo zIvg#is%rv`sIfGqLoR9wM=HSS; zU`0@D^&n&JG;j81hndDi4r{^+U2s8 z&d5|emw4jRD7+w80AM(MoJ1C^IsmtFp#S_urSa0aomo$uFzkb}xy^v5;7|pv&&1G( z{`e-`{h%IpQrV|D&5}^O;p(3}2d?P}i)wT36`~^yJCII-^0lx#;~9-x6cvm>LEjv- zcWv+d=q@Ton*t~KCTF~p8yV1nIBJJUog)(R?xl%UP>H&s(b?!vsL*tXk#L6c&oaI4 z#q87aXX57idxbi0;(8^*U#GA~vaR6jxv-Tz}!raA30x=+R2n7>3wJ z*g9P7IBkiqFsC_J9n(s1*ZDn{ebrrf@Ot5;L-7IMy5vNq%lX`_^8x^6`NR$?Rs3~i zd4s)mZn6G%_Zx3#-@Ad;yzeO4WZyGvvX~-3I@2kgoD~M9*{v={c8@;qQ0NAKGk71- z1o_$t@CH~K7fQK1n~Hr8FLtIZBr|m)B_ZcyB40ia-(fJfh4)$7ImNF-jgo@78oxS? z@V9QHDGOfv2U%AxrAYAn!{PV&8eTS{5hH&U-4C;#T^~&Y&Tg5scdbK()6c23cZpk- zd7ht_w;dj`me~^<5s}b)o+43E_H`~{nwU_COGH)Q0z!hkR)E|)acZCP1SlOptRSUw zbBEk@XoB^m8`@NatOl1-vXYN@OJM})>^%Lf` z8oILE&11K6n)h$Kinq}(JkBn~XKb#ye@_IL@?3t1AILVT=gNu>%+1p9Lb~DoxP?%` zvCIZZ_`b+fpkX@?6I`?qU(EMbxUlzbR~(AURCmnT_`T9mAv6M-7HUa zbr9fRpoY!Ur_s7k(A@fmZRcM!svF8*q$M8NyJ+>x0BrmxH0YzcrS%tABFtXwY_KoD zsnrc8NOK%ce}8B03YZ}xjCU$z0cNbbDX>DI&YAr6^>cd4W}N^mH>Hp8h$LPB(paAI?uyc_F1oBLUv=*6i9`RC$R$v>6*Qq3 zpq>V8pv*#z!1!j2iY>EN{-v*3Cy;z|>s@t3kwXZVbC(VX3DK0ljwaENn*z7-*9dd8 z*yoXv64@0?9E39u#k+AI4AGt6uMOy+WBAGd7dq0_1(WCqHhMo;-{1;bnTH6%>S{m; zx-4*MP?GFNfcCC6+e zy@@A%w(#+1Nxo4#Ce3n3Cjg!P7a0dh4&oBDNTo7qv0Q559a{I7!#3?7qKhTM+vcY@ zXxMZQDFl1x>>cbaF-xAc3%e(Z0*X!s$tAG`b3z8(o1a2CRI9y6e71rM%WM`8l{{h>) zmPyg)sW#=Ivj`oUyd5Ax5p`!=%+mVMQx_Iww@Z0+|A1y9Klx#CbC{-e$w)A8TL=7* z$WGNcO$;@x)3)Q+6~GS{o}$Ake04~Jz|N)%Rguc}-hYXJvDaUQrv>|y&jMnjC)EL@5uE(90ydjYznEJ2LPRJ) zJLp>6GGuphz;M?%qv_XAw$_qX-iiWObG?&sBUou$iR^$46H8gI!TAs4K#SKR3Hbus z0o%rSQ)q8x_qg{mn>06}3P14GH*qqToYuv;A3xCztmm$$lFF?yXKvoTUw(9-P51W# z^SpB-I(ofpatF*iQPu!47h&~oSeu-ac45jXXs|GySmF2273n@e%z1Tg%b?TWQNR1B zSDPQDRG(sOYgRtuh&^%Hm#PfB6ozKNA{L6K>h2YI$!DMZsWP`af$)y%L^)?!>-?ai zJp0>8!!~KYBq!M_*>)V`AIvo6kpks7@ z$eTHt0DpC6MwzqTa|-6%4AS>y|9YcMb9p#pKJyl(12@n50WDEslPyl=659+!3V|j7 zA<}Ns`ZDh@hNjI$z|)rXH}npHq^N)(h-Gp*bfDyhR|+4Jv9f@G`<^msgY+&*(=IPG zrjz>UO-kVA6M!RxQSEpl>xqW2zD@m>%A8pM2~l2}0x1qEACp2fP-2P>?QvLpNHUc+ zd|@>?_c@!DQm6H)BDEN)sWryLiXwn_s!i!X!_rex1uvRMfN3X*YG*yJg)}TJrk~q2 zA652>UIIUW$Lgs<44mN_dR`4wS#Nax!dSi%z+74+^d80R0nrr5Oi4#3SM5Lz0YJac zrmePw`9UgYS~h)4yPK+!pWnj^chR8h-ygt_i1KzOT`>U*^DG`5yFZ_0txp5fe0ps4 z>`<`M1wf$fD;Zk5K>L?F_1)Cq65rG=U!Ek(CI_zgW6LTe*-d2T7lKp)0pw*C3-63W z@`47y^0`_-_{f2t(`o&EnMLn3_2OyZ%N@qmgBo`!1Y8zG!QEt@%RaiZ zH+9pN{~S+OT2RjVCUXISE74vQ8gWt;LJ4kcLq; zYQjG6o+>hI(?vPEuUCFuU(lAf^yNnS^g0Lqq#9C|_iMPh1UxK#BKauP3XmK3 zy_xvP9a<_jCF-+AR*Ot?)uWvy|H~mW#`>5m8d3Ur-mkN|`F5v661OO8tuUgb1|I4E zK0WR@AW(aloMe?T<`{=+>`;y!z`dSgwzh&#!LQwt7+$B^T|m}AWhuaKr+VCy&-x@z z%!oQxW2~pa9iIr=N!IFFC9k&r{$@sBejQL+K{D+ymM=Z+OuO`R`xHFe1S*?j{OJT~ zqX9&n*~lf!Tr4l!?(Wo`v?*%R?8XVjFC9NXP3H}@0>{Su7WN}u|5{Eet;-~|=XYR( z+IuW2Aa--!(&Ek4?@*M6P*b7wn&~Vp z{p>-a1)5Jo>zykd``eU~RIFP^&p7h=s=ZqapDy1!S2;=P@u~cY9gefMRQID{dlC9S z_fO$~c=yj;+g-i=lU8HaGuJO8mEa!jlAsgoJ;aR#vx+vJZFy7HJo}pa{Rk*Qb!W~D zTic%Xq3}7Gt`UXsm64L;qa7|4;EE4ultSvk-gtMF&)dVe&YrzWnA(cwS2< zY4g{8d|8D30!2nG1GZXHLtiSIQI!ac2f{1l&JF>Usa&SWtx9vzAAL&r492vMz}SKR zh5f-B;GcE$WiET6W@$-41I8OYv-A6*0j>D8AGWDx)_2IJO6p7r0h2p{tZK&G*;Qh_ z^JTib(jtvczS-l%0>WRFD|QoD@w+Tm$*-dB(>?P0xjx9>YR`wG%e>EIVid^6@?x~3 z+RVPUCH1@H8%x%AjjXl2EdL9JUxGrrY?K^*aiUW`G;GQ#N1M@iuE39{qEj9kl|Z;n zFURTCi-*+DX4K>IJX>YRYksaV`xM#k7)<3wmPK8-f_gm3Zn;^5$Oywe_q#IrJ&oLR zQ7JbNk|zRAc*v|42%Zt@{h8`Nccw>ZYCnuTYs}I^I>+H-enZL&HoF$x=_ zTk0*EA?|WYt%gr*L~+xy#J3G2@CmA}{QUHS$&p^3zT)5fi>6 zqd@w2IR7Az@id6Bj`lRdnAGRY+y7b6(R_AXi81oE@ar$aT^TnC-%G}`GmHPE-D4ds zLBNwX^8Gz-wwMFf_ac^xRUz)&C7%S7=}gp4$-u{@hxr*)zntb(Smjxy;^)QBediRS zWnU>wO3&dA4>WU1=B_2|L%=G)7~T^{HIXqF*cIgL?oe=rU0YzCjA82jeh70!pztyw z5sAx9M*D+pI5D-p&CoYdgt~TAuvz3{#xQK4rK&v6fiWk3@7=mT=7`i_YIM>OC~nC5Elmk1-VnVWos>@;S2=(tc(-z->LWjauCP-w_t zr<5pz?-Wz!KKeX~d!UK)gkDP0EbipoDz$T!UnAqOg}UFzd*t2Hj28)~&`vGQMBtFJ zm+A;vfd0f?g8A#~cUYb3QX){0M$Z-yQbnt9oeyf&Ri82~hq0$ z2c3#uUN#cHpT#CU{cV(CDY&vITzxv`V-q~%0~N~2`PIG_w;)v`lWsWdTxyK^GH-37 zFsn09oKY(rtYz&SOXjov>BV7qHS*RCE1Wr6R!eVlI{MrsS;ly*kg?i$A@|Iz%UH!N zbi)zYQtm3Mzr~6#dsx&QZXG78v9O?V0{1WelU}vp!uOVD9xLMvBki%i;r7D&s%V3N z@W{Gcpt8Xy`d+k;jyl;o?Ed~d6n@`P})9x=P)9wkT@*b{dj)$s<9{)DcZFDMH6V==52}|U}?YM2R`WF_YCh}_g zU0uK)bOK#i7{H;BXo|($hlG*Kd#fnZx8n)EiNARqe6*zB2}BXL`k8)D#-Z*^Mx}%l5AL=?RiGuyubQ+5jU`)_9VS*$ zElLRse3GVyDW*Ffy_$yLv5de%8Pmx9A6M%yZcpE5!K1o_cboUpCnf6RMg2%&cDWey zLIs2{jO{j@J!uk}&oqLCy~<5WxS$=q)8aH9*=)3bRLjhYy*mrSk~zU%m?RXEYuzr) z+A@P09P7QFj&8sfodCH>we+OD)Ot7C z4-mgkd$qiG8j2a`#-mk)R&y=pd5_FNEq>3)kwT^9|LWWTD~3OxQxZLojt|<{9{qzU z{H^IhbZBMjR(a678q2F~z${ElfE^&F>&fOd6Ns|pC(3B=v52C8J%$0u8gQ$F|#`eZL?5jTYvyemqJ57$@bqRw@i zPWj-nCobHlnJnj$WAJz78WZj*4eCsAJ zReh({B=#z?#Tv6+(=B|mTDG+>yDul?QXT2_;O+>T`wlb)gQe8mC5E4dr4D7F|$s$36grNcOlJQPt%|neEoWr zYCEXaMI@hvh%xe-DfoRPXiq%YPK#@vKe?Oy_4&Q0@TuG&x5I<0=Co}K%hHi}cLv-^ zz*&$O5Pv%0%fGV#;ZlyUh_>&6?TGK;)OW^mv?u`9*((}1Dw~I9++fwJ2=~z7Kfc>P zWBh}{)xMlz!(v;kcoUjG1$y~7F8k5U%(NLgkH!X7mc53~TyKuW_0~!?*WxhFQdT5k zFeyg=2FwXQS+IDZvY5XA;NlK)DdwjCO=fyWU^NE1SL?VNIO5mQ3A|Fb+?#(uMLkWe zI6V6w2JFKcEkiJXjqDqo*|!|Bx$D&r)FgmxM9p78=?!n{3uo=2yv}mt!tS{Xd|gtC zE;S}ZS2;O0U8X)4=QHKQ#+TCZb}v21bv8#&>s)u1WJ)pGrMztiwE{*JllB`vmMXR=;xNo{Nio zDR{1)CR!G9{vD@4>D-D zVOZHWgo``6j6+iPBSC^5u6Jql9}1x2LgvjKyWi9d!^T}Ce_hcZ;_Fj@sO}&meNI-? zT_STIjuvRF)0Z}xDd)10X`jFqCL?6%GppN_5gv!J$;WE~%ULnrL&`#X{IhPRK%edK z{88(c_jWHHPoF(dr_3>Lyd2Q9R2Vo0{QdcwnQFbPlXUuYuI?t}Cj<`yFG8CZ-&FWY z0u{qb;ngA`8y+$b9BNe~HDcXKR-*W^#MRdqA3pa`4`)#P7yTdaGhx#1x!1+GoF?-N z-YJJWPXKMHVoerVNlg6u+LEU~@=ws#bA520F~?n(w@rrx-0Wf(3mxP>64=Mji<~Ft zZYmn8-Eo$DQfK{mI6t_cm5Y)2=tYrs^4^}*e!`B9Q`6Nf8+yUlE$_YgzhBX{ zIVu`9S<;EC4X8UtdcI;?6~D`uYD)zpP%D-l;SBJq*KRv0uvk9v%QtOCe=X`|)%au% ztw3bO#Fja^F6lwPszvWyNhF&J-p?4TZnv2CtHhliWH03lX3aDWT*v6~F{U^E;Sav) z)mel&?6w>iz3Z9JS?re-IyZO0hy2G!eV|>Wzt1xh)Yuu{?9x$__R;ZP*NrL2b~%zA zUsVb?%US+gWZ2T@_G_?WwX37CL;o`wISpIE4l9FXT5u5#{o)YBXMQ}yomJ>vJk5{# z1MP#EZX{pxJu@{tU$12~mdW$6bSl_?noAX%=YGnp4C=k!_)8=F*`?zlUGY?755$K| zZr@SB^h4bDYe#Swu=M@1!5(X?Z8=pI`IOn-S-!xisUR|Nl(el!$kotUIlGDfP?U=P zDH4KQIE$UQ^(d5pB7Ro-xI0ep`Lw0$(6`i$S#F6fpJjTCH*atsNS)DSZ^-g?gj@TzQFddl5&@UNMR z@0Phsf`8&puTDRO!wB^R3o2gO|5qy823$&1W%*w6DM+u$P)J{0APtoksZP@+oO#;L z-8&Om(`N4tRcc}wMgrQvuB|GyGOXU`H=gLW_(hQh=-4fGnxB#X-}kz07g0b(z-`wE zgw(Ek$f@TE?oH`_=aHqwobTWJFZ`s*dEuuvgetjon^#D2y807M2n*%Z{9iVF6kz6t?z*tUKDcj=nj*P~zIvifsemgdoxZ=EvQRGE40sS)bj%3)Rs!*er)(_)a{oQsOPkuIPqBl zcR>X~P?N*6JDDCZwID>frt4(@wu3ZYsHCMrqGDQaJx@6y zaaZf>ZYMo|EEEqxA)Y-{VjLuRnK?52ZTiBNtj+`23rnY(u0l5Jsjm`s2nnM5hAqB? n-?X^Ap!h#=JnWy6D;ONH_WqypnW~7Ee{B^74f(RSR$>1S9QE}E literal 0 HcmV?d00001 diff --git a/docs/images/local.png b/docs/images/local.png new file mode 100644 index 0000000000000000000000000000000000000000..2c04c81a47513e24dfb4997e459cb3aa72475f09 GIT binary patch literal 1902 zcmeAS@N?(olHy`uVBq!ia0y~yV4MKL9Be?5hW%z|fD~hKkh>G(&67IyKn_c~qpu?a z!^VE@KZ&eBzCyA`kS_y6l^Rfa3j@P1khT{L45bDP46hOx7_4S6Fo+k-*%fF5lweEp zc6VX;4}uH!E}sk(;VkfoEM{QPQwCwiilz2tKtc8rPhVH|m#lI^R(i=Pujc_ZZt`?- z45_&F_U_s4&{CP>AK$+}bLV2?94{l?jqHbd798t1u~fv>kw4(HKoiHMq^1R8dPysM zw`7KPD@-`Grq$)8T2qFw2aDsey4#IWLb}@MmzEOtz8zjJejI z|2;geH@mRK(cuSwPF?!9ml|ig15d4b`t7FCr|)Mf-mN~YzgO<^<(C=F4IlJp-ZB0j zrFr&S>VHS$@8{?Ic=&K}u=I1;(g-UlVV*aWk6#HmqO7!N%5S}>&57r)z4UjNsT2^< zJ8(At+?tciDhqeaYd!6}eEof+w=bF-3KZSkw@uX$cCIjcH>c+JnOHvt0Rge<0I%Zj z1$Cb;O}Th+dtUA9BUb5*930(~uJe3skH4MkTOrTHbZF71z_t&sSAWj6t^NDfdM>Ae z!iEmx%|7|}Y>SsGIyfxwU)nkGTHbqS2Zsd1Yh}^GyYuJR-7sS1;P~>o?;7v*y$_dK z0oiL^ZkAcD=Az=`2WHl6|6AO->0UGuJ3Tmj8LZoQdf|z00CI6STwkC2%q_zOYx4^uM#?^^YiT4i1O;teVNm z+G>CP^*1o+tWuh6CLqwTFLlZK4J<4Pg6HK0|IJv|z@VVOz`+4ROiT@69z@{(@<3sp zeQ&O<`20E0dhUrRWux9hn}WTBFPdE3;+C9pG{WVh{4y>M@g>&fX1m^QQ;hg$?iH!L z%XHUWzmnT;C1mCM)7GTUV0@;%hVh!Ev9=sfw(lZiYw4}F`P_^P->dhRS-(2eq$u%j z_KKZ5SIq5GWNC@J8s+}#PNcq-g3_+d#imAk1Ox@DjPBdbeAv|3(XnRh%DJtEKws^> zQ@B+;Z4E5BA*FViXi$U}0fkWCS7y z2Vlwp@xTJ~;z6v0*-mvH+#6sU$l*P_KTWNnSmb(3!-dyZKh5TSFh_gQu|jPQ4qMy0 zCyCkzD*Kv!Fm+w6<01RZSyEDHVeXjO)aG6r^{=t>d<2;HA zUmm=;a_r}tyKg0LZe%Want a managed solution? Checkout [Sourcebot Cloud](/docs/getting-started). - -Sourcebot is open source and can be self-hosted using our official [Docker image](https://github.com/sourcebot-dev/sourcebot/pkgs/container/sourcebot). - -## Quick Start Guide - -{/*@todo: record a self-hosting quick start guide - -*/} - - - - By default, Sourcebot requires a configuration file with a list of [code host connections](/docs/connections/overview) that specify what repositories should be **synced** (cloned and indexed). To get started, run the following command to create a starter `config.json`: - - ```bash - touch config.json - echo '{ - "$schema": "https://raw.githubusercontent.com/sourcebot-dev/sourcebot/main/schemas/v3/index.json", - "connections": { - // Comments are supported - "starter-connection": { - "type": "github", - "repos": [ - "sourcebot-dev/sourcebot" - ] - } - } - }' > config.json - ``` - - This config creates a single GitHub connection named `starter-connection` that specifies [Sourcebot](https://github.com/sourcebot-dev/sourcebot) as a repo to sync. - - - - If you're deploying Sourcebot behind a domain, you must set the [AUTH_URL](/self-hosting/configuration/environment-variables) environment variable - Sourcebot is packaged as a [single Docker image](https://github.com/sourcebot-dev/sourcebot/pkgs/container/sourcebot). In the same directory as `config.json`, run the following command to start your instance: - - ``` bash - docker run \ - -p 3000:3000 \ - --pull=always \ - --rm \ - -v $(pwd):/data \ - -e CONFIG_PATH=/data/config.json \ - --name sourcebot \ - ghcr.io/sourcebot-dev/sourcebot:latest - ``` - - Navigate to `localhost:3000` to start searching the Sourcebot repo. - - - **This command**: - - pulls the latest version of the `sourcebot` docker image. - - mounts the working directory to `/data` in the container to allow Sourcebot to persist data across restarts, and to access the `config.json`. In your local directory, you should see a `.sourcebot` folder created that contains all persistent data. - - runs any pending database migrations. - - starts up all services, including the webserver exposed on port 3000. - - reads `config.json` and starts syncing. - - - Hit an issue? Please let us know on [GitHub discussions](https://github.com/sourcebot-dev/sourcebot/discussions/categories/support) or by [emailing us](mailto:team@sourcebot.dev). - - - - Sourcebot has built-in authentication which gates your instance. The first account which is registered on a fresh Sourcebot deployment is made owner. - - Registration is performed using basic credentials which are stored encrypted within your deployment. To setup more authentication providers - check out the [auth docs](/self-hosting/configuration/authentication) - - - - - - Sourcebot supports indexing public & private code on the following code hosts: - - - - Missing your code host? [Submit a feature request on GitHub](https://github.com/sourcebot-dev/sourcebot/discussions/categories/ideas). - - - -## Architecture - -Sourcebot is shipped as a single docker container that runs a collection of services using [supervisord](https://supervisord.org/): - -![architecture diagram](/images/architecture_diagram.png) - -{/*TODO: outline the different services, how Sourcebot communicates with code hosts, and the different*/} - -Sourcebot consists of the following components: -- **Web Server** : main Next.js web application serving the Sourcebot UI. -- **Backend Worker** : Node.js process that incrementally syncs with code hosts (e.g., GitHub, GitLab etc.) and asynchronously indexes configured repositories. -- **Zoekt** : the [open-source](https://github.com/sourcegraph/zoekt), trigram indexing code search engine that powers Sourcebot under the hood. -- **Postgres** : transactional database for storing business-logic data. -- **Redis Job Queue** : fast in-memory store. Used with [BullMQ](https://docs.bullmq.io/) for queuing asynchronous work. -- **`.sourcebot/` cache** : file-system cache where persistent data is written. - -You can use managed Redis / Postgres services that run outside of the Sourcebot container by providing the `REDIS_URL` and `DATABASE_URL` environment variables, respectively. See the [configuration](/self-hosting/configuration) for more configuration options. - -## Scalability - -One of our design philosophies for Sourcebot is to keep our infrastructure [radically simple](https://www.radicalsimpli.city/) while balancing scalability concerns. Depending on the number of repositories you have indexed and the instance you are running Sourcebot on, you may experience slow search times or other performance degradations. Our recommendation is to vertically scale your instance by increasing the number of CPU cores and memory. - -Sourcebot does not support horizontal scaling at this time, but it is on our roadmap. If this is something your team would be interested in, please contact us at [team@sourcebot.dev](mailto:team@sourcebot.dev). - - -## Telemetry -By default, Sourcebot collects anonymized usage data through [PostHog](https://posthog.com/) to help us improve the performance and reliability of our tool. We don't collect or transmit any information related to your codebase. In addition, all events are [sanitized](https://github.com/sourcebot-dev/sourcebot/blob/HEAD/packages/web/src/app/posthogProvider.tsx) to ensure that no sensitive details (ex. ip address, query info) leave your machine. - -The data we collect includes general usage statistics and metadata such as query performance (e.g., search duration, error rates) to monitor the application's health and functionality. This information helps us better understand how Sourcebot is used and where improvements can be made. - -If you'd like to disable all telemetry, you can do so by setting the environment variable `SOURCEBOT_TELEMETRY_DISABLED` to `true`: - -```bash -docker run \ - -e SOURCEBOT_TELEMETRY_DISABLED=true \ - /* additional args */ \ - ghcr.io/sourcebot-dev/sourcebot:latest -``` - -If you disabled telemetry correctly, you'll see the following log when starting Sourcebot: - -```sh -Disabling telemetry since SOURCEBOT_TELEMETRY_DISABLED was set. -``` \ No newline at end of file diff --git a/docs/snippets/bitbucket-app-password.mdx b/docs/snippets/bitbucket-app-password.mdx index 34ea73614..778edfa6b 100644 --- a/docs/snippets/bitbucket-app-password.mdx +++ b/docs/snippets/bitbucket-app-password.mdx @@ -1,6 +1,6 @@ - Environment variables are only supported in a [declarative config](/self-hosting/configuration/declarative-config) and cannot be used in the web UI. + Environment variables are only supported in a [declarative config](/docs/configuration/declarative-config) and cannot be used in the web UI. 1. Add the `token` and `user` (username associated with the app password you created) properties to your connection config: ```json @@ -27,7 +27,7 @@ - Secrets are only supported when [authentication](/self-hosting/configuration/authentication) is enabled. + Secrets are only supported when [authentication](/docs/configuration/authentication) is enabled. 1. Navigate to **Secrets** in settings and create a new secret with your access token: diff --git a/docs/snippets/bitbucket-token.mdx b/docs/snippets/bitbucket-token.mdx index 262aadfc6..d353eddc5 100644 --- a/docs/snippets/bitbucket-token.mdx +++ b/docs/snippets/bitbucket-token.mdx @@ -1,6 +1,6 @@ - Environment variables are only supported in a [declarative config](/self-hosting/configuration/declarative-config) and cannot be used in the web UI. + Environment variables are only supported in a [declarative config](/docs/configuration/declarative-config) and cannot be used in the web UI. 1. Add the `token` property to your connection config: ```json @@ -25,7 +25,7 @@ - Secrets are only supported when [authentication](/self-hosting/configuration/authentication) is enabled. + Secrets are only supported when [authentication](/docs/configuration/authentication) is enabled. 1. Navigate to **Secrets** in settings and create a new secret with your PAT: diff --git a/packages/web/src/app/[domain]/settings/license/page.tsx b/packages/web/src/app/[domain]/settings/license/page.tsx index f46ecb5ed..266018202 100644 --- a/packages/web/src/app/[domain]/settings/license/page.tsx +++ b/packages/web/src/app/[domain]/settings/license/page.tsx @@ -33,7 +33,7 @@ export default async function LicensePage({ params: { domain } }: LicensePagePro

No License Found

- Check out the docs for more information. + Check out the docs for more information.

diff --git a/packages/web/src/lib/newsData.ts b/packages/web/src/lib/newsData.ts index a03669a5d..bca3a678b 100644 --- a/packages/web/src/lib/newsData.ts +++ b/packages/web/src/lib/newsData.ts @@ -11,7 +11,7 @@ export const newsData: NewsItem[] = [ unique_id: "sso", header: "SSO", sub_header: "We've added support for SSO providers", - url: "https://docs.sourcebot.dev/self-hosting/configuration/authentication", + url: "https://docs.sourcebot.dev/docs/configuration/authentication", }, { unique_id: "search-contexts", From 2f5b26f7b2b4a8e8fc9e209e81bc6a87da0078aa Mon Sep 17 00:00:00 2001 From: bkellam Date: Sat, 31 May 2025 10:56:04 -0700 Subject: [PATCH 02/16] wip --- docs/docs/connections/bitbucket-cloud.mdx | 4 ++++ docs/docs/connections/bitbucket-data-center.mdx | 4 ++++ docs/docs/overview.mdx | 9 ++++++--- 3 files changed, 14 insertions(+), 3 deletions(-) diff --git a/docs/docs/connections/bitbucket-cloud.mdx b/docs/docs/connections/bitbucket-cloud.mdx index 62cc4442b..0c431ce32 100644 --- a/docs/docs/connections/bitbucket-cloud.mdx +++ b/docs/docs/connections/bitbucket-cloud.mdx @@ -8,6 +8,10 @@ import BitbucketToken from '/snippets/bitbucket-token.mdx'; import BitbucketAppPassword from '/snippets/bitbucket-app-password.mdx'; import BitbucketSchema from '/snippets/schemas/v3/bitbucket.schema.mdx' + +Looking for docs on Bitbucket Data Center? See [this doc](/docs/connections/bitbucket-data-center). + + ## Examples diff --git a/docs/docs/connections/bitbucket-data-center.mdx b/docs/docs/connections/bitbucket-data-center.mdx index f6843600e..c76ca3839 100644 --- a/docs/docs/connections/bitbucket-data-center.mdx +++ b/docs/docs/connections/bitbucket-data-center.mdx @@ -8,6 +8,10 @@ import BitbucketToken from '/snippets/bitbucket-token.mdx'; import BitbucketAppPassword from '/snippets/bitbucket-app-password.mdx'; import BitbucketSchema from '/snippets/schemas/v3/bitbucket.schema.mdx' + +Looking for docs on Bitbucket Cloud? See [this doc](/docs/connections/bitbucket-cloud). + + ## Examples diff --git a/docs/docs/overview.mdx b/docs/docs/overview.mdx index 55a23bc13..484504acd 100644 --- a/docs/docs/overview.mdx +++ b/docs/docs/overview.mdx @@ -16,6 +16,7 @@ Search across millions of lines of code in seconds using Sourcebot's blazingly f - **Regex support:** Use regular expressions to find code with precision. - **Query language:** Scope searches to specific files, repos, languages, symbol definitions and more using a rich [query language](/docs/search/syntax-reference). +- **Branch search:** Specify a list of branches to search across ([docs](/docs/search/multi-branch-indexing)). - **Fast & scalable:** Sourcebot uses [trigram indexing](https://en.wikipedia.org/wiki/Trigram_search), allowing it to scale to massive codebases. - **Syntax highlighting:** Syntax highlighting support for over [100+ languages](https://github.com/sourcebot-dev/sourcebot/blob/57724689303f351c279d37f45b6406f1d5d5d5ab/packages/web/src/lib/codemirrorLanguage.ts#L125). - **Multi-repository:** Search across all of your repositories in a single search. @@ -53,12 +54,14 @@ Search across millions of lines of code in seconds using Sourcebot's blazingly f > -### Multi code-host support +### Cross code-host support -Connect your code from multiple code-host platforms and search them all from a single interface. +Connect your code from multiple code-host platforms and search across all of them from a single interface. -- **Auto re-indexing:** Sourcebot will take care of keeping it's index upto date with the latest changes. +- **Auto re-syncing:** Code-hosts will periodically sync with Sourcebot to pull the latest changes. +- **Flexible configuration:** Sourcebot uses a expressive [JSON schema](/docs/connections/overview) config format to specify exaclty what repositories to index (and what not to index). +- **Parallel indexing:** Repositories are indexed in parallel. From 48086fb3566c6b9ab786f8a2a1607d9cf508a41d Mon Sep 17 00:00:00 2001 From: msukkari Date: Sat, 31 May 2025 14:00:33 -0700 Subject: [PATCH 03/16] initial structured logs impl --- packages/backend/src/env.ts | 2 ++ packages/backend/src/logger.ts | 66 +++++++++++++++++++++++++++------- packages/web/src/env.mjs | 4 +++ 3 files changed, 59 insertions(+), 13 deletions(-) diff --git a/packages/backend/src/env.ts b/packages/backend/src/env.ts index 512c9dae7..0c0bfb639 100644 --- a/packages/backend/src/env.ts +++ b/packages/backend/src/env.ts @@ -23,6 +23,8 @@ export const env = createEnv({ server: { SOURCEBOT_ENCRYPTION_KEY: z.string(), SOURCEBOT_LOG_LEVEL: z.enum(["info", "debug", "warn", "error"]).default("info"), + SOURCEBOT_STRUCTURED_LOGGING_ENABLED: booleanSchema.default("false"), + SOURCEBOT_STRUCTURED_LOGGING_FILE: z.string().optional(), SOURCEBOT_TELEMETRY_DISABLED: booleanSchema.default("false"), SOURCEBOT_INSTALL_ID: z.string().default("unknown"), NEXT_PUBLIC_SOURCEBOT_VERSION: z.string().default("unknown"), diff --git a/packages/backend/src/logger.ts b/packages/backend/src/logger.ts index 1701d7e67..d30e23ce0 100644 --- a/packages/backend/src/logger.ts +++ b/packages/backend/src/logger.ts @@ -2,35 +2,75 @@ import winston, { format } from 'winston'; import { Logtail } from '@logtail/node'; import { LogtailTransport } from '@logtail/winston'; import { env } from './env.js'; +import { MESSAGE } from 'triple-beam'; -const { combine, colorize, timestamp, prettyPrint, errors, printf, label: labelFn } = format; +/** + * Logger configuration with support for structured JSON logging. + * + * When SOURCEBOT_STRUCTURED_LOGGING_ENABLED=true: + * - Console output will be in JSON format suitable for Datadog ingestion + * - Logs will include structured fields: timestamp, level, message, label, stack (if error) + * + * When SOURCEBOT_STRUCTURED_LOGGING_ENABLED=false (default): + * - Console output will be human-readable with colors + * - Logs will be formatted as: "timestamp level: [label] message" + */ +const { combine, colorize, timestamp, prettyPrint, errors, printf, label: labelFn, json } = format; + +const datadogFormat = format((info) => { + info.status = info.level.toLowerCase(); + + const msg = info[MESSAGE as unknown as string] as string | undefined; + if (msg) { + info.message = msg; + delete info[MESSAGE as unknown as string]; + } + + return info; +}); + +const humanReadableFormat = printf(({ level, message, timestamp, stack, label: _label }) => { + const label = `[${_label}] `; + if (stack) { + return `${timestamp} ${level}: ${label}${message}\n${stack}`; + } + return `${timestamp} ${level}: ${label}${message}`; +}); const createLogger = (label: string) => { + const isStructuredLoggingEnabled = env.SOURCEBOT_STRUCTURED_LOGGING_ENABLED === 'true'; + return winston.createLogger({ level: env.SOURCEBOT_LOG_LEVEL, format: combine( errors({ stack: true }), timestamp(), - prettyPrint(), labelFn({ label: label, }) ), transports: [ new winston.transports.Console({ - format: combine( - errors({ stack: true }), - colorize(), - printf(({ level, message, timestamp, stack, label: _label }) => { - const label = `[${_label}] `; - if (stack) { - return `${timestamp} ${level}: ${label}${message}\n${stack}`; - } - return `${timestamp} ${level}: ${label}${message}`; - }), - ), + format: isStructuredLoggingEnabled + ? combine( + datadogFormat(), + json() + ) + : combine( + colorize(), + humanReadableFormat + ), }), + ...(env.SOURCEBOT_STRUCTURED_LOGGING_FILE && isStructuredLoggingEnabled ? [ + new winston.transports.File({ + filename: env.SOURCEBOT_STRUCTURED_LOGGING_FILE, + format: combine( + datadogFormat(), + json() + ), + }), + ] : []), ...(env.LOGTAIL_TOKEN && env.LOGTAIL_HOST ? [ new LogtailTransport( new Logtail(env.LOGTAIL_TOKEN, { diff --git a/packages/web/src/env.mjs b/packages/web/src/env.mjs index 479acaa60..f1871d5d2 100644 --- a/packages/web/src/env.mjs +++ b/packages/web/src/env.mjs @@ -56,6 +56,10 @@ export const env = createEnv({ STRIPE_WEBHOOK_SECRET: z.string().optional(), STRIPE_ENABLE_TEST_CLOCKS: booleanSchema.default('false'), + // Logging + SOURCEBOT_LOG_LEVEL: z.enum(["debug", "info", "warn", "error"]).default("info"), + SOURCEBOT_STRUCTURED_LOGGING_ENABLED: booleanSchema.default('false'), + // Misc CONFIG_MAX_REPOS_NO_TOKEN: numberSchema.default(Number.MAX_SAFE_INTEGER), NODE_ENV: z.enum(["development", "test", "production"]), From fe347878da4f1bd40240a6c84892df88687088a4 Mon Sep 17 00:00:00 2001 From: msukkari Date: Sat, 31 May 2025 14:51:49 -0700 Subject: [PATCH 04/16] structured log docs --- docs/docs.json | 3 ++- docs/docs/more/structured-logging.mdx | 39 +++++++++++++++++++++++++++ packages/backend/src/logger.ts | 6 ++--- 3 files changed, 44 insertions(+), 4 deletions(-) create mode 100644 docs/docs/more/structured-logging.mdx diff --git a/docs/docs.json b/docs/docs.json index 4c099ccfd..12a64ac01 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -69,7 +69,8 @@ "pages": [ "docs/more/api-keys", "docs/more/roles-and-permissions", - "docs/more/mcp-server" + "docs/more/mcp-server", + "docs/more/structured-logging" ] }, { diff --git a/docs/docs/more/structured-logging.mdx b/docs/docs/more/structured-logging.mdx new file mode 100644 index 000000000..ef0f6e414 --- /dev/null +++ b/docs/docs/more/structured-logging.mdx @@ -0,0 +1,39 @@ +--- +title: Structured Logging +--- + +By default, Sourcebot will output logs to the console in a human readable format. If you'd like Sourcebot to output structured JSON logs, set the following env vars: + +- `SOURCEBOT_STRUCTURED_LOGGING_ENABLED` (default: `false`): Controls whether logs are in a structured JSON format +- `SOURCEBOT_STRUCTURED_LOGGING_FILE`: If structured logging is enabled and this env var is set, structured logs will be written to this file (ex. `/data/sourcebot.log`) + +### Structured log schema +```json +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "title": "SourcebotLog", + "properties": { + "level": { + "type": "string", + "description": "The log level (error, warning, info, debug)" + }, + "service": { + "type": "string", + "description": "The Sourcebot component that generated the log" + }, + "message": { + "type": "string", + "description": "The log message" + }, + "status": { + "type": "string", + "description": "The same value as the level field added for datadog support" + }, + "timestamp": { + "type": "string", + "description": "The timestamp of the log in ISO 8061 format" + } + } +} +``` \ No newline at end of file diff --git a/packages/backend/src/logger.ts b/packages/backend/src/logger.ts index d30e23ce0..2dcf0eaa3 100644 --- a/packages/backend/src/logger.ts +++ b/packages/backend/src/logger.ts @@ -20,6 +20,8 @@ const { combine, colorize, timestamp, prettyPrint, errors, printf, label: labelF const datadogFormat = format((info) => { info.status = info.level.toLowerCase(); + info.service = info.label; + delete info.label; const msg = info[MESSAGE as unknown as string] as string | undefined; if (msg) { @@ -46,9 +48,7 @@ const createLogger = (label: string) => { format: combine( errors({ stack: true }), timestamp(), - labelFn({ - label: label, - }) + labelFn({ label: label }) ), transports: [ new winston.transports.Console({ From 77a7ed8ee9c11f45644e31df60804831344a7bf5 Mon Sep 17 00:00:00 2001 From: msukkari Date: Sat, 31 May 2025 15:19:41 -0700 Subject: [PATCH 05/16] create logger package --- packages/backend/package.json | 4 +- packages/backend/src/bitbucket.ts | 4 +- packages/backend/src/connectionManager.ts | 4 +- packages/backend/src/gerrit.ts | 8 +-- packages/backend/src/gitea.ts | 4 +- packages/backend/src/github.ts | 4 +- packages/backend/src/gitlab.ts | 4 +- packages/backend/src/index.ts | 21 +++--- packages/backend/src/instrument.ts | 5 +- packages/backend/src/main.ts | 6 +- packages/backend/src/promClient.ts | 5 +- packages/backend/src/repoCompileUtils.ts | 4 +- packages/backend/src/repoManager.ts | 65 ++++++++++--------- packages/backend/src/zoekt.ts | 2 +- packages/db/package.json | 3 +- packages/db/tools/scriptRunner.ts | 13 ++-- .../scripts/migrate-duplicate-connections.ts | 9 ++- packages/db/tools/utils.ts | 10 ++- packages/logger/.gitignore | 2 + packages/logger/package.json | 24 +++++++ packages/logger/src/env.ts | 28 ++++++++ .../src/logger.ts => logger/src/index.ts} | 4 +- packages/logger/tsconfig.json | 23 +++++++ packages/mcp/package.json | 1 + packages/mcp/src/client.ts | 5 +- packages/mcp/src/index.ts | 9 ++- packages/web/package.json | 1 + packages/web/sentry.server.config.ts | 5 +- packages/web/src/actions.ts | 31 +++++---- .../web/src/app/api/(server)/health/route.ts | 8 ++- .../web/src/app/api/(server)/stripe/route.ts | 11 ++-- .../web/src/app/api/(server)/webhook/route.ts | 21 +++--- packages/web/src/app/login/page.tsx | 7 +- packages/web/src/auth.ts | 7 +- .../web/src/ee/features/billing/actions.ts | 5 +- .../searchContexts/syncSearchContexts.ts | 7 +- .../src/features/agents/review-agent/app.ts | 9 ++- .../review-agent/nodes/fetchFileContent.ts | 8 ++- .../nodes/generateDiffReviewPrompt.ts | 7 +- .../review-agent/nodes/generatePrReview.ts | 9 ++- .../review-agent/nodes/githubPrParser.ts | 11 ++-- .../review-agent/nodes/githubPushPrReviews.ts | 11 ++-- .../review-agent/nodes/invokeDiffReviewLlm.ts | 11 ++-- .../web/src/features/entitlements/server.ts | 9 ++- packages/web/src/initialize.ts | 17 +++-- yarn.lock | 22 +++++-- 46 files changed, 332 insertions(+), 156 deletions(-) create mode 100644 packages/logger/.gitignore create mode 100644 packages/logger/package.json create mode 100644 packages/logger/src/env.ts rename packages/{backend/src/logger.ts => logger/src/index.ts} (99%) create mode 100644 packages/logger/tsconfig.json diff --git a/packages/backend/package.json b/packages/backend/package.json index a7adf99a6..2360f6c06 100644 --- a/packages/backend/package.json +++ b/packages/backend/package.json @@ -23,8 +23,6 @@ }, "dependencies": { "@gitbeaker/rest": "^40.5.1", - "@logtail/node": "^0.5.2", - "@logtail/winston": "^0.5.2", "@octokit/rest": "^21.0.2", "@sentry/cli": "^2.42.2", "@sentry/node": "^9.3.0", @@ -32,6 +30,7 @@ "@sourcebot/crypto": "workspace:*", "@sourcebot/db": "workspace:*", "@sourcebot/error": "workspace:*", + "@sourcebot/logger": "workspace:*", "@sourcebot/schemas": "workspace:*", "@t3-oss/env-core": "^0.12.0", "@types/express": "^5.0.0", @@ -51,7 +50,6 @@ "prom-client": "^15.1.3", "simple-git": "^3.27.0", "strip-json-comments": "^5.0.1", - "winston": "^3.15.0", "zod": "^3.24.3" } } diff --git a/packages/backend/src/bitbucket.ts b/packages/backend/src/bitbucket.ts index 5ffdf7e0e..e204850c0 100644 --- a/packages/backend/src/bitbucket.ts +++ b/packages/backend/src/bitbucket.ts @@ -2,7 +2,7 @@ import { createBitbucketCloudClient } from "@coderabbitai/bitbucket/cloud"; import { createBitbucketServerClient } from "@coderabbitai/bitbucket/server"; import { BitbucketConnectionConfig } from "@sourcebot/schemas/v3/bitbucket.type"; import type { ClientOptions, ClientPathsWithMethod } from "openapi-fetch"; -import { createLogger } from "./logger.js"; +import { createLogger } from "@sourcebot/logger"; import { PrismaClient } from "@sourcebot/db"; import { getTokenFromConfig, measure, fetchWithRetry } from "./utils.js"; import * as Sentry from "@sentry/node"; @@ -13,7 +13,7 @@ import { SchemaRestRepository as ServerRepository } from "@coderabbitai/bitbucke import { processPromiseResults } from "./connectionUtils.js"; import { throwIfAnyFailed } from "./connectionUtils.js"; -const logger = createLogger("Bitbucket"); +const logger = createLogger('bitbucket'); const BITBUCKET_CLOUD_GIT = 'https://bitbucket.org'; const BITBUCKET_CLOUD_API = 'https://api.bitbucket.org/2.0'; const BITBUCKET_CLOUD = "cloud"; diff --git a/packages/backend/src/connectionManager.ts b/packages/backend/src/connectionManager.ts index 132b6f66e..ac31826c0 100644 --- a/packages/backend/src/connectionManager.ts +++ b/packages/backend/src/connectionManager.ts @@ -2,7 +2,7 @@ import { Connection, ConnectionSyncStatus, PrismaClient, Prisma } from "@sourceb import { Job, Queue, Worker } from 'bullmq'; import { Settings } from "./types.js"; import { ConnectionConfig } from "@sourcebot/schemas/v3/connection.type"; -import { createLogger } from "./logger.js"; +import { createLogger } from "@sourcebot/logger"; import { Redis } from 'ioredis'; import { RepoData, compileGithubConfig, compileGitlabConfig, compileGiteaConfig, compileGerritConfig, compileBitbucketConfig, compileGenericGitHostConfig } from "./repoCompileUtils.js"; import { BackendError, BackendException } from "@sourcebot/error"; @@ -32,7 +32,7 @@ type JobResult = { export class ConnectionManager implements IConnectionManager { private worker: Worker; private queue: Queue; - private logger = createLogger('ConnectionManager'); + private logger = createLogger('connection-manager'); constructor( private db: PrismaClient, diff --git a/packages/backend/src/gerrit.ts b/packages/backend/src/gerrit.ts index 1ecb4add1..25e3cfa7b 100644 --- a/packages/backend/src/gerrit.ts +++ b/packages/backend/src/gerrit.ts @@ -1,6 +1,6 @@ import fetch from 'cross-fetch'; import { GerritConnectionConfig } from "@sourcebot/schemas/v3/index.type" -import { createLogger } from './logger.js'; +import { createLogger } from '@sourcebot/logger'; import micromatch from "micromatch"; import { measure, fetchWithRetry } from './utils.js'; import { BackendError } from '@sourcebot/error'; @@ -33,7 +33,7 @@ interface GerritWebLink { url: string; } -const logger = createLogger('Gerrit'); +const logger = createLogger('gerrit'); export const getGerritReposFromConfig = async (config: GerritConnectionConfig): Promise => { const url = config.url.endsWith('/') ? config.url : `${config.url}/`; @@ -95,7 +95,7 @@ const fetchAllProjects = async (url: string): Promise => { try { response = await fetch(endpointWithParams); if (!response.ok) { - console.log(`Failed to fetch projects from Gerrit at ${endpointWithParams} with status ${response.status}`); + logger.error(`Failed to fetch projects from Gerrit at ${endpointWithParams} with status ${response.status}`); const e = new BackendException(BackendError.CONNECTION_SYNC_FAILED_TO_FETCH_GERRIT_PROJECTS, { status: response.status, }); @@ -109,7 +109,7 @@ const fetchAllProjects = async (url: string): Promise => { } const status = (err as any).code; - console.log(`Failed to fetch projects from Gerrit at ${endpointWithParams} with status ${status}`); + logger.error(`Failed to fetch projects from Gerrit at ${endpointWithParams} with status ${status}`); throw new BackendException(BackendError.CONNECTION_SYNC_FAILED_TO_FETCH_GERRIT_PROJECTS, { status: status, }); diff --git a/packages/backend/src/gitea.ts b/packages/backend/src/gitea.ts index 3c8d134f4..aefe1c24d 100644 --- a/packages/backend/src/gitea.ts +++ b/packages/backend/src/gitea.ts @@ -2,14 +2,14 @@ import { Api, giteaApi, HttpResponse, Repository as GiteaRepository } from 'gite import { GiteaConnectionConfig } from '@sourcebot/schemas/v3/gitea.type'; import { getTokenFromConfig, measure } from './utils.js'; import fetch from 'cross-fetch'; -import { createLogger } from './logger.js'; +import { createLogger } from '@sourcebot/logger'; import micromatch from 'micromatch'; import { PrismaClient } from '@sourcebot/db'; import { processPromiseResults, throwIfAnyFailed } from './connectionUtils.js'; import * as Sentry from "@sentry/node"; import { env } from './env.js'; -const logger = createLogger('Gitea'); +const logger = createLogger('gitea'); const GITEA_CLOUD_HOSTNAME = "gitea.com"; export const getGiteaReposFromConfig = async (config: GiteaConnectionConfig, orgId: number, db: PrismaClient) => { diff --git a/packages/backend/src/github.ts b/packages/backend/src/github.ts index d976fb8bb..376ed039f 100644 --- a/packages/backend/src/github.ts +++ b/packages/backend/src/github.ts @@ -1,6 +1,6 @@ import { Octokit } from "@octokit/rest"; import { GithubConnectionConfig } from "@sourcebot/schemas/v3/github.type"; -import { createLogger } from "./logger.js"; +import { createLogger } from "@sourcebot/logger"; import { getTokenFromConfig, measure, fetchWithRetry } from "./utils.js"; import micromatch from "micromatch"; import { PrismaClient } from "@sourcebot/db"; @@ -9,7 +9,7 @@ import { processPromiseResults, throwIfAnyFailed } from "./connectionUtils.js"; import * as Sentry from "@sentry/node"; import { env } from "./env.js"; -const logger = createLogger("GitHub"); +const logger = createLogger('github'); const GITHUB_CLOUD_HOSTNAME = "github.com"; export type OctokitRepository = { diff --git a/packages/backend/src/gitlab.ts b/packages/backend/src/gitlab.ts index 2981e50ed..f08b42186 100644 --- a/packages/backend/src/gitlab.ts +++ b/packages/backend/src/gitlab.ts @@ -1,6 +1,6 @@ import { Gitlab, ProjectSchema } from "@gitbeaker/rest"; import micromatch from "micromatch"; -import { createLogger } from "./logger.js"; +import { createLogger } from "@sourcebot/logger"; import { GitlabConnectionConfig } from "@sourcebot/schemas/v3/gitlab.type" import { getTokenFromConfig, measure, fetchWithRetry } from "./utils.js"; import { PrismaClient } from "@sourcebot/db"; @@ -8,7 +8,7 @@ import { processPromiseResults, throwIfAnyFailed } from "./connectionUtils.js"; import * as Sentry from "@sentry/node"; import { env } from "./env.js"; -const logger = createLogger("GitLab"); +const logger = createLogger('gitlab'); export const GITLAB_CLOUD_HOSTNAME = "gitlab.com"; export const getGitLabReposFromConfig = async (config: GitlabConnectionConfig, orgId: number, db: PrismaClient) => { diff --git a/packages/backend/src/index.ts b/packages/backend/src/index.ts index 149d3bd4d..4411fcbc0 100644 --- a/packages/backend/src/index.ts +++ b/packages/backend/src/index.ts @@ -8,31 +8,34 @@ import { AppContext } from "./types.js"; import { main } from "./main.js" import { PrismaClient } from "@sourcebot/db"; import { env } from "./env.js"; +import { createLogger } from "@sourcebot/logger"; + +const logger = createLogger('index'); // Register handler for normal exit process.on('exit', (code) => { - console.log(`Process is exiting with code: ${code}`); + logger.info(`Process is exiting with code: ${code}`); }); // Register handlers for abnormal terminations process.on('SIGINT', () => { - console.log('Process interrupted (SIGINT)'); - process.exit(130); + logger.info('Process interrupted (SIGINT)'); + process.exit(0); }); process.on('SIGTERM', () => { - console.log('Process terminated (SIGTERM)'); - process.exit(143); + logger.info('Process terminated (SIGTERM)'); + process.exit(0); }); // Register handlers for uncaught exceptions and unhandled rejections process.on('uncaughtException', (err) => { - console.log(`Uncaught exception: ${err.message}`); + logger.error(`Uncaught exception: ${err.message}`); process.exit(1); }); process.on('unhandledRejection', (reason, promise) => { - console.log(`Unhandled rejection at: ${promise}, reason: ${reason}`); + logger.error(`Unhandled rejection at: ${promise}, reason: ${reason}`); process.exit(1); }); @@ -60,12 +63,12 @@ main(prisma, context) await prisma.$disconnect(); }) .catch(async (e) => { - console.error(e); + logger.error(e); Sentry.captureException(e); await prisma.$disconnect(); process.exit(1); }) .finally(() => { - console.log("Shutting down..."); + logger.info("Shutting down..."); }); diff --git a/packages/backend/src/instrument.ts b/packages/backend/src/instrument.ts index 754c06413..926bf2aec 100644 --- a/packages/backend/src/instrument.ts +++ b/packages/backend/src/instrument.ts @@ -1,5 +1,8 @@ import * as Sentry from "@sentry/node"; import { env } from "./env.js"; +import { createLogger } from "@sourcebot/logger"; + +const logger = createLogger('instrument'); if (!!env.NEXT_PUBLIC_SENTRY_BACKEND_DSN && !!env.NEXT_PUBLIC_SENTRY_ENVIRONMENT) { Sentry.init({ @@ -8,5 +11,5 @@ if (!!env.NEXT_PUBLIC_SENTRY_BACKEND_DSN && !!env.NEXT_PUBLIC_SENTRY_ENVIRONMENT environment: env.NEXT_PUBLIC_SENTRY_ENVIRONMENT, }); } else { - console.debug("Sentry was not initialized"); + logger.debug("Sentry was not initialized"); } diff --git a/packages/backend/src/main.ts b/packages/backend/src/main.ts index 7a22e9e57..7a1aaad63 100644 --- a/packages/backend/src/main.ts +++ b/packages/backend/src/main.ts @@ -1,5 +1,5 @@ import { PrismaClient } from '@sourcebot/db'; -import { createLogger } from "./logger.js"; +import { createLogger } from "@sourcebot/logger"; import { AppContext } from "./types.js"; import { DEFAULT_SETTINGS } from './constants.js'; import { Redis } from 'ioredis'; @@ -14,7 +14,7 @@ import { SourcebotConfig } from '@sourcebot/schemas/v3/index.type'; import { indexSchema } from '@sourcebot/schemas/v3/index.schema'; import { Ajv } from "ajv"; -const logger = createLogger('main'); +const logger = createLogger('backend-main'); const ajv = new Ajv({ validateFormats: false, }); @@ -56,7 +56,7 @@ export const main = async (db: PrismaClient, context: AppContext) => { logger.info('Connected to redis'); }).catch((err: unknown) => { logger.error('Failed to connect to redis'); - console.error(err); + logger.error(err); process.exit(1); }); diff --git a/packages/backend/src/promClient.ts b/packages/backend/src/promClient.ts index 8e8c43723..058cfe0b3 100644 --- a/packages/backend/src/promClient.ts +++ b/packages/backend/src/promClient.ts @@ -1,5 +1,8 @@ import express, { Request, Response } from 'express'; import client, { Registry, Counter, Gauge } from 'prom-client'; +import { createLogger } from "@sourcebot/logger"; + +const logger = createLogger('prometheus-client'); export class PromClient { private registry: Registry; @@ -96,7 +99,7 @@ export class PromClient { }); this.app.listen(this.PORT, () => { - console.log(`Prometheus metrics server is running on port ${this.PORT}`); + logger.info(`Prometheus metrics server is running on port ${this.PORT}`); }); } diff --git a/packages/backend/src/repoCompileUtils.ts b/packages/backend/src/repoCompileUtils.ts index b2814ebf5..71dc89a76 100644 --- a/packages/backend/src/repoCompileUtils.ts +++ b/packages/backend/src/repoCompileUtils.ts @@ -9,7 +9,7 @@ import { SchemaRepository as BitbucketCloudRepository } from "@coderabbitai/bitb import { Prisma, PrismaClient } from '@sourcebot/db'; import { WithRequired } from "./types.js" import { marshalBool } from "./utils.js"; -import { createLogger } from './logger.js'; +import { createLogger } from '@sourcebot/logger'; import { BitbucketConnectionConfig, GerritConnectionConfig, GiteaConnectionConfig, GitlabConnectionConfig, GenericGitHostConnectionConfig } from '@sourcebot/schemas/v3/connection.type'; import { RepoMetadata } from './types.js'; import path from 'path'; @@ -20,7 +20,7 @@ import GitUrlParse from 'git-url-parse'; export type RepoData = WithRequired; -const logger = createLogger('RepoCompileUtils'); +const logger = createLogger('repo-compile-utils'); export const compileGithubConfig = async ( config: GithubConnectionConfig, diff --git a/packages/backend/src/repoManager.ts b/packages/backend/src/repoManager.ts index 6f357d157..d2ae05036 100644 --- a/packages/backend/src/repoManager.ts +++ b/packages/backend/src/repoManager.ts @@ -1,6 +1,6 @@ import { Job, Queue, Worker } from 'bullmq'; import { Redis } from 'ioredis'; -import { createLogger } from "./logger.js"; +import { createLogger } from "@sourcebot/logger"; import { Connection, PrismaClient, Repo, RepoToConnection, RepoIndexingStatus, StripeSubscriptionStatus } from "@sourcebot/db"; import { GithubConnectionConfig, GitlabConnectionConfig, GiteaConnectionConfig, BitbucketConnectionConfig } from '@sourcebot/schemas/v3/connection.type'; import { AppContext, Settings, repoMetadataSchema } from "./types.js"; @@ -28,12 +28,13 @@ type RepoGarbageCollectionPayload = { repo: Repo, } +const logger = createLogger('repo-manager'); + export class RepoManager implements IRepoManager { private indexWorker: Worker; private indexQueue: Queue; private gcWorker: Worker; private gcQueue: Queue; - private logger = createLogger('RepoManager'); constructor( private db: PrismaClient, @@ -113,12 +114,12 @@ export class RepoManager implements IRepoManager { this.promClient.pendingRepoIndexingJobs.inc({ repo: repo.id.toString() }); }); - this.logger.info(`Added ${orgRepos.length} jobs to indexQueue for org ${orgId} with priority ${priority}`); + logger.info(`Added ${orgRepos.length} jobs to indexQueue for org ${orgId} with priority ${priority}`); } }).catch((err: unknown) => { - this.logger.error(`Failed to add jobs to indexQueue for repos ${repos.map(repo => repo.id).join(', ')}: ${err}`); + logger.error(`Failed to add jobs to indexQueue for repos ${repos.map(repo => repo.id).join(', ')}: ${err}`); }); } @@ -176,7 +177,7 @@ export class RepoManager implements IRepoManager { if (connection.connectionType === 'github') { const config = connection.config as unknown as GithubConnectionConfig; if (config.token) { - const token = await getTokenFromConfig(config.token, connection.orgId, db, this.logger); + const token = await getTokenFromConfig(config.token, connection.orgId, db, logger); return { password: token, } @@ -186,7 +187,7 @@ export class RepoManager implements IRepoManager { else if (connection.connectionType === 'gitlab') { const config = connection.config as unknown as GitlabConnectionConfig; if (config.token) { - const token = await getTokenFromConfig(config.token, connection.orgId, db, this.logger); + const token = await getTokenFromConfig(config.token, connection.orgId, db, logger); return { username: 'oauth2', password: token, @@ -197,7 +198,7 @@ export class RepoManager implements IRepoManager { else if (connection.connectionType === 'gitea') { const config = connection.config as unknown as GiteaConnectionConfig; if (config.token) { - const token = await getTokenFromConfig(config.token, connection.orgId, db, this.logger); + const token = await getTokenFromConfig(config.token, connection.orgId, db, logger); return { password: token, } @@ -207,7 +208,7 @@ export class RepoManager implements IRepoManager { else if (connection.connectionType === 'bitbucket') { const config = connection.config as unknown as BitbucketConnectionConfig; if (config.token) { - const token = await getTokenFromConfig(config.token, connection.orgId, db, this.logger); + const token = await getTokenFromConfig(config.token, connection.orgId, db, logger); const username = config.user ?? 'x-token-auth'; return { username, @@ -228,23 +229,23 @@ export class RepoManager implements IRepoManager { // If the repo was already in the indexing state, this job was likely killed and picked up again. As a result, // to ensure the repo state is valid, we delete the repo if it exists so we get a fresh clone if (repoAlreadyInIndexingState && existsSync(repoPath) && !isReadOnly) { - this.logger.info(`Deleting repo directory ${repoPath} during sync because it was already in the indexing state`); + logger.info(`Deleting repo directory ${repoPath} during sync because it was already in the indexing state`); await promises.rm(repoPath, { recursive: true, force: true }); } if (existsSync(repoPath) && !isReadOnly) { - this.logger.info(`Fetching ${repo.displayName}...`); + logger.info(`Fetching ${repo.displayName}...`); const { durationMs } = await measure(() => fetchRepository(repoPath, ({ method, stage, progress }) => { - this.logger.debug(`git.${method} ${stage} stage ${progress}% complete for ${repo.displayName}`) + logger.debug(`git.${method} ${stage} stage ${progress}% complete for ${repo.displayName}`) })); const fetchDuration_s = durationMs / 1000; process.stdout.write('\n'); - this.logger.info(`Fetched ${repo.displayName} in ${fetchDuration_s}s`); + logger.info(`Fetched ${repo.displayName} in ${fetchDuration_s}s`); } else if (!isReadOnly) { - this.logger.info(`Cloning ${repo.displayName}...`); + logger.info(`Cloning ${repo.displayName}...`); const auth = await this.getCloneCredentialsForRepo(repo, this.db); const cloneUrl = new URL(repo.cloneUrl); @@ -263,12 +264,12 @@ export class RepoManager implements IRepoManager { } const { durationMs } = await measure(() => cloneRepository(cloneUrl.toString(), repoPath, ({ method, stage, progress }) => { - this.logger.debug(`git.${method} ${stage} stage ${progress}% complete for ${repo.displayName}`) + logger.debug(`git.${method} ${stage} stage ${progress}% complete for ${repo.displayName}`) })); const cloneDuration_s = durationMs / 1000; process.stdout.write('\n'); - this.logger.info(`Cloned ${repo.displayName} in ${cloneDuration_s}s`); + logger.info(`Cloned ${repo.displayName} in ${cloneDuration_s}s`); } // Regardless of clone or fetch, always upsert the git config for the repo. @@ -278,14 +279,14 @@ export class RepoManager implements IRepoManager { await upsertGitConfig(repoPath, metadata.gitConfig); } - this.logger.info(`Indexing ${repo.displayName}...`); + logger.info(`Indexing ${repo.displayName}...`); const { durationMs } = await measure(() => indexGitRepository(repo, this.settings, this.ctx)); const indexDuration_s = durationMs / 1000; - this.logger.info(`Indexed ${repo.displayName} in ${indexDuration_s}s`); + logger.info(`Indexed ${repo.displayName} in ${indexDuration_s}s`); } private async runIndexJob(job: Job) { - this.logger.info(`Running index job (id: ${job.id}) for repo ${job.data.repo.displayName}`); + logger.info(`Running index job (id: ${job.id}) for repo ${job.data.repo.displayName}`); const repo = job.data.repo as RepoWithConnections; // We have to use the existing repo object to get the repoIndexingStatus because the repo object @@ -296,7 +297,7 @@ export class RepoManager implements IRepoManager { }, }); if (!existingRepo) { - this.logger.error(`Repo ${repo.id} not found`); + logger.error(`Repo ${repo.id} not found`); const e = new Error(`Repo ${repo.id} not found`); Sentry.captureException(e); throw e; @@ -328,19 +329,19 @@ export class RepoManager implements IRepoManager { attempts++; this.promClient.repoIndexingReattemptsTotal.inc(); if (attempts === maxAttempts) { - this.logger.error(`Failed to sync repository ${repo.name} (id: ${repo.id}) after ${maxAttempts} attempts. Error: ${error}`); + logger.error(`Failed to sync repository ${repo.name} (id: ${repo.id}) after ${maxAttempts} attempts. Error: ${error}`); throw error; } const sleepDuration = 5000 * Math.pow(2, attempts - 1); - this.logger.error(`Failed to sync repository ${repo.name} (id: ${repo.id}), attempt ${attempts}/${maxAttempts}. Sleeping for ${sleepDuration / 1000}s... Error: ${error}`); + logger.error(`Failed to sync repository ${repo.name} (id: ${repo.id}), attempt ${attempts}/${maxAttempts}. Sleeping for ${sleepDuration / 1000}s... Error: ${error}`); await new Promise(resolve => setTimeout(resolve, sleepDuration)); } } } private async onIndexJobCompleted(job: Job) { - this.logger.info(`Repo index job for repo ${job.data.repo.displayName} (id: ${job.data.repo.id}, jobId: ${job.id}) completed`); + logger.info(`Repo index job for repo ${job.data.repo.displayName} (id: ${job.data.repo.id}, jobId: ${job.id}) completed`); this.promClient.activeRepoIndexingJobs.dec(); this.promClient.repoIndexingSuccessTotal.inc(); @@ -356,7 +357,7 @@ export class RepoManager implements IRepoManager { } private async onIndexJobFailed(job: Job | undefined, err: unknown) { - this.logger.info(`Repo index job for repo ${job?.data.repo.displayName} (id: ${job?.data.repo.id}, jobId: ${job?.id}) failed with error: ${err}`); + logger.info(`Repo index job for repo ${job?.data.repo.displayName} (id: ${job?.data.repo.id}, jobId: ${job?.id}) failed with error: ${err}`); Sentry.captureException(err, { tags: { repoId: job?.data.repo.id, @@ -396,7 +397,7 @@ export class RepoManager implements IRepoManager { data: { repo }, }))); - this.logger.info(`Added ${repos.length} jobs to gcQueue`); + logger.info(`Added ${repos.length} jobs to gcQueue`); }); } @@ -425,7 +426,7 @@ export class RepoManager implements IRepoManager { }, }); if (reposWithNoConnections.length > 0) { - this.logger.info(`Garbage collecting ${reposWithNoConnections.length} repos with no connections: ${reposWithNoConnections.map(repo => repo.id).join(', ')}`); + logger.info(`Garbage collecting ${reposWithNoConnections.length} repos with no connections: ${reposWithNoConnections.map(repo => repo.id).join(', ')}`); } //////////////////////////////////// @@ -448,7 +449,7 @@ export class RepoManager implements IRepoManager { }); if (inactiveOrgRepos.length > 0) { - this.logger.info(`Garbage collecting ${inactiveOrgRepos.length} inactive org repos: ${inactiveOrgRepos.map(repo => repo.id).join(', ')}`); + logger.info(`Garbage collecting ${inactiveOrgRepos.length} inactive org repos: ${inactiveOrgRepos.map(repo => repo.id).join(', ')}`); } const reposToDelete = [...reposWithNoConnections, ...inactiveOrgRepos]; @@ -458,7 +459,7 @@ export class RepoManager implements IRepoManager { } private async runGarbageCollectionJob(job: Job) { - this.logger.info(`Running garbage collection job (id: ${job.id}) for repo ${job.data.repo.displayName} (id: ${job.data.repo.id})`); + logger.info(`Running garbage collection job (id: ${job.id}) for repo ${job.data.repo.displayName} (id: ${job.data.repo.id})`); this.promClient.activeRepoGarbageCollectionJobs.inc(); const repo = job.data.repo as Repo; @@ -474,7 +475,7 @@ export class RepoManager implements IRepoManager { // delete cloned repo const { path: repoPath, isReadOnly } = getRepoPath(repo, this.ctx); if (existsSync(repoPath) && !isReadOnly) { - this.logger.info(`Deleting repo directory ${repoPath}`); + logger.info(`Deleting repo directory ${repoPath}`); await promises.rm(repoPath, { recursive: true, force: true }); } @@ -483,13 +484,13 @@ export class RepoManager implements IRepoManager { const files = readdirSync(this.ctx.indexPath).filter(file => file.startsWith(shardPrefix)); for (const file of files) { const filePath = `${this.ctx.indexPath}/${file}`; - this.logger.info(`Deleting shard file ${filePath}`); + logger.info(`Deleting shard file ${filePath}`); await promises.rm(filePath, { force: true }); } } private async onGarbageCollectionJobCompleted(job: Job) { - this.logger.info(`Garbage collection job ${job.id} completed`); + logger.info(`Garbage collection job ${job.id} completed`); this.promClient.activeRepoGarbageCollectionJobs.dec(); this.promClient.repoGarbageCollectionSuccessTotal.inc(); @@ -501,7 +502,7 @@ export class RepoManager implements IRepoManager { } private async onGarbageCollectionJobFailed(job: Job | undefined, err: unknown) { - this.logger.info(`Garbage collection job failed (id: ${job?.id ?? 'unknown'}) with error: ${err}`); + logger.info(`Garbage collection job failed (id: ${job?.id ?? 'unknown'}) with error: ${err}`); Sentry.captureException(err, { tags: { repoId: job?.data.repo.id, @@ -536,7 +537,7 @@ export class RepoManager implements IRepoManager { }); if (repos.length > 0) { - this.logger.info(`Scheduling ${repos.length} repo timeouts`); + logger.info(`Scheduling ${repos.length} repo timeouts`); await this.scheduleRepoTimeoutsBulk(repos); } } diff --git a/packages/backend/src/zoekt.ts b/packages/backend/src/zoekt.ts index 3294fea81..86a191e41 100644 --- a/packages/backend/src/zoekt.ts +++ b/packages/backend/src/zoekt.ts @@ -5,7 +5,7 @@ import { getRepoPath } from "./utils.js"; import { getShardPrefix } from "./utils.js"; import { getBranches, getTags } from "./git.js"; import micromatch from "micromatch"; -import { createLogger } from "./logger.js"; +import { createLogger } from "@sourcebot/logger"; import { captureEvent } from "./posthog.js"; const logger = createLogger('zoekt'); diff --git a/packages/db/package.json b/packages/db/package.json index 7af80bb4c..2fa2e5304 100644 --- a/packages/db/package.json +++ b/packages/db/package.json @@ -26,6 +26,7 @@ "dependencies": { "@prisma/client": "6.2.1", "@types/readline-sync": "^1.4.8", - "readline-sync": "^1.4.10" + "readline-sync": "^1.4.10", + "@sourcebot/logger": "workspace:*" } } diff --git a/packages/db/tools/scriptRunner.ts b/packages/db/tools/scriptRunner.ts index 0b86058d1..ef9bfdd35 100644 --- a/packages/db/tools/scriptRunner.ts +++ b/packages/db/tools/scriptRunner.ts @@ -2,6 +2,7 @@ import { PrismaClient } from "@sourcebot/db"; import { ArgumentParser } from "argparse"; import { migrateDuplicateConnections } from "./scripts/migrate-duplicate-connections"; import { confirmAction } from "./utils"; +import { createLogger } from "@sourcebot/logger"; export interface Script { run: (prisma: PrismaClient) => Promise; @@ -16,17 +17,19 @@ parser.add_argument("--url", { required: true, help: "Database URL" }); parser.add_argument("--script", { required: true, help: "Script to run" }); const args = parser.parse_args(); +const logger = createLogger('db-script-runner'); + (async () => { if (!(args.script in scripts)) { - console.log("Invalid script"); + logger.error("Invalid script"); process.exit(1); } const selectedScript = scripts[args.script]; - console.log("\nTo confirm:"); - console.log(`- Database URL: ${args.url}`); - console.log(`- Script: ${args.script}`); + logger.info("\nTo confirm:"); + logger.info(`- Database URL: ${args.url}`); + logger.info(`- Script: ${args.script}`); confirmAction(); @@ -36,7 +39,7 @@ const args = parser.parse_args(); await selectedScript.run(prisma); - console.log("\nDone."); + logger.info("\nDone."); process.exit(0); })(); diff --git a/packages/db/tools/scripts/migrate-duplicate-connections.ts b/packages/db/tools/scripts/migrate-duplicate-connections.ts index 7093e429e..fe3fa949c 100644 --- a/packages/db/tools/scripts/migrate-duplicate-connections.ts +++ b/packages/db/tools/scripts/migrate-duplicate-connections.ts @@ -1,6 +1,9 @@ import { Script } from "../scriptRunner"; import { PrismaClient } from "../../dist"; import { confirmAction } from "../utils"; +import { createLogger } from "@sourcebot/logger"; + +const logger = createLogger('migrate-duplicate-connections'); // Handles duplicate connections by renaming them to be unique. // @see: 20250320215449_unique_connection_name_constraint_within_org @@ -15,7 +18,7 @@ export const migrateDuplicateConnections: Script = { }, })).filter(({ _count }) => _count._all > 1); - console.log(`Found ${duplicates.reduce((acc, { _count }) => acc + _count._all, 0)} duplicate connections.`); + logger.info(`Found ${duplicates.reduce((acc, { _count }) => acc + _count._all, 0)} duplicate connections.`); confirmAction(); @@ -37,7 +40,7 @@ export const migrateDuplicateConnections: Script = { const connection = connections[i]; const newName = `${name}-${i + 1}`; - console.log(`Migrating connection with id ${connection.id} from name=${name} to name=${newName}`); + logger.info(`Migrating connection with id ${connection.id} from name=${name} to name=${newName}`); await prisma.connection.update({ where: { id: connection.id }, @@ -47,6 +50,6 @@ export const migrateDuplicateConnections: Script = { } } - console.log(`Migrated ${migrated} connections.`); + logger.info(`Migrated ${migrated} connections.`); }, }; diff --git a/packages/db/tools/utils.ts b/packages/db/tools/utils.ts index dbd08b007..a096ac9fb 100644 --- a/packages/db/tools/utils.ts +++ b/packages/db/tools/utils.ts @@ -1,9 +1,17 @@ import readline from 'readline-sync'; +import { createLogger } from "@sourcebot/logger"; + +const logger = createLogger('db-utils'); export const confirmAction = (message: string = "Are you sure you want to proceed? [N/y]") => { const response = readline.question(message).toLowerCase(); if (response !== 'y') { - console.log("Aborted."); + logger.info("Aborted."); process.exit(0); } } + +export const abort = () => { + logger.info("Aborted."); + process.exit(0); +}; diff --git a/packages/logger/.gitignore b/packages/logger/.gitignore new file mode 100644 index 000000000..96351007d --- /dev/null +++ b/packages/logger/.gitignore @@ -0,0 +1,2 @@ +dist/ +*.tsbuildinfo \ No newline at end of file diff --git a/packages/logger/package.json b/packages/logger/package.json new file mode 100644 index 000000000..2e2279a35 --- /dev/null +++ b/packages/logger/package.json @@ -0,0 +1,24 @@ +{ + "name": "@sourcebot/logger", + "version": "0.1.0", + "main": "dist/index.js", + "type": "module", + "private": true, + "scripts": { + "build": "tsc", + "postinstall": "yarn build" + }, + "dependencies": { + "@logtail/node": "^0.5.2", + "@logtail/winston": "^0.5.2", + "@t3-oss/env-core": "^0.12.0", + "dotenv": "^16.4.5", + "triple-beam": "^1.4.1", + "winston": "^3.15.0", + "zod": "^3.24.3" + }, + "devDependencies": { + "@types/node": "^22.7.5", + "typescript": "^5.7.3" + } +} diff --git a/packages/logger/src/env.ts b/packages/logger/src/env.ts new file mode 100644 index 000000000..5f582e0d2 --- /dev/null +++ b/packages/logger/src/env.ts @@ -0,0 +1,28 @@ +import { createEnv } from "@t3-oss/env-core"; +import { z } from "zod"; +import dotenv from 'dotenv'; + +// Booleans are specified as 'true' or 'false' strings. +const booleanSchema = z.enum(["true", "false"]); + +dotenv.config({ + path: './.env', +}); + +dotenv.config({ + path: './.env.local', + override: true +}); + +export const env = createEnv({ + server: { + SOURCEBOT_LOG_LEVEL: z.enum(["info", "debug", "warn", "error"]).default("info"), + SOURCEBOT_STRUCTURED_LOGGING_ENABLED: booleanSchema.default("false"), + SOURCEBOT_STRUCTURED_LOGGING_FILE: z.string().optional(), + LOGTAIL_TOKEN: z.string().optional(), + LOGTAIL_HOST: z.string().url().optional(), + }, + runtimeEnv: process.env, + emptyStringAsUndefined: true, + skipValidation: process.env.SKIP_ENV_VALIDATION === "1", +}); \ No newline at end of file diff --git a/packages/backend/src/logger.ts b/packages/logger/src/index.ts similarity index 99% rename from packages/backend/src/logger.ts rename to packages/logger/src/index.ts index 2dcf0eaa3..218f51924 100644 --- a/packages/backend/src/logger.ts +++ b/packages/logger/src/index.ts @@ -1,8 +1,8 @@ import winston, { format } from 'winston'; import { Logtail } from '@logtail/node'; import { LogtailTransport } from '@logtail/winston'; -import { env } from './env.js'; import { MESSAGE } from 'triple-beam'; +import { env } from './env.js'; /** * Logger configuration with support for structured JSON logging. @@ -84,4 +84,4 @@ const createLogger = (label: string) => { export { createLogger -}; \ No newline at end of file +}; \ No newline at end of file diff --git a/packages/logger/tsconfig.json b/packages/logger/tsconfig.json new file mode 100644 index 000000000..88ae91dd7 --- /dev/null +++ b/packages/logger/tsconfig.json @@ -0,0 +1,23 @@ +{ + "compilerOptions": { + "target": "ES2022", + "module": "Node16", + "moduleResolution": "Node16", + "lib": ["ES2023"], + "outDir": "dist", + "rootDir": "src", + "declaration": true, + "declarationMap": true, + "sourceMap": true, + "strict": true, + "noImplicitAny": true, + "strictNullChecks": true, + "esModuleInterop": true, + "forceConsistentCasingInFileNames": true, + "skipLibCheck": true, + "isolatedModules": true, + "resolveJsonModule": true + }, + "include": ["src/**/*"], + "exclude": ["node_modules", "dist"] +} \ No newline at end of file diff --git a/packages/mcp/package.json b/packages/mcp/package.json index fcd9fa201..a77dd1b16 100644 --- a/packages/mcp/package.json +++ b/packages/mcp/package.json @@ -18,6 +18,7 @@ }, "dependencies": { "@modelcontextprotocol/sdk": "^1.10.2", + "@sourcebot/logger": "workspace:*", "@t3-oss/env-core": "^0.13.4", "escape-string-regexp": "^5.0.0", "express": "^5.1.0", diff --git a/packages/mcp/src/client.ts b/packages/mcp/src/client.ts index ffde2d7bf..df5d9973c 100644 --- a/packages/mcp/src/client.ts +++ b/packages/mcp/src/client.ts @@ -2,9 +2,12 @@ import { env } from './env.js'; import { listRepositoriesResponseSchema, searchResponseSchema, fileSourceResponseSchema } from './schemas.js'; import { FileSourceRequest, FileSourceResponse, ListRepositoriesResponse, SearchRequest, SearchResponse, ServiceError } from './types.js'; import { isServiceError } from './utils.js'; +import { createLogger } from "@sourcebot/logger"; + +const logger = createLogger('mcp-client'); export const search = async (request: SearchRequest): Promise => { - console.error(`Executing search request: ${JSON.stringify(request, null, 2)}`); + logger.debug(`Executing search request: ${JSON.stringify(request, null, 2)}`); const result = await fetch(`${env.SOURCEBOT_HOST}/api/search`, { method: 'POST', headers: { diff --git a/packages/mcp/src/index.ts b/packages/mcp/src/index.ts index 00933bd51..fda34e4de 100644 --- a/packages/mcp/src/index.ts +++ b/packages/mcp/src/index.ts @@ -9,6 +9,9 @@ import { listRepos, search, getFileSource } from './client.js'; import { env, numberSchema } from './env.js'; import { TextContent } from './types.js'; import { base64Decode, isServiceError } from './utils.js'; +import { createLogger } from "@sourcebot/logger"; + +const logger = createLogger('mcp-server'); // Create MCP server const server = new McpServer({ @@ -75,7 +78,7 @@ server.tool( query += ` case:no`; } - console.error(`Executing search request: ${query}`); + logger.debug(`Executing search request: ${query}`); const response = await search({ query, @@ -215,10 +218,10 @@ server.tool( const runServer = async () => { const transport = new StdioServerTransport(); await server.connect(transport); - console.error('Sourcebot MCP server ready'); + logger.info('Sourcebot MCP server ready'); } runServer().catch((error) => { - console.error('Failed to start MCP server:', error); + logger.error('Failed to start MCP server:', error); process.exit(1); }); diff --git a/packages/web/package.json b/packages/web/package.json index 15e17fe4d..986e18dca 100644 --- a/packages/web/package.json +++ b/packages/web/package.json @@ -74,6 +74,7 @@ "@sourcebot/crypto": "workspace:*", "@sourcebot/db": "workspace:*", "@sourcebot/error": "workspace:*", + "@sourcebot/logger": "workspace:*", "@sourcebot/schemas": "workspace:*", "@ssddanbrown/codemirror-lang-twig": "^1.0.0", "@stripe/react-stripe-js": "^3.1.1", diff --git a/packages/web/sentry.server.config.ts b/packages/web/sentry.server.config.ts index a2049565e..548160c8c 100644 --- a/packages/web/sentry.server.config.ts +++ b/packages/web/sentry.server.config.ts @@ -3,6 +3,9 @@ // https://docs.sentry.io/platforms/javascript/guides/nextjs/ import * as Sentry from "@sentry/nextjs"; +import { createLogger } from "@sourcebot/logger"; + +const logger = createLogger('sentry-server-config'); if (!!process.env.NEXT_PUBLIC_SENTRY_WEBAPP_DSN && !!process.env.NEXT_PUBLIC_SENTRY_ENVIRONMENT) { Sentry.init({ @@ -13,5 +16,5 @@ if (!!process.env.NEXT_PUBLIC_SENTRY_WEBAPP_DSN && !!process.env.NEXT_PUBLIC_SEN debug: false, }); } else { - console.debug("[server] Sentry was not initialized"); + logger.debug("[server] Sentry was not initialized"); } diff --git a/packages/web/src/actions.ts b/packages/web/src/actions.ts index 3c4c37c62..9cd3ff55a 100644 --- a/packages/web/src/actions.ts +++ b/packages/web/src/actions.ts @@ -33,11 +33,14 @@ import { hasEntitlement } from "./features/entitlements/server"; import { getPublicAccessStatus } from "./ee/features/publicAccess/publicAccess"; import JoinRequestSubmittedEmail from "./emails/joinRequestSubmittedEmail"; import JoinRequestApprovedEmail from "./emails/joinRequestApprovedEmail"; +import { createLogger } from "@sourcebot/logger"; const ajv = new Ajv({ validateFormats: false, }); +const logger = createLogger('web-actions'); + /** * "Service Error Wrapper". * @@ -49,7 +52,7 @@ export const sew = async (fn: () => Promise): Promise => return await fn(); } catch (e) { Sentry.captureException(e); - console.error(e); + logger.error(e); return unexpectedError(`An unexpected error occurred. Please try again later.`); } } @@ -64,7 +67,7 @@ export const withAuth = async (fn: (userId: string) => Promise, allowSingl if (apiKey) { const apiKeyOrError = await verifyApiKey(apiKey); if (isServiceError(apiKeyOrError)) { - console.error(`Invalid API key: ${JSON.stringify(apiKey)}. Error: ${JSON.stringify(apiKeyOrError)}`); + logger.error(`Invalid API key: ${JSON.stringify(apiKey)}. Error: ${JSON.stringify(apiKeyOrError)}`); return notAuthenticated(); } @@ -75,7 +78,7 @@ export const withAuth = async (fn: (userId: string) => Promise, allowSingl }); if (!user) { - console.error(`No user found for API key: ${apiKey}`); + logger.error(`No user found for API key: ${apiKey}`); return notAuthenticated(); } @@ -97,7 +100,7 @@ export const withAuth = async (fn: (userId: string) => Promise, allowSingl ) { if (!hasEntitlement("public-access")) { const plan = getPlan(); - console.error(`Public access isn't supported in your current plan: ${plan}. If you have a valid enterprise license key, pass it via SOURCEBOT_EE_LICENSE_KEY. For support, contact ${SOURCEBOT_SUPPORT_EMAIL}.`); + logger.error(`Public access isn't supported in your current plan: ${plan}. If you have a valid enterprise license key, pass it via SOURCEBOT_EE_LICENSE_KEY. For support, contact ${SOURCEBOT_SUPPORT_EMAIL}.`); return notAuthenticated(); } @@ -1011,11 +1014,11 @@ export const createInvites = async (emails: string[], domain: string): Promise<{ const failed = result.rejected.concat(result.pending).filter(Boolean); if (failed.length > 0) { - console.error(`Failed to send invite email to ${email}: ${failed}`); + logger.error(`Failed to send invite email to ${email}: ${failed}`); } })); } else { - console.warn(`SMTP_CONNECTION_URL or EMAIL_FROM_ADDRESS not set. Skipping invite email to ${emails.join(", ")}`); + logger.warn(`SMTP_CONNECTION_URL or EMAIL_FROM_ADDRESS not set. Skipping invite email to ${emails.join(", ")}`); } return { @@ -1457,7 +1460,7 @@ export const createAccountRequest = async (userId: string, domain: string) => se } if (user.pendingApproval == false) { - console.warn(`User ${userId} isn't pending approval. Skipping account request creation.`); + logger.warn(`User ${userId} isn't pending approval. Skipping account request creation.`); return { success: true, existingRequest: false, @@ -1484,7 +1487,7 @@ export const createAccountRequest = async (userId: string, domain: string) => se }); if (existingRequest) { - console.warn(`User ${userId} already has an account request for org ${org.id}. Skipping account request creation.`); + logger.warn(`User ${userId} already has an account request for org ${org.id}. Skipping account request creation.`); return { success: true, existingRequest: true, @@ -1516,7 +1519,7 @@ export const createAccountRequest = async (userId: string, domain: string) => se }); if (!owner) { - console.error(`Failed to find owner for org ${org.id} when drafting email for account request from ${userId}`); + logger.error(`Failed to find owner for org ${org.id} when drafting email for account request from ${userId}`); } else { const html = await render(JoinRequestSubmittedEmail({ baseUrl: deploymentUrl, @@ -1541,11 +1544,11 @@ export const createAccountRequest = async (userId: string, domain: string) => se const failed = result.rejected.concat(result.pending).filter(Boolean); if (failed.length > 0) { - console.error(`Failed to send account request email to ${owner.email}: ${failed}`); + logger.error(`Failed to send account request email to ${owner.email}: ${failed}`); } } } else { - console.warn(`SMTP_CONNECTION_URL or EMAIL_FROM_ADDRESS not set. Skipping account request email to owner`); + logger.warn(`SMTP_CONNECTION_URL or EMAIL_FROM_ADDRESS not set. Skipping account request email to owner`); } } @@ -1612,7 +1615,7 @@ export const approveAccountRequest = async (requestId: string, domain: string) = }) for (const invite of invites) { - console.log(`Account request approved. Deleting invite ${invite.id} for ${request.requestedBy.email}`); + logger.info(`Account request approved. Deleting invite ${invite.id} for ${request.requestedBy.email}`); await tx.invite.delete({ where: { id: invite.id, @@ -1651,10 +1654,10 @@ export const approveAccountRequest = async (requestId: string, domain: string) = const failed = result.rejected.concat(result.pending).filter(Boolean); if (failed.length > 0) { - console.error(`Failed to send approval email to ${request.requestedBy.email}: ${failed}`); + logger.error(`Failed to send approval email to ${request.requestedBy.email}: ${failed}`); } } else { - console.warn(`SMTP_CONNECTION_URL or EMAIL_FROM_ADDRESS not set. Skipping approval email to ${request.requestedBy.email}`); + logger.warn(`SMTP_CONNECTION_URL or EMAIL_FROM_ADDRESS not set. Skipping approval email to ${request.requestedBy.email}`); } return { diff --git a/packages/web/src/app/api/(server)/health/route.ts b/packages/web/src/app/api/(server)/health/route.ts index 9dd2a1a00..ac1a2ba1d 100644 --- a/packages/web/src/app/api/(server)/health/route.ts +++ b/packages/web/src/app/api/(server)/health/route.ts @@ -1,7 +1,11 @@ 'use server'; -export const GET = async () => { - console.log('health check'); +import { createLogger } from "@sourcebot/logger"; + +const logger = createLogger('health-check'); + +export async function GET() { + logger.info('health check'); return Response.json({ status: 'ok' }); } diff --git a/packages/web/src/app/api/(server)/stripe/route.ts b/packages/web/src/app/api/(server)/stripe/route.ts index 8a466b7a7..b755fcfd4 100644 --- a/packages/web/src/app/api/(server)/stripe/route.ts +++ b/packages/web/src/app/api/(server)/stripe/route.ts @@ -5,6 +5,9 @@ import { prisma } from '@/prisma'; import { ConnectionSyncStatus, StripeSubscriptionStatus } from '@sourcebot/db'; import { stripeClient } from '@/ee/features/billing/stripe'; import { env } from '@/env.mjs'; +import { createLogger } from "@sourcebot/logger"; + +const logger = createLogger('stripe-webhook'); export async function POST(req: NextRequest) { const body = await req.text(); @@ -52,7 +55,7 @@ export async function POST(req: NextRequest) { stripeLastUpdatedAt: new Date() } }); - console.log(`Org ${org.id} subscription status updated to INACTIVE`); + logger.info(`Org ${org.id} subscription status updated to INACTIVE`); return new Response(JSON.stringify({ received: true }), { status: 200 @@ -80,7 +83,7 @@ export async function POST(req: NextRequest) { stripeLastUpdatedAt: new Date() } }); - console.log(`Org ${org.id} subscription status updated to ACTIVE`); + logger.info(`Org ${org.id} subscription status updated to ACTIVE`); // mark all of this org's connections for sync, since their repos may have been previously garbage collected await prisma.connection.updateMany({ @@ -96,14 +99,14 @@ export async function POST(req: NextRequest) { status: 200 }); } else { - console.log(`Received unknown event type: ${event.type}`); + logger.info(`Received unknown event type: ${event.type}`); return new Response(JSON.stringify({ received: true }), { status: 202 }); } } catch (err) { - console.error('Error processing webhook:', err); + logger.error('Error processing webhook:', err); return new Response( 'Webhook error: ' + (err as Error).message, { status: 400 } diff --git a/packages/web/src/app/api/(server)/webhook/route.ts b/packages/web/src/app/api/(server)/webhook/route.ts index 4f980d077..ee9d4dcc1 100644 --- a/packages/web/src/app/api/(server)/webhook/route.ts +++ b/packages/web/src/app/api/(server)/webhook/route.ts @@ -9,6 +9,9 @@ import { processGitHubPullRequest } from "@/features/agents/review-agent/app"; import { throttling } from "@octokit/plugin-throttling"; import fs from "fs"; import { GitHubPullRequest } from "@/features/agents/review-agent/types"; +import { createLogger } from "@sourcebot/logger"; + +const logger = createLogger('github-webhook'); let githubApp: App | undefined; if (env.GITHUB_APP_ID && env.GITHUB_APP_WEBHOOK_SECRET && env.GITHUB_APP_PRIVATE_KEY_PATH) { @@ -26,7 +29,7 @@ if (env.GITHUB_APP_ID && env.GITHUB_APP_WEBHOOK_SECRET && env.GITHUB_APP_PRIVATE throttle: { onRateLimit: (retryAfter: number, options: Required, octokit: Octokit, retryCount: number) => { if (retryCount > 3) { - console.log(`Rate limit exceeded: ${retryAfter} seconds`); + logger.warn(`Rate limit exceeded: ${retryAfter} seconds`); return false; } @@ -35,7 +38,7 @@ if (env.GITHUB_APP_ID && env.GITHUB_APP_WEBHOOK_SECRET && env.GITHUB_APP_PRIVATE } }); } catch (error) { - console.error(`Error initializing GitHub app: ${error}`); + logger.error(`Error initializing GitHub app: ${error}`); } } @@ -53,21 +56,21 @@ export const POST = async (request: NextRequest) => { const githubEvent = headers['x-github-event'] || headers['X-GitHub-Event']; if (githubEvent) { - console.log('GitHub event received:', githubEvent); + logger.info('GitHub event received:', githubEvent); if (!githubApp) { - console.warn('Received GitHub webhook event but GitHub app env vars are not set'); + logger.warn('Received GitHub webhook event but GitHub app env vars are not set'); return Response.json({ status: 'ok' }); } if (isPullRequestEvent(githubEvent, body)) { if (env.REVIEW_AGENT_AUTO_REVIEW_ENABLED === "false") { - console.log('Review agent auto review (REVIEW_AGENT_AUTO_REVIEW_ENABLED) is disabled, skipping'); + logger.info('Review agent auto review (REVIEW_AGENT_AUTO_REVIEW_ENABLED) is disabled, skipping'); return Response.json({ status: 'ok' }); } if (!body.installation) { - console.error('Received github pull request event but installation is not present'); + logger.error('Received github pull request event but installation is not present'); return Response.json({ status: 'ok' }); } @@ -81,15 +84,15 @@ export const POST = async (request: NextRequest) => { if (isIssueCommentEvent(githubEvent, body)) { const comment = body.comment.body; if (!comment) { - console.warn('Received issue comment event but comment body is empty'); + logger.warn('Received issue comment event but comment body is empty'); return Response.json({ status: 'ok' }); } if (comment === `/${env.REVIEW_AGENT_REVIEW_COMMAND}`) { - console.log('Review agent review command received, processing'); + logger.info('Review agent review command received, processing'); if (!body.installation) { - console.error('Received github issue comment event but installation is not present'); + logger.error('Received github issue comment event but installation is not present'); return Response.json({ status: 'ok' }); } diff --git a/packages/web/src/app/login/page.tsx b/packages/web/src/app/login/page.tsx index 1487d1822..4fc6f9c48 100644 --- a/packages/web/src/app/login/page.tsx +++ b/packages/web/src/app/login/page.tsx @@ -3,6 +3,9 @@ import { LoginForm } from "./components/loginForm"; import { redirect } from "next/navigation"; import { getProviders } from "@/auth"; import { Footer } from "@/app/components/footer"; +import { createLogger } from "@sourcebot/logger"; + +const logger = createLogger('login-page'); interface LoginProps { searchParams: { @@ -12,10 +15,10 @@ interface LoginProps { } export default async function Login({ searchParams }: LoginProps) { - console.log("Login page loaded"); + logger.info("Login page loaded"); const session = await auth(); if (session) { - console.log("Session found in login page, redirecting to home"); + logger.info("Session found in login page, redirecting to home"); return redirect("/"); } diff --git a/packages/web/src/auth.ts b/packages/web/src/auth.ts index 3aefd694b..f4e0aa82a 100644 --- a/packages/web/src/auth.ts +++ b/packages/web/src/auth.ts @@ -19,6 +19,7 @@ import { getSSOProviders, handleJITProvisioning } from '@/ee/sso/sso'; import { hasEntitlement } from '@/features/entitlements/server'; import { isServiceError } from './lib/utils'; import { ServiceErrorException } from './lib/serviceError'; +import { createLogger } from "@sourcebot/logger"; export const runtime = 'nodejs'; @@ -36,6 +37,8 @@ declare module 'next-auth/jwt' { } } +const logger = createLogger('web-auth'); + export const getProviders = () => { const providers: Provider[] = []; @@ -202,13 +205,13 @@ const onCreateUser = async ({ user }: { user: AuthJsUser }) => { if (env.AUTH_EE_ENABLE_JIT_PROVISIONING === 'true' && hasEntitlement("sso")) { const res = await handleJITProvisioning(user.id!, SINGLE_TENANT_ORG_DOMAIN); if (isServiceError(res)) { - console.error(`Failed to provision user ${user.id} for org ${SINGLE_TENANT_ORG_DOMAIN}: ${res.message}`); + logger.error(`Failed to provision user ${user.id} for org ${SINGLE_TENANT_ORG_DOMAIN}: ${res.message}`); throw new ServiceErrorException(res); } } else { const res = await createAccountRequest(user.id!, SINGLE_TENANT_ORG_DOMAIN); if (isServiceError(res)) { - console.error(`Failed to provision user ${user.id} for org ${SINGLE_TENANT_ORG_DOMAIN}: ${res.message}`); + logger.error(`Failed to provision user ${user.id} for org ${SINGLE_TENANT_ORG_DOMAIN}: ${res.message}`); throw new ServiceErrorException(res); } } diff --git a/packages/web/src/ee/features/billing/actions.ts b/packages/web/src/ee/features/billing/actions.ts index a5e8f1862..d7085049b 100644 --- a/packages/web/src/ee/features/billing/actions.ts +++ b/packages/web/src/ee/features/billing/actions.ts @@ -12,6 +12,9 @@ import { StatusCodes } from "http-status-codes"; import { ErrorCode } from "@/lib/errorCodes"; import { headers } from "next/headers"; import { getSubscriptionForOrg } from "./serverUtils"; +import { createLogger } from "@sourcebot/logger"; + +const logger = createLogger('billing-actions'); export const createOnboardingSubscription = async (domain: string) => sew(() => withAuth(async (userId) => @@ -98,7 +101,7 @@ export const createOnboardingSubscription = async (domain: string) => sew(() => subscriptionId: subscription.id, } } catch (e) { - console.error(e); + logger.error(e); return { statusCode: StatusCodes.INTERNAL_SERVER_ERROR, errorCode: ErrorCode.STRIPE_CHECKOUT_ERROR, diff --git a/packages/web/src/ee/features/searchContexts/syncSearchContexts.ts b/packages/web/src/ee/features/searchContexts/syncSearchContexts.ts index e662896d0..2c3b8594f 100644 --- a/packages/web/src/ee/features/searchContexts/syncSearchContexts.ts +++ b/packages/web/src/ee/features/searchContexts/syncSearchContexts.ts @@ -4,6 +4,9 @@ import { SINGLE_TENANT_ORG_ID, SOURCEBOT_SUPPORT_EMAIL } from "@/lib/constants"; import { prisma } from "@/prisma"; import { SearchContext } from "@sourcebot/schemas/v3/index.type"; import micromatch from "micromatch"; +import { createLogger } from "@sourcebot/logger"; + +const logger = createLogger('sync-search-contexts'); export const syncSearchContexts = async (contexts?: { [key: string]: SearchContext }) => { if (env.SOURCEBOT_TENANCY_MODE !== 'single') { @@ -13,7 +16,7 @@ export const syncSearchContexts = async (contexts?: { [key: string]: SearchConte if (!hasEntitlement("search-contexts")) { if (contexts) { const plan = getPlan(); - console.error(`Search contexts are not supported in your current plan: ${plan}. If you have a valid enterprise license key, pass it via SOURCEBOT_EE_LICENSE_KEY. For support, contact ${SOURCEBOT_SUPPORT_EMAIL}.`); + logger.error(`Search contexts are not supported in your current plan: ${plan}. If you have a valid enterprise license key, pass it via SOURCEBOT_EE_LICENSE_KEY. For support, contact ${SOURCEBOT_SUPPORT_EMAIL}.`); } return; } @@ -101,7 +104,7 @@ export const syncSearchContexts = async (contexts?: { [key: string]: SearchConte }); for (const context of deletedContexts) { - console.log(`Deleting search context with name '${context.name}'. ID: ${context.id}`); + logger.info(`Deleting search context with name '${context.name}'. ID: ${context.id}`); await prisma.searchContext.delete({ where: { id: context.id, diff --git a/packages/web/src/features/agents/review-agent/app.ts b/packages/web/src/features/agents/review-agent/app.ts index f62d77a99..1ea0cba98 100644 --- a/packages/web/src/features/agents/review-agent/app.ts +++ b/packages/web/src/features/agents/review-agent/app.ts @@ -6,6 +6,7 @@ import { env } from "@/env.mjs"; import { GitHubPullRequest } from "@/features/agents/review-agent/types"; import path from "path"; import fs from "fs"; +import { createLogger } from "@sourcebot/logger"; const rules = [ "Do NOT provide general feedback, summaries, explanations of changes, or praises for making good additions.", @@ -17,11 +18,13 @@ const rules = [ "If there are no issues found on a line range, do NOT respond with any comments. This includes comments such as \"No issues found\" or \"LGTM\"." ] +const logger = createLogger('review-agent'); + export async function processGitHubPullRequest(octokit: Octokit, pullRequest: GitHubPullRequest) { - console.log(`Received a pull request event for #${pullRequest.number}`); + logger.info(`Received a pull request event for #${pullRequest.number}`); if (!env.OPENAI_API_KEY) { - console.error("OPENAI_API_KEY is not set, skipping review agent"); + logger.error("OPENAI_API_KEY is not set, skipping review agent"); return; } @@ -42,7 +45,7 @@ export async function processGitHubPullRequest(octokit: Octokit, pullRequest: Gi hour12: false }).replace(/(\d+)\/(\d+)\/(\d+), (\d+):(\d+):(\d+)/, '$3_$1_$2_$4_$5_$6'); reviewAgentLogPath = path.join(reviewAgentLogDir, `review-agent-${pullRequest.number}-${timestamp}.log`); - console.log(`Review agent logging to ${reviewAgentLogPath}`); + logger.info(`Review agent logging to ${reviewAgentLogPath}`); } const prPayload = await githubPrParser(octokit, pullRequest); diff --git a/packages/web/src/features/agents/review-agent/nodes/fetchFileContent.ts b/packages/web/src/features/agents/review-agent/nodes/fetchFileContent.ts index cf6f6e03f..cb048ba5e 100644 --- a/packages/web/src/features/agents/review-agent/nodes/fetchFileContent.ts +++ b/packages/web/src/features/agents/review-agent/nodes/fetchFileContent.ts @@ -4,17 +4,19 @@ import { fileSourceResponseSchema } from "@/features/search/schemas"; import { base64Decode } from "@/lib/utils"; import { isServiceError } from "@/lib/utils"; import { env } from "@/env.mjs"; +import { createLogger } from "@sourcebot/logger"; +const logger = createLogger('fetch-file-content'); export const fetchFileContent = async (pr_payload: sourcebot_pr_payload, filename: string): Promise => { - console.log("Executing fetch_file_content"); + logger.debug("Executing fetch_file_content"); const repoPath = pr_payload.hostDomain + "/" + pr_payload.owner + "/" + pr_payload.repo; const fileSourceRequest = { fileName: filename, repository: repoPath, } - console.log(JSON.stringify(fileSourceRequest, null, 2)); + logger.debug(JSON.stringify(fileSourceRequest, null, 2)); const response = await getFileSource(fileSourceRequest, "~", env.REVIEW_AGENT_API_KEY); if (isServiceError(response)) { @@ -30,6 +32,6 @@ export const fetchFileContent = async (pr_payload: sourcebot_pr_payload, filenam context: fileContent, } - console.log("Completed fetch_file_content"); + logger.debug("Completed fetch_file_content"); return fileContentContext; } \ No newline at end of file diff --git a/packages/web/src/features/agents/review-agent/nodes/generateDiffReviewPrompt.ts b/packages/web/src/features/agents/review-agent/nodes/generateDiffReviewPrompt.ts index 131502265..251308779 100644 --- a/packages/web/src/features/agents/review-agent/nodes/generateDiffReviewPrompt.ts +++ b/packages/web/src/features/agents/review-agent/nodes/generateDiffReviewPrompt.ts @@ -1,8 +1,11 @@ import { sourcebot_diff, sourcebot_context, sourcebot_file_diff_review_schema } from "@/features/agents/review-agent/types"; import { zodToJsonSchema } from "zod-to-json-schema"; +import { createLogger } from "@sourcebot/logger"; + +const logger = createLogger('generate-diff-review-prompt'); export const generateDiffReviewPrompt = async (diff: sourcebot_diff, context: sourcebot_context[], rules: string[]) => { - console.log("Executing generate_diff_review_prompt"); + logger.debug("Executing generate_diff_review_prompt"); const prompt = ` You are an expert software engineer that excells at reviewing code changes. Given the input, additional context, and rules defined below, review the code changes and provide a detailed review. The review you provide @@ -39,6 +42,6 @@ export const generateDiffReviewPrompt = async (diff: sourcebot_diff, context: so ${JSON.stringify(zodToJsonSchema(sourcebot_file_diff_review_schema), null, 2)} `; - console.log("Completed generate_diff_review_prompt"); + logger.debug("Completed generate_diff_review_prompt"); return prompt; } \ No newline at end of file diff --git a/packages/web/src/features/agents/review-agent/nodes/generatePrReview.ts b/packages/web/src/features/agents/review-agent/nodes/generatePrReview.ts index c7ad81a3f..48f6bda03 100644 --- a/packages/web/src/features/agents/review-agent/nodes/generatePrReview.ts +++ b/packages/web/src/features/agents/review-agent/nodes/generatePrReview.ts @@ -2,9 +2,12 @@ import { sourcebot_pr_payload, sourcebot_diff_review, sourcebot_file_diff_review import { generateDiffReviewPrompt } from "@/features/agents/review-agent/nodes/generateDiffReviewPrompt"; import { invokeDiffReviewLlm } from "@/features/agents/review-agent/nodes/invokeDiffReviewLlm"; import { fetchFileContent } from "@/features/agents/review-agent/nodes/fetchFileContent"; +import { createLogger } from "@sourcebot/logger"; + +const logger = createLogger('generate-pr-review'); export const generatePrReviews = async (reviewAgentLogPath: string | undefined, pr_payload: sourcebot_pr_payload, rules: string[]): Promise => { - console.log("Executing generate_pr_reviews"); + logger.debug("Executing generate_pr_reviews"); const file_diff_reviews: sourcebot_file_diff_review[] = []; for (const file_diff of pr_payload.file_diffs) { @@ -32,7 +35,7 @@ export const generatePrReviews = async (reviewAgentLogPath: string | undefined, const diffReview = await invokeDiffReviewLlm(reviewAgentLogPath, prompt); reviews.push(...diffReview.reviews); } catch (error) { - console.error(`Error generating review for ${file_diff.to}: ${error}`); + logger.error(`Error generating review for ${file_diff.to}: ${error}`); } } @@ -44,6 +47,6 @@ export const generatePrReviews = async (reviewAgentLogPath: string | undefined, } } - console.log("Completed generate_pr_reviews"); + logger.debug("Completed generate_pr_reviews"); return file_diff_reviews; } \ No newline at end of file diff --git a/packages/web/src/features/agents/review-agent/nodes/githubPrParser.ts b/packages/web/src/features/agents/review-agent/nodes/githubPrParser.ts index 59ea158bb..b1cee1987 100644 --- a/packages/web/src/features/agents/review-agent/nodes/githubPrParser.ts +++ b/packages/web/src/features/agents/review-agent/nodes/githubPrParser.ts @@ -2,22 +2,25 @@ import { sourcebot_pr_payload, sourcebot_file_diff, sourcebot_diff } from "@/fea import parse from "parse-diff"; import { Octokit } from "octokit"; import { GitHubPullRequest } from "@/features/agents/review-agent/types"; +import { createLogger } from "@sourcebot/logger"; + +const logger = createLogger('github-pr-parser'); export const githubPrParser = async (octokit: Octokit, pullRequest: GitHubPullRequest): Promise => { - console.log("Executing github_pr_parser"); + logger.debug("Executing github_pr_parser"); let parsedDiff: parse.File[] = []; try { const diff = await octokit.request(pullRequest.diff_url); parsedDiff = parse(diff.data); } catch (error) { - console.error("Error fetching diff: ", error); + logger.error("Error fetching diff: ", error); throw error; } const sourcebotFileDiffs: (sourcebot_file_diff | null)[] = parsedDiff.map((file) => { if (!file.from || !file.to) { - console.log(`Skipping file due to missing from (${file.from}) or to (${file.to})`) + logger.debug(`Skipping file due to missing from (${file.from}) or to (${file.to})`) return null; } @@ -50,7 +53,7 @@ export const githubPrParser = async (octokit: Octokit, pullRequest: GitHubPullRe }); const filteredSourcebotFileDiffs: sourcebot_file_diff[] = sourcebotFileDiffs.filter((file) => file !== null) as sourcebot_file_diff[]; - console.log("Completed github_pr_parser"); + logger.debug("Completed github_pr_parser"); return { title: pullRequest.title, description: pullRequest.body ?? "", diff --git a/packages/web/src/features/agents/review-agent/nodes/githubPushPrReviews.ts b/packages/web/src/features/agents/review-agent/nodes/githubPushPrReviews.ts index 70ecd0433..e0a9e5973 100644 --- a/packages/web/src/features/agents/review-agent/nodes/githubPushPrReviews.ts +++ b/packages/web/src/features/agents/review-agent/nodes/githubPushPrReviews.ts @@ -1,8 +1,11 @@ import { Octokit } from "octokit"; import { sourcebot_pr_payload, sourcebot_file_diff_review } from "@/features/agents/review-agent/types"; +import { createLogger } from "@sourcebot/logger"; + +const logger = createLogger('github-push-pr-reviews'); export const githubPushPrReviews = async (octokit: Octokit, pr_payload: sourcebot_pr_payload, file_diff_reviews: sourcebot_file_diff_review[]) => { - console.log("Executing github_push_pr_reviews"); + logger.info("Executing github_push_pr_reviews"); try { for (const file_diff_review of file_diff_reviews) { @@ -25,13 +28,13 @@ export const githubPushPrReviews = async (octokit: Octokit, pr_payload: sourcebo }), }); } catch (error) { - console.error(`Error pushing pr reviews for ${file_diff_review.filename}: ${error}`); + logger.error(`Error pushing pr reviews for ${file_diff_review.filename}: ${error}`); } } } } catch (error) { - console.error(`Error pushing pr reviews: ${error}`); + logger.error(`Error pushing pr reviews: ${error}`); } - console.log("Completed github_push_pr_reviews"); + logger.info("Completed github_push_pr_reviews"); } \ No newline at end of file diff --git a/packages/web/src/features/agents/review-agent/nodes/invokeDiffReviewLlm.ts b/packages/web/src/features/agents/review-agent/nodes/invokeDiffReviewLlm.ts index 2a7038048..c726ba01a 100644 --- a/packages/web/src/features/agents/review-agent/nodes/invokeDiffReviewLlm.ts +++ b/packages/web/src/features/agents/review-agent/nodes/invokeDiffReviewLlm.ts @@ -2,12 +2,15 @@ import OpenAI from "openai"; import { sourcebot_file_diff_review, sourcebot_file_diff_review_schema } from "@/features/agents/review-agent/types"; import { env } from "@/env.mjs"; import fs from "fs"; +import { createLogger } from "@sourcebot/logger"; + +const logger = createLogger('invoke-diff-review-llm'); export const invokeDiffReviewLlm = async (reviewAgentLogPath: string | undefined, prompt: string): Promise => { - console.log("Executing invoke_diff_review_llm"); + logger.debug("Executing invoke_diff_review_llm"); if (!env.OPENAI_API_KEY) { - console.error("OPENAI_API_KEY is not set, skipping review agent"); + logger.error("OPENAI_API_KEY is not set, skipping review agent"); throw new Error("OPENAI_API_KEY is not set, skipping review agent"); } @@ -39,10 +42,10 @@ export const invokeDiffReviewLlm = async (reviewAgentLogPath: string | undefined throw new Error(`Invalid diff review format: ${diffReview.error}`); } - console.log("Completed invoke_diff_review_llm"); + logger.debug("Completed invoke_diff_review_llm"); return diffReview.data; } catch (error) { - console.error('Error calling OpenAI:', error); + logger.error('Error calling OpenAI:', error); throw error; } } \ No newline at end of file diff --git a/packages/web/src/features/entitlements/server.ts b/packages/web/src/features/entitlements/server.ts index 2eb3163f3..298098faa 100644 --- a/packages/web/src/features/entitlements/server.ts +++ b/packages/web/src/features/entitlements/server.ts @@ -3,6 +3,9 @@ import { Entitlement, entitlementsByPlan, Plan } from "./constants" import { base64Decode } from "@/lib/utils"; import { z } from "zod"; import { SOURCEBOT_SUPPORT_EMAIL } from "@/lib/constants"; +import { createLogger } from "@sourcebot/logger"; + +const logger = createLogger('entitlements'); const eeLicenseKeyPrefix = "sourcebot_ee_"; export const SOURCEBOT_UNLIMITED_SEATS = -1; @@ -22,7 +25,7 @@ const decodeLicenseKeyPayload = (payload: string): LicenseKeyPayload => { const payloadJson = JSON.parse(decodedPayload); return eeLicenseKeyPayloadSchema.parse(payloadJson); } catch (error) { - console.error(`Failed to decode license key payload: ${error}`); + logger.error(`Failed to decode license key payload: ${error}`); process.exit(1); } } @@ -49,13 +52,13 @@ export const getPlan = (): Plan => { if (licenseKey) { const expiryDate = new Date(licenseKey.expiryDate); if (expiryDate.getTime() < new Date().getTime()) { - console.error(`The provided license key has expired (${expiryDate.toLocaleString()}). Falling back to oss plan. Please contact ${SOURCEBOT_SUPPORT_EMAIL} for support.`); + logger.error(`The provided license key has expired (${expiryDate.toLocaleString()}). Falling back to oss plan. Please contact ${SOURCEBOT_SUPPORT_EMAIL} for support.`); process.exit(1); } return licenseKey.seats === SOURCEBOT_UNLIMITED_SEATS ? "self-hosted:enterprise-unlimited" : "self-hosted:enterprise"; } else { - console.info(`No valid license key found. Falling back to oss plan.`); + logger.info(`No valid license key found. Falling back to oss plan.`); return "oss"; } } diff --git a/packages/web/src/initialize.ts b/packages/web/src/initialize.ts index 545ef9555..5a9df2d8f 100644 --- a/packages/web/src/initialize.ts +++ b/packages/web/src/initialize.ts @@ -15,6 +15,9 @@ import { createGuestUser, setPublicAccessStatus } from '@/ee/features/publicAcce import { isServiceError } from './lib/utils'; import { ServiceErrorException } from './lib/serviceError'; import { SOURCEBOT_SUPPORT_EMAIL } from "@/lib/constants"; +import { createLogger } from "@sourcebot/logger"; + +const logger = createLogger('web-initialize'); const ajv = new Ajv({ validateFormats: false, @@ -73,7 +76,7 @@ const syncConnections = async (connections?: { [key: string]: ConnectionConfig } } }); - console.log(`Upserted connection with name '${key}'. Connection ID: ${connectionDb.id}`); + logger.info(`Upserted connection with name '${key}'. Connection ID: ${connectionDb.id}`); // Re-try any repos that failed to index. const failedRepos = currentConnection?.repos.filter(repo => repo.repo.repoIndexingStatus === RepoIndexingStatus.FAILED).map(repo => repo.repo.id) ?? []; @@ -104,7 +107,7 @@ const syncConnections = async (connections?: { [key: string]: ConnectionConfig } }); for (const connection of deletedConnections) { - console.log(`Deleting connection with name '${connection.name}'. Connection ID: ${connection.id}`); + logger.info(`Deleting connection with name '${connection.name}'. Connection ID: ${connection.id}`); await prisma.connection.delete({ where: { id: connection.id, @@ -142,12 +145,12 @@ const syncDeclarativeConfig = async (configPath: string) => { const hasPublicAccessEntitlement = hasEntitlement("public-access"); const enablePublicAccess = config.settings?.enablePublicAccess; if (enablePublicAccess !== undefined && !hasPublicAccessEntitlement) { - console.error(`Public access flag is set in the config file but your license doesn't have public access entitlement. Please contact ${SOURCEBOT_SUPPORT_EMAIL} to request a license upgrade.`); + logger.error(`Public access flag is set in the config file but your license doesn't have public access entitlement. Please contact ${SOURCEBOT_SUPPORT_EMAIL} to request a license upgrade.`); process.exit(1); } if (hasPublicAccessEntitlement) { - console.log(`Setting public access status to ${!!enablePublicAccess} for org ${SINGLE_TENANT_ORG_DOMAIN}`); + logger.info(`Setting public access status to ${!!enablePublicAccess} for org ${SINGLE_TENANT_ORG_DOMAIN}`); const res = await setPublicAccessStatus(SINGLE_TENANT_ORG_DOMAIN, !!enablePublicAccess); if (isServiceError(res)) { throw new ServiceErrorException(res); @@ -179,7 +182,7 @@ const pruneOldGuestUser = async () => { }, }); - console.log(`Deleted old guest user ${guestUser.userId}`); + logger.info(`Deleted old guest user ${guestUser.userId}`); } } @@ -227,7 +230,7 @@ const initSingleTenancy = async () => { // watch for changes assuming it is a local file if (!isRemotePath(configPath)) { watch(configPath, () => { - console.log(`Config file ${configPath} changed. Re-syncing...`); + logger.info(`Config file ${configPath} changed. Re-syncing...`); syncDeclarativeConfig(configPath); }); } @@ -237,7 +240,7 @@ const initSingleTenancy = async () => { const initMultiTenancy = async () => { const hasMultiTenancyEntitlement = hasEntitlement("multi-tenancy"); if (!hasMultiTenancyEntitlement) { - console.error(`SOURCEBOT_TENANCY_MODE is set to ${env.SOURCEBOT_TENANCY_MODE} but your license doesn't have multi-tenancy entitlement. Please contact ${SOURCEBOT_SUPPORT_EMAIL} to request a license upgrade.`); + logger.error(`SOURCEBOT_TENANCY_MODE is set to ${env.SOURCEBOT_TENANCY_MODE} but your license doesn't have multi-tenancy entitlement. Please contact ${SOURCEBOT_SUPPORT_EMAIL} to request a license upgrade.`); process.exit(1); } } diff --git a/yarn.lock b/yarn.lock index 2ae1187d2..e225fee14 100644 --- a/yarn.lock +++ b/yarn.lock @@ -5759,8 +5759,6 @@ __metadata: resolution: "@sourcebot/backend@workspace:packages/backend" dependencies: "@gitbeaker/rest": "npm:^40.5.1" - "@logtail/node": "npm:^0.5.2" - "@logtail/winston": "npm:^0.5.2" "@octokit/rest": "npm:^21.0.2" "@sentry/cli": "npm:^2.42.2" "@sentry/node": "npm:^9.3.0" @@ -5768,6 +5766,7 @@ __metadata: "@sourcebot/crypto": "workspace:*" "@sourcebot/db": "workspace:*" "@sourcebot/error": "workspace:*" + "@sourcebot/logger": "workspace:*" "@sourcebot/schemas": "workspace:*" "@t3-oss/env-core": "npm:^0.12.0" "@types/argparse": "npm:^2.0.16" @@ -5796,7 +5795,6 @@ __metadata: tsx: "npm:^4.19.1" typescript: "npm:^5.6.2" vitest: "npm:^2.1.9" - winston: "npm:^3.15.0" zod: "npm:^3.24.3" languageName: unknown linkType: soft @@ -5835,6 +5833,22 @@ __metadata: languageName: unknown linkType: soft +"@sourcebot/logger@workspace:*, @sourcebot/logger@workspace:packages/logger": + version: 0.0.0-use.local + resolution: "@sourcebot/logger@workspace:packages/logger" + dependencies: + "@logtail/node": "npm:^0.5.2" + "@logtail/winston": "npm:^0.5.2" + "@t3-oss/env-core": "npm:^0.12.0" + "@types/node": "npm:^22.7.5" + dotenv: "npm:^16.4.5" + triple-beam: "npm:^1.4.1" + typescript: "npm:^5.7.3" + winston: "npm:^3.15.0" + zod: "npm:^3.24.3" + languageName: unknown + linkType: soft + "@sourcebot/mcp@workspace:packages/mcp": version: 0.0.0-use.local resolution: "@sourcebot/mcp@workspace:packages/mcp" @@ -15525,7 +15539,7 @@ __metadata: languageName: node linkType: hard -"triple-beam@npm:^1.3.0": +"triple-beam@npm:^1.3.0, triple-beam@npm:^1.4.1": version: 1.4.1 resolution: "triple-beam@npm:1.4.1" checksum: 10c0/4bf1db71e14fe3ff1c3adbe3c302f1fdb553b74d7591a37323a7badb32dc8e9c290738996cbb64f8b10dc5a3833645b5d8c26221aaaaa12e50d1251c9aba2fea From 337eedc633b163e9c059d97487e378c693a35361 Mon Sep 17 00:00:00 2001 From: msukkari Date: Sat, 31 May 2025 15:24:50 -0700 Subject: [PATCH 06/16] add news entry for structured logging --- packages/web/src/lib/newsData.ts | 42 ++++++++++++++++++-------------- 1 file changed, 24 insertions(+), 18 deletions(-) diff --git a/packages/web/src/lib/newsData.ts b/packages/web/src/lib/newsData.ts index bca3a678b..1573e078e 100644 --- a/packages/web/src/lib/newsData.ts +++ b/packages/web/src/lib/newsData.ts @@ -1,22 +1,28 @@ import { NewsItem } from "./types"; export const newsData: NewsItem[] = [ - { - unique_id: "code-nav", - header: "Code navigation", - sub_header: "Built in go-to definition and find references", - url: "https://docs.sourcebot.dev/docs/search/code-navigation" - }, - { - unique_id: "sso", - header: "SSO", - sub_header: "We've added support for SSO providers", - url: "https://docs.sourcebot.dev/docs/configuration/authentication", - }, - { - unique_id: "search-contexts", - header: "Search contexts", - sub_header: "Filter searches by groups of repos", - url: "https://docs.sourcebot.dev/docs/search/search-contexts" - } + { + unique_id: "structured-logging", + header: "Structured logging", + sub_header: "We've added support for structured logging", + url: "https://docs.sourcebot.dev/docs/more/structured-logging" + }, + { + unique_id: "code-nav", + header: "Code navigation", + sub_header: "Built in go-to definition and find references", + url: "https://docs.sourcebot.dev/docs/search/code-navigation" + }, + { + unique_id: "sso", + header: "SSO", + sub_header: "We've added support for SSO providers", + url: "https://docs.sourcebot.dev/docs/configuration/authentication", + }, + { + unique_id: "search-contexts", + header: "Search contexts", + sub_header: "Filter searches by groups of repos", + url: "https://docs.sourcebot.dev/docs/search/search-contexts" + } ]; \ No newline at end of file From af6534f71f405b3198616cfee780afb3a1efda39 Mon Sep 17 00:00:00 2001 From: msukkari Date: Sat, 31 May 2025 15:44:46 -0700 Subject: [PATCH 07/16] add logger package to dockerfile and cleanup --- Dockerfile | 5 +++++ docs/docs/configuration/environment-variables.mdx | 4 +++- packages/backend/src/env.ts | 3 --- packages/web/src/env.mjs | 4 ---- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/Dockerfile b/Dockerfile index e2c9c2391..c2f3c398e 100644 --- a/Dockerfile +++ b/Dockerfile @@ -42,11 +42,13 @@ COPY ./packages/db ./packages/db COPY ./packages/schemas ./packages/schemas COPY ./packages/crypto ./packages/crypto COPY ./packages/error ./packages/error +COPY ./packages/logger ./packages/logger RUN yarn workspace @sourcebot/db install RUN yarn workspace @sourcebot/schemas install RUN yarn workspace @sourcebot/crypto install RUN yarn workspace @sourcebot/error install +RUN yarn workspace @sourcebot/logger install # ------------------------------------ # ------ Build Web ------ @@ -89,6 +91,7 @@ COPY --from=shared-libs-builder /app/packages/db ./packages/db COPY --from=shared-libs-builder /app/packages/schemas ./packages/schemas COPY --from=shared-libs-builder /app/packages/crypto ./packages/crypto COPY --from=shared-libs-builder /app/packages/error ./packages/error +COPY --from=shared-libs-builder /app/packages/logger ./packages/logger # Fixes arm64 timeouts RUN yarn workspace @sourcebot/web install @@ -128,6 +131,7 @@ COPY --from=shared-libs-builder /app/packages/db ./packages/db COPY --from=shared-libs-builder /app/packages/schemas ./packages/schemas COPY --from=shared-libs-builder /app/packages/crypto ./packages/crypto COPY --from=shared-libs-builder /app/packages/error ./packages/error +COPY --from=shared-libs-builder /app/packages/logger ./packages/logger RUN yarn workspace @sourcebot/backend install RUN yarn workspace @sourcebot/backend build @@ -209,6 +213,7 @@ COPY --from=shared-libs-builder /app/packages/db ./packages/db COPY --from=shared-libs-builder /app/packages/schemas ./packages/schemas COPY --from=shared-libs-builder /app/packages/crypto ./packages/crypto COPY --from=shared-libs-builder /app/packages/error ./packages/error +COPY --from=shared-libs-builder /app/packages/logger ./packages/logger # Configure dependencies RUN apk add --no-cache git ca-certificates bind-tools tini jansson wget supervisor uuidgen curl perl jq redis postgresql postgresql-contrib openssl util-linux unzip diff --git a/docs/docs/configuration/environment-variables.mdx b/docs/docs/configuration/environment-variables.mdx index ade59913f..d6a24285c 100644 --- a/docs/docs/configuration/environment-variables.mdx +++ b/docs/docs/configuration/environment-variables.mdx @@ -26,7 +26,9 @@ The following environment variables allow you to configure your Sourcebot deploy | `SMTP_CONNECTION_URL` | `-` |

The url to the SMTP service used for sending transactional emails. See [this doc](/docs/configuration/transactional-emails) for more info.

| | `SOURCEBOT_ENCRYPTION_KEY` | Automatically generated at startup if no value is provided. Generated using `openssl rand -base64 24` |

Used to encrypt connection secrets and generate API keys.

| | `SOURCEBOT_LOG_LEVEL` | `info` |

The Sourcebot logging level. Valid values are `debug`, `info`, `warn`, `error`, in order of severity.

| -| `SOURCEBOT_TELEMETRY_DISABLED` | `false` |

Enables/disables telemetry collection in Sourcebot. See [this doc](/self-hosting/security/telemetry) for more info.

| +| `SOURCEBOT_STRUCTURED_LOGGING_ENABLED` | `false` |

Enables/disable structured JSON logging. See [this doc](/docs/configuration/transactional-emails) for more info.

| +| `SOURCEBOT_STRUCTURED_LOGGING_FILE` | - |

Optional file to log to if structured logging is enabled

| +| `SOURCEBOT_TELEMETRY_DISABLED` | `false` |

Enables/disables telemetry collection in Sourcebot. See [this doc](/docs/more/structured-logging) for more info.

| | `TOTAL_MAX_MATCH_COUNT` | `100000` |

The maximum number of matches per query

| | `ZOEKT_MAX_WALL_TIME_MS` | `10000` |

The maximum real world duration (in milliseconds) per zoekt query

| diff --git a/packages/backend/src/env.ts b/packages/backend/src/env.ts index 0c0bfb639..40a6a1cc0 100644 --- a/packages/backend/src/env.ts +++ b/packages/backend/src/env.ts @@ -22,9 +22,6 @@ dotenv.config({ export const env = createEnv({ server: { SOURCEBOT_ENCRYPTION_KEY: z.string(), - SOURCEBOT_LOG_LEVEL: z.enum(["info", "debug", "warn", "error"]).default("info"), - SOURCEBOT_STRUCTURED_LOGGING_ENABLED: booleanSchema.default("false"), - SOURCEBOT_STRUCTURED_LOGGING_FILE: z.string().optional(), SOURCEBOT_TELEMETRY_DISABLED: booleanSchema.default("false"), SOURCEBOT_INSTALL_ID: z.string().default("unknown"), NEXT_PUBLIC_SOURCEBOT_VERSION: z.string().default("unknown"), diff --git a/packages/web/src/env.mjs b/packages/web/src/env.mjs index f1871d5d2..479acaa60 100644 --- a/packages/web/src/env.mjs +++ b/packages/web/src/env.mjs @@ -56,10 +56,6 @@ export const env = createEnv({ STRIPE_WEBHOOK_SECRET: z.string().optional(), STRIPE_ENABLE_TEST_CLOCKS: booleanSchema.default('false'), - // Logging - SOURCEBOT_LOG_LEVEL: z.enum(["debug", "info", "warn", "error"]).default("info"), - SOURCEBOT_STRUCTURED_LOGGING_ENABLED: booleanSchema.default('false'), - // Misc CONFIG_MAX_REPOS_NO_TOKEN: numberSchema.default(Number.MAX_SAFE_INTEGER), NODE_ENV: z.enum(["development", "test", "production"]), From 34105cd51adeb11642b92a40523b98870f8d791f Mon Sep 17 00:00:00 2001 From: bkellam Date: Sat, 31 May 2025 16:55:31 -0700 Subject: [PATCH 08/16] add gh workflow for catching broken links --- .github/workflows/docs-broken-links.yml | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 .github/workflows/docs-broken-links.yml diff --git a/.github/workflows/docs-broken-links.yml b/.github/workflows/docs-broken-links.yml new file mode 100644 index 000000000..3c9cdd835 --- /dev/null +++ b/.github/workflows/docs-broken-links.yml @@ -0,0 +1,25 @@ +name: Check for broken links in docs + +on: + pull_request: + branches: ["main"] + paths: + - "docs/**" + +jobs: + check-links: + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Use Node.Js + uses: actions/setup-node@v4 + with: + node-version: '20.x' + + - name: Install Mintlify CLI + run: npm i -g mintlify + + - name: Check for broken links + run: mintlify broken-links From c96d2cde896666d058af86742a574d7f991fd24b Mon Sep 17 00:00:00 2001 From: bkellam Date: Sat, 31 May 2025 16:56:39 -0700 Subject: [PATCH 09/16] further wip --- CHANGELOG.md | 14 +-- docs/development.mdx | 107 ------------------ docs/docs.json | 59 +++++----- .../{authentication.mdx => auth/overview.mdx} | 11 +- .../auth}/roles-and-permissions.mdx | 0 .../configuration/environment-variables.mdx | 13 ++- docs/docs/configuration/tenancy.mdx | 2 +- .../configuration/transactional-emails.mdx | 2 +- docs/docs/connections/gitea.mdx | 2 +- docs/docs/connections/github.mdx | 2 +- docs/docs/connections/gitlab.mdx | 2 +- docs/docs/connections/local-repos.mdx | 4 - docs/docs/connections/overview.mdx | 18 +-- ...k-start-guide.mdx => deployment-guide.mdx} | 60 +++++++--- docs/docs/{ => features}/agents/overview.mdx | 8 +- .../{ => features}/agents/review-agent.mdx | 2 +- .../{search => features}/code-navigation.mdx | 7 +- docs/docs/{more => features}/mcp-server.mdx | 28 +++-- .../search/multi-branch-indexing.mdx | 0 .../{ => features}/search/search-contexts.mdx | 7 +- .../search/syntax-reference.mdx | 2 +- docs/docs/more/api-keys.mdx | 8 -- docs/docs/overview.mdx | 60 ++++++++-- docs/docs/upgrade/v3-to-v4-guide.mdx | 4 +- docs/snippets/bitbucket-app-password.mdx | 2 +- docs/snippets/bitbucket-token.mdx | 2 +- docs/snippets/license-key-required.mdx | 4 + docs/snippets/schemas/v3/index.schema.mdx | 2 +- packages/mcp/README.md | 4 +- packages/schemas/src/v3/index.schema.ts | 2 +- packages/schemas/src/v3/index.type.ts | 2 +- packages/web/src/app/[domain]/agents/page.tsx | 2 +- .../browse/components/bottomPanel.tsx | 2 +- .../src/app/login/components/loginForm.tsx | 20 +++- packages/web/src/env.mjs | 6 + packages/web/src/lib/newsData.ts | 6 +- schemas/v3/index.json | 2 +- 37 files changed, 218 insertions(+), 260 deletions(-) delete mode 100644 docs/development.mdx rename docs/docs/configuration/{authentication.mdx => auth/overview.mdx} (92%) rename docs/docs/{more => configuration/auth}/roles-and-permissions.mdx (100%) rename docs/docs/{quick-start-guide.mdx => deployment-guide.mdx} (53%) rename docs/docs/{ => features}/agents/overview.mdx (60%) rename docs/docs/{ => features}/agents/review-agent.mdx (99%) rename docs/docs/{search => features}/code-navigation.mdx (79%) rename docs/docs/{more => features}/mcp-server.mdx (89%) rename docs/docs/{ => features}/search/multi-branch-indexing.mdx (100%) rename docs/docs/{ => features}/search/search-contexts.mdx (94%) rename docs/docs/{ => features}/search/syntax-reference.mdx (93%) delete mode 100644 docs/docs/more/api-keys.mdx create mode 100644 docs/snippets/license-key-required.mdx diff --git a/CHANGELOG.md b/CHANGELOG.md index 0f0a1bac3..d8488bd12 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,14 +22,14 @@ Sourcebot V4 introduces authentication, performance improvements and code naviga ### Changed - [**Breaking Change**] Authentication is now required by default. Notes: - When setting up your instance, email / password login will be the default authentication provider. - - The first user that logs into the instance is given the `owner` role. ([docs](https://docs.sourcebot.dev/docs/more/roles-and-permissions)). + - The first user that logs into the instance is given the `owner` role. ([docs](https://docs.sourcebot.dev/docs/configuration/auth/roles-and-permissions)). - Subsequent users can request to join the instance. The `owner` can approve / deny requests to join the instance via `Settings` > `Members` > `Pending Requests`. - If a user is approved to join the instance, they are given the `member` role. - - Additional login providers, including email links and SSO, can be configured with additional environment variables. ([docs](https://docs.sourcebot.dev/docs/configuration/authentication)). + - Additional login providers, including email links and SSO, can be configured with additional environment variables. ([docs](https://docs.sourcebot.dev/docs/configuration/auth/overview)). - Clicking on a search result now takes you to the `/browse` view. Files can still be previewed by clicking the "Preview" button or holding `Cmd` / `Ctrl` when clicking on a search result. [#315](https://github.com/sourcebot-dev/sourcebot/pull/315) ### Added -- [Sourcebot EE] Added search-based code navigation, allowing you to jump between symbol definition and references when viewing source files. [Read the documentation](https://docs.sourcebot.dev/docs/search/code-navigation). [#315](https://github.com/sourcebot-dev/sourcebot/pull/315) +- [Sourcebot EE] Added search-based code navigation, allowing you to jump between symbol definition and references when viewing source files. [Read the documentation](https://docs.sourcebot.dev/docs/features/code-navigation). [#315](https://github.com/sourcebot-dev/sourcebot/pull/315) - Added collapsible filter panel. [#315](https://github.com/sourcebot-dev/sourcebot/pull/315) - Added Sourcebot API key management for external clients. [#311](https://github.com/sourcebot-dev/sourcebot/pull/311) @@ -44,7 +44,7 @@ Sourcebot V4 introduces authentication, performance improvements and code naviga ## [3.2.0] - 2025-05-12 ### Added -- Added AI code review agent [#298](https://github.com/sourcebot-dev/sourcebot/pull/298). Checkout the [docs](https://docs.sourcebot.dev/docs/agents/review-agent) for more information. +- Added AI code review agent [#298](https://github.com/sourcebot-dev/sourcebot/pull/298). Checkout the [docs](https://docs.sourcebot.dev/docs/features/agents/review-agent) for more information. ### Fixed - Fixed issue with repos appearing in the carousel when they fail indexing for the first time. [#305](https://github.com/sourcebot-dev/sourcebot/pull/305) @@ -122,11 +122,11 @@ Sourcebot v3 is here and brings a number of structural changes to the tool's fou ### Added - Added parallelized repo indexing and connection syncing via Redis & BullMQ. See the [architecture overview](https://docs.sourcebot.dev/docs/overview#architecture). - Added repo indexing progress indicators in the navbar. -- Added authentication support via OAuth or email/password. For instructions on enabling, see [this doc](https://docs.sourcebot.dev/docs/configuration/authentication). -- Added the following UI for managing your deployment when **[auth is enabled](https://docs.sourcebot.dev/docs/configuration/authentication)**: +- Added authentication support via OAuth or email/password. For instructions on enabling, see [this doc](https://docs.sourcebot.dev/docs/configuration/auth/overview). +- Added the following UI for managing your deployment when **[auth is enabled](https://docs.sourcebot.dev/docs/configuration/auth/overview)**: - connection management: create and manage your JSON configs via a integrated web-editor. - secrets: import personal access tokens (PAT) into Sourcebot (AES-256 encrypted). Reference secrets in your connection config by name. - - team & invite management: invite users to your instance to give them access. Configure team [roles & permissions](https://docs.sourcebot.dev/docs/more/roles-and-permissions). + - team & invite management: invite users to your instance to give them access. Configure team [roles & permissions](https://docs.sourcebot.dev/docs/configuration/auth/roles-and-permissions). - Added multi-tenancy support. See [this doc](https://docs.sourcebot.dev/self-hosting/more/tenancy). ### Removed diff --git a/docs/development.mdx b/docs/development.mdx deleted file mode 100644 index fc3fec155..000000000 --- a/docs/development.mdx +++ /dev/null @@ -1,107 +0,0 @@ ---- -title: 'Development' -description: 'Preview changes locally to update your docs' ---- - - - **Prerequisite**: Please install Node.js (version 19 or higher) before proceeding.
- Please upgrade to ```docs.json``` before proceeding and delete the legacy ```mint.json``` file. - - -Follow these steps to install and run Mintlify on your operating system: - -**Step 1**: Install Mintlify: - - - - ```bash npm - npm i -g mintlify - ``` - -```bash yarn -yarn global add mintlify -``` - - - -**Step 2**: Navigate to the docs directory (where the `docs.json` file is located) and execute the following command: - -```bash -mintlify dev -``` - -A local preview of your documentation will be available at `http://localhost:3000`. - -### Custom Ports - -By default, Mintlify uses port 3000. You can customize the port Mintlify runs on by using the `--port` flag. To run Mintlify on port 3333, for instance, use this command: - -```bash -mintlify dev --port 3333 -``` - -If you attempt to run Mintlify on a port that's already in use, it will use the next available port: - -```md -Port 3000 is already in use. Trying 3001 instead. -``` - -## Mintlify Versions - -Please note that each CLI release is associated with a specific version of Mintlify. If your local website doesn't align with the production version, please update the CLI: - - - -```bash npm -npm i -g mintlify@latest -``` - -```bash yarn -yarn global upgrade mintlify -``` - - - -## Validating Links - -The CLI can assist with validating reference links made in your documentation. To identify any broken links, use the following command: - -```bash -mintlify broken-links -``` - -## Deployment - - - Unlimited editors available under the [Pro - Plan](https://mintlify.com/pricing) and above. - - -If the deployment is successful, you should see the following: - - - - - -## Code Formatting - -We suggest using extensions on your IDE to recognize and format MDX. If you're a VSCode user, consider the [MDX VSCode extension](https://marketplace.visualstudio.com/items?itemName=unifiedjs.vscode-mdx) for syntax highlighting, and [Prettier](https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode) for code formatting. - -## Troubleshooting - - - - - This may be due to an outdated version of node. Try the following: - 1. Remove the currently-installed version of mintlify: `npm remove -g mintlify` - 2. Upgrade to Node v19 or higher. - 3. Reinstall mintlify: `npm install -g mintlify` - - - - - Solution: Go to the root of your device and delete the \~/.mintlify folder. Afterwards, run `mintlify dev` again. - - - -Curious about what changed in the CLI version? [Check out the CLI changelog.](https://www.npmjs.com/package/mintlify?activeTab=versions) diff --git a/docs/docs.json b/docs/docs.json index 4c099ccfd..c4a5b7995 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -21,8 +21,30 @@ "group": "Getting Started", "pages": [ "docs/overview", - "docs/quick-start-guide", - "docs/license-key" + "docs/deployment-guide" + ] + }, + { + "group": "Features", + "pages": [ + { + "group": "Search", + "pages": [ + "docs/features/search/syntax-reference", + "docs/features/search/multi-branch-indexing", + "docs/features/search/search-contexts" + ] + }, + "docs/features/code-navigation", + "docs/features/mcp-server", + { + "group": "Agents", + "tag": "experimental", + "pages": [ + "docs/features/agents/overview", + "docs/features/agents/review-agent" + ] + } ] }, { @@ -43,35 +65,18 @@ "docs/connections/request-new" ] }, + "docs/license-key", "docs/configuration/environment-variables", - "docs/configuration/authentication", + { + "group": "Authentication", + "pages": [ + "docs/configuration/auth/overview", + "docs/configuration/auth/roles-and-permissions" + ] + }, "docs/configuration/transactional-emails" ] }, - { - "group": "Search", - "pages": [ - "docs/search/syntax-reference", - "docs/search/multi-branch-indexing", - "docs/search/code-navigation", - "docs/search/search-contexts" - ] - }, - { - "group": "Agents", - "pages": [ - "docs/agents/overview", - "docs/agents/review-agent" - ] - }, - { - "group": "More", - "pages": [ - "docs/more/api-keys", - "docs/more/roles-and-permissions", - "docs/more/mcp-server" - ] - }, { "group": "Upgrade", "pages": [ diff --git a/docs/docs/configuration/authentication.mdx b/docs/docs/configuration/auth/overview.mdx similarity index 92% rename from docs/docs/configuration/authentication.mdx rename to docs/docs/configuration/auth/overview.mdx index 3604f5f47..a64787629 100644 --- a/docs/docs/configuration/authentication.mdx +++ b/docs/docs/configuration/auth/overview.mdx @@ -1,13 +1,10 @@ --- -title: Authentication -sidebarTitle: Authentication +title: Overview --- -Make sure the `AUTH_URL` environment variable is [configured correctly](/self-hosting/configuration) when using Sourcebot behind a domain. - Sourcebot has built-in authentication that gates access to your organization. OAuth, email codes, and email / password are supported. -The first account that's registered on a Sourcebot deployment is made the owner. All other users who register must be [approved](/docs/configuration/authentication#approving-new-members) by the owner. +The first account that's registered on a Sourcebot deployment is made the owner. All other users who register must be [approved](/docs/configuration/auth/overview#approving-new-members) by the owner. ![Login Page](/images/login.png) @@ -16,7 +13,7 @@ The first account that's registered on a Sourcebot deployment is made the owner. All account registrations after the first account must be approved by the owner. The owner can see all join requests by going into **Settings -> Members**. -If you have an [enterprise license](/docs/license-key), you can enable [AUTH_EE_ENABLE_JIT_PROVISIONING](/docs/configuration/authentication#enterprise-authentication-providers) to +If you have an [enterprise license](/docs/license-key), you can enable [AUTH_EE_ENABLE_JIT_PROVISIONING](/docs/configuration/auth/overview#enterprise-authentication-providers) to have Sourcebot accounts automatically created and approved on registration. You can setup emails to be sent when new join requests are created/approved by configurating [transactional emails](/docs/configuration/transactional-emails) @@ -43,6 +40,8 @@ See [transactional emails](/docs/configuration/transactional-emails) for more de ## Enterprise Authentication Providers +If you're deploying Sourcebot behind a domain, you must set the [AUTH_URL](/docs/configuration/environment-variables) environment variable to use these providers. + The following authentication providers require an [enterprise license](/docs/license-key) to be enabled. By default, a new user registering using these providers must have their join request accepted by the owner of the organization to join. To allow a user to join automatically when diff --git a/docs/docs/more/roles-and-permissions.mdx b/docs/docs/configuration/auth/roles-and-permissions.mdx similarity index 100% rename from docs/docs/more/roles-and-permissions.mdx rename to docs/docs/configuration/auth/roles-and-permissions.mdx diff --git a/docs/docs/configuration/environment-variables.mdx b/docs/docs/configuration/environment-variables.mdx index ade59913f..99902b686 100644 --- a/docs/docs/configuration/environment-variables.mdx +++ b/docs/docs/configuration/environment-variables.mdx @@ -1,17 +1,18 @@ --- -title: Environment Variables -sidebarTitle: Environment Variables +title: Environment variables +sidebarTitle: Environment variables +mode: "wide" --- -This page provides a detailed reference of all environment variables supported by Sourcebot. If you're just looking to get up and running, we recommend starting with the [getting started](/self-hosting/overview) guide instead. +This page provides a detailed reference of all environment variables supported by Sourcebot. If you're just looking to get up and running, we recommend starting with the [deployment guide](/docs/deployment-guide) instead. ### Core Environment Variables The following environment variables allow you to configure your Sourcebot deployment. | Variable | Default | Description | | :------- | :------ | :---------- | -| `AUTH_CREDENTIALS_LOGIN_ENABLED` | `true` |

Enables/disables authentication with basic credentials. Username and passwords are stored encrypted at rest within the postgres database. Checkout the [auth docs](/docs/configuration/authentication) for more info

| -| `AUTH_EMAIL_CODE_LOGIN_ENABLED` | `false` |

Enables/disables authentication with a login code that's sent to a users email. `SMTP_CONNECTION_URL` and `EMAIL_FROM_ADDRESS` must also be set. Checkout the [auth docs](/docs/configuration/authentication) for more info

| +| `AUTH_CREDENTIALS_LOGIN_ENABLED` | `true` |

Enables/disables authentication with basic credentials. Username and passwords are stored encrypted at rest within the postgres database. Checkout the [auth docs](/docs/configuration/auth/overview) for more info

| +| `AUTH_EMAIL_CODE_LOGIN_ENABLED` | `false` |

Enables/disables authentication with a login code that's sent to a users email. `SMTP_CONNECTION_URL` and `EMAIL_FROM_ADDRESS` must also be set. Checkout the [auth docs](/docs/configuration/auth/overview) for more info

| | `AUTH_SECRET` | Automatically generated at startup if no value is provided. Generated using `openssl rand -base64 33` |

Used to validate login session cookies

| | `AUTH_URL` | - |

URL of your Sourcebot deployment, e.g., `https://example.com` or `http://localhost:3000`.

| | `CONFIG_PATH` | `-` |

The container relative path to the declerative configuration file. See [this doc](/docs/configuration/declarative-config) for more info.

| @@ -26,7 +27,7 @@ The following environment variables allow you to configure your Sourcebot deploy | `SMTP_CONNECTION_URL` | `-` |

The url to the SMTP service used for sending transactional emails. See [this doc](/docs/configuration/transactional-emails) for more info.

| | `SOURCEBOT_ENCRYPTION_KEY` | Automatically generated at startup if no value is provided. Generated using `openssl rand -base64 24` |

Used to encrypt connection secrets and generate API keys.

| | `SOURCEBOT_LOG_LEVEL` | `info` |

The Sourcebot logging level. Valid values are `debug`, `info`, `warn`, `error`, in order of severity.

| -| `SOURCEBOT_TELEMETRY_DISABLED` | `false` |

Enables/disables telemetry collection in Sourcebot. See [this doc](/self-hosting/security/telemetry) for more info.

| +| `SOURCEBOT_TELEMETRY_DISABLED` | `false` |

Enables/disables telemetry collection in Sourcebot. See [this doc](/docs/overview.mdx#telemetry) for more info.

| | `TOTAL_MAX_MATCH_COUNT` | `100000` |

The maximum number of matches per query

| | `ZOEKT_MAX_WALL_TIME_MS` | `10000` |

The maximum real world duration (in milliseconds) per zoekt query

| diff --git a/docs/docs/configuration/tenancy.mdx b/docs/docs/configuration/tenancy.mdx index ddd98d98b..b70412a9c 100644 --- a/docs/docs/configuration/tenancy.mdx +++ b/docs/docs/configuration/tenancy.mdx @@ -4,7 +4,7 @@ sidebarTitle: Multi tenancy --- If you're switching from single-tenant mode, delete the Sourcebot cache (the `.sourcebot` folder) before starting. -[Authentication](/docs/configuration/authentication) must be enabled to enable multi tenancy mode +[Authentication](/docs/configuration/auth/overview) must be enabled to enable multi tenancy mode Multi tenancy allows your Sourcebot deployment to have **multiple organizations**, each with their own set of members and repos. To enable multi tenancy mode, define an environment variable named `SOURCEBOT_TENANCY_MODE` and set its value to `multi`. When multi tenancy mode is enabled: diff --git a/docs/docs/configuration/transactional-emails.mdx b/docs/docs/configuration/transactional-emails.mdx index 9a18cf45e..53fea615a 100644 --- a/docs/docs/configuration/transactional-emails.mdx +++ b/docs/docs/configuration/transactional-emails.mdx @@ -7,7 +7,7 @@ To enable transactional emails in your deployment, set the following environment - Send emails when new members are invited - Send emails when organization join requests are created/accepted -- Log into the Sourcebot deployment using [email codes](self-hosting/more/authentication#email-codes) +- Log into the Sourcebot deployment using [email codes](/docs/configuration/auth/overview#email-codes) | Variable | Description | | :------- | :---------- | diff --git a/docs/docs/connections/gitea.mdx b/docs/docs/connections/gitea.mdx index 0a85eaeac..8869ebff8 100644 --- a/docs/docs/connections/gitea.mdx +++ b/docs/docs/connections/gitea.mdx @@ -108,7 +108,7 @@ Next, provide the access token via the `token` property, either as an environmen - Secrets are only supported when [authentication](/docs/configuration/authentication) is enabled. + Secrets are only supported when [authentication](/docs/configuration/auth/overview) is enabled. 1. Navigate to **Secrets** in settings and create a new secret with your PAT: diff --git a/docs/docs/connections/github.mdx b/docs/docs/connections/github.mdx index 525622331..2a86cecc4 100644 --- a/docs/docs/connections/github.mdx +++ b/docs/docs/connections/github.mdx @@ -137,7 +137,7 @@ Next, provide the PAT via the `token` property, either as an environment variabl - Secrets are only supported when [authentication](/docs/configuration/authentication) is enabled. + Secrets are only supported when [authentication](/docs/configuration/auth/overview) is enabled. 1. Navigate to **Secrets** in settings and create a new secret with your PAT: diff --git a/docs/docs/connections/gitlab.mdx b/docs/docs/connections/gitlab.mdx index 3ca971a58..d0131e503 100644 --- a/docs/docs/connections/gitlab.mdx +++ b/docs/docs/connections/gitlab.mdx @@ -142,7 +142,7 @@ Next, provide the PAT via the `token` property, either as an environment variabl - Secrets are only supported when [authentication](/docs/configuration/authentication) is enabled. + Secrets are only supported when [authentication](/docs/configuration/auth/overview) is enabled. 1. Navigate to **Secrets** in settings and create a new secret with your PAT: diff --git a/docs/docs/connections/local-repos.mdx b/docs/docs/connections/local-repos.mdx index 23a205146..580908861 100644 --- a/docs/docs/connections/local-repos.mdx +++ b/docs/docs/connections/local-repos.mdx @@ -5,10 +5,6 @@ icon: folder import GenericGitHost from '/snippets/schemas/v3/genericGitHost.schema.mdx' - -This feature is only supported when [self-hosting](/self-hosting/overview). - - Sourcebot can sync code from generic git repositories stored in a local directory. This can be helpful in scenarios where you already have a large number of repos already checked out. Local repositories are treated as **read-only**, meaing Sourcebot will **not** `git fetch` new revisions. ## Getting Started diff --git a/docs/docs/connections/overview.mdx b/docs/docs/connections/overview.mdx index fc5b41854..3a15ffa4a 100644 --- a/docs/docs/connections/overview.mdx +++ b/docs/docs/connections/overview.mdx @@ -5,23 +5,7 @@ sidebarTitle: Overview import SupportedPlatforms from '/snippets/platform-support.mdx' -To connect your code to Sourcebot you create **connections**. A **connection** is a configuration object that describes how Sourcebot should fetch information from a supported code host. - -There are two ways to define connections: - - - - This is only supported when self-hosting, and is the default mechanism to define connections. Connections are defined in a [JSON file](/docs/configuration/declarative-config) - and the path to the file is provided through the `CONFIG_PATH` environment variable - - - This is the only way to define connections when using Sourcebot Cloud, and can be configured when self-hosting by enabling [authentication](/docs/configuration/authentications). - - In this method, connections are defined and managed within the webapp: - - ![Connections page](/images/connection_page.png) - - +{/* TODO: document the concept of connections, the JSON schema, etc. */} ### Supported code hosts diff --git a/docs/docs/quick-start-guide.mdx b/docs/docs/deployment-guide.mdx similarity index 53% rename from docs/docs/quick-start-guide.mdx rename to docs/docs/deployment-guide.mdx index 621fa9ec7..694a960eb 100644 --- a/docs/docs/quick-start-guide.mdx +++ b/docs/docs/deployment-guide.mdx @@ -2,29 +2,43 @@ title: "Deployment guide" --- -## Quick Start Guide +import SupportedPlatforms from '/snippets/platform-support.mdx' + +The following guide will walk you through the steps to deploy Sourcebot on your own infrastructure. Sourcebot is distributed as a [single docker container](/docs/overview#architecture) that can be deployed to a k8s cluster, a VM, or any platform that supports docker. + + +## Walkthrough video +--- + +Watch this 1:51 minute video to get a quick overview of how to deploy Sourcebot using Docker. -{/* -*/} + +## Step-by-step guide +--- + +Hit an issue? Please let us know on [GitHub discussions](https://github.com/sourcebot-dev/sourcebot/discussions/categories/support) or by [emailing us](mailto:team@sourcebot.dev). - - By default, Sourcebot requires a configuration file with a list of [code host connections](/docs/connections/overview) that specify what repositories should be **synced** (cloned and indexed). To get started, run the following command to create a starter `config.json`: + + - Docker -> use [Docker Desktop](https://www.docker.com/products/docker-desktop/) on Mac or Windows. + + + Create a `config.json` file that tells Sourcebot which repositories to sync and index: ```bash touch config.json echo '{ "$schema": "https://raw.githubusercontent.com/sourcebot-dev/sourcebot/main/schemas/v3/index.json", "connections": { - // Comments are supported + // "starter-connection": { "type": "github", "repos": [ @@ -35,12 +49,11 @@ title: "Deployment guide" }' > config.json ``` - This config creates a single GitHub connection named `starter-connection` that specifies [Sourcebot](https://github.com/sourcebot-dev/sourcebot) as a repo to sync. + This config creates a single GitHub connection named `starter-connection` that specifies [Sourcebot](https://github.com/sourcebot-dev/sourcebot) as a repo to sync. [Learn more about the config file](/docs/connections/overview). - If you're deploying Sourcebot behind a domain, you must set the [AUTH_URL](/docs/configuration/environment-variables) environment variable - Sourcebot is packaged as a [single Docker image](https://github.com/sourcebot-dev/sourcebot/pkgs/container/sourcebot). In the same directory as `config.json`, run the following command to start your instance: + In the same directory as `config.json`, run the following command to start your instance: ``` bash docker run \ @@ -64,16 +77,14 @@ title: "Deployment guide" - reads `config.json` and starts syncing. - Hit an issue? Please let us know on [GitHub discussions](https://github.com/sourcebot-dev/sourcebot/discussions/categories/support) or by [emailing us](mailto:team@sourcebot.dev). - - Sourcebot has built-in authentication which gates your instance. The first account which is registered on a fresh Sourcebot deployment is made owner. - - Registration is performed using basic credentials which are stored encrypted within your deployment. To setup more authentication providers - check out the [auth docs](/docs/configuration/authentication) + + You're all set! Navigate to `http://localhost:3000` to login and start searching. - + + The first account which is registered on a fresh Sourcebot deployment is given the [owner role](/docs/configuration/auth/roles-and-permissions). + @@ -83,4 +94,17 @@ title: "Deployment guide" Missing your code host? [Submit a feature request on GitHub](https://github.com/sourcebot-dev/sourcebot/discussions/categories/ideas). - \ No newline at end of file + + + +## Next steps +--- + + + + Learn more about how to connect your code to Sourcebot. + + + Learn more about how to setup SSO, email codes, and other authentication providers. + + \ No newline at end of file diff --git a/docs/docs/agents/overview.mdx b/docs/docs/features/agents/overview.mdx similarity index 60% rename from docs/docs/agents/overview.mdx rename to docs/docs/features/agents/overview.mdx index 86fce12d0..3f5a82c6b 100644 --- a/docs/docs/agents/overview.mdx +++ b/docs/docs/features/agents/overview.mdx @@ -3,15 +3,15 @@ title: "Agents Overview" sidebarTitle: "Overview" --- - -Have an idea for an agent that we haven't built? Submit a [feature request](https://github.com/sourcebot-dev/sourcebot/discussions/categories/feature-requests) on our GitHub - + +Agents are currently a experimental feature. Have an idea for an agent that we haven't built? Submit a [feature request](https://github.com/sourcebot-dev/sourcebot/discussions/categories/feature-requests) on our GitHub. + Agents are automations that leverage the code indexed on Sourcebot to perform a specific task. Once you've setup Sourcebot, check out the guides below to configure additional agents. - + An AI agent that reviews your PRs to identify issues \ No newline at end of file diff --git a/docs/docs/agents/review-agent.mdx b/docs/docs/features/agents/review-agent.mdx similarity index 99% rename from docs/docs/agents/review-agent.mdx rename to docs/docs/features/agents/review-agent.mdx index 1d81d58f8..ae0b3d1ec 100644 --- a/docs/docs/agents/review-agent.mdx +++ b/docs/docs/features/agents/review-agent.mdx @@ -10,7 +10,7 @@ codebase that the agent may fetch to perform the review. This agent provides codebase-aware reviews for your PRs. For each diff, this agent fetches relevant context from Sourcebot and feeds it into an LLM for a detailed review of your changes. -The AI Code Review Agent is [open source](https://github.com/sourcebot-dev/sourcebot/tree/main/packages/web/src/features/agents/review-agent) and packaged in [Sourcebot](https://github.com/sourcebot-dev/sourcebot). To get started using this agent, [deploy Sourcebot](/self-hosting/overview) +The AI Code Review Agent is [open source](https://github.com/sourcebot-dev/sourcebot/tree/main/packages/web/src/features/agents/review-agent) and packaged in [Sourcebot](https://github.com/sourcebot-dev/sourcebot). To get started using this agent, [deploy Sourcebot](/docs/deployment-guide) and then follow the configuration instructions below. ![AI Code Review Agent Example](/images/review_agent_example.png) diff --git a/docs/docs/search/code-navigation.mdx b/docs/docs/features/code-navigation.mdx similarity index 79% rename from docs/docs/search/code-navigation.mdx rename to docs/docs/features/code-navigation.mdx index a18c851ff..6720556d4 100644 --- a/docs/docs/search/code-navigation.mdx +++ b/docs/docs/features/code-navigation.mdx @@ -4,10 +4,9 @@ sidebarTitle: Code navigation --- import SearchContextSchema from '/snippets/schemas/v3/searchContext.schema.mdx' +import LicenseKeyRequired from '/snippets/license-key-required.mdx' - -This feature is only available in [Sourcebot cloud](app.sourcebot.dev) or with an active Enterprise license when [self-hosting](/self-hosting). Please add your [license key](/docs/license-key) to activate it. - + **Code navigation** allows you to jump between symbol definition and references when viewing source files in Sourcebot. This feature is enabled **automatically** when a valid license key is present and works with all popular programming languages. @@ -25,7 +24,7 @@ This feature is only available in [Sourcebot cloud](app.sourcebot.dev) or with a ## How does it work? -Code navigation is **search-based**, meaning it uses the same code search engine and [query language](/docs/search/syntax-reference) to estimate a symbol's references and definitions. We refer to these estimations as "search heuristics". We have two search heuristics to enable the following operations: +Code navigation is **search-based**, meaning it uses the same code search engine and [query language](/docs/features/search/syntax-reference) to estimate a symbol's references and definitions. We refer to these estimations as "search heuristics". We have two search heuristics to enable the following operations: ### Find references Given a `symbolName`, along with information about the file the symbol is contained within (`git_revision`, and `language`), runs the following search: diff --git a/docs/docs/more/mcp-server.mdx b/docs/docs/features/mcp-server.mdx similarity index 89% rename from docs/docs/more/mcp-server.mdx rename to docs/docs/features/mcp-server.mdx index 378e82800..b18781b77 100644 --- a/docs/docs/more/mcp-server.mdx +++ b/docs/docs/features/mcp-server.mdx @@ -3,21 +3,25 @@ title: Sourcebot MCP server (@sourcebot/mcp) sidebarTitle: Sourcebot MCP server --- - -This feature is only available when [self-hosting](/self-hosting) - - The [Model Context Protocol](https://modelcontextprotocol.io/introduction) (MCP) is an open standard for providing context to LLMs. The [@sourcebot/mcp](https://www.npmjs.com/package/@sourcebot/mcp) package is a MCP server that enables LLMs to interface with your Sourcebot instance, enabling MCP clients like Cursor, Vscode, and others to have context over your entire codebase. ## Getting Started - Follow the self-hosting [quick start guide](/self-hosting/overview#quick-start-guide) to launch Sourcebot and get your code indexed. The host url of your instance (e.g., `http://localhost:3000`) is passed to the MCP server via the `SOURCEBOT_HOST` url. + Follow the [deployment guide](/docs/deployment-guide) to launch Sourcebot and get your code indexed. The host url of your instance (e.g., `http://localhost:3000`) is passed to the MCP server via the `SOURCEBOT_HOST` url. If a host is not provided, then the server will fallback to using the demo instance hosted at https://demo.sourcebot.dev. You can see the list of repositories indexed [here](https://demo.sourcebot.dev/~/repos). Add additional repositories by [opening a PR](https://github.com/sourcebot-dev/sourcebot/blob/main/demo-site-config.json). + + Create an API key to allow the MCP server to query your Sourcebot instance. To create an API key, login to your Sourcebot instance and navigate to **Settings -> API Keys**: + + ![API Keys UI](/images/api_key.png) + + Copy the API key and set it as the `SOURCEBOT_API_KEY` environment variable. + + Ensure you have [Node.js](https://nodejs.org/en) >= v18.0.0 installed. @@ -39,7 +43,8 @@ The [Model Context Protocol](https://modelcontextprotocol.io/introduction) (MCP) "command": "npx", "args": ["-y", "@sourcebot/mcp@latest" ], "env": { - "SOURCEBOT_HOST": "http://localhost:3000" + "SOURCEBOT_HOST": "http://localhost:3000", + "SOURCEBOT_API_KEY": "your-api-key" } } } @@ -62,7 +67,8 @@ The [Model Context Protocol](https://modelcontextprotocol.io/introduction) (MCP) "command": "npx", "args": ["-y", "@sourcebot/mcp@latest" ], "env": { - "SOURCEBOT_HOST": "http://localhost:3000" + "SOURCEBOT_HOST": "http://localhost:3000", + "SOURCEBOT_API_KEY": "your-api-key" } } } @@ -86,7 +92,8 @@ The [Model Context Protocol](https://modelcontextprotocol.io/introduction) (MCP) "command": "npx", "args": ["-y", "@sourcebot/mcp@latest"], "env": { - "SOURCEBOT_HOST": "http://localhost:3000" + "SOURCEBOT_HOST": "http://localhost:3000", + "SOURCEBOT_API_KEY": "your-api-key" } } } @@ -102,7 +109,7 @@ The [Model Context Protocol](https://modelcontextprotocol.io/introduction) (MCP) Run the following command: ```sh - claude mcp add sourcebot -e SOURCEBOT_HOST=http://localhost:3000 -- npx -y @sourcebot/mcp@latest + claude mcp add sourcebot -e SOURCEBOT_HOST=http://localhost:3000 -e SOURCEBOT_API_KEY=your-api-key -- npx -y @sourcebot/mcp@latest ``` Replace `http://localhost:3000` with wherever your Sourcebot instance is hosted. @@ -119,7 +126,8 @@ The [Model Context Protocol](https://modelcontextprotocol.io/introduction) (MCP) "command": "npx", "args": ["-y", "@sourcebot/mcp@latest"], "env": { - "SOURCEBOT_HOST": "http://localhost:3000" + "SOURCEBOT_HOST": "http://localhost:3000", + "SOURCEBOT_API_KEY": "your-api-key" } } } diff --git a/docs/docs/search/multi-branch-indexing.mdx b/docs/docs/features/search/multi-branch-indexing.mdx similarity index 100% rename from docs/docs/search/multi-branch-indexing.mdx rename to docs/docs/features/search/multi-branch-indexing.mdx diff --git a/docs/docs/search/search-contexts.mdx b/docs/docs/features/search/search-contexts.mdx similarity index 94% rename from docs/docs/search/search-contexts.mdx rename to docs/docs/features/search/search-contexts.mdx index 6d222aa70..e8b7aab42 100644 --- a/docs/docs/search/search-contexts.mdx +++ b/docs/docs/features/search/search-contexts.mdx @@ -4,10 +4,9 @@ sidebarTitle: Search contexts --- import SearchContextSchema from '/snippets/schemas/v3/searchContext.schema.mdx' +import LicenseKeyRequired from '/snippets/license-key-required.mdx' - -This feature is only available when [self-hosting](/self-hosting) with an active Enterprise license. Please add your [license key](/docs/license-key) to activate it. - + A **search context** is a user-defined grouping of repositories that helps focus searches on specific areas of your codebase, like frontend, backend, or infrastructure code. Some example queries using search contexts: @@ -105,7 +104,7 @@ Like other prefixes, contexts can be negated using `-` or combined using `or`: - `-context:web` excludes frontend repositories from results - `( context:web or context:backend )` searches across both frontend and backend code -See [this doc](/docs/search/syntax-reference) for more details on the search query syntax. +See [this doc](/docs/features/search/syntax-reference) for more details on the search query syntax. ## Schema reference diff --git a/docs/docs/search/syntax-reference.mdx b/docs/docs/features/search/syntax-reference.mdx similarity index 93% rename from docs/docs/search/syntax-reference.mdx rename to docs/docs/features/search/syntax-reference.mdx index 30cfb1096..cde52d0e2 100644 --- a/docs/docs/search/syntax-reference.mdx +++ b/docs/docs/features/search/syntax-reference.mdx @@ -32,4 +32,4 @@ Expressions can be prefixed with certain keywords to modify search behavior. Som | `rev:` | Filter results from a specific branch or tag. By default **only** the default branch is searched. | `rev:beta` - Filter results to branches that match regex `/beta/` | | `lang:` | Filter results by language (as defined by [linguist](https://github.com/github-linguist/linguist/blob/main/lib/linguist/languages.yml)). By default all languages are searched. | `lang:TypeScript` - Filter results to TypeScript files
`-lang:YAML` - Ignore results from YAML files | | `sym:` | Match symbol definitions created by [universal ctags](https://ctags.io/) at index time. | `sym:\bmain\b` - Filter results to symbols that match regex `/\bmain\b/` | -| `context:` | Filter results to a predefined [search context](/docs/search/search-contexts). | `context:web` - Filter results to the web context
`-context:pipelines` - Ignore results from the pipelines context | \ No newline at end of file +| `context:` | Filter results to a predefined [search context](/docs/features/search/search-contexts). | `context:web` - Filter results to the web context
`-context:pipelines` - Ignore results from the pipelines context | \ No newline at end of file diff --git a/docs/docs/more/api-keys.mdx b/docs/docs/more/api-keys.mdx deleted file mode 100644 index 4aa31a695..000000000 --- a/docs/docs/more/api-keys.mdx +++ /dev/null @@ -1,8 +0,0 @@ ---- -title: API Keys ---- - -An API Key is required when querying Sourcebot outside the context of the web app client (ex. MCP server, review agent). To create an API key, login to your Sourcebot instance and navigate to -**Settings -> API Keys**: - -![API Keys UI](/images/api_key.png) \ No newline at end of file diff --git a/docs/docs/overview.mdx b/docs/docs/overview.mdx index 484504acd..d6f1d4946 100644 --- a/docs/docs/overview.mdx +++ b/docs/docs/overview.mdx @@ -2,9 +2,16 @@ title: "Overview" --- -import SupportedPlatforms from '/snippets/platform-support.mdx' +> Sourcebot is an open-source ([GitHub](https://github.com/sourcebot-dev/sourcebot)), self-hosted code search tool that is purpose built to help teams find and navigate code quickly, at scale. -Sourcebot is an [open-source](https://github.com/sourcebot-dev/sourcebot) code search tool that is purpose built to search multi-million line codebases in seconds. It integrates with [GitHub](/docs/connections/github), [GitLab](/docs/connections/gitlab), and [other platforms](/docs/connections). + + + - **Open-source:** + - **Fast:** Search across millions of lines of code in seconds using Sourcebot's blazingly fast indexed search. + - **Scalable:** Sourcebot uses [trigram indexing](https://en.wikipedia.org/wiki/Trigram_search), allowing it to scale to massive codebases. + - **Self-hosted:** Sourcebot is designed to be easily self-hosted, allowing you to deploy it onto your own infrastructure, keeping your code private and secure. + + ## Features --- @@ -15,8 +22,8 @@ Search across millions of lines of code in seconds using Sourcebot's blazingly f - **Regex support:** Use regular expressions to find code with precision. -- **Query language:** Scope searches to specific files, repos, languages, symbol definitions and more using a rich [query language](/docs/search/syntax-reference). -- **Branch search:** Specify a list of branches to search across ([docs](/docs/search/multi-branch-indexing)). +- **Query language:** Scope searches to specific files, repos, languages, symbol definitions and more using a rich [query language](/docs/features/search/syntax-reference). +- **Branch search:** Specify a list of branches to search across ([docs](/docs/features/search/multi-branch-indexing)). - **Fast & scalable:** Sourcebot uses [trigram indexing](https://en.wikipedia.org/wiki/Trigram_search), allowing it to scale to massive codebases. - **Syntax highlighting:** Syntax highlighting support for over [100+ languages](https://github.com/sourcebot-dev/sourcebot/blob/57724689303f351c279d37f45b6406f1d5d5d5ab/packages/web/src/lib/codemirrorLanguage.ts#L125). - **Multi-repository:** Search across all of your repositories in a single search. @@ -35,7 +42,7 @@ Search across millions of lines of code in seconds using Sourcebot's blazingly f ### Code Navigation -[Code navigation](/docs/search/code-navigation) helps you jump between symbol definitions and references quickly when browsing source code in Sourcebot. +[Code navigation](/docs/features/code-navigation) helps you jump between symbol definitions and references quickly when browsing source code in Sourcebot. - **Hover popover:** Hovering over a symbol reveals the symbol's definition signature in a inline preview. @@ -85,17 +92,47 @@ Connect your code from multiple code-host platforms and search across all of the ### Authentication -TODO +Sourcebot comes with built-in support for authentication via [email/password](/docs/configuration/auth/overview#email-%2F-password), [email codes](/docs/configuration/auth/overview#email-codes), and various [SSO providers](/docs/configuration/auth/overview#enterprise-authentication-providers). -### Light / Dark mode support + +- **Configurable auth providers:** Configure the auth providers that are available to your team. +- **SSO:** Support for various SSO providers. +- **_(coming soon)_ RBAC:** Role-based access control for managing user permissions. +- **_(coming soon)_ Code host permission syncing:** Sync permissions from GitHub, Gitlab, etc. to Sourcebot. +- **_(coming soon)_ Audit logs:** Audit logs for all actions performed on Sourcebot, such as user login, search, etc. + -TODO + ### Self-hosted -TODO +Sourcebot is designed to be easily self-hosted, allowing you to deploy it onto your own infrastructure, keeping your code private and secure. + + +- **Easy deployment:** Sourcebot is shipped as a [single docker container](https://github.com/sourcebot-dev/sourcebot/pkgs/container/sourcebot) that can be deployed to a k8s cluster, a VM, or any other platform that supports docker. +- **Secure:** Your code **never** leaves your infrastructure. +- **No-vendor lock-in:** Avoid dependency on a third-party SaaS provider; you can modify, extend, or migrate your deployment as needed. + + +## Get started +--- + + + + + + + ## Architecture +--- Sourcebot is shipped as a single docker container that runs a collection of services using [supervisord](https://supervisord.org/): @@ -111,9 +148,10 @@ Sourcebot consists of the following components: - **Redis Job Queue** : fast in-memory store. Used with [BullMQ](https://docs.bullmq.io/) for queuing asynchronous work. - **`.sourcebot/` cache** : file-system cache where persistent data is written. -You can use managed Redis / Postgres services that run outside of the Sourcebot container by providing the `REDIS_URL` and `DATABASE_URL` environment variables, respectively. See the [configuration](/self-hosting/configuration) for more configuration options. +You can use managed Redis / Postgres services that run outside of the Sourcebot container by providing the `REDIS_URL` and `DATABASE_URL` environment variables, respectively. See the [environment variables](/docs/configuration/environment-variables) doc for more configuration options. ## Scalability +--- One of our design philosophies for Sourcebot is to keep our infrastructure [radically simple](https://www.radicalsimpli.city/) while balancing scalability concerns. Depending on the number of repositories you have indexed and the instance you are running Sourcebot on, you may experience slow search times or other performance degradations. Our recommendation is to vertically scale your instance by increasing the number of CPU cores and memory. @@ -121,6 +159,8 @@ Sourcebot does not support horizontal scaling at this time, but it is on our roa ## Telemetry +--- + By default, Sourcebot collects anonymized usage data through [PostHog](https://posthog.com/) to help us improve the performance and reliability of our tool. We don't collect or transmit any information related to your codebase. In addition, all events are [sanitized](https://github.com/sourcebot-dev/sourcebot/blob/HEAD/packages/web/src/app/posthogProvider.tsx) to ensure that no sensitive details (ex. ip address, query info) leave your machine. The data we collect includes general usage statistics and metadata such as query performance (e.g., search duration, error rates) to monitor the application's health and functionality. This information helps us better understand how Sourcebot is used and where improvements can be made. diff --git a/docs/docs/upgrade/v3-to-v4-guide.mdx b/docs/docs/upgrade/v3-to-v4-guide.mdx index c51da3a9d..93477157b 100644 --- a/docs/docs/upgrade/v3-to-v4-guide.mdx +++ b/docs/docs/upgrade/v3-to-v4-guide.mdx @@ -22,7 +22,7 @@ Please note that the following features are no longer supported in v4: When you visit your new deployment you'll be presented with a sign-in page. Sourcebot now requires authentication, and all users must register and sign-in to the deployment. The first account that's registered will be made the owner. By default, you can register using basic credentials which will be stored encrypted within the postgres DB connected to Sourcebot. Check out - the [auth docs](/docs/configuration/authentication) to setup additional auth providers. + the [auth docs](/docs/configuration/auth/overview) to setup additional auth providers. @@ -36,7 +36,7 @@ Please note that the following features are no longer supported in v4: ![Pending Approval Page](/images/pending_approval.png) - The owner can view and approve join requests by navigating to **Settings -> Members**. Automatic provisioning of accounts is supported when using SSO/Oauth providers, check out the [auth docs](/docs/configuration/authentication#enterprise-authentication-providers) for more info + The owner can view and approve join requests by navigating to **Settings -> Members**. Automatic provisioning of accounts is supported when using SSO/Oauth providers, check out the [auth docs](/docs/configuration/auth/overview#enterprise-authentication-providers) for more info diff --git a/docs/snippets/bitbucket-app-password.mdx b/docs/snippets/bitbucket-app-password.mdx index 778edfa6b..1f52e79c1 100644 --- a/docs/snippets/bitbucket-app-password.mdx +++ b/docs/snippets/bitbucket-app-password.mdx @@ -27,7 +27,7 @@ - Secrets are only supported when [authentication](/docs/configuration/authentication) is enabled. + Secrets are only supported when [authentication](/docs/configuration/auth/overview) is enabled. 1. Navigate to **Secrets** in settings and create a new secret with your access token: diff --git a/docs/snippets/bitbucket-token.mdx b/docs/snippets/bitbucket-token.mdx index d353eddc5..8b7e1db67 100644 --- a/docs/snippets/bitbucket-token.mdx +++ b/docs/snippets/bitbucket-token.mdx @@ -25,7 +25,7 @@ - Secrets are only supported when [authentication](/docs/configuration/authentication) is enabled. + Secrets are only supported when [authentication](/docs/configuration/auth/overview) is enabled. 1. Navigate to **Secrets** in settings and create a new secret with your PAT: diff --git a/docs/snippets/license-key-required.mdx b/docs/snippets/license-key-required.mdx new file mode 100644 index 000000000..603994fd9 --- /dev/null +++ b/docs/snippets/license-key-required.mdx @@ -0,0 +1,4 @@ + + +This feature is only available with an active Enterprise license. Please add your [license key](/docs/license-key) to activate it. + \ No newline at end of file diff --git a/docs/snippets/schemas/v3/index.schema.mdx b/docs/snippets/schemas/v3/index.schema.mdx index 11e18f559..79bcda808 100644 --- a/docs/snippets/schemas/v3/index.schema.mdx +++ b/docs/snippets/schemas/v3/index.schema.mdx @@ -188,7 +188,7 @@ }, "contexts": { "type": "object", - "description": "[Sourcebot EE] Defines a collection of search contexts. This is only available in single-tenancy mode. See: https://docs.sourcebot.dev/docs/search/search-contexts", + "description": "[Sourcebot EE] Defines a collection of search contexts. This is only available in single-tenancy mode. See: https://docs.sourcebot.dev/docs/features/search/search-contexts", "patternProperties": { "^[a-zA-Z0-9_-]+$": { "$schema": "http://json-schema.org/draft-07/schema#", diff --git a/packages/mcp/README.md b/packages/mcp/README.md index e74f6499d..1b0a07712 100644 --- a/packages/mcp/README.md +++ b/packages/mcp/README.md @@ -2,7 +2,7 @@ [![Sourcebot](https://img.shields.io/badge/Website-sourcebot.dev-blue)](https://sourcebot.dev) [![GitHub](https://img.shields.io/badge/GitHub-sourcebot--dev%2Fsourcebot-green?logo=github)](https://github.com/sourcebot-dev/sourcebot) -[![Docs](https://img.shields.io/badge/Docs-docs.sourcebot.dev-yellow)](https://docs.sourcebot.dev/docs/more/mcp-server) +[![Docs](https://img.shields.io/badge/Docs-docs.sourcebot.dev-yellow)](https://docs.sourcebot.dev/docs/features/mcp-server) [![npm](https://img.shields.io/npm/v/@sourcebot/mcp)](https://www.npmjs.com/package/@sourcebot/mcp) The Sourcebot MCP server gives your LLM agents the ability to fetch code context across thousands of repos hosted on [GitHub](https://docs.sourcebot.dev/docs/connections/github), [GitLab](https://docs.sourcebot.dev/docs/connections/gitlab), [BitBucket](https://docs.sourcebot.dev/docs/connections/bitbucket-cloud) and [more](#supported-code-hosts). Ask your LLM a question, and the Sourcebot MCP server will fetch relevant context from its index and inject it into your chat session. Some use cases this unlocks include: @@ -159,7 +159,7 @@ The Sourcebot MCP server gives your LLM agents the ability to fetch code context
-For a more detailed guide, checkout [the docs](https://docs.sourcebot.dev/docs/more/mcp-server). +For a more detailed guide, checkout [the docs](https://docs.sourcebot.dev/docs/features/mcp-server). ## Available Tools diff --git a/packages/schemas/src/v3/index.schema.ts b/packages/schemas/src/v3/index.schema.ts index 925b27a82..35e7a4fe6 100644 --- a/packages/schemas/src/v3/index.schema.ts +++ b/packages/schemas/src/v3/index.schema.ts @@ -187,7 +187,7 @@ const schema = { }, "contexts": { "type": "object", - "description": "[Sourcebot EE] Defines a collection of search contexts. This is only available in single-tenancy mode. See: https://docs.sourcebot.dev/docs/search/search-contexts", + "description": "[Sourcebot EE] Defines a collection of search contexts. This is only available in single-tenancy mode. See: https://docs.sourcebot.dev/docs/features/search/search-contexts", "patternProperties": { "^[a-zA-Z0-9_-]+$": { "$schema": "http://json-schema.org/draft-07/schema#", diff --git a/packages/schemas/src/v3/index.type.ts b/packages/schemas/src/v3/index.type.ts index 4506c611d..d239245f1 100644 --- a/packages/schemas/src/v3/index.type.ts +++ b/packages/schemas/src/v3/index.type.ts @@ -16,7 +16,7 @@ export interface SourcebotConfig { $schema?: string; settings?: Settings; /** - * [Sourcebot EE] Defines a collection of search contexts. This is only available in single-tenancy mode. See: https://docs.sourcebot.dev/docs/search/search-contexts + * [Sourcebot EE] Defines a collection of search contexts. This is only available in single-tenancy mode. See: https://docs.sourcebot.dev/docs/features/search/search-contexts */ contexts?: { [k: string]: SearchContext; diff --git a/packages/web/src/app/[domain]/agents/page.tsx b/packages/web/src/app/[domain]/agents/page.tsx index 7f283ede8..3bbccd0c3 100644 --- a/packages/web/src/app/[domain]/agents/page.tsx +++ b/packages/web/src/app/[domain]/agents/page.tsx @@ -9,7 +9,7 @@ const agents = [ name: "Review Agent", description: "An AI code review agent that reviews your PRs. Uses the code indexed on Sourcebot to provide codebase-wide context.", requiredEnvVars: ["GITHUB_APP_ID", "GITHUB_APP_WEBHOOK_SECRET", "GITHUB_APP_PRIVATE_KEY_PATH", "OPENAI_API_KEY"], - configureUrl: "https://docs.sourcebot.dev/docs/agents/review-agent" + configureUrl: "https://docs.sourcebot.dev/docs/features/agents/review-agent" }, ]; diff --git a/packages/web/src/app/[domain]/browse/components/bottomPanel.tsx b/packages/web/src/app/[domain]/browse/components/bottomPanel.tsx index 86147ee7a..90ec4b742 100644 --- a/packages/web/src/app/[domain]/browse/components/bottomPanel.tsx +++ b/packages/web/src/app/[domain]/browse/components/bottomPanel.tsx @@ -18,7 +18,7 @@ import { useRouter } from "next/navigation"; export const BOTTOM_PANEL_MIN_SIZE = 35; export const BOTTOM_PANEL_MAX_SIZE = 65; -const CODE_NAV_DOCS_URL = "https://docs.sourcebot.dev/docs/search/code-navigation"; +const CODE_NAV_DOCS_URL = "https://docs.sourcebot.dev/docs/features/code-navigation"; export const BottomPanel = () => { const panelRef = useRef(null); diff --git a/packages/web/src/app/login/components/loginForm.tsx b/packages/web/src/app/login/components/loginForm.tsx index 132e41ec3..bce690258 100644 --- a/packages/web/src/app/login/components/loginForm.tsx +++ b/packages/web/src/app/login/components/loginForm.tsx @@ -1,9 +1,8 @@ 'use client'; -import { Button } from "@/components/ui/button"; import Image from "next/image"; import { signIn } from "next-auth/react"; -import { Fragment, useCallback, useMemo } from "react"; +import { Fragment, useCallback, useMemo, useState } from "react"; import { Card } from "@/components/ui/card"; import { cn, getAuthProviderInfo } from "@/lib/utils"; import { MagicLinkForm } from "./magicLinkForm"; @@ -14,6 +13,7 @@ import useCaptureEvent from "@/hooks/useCaptureEvent"; import DemoCard from "@/app/[domain]/onboard/components/demoCard"; import Link from "next/link"; import { env } from "@/env.mjs"; +import { LoadingButton } from "@/components/ui/loading-button"; const TERMS_OF_SERVICE_URL = "https://sourcebot.dev/terms"; const PRIVACY_POLICY_URL = "https://sourcebot.dev/privacy"; @@ -27,7 +27,9 @@ interface LoginFormProps { export const LoginForm = ({ callbackUrl, error, providers }: LoginFormProps) => { const captureEvent = useCaptureEvent(); const onSignInWithOauth = useCallback((provider: string) => { - signIn(provider, { redirectTo: callbackUrl ?? "/" }); + signIn(provider, { + redirectTo: callbackUrl ?? "/" + }); }, [callbackUrl]); const errorMessage = useMemo(() => { @@ -137,15 +139,21 @@ const ProviderButton = ({ onClick: () => void; className?: string; }) => { + const [isLoading, setIsLoading] = useState(false); + return ( - + ) } diff --git a/packages/web/src/env.mjs b/packages/web/src/env.mjs index 479acaa60..60996de9d 100644 --- a/packages/web/src/env.mjs +++ b/packages/web/src/env.mjs @@ -26,20 +26,26 @@ export const env = createEnv({ // Enterprise Auth AUTH_EE_ENABLE_JIT_PROVISIONING: booleanSchema.default('false'), + AUTH_EE_GITHUB_CLIENT_ID: z.string().optional(), AUTH_EE_GITHUB_CLIENT_SECRET: z.string().optional(), AUTH_EE_GITHUB_BASE_URL: z.string().optional(), + AUTH_EE_GITLAB_CLIENT_ID: z.string().optional(), AUTH_EE_GITLAB_CLIENT_SECRET: z.string().optional(), AUTH_EE_GITLAB_BASE_URL: z.string().default("https://gitlab.com"), + AUTH_EE_GOOGLE_CLIENT_ID: z.string().optional(), AUTH_EE_GOOGLE_CLIENT_SECRET: z.string().optional(), + AUTH_EE_OKTA_CLIENT_ID: z.string().optional(), AUTH_EE_OKTA_CLIENT_SECRET: z.string().optional(), AUTH_EE_OKTA_ISSUER: z.string().optional(), + AUTH_EE_KEYCLOAK_CLIENT_ID: z.string().optional(), AUTH_EE_KEYCLOAK_CLIENT_SECRET: z.string().optional(), AUTH_EE_KEYCLOAK_ISSUER: z.string().optional(), + AUTH_EE_MICROSOFT_ENTRA_ID_CLIENT_ID: z.string().optional(), AUTH_EE_MICROSOFT_ENTRA_ID_CLIENT_SECRET: z.string().optional(), AUTH_EE_MICROSOFT_ENTRA_ID_ISSUER: z.string().optional(), diff --git a/packages/web/src/lib/newsData.ts b/packages/web/src/lib/newsData.ts index bca3a678b..c67b8b5d2 100644 --- a/packages/web/src/lib/newsData.ts +++ b/packages/web/src/lib/newsData.ts @@ -5,18 +5,18 @@ export const newsData: NewsItem[] = [ unique_id: "code-nav", header: "Code navigation", sub_header: "Built in go-to definition and find references", - url: "https://docs.sourcebot.dev/docs/search/code-navigation" + url: "https://docs.sourcebot.dev/docs/features/code-navigation" }, { unique_id: "sso", header: "SSO", sub_header: "We've added support for SSO providers", - url: "https://docs.sourcebot.dev/docs/configuration/authentication", + url: "https://docs.sourcebot.dev/docs/configuration/auth/overview", }, { unique_id: "search-contexts", header: "Search contexts", sub_header: "Filter searches by groups of repos", - url: "https://docs.sourcebot.dev/docs/search/search-contexts" + url: "https://docs.sourcebot.dev/docs/features/search/search-contexts" } ]; \ No newline at end of file diff --git a/schemas/v3/index.json b/schemas/v3/index.json index ed6f182b4..655a44661 100644 --- a/schemas/v3/index.json +++ b/schemas/v3/index.json @@ -83,7 +83,7 @@ }, "contexts": { "type": "object", - "description": "[Sourcebot EE] Defines a collection of search contexts. This is only available in single-tenancy mode. See: https://docs.sourcebot.dev/docs/search/search-contexts", + "description": "[Sourcebot EE] Defines a collection of search contexts. This is only available in single-tenancy mode. See: https://docs.sourcebot.dev/docs/features/search/search-contexts", "patternProperties": { "^[a-zA-Z0-9_-]+$": { "$ref": "#/definitions/SearchContext" From 6df4b119b6a8e3e81ac6130f4fd46dffc41e1370 Mon Sep 17 00:00:00 2001 From: bkellam Date: Sat, 31 May 2025 16:59:04 -0700 Subject: [PATCH 10/16] fix --- .github/workflows/docs-broken-links.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/docs-broken-links.yml b/.github/workflows/docs-broken-links.yml index 3c9cdd835..4e410c496 100644 --- a/.github/workflows/docs-broken-links.yml +++ b/.github/workflows/docs-broken-links.yml @@ -22,4 +22,5 @@ jobs: run: npm i -g mintlify - name: Check for broken links + working-directory: docs run: mintlify broken-links From aa834a013a4fbb61ff4f71ed628735e007af554f Mon Sep 17 00:00:00 2001 From: bkellam Date: Sun, 1 Jun 2025 15:17:59 -0700 Subject: [PATCH 11/16] further wip on docs --- docs/docs/connections/overview.mdx | 66 ++++++++++++++++++++++++++++-- docs/docs/deployment-guide.mdx | 9 ---- docs/docs/license-key.mdx | 19 ++++++++- docs/docs/overview.mdx | 22 ++++++++-- 4 files changed, 99 insertions(+), 17 deletions(-) diff --git a/docs/docs/connections/overview.mdx b/docs/docs/connections/overview.mdx index 3a15ffa4a..1167fab56 100644 --- a/docs/docs/connections/overview.mdx +++ b/docs/docs/connections/overview.mdx @@ -4,11 +4,71 @@ sidebarTitle: Overview --- import SupportedPlatforms from '/snippets/platform-support.mdx' +import ConfigSchema from '/snippets/schemas/v3/index.schema.mdx' -{/* TODO: document the concept of connections, the JSON schema, etc. */} +A **connection** in Sourcebot represents a link to a code host (such as GitHub, GitLab, Bitbucket, etc.). Each connection defines how Sourcebot should authenticate and interact with a particular host, and which repositories to sync and index from that host. Connections are uniquely identified by their name. -### Supported code hosts +A JSON configuration file is used to specify connections. For example: + +```json +// Specifies two connections: +{ + "$schema": "https://raw.githubusercontent.com/sourcebot-dev/sourcebot/main/schemas/v3/index.json", + "connections": { + // 1. A connection to GitHub.com + "github-connection": { + "type": "github", + "repos": [ + "sourcebot-dev/sourcebot" + ], + "token": { + "env": "GITHUB_TOKEN" + } + }, + // 2. A self-hosted GitLab instance + "gitlab-connection": { + "type": "gitlab", + "url": "https://gitlab.example.com", + "groups": [ + "my-group", + "my-other-group/sub-group" + ], + "token": { + "env": "GITLAB_TOKEN" + } + } + } +} +``` + +Configuration files must conform to the [JSON schema](https://raw.githubusercontent.com/sourcebot-dev/sourcebot/main/schemas/v3/index.json). + +When running Sourcebot, this file must be mounted in a volume that is accessible to the container, with it's path specified in the `CONFIG_FILE` environment variable. For example: + +```bash +docker run \ + -v $(pwd)/config.json:/data/config.json \ + -e CONFIG_FILE=/data/config.json \ + ... \ # other config + ghcr.io/sourcebot-dev/sourcebot:latest +``` + +## Getting started +--- + +To get started, pick a platform below and follow the instructions to connect your code. -Missing your code host? [Submit a feature request on GitHub](https://github.com/sourcebot-dev/sourcebot/discussions/categories/ideas). \ No newline at end of file +Missing your code host? [Submit a feature request on GitHub](https://github.com/sourcebot-dev/sourcebot/discussions/categories/ideas). + + +## Schema reference +--- + + +[schemas/v3/index.json](https://github.com/sourcebot-dev/sourcebot/blob/main/schemas/v3/index.json) + + + + diff --git a/docs/docs/deployment-guide.mdx b/docs/docs/deployment-guide.mdx index 694a960eb..016a25cd6 100644 --- a/docs/docs/deployment-guide.mdx +++ b/docs/docs/deployment-guide.mdx @@ -86,15 +86,6 @@ Watch this 1:51 minute video to get a quick overview of how to deploy Sourcebot The first account which is registered on a fresh Sourcebot deployment is given the [owner role](/docs/configuration/auth/roles-and-permissions). - - - Sourcebot supports indexing public & private code on the following code hosts: - - - - Missing your code host? [Submit a feature request on GitHub](https://github.com/sourcebot-dev/sourcebot/discussions/categories/ideas). - - ## Next steps diff --git a/docs/docs/license-key.mdx b/docs/docs/license-key.mdx index 751291a84..f48d71674 100644 --- a/docs/docs/license-key.mdx +++ b/docs/docs/license-key.mdx @@ -7,10 +7,11 @@ sidebarTitle: License key If you'd like a trial license, [reach out](https://www.sourcebot.dev/contact) and we'll send one over within 24 hours -All core Sourcebot features are available in Sourcebot OSS (MIT Licensed). Some additional features require a license key. See the [pricing page](https://www.sourcebot.dev/pricing) for more details. +All core Sourcebot features are available in Sourcebot OSS (MIT Licensed) without any limits. Some additional features require a license key. See the [pricing page](https://www.sourcebot.dev/pricing) for more details. ## Activating a license key +--- After purchasing a license key, you can activate it by setting the `SOURCEBOT_EE_LICENSE_KEY` environment variable. @@ -21,6 +22,22 @@ docker run \ ghcr.io/sourcebot-dev/sourcebot:latest ``` +## Feature availability +--- + +| Feature | OSS | Licensed | +|:---------|:-----|:----------| +| [Search](/docs/features/search/overview) | ✅ | ✅ | +| [Full code host support](/docs/connections/overview) | ✅ | ✅ | +| [MCP Server](/docs/features/mcp-server) | ✅ | ✅ | +| [Agents](/docs/features/agents/overview) | ✅ | ✅ | +| [Login with credentials](/docs/configuration/auth/overview) | ✅ | ✅ | +| [Login with email codes](/docs/configuration/auth/overview) | ✅ | ✅ | +| [Login with SSO](/docs/configuration/auth/overview#enterprise-authentication-providers) | 🛑 | ✅ | +| [Code navigation](/docs/features/code-navigation) | 🛑 | ✅ | +| [Search contexts](/docs/features/search/search-contexts) | 🛑 | ✅ | + + ## Questions? If you have any questions regarding licensing, please [contact us](https://www.sourcebot.dev/contact). \ No newline at end of file diff --git a/docs/docs/overview.mdx b/docs/docs/overview.mdx index d6f1d4946..e23f4fe06 100644 --- a/docs/docs/overview.mdx +++ b/docs/docs/overview.mdx @@ -6,16 +6,21 @@ title: "Overview" - - **Open-source:** - - **Fast:** Search across millions of lines of code in seconds using Sourcebot's blazingly fast indexed search. - - **Scalable:** Sourcebot uses [trigram indexing](https://en.wikipedia.org/wiki/Trigram_search), allowing it to scale to massive codebases. - - **Self-hosted:** Sourcebot is designed to be easily self-hosted, allowing you to deploy it onto your own infrastructure, keeping your code private and secure. + - **Full-featured search:** Fast indexed-based search with regex support, filters, branch search, boolean logic, and more. + - **Self-hosted:** Ships as a single [docker container](https://github.com/sourcebot-dev/sourcebot/pkgs/container/sourcebot) that can be deployed anywhere. + - **Modern design:** Light/Dark mode, vim keybindings, keyboard shortcuts, syntax highlighting, etc. + - **Scalable:** Scales to millions of lines of code. + - **Open-source:** Core features are MIT licensed, no vendor lock-in. ## Features --- + + Find an overview of all Sourcebot features below. For details, see the individual documentation pages. + + ### Fast-indexed based search Search across millions of lines of code in seconds using Sourcebot's blazingly fast indexed search. Find exactly what you are looking for with regular expressions, search filters, boolean logic, and more. @@ -157,6 +162,15 @@ One of our design philosophies for Sourcebot is to keep our infrastructure [radi Sourcebot does not support horizontal scaling at this time, but it is on our roadmap. If this is something your team would be interested in, please contact us at [team@sourcebot.dev](mailto:team@sourcebot.dev). +## License key +--- + +Sourcebot's core features are available under an [MIT license](https://github.com/sourcebot-dev/sourcebot/blob/HEAD/LICENSE) without any limits. Some [additional features](/docs/license-key#feature-availability) such as SSO and code navigation require a [license key](/docs/license-key). + + + + + ## Telemetry --- From e24b7d951169f73b2d9f32808096e95f22247f69 Mon Sep 17 00:00:00 2001 From: msukkari Date: Mon, 2 Jun 2025 10:17:01 -0700 Subject: [PATCH 12/16] review feedback --- docs/docs/configuration/environment-variables.mdx | 2 +- docs/docs/connections/overview.mdx | 2 +- packages/logger/src/index.ts | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/docs/configuration/environment-variables.mdx b/docs/docs/configuration/environment-variables.mdx index da59c67f9..199a214d6 100644 --- a/docs/docs/configuration/environment-variables.mdx +++ b/docs/docs/configuration/environment-variables.mdx @@ -27,7 +27,7 @@ The following environment variables allow you to configure your Sourcebot deploy | `SMTP_CONNECTION_URL` | `-` |

The url to the SMTP service used for sending transactional emails. See [this doc](/docs/configuration/transactional-emails) for more info.

| | `SOURCEBOT_ENCRYPTION_KEY` | Automatically generated at startup if no value is provided. Generated using `openssl rand -base64 24` |

Used to encrypt connection secrets and generate API keys.

| | `SOURCEBOT_LOG_LEVEL` | `info` |

The Sourcebot logging level. Valid values are `debug`, `info`, `warn`, `error`, in order of severity.

| -| `SOURCEBOT_STRUCTURED_LOGGING_ENABLED` | `false` |

Enables/disable structured JSON logging. See [this doc](/docs/configuration/transactional-emails) for more info.

| +| `SOURCEBOT_STRUCTURED_LOGGING_ENABLED` | `false` |

Enables/disable structured JSON logging. See [this doc](/docs/configuration/structured-logging) for more info.

| | `SOURCEBOT_STRUCTURED_LOGGING_FILE` | - |

Optional file to log to if structured logging is enabled

| | `SOURCEBOT_TELEMETRY_DISABLED` | `false` |

Enables/disables telemetry collection in Sourcebot. See [this doc](/docs/overview.mdx#telemetry) for more info.

| | `TOTAL_MAX_MATCH_COUNT` | `100000` |

The maximum number of matches per query

| diff --git a/docs/docs/connections/overview.mdx b/docs/docs/connections/overview.mdx index 6cb894a21..e095b6d6f 100644 --- a/docs/docs/connections/overview.mdx +++ b/docs/docs/connections/overview.mdx @@ -43,7 +43,7 @@ A JSON configuration file is used to specify connections. For example: Configuration files must conform to the [JSON schema](#schema-reference). -When running Sourcebot, this file must be mounted in a volume that is accessible to the container, with it's path specified in the `CONFIG_PATH` environment variable. For example: +When running Sourcebot, this file must be mounted in a volume that is accessible to the container, with its path specified in the `CONFIG_PATH` environment variable. For example: ```bash docker run \ diff --git a/packages/logger/src/index.ts b/packages/logger/src/index.ts index 218f51924..d3998d2cf 100644 --- a/packages/logger/src/index.ts +++ b/packages/logger/src/index.ts @@ -21,12 +21,12 @@ const { combine, colorize, timestamp, prettyPrint, errors, printf, label: labelF const datadogFormat = format((info) => { info.status = info.level.toLowerCase(); info.service = info.label; - delete info.label; + info.label = undefined; const msg = info[MESSAGE as unknown as string] as string | undefined; if (msg) { info.message = msg; - delete info[MESSAGE as unknown as string]; + info[MESSAGE as unknown as string] = undefined; } return info; From 3e0d3b48ac4ce0bc692332fd6cc6d267903a92b8 Mon Sep 17 00:00:00 2001 From: msukkari Date: Mon, 2 Jun 2025 10:19:07 -0700 Subject: [PATCH 13/16] remove logger dep from mcp package --- packages/mcp/package.json | 1 - packages/mcp/src/client.ts | 5 +---- packages/mcp/src/index.ts | 9 +++------ 3 files changed, 4 insertions(+), 11 deletions(-) diff --git a/packages/mcp/package.json b/packages/mcp/package.json index a77dd1b16..fcd9fa201 100644 --- a/packages/mcp/package.json +++ b/packages/mcp/package.json @@ -18,7 +18,6 @@ }, "dependencies": { "@modelcontextprotocol/sdk": "^1.10.2", - "@sourcebot/logger": "workspace:*", "@t3-oss/env-core": "^0.13.4", "escape-string-regexp": "^5.0.0", "express": "^5.1.0", diff --git a/packages/mcp/src/client.ts b/packages/mcp/src/client.ts index df5d9973c..c7d7d230c 100644 --- a/packages/mcp/src/client.ts +++ b/packages/mcp/src/client.ts @@ -2,12 +2,9 @@ import { env } from './env.js'; import { listRepositoriesResponseSchema, searchResponseSchema, fileSourceResponseSchema } from './schemas.js'; import { FileSourceRequest, FileSourceResponse, ListRepositoriesResponse, SearchRequest, SearchResponse, ServiceError } from './types.js'; import { isServiceError } from './utils.js'; -import { createLogger } from "@sourcebot/logger"; - -const logger = createLogger('mcp-client'); export const search = async (request: SearchRequest): Promise => { - logger.debug(`Executing search request: ${JSON.stringify(request, null, 2)}`); + console.debug(`Executing search request: ${JSON.stringify(request, null, 2)}`); const result = await fetch(`${env.SOURCEBOT_HOST}/api/search`, { method: 'POST', headers: { diff --git a/packages/mcp/src/index.ts b/packages/mcp/src/index.ts index fda34e4de..8a15e93e3 100644 --- a/packages/mcp/src/index.ts +++ b/packages/mcp/src/index.ts @@ -9,9 +9,6 @@ import { listRepos, search, getFileSource } from './client.js'; import { env, numberSchema } from './env.js'; import { TextContent } from './types.js'; import { base64Decode, isServiceError } from './utils.js'; -import { createLogger } from "@sourcebot/logger"; - -const logger = createLogger('mcp-server'); // Create MCP server const server = new McpServer({ @@ -78,7 +75,7 @@ server.tool( query += ` case:no`; } - logger.debug(`Executing search request: ${query}`); + console.debug(`Executing search request: ${query}`); const response = await search({ query, @@ -218,10 +215,10 @@ server.tool( const runServer = async () => { const transport = new StdioServerTransport(); await server.connect(transport); - logger.info('Sourcebot MCP server ready'); + console.info('Sourcebot MCP server ready'); } runServer().catch((error) => { - logger.error('Failed to start MCP server:', error); + console.error('Failed to start MCP server:', error); process.exit(1); }); From 955fec72d26231a33172e70289fb7fb7bb8aafc2 Mon Sep 17 00:00:00 2001 From: msukkari Date: Mon, 2 Jun 2025 10:32:14 -0700 Subject: [PATCH 14/16] fix build errors --- packages/db/package.json | 4 ++-- yarn.lock | 2 ++ 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/packages/db/package.json b/packages/db/package.json index 2fa2e5304..8dc07d69d 100644 --- a/packages/db/package.json +++ b/packages/db/package.json @@ -25,8 +25,8 @@ }, "dependencies": { "@prisma/client": "6.2.1", + "@sourcebot/logger": "workspace:*", "@types/readline-sync": "^1.4.8", - "readline-sync": "^1.4.10", - "@sourcebot/logger": "workspace:*" + "readline-sync": "^1.4.10" } } diff --git a/yarn.lock b/yarn.lock index e225fee14..80a79b2ad 100644 --- a/yarn.lock +++ b/yarn.lock @@ -5814,6 +5814,7 @@ __metadata: resolution: "@sourcebot/db@workspace:packages/db" dependencies: "@prisma/client": "npm:6.2.1" + "@sourcebot/logger": "workspace:*" "@types/argparse": "npm:^2.0.16" "@types/readline-sync": "npm:^1.4.8" argparse: "npm:^2.0.1" @@ -5947,6 +5948,7 @@ __metadata: "@sourcebot/crypto": "workspace:*" "@sourcebot/db": "workspace:*" "@sourcebot/error": "workspace:*" + "@sourcebot/logger": "workspace:*" "@sourcebot/schemas": "workspace:*" "@ssddanbrown/codemirror-lang-twig": "npm:^1.0.0" "@stripe/react-stripe-js": "npm:^3.1.1" From a2e0efc3c69fc9692e8bd9ea31857f429d48d5b5 Mon Sep 17 00:00:00 2001 From: msukkari Date: Mon, 2 Jun 2025 11:08:31 -0700 Subject: [PATCH 15/16] add back auth_url warning --- docs/docs/configuration/auth/overview.mdx | 4 ++-- docs/docs/deployment-guide.mdx | 3 +++ 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/docs/configuration/auth/overview.mdx b/docs/docs/configuration/auth/overview.mdx index a64787629..c89299ba1 100644 --- a/docs/docs/configuration/auth/overview.mdx +++ b/docs/docs/configuration/auth/overview.mdx @@ -2,6 +2,8 @@ title: Overview --- +If you're deploying Sourcebot behind a domain, you must set the [AUTH_URL](/docs/configuration/environment-variables) environment variable. + Sourcebot has built-in authentication that gates access to your organization. OAuth, email codes, and email / password are supported. The first account that's registered on a Sourcebot deployment is made the owner. All other users who register must be [approved](/docs/configuration/auth/overview#approving-new-members) by the owner. @@ -40,8 +42,6 @@ See [transactional emails](/docs/configuration/transactional-emails) for more de ## Enterprise Authentication Providers -If you're deploying Sourcebot behind a domain, you must set the [AUTH_URL](/docs/configuration/environment-variables) environment variable to use these providers. - The following authentication providers require an [enterprise license](/docs/license-key) to be enabled. By default, a new user registering using these providers must have their join request accepted by the owner of the organization to join. To allow a user to join automatically when diff --git a/docs/docs/deployment-guide.mdx b/docs/docs/deployment-guide.mdx index 5c6c05a46..63395ab45 100644 --- a/docs/docs/deployment-guide.mdx +++ b/docs/docs/deployment-guide.mdx @@ -53,6 +53,9 @@ Watch this 1:51 minute video to get a quick overview of how to deploy Sourcebot + If you're deploying Sourcebot behind a domain, you must set the [AUTH_URL](/docs/configuration/environment-variables) environment variable. + + In the same directory as `config.json`, run the following command to start your instance: ``` bash From 6a99ecf8d36c5b7cc43ebd5c193dfa0f60cdd52f Mon Sep 17 00:00:00 2001 From: msukkari Date: Mon, 2 Jun 2025 11:10:58 -0700 Subject: [PATCH 16/16] fix sidebar title consistency --- docs/docs/configuration/auth/roles-and-permissions.mdx | 1 + docs/docs/configuration/structured-logging.mdx | 2 +- docs/docs/features/agents/review-agent.mdx | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/docs/configuration/auth/roles-and-permissions.mdx b/docs/docs/configuration/auth/roles-and-permissions.mdx index 43be9319e..b639b521b 100644 --- a/docs/docs/configuration/auth/roles-and-permissions.mdx +++ b/docs/docs/configuration/auth/roles-and-permissions.mdx @@ -1,5 +1,6 @@ --- title: Roles and Permissions +sidebarTitle: Roles and permissions --- Looking to sync permissions with your identify provider? We're working on it - [reach out](https://www.sourcebot.dev/contact) to us to learn more diff --git a/docs/docs/configuration/structured-logging.mdx b/docs/docs/configuration/structured-logging.mdx index ef0f6e414..65fd06a0b 100644 --- a/docs/docs/configuration/structured-logging.mdx +++ b/docs/docs/configuration/structured-logging.mdx @@ -1,5 +1,5 @@ --- -title: Structured Logging +title: Structured logging --- By default, Sourcebot will output logs to the console in a human readable format. If you'd like Sourcebot to output structured JSON logs, set the following env vars: diff --git a/docs/docs/features/agents/review-agent.mdx b/docs/docs/features/agents/review-agent.mdx index ae0b3d1ec..74582c537 100644 --- a/docs/docs/features/agents/review-agent.mdx +++ b/docs/docs/features/agents/review-agent.mdx @@ -1,6 +1,6 @@ --- title: AI Code Review Agent -sidebarTitle: AI Code Review Agent +sidebarTitle: AI code review agent ---