diff --git a/README.md b/README.md index 8a4bad0b0f..a81f0c6fa0 100644 --- a/README.md +++ b/README.md @@ -22,9 +22,9 @@ For getting started please check the [tutorial example](./tutorial). * [Debezium - Monitoring](./monitoring): Monitoring a Debezium instance * [Debezium - Auto-Creation of Topics](./topic-auto-create): Auto-creating Debezium change data topics * [Debezium - Outbox Pattern](./outbox): Implement the "outbox pattern", an approach for letting services communicate in an asynchronous and reliable fashion +* [Debezium Management Platform](./debezium-platform): How to use the debezium-platform to create a data pipeline to stream chnages from a PostgreSQL database to Apache Kafka broker in kubernetes cluster. * [Debezium - Saga Pattern](./saga): How to implement the [Saga pattern](https://microservices.io/patterns/data/saga.html) for realizing distributed transactions across multiple microservices * [Debezium - Testing](./testcontainers): How to implement an integration test for your CDC set-up using [Testcontainers](https://www.testcontainers.org/) -* [Debezium - User Interface](./ui-demo): How to deploy Debezium UI * [Debezium Connect - Using Kafka with SSL enabled](./kafka-ssl): How to configure Debezium Connect to use a Kafka instance with SSL enabled * [Debezium Server - Using Google Cloud Pub/Sub](./debezium-server/debezium-server-sink-pubsub): How to deploy [Debezium Server](https://debezium.io/documentation/reference/stable/operations/debezium-server.html) using Postgres, MongoDB, and MySQL as data sources and [Google Cloud Pub/Sub](https://cloud.google.com/pubsub/docs) as a destination * [Debezium Server - Using storage for offset and schema history](./debezium-server/debezium-server-mysql-redis-pubsub): How to deploy [Debezium Server](https://debezium.io/documentation/reference/stable/operations/debezium-server.html) using MySQL as data sources, Redis and storage for offset and schema history, and [Google Cloud Pub/Sub emulator](https://cloud.google.com/pubsub/docs) as a destination diff --git a/debezium-platform/postgresql-kafka-example/README.md b/debezium-platform/postgresql-kafka-example/README.md new file mode 100644 index 0000000000..69fedd3200 --- /dev/null +++ b/debezium-platform/postgresql-kafka-example/README.md @@ -0,0 +1,199 @@ +Using Debezium-platfrom to manage and stream changes +=== +This example will walk you through on how to use the Debezium Management Platform to manage and stream changes from a PostgreSQL database into Apache Kafka. + + +Preparing the Environment +--- +As the first step we will provision a local Kubernetes cluster using [minikube](https://minikube.sigs.k8s.io/docs/) and will install an ingress controller. For this example, considering a local setup, we will use the `/etc/hosts` to resolve the domain. +The following script, when executed, will use minikube to provision a local k8s cluster named `debezium` and will add the required ingress controllers. It will also update the `/etc/hosts` to add the domain url. + +```sh +./create-environment.sh +``` +> **_NOTE:_** +If you are using minikube on Mac, you need also to run the `minikube tunnel -p debezium` command. For more details see [this](https://minikube.sigs.k8s.io/docs/drivers/docker/#known-issues) and [this](https://stackoverflow.com/questions/70961901/ingress-with-minikube-working-differently-on-mac-vs-ubuntu-when-to-set-etc-host). + +Now that you have the required k8s environment setup, its time to fire the required infra for this example. As we will be using PostgreSQL database and the Apache Kafka broker as source and the destination for our pipeline. The following script will create a dedicated namespace `debezium-platform` and use it going forward for further installations of our example. It will also provision the PostgreSQL database and the Apache Kafka broker. + +```shell +./setup-infra.sh +``` + +you can check the required infra is up and running + +```shell + +$ kubectl get pods + +NAME READY STATUS RESTARTS AGE +dbz-kafka-dual-role-0 1/1 Running 0 98s +dbz-kafka-entity-operator-9f4d8fbc4-twq7j 1/2 Running 0 14s +postgresql-85cc668d48-pjn58 1/1 Running 0 4m2s +strimzi-cluster-operator-7dc6fbcbf5-h28dl 1/1 Running 0 3m59s + +``` + +Deploying Debezium Management Platform +--- +We will install debezium-platfrom platform through helm + +```shell +helm repo add debezium https://charts.debezium.io && +helm install debezium-platform debezium/debezium-platform --version 3.1.0-final --set database.enabled=true --set domain.url=platform.debezium.io + +``` + +- `domain.url` is the only required property; it is used as host in the Ingress definition. +- `database.enabled` property is used. This property helps to simplify deployment in testing environments by automatically deploying the PostgreSQL database that is required by the conductor service. When deploying in a production environment, do not enable automatic deployment of the PostgreSQL database. Instead, specify an existing database instance, by setting the database.name, database.host, and other properties required to connect to the database. + +```shell + +$ kubectl get pods + +NAME READY STATUS RESTARTS AGE +conductor-7c48c54c5c-rmjw9 1/1 Running 0 4m24s +dbz-kafka-dual-role-0 1/1 Running 0 6m7s +dbz-kafka-entity-operator-54dd7cc446-k8cfh 2/2 Running 0 5m9s +debezium-operator-666f7b44d9-6tf4n 1/1 Running 0 4m24s +postgres-69c4c64ff5-2tfmw 1/1 Running 0 4m24s +postgresql-85cc668d48-xtlsw 1/1 Running 0 8m12s +stage-6c64f68df6-cfhjs 1/1 Running 0 4m24s +strimzi-cluster-operator-7dc6fbcbf5-wkqgz 1/1 Running 0 8m9s + +``` + +After all pods are running you should access the Debezium-platform-stage(UI) from `http://platform.debezium.io/`, now you have completed the installing and running the debezium-platform part. + + +Using the debezium-platfrom-stage(UI) for setting up our data pipeline +--- +Now once you have running platfrom-stage(UI), we will create a data pipeline and all its resources i.e source, destination and transform(as per need) thru it. You will see different side navigation option to configure them. + +For this demo, see the connection properties you can use for each connector type as illustrated below: + +### Source + #### PostgreSQL + + ```shell + { + "name": "test-source", + "description": "postgreSQL database", + "type": "io.debezium.connector.postgresql.PostgresConnector", + "schema": "schema123", + "vaults": [], + "config": { + "topic.prefix": "inventory", + "database.port": 5432, + "database.user": "debezium", + "database.dbname": "debezium", + "database.hostname": "postgresql", + "database.password": "debezium", + "schema.include.list": "inventory" + } +} + + ``` + +![PostgreSQL Connnector](./resources/source.png) + + +### Destination + ```shell + { + "name": "test-destination", + "description": "Kafka destination", + "type": "kafka", + "schema": "schema123", + "vaults": [], + "config": { + "producer.key.serializer": "org.apache.kafka.common.serialization.StringSerializer", + "producer.value.serializer": "org.apache.kafka.common.serialization.StringSerializer", + "producer.bootstrap.servers": "dbz-kafka-kafka-bootstrap.debezium-platform:9092" + } +} + + ``` + + ![Kafka Connnector](./resources/destination.png) + +### Transform + +**Transform class**: o.debezium.transforms.ExtractNewRecordState +**Transform name**: Debezium marker +**Description**: Extract Debezium payload +**Adds the specified fields to the header if they exist**: db,table +**Adds the specified field(s) to the message if they exist.**: op +**Predicate type**: org.apache.kafka.connect.transforms.predicates.TopicNameMatches +**Pattern**: inventory.inventory.products + + ```shell + { + "config": { + "add.fields": "op", + "add.headers": "db,table" + }, + "description": "Extract Debezium payload", + "name": "Debezium marker", + "predicate": { + "config": { + "pattern": "inventory.inventory.products" + }, + "negate": false, + "type": "org.apache.kafka.connect.transforms.predicates.TopicNameMatches" + }, + "schema": "string", + "type": "io.debezium.transforms.ExtractNewRecordState", + "vaults": [] +} + + ``` + + ![ExtractNewRecordState](./resources/transform.png) + +### Pipeline + +Now that you have all the required resources setup, we can proceed on creating the final data pipeline. + +#### Pipeline designer + ![Pipeline Designer](./resources/pipeline_designer.png) + +#### Pipeline configuration + ![Pipeline Configuration](./resources/pipeline_configuration.png) + + +After creating the pipeline in the UI you can check the k8 pods and should see the pipeline pods `test-pipeline-*` + + ```shell +$ kubectl get pods +NAME READY STATUS RESTARTS AGE +conductor-7c48c54c5c-rmjw9 1/1 Running 0 9m51s +dbz-kafka-dual-role-0 1/1 Running 0 11m +dbz-kafka-entity-operator-54dd7cc446-k8cfh 2/2 Running 0 10m +debezium-operator-666f7b44d9-6tf4n 1/1 Running 0 9m51s +postgres-69c4c64ff5-2tfmw 1/1 Running 0 9m51s +postgresql-85cc668d48-xtlsw 1/1 Running 0 13m +stage-6c64f68df6-cfhjs 1/1 Running 0 9m51s +strimzi-cluster-operator-7dc6fbcbf5-wkqgz 1/1 Running 0 13m +test-pipeline-645bddd8df-86r4g 1/1 Running 0 116s + ``` + + + +Verifying Change Events +--- +You can verify that the data pipeline instance `test-pipeline` deployed in the previous section consumed all initial data from the database with the following command: + +```sh + +kubectl exec -n debezium-platform -it dbz-kafka-dual-role-0 -- ./bin/kafka-console-consumer.sh --bootstrap-server=localhost:9092 --topic inventory.inventory.products --from-beginning --max-messages 5 + +``` + +Cleanup +--- +To remove the Kubernetes environment used in this tutorial, execute the cleanup script: + +```sh +./clean-up.sh +``` \ No newline at end of file diff --git a/debezium-platform/postgresql-kafka-example/clean-up.sh b/debezium-platform/postgresql-kafka-example/clean-up.sh new file mode 100644 index 0000000000..6b68af3692 --- /dev/null +++ b/debezium-platform/postgresql-kafka-example/clean-up.sh @@ -0,0 +1,5 @@ +#! /usr/bin/env bash + +source env.sh + +minikube delete -p $CLUSTER \ No newline at end of file diff --git a/debezium-platform/postgresql-kafka-example/create-environment.sh b/debezium-platform/postgresql-kafka-example/create-environment.sh new file mode 100644 index 0000000000..b8d59231ee --- /dev/null +++ b/debezium-platform/postgresql-kafka-example/create-environment.sh @@ -0,0 +1,71 @@ +#! /usr/bin/env bash + +source env.sh + +# Update /etc/hosts to resolve the domain +echo ">>> Checking and updating /etc/hosts entry for platform domain..." + +IP=$(kubectl cluster-info | sed -n 's/.*https:\/\/\([0-9.]*\).*/\1/p' | head -n 1) +HOSTNAME=${DEBEZIUM_PLATFORM_DOMAIN:-platform.debezium.io} +EXISTING=$(grep "$HOSTNAME" /etc/hosts) + +if [ -z "$EXISTING" ]; then + echo "$IP $HOSTNAME" | sudo tee -a /etc/hosts + echo "Added new entry: $IP $HOSTNAME" +else + EXISTING_IP=$(echo "$EXISTING" | awk '{print $1}') + if [ "$EXISTING_IP" != "$IP" ]; then + echo "WARNING: $HOSTNAME is already associated with IP $EXISTING_IP" + echo "Current kubectl IP is $IP" + read -p "Do you want to update the entry? (y/n) " -r + if [[ $REPLY =~ ^[Yy]$ ]]; then + sudo sed -i "/$HOSTNAME/d" /etc/hosts + echo "$IP $HOSTNAME" | sudo tee -a /etc/hosts + echo "Updated hosts file with new IP" + else + echo "Hosts file not modified" + fi + else + echo "Entry for $HOSTNAME already exists with the same IP" + fi +fi + +echo ">>> Creating minikube cluster 'debezium'..." +minikube start -p $CLUSTER --addons ingress + +echo ">>> Waiting for minikube components to be ready..." +# Wait for all minikube components to show expected status +MAX_ATTEMPTS=30 +ATTEMPT=1 +while [[ $ATTEMPT -le $MAX_ATTEMPTS ]]; do + STATUS=$(minikube status --profile "$CLUSTER") + + if echo "$STATUS" | grep -q "host: Running" && + echo "$STATUS" | grep -q "kubelet: Running" && + echo "$STATUS" | grep -q "apiserver: Running" && + echo "$STATUS" | grep -q "kubeconfig: Configured"; then + echo ">>> Minikube components are ready" + break + else + echo "Waiting... ($ATTEMPT/$MAX_ATTEMPTS)" + sleep 5 + ((ATTEMPT++)) + fi +done + +if [[ $ATTEMPT -gt $MAX_ATTEMPTS ]]; then + echo "❌ Timed out waiting for minikube to be ready" + exit 1 +fi + +echo ">>> Waiting for Kubernetes environment to be ready..." +kubectl wait --for=condition=Ready nodes --all --timeout=300s + +echo ">>> Kubernetes environment is ready" + + +# Only run minikube tunnel on macOS +if [[ "$(uname)" == "Darwin" ]]; then + echo ">>> Starting 'minikube tunnel' in the background (macOS only)..." + minikube tunnel -p "$CLUSTER" +fi \ No newline at end of file diff --git a/debezium-platform/postgresql-kafka-example/destination-kafka/001_kafka.yml b/debezium-platform/postgresql-kafka-example/destination-kafka/001_kafka.yml new file mode 100644 index 0000000000..6fe30ba9ce --- /dev/null +++ b/debezium-platform/postgresql-kafka-example/destination-kafka/001_kafka.yml @@ -0,0 +1,48 @@ +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaNodePool +metadata: + name: dual-role + labels: + strimzi.io/cluster: dbz-kafka +spec: + replicas: 1 + roles: + - controller + - broker + storage: + type: jbod + volumes: + - id: 0 + type: ephemeral + kraftMetadata: shared +--- + +apiVersion: kafka.strimzi.io/v1beta2 +kind: Kafka +metadata: + name: dbz-kafka + annotations: + strimzi.io/node-pools: enabled + strimzi.io/kraft: enabled +spec: + kafka: + version: 3.8.0 + metadataVersion: 3.8-IV0 + listeners: + - name: plain + port: 9092 + type: internal + tls: false + - name: tls + port: 9093 + type: internal + tls: true + config: + offsets.topic.replication.factor: 1 + transaction.state.log.replication.factor: 1 + transaction.state.log.min.isr: 1 + default.replication.factor: 1 + min.insync.replicas: 1 + entityOperator: + topicOperator: {} + userOperator: {} \ No newline at end of file diff --git a/debezium-platform/postgresql-kafka-example/env.sh b/debezium-platform/postgresql-kafka-example/env.sh new file mode 100644 index 0000000000..ea3c21ad13 --- /dev/null +++ b/debezium-platform/postgresql-kafka-example/env.sh @@ -0,0 +1,4 @@ +CLUSTER=debezium +NAMESPACE=debezium-platform +DEBEZIUM_PLATFORM_DOMAIN=platform.debezium.io +TIMEOUT=300s \ No newline at end of file diff --git a/debezium-platform/postgresql-kafka-example/resources/destination.png b/debezium-platform/postgresql-kafka-example/resources/destination.png new file mode 100644 index 0000000000..e1cd8b071f Binary files /dev/null and b/debezium-platform/postgresql-kafka-example/resources/destination.png differ diff --git a/debezium-platform/postgresql-kafka-example/resources/pipeline_configuration.png b/debezium-platform/postgresql-kafka-example/resources/pipeline_configuration.png new file mode 100644 index 0000000000..0761a5d50b Binary files /dev/null and b/debezium-platform/postgresql-kafka-example/resources/pipeline_configuration.png differ diff --git a/debezium-platform/postgresql-kafka-example/resources/pipeline_designer.png b/debezium-platform/postgresql-kafka-example/resources/pipeline_designer.png new file mode 100644 index 0000000000..ec72d4ef48 Binary files /dev/null and b/debezium-platform/postgresql-kafka-example/resources/pipeline_designer.png differ diff --git a/debezium-platform/postgresql-kafka-example/resources/source.png b/debezium-platform/postgresql-kafka-example/resources/source.png new file mode 100644 index 0000000000..71f6501797 Binary files /dev/null and b/debezium-platform/postgresql-kafka-example/resources/source.png differ diff --git a/debezium-platform/postgresql-kafka-example/resources/transform.png b/debezium-platform/postgresql-kafka-example/resources/transform.png new file mode 100644 index 0000000000..137eb689b5 Binary files /dev/null and b/debezium-platform/postgresql-kafka-example/resources/transform.png differ diff --git a/debezium-platform/postgresql-kafka-example/setup-infra.sh b/debezium-platform/postgresql-kafka-example/setup-infra.sh new file mode 100644 index 0000000000..626564729e --- /dev/null +++ b/debezium-platform/postgresql-kafka-example/setup-infra.sh @@ -0,0 +1,48 @@ +#! /usr/bin/env bash + +source env.sh + +NAMESPACE="debezium-platform" + +echo ">>> Creating namespace: $NAMESPACE" +kubectl create ns "$NAMESPACE" || echo "Namespace $NAMESPACE already exists" + +echo ">>> Setting kubectl context to use namespace: $NAMESPACE" +kubectl config set-context --current --namespace="$NAMESPACE" + +echo ">>> Deploying PostgreSQL..." +kubectl create -f ./source-database/001_postgresql.yml -n $NAMESPACE & + +echo ">>> Deploying Strimzi Kafka Operator..." +( + helm repo add strimzi https://strimzi.io/charts/ + helm repo update strimzi + helm install strimzi-operator strimzi/strimzi-kafka-operator \ + --version 0.44.0 \ + --namespace "$NAMESPACE" + + echo ">>> Waiting for Strimzi operator pod to be ready..." + kubectl wait pod -n "$NAMESPACE" -l name=strimzi-cluster-operator \ + --for=condition=Ready --timeout=300s + + echo ">>> Applying Kafka cluster YAML..." + kubectl create -f ./destination-kafka/001_kafka.yml +) & + +# Wait for both background jobs to finish +wait + +echo ">>> Waiting for Database and Kafka to be ready" +echo ">>> Waiting for database..." +kubectl wait pod -n "$NAMESPACE" -l app=postgres \ + --for=condition=Ready --timeout=300s + +echo ">>> Waiting for Kafka broker pod to be ready 2..." +kubectl wait pod -n "$NAMESPACE" -l strimzi.io/name=dbz-kafka-kafka --for=condition=Ready --timeout=300s + +# Kafka entity operator pod +echo ">>> Waiting for Kafka broker pod to be ready 2..." +kubectl wait pod -n "$NAMESPACE" -l strimzi.io/name=dbz-kafka-entity-operator --for=condition=Ready --timeout=300s + + +echo "✅ Done: PostgreSQL and Kafka are up and running!" diff --git a/debezium-platform/postgresql-kafka-example/source-database/001_postgresql.yml b/debezium-platform/postgresql-kafka-example/source-database/001_postgresql.yml new file mode 100644 index 0000000000..e44d88d22d --- /dev/null +++ b/debezium-platform/postgresql-kafka-example/source-database/001_postgresql.yml @@ -0,0 +1,73 @@ +apiVersion: v1 +kind: Secret +metadata: + name: postgresql-credentials +type: opaque +data: + POSTGRES_DB: ZGViZXppdW0= + POSTGRES_USER: ZGViZXppdW0= + POSTGRES_PASSWORD: ZGViZXppdW0= +--- +kind: Deployment +apiVersion: apps/v1 +metadata: + name: postgresql + labels: + app: postgresql +spec: + replicas: 1 + selector: + matchLabels: + app: postgresql + deployment: postgresql + template: + metadata: + labels: + app: postgresql + deployment: postgresql + spec: + containers: + - resources: {} + name: postgresql + envFrom: + - secretRef: + name: postgresql-credentials + ports: + - containerPort: 5432 + protocol: TCP + imagePullPolicy: IfNotPresent + livenessProbe: + initialDelaySeconds: 30 + tcpSocket: + port: 5432 + timeoutSeconds: 1 + readinessProbe: + exec: + command: + - "/bin/sh" + - "-i" + - "-c" + - "PGPASSWORD=${POSTGRES_PASSWORD} /usr/bin/psql -w -U ${POSTGRES_USER} -d ${POSTGRES_DB} -c 'SELECT 1'" + initialDelaySeconds: 5 + timeoutSeconds: 1 + terminationMessagePolicy: File + terminationMessagePath: /dev/termination-log + image: quay.io/debezium/example-postgres:latest + restartPolicy: Always + terminationGracePeriodSeconds: 30 + dnsPolicy: ClusterFirst + strategy: + type: Recreate +--- +apiVersion: v1 +kind: Service +metadata: + name: postgresql +spec: + selector: + app: postgresql + deployment: postgresql + ports: + - name: db + port: 5432 + targetPort: 5432 diff --git a/ui-demo/README.md b/ui-demo/README.md deleted file mode 100644 index b0a1aeb6ea..0000000000 --- a/ui-demo/README.md +++ /dev/null @@ -1,109 +0,0 @@ -# Debezium UI Example - -This demo automatically deploys the topology of services as defined in the [Debezium Tutorial](../tutorial) along with Debezium UI. You can use the Debezium UI to view available connectors and create new connectors. - - -## Prerequisites - -The Debezium UI is only available with Debezium version 1.5 and above. - - -## Launching UI - -Launch all the required components for the topology as defined in [tutorial example](../tutorial) - -```shell -export DEBEZIUM_VERSION=2.1 -docker-compose up -d - -Creating db-mysql ... done -Creating db-pg ... done -Creating zookeeper ... done -Creating db-mongo ... done -Creating ui-demo_mongo-initializer_1 ... done -Creating kafka ... done -Creating connect ... done -Creating debezium-ui ... done - -``` - - -The Debezium UI will be available on http://localhost:8080 - -## Create connectors via UI - -You can create Postgres, Mongo DB, MySQL & SQL Server (coming soon) connectors using the Debezium UI. Create the connectors using the *Create connector* wizard in the Debezium UI. The first two steps of the wizard are mandatory (i.e. selecting the "Connector type" in step 1, then entering the basic connection "Properties" in step 2). Additional steps in the wizard allow you to configure a variety of optional properties for each connector type. - -For this demo, see the connection properties you can use for each connector type as illustrated below: -### MySQL -**Topic Prefix**: inventory -**Cluster ID**: 12345 -**Hostname**: db-mysql -**User**: debezium -**Password**: dbz -**Kafka broker addresses**: kafka:9092 -**Database schema history topic name**: dbhistory.inventory - -![MySQL Connnector](connMySQL.png) - -### PostgreSQL -**Topix prefix**: fulfillment -**Hostname**: db-pg -**User**: postgres -**Password**: postgres -**Database**:postgres - -![PostgreSQL Connector](connPostgres.png) - -### MongoDB -**Topic prefix**: inventory_mongo -**Hosts**: db-mongo:27017 -**User**: admin -**Password**: admin - -![MongoDB Connector](connMongo.png) - -## Command line clients -After the connectors are registered and running, you can interact via cli to query and make changes to the dbs - -### MySQL -To connect with MySQL via cli: -```shell -docker exec -it db-mysql bash -c 'mysql -u $MYSQL_USER -p$MYSQL_PASSWORD inventory' - -``` - -### PostgreSQL -To connect with PostgreSQL via cli: -```shell -docker exec -it db-pg bash -c 'psql -U $POSTGRES_USER $POSTGRES_PASSWORD' - -``` - -### MongoDB -To connect with MongoDB via cli: -```shell -mongo -u admin -p admin --authenticationDatabase admin localhost:37017/inventory - -``` - - -## Examine the change events -You can examine change events which occur due to database changes - -```shell -# Open in a new terminal -# Viewing the change events in the kafka container -docker exec -it kafka bash - -./bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic [TOPIC_NAME] --from-beginning - -``` - -## Stopping All Services -When finished with the demo, stop all of the services. - -```console -$ docker-compose down -``` - diff --git a/ui-demo/connMongo.png b/ui-demo/connMongo.png deleted file mode 100644 index d5d7fc9f18..0000000000 Binary files a/ui-demo/connMongo.png and /dev/null differ diff --git a/ui-demo/connMySQL.png b/ui-demo/connMySQL.png deleted file mode 100644 index 834516155f..0000000000 Binary files a/ui-demo/connMySQL.png and /dev/null differ diff --git a/ui-demo/connPostgres.png b/ui-demo/connPostgres.png deleted file mode 100644 index 4e7b1b070d..0000000000 Binary files a/ui-demo/connPostgres.png and /dev/null differ diff --git a/ui-demo/docker-compose.yaml b/ui-demo/docker-compose.yaml deleted file mode 100644 index 64769da091..0000000000 --- a/ui-demo/docker-compose.yaml +++ /dev/null @@ -1,93 +0,0 @@ -version: '3' -services: - zookeeper: - container_name: zookeeper - image: quay.io/debezium/zookeeper:${DEBEZIUM_VERSION} - networks: - - ui-network - kafka: - container_name: kafka - image: quay.io/debezium/kafka:${DEBEZIUM_VERSION} - ports: - - "9092:9092" - depends_on: - - zookeeper - environment: - - ZOOKEEPER_CONNECT=zookeeper:2181 - networks: - - ui-network - db-pg: - container_name: db-pg - image: quay.io/debezium/example-postgres:${DEBEZIUM_VERSION} - ports: - - "65432:5432" - environment: - - POSTGRES_USER=postgres - - POSTGRES_PASSWORD=postgres - networks: - - ui-network - db-mysql: - container_name: db-mysql - image: quay.io/debezium/example-mysql:${DEBEZIUM_VERSION} - ports: - - "63306:3306" - environment: - - MYSQL_ROOT_PASSWORD=debezium - - MYSQL_USER=mysqluser - - MYSQL_PASSWORD=mysql - networks: - - ui-network - db-mongo: - container_name: db-mongo - image: mongo:5.0 - command: ["mongod", "--replSet", "rs0", "--auth"] - ports: - - "37017:27017" - volumes: - - ./resources/initialize-mongo-single.js:/docker-entrypoint-initdb.d/initialize-mongo-single.js - environment: - - MONGO_INITDB_ROOT_USERNAME=admin - - MONGO_INITDB_ROOT_PASSWORD=admin - - MONGO_INITDB_DATABASE=admin - networks: - - ui-network - mongo-initializer: - image: mongo:5.0 - networks: - - ui-network - depends_on: - - db-mongo - command: bash -c 'sleep 20 && mongo -u admin -p admin --authenticationDatabase admin db-mongo:27017/inventory --eval "rs.initiate();"' - connect: - container_name: connect - image: quay.io/debezium/connect:nightly - ports: - - "8083:8083" - depends_on: - - kafka - - db-pg - environment: - - BOOTSTRAP_SERVERS=kafka:9092 - - GROUP_ID=1 - - CONFIG_STORAGE_TOPIC=my_connect_configs - - OFFSET_STORAGE_TOPIC=my_connect_offsets - - STATUS_STORAGE_TOPIC=my_connect_statuses - - ENABLE_DEBEZIUM_KC_REST_EXTENSION=true - - ENABLE_DEBEZIUM_SCRIPTING=true - - CONNECT_REST_EXTENSION_CLASSES=io.debezium.kcrestextension.DebeziumConnectRestExtension - networks: - - ui-network - debezium-ui: - container_name: debezium-ui - image: quay.io/debezium/debezium-ui:${DEBEZIUM_VERSION} - ports: - - "8080:8080" - environment: - - KAFKA_CONNECT_URIS=http://connect:8083 - depends_on: - - connect - networks: - - ui-network -networks: - ui-network: - external: false diff --git a/ui-demo/resources/initialize-mongo-single.js b/ui-demo/resources/initialize-mongo-single.js deleted file mode 100644 index 834c885b5e..0000000000 --- a/ui-demo/resources/initialize-mongo-single.js +++ /dev/null @@ -1,69 +0,0 @@ -let error = true - -let res = [ - db = db.getSiblingDB('inventory'), - - db.customers.drop(), - db.orders.drop(), - db.products.drop(), - - db = db.getSiblingDB('admin'), - - db.runCommand({ - createRole: "listDatabases", - privileges: [ - { resource: { cluster : true }, actions: ["listDatabases"]} - ], - roles: [] - }), - - db.createUser({ - user: 'debezium', - pwd: 'dbz', - roles: [ - { role: "readWrite", db: "inventory" }, - { role: "read", db: "local" }, - { role: "listDatabases", db: "admin" }, - { role: "read", db: "config" }, - { role: "read", db: "admin" } - ] - }), - - db = db.getSiblingDB('inventory'), - - db.products.insert([ - { _id : NumberLong("101"), name : 'scooter', description: 'Small 2-wheel scooter', weight : 3.14, quantity : NumberInt("3") }, - { _id : NumberLong("102"), name : 'car battery', description: '12V car battery', weight : 8.1, quantity : NumberInt("8") }, - { _id : NumberLong("103"), name : '12-pack drill bits', description: '12-pack of drill bits with sizes ranging from #40 to #3', weight : 0.8, quantity : NumberInt("18") }, - { _id : NumberLong("104"), name : 'hammer', description: "12oz carpenter's hammer", weight : 0.75, quantity : NumberInt("4") }, - { _id : NumberLong("105"), name : 'hammer', description: "14oz carpenter's hammer", weight : 0.875, quantity : NumberInt("5") }, - { _id : NumberLong("106"), name : 'hammer', description: "16oz carpenter's hammer", weight : 1.0, quantity : NumberInt("0") }, - { _id : NumberLong("107"), name : 'rocks', description: 'box of assorted rocks', weight : 5.3, quantity : NumberInt("44") }, - { _id : NumberLong("108"), name : 'jacket', description: 'water resistent black wind breaker', weight : 0.1, quantity : NumberInt("2") }, - { _id : NumberLong("109"), name : 'spare tire', description: '24 inch spare tire', weight : 22.2, quantity : NumberInt("5") } - ]), - - db.customers.insert([ - { _id : NumberLong("1001"), first_name : 'Sally', last_name : 'Thomas', email : 'sally.thomas@acme.com' }, - { _id : NumberLong("1002"), first_name : 'George', last_name : 'Bailey', email : 'gbailey@foobar.com' }, - { _id : NumberLong("1003"), first_name : 'Edward', last_name : 'Walker', email : 'ed@walker.com' }, - { _id : NumberLong("1004"), first_name : 'Anne', last_name : 'Kretchmar', email : 'annek@noanswer.org' } - ]), - - db.orders.insert([ - { _id : NumberLong("10001"), order_date : new ISODate("2016-01-16T00:00:00Z"), purchaser_id : NumberLong("1001"), quantity : NumberInt("1"), product_id : NumberLong("102") }, - { _id : NumberLong("10002"), order_date : new ISODate("2016-01-17T00:00:00Z"), purchaser_id : NumberLong("1002"), quantity : NumberInt("2"), product_id : NumberLong("105") }, - { _id : NumberLong("10003"), order_date : new ISODate("2016-02-19T00:00:00Z"), purchaser_id : NumberLong("1002"), quantity : NumberInt("2"), product_id : NumberLong("106") }, - { _id : NumberLong("10004"), order_date : new ISODate("2016-02-21T00:00:00Z"), purchaser_id : NumberLong("1003"), quantity : NumberInt("1"), product_id : NumberLong("107") } - ]), - - error = false -] - -printjson(res) - -if (error) { - print('Error, exiting!') - quit(1) -} -