diff --git a/services/dbm/Dockerfile b/services/dbm/Dockerfile index fc84a5f5..ab06bf9d 100644 --- a/services/dbm/Dockerfile +++ b/services/dbm/Dockerfile @@ -24,4 +24,4 @@ ENV FLASK_APP=dbm.py # Start the app using ddtrace so we have profiling and tracing ENTRYPOINT ["ddtrace-run"] -CMD gunicorn --bind 0.0.0.0:7578 dbm:app +CMD ["gunicorn", "--bind", "0.0.0.0:7595", "dbm:app"] diff --git a/services/dbm/README.md b/services/dbm/README.md index 9b811f3d..8c44069e 100644 --- a/services/dbm/README.md +++ b/services/dbm/README.md @@ -35,6 +35,8 @@ The database schema can be found in the `models.py` file with these models: ## Adding DBM to your project +### Docker Compose + To add this service to your project, add this definition to your docker-compose file: ```yaml @@ -56,6 +58,7 @@ dbm: - DD_LOGS_INJECTION=true - DD_PROFILING_ENABLED=true - DD_APPSEC_ENABLED=true + - DD_DBM_PROPAGATION_MODE=full volumes: - './services/dbm:/app' networks: @@ -79,14 +82,6 @@ postgres: - POSTGRES_HOST_AUTH_METHOD=trust - POSTGRES_USER - POSTGRES_PASSWORD - - DD_ENV=${DD_ENV-dev} - - DD_SERVICE=postgres - - DD_VERSION=${DD_VERSION_POSTGRES-15} - - DD_AGENT_HOST=dd-agent - - DD_DBM_PROPAGATION_MODE=full - - DD_LOGS_INJECTION=true - - DD_RUNTIME_METRICS_ENABLED=true - - DD_PROFILING_ENABLED=true labels: com.datadoghq.tags.env: '${DD_ENV-dev}' com.datadoghq.tags.service: 'postgres' @@ -148,3 +143,135 @@ Also update the Datadog agent service definition to include the following enviro DD_DBM_PROPAGATION_MODE=full ``` +#### Script files + +There are two shell script files in the `./scripts` directory. This need to be added to the lab files. Ensure they are executable `chmod +x `. + +These scripts use `psql` to query the database. They can be added as a cron job to continually run on the host. Use the following command to install the required programs. + +```bash +apt update +apt install -y postgresql-client cron +``` + +Add the script to cron: + +```bash +echo "* * * * * /root/dbm_query_one.sh > /dev/null 2>&1" |crontab - +(crontab -l;echo "*/2 * * * * /root/dbm_query_two.sh > /dev/null 2>&1") |crontab - +``` + +### Kubernetes + +The `dbm/k8s-manifest` directory is additive to the version at the root of the repo. These files will add the `store-dbm` service and the configurations needed to collect DBM data. + +In this scenario, Storedog and the Datadog Agent are expected to be in the same namespace. The steps below assume that Storedog will run in the `default` namespace. This simplifies Postgres log collection. + +The manifest YAML files use environment variables such as ${DD_ENV} and ${REGISTRY_URL}. These are substituted using envsubst during deployment. If you prefer, you can edit the YAMLs directly to hardcode these values. + +#### A note on log collection + +PostgreSQL default logging is to `stderr`, and logs do not include detailed information. Information on collecting more detailed logs are in the [DBM Postgres configuration](https://docs.datadoghq.com/database_monitoring/setup_postgres/selfhosted/?tab=postgres15#collecting-logs-optional) documentation. These more detailed logs are written to a file which is then collected by the Datadog Agent. Running the agent and Postgres in the same namespace allows for a volume to be shared between the two pods. Running Storedog in a different namespace would require a more complex configuration for shared volumes using NFS or cloud storage. This is beyond the scope of running the store-dbm service for our lab environments. + +#### Deploy the Datadog Operator + +1. Install the Datadog Operator with Helm: + +```bash +helm repo add datadog https://helm.datadoghq.com +helm repo update +helm install my-datadog-operator datadog/datadog-operator +``` + +2. Create a Kubernetes secret with your Datadog API and app keys: + +```bash +kubectl create secret generic datadog-secret --from-literal api-key=$DD_API_KEY --from-literal app-key=$DD_APP_KEY +``` + +#### Deploy Storedog with store-dbm + +These steps ensure that dependencies are applied first. For example the storage volumes are created before the pods that will use them. + +1. Begin by merging the files from `dbm/k8s-manifest` into the main manifests directory. Run the following command at the root of the registry: + +```bash +cp -a services/dbm/k8s-manifests/. k8s-manifests/ +``` + +2. In addition to the usual environment variables used with Storedog, set `DD_VERSION_DBM=1.0.0`: + +```bash +export DD_VERSION_DBM=1.0.0 +``` + +3. Create the `fake-traffic` namespace: + +```bash +kubectl create namespace fake-traffic +``` + +4. Apply secrets for each namespace: + +```bash +kubectl apply -n default -f k8s-manifests/storedog-app/secrets/shared-secrets.yaml +kubectl apply -n fake-traffic -f k8s-manifests/storedog-app/secrets/shared-secrets.yaml +``` + +5. Apply the storage provisioner and the ingress controller. + +```bash +kubectl apply -R -f k8s-manifests/cluster-setup/ +``` + +6. Apply the Datadog Agent manifest: + +> [!IMPORTANT] +> This assumes you've already installed the Datadog Operator and set the API key. + +```bash +envsubst '${DD_ENV}' < k8s-manifests/datadog/datadog-agent.yaml | kubectl apply -f - +``` + +> [!NOTE] +> Use `kubectl get pods` and wait till the agent is running before continuing. This will ensure that traces are collected from all services. + +7. Deploy Storedog application components based on dependency. + +> [!NOTE] +> Applying all manifests at once can cause delays in service start. This command will apply all Storedog manifests in the default namespace. +> +> ```bash +> for file in k8s-manifests/storedog-app/**/*.yaml; do envsubst < "$file" | kubectl apply -f -; done +> ``` + +8. Apply ConfigMaps: + +```bash +kubectl apply -R -f k8s-manifests/storedog-app/configmaps/ +``` + +9. Apply StatefulSets (Redis and Postgres): + +```bash +for file in k8s-manifests/storedog-app/statefulsets/*.yaml; do envsubst < "$file" | kubectl apply -f -; done +``` + +10. Apply Storedog services: + +```bash +for file in k8s-manifests/storedog-app/deployments/*.yaml; do envsubst < "$file" | kubectl apply -f -; done +``` + +11. Apply Ingress for external traffic: + +```bash +for file in k8s-manifests/storedog-app/ingress/*.yaml; do envsubst < "$file" | kubectl apply -f -; done +``` + +12. Apply the ConfigMap and services to generate fake traffic. + +```bash +kubectl apply -f k8s-manifests/storedog-app/configmaps/shared-config.yaml -n fake-traffic +for file in k8s-manifests/fake-traffic/*.yaml; do envsubst '${REGISTRY_URL} ${SD_TAG}' < "$file" | kubectl apply -n fake-traffic -f -; done +``` diff --git a/services/dbm/bootstrap.py b/services/dbm/bootstrap.py index 05d005e1..1cf06f46 100644 --- a/services/dbm/bootstrap.py +++ b/services/dbm/bootstrap.py @@ -1,4 +1,5 @@ from flask import Flask +from flask_cors import CORS from models import Items, Preorder_Items, db from faker import Faker import random @@ -13,6 +14,7 @@ def create_app(): """Create a Flask application""" app = Flask(__name__) + CORS(app) app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql://' + \ DB_USERNAME + ':' + DB_PASSWORD + '@' + DB_HOST + '/' + DB_USERNAME app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False diff --git a/services/dbm/k8s-manifests/cluster-setup/storage/shared-pg-logs-pvc.yaml b/services/dbm/k8s-manifests/cluster-setup/storage/shared-pg-logs-pvc.yaml new file mode 100644 index 00000000..45a9b76c --- /dev/null +++ b/services/dbm/k8s-manifests/cluster-setup/storage/shared-pg-logs-pvc.yaml @@ -0,0 +1,11 @@ +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: shared-pg-logs +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + storageClassName: standard diff --git a/services/dbm/k8s-manifests/datadog/datadog-agent.yaml b/services/dbm/k8s-manifests/datadog/datadog-agent.yaml new file mode 100644 index 00000000..69d11551 --- /dev/null +++ b/services/dbm/k8s-manifests/datadog/datadog-agent.yaml @@ -0,0 +1,117 @@ +apiVersion: datadoghq.com/v2alpha1 +kind: DatadogAgent +metadata: + name: datadog +spec: + global: + secretBackend: + command: "/readsecret_multiple_providers.sh" + enableGlobalPermissions: true + clusterName: storedog-k8s + site: datadoghq.com + kubelet: + tlsVerify: false + credentials: + apiSecret: + secretName: datadog-secret + keyName: api-key + appSecret: + secretName: datadog-secret + keyName: app-key + features: + logCollection: # Logs + enabled: true + containerCollectAll: true + clusterChecks: # Required for integrations + enabled: true + # Datadog security features + # cspm: # Cloud Security Posture Management + # enabled: true + # hostBenchmarks: + # enabled: true + # cws: # Cloud Workload Security + # enabled: true + # sbom: # Software Bill of Materials + # enabled: true + # containerImage: + # enabled: true + override: + nodeAgent: + containers: + agent: + volumeMounts: + - name: shared-pg-logs # For Postgres log collection + mountPath: /var/log/pg_log_shared + readOnly: true + env: + volumes: + - name: shared-pg-logs # For Postgres log collection + persistentVolumeClaim: + claimName: shared-pg-logs + extraConfd: + configDataMap: # Integration configurations + nginx_ingress_controller.yaml: |- # nginx-ingress-controller integration + ad_identifiers: + - controller + init_config: + instances: + - prometheus_url: http://%%host%%:10254/metrics + collect_nginx_histograms: true + logs: + - service: controller + source: nginx-ingress-controller + nginx.yaml: |- # nginx integration on the nginx-ingress-controller + ad_identifiers: + - controller + init_config: + instances: + - nginx_status_url: http://%%host%%:18080/nginx_status + clusterAgent: + extraConfd: + configDataMap: + postgres.yaml: |- # Postgres cluster check for DBM + cluster_check: true + init_config: + instances: + - dbm: true + host: postgres + port: 5432 + username: ENC[k8s_secret@default/storedog-secrets/POSTGRES_INTEGRATION_USER] + password: ENC[k8s_secret@default/storedog-secrets/POSTGRES_INTEGRATION_PASSWORD] + tags: + - env:${DD_ENV} + - service:store-db + relations: + - relation_name: advertisement + - relation_name: discount + - relation_name: items + - relation_name: preorder_items + - relation_name: influencer + query_samples: + enabled: true + explain_parameterized_queries: true + max_relations: 400 + collect_function_metrics: true + collection_interval: 1 + collect_schemas: + enabled: true + collect_settings: + enabled: true + - dbm: true + host: postgres + port: 5432 + username: ENC[k8s_secret@default/storedog-secrets/POSTGRES_INTEGRATION_USER] + password: ENC[k8s_secret@default/storedog-secrets/POSTGRES_INTEGRATION_PASSWORD] + dbname: storedog_db + relations: + - relation_regex: spree_.* + query_samples: + enabled: true + explain_parameterized_queries: true + max_relations: 400 + collect_function_metrics: true + collection_interval: 1 + collect_schemas: + enabled: true + collect_settings: + enabled: true diff --git a/services/dbm/k8s-manifests/fake-traffic/dbm-env-configmap.yaml b/services/dbm/k8s-manifests/fake-traffic/dbm-env-configmap.yaml new file mode 100644 index 00000000..22f916a5 --- /dev/null +++ b/services/dbm/k8s-manifests/fake-traffic/dbm-env-configmap.yaml @@ -0,0 +1,7 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: dbm-env + namespace: fake-traffic +data: + FAKETRAFFIC_POSTGRES_HOST: postgres.default.svc.cluster.local diff --git a/services/dbm/k8s-manifests/fake-traffic/dbm-query-one-cronjob.yaml b/services/dbm/k8s-manifests/fake-traffic/dbm-query-one-cronjob.yaml new file mode 100644 index 00000000..cb3da864 --- /dev/null +++ b/services/dbm/k8s-manifests/fake-traffic/dbm-query-one-cronjob.yaml @@ -0,0 +1,34 @@ +apiVersion: batch/v1 +kind: CronJob +metadata: + name: dbm-query-one + namespace: fake-traffic +spec: + schedule: "* * * * *" + jobTemplate: + spec: + template: + metadata: + annotations: + ad.datadoghq.com/dbm-query-one.logs: '[{"source": "postgresql", "service": "cronjob"}]' + spec: + containers: + - name: dbm-query-one + image: ${REGISTRY_URL}/postgres:${SD_TAG} + command: ["/bin/sh", "/scripts/dbm_query_one.sh"] + envFrom: + - configMapRef: + name: storedog-config + - configMapRef: + name: dbm-env + - secretRef: + name: storedog-secrets + volumeMounts: + - name: scripts + mountPath: /scripts + restartPolicy: OnFailure + volumes: + - name: scripts + configMap: + name: dbm-scripts + defaultMode: 0755 diff --git a/services/dbm/k8s-manifests/fake-traffic/dbm-query-two-cronjob.yaml b/services/dbm/k8s-manifests/fake-traffic/dbm-query-two-cronjob.yaml new file mode 100644 index 00000000..6697258f --- /dev/null +++ b/services/dbm/k8s-manifests/fake-traffic/dbm-query-two-cronjob.yaml @@ -0,0 +1,34 @@ +apiVersion: batch/v1 +kind: CronJob +metadata: + name: dbm-query-two + namespace: fake-traffic +spec: + schedule: "*/2 * * * *" + jobTemplate: + spec: + template: + metadata: + annotations: + ad.datadoghq.com/dbm-query-two.logs: '[{"source": "postgresql", "service": "cronjob"}]' + spec: + containers: + - name: dbm-query-two + image: ${REGISTRY_URL}/postgres:${SD_TAG} + command: ["/bin/sh", "/scripts/dbm_query_two.sh"] + envFrom: + - configMapRef: + name: storedog-config + - configMapRef: + name: dbm-env + - secretRef: + name: storedog-secrets + volumeMounts: + - name: scripts + mountPath: /scripts + restartPolicy: OnFailure + volumes: + - name: scripts + configMap: + name: dbm-scripts + defaultMode: 0755 diff --git a/services/dbm/k8s-manifests/fake-traffic/dbm-scripts-configmap.yaml b/services/dbm/k8s-manifests/fake-traffic/dbm-scripts-configmap.yaml new file mode 100644 index 00000000..f275c569 --- /dev/null +++ b/services/dbm/k8s-manifests/fake-traffic/dbm-scripts-configmap.yaml @@ -0,0 +1,47 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: dbm-scripts + namespace: fake-traffic +data: + dbm_query_one.sh: | + #!/bin/sh + set -f + QUERY="BEGIN; LOCK TABLE items IN EXCLUSIVE MODE;" + UPDATE_QUERY=$(cat < random() * 7000 LIMIT 1); + SELECT pg_sleep(1); + EOF + ) + for i in {1..15} + do + for j in {1..20} + do + QUERY="$QUERY$UPDATE_QUERY" + done + QUERY="$QUERY COMMIT;" + PGPASSWORD="$POSTGRES_PASSWORD" psql -U "$POSTGRES_USER" -d postgres -h "$FAKETRAFFIC_POSTGRES_HOST" -c "$QUERY" + QUERY="BEGIN; LOCK TABLE items IN EXCLUSIVE MODE;" # Reset for next iteration + done + dbm_query_two.sh: | + #!/bin/sh + set -f + QUERY="BEGIN; LOCK TABLE items IN EXCLUSIVE MODE;" + UPDATE_QUERY=$(cat < random() * 7000 ORDER BY RANDOM() LIMIT 1); + EOF + ) + for i in {1..15} + do + for j in {1..20} + do + QUERY="$QUERY$UPDATE_QUERY" + done + QUERY="$QUERY COMMIT;" + PGPASSWORD="$POSTGRES_PASSWORD" psql -U "$POSTGRES_USER" -d postgres -h "$FAKETRAFFIC_POSTGRES_HOST" -c "$QUERY" + QUERY="BEGIN; LOCK TABLE items IN EXCLUSIVE MODE;" # Reset for next iteration + done diff --git a/services/dbm/k8s-manifests/storedog-app/configmaps/feature-flags-config.yaml b/services/dbm/k8s-manifests/storedog-app/configmaps/feature-flags-config.yaml new file mode 100644 index 00000000..40bbf97b --- /dev/null +++ b/services/dbm/k8s-manifests/storedog-app/configmaps/feature-flags-config.yaml @@ -0,0 +1,35 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: feature-flags-config + namespace: default + labels: + app: frontend + managed-by: storedog + purpose: feature-flags + environment: development + tier: application +data: + featureFlags.config.json: | + [ + { + "id": "1", + "name": "dbm", + "active": true + }, + { + "id": "2", + "name": "error-tracking", + "active": false + }, + { + "id": "3", + "name": "api-errors", + "active": false + }, + { + "id": "4", + "name": "product-card-frustration", + "active": false + } + ] \ No newline at end of file diff --git a/services/dbm/k8s-manifests/storedog-app/deployments/dbm.yaml b/services/dbm/k8s-manifests/storedog-app/deployments/dbm.yaml new file mode 100644 index 00000000..90fa35cc --- /dev/null +++ b/services/dbm/k8s-manifests/storedog-app/deployments/dbm.yaml @@ -0,0 +1,84 @@ +apiVersion: v1 +kind: Service +metadata: + name: dbm +spec: + ports: + - port: 7595 + targetPort: 7595 + name: http + selector: + app: dbm +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: dbm +spec: + replicas: 1 + selector: + matchLabels: + app: dbm + template: + metadata: + labels: + app: dbm + annotations: + ad.datadoghq.com/dbm.logs: '[{"source": "python"}]' + spec: + volumes: + - name: app-volume + emptyDir: {} + - name: apmsocketpath + hostPath: + path: /var/run/datadog/ + initContainers: + - name: wait-for-db + image: busybox + command: ['sh', '-c', 'until nc -z postgres 5432; do echo waiting for postgres; sleep 2; done;'] + containers: + - name: dbm + image: ${REGISTRY_URL}/dbm:${SD_TAG} + ports: + - containerPort: 7595 + env: + - name: FLASK_APP + value: "dbm.py" + - name: FLASK_DEBUG + value: "0" + - name: POSTGRES_PASSWORD + valueFrom: + secretKeyRef: + name: storedog-secrets + key: POSTGRES_PASSWORD + - name: POSTGRES_USER + valueFrom: + configMapKeyRef: + name: storedog-config + key: POSTGRES_USER + - name: POSTGRES_HOST + value: postgres + - name: DD_ENV + value: ${DD_ENV} + - name: DD_SERVICE + value: storedog-dbm + - name: DD_VERSION + value: ${DD_VERSION_DBM} + - name: DD_DBM_PROPAGATION_MODE + value: full + - name: DD_LOGS_INJECTION + value: "true" + - name: DD_PROFILING_ENABLED + value: "true" + - name: DD_APPSEC_ENABLED + value: "true" + resources: + requests: + memory: "128Mi" + cpu: "100m" + limits: + memory: "256Mi" + cpu: "200m" + volumeMounts: + - name: apmsocketpath + mountPath: /var/run/datadog diff --git a/services/dbm/k8s-manifests/storedog-app/secrets/shared-secrets.yaml b/services/dbm/k8s-manifests/storedog-app/secrets/shared-secrets.yaml new file mode 100644 index 00000000..093bafe8 --- /dev/null +++ b/services/dbm/k8s-manifests/storedog-app/secrets/shared-secrets.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +kind: Secret +metadata: + name: storedog-secrets +type: Opaque +stringData: + POSTGRES_PASSWORD: postgres # Change this in production + DB_PASSWORD: postgres # Change this in production + POSTGRES_INTEGRATION_USER: datadog # For Datadog integration + POSTGRES_INTEGRATION_PASSWORD: datadog # For Datadog integration diff --git a/services/dbm/k8s-manifests/storedog-app/statefulsets/postgres.yaml b/services/dbm/k8s-manifests/storedog-app/statefulsets/postgres.yaml new file mode 100644 index 00000000..d7866592 --- /dev/null +++ b/services/dbm/k8s-manifests/storedog-app/statefulsets/postgres.yaml @@ -0,0 +1,79 @@ +apiVersion: v1 +kind: Service +metadata: + name: postgres +spec: + ports: + - port: 5432 + name: postgres + clusterIP: None + selector: + app: postgres +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: postgres + labels: + tags.datadoghq.com/env: ${DD_ENV} + tags.datadoghq.com/service: store-db + tags.datadoghq.com/version: "15.0" +spec: + serviceName: postgres + replicas: 1 + selector: + matchLabels: + app: postgres + template: + metadata: + labels: + app: postgres + tags.datadoghq.com/env: ${DD_ENV} + tags.datadoghq.com/service: store-db + tags.datadoghq.com/version: "15.0" + annotations: + ad.datadoghq.com/postgres.logs: | + [{ + "source": "postgresql", + "service": "store-db", + "auto_multi_line_detection": true, + "path": "/var/log/pg_log_shared/postgresql-*.json", + "type": "file" + }] + spec: + containers: + - name: postgres + image: ${REGISTRY_URL}/postgres:${SD_TAG} + ports: + - containerPort: 5432 + name: postgres + env: + - name: POSTGRES_HOST_AUTH_METHOD + value: trust + - name: POSTGRES_USER + valueFrom: + configMapKeyRef: + name: storedog-config + key: POSTGRES_USER + - name: POSTGRES_PASSWORD + valueFrom: + secretKeyRef: + name: storedog-secrets + key: POSTGRES_PASSWORD + volumeMounts: + - name: postgres-data + mountPath: /var/lib/postgresql/data + - name: shared-pg-logs + mountPath: /var/log/pg_log + volumes: + - name: shared-pg-logs + persistentVolumeClaim: + claimName: shared-pg-logs + volumeClaimTemplates: + - metadata: + name: postgres-data + spec: + accessModes: [ "ReadWriteOnce" ] + resources: + requests: + storage: 10Gi diff --git a/services/dbm/scripts/dbm_query_one.sh b/services/dbm/scripts/dbm_query_one.sh new file mode 100755 index 00000000..3fab264e --- /dev/null +++ b/services/dbm/scripts/dbm_query_one.sh @@ -0,0 +1,23 @@ +#!/usr/bin/bash + +# disable path expansion so we can use a literal * +set -f +QUERY="LOCK TABLE items IN EXCLUSIVE MODE;" +UPDATE_QUERY=$(cat < random() * 7000 LIMIT 1); + SELECT pg_sleep(1); +EOF +) + +# Loop for running the query +for i in {1..15} +do + # Loop for building the query + for j in {1..20} + do + QUERY+="${UPDATE_QUERY}" + done + psql -U postgres -d postgres -h postgres -c "$QUERY" +done diff --git a/services/dbm/scripts/dbm_query_two.sh b/services/dbm/scripts/dbm_query_two.sh new file mode 100755 index 00000000..a8764e03 --- /dev/null +++ b/services/dbm/scripts/dbm_query_two.sh @@ -0,0 +1,22 @@ +#!/usr/bin/bash + +# disable path expansion so we can use a literal * +set -f +QUERY="LOCK TABLE items IN EXCLUSIVE MODE;" +UPDATE_QUERY=$(cat < random() * 7000 ORDER BY RANDOM() LIMIT 1); +EOF +) + +# Loop for running the query +for i in {1..15} +do + # Loop for building the query + for j in {1..20} + do + QUERY+="${UPDATE_QUERY}" + done + psql -U postgres -d postgres -h localhost -c "$QUERY" +done