Data prepper pipelines (#105)

YANG-DB · julianocosta89 · cartersocha · web-flow · commit a092b0934f34 · 2023-11-20T14:21:22.000-08:00
* Initial setup of Observability OTEL demo using Opensearch as store * Initial setup of Observability OTEL demo using Opensearch as store Signed-off-by: YANGDB <yang.db.dev@gmail.com> * Initial setup of Observability OTEL demo using Opensearch as store Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add curl to the basic docker O/S Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add monitoring & opinionated assets for this demo Signed-off-by: YANGDB <yang.db.dev@gmail.com> * export F/E & Load-Generator port services Signed-off-by: YANGDB <yang.db.dev@gmail.com> * fix jaeger service name Signed-off-by: YANGDB <yang.db.dev@gmail.com> * adding nginx as frontend proxy, fluent-bit.conf as log emitter Signed-off-by: YANGDB <yang.db.dev@gmail.com> * adding nginx & fluent-bit docs Signed-off-by: YANGDB <yang.db.dev@gmail.com> * adding integrationbs docker-compose service for assets loading into opensearch / dashboards Signed-off-by: YANGDB <yang.db.dev@gmail.com> * adding datasource for query prometheus Signed-off-by: YANGDB <yang.db.dev@gmail.com> * adding architecture.md including images Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add images and service links Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add jaeger-agent container name Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add data-prepper support for traces ingestion & serviceMap creation Signed-off-by: YANGDB <yang.db.dev@gmail.com> * fix auth issues for data-prepper Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add documentation for different ingestion capabilities Signed-off-by: YANGDB <yang.db.dev@gmail.com> * Initial setup of Observability OTEL demo using Opensearch as store Signed-off-by: YANGDB <yang.db.dev@gmail.com> * Initial setup of Observability OTEL demo using Opensearch as store Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add curl to the basic docker O/S Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add monitoring & opinionated assets for this demo Signed-off-by: YANGDB <yang.db.dev@gmail.com> * [shippingservice] update rust version and dependencies (open-telemetry#865) * update rust version and dependencies * Changelog --------- Co-authored-by: Carter Socha <43380952+cartersocha@users.noreply.github.com> * [load generator] Bump loagen dependencies (open-telemetry#869) * Bump loagen dependencies * Changelog * [grafana] fix demo dashboard to be compatible with spanmetrics connector (open-telemetry#874) * [grafana] fix demo dashboard to be compatible with spanmetrics connector Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com> * add changelog entry Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com> * fix some naming Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com> --------- Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com> * export F/E & Load-Generator port services Signed-off-by: YANGDB <yang.db.dev@gmail.com> * [kafka] remove KRaft mode support workarounds (open-telemetry#880) as the latest confluentinc/cp-kafka:7.4.0-1-ubi8 already provides built-in support * enabling batch span processor metrics for quoteservice (open-telemetry#878) * enabling batch span processor metrics for quoteservice PHP's batch span processor can emit metrics. There is not yet an async implementation, so use react's event loop to periodically flush traces and metrics. Add a grafana chart for some of the batch span processor metrics: dropped, pending, processed, queued * update changelog * revert change in selected graph * Update dependencies --------- Co-authored-by: Juliano Costa <julianocosta89@outlook.com> * Generate random errors in cartservice (open-telemetry#824) * Generate random errors from cart service * Update migration script? * Update changelog * Update changelog * Add Copyright to FeatureFlagHelper.cs --------- Co-authored-by: Carter Socha <43380952+cartersocha@users.noreply.github.com> Co-authored-by: Juliano Costa <julianocosta89@outlook.com> * fix jaeger service name Signed-off-by: YANGDB <yang.db.dev@gmail.com> * adding nginx as frontend proxy, fluent-bit.conf as log emitter Signed-off-by: YANGDB <yang.db.dev@gmail.com> * adding nginx & fluent-bit docs Signed-off-by: YANGDB <yang.db.dev@gmail.com> * adding integrationbs docker-compose service for assets loading into opensearch / dashboards Signed-off-by: YANGDB <yang.db.dev@gmail.com> * [currencyservice] Fix OTel C++ build and update OTel version (open-telemetry#886) * Fix OTel C++ build and update OTel version * changelog * adding datasource for query prometheus Signed-off-by: YANGDB <yang.db.dev@gmail.com> * adding architecture.md including images Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add images and service links Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add jaeger-agent container name Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add data-prepper support for traces ingestion & serviceMap creation Signed-off-by: YANGDB <yang.db.dev@gmail.com> * fix auth issues for data-prepper Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add documentation for different ingestion capabilities Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add assets manager flast app for loading the integration assets Signed-off-by: YANGDB <yang.db.dev@gmail.com> * remove experimental assets mgr app Signed-off-by: YANGDB <yang.db.dev@gmail.com> * Opensearch otel demo observability tutorial (#5) * add tutorial for using observability and diagnostic of a real use case * adding otel landing page markdown dashboard * adding tutorial markdown dashboards and documents * adding Observability Introduction.md tutorial markdown dashboards and documents * add dependencies to the docker-compose.yml * exposing docker ports locally Signed-off-by: YANGDB <yang.db.dev@gmail.com> --------- Signed-off-by: YANGDB <yang.db.dev@gmail.com> * Add .whitesource configuration file * Update local-config.md Co-authored-by: Joshua Li <joshuali925@gmail.com> Signed-off-by: YANGDB <yang.db.dev@gmail.com> * Update README.md update fluent-bit README.md file reference to the config Signed-off-by: YANGDB <yang.db.dev@gmail.com> * chore(deps): update dependency com.fasterxml.jackson.core:jackson-databind to v2.13.4.1 (#4) Co-authored-by: mend-for-github.zerozr99.workers.dev[bot] <50673670+mend-for-github.zerozr99.workers.dev[bot]@users.noreply.github.com> * chore(deps): update dependency io.grpc:grpc-protobuf to v1.45.2 (#5) Co-authored-by: mend-for-github.zerozr99.workers.dev[bot] <50673670+mend-for-github.zerozr99.workers.dev[bot]@users.noreply.github.com> * chore(deps): update dependency org.apache.kafka:kafka-clients to v3.4.0 (#6) Co-authored-by: mend-for-github.zerozr99.workers.dev[bot] <50673670+mend-for-github.zerozr99.workers.dev[bot]@users.noreply.github.com> * chore(deps): update dependency requests to v2.31.0 (#7) Co-authored-by: mend-for-github.zerozr99.workers.dev[bot] <50673670+mend-for-github.zerozr99.workers.dev[bot]@users.noreply.github.com> * update url for the new opensearch otel demo repo update integration script to log into a file update the README.md for the otel-collector pipline descrition update ports for the otel-col health-check add memory_limiter / servicegraph processors for otel-col Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update url for the new opensearch otel demo repo update integration script to log into a file update the README.md for the otel-collector pipline descrition update ports for the otel-col health-check add memory_limiter / servicegraph processors for otel-col Signed-off-by: YANGDB <yang.db.dev@gmail.com> * Update docker-compose.yml fixing the issues with latest OS images that cause the demo to break Signed-off-by: Anirudha (Ani) Jadhav <anirudha@nyu.edu> * fix integrations script connections retry add integrations docker service dependencies on the O/S cluster nodes Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update bytes map to long and status to int (#11) Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * Update README.md update the ports according to the .env variables Signed-off-by: YANGDB <yang.db.dev@gmail.com> * Integration config parser support (#12) * update integration script to load all assets from data.ini file Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update url to support both text & keyword Signed-off-by: YANGDB <yang.db.dev@gmail.com> --------- Signed-off-by: YANGDB <yang.db.dev@gmail.com> * Initial setup of Observability OTEL demo using Opensearch as store Signed-off-by: YANGDB <yang.db.dev@gmail.com> * Initial setup of Observability OTEL demo using Opensearch as store Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add curl to the basic docker O/S Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add monitoring & opinionated assets for this demo Signed-off-by: YANGDB <yang.db.dev@gmail.com> * export F/E & Load-Generator port services Signed-off-by: YANGDB <yang.db.dev@gmail.com> * fix jaeger service name Signed-off-by: YANGDB <yang.db.dev@gmail.com> * adding nginx as frontend proxy, fluent-bit.conf as log emitter Signed-off-by: YANGDB <yang.db.dev@gmail.com> * adding nginx & fluent-bit docs Signed-off-by: YANGDB <yang.db.dev@gmail.com> * adding integrationbs docker-compose service for assets loading into opensearch / dashboards Signed-off-by: YANGDB <yang.db.dev@gmail.com> * adding datasource for query prometheus Signed-off-by: YANGDB <yang.db.dev@gmail.com> * adding architecture.md including images Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add images and service links Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add jaeger-agent container name Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add data-prepper support for traces ingestion & serviceMap creation Signed-off-by: YANGDB <yang.db.dev@gmail.com> * fix auth issues for data-prepper Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add documentation for different ingestion capabilities Signed-off-by: YANGDB <yang.db.dev@gmail.com> * Initial setup of Observability OTEL demo using Opensearch as store Signed-off-by: YANGDB <yang.db.dev@gmail.com> * Initial setup of Observability OTEL demo using Opensearch as store * Initial setup of Observability OTEL demo using Opensearch as store Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add monitoring & opinionated assets for this demo Signed-off-by: YANGDB <yang.db.dev@gmail.com> * export F/E & Load-Generator port services Signed-off-by: YANGDB <yang.db.dev@gmail.com> * fix jaeger service name Signed-off-by: YANGDB <yang.db.dev@gmail.com> * adding nginx as frontend proxy, fluent-bit.conf as log emitter Signed-off-by: YANGDB <yang.db.dev@gmail.com> * adding nginx & fluent-bit docs Signed-off-by: YANGDB <yang.db.dev@gmail.com> * adding integrationbs docker-compose service for assets loading into opensearch / dashboards Signed-off-by: YANGDB <yang.db.dev@gmail.com> * adding datasource for query prometheus Signed-off-by: YANGDB <yang.db.dev@gmail.com> * adding architecture.md including images Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add images and service links Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add jaeger-agent container name Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add data-prepper support for traces ingestion & serviceMap creation Signed-off-by: YANGDB <yang.db.dev@gmail.com> * fix auth issues for data-prepper Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add documentation for different ingestion capabilities Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add assets manager flast app for loading the integration assets Signed-off-by: YANGDB <yang.db.dev@gmail.com> * remove experimental assets mgr app Signed-off-by: YANGDB <yang.db.dev@gmail.com> * Opensearch otel demo observability tutorial (#5) * add tutorial for using observability and diagnostic of a real use case * adding otel landing page markdown dashboard * adding tutorial markdown dashboards and documents * adding Observability Introduction.md tutorial markdown dashboards and documents * add dependencies to the docker-compose.yml * exposing docker ports locally Signed-off-by: YANGDB <yang.db.dev@gmail.com> --------- Signed-off-by: YANGDB <yang.db.dev@gmail.com> * Update local-config.md Co-authored-by: Joshua Li <joshuali925@gmail.com> Signed-off-by: YANGDB <yang.db.dev@gmail.com> * Update README.md update fluent-bit README.md file reference to the config Signed-off-by: YANGDB <yang.db.dev@gmail.com> * Add .whitesource configuration file * chore(deps): update dependency com.fasterxml.jackson.core:jackson-databind to v2.13.4.1 (#4) Co-authored-by: mend-for-github.zerozr99.workers.dev[bot] <50673670+mend-for-github.zerozr99.workers.dev[bot]@users.noreply.github.com> * chore(deps): update dependency io.grpc:grpc-protobuf to v1.45.2 (#5) Co-authored-by: mend-for-github.zerozr99.workers.dev[bot] <50673670+mend-for-github.zerozr99.workers.dev[bot]@users.noreply.github.com> * chore(deps): update dependency org.apache.kafka:kafka-clients to v3.4.0 (#6) Co-authored-by: mend-for-github.zerozr99.workers.dev[bot] <50673670+mend-for-github.zerozr99.workers.dev[bot]@users.noreply.github.com> * chore(deps): update dependency requests to v2.31.0 (#7) Co-authored-by: mend-for-github.zerozr99.workers.dev[bot] <50673670+mend-for-github.zerozr99.workers.dev[bot]@users.noreply.github.com> * update url for the new opensearch otel demo repo update integration script to log into a file update the README.md for the otel-collector pipline descrition update ports for the otel-col health-check add memory_limiter / servicegraph processors for otel-col Signed-off-by: YANGDB <yang.db.dev@gmail.com> * Update docker-compose.yml fixing the issues with latest OS images that cause the demo to break Signed-off-by: Anirudha (Ani) Jadhav <anirudha@nyu.edu> * fix integrations script connections retry add integrations docker service dependencies on the O/S cluster nodes Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update bytes map to long and status to int (#11) Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * Update README.md update the ports according to the .env variables Signed-off-by: YANGDB <yang.db.dev@gmail.com> * Integration config parser support (#12) * update integration script to load all assets from data.ini file Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update url to support both text & keyword Signed-off-by: YANGDB <yang.db.dev@gmail.com> --------- Signed-off-by: YANGDB <yang.db.dev@gmail.com> * upstream updates Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update http.template mapping (#15) fix dependency in the docker-compose missing env var for cartservices add extra logging into the integration.py * adding standard opensearch github dev / maintenance spec: (#16) * adding standard opensearch github dev / maintenance spec: - CONTRIBUTING.md - MAINTAINERS.md - CODE_OF_CONDUCT.md ... Signed-off-by: YANGDB <yang.db.dev@gmail.com> --------- Signed-off-by: YANGDB <yang.db.dev@gmail.com> * fixing opensearch demo link for OpenTelemetry (#14) Signed-off-by: Mitchell Gale <Mitchell.Gale@improving.com> * upstream updates (#17) * Regenerate kubernetes manifest and add autogenerate comment (open-telemetry#909) Co-authored-by: Pierre Tessier <pierre@pierretessier.com> * [loadgenerator] run load through frontend proxy (Envoy) (open-telemetry#914) * Run load through frontendProxy Signed-off-by: Pierre Tessier <pierre@pierretessier.com> * Run load through frontendProxy Signed-off-by: Pierre Tessier <pierre@pierretessier.com> --------- Signed-off-by: Pierre Tessier <pierre@pierretessier.com> Co-authored-by: Austin Parker <austin@ap2.io> * Fix error sh in README.md (open-telemetry#926) --------- Signed-off-by: Pierre Tessier <pierre@pierretessier.com> Co-authored-by: isabel <isaschm@users.noreply.github.com> Co-authored-by: Pierre Tessier <pierre@pierretessier.com> Co-authored-by: Austin Parker <austin@ap2.io> Co-authored-by: Lancelot <lancel.huang.cs@gmail.com> * add untriaged gh workflow (#18) Signed-off-by: YANGDB <yang.db.dev@gmail.com> * [cartservice] Update .NET OTel to 1.5.0 (open-telemetry#935) (#23) * [cartservice] Update OTel .NET to 1.5.0 * [cartservice] Simplify resource configuration TelemetrySdk and EnvVariable are defaults * Update changelog Co-authored-by: Piotr Kiełkowicz <pkiekowicz@splunk.com> * add red metrics from spans (#21) * add spanmetrics namespace (span.metrics) add description for all metrics in prometheus Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add spanmetrics namespace (span.metrics) add description for all metrics in prometheus Signed-off-by: YANGDB <yang.db.dev@gmail.com> --------- Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add integration dependency on the catalog repository for getting all the schema related index and component templates mapping (#29) Signed-off-by: YANGDB <yang.db.dev@gmail.com> * fix data-prepper image to a stable version update fluent-bit.conf parser to emmit a different field name adding support for prometheus-opensearch exporter Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add error wrapping for integration script Signed-off-by: YANGDB <yang.db.dev@gmail.com> Signed-off-by: YANGDB <yang.db.dev@gmail.com> * Upstream changes needed merge (#41) * Commit generated Go code and bump dependencies (open-telemetry#946) * accountingservice: commit generated code bump deps * license-checker to ignore genproto * Update README.md * Add comment to tools.go * checkoutservice: commit generated code bump dependencies * productcatalogservice: commit generated code bump dependencies * Configure Renovate (open-telemetry#931) * Add renovate.json * only update otel deps * update renovate config * update renovate config --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: austinlparker <austin@ap2.io> Co-authored-by: Carter Socha <43380952+cartersocha@users.noreply.github.com> * [cartservice] code cleanup (open-telemetry#943) * File scope namespaces * Sort modifiers * Remove redundant type declaration * Avoid hiding variables * join declaration and assignment * Use standard .NET convention for fields and consts * inline out variable * object initializer * collection initializer * drop unused using * drop unused parameters * remove redundant field initializer --------- Co-authored-by: Austin Parker <austin@ap2.io> * Add Axiom to list of vendors (open-telemetry#963) We would very much like to add [Axiom](axiom.co) to the list of vendors in the README. I linked a public demo dashboard that is feed by the demo. --------- Signed-off-by: YANGDB <yang.db.dev@gmail.com> Co-authored-by: Robert Pająk <pellared@hotmail.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: austinlparker <austin@ap2.io> Co-authored-by: Carter Socha <43380952+cartersocha@users.noreply.github.com> Co-authored-by: Piotr Kiełkowicz <pkiekowicz@splunk.com> Co-authored-by: Lukas Malkmus <mail@lukasmalkmus.com> * Update dependency io.grpc:grpc-netty to v1.50.3 (#47) Co-authored-by: mend-for-github.zerozr99.workers.dev[bot] <50673670+mend-for-github.zerozr99.workers.dev[bot]@users.noreply.github.com> * upstream updates (#52) Signed-off-by: YANGDB <yang.db.dev@gmail.com> * Update frauddetectionservice (#55) Co-authored-by: mend-for-github.zerozr99.workers.dev[bot] <50673670+mend-for-github.zerozr99.workers.dev[bot]@users.noreply.github.com> * Update adservice (#48) Co-authored-by: mend-for-github.zerozr99.workers.dev[bot] <50673670+mend-for-github.zerozr99.workers.dev[bot]@users.noreply.github.com> * update upstream changes Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update upstream changes Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update opensearch for 2.9 Signed-off-by: YANGDB <yang.db.dev@gmail.com> * fix: Fixing tracetest configuration (#65) * fix: Fixing tracetest configuration * fix: Fixing tracetest configuration Signed-off-by: Oscar Reyes <oscar-rreyes1@hotmail.com> --------- Signed-off-by: Oscar Reyes <oscar-rreyes1@hotmail.com> * chore(deps): update loadgenerator (#67) Co-authored-by: mend-for-github.zerozr99.workers.dev[bot] <50673670+mend-for-github.zerozr99.workers.dev[bot]@users.noreply.github.com> * update upstream changes - remove 'opensearch integration' service - update the opensearch dashboard - add opensearch+dashboard to the minimized docker-compose.yml Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update upstream changes - remove 'opensearch integration' service - update the opensearch dashboard - add opensearch+dashboard to the minimized docker-compose.yml Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update upstream changes - remove 'opensearch integration' service - update the opensearch dashboard - add opensearch+dashboard to the minimized docker-compose.yml - update opensearch to use a single node Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update upstream changes - remove 'opensearch integration' service - update the opensearch dashboard - add opensearch+dashboard to the minimized docker-compose.yml - update opensearch to use a single node Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update upstream changes - remove 'opensearch integration' service - update the opensearch dashboard - add opensearch+dashboard to the minimized docker-compose.yml - update opensearch to use a single node Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update upstream changes - remove 'opensearch integration' service - update the opensearch dashboard - add opensearch+dashboard to the minimized docker-compose.yml - update opensearch to use a single node Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update upstream changes - remove 'opensearch integration' service - update the opensearch dashboard - add opensearch+dashboard to the minimized docker-compose.yml - update opensearch to use a single node Signed-off-by: YANGDB <yang.db.dev@gmail.com> * remove https in favor of http Signed-off-by: YANGDB <yang.db.dev@gmail.com> * restore https Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update data-prepper config files Signed-off-by: YANGDB <yang.db.dev@gmail.com> * fix opensearch service name for jaeger-query Signed-off-by: YANGDB <yang.db.dev@gmail.com> * remove jaeger-query Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update data-prepper pipelines Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update data-prepper pipelines Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update data-prepper pipelines Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update data-prepper pipelines Signed-off-by: YANGDB <yang.db.dev@gmail.com> --------- Signed-off-by: YANGDB <yang.db.dev@gmail.com> Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com> Signed-off-by: Anirudha (Ani) Jadhav <anirudha@nyu.edu> Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> Signed-off-by: Mitchell Gale <Mitchell.Gale@improving.com> Signed-off-by: Pierre Tessier <pierre@pierretessier.com> Signed-off-by: Oscar Reyes <oscar-rreyes1@hotmail.com> Co-authored-by: Juliano Costa <julianocosta89@outlook.com> Co-authored-by: Carter Socha <43380952+cartersocha@users.noreply.github.com> Co-authored-by: Ziqi Zhao <zhaoziqi9146@gmail.com> Co-authored-by: Petr Styblo <styblope@gmail.com> Co-authored-by: Brett McBride <brett@deakin.edu.au> Co-authored-by: Alan West <3676547+alanwest@users.noreply.github.com> Co-authored-by: mend-for-github.zerozr99.workers.dev[bot] <50673670+mend-for-github.zerozr99.workers.dev[bot]@users.noreply.github.com> Co-authored-by: Joshua Li <joshuali925@gmail.com> Co-authored-by: Anirudha (Ani) Jadhav <anirudha@nyu.edu> Co-authored-by: Shenoy Pratik <sgguruda@amazon.com> Co-authored-by: Mitchell Gale <Mitchell.gale@improving.com> Co-authored-by: isabel <isaschm@users.noreply.github.com> Co-authored-by: Pierre Tessier <pierre@pierretessier.com> Co-authored-by: Austin Parker <austin@ap2.io> Co-authored-by: Lancelot <lancel.huang.cs@gmail.com> Co-authored-by: Piotr Kiełkowicz <pkiekowicz@splunk.com> Co-authored-by: Robert Pająk <pellared@hotmail.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Lukas Malkmus <mail@lukasmalkmus.com> Co-authored-by: Oscar Reyes <oscar-rreyes1@hotmail.com>
diff --git a/docker-compose.yml b/docker-compose.yml
@@ -232,15 +232,7 @@ services:
       - OTEL_SERVICE_NAME=featureflagservice
       - DATABASE_URL=ecto://ffs:ffs@ffs_postgres:5432/ffs
     healthcheck:
-      test:
-        [
-          "CMD",
-          "curl",
-          "-H",
-          "baggage: synthetic_request=true",
-          "-f",
-          "http://localhost:${FEATURE_FLAG_SERVICE_PORT}",
-        ]
+      test: ["CMD", "curl", "-H", "baggage: synthetic_request=true", "-f", "http://localhost:${FEATURE_FLAG_SERVICE_PORT}"]
     depends_on:
       ffs_postgres:
         condition: service_healthy
@@ -455,7 +447,7 @@ services:
     deploy:
       resources:
         limits:
-          memory: 500M # This is high to enable supporting the recommendationCache feature flag use case
+          memory: 500M               # This is high to enable supporting the recommendationCache feature flag use case
     restart: unless-stopped
     ports:
       - "${RECOMMENDATION_SERVICE_PORT}"
@@ -773,7 +765,6 @@ services:
     container_name: dataprepper
     volumes:
       - /data/service-map/
-      - ./src/dataprepper/trace_analytics_no_ssl_2x.yml:/usr/share/data-prepper/pipelines/trace_analytics_no_ssl_2x.yml
       - ./src/dataprepper/pipelines.yaml:/usr/share/data-prepper/pipelines/pipelines.yaml
       - ./src/dataprepper/data-prepper-config.yaml:/usr/share/data-prepper/config/data-prepper-config.yaml
     ports:
diff --git a/src/dataprepper/pipelines.yaml b/src/dataprepper/pipelines.yaml
@@ -15,3 +15,41 @@ demo-pipeline:
         username: admin
         password: admin
         index: otel
+
+entry-pipeline:
+  delay: "100"
+  source:
+    otel_trace_source:
+      ssl: false
+  sink:
+    - pipeline:
+        name: "raw-pipeline"
+    - pipeline:
+        name: "service-map-pipeline"
+raw-pipeline:
+  source:
+    pipeline:
+      name: "entry-pipeline"
+  processor:
+    - otel_trace_raw:
+  sink:
+    - opensearch:
+        hosts: [ "https://opensearch:9200" ]
+        insecure: true        
+        username: "admin"
+        password: "admin"
+        index_type: trace-analytics-raw
+service-map-pipeline:
+  delay: "100"
+  source:
+    pipeline:
+      name: "entry-pipeline"
+  processor:
+    - service_map_stateful:
+  sink:
+    - opensearch:
+        hosts: ["https://opensearch:9200"]
+        insecure: true
+        username: "admin"
+        password: "admin"
+        index_type: trace-analytics-service-map
