AWS SDK v3 traces not showing

[haimprecise]

What version of OpenTelemetry are you using?

@opentelemetry/core: 1.12.0
@opentelemetry/api: 1.3.0
@opentelemetry/instrumentation-aws-sdk: 0.34.1

What version of Node are you using?

18.14.0-alpine

What did you do?

Updated from AWS SDK v2 to v3:

"@aws-sdk/client-s3": 3.321.1
"@aws-sdk/client-sns": 3.321.1
"@aws-sdk/client-sqs": 3.321.1

And deployed to our AKS v1.23

What did you expect to see?

Exported traces for AWS SQS messages

What did you see instead?

No traces at all

Additional context

When debugging, the patchV3 functions are not called at all.

[Flarna]

Are you using some bundler like esbuild/webpack (see here)?

Are you creating the AWS SDK instrumentation before the AWS SDK module is required?

Is your application using CJS or ESM modules?

[haimprecise]

Were are using CJS, but the OpenTelemtry is wrapped with a NestJS Module implementing its OnApplicationBootstrap, It is called once all modules have been initialized, but before listening for connections.

According to the opentelemetry-instrumentation docs, it seems like the mechanism won't kick in, am I right?

[Flarna]

Not an expert on NestJS but it sounds like thats to late. Most stable approach is to move OTel init into a dedicated file and preload it via -r command line argument.

[haimprecise]

Okay I will try call the instrumentation code before the app bootstrap.
I will share the results.

Thanks

[haimprecise]

I call the Opentelemtry code before my app in instantiated.

This is what I use:

this.otelExporter = new PrecisePrometheusExporter({
port: 8081,
});

  this.otelSDK = new NodeSDK({
    metricReader: this.otelExporter,
    spanProcessor: new BatchSpanProcessor(
      !!process.env.NODE_ENV &&
      process.env.NODE_ENV?.toUpperCase() !== "LOCAL"
        ? new JaegerExporter()
        : new ConsoleSpanExporter()
    ),
    contextManager: new AsyncLocalStorageContextManager(),
    textMapPropagator: new CompositePropagator({
      propagators: [
        new JaegerPropagator(),
        new W3CTraceContextPropagator(),
        new W3CBaggagePropagator(),
        new B3Propagator(),
        new B3Propagator({
          injectEncoding: B3InjectEncoding.MULTI_HEADER,
        }),
      ],
    }),
    instrumentations: [
      new HttpInstrumentation({
        ignoreIncomingPaths: ["/health", "/metrics", /^\/api.*$/],
        ignoreOutgoingUrls: [/^.*amazonaws\.com.*$/, /^.*unleash.*$/],
      }),
      new GrpcInstrumentation(),
      new RedisInstrumentation(),
      new ExpressInstrumentation(),
      new DnsInstrumentation(),
      new AwsInstrumentation({
        suppressInternalInstrumentation: true,
      }),
      new PrismaInstrumentation(),
      new NestInstrumentation(),
      new PinoInstrumentation(),
    ],
  });
  
  this.otelSDK.start();

I can verify with debugging that the instrumentation code is triggered before the AWS imports.
However, the patch functions are not called.

Here is my tsconfig.json:

  "compilerOptions": {
    "module": "commonjs",
    "declaration": true,
    "removeComments": true,
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "allowSyntheticDefaultImports": true,
    "target": "es2020",
    "sourceMap": true,
    "outDir": "./dist",
    "baseUrl": "./",
    "incremental": true,
    "allowJs": true,
    "esModuleInterop": true,
    "moduleResolution": "node"
  }
}```

and I'm also excluding the `node_modules` in the `tsconfig.build.json`.

Any ideas why it's not working?

[Flarna]

fyi @carolabadeer @blumamir - seems like instrumentation of AWS SDK has some issue here

[carolabadeer]

Can you try adding the AWSXRayPropagator to the list of propagators?

[haimprecise]

I've added them but no change.

The issue seems at the level of patching via middleware-stack, non of the patchV3 functions are triggered, although I start the NodeSDK prior to my app initialization.

[dyladan]

@carolabadeer I'm going to assign this to you since you're the code owner. Can you please categorize this as P1-P4 and I'll add the appropriate label.

[haimprecise]

Hey @carolabadeer,
Any updates? Can I assist?