usage reporting: add sendTraces, rename sendErrorsInTraces, etc

- We've decided to leave the usage reporting defaults in AS4 as "field
  level instrumentation on for all operations, send some operations as
  traces". But we'd like to at least provide a non-experimental
  mechanism for entirely disabling sending traces to Studio. So there's
  a new ApolloServerPluginUsageReporting: `sendTraces: false`.
- Rename the (new in AS4) usage reporting option `sendErrorsInTraces` to
  `sendErrors`, because it does also affect error statistics in stats
  reports if your `transform` function returns `null`.
- Remove Apollo-internal `internal_includeTracesContributingToStats`
  option. This enabled some internal consistency monitoring which we are
  no longer paying attention to.
- If you enable `debugPrintReports`, send the reports as `info` rather
  than `warn`, which primarily lets us delete some large comments (and
  makes sense because these debug messages are not warnings).

Fixes #6051. Fixes #6078.

Author: glasser

.changeset/healthy-glasses-reflect.md

@@ -0,0 +1,5 @@
+---
+"@apollo/server": patch
+---
+
+New usage reporting option `sendTraces: false` to only send usage reports as aggregated statistics, not per-request traces.

.changeset/many-plums-smile.md

@@ -0,0 +1,5 @@
+---
+"@apollo/server": patch
+---
+
+Remove Apollo-internal `internal_includeTracesContributingToStats`. This should not have been used other than inside Apollo's own servers.

.changeset/short-lies-kneel.md

@@ -0,0 +1,5 @@
+---
+"@apollo/server": patch
+---
+
+The usage reporting option `debugPrintReports` now displays reports via `logger.info` rather than `logger.warn`.

.changeset/silly-icons-taste.md

@@ -0,0 +1,6 @@
+---
+"@apollo/server-integration-testsuite": patch
+"@apollo/server": patch
+---
+
+Rename usage reporting option `sendErrorsInTraces` (added in 4.0.0-alpha.4) to `sendErrors`, as it also affects error statistics outside of traces.

docs/source/data/errors.mdx

@@ -438,7 +438,7 @@ You can use Apollo Studio to analyze your server's error rates. By default, the
 
 If you _do_ want error information sent to Studio, you can send every error, or you can modify or redact specific errors before they're transmitted.
 
-To send all errors to Studio you can pass `{ unmodified: true }` to `sendErrorsInTraces`, like so:
+To send all errors to Studio you can pass `{ unmodified: true }` to `sendErrors`, like so:
 
 ```ts {7}
 new ApolloServer({
@@ -447,23 +447,23 @@ new ApolloServer({
     ApolloServerPluginUsageReporting({
       // If you pass unmodified: true to the usage reporting
       // plugin, Apollo Studio receives ALL error details
-      sendErrorsInTraces: { unmodified: true },
+      sendErrors: { unmodified: true },
     }),
   ],
 });
 ```
 
-If you want to report specific errors or modify an error before reporting it, you can pass a function to the `sendErrorsInTraces.transform` option, like so:
+If you want to report specific errors or modify an error before reporting it, you can pass a function to the `sendErrors.transform` option, like so:
 
 ```ts {4-6}
 new ApolloServer({
   // etc.
   plugins: [
     ApolloServerPluginUsageReporting({
-      sendErrorsInTraces: {
+      sendErrors: {
         transform: (err) => {
           if (err.extensions.code === 'MY_CUSTOM_CODE') {
-            // returning null will skip reporting this error 
+            // returning null will skip reporting this error
             return null;
           }
 
@@ -483,6 +483,8 @@ The function you pass to `transform` is called for each error (`GraphQLError`) t
 - Return a modified form of the error (e.g., by changing the `err.message` to remove potentially sensitive information)
 - Return `null` to prevent the error from being reported entirely
 
+Note that returning `null`` also affects Studio's aggregated statistics about how many operations contain errors and at what paths those errors appear.
+
 [As mentioned above](#for-client-responses), you can use the `unwrapResolverError` (from `@apollo/server/errors`) to remove the `GraphQLError` wrapping an original error.
 
 <!-- TODO(AS4) Replace link once plugin docs are updated -->
@@ -502,7 +504,7 @@ const server = new ApolloServer({
   plugins: [
     // highlight-start
     ApolloServerPluginUsageReporting({
-      sendErrorsInTraces: {
+      sendErrors: {
         transform: (err) => {
           // Return `null` to avoid reporting `UNAUTHENTICATED` errors
           if (err.extensions.code === 'UNAUTHENTICATED') {
@@ -536,7 +538,7 @@ const server = new ApolloServer({
   resolvers,
   plugins: [
     ApolloServerPluginUsageReporting({
-      sendErrorsInTraces: {
+      sendErrors: {
         transform: (err) => {
           // Using a more stable, known error extension (e.g. `err.code`) would be
           // more defensive, however checking the `message` might serve most needs!
@@ -565,7 +567,7 @@ If you _do_ want to send an error's details to Apollo Studio, but need to redact
 
 For example, if there is personally identifiable information in the error `message`, like an API key:
 
-```ts 
+```ts
 import { GraphQLError } from 'graphql';
 
 throw new GraphQLError(
@@ -584,7 +586,7 @@ const server = new ApolloServer({
   resolvers,
   plugins: [
     ApolloServerPluginUsageReporting({
-      sendErrorsInTraces: {
+      sendErrors: {
         transform: (err) => {
           // Make sure that a specific pattern is removed from all error messages.
           err.message = err.message.replace(/x-api-key:[A-Z0-9-]+/, 'REDACTED');

docs/source/migration.mdx

@@ -936,7 +936,7 @@ ApolloServerPluginUsageReporting({
 
 In Apollo Server 3, you can specify a function to rewrite errors before sending them to Apollo's server via the `rewriteError` option to `ApolloServerPluginUsageReporting` (for monoliths) and `ApolloServerPluginInlineTrace` (for subgraphs).
 
-In Apollo Server 4, you specify the same function as the `transform` option on the `sendErrorsInTraces` option to `ApolloServerPluginUsageReporting` and the `includeErrors` option to `ApolloServerPluginInlineTrace`.
+In Apollo Server 4, you specify the same function as the `transform` option on the `sendErrors` option to `ApolloServerPluginUsageReporting` and the `includeErrors` option to `ApolloServerPluginInlineTrace`.
 
 (Additionally, the [default behavior has changed to mask errors](#usage-reporting-and-inline-trace-plugins-mask-errors-by-default).)
 
@@ -962,7 +962,7 @@ you can now write:
 // monoliths
 new ApolloServer({
   plugins: [ApolloServerPluginUsageReporting({
-    sendErrorsInTraces: { transform: rewriteError },
+    sendErrors: { transform: rewriteError },
   })],
   // ...
 })
@@ -1599,7 +1599,7 @@ To restore the Apollo Server 3 behavior, you can pass `{ unmodified: true }` to
 // monoliths
 new ApolloServer({
   plugins: [ApolloServerPluginUsageReporting({
-    sendErrorsInTraces: { unmodified: true },
+    sendErrors: { unmodified: true },
   })],
   // ...
 })
@@ -1613,7 +1613,7 @@ new ApolloServer({
 })
 ```
 
-(As [described above](#rewriteerror-plugin-option), the `rewriteError` option has been replaced by a `transform` option on `sendErrorsInTraces` or `includeErrors`.)
+(As [described above](#rewriteerror-plugin-option), the `rewriteError` option has been replaced by a `transform` option on `sendErrors` or `includeErrors`.)
 
 ## Renamed packages
 

packages/integration-testsuite/src/apolloServerTests.ts

@@ -1139,14 +1139,14 @@ export function defineIntegrationTestSuiteApolloServerTests(
           });
 
           describe('error munging', () => {
-            describe('sendErrorsInTraces', () => {
+            describe('sendErrors', () => {
               it('new error', async () => {
                 throwError.mockImplementationOnce(() => {
                   throw new Error('transform nope');
                 });
 
                 await setupApolloServerAndFetchPair({
-                  sendErrorsInTraces: {
+                  sendErrors: {
                     transform: () =>
                       new GraphQLError('rewritten as a new error'),
                   },
@@ -1191,7 +1191,7 @@ export function defineIntegrationTestSuiteApolloServerTests(
                 });
 
                 await setupApolloServerAndFetchPair({
-                  sendErrorsInTraces: {
+                  sendErrors: {
                     transform: (err) => {
                       err.message = 'rewritten as a modified error';
                       return err;
@@ -1236,7 +1236,7 @@ export function defineIntegrationTestSuiteApolloServerTests(
                 });
 
                 await setupApolloServerAndFetchPair({
-                  sendErrorsInTraces: { transform: () => null },
+                  sendErrors: { transform: () => null },
                 });
 
                 const result = await apolloFetch({
@@ -1271,7 +1271,7 @@ export function defineIntegrationTestSuiteApolloServerTests(
               });
 
               await setupApolloServerAndFetchPair({
-                sendErrorsInTraces: {
+                sendErrors: {
                   // @ts-expect-error (not allowed to be undefined)
                   transform: () => undefined,
                 },
@@ -1319,7 +1319,7 @@ export function defineIntegrationTestSuiteApolloServerTests(
               });
 
               await setupApolloServerAndFetchPair({
-                sendErrorsInTraces: {
+                sendErrors: {
                   unmodified: true,
                 },
               });
@@ -1370,7 +1370,7 @@ export function defineIntegrationTestSuiteApolloServerTests(
               });
 
               await setupApolloServerAndFetchPair({
-                sendErrorsInTraces: {
+                sendErrors: {
                   masked: true,
                 },
               });

packages/server/src/__tests__/plugin/usageReporting/plugin.test.ts

@@ -174,6 +174,24 @@ describe('end-to-end', () => {
     ).toBeTruthy();
   });
 
+  it('sendTraces: false', async () => {
+    const { report } = await runTest({ pluginOptions: { sendTraces: false } });
+
+    expect(Object.keys(report.tracesPerQuery)).toHaveLength(1);
+    expect(Object.keys(report.tracesPerQuery)[0]).toMatch(/^# q\n/);
+    const tracesAndStats = Object.values(report.tracesPerQuery)[0]!;
+    expect(tracesAndStats.trace).toHaveLength(0);
+    expect(tracesAndStats.statsWithContext).toHaveLength(1);
+    const contextualizedStats = (
+      tracesAndStats.statsWithContext as ContextualizedStats[]
+    )[0]!;
+    expect(contextualizedStats.queryLatencyStats?.requestCount).toBe(1);
+    expect(
+      contextualizedStats.perTypeStat['User'].perFieldStat?.['name']
+        .observedExecutionCount,
+    ).toBe(1);
+  });
+
   [
     {
       testName: 'fails parse for non-parsable gql',

packages/server/src/plugin/usageReporting/options.ts

@@ -14,23 +14,63 @@ export interface ApolloServerPluginUsageReportingOptions<
 > {
   //#region Configure exactly which data should be sent to Apollo.
   /**
-   * By default, Apollo Server does not send the values of any GraphQL variables to Apollo's servers, because variable
-   * values often contain the private data of your app's users. If you'd like variable values to be included in traces, set this option.
-   * This option can take several forms:
+   * Apollo Server's usage reports describe each individual request in one of
+   * two ways: as a "trace" (a detailed description of the specific request,
+   * including a query plan and resolver tree with timings and errors, as well
+   * as optional details like variable values and HTTP headers), or as part of
+   * aggregated "stats" (where invocations of the same operation from the same
+   * client program are aggregated together rather than described individually).
+   * Apollo Server uses an heuristic to decide which operations to describe as
+   * traces and which to aggregate as stats.
+   *
+   * By setting the `sendTraces` option to `false`, Apollo Server will describe
+   * *all* operations as stats; individual requests will never be broken out
+   * into separate traces. If you set `sendTraces: false`, the Traces view in
+   * Apollo Studio will not show any traces, but all other Studio functionality
+   * should be unaffected.
+   *
+   * Note that the values of `sendVariableValues`, `sendHeaders`, and
+   *  `sendUnexecutableOperationDocuments` are irrelevant if you set
+   *  `sendTraces: false`, because those options control data that is contained
+   *  only in traces and not in stats.
+   *
+   * Setting `sendTraces: false` does *NOT* imply `fieldLevelInstrumentation:
+   * 0`. Apollo Server can still take advantage of field-level instrumentation
+   * (either directly for monolith servers, or via federated tracing for
+   * Gateways) in order to accurately report field execution usage in "stats".
+   * This option only controls whether data is sent to Apollo's servers as
+   * traces, not whether traces are internally used to learn about usage.
+   */
+  sendTraces?: boolean;
+
+  /**
+   * By default, Apollo Server does not send the values of any GraphQL variables
+   * to Apollo's servers, because variable values often contain the private data
+   * of your app's users. If you'd like variable values to be included in
+   * traces, set this option. This option can take several forms:
    * - { none: true }: don't send any variable values (DEFAULT)
    * - { all: true}: send all variable values
-   * - { transform: ... }: a custom function for modifying variable values. Keys added by the custom function will
-   *    be removed, and keys removed will be added back with an empty value. For security reasons, if an error occurs within this function, all variable values will be replaced with `[PREDICATE_FUNCTION_ERROR]`.
-   * - { exceptNames: ... }: a case-sensitive list of names of variables whose values should not be sent to Apollo servers
-   * - { onlyNames: ... }: A case-sensitive list of names of variables whose values will be sent to Apollo servers
+   * - { transform: ... }: a custom function for modifying variable values. Keys
+   *    added by the custom function will be removed, and keys removed will be
+   *    added back with an empty value. For security reasons, if an error occurs
+   *    within this function, all variable values will be replaced with
+   *    `[PREDICATE_FUNCTION_ERROR]`.
+   * - { exceptNames: ... }: a case-sensitive list of names of variables whose
+   *   values should not be sent to Apollo servers
+   * - { onlyNames: ... }: A case-sensitive list of names of variables whose
+   *   values will be sent to Apollo servers
+   *
+   * Defaults to not sending any variable values if both this parameter and the
+   * deprecated `privateVariables` are not set. The report will indicate each
+   * private variable key whose value was redacted by { none: true } or {
+   * exceptNames: [...] }.
    *
-   * Defaults to not sending any variable values if both this parameter and
-   * the deprecated `privateVariables` are not set. The report will
-   * indicate each private variable key whose value was redacted by { none: true } or { exceptNames: [...] }.
+   * The value of this option is not relevant if you set `sendTraces: false`,
+   * because variable values only appear in traces.
    */
   sendVariableValues?: VariableValueOptions;
   /**
-   * By default, Apollo Server does not send the list of HTTP headers and values
+   * By default, Apollo Server does not send the HTTP request headers and values
    * to Apollo's servers, as these headers may contain your users' private data.
    * If you'd like this information included in traces, set this option. This
    * option can take several forms:
@@ -44,6 +84,9 @@ export interface ApolloServerPluginUsageReportingOptions<
    *
    * Unlike with sendVariableValues, names of dropped headers are not reported.
    * The headers 'authorization', 'cookie', and 'set-cookie' are never reported.
+   *
+   * The value of this option is not relevant if you set `sendTraces: false`,
+   * because request headers only appear in traces.
    */
   sendHeaders?: SendValuesBaseOptions;
   /**
@@ -65,10 +108,12 @@ export interface ApolloServerPluginUsageReportingOptions<
    *   (either a new error, or its potentially-modified argument) or `null`.
    *   This error is used in the report to Apollo servers; if `null`, the error
    *   is not included in traces or error statistics.
+   *
+   * If you set `sendTraces: false`, then the only relevant aspect of this
+   * option is whether you return `null` from a `transform` function or not
+   * (which affects aggregated error statistics).
    */
-  sendErrorsInTraces?: SendErrorsOptions;
-
-  // We should strongly consider changing the default to false in AS4.
+  sendErrors?: SendErrorsOptions;
 
   /**
    * This option allows you to choose if Apollo Server should calculate detailed
@@ -127,8 +172,9 @@ export interface ApolloServerPluginUsageReportingOptions<
    * (Note that returning true here does *not* mean that the data derived from
    * field-level instrumentation must be transmitted to Apollo Studio's servers
    * in the form of a trace; it may still be aggregated locally to statistics.
-   * But either way this operation will contribute to the "field executions"
-   * statistic and timing hints.)
+   * Similarly, setting `sendTraces: false` does not affect
+   * `fieldLevelInstrumentation`. But either way this operation will contribute
+   * to the "field executions" statistic and timing hints.)
    *
    * The default `fieldLevelInstrumentation` is a function that always returns
    * true.
@@ -211,6 +257,9 @@ export interface ApolloServerPluginUsageReportingOptions<
    * and the operation name and signature will always be reported with a constant
    * identifier. Whether the operation was a parse failure or a validation
    * failure will be embedded within the stats report key itself.
+   *
+   * The value of this option is not relevant if you set `sendTraces: false`,
+   * because unexecutable operation documents only appear in traces.
    */
   sendUnexecutableOperationDocuments?: boolean;
 
@@ -225,6 +274,9 @@ export interface ApolloServerPluginUsageReportingOptions<
    * Apollo's servers perform their own sampling on received traces; not all
    * traces sent to Apollo's servers can be later retrieved via the trace UI.)
    *
+   * If you just want to send all operations as stats, set `sendTraces: false`
+   * instead of using this experimental hook.
+   *
    * This option is highly experimental and may change or be removed in future
    * versions.
    */
@@ -298,7 +350,7 @@ export interface ApolloServerPluginUsageReportingOptions<
   /**
    * If set, prints all reports as JSON when they are sent. (Note that for
    * technical reasons, traces embedded in a report are printed separately when
-   * they are added to a report.)
+   * they are added to a report.) Reports are sent through `logger.info`.
    */
   debugPrintReports?: boolean;
   /**
@@ -307,12 +359,6 @@ export interface ApolloServerPluginUsageReportingOptions<
    * about how the signature relates to the operation you executed.
    */
   calculateSignature?: (ast: DocumentNode, operationName: string) => string;
-  /**
-   * This option includes extra data in reports that helps Apollo validate the
-   * stats generation code in this plugin. Do not set it; the only impact on
-   * your app will be a decrease in performance.
-   */
-  internal_includeTracesContributingToStats?: boolean;
   //#endregion
 }