[BUG] Unoptimized text field used for traceGroup

[dlvenable]

Describe the bug

Saving trace data to OpenSearch is producing the following error:

[illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [traceGroup] in order to load field data by uninverting the inverted index. Note that this can use significant memory.

Environment (please complete the following information):

• OS: Docker image
• Version 1.2.1
• OpenSearch: 1.0
• OTel Collector 0.37.0

[alexis-spx]

Hi this issue was opened on my behalf after reporting the issue to AWS support. They have now worked out what it was about my set up that was causing the issue.

We had a default index template that was matching all indexes.

GET _index_template/default_index_template
{
  "index_templates" : [
    {
      "name" : "default_index_template",
      "index_template" : {
        "index_patterns" : [
          "*"
        ],
        "template" : {
          "settings" : {
            "index" : {
              "number_of_shards" : "4",
              "default_pipeline" : "log_ingest_lag"
            }
          }
        },
        "composed_of" : [ ],
        "priority" : 1
      }
    }
  ]
}

From the Elasticsearch 7.10 documentation :
"When a composable template matches a given index it always takes precedence over a legacy template. If no composable template matches, a legacy template may still match and be applied."

We have worked around this issue for now by copying the templates that data-prepper is creating (otel-v1-apm-span-index-template, otel-v1-apm-service-map-index-template) and creating equivalent composable templates.

PUT _index_template/otel_v1_apm_service_map_index_template_comp...
PUT _index_template/otel_v1_apm_span_index_template_comp...

The templates that data-prepper are creating are legacy templates, could data-prepper be updated to create the new composable template type please?

Cheers.

[dlvenable]

@alexis-spx , I'm glad you were able to find a workaround.

I created #1275 to track this as a feature request. Please read it and add any feedback or comments that you have.

For any other users who hit this situation, the comment above provides a good workaround: Create a manual index template based on the one Data Prepper provides.

I'll close this bug in favor of using #1275.

[alexis-spx]

Thanks @dlvenable looks like a great proposal.