[BUG] Data-Prepper is still using the deprecated _template path when creating ES templates

[thomass4t]

Describe the bug
When using a sink with the parameters template_file and ism_policy_file, an OpenSearch template is automatically created. However, this is not visible in the dashboard because it uses the old path _template instead of the new one _index_template
It's probably more an improvement to stay compatible with newer versions.

To Reproduce
Steps to reproduce the behavior:

1. Configure a sink, e.g.

  sink:
    - opensearch:
        hosts: [ "https://127.0.0.1:9200" ]
        index: "test_logs"
        number_of_shards: 1
        number_of_replicas: 0
        template_file: "/opt/data-prepper/pipelines/log.template"
        ism_policy_file: "/opt/data-prepper/pipelines/log.policy"

3. The log.template looks like this:

{
  "version": 1,
  "settings": {
      "index": {
        "opendistro": {
          "index_state_management": {
            "rollover_alias": "test_logs"
          }
        },
        "number_of_shards": "1",
        "number_of_replicas": "0"
      }
  },
  "mappings": {
      "properties": {
        "date": {
          "type": "float"
        },
        "level": {
          "type": "keyword"
        },
        "log": {
          "type": "keyword"
        },
        "logger": {
          "type": "text"
        },
        "message": {
          "type": "text"
        },
        "time": {
          "type": "date"
        }
      }
    }
}

5. GET _template shows the new template, created by data-prepper. Deletion only works in the path _template/ and not in _index_template/.

Screenshots

Environment (please complete the following information):

• OS: Ubuntu 22.04 LTS
• Data-prepper: 2.3.2
• OpenSearch 2.8.0

Additional context

[dlvenable]

@thomass4t ,

Data Prepper does use the _template path by default. However, it also support _index_templates if configured. This was added in Data Prepper 2.3, so it is fairly recent.

See the template_type documentation in the opensearch sink documentation.

So in your example you should be able to set:

sink:
    - opensearch:
        hosts: [ "https://127.0.0.1:9200" ]
        index: "test_logs"
        number_of_shards: 1
        number_of_replicas: 0
        template_type: index-template
        template_file: "/opt/data-prepper/pipelines/log.template"
        ism_policy_file: "/opt/data-prepper/pipelines/log.policy"

[thomass4t]

@dlvenable
Thanks for your quick help :)
I adjusted the sink as you suggested. The template is now created in the new path index-template.

However, my template file doesn't seem to work any more.
The mapping and replicas are not transferred to the template.

Perhaps I need to change the template to composable ones. I will try to adjust the template in the next days.

Thanks for your help so far! 👍

[dlvenable]

@thomass4t , Yes, the template file you supply must be in the correct structure for the composable index templates. Data Prepper does not attempt to map it.

We have a couple of examples that are used for trace analytics which you can look at to see some of the differences. I also suggest that you review the documentation for OpenSearch.

Composable index template:

https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/src/main/resources/index-template/otel-v1-apm-service-map-index-template.json

V1 template:

https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/src/main/resources/otel-v1-apm-service-map-index-template.json

[thomass4t]

@dlvenable
Great! That example was exactly, what I was looking for 🙂
I already tested it and it worked!
Thanks a lot for your support and have a great day!