Feature proposal: zero-downtime reindex

[David-Guillot]

Hi there,

We have, in a private repository, an implementation of zero-downtime OpenSearch reindex of a whole collection of documents coming from an external source (possibly a Django model). It's based on:

• Putting a suffix on a Document's OpenSearch index (e.g. a timestamp)
• Setting an OpenSearch index alias on the current Document's index
• Re-indexing (e.g. after a deployment that changes the Documents schema) on a new index while the current one is still live
• Swapping the index alias
• Deleting obsolete indices

Do you think such feature could be a part of this lib?

[qcoumes]

That sounds really useful, and I would use it for sure, but I'm afraid the fact that the index is actually an alias could confuse people at some point.

I think it would be better if the lib could first manage aliases. This could be done through some changes to management commands. I think of 3 possibilities:

• Some changes to manage.py opensearch list to at least display aliases information.
• A new action of manage.py opensearch index.
• A new subcommand manage.py opensearch alias.

Once there is a way to manage (or at least display) aliases, I'd gladly accept your zero-downtime reindex feature.

[David-Guillot]

Thanks for your feedback, yes this feature is cool but in its current implementation, which is private, is a bit magical. We need a more explicit implementation, thanks for the leads on that 👍

[David-Guillot]

Hi @qcoumes ,

It's been a long time, but I haven't forgotten this project. I had some time to think about it these days, and I'd like to know if the following presentation would be acceptable from your point of view:

• Creating indices with the will to make them "versioned":
  
• Listing indices that are "versioned":
  

Of course this is still work-in-progress because the alias to make event point to event_20220810093110 is not created yet, but this is the easy part. For sure, as soon as the alias would be created, the index would appear with a [x] in the listing.

If this seems clear enough to you, I should be able to open a PR in the next few days 😉

[qcoumes]

Seems very promising