Proposal: Fork and maintain elasticsearch-rails

[mcoms]

What kind of business use case are you trying to solve? What are your requirements?

I'd like to transition to OpenSearch from Elasticsearch, inside of a Rails project. These projects almost exclusively use the elastic/elasticsearch-rails gem, rather than the ruby client directly.

What is the problem? What is preventing you from meeting the requirements?
There is no comparable or compatible version of this gem maintained by the OpenSearch community.

What are you proposing? What do you suggest we do to solve the problem or improve the existing situation?
That the opensearch-project team fork and maintain an opensearch version of elastic/elasticsearch-rails (this codebase comprises three gems: elasticsearch-model, elasticsearch-persistence, elasticsearch-rails which work together).

What are your assumptions or prerequisites?
That the team would be prepared to commit the resources to doing this; and that the vast majority of ruby projects interacting with opensearch will be using rails and would therefore benefit.

What are remaining open questions?
None.

[brijos]

I see there are 29 likes on this issue. Is anyone available to contribute and/or maintain?

[calvincorreli]

I'd potentially be open to have my team step up and maintain this. Doesn't seem like it'd be a ton of work. Let me check with the team.

[epugh]

If this doesn't gain traction, another route to go is the wonderful https://github.com/ankane/searchkick library... Currently testing against OS and ES!

[simi]

@epugh that looks ok, but it is compatible on data level? Or do we need to build index from scratch?

[epugh]

I'm sure there will be some differences, as I don't know what specific assumptions elasticsearch-rails makes... I know though that this library is very widely used.

It may be one of those things where, if you only have a fairly simple implementation of elasticsearch-rails, then swapping should be easy. But if you've used every last feature the gem offers, then migrating may be tougher.

I will say, traditionally in search, migrating from one search engine to another is pretty common task for folks! And a good chance to rethink some things....

Much simpler then say swapping database platforms!!!

[wbeckler]

@epugh Are there things you would suggest to guide people toward searchkick, such as adding links in the opensearch-ruby documentation or readme etc.?

[simi]

@wbeckler feel free to inspire at rubygems/rubygems.org#3199, it is almost ready for merge. We can try to write dome some steps for upgrade.

[wbeckler]

Nice! Those are all good catches. Does anyone else have thoughts on elevating searchkick versus bringing in elasticsearch-rails?

[epugh]

@epugh Are there things you would suggest to guide people toward searchkick, such as adding links in the opensearch-ruby documentation or readme etc.?

I think both of those are great suggestions. I noticed the only page that highlights the ecosystem around OS, https://www.opensearch.org/community_projects, is kind of buried deep on the main site. The text around "A community-driven approach" could maybe highlight the number of related projects with a link?

I believe that a differentiator for OS versus ES is the community and more open ecosystem?

[mcoms]

Searchkick is a nice gem but it's the answer to a different question. The two gems have different APIs and different use cases.

In my case, I'm not writing a greenfield project which needs common search features; I'm migrating a number of production apps which have complex search behaviour using the elasticsearch-rails framework bindings for elasticsearch, to OpenSearch. These apps have years of code targetting the existing API.

It's a common pattern in Rails to have a thing-rails gem with framework bindings for your thing. When I first raised this issue in January, it seemed as though forgetting about an opensearch-rails gem was a simple oversight rather than a conscious choice: elasticsearch-rails is how Elastic intended Rails projects to integrate with elasticsearch, accordingly it's how most Rails apps are built.

Telling people to rewrite all their search code for a different gem's API, and hoping it has feature parity and that they'll be able to do everything they're already doing, is likely to be too high a risk and too much work for teams to take on; it'll deter them form switching to OpenSearch.

Gem	Downloads (RubyGems.org)
elasticsearch	95,968,004 (will include both figures below)
elasticsearch-rails	31,573,512
searchkick	9,786,846
opensearch-ruby	668,315

Whilst I think that searchkick is a great option if you're looking to add common search features to a brand new app, in my case I simply cannot switch to the OpenSearch ecosystem without a comparable opensearch-rails gem. If the project provided one, migrating a Rails app to OpenSearch would be a simple change in an app's Gemfile (as it is for the opensearch-ruby gem) and would result in a boost in uptake; rather than months of work (to rewrite complex search logic for another library like searchkick) which may not ultimately be successful.

[simi]

@mcoms I do understand your problems. Personally I was surprised to see opensearch-rails is not even planned from the OS side. I was thinking about 2 ways to migrate rubygems.org.

1. fork elasticsearch-rails, migrate to opensearch-rails and handover this back to opensearch (I was told somewhere this would be accepted)
2. migrate to other gem (like searchkick)

Initially I was about to go with 1., but then I realized I don't want to spend my time contributing to billion dollar company projects. I have explored searchkick and it is really close to elasticsearch-rails. API differs a little only.

Instead of waiting for opensearch-rails maybe we can improve documentation to searchkick regarding migration or even create small layer (separated gem, maybe merged later to searchkick) to support original API and just translate calls to searchkick API.

[dblock]

@wbeckler should chime in on picking up opensearch-rails by the existing team of client maintainers at Amazon, but don't let anyone stop you from forking and maintaining opensearch-rails outside of this org - we can always move it here later