Use new jql endpoint with token based paging

[fuzzkat]

Based on an idea in #467 by @freibuis

This change affects versions 2 and 3 of the API.

The Jira API has been updated so that the search endpoint with ?jql parameter has been completely removed (it was deprecated back in October 2024). This endpoint was replaced with a more performant one at search/jql?jql=. As well as moving the endpoint, the format and parameters has changed. Most critically the endpoint now returns a nextPageToken which must be used to request the next page of data.

This fix modifies the Issue.jql method to loop through the pages inside the gem, instead of expecting the start_at parameter to be passed in. This avoids returning the nextPageToken back to the consumer, which would mean introducing a new return object type or structure.

[levindixon]

Thanks for working on this @fuzzkat! I've added a note about the fields param in the other PR that aims to address this issue which may or may not be relevant to you here: #467 (comment)

[freibuis]

@fuzzkat 👍 I didn't even noticed the projects jql.

Thanks for the acknowledgement.
I did hit a wall as I would have to return the next page in the object. So I moved it into the method.. this also streamed lines a few other processes

In my testing I did hit a wall with very very large jql queries. I would get a 429 rate limiting. So I was working on checking for 429 codes. But gave up

I'll will be withdrawing my pr for this one

[fuzzkat]

I've added a note about the fields param in the other PR - @levindixon

Yes, I'm not sure how that should be handled. I'm using v2 (the default) and you can pass in the required fields, but that behaviour may surprise users if all formerly visible fields disappear! It didn't take me long to realise that the fields parameter was needed.

Since the gem is providing a somewhat standardised interface to the API, perhaps it's okay for fields to be a required parameter for all versions. The alternative would be to parse all the navigable fields and build a fields param internally, but that seems like a retrograde step.

[fuzzkat]

I was working on checking for 429 codes. - @freibuis

My personal use-case has been solved, as we don't request that much data, but I can see that for general use this solution is sub-optimal. Bringing rate-limiting inside the gem is certainly a side-effect of hiding the workings of the API behind the gem's interface.

Options that I can think of;

• Rate limit our requests when we hit the rate limit response and delay the return from the gem. This would make the Gem even less responsive (it will be pretty slow for large queries with this solution anyway).
• Stop requesting more pages when we hit the rate-limit response (you probably don't need them anyway and should rethink your query).
• Return the next page token as well as the current page of data from the API. This would require a significant change to the gem's API. Tuple method responses are not natively handled in ruby, but a simple hash or Struct would do.

Probably the latter is the most sane option, it's just a shame that it renders all old versions incompatible. Mind you, that's really down to Atlassian making a breaking change on an old API version!

[fuzzkat]

It would be good to get a steer from @bobbrodie about which option would be preferred by sumoheavy.

[bobbrodie]

Thank you so much @fuzzkat -- I'm thinking this through :)

[ashkulz]

This PR, along with adding fields: ["*navigable"] in the calling code fixed it for us -- I changed our Gemfile to use it via:

gem 'jira-ruby', github: 'sumoheavy/jira-ruby', ref: 'refs/pull/468/head'

Thanks, @fuzzkat and @levindixon!

[SamC-Apadmi]

Thanks @ashkulz you've saved my skin there!

We depend pretty heavily on this package for a lot of pretty critical stuff, is there anything I can do to help with getting this in @bobbrodie ?

[bobbrodie]

This is looking great, @fuzzkat -- thanks for the contribution! This gem doesn't get a ton of traction these days, so I really appreciate it :)

Just one small thing - could you update rubocop.yml to change Naming/PredicateName to Naming/PredicatePrefix? I think that should clear up our errors and let us merge.

Once we merge, I think we could do a real release of 3.0.0 and take it out of beta.

[freibuis]

I have had a thought about the 429 rate limiting.
Since I am doing a few cloud -> cloud migrations (merging 2 sites into 1) this kind of needs to happen. some sites that I merge have 100k records to move.. you will hit the limits pretty quick

I have gotten around this by doing the JQL call in batches. JQL key between 1-499 sleep 1 second. then call key 500 -> 999 etc

I think there needs to be an example of this new method + examples on how to rate limit using jql queries

[kikin81]

@bobbrodie hey! Thanks for merging this fix. I was corious of the timeline to release this as part of 3.0.0? Any chance we can have a pre-release tag in case it's not ready?
Thanks!

[bobbrodie]

Hey of course @kikin81 , an even bigger thanks to everyone who worked on this. The gem doesn't get a ton of attention from folks these days, but it feels like Rails is having a bit of a comeback which I like to see :)

We're working on getting this tagged and up to RubyGems today!

[bobbrodie]

And 3.0.0 is live! If anyone runs into any issues, please give me a shout and I'll take a look. Thank you all again so much for the kind contributions, I really, really appreciate it. And, it's nice to see the gem getting some use!

[kikin81]

@bobbrodie thank you so much for this! Really appreciate it!

[freibuis]

for those whom are reading .. version 3 has breaking changes.. duh...

love how this got snuck into version 3 ❤️

Thanks @fuzzkat for taking my idea and putting it on another level. the amount of time this has saved me. if we ever meet.. 🍺 is yours.

[fuzzkat]

Thanks @bobbrodie for getting this merged into v3.0.0, we all appreciate the fast turnaround - there's plenty of us who still love Rails!