Performance Degradation in 2.1.0.0.3

[CharlieC3]

Describe the bug
Based on our metrics, we're seeing a large regression in HTTP response time performance when we upgraded from 2.1.0.0.2 to 2.1.0.0.3 for stacks blockchain nodes receiving HTTP traffic.

We upgraded our nodes around 1500ET on April 17th. CPU and memory usage has stayed the same before and after the upgrade.

HTTP GET response time increased for 200s and 404s. Some affected endpoints I've seen are:

• /v2/accounts/*
• /v2/info
• /v2/pox
  
  

HTTP POST response time increased for 200s and 400s. Some affected endpoints I've seen are:

• /v2/transactions
• /v2/contracts/call-read/* (lots of different contracts affected, not just one or a few in particular)
• /v2/map_entry/SP000000000000000000002Q6VF78/pox/delegation-state?proof=0
• /v2/fees/transaction

[kantai]

I'll look through our logs to try to see what's driving the performance issues here, but I'm very skeptical that this could be related to 2.1.0.0.3 -- the patch did not touch any codepaths used by follower nodes.

[pavitthrap]

Performance degradations went away after API release was downgraded. It seems like the issue did not emerge from the node upgrade.

More analysis needed.

[AshtonStephens]

First, @kantai you said:

I'll look through our logs to try to see what's driving the performance issues here, but I'm very skeptical that this could be related to 2.1.0.0.3 -- the patch did not touch any codepaths used by follower nodes.

Then @pavitthrap you said:

Performance degradations went away after API release was downgraded.

This sounds like the issue was caused by the API release upgrade. Am I misunderstanding something? Is it possible that the upgrade causes a memory issue or something else on the hosts that could tangentially cause large http times?

[kantai]

Pretty unlikely -- the stacks-blockchain-api itself isn't handling these requests: these are being handled by the stacks-node RPC handler. My best guess is that request patterns changed a little bit which led to a change in the performance of the node.

[CharlieC3]

Also adding some further perspective, this issue does not mark the first of "poor HTTP performance" for the stacks-node. Rather it should be interpreted as a time when things went from poor to worse.
Agreed with @kantai's statement that this may have been a result of a change in user behavior; it's unclear to me and given the issue's age we likely won't be able to extract any more contextual data from that time.