Improve HTTP connection handling

[sg70]

• All tests passed. If this feature is not already covered by the tests, I added new tests.
• All static analysis checks passed.
• This pull request is on the dev branch.
• I used gofmt for formatting the code before submitting the pull request.

---

Closes jfrog/jfrog-cli-artifactory#290

What does this MR do?

This MR enables the underlying http.client to reuse HTTP connections.

Why was this MR needed?

The relevant part of the HTTP client implementation has remained unchanged since 2017. It is unclear why req.Close was set to true, which forces the Go http.Client to close the connection immediately after the response is received. Reusing established HTTP(S) connections is a common best practice and can significantly improve performance.

The function createDefaultHttpTransport in http/httpclient/clientBuilder.go uses the same defaults as the original DefaultTransport implementation in the net/http package. Since ForceAttemptHTTP2 was added as a default in the net/http package after 2017, I added this option as well.

After performance testing, I set the parameter MaxIdleConnsPerHost to 6, aligning with modern web browsers’ common default and providing a balanced improvement in connection reuse, especially when HTTP/2 is not used. The previous default of 2 resulted in a high number of short-lived TCP connections under multi-threaded workloads.

Performance test case

Download a directory from Artifactory with 4591 files ranging from 100 Bytes to 21 MByte (total size 892.13 MByte) with a max connection throughput of 350 Mbit/s. Tests were conducted on the internet where the Artifactory server is hosted at AWS behind an AWS ALB. Client and AWS region are in the same country. The jfrog-cli was used with its default configuration. All test were executed multiple times.

current jfrog-cli release (2.78.10)
Download as directory using jfrog-cli with options --include-dirs --quiet --threads 3: 221 secs
Download as directory using jfrog-cli with options --include-dirs --quiet --threads 10 : 97 secs
Download as directory using jfrog-cli with options --include-dirs --quiet --threads 20 : 69 secs

with the change of the PR applied:
Download as directory using jfrog-cli with options --include-dirs --quiet --threads 3: 124 secs
Download as directory using jfrog-cli with options --include-dirs --quiet --threads 10 : 64 secs
Download as directory using jfrog-cli with options --include-dirs --quiet --threads 20 : 46 secs

During the test runs with the change of the PR applied, the number of TCP connections was stable, and TCP connections got reused. Furthermore the total amount of open TCP connections when using HTTP2 was not more than 4 connections.

Potential Drawbacks

While reusing connections generally improves performance, it's important to note that the Go http.Client has set a IdleConnTimeout of 90 seconds. This means that idle connections will be closed after 90 seconds of inactivity, preventing them from becoming stale indefinitely. Therefore, the risk of long-lived connections causing issues is mitigated by this timeout. The benefits of connection reuse are expected to outweigh any potential drawbacks in most common scenarios.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[sg70]

I have read the CLA Document and I hereby sign the CLA.

[sg70]

@ehl-jf can have a look on this PR?

[github-actions]

---

🐸 JFrog Frogbot