fix: respect user configuration for work with status codes

[Mantisus]

Description

Fixes HTTP status code configuration behavior:

• additional_http_error_status_codes: Properly triggers errors and retries when specified status codes are encountered
• ignore_http_error_status_codes: Correctly treats specified error status codes as successful responses

Issues

• Closes: Unable to configure retry behavior for status codes 400-499 #756
• closes Refactor handling of ignored_http_status_codes and SessionError #708

[Mantisus]

I have already discussed this with @janbuchar. This PR should not introduce any breaking changes. If additional_http_error_status_codes and ignore_http_error_status_codes are not specified, the previous behavior is preserved.

Status codes were previously discussed in PR: #498

However, we have differences with the JS version of crawlee:

Python:

• 400-499 codes cause errors without retries

JS:

• 400-499 codes are treated as successful
• Exceptions:
  • 401, 403, 429 - cause errors with retries
  • 406 - causes error without retries and ignores ignoreHttpErrorStatusCodes parameter

[janbuchar]

Two questions, otherwise LGTM... much better!

[vdusek]

Just one last naming thing, otherwise LGTM.

[vdusek]

However, we have differences with the JS version of crawlee:

Python:

400-499 codes cause errors without retries
JS:

400-499 codes are treated as successful
Exceptions:
401, 403, 429 - cause errors with retries
406 - causes error without retries and ignores ignoreHttpErrorStatusCodes parameter

Any ideas how we want to align it? I believe we do not have issue for that, correct? cc @janbuchar

[janbuchar]

However, we have differences with the JS version of crawlee:
Python:
400-499 codes cause errors without retries
JS:
400-499 codes are treated as successful
Exceptions:
401, 403, 429 - cause errors with retries
406 - causes error without retries and ignores ignoreHttpErrorStatusCodes parameter

Any ideas how we want to align it? I believe we do not have issue for that, correct? cc @janbuchar

I believe apify/crawlee#812 tracks this. I also think that what we do in this PR is the correct way to handle things. The "success on 4xx" in JS is a crutch that prevented retries on such errors, and we can do that more cleanly now (in both implementations).

How does the Python version handle 406 again? Can it be ignored via ignoreHttpErrorStatusCodes?

[janbuchar]

One more thing - it looks like there are no tests for this behavior - could you add some? test_http_crawler would be an appropriate place I guess.