Incorrect maximum encryption size for RSA OAEP

[mschrag]

Incorrect maximum encryption size for RSA OAEP

Checklist

• I've run tests to see all new and existing tests pass
• I've followed the code style of the rest of the project
• I've read the Contribution Guidelines
• I've updated the documentation if necessary

If you've made changes to gyb files

• [n/a] I've run ./scripts/generate_boilerplate_files_with_gyb.sh and included updated generated files in a commit of this pull request

Motivation:

The maximumEncryptSize function is hardcoded to use 42 as the hash offset, but the RFC actually says it's "2hLen-2" so 42 is only valid for SHA1. SHA256 should be 62 (232-2). This adds a hash length onto the Digest enum, which can then be used in the length calculation.

In writing the tests for this, I also stumbled on the unsafe PEM representation for RSA Public Keys don't allow 1024-bit keys as documented, so this also fixes that.

Modifications:

• Added a hashBitLength to the RSA Digest which is then used in the maximumEncryptSize to properly compute the maximum length
• Corrected the minimum key size for RSA unsafe PEM public keys to 1024 from 2048

Result:

• The maximumEncryptSize function will return the expected value for RSA OAEP SHA256 keys. I don't believe anything calls this internally, so this would be for external consumers.
• Unsafe construction of RSA public keys now allow 1024 bit keys. It was documented as supporting them, but the check was still 2048 (probably copy-paste error from the safe variant). This only expands the potential uses, so it shouldn't introduce any new failures.

[Lukasa]

Nice one, thanks!