str and [u8] should hash the same

[chris-morgan]

str and [u8] hash differently and have done since the solution to #5257.

This is inconvenient in that it leads to things like the StrTendril type (from the tendril crate) hashing like a [u8] rather than like a str; one is thus unable to happily implement Borrow<str> on it in order to make HashMap<StrTendril, _> palatable.

[u8] gets its length prepended, while str gets 0xff appended. Sure, one u8 rather than one usize is theoretically cheaper to hash; but marginally so only, marginally so. I see no good reason why they should use different techniques, and so I suggest that str should be changed to use the hashing technique of [u8]. This will prevent potential nasty surprises and bring us back closer to the blissful land of “str is just [u8] with a UTF-8 guarantee”.

Hashes being internal matters, I do not believe this should be considered a breaking change, but it would still probably be a release-notes-worthy change as it could conceivably break eldritch codes.

[SimonSapin]

👍

[Gankra]

I'm all for this change as long as it's benchmarked to not cause a noticeable regression. (I agree with your assessment that it shouldn't)

[bluss]

Hashing an strings of length n will be n + 1 bytes before and n + 8 bytes after. Siphash adds one byte and hashes in 8-byte blocks, so very short strings bumped up to the next size, using 2 * 2 + 4 = 8 siphash rounds vs. previous 2 * 1 + 4 = 6. I'm sure you can measure the difference.

Also, this is a change to what the Hash trait emits for str, so it's a breaking change no matter which hash function is used. I understand it can be a minor change, I think it's fine to change this in a new rust release.

[arthurprs]

This would be optimized if rust-lang/rfcs#1666 is accepted.

[chris-morgan]

@arthurprs I don’t see how rust-lang/rfcs#1666 would have any bearing on this. All this should need is

diff --git a/src/libcore/hash/mod.rs b/src/libcore/hash/mod.rs
index 051eb97..85427d0 100644
--- a/src/libcore/hash/mod.rs
+++ b/src/libcore/hash/mod.rs
@@ -328,8 +328,7 @@ mod impls {
     #[stable(feature = "rust1", since = "1.0.0")]
     impl Hash for str {
         fn hash<H: Hasher>(&self, state: &mut H) {
-            state.write(self.as_bytes());
-            state.write_u8(0xff)
+            self.as_bytes().hash(state)
         }
     }

[arthurprs]

Yes, but &[u8].hash() is internally

    #[stable(feature = "rust1", since = "1.0.0")]
    impl<T: Hash> Hash for [T] {
        fn hash<H: Hasher>(&self, state: &mut H) {
            self.len().hash(state);
            Hash::hash_slice(self, state)
        }
    }

And that self.len().hash(state); would be optimized to a single branch (or nothing at all if LLVM is that that) if the RFC is accepted.

[dtolnay]

I agree that it would be desirable for str and [u8] to hash the same. Does anyone have an estimate of the worst case, how much this would affect performance of realistic usage of hash map with string keys?