Document missing safe abstractions

[Shnatsel]

Some crates, e.g. reqwest (see #5) clearly indicate the need for better safe abstractions, as their logic cannot be expressed in terms of the existing ones.

The worst offender by far is the Read trait which requires an initialized slice to write to, but initializing a slice is costly, so people just throw uninitialized slices at it and hope for the best. This unsafety could be encapsulated by appending to a Vec-like fixed-size structure.

Another such case is rust-lang/rfcs#2714

I'm thinking of filing issues on https://github.com/rust-lang/rust/ and then linking to them from some markdown file or from this issue. Thoughts?

[Lokathor]

I think part of this is that you're supposed to reuse a buffer, not just make it, read to it, and throw it away.

perhaps their code could be made to reuse buffers more often and average out the cost of the initial zero initialization?

[Shnatsel]

If initialization is done only once it comes without any overhead because the allocator can request already-zeroed memory from the OS. It's the subsequent initializations that require memset().

[Lokathor]

That only holds if no one else has allocated and deallocated before you create your vec. With how box happy a lot of Rust code sadly is, you cannot assume that.

[64]

If initialization is done only once it comes without any overhead because the allocator can request already-zeroed memory from the OS. It's the subsequent initializations that require memset().

This is not always the case. Not every single fresh allocation comes directly from the OS (that’s exactly what allocators are supposed to avoid, as it’s very slow).

[tarcieri]

This unsafety could be encapsulated by appending to a Vec-like fixed-size structure.

There are several crates of this nature. heapless::Vec comes to mind for me:

https://docs.rs/heapless/latest/heapless/struct.Vec.html

[Shnatsel]

Oh, another one! There are also:

• https://crates.io/crates/buffer
• https://crates.io/crates/uninit
• https://crates.io/crates/static-alloc

[Shnatsel]

Another use case for fixed-capacity Vec-like memory view - flate2 needs it to become 100% safe. See #32

[Shnatsel]

Regarding the "fixed-capacity Vec" approach: tokio is working on an AsyncRead trait that writes to a Vec-like structure instead of a slice. This lets us sidestep the thorny issues around uninitialized memory, since it's all safely encapsulated. I'm not sure if the capacity is bounded, so I've left some comments on the PR to clarify that: tokio-rs/tokio#1744

[DoumanAsh]

The worst offender by far is the Read trait which requires an initialized slice to write to, but initializing a slice is costly, so people just throw uninitialized slices at it and hope for the best.

Being able to write uninit mem is must have for any performance oriented code
I do not believe there is any way to completely avoid it.
It should not be UB to write uninit &mut [T], which is de-facto safe and uninitt &[T] should be UB only when you read. (I believe right now it is UB?)

[danielhenrymantilla]

@DoumanAsh the problem does not lie in writing to a &mut [u8] being UB, but in the Read trait requiring a &mut [u8] to write to it.

Read ought to be reworked to have the API offered in https://docs.rs/uninit

[DoumanAsh]

I do not see a particular problem with current Read and it would be hard to remove something that is 'broken' due to backward compatibility

Well you can extend Read to have something like read_into_uninit but it needs to have default impl and there is not much point to it anyway.
Read cannot break backward compatibility

[64]

I do not see a particular problem with the current Read

The problem (IIUC) is that a caller to a generic Read implementation cannot pass uninitialised memory to be filled in, as all the relevant methods are not marked unsafe (or don’t accept a structure capable of preventing uninit mem accesses like Shnatsel is describing).

So to be safe everyone has to pass initialised memory which wastes some cycles.

[Lokathor]

Yeah, what we want is unsafe trait ReadToUninit that takes &mut [MaybeUninit<u8>] and then assures that however much it says it read is now safe to use as initialized.