format=raw for vault read

[sebastiang]

Some secret values are binary blobs, like pfx format private keys. Is there a straightforward facility for storing these in vault? It can be jury-rigged with base64 encoding, but especially from the command line it can be awkward.

[mitchellh]

Vault actually should already be able to do this. It doesn't introspect or consider the values text at all. vault read may struggle with it. But you should be able to do this, probably:

vault write secret/foo @binarydata.bin

And you can read it using the API. I suppose we can add a flag to vault read to output binary though...

Going to keep this open so I can test.

[sebastiang]

It's vault read that's the concern; I should have been more precise. You're right that you can stuff it in fine, but the text-based cmd line export is hard to parse in that case. format=raw or format=binary might be handy.

[sebastiang]

I took the liberty of renaming the issue to reflect the real desire: a simple way to read a binary value as easily as one can write it

[markpeek]

Has this been resolved with commit c6b5d95?

$ cat data.bin
testing
    1 2 3
$ hexdump data.bin
0000000 74 65 73 74 69 6e 67 0a 09 31 20 32 20 33 0a
000000f
$ vault write secret/foo value=@data.bin
Success! Data written to: secret/foo
$ vault read -field=value secret/foo
testing
    1 2 3

$ vault read -field=value secret/foo | hexdump
0000000 74 65 73 74 69 6e 67 0a 09 31 20 32 20 33 0a 0a
0000010

Not sure if the extra trailing line feed would cause issues with @sebastiang use case.

[jefferai]

Yes. I'm going to close this in favor of #583 which is tracking the carriage return issue.

[arthurtsang]

hi @sebastiang, did you store the pfx files on vault successfully? I'm having the same issue but the file content is vastly different when i read it back from vault. want to know if you have any luck getting it work without encoding the file first? (actually using uuencode failed too, i've to use uuencode -m)

[sebastiang]

I ended up just using PEM (text) formatted private keys, which most libraries can accept if you talk to them the right way.
On February 10, 2016 at 11:32:31 AM, Arthur (notifications@github.com) wrote:

hi @sebastiang, did you store the pfx files on vault successfully? I'm having the same issue but the file content is vastly different when i read it back from vault. want to know if you have any luck getting it work without encoding the file first? (actually using uuencode failed too, i've to use uuencode -m)

—
Reply to this email directly or view it on GitHub.

[arthurtsang]

in case anyone is interested, this is how i upload the keys. uuencode -m foo.pfx /dev/stdout | jq -Rs '{"value":.}' | vault write "secret/foo" -

[jefferai]

@arthurtsang The content should certainly not be "vastly" different! Please note that the data you're passing in is being shoved into a JSON API, so we recommend base64 encoding. Here's an example:

$ cat gpgkeybin | base64 | vault write "secret/foo" foo=-
Success! Data written to: secret/foo
$ vault read -field=foo secret/foo | base64 -d > gpgkeybin2
$ sha256sum gpgkeybin gpgkeybin2
c7b3077233385943e111e82c2056ac15c74db3ea82a406197315302f027538bf  gpgkeybin
c7b3077233385943e111e82c2056ac15c74db3ea82a406197315302f027538bf  gpgkeybin2

[jefferai]

@arthurtsang It works for me with uuencode's -m flag too.

My guess is that you are formatting a JSON input to the command, but when you get the data back from Vault you are not parsing it as JSON and pulling the value out:

$ uuencode -m gpgkeybin /dev/stdout | jq -R -s '{"value":.}' | vault write "secret/foo" foo=-
Success! Data written to: secret/foo
$ vault read -field=foo secret/foo | jq -r '.value' | uudecode -o /dev/stdout > gpgkeybin2
$ sha256sum gpgkeybin gpgkeybin2
c7b3077233385943e111e82c2056ac15c74db3ea82a406197315302f027538bf  gpgkeybin
c7b3077233385943e111e82c2056ac15c74db3ea82a406197315302f027538bf  gpgkeybin2

[arthurtsang]

@jefferai, thanks. uuencode -m is just base64 encoding. but without the flag -m, when I read back the data from vault, it will contains a lot of (MISSING) in the content. that's what breaking some binary files and contribute to the vastly difference I observed above.