Align Java and C++ RecordBatch data and metadata layout

[toddfarmer]

Note: This issue was originally created as ARROW-384. Please see the migration documentation for further details.

Original Issue Description:

layout on C++ side:

<buffers> <metadata> <metadata size: int32>

and on the java side:

<metadata> <buffers>

In the file format the footer has a Block info that contains the metadata length.
https://github.com/apache/arrow/blob/f082b17323354dc2af31f39c15c58b995ba08360/format/File.fbs#L36
See:
apache/arrow#211 (comment)

Related issues:

• Fix string/binary integration tests #344 (is blocked by)

Migrated issue participants:

Reporter: Julien Le Dem (julienledem)
Assignee: Wes McKinney (wesm)

[toddfarmer]

Note: Comment by Julien Le Dem (julienledem):
Here is a 3rd suggestion:

<metadata size: int32> <metadata> <buffers> 

Since we probably know the size of the flatbuffer already before writing.

[~wesmckinn]: One thing I am missing is that in the context of shared memory, buffers are not contiguous.

[toddfarmer]

Note: Comment by Jacques Nadeau (jnadeau):
+1 on this approach.

[toddfarmer]

Note: Comment by Wes McKinney (wesm):
This seems reasonable, and saves you from always requiring the metadata size.

If we look at the flatbuffers IDL for the RecordBatch and Buffer metadata (https://github.com/apache/arrow/blob/master/format/Message.fbs#L210), it says the offset is "The relative offset into the shared memory page where the bytes for this buffer starts". This is somewhat rigid because it means that file-like record batches are not easily relocatable – by that definition, the offset would need to be the position in the file relative to the start, not the start of the record batch. This is what the C++ implementation is doing now.

Here's my idea: add an enum flag to RecordBatch that indicates whether the buffer offsets are absolute (relative to the start of the file or shared memory block) or relative to a contiguous blob of bytes (what the Java file implementation is doing now). The latter is not good necessarily for shared memory because it presumes contiguousness, but it also makes record batches relocatable when they are (e.g. in a file-like setting).

Relocatable record batch metadata / fully relative offsets is also better for RPC / socket-based exchange (which is effectively the same as sending a segment of the current "file format"), so that's another argument for adding that as an option.

I don't think I can make an argument that either absolute (needed for general shared memory IPC) or relative (better for file / RPC) offsets should be the only option available to the exclusion of the other.

[toddfarmer]

Note: Comment by Wes McKinney (wesm):
On this layout:

<metadata size: int32> <metadata> <buffers> 

One issue we need to address is alignment / padding. The <metadata size: int32> <metadata> block should be 64-byte aligned. I don't think that the flatbuffers library cares about the alignment, but the buffer block should start on a 64-byte offset. Whatever we decide we will want to write this up in the format docs

[toddfarmer]

Note: Comment by Wes McKinney (wesm):
I talked this proposal through with Julien yesterday and we should be able to manage without additional metadata by treating the contiguous buffer block written as part of the file format as a "memory page" for the purposes of record batch reconstruction – so we will not consider the offset from the beginning of the physical file to be semantically meaningful. This is distinct from the shared memory IPC case where the buffers may occur in many different places in one or more shared memory regions.

I'll write up some details under format/ about this more precisely in the course of implementing this in my patch for ARROW-363.

[toddfarmer]

Note: Comment by Wes McKinney (wesm):
This was partially addressed in apache/arrow@e3c167b. Leaving this open until we have some more functioning integration tests

[toddfarmer]

Note: Comment by Wes McKinney (wesm):
This was resolved in apache/arrow@e3c167b – other incompatibilities are likely to be data-type-specific from here on out