cram support

[brentp]

we are starting to have a push to shift to CRAM over BAM for storage reasons. I know this will be a huge development effort, but perhaps we can start with a few discrete tasks. Here are some obvious ones:

• implement golomb-rice en/decoding
• implement itf8 en/decoding
• Elias gamma en/decoding
• sub exponential en/decoding

I could prototype some of those if we agree on an API and if there is interest in having cram.

[kortschak]

I have no objection to CRAM support, it's just that I don't use it and the spec is pretty big. In the long run I guess we should support it.

Would you like to write up a proposal for the API here?

[brentp]

Yes, I will write up a proposal here. A cursory look shows that a cursory look is insufficient for me to get a good handle on what is needed, but I'll tackle this in January.

[brentp]

For the codecs I listed above, do you agree that BinaryUnmarshaler is the obvious interface?

there will be a lot of machinery around that for CRAM, but implementing each of those is no small task either...

[kortschak]

For the codecs I listed above, do you agree that BinaryUnmarshaler is the obvious interface?

That seems fair in the first instance.

[kortschak]

Does that have implications on the bam writing method name for hts/bam and the implementation of BinaryUnmarshaler for sam.Record?

[brentp]

I'm not sure.
After looking at it more, I thought maybe

type Decoder interface {
    UnmarshalInts([]int) (n, err)
    UnmarshalBytes([]bytes) (n, err)
}

might be a better interface unless we unsafely convert the []bytes.

After that, I was struck by the amount of work this will be.

[kortschak]

After that, I was struck by the amount of work this will be.

Indeed. I spent a day reading and sketching about a year back and decided I didn't care enough.

[sb10]

I'll note that the Sanger Institute now produces cram as our "raw" sequencing data, that the EBI's sequencing archive is pushing toward cram (and may have to go back and convert everything to cram to save space), and major collaborative projects tend to use cram as their interchange format.

Sooner rather than later, cram support is going to be vital. I'd be interested in investigating the use of this go API for use at the Sanger, but without cram support it's a non-starter. I'd suggest looking at how C htslib added sort-of-seamless cram support (in a relatively short amount of time?), and make cram/bam support in this API even more seamless.

[kortschak]

@sb10 I had a look at the htslib cram fs and it is not trivial - as is the case for the spec. Yes, it's something that we should have (hence this issue), but developer resources are limited. Just the codec code is 6.3ksloc of dense htslib-style C and the entire fs is 14.5ksloc. An alternative I used previously would be to interface to the C (this is what sambamba does) - this is not an option here since we want hts to be pure Go and the data representation differences would require significant massaging at the interface.

[jkbonfield]

The are two C implementations with a shared common history:

https://github.com/samtools/htslib/tree/develop/cram
https://github.com/jkbonfield/io_lib/tree/master/io_lib

Io_lib was the original, which got copied to htslib, but some divergence happened immediately thereafter (htslib-ifying the code basically). Both have been maintained since then.

My code can never generate gamma, elias gamma or subexponential coded data and frankly it was an accident that I left in the beta code. Furthermore it only uses huffman in the most trivial case of one single symbol with 0-bit length code. It's entirely up to you which you use for the encoder too - so start with the minimal necessary. I'm not sure which codecs the Java implementation can use, but it's reduced over time. The basic premise now is that ideally the CORE block shouldn't be used at all as it harms partial the decoding ability (eg "I don't care about qualities and aux tags, but get me the rest").

I'd also suggest you use the cram_dump -v tool from Staden io_lib as this will basically walk you through the steps being taken when decoding a record. I'd still be interested though to know where this explains something that the specification didn't! We fixed a lot of omissions in the early days, but for sure there are many more in the spec left, but it's harder to spot them once you've learnt how it works.

PS. Learn from my mistake - separate out reference handling into its own module, preferably not CRAM specific as lots of tools need reference querying. It's a complicated enough beast by itself that it warrants being its own thing. I think that code has been my biggest source of bugs!

[sb10]

@kortschak I agree that remaining pure Go is an important feature.

If you do find the time and resources to work on this though, as @jkbonfield alludes to, the current implementations are more important than the spec (which may need to be improved).

So just stick to codecs that are already implemented in Go (if possible); James told me the main ones used right now are gzip, bzip2, lzma, rANS, huffman in single-code-only mode (so not huffman at all; just "const").

As long as it passes round-trip tests with samtools, it will be good enough for use.

[brentp]

@jkbonfield thanks for the insight. so what codecs are seen in the wild?
just golomb-rice (and itf8?)?

@kortschak maybe biogo/hts could get a summer of code student to work on cram support.

[brentp]

Here's my immediate method for supporting cram in the short term:

type reader struct {
    io.ReadCloser
    cmd *exec.Cmd
}

func (r *reader) Close() error {
    if err := r.cmd.Wait(); err != nil {
        return err
    }
    return r.ReadCloser.Close()
}

// NewReader returns a bam.Reader from any path that samtools can read.
func NewReader(path string, rd int, fasta string) (*bam.Reader, error) {
    cmd := exec.Command("samtools", "view", "-T", fasta, "-b", "-u", "-h", path)
    cmd.Stderr = os.Stderr
    pipe, err := cmd.StdoutPipe()
    if err != nil {
        return nil, err
    }
    if err = cmd.Start(); err != nil {
        pipe.Close()
        return nil, err
    }
    cr := &reader{ReadCloser: pipe, cmd: cmd}
    return bam.NewReader(cr, rd)
}

[kortschak]

Before we have functionality, I'd like to see a proposal for the API. If it is the case that the API will be exactly the same as it is now, them that's fine, but it needs to be discussed. For an idea of what I'm looking for see some of the briefer golang proposals.

[jkbonfield]

API-wise, things to consider for reading would be a way to indicate which fields are required by the caller and which are not (eg maybe we don't care about qualities, or aux tags) and whether there is a need to generate MD and NM tags. This is irrelevant in BAM as it costs the same time regardless, but can be significant in CRAM (consider samtools flagstat timings).

On the encoding side there are more options than BAMs compression level; basic CRAM compression level too of course but also how many reads/bases per slice, whether to do reference based encoding, embedded references or referenceless, what compression codecs you are willing to make use of (zlib for sure, but also maybe bzip2 and/or lzma have their uses and are part of CRAM spec).

These (except currently bzip2/lzma) are all controllable directly from samtools command line too via the --input-fmt-option and --output-fmt-option arguments, or as -I and -O format adjustments. Eg -O cram,seqs_per_slice=1000. So you can still generate an API capable of setting these things while falling back to an external implementation for now.