Update desitarget.io.read_mtl_ledger() to read gzipped files

[geordie666]

It would be convenient to gzip the alt-MTL ledgers, as this would make them about 4x smaller.

However, to speed up the reading of the data MTLs, desitarget.io.read_mtl_ledger() parses the ledgers as pure-ascii files in some places rather than doing a simple Table read.

So, we should update desitarget.io.read_mtl_ledger() to detect that a ledger is in .gz format and, if so, read it using a pure Table read.

[sbailey]

A few comments:

• Table.read('mtl-blat.ecsv.gz', format='ascii.ecsv') is slightly faster than reading non-gzipped Table.read('mtl-blat.ecsv', format='ascii.ecsv'), presumably because the gunzipping overhead is offset by reading 4x fewer bytes from disk.
• It looks like desitarget.io.read_one_mtl_ledger does the manual header parsing because Table.read(filename, comment='#', format='ascii.basic') is ~4x faster than format='ascii.ecsv. However, if you include the time spent manually parsing the header, there is barely any win at all.

i.e. it is possible that the timing optimizations here date back to an earlier version of astropy and simply collapsing this all down to mtl = Table.read(filename, format='ascii.ecsv') might be sufficient to handle both gzipped and not without extra custom handling and without a performance penalty.

[sbailey]

It looks like this was implemented in PR #858 with an additional bugfix in #859. Since the MTL reading is apparently more subtle than a vanilla read of an ecsv file, I suggest adding a few unit tests to confirm that read_one_mtl_ledger continues to work with the various formats in play: regular, gzipped, override, possibly other. These could be mini copies kept as test-data in desitarget itself to enable testing on GitHub, or they could be NERSC-specific tests run on real example ledgers as part of nightly tests.

[geordie666]

Agreed: I'll make sure I do that before closing this issue.