Skip to content

Issue 71: better control of recordLength value, and related error#72

Merged
JulienPeloton merged 1 commit intomasterfrom
issues/71/recordlength
May 13, 2019
Merged

Issue 71: better control of recordLength value, and related error#72
JulienPeloton merged 1 commit intomasterfrom
issues/71/recordlength

Conversation

@JulienPeloton
Copy link
Copy Markdown
Member

@JulienPeloton JulienPeloton commented May 12, 2019

Fix issue #71, by throwing an error if one specifies recordLength < rowSize, and adapting recordLength otherwise (default is recordLength = 1024 Bytes).

Let suppose we have a file with row size 2 KB (> 1 KB).

Case 1: default behaviour

# will work (was not working previously)
# The recordLength is equal to row size.
df = spark.read.format("fits")\
  .option("hdu", 1)\
  .load("path/to/my.fits")

Case 2: specifying a large recordLength

# will work (was working previously)
# The recordLength is equal to 5 KB.
df = spark.read.format("fits")\
  .option("hdu", 1)\
  .option("recordlength", 5*1024)\
  .load("path/to/my.fits")

Case 3: specifying a too small recordLength

# will throw an error (was throwing a cryptic error previously)
df = spark.read.format("fits")\
  .option("hdu", 1)\
  .option("recordlength", 1024)\
  .load("path/to/my.fits")
...
Py4JJavaError: An error occurred while calling o35.load.
: java.lang.AssertionError:
          You specified a recordLength option too small compared to the FITS row size:
          1024 B < 2048 B
	at com.astrolabsoftware.sparkfits.FitsLib$Fits.<init>(FitsLib.scala:196)
...

…owSize, and adapting recordLength otherwise.
@JulienPeloton JulienPeloton added bug Something isn't working BlockOrSplit IO labels May 12, 2019
@codecov-io
Copy link
Copy Markdown

Codecov Report

Merging #72 into master will increase coverage by 0.11%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #72      +/-   ##
==========================================
+ Coverage   90.11%   90.22%   +0.11%     
==========================================
  Files           9        9              
  Lines         526      532       +6     
  Branches       99       99              
==========================================
+ Hits          474      480       +6     
  Misses         52       52
Impacted Files Coverage Δ
.../astrolabsoftware/sparkfits/FitsRecordReader.scala 75.7% <100%> (ø) ⬆️
...scala/com/astrolabsoftware/sparkfits/FitsLib.scala 97.51% <100%> (+0.09%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update de56561...cbd7cf1. Read the comment docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

BlockOrSplit bug Something isn't working IO

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants