Skip to content

Issue-62: Add possibility to load a list of comma-separated files.#63

Merged
JulienPeloton merged 1 commit intomasterfrom
inputPath
Dec 3, 2018
Merged

Issue-62: Add possibility to load a list of comma-separated files.#63
JulienPeloton merged 1 commit intomasterfrom
inputPath

Conversation

@JulienPeloton
Copy link
Copy Markdown
Member

What is new in this PR?

This PR allows to load a list of comma-separated FITS files as described in #62:

// Scala
val fns = "/path/to/file1.fits,/path/to/file2.fits,..."
val df = spark.read.format("fits").option("hdu", 1).load(fns)
# Python
fns = "/path/to/file1.fits,/path/to/file2.fits,..."
df = spark.read.format("fits").option("hdu", 1).load(fns)

How this has been tested?

New unit test added.

Other changes?

Bump spark version to 2.4.0 for the build.

@codecov-io
Copy link
Copy Markdown

Codecov Report

Merging #63 into master will increase coverage by 0.06%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #63      +/-   ##
==========================================
+ Coverage   89.37%   89.44%   +0.06%     
==========================================
  Files           9        9              
  Lines         480      483       +3     
  Branches       87       90       +3     
==========================================
+ Hits          429      432       +3     
  Misses         51       51
Impacted Files Coverage Δ
...strolabsoftware/sparkfits/FitsSourceRelation.scala 97.46% <100%> (+0.09%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3442a2c...aecc184. Read the comment docs.

@JulienPeloton
Copy link
Copy Markdown
Member Author

Note however that loading a list of paths turns out to not be as easy as I would think.
The input parameters are implicitly read as Map[String, String] (from RelationProvider), hence requiring an array requires some deeper changes. In addition, Spark Data Source API is evolving quickly to V2 (spark-fits is V1), which makes things highly confusing for me :-(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants