Skip to content

Updated the Spark SQL Programming guide with Custom object encoding for Dataset and unsupported operation error handling #16997

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HarshSharma8 wants to merge 5 commits into from
Closed

Updated the Spark SQL Programming guide with Custom object encoding for Dataset and unsupported operation error handling #16997

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
103906f
Updated the SQL programming guide to explain about the Encoding opera…
HarshSharma8 Feb 20, 2017
9c8f63c
Updated the docs to match the voice of my updates in SQL Programming …
HarshSharma8 Feb 20, 2017
7a539a7
Modified the content and replaced the code block inside back-ticks in…
HarshSharma8 Feb 21, 2017
d49ae50
Updated the content to provide object name
HarshSharma8 Feb 21, 2017
c2fd0ad
Updated the content to provide object name
HarshSharma8 Feb 21, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/sql-programming-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -297,6 +297,9 @@ reflection and become the names of the columns. Case classes can also be nested
types such as `Seq`s or `Array`s. This RDD can be implicitly converted to a DataFrame and then be
registered as a table. Tables can be used in subsequent SQL statements.

Spark Encoders are used to convert a JVM object to Spark SQL representation. When we want to make a datase, Spark requires an encoder which takes the form Encoder[T] where T is the type we want to be encoded. When we try to create dataset with a custom type of object, then may result into <b>java.lang.UnsupportedOperationException: No Encoder found for Object-Name</b>.
Copy link
Member

@srowen srowen Feb 20, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's minor, but there are enough problems with the text to call it out. Please match the voice of the other text and avoid 'we'. Typos: "datase", "spark sql" and "kryo" for example. Use back-ticks to consistently format code if you're going to. What is Object-Name?

Copy link
Author

@HarshSharma8 HarshSharma8 Feb 20, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello srowen,
I have updated the content to match the void of the content, you can have another look at it.

To overcome this problem, we use the kryo encoder. It generally tells spark sql to encode our custom object, so that the operation could find this encoded object.

{% include_example schema_inferring scala/org/apache/spark/examples/sql/SparkSQLExample.scala %}
</div>

Expand Down