[SPARK-33669] Wrong error message from YARN application state monitor when sc.stop in yarn client mode

[sqlwindspeaker]

What changes were proposed in this pull request?

This change make InterruptedIOException to be treated as InterruptedException when closing YarnClientSchedulerBackend, which doesn't log error with "YARN application has exited unexpectedly xxx"

Why are the changes needed?

For YarnClient mode, when stopping YarnClientSchedulerBackend, it first tries to interrupt Yarn application monitor thread. In MonitorThread.run() it catches InterruptedException to gracefully response to stopping request.

But client.monitorApplication method also throws InterruptedIOException when the hadoop rpc call is calling. In this case, MonitorThread will not know it is interrupted, a Yarn App failed is returned with "Failed to contact YARN for application xxxxx; YARN application has exited unexpectedly with state xxxxx" is logged with error level. which confuse user a lot.

Does this PR introduce any user-facing change?

Yes

How was this patch tested?

very simple patch, seems no need?

[HyukjinKwon]

cc @tgravescs and @mridulm FYI

[tgravescs]

ok to test

[SparkQA]

Test build #132372 has finished for PR 30617 at commit 4cdad1b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36973/

[mridulm]

Looks good to me, but will let @tgravescs take a look as well.

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36973/

[sqlwindspeaker]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36973/

seems not related with this patch

[tgravescs]

changes look fine.

I assume you manually tested?

[SparkQA]

Test build #132453 has finished for PR 30617 at commit 57ea973.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37054/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37054/

[sqlwindspeaker]

changes look fine.

I assume you manually tested?

Yes，but not fully。
This issue consist of 2 parts: hadoop ipc will raise InterruptedIOException, and InterruptedIOException should be handled correctly.

I test part 2 by raising an InterruptedIOException in client.monitorApplication， but for part 1 I don't find a easy way to mock.

But I believe the error trace is enough to show it will raise InterruptedIOException.

[mridulm]

Thanks @sqlwindspeaker, merging to master and 3.1