[SPARK-51691][CORE][TESTS] SerializationDebugger should swallow exception when try to find the reason of serialization problem

[summaryzb]

What changes were proposed in this pull request?

Catch SparkRuntimeException when deep into serialization exception stack during unit test.

Why are the changes needed?

Present a clearer serialization exception stack hierarchy during test, toString implementation of TreeNode may throw exception since SQLConf.get check. It is helpful for debug the real problem

Does this PR introduce any user-facing change?

Yes, but it only take effect in unit test.
User will see the direct serialization exception and the reference chain beyond the root cause.
Before this pr, user will get confuse when unrelated exception is shown

WARN org.apache.spark.serializer.SerializationDebugger: Exception in serialization debugger
org.apache.spark.SparkRuntimeException: Cannot get SQLConf inside scheduler event loop thread.
    at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotGetSQLConfInSchedulerEventLoopThreadError(QueryExecutionErrors.scala:2002)
    at org.apache.spark.sql.internal.SQLConf$.get(SQLConf.scala:225)
    at org.apache.spark.sql.execution.ScalarSubquery.toString(subquery.scala:69)
    at java.lang.String.valueOf(String.java:2994)
    at scala.collection.mutable.StringBuilder.append(StringBuilder.scala:203)
    at scala.collection.immutable.Stream.addString(Stream.scala:701)
    at scala.collection.TraversableOnce.mkString(TraversableOnce.scala:377)
org.apache.spark.SparkException: Job aborted due to stage failure: Task not serializable: java.io.NotSerializableException: 
org.apache.spark.SimpleFutureAction

After this pr

org.apache.spark.SparkException: Job aborted due to stage failure: Task not serializable: java.io.NotSerializableException: org.apache.spark.SimpleFutureAction
Serialization stack:
	- object not serializable (class: org.apache.spark.SimpleFutureAction, value: org.apache.spark.SimpleFutureAction@4050649d)
	- writeObject data (class: java.util.concurrent.ConcurrentHashMap)
	- object (class java.util.concurrent.ConcurrentHashMap)
	....(not shown)

How was this patch tested?

Pass GitHub Actions

Was this patch authored or co-authored using generative AI tooling?

No

[LuciferYang]

Is it possible to add a new test case? Or which existing test case covers this scenario?

[summaryzb]

Add test SPARK-51691 improveException swallow underlying exception to cover this scenario

[LuciferYang]

can we just add a branch like case s: Object with java.io.Serializable if Utils.isTesting =>

[summaryzb]

This simulate the behavior of toString implementation of TreeNode, since SQLConf.get may throw exception

[LuciferYang]

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

Lines 228 to 229 in 554d678

	} else if (Utils.isTesting) {
	throw QueryExecutionErrors.cannotGetSQLConfInSchedulerEventLoopThreadError()

So is this an issue that can only occur during testing?

[summaryzb]

Yes, it only throw exception in during test

[LuciferYang]

@summaryzb Could you update the description of the pr to clarify that the changes in this PR are intended to present a clearer exception stack hierarchy in the tests? (Is my understanding correct?)
Additionally, I think it would be appropriate to add the [TESTS] tag to the PR title.

[summaryzb]

Update the title and description to clarify that this is a unit test issue

[LuciferYang]

+1, LGTM
Let's wait for another 48 hours, If no one has any other comments, I will merge it.

[the-sakthi]

LGTM