lokeshj1703 commented on code in PR #18132:
URL: https://github.com/apache/hudi/pull/18132#discussion_r3050114505
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala:
##########
@@ -97,8 +97,11 @@ object AvroConversionUtils {
* TODO convert directly from GenericRecord into InternalRow instead
*/
def createDataFrame(rdd: RDD[GenericRecord], schemaStr: String, ss:
SparkSession): Dataset[Row] = {
- if (rdd.isEmpty()) {
- ss.emptyDataFrame
+ // Avoid calling isEmpty() which can cause serialization issues with
Ordering$Reverse
+ // Check partition count instead, which doesn't require task serialization
+ val structType = convertAvroSchemaToStructType(new
Schema.Parser().parse(schemaStr))
Review Comment:
The code is different in 0.15.1. This optimisation is not required there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
