[GitHub] [pinot] rajagopr commented on a diff in pull request #11540: Keep header parsing logic the same in default CSVRecordReader and the line iterator

[via GitHub]

rajagopr commented on code in PR #11540:
URL: https://github.com/apache/pinot/pull/11540#discussion_r1323399278


##########
pinot-plugins/pinot-input-format/pinot-csv/src/main/java/org/apache/pinot/plugin/inputformat/csv/CSVRecordReader.java:
##########
@@ -104,50 +103,42 @@ public void init(File dataFile, @Nullable Set<String> 
fieldsToRead, @Nullable Re
         }
       }
       char delimiter = config.getDelimiter();
-      format = format.withDelimiter(delimiter);
+      format = format.builder().setDelimiter(delimiter).build();
 
       if (config.isSkipUnParseableLines()) {
         _useLineIterator = true;
       }
 
-      String csvHeader = config.getHeader();
-      if (csvHeader == null) {
-        format = format.withHeader();
-      } else {
-        // do not validate header if using the line iterator
-        if (_useLineIterator) {
-          String[] header = StringUtils.split(csvHeader, delimiter);
-          setHeaderMap(header);
-          format = format.withHeader(header);
-          _isHeaderProvided = true;
-        } else {
-          // validate header for the delimiter before splitting
-          validateHeaderForDelimiter(delimiter, csvHeader, format);
-          format = format.withHeader(StringUtils.split(csvHeader, delimiter));
-        }
-      }
-      format = format.withSkipHeaderRecord(config.isSkipHeader());
+      _isHeaderProvided = config.getHeader() != null;

Review Comment:
   Added the check back on the default CSVRecordReader.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org