npawar commented on a change in pull request #6332:
URL: https://github.com/apache/incubator-pinot/pull/6332#discussion_r541176761
##########
File path:
pinot-spi/src/main/java/org/apache/pinot/spi/ingestion/batch/BatchConfig.java
##########
@@ -28,73 +28,57 @@
* Provides all config related to the batch data source, as configured in the
table config's ingestion config
*/
public class BatchConfig {
+ private final Map<String, String> _batchConfigMap;
+
private final String _tableNameWithType;
- private final String _type;
private final String _inputDirURI;
private final String _outputDirURI;
- private final String _fsClassName;
- private final Map<String, String> _fsProps = new HashMap<>();
+ private final String _inputFsClassName;
+ private final Map<String, String> _inputFsProps = new HashMap<>();
+ private final String _outputFsClassName;
+ private final Map<String, String> _outputFsProps = new HashMap<>();
private final FileFormat _inputFormat;
private final String _recordReaderClassName;
private final String _recordReaderConfigClassName;
private final Map<String, String> _recordReaderProps = new HashMap<>();
- private final Map<String, String> _batchConfigMap = new HashMap<>();
-
public BatchConfig(String tableNameWithType, Map<String, String>
batchConfigsMap) {
- _tableNameWithType = tableNameWithType;
-
- _type = batchConfigsMap.get(BatchConfigProperties.BATCH_TYPE);
- Preconditions.checkState(_type != null, "Property: %s cannot be null",
BatchConfigProperties.BATCH_TYPE);
+ _batchConfigMap = batchConfigsMap;
- String inputDirURIKey =
BatchConfigProperties.constructBatchProperty(_type,
BatchConfigProperties.INPUT_DIR_URI);
- _inputDirURI = batchConfigsMap.get(inputDirURIKey);
- Preconditions.checkState(_inputDirURI != null, "Property: %s cannot be
null", inputDirURIKey);
+ _tableNameWithType = tableNameWithType;
- String outputDirURIKey =
BatchConfigProperties.constructBatchProperty(_type,
BatchConfigProperties.OUTPUT_DIR_URI);
- _outputDirURI = batchConfigsMap.get(outputDirURIKey);
- Preconditions.checkState(_outputDirURI != null, "Property: %s cannot be
null", outputDirURIKey);
+ _inputDirURI = batchConfigsMap.get(BatchConfigProperties.INPUT_DIR_URI);
+ Preconditions.checkState(_inputDirURI != null, "Property: %s cannot be
null", BatchConfigProperties.INPUT_DIR_URI);
Review comment:
As discussed offline, please relax this requirement of input and output
dir, so that user can pass in their own BatchConfig with null input and output
for file uploads
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
