jyothsnakonisa commented on code in PR #181:
URL:
https://github.com/apache/cassandra-analytics/pull/181#discussion_r2923726305
##########
cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/bulkwriter/BulkSparkConf.java:
##########
@@ -207,6 +208,7 @@ public BulkSparkConf(SparkConf conf, Map<String, String>
options, @Nullable Logg
this.ttl = MapUtils.getOrDefault(options, WriterOptions.TTL.name(),
null);
this.timestamp = MapUtils.getOrDefault(options,
WriterOptions.TIMESTAMP.name(), null);
this.quoteIdentifiers = MapUtils.getBoolean(options,
WriterOptions.QUOTE_IDENTIFIERS.name(), false, "quote identifiers");
+ this.skipSecondaryIndexCheck = MapUtils.getBoolean(options,
WriterOptions.SKIP_SECONDARY_INDEX_CHECK.name(), false, "skip secondary index
check");
Review Comment:
Should the default value be true?
##########
cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/bulkwriter/TableSchema.java:
##########
@@ -79,7 +80,15 @@ public TableSchema(StructType dfSchema,
this.quoteIdentifiers = quoteIdentifiers;
validateDataFrameCompatibility(dfSchema, tableInfo);
- validateNoSecondaryIndexes(tableInfo);
+ // If a table has indexes on it, some external process (application,
DB, etc.) is responsible for rebuilding
+ // indexes on the table after the bulk write completes; cassandra does
this as part of the SSTable import
+ // process today. 2i and SAI have different ergonomics here regarding
if stale data is served during index build;
+ // ultimately we want the bulk writer to also write native SAI index
files alongside sstables but until
+ // then, this is allowable and fine for users who Know What They're
Doing.
+ if (!skipSecondaryIndexCheck)
Review Comment:
Can you please add a test if possible that check this behavior, basically
the if block
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
