yifan-c commented on code in PR #181:
URL:
https://github.com/apache/cassandra-analytics/pull/181#discussion_r2920791143
##########
cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/bulkwriter/WriterOptions.java:
##########
@@ -132,4 +132,12 @@ public enum WriterOptions implements WriterOption
* - a failure otherwise
*/
JOB_TIMEOUT_SECONDS,
+ /**
+ * Option to bypass the secondary index validation check during bulk write
job setup.
+ * By default, bulk writes to tables with secondary indexes are rejected.
+ * Setting this option to {@code true} allows bulk writes to proceed on
tables that have secondary indexes,
+ * with the understanding that the secondary indexes will NOT be updated
by the bulk write and must be
+ * rebuilt separately after the job completes.
+ */
+ SKIP_SECONDARY_INDEX_CHECK,
Review Comment:
How about defining it in the spark conf, instead of write options?
The rationale is that the toggle is to for advanced use case and not
directly related to the write behavior. There is another existing spark conf,
`org.apache.cassandra.spark.bulkwriter.BulkSparkConf#SKIP_CLEAN` that skips
cleaning up SSTable when job fails.
Admittedly, there are some existing inconsistency of where to have conf and
where to have write options in the project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
