RahmanQureshi commented on code in PR #4479:
URL: https://github.com/apache/iceberg/pull/4479#discussion_r1582396495
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreamingRead3.java:
##########
@@ -138,6 +143,57 @@ public void
testReadStreamOnIcebergTableWithMultipleSnapshots() throws Exception
Assertions.assertThat(actual).containsExactlyInAnyOrderElementsOf(Iterables.concat(expected));
}
+ @Test
+ public void
testReadStreamOnIcebergTableWithMultipleSnapshots_WithNumberOfFiles_1()
+ throws Exception {
+ appendDataAsMultipleSnapshots(TEST_DATA_MULTIPLE_SNAPSHOTS);
+
+ Assert.assertEquals(
+ 6,
+ microBatchCount(
+
ImmutableMap.of(SparkReadOptions.STREAMING_MAX_FILES_PER_MICRO_BATCH, "1")));
+ }
+
+ @Test
+ public void
testReadStreamOnIcebergTableWithMultipleSnapshots_WithNumberOfFiles_2()
+ throws Exception {
+ appendDataAsMultipleSnapshots(TEST_DATA_MULTIPLE_SNAPSHOTS);
+
+ Assert.assertEquals(
+ 3,
+ microBatchCount(
+
ImmutableMap.of(SparkReadOptions.STREAMING_MAX_FILES_PER_MICRO_BATCH, "2")));
+ }
+
+ @Test
+ public void
testReadStreamOnIcebergTableWithMultipleSnapshots_WithNumberOfRows_1()
+ throws Exception {
+ appendDataAsMultipleSnapshots(TEST_DATA_MULTIPLE_SNAPSHOTS);
+
+ // only 1 micro-batch will be formed and we will read data partially
+ Assert.assertEquals(
+ 1,
Review Comment:
@singhpk234 👋 Can you help me understand why the expected result here 1 and
not 7? For example, the test "WithNumberOfRows_4" expects all of the test data
to have been processed in 2 micro batches
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
