xixipi-lining opened a new pull request, #1225:
URL: https://github.com/apache/iceberg-go/pull/1225
## Problem Statement
Currently, when `partitionedFanoutWriter.Write` completes successfully, it
waits for all upstream producers to finish (`fanoutWorkers.Wait()`), and then
attempts a graceful shutdown by calling `writerFactory.closeAll()`.
However, `writerFactory.closeAll()` iterates through the
`RollingDataWriter`s and calls `closeAndWait()` on each. The `close()` method
does the following:
```go
func (r *RollingDataWriter) close() {
r.cancel() // Problem here!
close(r.recordCh)
}
```
By immediately calling `r.cancel()`, it actively cancels the context passed
to the background `stream` goroutine. Since `recordCh` is buffered (size 64),
it may still contain unprocessed `RecordBatch` items. If the underlying Arrow
functions (e.g., `compute.SortRecordBatch`) or Parquet writers check
`ctx.Done()` during this draining phase, they will abort immediately with a
`context.Canceled` error. This converts what should be a successful, graceful
shutdown into an error, and silently drops the remaining buffered records.
## Proposed Solution
To support proper channel draining, `RollingDataWriter` should rely on
`close(r.recordCh)` as the primary signal for the `stream` goroutine to process
the remaining buffered records and exit naturally.
In this PR:
- We remove `r.cancel()` from `close()`.
- We move `r.cancel()` to the end of `closeAndWait()`, after `r.wg.Wait()`
has returned and all residual batches have been successfully processed and
flushed to Parquet.
This ensures that the memory buffer is correctly fully drained, preventing
silent data loss or unexpected `context.Canceled` panics during the
finalization phase. The overall timeout/abort mechanism is still safely
controlled by the parent context passed into the `Write` function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
