twuebi opened a new issue, #1510: URL: https://github.com/apache/arrow-go/issues/1510
### Describe the bug, including details regarding any error messages, version, and platform. `rowGroupWriter.Close` returns as soon as `checkRowsWritten` reports a row-count mismatch, before closing any column writer (https://github.com/apache/arrow-go/blob/32947b25/parquet/file/row_group_writer.go#L239-L242). The open column writers never release their encoders, dictionary memo tables, level buffers or bits buffers. The error is cached, so a later `Close` returns it without freeing anything. With `memory.CheckedAllocator` the bytes stay charged, and with the cgo mallocator they leak. The mismatch isn't only caused by misuse. When a sink fails partway through a column that isn't the first in a serial row group, the column has written fewer rows than the earlier ones. `pqarrow.FileWriter.Write` then calls `Close` (https://github.com/apache/arrow-go/blob/32947b25/parquet/pqarrow/file_writer.go#L384-L386), which hits the mismatch and leaves that column writer unclosed. The repros below run with https://github.com/apache/arrow-go/pull/1506 applied, which fixes the other close-path leaks. Buffered row group with mismatched row counts, in `parquet/file` (`newPlainAndDictStringSchema` and `writeStringColumn` are the helpers from #1506's `file_writer_test.go`, and `writeStringColumn` writes 300 rows); 2304 bytes stay charged: ```go mem := memory.NewCheckedAllocator(memory.DefaultAllocator) var sink bytes.Buffer writer, _ := file.NewParquetWriterWithError(&sink, newPlainAndDictStringSchema(t), file.WithWriterProps(parquet.NewWriterProperties(parquet.WithAllocator(mem)))) rgw, _ := writer.AppendBufferedRowGroupChecked() c0, _ := rgw.Column(0) writeStringColumn(c0) // 300 rows c1, _ := rgw.Column(1) c1.(*file.ByteArrayColumnChunkWriter).WriteBatch([]parquet.ByteArray{[]byte("a")}, []int16{1}, nil) rgw.Close() // row mismatch for buffered row group: 0, column: 1, count expected: 300, actual: 1 writer.Close() mem.AssertSize(t, 0) // got=2304 ``` Sink failure in the middle of the second column, through pqarrow; 64 bytes stay charged: ```go type failAfter struct{ n, limit int } func (s *failAfter) Write(p []byte) (int, error) { if s.n+len(p) > s.limit { return 0, errors.New("sink full") } s.n += len(p) return len(p), nil } mem := memory.NewCheckedAllocator(memory.DefaultAllocator) schema := arrow.NewSchema([]arrow.Field{ {Name: "a", Nullable: true, Type: arrow.BinaryTypes.String}, {Name: "b", Nullable: true, Type: arrow.BinaryTypes.String}, }, nil) bldr := array.NewRecordBuilder(mem, schema) for i := 0; i < 300; i++ { bldr.Field(0).(*array.StringBuilder).Append("row-" + strconv.Itoa(i)) bldr.Field(1).(*array.StringBuilder).Append("row-" + strconv.Itoa(i)) } rec := bldr.NewRecordBatch() bldr.Release() writer, _ := pqarrow.NewFileWriter(schema, &failAfter{limit: 3294}, parquet.NewWriterProperties(parquet.WithAllocator(mem), parquet.WithDictionaryDefault(false), parquet.WithDataPageSize(1024), parquet.WithBatchSize(100)), pqarrow.NewArrowWriterProperties(pqarrow.WithAllocator(mem))) writer.Write(rec) // unknown error type: sink full writer.Close() rec.Release() mem.AssertSize(t, 0) // got=64 ``` Both repros pass once `rowGroupWriter.Close` records the mismatch error and still closes every column writer. The open question is whether closing the column writers there is acceptable: it writes their pending pages to the sink for a row group that is already known to be invalid. The file never gets a valid footer because the error is cached. The alternative is a release-only path that discards pending pages without writing them. Seen on main at 32947b25, macOS arm64, Go 1.26.10. ### Component(s) Parquet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
