JamesGuthrie commented on PR #3212: URL: https://github.com/apache/thrift/pull/3212#issuecomment-3334844392
For additional context and real-world anchoring, the memory bloat was discovered in the context of analysing memory usage when reading parquet files with the apache/arrow-go library. The following screenshots show the heap profile of opening a ~1GB parquet file with 18300 row groups using [apache/arrow-go/v18/parquet/file.NewParquetReader](https://pkg.go.dev/github.com/apache/arrow-go/v18/parquet/file#NewParquetReader), both before and after this change. The code used to obtain these results: ```go package main import ( "os" "runtime" "runtime/pprof" "github.com/apache/arrow-go/v18/parquet" "github.com/apache/arrow-go/v18/parquet/file" ) func main() { f, err := os.Open("/path/to/sensor_readings_1gb.parquet") if err != nil { panic(err.Error()) } r, err := file.NewParquetReader(f, file.WithReadProps(&parquet.ReaderProperties{ BufferSize: 4 * 1024 * 1024, BufferedStreamEnabled: true, })) if err != nil { panic(err.Error()) } p, err := os.Create("/path/to/heap.pprof") if err != nil { panic(err.Error()) } defer p.Close() runtime.GC() err = pprof.WriteHeapProfile(p) if err != nil { panic(err.Error()) } runtime.GC() runtime.KeepAlive(r) } ``` The test file is available at this link: https://jg-test-document-public-bucket.s3.eu-central-1.amazonaws.com/sensor_readings_1gb.parquet Before: <img width="1279" height="587" alt="Screenshot 2025-09-25 at 17 49 38" src="https://github.com/user-attachments/assets/beba7f4c-44dd-4d91-8662-952c403e72bb"; /> After: <img width="1279" height="587" alt="Screenshot 2025-09-25 at 17 49 49" src="https://github.com/user-attachments/assets/0a259c72-7b45-41b6-aa08-74a770e9734a"; /> Note how memory usage of the parquet file comes primarily from metadata, the size of which is reduced from ~460MB to ~60MB. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
