hamilton-earthscope opened a new pull request, #627: URL: https://github.com/apache/iceberg-go/pull/627
In #622 , specifically commit 5eb2aba5c1afc071405d89a4e43c17a9939b5435, I introduced a map to cache partition type info. However, during testing I did not account for the fact that this map could be written to in parallel by the fanout writer when there are more than 1 worker. Introducing a mutex on the `PartitionSpec` struct to protect the cache writes would require changing all pass-by-value to pass-by-ref and this would touch a lot of code. Instead, I just removed the cache. The other optimizations in #622 introduced after 5eb2aba5c1afc071405d89a4e43c17a9939b5435 obviate the need for this cache. I am seeing no discrepancy in write throughput with `go test -run=NONE -bench=BenchmarkPartitionedWriteThroughput -benchtime=5s -benchmem` before/after this change: the difference in records written/sec is noise (sometimes ever-so-slightly higher, sometimes ever-so-slightly lower). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
