zeroshade commented on issue #438: URL: https://github.com/apache/iceberg-go/issues/438#issuecomment-2906997287
So, looking at the code a bit. One thing that stands out to me as a potential cause of this is that we do `defer counter.W.(io.Closer).Close()` while passing `counter.Count` to the `ToManifestFile` call. The reason why this could be the source is because, in this case, the underlying writer is an `*ocf.Encoder` which will call `Flush` during the call to `Close`. So if we're not fully flushed before calling `wr.ToManifestFile(path, counter.Count)`, then `counter.Count` won't account for the number of bytes that have yet to be flushed, resulting in the incorrect value being written to the manifest. This could also explain the similarity in the sizes that were written depending on how the writes used by hamba/avro/v2/ocf are buffered. I'll try to replicate this situation and then see if forcibly flushing before the call to `ToManifestFile` fixes the issue. If so, then I'll see what a good ergonomic solution to this would be. The entire purpose of using the wrapping write counter is to avoid having to perform the extra request to the object storage to find the size of the written file, instead just tracking the number of bytes we actually write out. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
