hcrosse opened a new issue, #889: URL: https://github.com/apache/iceberg-go/issues/889
### Problem The Iceberg Java implementation used \`added_data_files_count\`, \`existing_data_files_count\`, and \`deleted_data_files_count\` as field names for manifest list entries (IDs 504/505/506) from its initial release until apache/iceberg#5338 fixed the names to match the spec in the 1.4 release (merged 2023-10-02). Any table written before that fix, or by an engine still on an older Java version (Athena, some Trino deployments, etc.), has the legacy names embedded in the writer schema of every manifest list OCF file. ### Symptom When iceberg-go reads such a manifest list, the three count fields decode to zero regardless of their actual values. Downstream this causes \`HasAddedFiles()\` and \`HasExistingFiles()\` to return \`false\`, so fast-append operations silently drop all inherited manifests and \`Table.Scan()\` returns no data. ### Root cause hamba/avro's OCF decoder resolves writer-schema field names against struct \`avro:\` tags. When the writer schema says \`added_data_files_count\` and the struct tag says \`avro:"added_files_count"\`, there is no match and the struct field stays zero. ### Fix Add parallel struct fields with the legacy tags and coalesce in the accessors: \`\`\`go AddedFilesCount int32 \`avro:"added_files_count"\` AddedDataFilesCount int32 \`avro:"added_data_files_count"\` // pre-1.4 Java name \`\`\` \`AddedDataFiles()\` returns \`AddedFilesCount\` when non-zero, falling back to \`AddedDataFilesCount\`. Same pattern for existing/deleted. Read-path only, no write-path or schema-registration changes needed. ### Relation to existing issue apache/iceberg-go#869 fixes the downstream data-loss (manifest inheritance filter in fast-append). This issue fixes the upstream root cause: zero counts when reading pre-1.4 manifest lists in the first place. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
