geruh commented on issue #9604: URL: https://github.com/apache/iceberg/issues/9604#issuecomment-1928799128
Based on my current understanding, the `InMemoryCatalog` serves as the supporting catalog for the RESTCatalog. When a table is created the RESTTableOperations, returns a ResolvingFileIO object that encapsulates a HadoopFileIO instance. In scenarios like the one demonstrated by the testAppendFiles method (within the REST catalog context), after committing changes, it appears that only InMemoryFileIO is used for referencing metadata files. <img width="1208" alt="Screenshot 2024-02-05 at 8 47 05 PM" src="https://github.com/apache/iceberg/assets/19402187/21cbeebe-5e1f-4dee-820b-438063b6bdc3";> However, attempting to access these metadata files with the REST table's HadoopFileIO results in a NotFoundException. This issue does not occur when accessing the files through the backendCatalog, indicating that HadoopFileIO can access files that InMemoryFileIO cannot. `ResolvingFileIO` <img width="1204" alt="Screenshot 2024-02-05 at 8 52 05 PM" src="https://github.com/apache/iceberg/assets/19402187/bdfe9909-f6ca-4244-8863-1f4de1b0c68e";> `InMemoryFileIO` <img width="1265" alt="Screenshot 2024-02-05 at 8 53 31 PM" src="https://github.com/apache/iceberg/assets/19402187/2b6f4957-5eca-4887-b67c-9cb656e0ab7a";> Additionally, when attempting to read a manifest list or snapshot created on the table using InMemoryFileIO, a NotFoundException is also encountered. <img width="1214" alt="Screenshot 2024-02-05 at 8 56 28 PM" src="https://github.com/apache/iceberg/assets/19402187/74a1d8ec-0cb1-4215-bfd0-d0923bbc8cc1";> It's also worth noting that the test methods utilizing this functionality in the TestRESTCatalog class initiate a new instance of InMemoryFileIO for each test, following the conf passed in to the catalog. https://github.com/apache/iceberg/blob/main/core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java#L2323 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
