theamit45 commented on issue #12099: URL: https://github.com/apache/iceberg/issues/12099#issuecomment-2613324335
### Context - **Current Behavior:** List APIs return all records in a single response without pagination. - **Observed Issues:** Large datasets cause performance issues, including slower response times, increased memory consumption, and higher server load. - **User Impact:** Developers consuming these APIs face difficulties in efficiently handling and processing large responses, requiring additional client-side logic to mitigate issues. ### Expected Behavior - Implement pagination in List APIs to limit the number of records returned per response. - Provide query parameters like `page`, `limit`, or `offset` for clients to specify pagination preferences. - Include metadata in the response, such as `total records`, `current page`, and `total pages`, to improve usability. ### Proposed Solution 1. Add support for pagination to all List APIs using a standard approach (e.g., offset-based or cursor-based pagination). 2. Update the response format to include pagination metadata (e.g., `page`, `limit`, `total_records`, `total_pages`). 3. Ensure backward compatibility by maintaining the current behavior for clients that do not provide pagination parameters. 4. Update documentation to reflect the new pagination functionality and include examples of paginated API usage. 5. Add test cases to validate the correctness of paginated responses, covering edge cases like empty datasets, invalid page numbers, and large datasets. ### Benefits - **Improved Performance:** Pagination reduces the response size and optimizes server resource usage. - **Scalability:** APIs become more scalable and maintain performance with increasing dataset sizes. - **Better Developer Experience:** Developers can handle data more effectively, with less need for client-side workarounds. ### Additional Considerations - Benchmark the performance of different pagination strategies to choose the most efficient implementation. - Ensure that paginated APIs work seamlessly with other query parameters, such as filters and sorting. --- Was this feedback helpful? **Reviewer ID:** 0c6cda -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
