singhpk234 commented on code in PR #15595:
URL: https://github.com/apache/iceberg/pull/15595#discussion_r2934439471
##########
core/src/main/java/org/apache/iceberg/rest/RESTTableScan.java:
##########
@@ -74,16 +71,15 @@ class RESTTableScan extends DataTableScan {
.build();
private final RESTClient client;
- private final Map<String, String> headers;
- private final TableOperations operations;
- private final Table table;
- private final ResourcePaths resourcePaths;
- private final TableIdentifier tableIdentifier;
- private final Set<Endpoint> supportedEndpoints;
+ private final Supplier<Map<String, String>> headers;
+ private final String planTableScanPath;
+ private final Function<String, String> planPath;
+ private final String fetchScanTasksPath;
+ private final boolean supportsAsync;
+ private final boolean supportsCancel;
+ private final boolean supportsFetchTasks;
Review Comment:
One can prewarm the caches is one thing i was thinking ? orthogonally there
may be cases where some one is producing a governed set in a sense they gave
plan-id and took a while to materialize, until the client polls, one they are
done producing they just give the whole list of materialized, then they give it
one go (I totally agree, we are still constrained by size of the payload here),
which would be ok to it in spark side semantics where we wait to collect all
the splits on the driver before handling of to the executor, rather than having
a streaming splits like Trino ?
I can say this falls under example of "bad architecture" but i believe these
are possible as of today.
I think one other way to put is its the **server** who is orchestrating the
client to come to them by giving them the planTask, they are in their full
liberty to not do so and achieve the architecture they want and co-exit and
from the spec / ref impl perspective we can take a stand that this is what we
recommend, though i feel its a bit strict but i can understand how this
simplifies things
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
