laskoviymishka commented on issue #2044:
URL: https://github.com/apache/iceberg-go/issues/2044#issuecomment-5845305857
Thanks @zeroshade, happy to coordinate here. I did a quick prototype first
to get real numbers before we settle on ordering.
What I found:
For `_ "catalog/rest"` on recent main, the main dependency groups are
roughly:
- AWS SDK: ~55 packages
- arrow-go: ~44
- parquet: ~15
- substrait: ~11
- geoarrow: 1
So the biggest dependency for a REST/metadata-only consumer is actually AWS,
pulled in only for SigV4. Removing Arrow alone does not get us close to a tiny
binary; I get roughly 33–37 MB as the floor. The main Arrow wins are js/wasm
support, dropping substrait, and a cleaner graph.
I also have two small changes working locally:
1. Drop `pterm` from `table`
Use `text/tabwriter` for schema compatibility output. This removes
`atomicgo.dev/keyboard` and makes `table` / `catalog/rest` build on js/wasm. No
API change.
2. Make SigV4 optional
Move AWS signing into `catalog/rest/sigv4`, with a small signer interface
in `catalog/rest`. This removes AWS SDK from REST unless SigV4 is used.
`rest.WithAwsConfig` changes here, so I think this belongs before v1.
For ordering, I’d do:
1. testkit first
2. pterm + substrait/Arrow isolation
3. SigV4 in parallel
4. decide separately whether a fully Arrow-free metadata core is actually a
v1 goal
The last one is the harder part. `VariantLiteral` and `DecimalLiteral`
expose parquet/Arrow types today, so removing the remaining dependency means
changing the public Literal API. If zero-Arrow core is a v1 goal, we should
decide that now; otherwise I’d stop before that.
Small scope-creep / FYI for the v1 discussion: since we are already looking
at pre-v1 API breaks, there are a few other things probably worth tracking in a
separate v1 cleanup issue rather than mixing into this one:
- `io.IO` context handling
- catalog capability interfaces for views / `RegisterTable`
- duplicate table write API names
- a couple of small public leaks / awkward names
- removing the deprecated compatibility shims that are still around
I would not block v1 on the full `table` package split or on removing Arrow
from the scan path. Those feel like separate follow-ups.
I pushed the two prototypes as PRs (#2060 and #2061) so we have something
concrete to review. @nssalian for visibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
