liurenjie1024 closed issue #522: Improvement: reuse the tcp connection when
plan files
URL: https://github.com/apache/iceberg-rust/issues/522
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the spec
Xuanwo commented on issue #522:
URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2270991646
> I also have some local code that reuses the same OpenDAL operator rather
than creating a new one each time. I'd not submitted it yet as I wasn't sure of
the validity of doing tha
sdd commented on issue #522:
URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2270982857
I also have some local code that reuses the same OpenDAL operator rather
than creating a new one each time. I'd not submitted it yet as I wasn't sure of
the validity of doing that in
Xuanwo commented on issue #522:
URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2270676235
Also, cc @sdd, who is focusing on the iceberg benchmark now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and us
chenzl25 commented on issue #522:
URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2270672894
I have already verified that using a global HTTP client works in this issue
and the performance improvement is impressive (about 5 times faster: from
500+ms to 100+ms).
http
Xuanwo commented on issue #522:
URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2270668680
> Which opendal's version do we need to bump into?
I'm guessing it will be included in our next release `0.49`.
--
This is an automated message from the Apache Git Service.
chenzl25 commented on issue #522:
URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2270665390
> We will need
[apache/opendal#4967](https://github.com/apache/opendal/issues/4967) for this.
I'm working on it now.
Which opendal's version do we need to bump into?
--
Xuanwo commented on issue #522:
URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2270617271
We will need https://github.com/apache/opendal/issues/4967 for this. I'm
working on it now.
--
This is an automated message from the Apache Git Service.
To respond to the message
Xuanwo commented on issue #522:
URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2270424332
Thanks for the sharing. I believe this can be addressed by resuing the same
http client.
--
This is an automated message from the Apache Git Service.
To respond to the message, p
chenzl25 commented on issue #522:
URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2270355507
However, risingwave the database use iceberg-rust seems can't reuse any TCP
connection. You can see that every HTTP request is in its own TCP connection.
https://github.com
chenzl25 commented on issue #522:
URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2270352785
I have tested duckdb and clickhouse to scan an iceberg table in MinIO as
well. They can reuse a TCP connection to send HTTP requests. BTW, clickhouse
can reuse connections aggres
chenzl25 commented on issue #522:
URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2270330523
Great, Looking forward to it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
Xuanwo commented on issue #522:
URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2269056564
Hi, OpenDAL handles those connection-related tasks (by reqwest).
Currently, FileIO builds new operators every time:
https://github.com/apache/iceberg-rust/blob/4083f8
liurenjie1024 commented on issue #522:
URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2269007589
> Considering I am trying to read an iceberg table from S3. Currently,
`plan_files()` seems unable to reuse the TCP connection for HTTP requests. It
will lead to a relativel
14 matches
Mail list logo
