[GitHub] [iceberg] chenwyi2 opened a new issue, #6326: estimateStatistics cost mush time to compute stats

[GitBox]

chenwyi2 opened a new issue, #6326:
URL: https://github.com/apache/iceberg/issues/6326

   ### Query engine
   
   spark 3.1
   
   ### Question
   
   spark driver will cost too much time to computes stats since listing all the 
files to getstats,and the thread is below:
   
![reliao_img_1669864216597](https://user-images.githubusercontent.com/19389434/204957073-f7672b97-4ad5-418a-a5b1-2f50ec020146.png)
    and the code is 
   ````java
   for (CombinedScanTask task : tasks()) {
         for (FileScanTask file : task.files()) {
           // TODO: if possible, take deletes also into consideration.
           double fractionOfFileScanned = ((double) file.length()) / 
file.file().fileSizeInBytes();
           numRows += (fractionOfFileScanned * file.file().recordCount());
         }
       }
   ````
   maybe we can add an option to disable estimateStatistics ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org