Hi all, Is ShortCircuit read faster than legacy read which goes through data nodes?
I have evaluated SCR and legacy local read on both HDD and NVMe SSD. However, I have not seen any results that SCR is faster than legacy. Rather, SCR was slower than legacy when using NVMe SSDs because of the operation that initially to get the file descriptors. When I configured SCR, getBlockReader() elapsed time is slower than legacy local read. When I used NVMe SSD, I also found that DFSInputStream: dataIn.read() time is really similar to hardware limit. (8MB/0.00289sec) = 2800MB/s I checked the logs that the execution time measured by the application took 5ms to process 8mb. There is a 3 ms runtime difference between blockReader.doRead () in DFSInputStream.java and dataIn.read () in BlockReader.java. Where is this 3ms difference from? Thank you ᐧ
