Interesting benchmark. Thank you, Daegyu. Can you try a larger file too? Like 128mb or 1gb? HDFS is not optimized for smaller files.
What did you use for benchmark? Daegyu Han <[email protected]>於 2019年8月29日 週四，下午11:40寫道： > Hi all, > > Is ShortCircuit read faster than legacy read which goes through data nodes? > > I have evaluated SCR and legacy local read on both HDD and NVMe SSD. > However, I have not seen any results that SCR is faster than legacy. > > Rather, SCR was slower than legacy when using NVMe SSDs because of the > operation that initially to get the file descriptors. > > When I configured SCR, getBlockReader() elapsed time is slower than legacy > local read. > > When I used NVMe SSD, > I also found that DFSInputStream: dataIn.read() time is really similar to > hardware limit. > (8MB/0.00289sec) = 2800MB/s > > I checked the logs that the execution time measured by the application > took 5ms to process 8mb. > > There is a 3 ms runtime difference between blockReader.doRead () in > DFSInputStream.java and dataIn.read () in BlockReader.java. > Where is this 3ms difference from? > > Thank you > ᐧ >
