Thanks, Sam. The root cause is from different OpenMP library. Intel OpenMP will provide better performance as your data shown.
Regarding release, since the license issue[1], we can't ship Intel OpenMP in the binary, but the most of performance boost from MKLDNN is still available. I think it should be acceptable to release 1.6 with MKLDNN + GNU OpenMP for suboptimal performance. To achieve the best performance, user should build from source to enable more advanced features like Intel MKL, Intel OpenMP, AVX512. Thanks, --Patric [1] https://www.apache.org/legal/resolved.html#category-x > -----Original Message----- > From: Skalicky, Sam <[email protected]> > Sent: Wednesday, December 11, 2019 1:36 PM > To: [email protected] > Cc: Keshavan, Arjuna <[email protected]>; Harish, Nihal > <[email protected]> > Subject: Performance regression from removing libiomp5.so > > Hi MXNet community, > > I would like to bring your attention to the performance regression that was > found [1] between 1.5.1 and 1.6.0 due to removing the libiomp5.so library > due to licensing issues. This change was made since this library has a > category > x license [2] that is not compatible with the MXNet Apache > license/distribution. > > We found that using OpenBLAS instead of MKL BLAS caused a regression > from 1500 samples/sec to 1300 samples/sec a 13.3% regression in training > speed for a resnet18 training benchmark on a C5.18xlarge EC2 instance (with > 72 cores). Rebuilding with MKL BLAS showed an increase in performance to > 1600 samples/sec in the 1.6.0 branch. > > Please provide your feedback on the licensing issue (are there any work- > arounds) and the tradeoff in performance (is the benefit worth trying to > include back into MXNet builds). > > Thanks to the efforts of the following folks for working on this issue (in no > particular order): > Patric Zhao > Amol Lele > Tao Lv A > Pedro Larroy > Nihal Harish > Chai Bapat > Arjuna Keshavan > Rong Zhang > > Thanks! > Sam > > [1] https://github.com/apache/incubator-mxnet/issues/16891 > [2] https://www.apache.org/legal/resolved.html#category-x
