Jasper: Thanks! Unfortunately it looks like our problems are different — mine began with CentOS 7.7, while you seem to be fine with 7.9 and are only seeing issues with CentOS 8 (if I understand you correctly).
The problem I’m seeing is very poor performance — network transfers at 6KB/s (vs. 500K-1MB/s) — and appears to be specific to Samba. What is odd is that the exact same configuration runs fine if I boot with an earlier version of the kernel (-957 or below). It’s nevertheless possible that the two issues are related — if I’m able to come up with any clues, I’ll be sure to post them back here. Thanks again for the help! Bill > On Apr 9, 2022, at 8:03 PM, Jasper Jaspers <[email protected]> wrote: > >> Here’s the info you requested. What issues are you seeing? >> >> >> >> RHEL 7: >> >> uname -a >> >> Linux cas00 3.10.0-1160.11.1.el7.oe.x86_64 #1 SMP Mon Jan 18 19:15:21 >> EST 2021 x86_64 x86_64 x86_64 GNU/Linux >> >> >> >> lsb_release -a >> >> LSB Version: >> :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch >> >> Distributor ID: RedHatEnterpriseServer >> >> Description: Red Hat Enterprise Linux Server release 7.9 (Maipo) >> >> Release: 7.9 >> >> Codename: Maipo >> >> >> >> RHEL8: >> >> uname -a >> >> Linux cas00 4.18.0-348.12.2.el8_5.x86_64 #1 SMP Mon Jan 17 07:06:06 >> EST 2022 x86_64 x86_64 x86_64 GNU/Linux >> >> >> >> lsb_release -a >> >> LSB Version: :core-4.1-amd64:core-4.1-noarch >> >> Distributor ID: RedHatEnterprise >> >> Description: Red Hat Enterprise Linux release 8.5 (Ootpa) >> >> Release: 8.5 >> >> Codename: Ootp >> > > On Saturday, April 9, 2022, Bill Torpey <[email protected] > <mailto:[email protected]>> wrote: > Hi Jasper: > > I’d be VERY interested in knowing what version of RH7 you’re running: > > uname -a > > lsb_release -a > > I’m seeing weird network performance on CentOS 7, as of kernel version 1062 > (part of 7.7) —earlier kernel version 957 (part of 7.6) is fine. > > Curious if that’s related to what you are seeing. I’ve been trying to run > this down for a while, and any clues would be most welcome. > > Thanks! > > Bill > > > On Apr 8, 2022, at 9:56 PM, Jasper Jaspers <[email protected] > > <mailto:[email protected]>> wrote: > > > > I’m noticing a difference in performance between zeromq running on Linux > > RHEL7 and RHEL8. I’m running with zmq version 4.3.4. > > > > I have a sample program to replicate the issues were seeing on our full > > system. The application uses a simple REQ/REP pattern with an INPROC > > socket. The client repeatedly (every 50 milliseconds) sends 3 message > > parts ( 100, 200, and 500000 bytes) and then waits for the reply. The > > server receives the message parts and sends a basic reply message of 1 byte > > back. I’m also changing the thread policy/priority (SCHED_FIFO) to minimize > > latency due to other things running on the node. Im capturing the time it > > takes for each client transaction to complete. > > > > When this is only app running on the node each message transaction takes > > about 1-10 microseconds. > > > > To simulate other lower priority load on the node I’m running a > > compilation which puts a bunch of cpu usage on every core. When this is > > running in the background I’m seeing a difference between RHEL7 and RHEL8. > > > > Here’s results from running test for about 10 minutes: > > > > On RHEL7 I saw 5 transactions that took over 1 milliseconds, with the > > highest being 2.5 milliseconds. > > > > On RHEL8 I saw about 100 transactions that took over 1 milliseconds with > > most around 10 milliseconds and the highest around 20 milliseconds. > > > > After seeing this I looked into where the time was being spent and found > > that for each transaction above 1 milliseconds all the time was spent in > > the client’s zmq_rcvmsg call waiting for the reply. > > > > Given that the application is running with real time priority I wouldn’t > > expect there to be that much additional latency due to other lower priority > > applications running on the node. Do you agree with that assessment? And > > if so do you think the issue is within zmq or within the Linux OS? Given > > that this works much better in RHEL7 I’ll leaning toward the OS. If so, any > > ideas what I could look for, or do you suggest contacting red hat directly > > for support ? > > > > _______________________________________________ > > zeromq-dev mailing list > > [email protected] <mailto:[email protected]> > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > <https://lists.zeromq.org/mailman/listinfo/zeromq-dev> > > _______________________________________________ > zeromq-dev mailing list > [email protected] <mailto:[email protected]> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > <https://lists.zeromq.org/mailman/listinfo/zeromq-dev> > _______________________________________________ > zeromq-dev mailing list > [email protected] > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________ zeromq-dev mailing list [email protected] https://lists.zeromq.org/mailman/listinfo/zeromq-dev
