[tcpdump-workers] Huge latency increase libpcap-1.4.0 -> libpcap-1.5.3
I have some programs that suffered terrible performance degradation when moving from Ubuntu 12.04 to Ubuntu 14.04. Originally I blamed Ubuntu 14.04 for the performance hit and simply stuck with 12.04, however when I updated libpcap to support hardware timestamps, the performance degraded again. I then reduced the problem to a program that simple returned ICMP ECHO replies when requests were received. Under 12.04, this program works very well (with reported responses from ping of under 500 usec), however, if I point the libpcap.so files to the downloaded version 1.5.3 (originally tested with 1.6.2 and problem seen there as well), the response times jump to sometimes over 100 msec. Similarly, if I run the program on Ubuntu 14.04 I see response times well over 10 msec (and actually miss many requests). If I simply replace the links to libpcap.so.1.5.3 with links to libpcap.so.1.4.0 the responses return to sub 300 usec. Is this behavior expected? Do I need to redo my pcap handle setup if moving from libpcap-1.4.0 to libpcap-1.5.3 (or libpcap-1.6.2)? Any guidance is greatly appreciated. Any troubleshooting I can do to help out? Thanks, Steve ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] Huge latency increase libpcap-1.4.0 -> libpcap-1.5.3
On Fri, 14 Nov 2014, Guy Harris wrote: On Nov 14, 2014, at 1:17 PM, Steve Bourland wrote: I have some programs that suffered terrible performance degradation when moving from Ubuntu 12.04 to Ubuntu 14.04. Are those programs capturing and processing network traffic, are they sending packets and expecting to receive a response, or are they doing something else? They are essentially servers, listening for specific packets and then building and sending the appropriate reply to the client. Is the performance issue one of high latency or of dropped packets when capturing? Sorry, the main concern I have is high latency, I only saw the dropped packets when I started changing the links in /usr/lib/x86_64-linux-gnu/ so I am fairly confident I messed something up there. I then reduced the problem to a program that simple returned ICMP ECHO replies when requests were received. So that's a program sending packets and expecting to receive a response... It simply replaces the stack and replies to ICMP ECHO REQUESTs using libpcap. I did that so I could just use ping as the stimulous and have automatic statistics. Under 12.04, this program works very well (with reported responses from ping of under 500 usec), however, if I point the libpcap.so files to the downloaded version 1.5.3 (originally tested with 1.6.2 and problem seen there as well), the response times jump to sometimes over 100 msec. ...and the issue you're talking about there is latency. Yes, sorry, the times reported are reported from the other machine's (stock) ping command. What timeout are you specifying with pcap_open_live() or pcap_set_timeout()? In the test program I am using 1 msec as the timeout for pcap_open_live...but then have a selectable fd and am using a select call to let me know when a packet arrives. Note that: For programs passively capturing and processing network traffic, you probably want a long timeout (tcpdump uses 1 second, Wireshark uses 100 ms or so), so that as many packets are delivered per wakeup as possible. For programs sending packets and expecting a response for each packet, you probably want a *low* timeout (note that 0 is *not* a valid timeout - it may mean "no timeout" in the sense that you won't see packets until the capture mechanism's buffer fills up, which could take an indefinitely long period of time) or, with newer versions of libpcap, you might want to use "immediate mode", instead, in which no timeout-based packet buffering is done - packets are delivered immediately. pcap_set_immediate_mode() first appeared in libpcap 1.5.0. This is exactly the information I was hoping you would be able to provide. I will try this out and report back with my findings as soon as possible, but I expect you have identified where my problem lies. Thank you very much, Steve ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] Huge latency increase libpcap-1.4.0 -> libpcap-1.5.3
On Fri, 14 Nov 2014, Guy Harris wrote: On Nov 14, 2014, at 2:08 PM, Steve Bourland wrote: On Fri, 14 Nov 2014, Guy Harris wrote: On Nov 14, 2014, at 1:17 PM, Steve Bourland wrote: I have some programs that suffered terrible performance degradation when moving from Ubuntu 12.04 to Ubuntu 14.04. Are those programs capturing and processing network traffic, are they sending packets and expecting to receive a response, or are they doing something else? They are essentially servers, listening for specific packets and then building and sending the appropriate reply to the client. OK, that's the same case, from the point of view of using libpcap, as the client case of sending packets and expecting to receive a response. Is the performance issue one of high latency or of dropped packets when capturing? Sorry, the main concern I have is high latency, As I suspected. What timeout are you specifying with pcap_open_live() or pcap_set_timeout()? In the test program I am using 1 msec as the timeout for pcap_open_live...but then have a selectable fd and am using a select call to let me know when a packet arrives. Well, 1 ms is the smallest that the timeout can get. For PF_PACKET sockets, which are what libpcap uses on Linux, the TPACKET_V3 mechanism, which is the default in 1.5.0 and later and which works better for the packet-capture case, doesn't work well for the request-and-response application case. I would suggest that: if the version of libpcap with which you're building has the pcap_set_immediate_mode() API, use pcap_create(), pcap_set_immediate_mode(), and pcap_activate() rather than pcap_open_live(), as libpcap falls back to TPACKET_V2 in immediate mode; otherwise, libpcap also doesn't support TPACKET_V3, so that's not an issue - keep using pcap_open_live() (unless you're already using pcap_create()/pcap_activate(), which first appeared in 1.0.0). This may require a configure script, if you're OK with building from source different versions of the program for different machines, or some dlopen()/dlsym() trickery if you want to make a binary that'll work with both libpcap 1.5-and-later and earlier versions. Initial testing on an Ubuntu 14.04 system showed that the pcap_create, pcap_set_immediate_mode, pcap_activate fixed the problems beautifully for me. The simplified icmp "server" went from horrible latencies and losses to a mean round trip time of 298 usec (reported by ping on a 14.04 server system) and zero losses. Thanks so much for you quick help, and in terms of the full program, I am very fortunate that it will be compiled and deployed on a per system basis so I don't need to worry about a generic binary and can sort out a configure script or Makefile trickery. Thanks for answering within a few hours a question that has been in the back of my mind for months. ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] tcpdump-workers Digest, Vol 72, Issue 3
If you have the server's certificate, wireshark has the capability to decrypt SSL traffic captured with tcpdump, but you must have the certificate and the start of the tcp session. On Sun, 8 Jul 2018, tcpdump-workers-requ...@lists.tcpdump.org wrote: Send tcpdump-workers mailing list submissions to tcpdump-workers@lists.tcpdump.org To subscribe or unsubscribe via the World Wide Web, visit https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers or, via email, send a message with subject or body 'help' to tcpdump-workers-requ...@lists.tcpdump.org You can reach the person managing the list at tcpdump-workers-ow...@lists.tcpdump.org When replying, please edit your Subject line so it is more specific than "Re: Contents of tcpdump-workers digest..." Today's Topics: 1. Re: Packet capture of SSL traffic (Kaushal Shriyan) -- Message: 1 Date: Sun, 8 Jul 2018 10:53:40 +0530 From: Kaushal Shriyan To: g...@alum.mit.edu Cc: tcpdump-workers@lists.tcpdump.org Subject: Re: [tcpdump-workers] Packet capture of SSL traffic Message-ID: Content-Type: text/plain; charset="UTF-8" Thanks! Guy Harris for the explanation. Are there any tools which can decrypt SSL traffic once i do the packet capture of SSL traffic using tcpdump? I look forward to hearing from you. Best Regards, Kaushal On Sat, Jul 7, 2018 at 6:23 AM Guy Harris wrote: On Jul 5, 2018, at 11:18 AM, Kaushal Shriyan wrote: > Is there a way to run tcpdump to do packet capture on SSL traffic? Yes. Plug the machine running tcpdump into a network on which SSL traffic is being sent, in a fashion that allows it to see that traffic (bearing in mind, for example, that capturing third-party traffic on a switched network may be difficult or impossible), and run tcpdump, with the -w flag, so that it saves the traffic to a file, and either with no filter or with a filter that matches the SSL traffic. If you mean "is there a way to run tcpdump so that it can *dissect* SSL traffic", rather than just being able to put undissected raw packet contents, including SSL packets, into a file to be read by another program, the answer is "no" - tcpdump doesn't currently include the ability to decrypt SSL traffic. (I.e., there's more to being able to analyze traffic than just being able to capture it) -- Subject: Digest Footer ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers -- End of tcpdump-workers Digest, Vol 72, Issue 3 ** ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
[tcpdump-workers] pcap_inject change?
I am expecting this is a change outside of libpcap given the lack of a note in the change log, but we are moving from Ubuntu 16.04 (well, really 12.04) to 18.04 and have starting having packets flagged for Frame Check Sequence errors when captured and analyzed in Wireshark. I have a simple ARP sending program for testing and have found that under Ubuntu 18.04 (libpcap version 1.8.1) an additional 14 bytes appear to be injected, with non-zero data that is triggering the errors in the dissector (ie, if I call pcap_inject with size argument 50, 64 bytes are captured, if called with size argument 60, 74 are captured). On matching hardware under Ubuntu 16.04 (libpcap 1.7.4), pcap_inject with size 50 results in 60 bytes on the wire (expected minimum packet size) and the padding is all zeros. Both machines are using the same Intel e1000e driver versions (3.2.6-k). Has anyone else seen this or have a workaround? (pcap handle is being created with pcap_open_live vs. pcap_create as this is an older program.) Thanks, Steve ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] pcap_inject change?
On Tue, 11 Sep 2018, Michael Richardson wrote: Steve Bourland wrote: > are captured, if called with size argument 60, 74 are captured). On > matching hardware under Ubuntu 16.04 (libpcap 1.7.4), pcap_inject with > size 50 results in 60 bytes on the wire (expected minimum packet size) > and the padding is all zeros. Both machines are using the same Intel > e1000e driver versions (3.2.6-k). Has anyone else seen this or have a > workaround? It sounds like different kernel packet capture mechanisms are being used. The pcap_lib_version string will tell you what's compiled in, but not which one is used. (TPACKET3 vs 2, etc.) There was no way in earlier versions of libpcap to even know which was used. I thought we had a way in 1.9, but I can't find it... so I think that I'm wrong, I'm just remembering that we discussed that we should? The pcap_lib_version string will tell you what's compiled in, but not which one is used. -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works| network architect [ ] m...@sandelman.ca http://www.sandelman.ca/| ruby on rails[ I'm a little confused, why would the capture mechanism matter for the pcap_inject call? I am capturing both senders packets on the same machine (a single tcpdump call). I was thinking my next move would be to grab the 18.04 kernel and install it on the 16.04 machine to see if that triggers the behavior on the 16.04 machine. ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] pcap_inject change?
On Tue, 11 Sep 2018, Michael Richardson wrote: Steve Bourland wrote: > are captured, if called with size argument 60, 74 are captured). On > matching hardware under Ubuntu 16.04 (libpcap 1.7.4), pcap_inject with > size 50 results in 60 bytes on the wire (expected minimum packet size) > and the padding is all zeros. Both machines are using the same Intel > e1000e driver versions (3.2.6-k). Has anyone else seen this or have a > workaround? It sounds like different kernel packet capture mechanisms are being used. The pcap_lib_version string will tell you what's compiled in, but not which one is used. (TPACKET3 vs 2, etc.) There was no way in earlier versions of libpcap to even know which was used. I thought we had a way in 1.9, but I can't find it... so I think that I'm wrong, I'm just remembering that we discussed that we should? The pcap_lib_version string will tell you what's compiled in, but not which one is used. -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works| network architect [ ] m...@sandelman.ca http://www.sandelman.ca/| ruby on rails[ OK, as I expected (feared), when I brought the 16.04 machine from kernel 4.4.0-109 to 4.15.0-34 it started injecting the "extra" 14 bytes, so it looks like it is a change in the kernel handling of the injection. Does anyone have any pointers on how to handle this? Thanks, Steve ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] pcap_inject change?
On Tue, 11 Sep 2018, Michael Richardson wrote: Steve Bourland wrote: > I'm a little confused, why would the capture mechanism matter for the > pcap_inject call? I am capturing both senders packets on the same > machine (a single tcpdump call). I was thinking my next move would be > to grab the 18.04 kernel and install it on the 16.04 machine to see if > that triggers the behavior on the 16.04 machine. I'm sorry, I mis-understood. I read pcap_inject, then immediately forgot that point... then understood that you were saying that there was garbage at the end of the captured packets! So I have no idea. What kernel versions are involved? What does strace on each show for the system call that sends the packet? -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works| network architect [ ] m...@sandelman.ca http://www.sandelman.ca/| ruby on rails[ No problem, I have found the 16.04 (using 4.4.0-) was "good" but once I installed kernel 4.15.0-34 it started misbehaving. Now realized I have a third machine (does have different Intel NIC, but using the same driver, so a bit of a difference there) that is "well behaved" running 18.04...with kernel 4.15.0-32, so updating it to 4.15.0-34 to see if that "ruins" its behavior Yes, things broke moving from 4.15.0-32 to 4.15.0-34, so it looks like the change came with the move from -32 to -33 (the original machines showing the problem have the -33 kernel installed). These kernels are what come with Ubuntu 18.04 from Canonical. Should I file a bug with them or do you have "better pull" and enough information to follow up on this? I am more than happy to continue to follow this, but I am going to be an unknown to folks vs. you. Thanks so much for your help, Steve ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
