[tcpdump-workers] reconstruct HTTP requests in custom sniffer
Hi, I am asked to write a custom sniffer with libpcap on Linux that has to handle a load of 50.000 packets per second. The sniffer has to detect all HTTP requests and dump the URI with additional information, such as request size and possibly response time/size. The packets, destined for the load-balancer, are duplicated by the switch using port-mirroring to my own machine. It is important that our solution is 100% non-intrusive to the web application being monitored. Probably I need to access the POST data of certain HTTP requests. Because HTTP requests are, obviously, broken into multiple packets, is it feasible to reconstruct the whole HTTP request with POST data from multiple packets? Regarding the load of 50.000 packets a second, is this expected to be a problem? Any feedback is very appreciated! Cheers, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] pcap_lib_version problem while installing DAQ
On Dec 27, 2010, at 10:26 PM, Appaji_Peruri wrote: > I am having a problem while installing DAQ which is used by snort . DAQ > package is searching for the function pcap_lib_version and returning the > following error . > > checking for pcap_lib_version... checking for pcap_lib_version in -lpcap... no > >ERROR! Libpcap library version >= 1.0.0 not found. >Get it from http://www.tcpdump.org > > My query is whether libpcap 1.1 includes the above function or not . Yes, it does. It has been in libpcap since one of the 0.8.x releases, and has never been removed. > if so how to verify the same. As you say "libpcap" rather than "WinPcap", I'm assuming this is on some flavor of UN*X (Solaris, Linux, Mac OS X, HP-UX, AIX, *BSD, etc.) rather than on Windows. Step 1: find where the libpcap library is installed. Step 2: run "nm -p" on that library and pipe the result through "grep pcap_lib_version". That should show that it does, in fact, define pcap_lib_version(). The problem here is that DAQ's configure script is buggy - it appears to be assuming that if you're building a program with -lpcap, you won't, for example, have to build it with -lnl. That assumption is not necessarily true; if, on Linux, you're linking with a static version of libpcap, for example, or if you're linking with a dynamic version that wasn't itself linked with -lnl, you might also have to link with -lnl, as libpcap uses -lnl. Similarly, it might be assuming that you don't have to build it with -lsocket and -lnsl, but that's not true on Solaris. Are you doing this on some Linux distribution? If so, what distribution is it, and what version of that distribution is this? If not, what operating system are you doing this on, and what version of that operating system is it? Did that version of that operating system/distribution come with libpcap? If so, are you using the version of libpcap that came with the operating system, or did you build and install libpcap yourself? (If the OS didn't come with libpcap, you presumably built and installed it yourself.) If you built and installed libpcap yourself, where is it installed - and did you build and install a shared (dynamic) version, or just a static version? If it came with the OS, does it come with a shared version, a static version, or both?- This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] pcap_lib_version problem while installing DAQ
On Dec 28, 2010, at 11:03 AM, Guy Harris wrote: > Are you doing this on some Linux distribution? If so, what distribution is > it, and what version of that distribution is this? > > If not, what operating system are you doing this on, and what version of that > operating system is it? > > Did that version of that operating system/distribution come with libpcap? If > so, are you using the version of libpcap that came with the operating system, > or did you build and install libpcap yourself? (If the OS didn't come with > libpcap, you presumably built and installed it yourself.) > > If you built and installed libpcap yourself, where is it installed - and did > you build and install a shared (dynamic) version, or just a static version? > > If it came with the OS, does it come with a shared version, a static version, > or both? ...and the configure script probably created a config.log file; what are the contents of that file?- This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] reconstruct HTTP requests in custom sniffer
On 2010-12-28 17:22, Andrej van der Zee wrote: > I am asked to write a custom sniffer with libpcap on Linux that has to > handle a load of 50.000 packets per second. The sniffer has to detect all > HTTP requests and dump the URI with additional information, such as request > size and possibly response time/size. The packets, destined for the > load-balancer, are duplicated by the switch using port-mirroring to my own > machine. It is important that our solution is 100% non-intrusive to the web > application being monitored. > > Probably I need to access the POST data of certain HTTP requests. Because > HTTP requests are, obviously, broken into multiple packets, is it feasible > to reconstruct the whole HTTP request with POST data from multiple packets? > > Regarding the load of 50.000 packets a second, is this expected to be a > problem? > > Any feedback is very appreciated! See urlsnarf: http://monkey.org/~dugsong/dsniff/ I don't think it does POST data but it may be a good starting point. -- Jefferson Ogata National Oceanographic Data Center You can't step into the same river twice. -- Herakleitos - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] reconstruct HTTP requests in custom sniffer
Hi, I have implemented a HTTP parser one year ago. I remembered that when the parser calculate the request-response latency, inspect the interested fields but do not record or dump them, the speed will reach about 2Gbps on a single core, and 8 Gbps on 6 cores. I think a 0.05Mpps parser is an easy work. However, as you said you had to reconstruct the whole HTTP request with POST data, that will be a different story. You need to store the previous packets and do a memcpy() operation to concatenate them when latter packets are received. In my experience, the cost is huge, especially the memcpy operation. It depends on how many packets are such kind of cross-packet POST requests. Usual GET requests do not have this issue. Hope it helps! --Kay On Wed, Dec 29, 2010 at 1:22 AM, Andrej van der Zee < andrejvander...@gmail.com> wrote: > Hi, > > I am asked to write a custom sniffer with libpcap on Linux that has to > handle a load of 50.000 packets per second. The sniffer has to detect all > HTTP requests and dump the URI with additional information, such as request > size and possibly response time/size. The packets, destined for the > load-balancer, are duplicated by the switch using port-mirroring to my own > machine. It is important that our solution is 100% non-intrusive to the web > application being monitored. > > Probably I need to access the POST data of certain HTTP requests. Because > HTTP requests are, obviously, broken into multiple packets, is it feasible > to reconstruct the whole HTTP request with POST data from multiple packets? > > Regarding the load of 50.000 packets a second, is this expected to be a > problem? > > Any feedback is very appreciated! > > Cheers, > Andrej > - > This is the tcpdump-workers list. > Visit https://cod.sandelman.ca/ to unsubscribe. > - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
[tcpdump-workers] Request for new DLT number
I've been looking through all of the DLT decoders looking for one that has just the DLT number in the header but I couldn't find one. Is there an existing DLT that matches this description? Otherwise, I'd like to request DLT_DLT (or something like that) be allocated to represent a 4 byte (network order) integer value that describes the DLT of the following data. In pcap files, it would roughly translate to the following being possible: [pcap file header, dlt = DLT_DLT] [pcap header with time stamp] [4 bytes, = DLT_EN10MB] [ethernet packet] [pcap header with time stamp] [4 bytes, = DLT_PPP] [ppp packet] Yes, I understand that "next gen pcap" can do this, no I don't want to use "next gen pcap" because that amount of change is just too big. Darren - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] Request for new DLT number
This is what PPI does. http://www.cacetech.com/documents/PPI%20Header%20format%201.0.10.pdf There is already a DLT for PPI (DLT_PPI). The only difference from your solution is that the minimum header before the packet is 8 bytes (instead of 4). The advantage is that Wireshark already supports this DLT. Have a nice day GV -Original Message- From: tcpdump-workers-ow...@lists.tcpdump.org [mailto:tcpdump-workers-ow...@lists.tcpdump.org] On Behalf Of Darren Reed Sent: Tuesday, December 28, 2010 7:02 PM To: tcpdump-workers@lists.tcpdump.org Subject: [tcpdump-workers] Request for new DLT number I've been looking through all of the DLT decoders looking for one that has just the DLT number in the header but I couldn't find one. Is there an existing DLT that matches this description? Otherwise, I'd like to request DLT_DLT (or something like that) be allocated to represent a 4 byte (network order) integer value that describes the DLT of the following data. In pcap files, it would roughly translate to the following being possible: [pcap file header, dlt = DLT_DLT] [pcap header with time stamp] [4 bytes, = DLT_EN10MB] [ethernet packet] [pcap header with time stamp] [4 bytes, = DLT_PPP] [ppp packet] Yes, I understand that "next gen pcap" can do this, no I don't want to use "next gen pcap" because that amount of change is just too big. Darren - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe. - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] reconstruct HTTP requests in custom sniffer
Hi, > See urlsnarf: > > http://monkey.org/~dugsong/dsniff/ > > I don't think it does POST data but it may be a good starting point. > > Thanks, this seems to be very useful. It uses libnids which *hopefully* enables me to re-assemble the HTTP request + POST data from raw packets with little effort. Cheers, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] reconstruct HTTP requests in custom sniffer
Hi, I have implemented a HTTP parser one year ago. I remembered that when the > parser calculate the request-response latency, inspect the interested > fields > but do not record or dump them, the speed will reach about 2Gbps on a > single > core, and 8 Gbps on 6 cores. I think a 0.05Mpps parser is an easy work. > Thanks, that sounds promising. > > However, as you said you had to reconstruct the whole HTTP request with > POST > data, that will be a different story. You need to store the previous > packets > and do a memcpy() operation to concatenate them when latter packets are > received. In my experience, the cost is huge, especially the memcpy > operation. It depends on how many packets are such kind of cross-packet > POST > requests. Usual GET requests do not have this issue. > Hopefully libnids can do this for me efficiently... Cheers, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
