[tcpdump-workers] reading .cap files
Hi, Sorry if it has been asked before. I need to read .cap files produced by tcpdump from c/c++. More specific, I need to read the timestamp, the protocol, the number of bytes of the package (including the data) and the destination IP of each package in .cap. My questions are: * I have no control over the version of tcpdump that is being used for the generation of .cap files. Do I have to write different code for many versions of tcpdump? * What library can I use? Thank you, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
[tcpdump-workers] two general questions tcpdump
Hi, I could not find any users-list for tcpdump, so I am sorry if I offend anybody. I have two questions about tcpdump: 1) I get many UDP packages that have an IP that is not bound to one of my interfaces, like this one (the local IP is 10.69.26.61.22): 1240473922.435472 00:1a:64:79:b9:4c > ff:ff:ff:ff:ff:ff, IPv4, length 550: 10.69.26.35.32768 > 10.69.26.255.9900: UDP, length 504 Now it turns out that 10.69.26.35 is the IP of my gateway. I was wondering why they are logged. 2) Is it possible to get the options of tcpdump that were given on the command from a .cap file? The problem is that I get .cap files from another department, and I want to check the interface(s) and IP(s) (and maybe other usefull info). Thank you, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
[tcpdump-workers] local timestamp recovery of .cap files
Hi, I am having a problem with the timestamps in .cap files. I receive .cap files captured on machines in a different timezone (GMT +1 or GMT +3). When I do a "tcpdump -r en0.cap -n -" then the timestamps are corrected to my local timezone (GMT +8 or GMT +9). The problem is that I need the timezone correction for the capturing machines (GMT +1 or GMT +3) and not of my local machine. I know I can do the timezone correction manually by using -tt instead of - and add +1 or +3 hours, but this is error-prone. Is there a way to calculate this kind of recovery of the timestamp automatically? Thank you, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] local timestamp recovery of .cap files
Thanks a lot for your email. I wish .cap files stored some meta-information such as local timezone, IP address, etc. Well, that's just my bad luck. Cheers, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] local timestamp recovery of .cap files
Hi, > 2) does, but "helpfully" converts the time to local time (in which > case, whoever decided to be "helpful" needs to be hit with said sock). I found that tcpdump with - converts to local time, but tcpdump -tt report GMT. > > However, even with standard pcap files, which have GMT time stamps, one > might want to be able to display the time stamps in the time zone in which > the capture was done rather than in the time zone in which it's being read; > that's what the original poster wanted. Exactly. >Storing time zone information in > the file, rather than getting it out of band (e.g., asking whoever sent you > the file where they captured it) isn't a requirement, but it could be a > convenience. Would be convenient indeed. Cheers, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
[tcpdump-workers] deduct local IP address from cap-file
Hi, Is it by any means possible to deduct the local IP address from a cap-file? With local I mean the IP address that is physically bound to the machine where tcpdump is ran. Thank you, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
[tcpdump-workers] estimate #packets in pcap file
Hi, I am looking for way to estimate the number of packages in a pcap file without traversing throu all packages with pcap_loop(). It does noet have to be precise, just an estimate. Is there a way? Thank you, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
[tcpdump-workers] deduct local IPs from pcap-files, possible?
Hi, Sorry for asking again, but I got no useful answer last time. Hopefully more luck this time... I receive many pcap-files from our clients. Now I am constructing an algorithm using libpcap that deducts time differences between the servers by matching packets on both ends of the connection and comparing timestamps (neglecting latencies). Every server produces one pcap-file that listens to all interfaces of the local machine. I found a way to calculate the time differences between the IPs, but I cannot tell if a particular server is ahead or behind in time. To be able to do this, I need to deduct the local IPs that belong to the server that produced the pcap-file. The "problem" is that on a particular server all incoming and outgoing packets are sniffed, hence the local IPs will appear as "src" and "dst" in the IP-packets. I am looking for a way to deduct the local IPs anyway, but need a push in the right direction (if it is possible at all). By the way, I am using libpcap and loop though all TCP and UDP packets. Thank you, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
[tcpdump-workers] tcp sequence and ack number with libpcap
Hi,
I am trying to get the TCP sequence and ack number of TCP packets. Somehow I
get different values than "tcpdump -vv" does. The numbers are way too big
all the time. Source and destination ports are just fine. Below the relevant
code. I studied the tcpdump source code but can't find why. Please help, I
am stuck!
Thank you,
Andrej
#define ETHER_HDRLEN14
typedef u_int32_t tcp_seq;
struct tcphdr {
u_int16_t th_sport; /* source port */
u_int16_t th_dport; /* destination port */
tcp_seq th_seq; /* sequence number */
tcp_seq th_ack; /* acknowledgement number */
u_int8_tth_offx2; /* data offset, rsvd */
u_int8_tth_flags;
u_int16_t th_win; /* window */
u_int16_t th_sum; /* checksum */
u_int16_t th_urp; /* urgent pointer */
};
static void handle_packet(unsigned char * ifile, const struct pcap_pkthdr *
h, const u_char * sp)
{
const struct ip * ip = (struct ip *) (sp + ETHER_HDRLEN);
const struct tcphdr * tcp_hdr = (const struct tcphdr *)(sp + ETHER_HDRLEN
+ sizeof(struct iphdr));
tcp_seq seq = htonl(tcp_hdr->th_seq);
tcp_seq ack = htonl(tcp_hdr->th_ack);
fprintf(stdout,"seq %u ack %u", seq, ack);
}
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] tcp sequence and ack number with libpcap
Hi,
Thanks for your replies.
Is there a simple way to get the relative numbers as well? My problem
is that i am trying to match tcp packets on both ends of the
conversation, ie from two pcap files captuered on two different hosts.
I use the seq and ack numbers, among other fields, to do the matching.
But somehow for two particulare tcpdump files the seq and ack never
match in my program. Though, when i match packets manually with
"tcpdump -vv" i can. So now i am hoping that a relative seq and ack
number would solve my misunderstanding. Or is this silly?
Thank you,
Andrej
On 2010/08/19, at 23:48, Mark Bednarczyk wrote:
Its probably because tcpdump reports seq/ack numbers as relative
from the start of that particular tcp stream and what you are
getting from the structure are raw/absolute numbers.
The other typical problem with reading the structure field is to
read it using little-endian byte encoding instead of big-endian
(network byte order), but I see you use htonl in your code which is
correct.
Try -S option with tcpdump to get "absolute" numbers.
Cheers,
mark...
--- On Thu, 8/19/10, Andrej van der Zee
wrote:
From: Andrej van der Zee
Subject: [tcpdump-workers] tcp sequence and ack number with libpcap
To: tcpdump-workers@lists.tcpdump.org
Date: Thursday, August 19, 2010, 9:23 AM
Hi,
I am trying to get the TCP sequence and ack number of TCP
packets. Somehow I
get different values than "tcpdump -vv" does. The numbers
are way too big
all the time. Source and destination ports are just fine.
Below the relevant
code. I studied the tcpdump source code but can't find why.
Please help, I
am stuck!
Thank you,
Andrej
#define ETHER_HDRLEN14
typedef u_int32_t tcp_seq;
struct tcphdr {
u_int16_t
th_sport;
/* source port */
u_int16_t
th_dport;
/* destination port */
tcp_seq
th_seq;
/* sequence number */
tcp_seq
th_ack;
/* acknowledgement number
*/
u_int8_tth_offx2;
/* data
offset, rsvd */
u_int8_tth_flags;
u_int16_t
th_win;
/* window */
u_int16_t
th_sum;
/* checksum */
u_int16_t
th_urp;
/* urgent pointer */
};
static void handle_packet(unsigned char * ifile, const
struct pcap_pkthdr *
h, const u_char * sp)
{
const struct ip * ip = (struct ip *) (sp +
ETHER_HDRLEN);
const struct tcphdr * tcp_hdr = (const struct tcphdr
*)(sp + ETHER_HDRLEN
+ sizeof(struct iphdr));
tcp_seq seq = htonl(tcp_hdr->th_seq);
tcp_seq ack = htonl(tcp_hdr->th_ack);
fprintf(stdout,"seq %u ack %u", seq, ack);
}
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] tcp sequence and ack number with libpcap
Hi, > I think you are performing your byte ordering conversion wrong. Seq and > > Ack values are transmitted in network byte order so you need to perform > > a "network to host long" conversion, and for that, you need to user > > ntohl(), not htonl(). > > On standard platforms with byte order 1234 or 4321, there is no difference. > > Yes I noticed that but it does not make a difference indeed. Thanks, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] tcp sequence and ack number with libpcap
Hi,
>> static void handle_packet(unsigned char * ifile, const struct pcap_pkthdr
>> *
>> h, const u_char * sp)
>> {
>> const struct ip * ip = (struct ip *) (sp + ETHER_HDRLEN);
>> const struct tcphdr * tcp_hdr = (const struct tcphdr *)(sp + ETHER_HDRLEN
>> + sizeof(struct iphdr));
>>
>
> You are assuming that the IP header length is constant (sizeof(struct
> iphdr)), but it's not. You need to get the IP header length from the the IP
> header itself.
>
>
Source port and dest number seem to be ok, so I guess this is not the
problem. Nevertheless, I tried the code below but it does not make a
difference. Why do I get those weird seq and ack numbers? I am really
stuck...
Thanks,
Andrej
struct ip {
u_int8_tip_vhl; /* header length, version */
#define IP_V(ip)(((ip)->ip_vhl & 0xf0) >> 4)
#define IP_HL(ip) ((ip)->ip_vhl & 0x0f)
u_int8_tip_tos; /* type of service */
u_int16_t ip_len; /* total length */
u_int16_t ip_id; /* identification */
u_int16_t ip_off; /* fragment offset field */
#define IP_DF 0x4000/* dont fragment flag */
#define IP_MF 0x2000/* more fragments flag */
#define IP_OFFMASK 0x1fff /* mask for fragmenting bits */
u_int8_tip_ttl; /* time to live */
u_int8_tip_p; /* protocol */
u_int16_t ip_sum; /* checksum */
struct in_addr ip_src,ip_dst; /* source and dest address */
};
static void handle_packet(unsigned char * ifile, const struct pcap_pkthdr *
h, const u_char * sp)
{
const struct ip * ip = (struct ip *) (sp + ETHER_HDRLEN);
const struct tcphdr * tcp_hdr = (const struct tcphdr *)(sp + ETHER_HDRLEN
+ IP_HL(ip));
tcp_seq seq = ntohl(tcp_hdr->th_seq);
tcp_seq ack = ntohl(tcp_hdr->th_ack);
fprintf(stdout,"seq %u ack %u", seq, ack);
}
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] tcp sequence and ack number with libpcap
Hi, If I get "bad cksum 0 (->4327)!" for packets with "tcpdump -vv" like below, do I have to take special measures to read fields like seq and ack from the TCP header? 17:55:47.657974 IP (tos 0x0, ttl 128, id 13151, offset 0, flags [DF], proto TCP (6), length 1420, bad cksum 0 (->4327)!) 172.16.123.4.80 > 193.34.150.174.38357: Flags [.], seq 1:1369, ack 436, win 65100, options [nop,nop,TS val 53409804 ecr 3029480193], length 1368 Thank you, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] tcp sequence and ack number with libpcap
Hi, Source port and dest number seem to be ok, so I guess this is not the >> problem. Nevertheless, I tried the code below but it does not make a >> difference. Why do I get those weird seq and ack numbers? I am really >> stuck... >> > > Can you provide some examples of those "weird seq and ack numbers"? Thanks for your reply. With weird I meant different than obtained with "tcpdump -vv". There numbers are much too high: seq 101688001 ack 580300460 seq 103252140 ack 276497601 seq 101689793 ack 580300460 seq 101592513 ack 580300460 seq 102902956 ack 276497601 seq 102902700 ack 276497601 seq 101689281 ack 580300460 seq 101689025 ack 580300460 seq 102902444 ack 276497601 seq 101688769 ack 580300460 With "tcpdump -r -n -vv tcp" I get: 17:53:35.347343 IP (tos 0x10, ttl 64, id 40919, offset 0, flags [DF], proto TCP (6), length 92) 193.34.150.174.22 > 83.247.48.159.52238: Flags [P.], seq 949215706:949215758, ack 3908965070, win 80, length 52 17:53:35.347348 IP (tos 0x10, ttl 64, id 40920, offset 0, flags [DF], proto TCP (6), length 156) 193.34.150.174.22 > 83.247.48.159.52238: Flags [P.], seq 52:168, ack 1, win 80, length 116 17:53:35.367017 IP (tos 0x0, ttl 122, id 8778, offset 0, flags [DF], proto TCP (6), length 40) 83.247.48.159.52238 > 193.34.150.174.22: Flags [.], cksum 0xb0f5 (correct), seq 1, ack 52, win 16356, length 0 17:53:35.568407 IP (tos 0x0, ttl 122, id 8779, offset 0, flags [DF], proto TCP (6), length 40) 83.247.48.159.52238 > 193.34.150.174.22: Flags [.], cksum 0xb09e (correct), seq 1, ack 168, win 16327, length 0 17:53:35.572654 IP (tos 0x0, ttl 122, id 8780, offset 0, flags [DF], proto TCP (6), length 76) 83.247.48.159.49808 > 193.34.150.174.22: Flags [P.], cksum 0x035d (correct), seq 3237258086:3237258122, ack 1238688284, win 16347, length 36 17:53:35.572666 IP (tos 0x10, ttl 64, id 29749, offset 0, flags [DF], proto TCP (6), length 40) 193.34.150.174.22 > 83.247.48.159.49808: Flags [.], cksum 0x7fed (correct), seq 1, ack 36, win 105, length 0 17:53:48.459350 IP (tos 0x0, ttl 122, id 8813, offset 0, flags [DF], proto TCP (6), length 76) 83.247.48.159.52238 > 193.34.150.174.22: Flags [P.], cksum 0x795e (correct), seq 1:37, ack 168, win 16327, length 36 17:53:48.527273 IP (tos 0x10, ttl 64, id 40921, offset 0, flags [DF], proto TCP (6), length 40) 193.34.150.174.22 > 83.247.48.159.52238: Flags [.], cksum 0xeff1 (correct), seq 168, ack 37, win 80, length 0 17:53:50.581688 IP (tos 0x0, ttl 122, id 8816, offset 0, flags [DF], proto TCP (6), length 76) 83.247.48.159.49808 > 193.34.150.174.22: Flags [P.], cksum 0x7fa1 (correct), seq 36:72, ack 1, win 16347, length 36 17:53:50.581701 IP (tos 0x10, ttl 64, id 29750, offset 0, flags [DF], proto TCP (6), length 40) 193.34.150.174.22 > 83.247.48.159.49808: Flags [.], cksum 0x7fc9 (correct), seq 1, ack 72, win 105, length 0 - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] tcp sequence and ack number with libpcap
Hi, Hi Andrej, > > Several others have already mentioned it -- tcpdump is using relative > sequence numbers to make it easier to read the output. Large sequence > numbers are perfectly valid (after all, they are 32-bit unsigned numbers). > > Use the -S argument to tcpdump and you'll see tcpdump report large sequence > numbers as well, just as your application does. > > The -S options does not give me the same results either. I did another run with -S and printed the timestamps and length of the packets to absolutely make sure that we are comparing the same thing. Still big differences. This is killing me. 17:53:35.347343 seq 113135041 ack 580300371 len 92 17:53:35.347348 seq 113118401 ack 580300371 len 156 17:53:35.367017 seq 100802387 ack 4147158977 len 40 17:53:35.568407 seq 100802131 ack 4147158977 len 40 17:53:35.572654 seq 100792659 ack 4147158977 len 76 17:53:35.572666 seq 116007873 ack 580300371 len 40 17:53:48.459350 seq 100784211 ack 4147158977 len 76 17:53:48.527273 seq 113147841 ack 580300371 len 40 17:53:50.581688 seq 100783443 ack 4147158977 len 76 and...@ubuntu:~/caps$ tcpdump -r client_1_20100818115534.cap -S -n -vv tcp | head -n 20 reading from file client_1_20100818115534.cap, link-type EN10MB (Ethernet) 17:53:35.347343 IP (tos 0x10, ttl 64, id 40919, offset 0, flags [DF], proto TCP (6), length 92) 193.34.150.174.22 > 83.247.48.159.52238: Flags [P.], seq 949215706:949215758, ack 3908965070, win 80, length 52 17:53:35.347348 IP (tos 0x10, ttl 64, id 40920, offset 0, flags [DF], proto TCP (6), length 156) 193.34.150.174.22 > 83.247.48.159.52238: Flags [P.], seq 949215758:949215874, ack 3908965070, win 80, length 116 17:53:35.367017 IP (tos 0x0, ttl 122, id 8778, offset 0, flags [DF], proto TCP (6), length 40) 83.247.48.159.52238 > 193.34.150.174.22: Flags [.], cksum 0xb0f5 (correct), seq 3908965070, ack 949215758, win 16356, length 0 17:53:35.568407 IP (tos 0x0, ttl 122, id 8779, offset 0, flags [DF], proto TCP (6), length 40) 83.247.48.159.52238 > 193.34.150.174.22: Flags [.], cksum 0xb09e (correct), seq 3908965070, ack 949215874, win 16327, length 0 17:53:35.572654 IP (tos 0x0, ttl 122, id 8780, offset 0, flags [DF], proto TCP (6), length 76) 83.247.48.159.49808 > 193.34.150.174.22: Flags [P.], cksum 0x035d (correct), seq 3237258086:3237258122, ack 1238688284, win 16347, length 36 17:53:35.572666 IP (tos 0x10, ttl 64, id 29749, offset 0, flags [DF], proto TCP (6), length 40) 193.34.150.174.22 > 83.247.48.159.49808: Flags [.], cksum 0x7fed (correct), seq 1238688284, ack 3237258122, win 105, length 0 17:53:48.459350 IP (tos 0x0, ttl 122, id 8813, offset 0, flags [DF], proto TCP (6), length 76) 83.247.48.159.52238 > 193.34.150.174.22: Flags [P.], cksum 0x795e (correct), seq 3908965070:3908965106, ack 949215874, win 16327, length 36 17:53:48.527273 IP (tos 0x10, ttl 64, id 40921, offset 0, flags [DF], proto TCP (6), length 40) 193.34.150.174.22 > 83.247.48.159.52238: Flags [.], cksum 0xeff1 (correct), seq 949215874, ack 3908965106, win 80, length 0 17:53:50.581688 IP (tos 0x0, ttl 122, id 8816, offset 0, flags [DF], proto TCP (6), length 76) 83.247.48.159.49808 > 193.34.150.174.22: Flags [P.], cksum 0x7fa1 (correct), seq 3237258122:3237258158, ack 1238688284, win 16347, length 36 17:53:50.581701 IP (tos 0x10, ttl 64, id 29750, offset 0, flags [DF], proto TCP (6), length 40) 193.34.150.174.22 > 83.247.48.159.49808: Flags [.], cksum 0x7fc9 (correct), seq 1238688284, ack 3237258158, win 105, length 0 - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] tcp sequence and ack number with libpcap
Hi, const struct tcphdr * tcp_hdr = (const struct tcphdr *)(sp + ETHER_HDRLEN > + IP_HL(ip)); > > This is surely wrong. > The size of the IP header is IP_HL(ip)*4 not IP_HL(ip) > > Thank you very much! Now I do get the same seq and ack number for my app and tcpdump -vv. Thanks, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] tcp sequence and ack number with libpcap
Hi, > const struct tcphdr * tcp_hdr = (const struct tcphdr *)(sp + ETHER_HDRLEN >> + IP_HL(ip)); >> >> This is surely wrong. >> The size of the IP header is IP_HL(ip)*4 not IP_HL(ip) >> >> > Thank you very much! Now I do get the same seq and ack number for my app > and tcpdump -vv. > > Though, unfortunately I turn out to need the relative sequence and ack number. How do I get those? Thank you, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] tcp sequence and ack number with libpcap
Hi Ronnie, Note: > The relative numbers are not part of the TCP protocol. > The relative numbers does not actually exist. Only the absolute > numbers exist in the packets. > > Relative numbers are an invention inside tcpdump to make the printed > numbers easier to read for humans. > > > Thank you, I got it working now. Your explanation and the tcpdump source-code made it work... Cheers, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
[tcpdump-workers] sniffing HTTP traffic to load-balancer on a dedicated machine
Hi, I am looking for a solution that sniffs all HTTP traffic to the load-balancer in a multi-tier web application, but WITHOUT starting tcpdump on the load-balancer itself. The sniffing has to be isolated on a dedicated machine, without having to change anything to the web application itself. If the dedicated sniff-machine shuts down, the web application CANNOT be affected. The dedicated sniff-machine should run Linux, preferably Ubuntu. I am looking for a cheap solution in terms of hardware. But most important is that the solution is non-intrusive to the web application to be sniffed. I am sure I am not the first, so any help appreciated! Thank you, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] sniffing HTTP traffic to load-balancer on a
Hi, Thanks for your emails. I am not sure the type of switch but I am going to try to find out. Will take some time though. The idea is to sniff all incoming/outgoing traffic on the WAN side of the load-balancer, I mean all external traffic of users that visit the web site hosted through the load-balancer. Does this change anything regarding the use of "port mirroring"? Cheers, Andrej On Tue, Nov 2, 2010 at 3:44 PM, Guy Harris wrote: > > On Nov 1, 2010, at 8:57 PM, Andrej van der Zee wrote: > >> Hi, >> >> I am looking for a solution that sniffs all HTTP traffic to the >> load-balancer in a multi-tier web application, but WITHOUT starting >> tcpdump on the load-balancer itself. > > Does the load balancer support some form of "mirror port"? > > If so, you might be able to capture on that (although that port would have to > support the *total* aggregate traffic through the load balancer...). > > If not, is there only one network feeding into the load balancer? > > If so, can you capture on that network with, for example, some sort of > tap? > > If not, you presumably can't do this with a single tap, as you'd > either have to tap on all the networks going *into* the load balancer or *out > of* the load balancer. You might try running multiple captures and merging > them.- > This is the tcpdump-workers list. > Visit https://cod.sandelman.ca/ to unsubscribe. > -- Andrej van der Zee Koenji-minami 2-40-19A Suginami-ku, Tokyo 166-0003 JAPAN Mobile: +81-(0)80-65251092 Phone/Fax: +81-(0)3-3318-3155 - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] sniffing HTTP traffic to load-balancer on a
Hi,
Thanks for pushing me in the right direction.
I will have to find out the network-layout of our client somehow, but
most likely it is not directly connected to the WAN. Thanks again, I
will get back as soon as I have more info.
Cheers,
Andrej
On Tue, Nov 2, 2010 at 4:15 PM, Guy Harris wrote:
>
> On Nov 2, 2010, at 12:05 AM, Andrej van der Zee wrote:
>
>> The idea is to sniff all incoming/outgoing traffic on the WAN side of
>> the load-balancer,
>
> Is the "WAN side" implemented as:
>
> some form of WAN (a T{n} or E{n} serial line, or an OC{n} or STM{n}
> optical link) going directly into the load balancer;
>
> an Ethernet coming out of some flavor of WAN-to-Ethernet
> router/switch/whatever;
>
> multiple Ethernets coming out of such a device;
>
> something else?
>
>> I mean all external traffic of users that visit the
>> web site hosted through the load-balancer. Does this change anything
>> regarding the use of "port mirroring"?
>
> If the WAN side is a WAN going directly into the load balancer, and you want
> to capture traffic on the WAN side, that wouldn't be done with "port
> mirroring" unless the load balancer can funnel copies of all WAN-side traffic
> into an Ethernet port; in that case, you'd need to somehow capture on the WAN
> side, e.g. with a DAG card from Endace:
>
> http://www.endace.com/endace-dag-high-speed-packet-capture-cards.html
>
> If it's an Ethernet or multiple Ethernets coming some routing device, it
> might be possible to have *that* device mirror the WAN traffic to a port -
> and if it's only one Ethernet, you could try just tapping that Ethernet.-
> This is the tcpdump-workers list.
> Visit https://cod.sandelman.ca/ to unsubscribe.
>
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.
[tcpdump-workers] tcpdump and timestamps
Hi, Today I received a tcpdump file from a client with timestamps that did not correspond to the system clock. If I remember correctly, tcpdump does not store complete timestamps but only a delta compared to the first timestamp. I guess tcpdump does not read the system clock every time, but has its own mechanisms. My question is, how does tcpdump calculate its timestamps? Thank you, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] tcpdump and timestamps
Hi Gary, Thank you for your clear reply, as always. > > WinDump, the Windows port of tcpdump, uses WinPcap, the Windows port of > libpcap. The time stamps come from the WinPcap driver, which might, > depending on how it's configured, read the system clock for each packet, or > might read it when it starts and, for each packet, add a value from the > performance counter to it. In the latter case, the time stamps might drift > from the system clock value. > I just read that putting the registry value for the key below on 2 lets the driver use system clock. HKLM\System\CurrentControlSet\Services\NPF\TimestampMode Thank you, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
[tcpdump-workers] reconstruct HTTP requests in custom sniffer
Hi, I am asked to write a custom sniffer with libpcap on Linux that has to handle a load of 50.000 packets per second. The sniffer has to detect all HTTP requests and dump the URI with additional information, such as request size and possibly response time/size. The packets, destined for the load-balancer, are duplicated by the switch using port-mirroring to my own machine. It is important that our solution is 100% non-intrusive to the web application being monitored. Probably I need to access the POST data of certain HTTP requests. Because HTTP requests are, obviously, broken into multiple packets, is it feasible to reconstruct the whole HTTP request with POST data from multiple packets? Regarding the load of 50.000 packets a second, is this expected to be a problem? Any feedback is very appreciated! Cheers, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] reconstruct HTTP requests in custom sniffer
Hi, > See urlsnarf: > > http://monkey.org/~dugsong/dsniff/ > > I don't think it does POST data but it may be a good starting point. > > Thanks, this seems to be very useful. It uses libnids which *hopefully* enables me to re-assemble the HTTP request + POST data from raw packets with little effort. Cheers, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] reconstruct HTTP requests in custom sniffer
Hi, I have implemented a HTTP parser one year ago. I remembered that when the > parser calculate the request-response latency, inspect the interested > fields > but do not record or dump them, the speed will reach about 2Gbps on a > single > core, and 8 Gbps on 6 cores. I think a 0.05Mpps parser is an easy work. > Thanks, that sounds promising. > > However, as you said you had to reconstruct the whole HTTP request with > POST > data, that will be a different story. You need to store the previous > packets > and do a memcpy() operation to concatenate them when latter packets are > received. In my experience, the cost is huge, especially the memcpy > operation. It depends on how many packets are such kind of cross-packet > POST > requests. Usual GET requests do not have this issue. > Hopefully libnids can do this for me efficiently... Cheers, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] reconstruct HTTP requests in custom sniffer
Hi Cedric, > Looks very similar to : > > http://github.com/securactive/junkie > > Is the intention of junkie to follow TCP streams and reassemble complete HTTP requests/responses from the packets? How far is this implemented? > if you can live with the AGPL, maybe we could join forces ? > At first sight it sounds tempting, as I have no intention to re-invent the wheel again. Though, the project "libnids" already seems to follow TCP streams and fully re-assemble HTTP requests/responses. Though, in some of our side-projects we need to follow TCP streams with truncated packets and libnids is not designed for this. It would be nice to use one solution for all our projects, and maybe junkie could solve this. Cheers, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] reconstruct HTTP requests in custom sniffer
Hi Cedric, > TCP reordering, IP fragmentation and buffering of stream is not present on > github > yet but is implemented and is being reviewed. I can push on github if you > want to > have a look. Concerning HTTP, for now we only fetch hostname and URL but > were > asked to capture the whole request including POST parameters so this is > going > to be done in a way or another. > Is there anything to say about a rough time-schedule? If we decide to use junkie, it would be nice to step in *after* the review. Then my experiences could serve to test the reviewed code, rather than pre-mature code. > > Though, in some of > > our side-projects we need to follow TCP streams with truncated packets > and > > libnids is not designed for this. > > Junkie tolerate a certain amount of truncation, but any complex parser will > certainly fail in this situation. > In some of our projects, we are only interested in the length of HTTP requests and responses therefor reassembling the whole requests would be overkill, as the segment lengths can be read from the TCP headers of packets in a TCP stream, obviously. In other projects, we definitely have to access the POST data need full-reassembly. Depending on the project, a different parsing-behavior is wanted. Will such behavior be configurable without having to write my own patches against junkie? > > > It would be nice to use one solution for > > all our projects, and maybe junkie could solve this. > > Honestly I can't recommend one over the other. Junkie has certainly more > bugs > since it's younger, but in other hand it's backed by a company so you have > at least 1 coder full time on it so the bugs can disapear pretty fast :-) > I do not mind a few bugs and get my hands in the mud :) One last concern is the licensing constraints. Suppose my company decides to use junkie and I will participate in bug fixes, real-life testing and who knows to what extend, then what are the constraints? Obviously we will be using junkie for our own sake, and the software build on top of junkie cannot be open-source, unfortunately. Moreover, I am not very familiar with licensing, but can we build our own software on top of junkie without financial obligations? Thank you, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] reconstruct HTTP requests in custom sniffer
Hi Cedric, > Support for TCP segmentation as well as new parsers that use this > feature should be pushed before end of week. Concerning the capture of > POST messages we should probably start working on this in february (this > is a small company so no schedule is ever definitive, so no promise). > I guess we just have to wait awhile then and see how things develop. We are a small company too, and the projects concerning POST re-assembly need some time to take off. After TCP segmentation is pushed, I will make some time to get a first feel of junkie. > > > In some of our projects, we are only interested in the length of HTTP > > requests and responses therefor reassembling the whole requests would be > > overkill, as the segment lengths can be read from the TCP headers of > packets > > in a TCP stream, obviously. > > Yes, in theory we could follow the sizes associated with each request quite > precisely even with truncated packets as long as the "Content-length" > header lines are present. Another way is too follow the TCP stream and summarize the payload lengths in the TCP headers instead of using HTTP headers, ignoring retransmitted packets. > To be honest, truncated packets were > introduced very recently and were not tested much (since we do not > require this feature), thus I'm not certain junkie is very robust in this > regard ; but I'm going to check. > Nice test-case for us would be to check if one could get the HTTP request/response lengths as described above. Cheers, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
[tcpdump-workers] live capture Ethernet gives me zero-packets
Hi,
I am trying to get started with a live capture using libpcap. Somehow I
cannot get the contents of a packet. It seems that all bytes are zeroed.
Here is the source code:
#include
#include
#include
void callback(u_char * user, const struct pcap_pkthdr* pkthdr, const u_char
* pkt)
{
fprintf(stdout, "caplen=%d, pkt[12]=%d, pkt[13]=%d\n", pkthdr->caplen,
pkt[12], pkt[13]);
}
int main(int argc,char **argv)
{
char errbuf[PCAP_ERRBUF_SIZE];
pcap_t * pd;
if(NULL == (pd = pcap_open_live("any", BUFSIZ, 1, 0, errbuf)))
{
printf("pcap_open_live(): %s\n", errbuf);
exit(1);
}
pcap_loop(pd, -1, callback, NULL);
return 0;
}
Output is like this:
caplen=1416, pkt[12]=0, pkt[13]=0
caplen=56, pkt[12]=0, pkt[13]=0
caplen=1416, pkt[12]=0, pkt[13]=0
caplen=1416, pkt[12]=0, pkt[13]=0
etc
I am running Ubuntu Maverick 64bit with kernel 2.6.35-28-generic. Note that
reading from a savefile works just fine, so my guess it has something to do
with memory mapping between kernel and user space.
Thank you,
Andrej
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] live capture Ethernet gives me zero-packets
Hi Guy, Thanks for the email. > (No, the "any" device doesn't give you Ethernet packets, even if, at the > time you start the capture, the only interfaces on your machine are Ethernet > interfaces. If you want to capture on a particular Ethernet device, use its > name, e.g. "eth0", in which case you'll presumably get packets that have > Ethernet headers - although you should probably check the value returned by > pcap_datalink() whenever you do any pcap_open call, including > pcap_open_offline() to read from a savefile, or when you do > pcap_create()/.../pcap_activate().)- > Indeed I assumed that since I have only ethernet interfaces that the link-type for any would be EN10MB. Now I now this is false on Linux when using "any". Greets, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
[tcpdump-workers] DCERPC
Hi, I wrote a sniffer using libpcap that re-assembles TCP streams to enable HTTP request/response re-assembly. It works fine except when DCERPC-packets are found in the middle of a data-transfer between an HTTP client and server (example of such a DCERPC-packet see below, captured with Wireshark). Why do these packets show up (not often though) in the middle of an HTTP stream? How can I recognize these packets using libpcap? Thank you, Andrej Frame 461 (11282 bytes on wire, 11282 bytes captured) Arrival Time: Apr 13, 2011 21:54:10.076378000 [Time delta from previous captured frame: 0.29000 seconds] [Time delta from previous displayed frame: 0.29000 seconds] [Time since reference or first frame: 34.142183000 seconds] Frame Number: 461 Frame Length: 11282 bytes Capture Length: 11282 bytes [Frame is marked: False] [Protocols in frame: eth:ip:tcp:http:dcerpc] [Coloring Rule Name: Checksum Errors] [Coloring Rule String: cdp.checksum_bad==1 || edp.checksum_bad==1 || ip.checksum_bad==1 || tcp.checksum_bad==1 || udp.checksum_bad==1 || mstp.checksum_bad==1] Ethernet II, Src: Dell_99:6d:be (b8:ac:6f:99:6d:be), Dst: All-HSRP-routers_12 (00:00:0c:07:ac:12) Internet Protocol, Src: 85.17.148.22 (85.17.148.22), Dst: 175.105.93.20 (175.105.93.20) Transmission Control Protocol, Src Port: http (80), Dst Port: 53444 (53444), Seq: 1885021513, Ack: 2538648414, Len: 11216 Hypertext Transfer Protocol DCE RPC Request, Fragment: Mid, FragLen: 5, Call: 2236416 Version: 5 Version (minor): 0 Packet type: Request (0) Packet Flags: 0x00 0... = Object: Not set .0.. = Maybe: Not set ..0. = Did Not Execute: Not set ...0 = Multiplex: Not set 0... = Reserved: Not set .0.. = Cancel Pending: Not set ..0. = Last Frag: Not set ...0 = First Frag: Not set Data Representation: Byte order: Big-endian (0) Character: ASCII (0) Floating-point: IEEE (0) Frag Length: 5 Auth Length: 16400 Call ID: 2236416 [Unreassembled Packet [incorrect TCP checksum]: DCERPC] [Expert Info (Warn/Reassemble): Unreassembled Packet (Exception occurred)] [Message: Unreassembled Packet (Exception occurred)] [Severity level: Warn] [Group: Reassemble] [DCE RPC: 11211 bytes left, desegmentation might follow] - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
[tcpdump-workers] only outbound traffic
Hi, I am facing an issue that I cannot solve... I was hoping somebody could get me back on track... I have a ProCurve J9021A Switch 2810-24G and configured port A to be mirrored to port B. When I run tcpdump on port B, I can only see the outgoing traffic on port A, but no incoming traffic at all! The configuration menu of the switch does not have any options for egress and/or ingress. According to the manual, it is supposed to do both at all times. I also turned off all Firewalls, but still no luck. What am I forgetting? Thank you, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] only outbound traffic
Hi, > I have a ProCurve J9021A Switch 2810-24G and configured port A to be > mirrored to port B. When I run tcpdump on port B, I can only see the > outgoing traffic on port A, but no incoming traffic at all! The > configuration menu of the switch does not have any options for egress > and/or ingress. According to the manual, it is supposed to do both at > all times. I also turned off all Firewalls, but still no luck. What am > I forgetting? Well, I experienced VLAN-tags for the first time... This solved my problem for listening on port 3306: tcpdump -i eth1 "(port 3306) or (vlan and port 3306)" Is there any documentation on how libpcap/tcpdump/BPF deal with VLAN tags? Its still a bit of a mystery to me... Thank you, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] only outbound traffic
Hi, > > Does this help? > > https://blog.wireshark.org/2009/10/capture-filters-and-offsets Yes it does. Makes me wonder though why BPF was not extended with an "offset" keyword. Thanks for the link! Andrej- This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] pcap anonymizer
With tcprewrite you can change ips too. Not sure if it updates checksums though... Andrej > - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] only outbound traffic
Hi,
>
> Why would an "offset" keyword be better in the filtering language than, say,
> the "vlan" keyword it already has? You'd still have to do the same sort of
> special stuff, but it'd be a more manual operation. (I.e., why would saying
> "offset {length of VLAN tag}" be better than "vlan"?)
>
Its more explicit too me. It is not really intuitive that "port 80 and vlan"
and " vlan and port 80" gives different results, until you realize that vlan
increases the ether type offset.
> The ideal would be a filtering language wherein having the filter code in the
> kernel skip past VLAN tags automatically was cheap. Perhaps a language (not
> a language for users to express filters, but a language into which to compile
> the filters the user expresses) that makes it impossible to specify infinite
> loops, combined with a JIT to make loops reasonably efficient (there already
> exist JITs for x86-32 and x86-64 on some platforms, e.g. Windows and
> FreeBSD), would be the right way to handle VLANs and IPv6 protocol chains and
> perhaps even filters at higher protocol levels.-
If you say so ;)
Arent there any special port mirroring NICS out there that remove those VLAN
tags?
Cheers,
Andrej-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.
[tcpdump-workers] libpcap and certificates
Hi, I would like to ask if somebody could point me to information on howto hook up libpcap to sniff a secure website on HTTPS. I understand I have to decrypt the packets probably using openssl, but I wonder if there is some howto to guide me. I am on the latest Ubuntu. Cheers, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] libpcap and certificates
>> I would like to ask if somebody could point me to information on howto >> hook up libpcap to sniff a secure website on HTTPS. I understand I >> have to decrypt the packets probably using openssl, but I wonder if >> there is some howto to guide me. I am on the latest Ubuntu. > > Last I checked, tcpdump supports decrypting IPSec, but not SSL. > However, Wireshark decrypts SSL. You'll need the SSL private key of > the webserver to do Yes i am aware Wireshark can do it if you provide the private key, but i need to decrypt packets in my own sniffer based on libpcap. I was hoping i am not the first and somebody could gimme some startup pointers, so that i dont have to go through the Wireshark sources... But maybe i am at the wrong place and should turn to the openssl forum instead... Cheers, Andrej- This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] libpcap and certificates
> Actually the SourceForge repository has more recent activity: > > http://sourceforge.net/projects/ssldump/develop Thanks, ssldump seems like a good starting point. Cheers, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
[tcpdump-workers] IP range implementation given an IP address and subnet mask
Hi, I would like to implement an IP address range filter similar to the BPF capture filter that takes an IP address and subnet mask (like in "net 192.168.0.0 and mask 255.255.255.0"). What would be the best way to calculate this in C? Or in which file can I find the BPF implementation of this expression? Cheers, Andrej - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Re: [tcpdump-workers] IP range implementation given an IP address and subnet mask
> I would like to implement an IP address range filter similar to the
> BPF capture filter that takes an IP address and subnet mask (like in
> "net 192.168.0.0 and mask 255.255.255.0"). What would be the best way
> to calculate this in C? Or in which file can I find the BPF
> implementation of this expression?
Ok, pretty straightforward using the CIDR notation:
bool in_cidr_subnet(unsigned long ip_addr,
unsigned long cidr_network_addr,
unsigned short cidr_bits)
{
const unsigned long mask = ntohl(((unsigned long) -1) << (32-cidr_bits));
assert(cidr_network_addr & mask == cidr_network_addr);
const unsigned long ip_network_addr = ip_addr & mask;
return cidr_network_addr == ip_network_addr;
}
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.
