Re: [tcpdump-workers] Proposed new pcap format

[Stephen Donnelly]

Jefferson Ogata wrote:
Something keeps bugging me, and I just want to throw it out there for 
the mad dogs to tear into little bloody pieces:

Given all the desirable options people are looking for in this, and the 
need for future growth, I think we should seriously consider an 
XML-based format. Besides making it easy, format-wise, to include many 
optional features and types of metadata, programs could also embed 
decoded frame and protocol information in appropriate elements, right 
within the capture file.

Yes, fully fledged decoded captures would use a lot of extra disk, but a 
raw no-frills capture could be recorded with maybe only 50% or so overhead.
50% extra space and 50% extra disk bandwidth cost? So my 250 Megabyte per second 
pcap stream to disk becomes 375MB/s?

I'm not keen so far, although I understand the desire for flexibility. I think 
the file format needs to be as space efficient per record as possible. Extra 
information can still be carried in 'file headers', 'metadata packets', or 
attached to each packet record in *optional* metadata fields that can be omitted 
for space/speed.

Stephen.
--
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---
-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] Proposed new pcap format

[Stephen Donnelly]

Ronnie Sahlberg wrote:
Yes, fully fledged decoded captures would use a lot of extra disk, but a
raw no-frills capture could be recorded with maybe only 50% or so
overhead.

50% extra space and 50% extra disk bandwidth cost? So my 250 Megabyte per
second pcap stream to disk becomes 375MB/s?
I would be extatic if the captures people send to me would be tiny 250
MByte.
Some people work with captures that are 1GByte in size and bigger.
If you read my email again, you will notice I said 250 megabytes per second. 
This is an expression of a data rate, not a file size. The captured file size 
will be determined by the rate, multiplied by the capture time. E.g. 250MB/s * 
24 hours.

Stephen.
--
---
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---
-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] Proposed new pcap format

[Stephen Donnelly]

Jefferson Ogata wrote:
50% extra space and 50% extra disk bandwidth cost? So my 250
Megabyte per second
pcap stream to disk becomes 375MB/s?
...

Raw packet data would typically be base64-encoded. This expands data by 
33%; three octets become four. You don't have to write one octet as two.

In any case, if you're trying to capture every packet off the wire, you 
might not want to use the newer binary pcap format under discussion 
either. It's looking to impose some not insignificant overhead as well.
When capturing network data at hundreds of megabytes per second for extended 
periods and hence dealing with hundreds of gigabytes of captured data at a time, 
even 33% overhead is very expensive in storage space and disk bandwidth, as well 
as the cpu time required to perform XML output with base-64 encoding.

This is why my interest in the new format is to encourage keeping the fixed 
overhead per packet record small. This can be done by a) keeping per-packet meta 
data optional where possible, and b) keeping space efficiency in mind when 
encoding per packet (meta)data.

It may well be true that for analysis XML is useful either internally for 
processing, or for results, but libpcap is primarily about packet capture.

Stephen.
--
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---
-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] Are all traces captured by dag card in "tcpdump"

[Stephen Donnelly]

ice ice wrote:
I have a trace saying
"Data provided by WAND Research Group using the dag interface card
OC48 data analysis required CAIDA's CoralReef software suite."
I am confused by the statement of "OC48 data analysis required CAIDA's 
CoralReef software suite".

It seems to me that traces captured by dag card are collections of 
packet headers. And I can use Tcpdump or CoralReef libary in reading the 
packet information from the trace. And I even can directly read header 
by header (IP+TCP/UDP/or other+..) from the trace by my own program, and 
interpret the information in packet by matching the structure specified 
in RFC.

Then why "OC48 data analysis required CAIDA's CoralReef software suite"?
I apply the tcpdump on the trace, it also can print out the packet 
information. But when I write my own program to parse through the trace, 
I can not get right information. Why is that?
If tcpdump can parse the file, there is a good chance it is in 'libpcap' 
format. You can tell easily by running 'file yourfilename', e.g.

$ file /usr/var/tmp/foo.pcap
/usr/var/tmp/foo.pcap: tcpdump capture file (little-endian) - version 2.4 
(Ethernet, capture length 68)

DAG cards have their own native format as well, but the research group may 
have converted the traces to libpcap format for public convienience. 
Perhaps they did this using CoralReef.

How are you attempting to parse it if you are having trouble? Note you 
shouldn't assume it uses DLT_EN10MB.

Stephen.
--
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---
-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] Patch to print out IP data in PPP HDLC packets

[Stephen Donnelly]

Normally a NIC or interface providing access to an HDLC link/network would 
perform de-bytestuffing internally before the packets ever get to libpcap. 
Obviously de-bytestuffing a packet twice can corrupt data. Do we really 
want to have de-bytestuffing code in libpcap?

Surely at minimum it should be off by default and selectable as an option 
perhaps, if necessary?

Stephen.
Hannes Gredler wrote:
On Thu, Jul 01, 2004 at 09:32:26PM +1000, Darren Reed wrote:
| I've been using this patch to print IP packets inside PPP HDLC
| frames found in raw 1xRTT traffic.  I've been able to find few
| details on the actual PPP header format apart from what "0x7eff"
| means and observing traffic for 0x7e21.  The end result is extra
| output of the form "{ PPP HDLC IP 1.2.3.4 > 2.3.4.5: GREv1call 0}"
| 
| It may not be particularly efficient because it malloc's a new
| buffer for each packet (rather than using a static buffer) but
| better that than limit the program's capabilities w.r.t recursive
| decoding was what I decided.
| 
| Darren

darren,
thanks for your submission - i have checked in the attached patch;
/hannes

Index: print-ppp.c
===
RCS file: /tcpdump/master/tcpdump/print-ppp.c,v
retrieving revision 1.95
diff -u -r1.95 print-ppp.c
--- print-ppp.c	2 Jul 2004 06:32:47 -	1.95
+++ print-ppp.c	2 Jul 2004 20:15:32 -
@@ -47,6 +47,7 @@
 
 #include 
 #include 
+#include 
 
 #include "interface.h"
 #include "extract.h"
@@ -370,6 +371,7 @@
 static int print_ccp_config_options (const u_char *p, int);
 static int print_bacp_config_options (const u_char *p, int);
 static void handle_ppp (u_int proto, const u_char *p, int length);
+static void ppp_hdlc(const u_char *p, int length);
 
 /* generic Control Protocol (e.g. LCP, IPCP, CCP, etc.) handler */
 static void
@@ -1052,10 +1054,81 @@
 }
 
 
+static void
+ppp_hdlc(const u_char *p, int length)
+{
+	u_char *b, *s, *t, c;
+	int i, proto;
+	const void *se;
+
+	b = (u_int8_t *)malloc(length);
+	if (b == NULL)
+		return;
+
+	/*
+	 * Unescape all the data into a temporary, private, buffer.
+	 * Do this so that we dont overwrite the original packet
+	 * contents.
+	 */
+	for (s = (u_char *)p, t = b, i = length; i > 0; i--) {
+		c = *s++;
+		if (c == 0x7d) {
+			if (i > 1) {
+i--;
+c = *s++ ^ 0x20;
+			} else
+continue;
+		}
+		*t++ = c;
+	}
+
+	se = snapend;
+	snapend = t;
+
+/* now lets guess about the payload codepoint format */
+proto = *b; /* start with a one-octet codepoint guess */
+
+switch (proto) {
+case PPP_IP:
+ip_print(b+1, t - b - 1);
+goto cleanup;
+#ifdef INET6
+case PPP_IPV6:
+ip6_print(b+1, t - b - 1);
+goto cleanup;
+#endif
+default: /* no luck - try next guess */
+break;
+}
+
+proto = EXTRACT_16BITS(b); /* next guess - load two octets */
+
+switch (proto) {
+case 0xff03: /* looks like a PPP frame */
+proto = EXTRACT_16BITS(b+2); /* load the PPP proto-id */
+handle_ppp(proto, b+4, t - b - 4);
+break;
+default: /* last guess - proto must be a PPP proto-id */
+handle_ppp(proto, b+2, t - b - 2);
+break;
+}
+
+cleanup:
+snapend = se;
+	free(b);
+return;
+}
+
+
 /* PPP */
 static void
 handle_ppp(u_int proto, const u_char *p, int length)
 {
+if ((proto & 0xff00) == 0x7e00) {/* is this an escape code ? */
+ppp_hdlc(p-1, length);
+return;
+}
+
 	switch (proto) {
 	case PPP_LCP:
 	case PPP_IPCP:

--
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---
-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] New magic number

[Stephen Donnelly]

Guy Harris wrote:
 > When do you expect the new format will be available?
I don't think we have a date yet.  I think we'd like to finish up the
specification soon; it'll take longer to implement APIs to use all the
capabilities, although we could probably add the ability to read those
files - or, at least, files in that format that don't have captures from
more than one network interface - sooner than that, with the existing
APIs (which won't show all the data available in the file).
 > If I can help, let me know.
The current specification for the new format can be found at
http://www.tcpdump.org/pcap/pcap.html
or
http://www.tcpdump.org/pcap/pcap.txt
Send any suggestions you have to the list.
I note this draft is due to expire at the end of August, is there a 
repository of changes made since the draft was published?

Is this list the primary discussion forum for this draft? I see a reference 
to 'Network Working Group', is this an IETF body?

Regards,
Stephen.
--
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---
-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] number of concurrent TCP sessions

[Stephen Donnelly]

tcpdump may not be the right tool for the job, but considerable work has 
been done on IP flows.

You might want to look at tcptrace, or a flows analysis package like 
Coralreef, or a flow probe like fprobe or ntop.

http://jarok.cs.ohiou.edu/software/tcptrace/tcptrace.html
http://www.caida.org/tools/measurement/coralreef/
http://fprobe.sourceforge.net/
http://www.ntop.org/
Regards,
Stephen.
César Cárdenas wrote:
Many Thanks Kiss, Dear all:
I am using windump (windows 2000)...
I really appreciate if you could say me how I can determine the number of
concurrent TCP connections?
César

-- Mensaje Original --
Date: Tue, 24 Aug 2004 19:57:36 +0200 (CEST)
From: Kiss Karoly <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Subject: Re: [tcpdump-workers] number of concurrent TCP sessions
Reply-To: [EMAIL PROTECTED]
On Tue, 24 Aug 2004, [iso-8859-1] César Cárdenas wrote:

Dear all:
In a captured file I found '.', S, F and FP flags...
According to the manual:
flag = '.' and data-seqno = '1' implies the first time tcpdump sees a
TCP
conversation.

flag = 'S' and 'win (value)' stands for the beginning of a TCP conversation
flag = 'F" implies FIN (end) and flag = 'FP' I guess implies Fin/Pushed
(anyway end)
I computed the number of concurrent TCP conversations throughout the ti
e
by adding a '1' each time I found a 'S' and substractin a '1' each time
I found a 'F' or a 'FP'
By doing this the number of concurrent TCP connections decreases linearly
in a negative way through the time.
Am I determining in a c
rrect way the number of Concurrent TCP connections?
I really appreciate if you could suggest me how to determine the number
of concurrent TCP connections?
Please accept mys best regards,
Cesar Cardenas
-
This is the tcpdump-work
rs list.
Visit https://lists.sandelman.ca/ to unsubscribe.

You forgot to mention the system you are using but if wou use linux and
have connection tracking enabled in the kernel ( module ip_conntrack )
then it's much easyer if you do a wc -l
/proc/net/tcp
This will give you the number of connections pretty accurately.
But be careful with using ip_conntrack because it makes your box
vulnerable to SYN flood attacks.
Regards
Karoly Kiss
-
--
---
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---
-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] PCAP Timestamp - HWClock or SWClock?

[Stephen Donnelly]

Guy Harris wrote:
J.O. Leger wrote:
Is the timestamp in pcap_pkthdr from the hardware clock or the 
software clock?

The timestamp is from whatever it's from. :-)

If you're capturing from a DAG card from Endace (recent versions of 
libpcap can be built with support for DAG cards), I suspect the time 
stamp might come from a purely hardware clock.
That's correct, the cards have a hardware clock used for timestamping 
onboard. It records a timestamp at high resolution, the libpcap code 
'degrades' this to microseconds which is the libpcap native resolution.

We have been asked 'why libpcap doesn't support higher resolution 
timestamps'. Since there is no standard support for that in libpcap at 
present we are waiting for 'NG pcap' to surface.

I suspect, however, that you didn't ask a question the answer to which 
tells you what you ultimately really want to know.

If what you *really* want to know is "how accurate are the time 
stamps?", then the answer is "if you care enough about that to ask, they 
probably aren't as accurate as you'd like", because
-good points about software timestamp limitations snipped-
DAG cards capture their timestamps at the beginning of the packet. For 
Ethernet this is generally the SFD byte. I'm happy to discuss specifics 
off-list if people are interested.

Stephen.
--
---
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---
-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] [ANNOUNCE] NTAR - PCAP next generation dump

[Stephen Donnelly]

I'm a bit confused about naming. How does NTAR stand with respect to libpcap?

Is it in fact libpcap 1.0? Or just an independent implementation of the 
proposed/agreed format for 'libpcap-ng'?


Can NTAR read 'old' format libpcap traces?

What's next for libpcap development, is there the intent for a new version 
of libpcap to also process the new format? Independently of NTAR? With or 
without backwards compatibility at the file reading or API levels?


Thanks,
Stephen.

Gianluca Varenni wrote:

Hi all.

This mail is to announce the birth of the NTAR project. NTAR stands for
Network Trace Archival and Retrieval library, and is an implementation of
the PCAP next generation dump file format, that was proposed and discussed
last year by several folks on the libpcap/tcpdump and WinPcap mailing 
lists.

The library is released under the 3-clause/BSD license.

The URL of the project is:

http://www.winpcap.org/ntar

On this website you can find
- the source file of the library (both for windows and *nix)
- the HTML documentation of the API (generated with doxygen from the
 commented source files), both for the user  and for someone wanting to
 extend it. The docs contain some tests/examples that you can look to
 get an idea of how the library works.
- An updated version of the PCAP draft specifying the file format.
 The original draft of the file format is available at
http://www.tcpdump.org/pcap/pcap.html



I'm seeking contributors to improve the library in terms of
- testing the library on different platforms and operating systems
- reviewing the API
- implementing new extensions to the library.

A mailing list, [EMAIL PROTECTED], has been created for NTAR-related
discussions. People interested in this project are welcome to join it, the
mailman web interface to subscribe is available at

https://www.winpcap.org/mailman/listinfo/ntar-workers

Have a nice day
Gianluca Varenni

--
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---
-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.

Re: [ntar-workers] Re: [tcpdump-workers] [ANNOUNCE] NTAR - PCAP next

[Stephen Donnelly]

Guy Harris wrote:
What's next for libpcap development, is there the intent for a new  
version of libpcap to also process the new format?


Yes, and to add additional capabilities as well.


Independently of NTAR?


At least as I see it, it'd come out at the same time, or after, NTAR,  
and would use NTAR to handle the low-level details of reading pcap-NG  
files.


With or without backwards compatibility at the file reading or API  
levels?


With backwards compatibility at the file reading layer, so that it  can 
read old libpcap files.


With backwards compatibility at the API layer, *but* with some  
additional new APIs to expose the full capabilities of the new format  
and to add additional capabilities.


Thanks for the clarification, it can be difficult to understand the libpcap 
'roadmap' from out here. Your suggestions all sound perfectly reasonable to me.


(Are there any API changes that  
would allow capturing with libpcap on a DAG device to run at a speed  
closer to that of capturing using the native DAG interface?)


Curiously enough I have just been looking at libpcap performance with DAG 
cards. The answer is that the current libpcap DAG code works pretty well, 
it's not much slower than the native interface. There are however some 
changes I'm planning that will improve things a bit more. So far my 
proposed changes affect only the library internals, they do not require 
changes to the libpcap API.


Regards,
Stephen.
--
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---
-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] problem with parsing Leipzig-I trace

[Stephen Donnelly]

From the web pages you mentioned, the Leipzig-I trace page says that it 
was taken from a Packet over SONET link. Did you try the "dagbpf -p" flag 
for PoS?


Regards,
Stephen.

Zhen Wu wrote:

Hello, everyone:

I am using dagtools and tcpdump to parse the Leipzig-I trace. The output 
is NOT what I expected. Using the same command, I can successfully parse 
the Auckland-IV trace.


Anyone can help me??? Thanks a lot!

Zhen


output from parsing Leipzig-I trace, from 
"http://pma.nlanr.net/Special/leip1.html";

--
$ zcat 20021125-14-0.gz | /usr/local/dagtools-0.8.1/pcap/dagbpf -v | 
/usr/local/sbin/tcpdump -n -tt -r -| more

dagbpf: verbose: header
dagbpf: verbose: sloop
reading from file -, link-type ATM_RFC1483 (RFC 1483 IP-over-ATM)
1038229200.000249 sap 02 > sap 0a 83/P
1038229200.000275 sap 00 > sap 0a rnr (r=59,F)
1038229200.000312 sap 02 > sap 0a 83/P
1038229200.000385 sap 02 > sap 0a 83/P

output from parsing Auckland-IV trace, from 
"http://pma.nlanr.net/Traces/long/auck4.html";

--
$ zcat 20010309-02-0.gz | /usr/local/dagtools-0.8.1/pcap/dagbpf -v | 
/usr/local/sbin/tcpdump -n -tt -r - | more

dagbpf: verbose: header
dagbpf: verbose: sloop
reading from file -, link-type ATM_RFC1483 (RFC 1483 IP-over-ATM)
984056400.009423 IP 10.0.45.255.80 > 10.0.0.53.4608: . ack 397996760 win 
8760
984056400.012529 IP 10.0.45.255.80 > 10.0.0.53.4608: P 0:159(159) ack 1 
win 8760
984056400.012546 IP 10.0.45.255.80 > 10.0.0.53.4608: F 159:159(0) ack 1 
win 8760
984056400.013616 IP 10.2.179.148.2875 > 10.0.1.19.80: . ack 584221866 
win 31856 <[|tcp]>


The version of my tcpdump
--
tcpdump version current-cvs.tcpdump.org.2004.06.20
libpcap version 0.7

-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.



--
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---
-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] problem with parsing Leipzig-I trace

[Stephen Donnelly]

Hi Zhen,

Well, 0x21 is the IANA PPP DLL protocol number for Internet Protocol 
version 4, so perhaps the link in question is RFC2615 PPP over SONET rather 
than PoS/Cisco HDLC.


You could try changing DLT_CHDLC to DLT_PPP_SERIAL in dagbpf.c and 
recompiling. Alternatively contact NLANR PMA via the feedback address on 
the webpage you mentioned and ask them about how to process the trace.


Regards,
Stephen.

Zhen Wu wrote:

Yes. I tried different flags. Here is what I got by trying "dagbpf -p".
-
$ zcat 20021125-14-0.gz | /usr/local/dagtools-0.8.1/pcap/dagbpf -p  
| /usr/local/sbin/tcpdump -n -tt -r -| more

reading from file -, link-type C_HDLC (Cisco HDLC)
1038229200.000249 unknown CHDLC protocol (0x0021)
1038229200.000275 unknown CHDLC protocol (0x0021)
1038229200.000312 unknown CHDLC protocol (0x0021)
1038229200.000385 unknown CHDLC protocol (0x0021)

There must be a way to read/parse the Leipzig-I trace. I just couldn't 
figure it out. I would appreciate any suggestion. Thanks!


Zhen

On Jul 10, 2005, at 2:19 PM, Stephen Donnelly wrote:

From the web pages you mentioned, the Leipzig-I trace page says that 
it was taken from a Packet over SONET link. Did you try the "dagbpf 
-p" flag for PoS?


Regards,
Stephen.

Zhen Wu wrote:


Hello, everyone:
I am using dagtools and tcpdump to parse the Leipzig-I trace. The 
output is NOT what I expected. Using the same command, I can 
successfully parse the Auckland-IV trace.

Anyone can help me??? Thanks a lot!
Zhen
output from parsing Leipzig-I trace, from 
"http://pma.nlanr.net/Special/leip1.html";

--
$ zcat 20021125-14-0.gz | /usr/local/dagtools-0.8.1/pcap/dagbpf 
-v | /usr/local/sbin/tcpdump -n -tt -r -| more

dagbpf: verbose: header
dagbpf: verbose: sloop
reading from file -, link-type ATM_RFC1483 (RFC 1483 IP-over-ATM)
1038229200.000249 sap 02 > sap 0a 83/P
1038229200.000275 sap 00 > sap 0a rnr (r=59,F)
1038229200.000312 sap 02 > sap 0a 83/P
1038229200.000385 sap 02 > sap 0a 83/P
output from parsing Auckland-IV trace, from 
"http://pma.nlanr.net/Traces/long/auck4.html";

--
$ zcat 20010309-02-0.gz | /usr/local/dagtools-0.8.1/pcap/dagbpf 
-v | /usr/local/sbin/tcpdump -n -tt -r - | more

dagbpf: verbose: header
dagbpf: verbose: sloop
reading from file -, link-type ATM_RFC1483 (RFC 1483 IP-over-ATM)
984056400.009423 IP 10.0.45.255.80 > 10.0.0.53.4608: . ack 397996760 
win 8760
984056400.012529 IP 10.0.45.255.80 > 10.0.0.53.4608: P 0:159(159) ack 
1 win 8760
984056400.012546 IP 10.0.45.255.80 > 10.0.0.53.4608: F 159:159(0) ack 
1 win 8760
984056400.013616 IP 10.2.179.148.2875 > 10.0.1.19.80: . ack 584221866 
win 31856 <[|tcp]>

The version of my tcpdump
--
tcpdump version current-cvs.tcpdump.org.2004.06.20
libpcap version 0.7
-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.




--
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---
-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.



-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.



--
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---
-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] user provided packet buffer

[Stephen Donnelly]

On Mon, 2005-08-15 at 13:31 -0500, Randy Smith wrote:
> Hello,
> 
> I'm using libpcap to capture packets, but by default the packets are 
> stored in a buffer allocated "inside" the pcap library (e.g., the 
> "buffer" pointer of the pcap_t struct).
> 
> Does the pcap library provide any API's that allow the user to specify 
> his/her own pcap_pkthdr buffer and capture buffer?
> 
> I couldn't find any, and looking through the source code itself it 
> appears that there is no way for me to provide my own buffers.
> 
> I've used the libpcap API a bit and am familiar with it from that 
> perspective.  If I choose to extend the library by adding a routine to 
> accept user defined buffers, are there any reasons that you are aware of 
> a priori why this would fail?

It certainly wouldn't work with DAG cards, which provide libpcap readers
with direct pointers to buffers memory mapped from kernel space.

Implementing a 'user buffer' scheme as you described would require an
inefficient packet copy operation into the supplied buffer.

I expect other high-performance libpcap implementations that also use
memory mapping and would have similar problems.

Why is it that you want packets in user allocated buffers? It seems to
me that requiring the user to do their own explicit copies when required
is not unreasonable.

Regards,
Stephen.
-- 
---
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] Paquets smaller than 64 bytes

[Stephen Donnelly]

On Wed, 2005-11-23 at 23:49 +0100, David Rosal wrote:
> Gianluca Varenni wrote:
> > The minimum ethernet frame length is 64 bytes *if* you include the FCS. 
> > Unfortunately, most of the network cards strip the FCS before the packet 
> > reaches the host, so the actual minimum frame length that you see with 
> > libpcap is actually 60 bytes.
> > 
> > Hope it helps
> 
> Yes it does. Thanks.
> But a question arises: If my network card is stripping the FCS (and it 
> seems to do so), may I suppose that this is done for all packets? In 
> other words, it is safe to add 4 bytes to the sizes of *all* captured 
> packets to get the sizes on wire?

You can also add an unknown number of bytes of preamble (typ. 8), and 12
bytes of Inter-frame Gap if you like. Depends what you mean by 'On the
wire'.

Stephen.
-- 
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] What is the main reason in absent append

[Stephen Donnelly]

On Thu, 2006-02-16 at 20:17 +0300, Mikhail Manuylov wrote:
> Hi there,
> 
> All I wonder is why tcpdump still hasn't any binary dump append feature.

The biggest problem I imagine is that the resulting file would have only
one header block, so the configuration of the capture for the appended
records would have to be the same as for the original file.

I'm not sure how you could check for or enforce this?

The 'NTAR' file format intended for pcap-ng supports directly appending
capture files together, allowing new header blocks to redescribe the
interfaces and capture parameters.

Stephen.
-- 
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] pcap file format documentation

[Stephen Donnelly]

It may be worth noting (AFAIK) the libpcap file format is intended to be
opaque, with access for read/writing provided only by libpcap itself.

This allows the implementation of the file format to be changed by the
libpcap maintainers, while remaining transparent to the user.

If you write your own code to read/write the current libpcap file format
it may not deal with older files or with potential new changes (aka
pcap-ng, pcap 1.0, NTAR etc)

Stephen.

On Sun, 2006-03-19 at 17:59 -0800, Don Morrison wrote:
> Hello,
> 
> Is there documentation describing the pcap file formats (other than
> the libpcap source)?
> 
> Thanks,
> Don
> -
> This is the tcpdump-workers list.
> Visit https://lists.sandelman.ca/ to unsubscribe.
-- 
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] pcap file format documentation

[Stephen Donnelly]

Hi Don,

That sounds quite likely. This may well be a case where you need to edit
the file directly, and it seems unlikely that the compatibility issues I
mentioned would be a problem.

Alternatively have you looked to see if NetDude will do what you want?

Stephen.

On Sun, 2006-03-19 at 20:43 -0800, Don Morrison wrote:
> Hi Stephen,
> 
> Here's the problem.  I'm dealing with corrupted pcap files, where the
> last packet was partially written, but it's not of interest and all I
> want to do is truncate the last packet.  My assumption is that
> libpcap's API will not allow me to deal with this since programs that
> are dependent on it (tcpdump, ethereal) hang when attempting to open
> any such file.  Is this assumption incorrect?
> 
> Thanks,
> Don
> 
> On 3/19/06, Stephen Donnelly <[EMAIL PROTECTED]> wrote:
> > It may be worth noting (AFAIK) the libpcap file format is intended to be
> > opaque, with access for read/writing provided only by libpcap itself.
> >
> > This allows the implementation of the file format to be changed by the
> > libpcap maintainers, while remaining transparent to the user.
> >
> > If you write your own code to read/write the current libpcap file format
> > it may not deal with older files or with potential new changes (aka
> > pcap-ng, pcap 1.0, NTAR etc)
> >
> > Stephen.
> >
> > On Sun, 2006-03-19 at 17:59 -0800, Don Morrison wrote:
> > > Hello,
> > >
> > > Is there documentation describing the pcap file formats (other than
> > > the libpcap source)?
> > >
> > > Thanks,
> > > Don
> > > -
> > > This is the tcpdump-workers list.
> > > Visit https://lists.sandelman.ca/ to unsubscribe.
> > --
> > ---
> > Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
> > Endace Technology Ltd   phone: +64 7 839 0540
> > Hamilton, New Zealand   cell:  +64 21 1104378
> > ---
> >
> > -
> > This is the tcpdump-workers list.
> > Visit https://lists.sandelman.ca/ to unsubscribe.
> >
> -
> This is the tcpdump-workers list.
> Visit https://lists.sandelman.ca/ to unsubscribe.
-- 
---
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] [RESEND][PATCH] enable sniff on USB ports

[Stephen Donnelly]

Is this still the right root?

:pserver:[EMAIL PROTECTED]:/tcpdump/master

I'm getting:

...
cvs update: Updating pcap
cvs update: failed to create lock directory for
`/tcpdump/master/libpcap/pcap' (/tcpdump/master/libpcap/pcap/#cvs.lock):
Permission denied
cvs update: failed to obtain dir lock in repository
`/tcpdump/master/libpcap/pcap'
cvs [update aborted]: read lock failed - giving up

Regards,
Stephen.

On Wed, 2006-10-04 at 17:51 +0200, Hannes Gredler wrote:
> paolo,
> 
> checked in.
> can you make a fresh checkout and verify if everything is working as expected 
> ?
> 
> tx,
> 
> /hannes
-- 
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] [RESEND][PATCH] enable sniff on USB ports

[Stephen Donnelly]

On Wed, 2006-10-04 at 16:53 -0400, Michael Richardson wrote:
> >>>>> "Stephen" == Stephen Donnelly <[EMAIL PROTECTED]> writes:
> Stephen> (/tcpdump/master/libpcap/pcap/#cvs.lock): Permission denied
> 
>   Appologies. the lockdir stuff got lost.
>   Try now.

Working now, thanks.

Stephen.
-- 
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] Headroom

[Stephen Donnelly]

On Fri, 2006-10-06 at 13:46 -0400, Harley Stenzel wrote:
> Greetings--
> 
> Has the idea of headroom for libpcap-based stack applications been
> discussed before?  The idea is to have a known amount of empty/usable
> space in front of the buffer given to the application by the
> pcap_dispatch callback, pcap_next(), or pcap_next_ex().
> 
> It looks like it'd be a ((pcap_t *)p)->headroom, which would be added
> to p->offset and p->bufsize for malloc.
> 
> p->offset would still have to guarantee alignment for for the data,
> not for the beginning of the buffer.
> 
> Headroom would have to be specified at pcap_open_live() or
> pcap_open_offline() time, so it could alter the pcap_t initialization
> logic. I suppose the best approach would be to make an additional
> variant of those functions, so the prototype changes don't ripple to
> other apps.  Possibly have it be an additional options structure,
> which may be null to get the current behavior.
> 
> Any thoughts?

Wouldn't work with Endace DAG cards, as libpcap accesses memory mapped
packet buffers directly.

If this feature is accepted into libpcap there would need to be a way to
'reject' the option, perhaps via a specific function call like
pcap_setnonblock()?

Stephen.
-- 
---
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] print-tcp.c: remove commas from output, to

[Stephen Donnelly]

On Wed, 2007-01-17 at 11:56 -0800, Rick Jones wrote:
> Kevin Steves wrote:
> > commas aren't used in tcp fields so remove these that are
> > before and after cksum.
> 
> i'm not necessarily trying to stop the patch, but while it may not be 
> consistent with other output, maintaining "consistency" with previous 
> versions has the nice property of being less likely to break someones 
> scripts no?

I'm pretty sure we've had this debate on this list before. The
conclusion was that the output of tcpdump is intended to be human
readable, not machine readable.

My understanding was that the text output of tcpdump is not
guaranteed/expected to maintain the same exact format over time so
people writing scripts based on it do so at their own risk.

People wanting to write scripts should use a machine readable format
such as XML, or dissect packets directly with a protocol analyser class
etc.

Stephen.
-- 
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] Failing to capture packets....

[Stephen Donnelly]

Most commercial NICs only deliver well formed Ethernet packets (valid
size, correct FCS) to the operating system, and also generally strip out
Ethernet control frames such as 'pause' indications.

Because of this Libpcap would not have any data to deliver to the
capturing application.

You may have to get hold of low level Ethernet test equipment in order
to determine exactly what is there.

Stephen.

On Tue, 2007-01-23 at 12:11 -0600, Paul Armor wrote:
> Hi,
> after Guy's last email where he states:
> "Tcpdump supports capturing *all* network traffic;"
> 
> I feel compelled to again ask if anyone can offer any suggestions on how I 
> can achieve my goal of capturing the data I'm seeing on my ethernet 
> segment...  To what I wrote earlier I'd also add that the Force10 switch 
> that's upstream of these systems records a lot of "throttle" packets on 
> that interface.
> 
> Any help would be appreciated!
> 
> Thanks!
> Paul
> 
> 
> On Fri, 19 Jan 2007, Paul Armor wrote:
> 
> > Hi,
> > I've got a problem that's strange on various levels and using tcpdump isn't 
> > as helpful as I'd have hoped.  Can anyone offer suggestions on how to 
> > capture/interpret my bad data on the wire?  I'm trying to capture from any 
> > of 
> > a few other machines with Broadcomm chips, and am wondering if there's a 
> > limitation to hardware/driver that prevents tcpdump/libpcap from "seeing" 
> > that data?
> >
> > Generally speaking, I'm trying to capture data on the wire that's coming 
> > from 
> > a computer that's crashed.  That sounds simple enough...
> >
> > BUT, here's the rub... the driver and thus tcpdump/ethereal don't recognize 
> > any "packets", but there's data spraying on the wire, so I don't think 
> > they're at all properly formed ethernet packets.  Here's some interesting 
> > ifconfig (linux 2.6) output:
> >
> > eth0  Link encap:Ethernet  HWaddr 00:14:22:D1:16:B1
> >  RX packets:2491 errors:0 dropped:0 overruns:0 frame:21
> >  TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
> >  collisions:0 txqueuelen:1000
> >  RX bytes:2910464 (2.7 MiB)  TX bytes:492 (492.0 b)
> >
> >
> > eth0  Link encap:Ethernet  HWaddr 00:14:22:D1:16:B1
> >  RX packets:2491 errors:0 dropped:0 overruns:0 frame:21
> >  TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
> >  collisions:0 txqueuelen:1000
> >  RX bytes:2940992 (2.8 MiB)  TX bytes:492 (492.0 b)
> >
> > Note how RX packets does NOT increase, while RX bytes does.  These two 
> > ifconfig's were run about 1 sec apart from another machine attached via 
> > Xover.  I didn't pay attention to the occurance of the "frame" pkts...
> >
> > How this happens is that I've got a large number of machines running a 
> > Fedora 
> > install, and certain users jobs are able to tickle a problem with 
> > memory/memory-controller/CPU (everybody's blaming everybody else), which 
> > sometimes (~60% of the time) causes a crashed machine (a Machine Check 
> > Exception) to start spraying the network with crap.  This crap causes a 
> > broadcast/multicast cache/buffer to overflow on a big Force 10 switch, 
> > which 
> > causes other machines to "drop off the network" (as ARP fails, etc).
> >
> > I suspect a problem with BIOS on motherboard or firmware on embedded 
> > ethernet 
> > controller (Broadcomm (BCM95704A6) rev 2100 PHY(5704))... and am looking 
> > for 
> > evidence.
> >
> > ANY help/suggestions would be greatly appreciated!
> >
> > Thanks!
> > Paul
> >
> >
> k
> -
> This is the tcpdump-workers list.
> Visit https://cod.sandelman.ca/ to unsubscribe.
-- 
---
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] Request for a new DLT for MTP2 with FCS

[Stephen Donnelly]

On Tue, 2007-02-06 at 18:18 +0100, [EMAIL PROTECTED]
wrote:
>   Hello,

> I need a new datalink value (LINKTYPE_MTP2_FCS) to manage MTP2 frame with
> Checksum.
> (See http://www.wireshark.org/lists/wireshark-dev/200702/msg00206.html )
> 
> Once I will have the value, I will provide the patch for libpcap.

Endace DAG cards (for instance) can capture MTP2 with or without FCS,
does your proposal mean that MTP2 without FCS should only use
LINKTYPE_MTP2 and MTP2 with FCS should only use LINKTYPE_MTP2_FCS?

What about other linktypes, such as Ethernet?

The issue of whether captured packets should include FCS and how this is
reported in libpcap seems to be a more generic question. I'm not sure
that duplicating all existing DLTs by adding _FCS (or tripling with
_NOFCS as well) at the end would be accepted by the libpcap community. 

If people are generally happy with adding LINKTYPE_MTP2_FCS as a special
case I have no problem, and Endace can support both linktypes.

Stephen.
-- 
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] Request for a new DLT for MTP2 with FCS

[Stephen Donnelly]

As a question to the community in general, is it true that the link
layer checksum is not normally included in libpcap records, and that
only frames with valid L2 checksums are normally captured?

If so it seems clear that DLT_MTP2 (and other) link types should not
have the L2 checksum present appended to their records. Varying from
this practice would cause problems for applications which do not expect
the extra bytes.

If there is a special case for MTP2 where capturing the L2 checksum is
desirable, then I agree that it makes sense to add a new DLT value to
distinguish this.

It would only be necessary to add *_FCS DLTs where people specifically
request the ability to capture the checksum. This would keep the number
of new DLTs required to the minimum.

Regards,
Stephen

On Thu, 2007-02-08 at 19:44 +0100, [EMAIL PROTECTED]
wrote:
> I agree with you, the problem I am speaking about, seems to be a common
> problem.
> It is very difficult to differentiate a frame with FCS (Frame Check
> Sequence) and a frame without, if we are just looking at the linktype.
> But, with the current libpcap, I do not see other solutions, than using
> different linktypes.
> 
> Is it really a problem to create new linktypes, just for such purpose ?
> (I understood that the linktypes are coded on 4 bytes )
> 
> Regards
> Florent

-- 
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] Request for a new DLT for MTP2 with FCS

[Stephen Donnelly]

It seems that if it is worth making the change, it is also worth using a
couple of bits to indicate whether a 16 or 32-bit CRC/FCS is present as
Guy suggested. This could then be used on linktypes such as PPP_SERIAL
which can have either length.

Stephen.

On Mon, 2007-02-19 at 19:59 +0100, [EMAIL PROTECTED]
wrote:
>   Hello Guy,
> 
> 
> I have made the patch, so the 2 last bytes of the network datalink are now
> used as an extension of the linktype
> We can now consider the network datalink (32 bits) as a link type (16bits)
> and an extension (16 bits).
> On the extension, I just indicate that the FCS are present with the last
> bit set to TRUE.
> 
> For the patch, I did add a mask to get only the 2 lower bytes for the
> linktype. I don't know if this will not lead to incompatibilities ?
> You said, NetBSD uses the 16 upper bits, but in libpcap code, I have seen
> any specific treatment.
> Did I missed something. Maybe I  should  use the mask only for a restricted
> list of DLT to be sure I will not create new problems ?
> 
> (See attached file: pcap_linktype_ext.zip)
> 
> I have made the patch for wireshark too, and there is a lot of impact in
> the structures, like data frame or capture_file.
> But it works, the FCS are integrated in the pcap file without any
> configuration from the end users, and the FCS are correctly decoded.
> 
> Best regards
> Florent
> 
> 
> 
>   
>   
>   Guy Harris  
>   
>   <[EMAIL PROTECTED]>   To:  
> tcpdump-workers@lists.tcpdump.org   
>   Sent by: cc:
>   
>   [EMAIL PROTECTED] Subject: Re: 
> [tcpdump-workers] Request for a new DLT for MTP2 with FCS   
>   tcpdump.org  (repost)   
>   
>   
>   
>   
>   
>   19/02/2007 05:58
>   
>   Please respond to   
>   
>   tcpdump-workers 
>   
>   
>   
> 
> 
> 
> 
> [EMAIL PROTECTED] wrote:
> 
> > As requested few days before, I would like to use a new DLT to
> distinguish
> > between MTP2 without FCS (the current DLT_MTP2) and MTP2 with FCS.
> 
> Or perhaps the link type value in the file header should be interpreted
> as having bitfields, with the lower 16 bits being the link layer type,
> and an indication of whether there's an FCS present being somewhere in
> the upper 16 bits.
> 
> NetBSD already uses the upper 16 bits for its own purpose - if the upper
> 16 bits are 0x0224, the lower 16 bits are a NetBSD address family value.
>   (Given that AF_INET6, for example, has at least 3 different values on
> various BSD-flavored OSes, 0x0224 should be treated as NetBSD-specific,
> with other values used for other OSes.)
> 
> We could, for example, use the uppermost nibble as an FCS length
> indication, with the bit below it being an indication of whether the FCS
> length is known or not.  That doesn't touch any of the bits in 0x0224.
> 
> For all current DLT_ values, the bit would be clear, so the FCS length
> isn't known; that's the case for Ethernet, as not only is it not known
> whether any given DLT_EN10MB capture has FCSes in the packets or not
> (some do, some don't), it's not even known which *packets* in a capture
> that does have FCSes (packets sent by the machine doing the capture
> don't, but there's not a per-packet way of indicating that).
> 
> I think it would be possible to make this work with pcap-NG as well.
> 
> This has the advantage that "what is the link-layer header?" and "do
> frames have FCSes?" are separa

[tcpdump-workers] [PATCH] DAG card support update

[Stephen Donnelly]

This patch introduces support for the DAG ERF type
TYPE_COLOR_MC_HDLC_POS.

The patch also allows appropriate DAG cards (DAG 3.7T, DAG 7.1S) to
optionally produce DLT_MTP2_WITH_PHDR (139) traces when capturing from
channelised HDLC links, as an alternative to DLT_MTP2 (140). When using
the new DLT, the 'DAG channel' is recorded in the pcap record pseudo
header as the 'link_number'.

Basic BPF filtering support for DLT_MTP2_WITH_PHDR is also added.

DLT_MTP2_WITH_PHDR is currently handled by Wireshark (svn?), but not
tcpdump.

This patch was tested with assistance from Florent Drouin.

Regards,
Stephen.
-- 
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---
Index: gencode.c
===
RCS file: /tcpdump/master/libpcap/gencode.c,v
retrieving revision 1.283
diff -u -r1.283 gencode.c
--- gencode.c	14 Jun 2007 20:55:44 -	1.283
+++ gencode.c	20 Jun 2007 22:39:00 -
@@ -1267,6 +1267,17 @@
 		off_nl_nosnap = -1;
 		return;
 
+	case DLT_MTP2_WITH_PHDR:
+		off_li = 6;
+		off_sio = 7;
+		off_opc = 8;
+		off_dpc = 8;
+		off_sls = 11;
+		off_linktype = -1;
+		off_nl = -1;
+		off_nl_nosnap = -1;
+		return;
+
 #ifdef DLT_PFSYNC
 	case DLT_PFSYNC:
 		off_linktype = -1;
@@ -7010,14 +7021,16 @@
 	switch (type) {
 
 	case M_FISU:
-		if (linktype != DLT_MTP2)
+		if ( (linktype != DLT_MTP2) &&
+		 (linktype != DLT_MTP2_WITH_PHDR) )
 			bpf_error("'fisu' supported only on MTP2");
 		/* gen_ncmp(offrel, offset, size, mask, jtype, reverse, value) */
 		b0 = gen_ncmp(OR_PACKET, off_li, BPF_B, 0x3f, BPF_JEQ, 0, 0);
 		break;
 
 	case M_LSSU:
-		if (linktype != DLT_MTP2)
+		if ( (linktype != DLT_MTP2) &&
+		 (linktype != DLT_MTP2_WITH_PHDR) )
 			bpf_error("'lssu' supported only on MTP2");
 		b0 = gen_ncmp(OR_PACKET, off_li, BPF_B, 0x3f, BPF_JGT, 1, 2);
 		b1 = gen_ncmp(OR_PACKET, off_li, BPF_B, 0x3f, BPF_JGT, 0, 0);
@@ -7025,7 +7038,8 @@
 		break;
 
 	case M_MSU:
-		if (linktype != DLT_MTP2)
+		if ( (linktype != DLT_MTP2) &&
+		 (linktype != DLT_MTP2_WITH_PHDR) )
 			bpf_error("'msu' supported only on MTP2");
 		b0 = gen_ncmp(OR_PACKET, off_li, BPF_B, 0x3f, BPF_JGT, 0, 2);
 		break;
Index: pcap-dag.c
===
RCS file: /tcpdump/master/libpcap/pcap-dag.c,v
retrieving revision 1.28
diff -u -r1.28 pcap-dag.c
--- pcap-dag.c	1 Feb 2007 02:58:39 -	1.28
+++ pcap-dag.c	20 Jun 2007 22:39:00 -
@@ -49,6 +49,18 @@
 #define ATM_CELL_SIZE		52
 #define ATM_HDR_SIZE		4
 
+/*
+ * A header containing additional MTP information.
+ */
+#define MTP2_SENT_OFFSET		0	/* 1 byte */
+#define MTP2_ANNEX_A_USED_OFFSET	1	/* 1 byte */
+#define MTP2_LINK_NUMBER_OFFSET		2	/* 2 bytes */
+#define MTP2_HDR_LEN			4	/* length of the header */
+
+#define MTP2_ANNEX_A_NOT_USED  0
+#define MTP2_ANNEX_A_USED  1
+#define MTP2_ANNEX_A_USED_UNKNOWN  2
+
 /* SunATM pseudo header */
 struct sunatm_hdr {
 	unsigned char	flags;		/* destination and traffic type */
@@ -78,6 +90,10 @@
 #define dag_platform_finddevs pcap_platform_finddevs
 #endif /* DAG_ONLY */
 
+#define MAX_DAG_PACKET 65536
+
+static unsigned char TempPkt[MAX_DAG_PACKET];
+
 static int dag_setfilter(pcap_t *p, struct bpf_program *fp);
 static int dag_stats(pcap_t *p, struct pcap_stat *ps);
 static int dag_set_datalink(pcap_t *p, int dlt);
@@ -340,6 +356,9 @@
 caplen = packet_len;
 			}
 			break;
+#ifdef TYPE_COLOR_MC_HDLC_POS
+		case TYPE_COLOR_MC_HDLC_POS:
+#endif
 #ifdef TYPE_MC_HDLC
 		case TYPE_MC_HDLC:
 			packet_len = ntohs(header->wlen);
@@ -348,7 +367,20 @@
 			if (caplen > packet_len) {
 caplen = packet_len;
 			}
+			/* jump the MC_HDLC_HEADER */
 			dp += 4;
+			if (p->linktype == DLT_MTP2_WITH_PHDR) {
+/* Add the MTP2 Pseudo Header */
+caplen += MTP2_HDR_LEN;
+packet_len += MTP2_HDR_LEN;
+
+TempPkt[MTP2_SENT_OFFSET] = 0;
+TempPkt[MTP2_ANNEX_A_USED_OFFSET] = MTP2_ANNEX_A_USED_UNKNOWN;
+*(TempPkt+MTP2_LINK_NUMBER_OFFSET) = ((header->rec.mc_hdlc.mc_header>>16)&0x01);
+*(TempPkt+MTP2_LINK_NUMBER_OFFSET+1) = ((header->rec.mc_hdlc.mc_header>>24)&0xff);
+memcpy(TempPkt+MTP2_HDR_LEN, dp, caplen);
+dp = TempPkt;
+			}
 			break;
 #endif
 		default:
@@ -383,6 +415,10 @@
 		case TYPE_DSM_COLOR_ETH:
 			break;
 #endif
+#ifdef TYPE_COLOR_MC_HDLC_POS
+		case TYPE_COLOR_MC_HDLC_POS:
+			break;
+#endif
 
 		default:
 			if (header->lctr) {
@@ -936,6 +972,9 @@
 p->linktype = DLT_ATM_RFC1483;
 			break;
 
+#ifdef TYPE_COLOR_MC_HDLC_POS
+		case TYPE_COLOR_MC_HDLC_POS:
+#endif
 #ifdef TYPE_MC

Re: [tcpdump-workers] Packet capture performance comparison of

[Stephen Donnelly]

On Wed, 2007-06-27 at 22:00 +, Jefferson Ogata wrote:
> Greetings.
> 
> I've read Fabian Schneider's thesis "Performance evaluation of packet
> capturing systems for high-speed networks", which compares capture
> performance under variable testing and generally finds that dual-core
> Opterons perform somewhat better under heavy capture load than dual-core
> Xeons. But now that quad-core Xeons are available, I'm curious whether
> anyone has measured capture improvement using four cores. I should
> expect four cores to do better, but I'd be interested in any empirical
> results to that effect. I'm wondering, for example, how close a box with
> a couple of dual-port PCIe Gb NICs (Endace or nPulse) and dual quad-core
> processors could come to 4Gb/s aggregate capture speed, while writing
> some packets to disk. Has anyone out there put together such a box and
> come up with some performance statistics?

Hopefully this won't be considered too commercial a response, but I
think it's worth replying to the list so that the response is available
as well as the question.

The DAG 4.5G4 card is a 4-port Gigabit Ethernet Network Measurement
Interface Card that is specifically designed to capture packets from
four GE interfaces at up to 100% line rate, regardless of the packet
size, e.g. up to 5,952,381 pps for 64-byte packets.

The capture process is DMA based and uses very little CPU time, so even
a single core is sufficient.

The drivers allow a large amount of host memory to be used for
buffering, but capture to disk performance is ultimately dependent on
the storage subsystem. Provided the average sustained disk write
performance is sufficient, 4Gbps can be captured to disk.

Endace also offers disk capture appliances which provide this level of
performance.

Unfortunately I'm not aware of any recent independent test publications.

Regards,
Stephen.
-- 
---
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] Packet capture performance comparison of

[Stephen Donnelly]

On Thu, 2007-06-28 at 03:09 +, Jefferson Ogata wrote:
> Stephen Donnelly wrote:
> > On Wed, 2007-06-27 at 22:00 +, Jefferson Ogata wrote:
> >> some packets to disk. Has anyone out there put together such a box and
> >> come up with some performance statistics?
> [snip]
> > Endace also offers disk capture appliances which provide this level of
> > performance.
> > 
> > Unfortunately I'm not aware of any recent independent test publications.
> 
> Hmm. I wonder if that's because your company requires signing an NDA for
> getting eval gear. :^/

I can't comment on that, although I think it is fairly common practice.

There have been a few university papers published, as you noted. Here is
another thesis "On the Quality of Computer Network Measurements":

http://www.bth.se/fou/forskinfo.nsf/6753b78eb2944e0ac1256608004f0535/6f935ee3049f95d7c125709800464cac?OpenDocument

Regards,
Stephen.
-- 
---
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

[tcpdump-workers] DLT assignment request

[Stephen Donnelly]

Libpcap supports live capture from Endace DAG cards from wide variety of
network link types. For example a capture natively using the Endace ERF
TYPE_HDLC_POS type can select DLT_CHDLC, DLT_PPP_SERIAL, DLT_FRELAY,
DLT_MTP2, or DLT_MTP2_WITH_PHDR depending on the nature of the actual
link.

Endace also has two DLTs defined, DLT_ERF_ETH and DLT_ERF_POS.
DLT_ERF_POS encapsulates a complete ERF TYPE_POS_HDLC record, while
DLT_ERF_ETH encapsulates a complete ERF TYPE_ETH record. These types
were only experimental and are not used in any production code.

Florent Drouin from Alcatel-Lucent has been working on improving the ERF
support in Wireshark. As part of this work we would like to request a
new DLT (DLT_ERF) which would encapsulate a single ERF record of any ERF
type. DLT_ERF would be available in addition to the existing DLT
choices. Forthcoming patches for Wireshark will allow it to completely
decode this type, including all of the ERF pseudoheader information.

As the DLT_ERF could encapsulate a variety of link types we would not
support BPF filtering of DLT_ERF records, at least initially. Wireshark
'display filters' will be able to act on this type and when reading
directly from ERF format files.

The only alternative I can see would be assigning new DLTs on a 1:1
basis with ERF types, however there are already 19 ERF types defined and
I feel this would unnecessarily consume/pollute the DLT namespace.

Comments, questions, objections welcome.

Regards,
Stephen.
-- 
---
    Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

[tcpdump-workers] DLT assignment request

[Stephen Donnelly]

I have sent this to the tcpdump-workers list but there is no response
yet so I will copy it here as it impacts Wireshark also.

Libpcap supports live capture from Endace DAG cards from wide variety of
network link types. For example a capture natively using the Endace ERF
TYPE_HDLC_POS type can select DLT_CHDLC, DLT_PPP_SERIAL, DLT_FRELAY,
DLT_MTP2, or DLT_MTP2_WITH_PHDR depending on the nature of the actual
link.

Florent Drouin from Alcatel-Lucent has been working on improving the ERF
support in Wireshark. As part of this work we would like to request a
new DLT (DLT_ERF) which would encapsulate a single ERF record of any ERF
type. DLT_ERF would be available in addition to the existing DLT
choices. Forthcoming patches for Wireshark will allow it to completely
decode this type, including all of the ERF pseudoheader information.

As the DLT_ERF could encapsulate a variety of link types we would not
support BPF filtering of DLT_ERF records, at least initially. Wireshark
'display filters' will be able to act on this pcap type and also when
reading directly from ERF format files.

The only alternative I can see would be assigning new DLTs on a 1:1
basis with ERF types, however there are already 19 ERF types defined and
I feel this would unnecessarily consume/pollute the libpcap DLT
namespace.

Comments, questions, objections welcome.

Regards,
Stephen.
-- 
---
    Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] DLT assignment request

[Stephen Donnelly]

On Tue, 2007-08-07 at 16:55 -0700, Guy Harris wrote:
> On Jul 25, 2007, at 1:57 PM, Stephen Donnelly wrote:
> 
> > Florent Drouin from Alcatel-Lucent has been working on improving the  
> > ERF
> > support in Wireshark. As part of this work we would like to request a
> > new DLT (DLT_ERF) which would encapsulate a single ERF record of any  
> > ERF
> > type. DLT_ERF would be available in addition to the existing DLT
> > choices. Forthcoming patches for Wireshark will allow it to completely
> > decode this type, including all of the ERF pseudoheader information.
> 
> I've assigned 197 for DLT_ERF.

Thanks Guy.

Florent is currently on leave but when he returns we will submit patches
using the new DLT assignment.

Regards,
Stephen.
-- 
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] Endace DAG card

[Stephen Donnelly]

On Wed, 2007-09-26 at 10:09 +0200, Michele Sciuto wrote:
> Hi,
> I'm using a DAG 4.3GE card to capture traffic and I just have to clarify 
> some points.
> 
> Since I need the acquired data in libpcap format, I was wondering if it 
> would be better to use dagsnap followed by dagconvert or, alternatively, 
> tcpdump with the dag patch. Does somebody know which is the best choice, 
> considering performance?

Generally the highest possible performance writing to disk would be to
use dagsnap with the -j option capturing in ERF format.

>From an informational viewpoint, you are better to process in ERF format
directly where possible, as the conversion to pcap format loses
timestamp precision and some packet metadata.

The upcoming DLT_ERF works around the information loss issue but still
imposes memory, bandwidth and cpu overheads compared to using ERF
directly.

If you do need to process in pcap format, you can either capture in ERF
format and post-convert (dagsnap then dagconvert or editcap), or capture
directly with dagconvert (dagconvert -d), or use libpcap based software
such as tcpdump or tshark.

> And again, so far I can see the card as /dev/dag0 and I just used 
> dagsnap to capture data. Somehow, is it possible to see the dag 
> interface with ifconfig (I mean configuring it in 
> /etc/network/interfaces), in order to use netstat to check the counters?

No, as the DAG is not presented as a network interface to Linux. dagfour
or dagconfig can be used, or you can access the statistics via the DAG
configuration and status API from your own software.

Regards,
Stephen.
-- 
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

[tcpdump-workers] [PATCH] Recent checkin breaks build

[Stephen Donnelly]

Unmatched parenthesis.

Index: pcap/pcap.h
===
RCS file: /tcpdump/master/libpcap/pcap/pcap.h,v
retrieving revision 1.3
diff -u -r1.3 pcap.h
--- pcap/pcap.h 29 Sep 2007 19:33:29 -  1.3
+++ pcap/pcap.h 4 Oct 2007 20:53:48 -
@@ -128,7 +128,7 @@
  */
 #define LT_FCS_LENGTH_PRESENT(x)   ((x) & 0x0400)
 #define LT_FCS_LENGTH(x)   (((x) & 0xF000) >> 28)
-#define LT_FCS_DATALINK_EXT(x) (((x) & 0xF) << 28) |
0x0400)
+#define LT_FCS_DATALINK_EXT(x) x) & 0xF) << 28) |
0x0400)

 typedef enum {
PCAP_D_INOUT = 0,

-- 
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

[tcpdump-workers] [PATCH] dag updates

[Stephen Donnelly]

The attached patch includes:

  * Improved error checking in dag_read().
  * More efficient dag_platform_finddevs().
  * Support for new DAG API function dag_get_stream_erf_types().
  * Fix to pcap/pcap.h mismatched parenthesis.

As changes are made to configure.in, configure will also need to be
regenerated using the preferred autoconf version.

Stephen.
-- 
---
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---
? pcap-dag-vm.c
Index: config.h.in
===
RCS file: /tcpdump/master/libpcap/config.h.in,v
retrieving revision 1.29
diff -u -r1.29 config.h.in
--- config.h.in	12 Sep 2007 19:09:50 -	1.29
+++ config.h.in	4 Oct 2007 22:00:39 -
@@ -19,6 +19,9 @@
 /* define if you have dag_get_erf_types() */
 #undef HAVE_DAG_GET_ERF_TYPES
 
+/* define if you have dag_get_stream_erf_types() */
+#undef HAVE_DAG_GET_STREAM_ERF_TYPES
+
 /* define if you have streams capable DAG API */
 #undef HAVE_DAG_STREAMS_API
 
Index: configure.in
===
RCS file: /tcpdump/master/libpcap/configure.in,v
retrieving revision 1.137
diff -u -r1.137 configure.in
--- configure.in	12 Sep 2007 19:09:50 -	1.137
+++ configure.in	4 Oct 2007 22:00:39 -
@@ -592,8 +592,9 @@
 	LDFLAGS="-L$dag_lib_dir"
 	AC_CHECK_LIB([dag], [dag_attach_stream], [dag_version="2.5.x"], [dag_version="2.4.x"])
 	AC_CHECK_LIB([dag],[dag_get_erf_types], [
-		AC_DEFINE(HAVE_DAG_GET_ERF_TYPES, 1, [define if you have dag_get_erf_types()])]
-	)
+		AC_DEFINE(HAVE_DAG_GET_ERF_TYPES, 1, [define if you have dag_get_erf_types()])])
+	AC_CHECK_LIB([dag],[dag_get_stream_erf_types], [
+		AC_DEFINE(HAVE_DAG_GET_STREAM_ERF_TYPES, 1, [define if you have dag_get_stream_erf_types()])])
 	LDFLAGS=$saved_ldflags
 
 	if test "$dag_version" = 2.5.x; then
Index: pcap-dag.c
===
RCS file: /tcpdump/master/libpcap/pcap-dag.c,v
retrieving revision 1.30
diff -u -r1.30 pcap-dag.c
--- pcap-dag.c	29 Sep 2007 19:33:29 -	1.30
+++ pcap-dag.c	4 Oct 2007 22:00:40 -
@@ -220,7 +220,9 @@
 		 * If non-block is specified it will return immediately. The user
 		 * is then responsible for efficiency.
 		 */
-		p->md.dag_mem_top = dag_advance_stream(p->fd, p->md.dag_stream, &(p->md.dag_mem_bottom));
+		if ( NULL == (p->md.dag_mem_top = dag_advance_stream(p->fd, p->md.dag_stream, &(p->md.dag_mem_bottom))) ) {
+		 return -1;
+		}
 #else
 		/* dag_offset does not support timeouts */
 		p->md.dag_mem_top = dag_offset(p->fd, &(p->md.dag_mem_bottom), flags);
@@ -783,10 +785,13 @@
 }
 
 /*
- * Simply submit all possible dag names as candidates.
- * pcap_add_if() internally tests each candidate with pcap_open_live(),
- * so any non-existent devices are dropped.
- * For 2.5 try all rx stream names as well.
+ * Previously we just generated a list of all possible names and let
+ * pcap_add_if() attempt to open each one, but with streams this adds up
+ * to 81 possibilities which is inefficient.
+ *
+ * Since we know more about the devices we can prune the tree here.
+ * pcap_add_if() will still retest each device but the total number of
+ * open attempts will still be much less than the naive approach.
  */
 int
 dag_platform_finddevs(pcap_if_t **devlistp, char *errbuf)
@@ -794,30 +799,46 @@
 	char name[12];	/* XXX - pick a size */
 	int ret = 0;
 	int c;
+	char dagname[DAGNAME_BUFSIZE];
+	int dagstream;
+	int dagfd;
 
 	/* Try all the DAGs 0-9 */
 	for (c = 0; c < 9; c++) {
 		snprintf(name, 12, "dag%d", c);
-		if (pcap_add_if(devlistp, name, 0, NULL, errbuf) == -1) {
-			/*
-			 * Failure.
-			 */
-			ret = -1;
-		}
-#ifdef HAVE_DAG_STREAMS_API
+		if (-1 == dag_parse_name(name, dagname, DAGNAME_BUFSIZE, &dagstream))
 		{
-			int stream;
-			for(stream=0;stream<16;stream+=2) {
-snprintf(name,  10, "dag%d:%d", c, stream);
-if (pcap_add_if(devlistp, name, 0, NULL, errbuf) == -1) {
-	/*
-	 * Failure.
-	 */
-	ret = -1;
-}
-			}
+			return -1;
 		}
+		if ( (dagfd = dag_open(dagname)) >= 0 ) {
+			if (pcap_add_if(devlistp, name, 0, NULL, errbuf) == -1) {
+/*
+ * Failure.
+ */
+ret = -1;
+			}
+#ifdef HAVE_DAG_STREAMS_API
+			{
+int stream, rxstreams, found=0;
+rxstreams = dag_rx_get_stream_count(dagfd);
+for(stream=0;stream<16;stream+=2) {
+	if (0 == dag_attach_stream(dagfd, stream, 0, 0)) {
+		dag_detach_stream(dagfd, stream);
+
+		snprintf(name,  10, "dag%d:%d", c, stream);
+		if (pcap_add_if(devlistp, nam

Re: [tcpdump-workers] Creation of libpcap 1.0 and tcpdump

[Stephen Donnelly]

On Mon, 2007-11-05 at 20:47 -0500, Ken Bantoft wrote:

> I'll prep a 'release candidate' for 4.0/1.0 in the next few days.  I  
> know we normally don't do 'betas' for either, but given the large  
> amount of recent changes, I think it's a safe idea (and it avoids  
> 4.0.1. and 1.0.1 48hours after release!)

A release candidate sounds like a good idea. Could easily give it a week
or two to settle before finalising it.

Stephen
-- 
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] NIC / driver performance with libpcap

[Stephen Donnelly]

On Wed, 2008-01-09 at 17:12 +0100, Fabian Schneider wrote:
> Hi Andy,

> > The two metrics I looking at now are:
> >
> > - What throughput can I get before seeing dropped packets
> > - CPU usage
>   
> maybe you want to take a look at [1] where I have done exactly this for a
> special systems with Intel cards. If you want to read more background on  
> this have a look at [2] where the measurement setup is explained in more  
> detail.
>   
> We are trying to gather all things that enable high performance capturing
> on the following web page:
> 
> http://www.net.t-labs.tu-berlin.de/research/hppc/
> 
> Everybody is welcome to supply further and newer hint and tips on this 
> topic. 
> 
> [1] http://www.net.t-labs.tu-berlin.de/papers/SWF-PCCH10GEE-07.pdf
> [2] http://www.net.t-labs.tu-berlin.de/papers/S-PEPCSHN-05.pdf

Hi Fabian,

I was interested to see that your paper reported strikingly similar
results to my own investigations.

It seems that modern NIC cards on modern OSs can capture at 1Gbps,
provided the packet sizes are large (>1000B). As packet sizes decrease
the CPU load increases until the system drops packets. The limit
obviously varies with OS and system specifications, but seems to be in
the few hundred thousand packets per second range, leading to
significant packet drops for 'typical' packet size distributions even at
only 1Gbps.

If you'll excuse the commercial slant, you may be interested in my
whitepaper. Disclaimer: I work for Endace, a company that makes hardware
specialised for network packet capture.

http://www.endace.com/assets/docs/accelerated/DAGPacketCapturePerformance.pdf

Regards,
Stephen.
-- 
---
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] tcpdump problem with DAG card

[Stephen Donnelly]

On Wed, 2008-01-09 at 17:25 -0800, Guy Harris wrote:
> On Jan 9, 2008, at 3:37 PM, lei wei wrote:
> 
> > I'm actually trying to get Argus working with DAG but argus still  
> > can't read
> > anything from it.
> 
>  From a quick look at the source to Argus 2.0.6, it appears to be  
> assuming that you can do a select() on the result of pcap_fileno(),  
> which, as far as I know, is *NOT* the case for DAG devices; I don't  
> think the DAG driver supports select() or poll().  That might cause it  
> (and other applications using select() or poll() on pcap streams)  
> never to see any incoming packets, or to fail in other ways.
> 
> Newer versions of libpcap (including 0.9.x) have  
> pcap_get_selectable_fd(), which returns a file descriptor on which you  
> can do select(), if such a descriptor exists, or -1, if no such file  
> descriptor exists.

I agree. From ArgusGetPackets() in ArgusOutput.c it appears that Argus
is intended to operate over multiple pcap interfaces, but incorrectly
assumes that pcap descriptors are always selectable.

Because Argus does not check for selectable descriptors and work around
any non-selectable descriptors it is not possible to use Argus with DAG
cards without further modification.

Curiously under CYGWIN it does not assume selectable descriptors, but
apparently works with only one interface in this case. It may be
possible to use this as the basis for non-selectable descriptors in
general.

Stephen.
-- 
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] tcpdump problem with DAG card

[Stephen Donnelly]

On Thu, 2008-01-10 at 14:53 +1300, Stephen Donnelly wrote:
> On Wed, 2008-01-09 at 17:25 -0800, Guy Harris wrote:
> > On Jan 9, 2008, at 3:37 PM, lei wei wrote:
> > 
> > > I'm actually trying to get Argus working with DAG but argus still  
> > > can't read
> > > anything from it.
> > 
> >  From a quick look at the source to Argus 2.0.6, it appears to be  
> > assuming that you can do a select() on the result of pcap_fileno(),  
> > which, as far as I know, is *NOT* the case for DAG devices; I don't  
> > think the DAG driver supports select() or poll().  That might cause it  
> > (and other applications using select() or poll() on pcap streams)  
> > never to see any incoming packets, or to fail in other ways.
> > 
> > Newer versions of libpcap (including 0.9.x) have  
> > pcap_get_selectable_fd(), which returns a file descriptor on which you  
> > can do select(), if such a descriptor exists, or -1, if no such file  
> > descriptor exists.
> 
> I agree. From ArgusGetPackets() in ArgusOutput.c it appears that Argus
> is intended to operate over multiple pcap interfaces, but incorrectly
> assumes that pcap descriptors are always selectable.
> 
> Because Argus does not check for selectable descriptors and work around
> any non-selectable descriptors it is not possible to use Argus with DAG
> cards without further modification.
> 
> Curiously under CYGWIN it does not assume selectable descriptors, but
> apparently works with only one interface in this case. It may be
> possible to use this as the basis for non-selectable descriptors in
> general.

As a workaround, pretending to be CYGWIN gets Argus running. Changing
#if defined(CYGWIN) to #if 1 at line 1797 in
argus-3.0.0/argus/ArgusSource.c in order to use the non-select version
of the code seems to work okay, although Argus will only be able to read
from one interface.

Cross-posting to the Argus list in case someone wants to have a better
go at fixing this upstream.

Stephen.
-- 
---
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] Which versions of pcap files accept

[Stephen Donnelly]

On Tue, 2008-03-11 at 01:04 -0700, Guy Harris wrote:
> vcarela wrote:
> 
> > The problem is that if I capture with wireshark a trace from my eth0
> > connection and I save it as a "Wireshark/tcpdump/...-libpcap" file. Then
> > when I run the sniffer with this pcap trace the sniffer runs properly. 
> > But if I open a .erf trace from a DAG card with wireshark and I save it
> > as a "Wireshark/tcpdump/...-libpcap" when I run this trace in the
> > sniffer no packets are dispatched.
> 
> When read an ERF trace, save it with a recent build of Wireshark as a 
> libpcap-format file, and run a (slightly modified, so it compiles) 
> version of your program, it prints
> 
>   Error compilando el filtro 'ip'
> 
> without even trying to read the file.
> 
> Recent versions of Wireshark save ERF files as libpcap files with a 
> packet type of DLT_ERF, and the filter compiler in libpcap doesn't 
> support DLT_ERF.

I wonder if that is the best approach? On the plus side it avoids losing
information such as timestamp precision, but on the downside it is not
widely interoperable.

If the user's purpose in saving to libpcap format is to use the file
with another program then saving to DLT_ERF may not be useful.

When you save a capture in libpcap format Wireshark doesn't prompt you
for which DLT to use? How does it decide which DLT is appropriate?

Stephen
-- 
---
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

[tcpdump-workers] Patch to fix DAG support in HEAD

[Stephen Donnelly]

pcap-dag.c 1.37 doesn't compile after changes to support the new
'activate' model.

Small patch which should address the issues.

Stephen.
-- 
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---
? autom4te.cache
? libpcap.so.0.9-PRE-CVS
Index: pcap-dag.c
===
RCS file: /tcpdump/master/libpcap/pcap-dag.c,v
retrieving revision 1.37
diff -u -r1.37 pcap-dag.c
--- pcap-dag.c	4 Apr 2008 19:37:45 -	1.37
+++ pcap-dag.c	8 Apr 2008 01:05:58 -
@@ -560,8 +560,8 @@
 /*
  *  Get a handle for a live capture from the given DAG device.  Passing a NULL
  *  device will result in a failure.  The promisc flag is ignored because DAG
- *  cards are always promiscuous.  The to_ms parameter is also ignored as it is
- *  not supported in hardware.
+ *  cards are always promiscuous.  The to_ms parameter is used in setting the
+ *  API polling parameters.
  *  
  *  snaplen is now also ignored, until we get per-stream slen support. Set
  *  slen with approprite DAG tool BEFORE pcap_activate().
@@ -650,11 +650,11 @@
 	 */
 	mindata = 65536;
 
-	/* Obey to_ms if supplied. This is a good idea!
+	/* Obey md.timeout (was to_ms) if supplied. This is a good idea!
 	 * Recommend 10-100ms. Calls will time out even if no data arrived.
 	 */
-	maxwait.tv_sec = to_ms/1000;
-	maxwait.tv_usec = (to_ms%1000) * 1000;
+	maxwait.tv_sec = handle->md.timeout/1000;
+	maxwait.tv_usec = (handle->md.timeout%1000) * 1000;
 
 	if (dag_set_stream_poll(handle->fd, handle->md.dag_stream,
 mindata, &maxwait, &poll) < 0) {
@@ -734,7 +734,7 @@
 			if ((n = atoi(s)) == 0 || n == 16 || n == 32) {
 handle->md.dag_fcs_bits = n;
 			} else {
-snprintf(ebuf, PCAP_ERRBUF_SIZE,
+snprintf(handle->errbuf, PCAP_ERRBUF_SIZE,
 	"pcap_activate %s: bad ERF_FCS_BITS value (%d) in environment\n", device, n);
 goto failstop;
 			}
@@ -790,7 +790,7 @@
 
 #ifdef HAVE_DAG_STREAMS_API 
 failstop:
-	if (dag_stop_stream(handle->fd, handle->md.dag_stream) < 0)
+	if (dag_stop_stream(handle->fd, handle->md.dag_stream) < 0) {
 		fprintf(stderr,"dag_stop_stream: %s\n", strerror(errno));
 	}
 	
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

[tcpdump-workers] tcpdump display/decode bug?

[Stephen Donnelly]

I recently came across some packets which tcpdump appears to display
incorrectly.

Is tcpdump incorrectly invoking some heuristic dissector, or is there
another reason?

$ tcpdump -n -r tcpdump-error.pcap
reading from file tcpdump-error.pcap, link-type EN10MB (Ethernet)
08:35:24.570337 vlan 506, p 0, IP 10.143.146.4.22966 > 10.36.62.45.7098:
UDP, length 311
08:35:24.570387 vlan 179, p 0, IP 85.254.4.128 > 223.117.196.0: at-#182
673
08:35:24.570393 vlan 506, p 0, IP 85.254.4.128 > 223.117.196.0: at-#182
673
08:35:24.570399 vlan 179, p 0, IP 10.143.146.4.31200 > 10.36.69.80.6988:
UDP, length 189

$ tshark -n -r tcpdump-error.pcap
  1   0.00 10.143.146.4 -> 10.36.62.45  UDP Source port: 22966
Destination port: 7098
  2   0.50 10.143.146.4 -> 10.36.53.122 UDP Source port: 8756
Destination port: 16622
  3   0.56 10.143.146.4 -> 10.36.53.122 UDP Source port: 8756
Destination port: 16622
  4   0.62 10.143.146.4 -> 10.36.69.80  UDP Source port: 31200
Destination port: 6988

$ tcpdump -V
tcpdump version 3.9.8
libpcap version 0.9.8

$ tshark -v
TShark 1.0.99 (SVN Rev 25740)

Compiled with GLib 2.16.3, with libpcap 0.9-PRE-CVS, with libz 1.2.3.3,
with
POSIX capabilities (Linux), with libpcre 7.4, without SMI, with ADNS,
without
Lua, with GnuTLS 2.0.4, with Gcrypt 1.2.4, with MIT Kerberos.

Running on Linux 2.6.24-12-generic, with libpcap version 0.9-PRE-CVS.

Built using gcc 4.2.3 (Ubuntu 4.2.3-2ubuntu7).


Any assistance appreciated.

Stephen.
-- 
-------
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] tcpdump display/decode bug?

[Stephen Donnelly]

On Wed, 2008-07-30 at 20:07 -0700, Guy Harris wrote:
> On Jul 30, 2008, at 2:12 PM, Stephen Donnelly wrote:
> 
> > I recently came across some packets which tcpdump appears to display
> > incorrectly.
> >
> > Is tcpdump incorrectly invoking some heuristic dissector, or is there
> > another reason?
> 
> I guess that's what I'd call it.
> 
> tcpdump assumes that packets to or from certain ports might be KIP- 
> encapsulated AppleTalk packets (KIP = "Kinetics IP"); see the tcpdump  
> man page (look for "KIP AppleTalk (DDP in UDP)"), or RFC 1243:
> 
>   4.7.  The Kinetics Internet Protocol Group
> 
>  The Kinetics Internet Protocol (KIP) is a protocol for encapsulating
>  and routing AppleTalk datagrams over an IP internet.  This name is
>  historical.  The KIP group manages the KIP routing protocol as well
>  as the routing tables generated by this protocol.
> 
> It uses a heuristic to check, but the heuristic is really weak (it  
> checks whether, if the payload were an AppleTalk LLAP packet, the type  
> would be DDP, so it checks one count 'em one byte in the entire  
> payload).

Okay, the explanation makes sense. We just had bad luck with our
packets looking like candidates for KIP.

Tcpdump doesn't have a way of configuring/disabling heuristic dissectors
like this, without hacking the code?

Stephen.
-- 
---
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] does "port 25" work?

[Stephen Donnelly]

On Thu, 2008-07-31 at 14:29 -0400, U. George wrote:
> Guy Harris wrote:
> > 
> > On Jul 31, 2008, at 10:48 AM, U. George wrote:
> > 
> >> why does adding the "PORT" conditional also modify the wild-card 
> >> aspects of "ethernet type"
> > 
> > To what "wild-card aspects of 'ethernet type'" are you referring?
> > 
> > If you say "port domain", that can only match TCP or UDP packets, which 
> > means it can only match IP packets, which means it *cannot* match 
> > arbitrary Ethernet types.
> > -
> > This is the tcpdump-workers list.
> > Visit https://cod.sandelman.ca/ to unsubscribe.
> > 
> > 
> if i say this:
>   tcpdump  -n -v -i eth1
> 
> i get a log of: ether type * and port *, ie the PPPoE data.
> 
> If i say
>   tcpdump  -n -v -i eth1 port domain
> 
> i get a filter of ether (type UDP or TCP) and port domain, and no PPPoE data

The filter "port domain" on an Ethernet interface (on my box) generates
a BPF filter that looks for Ethertype 0x86dd for IPv6 OR 0x0800 for
IPv4. It doesn't look for PPPoE, VLANs, GRE or anything else, because
you didn't specify that in your filter.

# tcpdump  -d -n -v -i eth0 port domain
(000) ldh  [12]
(001) jeq  #0x86dd  jt 2jf 10
(002) ldb  [20]
(003) jeq  #0x84jt 6jf 4
(004) jeq  #0x6 jt 6jf 5
(005) jeq  #0x11jt 6jf 23
(006) ldh  [54]
(007) jeq  #0x35jt 22   jf 8
(008) ldh  [56]
(009) jeq  #0x35jt 22   jf 23
(010) jeq  #0x800   jt 11   jf 23
(011) ldb  [23]
(012) jeq  #0x84jt 15   jf 13
(013) jeq  #0x6 jt 15   jf 14
(014) jeq  #0x11jt 15   jf 23
(015) ldh  [20]
(016) jset #0x1fff  jt 23   jf 17
(017) ldxb 4*([14]&0xf)
(018) ldh  [x + 14]
(019) jeq  #0x35jt 22   jf 20
(020) ldh  [x + 16]
(021) jeq  #0x35jt 22   jf 23
(022) ret  #96
(023) ret  #0

The filter "pppoes and port domain" generates a BPF filter which looks
for Ethertype 0x8864 for PPPOE traffic.

# tcpdump  -d -n -v -i eth0 pppoes and port domain
(000) ldh  [12]
(001) jeq  #0x8864  jt 2jf 25
(002) ldh  [20]
(003) jeq  #0x57jt 4jf 12
(004) ldb  [28]
(005) jeq  #0x84jt 8jf 6
(006) jeq  #0x6 jt 8jf 7
(007) jeq  #0x11jt 8jf 25
(008) ldh  [62]
(009) jeq  #0x35jt 24   jf 10
(010) ldh  [64]
(011) jeq  #0x35jt 24   jf 25
(012) jeq  #0x21jt 13   jf 25
(013) ldb  [31]
(014) jeq  #0x84jt 17   jf 15
(015) jeq  #0x6 jt 17   jf 16
(016) jeq  #0x11jt 17   jf 25
(017) ldh  [28]
(018) jset #0x1fff  jt 25   jf 19
(019) ldxb 4*([22]&0xf)
(020) ldh  [x + 22]
(021) jeq  #0x35jt 24   jf 22
(022) ldh  [x + 24]
(023) jeq  #0x35jt 24   jf 25
(024) ret  #96
(025) ret  #0

>From the man page of a recent tcpdump version:

pppoes True if the packet is a PPP-over-Ethernet Session packet
(Ethernet type 0x8864).  Note that the first  pppoes  keyword
encountered  in  expression changes the decoding offsets for the
remainder of expression on the assumption that the packet is a PPPoE
session packet.

# tcpdump --version
tcpdump version 3.9.8
libpcap version 0.9.8

Stephen
-- 
---
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] does "port 25" work?

[Stephen Donnelly]

On Thu, 2008-07-31 at 23:26 -0400, U. George wrote:
> > 
> > The filter "port domain" on an Ethernet interface (on my box) generates
> > a BPF filter that looks for Ethertype 0x86dd for IPv6 OR 0x0800 for
> > IPv4. It doesn't look for PPPoE, VLANs, GRE or anything else, because
> > you didn't specify that in your filter.
> > 
> Actually I didnt specify 0x86dd or 0x0800 either. I did specify device 
> eth1 AND i did specify port domain. I dont care for ethertype filtering 
> as it is not germane.
> If PPPoE has port settings, then PPPoE packets should be filtered also. 
> If VLANS, or GRE, or anything else has port designations, then that 
> should be included in the filtering.

I didn't say that you did; I told you what happens when you specify that
filter. That explains the behaviour that you saw, which is expected.

>  From a users point of view, if tcpdump can print the packet with out 
> any ethertype options, then one should also be able to compare/match 
> pieces  of the data stream without the use of or knowledge of ethertype 
> specifics. The only item of significance ( for me ) is "port domain" 
> from the specific interface. From my point of view, ethertype is wild, 
> ip is wild, protocol is wild, and everything else is wild - with the 
> exception of the port designation.
> Its just intuitive.

That may be true, but it isn't the way tcpdump works.

Perhaps you should try Wireshark, you may find its 'display filters'
more user friendly.

http://www.wireshark.org

Stephen.
-- 
---
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] tcpdump and wireshark

[Stephen Donnelly]

On Mon, 2008-09-22 at 18:18 +0400, Dmitry wrote:
> Yeah! You´re right!
> 
> Dumping packets via tcpdump to file, I can choose packet and cut out payload
> starting from 0x0042
> Therefore It could be done via dd utility and some scripting avoiding
> libpcap.
> 
> Via tcpflow I can dump sessions. That´s more convenient.
> 
> Thanks in advance!
> 
> It would be better to make tcpdump available dump payloads.

Simply concatenating packet payloads together to recover binary objects
is often insufficient. TCP packets may be lost, duplicated, or received
out of order. The TCP/IP stack can put the object back together, but
looking at the payloads in order will not always work.

If you are on a LAN where there is essentially no packet loss or
reordering you may be lucky, but you are less likely to be lucky on the
public Internet.

tcpflow understands TCP sessions and 'does the right thing' in order to
extract the original binary data. Putting the same functionality into
tcpdump would be duplication, and this is already handled by Wireshark
in any case.

Stephen.

> Dmitry
> 
> 
> On Mon, Sep 22, 2008 at 2:12 PM, <[EMAIL PROTECTED]> wrote:
> 
> >
> > > And now my question is:
> > > can tcpdump extract payloads from packets, or it just extracting headers?
> >
> > No, tcpdump by itself can't. But that's what tcpflow does.
> >
> >Regards,
> >
> >   Marco.
> > -
> > This is the tcpdump-workers list.
> > Visit https://cod.sandelman.ca/ to unsubscribe.
> >
> -
> This is the tcpdump-workers list.
> Visit https://cod.sandelman.ca/ to unsubscribe.
> 
-- 
---
Stephen Donnelly BCMS PhD   email: [EMAIL PROTECTED]
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] TCPDUMP 4.0.1rc1 and LIBPCAP 1.0.1rc1

[Stephen Donnelly]

The default install path is probably /usr/local, have you checked
if /usr/local/lib is in your library path?

See /etc/ld.so.conf.d/*.conf

Stephen.

On Sun, 2008-12-21 at 01:25 -0800, Gianluca Varenni wrote:
> I have some problems using the shared version of libpcap.
> 
> Environment: fresh installation of a Debian 4.0r5 x86
> 
> I compiled the 1.0.1rc1 sources with
> 
> make shared;make install;make install-shared
> 
> (BTW, you need to run both "install" and "install-shared" because the latter 
> doesn't install the include files)
> 
> After that I successfully compiled the following test app:
> 
> 
> #include "pcap.h"
> 
> int main()
> {
>  pcap_if_t *alldevs;
>  pcap_if_t *d;
>  char errbuf[PCAP_ERRBUF_SIZE+1];
> 
>  /* Retrieve the device list */
>  if(pcap_findalldevs(&alldevs, errbuf) == -1)
>  {
>   fprintf(stderr,"Error in pcap_findalldevs: %s\n", errbuf);
>   return -1;
>  }
> 
>  /* Free the device list */
>  pcap_freealldevs(alldevs);
> 
>  return 1;
> }
> 
> 
> with "gcc test.c -lpcap"
> 
> When I run it, I get
> 
> ./a.out: error while loading shared libraries: libpcap.so.1: cannot open 
> shared  object file: No such file or directory
> 
> I'm not an expert about linux shared objects, maybe I'm doing something 
> wrong.
> Is it my fault or something within the installation of the libpcap shared 
> object?
> 
> Thanks
> GV
> 
> 
> 
> 
> 
> 
> - Original Message - 
> From: "Ken Bantoft" 
> To: 
> Sent: Tuesday, December 02, 2008 12:14 PM
> Subject: [tcpdump-workers] TCPDUMP 4.0.1rc1 and LIBPCAP 1.0.1rc1 available 
> for testing
> 
> 
> >
> > Hi -workers,
> >
> > http://www.tcpdump.org/beta/  has the Release Candidates - they  contain 
> > everything including the dlpisubs.c patch from today.
> >
> > Please give them a test run if you can - provided no bugs, I'm  planning 
> > to release them for Monday, Dec 8th.
> >
> >
> > Ken
> > -
> > This is the tcpdump-workers list.
> > Visit https://cod.sandelman.ca/ to unsubscribe. 
> 
> -
> This is the tcpdump-workers list.
> Visit https://cod.sandelman.ca/ to unsubscribe.
-- 
---
Stephen Donnelly BCMS PhD   email: s...@endace.com
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] Hardware timestamp ?

[Stephen Donnelly]

On Mon, 2009-03-02 at 12:24 -0600, Pierre Karampournis wrote:
> Guy Harris wrote :
> >
> > On Feb 27, 2009, at 9:03 AM, Pierre KARAMPOURNIS wrote:
> >
> >> I worked on old Linux Kernel versions so I will try the latest ones 
> >> to see
> >> hardware timestamping. So now I have to search for Network cards 
> >> which can
> >> timestamp the packets with nanosecond resolution (Endace DAG cards can
> >> apparently do that)
> >
> > Yes, but they're not regular networking cards, so they won't use that 
> > kernel code path on Linux; they supply packets through their own API, 
> > and that API does provide nanosecond timestamps, which libpcap turns 
> > into microsecond timestamps.
> >
> > If you want a network card that supplies its own timestamps *and* 
> > functions as a regular network adapter rather than a capture-only 
> > adapter, it would need to have a Linux driver that time-stamped the 
> > skbuff containing the packet.  I don't know whether any cards of that 
> > sort exist, or whether they have Linux drivers that do that (if they 
> > have open-source Linux drivers that don't, you could perhaps modify 
> > them to do so).
> >
> > If, as you said, you're planning to capture traffic on a 
> > gigabit-or-faster network, you might *want* a capture-only NIC for 
> > that purpose, in which case the Endace cards would be sufficient.

> I only need to capture data so endace cards should do the job. I will 
> also modify libpcap to keep the original timestamps with nanosecond 
> resolution so I consider the problem "solved".
> I didn't know the Endace devices, I was not searching in that direction, 
> thank you for your help.

Endace DAG cards capture into our Extensible Record Format (ERF) format,
bypassing the kernel, network stack, and libpcap for efficiency. The ERF
format has a high resolution time stamp in units of 2^-32s, or an
ultimate resolution of ~233ps. The actual resolution depends on the link
type; for Gigabit Ethernet it would typically be 2^-26s (~15ns). The ERF
format also provides additional metadata per frame such as which capture
port the apcket was received on, packet loss tracking, packet errors
etc.

Wireshark for example reads ERF format files directly, so conversion via
libpcap is not necessary (and would be less efficient).

Libpcap 1.0 and trunk can perform 'live' captures from DAG cards while
bypassing the kernel. In this case Ethernet ERF records are converted to
libpcap DLT_EN10MB on the fly, which loses time stamp resolution
(libpcap is microsecond only) and other ERF metadata.

Alternatively libpcap can capture into DLT_ERF. This encapsulates the
ERF record within a libpcap record, hence preserving the original time
stamp and metadata. Wireshark can read this pcap DLT and will use the
ERF record information inside in preference to the pcap packet header
information.

Some DAG cards can also transmit packets at specific times. They use the
time stamp from the ERF record to determine when to transmit the packet
in hardware. This allows for very accurate capture and 'replay' of
network traffic. The inter-packet timing is preserved and regenerated
with high accuracy, typically orders of magnitude better than
software-only approaches.

Regards,
Stephen
-- 
---
Stephen Donnelly BCMS PhD   email: s...@endace.com
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 530 770
---

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

[tcpdump-workers] Pull request

[Stephen Donnelly]

git://github.com/sfd/libpcap.git

Updating Endace DAG ERF support.
--
---
Stephen Donnelly BCMS PhD   email: s...@endace.com
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] ATM raw format data-link level type code in

[Stephen Donnelly]

Miguel Casas-Sanchez wrote:

Hi all,

I have been messing around with data dumps into libpcap file format 
(major=2, minor=4) for testing my traffic generation scripts, and so far 
it went very good: I could create different streams, dump them into 
file(s), and parse them back using either windump/tcpdump/wireshark.


Now I'm generating ATM cells with AAL5 encapsulated traffic inside (as 
per RFC 2684, not RFC 1483 which is superseded) and cannot seem to find 
the appropriate DLT_ for "raw atm".


There are, in pcap/bpf.h, the following related DLT modes:

   #define DLT_ATM_RFC148311/* LLC-encapsulated ATM */
   #define DLT_ATM_CLIP   19/* Linux Classical-IP over ATM */
   #define DLT_SUNATM123/* Solaris+SunATM */

   DLT_ATM_RFC1483 expects the "packet data" in the dump file, to be 
stripped of the atm cell header, and most likely of the AAL5 epilog.


   DLT_ATM_CLIP expects the "packet data" in the dump file, to be only 
an IP packet, so basically is the same as the previous case.


   DLT_SUNATM expects another type of format for "packet data": 1 byte 
of traffic type indication, 1 byte VPI and 2-byte VCI, then the atm cell 
payload.



I would like to know if anyone implemented/had though about adding a 
DLT_ATM_RAW or similar, to dump raw ATM cells as they come from the 
network. If necessary/interesting I could add them myself.


You are right, I haven't seen direct support for ATM cells.

For reassembled frames we can convert into the above types, but for raw 
cells we would generally use our own format.


I would support adding a DLT_ATM_CELLS or similar.

There is always the question of what metadata per cell is appropriate. 
We capture a 4-byte cell header without the HCS. In some cases we have a 
'Physical Port ID' which would be useful.


Stephen.
--
-------
Stephen Donnelly BCMS PhD   email: s...@endace.com
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] Memory leak in libpcap (top of tree) and/or

[Stephen Donnelly]

Guy Harris wrote:

Spot the potential leaks:

$ egrep -l 'mmap\(' *.c
pcap-bpf.c
pcap-dag.c
pcap-linux.c
pcap-usb-linux.c

$ egrep -l 'munmap\(' *.c
pcap-bpf.c
pcap-linux.c

I suspect the close routines for Linux USB devices - and DAG devices - 
need to unmap any buffer they have.


I believe the current code to support DAG cards correctly cleans up and 
unmaps any memmapped space.


The dag_mmap() call is a legacy interface that is cleaned up inside 
dag_close(), so there no matching *munmap() call. DAG Streams API code 
uses dag_attach_stream() and dag_detach_stream() to handle 
mapping/unmapping.


Stephen.
--
-------
Stephen Donnelly BCMS PhD   email: s...@endace.com
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [PATCH] Re: [tcpdump-workers] Bug: Counting dropped packets in

[Stephen Donnelly]

Dustin Spicuzza wrote:


So after reading the libpcap and kernel source, I see that this is
actually how its supposed to work. But it *seems* like it would be quite
nice if we could grab the interface driver drop statistics as well as
drops resulting from the capture buffer filling up.

I'm looking for a way to query the kernel about this number -- should I
bother submitting a patch if I can get it, or is the current behavior
more desired?


It appears that the only way to get this number is by looking at
/proc/stats/dev, so I added something that can parse it for the correct
interface, and get the drop stats from there. Since those are cumulative
over the boot time of the system, it stores a number and does
incremental updates of the number. It doesn't do this when not in
promiscuous mode however, since that doesn't make any sense.

It works for me, let me know what you think.


The current 'drop' count in libpcap is not intuitive, and frequently 
arguably undercounts since it does not include 'rx buffer overflow' and 
similar interface/OS specific packet loss. OTOH, the documentation is 
quite clear about what it does count I think.


I would not recommend changing the definition or Linux implementation at 
this point. It would change the behaviour for existing applications and 
would only cause further confusion.


I think the current libpcap statistics interface is not very useful 
because it is underspecified and varies too much between platforms. 
Hopefully for pcap-ng (the portable packet capture library, not the file 
format) we can get a better live statistics interface which is more 
tightly defined and can conserve meaning across platforms.


E.g. we could track at least:

Packets received per interface
Packets dropped in hardware (due to packet errors)
Packets dropped in hardware (buffering/other reasons)
Packets dropped by driver
Packets dropped by kernel
Packets discarded by filter
Packets dropped by ring buffer (kernel-user space transport)
Packets read by application

and indicate which counters are available on the platform at run time.

Regards,
Stephen
--
-------
Stephen Donnelly BCMS PhD   email: s...@endace.com
Endace Technology Ltd   phone: +64 7 839 0540
Hamilton, New Zealand   cell:  +64 21 1104378
---
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

[tcpdump-workers] Pull request for DAG updates

[Stephen Donnelly]

I have submitted a pull request to mcr's github tree.

https://github.com/mcr/libpcap/pull/1

There are 2 changes. The dag_platform_finddevs() function is updated to 
improve the search space and efficiency.


Secondly the build process moves to 'pcap-config' for external library 
dependencies instead of statically linking in external binary objects. 
The changes to configure.in will require rebuilding configure with the 
reference tool chain.


If the next release from master is likely to be some time, it would be 
worth merging to libpcap_1.1. I can provide this as well if needed.


Regards,
Stephen.
--

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] pcap anonymizer

[Stephen Donnelly]

On 29/04/11 19:12, Guy Harris wrote:

On Apr 28, 2011, at 3:31 PM, Michael Richardson wrote:


Unless someone says that there is something else out there, I'm going to
write an (IPv4) pcap file anonymizer.  I won't make the first version
efficient.

The Internet Traffic Archive has some anonymizing software:

http://ita.ee.lbl.gov/html/software.html

http://ita.ee.lbl.gov/html/contrib/tcpdpriv.html

http://ita.ee.lbl.gov/html/contrib/sanitize.html

and some other stuff that turned up from a search for

pcap anonymizer

include:


http://crawdad.cs.dartmouth.edu/meta.php?name=tools/sanitize/generic/AnonTool

http://www.tm.uka.de/software/pktanon/


There is also Crypto-PAn.

http://www.cc.gatech.edu/computing/Networking/projects/cryptopan/


CAIDA has a useful taxonomy of tools here:

http://www.caida.org/tools/taxonomy/anontaxonomy.xml


Stephen
--
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] [Wireshark-dev] Multiple interface capture device support in

[Stephen Donnelly]

On 06/06/12 22:03, Guy Harris wrote:

On Jun 5, 2012, at 8:04 PM, Stephen Donnelly wrote:

I've posted an 'experimental' patch/hack to dumpcap in Bug #7300.

https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=7300

The dumpcap implementation assumes that there is a one-to-one mapping between 
capture sources (pipe or pcap device) and physical interfaces, and so assigns 
one pcap-NG 'Interface Id' per source. This is fine for conventional capture 
sources, but does not support devices that represent more than one physical 
interface well.

...such as the Linux "any" device.


Good point, this is another case. Could PPI records come from multiple 
physical interfaces as well?


Does the linux 'any' device include a pseudo-header to indicate which 
interface each frame was captured on?


Is there a way to determine (before capture starts) how many interfaces 
will be captured from, or any details about them? This may require a new 
libpcap API.


Stephen.
--
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

[tcpdump-workers] pcap FCS length and LT_FCS_DATALINK_EXT()

[Stephen Donnelly]

Hi Guy,

In 2007 in libpcap afbb1ce7 you committed some code (possibly from Florent 
Drouin) adding the LT_FCS_DATALINK_EXT mechanism to record whether the capture 
includes information about captured FCS length, and if so what length it is.

I believe that currently only the DAG capture code supports this, and as 
implemented the LT_FCS_LENGTH is in units of 16-bits, e.g. 16-bit FCS returns 
1, and 32-bit FCS returns 2?

I don't think Wireshark's pcap wiretap code currently uses this mechanism to 
check for FCS length, does it make sense to add it?

If it is supported in Wireshark, it might be more widely implemented. It seems 
to solve a genuine problem, and avoids the proliferation of DLTs for WITH and 
WITHOUT FCS.

Regards,
Stephen.

___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

[tcpdump-workers] Fix DAG Stream support in dag_create()

[Stephen Donnelly]

It appears that when "Have non-interface modules take responsibility for 
identifying their devices" 2426611 
 
was committed, the heuristic for DAG device names was insufficient.


https://github.com/the-tcpdump-group/libpcap/commit/2426611584e9099af5f98d18ef37337df9bef025

The impact is that libpcap cannot capture from DAG streams other than 0, 
which will significantly affect some users.


This issue also affects libpcap_1.4 branch.

DAG capture device names are of the format dagN or dagN:M where N is the 
device number and M is the stream number. The former implies stream 0.


Thanks to joseph.cicc...@emulex.com  
for his initial patch.


I have created a pull request at github:
https://github.com/the-tcpdump-group/libpcap/pull/313

Regards,
Stephen.


___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

[tcpdump-workers] Pending pull request #378

[Stephen Donnelly]

Hi, I have had a pull request in the queue on github since August:

https://github.com/the-tcpdump-group/libpcap/pull/378

This does include some ideally separate things, a bug fix, and some 
improvements. Is there anything blocking this pull request? Is more information 
required, or should I rebase and separate the items? Has development moved to a 
different tree somewhere?

I notice there quite a few open pull requests in general, are there blocking 
issues with these, or insufficient resources to merge them?

Regards,
Stephen

___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

[tcpdump-workers] 1.9.0 release progress

[Stephen Donnelly]

Hi,

I see 1.9.0 is up to rc2 as of 25th June, how is it going? Is there anything we 
can do to assist?

This fixes a serious bug in 1.8.1 for us, so keen to see a new release!

Regards,
Stephen

___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] [libpcap] Problem with version 1.9.0

[Stephen Donnelly]

>Behalf Of Francois-Xavier Le Bail
>On 23/07/2018 15:33, Michael Richardson wrote:
>> Francois-Xavier Le Bail via tcpdump-security wrote:
>> > Need autoreconf.
>> > And 1.9.1 ?
>>
>> Let's do 1.9.1 in September.
>
>Why not this week to have a proper version tag ?

Agreed, seems like a simple fix. Better now than later?

Stephen

___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers