Re: [tcpdump-workers] Legacy Linux kernel support

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Oct 21, 2019, at 5:59 PM, Mario Rugiero via tcpdump-workers 
 wrote:

> I think it's time to summarize, and to propose one last idea.
> I'm following the thread again to try and be as accurate as possible,
> but of course any objections are welcomed.
> 
> - The oldest officially supported kernel is 3.16, as this is the
> oldest LTS according to kernel.org.
> - Users of the library must be properly informed when their
> environment is unsupported, as well as the last version supporting it.
>  This should be done both at compile-time and at run-time.
> - SOCK_PACKET goes away. This is already done in master.
> - TPACKET_V1 goes away.

This includes the hack to handle 32-bit userland running on top of a 64-bit 
kernel; TPACKET_V2 eliminated that problem by making the flags field a 32-bit 
integer, even on 64-bit architectures, in the data structures shared between 
the kernel and userland.

I.e., we also remove the internal "TPACKET_V1_64" support.

> - TPACKET_V2 stays for immediate-mode support.
>  - As a side-effect, RHEL6 remains supported.

So RHEL6's kernel is pre-3.16 and thus doesn't support TPACKET_V3?

>  - The idea of exploring using non-memory-mapped sockets for this was
> proposed, and it would be interesting to follow-up.
> For this, I was supposed to check whether that makes a difference
> regarding how the kernel implements it.
> - The workaround for TPACKET_V3's bug stays, as the fix was only
> introduced in 3.19.
> - We should explore reaching a solution to immediate-mode that doesn't
> require TPACKET_V2.
>  - It has to be noted, tho, that any changes to allow that aren't
> unlikely to be back-ported to older kernels, so we'd still need
> TPACKET_V2 for the time being. It'd be a bet for the future.

So you're talking about a TPACKET_V5, or changes to TPACKET_V3 or TPACKET_V4, 
to support immediate mode in memory-mapped capture, as opposed to using 
non-memory-mapped sockets?

> - Just to acknowledge it, it was proposed to research on whether
> support for AF_XDP makes sense. I think that belongs to its own
> discussion, tho.

Yes, that's a different mechanism from AF_PACKET.

Does it allow receiving copies of packets that are also handed either to the 
kernel networking stack or to other AF_XDP sockets for regular input 
processing? That would be needed to allow it to be used for packet sniffing.

> Now, the idea goes along with the last item.
> I was thinking of proposing a new option for TPACKET_V3 sockets to set
> a deadline.
> I haven't completely decided on the details, but basically it would
> behave somewhat like this:
> - The deadline can take three types of value:
>  - (-1): no deadline, wait until a block is full before marking as
> ready for user space. This would be the default, so no existing
> programs change their behavior.

That sounds like the behavior with a timeout set to 0 (with PF_PACKET sockets, 
BPF devices, and Solaris DLPI)

>  - (0): expose packets as soon as they arrive. This would act more or
> less as previous AF_PACKET versions work.

That sounds like immediate mode.

>  - A positive integer: this would be how long to wait

If you mean "deliver a block if it's full or if the timeout expires", that 
sounds like the behavior with a non-zero timeout.

So how does this differ from the regular timeout mechanism?

> (I haven't decided on the unit, I'm guessing microseconds should work).

The units are milliseconds in pcap_set_timeout() and with BPF devices.

>I'm not sure regarding what the deadline is set, but I'm thinking
> since the first packet in the block arrived.

At least for BPF devices, it's "since a read or select was done", rather than 
"since the first packet in the block arrived"; I think it might be "since the 
first packet in the block arrived" with Solaris DLPI.

> Any ideas on this?
> How should we keep the discussion in sync between the two lists?
> Should I CC the participants on this list on the RFC on the kernel list?

"The two lists" being this list and some Linux list?

Is the Linux list linux-netdev?
--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Legacy Linux kernel support

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Oct 22, 2019, at 11:38 AM, Mario Rugiero  wrote:

> El mar., 22 oct. 2019 a las 15:08, Guy Harris () escribió:
> 
>> On Oct 21, 2019, at 5:59 PM, Mario Rugiero via tcpdump-workers 
>>  wrote:
>>> - TPACKET_V2 stays for immediate-mode support.
>>> - As a side-effect, RHEL6 remains supported.
>> 
>> So RHEL6's kernel is pre-3.16 and thus doesn't support TPACKET_V3?
> 
> It's 2.6 series (2.6.32?), and TPACKET_V3 was introduced in 3.2.

I.e., the goal for libpcap support on Linux should be something such as

it should work on min({kernel for oldest supported enterprise 
distribution}, {oldest "longterm maintenance" kernel release from kernel.org})

>>> Now, the idea goes along with the last item.
>>> I was thinking of proposing a new option for TPACKET_V3 sockets to set
>>> a deadline.

...

>> So how does this differ from the regular timeout mechanism?
> 
> This mechanism would be for the AF_PACKET driver in the Linux kernel,
> not for libpcap.
> libpcap would only either set a small non-zero deadline on TPACKET_Vx
> (x >= 3) or 0 for immediate mode, and just use the default behavior
> for non-immediate mode.
> The similarity with what libpcap does is not a coincidence.

OK, so TPACKET_V3 currently supports something similar to what BPF devices 
support, i.e. "deliver a block if it's full or if the timeout expires".  The 
timeout is in the tp_retire_blk_tov field of a tpacket_req3 structure, as 
handed to a SOL_PACKET/PACKET_RX_RING setsockopt() call.  It's in units of 
milliseconds; it doesn't refer to inter-packet spacing, but to the age of the 
block.

Currently it doesn't deliver empty blocks; libpcap can handle either "delivers 
empty blocks" (as that's what BPF devices do) or "doesn't deliver empty blocks" 
(as that's what TPACKET_V3 currently does).

The main difference is whether the timeout times out even if no packets are 
available; I guess code that wants to be woken up periodically, perhaps to do 
other work, even if there's no traffic that passes the filter would prefer 
"time out even if no packets are available".

>> Is the Linux list linux-netdev?
> 
> Yes.

OK, I guess I'll have to go back to reading that list.  (It's a very heavy 
traffic list, and 99.999% of it isn't relevant to packet capture - all 
that matters to me is 1) PF_PACKET stuff and 2) stuff involving device modes 
such as some ethtool features and monitor-mode/radiotap support - so I just 
look at it on occasion.)

> Addendum: I missed one, replacing some device detection boilerplate.
> Initially. `if_nameindex` was proposed, but there's already the
> `getifaddr` based implementation that should detect all Linux
> interfaces usable by pcap by 2.3.x due to the fact that it counts
> AF_PACKET addresses, so we should be able to just remove the
> '/proc/net' and '/sys/class/net' crawling when we start expecting 2.4.

I.e., getifaddr() will find interfaces with no networking-layer addresses (no 
IPv4/IPv6/etc.) on 2.4 and later kernels?

--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Legacy Linux kernel support

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Oct 22, 2019, at 1:22 PM, Mario Rugiero  wrote:

> El mar., 22 oct. 2019 a las 16:02, Guy Harris () escribió:
>> I.e., the goal for libpcap support on Linux should be something such as
>> 
>>it should work on min({kernel for oldest supported enterprise 
>> distribution}, {oldest "longterm maintenance" kernel release from 
>> kernel.org})
>> 
> I'm more inclined to oldest longterm from kernel.org only, but I guess so.

If RHEL 6 matters, oldest longterm from kernel.org only doesn't work, because 
RHEL 6 runs 2.6.32, according to

https://access.redhat.com/articles/3078

so if we're going to support only the oldest longterm maintenance kernel from 
kernel.org, we're not going to support RHEL 6 unless TPACKET_V3 has been back 
ported to the RHEL 6 kernel.

If it's not backported, *and* we continue to use TPACKET_V2 for immediate mode, 
then RHEL 6 happens to still be supported to that extent.

However, if we require any *other* mechanisms that aren't present in the RHEL 6 
kernel, that means no RHEL 6 support.

So I wouldn't claim RHEL 6 support solely on the basis of continued TPACKET_V2 
support - don't rely on the side effect.

if we're going to support
> 
>> OK, so TPACKET_V3 currently supports something similar to what BPF devices 
>> support, i.e. "deliver a block if it's full or if the timeout expires".  The 
>> timeout is in the tp_retire_blk_tov field of a tpacket_req3 structure, as 
>> handed to a SOL_PACKET/PACKET_RX_RING setsockopt() call.  It's in units of 
>> milliseconds; it doesn't refer to inter-packet spacing, but to the age of 
>> the block.
>> 
>> Currently it doesn't deliver empty blocks; libpcap can handle either 
>> "delivers empty blocks" (as that's what BPF devices do) or "doesn't deliver 
>> empty blocks" (as that's what TPACKET_V3 currently does).
>> 
>> The main difference is whether the timeout times out even if no packets are 
>> available; I guess code that wants to be woken up periodically, perhaps to 
>> do other work, even if there's no traffic that passes the filter would 
>> prefer "time out even if no packets are available".
>> 
> I see. We would want a way to signal we want time outs regardless of
> blocks being empty, then, right?

Either that, or just change TPACKET_V3 to do that.

Originally, TPACKET_V3 delivered wakeups in a bogus fashion:

https://www.spinics.net/lists/netdev/msg290837.html

(that's the problem we're working around).  The original developer of 
TPACKET_V3 claimed that empty blocks have to be delivered:

https://www.spinics.net/lists/netdev/msg291734.html

but didn't indicate why, so I tried to infer from the patch:

https://www.spinics.net/lists/netdev/msg291734.html

I have no record of a response (and, for whatever reason, his original message 
didn't show up in the netdev archives).

The bogus wakeups were fixed by a later patch:

https://www.spinics.net/lists/netdev/msg315231.html

That also eliminated delivery of empty blocks to the user.  A response said 
"This change would break existing applications that have come
to depend on the periodic signal.":

https://www.spinics.net/lists/netdev/msg315418.html

to which I responded:

https://www.spinics.net/lists/netdev/msg315425.html

The author of the patch then responded:

https://www.spinics.net/lists/netdev/msg315510.html

https://www.spinics.net/lists/netdev/msg315528.html

The code currently has the patch, and doesn't deliver empty blocks.

>> OK, I guess I'll have to go back to reading that list.  (It's a very heavy 
>> traffic list, and 99.999% of it isn't relevant to packet capture - 
>> all that matters to me is 1) PF_PACKET stuff and 2) stuff involving device 
>> modes such as some ethtool features and monitor-mode/radiotap support - so I 
>> just look at it on occasion.)
>> 
> Wouldn't CC'ing you keep you on the loop already?

It might, as long as everybody keeps me on the CC list.

It doesn't look as if you've sent anything to netdev yet about tpacket changes.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Legacy Linux kernel support

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Oct 22, 2019, at 2:24 PM, Mario Rugiero  wrote:

> El mar., 22 oct. 2019 a las 18:01, Guy Harris () escribió:
>> 
>> If RHEL 6 matters, oldest longterm from kernel.org only doesn't work, 
>> because RHEL 6 runs 2.6.32, according to
>> 
>>https://access.redhat.com/articles/3078
>> 
>> so if we're going to support only the oldest longterm maintenance kernel 
>> from kernel.org, we're not going to support RHEL 6 unless TPACKET_V3 has 
>> been back ported to the RHEL 6 kernel.
>> 
>> If it's not backported, *and* we continue to use TPACKET_V2 for immediate 
>> mode, then RHEL 6 happens to still be supported to that extent.
>> 
>> However, if we require any *other* mechanisms that aren't present in the 
>> RHEL 6 kernel, that means no RHEL 6 support.
>> 
>> So I wouldn't claim RHEL 6 support solely on the basis of continued 
>> TPACKET_V2 support - don't rely on the side effect.
>> 
> Exactly. I'm against supporting it if it requires extra work. I don't
> think libpcap 1.10 is an absolute need in a scenario where you have to
> deal with RHEL 6, except possibly for security fixes, but those will
> have to be backported by Red Hat anyway.

So we'll say "oldest longterm maintenance kernel from kernel.org", and if it 
also happens to work on your enterprise Linux with a pre-3.16, consider 
yourself lucky; we won't make any effort to support RHEL 6 or other enterprise 
distribution releases with pre-3.16 kernels.

>> Either that, or just change TPACKET_V3 to do that.
>> 
> Yes, that's what I was proposing.

The proposal was "We would want a way to signal we want time outs regardless of 
blocks being empty, then, right?"; I was suggesting just delivering empty 
blocks no matter what, if there's code that depends on it.  Libpcap itself can 
work either way, and most libpcap applications used on both Linux and a 
platform with BPF devices can work either way, as they don't get timeouts with 
an empty buffer on Linux and they do get them with BPF, so I'm not sure there's 
a strong need to have TPACKET_V3 support both with an option to specify which 
one.

>> Originally, TPACKET_V3 delivered wakeups in a bogus fashion:
>> 
>> ...
>> 
>> The code currently has the patch, and doesn't deliver empty blocks.
>> 
> I'll read these carefully later, but my take on it is that TPACKET_V3
> used to support our use case, so in principle a patch to restore it
> could be accepted.

libpcap's use case doesn't require delivery of empty blocks; we make no promise 
that pcap_dispatch() or pcap_next() or pcap_next_ex() will return within the 
specified timeout interval.  I don't think Solaris DLPI with bufmod delivers 
empty packets, for example.

There may still be some programs that expect pcap_dispatch() or pcap_next() or 
pcap_next_ex() to return after the timeout expires, but since that doesn't 
happen on Linux with TPACKET_V3, most programs that used to do so have probably 
been changed not to do so.

There may be programs directly using PF_PACKET sockets with TPACKET_V3 that 
expected empty blocks to be delivered, but given that 3.19 was released on 
2015-02-08, and contained the patch that caused empty blocks not to be 
delivered, I suspect most programs that used to do no longer do so.

> I find it unclear whether it is the ability of posting of empty blocks
> that would break use cases or its absence from the previous paragraph,
> but I guess I'll know after reading the mails.

It's the *absence* of empty block delivery that was cited as potentially 
breaking code.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Legacy Linux kernel support

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Oct 23, 2019, at 12:26 AM, Anders Broman  wrote:

> El mar., 22 oct. 2019 a las 15:08, Guy Harris () escribió:
>> 
>>> Does it allow receiving copies of packets that are also handed either to 
>>> the kernel networking stack or to other AF_XDP sockets for regular input 
>>> processing? That would be needed to allow it to be used for packet sniffing.
>> 
>> I haven't looked into it yet, but I'll keep that in mind when I do.
> 
> Is that needed for an interface used ONLY for sniffing? E.g connected to a 
> span/mirror port or a tap? One of the interesting features here is packet 
> filtering on the NIC.

No, but there's no API in libpcap to indicate that the code is using it for 
passive sniffing on an interface used only for sniffing rather than for active 
sniffing of the traffic on a live interface, so if AF_XDP works only for 
passive sniffing there's no way to know whether it can be used or not.
--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Legacy Linux kernel support

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Oct 23, 2019, at 1:43 AM, Anders Broman  wrote:

> On Oct 23, 2019, at 12:26 AM, Anders Broman  
> wrote:
> 
>>> El mar., 22 oct. 2019 a las 15:08, Guy Harris () 
>>> escribió:
 
> Does it allow receiving copies of packets that are also handed either to 
> the kernel networking stack or to other AF_XDP sockets for regular input 
> processing? That would be needed to allow it to be used for packet 
> sniffing.
 
 I haven't looked into it yet, but I'll keep that in mind when I do.
>>> 
>>> Is that needed for an interface used ONLY for sniffing? E.g connected to a 
>>> span/mirror port or a tap? One of the interesting features here is packet 
>>> filtering on the NIC.
>> 
>> No, but there's no API in libpcap to indicate that the code is using it for 
>> passive sniffing on an interface used only for sniffing rather than for 
>> active sniffing of the traffic on a live interface, so if AF_XDP works only 
>> for passive sniffing >there's no way to know whether it can be used or not.
> 
> Isn't this very similar to WiFi monitor mode?

Wi-Fi monitor mode isn't necessarily passive sniffing on an interface used only 
for sniffing - at least some OSes and drivers allow interfaces to be in monitor 
mode while the interface is still associated with a network.  (I just did 
"tcpdump -I" on my MacBook Pro's Broadcom BCM43xx-based Wi-Fi adapter in one 
window, and "ping my.isp.net" in another window, on Mojave, and it worked.)

So it's not similar in that sense.

> How is that handled? Perhaps new libpcap API are needed for AF_XDP.

If there are two capture mechanisms, with one of them giving higher performance 
but not allowing the adapter to run as a regular network interface, then there 
would need to be an API in libpcap to allow that mechanism to be used - and 
there would need to be UI in programs using libpcap to allow that to be 
requested.

It's not yet determined whether AF_XDP does that.

> An API for dedicated capture interface might be useful for other stuff to 
> like CPU pinning, setting up the
> Interface for sniffing programmatically trough ethtool(Offloading, etc).

By "Offloading" do you mean *disabling* offloading?  That's something that 
might be used if you're *not* doing passive sniffing, if you want to see what 
the machine is sending on the wire.  (No, that won't handle the case where the 
offloading is the source of the problem, i.e. where disabling offloading makes 
the problem go away, but for *that* you'll need a network tap and either 
another machine or another adapter on your machine.)
--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

[tcpdump-workers] LGTM.com alerts about libpcap and tcpdump

[Guy Harris via tcpdump-workers]

--- Begin Message ---
(Looks Good To Me? Little Goggle-eyed Tiny Monkey?)

https://lgtm.com

has a number of open-source projects that it scans, including

https://lgtm.com/projects/g/the-tcpdump-group/libpcap/?mode=list

(mostly complaints about internal headers lacking header guards) and

https://lgtm.com/projects/g/the-tcpdump-group/tcpdump/?mode=list

(a lot of complaints about using non-thread-safe time_t interpretation 
routines; if we care about libnetdissect being thread-safe, that's an issue, 
and another pile of complaints about headers lacking header guards).--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Capturing external packets sent to loopback (FreeBSD) ?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Feb 24, 2020, at 6:15 AM, Ray Bellis via tcpdump-workers 
 wrote:

> I've got a daemon that listens on a virtual IP address, that is itself
> attached to a cloned loopback interface on FreeBSD.

What do you mean by "loopback" here?  The term "loopback interface" generally 
means "fake interface that sends packets from the machine to itself" on UN*Xes, 
e.g. the lo0 interface on most UN*Xes or just lo on Linux.  Is that the type of 
interface on which you're capturing?

If so...

> Packets for that daemon could be arriving from multiple physical
> interfaces, and that's what's taxing me just now.
> 
> I would -really- like to be able to process those packets without
> needing a per-interface thread or process (or other code changes) but it
> seems FreeBSD (and Linux FWIW) will only capture packets on that
> loopback IP that also -originated- on the loopback.

...why would packets be delivered on the loopback interface if they arrived on 
a physical interface?
--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Capturing external packets sent to loopback (FreeBSD) ?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Feb 24, 2020, at 9:44 AM, Ray Bellis via tcpdump-workers 
 wrote:

> I never considered "any" !   But you appear to be suggesting it's not
> available in FreeBSD ?

It's not.

In Linux, packet capture is done with sockets created with a protocol family of 
PF_PACKET.  Those sockets *can* be bound to a network interface, but if you 
*don't* bind the socket, it gets packets from all interfaces.  That's how the 
"any" device is implemented.

In *BSD/Darwin, packet capture is done with BPF devices (/dev/bpfN or, on 
systems with cloning BPF devices, a device you get from opening /dev/bpf).  
Those devices *must* be bound to a network interface; an unbound device won't 
provide any packets.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] snprintf in libpcap

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Mar 2, 2020, at 1:22 PM, Michael Richardson via tcpdump-workers 
 wrote:

> Back in 2016, we note in the CHANGES:
>   Replace sprintf() with pcap_snprintf().
> 
> but, we have no prototype for this, and apparently no definition, and we use
> snprintf() everywhere.  I'm trying to merge Ray's changes which are rebased
> into pull request #914.
> 
> I think that we just use "snprintf()" now.

In the master branch, we require versions of Visual Studio that support a 
C99-compatible snprintf(), and we also require C99-compatible snprintf() 
support on UN*Xes.

Earlier versions 1) had pcap_snprintf() and 2) either declared it and provided 
it, or just #defined pcap_snprintf to be snprintf.

Note that #914 also uses structure initializations that initialize fields by 
name, which is also a C99-ism and which I think sufficiently recent versions of 
Visual Studio support; I'm OK with requiring support for that (1990 called, it 
wants its C standard back...).--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] snprintf in libpcap

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Mar 2, 2020, at 1:51 PM, Guy Harris via tcpdump-workers 
 wrote:

> On Mar 2, 2020, at 1:22 PM, Michael Richardson via tcpdump-workers 
>  wrote:
> 
>> Back in 2016, we note in the CHANGES:
>>  Replace sprintf() with pcap_snprintf().
>> 
>> but, we have no prototype for this, and apparently no definition, and we use
>> snprintf() everywhere.  I'm trying to merge Ray's changes which are rebased
>> into pull request #914.
>> 
>> I think that we just use "snprintf()" now.
> 
> In the master branch, we require versions of Visual Studio that support a 
> C99-compatible snprintf(), and we also require C99-compatible snprintf() 
> support on UN*Xes.
> 
> Earlier versions 1) had pcap_snprintf() and 2) either declared it and 
> provided it, or just #defined pcap_snprintf to be snprintf.

Or are you thinking of asprintf()?  I didn't *think* that was available on 
Windows, so we would supply our own implementation, but AppVeyor appears to be 
finding it in recent builds.
--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] snprintf in libpcap

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Mar 2, 2020, at 2:08 PM, Guy Harris via tcpdump-workers 
 wrote:

> Or are you thinking of asprintf()?  I didn't *think* that was available on 
> Windows, so we would supply our own implementation, but AppVeyor appears to 
> be finding it in recent builds.

OK, it's finding it *with MinGW64*; it's not finding it with MSVC.

Some more cleanup is needed for MinGW64.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Linktype allocation for ATSC link layer protocol

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Mar 20, 2020, at 2:39 PM, Nick Kelsey via tcpdump-workers 
 wrote:

> Pull request created (#919):
> 
> https://github.com/the-tcpdump-group/libpcap/pull/919

I've added a proposed description as a comment to the pull request; please 
check whether my assumptions are correct and whether the description would work 
for that format's entry on https://www.tcpdump.org/linktypes.html.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Linktype allocation for ATSC link layer protocol

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Mar 20, 2020, at 11:14 PM, Nick Kelsey  wrote:

> I figure there will be interest in this protocol as we start to see ATSC 3.0 
> test broadcasts. Our ATSC 3.0 hardware can output pcap data over HTTP in real 
> time so ALP can be captured to file with a simple curl/wget command.

An additional possibility might be to implement the rpcap protocol, but I 
should probably get around to documenting that protocol before I ask you to do 
that. :-)--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

[tcpdump-workers] New RFCs for 1) pcap file format and 2) rpcapd protocol?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
There should probably be RFC-style specifications for 1) the pcap file format 
and 2) the rpcapd protocol used for remote capturing.

Currently, on GitHub, there's a "pcapng" team:

https://github.com/pcapng

with one repository containing the pcapng specification, and a 
"the-tcpdump-group" team:

https://github.com/the-tcpdump-group

with repositories for libpcap, tcpdump, and the tcpdump.org Web site.

It makes sense to me to keep those specifications on a site such as GitHub; 
GitHub comes to mind first because that's where pcapng currently is.

The options I see are:

1) add them as repositories to the pcapng team;

2) add them as repositories to the the-tcpdump-group team;

3) give them each their own teams.

I see pcapng - and the pcap file format and rpcapd protocol - as not being 
directly tied to libpcap.  *Historically*, pcap originated as the format that 
libpcap read and wrote, and rpcap was a protocol initially implemented in the 
WinPcap derivative of libpcap, but:

1) pcapng arose independently, and one of the earliest implementations 
was in Wireshark (where the internal APIs were easier to change; libpcap's 
support currently works through the existing API, but that hides a lot of the 
capabilities of pcapng);

2) code other than libpcap code reads and writes pcap files (including, 
but not limited to, Wireshark's code);

3) some devices either implement an rpcap server or could perhaps 
usefully do so, and they might have reasons to have independent implementations 
rather than basing their implementations on libpcap's rpcapd.

So I'm not inclined to go with option 2) - and if we do go with option 2), 
whatever arguments are offered for that would probably apply to pcapng as well, 
so it would, in that case, make sense to move the pcapng repository to that 
team as well.

1) has the slight disadvantage that the name for the team suggests it's for 
pcapng only; it appears that teams can be renamed:


https://help.github.com/en/github/setting-up-and-managing-organizations-and-teams/renaming-a-team

Were we to rename it, I don't know what would be a good new name.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] New RFCs for 1) pcap file format and 2) rpcapd protocol?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Mar 21, 2020, at 2:31 PM, Mario Rugiero via tcpdump-workers 
 wrote:

> El sáb., 21 mar. 2020 18:15, Guy Harris via tcpdump-workers 
>  escribió:
> 
>> 1) has the slight disadvantage that the name for the team suggests it's
>> for pcapng only; it appears that teams can be renamed:
>> 
>> 
>> https://help.github.com/en/github/setting-up-and-managing-organizations-and-teams/renaming-a-team
>> 
>> Were we to rename it, I don't know what would be a good new name.
>> 
> I'd be careful with this option, as it may affect downstream projects, as
> some download links that may be used for automated download (as part of a
> build process) could break. If there's anything downloadable I wouldn't
> change the team's name unless we make sure the old links remain accessible.

Its one repository contains the pcapng spec, no source code, so that particular 
example probably less likely to happen than would be the case for other teams.

But what it would break are links to the pcapng spec on other pages.

Perhaps changing the *description* of the team would suffice.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] New RFCs for 1) pcap file format and 2) rpcapd protocol?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Mar 21, 2020, at 2:14 PM, Guy Harris via tcpdump-workers 
 wrote:

> The options I see are:

4) add a new team for rpcap, as it's a protocol rather than a file format, and 
thus only indirectly tied to pcap/pcapng, and putting the pcap format in the 
pcapng team because you can't have a pcap*ng* without having had a pcap in the 
first place.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] [Wireshark-dev] New RFCs for 1) pcap file format and 2) rpcapd protocol?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Mar 22, 2020, at 9:49 AM, Michael Tuexen  
wrote:

> I would support this. However, last time I tried this, I was not successful.
> There were not very interested in defining a file format...

They've done so in the past:

https://tools.ietf.org/html/rfc1761 (and 
https://tools.ietf.org/html/rfc3827)

and, for non-capture-file formats:

https://tools.ietf.org/html/rfc8536#section-3--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] New RFCs for 1) pcap file format and 2) rpcapd protocol?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Mar 21, 2020, at 2:38 PM, Guy Harris via tcpdump-workers 
 wrote:

> On Mar 21, 2020, at 2:14 PM, Guy Harris via tcpdump-workers 
>  wrote:
> 
>> The options I see are:
> 
> 4) add a new team for rpcap, as it's a protocol rather than a file format, 
> and thus only indirectly tied to pcap/pcapng, and putting the pcap format in 
> the pcapng team because you can't have a pcap*ng* without having had a pcap 
> in the first place.

5) Treat rpcap as "remote procedure call for libpcap" and put it under the 
the-tcpdump-group team, and put pcap under the pcapng team as per the same 
reason as 4).--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] New RFCs for 1) pcap file format and 2) rpcapd protocol?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Mar 22, 2020, at 10:29 AM, Guy Harris via tcpdump-workers 
 wrote:

> 5) ... and put pcap under the pcapng team as per the same reason as 4).

Done.  It's pointed to by

https://github.com/pcapng/pcapng

and the XML source is at


https://github.com/pcapng/pcapng/blob/master/draft-gharris-opsawg-pcap.xml

Feel free to comment, add additional authors, etc..

(Note that its purpose is to document the *existing* format, rather than be a 
source of proposed changes.)--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] New RFCs for 1) pcap file format and 2) rpcapd protocol?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Mar 26, 2020, at 7:10 PM, Guy Harris via tcpdump-workers 
 wrote:

> (Note that its purpose is to document the *existing* format, rather than be a 
> source of proposed changes.)

...proposed changes to the format.

I am, however, planning on proposing a mechanism for vendor-specific "custom" 
link-layer types, inspired by the pcapng "custom" block types and option types, 
for use in both pcap and pcapng.  That'll be in a separate message.

As with pcapng block types and options, anything of *general* use that a vendor 
would like multiple programs to be able to handle should probably be proposed 
as a part of the standard, with a register link-layer type/block number/option 
number.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

[tcpdump-workers] "Custom" link-layer types for pcap and pcapng

[Guy Harris via tcpdump-workers]

--- Begin Message ---
Here's the proposal for "custom" link-layer types I threatened^Wpromised in my 
earlier email.

A link-layer type value of 0x will be reserved as LINKTYPE_CUSTOM, with 
libpcap offering a DLT_CUSTOM.

A custom link-layer type has a 32-bit IANA-registered Private Enterprise Number 
(PEN):

https://en.wikipedia.org/wiki/Private_Enterprise_Number

https://www.iana.org/assignments/enterprise-numbers/enterprise-numbers

as is used for custom pcapng blocks and options.

We could either 1) also say it should have a 32-bit per-vendor link-layer type 
number or 2) say that if the vendor plans to use it for more than one type of 
link-layer header, they have to arrange that the link-layer header should begin 
with information necessary to determine what the *rest* of the link-layer 
header is.

2) is more along the lines of the way custom block and options work.  However:

every non-custom block includes a block type, and every non-custom 
option has an option type, but not every *block* in a capture file has a 
link-layer header type - a pcap header has a link-layer type that applies to 
all packets in the file and a pcapng IDB has a link-layer type that applies to 
all packets on that interface;

knowing the link-layer type up front makes it easier to generate BPF 
filter code for an interface, if we support these types for live capture (or if 
the vendor's private capture mechanism supports it);

so I'm inclined to go with 1).

In that model:

in pcap files, if the lower 16 bits of the 32-bit link-layer type value 
is 0x, the two "Reserved" fields (which were formerly a rarely-if-ever-used 
and not-supported-by-libpcap-or-Wireshark time zone offset and time stamp 
resolution) MUST contain the PEN and vendor-specific link-layer type;

in pcapng file, if the link-layer type in an IDB is 0x, the IDB 
*MUST* contain a new option, containing the PEN and vendor-specific link-layer 
type.

Given that it's for *two* capture file formats, these lists are probably better 
places for discussion than having two pull requests and discussing them in 
comments there.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] [tcpdump] About smiInit() parameter

[Guy Harris via tcpdump-workers]

--- Begin Message ---


> On Mar 30, 2020, at 10:24 AM, Francois-Xavier Le Bail 
>  wrote:
> 
> Hi Guy,
> 
> We have:
> $ git grep -n '"tcpdump"'
> netdissect.c:72:smiInit("tcpdump");
> 
> netdissect.c is a part of libnetdissect.
> 
> Should we use
> smiInit("libnetdissect");
> or something else to separate the lib and the tool ?

The man page at

https://www.ibr.cs.tu-bs.de/projects/libsmi/smi_config.html

says:

int smiInit(const char *tag);

...

The smiInit() function should be the first SMI function called in an 
application.  It initializes its internal structures. If tag is not NULL, the 
global configuration file and (on UNIX systems) a user configuration file are 
read implicitly, if existent.  All global statements and those statements with  
a tag (a ``tag: '' prefix) that matches the tag argument are executed.  (see 
also CONFIGURATION FILES below).  

I don't know if anybody's actually *using* that particular libsmi feature, but 
smiInit has been called with "tcpdump" as an argument since at least tcpdump 
0.7.  For what it's worth, the configuration file example they give in the man 
page *does* have a "tcpdump:" statement in it:

Example configuration:

  #
  # $HOME/.smirc
  #

 # add a private directory
  path :/usr/home/strauss/lib/mibs

 # don’t show any errors by default
  level 0

 # preload some basic modules
  load SNMPv2-SMI
  load SNMPv2-TC
  load SNMPv2-CONF

 # want to make smilint shout
  smilint: level 8

 # but please don’t claim about
  # any names longer than 32 chars
  smilint: hide namelength-32

 tcpdump: load DISMAN-SCRIPT-MIB

 smiquery: load IF-MIB
  smiquery: load DISMAN-SCRIPT-MIB

so there might be some who have "tcpdump:" statements in libsmi configuration 
files.

So either

1) we should consider leaving "tcpdump" there

or

2) we should make the tag an argument to nd_init and pass it "tcpdump" 
from tcpdump

although the latter means that any new program using libnetdissect might 
dissect SNMP packets differently from tcpdump, whether that's a bug or a 
feature.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] New RFCs for 1) pcap file format and 2) rpcapd protocol?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Mar 22, 2020, at 11:40 AM, Francois-Xavier Le Bail 
 wrote:

> On 22/03/2020 18:29, Guy Harris via tcpdump-workers wrote:
> 
>> 5) Treat rpcap as "remote procedure call for libpcap" and put it under the 
>> the-tcpdump-group team, and put pcap under the pcapng team as per the same 
>> reason as 4).
> 
> Ok.

This would require the rules in the pcapng repository Makefile to be in the 
libpcap Makefile.in.

Michael, would that be inconvenient to maintain?

--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] New RFCs for 1) pcap file format and 2) rpcapd protocol?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Apr 1, 2020, at 11:51 AM, Michael Richardson  wrote:

> Guy Harris via tcpdump-workers  wrote:
>>> 
>>>> 5) Treat rpcap as "remote procedure call for libpcap" and put it under the 
>>>> the-tcpdump-group team, and put pcap under the pcapng team as per the same 
>>>> reason as 4).
>>> 
>>> Ok.
> 
>> This would require the rules in the pcapng repository Makefile to be in
>> the libpcap Makefile.in.
> 
>> Michael, would that be inconvenient to maintain?
> 
> Yes, I think it would be a pain, but it could be done.
> I certainly would not use the complex lib/* makefile if we really wanted to
> do that.   What about cmake side of things?

It's complex, too - and I don't know whether anybody's done a CMake version of 
that.

Would it make more sense to put the rpcap I-D into either 1) a separate team 
and repository or 2) another repository in the pcapng group, so that it has its 
own Makefile?--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Legacy Linux kernel support

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Apr 1, 2020, at 4:14 PM, Mario Rugiero via tcpdump-workers 
 wrote:

> I haven't yet been able to test it, which is why I've been delaying
> writing about this,
> but these two commits[0][1], which according to these threads[2][3]
> are the ones fixing
> the timeout issue, have been applied to CentOS 7 default kernel,
> 3.10.0-1062.el7.

So this is about the mmapped PF_PACKET bug where it delivers an empty buffer 
when it times out, and does *not* wake up the reader when a buffer is 
delivered, causing empty buffers to pile up unread, and forcing libpcap to 
periodically poll the mmapped ring buffer to see if anything was delivered 
without notification?

If so, then this:


https://github.com/torvalds/linux/commit/da413eec729dae5dcb150e2eb34c5e7e5e4e1b49

i.e. [0] fixes it, by doing wakeups when a buffer is *delivered* (rather than 
when a packet is *added* to the buffer, which is pointless) and this:


https://github.com/torvalds/linux/commit/41a50d621a321b4c15273cc1b5ed41437f4acdfb

i.e. [1] arranges that empty buffers aren't delivered when a timeout occurs.

has_broken_tpacket_v3() is checking whether the kernel has that fix or not, 
based on the kernel version number; to quote the comment in front of it:

 * Some versions of TPACKET_V3 have annoying bugs/misfeatures
 * around which we have to work.  Determine if we have those
 * problems or not.
 * 3.19 is the first release with a fixed version of
 * TPACKET_V3.  We treat anything before that as
 * not having a fixed version; that may really mean
 * it has *no* version.

so it checks for versions prior to 3.19.

That routine is used in set_poll_timeout(); if it returns "true" (a non-zero 
value), meaning TPACKET_V3 has the bug, the poll() done to wait for packets to 
arrive is given a non-zero timeout, so that the socket is checked for packets 
even if no wakeup is delivered by the socket, to drain the empty buffers.

> It can also be seen by comparing vanilla 3.10[4] with the CentOS
> sources[5], diffing
> the files located at net/packet/af_packet.c.
> This *should* mean it works without the workaround.

It should work without the workaround...

...on systems with the fix.

*NOT* all versions of the Linux kernel with TPACKET_V3 necessarily have the 
fix.  Version 3.19 and later have the fix; earlier versions have it *only* if 
whoever built the kernel (e.g., the distribution developer) backported the fix.

So we can't assume the workaround is unnecessary, and remove it, unless we 
either 1) require a 3.19 or later kernel if we're going to use TPACKET_V3 or 2) 
*somehow* can detect kernels to which the fix was backported.
--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

[tcpdump-workers] What's the "link level header" in "minus its link level header" for the -x flag?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
(Opening this up to the full tcpdump-workers list, to get more user input.)

On Apr 30, 2020, at 11:40 AM, Francois-Xavier Le Bail 
 wrote:

> The tcpdump manual states:
> 
>   -x When parsing and printing, in addition to printing  the  headers
>  of  each  packet,  print the data of each packet (minus its link
>  level header) in hex.  The  smaller  of  the  entire  packet  or
>  snaplen  bytes  will  be  printed.  Note that this is the entire
>  link-layer packet, so for link layers that pad (e.g.  Ethernet),
>  the  padding  bytes  will  also be printed when the higher layer
>  packet is shorter than the required padding.
> 
> In "minus its link level header" (singular, thus one header), link level 
> header should be understood
> as the DLT link level header ?
> 
> E.g. for "IP over Fibre Channel printer" (print-ipfc.c), the LL header length 
> is IPFC_HDRLEN (16) or
> caplen if the packet is truncated ?
> 
> I ask the question because sometimes some other LL length are taken in 
> account (LLC, etc.).
> I think it is confusing to mix in the "minus its link level header" the DLT 
> LL and other upper layer
> link layers.
> 
> We should just take in account the pseudo-header length in some cases e.g. 
> DLT_NETANALYZER,
> DLT_NETANALYZER_TRANSPARENT, etc., added to Ethernet header length.

My *guess* is that the most *useful* interpretation of "link level header" is 
"whatever, in an IP packet, would come before the IP header".

So that'd include, for example, the LLC header.

It would also, of course, take into account any metadata pseudo-headers, such 
as the NetAlyzer headers the radiotap header for 802.11.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] decode MPLS-contained packets?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On May 5, 2020, at 3:15 AM, Gert Doering via tcpdump-workers 
 wrote:

> tcpdump's print-mpls.c already does "if I know what upper-layer protocol
> is in here, I call the appropriate printer".  But there is no well-defined
> type field, so it fails for my packets, and and falls back to "hexdump"
> (good enough).

OK, so what *shark's MPLS dissector does is:

1) see whether the label indicates the next protocol - but very few label 
values are registered in the dissector table for that, so that's *not* what's 
being used for your dissector (most dissectors just register for the "decode 
as" feature, so the user can say "decode this label as...");

2) if that fails, "use the 1st nibble logic (see BCP 4928, RFC 4385 and 5586)".

The "1st nibble logic" is:

if the upper nibble of the first octet is 4, it's IPv4';

if the upper nibble of the first octet is 6, it's IPv6;

if the upper nibble of the first octet is 1, dissect it as "PW 
Associated Channel Header Management Communication Network (MCN) dissection as 
per RFC 5718" (with a comment that "this could be from an Ethernet pseudo-wire 
without a control word, with the MAC address's first nibble being 1.";

if the upper nibble of the first octet is 0, then "If this is an 
Ethernet pseudo-wire, this could either be Ethernet without a control word and 
with the first nibble of the destination MAC address being 0 or it could be 
Ethernet with a control word.  Let the "pw_eth_heuristic" dissector try to 
figure it out.".

The heuristics that dissector does are

if the first 12 bytes in the payload look like two valid ethernet 
addresses", treat it as Ethernet with no control word;

otherwise, if the first octet is 0, treat it as Ethernet *with* a 
control word;

otherwise, treat it as Ethernet without a control word.

"Looks like a valid Ethernet address" is defined as "the first three octets 
appear in Wireshark's file giving manufacturer names for OUIs".  Tcpdump 
*currently* doesn't have such a file.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] decode MPLS-contained packets?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On May 5, 2020, at 11:36 AM, Gert Doering via tcpdump-workers 
 wrote:

> So, given that the first 16 bits are "4 bit always 0, and 12 bits
> reserved-must-be-set-to-0", using these as heuristics for "if two 0-bytes
> are following the MPLS headers, it's a control word, so we skip 4 bytes
> and the rest is a regular Ethernet packet" should work.

Wireshark looks only at the uppermost nibble, as per my earlier mail, probably 
to make it "future-proof" against the reserved bits being used later.

However, it also has the "do the upper three octets, and the three octets thee 
octets after that, look like OUIs" test.

(Note that 00:00:0C is Cisco, so "two 0 bytes following the MPLS headers" isn't 
*guaranteed* to work as a way of identifying control words.  Wireshark's manuf 
file also shows 00:00:17 as being Oracle, 00:00:0F as being NeXT so that may 
now be used by Apple if NeXT didn't use it up, and 00:00:F0 is Samsung 
Electronics, so there might be others in that range.)--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] decode MPLS-contained packets?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On May 7, 2020, at 12:04 AM, Francois-Xavier Le Bail via tcpdump-workers 
 wrote:

> On 07/05/2020 08:53, Guy Harris via tcpdump-workers wrote:
> 
>> "Looks like a valid Ethernet address" is defined as "the first three octets 
>> appear in Wireshark's file giving manufacturer names for OUIs".
> What if the destination address is ff:ff:ff:ff:ff:ff (broadcast) for e.g. ARP 
> request ?
> Or some multicast address ?

In this *particular* case, that test is done only if the uppermost nibble of 
the uppermost octet is 0, so that would only be the case for the source 
address, which is less likely to be a group address than the destination 
address.  There may be other places where that heuristic dissector is used, 
however.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] decode MPLS-contained packets?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On May 5, 2020, at 1:01 PM, Francois-Xavier Le Bail via tcpdump-workers 
 wrote:

> Wireshark MPLS heuristic is not perfect and has been criticized but is still 
> there :-) hopefully
> correctly parsing your data as well.

*No* heuristic will be perfect here.

> For tcpdump maybe a -T based approach is better?
> 
> -T mpls (+ 1st nibble logic for IPv4/IPv6)
> -T ethpw
> -T ethpwnocw

That's a thought, but -T should probably get a bunch of work done on it.

It's documented as

 -T type
  Force  packets  selected  by  "expression" to be interpreted the
  specified type.

but it has nothing to do with the filter expression - that really means "force 
all packets to be interpreted [as] the specified type", because "expression" 
controls which packets tcpdump sees.  It doesn't *entirely* work that way, but 
the places where it's checked - for example,  the TCP and UDP dissectors - just 
do "if somebody specified a -T flag, dissect *everything* as that protocol".

A mechanism where you could do something such as "-T tcp:1073:{protocol}" to 
force traffic to TCP port 1073 to be dissected as the specified protocol might 
be useful; in this case, we'd do something such as "-T mpls:{protocol}" to 
force *all* MPLS packets to be dissected as the specified protocol, and "-T 
mpls:{label}:{protocol}", to force packets with a particular label to be 
dissected as the specified protocol (which might mean you'd have to run tcpdump 
twice - once to see what the label is, and once to decode the label.

That would require a mechanism similar to Wireshark's "run this script on all 
dissector files to find their registration routines and call those at run time" 
to build a table of the first items in the "-T" string; once we've gone there, 
we might as well use it to build a table of the {protocol} values as well, and 
we could then just have a mechanism similar to Wireshark's dissector table 
mechanism, replacing the big switch statements (back in the very early days, 
Wireshark - still called "Ethereal" at the time - used switch statements as 
well; that was replaced by a dissector table mechanism) with dissector tables, 
which would mean that a new "runs atop TCP" or "runs atop UDP" dissector could 
just register and the switch statement wouldn't have to be changed.

We could start out by adding an explicit list of items, just as we currently 
have an explicit list of {protocol} values.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Compile libpcap with DLT_LINUX_SLL2

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Mar 31, 2020, at 2:10 AM, Francois-Xavier Le Bail 
 wrote:

> On 31/03/2020 00:04, Petr Vorel wrote:
>> Hi Guy,
>> 
 BTW man pages (pcap.3pcap.in, pcap_datalink.3pcap.in, pcap_loop.3pcap and
 pcap_next_ex.3pcap) mention only DLT_LINUX_SLL.
>> 
>>> Fixed in commit ffb99eceefd31771a4aa89f0da5d02a3c53cfd03.
>> Thanks a lot!
>> 
>> BTW how about DLT_LINUX_SLL2 as the default? What does it block?
> 
> To avoid breaking program that cannot use SLL2, could we have an API like
> pcap_set_cooked_default_ssl2() or pcap_set_default_ssl2_on_any() and use it 
> in tcpdump?

Or have an API like pcap_set_datalink() and use it in tcpdump.

Which we already do:

tcpdump -y LINUX_SLL -w /tmp/linux_sll_not_linux_sll2.pcap--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Compile libpcap with DLT_LINUX_SLL2

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Mar 31, 2020, at 2:10 AM, Francois-Xavier Le Bail 
 wrote:

> To avoid breaking program that cannot use SLL2,

Note, by the way, that one program that can't dissect LINKTYPE_LINUX_SLL2 
packets is named "Wireshark".

No, nobody appears to have contributed a change to add support to 
epan/dissectors/packet-sll.c yet.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Compile libpcap with DLT_LINUX_SLL2

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On May 8, 2020, at 10:47 AM, Guy Harris via tcpdump-workers 
 wrote:

> No, nobody appears to have contributed a change to add support to 
> epan/dissectors/packet-sll.c yet.

I just did; it was a significant enough change to a bit of infrastructure used 
by other dissectors that it's probably not a candidate for backporting, but it 
should show up in the 3.4 release, which should come out some time this year.

The main clients of libpcap on Linux are probably:

1) tcpdump - already supports SLL2 in the main branch, so will be 
supported in 5.0;

2) Wireshark - now supports SLL2 in the main branch, so will be 
supported in 3.4;

3) some others - Snort?  Scapy?  Bro^WZeek?  Any other ideas?

   Snort and its daq library, as of the latest release tarballs 
understand SLL but not SLL2.

   Snort3/Snort++ and its daq library appear to understand neither in 
the master branch.

   Scapy appears to understand SLL but not SLL2 in the master branch.

   Zeek appears to have a tiny bit of understanding of SLL but not SLL2 
in the master branch; it might be that it runs with a filter such as "ip" or 
"ip or ip6", so all it needs to know is how to skip over the link-layer header, 
which it does.

So, for now, I guess defaulting to SLL2 in tcpdump is the best answer, as it 
won't surprise any other software's live capture.  If new versions of various 
Linux distribution come out with the new tcpdump at about the same time as the 
new Wireshark comes out, it shouldn't cause problems with Wireshark reading 
tcpdump captures.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] LINUX_SLL2 printing update

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On May 8, 2020, at 9:57 PM, Francois-Xavier Le Bail via tcpdump-workers 
 wrote:

> For a quick look, I don't need 'ifindex N', but I need 'In/Out,...'
> 
> Thus I propose to print:

+1
--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Compile libpcap with DLT_LINUX_SLL2

[Guy Harris via tcpdump-workers]

--- Begin Message ---
BTW, having just implemented SLL2 support in Wireshark, the layout of the 
header really doesn't work as well as I'd like with ARPHRD_NETLINK packets.

I'd prefer something like

struct header {
uint16_t hatype;/* link-layer address type */
uint8_t  pkttype;   /* packet type */
uint8_t  halen; /* link-layer address length */
uint8_t  addr[SLL_ADDRLEN]; /* link-layer address */
int32_t  if_index;  /* 1-based interface index */
uint16_t hatype_specific;   /* dependent on sll3_hatype */
uint16_t protocol;  /* protocol */
};

because

1) It puts the protocol field *after* the hatype field, and right before the 
payload, so that, for packets with an hatype of ARPHRD_NETLINK, we can treat 
everything up to the if_index field as the cooked-mode header, and have the 
dissector for ARPHRD_NETLINK-over-SLL treat the hatype_specific and protocol 
fields as fields for *it* to dissect.  For that ARPHRD_ type, the protocol is a 
Netlink protocol type, so it really should be analyzed by the code that 
understands Netlink messages.

2) It provides a field to handle various annoyances in the way that packets are 
provided to PF_PACKET sockets.  In particular, Netlink messages are in the host 
byte order of the machine doing the capturing, so, for ARPHRD_NETLINK, we can 
have libpcap put the value 0x0123 in that field, in *host* byte order, so the 
code that processes the packets can just see whether that field's value is 
0x0123 or 0x3210 and, based on that, determine whether the messages need to be 
byte-swapped.  (Remember, somebody might capture Netlink traffic on a machine 
with one byte order but read the capture on a machine with the opposite byte 
order.)

Is SLL2 sufficiently established that we'd have to introduce an SLL3 type, or 
can we just change SLL2 at this point?--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] [tcpdump] Keep win32/prj/WinDump.dsp ?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On May 24, 2020, at 4:37 AM, Francois-Xavier Le Bail via tcpdump-workers 
 wrote:

> 15 printers are missing in win32/prj/WinDump.dsp.
> Does anyone use it? Any reason to keep it ?

Note that the *supported* way to build tcpdump (and libpcap) on Windows is with 
CMake (which can more easily be kept up-to-date by UN*X users adding a new 
dissector than can Windows project files, and which are tested by the 
Travis/AppVeyor CI builds).  CMake can generate project files for several 
versions of Visual Studio, as well as build files for various other build 
systems.

See


https://github.com/the-tcpdump-group/tcpdump/blob/master/doc/README.Win32.md

for information on building tcpdump on Windows with Visual Studio (and


https://github.com/the-tcpdump-group/libpcap/blob/master/doc/README.Win32.md

for information on building libpcap on Windows with Visual Studio).--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] [AiG-CERT #104737] DLT value

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On May 29, 2020, at 3:23 AM, Airbus CERT via tcpdump-workers 
 wrote:

> I would like to request you to get a DTL value for the PR 
> https://github.com/the-tcpdump-group/libpcap/pull/934. 
> This PR intend to add ETW capture for libpcap.

So is each packet an Event Tracing for Windows:

https://docs.microsoft.com/en-us/windows/win32/etw/event-tracing-portal

record of some sort?  If so, where is the format of that record defined?--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] [AiG-CERT #104737] DLT value

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Jun 2, 2020, at 12:22 AM, Airbus CERT via tcpdump-workers 
 wrote:

> Yes exactly each packet is an event. The layout of the event is 
> https://docs.microsoft.com/en-us/windows/win32/api/evntcons/ns-evntcons-event_header
>  and 
> https://docs.microsoft.com/en-us/windows/win32/api/evntcons/ns-evntcons-event_header_extended_data_item.
>  But we aligned this format with the ETL (serialization use by microsoft) 
> which is not well documented.

Is it documented at all?

The description of a given LINKTYPE_/DLT_ value on

https://www.tcpdump.org/linktypes.html

and the pages linked to by that description must be sufficient to allow 
somebody to write code to, at minimum, parse the link-layer headers, without 
ever looking at Wireshark or tcpdump code.
--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] [tcpdump] Keep win32/prj/WinDump.dsp ?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Jun 8, 2020, at 12:24 PM, Francois-Xavier Le Bail 
 wrote:

> Thus all the files in win32/prj/ could be removed?
> (WinDump.dsp  WinDump.dsw  WinDump.sln  WinDump.vcproj)

I have no problem removing them and requiring Windows users to ue CMake, 
especially given that newer versions of Visual Studio has CMake as an 
installable component.

If nobody else has said "no, I need them!" - and volunteered to take 
responsibility for maintaining them! - I'd say we should get rid of them, as 
we're not maintaining them.  (CMake files 1) can handle multiple versions of 
Visual Studio and 2) are intended to be maintainable by people using a text 
editor rather than generated by a big IDE that runs primarily, if not 
exclusively, on Windows.)--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] [AiG-CERT #104737] DLT value

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Jun 2, 2020, at 12:58 AM, Airbus CERT via tcpdump-workers 
 wrote:

> The layout is 
> https://docs.microsoft.com/en-us/windows/win32/api/evntcons/ns-evntcons-event_header

So each packet's data starts with, in order:

a 2-octet event record size;
a 2-octet header type;
a 2-octet flag word;
a 2-octet indication of the format of the event data;
a 4-octet thread ID;
a 4-octet process ID;
an 8-octet time stamp;
a 16-octet UUID for the event provider;
a sequence of:
a 2-octet event identifier;
a 1-octet event version;
a 1-octet event channel;
a 1-octet event level;
a 1-octet event opcode;
a 2-octet task identifier;
an 8-octet keyword bitmask;
either:
a 4-octet elapsed kernel CPU time followed by a 4-octet elapsed 
user CPU time;
or:
an 8-octet elapsed user-mode CPU time;
a 16-octet UUID for an activity.

What byte order are the numerical values in?  Little-endian?

> following by one or more 
> https://docs.microsoft.com/en-us/windows/win32/api/evntcons/ns-evntcons-event_header_extended_data_item
>  depending of the flag _EVENT_HEADER.Flags.

So that's one or more of, in order:

2 reserved octets;
a 2-octet extended data type value;
2 reserved octets;
a 2-octet extended data size value;

presumably immediately followed by the octets of the extended data.

What byte order are the numerical values in?  Little-endian?

If the number of octets of extended data isn't a multiple of 8, is there any 
padding after it?

And do the same rules used to generate those data layouts - and the same choice 
of byte order - apply for the structures in the extended data?
--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

[tcpdump-workers] Reading capture files with an unknown link-layer header type

[Guy Harris via tcpdump-workers]

--- Begin Message ---
François checked in a change to tcpdump so that, if it's handed a capture file 
with a link-layer header type for which it has no dissector, it just dumps the 
packet data in hex, rather than failing with an indication that the header type 
isn't supported.

However, pcap_compile(), in *libpcap*, will fail with an unknown header type - 
and tcpdump always hands a filter to pcap_compile(), even if it's a null string 
(which means "accept every packet").

It doesn't fail with *known* filter types for which most filters are 
unsupported, it just rejects most of them (other than "link[M:N]").

Is there any reason *not* handle link-layer types unknown to libpcap in 
pcap_compile()?--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Proposed update to DLT_BLUETOOTH_LE_LL_WITH_PHDR

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Jul 9, 2020, at 1:46 PM, Sultan Khan  wrote:

> Through discussions with Joakim Anderson (of Nordic) and Mike Ryan (Ubertooth 
> developer), and going through several iterations of proposed protocol 
> updates, I/we came up with this: 
> https://gistcdn.githack.com/sultanqasim/8b6561309f5934f084a0d938ae733b7a/raw/LINKTYPE_BLUETOOTH_LE_LL_WITH_PHDR.html

In the last paragraph, it says:

For packets using the LE Coded PHY as defined in the Bluetooth Core 
Specification v5.2, Volume 6, Part B, Section 2.2, the Coding Indicator (CI) is 
represented by the two least significant bits of a dedicated coding indicator 
byte between the Access Address and PDU. Packets received using the LE Coded 
PHY are represented in an uncoded form, so the TERM1 and TERM2 coding 
terminators are not included in the LE packet field.

Perhaps that's a bit clearer if stated as

For packets using the LE Coded PHY as defined in the Bluetooth Core 
Specification v5.2, Volume 6, Part B, Section 2.2, the LE Packet is represented 
as the Coding Indicator (CI), stored in a one-octet field with the lower 2 bits 
containing the CI value, immediately followed by the PDU and the CRC.  Packets 
received using the LE Coded PHY are represented in an uncoded form, so the 
TERM1 and TERM2 coding terminators are not included in the LE packet field.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Proposed update to DLT_BLUETOOTH_LE_LL_WITH_PHDR

[Guy Harris via tcpdump-workers]

--- Begin Message ---
A couple more editorial comments:

In the description of the bits in the Flags field, I'd describe the 0x3000 bits 
as "PDU type dependent", and, after they're listed indicate that:

For PDU types other than type 1 (auxiliary advertising), the PDU type 
dependent field indicates the checked status of the MIC portion of the 
decrypted packet:

* 0x1000 indicates the MIC portion of the decrypted LE Packet 
was checked
* 0x2000 indicates the MIC portion of the decrypted LE Packet 
passed its check

For PDU type 1 (auxiliary advertising, the PDU type dependent field 
indicates the auxiliary advertisement type:

* 0x: AUX_ADV_IND
* 0x1000: AUX_CHAIN_IND
* 0x2000: AUX_SYNC_IND
* 0x3000: AUX_SCAN_RSP

I'd redo the last two paragraphs as:

The LE Packet field follows the previous fields. All multi-octet values 
in the LE Packet are always expressed in little-endian format, as is the normal 
Bluetooth practice.

For packets using the LE Uncoded PHYs (LE 1M PHY and LE 2M PHY) as 
defined in the Bluetooth Core Specification v5.2, Volume 6, Part B, Section 
2.1, it is represented as the four-octet access address, immediately followed 
by the PDU and CRC; it does not include the preamble.

For packets using the LE Coded PHY as defined in the Bluetooth Core 
Specification v5.2, Volume 6, Part B, Section 2.2, the LE Packet is represented 
as the four-octet access address, followed by the Coding Indicator (CI), stored 
in a one-octet field with the lower 2 bits containing the CI value, immediately 
followed by the PDU and the CRC; it does not include the preamble. Packets 
using the LE Coded PHY are represented in an uncoded form, so the TERM1 and 
TERM2 coding terminators are not included in the LE packet field.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Proposed update to DLT_BLUETOOTH_LE_LL_WITH_PHDR

[Guy Harris via tcpdump-workers]

--- Begin Message ---
For an advertising physical channel PDU, it appears that the PDU type is in the 
least-significant 4 bits of the PDU header.

Is that not present in an auxiliary advertising packet?--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Proposed update to DLT_BLUETOOTH_LE_LL_WITH_PHDR

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Jul 10, 2020, at 2:57 PM, Sultan Khan  wrote:

> Link to the updated version of the spec with the latest changes: 
> https://gistcdn.githack.com/sultanqasim/8b6561309f5934f084a0d938ae733b7a/raw/199fb1867642c927f768fe7d67dae2a639acb48e/LINKTYPE_BLUETOOTH_LE_LL_WITH_PHDR.html

So

https://www.tcpdump.org/linktypes.html

currently links to

http://www.whiterocker.com/bt/LINKTYPE_BLUETOOTH_LE_LL_WITH_PHDR.html

for LINKTYPE_BLUETOOTH_LE_LL_WITH_PHDR.  What should it link to now?





--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Proposed update to DLT_BLUETOOTH_LE_LL_WITH_PHDR

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Jul 13, 2020, at 9:02 AM, Sultan Khan  wrote:

> Thanks Chris. I’ll make a pull request to tcpdump-htdocs later today, and 
> I’ll include a link to the previous version of the spec as an archive.org 
> link to the old one on whiterocker.com.

The new version is a superset of the old version, so that any header that 
conforms to the old version also conforms to the new version, right?

If so, I don't see any need for an archive.org link to the old version.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Proposed update to DLT_BLUETOOTH_LE_LL_WITH_PHDR

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Jul 13, 2020, at 8:09 AM, Sultan Khan  wrote:

> Hmm. Chris Kilgour (whiterocker) originally created the spec, and the version 
> on tcpdump.org was just a backup copy. Now, Chris has said that he is no 
> longer active in the Bluetooth LE sniffing space, and he doesn’t want to be 
> in charge of the spec any more.

Does this also apply to the LINKTYPE_BLUETOOTH_BREDR_BB specification?--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] tcpslice licence

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Aug 3, 2020, at 12:33 PM, Denis Ovsienko via tcpdump-workers 
 wrote:
> 
> Whilst updating the description of files in tcpslice (the little
> relative of tcpdump) repository, it came to my attention that it does
> not have the customary LICENSE file. I have looked through the .c
> and .h files and they contain the following boilerplates:
> 
> * a 4-clause BSD-style licence seemingly derived from the so-called
>  LBNL 3-clause BSD: https://opensource.org/BSD-3-Clause-LBNL
> * a 3-clause BSD licence with the same text as above and two clauses
>  merged together
> * GPL2+
> 
> Would it be difficult to tell which licence is the right one for the
> program, and to say it in a LICENSE file for clarity?

The first step I'd take would be to get rid of the GPLed headers in favor of 
BSD-licensed headers, e.g. taking the ip.h, tcp.h, and udp.h headers from 
tcpdump and changing the code to work with them.

What remains are:

1) files such as tcpslice.c, which have a 3-clause variation of the original 
4-clause BSD license:

https://spdx.org/licenses/BSD-4-Clause.html

that puts the fourth clause ("don't use our name to endorse or promote products 
derived from this software without specific prior written permission") in a 
separate sentence, with no number, after the third clause ("give us credit by 
name");

2) files such as sessions.c, which have a 3-clause BSD license:

https://spdx.org/licenses/BSD-3-Clause.html

(with a slight wording tweak - just "The name of the author" rather than 
"Neither the name of the copyright holder nor the names of its contributors", 
probably because the copyright holder is the only contributor).

The 3-clause variation of the original 4-clause BSD license has the 
"advertising clause" ("All advertising materials mentioning features or use of 
this software must display the following acknowledgement: This product includes 
software developed by {XXX}.").

However, the 3-clause LBNL license you mention above is different - it's the 
LBNL version of the 3-clause BSD license, that has 3 numbered clauses because 
it doesn't have the advertising clause, not because it doesn't give the fourth 
clause of the original 4-clause BSD license a number.

A while ago, I tried contacting people at LBNL to see whether the big BSD "we 
hereby drop the advertising clause" letter applied to code licensed by LBNL.  I 
seem to remember not getting a definitive answer; I can't find *any* answer in 
my mail any more.  (Time to run find | xargs egrep on my mail directory?)

However, the 3-clause LBNL license *does* remove the clause - *and* the page 
you cite gives

> License Steward: 
> Sebastian Ainslie
> Principal Commercialization & Licensing Lead
> Computing Sciences Area & Energy Geosciences Division
> Intellectual Property Office, Lawrence Berkeley National Laboratory

so I'll try contacting Mr. Ainslie to see whether we can replace the 
3-clause-plus-one-unnumbered-clause LBL license with the 3-clause LB(N)L 
license in libpcap, tcpdump, and tcpslice.

--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

[tcpdump-workers] About DLT_LANE8023 and lane_if_print()

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Aug 4, 2020, at 1:28 PM, Francois-Xavier Le Bail  
wrote:

> lane_if_print() in print-lane.c
> (Added by 77b2a4405561467f66a3dfb0f8ce2b0eaa5ebaf9 in Sun Nov 21 1999 "print 
> of ATM LanEmulation")
> is called for DLT_LANE8023:
> 
> print.c-56-#ifdef DLT_LANE8023
> print.c:57: { lane_if_print,DLT_LANE8023 },
> print.c-58-#endif
> (Added by 777892a591065d32fb8744675574f9214398283a in Sun Nov 21 1999 "add 
> lane and cip printing")
> 
> But DLT_LANE8023 was never defined in libpcap nor tcpdump.

A comment in pcap/dlt.h says:

/*
* 17 was used for DLT_PFLOG in OpenBSD; it no longer is.
*
* It was DLT_LANE8023 in SuSE 6.3, so we defined LINKTYPE_PFLOG
* as 117 so that pflog captures would use a link-layer header type
* value that didn't collide with any other values.  On all
* platforms other than OpenBSD, we defined DLT_PFLOG as 117,
* and we mapped between LINKTYPE_PFLOG and DLT_PFLOG.
*
* OpenBSD eventually switched to using 117 for DLT_PFLOG as well.
*
* Don't use 17 for anything else.
*/

However, I downloaded ISO disk 6 from

https://archive.org/download/SuSE6.3-full

mounted it (macOS diskimages-helper for the win!), copied libpcap-0.4a6.spm, 
turned it into a cpis file with rpm2cpio, and extracted the contents; I can't 
see DLT_LANE8023 in either the source (which may be a vanilla version of 
libpcap 0.4a6, often mistakenly thought to be the last libpcap from LBL - 0.4 
non-alpha was) or in the SuSE patch, so either

1) there was no DLT_LANE8023 in SuSE 6.3;

2) there was, but it wasn't in libpcap;

3) there was, but it wasn't in *that* libpcap, it was in some *other* 
libpcap (but I couldn't find any other libpcap);

4) that's not an image of SuSE 6.3.

So I checked my mailbox, and found a message from 2000(!) to the ethereal-dev 
mailing list:

https://www.wireshark.org/lists/ethereal-users/28/msg00159.html

in which, among other things, I said:

> So I downloaded an RPM from SuSE's Web site, and the "bpf.h" in it says:
> 
>   /* Warning: not binary compatible with ANK libpcap !!! */
>   #define DLT_LANE802317  /* LANE 802.3(Ethernet) */
>   #define DLT_CIP 18  /* ATM Classical IP */

and

> And then, in Linuxland:
> 
>   We have Alexey's patches - which may just have picked stuff up
>   from elsewhere - which add
> 
>   #define DLT_LANE802315  /* LANE 802.3(Ethernet) */
>   #define DLT_CIP 16  /* ATM Classical IP */
> 
>   We have the ISDN4Linux patches, which add
> 
>   #define DLT_I4L_RAWIP   15  /* isdn4linux: rawip */
>   #define DLT_I4L_IP  16  /* isdn4linux: ip */
> 
>   And now we have SuSE's, which add the ISDN4Linux stuff, and then
>   add the stuff from Alexey's patches *but with different
>   numbers from the ones in his patches*.

I'm not sure what RPM that was, but the idea was, presumably, that *if* you 
built tcpdump on a system that *did* define DLT_LANE8023 in *its* libpcap, and 
used *its* libpcap, it could print packets that used DLT_LANE8023.

("Alexey"/"ANK" is Alexey Kuznetzov who, among other things, created the 
"turbopacket" patch to the Linux PF_PACKET socket code; that eventually got 
into the mainline kernel - the "T" in "TPACKET_V[123]" stands, I think, for 
"turbo".)

> What to do with this?

As far as I know, neither DLT_LANE8023 nor DLT_CIP are still around in any 
Linux distribution, so nuking support for that's OK with me.  I'm not seeing 
any support for either of them in Wireshark.

Current OpenSuSe Leap 15.2 does not have DLT_LANE8023 or DLT_CIP.

Is there any reason to keep the code to handle those DLT_ values around?--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] pcap_compile_nopcap() not in man pages

[Guy Harris via tcpdump-workers]

--- Begin Message ---
(Your first attempt seems to have worked - finally.  Perhaps Michael cleared 
the backlog?)

On Aug 10, 2020, at 4:24 AM, Denis Ovsienko via tcpdump-workers 
 wrote:

> It turns out, pcap_compile_nopcap() has been a part of the libpcap API
> since version 0.5 (June 2000), but it is not even mentioned anywhere in
> the man pages. The existing pcap_compile.3pcap man page seems to be the
> best place to add this information, since the two functions are
> similar. Would it be the right thing to do?

The problem with pcap_compile_nopcap() is that it provides no way to return an 
error message if it fails, unlike pcap_open_dead() combined with 
pcap_compile(), where you can use pcap_geterr() before closing the pcap_t.

An additional problem is that NetBSD fixed this by adding an error-buffer 
pointer argument, but that meant that NetBSD's pcap_compile_nopcap() was 
unfixably incompatible with the one in other OSes.  They've shifted to the 
compatible API, at the cost of not being able to get an error string.

So, for now, my inclination is to 1) deprecate pcap_compile_nopcap() (complete 
with marking it as deprecated, so code that uses it gets a compile-time warning 
on compilers where the deprecation macro is supported) and 2) not document it.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Using libnetdissect in other code, outside tcpdump source tree

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Aug 11, 2020, at 4:55 AM, Bill Fenner via tcpdump-workers 
 wrote:

> Is there a plan for a public face for libnetdissect?

At some point we should probably do that.

(Back in the late '90's, I discovered a program called tcpview, which was a 
Motif(!)-based GUI network analyzer based on modified tcpdump code, so people 
*have* used tcpdump's dissection code in their own programs.)

> I've tried teasing it
> out, and I ended up having to install:
> funcattrs.h print.h config.h netdissect.h ip.h ip6.h compiler-tests.h
> status-exit-codes.h
> in /usr/include/tcpdump/ in order to compile a libnetdissect-using program
> outside of the tcpdump source tree.

netdissect.h is the library's main API-declaration header.  print.h also 
declares functions that I'd consider part of libnetdissect's API; 
status-exit-codes.h is also part of that API.

For funcattrs.h and compiler-tests.h, libpcap installs equivalents in the 
include/pcap directory, for use by pcap.h.

We should probably have an include/libnetdissect directory in which we install 
netdissect.h and the headers it requires.

However, API-declaring headers should *NEVER* require config.h (there was a 
particularly horrible case with OpenBSD's version of libz, forcing a painful 
workaround in Wireshark:

/*
 * OK, now this is tricky.
 *
 * At least on FreeBSD 3.2, "/usr/include/zlib.h" includes
 * "/usr/include/zconf.h", which, if HAVE_UNISTD_H is defined,
 * #defines "z_off_t" to be "off_t", and if HAVE_UNISTD_H is
 * not defined, #defines "z_off_t" to be "long" if it's not
 * already #defined.
 *
 * In 4.4-Lite-derived systems such as FreeBSD, "off_t" is
 * "long long int", not "long int", so the definition of "z_off_t" -
 * and therefore the types of the arguments to routines such as
 * "gzseek()", as declared, with prototypes, in "zlib.h" - depends
 * on whether HAVE_UNISTD_H is defined prior to including "zlib.h"!
 *
 * It's not defined in the FreeBSD 3.2 "zlib", so if we include "zlib.h"
 * after defining HAVE_UNISTD_H, we get a misdeclaration of "gzseek()",
 * and, if we're building with "zlib" support, anything that seeks
 * on a file may not work.
 *
 * Other BSDs may have the same problem, if they haven't done something
 * such as defining HAVE_UNISTD_H in "zconf.h".
 *
 * "config.h" defines HAVE_UNISTD_H, on all systems that have it, and all
 * 4.4-Lite-derived BSDs have it.  Therefore, given that "zlib.h" is included
 * by "file_wrappers.h", that means that unless we include "zlib.h" before
 * we include "config.h", we get a misdeclaration of "gzseek()".
 *
 * Unfortunately, it's "config.h" that tells us whether we have "zlib"
 * in the first place, so we don't know whether to include "zlib.h"
 * until we include "config.h"
 *
 * A similar problem appears to occur with "gztell()", at least on
 * NetBSD.
 *
 * To add further complication, on recent versions, at least, of OpenBSD,
 * the Makefile for zlib defines HAVE_UNISTD_H.
 *
 * So what we do is, on all OSes other than OpenBSD, *undefine* HAVE_UNISTD_H
 * before including "wtap-int.h" (it handles including "zlib.h" if HAVE_ZLIB
 * is defined, and it includes "wtap.h", which we include to get the
 * WTAP_ERR_ZLIB values), and, if we have zlib, make "file_seek()" and
 * "file_tell()" subroutines, so that the only calls to "gzseek()" and
 * "gztell()" are in this file, which, by dint of the hackery described
 * above, manages to correctly declare "gzseek()" and "gztell()".
 *
 * On OpenBSD, we forcibly *define* HAVE_UNISTD_H if it's not defined.
 *
 * Hopefully, the BSDs will, over time, remove the test for HAVE_UNISTD_H
 * from "zconf.h", so that "gzseek()" and "gztell()" will be declared
 * with the correct signature regardless of whether HAVE_UNISTD_H is
 * defined, so that if they change the signature we don't have to worry
 * about making sure it's defined or not defined.
 *
 * DO NOT, UNDER ANY CIRCUMSTANCES, REMOVE THE FOLLOWING LINES, OR MOVE
 * THEM AFTER THE INCLUDE OF "wtap-int.h"!  Doing so will cause any program
 * using Wiretap to read capture files to fail miserably on a FreeBSD
 * 3.2 or 3.3 system - and possibly some other BSD systems - if zlib is
 * installed.  If you *must* have HAVE_UNISTD_H defined before including
 * "wtap-int.h", put "file_error()" into a file by itself, which can
 * cheerfully include "wtap.h" and get "gzseek()" misdeclared, and include
 * just "zlib.h" in this file - *after* undefining HAVE_UNISTD_H.
 */

Furthermore, the result of config.h may *also* reflect:

the compiler being used when it was generated, which means that it may 
not be appropriate on platforms with multiple compilers that would produce 
different config.h results, if you're compiling with a compiler other than the 
one used to generate config.h;

the instruction set used as the target when config.h was generated, 
which means that it may not be appropriate on platforms that support fat 
binaries, such as macOS (Apple now only support little-e

Re: [tcpdump-workers] Using libnetdissect in other code, outside tcpdump source tree

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Aug 12, 2020, at 1:31 PM, Guy Harris via tcpdump-workers 
 wrote:

> We should probably have an include/libnetdissect directory in which we 
> install netdissect.h and the headers it requires.

Or include/netdissect.

> However, API-declaring headers should *NEVER* require config.h (there was a 
> particularly horrible case with OpenBSD's version of libz, forcing a painful 
> workaround in Wireshark:

...

> so if anything in netdissect.h depends on config.h definitions, we should try 
> to fix that.

It looks like it's just declaring replacements for strlcat(), strlcpy(), 
strdup(), and strsep() if the platform doesn't provide them.  That should be 
done in a non-public header.

> That leaves ip.h and ip6.h; I'd have to check to see whether they should be 
> considered part of the API or not.

The comments are:

#include "ip.h" /* struct ip for nextproto4_cksum() */
#include "ip6.h" /* struct ip6 for nextproto6_cksum() */

so what should probably be done is have a header for *users* of libnetdissect 
and a separate header for *components* of libnetdissect; the latter can define 
more things.  (The latter would be a non-public header, unless we decide to 
support third-party dissector plugins; that would also mean we'd probably want 
to have something like Wireshark's dissector tables to which those plugins 
would add themselves.)--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] backward compatibility in pcap_loop(3PCAP)?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Aug 21, 2020, at 2:48 PM, Denis Ovsienko via tcpdump-workers 
 wrote:

> The man page says:
> 
>   (In  older  versions  of libpcap, the behavior when cnt was 0
>   was undefined; different platforms and devices  behaved
>   differently,  so  code that  must work with older versions of
>   libpcap should use -1, not 0, as the value of cnt.)
> 
> Would it make sense to move this paragraph to a BACKWARD COMPATIBILITY
> section and to tell which specific version started to recognise 0 as a
> valid value?

That's where other "some of what this manual page says doesn't apply to older 
versions of libpcap" items go, so it'd make sense.

The PACKET_COUNT_IS_UNLIMITED(), which is what pcap modules should now be 
using, was introduced in libpcap 1.5, so the first version where either 0 or -1 
should work is 1.5.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] tcpdump ack why become more 6 bytes

[Guy Harris via tcpdump-workers]

--- Begin Message ---
This is not a security issue; questions about tcpdump should be sent to 
tcpdump-workers@lists.tcpdump.org, which is where I'm sending this question.

On Sep 14, 2020, at 8:22 PM, Accepted <532876...@qq.com> wrote:

> hi, in this picture, I try to use tcpdump to get package when a new 
> connection become.
> but in three handshakes，the last ack why added more 6 bytes?

If that's Ethernet traffic, it's Ethernet padding.

An ACK-only TCP-over-IPv4 packet with no IP or TCP options has 20 bytes of IP 
header (the "45" at the beginning of the IP header says "IPv4, with a 20-byte 
header), 20 bytes of TCP header, and no TCP payload, for a total of 40 bytes.  
The Ethernet header is an additional 14 bytes, for a total of 54 bytes.

An Ethernet packet has a minimum size of 64 bytes, including the 4-byte CRC at 
the end of the packet; the CRC is normally not captured, so it doesn't show up 
in tcpdump.  The ACK-only packet must therefore have 6 bytes of padding before 
the 4-byte CRC, to be 64 bytes long.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] New Version Notification for draft-tuexen-opsawg-pcapng-02.txt

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Sep 28, 2020, at 12:06 PM, Michael Tuexen  wrote:

> Do we want to finally publish that? Up to now, I think the point was to
> find a home where it is substantially discussed and improved...

For example, unlike pcap, which is not easily changeable (you *can* change it, 
but that involves adding new magic numbers), pcapng can have new block types 
and option types.

There are extensible protocols with RFCs; that's handled with protocol 
registries:

https://www.iana.org/protocols

and with new I-Ds -> RFCs for extensions.  We'd have to set up registries for 
block and option types if we publish an RFC for pcapng.  We would *also* want a 
registry for link-layer header types, for both pcap and pcapng.

See, for example, RFC 1761

https://tools.ietf.org/html/rfc1761

which specifies the Sun snoop file format, and RFC 3827:

https://tools.ietf.org/html/rfc3827

which sets up a registry for snoop link-layer header types:


https://www.iana.org/assignments/snoop-datalink-types/snoop-datalink-types.xhtml#snoop-datalink-types-2

and adds some new entries to it.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] New Version Notification for draft-tuexen-opsawg-pcapng-02.txt

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Sep 28, 2020, at 12:06 PM, Michael Tuexen  wrote:

> On 28. Sep 2020, at 20:26, Michael Richardson  wrote:
> 
>> internet-dra...@ietf.org wrote:
>>> Diff:
>>> https://www.ietf.org/rfcdiff?url2=draft-tuexen-opsawg-pcapng-02
>> 
>> Hi, I have converted the xml to markdown.
> 
> Why? If we want to publish this, it will be published in xmlv3. So
> better to use that format earlier...

There are tools to convert Markdown to v2 or v3 RFC XML:

https://www.rfc-editor.org/pubprocess/tools/

so:

1) is it easier to edit Markdown or RFC XML?

2) is Markdown rich enough to do everything we want to do?

For 2), I note that


https://github.com/pcapng/pcapng/blob/master/draft-tuexen-opsawg-pcapng.md

has a bunch of stuff that GitHub isn't treating as markup, such as the stuff 
prior to the "Introduction" heading, and the tags such as "{::boilerplate 
bcp14}".  Is that an extension of Markdown not supported by GitHub's Markdown 
renderer but supported by some Markdown-to-RFC XML converter, or incomplete 
parts of the RFC XML to Markdown conversion?

In addition, the XML version at


https://github.com/pcapng/pcapng/blob/master/reference-draft-tuexen-opsawg-pcapng.xml

has some additional Decryption Secrets Block secret formats.  Those have data 
formats that *themselves* call for figures, and I'd been trying, at one point, 
to determine how to do that in RFC XML v2 format - it might require v3 format.  
Can that be handled with Markdown?--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] New Version Notification for draft-tuexen-opsawg-pcapng-02.txt

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Sep 28, 2020, at 1:42 PM, Michael Tuexen  wrote:

> Shouldn't we write up (I can work on an initial version) of
> a specification for .pcap.


https://github.com/pcapng/pcapng/blob/master/draft-gharris-opsawg-pcap.xml


http://xml2rfc.tools.ietf.org/cgi-bin/xml2rfc.cgi?url=https://raw.githubusercontent.com/pcapng/pcapng/master/draft-gharris-opsawg-pcap.xml&modeAsFormat=html/ascii&type=ascii
--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] New Version Notification for draft-tuexen-opsawg-pcapng-02.txt

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Sep 28, 2020, at 2:00 PM, Michael Tuexen  wrote:

> On 28. Sep 2020, at 22:48, Guy Harris  wrote:
> 
>> On Sep 28, 2020, at 1:42 PM, Michael Tuexen  wrote:
>> 
>>> Shouldn't we write up (I can work on an initial version) of
>>> a specification for .pcap.
>> 
>>  
>> https://github.com/pcapng/pcapng/blob/master/draft-gharris-opsawg-pcap.xml
> 
> Cool. Do you want to publish it as an RFC?

At some point.

Currently, I view it as "up for review by the community", and there have been 
pull requests from the community applied.

Should its publication coincide with the introduction of an IANA registry of 
link-layer data types (replacing the tcpdump.org one)?

Should we publish one RFC for the pcap format and one RFC that includes the 
current content of the registry?  (The latter would probably be much bigger 
than the former)--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] [OPSAWG] New Version Notification for draft-tuexen-opsawg-pcapng-02.txt

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Sep 29, 2020, at 7:14 PM, Qin Wu  wrote:

> Can you clarify what functionalities is missed for more modern applications? 
> Since it is enhancement to libpcap, do you expect all the future packet 
> capture tools support the format defined in this draft?

pcapng is a file format that's a replacement for pcap.

The current version of libpcap can read some pcapng files, but it only shows 
what can be shown through the existing pcap API, so most of the enhancements 
don't make a difference to programs using libpcap.  That version of libpcap 
cannot *write* pcapng files.

macOS's version of libpcap has undocumented APIs that allow macOS's tcpdump to 
read and write pcapng files.

Wireshark doesn't use libpcap to read capture files; it fully supports reading 
and writing pcapng files.

In the future, we would like to add new APIs to libpcap that support reading 
and writing pcapng files (and pcap files as well); the new APIs will make all 
of the added capabilities of pcapng available.  However, programs that use 
libpcap will have to be changed to use the new APIs in order to use those added 
capabilities.  tcpdump will probably be the first program updated to use them.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] libpcap error codes?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Oct 7, 2020, at 1:30 PM, Fernando Gont via tcpdump-workers 
 wrote:

> WHile using pcap_inject() in Linux, it is failing with "pcap_inject(): send: 
> Resource temporarily unavailable". In principle, one would expect that for 
> temporary problems (such as this one), one may one to wait a bit and retry.  
> So it would make sense to somehow be able to process the error 
> code/condition, and act differently depending on the error type.
> 
> Is there a way to get an error code (say, int value), as opposed to a text 
> describing it?

There isn't, other than looking at errno.

A new API could be added that returns a PCAP_ERROR_ value rather than -1 on 
error (so as not to break source or binary compatibility with code using the 
existing APIs).--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] libpcap error codes?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Oct 7, 2020, at 3:16 PM, Denis Ovsienko via tcpdump-workers 
 wrote:

> Do you mean to introduce a function like pcap_error(), which the
> developers would be able to interrogate if they need in use cases like
> this? Then existing functions could be slowly updated as needed to store
> the fault details somewhere for that function.

I was thinking of a new API for injecting packets, which would directly return 
a PCAP_ERROR_ value.  A pcap_last_error() routine would also handle that case, 
but if a new routine would return a success vs. failure value, it might as well 
return an error code, so pcap_last_error() would be useful only for existing 
routines.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] [pcap-ng-format] New Version Notification for draft-tuexen-opsawg-pcapng-02.txt

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Sep 28, 2020, at 4:28 PM, Michael Richardson  wrote:

> Guy Harris  wrote:
>> For 2), I note that
> 
>>  
>> https://github.com/pcapng/pcapng/blob/master/draft-tuexen-opsawg-pcapng.md
> 
>> has a bunch of stuff that GitHub isn't treating as markup, such as the
>> stuff prior to the "Introduction" heading, and the tags such as
>> "{::boilerplate bcp14}".  Is that an extension of Markdown not
>> supported by GitHub's Markdown renderer but supported by some
>> Markdown-to-RFC XML converter
> 
> Yes, kramdown-rfc2629. The MT Makefile does all the magic.

(Presumably "the MT Makefile" is the currently checked-in Martin Thompson 
Makefile in GitHub.)

So is there anything we do to arrange that the "Current committed version as 
..." links on the GitHub repository home page work again?

Should we, for now, put the XML documents back into the repository, and either

1) arrange, somehow, that GitHub automatically regenerate the XML 
documents if the Markdown documents are updated, if that's possible

or

2) have everybody who modifies the Markdown documents update the XML 
documents and check them in with the updated Markdown documents?

Or is there some site that will run kramdown-rfc2629 on a Markdown file and run 
xml2rfc on the result, along the lines of what xml2rfc.tools.ietf.org does?  I 
haven't gotten

https://xml2rfc.tools.ietf.org/experimental.html#kramdown

to work - I tried pasting 
https://raw.githubusercontent.com/pcapng/pcapng/master/draft-tuexen-opsawg-pcapng.md
 into the URL box, selecting "Window", and hitting Submit, but it didn't seem 
to work, it just popped up a blank window.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] [pcap-ng-format] New Version Notification for draft-tuexen-opsawg-pcapng-02.txt

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Oct 17, 2020, at 4:19 PM, Guy Harris  wrote:

> Or is there some site that will run kramdown-rfc2629 on a Markdown file and 
> run xml2rfc on the result, along the lines of what xml2rfc.tools.ietf.org 
> does?  I haven't gotten
> 
>   https://xml2rfc.tools.ietf.org/experimental.html#kramdown
> 
> to work - I tried pasting 
> https://raw.githubusercontent.com/pcapng/pcapng/master/draft-tuexen-opsawg-pcapng.md
>  into the URL box, selecting "Window", and hitting Submit, but it didn't seem 
> to work, it just popped up a blank window.

OK, that appears to go from kramdown-rfc2629 to xml2rfc XML format, but doesn't 
go the rest of the way; they don't appear to have a converter that takes 
kramdown-rfc2629 as input and gives you HTML/text/PDF as output.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] [pcap-ng-format] New Version Notification for draft-tuexen-opsawg-pcapng-02.txt

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Oct 17, 2020, at 6:01 PM, Michael Richardson  wrote:

> Guy Harris via tcpdump-workers  wrote:
> 
>> So is there anything we do to arrange that the "Current committed
>> version as ..." links on the GitHub repository home page work again?
> 
> Yes, there is a travis-ci process that generated the gh-pages.
> I haven't done that yet.  It was slightly involved before, but it has gotten
> significantly easier I'm told.  Travis-CI does the work.
> Oops, it is now "Circle" (CI) that is going it:
> 
>   https://github.com/martinthomson/i-d-template/blob/main/doc/REPO.md
>   _Automatic Update for Editor's Copy with Circle CI_

So, once that's done, there will be .xml files in the repository, built from 
the .md files?  (File, currently, but at some point I'll convert the pcap spec 
to  kramedown-rfc2629 as well.)

>> Or is there some site that will run kramdown-rfc2629 on a Markdown file
>> and run xml2rfc on the result, along the lines of what
>> xml2rfc.tools.ietf.org does?  I haven't gotten
> 
>> https://xml2rfc.tools.ietf.org/experimental.html#kramdown
> 
>> to work - I tried pasting
>> https://raw.githubusercontent.com/pcapng/pcapng/master/draft-tuexen-opsawg-pcapng.md
>> into the URL box, selecting "Window", and hitting Submit, but it didn't
>> seem to work, it just popped up a blank
> 
> generally, "gem install kramedown-rfc2629" is all you need to do.

Yes, that worked for me on macOS Catalina.  (I'm not sure whether Big Sur ships 
with Ruby or not; Apple wants to stop shipping the scripting language 
interpreters - or, at least, the ones not required by Single UNIX Spec 
conformance - probably because they don't want to be responsible for making it 
work *and* for keeping it up to date.  That's the tradeoff with "OS vendor 
provides third-party software" - the good news is it's there without having to 
download it, the bad news is that you may not be getting the latest version.)

But that means that I can generate the .xml files on my machine, for checking 
purposes.  I was trying to find a Web server that could be handed a URL for a 
kramedown-rfc2629 document and that would return an HTML/PDF/txt version of the 
document, for display in the browser, similar to what the xml2rfc server does 
for URLs pointing to an xml2rfc XML document, so that we could add links to 
README.me.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] [pcap-ng-format] New Version Notification for draft-tuexen-opsawg-pcapng-02.txt

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Oct 18, 2020, at 1:32 AM, Michael Tuexen  wrote:

> Just a note. I'm using the .xml format and put a link in the README.md, which 
> shows the .txt or .html file based on the current .xml.

Yes, that's what we were doing for the pcapng draft before switching to 
kramdown-rfc2629, and what we're still doing for the pcap draft, because I 
haven't yet switched it to kramdown-rfc2629.

That works for xml2rfc XML; it doesn't work for kramdown-rfc2629, because 
xml2rfc.tools.ietf.org doesn't have a converter that goes from kramdown-rfc2629 
directly to HTML/PDF/text.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] [RFC] Addition of link-layer header types for PCI, PCI-X, and PCI-Express

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Oct 21, 2020, at 1:56 PM, Aki Van Ness via tcpdump-workers 
 wrote:

> I'm working on a project that plans to store PCI and PCI-Express
> packets in the pcapng format as that's the most appropriate storage
> format and I really rather not roll something custom.
> 
> As such what are thoughts on adding Link-Layer types for PCI, PCI-X,
> and PCI-Express?

It seems reasonable, given that we have USB, Infiniband, and the DisplayPort 
AUX channel.

> And would you want to group all versions of PCI, PCI-X, and
> PCI-Express together or have them be their own values?

Would each version need its own LINKTYPE_ value, or would a single metadata 
header and payload suffice for all versions of PCI, all versions of PCI-X, and 
all versions of PCIe?  From a quick look at the Wikipedia pages for those, for 
what that's worth, they changes for each seem to be at the physical layer, with 
full or at least significant backwards compatibility, so, other than additional 
bits of metadata, would LINKTYPE_PCI, LINKTYPE_PCI_X, and LINKTYPE_PCI_EXPRESS 
suffice?

I'm assuming that the metadata would be different between PCI, PCI-X, and PCIe.
--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] pcap_lookupdev returning NULL

[Guy Harris via tcpdump-workers]

--- Begin Message ---
What happens if you put

printf("Version: %s\n", pcap_lib_version());

before the pcap_lookupdev() call?

It won't fix the pcap_lookupdev() call not to return NULL, but it'll indicate 
what version of libpcap your program is using, which might help determine what 
the problem is.  Let us know what the "Version:" output is.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] pcap_lookupdev returning NULL

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Nov 4, 2020, at 9:18 PM, Vaughan Wickham  wrote:

> Version: libpcap version 1.5.3

That's an older version (CentOS, proudly trailing-edge!), and only returns 
interfaces that the program can open.

Capturing on Linux generally requires, at minimum, the CAP_NET_RAW privilege, 
and finding devices might also require CAP_NET_ADMIN; root privilege will also 
work.  As such, you program will, by default, not be able to open *any* capture 
device, so:

1) if you were using a sufficiently more recent of libpcap, which 
return interfaces that the program doesn't have sufficient privileges to open 
(so that the user gets a "permission denied" error when trying to capture, 
which is somewhat clear about the underlying problem, rather than just not 
seeing any devices), you'd get "eth0" but then you'd get an error trying to 
open it (presumably that's why you're calling pcap_lookupdev());

2) you need to give your program sufficient privileges.

So try doing

sudo setcap cap_net_raw,cap_net_admin+eip {your program}

and then running the program.  ("cap_net_admin" might not be necessary with 
1.5.1.)--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] pcap_lookupdev returning NULL

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Nov 4, 2020, at 10:26 PM, Vaughan Wickham  wrote:

> In regards to your latest comments regarding
> 
> sudo setcap cap_net_raw,cap_net_admin+eip {your program}
> 
> Are you saying that I need to compile my program and then start the compiled 
> version with these arguments, from a terminal?

No.

You need to compile your program (within the IDE or on the command line), 
execute, on the command line, the command

sudo setcap cap_net_raw,cap_net_admin+eip {your program}

where {your program} is the path to the executable that was built, and then you 
can run the program from the command line or from the IDE.

> Alternatively, while I've been happy using CentOS as a development 
> environment up until now. As I'm planning on doing some work with pcap; if 
> there is a "better" distro for doing pcap development I'm more than happy to 
> build another development system using whatever flavour is easiest to develop 
> with.

Note that, as I said, getting a newer version of libpcap will *not* remove the 
requirement that you run your program with special privileges; all it means is 
that pcap_lookupdev() will not require the special privileges, but if you plan 
to *open* the device that it returns, your program will have to run with, at 
minimum, the cap_net_raw privileges.

And all that choosing a distribution other than CentOS will do is perhaps 
change the libpcap version.

> Basically I would like to be able build and execute within the IDE.

Unless you can arrange that the IDE run a special command, *as root*, as part 
of the build process, you won't be able to do everything within the IDE>

The command in question is "setcap cap_net_raw,cap_net_admin+eip {the program 
that was built}".  It will have to ask you for root privileges, which means 
that, if you want to avoid the command line, the IDE will have to run some GUI 
program that asks for your password, or the password of somebody with rights to 
run a program as root (that's what sudo, on the command line, does, but I don't 
know whether any version of sudo can do a GUI prompt when not run on the 
command line) and then run a command as root.

You will also have to have whatever privileges sudo, or the GUI program, 
requires you to have in order for it to allow you to run a program as root.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] pcap_lookupdev returning NULL

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Nov 5, 2020, at 1:04 AM, Vaughan Wickham  wrote:

> Appreciate all the info that you have provided.
> 
> Although it probably doesn't look like it from my questions; I did actually 
> read some tutorials prior to posting my initial question; and none made 
> reference to the need for:
> sudo setcap cap_net_raw,cap_net_admin+eip {your program} 
> 
> So I'm wondering if you can suggest some reading that I should review to 
> understand the basics of using libpcap.

I suspect most, if not all, tutorials spend little if any time discussing the 
platform-dependent permission issues with capturing traffic with libpcap; they 
probably focus on "how to write code using libpcap", not "how to arrange that 
your program have enough privileges to do something useful with libpcap".

The only discussions I can offer for the "permissions" issue are:

1) the "capture privileges" page of the Wireshark Wiki:


https://gitlab.com/wireshark/wireshark/-/wikis/CaptureSetup/CapturePrivileges

   and, for your case, this particular subsection of that page:


https://gitlab.com/wireshark/wireshark/-/wikis/CaptureSetup/CapturePrivileges#other-linux-based-systems-or-other-installation-methods

2) the main pcap man page:

https://www.tcpdump.org/manpages/pcap.3pcap.html

   in the subsection that begins with "Reading packets from a network 
interface may require that you have special privileges:".

> Also, where can I find an overview of the key differences between version 
> 1.5.3 and the current release?

There isn't one.  In this *particular* case, the difference (which may have 
been introduced before the current 1.9 version) is that pcap_findalldevs() 
(atop which pcap_lookupdev() is built) checks for operability in older releases 
and doesn't do so for newer releases.  However, as noted, the permissions 
required to open a device for capture does *not* differ (and *can't* differ - 
it's a requirement imposed by the OS kernel) between older and newer versions.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] [pcap-ng-format] draft-gharris-opsawg-pcap.txt --- IANA considerations

[Guy Harris via tcpdump-workers]

--- Begin Message ---
(Resent, from the correct address this time.)

On Dec 21, 2020, at 5:51 PM, Michael Richardson  wrote:

> The short of it is:
> 
> 1) reserve bits 16:28 of linktype as zero.

In pcap files, presumably; you have only bits 0:15 in pcapng IDBs.

Note that the registry is for both pcap and pcapng, so the specs should say 
that.

> 2) lower 32K Specification Required (any document),
>  upper 32K First Come First Served
> 
> Details:
> The Registry has three sections according to {{RFC8126}}:
> * values from 0 to 32767 are marked as Specification Required.
> *   except that values 147 to 162 are reserved for Private Use
> * values from 32768 to 65000 are marked as First-Come First-Served.
> * values from 65000 to 65536 are marked as Private Use.

Presumably "to 65535" - 65536 doesn't fit in the 16-bit pcapng field.

So, for FCFS, does that mean anybody who wants a linktype can just grab one?

And, as per my idea of using 65535 to mean "custom linktype", similar to pcapng 
custom blocks and options, with either:

the pcap file header/pcanng IDB option containing a Private Enterprise 
Number and private linktype number;

the pcap file header/pcanng IDB option containing a Private Enterprise 
Number, with any linktype specifier being in the link-level header;

the Private Enterprise Number and anything else being in the link-level 
header;

should we reserver 65535?

> I did some editing of the description field to shorten in a lot, but I got
> tired about 30% through the list, not sure if we should even include that
> column.
> There are many entries like:
>  LINKTYPE_PPP_ETHER  |   51   |PPPoE; per RFC 2516

That one's there for NetBSD; I *think* the packet contains just a PPPoE header 
and payload.  I may have to dig into the NetBSD code to see what they do.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] [pcap-ng-format] draft-gharris-opsawg-pcap.txt --- FCS length description

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Dec 21, 2020, at 4:31 PM, Michael Richardson  wrote:

> Hi, I have reworked the document that Guy put into XML describing the *PCAP*
> (not NG) format.   I found the text for LinkType to be confusing, and
> frankly, I think wrong.
> 
>   *  LinkType (32 bits): an unsigned value that defines, in the lower
>  16 bits, the link layer type of packets in the file, and
>  optionally indicates the length of the Frame Check Sequence (FCS)
>  of packets in the upper 16 bits.  The list of Standardized Link
>  Layer Type codes is available in [LINKTYPES].  If bit 5 is set,
>  bits 0 through 3 contain the length of the FCS field at the end of
>  all packets; if bit 5 is not set, the length of the FCS field at
>  the end of all packets is unknown.  Bit 4, and bits 6 through 15,
>  SHOULD be filled with 0 by pcap file writers, and MUST be ignored
>  by pcap file readers.

Perhaps that field should be called "LinkTypeandInfo", or something such as 
that, to indicate that only the lower 16 bits are the link type.  (Link-layer 
header types are shared by pcap and pcapng, and the link-layer header type in a 
pcapng Interface Description Block is 16 bits.)

> Looking at libpcap's pcap/pcap.h:
>   https://github.com/the-tcpdump-group/libpcap/blob/master/pcap/pcap.h#L217
> 
> we see:
> /*
> * Macros for the value returned by pcap_datalink_ext().
> *
> * If LT_FCS_LENGTH_PRESENT(x) is true, the LT_FCS_LENGTH(x) macro
> * gives the FCS length of packets in the capture.
> */
> #define LT_FCS_LENGTH_PRESENT(x)  ((x) & 0x0400)
> #define LT_FCS_LENGTH(x)  (((x) & 0xF000) >> 28)
> #define LT_FCS_DATALINK_EXT(x)x) & 0xF) << 28) | 
> 0x0400)
> 
> this suggests that the FCS length is really only 3 bits (maximum FCS size of
> 7 bytes?  Or does 0 indicate 8 bytes?  Ethernet is 4...).

0 indicates "no FCS present".

And, yes, the spec should indicate that.

> I see, however:
>   pcap-dag.c:
>p->linktype_ext = LT_FCS_DATALINK_EXT(pd->dag_fcs_bits/16);
> 
> I can find no other references.  So I guess Ethernet gets a value of 2 (*16 
> bits).

Yes, the length of the FCS is in 16-bit units.

And, yes, the spec should indicate that.

> I can't find any other uses.
> pcap_datalink_ext() is in pcap.c, but no the man page.
> 
> The code does not ignore bits 28:16 of the linktype field (the bits numbered
> 6:15 in the diagram).

They were originally intended for use with some stuff NetBSD was doing (I'd 
have to look into the history of the NetBSD code), but I think NetBSD stopped 
doing that.

> Since we are nowhere close to 64K link types, from looking at the pcap
> document only, we could make it 28-bits:
> BUT: pcapng has a 16-bit LinkType only, so we really need to limit 
> this to
> 16-bits OOPS.  I'll fix this in -01.
> 
> What I'm looking for in this email is:
> 1) confirmation that Linktype is 16-bits.

Yes.

> 2) some explanation of valid FCS values. Seems to be a multiple of 16-bits.
>   Is 0 valid?

Yes - it means "packets do not contain an FCS".

>  Or would that be indicated by LENGTH_PRESENT(x)==0?

*That* means "the FCS length, or whether there is an FCS, is unknown"; 
Wireshark does some heuristics to try to guess whether Ethernet packets have an 
FCS (I added those because, a long time ago, in a galaxy far far away, some 
Macs delivered Ethernet FCSes when capturing over BPF, and that messed up 
packet dissection in some cases).--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] [pcap-ng-format] draft-gharris-opsawg-pcap.txt --- FCS length description

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Dec 22, 2020, at 1:01 AM, Guy Harris  wrote:

> They were originally intended for use with some stuff NetBSD was doing (I'd 
> have to look into the history of the NetBSD code), but I think NetBSD stopped 
> doing that.

The commit message for the change that added the macros was:

commit afbb1ce7227dc5edb291f242ed8d95cd3762fc51
Author: Guy Harris 
Date:   Sat Sep 29 19:33:29 2007 +

Based on work from Florent Drouin, split the 32-bit link-layer type
field in a capture file into:

a 16-bit link-layer type field (it's 16 bits in pcap-NG, and
that'll probably be enough for the foreseeable future);

a 10-bit "class" field, indicating the group of link-layer type
values to which the link-layer type belongs - class 0 is for
regular DLT_ values, and class 0x224 grandfathers in the NetBSD
"raw address family" link-layer types;

a 6-bit "extension" field, storing information about the
capture, such an indication of whether the packets include an
FCS and, if so, how many bytes of FCS are present.

So what NetBSD had was a convention where a capture file could have a 
link-layer type that combined an AF_ value with some additional bits to 
distinguish the value from a regular LINKTYPE_ value; I don't know what AF_ 
values they supported for that, or where that code was, or whether it's still 
supported.
--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] [OPSAWG] [pcap-ng-format] draft-gharris-opsawg-pcap.txt --- IANA considerations

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Dec 22, 2020, at 8:36 AM, Michael Richardson  wrote:

> Guy Harris  wrote:
> 
>> And, as per my idea of using 65535 to mean "custom linktype", similar
>> to pcapng custom blocks and options, with either:
> 
> I'm happy with this proposal, but isn't it pcapng specific?

No - it's *cleaner* to implement in pcapng, as you can use Interface 
Description Block (IDB) options to provide the Private Enterprise Number (PEN) 
and an enterprise-specific encapsulation type, but:

if we go with the PEN and and enterprise-specific encapsulation type 
with IDB options, for pcap we can steal the former time stamp offset 
(Reserved1) and time stamp accuracy (Reserved2) fields, interpreting them as 
the PEN and enterprise-specific encapsulation type, respectively, if the link 
type is 65535;

if we go with the PEN as an IDB option, and say that if an enterprise 
wants more than one encapsulation type, they'd have to put a encapsulation type 
at the beginning of the payload, so, for pcap, we'd steal the former time stamp 
offset (Reserved1), interpreting it as the PEN if the link type is 65535;

if we go with putting the PEN and encapsulation type at the beginning 
of the payload, that would work the same way for pcap as it does for pcapng.

--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Performance impact with multiple pcap handlers on Linux

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Dec 22, 2020, at 2:05 PM, Linus Lüssing via tcpdump-workers 
 wrote:

> I was experimenting a bit with migrating from the use of
> pcap_offline_filter() to pcap_setfilter().
> 
> I was a bit surprised that installing for instance 500 pcap
> handlers

What is a "pcap handler" in this context?  An open live-capture pcap_t?

> with a BPF rule "arp" via pcap_setfilter() reduced
> the TCP performance of iperf3 over veth interfaces from 73.8 Gbits/sec
> to 5.39 Gbits/sec. Using only one or even five handlers seemed
> fine (71.7 Gbits/sec and 70.3 Gbits/sec).
> 
> Is that expected?
> 
> Full test setup description and more detailed results can be found
> here: https://github.com/lemoer/bpfcountd/pull/8

That talks about numbers of "rules" rather than "handlers".  It does speak of 
"pcap *handles*"; did you mean "handles", rather than "handlers"?

Do those "rules" correspond to items in the filter expression that's compiled 
into BPF code, or do they correspond to open `pcap_t`s?  If a "rule" 
corresponds to a "handle", then does it correspond to an open pcap_t?

Or do they correspond to an entire filter expression?

Does this change involve replacing a *single* pcap_t, on which you use 
pcap_offline_filter() with multiple different filter expressions, with 
*multiple* pcap_t's, with each one having a separate filter, set with 
pcap_setfilter()?  If so, note that this involves replacing a single file 
descriptor with multiple file descriptors, and replacing a single ring buffer 
into which the kernel puts captured packets with multiple ring buffers into 
*each* of which the kernel puts captured packets, which increases the amount of 
work the kernel does.

> PS: And I was also surprised that there seems to be a limit of
> only 510 pcap handlers on Linux.

"handlers" or "handles"?

If it's "handles", as in "pcap_t's open for live capture", and if you're 
switching from a single pcap_t to multiple pcap_t's, that means using more file 
descriptors (so that you may eventually run out) and more ring buffers (so that 
the kernel may eventually say "you're tying up too much wired memory for all 
those ring buffers").

In either of those cases, the attempt to open a pcap_t will eventually get an 
error; what is the error that's reported?--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Performance impact with multiple pcap handlers on Linux

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Dec 22, 2020, at 3:31 PM, Linus Lüssing  wrote:

> Basically we want to do live measurements of the overhead of the mesh
> routing protocol and measure and dissect the layer 2 broadcast traffic.
> To measure how much ARP, DHCP, ICMPv6 NS/NA/RS/RA, MDNS, LLDP overhead
> etc. we have.

OK, so I'm not a member of the bpf mailing list, so this message won't get to 
that list, but:

Given how general (e)BPF is in Linux, and given the number of places where you 
can add an eBPF program, and given the extensions added by the "(e)" part, it 
might be possible to:

construct a single eBPF program that matches all of those packet types;

provides, in some fashion, an indication of *which* of the packet types 
matched;

provides the packet length as well.

If you *only* care about the packet counts and packet byte counts, that might 
be sufficient if the eBPF program can be put into the right place in the 
networking stack - it would also mean that the Linux kernel wouldn't have to 
copy the packets (as it does for each PF_PACKET socket being used for 
capturing, and there's one of those for every pcap_t), and your program 
wouldn't have to read those packets.

libpcap won't help you there, as it doesn't even know about eBPF, much less 
about it's added capabilities, but it sounds as if this is a Linux-specific 
program, so that doesn't matter.  There may be a compiler allowing you to write 
a program to do what's described above and get it compiled into eBPF.

I don't know whether there's a place in the networking stack to which you can 
attach an eBPF probe to do this, but I wouldn't be surprised to find out that 
there is one.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] tcpdump build doc for Windows

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Jan 3, 2021, at 12:15 PM, Denis Ovsienko via tcpdump-workers 
 wrote:

> tcpdump source tree has a short file named "Readme.Win32", which was
> mostly updated on 8 Aug 2019, and a longer file named
> "doc/README.Win32.md", which was mostly updated on 5 Feb 2020. These
> seem to provide somewhat different instructions, perhaps it would be a
> good time to review that.

They're both reasonably up-to-date (they both mention Npcap and the use of 
CMake), but the latter is more detailed.

I'll check whether the first document says anything that's not mentioned in the 
second, and try to merge that into the second one.  Then we can probably get 
rid of the first one.

The top level README.md should probably point to doc/README.Win32.md (or 
README.Windows.md, given that 1) Windows can also be 64-bit and 2) 16-bit 
Windows is pretty much dead, so people are unlikely to get confused and say 
"OK, how do I build this 16-bit?", the answer to which is "we don't even 
support that on UN*X...".--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

[tcpdump-workers] So which is cooler - tcpdump on your wrist or tcpdump on your Mac's Touch Bar?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
$ curl -s 
https://opensource.apple.com/source/tcpdump/tcpdump-100/tcpdump.xcodeproj/project.pbxproj.auto.html
 | egrep SUPPORTED_PLATFORMS
SUPPORTED_PLATFORMS = "macosx iphoneos 
appletvos watchos bridgeos";
SUPPORTED_PLATFORMS = "macosx iphoneos 
appletvos watchos bridgeos";

(bridgeOS is the apparently-watchOS-derived OS that runs on the T-series chips 
that run the Touch Bar on Touch Bar Macs and that handle secure booting and 
possibly some other security stuff.)

I don't know whether it ships on iOS/iPadOS/tvOS/watchOS/bridgeOS or is just 
built to be used in-house.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] libpcap detection and linking in tcpdump

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Jan 7, 2021, at 9:35 AM, Bill Fenner via tcpdump-workers 
 wrote:

> These jobs are still failing, but now for a different reason.

Part of the problem is that pkg-config wasn't finding the locally-installed 
libpcap under /tmp, because PKG_CONFIG_PATH wasn't set to point to 
/tmp/lib/pkgconfig.

We're now doing that, and I re-enabled those jobs; so far, the GCC builds on 
Linux seem to be working for BUILD_LIBPCAP=yes CMAKE=yes.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] libpcap detection and linking in tcpdump

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Sep 9, 2020, at 9:07 AM, Denis Ovsienko via tcpdump-workers 
 wrote:

> The "Found pcap-config" message means that FindPCAP.cmake has not found
> libpcap by means of pkg-config, and the lack of the message means the
> pkg-config method worked. A few weeks ago Ubuntu 18.04 systems started
> to have the libpcap.pc file in the libpcap0.8-dev package, so on such a
> system "pkg-config --libs libpcap" now prints "-lpcap" and "pkg-config
> --cflags libpcap" prints an empty string, which makes sense.

It makes sense if you want to build with the *system* libpcap.

It does *not* make sense if you want to build with the libpcap that was built, 
and installed under /tmp, in the Travis build.  For *that*, you'd want 
"pkg-config --libs libpcap" to print "-L/tmp/lib -lpcap" and you'd want 
pkg-config --cflags libpcap" to print "-I /tmp/include".

That's because it's finding the libpcap.pc file in the libpcap0.8-dev package, 
not the one in /tmp/lib/pkgconfig; the latter one should do what we want.

I changed .travis.yml to run CMake with PKG_CONFIG_PATH=/tmp/lib/pkgconfig, 
which appears to make it find the right .pc file and thus to find the right 
libpcap.

> What does not make sense is that CMake seems to use non-pkg-config
> flags to tell if a specific feature is available,

It *should* be using the pkg-config flags - the code that tests for features 
just does

#
# libpcap/WinPcap/Npcap.
# First, find it.
#
find_package(PCAP REQUIRED)
include_directories(${PCAP_INCLUDE_DIRS})

cmake_push_check_state()

#
# Now check headers.
#
set(CMAKE_REQUIRED_INCLUDES ${PCAP_INCLUDE_DIRS})

#
# Check whether we have pcap/pcap-inttypes.h.
# If we do, we use that to get the C99 types defined.
#
check_include_file(pcap/pcap-inttypes.h HAVE_PCAP_PCAP_INTTYPES_H)

#
# Check for various functions in libpcap/WinPcap/Npcap.
#
cmake_push_check_state()
set(CMAKE_REQUIRED_LIBRARIES ${PCAP_LIBRARIES})

#
# Check for "pcap_list_datalinks()" and use a substitute version if
# it's not present.  If it is present, check for 
"pcap_free_datalinks()";
# if it's not present, we don't replace it for now.  (We could do so
# on UN*X, but not on Windows, where hilarity ensues if a program
# built with one version of the MSVC support library tries to free
# something allocated by a library built with another version of
# the MSVC support library.)
#
check_function_exists(pcap_list_datalinks HAVE_PCAP_LIST_DATALINKS)

...

cmake_pop_check_state()

which doesn't care whether PCAP_INCLUDE_DIRS and PCAP_LIBRARIES were set from 
pkg-config or pcap-config or manually poking the system.

> but uses pkg-config
> flags to compile and link the source,

*However*, the CMake documentation says about CMAKE_PREFIX_PATH:

Semicolon-separated list of directories specifying installation 
prefixes to be searched by the find_package(), find_program(), find_library(), 
find_file(), and find_path() commands.

and we're setting CMAKE_PREFIX_PATH, so if any of the include or library checks 
use CMAKE_PREFIX_PATH, then they might find headers for the libpcap installed 
in /tmp/libpcap, even though the build itself will use flags from the system 
libpcap.pc.

The CMake 3.19 documentation for FindPkgConfig, which is the module for using 
pkg-config:

https://cmake.org/cmake/help/v3.19/module/FindPkgConfig.html

says:

PKG_CONFIG_USE_CMAKE_PREFIX_PATH
Specifies whether pkg_check_modules() and pkg_search_module() 
should add the paths in the CMAKE_PREFIX_PATH, CMAKE_FRAMEWORK_PATH and 
CMAKE_APPBUNDLE_PATH cache and environment variables to thepkg-config search 
path.

If this variable is not set, this behavior is enabled by 
default if CMAKE_MINIMUM_REQUIRED_VERSION is 3.1 or later, disabled otherwise.

So we should either 1) require CMake 3.1 or later or 2) forcibly set 
PKG_CONFIG_USE_CMAKE_PREFIX_PATH to YES.  For now, my inclination is to do the 
latter.

Once all the other stuff I've checked in passes Travis, I'll try that instead 
of explicitly setting PKG_CONFIG_PATH. and see if that works.

> and when the system has one
> libpcap version installed as a package and another version that the
> user wants to build with, that very easily breaks (and even if it does
> not, the end result is not what the user was expecting).
> 
> Here are my steps to reproduce:
> 
> libpcap$ ./configure --enable-remote --prefix=/tmp/libpcap
> libpcap$ make
> libpcap$ make install
> tcpdumpbuild$ cmake -DWITH_CRYPTO="yes"
> -DCMAKE_PREFIX_PATH=/tmp/libpcap -DCMAKE_INSTALL_PREFIX=/tmp/libpcap
> /path/to/tcpdump_git_clone

Try that with

PKG_CONFIG_PATH=/tmp/libpcap/lib/pkgconfig cmake -DWITH_CRYPTO="yes" 
-DCMAKE_PREFIX_PATH=/tmp/libpcap -DC

Re: [tcpdump-workers] libpcap detection and linking in tcpdump

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Jan 7, 2021, at 3:21 PM, Guy Harris via tcpdump-workers 
 wrote:

> So we should either 1) require CMake 3.1 or later or 2) forcibly set 
> PKG_CONFIG_USE_CMAKE_PREFIX_PATH to YES.  For now, my inclination is to do 
> the latter.

That won't work - PKG_CONFIG_USE_CMAKE_PREFIX_PATH *isn't supported* prior to 
3.1.

3.1 dates back to 2015.  That might be sufficient to treat as a minimum.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] libpcap detection and linking in tcpdump

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Jan 7, 2021, at 5:41 PM, Denis Ovsienko via tcpdump-workers 
 wrote:

> 5 years of backward compatibility might be OK'ish, although from time
> to time I run into such "long-term support" systems that in practice
> mean someone keeps paying good money for "support" for 5-10 years, but
> they don't get bugs fixed or new software versions backported. In my
> own experience this tends to have something to do with RedHat Linux
> distributions.

Yeah, a lot of people are running old RHEL or CentOS - a lot of them keep 
asking about newer versions of Wireshark, because the older RHEL/CentOS 
versions provide Wireshark 2.x packages when Wireshark's already up to 3.4.2.

They probably provide old tcpdump, too, so people might want to build newer 
versions.

I've checked in a change that picks up some code from a newer version of the 
pkg-config CMake module to make it work with older versions.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

[tcpdump-workers] Stick with Travis for continuous integration, or switch?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
Travis CI is announcing on the travis-ci.org site that "... travis-ci.org will 
be shutting down in several weeks, with all accounts migrating to 
travis-ci.com. Please stay tuned here for more information."

They don't provide any information there.  However, at


https://travis-ci.community/t/build-delays-for-open-source-project/10272/26

they say

As was pointed out in Builds hang in queued state 3 linked to earlier 
in this topic, Travis is moving workers from travis-ci.org to travis-ci.com 1 
in preparation to fully close .org (or rather, make it read-only) around the 
New Year.

...

So you need to migrate to .com to stop experiencing delays. Note the 
caveats:

...

They claim that they'll still offer free service for free software:

Q. Will Travis CI be getting rid of free users? #

A. Travis CI will continue to offer a free tier for public or 
open-source repositories on travis-ci.com and will not be affected by the 
migration.

They also say here:

https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing

that

The upcoming pricing change will not affect those of you who are:

* Building on the Travis CI 1, 2 and 5 concurrency job plans 
who are building on Linux, Windows and experimental FreeBSD environments.
* GitHub Marketplace plans
* Grouped Accounts
* Enterprise customers (not building in our cloud environments)
* Builders on our premium or manual plans. Contact the Travis 
CI support team for more information.

but they also say that

The upcoming pricing change will affect those of you who are:

Building on the macOS environment

macOS builds need special care and attention. We want to make sure that 
builders on Mac have the highest quality experience at the fastest possible 
speeds. Therefore, we are separating out macOS usage from other build usage and 
offering a distinct add-on plan that will correlate directly to your macOS 
usage. Purchase only the credits you need and use them until you run out.

* $15 will buy you 25 000 credits (1 minute of mac build time 
costs 50 credits)
* Use your credits for macOS builds only when you need to run 
these
* Replenish your credits as you need them
* More special build environments that fall into this category 
will be available soon

which may mean that their "free tier" doesn't include macOS.

They also say:

Building on a public repositories only

We love our OSS teams who choose to build and test using TravisCI and 
we fully want to support that community. However, in recent months we have 
encountered significant abuse of the intention of this offering (increased 
activity of cryptocurrency miners, TOR nodes operators etc.). Abusers have been 
tying up our build queues and causing performance reductions for everyone. In 
order to bring the rules back to fair playing grounds, we are implementing some 
changes for our public build repositories.

* For those of you who have been building on public 
repositories (on travis-ci.com, with no paid subscription), we will upgrade you 
to our trial (free) plan with a 10K credit allotment (which allows around 1000 
minutes in a Linux environment).
* You will not need to change your build definitions when you 
are pointed to the new plan
* When your credit allotment runs out - we’d love for you to 
consider which of our plans will meet your needs.
* We will be offering an allotment of OSS minutes that will be 
reviewed and allocated on a case by case basis. Should you want to apply for 
these credits please open a request with Travis CI support stating that you’d 
like to be considered for the OSS allotment. Please include:
* Your account name and VCS provider (like 
travis-ci.com/github/[your account name] )
* How many credits (build minutes) you’d like to 
request (should your run out of credits again you can repeat the process to 
request more or discuss a renewable amount)
* Usage will be tracked under your account information so that 
you can better understand how many credits/minutes are being used

We haven't been building on travis-ci.com, so presumably the first item in the 
list doesn't apply.  If the "We will be offering an allotment..." part applies, 
the "should your run out of credits again you can repeat the process to request 
more or discuss a renewable amount" seems like a pain.

See also this comment:


https://travis-ci.community/t/org-com-migration-unexpectedly-comes-with-a-plan-change-for-oss-what-exactly-is-the-new-deal/10567/15

where the commenter says:

When I emailed support for credits, they gave this list of requirements 
for the so-called “

Re: [tcpdump-workers] libpcap detection and linking in tcpdump

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Jan 22, 2021, at 2:54 PM, Denis Ovsienko via tcpdump-workers 
 wrote:

> I have tested it again with the current master branches of libpcap and
> tcpdump. Both builds (with and without libpcap0.8-dev) now complete
> without errors.
> 
> However, in both cases the installed tcpdump fails to run because it
> is linked with libpcap.so.1. Which, as far as I can remember,
> previously somehow managed to resolve to the
> existing /tmp/libpcap/lib/libpcap.so.1, but not amymore:
> 
> $ /tmp/libpcap/bin/tcpdump --version
> /tmp/libpcap/bin/tcpdump: error while loading shared libraries:
> libpcap.so.1: cannot open shared object file: No such file or directory
> 
> $ ldd /tmp/libpcap/bin/tcpdump
>   linux-vdso.so.1 (0x7ffdc7ffe000)
>   libpcap.so.1 => not found
>   libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
> (0x7f34522ac000)
>   libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
> (0x7f3451ebb000)
>   libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
> (0x7f3451cb7000)
>   libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> (0x7f3451a98000)
>   /lib64/ld-linux-x86-64.so.2 (0x7f3452c6f000)
> 
> $ /tmp/libpcap/bin/pcap-config --libs
> -L/tmp/libpcap/lib -Wl,-rpath,/tmp/libpcap/lib -lpcap

So that *should* cause /tmp/libpcap/lib to be added to the executable's path, 
which *should* cause it to look in /tmp/libpcap/lib for shared libraries.

So, if there's a /tmp/libpcap/lib/libpcap.so.1 file, that's not happening, 
somehow.

I'll try experimenting with one of my Ubuntu VMs.

In the meantime, for some fun head-exploding reading, take a look at

https://en.wikipedia.org/wiki/Rpath

and perhaps some other documents found by a search for

lpath rpath linux--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Any way to filter ether address when type is LINUX_SLL?

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Jan 21, 2021, at 8:41 AM, Bill Fenner via tcpdump-workers 
 wrote:

> It would be perfectly reasonable (and fairly straightforward) to update
> libpcap to be able to filter on the Ethernet address in DLT_LINUX_SLL or
> DLT_LINUX_SLL2 mode.

Link-layer address, to be more accurate.

The good news is that, for Ethernet, that address appears to be the source 
address for all packets, incoming and outgoing, at least with the 5.6.7 kernel; 
I haven't checked the kernel code paths for other kernel versions.

That might also be the case for 802.11.

However, for FDDI, for example, it appears not to be set (it's marked as 
zero-length).

> There are already filters that match other offsets in
> the SLL or SLL2 header.  However, I don't think it could be done on live
> captures, only against a savefile.

At least as of 5.6.7, I don't see an SKF_ #define that would correspond to a 
link-layer address, so it appears that it's not possible to easily filter on 
the address in a live capture, at least not with an in-kernel filter.  As we're 
using cooked sockets (PF_PACKET/SOCK_DGRAM), the link-layer header isn't 
supplied to us, so we can't look at it ourselves.

I've been thinking about a world in which we have more pcapng-style APIs.  With 
a capture API that can deliver, for each packet, something similar to a pcapng 
Enhanced Packet Block, with an interface number from the capturing program can 
determine a link-layer header type, so that not all captured packet have to 
have the same link-layer header type, it might be possible to generate a filter 
program that:

could use one of the SKF_ magic offsets to fetch the "next protocol 
type" value for the protocol after the link-layer protocol, so 
link-layer-type-independent code could be used to check for common "next 
protocol type" values such as IPv4, IPv6, and ARP;

could use one of the SKF_ magic offsets to fetch the offset, relative 
to the beginning of the raw packet data, of the first byte past the link-layer 
header, so that link-layer-type-independent code could be used to check for 
anything at the next protocol layer (IP address, etc.);

could use one of the SKF_ magic offsets to fetch the ARPHRD_ type 
giving the link-layer header type, and, based on that run different code to 
check fields in the link-layer header.

This would be done by using a raw socket (PF_PACKET/SOCK_RAW) rather than a 
cooked socket.

With all of that, we could do live-capture filtering of MAC addresses (source 
*or* destination).

That's a lot of work, though.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] libpcap detection and linking in tcpdump

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Jan 22, 2021, at 7:11 PM, Guy Harris via tcpdump-workers 
 wrote:

> I'll try experimenting with one of my Ubuntu VMs.

Welcome to Shared Library Search Hell.

Most UN*Xes have a notion of RPATH (with, of course, different compiler 
command-line flags to set it).

pcap-config provides one if the shared library isn't going to be installed in 
/usr/lib.

The pkg-config file doesn't provide one, however, and some searching indicates 
that the pkg-config maintainers recommend *against* doing so.  They recommend 
using libtool when linking, instead.  Part of the problem here may be that 
setting the RPATH in an executable affects how it searches for *all* libraries, 
so it could affect which version of an unrelated library is found.

(The existence of libtool is an indication that shared libraries have gotten 
messy on UN*X.)

Perhaps for this particular case the right thing to do is to set 
LD_LIBRARY_PATH when running the temporarily-installed tcpdump.

The macOS linker appears to put absolute paths for shared libraries into the 
executable by default:

$ otool -L /bin/cat
/bin/cat:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, 
current version 1281.100.1)

so this may not be an issue there.

(Also, the existence of the term "DLL hell" is an indication that shared 
libraries have gotten messy on Windows, but I digress :-))--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Request to add MCTP and PCI_DOE to PCAP link type

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Dec 16, 2020, at 8:09 PM, Yao, Jiewen via tcpdump-workers 
 wrote:

> I write this email to request to below 2 link types.
> 
> 
>  1.  MCTP

...

> MCTP packet is defined in DMTF PMCI working group Management Component 
> Transport Protocol (MCTP) Base 
> Specification(https://www.dmtf.org/sites/default/files/standards/documents/DSP0236_1.3.1.pdf)
>  8.1 MCTP packet fields. It starts with MCTP transport header in Figure 4 - 
> Generic message fields.

So this is for MCTP messages, independent of the physical layer?

Presumably the not-a-multiple-of-8-bits fields in Table 1 go from the 
high-order bits to the low-order bits, so that the upper 4 bits of the first 
byte are the RSVD field and the lower 4 bits of the first byte are the Hdr 
version?

>  1.  PCI_DOE
> 
> PCI Data Object Exchange (DOE) is an industry standard defined by PCI-SIG 
> (https://pcisig.com/) Data Object Exchange (DOE) 
> ECN 
> (https://members.pcisig.com/wg/PCI-SIG/document/14143).

...

> PCI Data Object Exchange (DOE) is defined in PCI-SIG Data Object Exchange 
> (DOE) ECN (https://members.pcisig.com/wg/PCI-SIG/document/14143) 6.xx.1 Data 
> Objects. It starts with DOE Data Object Header 1 in Figure 6-x1: DOE Data 
> Object Format.

Unfortunately, I'm not a member of the PCI SIG, so I don't have an account to 
log in to in order to read that document.
--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Request to add MCTP and PCI_DOE to PCAP link type

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Dec 16, 2020, at 8:09 PM, Yao, Jiewen via tcpdump-workers 
 wrote:

> We did a prototype for the SpdmDump tool 
> (https://github.com/jyao1/openspdm/blob/master/Doc/SpdmDump.md). We can 
> generate a PCAP file and parse it offline.
> In our prototype, we use below definition:
> #define LINKTYPE_MCTP  290  // 0x0122
> #define LINKTYPE_PCI_DOE   291  // 0x0123
> If you can assign same number, it will be great.
> If different number is assigned, we will change our implementation 
> accordingly.

Different numbers will definitely be assigned, as 290 is already in use (in 
Wireshark, for example).  (Not everything was updated to reflect that; I've 
fixed that.)

You will probably be assigned 291 for LINKTYPE_MCTP and 292 for 
LINKTYPE_PCI_DOE; you should update your prototype for that for now.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Request for new LINKTYPE_* code LINKTYPE_AUERSWALD_LOG

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Feb 3, 2021, at 6:54 AM, developer--- via tcpdump-workers 
 wrote:

> We would like to request a dedicated LINKTYPE_* / DLT_* code.
> Auerswald is a major German telecommunications equipment manufacturer.
> We have implemented the option to capture (combined) network traffic and 
> logging information as pcap/pcapng in our soon to be released new product 
> line.
> 
> For development, we so far have used LINKTYPE_USER0 and would like to change 
> this to a proper code before the commercial release.
> 
> We also plan to publicly release the dissector and would like to make sure 
> both can be released with a proper code from the get go.
> The dissector we currently use is however only in lua.
> 
> Our preferred name would be
> LINKTYPE_AUERSWALD_LOG
> 
> If anyone is interested we can provide further information.

Please provide a detailed description of the packet format, sufficient to allow 
somebody to make a program such as tcpdump, or Wireshark, or anything else that 
reads pcap or pcapng files.
--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Request for new LINKTYPE_* code LINKTYPE_AUERSWALD_LOG

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Feb 4, 2021, at 3:41 AM, developer--- via tcpdump-workers 
 wrote:

> We currently use this code in our lua dissector to display (decoded) SIP 
> messages.
> 
> -- offsets will change with the new LINKTYPE
>if (buf(148,2):uint() == MSG_TYPE_SIP) then
>sadd("src_ip",0,16)
>sadd("src_port",16,2,"uint")
>sadd("dst_ip", 18,16)
>sadd("dst_port",34,2,"uint")
>Dissector.get("sip"):call(buf(msg_start, msg_len):tvb(), pinfo, 
> subtree)
>return
>end

In other words, the format of packets is:

IPv6 source address - 16 octets
source port - 2 octets
IPv6 destination address - 16 octets
destination port - 16 octets
SIP packet
--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Link Layer Type Request NETANALYZER_NG

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Mar 3, 2021, at 8:58 AM, Jan Adam via tcpdump-workers 
 wrote:

> for our new analysis product netANALYZER NG I would like to request a new 
> link-layer type value.
> 
> NETANALYZER_NG
> 
> The new Link-Layer-Type format is described as following:
> 
> Next-generation packet structure:
> +---+
> |   Payload |
> .   .
> .   .
> |   |
> +---+
> |   Footer  |
> |   |
> +---+
> 
> Next-gen footer description:
> 
> [16 bit]  Versionrepresents current structure version
> [64 bit]  Timestamp1 first timestamp in ns, UNIX time since 1.1.1970
> [64 bit]  Timestamp2 second timestamp in ns, UNIX time since 1.1.1970
> [32 bit]  TimestampAccuracy  actual accuracy of Timestamp1 and Timestamp2 in 
> ns. 0: actual accuracy is unknown

What do these two time stamps represent?  They presumably don't represent the 
packet arrival time, as both pcap and pcapng already provide that for all 
packets.

> [8 bit]   Representation identification of the following content

What are the possible values of this field, and what do those values signify?

> [32 bit]  SrcIdPart1 source identifier part 1
> [32 bit]  SrcIdPart2 source identifier part 2
> [8 bit]   SrcIdPart3 source identifier part 3
> [8 bit]   SrcIdPart4 source identifier part 3

So there's an 80-bit source identifier; what does that value signify?

> [64 bit]  VarId  variable identifier
> [64 bit]  VarState   variable error states, depending on 
> representation
> [8 bit]   VarTypevariable data type

What do those signify?

> [32 bit]  VarSizesize of raw frame payload

Presumably everything beyond that size is the footer; what are the contents of 
the footer?
--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] continuous integration status update

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Mar 3, 2021, at 2:30 PM, Denis Ovsienko via tcpdump-workers 
 wrote:

> A partial replacement for that service is ci.tcpdump.org, which is a
> buildbot instance doing Linux AArch64 builds for the github.com
> repositories.

So where is that hosted?  Are you hosting it yourself or hosting it on some 
cloud service?
--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Link Layer Type Request NETANALYZER_NG

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Mar 8, 2021, at 12:07 AM, Jan Adam via tcpdump-workers 
 wrote:

> We have created a public document on our website You can point to for the 
> description.
> 
> Here is the link:  https://kb.hilscher.com/x/brDJBw
> 
> It contains a more detailed description of the fields in the footer structure.
> It also contains a C – like structure definition of the footer.

So is "the variable" the same thing as "the payload"?--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Link Layer Type Request NETANALYZER_NG

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Mar 12, 2021, at 4:35 AM, Jan Adam  wrote:

>> So is "the variable" the same thing as "the payload"?
> 
> That is correct. To be more specific the payload is the value/content of the 
> variable.

Can the variable be anything *other* than a packet of some sort?  The current 
set of values for the variable listed in https://kb.hilscher.com/x/brDJBw:

0x01:   netANALYZER legacy frame
0x02:   Ethernet (may also be a re-assembled mpacket)
0x03:   mpacket
0x04:   PROFIBUS frame
0x05:   IO-Link frame

lists only packets of various types, but I was reading "variable" in the 
programming language sense, rather than in the sense that the total content has 
a "fixed part", that being the trailer, and a "variable part", that being the 
packet preceding the trailer.  Is the latter the sense in which the word 
"variable" should be understood?

It also appears that the boundary between the payload and the trailer would be 
determined by fetching the VarSize field at the end of the trailer.  The first 
VarSize bytes of the data would be the payload, and the remaining 
sizeof(footer) bytes would be the trailer.  Is that the case?

That would also indicate that the "captured length" value for a pcap record or 
a pcapng block containing NETANALYZER_NG data must be >= sizeof(footer), so 
that the entire footer is present.

This also means that NETANALYZER_NG data must *not* be cut off at the end by 
any "slicing" process, such as capturing with a "slice length"/"snapshot 
length".  Is it possible that the frame in the payload is "sliced" in that 
fashion?--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

[tcpdump-workers] ASAN - now for Windows

[Guy Harris via tcpdump-workers]

--- Begin Message ---

https://devblogs.microsoft.com/cppblog/address-sanitizer-for-msvc-now-generally-available/--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Request for new LINKTYPE_* code LINKTYPE_AUERSWALD_LOG

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Feb 12, 2021, at 4:49 AM, developer--- via tcpdump-workers 
 wrote:

> Sorry for the delay in responding, I had to look thru the code to make sure 
> all the information is accurate.
> 
> I looked into the option to change in particular the  msg_type 1 (decoded SIP 
> message),
> but for the short term, using export_pdu is not an option because of the 
> changes required.
> 
> The current implementation for the LINKTYPE_AUERSWALD_LOG uses the following 
> format:
> 
> Each package starts with a message header followed by the payload and a 0 
> byte.
> The message header currently has a fixed width of 150 bytes.
> For the future the length of the header might change based on the msg_type.
> (This will allow for less waste compared to the fix length)
> Whenever the content structure of the message or length of header is changed
> a new msg_type will be used.
> 
> /* Literals for displaying level are
> local level_tab = {
>  [0] = "Off",
>  [1] = "Error",
>  [2] = "Warning",
>  [3] = "Info",
>  [4] = "Debug",
>  [5] = "EDebug"
> }
> */
> 
> // The Offset in comments is just for convenience for writing wireshark 
> dissectors
> struct message_hdr {
>uint16_t msg_type;  // Offset 0
>uint32_t level; // Offset 2
>char category[32];  // Offset 6
>  union {   // New Offsetbase 38
>struct {// if msg_type == 0
>  int32_t  pid; // Offset 38
>  int32_t  tgid;// Offset 42
>  char procname[16];// Offset 46
>  char threadname[16];  // Offset 62
>  char class_name[32];  // Offset 78
>  char method[32];  // Offset 110
>  uint32_t lineno;  // Offset 142
>  uint32_t seqno;   // Offset 146
>};
>struct {// if msg_type == 1
>  char src_ip[16];  // Offset 38
>  uint16_t src_port;// Offset 54
>  char dst_ip[16];  // Offset 56
>  uint16_t dst_port;// Offset 72
>};
>  };
> } __attribute__((packed));// Total 150

So what are the byte orders of the 2-byte and 4byte integral fields?

What are the values that the msg_type field can have?

Presumably the values that the level field can have are 0-5, with the meanings 
indicated.

Presumably category is a string; what does that string signify?

What do the values signify in a message of message type 0?  They appear to 
indicate a thread within a process; do they also indicate the sending class and 
method?  Is linen a line of source code in the implememtation of that method, 
or something else?  What is the tgid? What is the sequence number?

What is the payload following the header?--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Link Layer Type Request NETANALYZER_NG

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Mar 15, 2021, at 9:04 AM, Jan Adam  wrote:

>> Can the variable be anything *other* than a packet of some sort?
> 
> There are only the mentioned 5 representations planned for pcap files since 
> this is what our capture device may capture into a pcap file. The 
> representation gives at least the ability to extend in the future. Do you 
> have anything specific in mind?

No.

>> It also appears that the boundary between the payload and the trailer would 
>> be determined by fetching the VarSize field at the end of the trailer.  The 
>> first VarSize bytes of the data would be the payload, and the remaining 
>> sizeof(footer) bytes would be the trailer.  Is that the case?
> 
> This is also correct. The remaining bytes of incl_len - VarSize is the footer 
> size.

If the fields of the footer are aligned on natural boundaries, the footer will 
be 72 bytes long; if they are *not* aligned, the footer will be 53 bytes long.

Are they aligned on natural boundaries?

Presumably VarSize is the same thing as PayloadSize?  If so, then presumably 
incl_len must be equal to VarSize + {either 53 or 72}.

> Some fields of the footer (like the ID) may seem to be redundant and not of 
> much purpose in the wireshark or tcpdump context but we use the footer 
> structure everywhere in our software stack. This way we eliminated converting 
> structures between different parts of our software when dealing with captured 
> data.

So what do the two time stamps indicate for the various various of 
Representation?

What do the four fields of the SrcID indicate for the various values of 
Representation?

What do the values of PayloadState indicate for the various values of 
Representation?

What other possible values of PayloadType are there?

>> This also means that NETANALYZER_NG data must *not* be cut off at the end by 
>> any "slicing" process, such as capturing with a "slice length"/"snapshot 
>> length".  Is it possible that the frame in the payload is "sliced" in that 
>> fashion?
> 
> Slicing a captured packet is not supported by our capturing device.

But some software can slice packets afterwards.  Either that would have to be 
forbidden (meaning editcap and, I think, tcpdump would have to check for 
LINKTYPE_NETANALYZER_NG/DLT_NETANALYZR_NG and refuse to do slicing), or they 
would have to 1) ensure that the slice size is >= the footer size and 2) do the 
slicing specially, removing bytes *before* the footer, so that if incl_len < 
VarSize + footer_size, (VarSize + footer_size) - incl_len bytes have been 
sliced off.
--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Link Layer Type Request NETANALYZER_NG

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Mar 22, 2021, at 7:33 AM, Jan Adam  wrote:

>> Are they aligned on natural boundaries?
> 
> No, it is not aligned but packet.  We use #pragma pack(1) for the footer 
> structure.

You should probably add that to the page with the structure definition.

>> What do the four fields of the SrcID indicate for the various values of 
>> Representation?
> 
> For Representation 0x01 to 0x05 their meaning is defined as following:
> tSrcId.ulPart1netANALYZER device number
> tSrcId.ulPart2netANALYZER serial number
> tSrcId.bPart4netANALYZER port number
> 
> For Representation 0x02 to 0x05
> tSrcId.bPart3netANALYZER TAP name (as character, e.g. 'A' = 0x41 or 'B')
> 
> For Representation 0x01
> tSrcId.bPart3netANALYZER TAP number

That should also be noted in the specification.

>> What other possible values of PayloadType are there?
> 
> The PayloadType has the following possible values but they are not usefull 
> for capturing network traffic. So the only value in the context of packet 
> data will be 0x0A which represents DATATYPE_OCTET_STRING.
> 
> #define VAR_DATATYPE_BOOLEAN0x01

...

> #define VAR_DATATYPE_NONE   0xff

It should also note that the other values are reserved and will not appear in 
pcap or pcapng files.

>>> Slicing a captured packet is not supported by our capturing device.
> 
>> But some software can slice packets afterwards.  Either that would have to 
>> be forbidden (meaning editcap and, I think, tcpdump would have to check for 
>> LINKTYPE_NETANALYZER_NG/DLT_NETANALYZR_NG and refuse to do slicing), or they 
>> would have to 1) ensure that the slice size is >= the footer size and 2) do 
>> the slicing specially, removing bytes *before* the footer, so that if 
>> incl_len < VarSize + footer_size, (VarSize + footer_size) - incl_len bytes 
>> have been sliced off.
> 
> Both might be possible path to take for slicing. In any case the PayloadSize 
> should also be adjusted when the payload length is changed in my opinion. Is 
> this a Problem?

So, with incl_len equal to {PayloadSize,VarSize} + 54, orig_len would be equal 
to {original PayloadSize} + 54, so the original payload size would be orig_len 
- 54.

That would allow the original size and the sliced size of the payload to be 
calculated, so that should work.

--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] ARM build slaves (tcpdump mirror in Germany)

[Guy Harris via tcpdump-workers]

--- Begin Message ---
On Mar 22, 2021, at 5:35 PM, Denis Ovsienko via tcpdump-workers 
 wrote:

> On Mon, 22 Mar 2021 19:00:31 +0100
> Harald Welte  wrote:

...

>> btw: I'm not sure if qemu full system emulation of e.g. ppc on a
>> x86_64 hardware would be an option, though.  I think
>> openbuildservice.org is doing that a lot for building packages on
>> less popular architectures.
> 
> QEMU was very useful for the NetBSD setup. NetBSD for some reason did
> not provide binary packages for 9.1/aarch64, and heavy non-default
> packages (LLVM, Clang, GCC 10) just do not compile on 1GB RAM of RPI3B
> (NetBSD release does not run on RPI4B), so the only way to compile
> these was in a QEMU VM with more RAM.
> 
> That said, on a Linux host with i7-3770 CPU the QEMU guest measured at
> 64% core-to-core CPU performance of an RPI3B. So after the initial
> setup a hardware Pi does a better job.

The main PowerPC/Power ISA buildbot we'd want would probably be ppcle64, as the 
ppcle64 implementation of some crypto library routines, as used by tcpdump, 
require strict adherence to the API documentation, e.g. 1) don't use the same 
buffer for encrypted and decrypted data and 2) provide all the necessary 
padding in the input buffer and leave enough room in the output buffer, as per

https://github.com/the-tcpdump-group/tcpdump/issues/814

64% isn't perfect, but it's a lot better than 10%, so if QEMUs' PPC64/64-bit 
Power ISA emulation supports both big-endian and little-endian mode, and runs 
with acceptable performance (anything in the range of 50% is probably good 
enough), and the emulation is faithful enough (which being able to boot ppc64le 
Linux would probably imply), that would probably be sufficient.

Having *some* big-endian machine would be useful primarily for tcpdump testing, 
to make sure there's no code that implicitly assumes it's running on a 
little-endian machine (which most developers probably have); any of SPARC, 
ppcbe, or s390/s390x would suffice for that.

SPARC has the additional advantage of trapping on unaligned accesses, so it'll 
also detect code that implicitly assumes that unaligned accesses work.  S/3x0 
hasn't required alignment since S/370 came out (unaligned accesses were an 
optional feature of S/360, but were made a standard feature in S/370), and I'm 
not sure PPC requires it.  We already have SPARC/Solaris 10 testing with 
OpenCSW, so that will fail on unaligned accesses; the only thing additional 
buildbots would do would be to give us Solaris 11 and Linux.--- End Message ---
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers