[tcpdump-workers] Regarding tcpdump pull request #614

[alice-cyberreboot]

Hi everyone!

I’m writing regarding a pull request I submitted (#614).

My workgroup is currently working on a project utilizing machine-learning and 
software-defined networking to detect and respond to malicious network 
activity. We are currently focused on internal Ethernet traffic, and one of our 
big challenges is capturing enough (network) data to sufficiently train our 
models. We are working with a number of organizations that wish to share data 
but want some basic levels of sanitization. Lack of modern, internal and benign 
traffic is a challenge for data science teams.

In order to better facilitate data sharing between collaborating organizations, 
we attempted to address some common privacy/sensitivity issues by expanding 
tcpdump to create the following options:

-  Strip out the packet payload after TCP/UDP headers; and

-  Mask external IP addresses (i.e., those not included in the RFC 5735 
reserved netblocks).

We have been using our modifications internally and they appear to be stable. 
Our initial testing using machine learning based on this approach was pretty 
successful, and we would like to open up our research to collaboration with 
other entities. Tcpdump is so common in our circles that when we suggested 
enhancing it everyone we work with agreed it was a great option. Our proposed 
modification performs the above operations when writing to a savefile. The two 
flags that I’ve added were:

-  -0 to zero out packet data after TCP/UDP headers

-  -00 to truncate the packet data entirely (this saves space for large 
packet captures)

-  -* [mask_ip] to mask external IP addresses with a user-specified IP.

In our enhancements these flags are available both when reading from an 
existing pcap file and when performing a live capture. The caveats are, this 
currently works solely for the Ethernet link layer (the scope of our project), 
the IPv6 protocol has not yet been supported, and it does not work when 
printing to screen (although the user will be warned at the outset). However, 
my workgroup would love to open this up to the rest of the open source 
community to facilitate broader information sharing and make network 
collections more accessible to data scientists.

If there are other enhancements that might be helpful toward this topic, please 
let me know!


Thanks,
Alice
(@lilchurro on github)

P.S. If folks are curious, we have published some of our work, including:
https://blog.cyberreboot.org/deep-session-learning-for-cyber-security-e7c0f6804b81


--
🙋 Alice Chang
👾 Cyber Reboot Software Engineer @ In-Q-Tel




"This e-mail, and any attachments hereto, may contain information that is 
privileged, proprietary, confidential and/or exempt from disclosure under law 
and are intended only for the designated addressee(s). If you are not the 
intended recipient of this message, or a person authorized to receive it on 
behalf of the intended recipient, you are hereby notified that you must not 
use, disseminate, copy in any form, or take any action based upon the email or 
information contained therein. If you have received this email in error, please 
permanently and immediately delete it and any copies of it, including any 
attachments, and promptly notify the sender at In-Q-Tel by reply e-mail, fax: 
703-248-3001, or phone: 703-248-3000. Thank you for your cooperation."
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Regarding tcpdump pull request #614

[alice-cyberreboot]

Good point!  Will fix this now.

---
🙋 al...@cyberreboot.org
👾 Cyber Reboot Software Engineer @ In-Q-Tel


-Original Message-
From: Guy Harris [mailto:g...@alum.mit.edu]
Sent: Tuesday, July 18, 2017 4:41 PM
To: alice-cyberreboot 
Cc: tcpdump-workers@lists.tcpdump.org
Subject: Re: [tcpdump-workers] Regarding tcpdump pull request #614

On Jul 18, 2017, at 12:44 PM, alice-cyberreboot  wrote:

> In our enhancements these flags are available both when reading from an 
> existing pcap file and when performing a live capture. The caveats are, this 
> currently works solely for the Ethernet link layer (the scope of our project),

So if you request the anonymization for any device with a DLT_ other than 
DLT_EN10MB, it'll fail and print a "that only works for Ethernet" message?

Otherwise, if it *doesn't* fail in that case, it's buggy and needs to be fixed.



"This e-mail, and any attachments hereto, may contain information that is 
privileged, proprietary, confidential and/or exempt from disclosure under law 
and are intended only for the designated addressee(s). If you are not the 
intended recipient of this message, or a person authorized to receive it on 
behalf of the intended recipient, you are hereby notified that you must not 
use, disseminate, copy in any form, or take any action based upon the email or 
information contained therein. If you have received this email in error, please 
permanently and immediately delete it and any copies of it, including any 
attachments, and promptly notify the sender at In-Q-Tel by reply e-mail, fax: 
703-248-3001, or phone: 703-248-3000. Thank you for your cooperation."
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

Re: [tcpdump-workers] Regarding tcpdump pull request #614

[alice-cyberreboot]

Should be fixed now; I created the new pull request #615 and closed #614, so 
that I could use a different branch moving forward.
Thanks for the guidance, Guy!

---
🙋 al...@cyberreboot.org
👾 Cyber Reboot Software Engineer @ In-Q-Tel


-Original Message-
From: Guy Harris [mailto:g...@alum.mit.edu]
Sent: Tuesday, July 18, 2017 4:41 PM
To: alice-cyberreboot 
Cc: tcpdump-workers@lists.tcpdump.org
Subject: Re: [tcpdump-workers] Regarding tcpdump pull request #614

On Jul 18, 2017, at 12:44 PM, alice-cyberreboot  wrote:

> In our enhancements these flags are available both when reading from an 
> existing pcap file and when performing a live capture. The caveats are, this 
> currently works solely for the Ethernet link layer (the scope of our project),

So if you request the anonymization for any device with a DLT_ other than 
DLT_EN10MB, it'll fail and print a "that only works for Ethernet" message?

Otherwise, if it *doesn't* fail in that case, it's buggy and needs to be fixed.



"This e-mail, and any attachments hereto, may contain information that is 
privileged, proprietary, confidential and/or exempt from disclosure under law 
and are intended only for the designated addressee(s). If you are not the 
intended recipient of this message, or a person authorized to receive it on 
behalf of the intended recipient, you are hereby notified that you must not 
use, disseminate, copy in any form, or take any action based upon the email or 
information contained therein. If you have received this email in error, please 
permanently and immediately delete it and any copies of it, including any 
attachments, and promptly notify the sender at In-Q-Tel by reply e-mail, fax: 
703-248-3001, or phone: 703-248-3000. Thank you for your cooperation."
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

[tcpdump-workers] Pull request # 615.

[alice-cyberreboot]

Hi there!

I finally got around to fixing my cloned repo again to get the TravisCI compile 
tests working, and have added some tests for my requested flags. I was 
wondering if I could get feedback on my branch, or if it was good to be pulled. 
It’s been merged with the latest version of your master branch.

Thanks!

Best,
Alice
---
🙋 al...@cyberreboot.org
👾 Cyber Reboot Software Engineer @ In-Q-Tel




"This e-mail, and any attachments hereto, may contain information that is 
privileged, proprietary, confidential and/or exempt from disclosure under law 
and are intended only for the designated addressee(s). If you are not the 
intended recipient of this message, or a person authorized to receive it on 
behalf of the intended recipient, you are hereby notified that you must not 
use, disseminate, copy in any form, or take any action based upon the email or 
information contained therein. If you have received this email in error, please 
permanently and immediately delete it and any copies of it, including any 
attachments, and promptly notify the sender at In-Q-Tel by reply e-mail, fax: 
703-248-3001, or phone: 703-248-3000. Thank you for your cooperation."
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

[tcpdump-workers] Packet sanitization and IP masking (PR #615)

[alice-cyberreboot]

Hello all,

In my attempt to get this pull request toward completion, I intend to rebase 
the branch’s commit history to one concise commit. Before I do so, however, I 
would like to elicit some suggestions (if any) regarding my proposed long 
options.

There are currently two main features to this PR: a masking of external IP 
addresses to some mask IP, and TCP/UDP payload sanitization in IPv4. Calling 
the former currently looks like [ --external-mask|-* mask_ip ] but I am 
currently planning to remove the -* short opt and change the long one to 
--mask-external-address. As for the latter feature, the packet sanitization 
works by using -0 to zero out the payload, and -00 to remove it completely. I’m 
planning to change this to --zero-tcpudp-payload and --no-tcpudp-payload, 
respectively. (The result will be that none of these features will be using a 
short opt.) Are there any objections and suggestions to these names?

Additionally, I know that there’s a lot of work being done to fix CVE issues. I 
was wondering if/how I could be of help in that effort.

Thanks! And thanks to the maintainers for their guidance thus far.

Alice

---
🙋 al...@cyberreboot.org
👾 Cyber Reboot Software Engineer @ In-Q-Tel





"This e-mail, and any attachments hereto, may contain information that is 
privileged, proprietary, confidential and/or exempt from disclosure under law 
and are intended only for the designated addressee(s). If you are not the 
intended recipient of this message, or a person authorized to receive it on 
behalf of the intended recipient, you are hereby notified that you must not 
use, disseminate, copy in any form, or take any action based upon the email or 
information contained therein. If you have received this email in error, please 
permanently and immediately delete it and any copies of it, including any 
attachments, and promptly notify the sender at In-Q-Tel by reply e-mail, fax: 
703-248-3001, or phone: 703-248-3000. Thank you for your cooperation."
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

[tcpdump-workers] Packet sanitization and IP masking (PR #615)

[alice-cyberreboot]

Resending, as it didn't seem to make the list the first time...




Hello all,

In my attempt to get this pull request toward completion, I intend to rebase 
the branch's commit history to one concise commit. Before I do so, however, I 
would like to elicit some suggestions (if any) regarding my proposed long 
options.

There are currently two main features to this PR: a masking of external IP 
addresses to some mask IP, and TCP/UDP payload sanitization in IPv4. Calling 
the former currently looks like [ --external-mask|-* mask_ip ] but I am 
currently planning to remove the -* short opt and change the long one to 
--mask-external-address. As for the latter feature, the packet sanitization 
works by using -0 to zero out the payload, and -00 to remove it completely. I'm 
planning to change this to --zero-tcpudp-payload and --no-tcpudp-payload, 
respectively. (The result will be that none of these features will be using a 
short opt.) Are there any objections and suggestions to these names?

Additionally, I know that there's a lot of work being done to fix CVE issues. I 
was wondering if/how I could be of help in that effort.

Thanks! And thanks to the maintainers for their guidance thus far.

Alice




"This e-mail, and any attachments hereto, may contain information that is 
privileged, proprietary, confidential and/or exempt from disclosure under law 
and are intended only for the designated addressee(s). If you are not the 
intended recipient of this message, or a person authorized to receive it on 
behalf of the intended recipient, you are hereby notified that you must not 
use, disseminate, copy in any form, or take any action based upon the email or 
information contained therein. If you have received this email in error, please 
permanently and immediately delete it and any copies of it, including any 
attachments, and promptly notify the sender at In-Q-Tel by reply e-mail, fax: 
703-248-3001, or phone: 703-248-3000. Thank you for your cooperation."
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers

[tcpdump-workers] Packet sanitization and IP masking (PR #615)

[alice-cyberreboot]

Hello again,

Here's a new update/summary of my PR:

-  Removed short options in favor of long ones for three features - 
zeroing out TCP/UDP payload in IPv4 packets (--zero-tcpudp-payload), removing 
said payloads completely (--no-tcpudp-payload), and masking external IP 
addresses to a given substitute IP (--mask-external-address mask_ip);

-  manpage documentation has been updated;

-  commits related to this PR have been consolidated into one commit 
message;

-  at the moment, currently up to date with upstream/master, including 
modifications resulting from ether.h being unavailable.

Hope that at this stage, the PR is now ready for proper review.

And again, if I can assist with work on fixing CVE-related issues, please let 
me know.


Thanks,
@lilchurro



"This e-mail, and any attachments hereto, may contain information that is 
privileged, proprietary, confidential and/or exempt from disclosure under law 
and are intended only for the designated addressee(s). If you are not the 
intended recipient of this message, or a person authorized to receive it on 
behalf of the intended recipient, you are hereby notified that you must not 
use, disseminate, copy in any form, or take any action based upon the email or 
information contained therein. If you have received this email in error, please 
permanently and immediately delete it and any copies of it, including any 
attachments, and promptly notify the sender at In-Q-Tel by reply e-mail, fax: 
703-248-3001, or phone: 703-248-3000. Thank you for your cooperation."
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers