On 2011-06-23 11:52, Stuart Henderson wrote:
On 2011-06-23, Magnus Rixtorp<[email protected]> wrote:
pass out quick log on $ext_if inet from 192.168.0.0/24 nat-to $ext_if
pass out quick log on $ext_if inet from 192.168.230.0/24 nat-to $ext_if
pass out quick log on $ext_if inet from 192.168.231.0/24 nat-to $ext_if
pass out quick log on $ext_if inet from 192.168.239.0/24 nat-to $ext_if
pass out quick log on $ext_if inet from 192.168.240.0/24 nat-to $ext_if
pass out quick log on $ext_if inet from 192.168.241.0/24 nat-to $ext_if
pass out quick log on $ext_if inet from 192.168.242.0/24 nat-to $ext_if
This probably isn't your problem, but that seems quite a lot of networks
to be natting behind a single IP especially with the default port range
(50001:65535). if you've got a lot of active natted states, the
search for a free port could involve a bunch of state searches
(pick a random port, lookup state to see if it's used, then search
sequentially for a free port looking up state each time).
So if you do have a lot of states you might want to either add more IPs
or increase the port range available (e.g. pass...nat-to $ext_if \
port 20000:65535) and adjust the net.inet.ip.port* sysctls for
connections coming from the firewall itself (to make sure you have
some free ports which don't conflict with the range used by that
PF rule).
No, thats not a real issue, since there may be alot of netowrks/ips in
those nats, but there is only 1-2 active hosts on those networks.
Jun 15 09:41:21 pbxfw /bsd: pf: state key linking mismatch! dir=OUT,
if=re0, stored af=2, a0: 130.244.190.46:5060, a1: 192.168.230.101:5060,
proto=17, found af=2, a0: 192.168.230.101:5060, a1:
187.170.255.239:5060, proto=17
Jun 17 12:02:55 pbxfw /bsd: pf: state key linking mismatch! dir=OUT,
if=re0, stored af=2, a0: 130.244.190.46:5060, a1: 192.168.230.101:5060,
proto=17, found af=2, a0: 192.168.230.101:5060, a1:
187.170.255.239:5060, proto=17
Is the only error output ive found on the problem.
So the problem, has to do with the ip 187.170.255.239,
239.255.170.187.in-addr.arpa domain name pointer
dsl-187-170-255-239-dyn.prod-infinitum.com.mx.
Our system has no relation at all with this ip.
But somehow our NAT translation at random intervals, decides to
redirects traffic to that ip instead of the intended destination.
Sofar we have primarily noted the problem towards 130.244.190.46 and
130.244.190.42, that are our providers sip gateways.
Since the only thing beeing used on the connection is a PBx solution.
A google on that perticular IP, gives a simular dmesg error output in
this post:
http://www.mail-archive.com/[email protected]/msg95116.html
But in his case, the system hangs, our system keeps on going.
And instead interferes with the connection of phonecalls.
since the problem was discovered ive set up pf to log the first packet
of every new state,
and then that is tcpdump thru tcpdump -n -e -ttt -s 1600 -vvv -XX to a
ascii log using the
http://www.openbsd.org/faq/pf/logging.html syslog method.
Jun 22 15:40:06.212694 rule 26/(match) [uid 0, pid 20284] pass in on
bge0: 130.244.190.46.5060> 212.247.80.66.5060: udp 442 (DF) [tos 0xb8]
(ttl 56, id 0, len 470)
0000: 45b8 01d6 0000 4000 3811 da02 82f4 be2e
E\M-8.\[email protected].\M-Z..\M-t\M->.
0010: d4f7 5042 13c4 13c4 01c2 f6b9 4259 4520
\M-T\M-wPB.\M-D.\M-D.\M-B\M-v\M-9BYE
0020: 7369 703a 3835 3933 4032 3132 2e32 3437 sip:[email protected]
0030: 2e38 302e 3636 2053 4950 2f32 .80.66 SIP/2
Jun 22 15:40:06.307515 rule 60/(match) [uid 0, pid 20284] pass in on
re0: 192.168.230.101.5060> 187.170.255.239.5060: udp 550 (ttl 64, id
33961, len 578)
0000: 4500 0242 84a9 0000 4011 9159 c0a8 e665
E..B.\M-)[email protected]\M-@\M-(\M-fe
0010: bbaa ffef 13c4 13c4 022e 9dc3 5349 502f
\M-;\M-*\M^?\M-o.\M-D.\M-D...\M-CSIP/
0020: 322e 3020 3230 3020 4f4b 0d0a 5669 613a 2.0 200 OK..Via:
0030: 2053 4950 2f32 2e30 2f55 4450 SIP/2.0/UDP
Considering this snippet alone, there's no indication of a problem
with PF; it looks to me like 192.168.230.101 is itself sending
packets to 187.170.255.239, maybe your PBX software is confused.
I would look at packets on the inbound/outbound interfaces rather
than pflog and see what addresses show up there. ("tcpdump -Xs1500
-nire0 port 5060" or something, and same for bge0).
The xxx.255.239 makes me wonder if the PBX is trying to do some
multicast thing and getting the byte-order wrong (239.255.xxx would
be a multicast address).
I have been taking a closer look on the packets, both on the external
bge0 itnerface,
and the internal re0.
And the problem happens when the packet transfers thru pf from bge0 to re0.
using the rule
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
5060 rdr-to 192.168.230.101
the packet on bge0 looks like this:
No. Time Source Destination Protocol
Length Info
6983 130.535506 130.244.190.42 212.247.80.66
SIP/SDP 1081 Request: INVITE sip:[email protected];user=phone,
with session description
Frame 6983: 1081 bytes on wire (8648 bits), 1081 bytes captured (8648 bits)
Ethernet II, Src: Cisco_79:39:0a (00:1d:a1:79:39:0a), Dst: Dell_7c:f3:91
(00:11:43:7c:f3:91)
Destination: Dell_7c:f3:91 (00:11:43:7c:f3:91)
Address: Dell_7c:f3:91 (00:11:43:7c:f3:91)
.... ...0 .... .... .... .... = IG bit: Individual address
(unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Source: Cisco_79:39:0a (00:1d:a1:79:39:0a)
Address: Cisco_79:39:0a (00:1d:a1:79:39:0a)
.... ...0 .... .... .... .... = IG bit: Individual address
(unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Type: IP (0x0800)
Internet Protocol Version 4, Src: 130.244.190.42 (130.244.190.42), Dst:
212.247.80.66 (212.247.80.66)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0xb8 (DSCP 0x2e: Expedited
Forwarding; ECN: 0x00: Not-ECT (Not ECN-Capable Transport))
1011 10.. = Differentiated Services Codepoint: Expedited
Forwarding (0x2e)
.... ..00 = Explicit Congestion Notification: Not-ECT (Not
ECN-Capable Transport) (0x00)
Total Length: 1067
Identification: 0x0000 (0)
Flags: 0x02 (Don't Fragment)
Fragment offset: 0
Time to live: 56
Protocol: UDP (17)
Header checksum: 0xd7b1 [correct]
Source: 130.244.190.42 (130.244.190.42)
Destination: 212.247.80.66 (212.247.80.66)
User Datagram Protocol, Src Port: sip (5060), Dst Port: sip (5060)
Source port: sip (5060)
Destination port: sip (5060)
Length: 1047
Checksum: 0x1030 [validation disabled]
Session Initiation Protocol
Request-Line: INVITE sip:[email protected];user=phone SIP/2.0
Method: INVITE
Request-URI: sip:[email protected];user=phone
Request-URI User Part: 0424507401
Request-URI Host Part: 212.247.80.66
[Resent Packet: False]
Message Header
Record-Route: <sip:130.244.190.42;lr;ftag=469095268>
Via: SIP/2.0/UDP 130.244.190.42;branch=z9hG4bKe904.69d17a12.0
Transport: UDP
Sent-by Address: 130.244.190.42
Branch: z9hG4bKe904.69d17a12.0
Via: SIP/2.0/UDP
212.151.144.8:5060;branch=z9hG4bKterm-785897-0424507401-0707957245-5102
Transport: UDP
Sent-by Address: 212.151.144.8
Sent-by port: 5060
Branch: z9hG4bKterm-785897-0424507401-0707957245-5102
From: 0707957245
<sip:[email protected]:5060;user=phone>;tag=469095268
SIP Display info: 0707957245
SIP from address: sip:[email protected]:5060;user=phone
SIP from address User Part: 0707957245
SIP from address Host Part: 212.151.144.8
SIP from address Host Port: 5060
SIP tag: 469095268
To: 0424507401
<sip:[email protected]:5060;user=phone>
SIP Display info: 0424507401
SIP to address:
sip:[email protected]:5060;user=phone
SIP to address User Part: 0424507401
SIP to address Host Part: sip-corporate.tele2.se
SIP to address Host Port: 5060
Call-ID: [email protected]
CSeq: 1 INVITE
Sequence Number: 1
Method: INVITE
Max-Forwards: 69
Supported: timer
Session-Expires: 1800
Min-SE: 1800
Contact: 0707957245 <sip:[email protected]:5060>
SIP Display info: 0707957245
Contact-URI: sip:[email protected]:5060
Contactt-URI User Part: 0707957245
Contact-URI Host Part: 212.151.144.8
Contact-URI Host Port: 5060
Allow:
INVITE,ACK,PRACK,SUBSCRIBE,BYE,CANCEL,NOTIFY,INFO,REFER,UPDATE
P-Asserted-Identity: 0707957245
<sip:[email protected];user=phone>
SIP Display info: 0707957245
SIP PAI Address: sip:[email protected];user=phone
SIP PAI User Part: 0707957245
SIP PAI Host Part: 212.151.144.8
Content-Type: application/sdp
Content-Length: 225
Message Body
Session Description Protocol
Session Description Protocol Version (v): 0
Owner/Creator, Session Id (o): - 52119343 0 IN IP4 130.244.1.7
Owner Username: -
Session ID: 52119343
Session Version: 0
Owner Network Type: IN
Owner Address Type: IP4
Owner Address: 130.244.1.7
Session Name (s): Cisco SDP 0
Connection Information (c): IN IP4 130.244.1.7
Connection Network Type: IN
Connection Address Type: IP4
Connection Address: 130.244.1.7
Time Description, active time (t): 0 0
Session Start Time: 0
Session Stop Time: 0
Media Description, name and address (m): audio 17070
RTP/AVP 8 0 99 102 101
Media Type: audio
Media Port: 17070
Media Protocol: RTP/AVP
Media Format: ITU-T G.711 PCMA
Media Format: ITU-T G.711 PCMU
Media Format: DynamicRTP-Type-99
Media Format: DynamicRTP-Type-102
Media Format: DynamicRTP-Type-101
Media Attribute (a): rtpmap:99 G.729a/8000
Media Attribute Fieldname: rtpmap
Media Format: 99
MIME Type: G.729a
Sample Rate: 8000
Media Attribute (a): rtpmap:102 G.729b/8000
Media Attribute Fieldname: rtpmap
Media Format: 102
MIME Type: G.729b
Sample Rate: 8000
Media Attribute (a): rtpmap:101 telephone-event/8000
Media Attribute Fieldname: rtpmap
Media Format: 101
MIME Type: telephone-event
Sample Rate: 8000
Media Attribute (a): fmtp:101 0-15
Media Attribute Fieldname: fmtp
Media Format: 101 [telephone-event]
Media format specific parameters: 0-15
The same packet, when it has gone thru pf, and now is leaving re0 looks
like this:
No. Time Source Destination Protocol
Length Info
74436 948.108526 187.170.255.239 192.168.230.101
SIP/SDP 1081 Request: INVITE sip:[email protected];user=phone,
with session description
Frame 74436: 1081 bytes on wire (8648 bits), 1081 bytes captured (8648 bits)
Ethernet II, Src: D-Link_b8:62:95 (f0:7d:68:b8:62:95), Dst:
Aastra_81:a4:ad (00:08:5d:81:a4:ad)
Destination: Aastra_81:a4:ad (00:08:5d:81:a4:ad)
Address: Aastra_81:a4:ad (00:08:5d:81:a4:ad)
.... ...0 .... .... .... .... = IG bit: Individual address
(unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Source: D-Link_b8:62:95 (f0:7d:68:b8:62:95)
Address: D-Link_b8:62:95 (f0:7d:68:b8:62:95)
.... ...0 .... .... .... .... = IG bit: Individual address
(unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Type: IP (0x0800)
Internet Protocol Version 4, Src: 187.170.255.239 (187.170.255.239),
Dst: 192.168.230.101 (192.168.230.101)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0xb8 (DSCP 0x2e: Expedited
Forwarding; ECN: 0x00: Not-ECT (Not ECN-Capable Transport))
1011 10.. = Differentiated Services Codepoint: Expedited
Forwarding (0x2e)
.... ..00 = Explicit Congestion Notification: Not-ECT (Not
ECN-Capable Transport) (0x00)
Total Length: 1067
Identification: 0x0000 (0)
Flags: 0x02 (Don't Fragment)
Fragment offset: 0
Time to live: 55
Protocol: UDP (17)
Header checksum: 0x0000 [incorrect, should be 0xdc61 (maybe caused
by "IP checksum offload"?)]
Source: 187.170.255.239 (187.170.255.239)
Destination: 192.168.230.101 (192.168.230.101)
User Datagram Protocol, Src Port: sip (5060), Dst Port: sip (5060)
Source port: sip (5060)
Destination port: sip (5060)
Length: 1047
Checksum: 0x13e0 [validation disabled]
Session Initiation Protocol
Request-Line: INVITE sip:[email protected];user=phone SIP/2.0
Method: INVITE
Request-URI: sip:[email protected];user=phone
Request-URI User Part: 0424507401
Request-URI Host Part: 212.247.80.66
[Resent Packet: False]
Message Header
Record-Route: <sip:130.244.190.42;lr;ftag=469095268>
Via: SIP/2.0/UDP 130.244.190.42;branch=z9hG4bKe904.69d17a12.0
Transport: UDP
Sent-by Address: 130.244.190.42
Branch: z9hG4bKe904.69d17a12.0
Via: SIP/2.0/UDP
212.151.144.8:5060;branch=z9hG4bKterm-785897-0424507401-0707957245-5102
Transport: UDP
Sent-by Address: 212.151.144.8
Sent-by port: 5060
Branch: z9hG4bKterm-785897-0424507401-0707957245-5102
From: 0707957245
<sip:[email protected]:5060;user=phone>;tag=469095268
SIP Display info: 0707957245
SIP from address: sip:[email protected]:5060;user=phone
SIP from address User Part: 0707957245
SIP from address Host Part: 212.151.144.8
SIP from address Host Port: 5060
SIP tag: 469095268
To: 0424507401
<sip:[email protected]:5060;user=phone>
SIP Display info: 0424507401
SIP to address:
sip:[email protected]:5060;user=phone
SIP to address User Part: 0424507401
SIP to address Host Part: sip-corporate.tele2.se
SIP to address Host Port: 5060
Call-ID: [email protected]
CSeq: 1 INVITE
Sequence Number: 1
Method: INVITE
Max-Forwards: 69
Supported: timer
Session-Expires: 1800
Min-SE: 1800
Contact: 0707957245 <sip:[email protected]:5060>
SIP Display info: 0707957245
Contact-URI: sip:[email protected]:5060
Contactt-URI User Part: 0707957245
Contact-URI Host Part: 212.151.144.8
Contact-URI Host Port: 5060
Allow:
INVITE,ACK,PRACK,SUBSCRIBE,BYE,CANCEL,NOTIFY,INFO,REFER,UPDATE
P-Asserted-Identity: 0707957245
<sip:[email protected];user=phone>
SIP Display info: 0707957245
SIP PAI Address: sip:[email protected];user=phone
SIP PAI User Part: 0707957245
SIP PAI Host Part: 212.151.144.8
Content-Type: application/sdp
Content-Length: 225
Message Body
Session Description Protocol
Session Description Protocol Version (v): 0
Owner/Creator, Session Id (o): - 52119343 0 IN IP4 130.244.1.7
Owner Username: -
Session ID: 52119343
Session Version: 0
Owner Network Type: IN
Owner Address Type: IP4
Owner Address: 130.244.1.7
Session Name (s): Cisco SDP 0
Connection Information (c): IN IP4 130.244.1.7
Connection Network Type: IN
Connection Address Type: IP4
Connection Address: 130.244.1.7
Time Description, active time (t): 0 0
Session Start Time: 0
Session Stop Time: 0
Media Description, name and address (m): audio 17070
RTP/AVP 8 0 99 102 101
Media Type: audio
Media Port: 17070
Media Protocol: RTP/AVP
Media Format: ITU-T G.711 PCMA
Media Format: ITU-T G.711 PCMU
Media Format: DynamicRTP-Type-99
Media Format: DynamicRTP-Type-102
Media Format: DynamicRTP-Type-101
Media Attribute (a): rtpmap:99 G.729a/8000
Media Attribute Fieldname: rtpmap
Media Format: 99
MIME Type: G.729a
Sample Rate: 8000
Media Attribute (a): rtpmap:102 G.729b/8000
Media Attribute Fieldname: rtpmap
Media Format: 102
MIME Type: G.729b
Sample Rate: 8000
Media Attribute (a): rtpmap:101 telephone-event/8000
Media Attribute Fieldname: rtpmap
Media Format: 101
MIME Type: telephone-event
Sample Rate: 8000
Media Attribute (a): fmtp:101 0-15
Media Attribute Fieldname: fmtp
Media Format: 101 [telephone-event]
Media format specific parameters: 0-15
So, it seems to be a PF issue, althou not a PF NAT issue,
since the problem is ingress, not egress.
Jun 22 15:40:06.307526 rule 0/(match) [uid 0, pid 20284] pass out on
bge0: 192.168.230.101.5060> 187.170.255.239.5060: udp 550 (ttl 63, id
33961, len 578, bad cksum 9159! differs by 100)
0000: 4500 0242 84a9 0000 3f11 9159 c0a8 e665
E..B.\M-)..?..Y\M-@\M-(\M-fe
0010: bbaa ffef 13c4 13c4 022e 9dc3 5349 502f
\M-;\M-*\M^?\M-o.\M-D.\M-D...\M-CSIP/
0020: 322e 3020 3230 3020 4f4b 0d0a 5669 613a 2.0 200 OK..Via:
0030: 2053 4950 2f32 2e30 2f55 4450 SIP/2.0/UDP
and on a side note, if anyone has a suggestion how to actually get the
complete package logged, and not just the first snap, it would be nice,
openbsd tcpdump seems to not support -s 0 as snaplen, to get the whole
thing.
see tcpdump(8) about -s (or ngrep has fairly clear formatting for
reading inside sip packets, "ngrep -d re0 -W byline port 5060",
though less information from the IP/TCP header is displayed).
anyway, that log snippet, is 130.244.190.46 asking us to setup a sip
connection with them on 5060,
but our respond to that ip, goes to 187.170.255.239. and the connection
fails.
another side note would be about the rampant amount of bad ckdsum on udp
traffic, if anyone would care to chime in about that.
Since about 98% of all udp packets get a bad cksum.
see tcpdump(8) about IP Checksum Offload.
but my main problem and concern is this 187.170.255.239, and why they
should get my phonecalls.
Regards
Magnus
