I have pf running on an OpenBSD 4.0 (patches 1-5, 7) router and I have one user with two Gentoo Linux machines with kernel 2.6.18 who is having troubles. Everyone else is having no problem at all. This user is having any tcp connection he makes dropped by the firewall. The state shows up when I run "pfctl -ss" but a sniff on both ends of the router shows that it is dropping the packets. If I set the debug level to loud I get the following output.
Gentoo and OpenBSD talking to each other Feb 13 15:35:41 titanium /bsd: pf: BAD state: TCP 10.10.10.224:22 10.10.10.224:22 10.8.0.6:14625 [lo=1438155416 high=1438171799 win=181 modulator=0] [lo=3399502493 high=3399502674 win=16384 modulator=0] 7:4 FPA seq=3399502493 ack=1438155416 len=776 ackskew=0 pkts=14:3 dir=in,rev Feb 13 15:35:41 titanium /bsd: pf: State failure on: 1 | Feb 13 15:35:43 titanium /bsd: pf: BAD state: TCP 10.10.10.224:22 10.10.10.224:22 10.8.0.6:11431 [lo=1182669220 high=1182684868 win=181 modulator=0] [lo=2952473521 high=2952473702 win=16384 modulator=0] 4:4 PA seq=2952473521 ack=1182668484 len=752 ackskew=736 pkts=4:2 dir=in,rev Feb 13 15:35:43 titanium /bsd: pf: State failure on: 1 | Feb 13 15:35:43 titanium /bsd: pf: BAD state: TCP 10.10.10.224:22 10.10.10.224:22 10.8.0.6:11431 [lo=1182669220 high=1182684868 win=181 modulator=0] [lo=2952473521 high=2952473702 win=16384 modulator=0] 4:4 PA seq=2952474273 ack=1182669220 len=24 ackskew=0 pkts=5:2 dir=in,rev Feb 13 15:35:43 titanium /bsd: pf: State failure on: 1 | Feb 13 15:35:44 titanium /bsd: pf: BAD state: TCP 10.10.10.224:22 10.10.10.224:22 10.8.0.6:11431 [lo=1182669220 high=1182685604 win=181 modulator=0] [lo=2952473521 high=2952473702 win=16384 modulator=0] 4:4 PA seq=2952473521 ack=1182669220 len=776 ackskew=0 pkts=5:3 dir=in,rev Feb 13 15:35:44 titanium /bsd: pf: State failure on: 1 | Feb 13 15:35:47 titanium /bsd: pf: BAD state: TCP 10.10.10.224:22 10.10.10.224:22 10.8.0.6:11431 [lo=1182669220 high=1182685604 win=181 modulator=0] [lo=2952473521 high=2952473702 win=16384 modulator=0] 4:4 PA seq=2952473521 ack=1182669220 len=776 ackskew=0 pkts=5:3 dir=in,rev Feb 13 15:35:47 titanium /bsd: pf: State failure on: 1 | The two gentoo machines trying to talk to each other Feb 13 14:55:42 titanium /bsd: pf: BAD state: TCP 10.10.10.224:22 10.10.10.224:22 10.8.0.98:54077 [lo=1806113440 high=1806113623 win=181 modulator=0] [lo=159381924 high=159382105 win=183 modulator=0] 4:4 PA seq=1806113440 ack=159381924 len=736 ackskew=0 pkts=3:3 dir=out,fwd Feb 13 14:55:42 titanium /bsd: pf: State failure on: 1 | Feb 13 14:55:42 titanium /bsd: pf: BAD state: TCP 10.10.10.224:22 10.10.10.224:22 10.8.0.98:54077 [lo=1806113440 high=1806113623 win=181 modulator=0] [lo=159381924 high=159382105 win=183 modulator=0] 4:4 PA seq=159381924 ack=1806113440 len=752 ackskew=0 pkts=3:3 dir=in,rev Feb 13 14:55:42 titanium /bsd: pf: State failure on: 1 | Feb 13 14:55:43 titanium /bsd: pf: BAD state: TCP 10.10.10.224:22 10.10.10.224:22 10.8.0.98:54077 [lo=1806113440 high=1806113623 win=181 modulator=0] [lo=159381924 high=159382105 win=183 modulator=0] 4:4 PA seq=1806113440 ack=159381924 len=736 ackskew=0 pkts=3:3 dir=out,fwd Feb 13 14:55:43 titanium /bsd: pf: State failure on: 1 | Feb 13 14:55:43 titanium /bsd: pf: BAD state: TCP 10.10.10.224:22 10.10.10.224:22 10.8.0.98:54077 [lo=1806113440 high=1806113623 win=181 modulator=0] [lo=159381924 high=159382105 win=183 modulator=0] 4:4 PA seq=159381924 ack=1806113440 len=752 ackskew=0 pkts=3:3 dir=in,rev Feb 13 14:55:43 titanium /bsd: pf: State failure on: 1 | Feb 13 14:55:43 titanium /bsd: pf: BAD state: TCP 10.10.10.224:22 10.10.10.224:22 10.8.0.98:54077 [lo=1806113440 high=1806113623 win=181 modulator=0] [lo=159381924 high=159382105 win=183 modulator=0] 4:4 PA seq=1806113440 ack=159381924 len=736 ackskew=0 pkts=3:3 dir=out,fwd Feb 13 14:55:43 titanium /bsd: pf: State failure on: 1 | Feb 13 14:55:43 titanium /bsd: pf: BAD state: TCP 10.10.10.224:22 10.10.10.224:22 10.8.0.98:54077 [lo=1806113440 high=1806113623 win=181 modulator=0] [lo=159381924 high=159382105 win=183 modulator=0] 4:4 PA seq=159381924 ack=1806113440 len=752 ackskew=0 pkts=3:3 dir=in,rev Feb 13 14:55:43 titanium /bsd: pf: State failure on: 1 | Feb 13 14:55:44 titanium /bsd: pf: BAD state: TCP 10.10.10.224:22 10.10.10.224:22 10.8.0.98:54077 [lo=1806113440 high=1806113623 win=181 modulator=0] [lo=159381924 high=159382105 win=183 modulator=0] 4:4 PA seq=1806113440 ack=159381924 len=736 ackskew=0 pkts=3:3 dir=out,fwd Feb 13 14:55:44 titanium /bsd: pf: State failure on: 1 | I am not quite sure exactly how to interpret this but it seemed to be an issue with tcp windows so I had him turn off these two settings on his linux box /proc/sys/net/ipv4/tcp_window_scaling /proc/sys/net/ipv4/tcp_sack After this it started working but seemed slow to him. Checking the pf debug I got the following now. Gentoo with SACK and window scaling off talking to a windows machine. Feb 13 17:02:29 titanium /bsd: pf: loose state match: TCP 10.8.0.98:43341 10.8.0.98:43341 10.10.12.40:443 [lo=2811501414 high=2811566845 win=5840 modulator=125064409] [lo=2131197667 high=2131201019 win=65431 modulator=2001531837] 10:10 R seq=2131197667 ack=2811501414 len=0 ackskew=0 pkts=23:16 Feb 13 17:02:29 titanium /bsd: pf: loose state match: TCP 10.10.12.40:443 10.10.12.40:443 10.8.0.98:43341 [lo=4132729504 high=4132732856 win=65431 modulator=0] [lo=2811501414 high=2811566845 win=5840 modulator=0] 10:10 R seq=4132729504 ack=2811501414 len=0 ackskew=0 pkts=16:22 Feb 13 17:02:44 titanium /bsd: pf: loose state match: TCP 10.8.0.98:43342 10.8.0.98:43342 10.10.12.40:443 [lo=2839702864 high=2839768295 win=5840 modulator=1261428551] [lo=326359370 high=326362722 win=65431 modulator=3635844913] 10:10 R seq=326359370 ack=2839702864 len=0 ackskew=0 pkts=23:16 Feb 13 17:02:44 titanium /bsd: pf: loose state match: TCP 10.10.12.40:443 10.10.12.40:443 10.8.0.98:43342 [lo=3962204283 high=3962207635 win=65431 modulator=0] [lo=2839702864 high=2839768295 win=5840 modulator=0] 10:10 R seq=3962204283 ack=2839702864 len=0 ackskew=0 pkts=16:22 So what is happening? It seems to me that either pf is broken or his linux kernel is broken and pf is catching it. Any ideas as to which is the cause? One other point I needs some clarification on, in my searching around I did find an article saying that you need the "flags S/SA" everytime you use keep state for tcp connections in your firewall rules. This didn't seem right to me but I tried it anyway just to see and it had no affect. What is the final word on this, should you always use "flags S/SA"? A couple of other points, I have tried various combinations of scrubing in my pf rules including turning it off with no luck. Also all other machines including other linux boxes work fine with this. If any more information is needed let me know. Thanks for the help! -- Tim Kuhlman Network Administrator ColoradoVnet.com
