Em 17-05-2014 16:26, Florenz Kley escreveu:
> On May 13, 2014, at 22:13, Giancarlo Razzolini <[email protected]> wrote:
>> go there, this e-mail would be too big. If you want I can elaborate more.
> yes please! do elaborate a bit!
>
> fl
>
Well,
I always wanted to write an article and post it on a blog. But never
really stopped everything and wrote it. I'll try to condense things on
this e-mail, but I'll also try to write a lengthier article someday.
First of all, you need to enable ip forwarding and, if you want your
firewall machine itself, to have multiple gateways in it's routing
table, you'll need to enable multipath routing. Both these are on
sysctl.conf. The multipath part is really only needed if you'll run
other services on your firewall itself. Such as a dns daemon, a proxy,
etc. The firewall must be able to reach the internet at all times, or
you'll have problems in this case.
Now, you need to determine how you will be able to tell if a
determined ISP link is down. This part is the one that gets more
different ways to accomplish. Somebody always will tell you to simply
ping an external IP address and get over with it. I really don't like it
and use it only as last resort. Anyway, one nice way to determine if a
link is down, is to use snmp. You can enable snmp on your ISP
router/modem/whatever, do an snmpwalk on it, determine which interfaces
it show. Then you trigger a disconnect and see which one of them are
down. These will be the ones you'll check. You can write a simple script
that does an snmpget on the interface oid and check if the state is up
or down. If your router do not have snmp, but does have another form of
remote administration, like a web interface or telnet, you can either
write a script that connects on it via http and go to the page where it
prints the internet connection status and check if it's connected. Or
you can write a expect script to connect to it via telnet and type the
commands that will show if the connection is up, or not. The external ip
pinging part is the easiest to implement, but also easily to get wrong.
You need to ping more than one IP address and only if all of them fail
to reply, them you can (hopefully) consider the link down. I hope you
got the idea. Almost forgot to mention, you need to script these to
simply return 0 or 1 to the shell that runs it. This is the way that
ifstated will be able to tell if the link is up or down. As you can see,
there are several ways to accomplish this.
The next part is to write the ifstated configuration. This is not as
hard as it seems. It's a state machine. It is a: do this, if this fails,
do that, but if it gets back, return to the previous state. Or do
something entirely different. You have at least n² states. n being the
number of internet links you have. So, if you have 2 links, you have at
least 4 states. If you have 3, 6 states, and so on. The reason is that
you need an state where all the links are on (the state where we hope it
will be most of the time) and one where all the links are off (zombie
apocalypse anyone?). Remember those scripts you wrote that monitor your
links? You'll configure them to run in your ifstated every x seconds.
The x here you'll define based on what you want to accomplish. Mine run
every 20 seconds. So basically you'll have some lines in the beginning
of your ifstated.conf that are something like this:
link_isp = '( "<path to script>" every 20 )'
The first state you define on ifstated will be the default state,
unless you override with the default_state option. So, the first state
will most likely be the state where all the links are up. You can run
things when you enter a state. I use this to update dynamic dns records,
to mail me when a link is down, reload/restart daemons that are bound to
a determined isp connection, etc. But the most important thing to do is
to change your pf rules when you change states. To do this, there are
lots of ways too. I use anchors. So, I have n²-1 anchor files. n being
the number of links you have. Because when all of them are down, there
is no point in changing your rules. You could point your users to a
internal web page telling them they have no internet, and, therefore, no
facebook for them. But this is out of the scope. Anyway, I use ifstated
to load anchors that change the pf rules to use the isp's that are
working. So how you point your users to different gateways? You use the
route-to option. This will point any machine using your firewall to the
gateways you choose. Since this will be up entirely to you, I'll not go
into details here. Just be sure to not confuse traffic that the machines
behind your firewall generate, from traffic that your firewall itself
generate. These need to be dealt differently, in the case of a link
change. As I mentioned before, if you only have pf on the firewall, you
can just use anchors and you are ok. If not, you'll need to change the
routes on the firewall itself, with multipath. In my case, I simply play
with the routing priority. But you can add/delete routes, pfctl -k to
kill states and trigger reconnections, etc.
Over the years I refined things, so I got better at detecting link
failures and I even have some scripts to detect not only link failures,
but when links are loosing too many packets or are slow, I can failover
them too. I even got further and I have a database, where I log the
events and so I can temporarily failover a link that is flapping between
states in a determined period of time. I send e-mails when links go
down, or when they come back. This e-mail got lengthy, so I'll stop
here. I hope you all got the idea. If anyone else could join in and
share their experiences too, it would be nice.
P.s.: If you are going to use CARP on top of this, it can work, but all
the carped machines must see all your ISP's router/modems/etc. I used a
separate switch for this, but there are other options too. CARP adds
complexity to the mix, for instance, you'll will need more states on
your ifstated.conf. Also, ifstated will most likely be running only at
one carp machine at a time. If not, they will need to "talk" with each
other some way. I used a central database back in the day, but, as
everything else, you have many options to accomplish this.
Cheers,
--
Giancarlo Razzolini
GPG: 4096R/77B981BC
