PuffySpace

PF - Firewall Redundancy (CARP and pfsync)

Introduction to CARP

CARP is the Common Address Redundancy Protocol. Its primary purpose is to allow multiple hosts on the same network segment to share an IP address. CARP is a secure, free alternative to the Virtual Router Redundancy Protocol (VRRP) and the Hot Standby Router Protocol (HSRP).

CARP works by allowing a group of hosts on the same network segment to share an IP address. This group of hosts is referred to as a "redundancy group." The redundancy group is assigned an IP address that is shared amongst the group members. Within the group, one host is designated the "master" and the rest as "backups." The master host is the one that currently "holds" the shared IP; it responds to any traffic or ARP requests directed towards that address. Each host may belong to more than one redundancy group at a time.

One common use for CARP is to create a group of redundant firewalls. The virtual IP that is assigned to the redundancy group is configured on client machines as the default gateway. In the event that the master firewall suffers a failure or is taken offline, the IP will move to one of the backup firewalls and service will continue unaffected.

CARP supports IPv4 and IPv6.

CARP Operation

The master host in the group sends regular advertisements to the local network so that the backup hosts know it's still alive. If the backup hosts don't hear an advertisement from the master for a set period of time, one of them will take over the duties of master (whichever backup host has the lowest configured advbase and advskew values).

It's possible for multiple CARP groups to exist on the same network segment. CARP advertisements contain the Virtual Host ID which allows group members to identify which redundancy group the advertisement belongs to.

In order to prevent a malicious user on the network segment from spoofing CARP advertisements, each group can be configured with a password. Each CARP packet sent to the group is then protected by an SHA1 HMAC.

Since CARP is its own protocol, it should have an explicit pass rule in filter rulesets: 

pass out on $carp_dev proto carp
$carp_dev should be the physical interface that CARP is communicating over.

Configuring CARP

Each redundancy group is represented by a carp(4) virtual network interface. As such, CARP is configured using ifconfig(8). 
# ifconfig carpN create
# ifconfig carpN vhid vhid [pass password] [carpdev carpdev] \
   [advbase advbase] [advskew advskew] [state state] [group|-group group] \
   ipaddress netmask mask
carpN
The name of the carp(4) virtual interface where N is an integer that represents the interface's number (e.g. carp10).

vhid
The Virtual Host ID. This is a unique number that is used to identify the redundancy group to other nodes in the group, and to distinguish between groups on the same network. Acceptable values are from 1 to 255. This must be the same on all members of the group.

password
The authentication password to use when talking to other CARP-enabled hosts in this redundancy group. This must be the same on all members of the group.

carpdev
This optional parameter specifies the physical network interface that belongs to this redundancy group. By default, CARP will try to determine which interface to use by looking for a physical interface that is in the same subnet as the ipaddress and mask combination given to the carp(4) interface.

advbase
This optional parameter specifies how often, in seconds, to advertise that we're a member of the redundancy group. The default is 1 second. Acceptable values are from 1 to 255.

advskew
This optional parameter specifies how much to skew the advbase when sending CARP advertisements. By manipulating advskew, the master CARP host can be chosen. The higher the number, the less preferred the host will be when choosing a master. The default is 0. Acceptable values are from 0 to 254.

state
Force a carp(4) interface into a certain state. Valid states are init, backup, and master.

group, -group
Add or remove a carp(4) interface to a certain interface group. By default, all carp interfaces are added to the carp group. Each group has a carpdemote counter affecting all carp interfaces belonging to that group. If one physical CARP-enabled interface goes down, CARP will increase the demotion counter by 1 on interface groups that the carp(4) interface is a member of, in effect causing all group members to fail-over together.

ipaddress
This is the shared IP address assigned to the redundancy group. This address does not have to be in the same subnet as the IP address on the physical interface (if present). This address needs to be the same on all hosts in the group, however.

mask
The subnet mask of the shared IP.

Further CARP behavior can be controlled via sysctl(8).

net.inet.carp.allow
Accept incoming CARP packets or not. Default is 1 (yes).

net.inet.carp.preempt
Allow hosts within a redundancy group that have a better advbase and advskew to preempt the master. net.inet.carp.preempt is 0 (disabled) by default.

net.inet.carp.log
Log state changes, bad packets and other errors. This value may be between 0 and 7, corresponding with syslog(3) priorities. The default is 2 (state changes only).

CARP Example

Here is an example CARP configuration: 
# sysctl net.inet.carp.allow=1
# echo 'net.inet.carp.allow=1' >> /etc/sysctl.conf
# ifconfig carp1 create
# ifconfig carp1 vhid 1 pass mekmitasdigoat carpdev em0 advskew 100 10.0.0.1 netmask 255.255.255.0
This sets up the following: The shared IP assigned to this group is 10.0.0.1/255.255.255.0.

Running ifconfig on carp1 shows the status of the interface. 

# ifconfig carp1
carp1: flags=8802<UP,BROADCAST,SIMPLEX,MULTICAST> mtu 1500
     carp: BACKUP carpdev em0 vhid 1 advbase 1 advskew 100
     groups: carp
     inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255

Introduction to pfsync

The pfsync(4) network interface exposes certain changes made to the pf(4) state table. By monitoring this device with tcpdump(8), state table changes can be observed in real time. In addition, the pfsync interface can send these state change messages out on the network so that other nodes running PF can merge the changes into their own state tables. Likewise, pfsync can also listen on the network for incoming messages.

pfsync Operation

By default, pfsync does not send or receive state table updates on the network; however, updates can still be monitored using tcpdump or similar tools on the local machine.

When pfsync is set up to send and receive updates on the network, the default behavior is to multicast updates out on the local network. All updates are sent without authentication. Best common practice is either:

  1. Connect the two nodes that will be exchanging updates back-to-back using a crossover cable and use that interface as the syncdev (see below)
  2. Use the ifconfig syncpeer option (see below) so that updates are unicast directly to the peer, then configure ipsec(4) between the hosts to secure the pfsync traffic
When updates are being sent and received on the network, pfsync packets should be passed in the filter ruleset: 
pass on $sync_if proto pfsync
$sync_if should be the physical interface that pfsync is communicating over.

Configuring pfsync

Since pfsync is a virtual network interface, it is configured using ifconfig(8). 
# ifconfig pfsyncN syncdev syncdev [syncpeer syncpeer] [defer|-defer]
pfsyncN
The name of the pfsync interface

syncdev
The name of the physical interface used to send pfsync updates out

syncpeer
This optional parameter specifies the IP address of a host to exchange pfsync updates with. By default, pfsync updates are multicast on the local network. This option overrides that behavior and instead unicasts the update to the specified syncpeer.

defer
If the defer flag is used, the initial packet of a new connection passing through the firewall will not be transmitted until either another pfsync system has acknowledged the state table addition, or a timeout has expired. This adds small delays but allows traffic to flow when more than one firewall might actively handle packets ("active/active"), e.g. with certain ospfd(8), bgpd(8), or carp(4) configurations.

pfsync Example

Here is an example pfsync configuration: 
# ifconfig pfsync0 syncdev em1 up
This enables pfsync on the em1 interface. Outgoing updates will be multicast on the network, allowing any other host running pfsync to receive them.

Combining CARP and pfsync for Failover

By combining the features of CARP and pfsync, a group of two or more firewalls can be used to create a highly-available, fully redundant firewall cluster. CARP handles the automatic failover of one firewall to another, while pfsync synchronizes the state table among all the firewalls. In the event of a failover, traffic can flow uninterrupted through the new master firewall.

An example scenario: two firewalls, fw1 and fw2. 

         +----| WAN/internet |----+
         |                        |
      em2|                        |em2
      +-----+                  +-----+
      | fw1 |-em1----------em1-| fw2 |
      +-----+                  +-----+
      em0|                        |em0
         |                        |
      ---+-------Shared LAN-------+---
The firewalls are connected back-to-back using a crossover cable on em1. Both are connected to the LAN on em0 and to a WAN/internet connection on em2. IP addresses are as follows: The network policy is that fw1 will be the preferred master.

To configure fw1, begin by enabling preemption and group interface failover: 

# sysctl net.inet.carp.preempt=1
# echo 'net.inet.carp.preempt=1' >> /etc/sysctl.conf
Configure pfsync: 
# ifconfig em1 10.10.10.1 netmask 255.255.255.0
# ifconfig pfsync0 syncdev em1
# ifconfig pfsync0 up
Configure CARP on the LAN side: 
# ifconfig carp1 create
# ifconfig carp1 vhid 1 carpdev em0 pass lanpasswd \
     172.16.0.100 netmask 255.255.255.0
Configure CARP on the WAN/internet side: 
# ifconfig carp2 create
# ifconfig carp2 vhid 2 carpdev em2 pass netpasswd \
     192.0.2.100 netmask 255.255.255.0
Then configure fw2 accordingly: 
# sysctl net.inet.carp.preempt=1
# echo 'net.inet.carp.preempt=1' >> /etc/sysctl.conf
# ifconfig em1 10.10.10.2 netmask 255.255.255.0
# ifconfig pfsync0 syncdev em1
# ifconfig pfsync0 up
# ifconfig carp1 create
# ifconfig carp1 vhid 1 carpdev em0 pass lanpasswd \
     advskew 128 172.16.0.100 netmask 255.255.255.0
# ifconfig carp2 create
# ifconfig carp2 vhid 2 carpdev em2 pass netpasswd \
     advskew 128 192.0.2.100 netmask 255.255.255.0

Operational Issues

Configuring CARP and pfsync During Boot

Since carp and pfsync are both types of network interfaces, they can be configured at boot by creating a hostname.if(5) file. The netstart startup script will take care of creating the interface and configuring it.

Examples:

/etc/hostname.carp1
inet 172.16.0.100 255.255.255.0 172.16.0.255 vhid 1 carpdev em0 pass lanpasswd
/etc/hostname.pfsync0
syncdev em1 up

Forcing Failover of the Master

There can be times when it's necessary to failover or demote the master node on purpose. Examples might include taking the master node down for maintenance or when troubleshooting a problem. The objective here is to gracefully fail over traffic to one of the backup hosts so that users do not notice any impact.

To failover a particular CARP group, shut down the carp interface on the master node. This will cause the master to advertise itself with an "infinite" advbase and advskew. The backup host(s) will see this and immediately take over the role of master. 

# ifconfig carp1 down
An alternative is to increase the advskew to a value that's higher than the advskew on the backup host(s). This will cause a failover but still allow the master to participate in the CARP group.

Another method of failover is to tweak the CARP demotion counter. The demotion counter is a measure of how "ready" a host is to become master of a CARP group. For example, while a host is in the middle of booting up it's a bad idea for it to become the CARP master until all interfaces have been configured, all network daemons have been started, etc. Hosts advertising a high demotion value will be less preferred as the master.

A demotion counter is stored in each interface group that the CARP interface belongs to. By default, all CARP interfaces are members of the "carp" interface group. The current value of a demotion counter can be viewed using ifconfig: 

# ifconfig -g carp
carp: carp demote count 0
In this example, the counter associated with the "carp" interface group is shown. When a CARP host advertises itself on the network, it takes the sum of the demotion counters for each interface group the carp interface belongs to and advertises that value as its demotion value.

Now assume the following example. Two firewalls running CARP with the following CARP interfaces:

The objective is to failover just the carp1 and carp2 groups to the secondary firewall.

First, assign each to a new interface group, in this case named "internal": 

# ifconfig carp1 group internal
# ifconfig carp2 group internal
# ifconfig internal
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
     carp: MASTER carpdev em0 vhid 1 advbase 1 advskew 100
     groups: carp internal
     inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
carp2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
     carp: MASTER carpdev em1 vhid 2 advbase 1 advskew 100
     groups: carp internal
     inet 10.0.1.1 netmask 0xffffff00 broadcast 10.0.1.255
Now increase the demotion counter for the "internal" group: 
# ifconfig -g internal
internal: carp demote count 0
# ifconfig -g internal carpdemote 50
# ifconfig -g internal
internal: carp demote count 50
The firewall will now gracefully failover on the carp1 and carp2 groups to the other firewall in the cluster while still remaining the master on carp3 and carp4. If the other firewall started advertising itself with a demotion value higher than 50, or if the other firewall stopped advertising altogether, then this firewall would again take over mastership on carp1 and carp2.

To fail back to the primary firewall, reverse the changes: 

# ifconfig -g internal -carpdemote 50
# ifconfig -g internal
internal: carp demote count 0
Network daemons such as bgpd(8) and sasyncd(8) make use of the demotion counter to ensure that the firewall does not become master until BGP sessions become established and IPsec SAs are synchronized.

Ruleset Tips

Filter the physical interface. As far as PF is concerned, network traffic comes from the physical interface, not the CARP virtual interface (i.e., carp0). So, write the rulesets accordingly. Don't forget that an interface name in a PF rule can be either the name of a physical interface or an address associated with that interface. For example, this rule could be correct: 
pass in on fxp0 inet proto tcp from any to carp0 port 22
But replacing the fxp0 with carp0 would not work as desired.

Don't forget to pass proto carp and proto pfsync!

Other References

Please see these other sources for more information: