IPSEC between OPNsense and pfSense

with one side behind Carrier-grade NAT or internal subnet

Published: 2022-11-12, Revised: 2023-03-11

TL;DR A site-to-site connection between pfSense/OPNsense with IPSEC is straight-forward. This post explains some of the peculiarities, needed to establish a connection, if one of the two sides is behind a Carrier-grade NAT or in an internal subnet. This can happen (e.g.) with an upstream WAN uplink that is dial-up, with a non-static IP. I also show how to do the routing with multiple subnets on one end, using Classless Inter-Domain Routing (CIDR). The different parts are largely separable, choose what's interesting to you.

Prerequisites#

1. Have a public domain (AAA) that is set up with a DNS API (e.g. Cloudflare). Your VPN remains private and no ports on either side will need to be exposed.
2. Have one site (in my case the "pfSense" side) with a static WAN address.
3. The other side ("OPNsense") just needs a WAN uplink, which can be dynamic and behind CG NAT or a private network

Hardware#

This post is not hardware specific, you can install pfSense and OPNsense on almost any computer. In the picture above you see my OPNsense setup, a Protectli mini computer ⁴. The WAN hooks up to a Wifi bridge that connects the Protectli, as a client, to another Wifi network with Internet access (a dynamic IP). I added a second WiFi to the LAN side because the Wifi module of Protectli is really only good for administrative work.

Another option would be the Apu2d4, with which I had great experiences with and using as the Site B example router.

DNS Setup#

A static site-to-site connection with IPSEC is commonly configured with two static IPs. This is one part of the security concept. With a dynamic IP on one side, the other side will need to be configured fully open. For this type of setup, a Road Warrior VPN setup (e.g. OpenVPN) is typically better.

There is a workaround, however, for setups where the dynamic IP changes just sometimes. Here, OPNsense can "announce" its external IP through a DNS API such as Cloudflare.

OPNsense has a service for updating DNS entries for various providers. Check out Services > Dynamic DNS.

You have one problem though when the OPNsense does not know its public IP because (e.g.) it itself sits in a private subnet.

In this case, use the following workaround.

1. Make sure you have an AAA record for a Top Level Domain (TLD). For management purposes, the nameservers of my domain link to Cloudflare. You can use any DNS service with an API here. Login to (e.g.) Cloudflare and go to your domain example.com > DNS.

2. Add a new entry with a random string.

Above, the nameservers for example.com point to Cloudflare, but no traffic is actually routed. We don't care about 99.99.99.99 - it does not matter whether routing traffic through Cloudflare is enabled or disabled for this guide. We only use the DNS API.

We want the subdomain jashdejvmiuqlachhsqaxs.siteb.example.com to point to the public WAN of our Protectli (so the IP can be queried by pfSense on the other side). Set the entry to DNS only and the start value to 0.0.0.0.

3. Create a new API Token under My Profile > API Tokens. If possible, you can limit this token to a subnet through Client IP Address Filtering, which increases security.

4. With your favorite shell of choice, login to OPNsense.

We are going to use a custom script, by Dominic Cerisano ⁵, that will
- (1) query the external WAN IP first and then
- (2) set this IP through the Cloudflare API for our DNS entry.

cd /usr/local/opnsense/scripts/
mkdir myscripts
cd myscripts
vi cloudflare-ddns.sh

Paste the following script from Dominic Cerisano ⁵ (or clone from the repo).

#!/bin/sh

AUTH_EMAIL=example@example.com
AUTH_KEY=** CF Authorization  key **
ZONE_ID=** CF Zone ID **
A_RECORD_NAME="dynamic"
A_RECORD_ID=** CF A-record ID from cloudflare-dns-id.sh **

# Retrieve the last recorded public IP address
IP_RECORD="/tmp/ip-record"
RECORDED_IP=`cat $IP_RECORD`

# Fetch the current public IP address
PUBLIC_IP=$(curl --silent https://api.ipify.org) || exit 1

# If the public ip has not changed, nothing needs to be done, exit.
if [ "$PUBLIC_IP" = "$RECORDED_IP" ]; then
    exit 0
fi

# Otherwise, your Internet provider changed your public IP again.
# Record the new public IP address locally
echo $PUBLIC_IP > $IP_RECORD

# Record the new public IP address on Cloudflare using API v4
RECORD=$(cat <<EOF
{ "type": "A",
  "name": "$A_RECORD_NAME",
  "content": "$PUBLIC_IP",
  "ttl": 180,
  "proxied": false }
EOF
)
curl "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records/$A_RECORD_ID" \
     -X PUT \
     -H "Content-Type: application/json" \
     -H "X-Auth-Email: $AUTH_EMAIL" \
     -H "X-Auth-Key: $AUTH_KEY" \
     -d "$RECORD"

PUBLIC_IP=$(curl --silent https://api.ipify.org) || exit 1

PUBLIC_IP=$(/sbin/ifconfig pppoe0 | grep "inet" | awk '/inet / { print $2 }')

You will need to update

• AUTH_EMAIL
• AUTH_KEY
• ZONE_ID
• A_RECORD_NAME
• A_RECORD_ID.

The first three are available in the Cloudflare dashboard.

A_RECORD_NAME is the subdomain we just created (e.g. jashdejvmiuqlachhsqaxs.siteb).

A_RECORD_ID must be queried using another script, provided by a friendly user on Github. ⁶

# get all Cloudflare record IDs, select the one for our subdomain
curl -X GET "https://api.cloudflare.com/client/v4/zones/**zoneid**/dns_records?type=A" \
     -H "X-Auth-Email: example@example.com" \
     -H "X-Auth-Key: ** CF Authorization  key **" \
     -H "Content-Type: application/json"

5. Test the script

chmod +x cloudflare-ddns.sh
sh cloudflare-ddns.sh

The script will

• (1) get the WAN IP through a query of api.ipify.org,
• (2) check if the IP did actually change (/tmp/ip-record).
• (3) If it did change, connect to Cloudflare and update the entry with the new IP.

Afterwards, login to Cloudflare and verify that the correct WAN is set.

6. Automate

If everything works, we will need to add this script to OPNsense cronjobs.

There is a system to hooking up scripts to the GUI. You can, of course, edit cron directly. I followed the docs. ⁷

Create a new action:

cd /usr/local/opnsense/service/conf/actions.d/
vi actions_cf.conf

Paste the following:

[update]
command:/usr/local/opnsense/scripts/myscripts/cloudflare-ddns.sh
description:Update CF DynDNS
parameters:
type:script
message:Updating Cloudflare DNS IP

Test the action:

rm /tmp/ip-record
service configd restart
configctl cf update

7. Login to the OPNsense WebGUI and activate the cronjob

Go to System > Settings > Cron.

Click Add and select the new action with the name Update CF DynDNS.

In the example, the script would run every hour at 59 Minutes.

You can check the logging under opnsense.siteb.example.com/ui/diagnostics/log/core/configd:

> 2022-01-30T06:40:00   Informational   configd.py  message ... [cf.update] returned OK 
> 2022-01-30T06:40:00   Notice  configd.py  [...] Updating Cloudflare DNS IP

8. Optionally, set crontab hook to run script after reboot

After reboot, it would be wise to check the Cloudflare DNS entry immediately because of a possible new WAN IP.

cp /usr/local/etc/rc.syshook.d/start/90-cron /usr/local/etc/rc.syshook.d/start/91-dyndns
vi /usr/local/etc/rc.syshook.d/start/91-dyndns

#!/bin/sh

echo -n "Updating Cloudflare DynDNS.. "
sleep 15 && sh /usr/local/opnsense/scripts/myscripts/cloudflare-ddns.sh

crontab -e

@reboot (sleep 15 && sh /usr/local/opnsense/scripts/myscripts/cloudflare-ddns.sh) > /dev/null

OPNsense IPSEC#

This part more or less follows the official OPNsense docs. ⁸

In the example below, the LAN IP of OPNsense is 192.168.179.1 and pfSense can be reached locally through 192.168.10.1.

Go to VPN > IPSEC and add a new phase1 entry.

Click save and add a new phase2 entry for the phase1 we just created.

pfSense IPSEC#

Again, nothing surprising here. You can follow the official pfSense docs. ¹⁰

Go to VPN > IPSEC and add a new phase1 entry. Configuration of both sides must match.

Firewall rules#

If you are using OPNsense in a private network, like in our example, you also need to disable the default rule to block private networks on WAN. ⁸ This is not necessary for pfSense, since it is available on a public and static IP in our example.

Go to Interfaces > WAN and uncheck “Block private networks”.

You will also need to add three rules for ingress traffic on the WAN interface for OPNsense, according to the docs.¹²

• Protocol ESP
• UDP Traffic on Port 500 (ISAKMP)
• UDP Traffic on Port 4500 (NAT-T)

Go to Firewall > Rules > WAN and add the three rules.

At this point you should see the IPSEC tunnel becoming available.

On pfSense, this will look very similar.

If not, restart both pfSense and OPNsense and give both some time to update IPs. Have a look at Debugging, where I list some common approaches.

Routing#

Routing Site B

Let's specify the routing part. In the example above, we have two obvious subnets, 192.168.10.0 (pfSense net) and 192.168.179.0 (OPNsense net).

The routing for these two basic networks are automatically added, when using the Install policy option in the IPSEC settings. Also note the docs.¹¹

Most Site-to-Site VPNs are policy-based, which means you define a local and a remote network (or group of networks).

However, nothing prevents us to have additional subnets on both sides, e.g. pfSense VLANs 192.168.20.1, 192.168.30.1 and 192.168.40.1. For instance, I use VLANs to separate my network into different security zones, which makes management much easier.

OPNsense, however, doesn't know which VLANs are on the pfSense side. In order to decide that traffic to (e.g.) the IP 192.168.40.50 (our imaginary nextcloud service from above) needs to be routed through the IPSEC tunnel, additional routing information must be added.

If you haven't noticed: We did this already, by using a subnet mask 192.168.0.0/17. This is a neat trick, using CIDR Subnet Mask Notation, that offers the benefits of using policy-based automatic routing while also allows to specify a range of selected subnets to be routed.

The bigger prefix 17 (Subnet Mask) means that OPNsense will route every packet for destination IPs < 192.168.127.0 through the IPSEC tunnel, and let pfSense on the other side decide what to do with these packets. Every packet for IPs >= 192.168.127.0 will not be routed through the tunnel.

Routing Site A

If you do not want to route traffic from Site A to Site B, nothing needs to be done on the pfSense side,

I wanted to access OPNsense from Site A, which requires adding a Gateway and Static Route.¹³

My observation was that this is not necessary on OPNsense anymore.

SSL#

SSL is not needed, but if you are doing any routing for private services you should set this up.

With the ACME plugin and a DNS API provider such as Cloudflare, setup is a matter of Minutes.

Go to Services > ACME Client > Settings and follow the instructions.

Furthermore, if clients on Site B are supposed to reach services on Site A, the DNS service must be configured.

Conclusion#

I used this setup successfully for over a year now for an IPSEC tunnel between the US and Germany.

Having a central routing device is much more convenient than setting up OpenVPN or WireGuard Road Warrior tunnels on all clients.

Any new client accessing the LAN/WiFi of the Protectli will automatically be served with the correct DNS entries and routed accordingly, either through IPSEC for private services or through the local WAN uplink for public traffic.

Bring your Protectli to any location, add two WiFi devices for WAN and LAN, and you have something akin to a Road-Warrior Setup for Groups! E.g. for Workshops, Prototype Demonstrations, or at Conferences. How cool is this?

Debugging#

Logs are your friend here. Check:

• Firewall logs
• IPSEC Logs (VPN > IPSEC > Log File > Debug)
• Use packet capture
• on clients, use ping or dig for debugging DNS