Forum Discussion

Deleted's avatar
Deleted
Jul 27, 2021

I can no longer connect to some of the nodes over pfSense IPSec running in Azure

This used to work flawlessly and I didn't change anything in the configuration, so this may have happened after upgrading to pfSense 2.5.2, or after some Azure changes.

Specifically, I can no longer connect to the Azure VMs which use the same subnet as my Azure pfSense instance. I can, however, connect to other VMs that reside in a Peered address space.

My infrastructure is:

  1. One Internet-facing pfSense instance in Azure, running IPSec (let's call it PF)

  2. Clients connecting to PF over IPSec

  3. Several VMs using the same subnet as the PF (let's call them VMA)

  4. Some other VMs using another peered subnet (VMB)

  5. PF is configured to pass all IPSec traffic (Local Network set to https://0.0.0.0/0)

I have since enabled the following, to no avail:

  1. IP forwarding on the LAN-facing PF NIC

  2. Azure UDR rule for https://0.0.0.0/0, making the next hop at PF appliance and associated it with the LAN subnet which all VMA use

What I have found out so far is that:

  1. the traffic originating from an IPSec client to VMA shows up in tcpdump on both enc0 and LAN interfaces in PF, and:

  • one-way traffic shows up in tcpdump on VMA *only* after enabling IP forwarding, but the VM is sending ARP request for the IP address and doesn't seem to be using the routing table:

 

15:19:00.578098 IP 10.1.1.1.52074 > 10.1.0.6.22: Flags [S], seq 2814773395, win 65535, options [mss 1198,nop,wscale 6,nop,nop,TS val 442515869 ecr 0,sackOK,eol], length 0
15:19:00.578191 ARP, Request who-has 10.1.1.1 tell 10.1.0.6, length 28

  • Tracerouting on VMA to any of the IPSec clients also confirms that VMA is unaware of the table rule I added.

  • there are no firewall rules blocking it in PF

  1. the traffic originating from an IPSec client to VMB shows up in tcpdump on both enc0 and LAN interfaces in PF, and:

  • two-way traffic does show up in tcpdump on VMB

  1. I can ping/connect to any of the VMs running on Azure or IPSec clients directly from my PF instance.

I have run out of the ideas on how to proceed here. So far it looks like everything is fine on pfSense end.

It does seem like routing issue, even though it used to work just fine up until recently.

 

EDIT1: I just checked the Effective Routes on one of the VM's NIC, and am seeing the Default Active 10.1.0.0/23 Virtual network route, which is listed first, while my UDR is last. The 10.1.0.0/23 is my LAN subnet, which includes the IPSec clients. What does one do in this case?

 

EDIT2: I can see that the communication with VMB works fine because of the effective rule that handles that: Default Active 10.2.0.0/24 VNet peering

 

EDIT3: I added a UDR targeting my IPSec clients specifically, which according to https://aidanfinn.com/?p=21480 should take precedence over the default rules, but still to no avail 😕

 

EDIT4: the Next Hop in Azure correctly shows PF as as a next hop from VMA.

No Replies

Resources