Reply
Member
Posts: 292
Registered: ‎01-22-2014
Kudos: 37
Solutions: 14

IPSEC tunnel acting strange (possible firewall issue)

Hi All,

I initially had trouble getting this tunnel running with a netgear router. The problem turned out to be the netgear and the tunnel now establishes.

I now have a strange issue that I can't for the life of me figure out. Fresh eyes may see the problem.

The tunnel is accepting all traffic to the remote network but only some of it back the other way. As an example I can ping from a host on either side of the VPN to another on the other side. I can traceroute from site A (main office) but not from site B (remote site).

I have set up a rule on the firewall to accept all outgoing traffic so I can see what was passting and am getting the following:

Jul 10 10:42:46 DDHC-Router kernel: [IN_WAN-default-A]IN=eth0 OUT=eth1 MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 src=10.20.7.75 DST=10.20.1.254 LEN=1406 TOS=0x00 PREC=0x00 TTL=63 ID=14479 DF PROTO=TCP SPT=58005 DPT=8080 WINDOW=2920 RES=0x00 ACK URGP=0
Jul 10 10:42:46 DDHC-Router kernel: [IN_WAN-default-A]IN=eth0 OUT=eth1 MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 src=10.20.7.75 DST=10.20.1.254 LEN=1406 TOS=0x00 PREC=0x00 TTL=63 ID=2537 DF PROTO=TCP SPT=58004 DPT=8080 WINDOW=2920 RES=0x00 ACK URGP=0
Jul 10 10:42:48 DDHC-Router kernel: [IN_WAN-default-A]IN=eth0 OUT=eth1 MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 src=10.20.7.70 DST=10.20.1.254 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=5930 DF PROTO=TCP SPT=33520 DPT=8080 WINDOW=5840 RES=0x00 SYN URGP=0
Jul 10 10:42:48 DDHC-Router kernel: [IN_WAN-default-A]IN=eth0 OUT=eth1 MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 src=10.20.7.70 DST=10.20.1.254 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=5931 DF PROTO=TCP SPT=33520 DPT=8080 WINDOW=2920 RES=0x00 ACK URGP=0
Jul 10 10:42:48 DDHC-Router kernel: [IN_WAN-default-A]IN=eth0 OUT=eth1 MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 src=10.20.7.70 DST=10.20.1.254 LEN=1264 TOS=0x00 PREC=0x00 TTL=63 ID=5933 DF PROTO=TCP SPT=33520 DPT=8080 WINDOW=2920 RES=0x00 ACK PSH URGP=0

 10.20.7.X is the Unifi APs trying to talk to the Controller on 10.20.1.254. As you can see this appears to be accepted but, alas, it never reaches the controller.

VPN Setup:

ddhcadmin@DDHC-Router# show vpn ipsec
 auto-firewall-nat-exclude enable
 esp-group FOO {
     compression disable
     lifetime 3600
     mode tunnel
     pfs dh-group2
     proposal 1 {
         encryption aes256
         hash sha1
     }
 }
 ike-group FOO {
     lifetime 28800
     proposal 1 {
         dh-group 2
         encryption aes256
         hash sha1
     }
 }
 ipsec-interfaces {
     interface eth1
 }
 logging {
     log-modes all
 }
 nat-networks {
     allowed-network 0.0.0.0/0 {
     }
 }
 nat-traversal enable
 site-to-site {
     peer 82.xxx.xxx.165 {
         authentication {
             mode pre-shared-secret
             pre-shared-secret Lulw0rthVPNAcce55
         }
         connection-type initiate
         ike-group FOO
         local-ip 82.xxx.xxx.50
         tunnel 1 {
             allow-nat-networks disable
             allow-public-networks disable
             esp-group FOO
             local {
                 subnet 10.20.7.0/24
             }
             remote {
                 subnet 10.20.0.0/22
             }
         }
     }
 }

 Firewall setup:

ddhcadmin@DDHC-Router# show firewall
 all-ping enable
 broadcast-ping disable
 ipv6-receive-redirects disable
 ipv6-src-route disable
 ip-src-route disable
 log-martians enable
 name IN_WAN {
     default-action accept
     description ""
 }
 name WAN_IN {
     default-action drop
     description "WAN to internal"
     enable-default-log
     rule 1 {
         action accept
         description "Allow Established Sessions"
         log disable
         protocol all
         state {
             established enable
             related enable
         }
     }
     rule 2 {
         action accept
         description "Allow Office Connections"
         log disable
         protocol all
         source {
             address 10.20.0.0/22
         }
         state {
             established enable
             invalid enable
             new enable
             related enable
         }
     }
     rule 3 {
         action drop
         description "drop invalid"
         log enable
         protocol all
         state {
             established disable
             invalid enable
             new disable
             related disable
         }
     }
 }
 name WAN_LOCAL {
     default-action drop
     description "WAN to router"
     rule 1 {
         action accept
         description "Accept Established sessions"
         log disable
         protocol all
         state {
             established enable
             related enable
         }
     }
     rule 2 {
         action accept
         description "Accept Office Connections"
         log disable
         protocol all
         source {
             address 10.20.0.0/22
         }
         state {
             established enable
             invalid disable
             new enable
             related enable
         }
     }
     rule 3 {
         action accept
         description "Office WAN Address"
         log disable
         protocol all
         source {
             address 82.xxx.xxx.165/32
         }
         state {
             established enable
             invalid disable
             new enable
             related enable
         }
     }
     rule 4 {
         action drop
         description "drop invalid"
         log disable
         state {
             invalid enable
         }
     }
 }
 receive-redirects disable
 send-redirects enable
 source-validation disable
 syn-cookies enable

 

 

 

Member
Posts: 292
Registered: ‎01-22-2014
Kudos: 37
Solutions: 14

Re: IPSEC tunnel acting strange (possible firewall issue)

Okay, found out something else. I though mabe NAT was causing issues (I do have exlude from NAT set on the VPN though).

When I run a traceroute with logging enabled on the masqurade rule I can see that when I trace to site A (10.20.0.0/22) it logs the destination as 8.8.8.8 Googles DNS server.

Is this the edgerouter behaving like it does not know the route? I'm confused.

Member
Posts: 292
Registered: ‎01-22-2014
Kudos: 37
Solutions: 14

Re: IPSEC tunnel acting strange (possible firewall issue)

[ Edited ]

Okay, I'm pretty sure I need an exlude from NAT rule, the auto exlude does not seem to work. Any tips?

Member
Posts: 292
Registered: ‎01-22-2014
Kudos: 37
Solutions: 14

Re: IPSEC tunnel acting strange (possible firewall issue)

Well after installing wireshark on the destination server I can observe the UDP datagrams from the traceroute getting to the server.

Is there a way to test why these are not showing?

A tracert from the server to the remote network works fine.

Member
Posts: 292
Registered: ‎01-22-2014
Kudos: 37
Solutions: 14

Re: IPSEC tunnel acting strange (possible firewall issue)

Please delete this mods.

It seems to be a similar issue to this post http://community.ubnt.com/t5/EdgeMAX/IPSec-performance-issue/m-p/903564#M34901

So I have added to that.

Previous Employee
Posts: 13,551
Registered: ‎06-10-2011
Kudos: 5465
Solutions: 1656
Contributions: 2

Re: IPSEC tunnel acting strange (possible firewall issue)

Are you sure it is the same issue? It sounds like in this case traceroute is not working in one direction for example, which doesn't sound like the performance issue discussed in the other thread?

Member
Posts: 292
Registered: ‎01-22-2014
Kudos: 37
Solutions: 14

Re: IPSEC tunnel acting strange (possible firewall issue)


@UBNT-ancheng wrote:

Are you sure it is the same issue? It sounds like in this case traceroute is not working in one direction for example, which doesn't sound like the performance issue discussed in the other thread?


HI @UBNT-ancheng 

The reason I think it is connected to the other post is because if I run a ping with a packet size of 1500 I will get a 40% or greater loss.

At 1492 this loss was at 77%.

At 1300 there was no loss.

It is as if the EdgeRouter is mangling the packets somehow.

Member
Posts: 292
Registered: ‎01-22-2014
Kudos: 37
Solutions: 14

Re: IPSEC tunnel acting strange (possible firewall issue)

Okay, I have decided that it is probably the netgear causing the problems so have stumped up the cash for an EdgeRouter Pro to purge myself of the Netgear.

I will let you know how it goes.

Highlighted
Emerging Member
Posts: 87
Registered: ‎09-25-2013
Kudos: 128
Solutions: 6

Re: IPSEC tunnel acting strange (possible firewall issue)

[ Edited ]

Ragarath,

 

I think I might know what is happening as I have been battling a similar issue. 

I believe that your initial hunch on the auto-firewall-nat-exclude may be to blame.  I have noticed that after a reboot that any device that is contstantly sending packets to an ip range that is destined to traverse any IPSec tunnel will somehow be caught in the NAT tables for around 20 minutes for some hosts, to forever for others. 

if I simply reboot every device that I can't ping or If I run the command  "clear connection-tracking"  it appears to resolve the issue until the next reboot. 

By manually adding a NAT exlude rule for each of my IPSec tunnels the problems dissapeared. 

I don't know at what point the exclude nat rule is created during bootup, but it seems that there is a space when the masquerade rule is created, begins to capture packets destined for a vpn tunnel, gets marked by the connection tracking, and then the exclude rule is created.  The problem appears to be that the connection tracking keeps the related traffic in the masquerade rule until the traffic stops long enough (a reboot) that it is then allowed to be sent to the exclude rule. 

I discovered all of this when some of my Shoretel SIP traffic was acting weird and couldn't complete calls.  I started ICMP pings between vpn's and ran a tcpdump on the wan interface and noticed that I was geting ICMP rejects from my ISP's router for traffic that should be going over the vpn. 

I have 24 ipsec/gre tunnels that have this issue after a reboot.  I use multiping to monitor all of my vpn connections and after a reboot non of the firewalls or hosts that I'm pinging will respond, however any host that wasn't being pinged during the bootup of the ERL I can ping successfully.  Then if I run "clear connection-tracking" all of the firewalls/hosts begin to respond.   

I have confirmed this behavior on v1.41, v1.5, and 1.6.0alpha3, however the 1.6 version differs in that the gre-IPSec connections don't seem to be affected, only the pure IPSec. 

Weird thing is that after about 20 minutes most of the hosts begin to respond to ICMP, but some traffic seems to be caught in the masquerade table forever. 

UBNT can you confirm this behavior?  

-Eli

Previous Employee
Posts: 13,551
Registered: ‎06-10-2011
Kudos: 5465
Solutions: 1656
Contributions: 2

Re: IPSEC tunnel acting strange (possible firewall issue)

Thanks for looking into this and providing the details! We'll need to see if we can replicate the issue and then look into it.

Member
Posts: 292
Registered: ‎01-22-2014
Kudos: 37
Solutions: 14

Re: IPSEC tunnel acting strange (possible firewall issue)

Hey @ejsearle I've not been able to look at this for a while but thanks for your post. I am planning on re-looking at this this week so I will try and duplicate your results.

Reply