Reply
Highlighted
New Member
Posts: 15
Registered: ‎12-01-2014
Solutions: 2

AWS IPSec VPN w/ VTI unable to keep stable connection

Hello,

 

I have an EdgeRouter Pro. I have IPSec tunnels configured between Amazon AWS and my office. The tunnels each use a vti interface (vti1 & vti2). After bringing up the tunnels the connection will be up and all traffic passes for some time, often 12-24 hours. After that one of the vti interfaces will show as Admin Down/Down and some traffic will no longer pass. This is how the vti interfaces look when traffic is partially broken:

 

admin@ER:~$ show interfaces vti vti1 brief
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface    IP Address                        S/L  Description
---------    ----------                        ---  -----------
vti1         169.254.255.42/30                 u/u
admin@ER:~$ show interfaces vti vti2 brief
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface    IP Address                        S/L  Description
---------    ----------                        ---  -----------
vti2         169.254.255.46/30                 A/D

 

To get all traffic working again, I must clear the ipsec peer associated with the vti that shows as down. Once I do that everything is working again:

admin@ER:~$ clear vpn ipsec-peer 54.240.217.164
Clearing tunnel vti with peer 54.240.217.164...
admin@ER:~$ show interfaces vti vti2 brief
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface    IP Address                        S/L  Description
---------    ----------                        ---  -----------
vti2         169.254.255.46/30                 u/u

 

 

When everything is working this is how they look:

admin@ER:~$ show interfaces vti vti1 brief
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface    IP Address                        S/L  Description
---------    ----------                        ---  -----------
vti1         169.254.255.42/30                 u/u
admin@ER:~$ show interfaces vti vti2 brief
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface    IP Address                        S/L  Description
---------    ----------                        ---  -----------
vti2         169.254.255.46/30                 u/u

 

 

My config follows the config here, although I found that link after I had configured my tunnels:

http://community.ubnt.com/t5/EdgeMAX/EdgeRouter-amp-Amazon-VPC-dynamically-routed-IPSec-tunnel/td-p/...

 

Can anyone give me some pointers on what is happening or how to get logs from when the vti changes to state "Down"?

 

Thank you,

James 

Previous Employee
Posts: 10,504
Registered: ‎06-09-2011
Kudos: 3124
Solutions: 945
Contributions: 16

Re: AWS IPSec VPN w/ VTI unable to keep stable connection

[ Edited ]

The VyOS guys had a patch for what sounds like a similar situation.  Maybe you could try their fix which was to edit /usr/lib/ipsec/vti-up-down.sh and change the following:

@@ -5,7 +5,7 @@
source /etc/default/vyatta
source /etc/default/locale
case "$PLUTO_VERB" in
-route-client)
+route-client | up-client)
/opt/vyatta/sbin/vyatta-vti-config.pl --updown --intf=$1 --action=up
;;
down-client)

 Basically add "| up-client" to the line with "route-client".

EdgeMAX Router Software Development
New Member
Posts: 15
Registered: ‎12-01-2014
Solutions: 2

Re: AWS IPSec VPN w/ VTI unable to keep stable connection

This fix has been applied to the EdgeRouter. I will let you know in a couple days if the connection stays up.

 

Thank you,

James

SuperUser
Posts: 20,367
Registered: ‎09-17-2013
Kudos: 5104
Solutions: 1455

Re: AWS IPSec VPN w/ VTI unable to keep stable connection

[ Edited ]

UBNT-stig wrote:

The VyOS guys had a patch for what sounds like a similar situation.  Maybe you could try their fix which was to edit /usr/lib/ipsec/vti-up-down.sh and change the following:

@@ -5,7 +5,7 @@
source /etc/default/vyatta
source /etc/default/locale
case "$PLUTO_VERB" in
-route-client)
+route-client | up-client)
/opt/vyatta/sbin/vyatta-vti-config.pl --updown --intf=$1 --action=up
;;
down-client)

 Basically add "! up-client" "| up-client" to the line with "route-client".


Had a typo there, stig  ... bang instead of a pipe would probably break in new and interesting ways though. Man Wink 

 

(not that I should speak ... about the only thing i'm GOOD at is making infinite loops Banghead)

New Member
Posts: 15
Registered: ‎12-01-2014
Solutions: 2

Re: AWS IPSec VPN w/ VTI unable to keep stable connection

Thanks dpurgert. I noticed the typo and used pipe instead of exclamation point. 

 

Thank you,

James

Previous Employee
Posts: 10,504
Registered: ‎06-09-2011
Kudos: 3124
Solutions: 945
Contributions: 16

Re: AWS IPSec VPN w/ VTI unable to keep stable connection

@dpurgert good catch!  I fixed my typo for furture reference.

EdgeMAX Router Software Development
New Member
Posts: 15
Registered: ‎12-01-2014
Solutions: 2

Re: AWS IPSec VPN w/ VTI unable to keep stable connection

It has now been a couple of days and the tunnles to AWS have remained up. I am not yet ready to call this fixed, but rarely have I seen the connection stay up for this long.

 

I will continue to post updates.

 

Thank you,

James Crow

Previous Employee
Posts: 10,504
Registered: ‎06-09-2011
Kudos: 3124
Solutions: 945
Contributions: 16

Re: AWS IPSec VPN w/ VTI unable to keep stable connection


the_crowbar wrote:

It has now been a couple of days and the tunnles to AWS have remained up. I am not yet ready to call this fixed, but rarely have I seen the connection stay up for this long.

 

I will continue to post updates.

 

Thank you,

James Crow


Thanks for the update.  Unfortunately the VyOS guys have reverted that patch since they claim is causes http://bugzilla.vyos.net/show_bug.cgi?id=291  We'll have to evaluate if there's a better fix for buy 291 than removing this patch.

EdgeMAX Router Software Development
New Member
Posts: 15
Registered: ‎12-01-2014
Solutions: 2

Re: AWS IPSec VPN w/ VTI unable to keep stable connection

Just another update. It has been a week since I restarted the router. (I actually have two from different sites that I made this change to.) I have not had to clear the ipsec peer on either one in 7 days.

 

This is better than it was before. I was clearing at least every 24-48 hours.

 

I looked at bug 291 and I do not seem to be hitting it. The other end of my tunnels is AWS so that may have something to do with it.

 

I will continue to update this thread as time goes on.

 

Thank you,

James

New Member
Posts: 4
Registered: ‎07-29-2014
Kudos: 6

Re: AWS IPSec VPN w/ VTI unable to keep stable connection

I have this same issue I believe. One side of tunnel believes it's up, the other side shows as down. Any developments? Disabling and re-enabling interface on the down side of the tunnel or disabling the interface and re-enable in gui fixes issue, temporarily. Restarting router with the up side of the tunnel does not prompt the down side to reconnect. I did not try the fix above because I didn't really understand the issue it causes about adding duplicate routes. When one side is down there are no connected routes showing for 10.60.204.0/24 network on either side of the tunnel I believe, I can't check for sure now because I fixed the issue after pasting below.

 

Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
vti60 10.60.204.204/24 A/D RDO

 

Interface IP Address S/L Description
--------- ---------- --- -----------
vti204 10.60.204.60/24 u/u CDO

 

New Member
Posts: 9
Registered: ‎07-23-2016

Re: AWS IPSec VPN w/ VTI unable to keep stable connection

I would also like to ask if there is an update on the issue. The EdgeMAX router I have already seems to have a newer kind of patch (I can see the "| up-client" added in the config), but still the tunnel connection drops every two or three days and needs to be "cleaned".

 

The logs do not show anything significant (to me at least), the vti interface just gets deactivated. Afterwards, I get messages that the tunnel has been established, but that is of no use since the interface is administratively down. I am posting the log (obfuscated IPs):

 

Apr 30 07:18:50 08[IKE] <peer-111.111.111.111-tunnel-vti|13> closing expired CHILD_SA peer-111.111.111.111-tunnel-vti{2} with SPIs c53f3b1e_i 86211b64_o and TS 0.0.0.0/0 === 0.0.0.0/0 
Apr 30 07:24:57 14[KNL] creating delete job for ESP CHILD_SA with SPI c7cefb34 and reqid {1}
Apr 30 07:24:57 14[IKE] <peer-222.222.222.222-tunnel-vti|12> closing expired CHILD_SA peer-222.222.222.222-tunnel-vti{1} with SPIs c7cefb34_i 9be4c806_o and TS 0.0.0.0/0 === 0.0.0.0/0 
Apr 30 07:24:57 09[KNL] creating delete job for ESP CHILD_SA with SPI 9be4c806 and reqid {1}
Apr 30 07:48:10 08[KNL] creating rekey job for ESP CHILD_SA with SPI c4b83f3f and reqid {2}
Apr 30 07:48:10 10[IKE] <peer-111.111.111.111-tunnel-vti|13> CHILD_SA peer-111.111.111.111-tunnel-vti{2} established with SPIs cd9c412b_i 5f4871f6_o and TS 0.0.0.0/0 === 0.0.0.0/0 
Apr 30 07:51:15 15[KNL] creating rekey job for ESP CHILD_SA with SPI c0b7c090 and reqid {1}
Apr 30 07:51:15 09[IKE] <peer-222.222.222.222-tunnel-vti|12> CHILD_SA peer-222.222.222.222-tunnel-vti{1} established with SPIs c2f50c35_i f4da5ca0_o and TS 0.0.0.0/0 === 0.0.0.0/0 
Apr 30 07:51:49 11[KNL] creating rekey job for ESP CHILD_SA with SPI 392793bc and reqid {2}
Apr 30 07:53:33 15[KNL] creating rekey job for ESP CHILD_SA with SPI 8feed07a and reqid {1}
Apr 30 08:01:51 08[KNL] creating delete job for ESP CHILD_SA with SPI c4b83f3f and reqid {2}
Apr 30 08:01:51 08[IKE] <peer-111.111.111.111-tunnel-vti|13> closing expired CHILD_SA peer-111.111.111.111-tunnel-vti{2} with SPIs c4b83f3f_i 392793bc_o and TS 0.0.0.0/0 === 0.0.0.0/0 
Apr 30 08:01:51 11[KNL] creating delete job for ESP CHILD_SA with SPI 392793bc and reqid {2}
Apr 30 08:09:11 04[KNL] creating delete job for ESP CHILD_SA with SPI c0b7c090 and reqid {1}
Apr 30 08:09:11 04[IKE] <peer-222.222.222.222-tunnel-vti|12> closing expired CHILD_SA peer-222.222.222.222-tunnel-vti{1} with SPIs c0b7c090_i 8feed07a_o and TS 0.0.0.0/0 === 0.0.0.0/0 
Apr 30 08:09:11 15[KNL] creating delete job for ESP CHILD_SA with SPI 8feed07a and reqid {1}
Apr 30 08:32:17 15[KNL] creating rekey job for ESP CHILD_SA with SPI cd9c412b and reqid {2}
Apr 30 08:32:17 14[IKE] <peer-111.111.111.111-tunnel-vti|13> CHILD_SA peer-111.111.111.111-tunnel-vti{2} established with SPIs ccf4f172_i 7869b410_o and TS 0.0.0.0/0 === 0.0.0.0/0 
Apr 30 08:34:42 14[KNL] creating rekey job for ESP CHILD_SA with SPI c2f50c35 and reqid {1}
Apr 30 08:34:43 13[IKE] <peer-222.222.222.222-tunnel-vti|12> CHILD_SA peer-222.222.222.222-tunnel-vti{1} established with SPIs c9a8511a_i e6748a40_o and TS 0.0.0.0/0 === 0.0.0.0/0 
Apr 30 08:37:51 09[KNL] creating rekey job for ESP CHILD_SA with SPI 5f4871f6 and reqid {2}
Apr 30 08:39:53 14[KNL] creating rekey job for ESP CHILD_SA with SPI f4da5ca0 and reqid {1}
Apr 30 08:48:10 16[IKE] <peer-111.111.111.111-tunnel-vti|13> closing CHILD_SA peer-111.111.111.111-tunnel-vti{2} with SPIs cd9c412b_i (28123 bytes) 5f4871f6_o (32647 bytes) and TS 0.0.0.0/0 === 0.0.0.0/0 
Apr 30 08:51:15 15[IKE] <peer-222.222.222.222-tunnel-vti|12> closing CHILD_SA peer-222.222.222.222-tunnel-vti{1} with SPIs c2f50c35_i (23003 bytes) f4da5ca0_o (32103 bytes) and TS 0.0.0.0/0 === 0.0.0.0/0 
Apr 30 09:17:40 14[KNL] creating rekey job for ESP CHILD_SA with SPI c9a8511a and reqid {1}
Apr 30 09:17:40 08[IKE] <peer-222.222.222.222-tunnel-vti|12> CHILD_SA peer-222.222.222.222-tunnel-vti{1} established with SPIs ccbe4864_i e2d41fc9_o and TS 0.0.0.0/0 === 0.0.0.0/0 
Apr 30 09:18:20 15[KNL] creating rekey job for ESP CHILD_SA with SPI ccf4f172 and reqid {2}
Apr 30 09:18:21 13[IKE] <peer-111.111.111.111-tunnel-vti|13> CHILD_SA peer-111.111.111.111-tunnel-vti{2} established with SPIs c500696c_i bab82cd1_o and TS 0.0.0.0/0 === 0.0.0.0/0 
Apr 30 09:19:21 14[KNL] creating rekey job for ESP CHILD_SA with SPI e6748a40 and reqid {1}
Apr 30 09:21:04 15[KNL] creating rekey job for ESP CHILD_SA with SPI 7869b410 and reqid {2}
Apr 30 09:32:17 04[IKE] <peer-111.111.111.111-tunnel-vti|13> closing CHILD_SA peer-111.111.111.111-tunnel-vti{2} with SPIs ccf4f172_i (30048 bytes) 7869b410_o (34632 bytes) and TS 0.0.0.0/0 === 0.0.0.0/0 
Apr 30 09:32:29 11[IKE] <peer-111.111.111.111-tunnel-vti|13> reauthenticating IKE_SA peer-111.111.111.111-tunnel-vti[13]
Apr 30 09:32:29 11[IKE] <peer-111.111.111.111-tunnel-vti|13> initiating Main Mode IKE_SA peer-111.111.111.111-tunnel-vti[14] to 111.111.111.111
Apr 30 09:32:30 12[IKE] <peer-111.111.111.111-tunnel-vti|14> IKE_SA peer-111.111.111.111-tunnel-vti[14] established between 1.2.3.4[1.2.3.4]...111.111.111.111[111.111.111.111]
Apr 30 09:33:00 08[IKE] <peer-111.111.111.111-tunnel-vti|14> closing CHILD_SA peer-111.111.111.111-tunnel-vti{2} with SPIs c500696c_i (9316 bytes) bab82cd1_o (10824 bytes) and TS 0.0.0.0/0 === 0.0.0.0/0 
Apr 30 09:33:01 09[KNL] interface vti0 deactivated

from then on:
May 28 13:43:14 04[IKE] <peer-111.111.111.111-tunnel-vti|108> initiating Main Mode IKE_SA peer-111.111.111.111-tunnel-vti[109] to 111.111.111.111
May 28 13:43:15 10[IKE] <peer-111.111.111.111-tunnel-vti|109> IKE_SA peer-111.111.111.111-tunnel-vti[109] established between 1.2.3.4[1.2.3.4]...111.111.111.111[111.111.111.111]
May 28 21:26:01 14[IKE] <peer-111.111.111.111-tunnel-vti|109> reauthenticating IKE_SA peer-111.111.111.111-tunnel-vti[109]
New Member
Posts: 9
Registered: ‎07-23-2016

Re: AWS IPSec VPN w/ VTI unable to keep stable connection

[ Edited ]

Update:

I am now trying to monitor whether the vti interfaces go into A/D status and clean the tunnel using a cron script. I will let you know results. The interface shuts down after 48-72 hours, so it is too early to say if this will do the trick.

 

 

#!/bin/bash
run=/opt/vyatta/bin/vyatta-op-cmd-wrapper
$run show interfaces vti | grep "A/D" | grep "vti0"
if [ $? == 0 ]
then
 $run clear vpn ipsec-peer 1.2.3.4
fi
$run show interfaces vti | grep "A/D" | grep "vti1"
if [ $? == 0 ]
then
 $run clear vpn ipsec-peer 5.6.7.8
fi
New Member
Posts: 9
Registered: ‎07-23-2016

Re: AWS IPSec VPN w/ VTI unable to keep stable connection

I could confirm that this is working.

Today one of the vti interfaces shut down again and the script above could bring it and the tunnel back up.

Reply