Site-to-Site established but no traffic [ Edited ]

[Torpi]

Hello,

this has been driving me crazy me for few days and I have not been able to find a solution after reading almost everything here in this forum. Any help to point me in the right direction would be appreciated!

So I have a Site-to-Site tunnel established (ER4 <--> Modem <Site-to-Site> ISP_Router <--> ER5Poe <--> Server with IPSEC VPN.

The problem is I cannot ping/access/etc... anyting from one site to the other. I have read this one (https://help.ubnt.com/hc/en-us/articles/115013382567-EdgeRouter-IPsec-Site-to-Site-VPN-behind-NAT) since I have double NAT on the router side because of my provider but that is not recognised as a valid command :

set vpn site-to-site peer 90.127.196.91 authentication remote-id 192. 168.1.30
The specified configuration node is not valid

Here is my status:

ubnt@ubnt:~$ sudo ipsec status
Routed Connections:
peer-levisip.dyndns.org-tunnel-1{1}: ROUTED, TUNNEL
peer-levisip.dyndns.org-tunnel-1{1}: 192.168.101.0/24 === 192.168.1.0/24
Security Associations (1 up, 0 connecting):
peer-levisip.dyndns.org-tunnel-1[1]: ESTABLISHED 23 minutes ago, 24.212.241.155[24.212.241.155]...90.127.196.91[192.168.1.30]
peer-levisip.dyndns.org-tunnel-1{1}: INSTALLED, TUNNEL, ESP in UDP SPIs: c716150d_i 6848285f_o
peer-levisip.dyndns.org-tunnel-1{1}: 192.168.101.0/24 === 192.168.1.0/24

ubnt@ubnt:~$ show vpn ipsec sa
peer-levisip.dyndns.org-tunnel-1: #1, ESTABLISHED, IKEv1, c72a977cedc23c05:93a7bfe210f771c2
local '24.212.241.155' @ 24.212.241.155
remote '192.168.1.30' @ 90.127.196.91
AES_CBC-256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024
established 1739s ago, reauth in 26506s
peer-levisip.dyndns.org-tunnel-1: #1, INSTALLED, TUNNEL-in-UDP, ESP:AES_CBC-256/HMAC_SHA1_96/MODP_1024
installed 1739 ago, rekeying in 810s, expires in 1861s
in c716150d, 0 bytes, 0 packets
out 6848285f, 60144 bytes, 720 packets, 4s ago
local 192.168.101.0/24
remote 192.168.1.0/24

ubnt@ubnt:~$ show vpn ipsec state
src 24.212.241.155 dst 90.127.196.91
proto esp spi 0x6848285f reqid 1 mode tunnel
replay-window 32 flag af-unspec
auth-trunc hmac(sha1) 0xb3f752fa0faad08632aa1e2738834d99751cb1a8 96
enc cbc(aes) 0xca72eeb10a18af38bd1d974c478c71dba443bde1c9575ae29724fbea860fbc1c
encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
src 90.127.196.91 dst 24.212.241.155
proto esp spi 0xc716150d reqid 1 mode tunnel
replay-window 32 flag af-unspec
auth-trunc hmac(sha1) 0xce8bb0516f7b41ee647d0af3893bc665a4ba9c67 96
enc cbc(aes) 0x118ba43c7dcbbcf3abb4853302d205477ec12378bf4a971ce920ee110438866d
encap type espinudp sport 4500 dport 4500 addr 0.0.0.0

The only "strange" thing I can see here is that there is no IPSec Interfaces listed here. But the IPSec Interface is noted as DEPRECATED in the Config Tree of the router.

ubnt@ubnt:~$ show vpn ipsec status
IPSec Process Running PID: 17891

1 Active IPsec Tunnels

IPsec Interfaces :
ubnt@ubnt:~$

The Tunnel has been set with the UI on the ER4 side.

Do you have any clue where to start to understand why a computer from the right subnet cannot acces the other side of the tunnel? Same thing from the router CLI directly.

Thank you so much for pointing me in the right direction and happy new year to you all!

Torpi

[redfive]

Try

set vpn ipsec site-to-site peer 90.127.196.91 authentication remote-id 192.168.1.30

And, on both routers

Spoiler

configure
set vpn ipsec nat-traversal enable
commit;save
exit
restart vpn

Cheers,

jonatha

[Torpi]

Thank you for pointing me in the right direction with the missing ipsec!

However, I cannot commit it because it complains about IKE group:

ubnt@ubnt# commit
[ vpn ]
[ vpn ipsec site-to-site peer 90.127.196.91 ike-group ]
VPN configuration error: No IKE group specified for peer "90.127.196.91".

The IP is the one corresponding to my.fqdn.to.the.vpn.com in my configuration.

Feeling a bit lost. Do you spot something wrong here? 

Thank you all!

vpn {
    ipsec {
        auto-firewall-nat-exclude enable
        esp-group FOO0 {
            compression disable
            lifetime 3600
            mode tunnel
            pfs enable
            proposal 1 {
                encryption aes256
                hash sha1
            }
        }
        ike-group FOO0 {
            ikev2-reauth no
            key-exchange ikev1
            lifetime 28800
            proposal 1 {
                dh-group 2
                encryption aes256
                hash sha1
            }
        }
        site-to-site {
            peer my.fqdn.to.the.vpn.com {
                authentication {
                    mode pre-shared-secret
                    pre-shared-secret ******
                }
                connection-type initiate
                description VPN-Site-abc
                ike-group FOO0
                ikev2-reauth inherit
                local-address any
                tunnel 1 {
                    allow-nat-networks disable
                    allow-public-networks disable
                    esp-group FOO0
                    local {
                        prefix 192.168.101.0/24
                    }
                    remote {
                        prefix 192.168.1.0/24
                    }
                }
            }
        }
    }
}

[Torpi]

OK, I figured out the command did not like the fqdn in the peer so I replaced it with the IP address and then was able to commit it.

Unfortunately that does not really change anything in the connection.

ubnt@ubnt:~$ sudo ipsec status                                                                                                      
Routed Connections:                                                                                                                 
peer-90.127.196.91-tunnel-1{1}:  ROUTED, TUNNEL                                                                                     
peer-90.127.196.91-tunnel-1{1}:   192.168.101.0/24 === 192.168.1.0/24                                                               
Security Associations (1 up, 0 connecting):                                                                                         
peer-90.127.196.91-tunnel-1[1]: ESTABLISHED 7 minutes ago, 24.212.241.155[24.212.241.155]...90.127.196.91[192.168.1.30]             
peer-90.127.196.91-tunnel-1{1}:  INSTALLED, TUNNEL, ESP in UDP SPIs: cf594c66_i 39a51b0a_o                                          
peer-90.127.196.91-tunnel-1{1}:   192.168.101.0/24 === 192.168.1.0/24                                                               


ubnt@ubnt:~$ show vpn ipsec sa                                                                                                      
peer-90.127.196.91-tunnel-1: #1, ESTABLISHED, IKEv1, 4d96acb92b5599bc:74ff46a3d30a5441                                              
  local  '24.212.241.155' @ 24.212.241.155                                                                                          
  remote '192.168.1.30' @ 90.127.196.91                                                                                             
  AES_CBC-256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024                                                                                  
  established 429s ago, reauth in 27408s                                                                                            
  peer-90.127.196.91-tunnel-1: #1, INSTALLED, TUNNEL-in-UDP, ESP:AES_CBC-256/HMAC_SHA1_96/MODP_1024                                 
    installed 429 ago, rekeying in 2221s, expires in 3171s                                                                          
    in  cf594c66,      0 bytes,     0 packets                                                                                       
    out 39a51b0a,  15600 bytes,   188 packets,     4s ago                                                                           
    local  192.168.101.0/24                                                                                                         
    remote 192.168.1.0/24                                                                                                           


ubnt@ubnt:~$ show vpn ipsec state                                                                                                   
src 24.212.241.155 dst 90.127.196.91                                                                                                
        proto esp spi 0x39a51b0a reqid 1 mode tunnel                                                                                
        replay-window 32 flag af-unspec                                                                                             
        auth-trunc hmac(sha1) 0xc05eb23b6da927bd57f6bb466e3a94331b2dcf48 96                                                         
        enc cbc(aes) 0x91173347c156940f9179f4af6d9513b4546e4550d0b28cf65106c869df6e129c                                             
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0                                                                      
src 90.127.196.91 dst 24.212.241.155                                                                                                
        proto esp spi 0xcf594c66 reqid 1 mode tunnel                                                                                
        replay-window 32 flag af-unspec                                                                                             
        auth-trunc hmac(sha1) 0x7e5fca17460b4b7f27d56e93d61b187cd43461e8 96                                                         
        enc cbc(aes) 0x17cba8c7bbc77b78b3bcda2d6d1503cae1abb015f51415d00dd95df7aa4b2e2f                                             
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0                                                                      
ubnt@ubnt:~$ show vpn ipsec status                                                                                                  


IPSec Process Running PID: 4544                                                                                                     
                                                                                                                                    
1 Active IPsec Tunnels                                                                                                              
                                                                                                                                    
IPsec Interfaces :                                                                                                                  
ubnt@ubnt:~$                                        

I am not sure to read correctly the logs. What should I get when it is proprely established? I have the impression the the tunnel is established but cannot connect nor ping anything in 192.168.1.0 subnet from 192.168.101.0 subnet (or from the router directly)

Coffee, thinking, and reading again documentations.

Thank you if you have any advice!

Torpi

[redfive]

Cn you post the full configs (sanitized, only where needed) of both routers ? With only snippets, is always hard the troubleshooting ...

Fastly, if on the remote router, the ip address  192.168.1.1 is on eth1, can you try to ping for eg.  the host 192.168.1.10 from an host of the 192.168.101.0/24 network, but before, on the remote router, issue

Spoiler

sudo tcpdump -ni eth1 host 192.168.101.10 and icmp

Do you see anything in the tcpdump's output ?
Cheers,
jonatha

[Torpi]

Thanks Jonatha, sure, it is attached. I removed non important FW Rules and NAT as well as passwords and mac addresses.

One thing I do not see compared to what you do with the CLI to set it up (like here https://help.ubnt.com/hc/en-us/articles/115011377588-EdgeRouter-Route-Based-Site-to-Site-IPsec-VPN) is step 7 and 8, setting the ip and route for vti. 

I tried from multiple machine the tcpdump but I do not receive anything.

The VPN on the MAIN SITE (SoftEther configured like this https://www.softether.org/4-docs/2-howto/9.L2TPIPsec_Setup_Guide_for_SoftEther_VPN_Server/6.Cisco_IO...) with works perfectly with I try to connect to it with Windows or iPhone VPN, so I don't think there is anything avout the FW there.

I am really curious about not having an interface here, what do you think?

ubnt@ubnt:~$ show vpn ipsec status
IPSec Process Running PID: 17891

1 Active IPsec Tunnels

IPsec Interfaces :
ubnt@ubnt:~$

Torpi

config.boot.MAIN_SITE.txt

config.boot.REMOTE_SITE.txt

[redfive]

Can you try, on the main site

Spoiler

configure
set protocols static route 192.168.101.0/24 next-hop 192.168.1.30
commit

[Torpi]

Make sense, since the VPN is this .30 server.

ubnt@ubnt:~$ configure                                                                                                                                                                                                                                   
[edit]                                                                                                                                                                                                                                                   
ubnt@ubnt# set protocols static route 192.168.101.0/24 next-hop 192.168.1.30                                                                                                                                                                             
[edit]                                                                                                                                                                                                                                                   
ubnt@ubnt# commit                                                                                                                                                                                                                                        
[ protocols static route 192.168.101.0/24 next-hop 192.168.1.30 ]                                                                                                                                                                                        
no ip static 192.168.101.0/24 192.168.1.30 fall-over bfd                                                                                                                                                                                                 
                                                                                                                                                                                                                                                         
[edit]                                                                                                                                                                                                                                                   
ubnt@ubnt# save                                                                                                                                                                                                                                          
Saving configuration to '/config/config.boot'...                                                                                                                                                                                                         
Done                                                                                                                                                                                                                                                     
[edit]                                                                                                                                                                                                                                                   
ubnt@ubnt# exit                                                                                                                                                                                                                                          
exit                                                                                                                                                                                                                                                     
ubnt@ubnt:~$ ping 192.168.101.103                                                                                                                                                                                                                        
PING 192.168.101.103 (192.168.101.103) 56(84) bytes of data.                                                                                                                                                                                             
^C                                                                                                                                                                                                                                                       
--- 192.168.101.103 ping statistics ---                                                                                                                                                                                                                  
9 packets transmitted, 0 received, 100% packet loss, time 8000ms 

The rule is applied, though, because from one computer on the main site, I have this now:

remi@griffin:~$ ping 192.168.101.103                                                                                                                                                                                                                     
PING 192.168.101.103 (192.168.101.103) 56(84) bytes of data.                                                                                                                                                                                             
From 192.168.1.1: icmp_seq=2 Redirect Host(New nexthop: 192.168.1.30)                                                                                                                                                                                    
From 192.168.1.1: icmp_seq=3 Redirect Host(New nexthop: 192.168.1.30)                                                                                                                                                                                    
From 192.168.1.1: icmp_seq=4 Redirect Host(New nexthop: 192.168.1.30)                                                                                                                                                                                    
From 192.168.1.1: icmp_seq=5 Redirect Host(New nexthop: 192.168.1.30)                                                                                                                                                                                    
From 192.168.1.1: icmp_seq=6 Redirect Host(New nexthop: 192.168.1.30)                                                                                                                                                                                    
From 192.168.1.1: icmp_seq=8 Redirect Host(New nexthop: 192.168.1.30)                                                                                                                                                                                    
^C                                                                                                                                                                                                                                                       
--- 192.168.101.103 ping statistics ---                                                                                                                                                                                                                  
9 packets transmitted, 0 received, 100% packet loss, time 8168ms 

[redfive]

Directly from the .30 host (the VPN server), are you able to ping something on the 192.168.101.0/24 network ? If yes, can you issue, on the remote

Spoiler

sudo tcpdump -ni eth1 host 192.168.1.30 and icmp

And then launch the ping's ?

[Torpi]

that's the thing, cannot ping anything from the .30 (VPN) to the remote .101.0 subnet. Cannot ping any computer nor the router.

Same thing from the remote .101.0 subnet, cannot ping anything (nor the router) in the main site .1.0 subnet.

I've tried with Automatically open firewall and excelude from NAT and removed 4500/500 and esp in WAN_LOCAL on remote but this does not change anything.

The only rule on remote site I have is IPSEC source 192.168.1.0/24 destination 192.168.101.0/24 all accept match ipsec.

Shoudl I need anything else here?

[redfive]

Can you add, on remote

Spoiler

configure
set firewall name WAN_LOCAL rule 30 action accept
set firewall name WAN_LOCAL rule 30 destination address 192.168.101.0/24
set firewall name WAN_LOCAL rule 30 source address 192.168.1.30
set firewall name WAN_LOCAL rule 30 ipsec match-ipsec
commit
set vpn ipsec nat-trasversal enable
commit

Then from the .1.30, ping 192.168.101.1 or, try an SSH session still on 192.168.101.1 ...

[Torpi]

Still no luck. What drives me nuts is that ipsec status shows established, installed tunnel ESP in UDP!

So, just let a small test computer that I am going to format on remote site, backup the remote router and I've been crazy enough to apply Default Action to Accept to both WAN_IN and WAN_LOCAL. Still cannot ping (or connect to) anything. 

Since the VPN Connection on remote site is the on the Router, I understand that it goes through WAN_LOCAL policy, right?

There is definitly no "in" data on the remote site:

ubnt@ubnt:~$ show vpn ipsec sa                                                                                                                                                                                                                           
peer-90.127.196.91-tunnel-1: #1, ESTABLISHED, IKEv1, a7622225f5725656:2391b3e2f1d47ba0                                                                                                                                                                   
  local  '24.212.241.155' @ 24.212.241.155                                                                                                                                                                                                               
  remote '192.168.1.30' @ 90.127.196.91                                                                                                                                                                                                                  
  AES_CBC-256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024                                                                                                                                                                                                       
  established 794s ago, reauth in 27275s                                                                                                                                                                                                                 
  peer-90.127.196.91-tunnel-1: #1, INSTALLED, TUNNEL-in-UDP, ESP:AES_CBC-256/HMAC_SHA1_96/MODP_1024                                                                                                                                                      
    installed 794 ago, rekeying in 1842s, expires in 2807s                                                                                                                                                                                               
    in  c22ea23e,      0 bytes,     0 packets                                                                                                                                                                                                            
    out 78fa1385,  26544 bytes,   316 packets,     4s ago                                                                                                                                                                                                
    local  192.168.101.0/24                                                                                                                                                                                                                              
    remote 192.168.1.0/24