Reply
Member
Posts: 106
Registered: ‎12-20-2011
Kudos: 17

Edgepoint Conntrack full but no reason for conntrack to be enabled

Good morning, I have had a edgepoint strop passing traffic multiple times since last Thursday and finally got a log dump from the last instance so that I can post for some help. I am going to adjust my connection tables to something that worked for an ER-8-Pro but I think it is a bandaid as the ER-8-Pro was doing NAT and needed connection tracking on and this edgepoint is not, it is just in normal route mode. 

 

I was reading that with UCRM and netflows connection tracking gets enabled, is UNMS the same? This issue was happening before I enabled UNMS, I only enabled it this last round so I could see if there was a huge CPU spike, and  I noticed that even though I cannot SSH or Web to the EP when the table is full, UNMS can still display stats and I can issue reboot command.

Thanks in advance to any help!

 

Here is the config

admin@CampbellEdgePoint:~$ show configuration
interfaces {
ethernet eth0 {
address 172.16.16.220/23
duplex auto
speed auto
vif 1609 {
address 172.16.9.1/24
description AF5ToMendes
mtu 1500
}
vif 1621 {
address 172.16.21.1/24
description "Campbell 21"
}
vif 1622 {
address 172.16.22.1/24
description "Campbell 22"
}
vif 1623 {
address 172.16.23.1/24
description "Campbell 23"
}
vif 1624 {
address 172.16.24.1/24
description "Campbell 24"
}
vif 1625 {
address 172.16.25.1/24
description "Campbell 25"
}
vif 1650 {
address 172.16.50.1/24
description "Campbell 50"
}
vif 1651 {
address 172.16.51.1/24
description "Campbell 51"
}
vif 1652 {
address 172.16.52.1/24
description "Campbell 52"
}
vif 1653 {
address 172.16.53.1/24
description "Campbell 53"
}
vif 1654 {
address 172.16.54.1/24
description "Campbell 54"
}
vif 1655 {
address 172.16.55.1/24
description "Campbell 55"
}
vif 1656 {
address 172.16.56.1/24
description "Campbell 56"
}
vif 1657 {
address 172.16.57.1/24
description "Campbell 57"
}
vif 1658 {
address 172.16.58.1/24
description "Campbell 58"
}
vif 1659 {
address 172.16.59.1/24
description "Campbell 59"
}
}
ethernet eth1 {
address 172.16.1.4/24
description Name1
duplex auto
poe {
output off
}
speed auto
}
ethernet eth2 {
address 172.16.2.4/24
description Name2
duplex auto
poe {
output off
}
speed auto
}
ethernet eth3 {
address 172.16.3.4/24
description Name3
duplex auto
poe {
output off
}
speed auto
}
ethernet eth4 {
address 172.16.7.1/24
description Name4Backhaul-Netonix2Port15
duplex auto
poe {
output off
}
speed auto
}
ethernet eth5 {
address 172.16.6.1/24
description name5Backhaul-Netonix2Port14
duplex auto
poe {
output off
}
speed auto
}
ethernet eth6 {
address 172.16.8.1/24
description name6Backhaul-Netonix2Port13
duplex auto
poe {
output off
}
speed auto
}
ethernet eth7 {
address 172.16.4.1/24
description name7Backhaul-Netonix1Port26
duplex auto
poe {
output off
}
speed auto
}
loopback lo {
}
}
protocols {
static {
route 0.0.0.0/0 {
next-hop 172.16.16.1 {
}
}
route 172.17.0.0/16 {
next-hop 172.16.7.4 {
description 1
}
}
route 172.18.0.0/16 {
next-hop 172.16.4.4 {
description 2
}
}
route 172.19.0.0/16 {
next-hop 172.16.6.4 {
description 3
}
}
route 172.20.0.0/16 {
next-hop 172.16.4.4 {
description 4
}
}
route 172.21.0.0/16 {
next-hop 172.16.8.4 {
description 5
}
}
route 172.22.0.0/16 {
next-hop 172.16.4.4 {
description 6
}
}
route 172.23.0.0/16 {
next-hop 172.16.4.4 {
description 7
}
}
route 172.24.0.0/16 {
next-hop 172.16.9.4 {
description 8
}
}
route 172.25.0.0/16 {
next-hop 172.16.4.4 {
description 9
}
}
route 172.30.0.0/16 {
next-hop 172.16.4.4 {
description CST
}
}
}
}
service {
dhcp-server {
disabled false
hostfile-update disable
shared-network-name 1 {
authoritative disable
subnet 172.16.21.0/24 {
default-router 172.16.21.1
dns-server 172.16.16.2
dns-server 172.16.16.3
lease 86400
start 172.16.21.100 {
stop 172.16.21.200
}
}
}
shared-network-name 2 {
authoritative disable
subnet 172.16.22.0/24 {
default-router 172.16.22.1
dns-server 172.16.16.2
dns-server 172.16.16.3
lease 86400
start 172.16.22.100 {
stop 172.16.22.200
}
}
}
shared-network-name 3 {
authoritative disable
subnet 172.16.23.0/24 {
default-router 172.16.23.1
dns-server 172.16.16.2
dns-server 172.16.16.3
lease 86400
start 172.16.23.100 {
stop 172.16.23.200
}
}
}
shared-network-name 4 {
authoritative disable
subnet 172.16.24.0/24 {
default-router 172.16.24.1
dns-server 172.16.16.2
dns-server 172.16.16.3
lease 86400
start 172.16.24.100 {
stop 172.16.24.200
}
}
}
shared-network-name 5 {
authoritative disable
subnet 172.16.25.0/24 {
default-router 172.16.25.1
dns-server 172.16.16.2
dns-server 172.16.16.3
lease 86400
start 172.16.25.100 {
stop 172.16.25.200
}
}
}
shared-network-name 6 {
authoritative disable
subnet 172.16.50.0/24 {
default-router 172.16.50.1
dns-server 172.16.16.2
dns-server 172.16.16.3
lease 86400
start 172.16.50.100 {
stop 172.16.50.200
}
static-mapping 7 {
ip-address 172.16.50.103
mac-address dc:9f:db:82:45:f8
}
}
}
shared-network-name 8 {
authoritative disable
subnet 172.16.51.0/24 {
default-router 172.16.51.1
dns-server 172.16.16.2
dns-server 172.16.16.3
lease 86400
start 172.16.51.100 {
stop 172.16.51.200
}
}
}
shared-network-name 9 {
authoritative disable
subnet 172.16.52.0/24 {
default-router 172.16.52.1
dns-server 172.16.16.2
dns-server 172.16.16.3
lease 86400
start 172.16.52.100 {
stop 172.16.52.200
}
}
}
shared-network-name 10 {
authoritative disable
subnet 172.16.53.0/24 {
default-router 172.16.53.1
dns-server 172.16.16.2
dns-server 172.16.16.3
lease 86400
start 172.16.53.100 {
stop 172.16.53.200
}
static-mapping 11 {
ip-address 172.16.53.136
mac-address f0:9f:c2:86:ed:14
}
}
}
shared-network-name 12 {
authoritative disable
subnet 172.16.54.0/24 {
default-router 172.16.54.1
dns-server 172.16.16.2
dns-server 172.16.16.3
lease 86400
start 172.16.54.100 {
stop 172.16.54.200
}
static-mapping qwe {
ip-address 172.16.54.107
mac-address 44:d9:e7:72:fa:fa
}
}
}
shared-network-name 13 {
authoritative disable
subnet 172.16.55.0/24 {
default-router 172.16.55.1
dns-server 172.16.16.2
dns-server 172.16.16.3
lease 86400
start 172.16.55.100 {
stop 172.16.55.200
}
}
}
shared-network-name 14 {
authoritative disable
subnet 172.16.56.0/24 {
default-router 172.16.56.1
dns-server 172.16.16.2
dns-server 172.16.16.3
lease 86400
start 172.16.56.100 {
stop 172.16.56.200
}
static-mapping 15 {
ip-address 172.16.56.134
mac-address f0:9f:c2:e4:08:ec
}
}
}
shared-network-name 16 {
authoritative disable
subnet 172.16.57.0/24 {
default-router 172.16.57.1
dns-server 172.16.16.2
dns-server 172.16.16.3
lease 86400
start 172.16.57.100 {
stop 172.16.57.200
}
}
}
shared-network-name 17 {
authoritative disable
subnet 172.16.58.0/24 {
default-router 172.16.58.1
dns-server 172.16.16.2
dns-server 172.16.16.3
lease 86400
start 172.16.58.100 {
stop 172.16.58.200
}
}
}
shared-network-name 18 {
authoritative disable
subnet 172.16.59.0/24 {
default-router 172.16.59.1
dns-server 172.16.16.2
dns-server 172.16.16.3
lease 86400
start 172.16.59.100 {
stop 172.16.59.200
}
}
}
use-dnsmasq disable
}
gui {
http-port 80
https-port 443
older-ciphers enable
}
ssh {
port 22
protocol-version v2
}
unms {
connection wss://unms.*****.com:443+fm10H5u2lEgiVkfGR8JGJ2ixqJUz1qxojf_b_BewWIsAAAAA+allowSelfSignedCertificate
}
}
system {
host-name EdgePoint
login {
user admin {
authentication {
encrypted-password ****************
plaintext-password ****************
}
full-name admin
level admin
}
}
name-server 172.16.16.3
ntp {
server 0.ubnt.pool.ntp.org {
}
server 1.ubnt.pool.ntp.org {
}
server 2.ubnt.pool.ntp.org {
}
server 3.ubnt.pool.ntp.org {
}
}
offload {
hwnat disable
ipv4 {
forwarding enable
vlan enable
}
}
syslog {
global {
facility all {
level notice
}
facility protocols {
level debug
}
}
}
time-zone America/Los_Angeles

 I have attached the log due to char limit.

Veteran Member
Posts: 7,614
Registered: ‎03-24-2016
Kudos: 1981
Solutions: 872

Re: Edgepoint Conntrack full but no reason for conntrack to be enabled

afaik, you can't disable conntrack

Try lowering tcp timeout, so aborted session don't eat up entire conn table

 

set system conntrack timeout tcp established 900

 

Member
Posts: 106
Registered: ‎12-20-2011
Kudos: 17

Re: Edgepoint Conntrack full but no reason for conntrack to be enabled

[ Edited ]

So I set it anyways but running the following commands produces errors until about the time the connection table fills up and the router becomes unreachable. That is why I assumed conntrack is not enabled at first.
Edit:

admin@CampbellEdgePoint:~$ cat /proc/sys/net/netfilter/nf_conntrack_max
262144
admin@CampbellEdgePoint:~$ cat /proc/sys/net/netfilter/nf_conntrack_count
0
Is what I get 99% of the day untill it starts counting then craps out

Member
Posts: 106
Registered: ‎12-20-2011
Kudos: 17

Re: Edgepoint Conntrack full but no reason for conntrack to be enabled

Still have 0 connections being tracked since last reboot,
admin@CampbellEdgePoint:~$ cat /proc/sys/net/netfilter/nf_conntrack_count
0

Would really appreciate someone from the ER team to chime in with why the EP would all of a sudden start counting connections and filling the table up. Is there a secondary table or command I can run that would show me more information?
Veteran Member
Posts: 7,614
Registered: ‎03-24-2016
Kudos: 1981
Solutions: 872

Re: Edgepoint Conntrack full but no reason for conntrack to be enabled

Connection count=0 is more suspicious to me.

 

Even without NAT , it should start counting all flows

Member
Posts: 106
Registered: ‎12-20-2011
Kudos: 17

Re: Edgepoint Conntrack full but no reason for conntrack to be enabled

I have 22 ER-8-Pros without Nat enabled for the school district I work for and they all show 0 connections with the same command. The only one that will print out a number greater than 0 is my infinity router with Nat enabled.

It is really strange to me that this edgepoint will be 0 for 2 or 3 days then all of a sudden have more than 260000 connections and not release them in a timely manner. I have added the 900 timeout and will see if the unit will stay online for the for see-able future but would really like to know what is causing the issue.
Member
Posts: 106
Registered: ‎12-20-2011
Kudos: 17

Re: Edgepoint Conntrack full but no reason for conntrack to be enabled

Had another router presumably fill contrack table last night....
Comparing the original config this a router that is directly up stream with similar config I am getting what I would expect with no NAT enabled.  @16again have you seen anything like this before?

 

admin@EdgePoint:~$ cat /proc/sys/net/netfilter/nf_conntrack_max
cat: can't open '/proc/sys/net/netfilter/nf_conntrack_max': No such file or directory
admin@EdgePoint:~$ cat /proc/sys/net/netfilter/nf_conntrack_count
cat: can't open '/proc/sys/net/netfilter/nf_conntrack_count': No such file or directory
 interfaces {
    ethernet eth0 {
        address 172.16.16.5/23
        description To_Internet
        duplex auto
        speed auto
    }
    ethernet eth1 {
        address 172.16.1.1/24
        description AF24BHC
        duplex auto
        poe {
            output off
        }
        speed auto
    }
    ethernet eth2 {
        address 172.16.2.1/24
        description CF11BHC
        duplex auto
        poe {
            output off
        }
        speed auto
    }
    ethernet eth3 {
        address 172.16.3.1/24
        description AF5BHC
        duplex auto
        poe {
            output off
        }
        speed auto
    }
    ethernet eth4 {
        duplex auto
        speed auto
    }
    ethernet eth5 {
        duplex auto
        speed auto
    }
    ethernet eth6 {
        duplex auto
        poe {
            output off
        }
        speed auto
    }
    ethernet eth7 {
        description ToEth0_OnER_Infinity
        duplex auto
        poe {
            output off
        }
        speed auto
    }
    loopback lo {
    }
}
protocols {
    static {
        bfd {
            interface eth3 {
            }
        }
        route 0.0.0.0/0 {
            next-hop 172.16.16.1 {
            }
        }
        route 172.16.4.0/24 {
            next-hop 172.16.1.4 {
                description LBackhaul
            }
        }
        route 172.16.6.0/24 {
            next-hop 172.16.1.4 {
                description S190Backhaul
            }
        }
        route 172.16.7.0/24 {
            next-hop 172.16.1.4 {
                description LBackhaul
            }
        }
        route 172.16.8.0/24 {
            next-hop 172.16.1.4 {
                description UpperBackhaul
            }
        }
        route 172.16.9.0/24 {
            next-hop 172.16.1.4 {
                description MBackhaul
            }
        }
        route 172.17.0.0/16 {
            next-hop 172.16.1.4 {
                description Leau
                disable
            }
            next-hop 172.16.16.220 {
                description LuBackup
            }
        }
        route 172.18.0.0/16 {
            next-hop 172.16.1.4 {
                description 7
                disable
            }
            next-hop 172.16.16.220 {
                description W8Backup
            }
        }
        route 172.19.0.0/16 {
            next-hop 172.16.1.4 {
                description S190
                disable
            }
            next-hop 172.16.16.220 {
                description StBackup
            }
        }
        route 172.20.0.0/16 {
            next-hop 172.16.1.4 {
                description Lower
                disable
            }
            next-hop 172.16.16.220 {
                description LowerBackup
            }
        }
        route 172.21.0.0/16 {
            next-hop 172.16.1.4 {
                description Upper
                disable
            }
            next-hop 172.16.16.220 {
                description UpperBackup
:
                description S190
                disable
            }
            next-hop 172.16.16.220 {
                description S190Backup
            }
        }
        route 172.20.0.0/16 {
            next-hop 172.16.1.4 {
                description LGlobe
                disable
            }
            next-hop 172.16.16.220 {
                description LeBackup
            }
        }
        route 172.21.0.0/16 {
            next-hop 172.16.1.4 {
                description UGlobe
                disable
            }
            next-hop 172.16.16.220 {
                description UppeeBackup
            }
        }
        route 172.22.0.0/16 {
            next-hop 172.16.1.4 {
                description Tebles
                disable
            }
            next-hop 172.16.16.220 {
                description TeBackup
            }
        }
        route 172.23.0.0/16 {
            next-hop 172.16.1.4 {
                description Splle
                disable
            }
            next-hop 172.16.16.220 {
                description SpleBackup
            }
        }
        route 172.24.0.0/16 {
            next-hop 172.16.1.4 {
                description Mes
                disable
            }
            next-hop 172.16.16.220 {
                description MBackup
            }
        }
        route 172.25.0.0/16 {
            next-hop 172.16.1.4 {
                description Lana
                disable
            }
            next-hop 172.16.16.220 {
                description Laackup
            }
        }
        route 172.25.10.0/24 {
            next-hop 172.16.1.10 {
                description Laouse
            }
        }
        route 172.30.0.0/16 {
            next-hop 172.16.1.4 {
                description CST
                disable
            }
            next-hop 172.16.16.220 {
                description CSTBackup
            }
        }
    }
}
service {
    gui {
        http-port 80
        https-port 443
        older-ciphers enable
    }
    ssh {
        port 22
        protocol-version v2
    }
    unms {
        connection wss://unms.***.com:443+tbAanpiV2oZZEzAqdz7W54eKDwoEj1mMJmmb23tIpkAAAAAA+allowSelfSignedCertificate
    }
}
system {
    domain-name ***.com
    host-name SEdgePoint
    login {
        user admin {
            authentication {
                encrypted-password ****************
                plaintext-password ****************
            }
            full-name Administrator
            level admin
        }
    }
    name-server 172.16.16.3
    ntp {
        server 0.ubnt.pool.ntp.org {
        }
        server 1.ubnt.pool.ntp.org {
        }
        server 2.ubnt.pool.ntp.org {
        }
        server 3.ubnt.pool.ntp.org {
        }
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone America/Los_Angeles

 

Veteran Member
Posts: 7,614
Registered: ‎03-24-2016
Kudos: 1981
Solutions: 872

Re: Edgepoint Conntrack full but no reason for conntrack to be enabled

fwiw,

 

On test setup over here on ER-X , I have disabled NAT , and conntrack table fills up normally

ubnt@ubnt:~$ cat /proc/sys/net/netfilter/nf_conntrack_max
262144
ubnt@ubnt:~$ cat /proc/sys/net/netfilter/nf_conntrack_count
85

 

ETH0 is "outside" network having internet access,

ETH1  is inside test-only network , single test PC attached

 

Guess mode:

Since yours stays at zero:  

Maybe your connections are all offloaded, and are no longer seen in conntrack count.

If offload engine capacity reaches its limit,  new connections are routed by CPU and visible in conntrack count.

Since you've lots of interfaces, I guess total number of connections might be huge.

You can test this theory, by testing with not off-loadable protocol.

Ping through firewall to a destination that doesn't reply, so reply doesn't remove conntrack entry.

Then check if conntrack count stays at zero

Member
Posts: 106
Registered: ‎12-20-2011
Kudos: 17

Re: Edgepoint Conntrack full but no reason for conntrack to be enabled

Hrmm, interesting idea on the offloading part.
EP with conntrack table file not found
IP offload module : loaded
IPv4
forwarding: enabled
vlan : disabled
pppoe : disabled
gre : disabled
IPv6
forwarding: disabled
vlan : disabled
pppoe : disabled

IPSec offload module: loaded

Traffic Analysis :
export : disabled
dpi : disabled
version : 1.302

EP with conntrack file found and 0 count

IP offload module : loaded
IPv4
forwarding: enabled
vlan : enabled
pppoe : disabled
gre : disabled
IPv6
forwarding: disabled
vlan : disabled
pppoe : disabled

IPSec offload module: loaded

Traffic Analysis :
export : disabled
dpi : disabled
version : 1.302


VLAN is the only difference. That and the tcp timeout timer set on the last one.
Veteran Member
Posts: 7,614
Registered: ‎03-24-2016
Kudos: 1981
Solutions: 872

Re: Edgepoint Conntrack full but no reason for conntrack to be enabled

For the ER where conntrack counters are non-existent, check if same kernel  modules are loaded using

sudo lsmod

Member
Posts: 106
Registered: ‎12-20-2011
Kudos: 17

Re: Edgepoint Conntrack full but no reason for conntrack to be enabled

[ Edited ]

One that Crashed originally with tcp timeout enabled

admin@CEdgePoint:~$ sudo lsmod
Module                  Size  Used by
ip6table_mangle         2060  1
ip6table_filter         1580  1
ip6table_raw            1504  1
ip6_tables             24240  3 ip6table_filter,ip6table_mangle,ip6table_raw
iptable_nat             3382  1
nf_conntrack_ipv4       9102  1
nf_defrag_ipv4          1419  1 nf_conntrack_ipv4
nf_nat_ipv4             5189  1 iptable_nat
iptable_mangle          1960  1
xt_CT                   4610  4
iptable_raw             1564  1
nf_nat_pptp             2242  0
nf_conntrack_pptp       4880  1 nf_nat_pptp
nf_conntrack_proto_gre     5575  1 nf_conntrack_pptp
nf_nat_h323             8095  0
nf_conntrack_h323      47417  1 nf_nat_h323
nf_nat_sip             10703  0
nf_conntrack_sip       34552  1 nf_nat_sip
nf_nat_proto_gre        2125  1 nf_nat_pptp
nf_nat_tftp             1078  0
nf_nat_ftp              1988  0
nf_nat                 17295  8 nf_nat_ftp,nf_nat_sip,nf_nat_proto_gre,nf_nat
nf_conntrack_tftp       4145  1 nf_nat_tftp
nf_conntrack_ftp        8679  1 nf_nat_ftp
nf_conntrack           74393  16 nf_nat_ftp,nf_nat_sip,xt_CT,nf_conntrack_proack_ipv4,nf_conntrack_pptp,nf_conntrack_tftp
8021q                  22855  0
garp                    7006  1 8021q
stp                     2037  1 garp
llc                     4673  2 stp,garp
ip_set_hash_net        32186  50
ip_set                 27169  1 ip_set_hash_net
nfnetlink               4565  1 ip_set
iptable_filter          1640  1
ip_tables              23962  4 iptable_filter,iptable_mangle,iptable_nat,ipt
x_tables               24251  9 ip6table_filter,xt_CT,ip6table_mangle,ip_tabl
cvm_ipsec_kame         56466  0
ipv6                  428069  80 ip6table_mangle,cvm_ipsec_kame
imq                     5825  0
cavium_ip_offload     152266  0
ubnt_nf_app            12733  1 cavium_ip_offload
tdts                  614810  2 cavium_ip_offload,ubnt_nf_app
octeon_rng              2074  0
rng_core                4400  2 octeon_rng
octeon_ethernet        66199  1 cavium_ip_offload
mdio_octeon             5259  1 octeon_ethernet
of_mdio                 3734  2 octeon_ethernet,mdio_octeon
ethernet_mem            5808  1 octeon_ethernet
octeon_common           3129  1 octeon_ethernet
ubnt_platform         111861  0
libphy                 26833  4 ubnt_platform,octeon_ethernet,mdio_octeon,of_
admin@CEdgePoint:~$ cat /proc/sys/net/netfilter/nf_conntrack_count
0

 

One that hasn't yet locked up

admin@SEdgePoint:~$ sudo lsmod
Module                  Size  Used by
ip_set_hash_net        32186  10
ip_set                 27169  1 ip_set_hash_net
nfnetlink               4565  1 ip_set
iptable_filter          1640  0
ip_tables              23962  1 iptable_filter
x_tables               24251  2 ip_tables,iptable_filter
cvm_ipsec_kame         56466  0
ipv6                  428069  51 cvm_ipsec_kame
imq                     5825  0
cavium_ip_offload     152266  0
ubnt_nf_app            12733  1 cavium_ip_offload
tdts                  614810  2 cavium_ip_offload,ubnt_nf_app
octeon_rng              2074  0
rng_core                4400  2 octeon_rng
octeon_ethernet        66199  1 cavium_ip_offload
mdio_octeon             5259  1 octeon_ethernet
of_mdio                 3734  2 octeon_ethernet,mdio_octeon
ethernet_mem            5808  1 octeon_ethernet
octeon_common           3129  1 octeon_ethernet
ubnt_platform         111861  0
libphy                 26833  4 ubnt_platform,octeon_ethernet,mdio_octeon,of_mdio
admin@SEdgePoint:~$ cat /proc/sys/net/netfilter/nf_conntrack_count
cat: can't open '/proc/sys/net/netfilter/nf_conntrack_count': No such file or directory
admin@SEdgePoint:~$

 

One that did crash last night but as of right now shows

admin@MEdgePoint8:~$ sudo lsmod
Module                  Size  Used by
ip_set_hash_net        32186  14
ip_set                 27169  1 ip_set_hash_net
nfnetlink               4565  1 ip_set
iptable_filter          1640  0
ip_tables              23962  1 iptable_filter
x_tables               24251  2 ip_tables,iptable_filter
cvm_ipsec_kame         56466  0
ipv6                  428069  43 cvm_ipsec_kame
imq                     5825  0
cavium_ip_offload     152266  0
ubnt_nf_app            12733  1 cavium_ip_offload
tdts                  614810  2 cavium_ip_offload,ubnt_nf_app
octeon_rng              2074  0
rng_core                4400  2 octeon_rng
octeon_ethernet        66199  1 cavium_ip_offload
mdio_octeon             5259  1 octeon_ethernet
of_mdio                 3734  2 octeon_ethernet,mdio_octeon
ethernet_mem            5808  1 octeon_ethernet
octeon_common           3129  1 octeon_ethernet
ubnt_platform         111861  0
libphy                 26833  4 ubnt_platform,octeon_ethernet,mdio_octeon,of_mdio
admin@MEdgePoint8:~$ cat /proc/sys/net/netfilter/nf_conntrack_count
cat: can't open '/proc/sys/net/netfilter/nf_conntrack_count': No such file or directory
admin@MEdgePoint8:~$
Member
Posts: 106
Registered: ‎12-20-2011
Kudos: 17

Re: Edgepoint Conntrack full but no reason for conntrack to be enabled

Router that has restarted but has new tcp timeout setting

Module                  Size  Used by
ip6table_mangle         2060  1
ip6table_filter         1580  1
ip6table_raw            1504  1
ip6_tables             24240  3 ip6table_filter,ip6table_mangle,ip6table_raw
iptable_nat             3382  1
nf_conntrack_ipv4       9102  1
nf_defrag_ipv4          1419  1 nf_conntrack_ipv4
nf_nat_ipv4             5189  1 iptable_nat
iptable_mangle          1960  1
xt_CT                   4610  4
iptable_raw             1564  1
nf_nat_pptp             2242  0
nf_conntrack_pptp       4880  1 nf_nat_pptp
nf_conntrack_proto_gre     5575  1 nf_conntrack_pptp
nf_nat_h323             8095  0
nf_conntrack_h323      47417  1 nf_nat_h323
nf_nat_sip             10703  0
nf_conntrack_sip       34552  1 nf_nat_sip
nf_nat_proto_gre        2125  1 nf_nat_pptp
nf_nat_tftp             1078  0
nf_nat_ftp              1988  0
nf_nat                 17295  8 nf_nat_ftp,nf_nat_sip,nf_nat_proto_gre,nf_nat
nf_conntrack_tftp       4145  1 nf_nat_tftp
nf_conntrack_ftp        8679  1 nf_nat_ftp
nf_conntrack           74393  16 nf_nat_ftp,nf_nat_sip,xt_CT,nf_conntrack_proack_ipv4,nf_conntrack_pptp,nf_conntrack_tftp
8021q                  22855  0
garp                    7006  1 8021q
stp                     2037  1 garp
llc                     4673  2 stp,garp
ip_set_hash_net        32186  50
ip_set                 27169  1 ip_set_hash_net
nfnetlink               4565  1 ip_set
iptable_filter          1640  1
ip_tables              23962  4 iptable_filter,iptable_mangle,iptable_nat,ipt
x_tables               24251  9 ip6table_filter,xt_CT,ip6table_mangle,ip_tabl
cvm_ipsec_kame         56466  0
ipv6                  428069  80 ip6table_mangle,cvm_ipsec_kame
imq                     5825  0
cavium_ip_offload     152266  0
ubnt_nf_app            12733  1 cavium_ip_offload
tdts                  614810  2 cavium_ip_offload,ubnt_nf_app
octeon_rng              2074  0
rng_core                4400  2 octeon_rng
octeon_ethernet        66199  1 cavium_ip_offload
mdio_octeon             5259  1 octeon_ethernet
of_mdio                 3734  2 octeon_ethernet,mdio_octeon
ethernet_mem            5808  1 octeon_ethernet
octeon_common           3129  1 octeon_ethernet
ubnt_platform         111861  0
libphy                 26833  4 ubnt_platform,octeon_ethernet,mdio_octeon,of_
admin@CEdgePoint:~$ cat /proc/sys/net/netfilter/nf_conntrack_count
0

One that  has not yet restarted but is simialr in config.

Module                  Size  Used by
ip_set_hash_net        32186  10
ip_set                 27169  1 ip_set_hash_net
nfnetlink               4565  1 ip_set
iptable_filter          1640  0
ip_tables              23962  1 iptable_filter
x_tables               24251  2 ip_tables,iptable_filter
cvm_ipsec_kame         56466  0
ipv6                  428069  51 cvm_ipsec_kame
imq                     5825  0
cavium_ip_offload     152266  0
ubnt_nf_app            12733  1 cavium_ip_offload
tdts                  614810  2 cavium_ip_offload,ubnt_nf_app
octeon_rng              2074  0
rng_core                4400  2 octeon_rng
octeon_ethernet        66199  1 cavium_ip_offload
mdio_octeon             5259  1 octeon_ethernet
of_mdio                 3734  2 octeon_ethernet,mdio_octeon
ethernet_mem            5808  1 octeon_ethernet
octeon_common           3129  1 octeon_ethernet
ubnt_platform         111861  0
libphy                 26833  4 ubnt_platform,octeon_ethernet,mdio_octeon,of_mdio
admin@SEdgePoint:~$ cat /proc/sys/net/netfilter/nf_conntrack_count
cat: can't open '/proc/sys/net/netfilter/nf_conntrack_count': No such file or directory

New one that has become unresponsive in the last 24 hours. and has been restarted.

Module                  Size  Used by
ip_set_hash_net        32186  14
ip_set                 27169  1 ip_set_hash_net
nfnetlink               4565  1 ip_set
iptable_filter          1640  0
ip_tables              23962  1 iptable_filter
x_tables               24251  2 ip_tables,iptable_filter
cvm_ipsec_kame         56466  0
ipv6                  428069  43 cvm_ipsec_kame
imq                     5825  0
cavium_ip_offload     152266  0
ubnt_nf_app            12733  1 cavium_ip_offload
tdts                  614810  2 cavium_ip_offload,ubnt_nf_app
octeon_rng              2074  0
rng_core                4400  2 octeon_rng
octeon_ethernet        66199  1 cavium_ip_offload
mdio_octeon             5259  1 octeon_ethernet
of_mdio                 3734  2 octeon_ethernet,mdio_octeon
ethernet_mem            5808  1 octeon_ethernet
octeon_common           3129  1 octeon_ethernet
ubnt_platform         111861  0
libphy                 26833  4 ubnt_platform,octeon_ethernet,mdio_octeon,of_mdio
admin@MEdgePoint8:~$ cat /proc/sys/net/netfilter/nf_conntrack_count
cat: can't open '/proc/sys/net/netfilter/nf_conntrack_count': No such file or directory

 

Veteran Member
Posts: 7,614
Registered: ‎03-24-2016
Kudos: 1981
Solutions: 872

Re: Edgepoint Conntrack full but no reason for conntrack to be enabled

lsmod output show all nf conntrack stuff not loaded , this explains why you don't have conntrack countrackers.

 

But throws a new question....why aren't the modules loaded to begin with?

 

With my config  posted earlier, even after a reboot,  those modules get loaded.   (I tried reboot, as previously the masquerade rule wasn't disabled , and thus modules were loaded )

Member
Posts: 106
Registered: ‎12-20-2011
Kudos: 17

Re: Edgepoint Conntrack full but no reason for conntrack to be enabled

Yeah, I am a little confused/concerned with the fact that it is being different.

The one thing I noticed is that the is the ones with conntrack loaded has vlan interfaces, maybe that triggers the conntrack?

Good news is the unit that was giving me issues has been solid since your tcp timeout suggestion so I am hoping that it will stay happy. It let my boss get a good night sleep the last few nights so kudos!

Wish someone from UBNT would chime with a little bit of the technical details but probably all off killing the zombies from the 1.10 patch notes!
Veteran Member
Posts: 7,614
Registered: ‎03-24-2016
Kudos: 1981
Solutions: 872

Re: Edgepoint Conntrack full but no reason for conntrack to be enabled

@UBNT-Fenng @UBNT-afomins 

What happens with these modules being (not) loaded ?

New Member
Posts: 28
Registered: ‎10-19-2009
Kudos: 6

Re: Edgepoint Conntrack full but no reason for conntrack to be enabled

Is there any update from Ubiquiti on this?  Should conntrack be on when an EdgeRouter is only routing and not doing anything related to NAT or Firewall rules?  

Regular Member
Posts: 508
Registered: ‎07-21-2010
Kudos: 91
Solutions: 6

Re: Edgepoint Conntrack full but no reason for conntrack to be enabled

No it should not!

 

But be aware that issuing iptables --list -vn -t nat will load nf_conntrack!

 

If you do not have a NOTRACK rule in iptables, your router will skrew up when table gets full!

 

I also run into that type of issue. My Routers locked up and i was not able to connect to that router via WebIF, SSH via IPv4! But you are able to ssh into that device via IPv6!

 

Sometimes my employees issue that iptables-command and after some hours the machine is "dead" and our customers start to open tickets "Internet is not working..." - Congratulations!

 

ssh fe80:xxxxxxxxxxx%eth1 from a direct neighbour gives you access to your router.

If you do not know the link-local address of your router, you can issue a arping to get the mac-address and build the link-local-address via that handy tool: http://www.sput.nl/internet/ipv6/ll-mac.html

rmmod nf_conntrack_ipv4 nf_conntrack_netlink nf_conntrack

 

and your routing is restored and you do not need to reboot.

 

PS: If nf_conntrack_ipv4 is loaded and you have 0 flows in your conntrack-table, this is because in RAW-Table there is a -j NOTRACK rule that will prevent a new flow-entry. I think that is new to v.1.10.0 that only natted packets get tracked. If you dont do a full state firewalling (related, established -> accept) then you do not need to know all current active flows!

Reply