Reply
New Member
bweisz
Posts: 21
Registered: ‎10-22-2013
Posts: 21
Kudos: 3
Registered: 10-22-2013

EdgeRouter POE - snmpd memory leak?

Hi all

Is anyone else seeing a memory leak in snmpd on 1.3.0?  I'm having to restart it every couple days.  Nothing special about the snmpd configuration...

bweisz@sw-kila-core0:~$ show version
Version:      v1.3.0
Build ID:     4605130
Build on:     10/11/13 17:54
Copyright:    2012-2013 Ubiquiti Networks, Inc.
HW model:     EdgeRouter PoE 5-Port
HW S/N:       DC9FDB80D225
Uptime:       16:43:17 up 5 days,  1:24,  1 user,  load average: 0.02, 0.03, 0.00


bweisz@sw-kila-core0# show service snmp
 community public {
     authorization ro
 }
 contact brandon@avioc.org
 location "Kila, MT"

top - 16:45:01 up 5 days,  1:26,  1 user,  load average: 0.00, 0.02, 0.00
Tasks:  61 total,   1 running,  60 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.3%us,  1.5%sy,  0.0%ni, 96.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    483856k total,   353596k used,   130260k free,    56628k buffers
Swap:        0k total,        0k used,        0k free,    90712k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
16128 snmp      20   0  138m 130m 2296 S    0 27.7  27:45.56 snmpd

Eventually it gets angry and starts spamming the log with failure to fork entries:

Oct 21 19:26:14 sw-kila-core0 sw-kila-core0 snmpd[1015]: fork: Cannot allocate memory
Oct 21 19:27:00 sw-kila-core0 sw-kila-core0 snmpd[1015]: last message repeated 2 times
Oct 21 19:28:10 sw-kila-core0 sw-kila-core0 snmpd[1015]: last message repeated 18 times
Oct 21 19:29:10 sw-kila-core0 sw-kila-core0 snmpd[1015]: last message repeated 10 times
....

All suggestions appreciated.

 

Regular Member
iamkinghenry
Posts: 384
Registered: ‎09-23-2013
Posts: 384
Kudos: 160
Solutions: 7
Registered: 09-23-2013

Re: EdgeRouter POE - snmpd memory leak?

Not so far but I won't force a restart and see how long the router stays up.  Only been up 2 days right now.

I did notice SNMP not connecting for 2 VLANs on PTRG that I had to delete and re-add to get them working.  Not sure what the issue was so hopefully it doesn't happen again.

Ubiquiti Employee
UBNT-stig
Posts: 4,541
Registered: ‎06-09-2011
Posts: 4541
Kudos: 1398
Solutions: 405
Contributions: 14
Registered: 06-09-2011

Re: EdgeRouter POE - snmpd memory leak?

Hmm, we did see a memory leak in the earlier version of snmpd (v1.3.0 has a newer net-snmpd), but so far I haven't seen a memory leak with v1.3.0.  Our corporate gateway has an uptime of 12 days and it's still holding around 12.6MB of memory (about what it started at).  Maybe you could post your config to see  what could be affecting snmp.

New Member
bweisz
Posts: 21
Registered: ‎10-22-2013
Posts: 21
Kudos: 3
Registered: 10-22-2013

Re: EdgeRouter POE - snmpd memory leak?

Thanks - config below...  Nothing fancy other than a bridged vlan to eth1..

 

bweisz@sw-kila-core0# show
 interfaces {
     bridge br0 {
         aging 300
         description "br0 - eth1,switch0.10"
         hello-time 2
         max-age 20
         priority 0
         stp false
     }
     ethernet eth0 {
         address 192.168.1.1/24
         duplex auto
         poe {
             output off
         }
         speed auto
     }
     ethernet eth1 {
         bridge-group {
             bridge br0
         }
         description "eth1 - NanoBridge M5"
         duplex auto
         poe {
             output 24v
         }
         speed auto
     }
     ethernet eth2 {
         description "eth2 - UniFi AP AC"
         duplex auto
         poe {
             output off
         }
         speed auto
     }
     ethernet eth3 {
         duplex auto
         poe {
             output off
         }
         speed auto
     }
     ethernet eth4 {
         description "eth4 - sw-kila-cs0"
         duplex auto
         poe {
             output off
         }
         speed auto
     }
     loopback lo {
     }
     switch switch0 {
         address 172.24.148.9/23
         ipv6 {
             address {
             }
         }
         switch-port {
             interface eth2
             interface eth3
             interface eth4
         }
         vif 2 {
             description "VLAN2 - TEST"
             mtu 1500
         }
         vif 10 {
             bridge-group {
                 bridge br0
             }
             description "VLAN10 - WAN"
         }
     }
 }
 service {
     gui {
         https-port 443
     }
     snmp {
         community public {
             authorization ro
         }
         contact brandon@avioc.org
         location "Kila, MT"
     }
     ssh {
         port 22
         protocol-version v2
     }
 }
 system {
     domain-name internal.avioc.org
     gateway-address 172.24.148.1
     host-name sw-kila-core0
     login {
         user bweisz {
             authentication {
                 encrypted-password xxxx
                 plaintext-password ""
                 }
             }
             full-name "Brandon Weisz"
             level admin
         }
         user ubnt {
             authentication {
                 encrypted-password xxxx
                 plaintext-password ""
             }
             level admin
         }
     }
     name-server 172.24.148.4
     name-server 172.24.148.2
     ntp {
         server 0.ubnt.pool.ntp.org {
         }
         server 1.ubnt.pool.ntp.org {
         }
         server 2.ubnt.pool.ntp.org {
         }
         server 72.250.181.65 {
         }
         server 172.24.148.2 {
         }
     }
     offload {
         ipv4 {
             vlan enable
         }
     }
     syslog {
         global {
             facility all {
                 level notice
             }
             facility protocols {
                 level err
             }
         }
         host 172.24.148.52 {
             facility all {
                 level info
             }
         }
     }
     time-zone America/Denver
 }
[edit]

 

New Member
bweisz
Posts: 21
Registered: ‎10-22-2013
Posts: 21
Kudos: 3
Registered: 10-22-2013

Re: EdgeRouter POE - snmpd memory leak?

Here is a quick idea of how fast its consuming memory... This is a sample after restarting snmpd:

bweisz@sw-kila-core0:~$ while :; do echo -n "`date +%T` " ; ps -p 2191 -o comm= -o size=,rss=,vsize=; sleep 600; done
15:06:16 snmpd            2868  4204  12620
15:16:16 snmpd            3368  4864  13120
15:26:16 snmpd            3872  5364  13624
15:36:17 snmpd            4372  5860  14124
15:46:17 snmpd            4748  6352  14500
15:56:17 snmpd            5248  6844  15000
16:06:17 snmpd            5752  7340  15504
16:16:17 snmpd            6260  7832  16012
16:26:17 snmpd            6772  8332  16524
16:36:17 snmpd            7276  8824  17028
16:46:17 snmpd            7784  9320  17536

 

New Member
bweisz
Posts: 21
Registered: ‎10-22-2013
Posts: 21
Kudos: 3
Registered: 10-22-2013

Re: EdgeRouter POE - snmpd memory leak?

The fast leak I have observed and reported appears to be caused by monomon utility from http://monomon.matf.de to query a standard ethernet interface on the EdgeRouter. 

I also upgraded the router to 1.4.0 alpha, and the fast leak is still present.

For now I can capture the same data with monomon pointed at a the other end of the link (FreeBSD router), and that does not leak memory....

I would certainly be curious if anyone else can reproduce the leak by using monomon.

 

 

Reply