Member
Posts: 151
Registered: ‎08-10-2012
Kudos: 102
Solutions: 10

New NSM5loco, high cpu and ethernet interrupt count

Deployed a new nanoStation M5 loco the other day, it's a client, powered through the power brick, connecting a Mikrotik RB260GSP switch and 2 UAP AC lite's into a network.  After a period of time the CPU goes to 100% and it stops passing traffic through to the switch.

 

bb-nsm5l.png

 

On further investigation the high cpu is caused by interrupts on the ethernet port:

 

Mem: 21008K used, 41108K free, 296K shrd, 1968K buff, 6164K cached
CPU:   0% usr   0% sys   0% nic   0% idle   0% io   0% irq  99% sirq
Load average: 1.15 1.07 1.01 3/28 964
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
    3     2 ubnt     RW       0   0%   0  96% [ksoftirqd/0]
  852     1 ubnt     S     7456  12%   0   2% /bin/infctld -m -c
  964   961 ubnt     R     2364   4%   0   1% top
  960   858 ubnt     S     2100   3%   0   1% /bin/dropbear -F -r /etc/persiste
    4     2 ubnt     SW       0   0%   0   0% [events/0]
XW.v6.1.6# cat /proc/interrupts 
           CPU0       
  2:   22039366            MIPS  cascade_sh, wifi0
  4: 3320250370            MIPS  eth0
  6:          0            MIPS  cascade
  7:   46964641            MIPS  timer
 16:          0        ATH MISC  ubnt-poll-aux-timer
 18:          0        ATH MISC  cascade
 19:         10        ATH MISC  serial
 20:          0        ATH MISC  Watchdog Panic Handler

ERR:          0

While it's functioning normally the interrupts on eth0 are in the low thousands. Once it stops passing traffic that's when it goes wild.

 

dmesg show this:

[18114.000000] NETDEV WATCHDOG: eth0 (): transmit queue 0 timed out
[18114.000000] athr_gmac_tx_timeout
[18114.002000] Trying to wake up queue. ACs free:
[18114.002000] [0] - 2
[18124.001000] athr_gmac_tx_timeout
[18124.003000] Trying to wake up queue. ACs free:
[18124.003000] [0] - 1
[18124.003000] ------------[ cut here ]------------
[18124.003000] WARNING: at /raid/build/devbox/tags/devbox-v6.1.6/openwrt/build_dir/linux-ar934x/ar724x-eth/include/athrs_mac_def.h:293 athr_gmac_direct_xmit+0x484/0x7fc [ar724x_eth]()
[18124.003000] TX ring OVERFLOW: tail 40 == head 40, nelem 128, queue running
[18124.003000] Modules linked in: ebt_mark ebt_mark_m ebt_arpnat ebt_vlan ebt_ip6 ebt_ip ebtable_nat ebtable_filter ebtable_broute ebtables rssi_leds(P) ubnt_poll(P) umac ath_dfs(P) ath_dev(P) ath_rate_atheros(P) ubnt_ewma_ht(P) ath_hal(P) asf(P) adf urd(P) x_tables pppoe pppox ppp_mppe ppp_async ppp_generic slhc crc_ccitt ar724x_eth libcrc32c crc32c sha1_generic michael_mic md5 hmac des_generic arc4 aes_generic ecb cbc ubnthal(P)
[18124.003000] Call Trace:
[18124.003000] [[8000b784]] dump_stack+0x8/0x34
[18124.003000] [[800251cc]] warn_slowpath_common+0x70/0x98
[18124.003000] [[80025234]] warn_slowpath_fmt+0x24/0x30
[18124.003000] [[83a87be0]] athr_gmac_direct_xmit+0x484/0x7fc [ar724x_eth]
[18124.003000] [[83a891d0]] athr_gmac_hard_start+0x94/0xc0 [ar724x_eth]
[18124.003000] [[80154994]] dev_hard_start_xmit+0x274/0x384
[18124.003000] [[80167fe8]] sch_direct_xmit+0x64/0x1b8
[18124.003000] [[80168238]] __qdisc_run+0xfc/0x134
[18124.003000] [[801574b8]] net_tx_action+0xe0/0x108
[18124.003000] [[8002ae20]] __do_softirq+0xb0/0x178
[18124.003000] [[8002af30]] do_softirq+0x48/0x68
[18124.003000] [[8002b1d0]] ksoftirqd+0x70/0x100
[18124.003000] [[8003b134]] kthread+0x7c/0x98
[18124.003000] [[800078ec]] kernel_thread_helper+0x10/0x18
[18124.003000] 
[18124.003000] ---[ end trace 1fd99e23855deb9b ]---
[18124.003000] ------------[ cut here ]------------
[18124.003000] WARNING: at /raid/build/devbox/tags/devbox-v6.1.6/openwrt/build_dir/linux-ar934x/ar724x-eth/include/athrs_mac_def.h:271 athr_gmac_direct_xmit+0x39c/0x7fc [ar724x_eth]()
[18124.003000] TX still owned by DMA: tail 40, head 40, nelem 128, queue running
[18124.003000] Modules linked in: ebt_mark ebt_mark_m ebt_arpnat ebt_vlan ebt_ip6 ebt_ip ebtable_nat ebtable_filter ebtable_broute ebtables rssi_leds(P) ubnt_poll(P) umac ath_dfs(P) ath_dev(P) ath_rate_atheros(P) ubnt_ewma_ht(P) ath_hal(P) asf(P) adf urd(P) x_tables pppoe pppox ppp_mppe ppp_async ppp_generic slhc crc_ccitt ar724x_eth libcrc32c crc32c sha1_generic michael_mic md5 hmac des_generic arc4 aes_generic ecb cbc ubnthal(P)
[18124.004000] Call Trace:
[18124.004000] [[8000b784]] dump_stack+0x8/0x34
[18124.004000] [[800251cc]] warn_slowpath_common+0x70/0x98
[18124.004000] [[80025234]] warn_slowpath_fmt+0x24/0x30
[18124.004000] [[83a87af8]] athr_gmac_direct_xmit+0x39c/0x7fc [ar724x_eth]
[18124.004000] [[83a891d0]] athr_gmac_hard_start+0x94/0xc0 [ar724x_eth]
[18124.004000] [[80154994]] dev_hard_start_xmit+0x274/0x384
[18124.004000] [[80167fe8]] sch_direct_xmit+0x64/0x1b8
[18124.004000] [[80168238]] __qdisc_run+0xfc/0x134
[18124.004000] [[801574b8]] net_tx_action+0xe0/0x108
[18124.004000] [[8002ae20]] __do_softirq+0xb0/0x178
[18124.004000] [[8002af30]] do_softirq+0x48/0x68
[18124.004000] [[8002b1d0]] ksoftirqd+0x70/0x100
[18124.004000] [[8003b134]] kthread+0x7c/0x98
[18124.004000] [[800078ec]] kernel_thread_helper+0x10/0x18
[18124.004000] 
[18124.004000] ---[ end trace 1fd99e23855deb9c ]---

I'm thinking it's probably just defective hardware and I need to swap it, but I thought I'd see if anybody else had any ideas...

 

Cheers

Ubiquiti Employee
Posts: 11,713
Registered: ‎11-27-2012
Kudos: 3748
Solutions: 773
Contributions: 73

Re: New NSM5loco, high cpu and ethernet interrupt count

@edwardg8  Could you try disabling Advanced Stats on the Advanced tab?  We have seen a few similar reports that are being worked on, but some are reporting that disabling Advanced Stats helps.  Please give that a try and let me know either way.

 

 

UBNT_Alternate_Logo.png
Ubiquiti Networks airMAX Support Team

Check out our ever-evolving Help Center for answers to many common questions!

FREE UBWA Student Guide-Great RF Primer!

Member
Posts: 151
Registered: ‎08-10-2012
Kudos: 102
Solutions: 10

Re: New NSM5loco, high cpu and ethernet interrupt count

Hi @UBNT-James

Thanks for the reply. I'd already tried that and tried disabling the multicast data option.

New Member
Posts: 1
Registered: ‎10-17-2017

Re: New NSM5loco, high cpu and ethernet interrupt count

 

 

 

New Member
Posts: 2
Registered: ‎09-26-2018

Re: New NSM5loco, high cpu and ethernet interrupt count

Hi,
I have the same issue with NSM5, I tried different configurations but the issue persists.

XW.v6.1.7# cat /proc/interrupts
CPU0
2: 8921997 MIPS cascade_sh, wifi0
4: 5560446 MIPS eth0
6: 0 MIPS cascade
7: 30459484 MIPS timer
16: 0 ATH MISC ubnt-poll-aux-timer
18: 0 ATH MISC cascade
19: 10 ATH MISC serial
20: 0 ATH MISC Watchdog Panic Handler

ERR: 0

********************

[110831.950000] ath_tx_start_dma: TXQ[1] stuck detected(len:50 lasTx 110830495+1455), trying to recover.
[154503.000000] NETDEV WATCHDOG: eth0 (): transmit queue 0 timed out
[154503.000000] AG934X_MAC:0: Tx Dma status eth0 : inactive
[154503.000000] mac:0 ret:1 RxFsm:25c00 TxFsm:ff900 RxFD:0 RxC:1 TxC:0
[154503.000000] Trying to wake up queue. ACs free: [0] - 2
[154503.000000] ------------[ cut here ]------------
[154503.000000] WARNING: at /raid/build/devbox/tags/devbox-v6.1.7/openwrt/build_dir/linux-ar934x/ar724x-eth/include/athrs_mac_def.h:254 athr_gmac_direct_xmit+0x3e4/0x88c [ar724x_eth]()
[154503.000000] TX ring FULL: tail 29 == head 27, nelem 128, queue running, tx inactive
[154503.000000] Modules linked in: ebt_mark ebt_mark_m ebt_arpnat ebt_vlan ebt_ip6 ebt_ip ebtable_nat ebtable_filter ebtable_broute ebtables rssi_leds(P) ubnt_poll(P) umac ath_dev(P) ath_rate_atheros(P) ubnt_ewma_ht(P) ath_hal(P) asf(P) adf urd(P) x_tables pppoe pppox ppp_mppe ppp_async ppp_generic slhc crc_ccitt ar724x_eth libcrc32c crc32c sha1_generic michael_mic md5 hmac des_generic arc4 aes_generic ecb cbc ubnthal(P)
[154503.000000] Call Trace:
[154503.000000] [[8000b784]] dump_stack+0x8/0x34
[154503.000000] [[800251cc]] warn_slowpath_common+0x70/0x98
[154503.000000] [[80025234]] warn_slowpath_fmt+0x24/0x30
[154503.000000] [[83a87c40]] athr_gmac_direct_xmit+0x3e4/0x88c [ar724x_eth]
[154503.000000] [[83a89314]] athr_gmac_hard_start+0x94/0xc0 [ar724x_eth]
[154503.000000] [[80154994]] dev_hard_start_xmit+0x274/0x384
[154503.000000] [[80167fe8]] sch_direct_xmit+0x64/0x1b8
[154503.000000] [[80168238]] __qdisc_run+0xfc/0x134
[154503.000000] [[801574b8]] net_tx_action+0xe0/0x108
[154503.000000] [[8002ae20]] __do_softirq+0xb0/0x178
[154503.000000] [[8002af30]] do_softirq+0x48/0x68
[154503.000000] [[8002b1d0]] ksoftirqd+0x70/0x100
[154503.000000] [[8003b134]] kthread+0x7c/0x98
[154503.000000] [[800078ec]] kernel_thread_helper+0x10/0x18
[154503.000000]
[154503.000000] ---[ end trace 594222ee087bbb2d ]---

*******

eth0 Link encap:Ethernet HWaddr F0:9F:C2:59Man Very Happy4:97
UP BROADCAST RUNNING PROMISC ALLMULTI MULTICAST MTU:1500 Metric:1
RX packets:7986502 errors:0 dropped:0 overruns:0 frame:0
TX packets:36815389 errors:0 dropped:0 overruns:19866 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1473645362 (1.3 GiB) TX bytes:2212943473 (2.0 GiB)

To recover the communication to LAN port, I need to reboot the device, so it causes outage for few minutes.

Has anyone found the solution?

Ubiquiti Employee
Posts: 11,317
Registered: ‎04-14-2017
Kudos: 2099
Solutions: 324

Re: New NSM5loco, high cpu and ethernet interrupt count

Have you checked that the cabling and device connected to this Ethernet port are free of errors?
New Member
Posts: 2
Registered: ‎09-26-2018

Re: New NSM5loco, high cpu and ethernet interrupt count

Thanks for your response.

 

The cable and POE was changed, but same issue. Also I changed the CPE router to discard it but same issue.

The cable was certified cat 6.

During the issue I can't see errors in eth0, however I see overruns.

Note the wireless link is working very good.

 

Which commands could you recommend to execute via CLI during the issue in order to detect it?

 

 

Ubiquiti Employee
Posts: 11,317
Registered: ‎04-14-2017
Kudos: 2099
Solutions: 324

Re: New NSM5loco, high cpu and ethernet interrupt count

@ubnt-Mindis do you have these commands to hand?
New Member
Posts: 2
Registered: ‎11-27-2015

Re: New NSM5loco, high cpu and ethernet interrupt count

did u resolve this issue? have the same troubles with our NSLM2 and NSLM5. @UBNT-SNK

Ubiquiti Employee
Posts: 11,317
Registered: ‎04-14-2017
Kudos: 2099
Solutions: 324

Re: New NSM5loco, high cpu and ethernet interrupt count

@UBNT-Mindas looks like I tagged you incorrectly, sorry.
New Member
Posts: 3
Registered: ‎02-18-2019

Re: New NSM5loco, high cpu and ethernet interrupt count

any luck on this issue? all similar issue have not been addressed yet.

Ubiquiti Employee
Posts: 11,317
Registered: ‎04-14-2017
Kudos: 2099
Solutions: 324

Re: New NSM5loco, high cpu and ethernet interrupt count

I'm afraid I do not have an update on this issue today.