Ubiquiti Employee
Posts: 11,657
Registered: ‎04-14-2017
Kudos: 2182
Solutions: 335

Re: M5 LAN not passing traffic even though WAN is up

There is no specific fix in this release for this issue, but I'm concerned that you're seeing it on 20 units. Do you have any support files from those we can take a look at?
Member
Posts: 264
Registered: ‎06-10-2009
Kudos: 8
Solutions: 3

Re: M5 LAN not passing traffic even though WAN is up

I just have a new case, one powerbeamM5 400, connected with powerbeamAC 400ISO, in mixed mode, I see in the dashboard of AC the connected station, traffic pass, but no ping to M5, and CPU fixed to 100%, no ssh, no http, some ping respond but not so much ... now after some hour, without success I can enter in the gear and no usefull log.

I download the sup file and update to 6.1.6 ... 

Attachment
New Member
Posts: 4
Registered: ‎11-23-2016

Re: M5 LAN not passing traffic even though WAN is up

@UBNT-SNK Just tested 6.1.6 last night and it was down again this morning. What firmware would you reccomend downgrading to?

 

We have hundreds in the field, so 20+ isn't so drastic. Seems to be only happening on new units in the last 4-5 months, unfortunately this has only recently come by desk and other colleagues where just replacing them with older units 5.6.X which seem to be unaffected. Due to that I don't have a collection of many sup files at this point but I will be saving more going forward.

 

Issue seems to be with units running 6.1.X but I need to gather more data on that. 

 

For now I have the sup from the same radio on 6.1.4 and 6.1.6.

Attachment
Ubiquiti Employee
Posts: 11,657
Registered: ‎04-14-2017
Kudos: 2182
Solutions: 335

Re: M5 LAN not passing traffic even though WAN is up

Thanks for the support files. You can try the signed version of 5.6.15 as a baseline to see if it's resolved there. In the meantime we will look into these.

@UBNT-Zy please use these support files when looking into this issue.
Ubiquiti Employee
Posts: 587
Registered: ‎08-20-2007
Kudos: 137
Solutions: 23

Re: M5 LAN not passing traffic even though WAN is up

 

New Member
Posts: 4
Registered: ‎11-23-2016

Re: M5 LAN not passing traffic even though WAN is up

@UBNT-SNK@UBNT-Zy

 

I have downgraded the radio from the first two sup files to 5.6.15 and will let you know if that one stays up.

 

However, I have some more sup files for you two from a different radio that was running 5.6.11, I had updated it to 5.6.15 last night and it failed again some time in the early morning.

 

Can't say if the CPU was pinned at 100% as 5.6.X does not display that info on the main tab but seems to be related as the WLAN was still up and all LAN traffic dropped to nothing, came back immediately on a reboot.

Attachment
New Member
Posts: 4
Registered: ‎11-23-2016

Re: M5 LAN not passing traffic even though WAN is up

@UBNT-Zy

 

Oops, didn't refresh this thread when I had just replied, didn't see your response.

 

This unit is currently at a very remote location, it will take a week or two before I'll be able to get it back in my hands to start the RMA process.

 

 

Ubiquiti Employee
Posts: 11,657
Registered: ‎04-14-2017
Kudos: 2182
Solutions: 335

Re: M5 LAN not passing traffic even though WAN is up

Please let me know if you have any issues with the RMA.
Ubiquiti Employee
Posts: 587
Registered: ‎08-20-2007
Kudos: 137
Solutions: 23

Re: M5 LAN not passing traffic even though WAN is up

@grant-hc

Thanks for the support file. My suggestion is not changing - please arrange F09FC2528E93 unit back for inspection.

Thank you.

 

Ubiquiti Employee
Posts: 11,657
Registered: ‎04-14-2017
Kudos: 2182
Solutions: 335

Re: M5 LAN not passing traffic even though WAN is up

I agree, and please let us know if you have any issues getting this unit back to us for inspection. We will provide a replacement to you.
Emerging Member
Posts: 61
Registered: ‎04-21-2009
Kudos: 3

Re: M5 LAN not passing traffic even though WAN is up

I have been seeing this for at least three months and possibly even as long as 15 months ago.

 

I have a PowerBeamM5 - 400 running XW 6.1.4 as a backhaul link connected to an EdgeRouter.  Wireless performance is excellent (consistently 99% or better CCQ and consistently 216 Mbps wireless Tx and Rx rates).  Throughput can easily approach 100 Mbps "down" and 10 Mbps "up".  This backhaul link runs flawlessly and then the PowerBeamM5 stops communicating over its LAN port with the EdgeRouter.  I have a tight ping watchdog configuration (pings every 12 seconds and reboots after 10 lost pings) which ultimately reboots the PowerBeamM5 and restores communication.

 

I started with an EdgeRouter-5-PoE and PowerBeamM5.  I replaced one and then the other and the problem persisted.

 

Next, I replaced Cat5e run between the EdgeRouter-5-PoE and the PowerBeamM5 and the problem persisted.

 

Next, I stopped using the PoE feature on the EdgeRouter-5-PoE and used a Ubiquiti 24 volt Gigabit PoE brick and the problem persisted.

 

Next, I replaced the EdgeRouter-5-PoE with an EdgeRouter-X-SFP and the problem persisted.

 

Finally, I replaced 250 feet of the Cat5e run with single-mode fiber, a media converter connected to the PowerBeamM5 - 400, and an optic in the SFP slot of the EdgeRouter-X-SFP and the problem still persists.

 

In summary I have replaced everything and even used a slightly different EdgeRouter and the problem persists.

 

When I periodically run the ethtool command on the PowerBeamM5, it indicates 1000 Mbps connection and the ifconfig command indicates zero interface errors.  And I get the same on the EdgeRouter-X-SFP.  Also, the PowerBeamM5 GUI looks normal indicating about 37% memory usage and anywhere from 36% to 58% CPU utilization.

 

The last time this happened, I ran the command sudo arp -i eth5 on the EdgeRouter and it took forever (like 30+ seconds) to return any results.  Perhaps there were no results to show because the PowerBeamM5 - 400 was in the process of rebooting???

 

This is a show stopper -- I cannot have my backhaul going down twice an evening for 5 minutes during peak usage and take 50 subscribers offline.  Ubiquiti needs to fix this ASAP.  And upgrading to AC radios is not an option since that requires two tower climbs and taking everyone offline for anywhere from 15 minutes (if I have two tower climbers) to 2 hours (if I have only one tower climber).

Roy Grove
President
ezwisp.com Inc.
Emerging Member
Posts: 61
Registered: ‎04-21-2009
Kudos: 3

Re: M5 LAN not passing traffic even though WAN is up

This morning I downgraded my wireless bridge (two PowerBeamM5s in bridge mode) to 5.6.15 (signed).  Within about 10 minutes of coming back online, the same (local to me) PowerBeamM5 that has been losing communication on its LAN port lost communication on its LAN port to my EdgeRouter-X-SFP.  The outage lasted a few minutes until the ping watchdog on that PowerBeamM5 rebooted it.

 

While that was happening, I had SSH running on the EdgeRouter and ran the command "sudo arp".  The command returned maybe one result within two seconds, returned maybe two more results over the next 30 to 60 seconds, and then promptly returned the remaining 20+ results when PowerBeamM5 resumed communication on its LAN port to my EdgeRouter.

 

I have no idea if that behavior on the EdgeRouter is normal while something strange is happening on one of its interfaces.  (In this case the interface is the "uplink" which moves traffic "upstream" to the gateway at the edge of my network.)

 

Traffic level seemed to be fairly light (a few Mbps???) at the time that the PowerBeamM5 stopped communication on its LAN port.  Once the link resumed operation, traffic was running between 30 and 40 Mbps going toward the LAN port.

Roy Grove
President
ezwisp.com Inc.
Emerging Member
Posts: 61
Registered: ‎04-21-2009
Kudos: 3

Re: M5 LAN not passing traffic even though WAN is up

The PowerBeamM5 at my location has been in station WDS mode and the remote PowerBeamM5 has been in access point WDS mode.  (And since there was only two devices they were a point-to-point bridge.)  This morning I changed the PowerBeamM5 at the remote end from access point WDS mode to station WDS mode and I replaced the PowerBeamM5 at my location with a PowerBeam5-AC in access point mode.

 

Now I am seeing large spikes in CPU utilization in the PowerBeamM5 (in station WDS mode) at the remote end.  The CPU spikes seem to track quite well with traffic.  Here are approximate traffic and CPU utilization levels:

traffic ----- CPU utilization

40 Mbps   70%

50 Mbps   80%

60 Mbps   90%

 

Is the PowerBeamM5 simply running out of processing power as traffic levels approach/exceed 70+ Mbps???  Or is there a firmware defect that consumes processing power when traffic levels increase?

 

Again, these two devices are a simple wireless bridge with a simple configuration (all through the Graphical User Interface) and no routing.

Roy Grove
President
ezwisp.com Inc.
Ubiquiti Employee
Posts: 11,657
Registered: ‎04-14-2017
Kudos: 2182
Solutions: 335

Re: M5 LAN not passing traffic even though WAN is up

Inspecting the CPU usage can be deceiving. When you say 70Mbps is that aggregate or are you seeing that in both directions?
New Member
Posts: 1
Registered: ‎03-21-2018

Re: M5 LAN not passing traffic even though WAN is up

Hello UBNT-SNK,

 

70 Mbps traffic was in one direction, and there was something like 5 Mbps in the other direction.  Thus, total aggregate traffic was about 75 Mbps.

 

Here is a "visual" depiction of my original wireless bridge link:

 

local (station WDS) ----------------- remote (access point WDS)

PowerBeamM5-400  <----- 2 miles ------> PowerBeamM5-400

<-------------------------------- 70 Mbps ----------------------------<<<

>>> ---------------------------- 5 Mbps --------------------------------->

Wireless (symbol) rate rock solid at 216 Mbps both directions.

Transmit CCQ rock solid between 99% and 100% both directions.

 

 

Here is a "visual" depiction of my UPDATED wireless bridge link where I

replaced one PowerBeamM5 - 400 with a PowerBeam5 - AC:

 

local (access point) ----------------------- remote (station WDS)

PowerBeam5 - AC  <----- 2 miles ------> PowerBeamM5-400

<-------------------------------- 70 Mbps ----------------------------<<<

>>> ---------------------------- 5 Mbps --------------------------------->

sustained CPU utilization about 90% on PowerBeamM5 - 400

sustained CPU utilization hitting roughly 30% on PowerBeam5 - AC

wireless rates and transmit CCQ equal or better than original configuration.

 

Note that the wireless bridges at both ends connect to EdgeRouter-X-SFPs and the LAN links are 1,000 Mbps (Gbit) Ethernet.  Both EdgeRouter-X-SFPs and both wireless bridges indicate zero interface errors.  This is the case in my original configuration as well as my updated configuration.

 

Thank you.

Emerging Member
Posts: 61
Registered: ‎04-21-2009
Kudos: 3

Re: M5 LAN not passing traffic even though WAN is up

Hello UBNT-SNK,

 

That previous reply from "dexterumc" is actually from me.  I had logged into the Ubiquiti environment last night as a consultant to a client and did not realize that I was still logged in as the client's consultant.

Roy Grove
President
ezwisp.com Inc.
Emerging Member
Posts: 61
Registered: ‎04-21-2009
Kudos: 3

Re: M5 LAN not passing traffic even though WAN is up

Hello UBNT-SNK,

 

That reply from "dexterumc" is actually from me -- answering your question about 70 Mbps traffic flow.

 

(I had logged into the Ubiquiti environment last night as a consultant to a client and did not realize that I was still logged in as the client's consultant.)

Roy Grove
President
ezwisp.com Inc.
New Member
Posts: 36
Registered: ‎11-12-2014
Kudos: 1
Solutions: 1

Re: M5 LAN not passing traffic even though WAN is up

Hi guys,

 

I've see the same issue in some devices. Unfortunately rebooting device temporary solve the problem so I think that customers have learned to reboot device instead to call support Man Sad

 

Anyway I can report my today case. I hope it could help solving issue.

 

Nannostation M5

Firmware 6.1.7

Sometimes it stop passing data on ethernet or better... In LAN0 graph, in Main page, I see few bytes in RX but zero bytes in TX (flat graph to zero). If I log into device in ssh and run tcpdump on eth0 I see this:

 

09:03:32.678371 ARP, Request who-has 192.168.1.1 tell 192.168.1.149, length 46
09:03:32.678464 ARP, Reply 192.168.1.1 is-at 80:2a:a8:0f:2b:a3, length 28
09:03:33.678384 ARP, Request who-has 192.168.1.1 tell 192.168.1.149, length 46
09:03:33.678477 ARP, Reply 192.168.1.1 is-at 80:2a:a8:0f:2b:a3, length 28

 

So Nanostation receives data from customer device (in this case Netgear router) and it seems to send data but no data really pass through the ethernet to the router.

 

I had enabled log but in log/messages there's nothing.

 

If I run "top" ksoftirqd/0 uses 87% Cpu ans the whole cpu is at 100% but sometimes I've seen this Nanostation stuck also with normal cpu usage.

 

If you need I've dowloaded support file.

 

I hope I can help in solve this issue... and I hope you can help me!

 

Thanks

Ubiquiti Employee
Posts: 592
Registered: ‎12-21-2017
Kudos: 104
Solutions: 16

Re: M5 LAN not passing traffic even though WAN is up

@3m4x - can you paste the dmesg output after the issue starts? /var/log/messages would be usefull as well.

New Member
Posts: 36
Registered: ‎11-12-2014
Kudos: 1
Solutions: 1

Re: M5 LAN not passing traffic even though WAN is up

@UBNT-MindasSorry for my late reply...

 

I just saw the issue:

 

- cpu at 100%

- no traffic on LAN

 

I got may data:

 

- screenshot

- logs

- dmesg output

- top output

- ps output

 

but files are to big to paste. How could I send you?