Reply
Emerging Member
Posts: 51
Registered: ‎11-01-2017
Kudos: 7

Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

[ Edited ]

I'm seeing a concerning issue where a number of our CPEs are suddenly unmanageable via both SSH and HTTP/HTTPS.  I can ping them, and they seem to be passing traffic as normal, but SSH just hangs indefinitely, and HTTP redirects to HTTPS, but I can't ever get past the SSL warning page. I just hangs at "Waiting for".  Based on the number of radios that have dropped offline in AirControl I would estimate at least 100 radios are impacted currently.  

 

I've tried ssh from an adjacent radio on the same AP with Client Isolation disabled so I'm confident that it's not an internal networking issue that I'm fighting.  

 

Has anyone else seen this behavior?  Is this some new exploit I'm not yet aware of?  

 

The CPE I'm troubleshooting with currently is a NanoStation M5 running v6.1.6 connected to a Prism 5AC Gen2 running v8.5.1.  I've seen this on both M and AC gear running various firmware versions.

 

The CPEs impacted have different passwords, so I don't think its as simple as a compromised password.  

 

Help!

Emerging Member
Posts: 51
Registered: ‎11-01-2017
Kudos: 7

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

[ Edited ]

NMAP shows 22/tcp and 10001/tcp as the only open ports. 

 

SSH attempts return this error "ssh_exchange_identification: read: Connection reset by peer"

 

WGET shows the redirect from http to https, and the 302 redirect to /login.cgi?uri=/  

The final line where it hang shows "HTTP request sent, awaiting response... "

Ubiquiti Employee
Posts: 8,579
Registered: ‎04-14-2017
Kudos: 1622
Solutions: 246

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

I'm not aware of any active exploits like this for those software versions, and I think this is more likely to be a network issue of some type that is occurring. Maybe something like broadcast traffic on your management VLAN.
Emerging Member
Posts: 51
Registered: ‎11-01-2017
Kudos: 7

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

I'm seeing this on some radios but not all radios on the same AP and in the same subnet.  The issue is consistently present with one CPE, and not with the next.  

Emerging Member
Posts: 51
Registered: ‎11-01-2017
Kudos: 7

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

This is happening across multiple vlans and subnets all over the network.  I have a tech going to swap out  a CPE and bring back the impacted unit so I can diagnose further.  

SuperUser
Posts: 5,797
Registered: ‎08-26-2009
Kudos: 1771
Solutions: 55

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

Uhh, is the management IP available via public internet IP? Or available to your other customers on private IP? both are no-no, the first is obviously much higher exposure...

Ubiquiti Employee
Posts: 11,006
Registered: ‎11-27-2012
Kudos: 3474
Solutions: 715
Contributions: 73

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

Are you running the custom script versions of these firmware builds?  If not, you should be able to just reboot the radio to clear out anything running on the radios.  Assuming no passwords have been changed, you should be able to get access.

 

Make sure to account for other devices like airRouter, aiurGateways and Toughswitches as they are often overlooked (from what I've seen).  

UBNT_Alternate_Logo.png
Ubiquiti Networks airMAX Support Team

Check out our ever-evolving Help Center for answers to many common questions!

FREE UBWA Student Guide-Great RF Primer!

Emerging Member
Posts: 51
Registered: ‎11-01-2017
Kudos: 7

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

[ Edited ]

No, everything is firewalled off and isolated so there is no clear path in from the outside. We also use non standard HTTP and HTTPS ports to help prevent these sorts of things.

Emerging Member
Posts: 51
Registered: ‎11-01-2017
Kudos: 7

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

UBNT-James,

     Thanks for the info.  I wasn't aware that the standard firmware would clear out the files on reboot.  A reboot did bring it back online in this case, but clearly something is causing this.  The only other thing I've thought of is that we just upgraded our AirControl server to the latest beta version around the same time this started happening.  I wonder if something it's doing is causing this issue.  

 

With aproximatly 10,000 UBNT devices on the network it's difficult to get every device.  AirControl has helped considerably with this effort, but you're correct.  One customer airRouter with ancient firmware is all it takes to get a bunch of current devices compromised.  

 

I'll let you know what we find as we dig deeper.  

Ubiquiti Employee
Posts: 11,006
Registered: ‎11-27-2012
Kudos: 3474
Solutions: 715
Contributions: 73

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs


@aaronjclark wrote:

UBNT-James,

     Thanks for the info.  I wasn't aware that the standard firmware would clear out the files on reboot.  A reboot did bring it back online in this case, but clearly something is causing this.  The only other thing I've thought of is that we just upgraded our AirControl server to the latest beta version around the same time this started happening.  I wonder if something it's doing is causing this issue.  

 

With aproximatly 10,000 UBNT devices on the network it's difficult to get every device.  AirControl has helped considerably with this effort, but you're correct.  One customer airRouter with ancient firmware is all it takes to get a bunch of current devices compromised.  

 

I'll let you know what we find as we dig deeper.  


Please let me know how it goes @aaronjclark

UBNT_Alternate_Logo.png
Ubiquiti Networks airMAX Support Team

Check out our ever-evolving Help Center for answers to many common questions!

FREE UBWA Student Guide-Great RF Primer!

Emerging Member
Posts: 51
Registered: ‎11-01-2017
Kudos: 7

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

James,

      I have the radio in my lab now, but as you mentioned it seems to wipe the issue on reboot.  That means I can't bring one back to troubleshoot.  The number of offline devices does not seem to be increasing currently.  I'm going to have the cusotmers power cycle the CPEs and I'll let you know if any of them don't come back.  In the mean time I'd be happy to get you in front of one via teamviewer if it would do any good.

 

Thank you

Emerging Member
Posts: 79
Registered: ‎12-30-2010
Kudos: 29

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

I am having the same issue.  I have about 10 right now showing the same symptoms. Firmware versions are 6.1.3, 6.1.7 or 8.4.3.

 

It seems the only way to get in to them is a power cycle.

Emerging Member
Posts: 51
Registered: ‎11-01-2017
Kudos: 7

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

Are you running AirControl, and if so which version?

Ubiquiti Employee
Posts: 11,006
Registered: ‎11-27-2012
Kudos: 3474
Solutions: 715
Contributions: 73

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

I would try updating any AC units seeing this to 8.5.6-BETA2 as there is an AC2 mem issue addressed in this release.  

 

Make sure all your other devices are up to date.  This includes airRouters, airGateways and Toughswitches.

 

If management interfaces are available to the Internet, don't.

UBNT_Alternate_Logo.png
Ubiquiti Networks airMAX Support Team

Check out our ever-evolving Help Center for answers to many common questions!

FREE UBWA Student Guide-Great RF Primer!

Emerging Member
Posts: 51
Registered: ‎11-01-2017
Kudos: 7

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

UBNT-James,

       While I can't imagine there is much shared code, this feels a lot like the issue we are having on the AF11fx radios where we randomly lose management access from the RF side requiring a power cycle to resolve.  The main symptomatic difference here is that these CPEs are so far not managable localy either, while on the AF11fx we can manage them over the wired ethernet interface.

 

https://community.ubnt.com/t5/airFiber/AF11fx-stops-responding-to-ping-and-management-from-the-wirel...

 

Thanks

-Aaron

Emerging Member
Posts: 79
Registered: ‎12-30-2010
Kudos: 29

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs


@aaronjclark wrote:

Are you running AirControl, and if so which version?


Yes, v 2.1.1-Beta-180525-1045

 

This did seem to start happening around the time I updated AirControl.

Emerging Member
Posts: 51
Registered: ‎11-01-2017
Kudos: 7

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

That is exctly the same version of AirControl we are running.

Emerging Member
Posts: 79
Registered: ‎12-30-2010
Kudos: 29

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs


@UBNT-James wrote:

I would try updating any AC units seeing this to 8.5.6-BETA2 as there is an AC2 mem issue addressed in this release.  

 

Make sure all your other devices are up to date.  This includes airRouters, airGateways and Toughswitches.

 

If management interfaces are available to the Internet, don't.


I will give that a try later on, right now I only have 1 AC unit with the issue, the rest are all XM or XW.

 

--David

Emerging Member
Posts: 51
Registered: ‎11-01-2017
Kudos: 7

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

UBNT-James,

      Can you confirm if the AC2 related memory issue you are referring to is resolved in 8.5.7, or if we need to go back to 8.5.6 Beta2?

 

Thank you

-Aaron

Ubiquiti Employee
Posts: 11,006
Registered: ‎11-27-2012
Kudos: 3474
Solutions: 715
Contributions: 73

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs


@aaronjclark wrote:

UBNT-James,

      Can you confirm if the AC2 related memory issue you are referring to is resolved in 8.5.7, or if we need to go back to 8.5.6 Beta2?

 

Thank you

-Aaron


No, it is not.  I know the version numbering seems to indicate it should be resolved (since 8.5.7 > 8.5.6), but he had to push out 8.5.7 as a hotfix to address an ethernet issue on dual port airMAX AC radios.  We will be renumbering 8.5.6-beta branch to 8.5.8).

 

 

UBNT_Alternate_Logo.png
Ubiquiti Networks airMAX Support Team

Check out our ever-evolving Help Center for answers to many common questions!

FREE UBWA Student Guide-Great RF Primer!

Reply