Reply
Established Member
Posts: 2,552
Registered: ‎09-11-2009
Kudos: 982
Solutions: 55

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

@aaronjclark@efairbairn@UBNT-Mindas

 

Same exact story here, we are having issues with IP conflicts now as well from our DHCP server.

 

The problem is, the only way to resolve this as far as I can tell is to physically reboot each radio. I am really hoping someone from UBNT has a magic way to get into the radios so we dont have to call all these people.

Established Member
Posts: 2,552
Registered: ‎09-11-2009
Kudos: 982
Solutions: 55

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

[ Edited ]

One quick update, we actually DID have 10001 UDP blocked at our firewall. Perhaps we need to add TCP to that as well?

 

So our management ports are all blocked, 10001 UDP is blocked, and AC2 server is completely blocked from the outside world.

Emerging Member
Posts: 46
Registered: ‎11-01-2017
Kudos: 5

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

[ Edited ]

Can your CPEs access each other?  We have seen issues where one customers AirRouter or other ancient UBNT device with out of date firmware gets compromised and scans for neighbors to do the same to.  We have even seen them take current fimrware radios and downgrade them to old firmware with more vulnerabilies.  

Established Member
Posts: 2,552
Registered: ‎09-11-2009
Kudos: 982
Solutions: 55

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

We do have a few sectors where we had to shut of client isolation so they could access cameras on another site etc...

 

Any older UBNT firmware that is sneaking out there not in AC2 for us is on a different password. We actually had a similiar incident like this 6 months ago, so we upgraded all the firmware and changed the password on all managed devices.

 

Guess its time to go MPLS and bridge the CPE's with privates!

Emerging Member
Posts: 46
Registered: ‎11-01-2017
Kudos: 5

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

@Wifimax,

        It looks like discovery is responding on both UDP and TCP now.  We are now blocking both.

Established Member
Posts: 2,552
Registered: ‎09-11-2009
Kudos: 982
Solutions: 55

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

Same here, I think that was likely the issue.

Member
Posts: 193
Registered: ‎06-09-2014
Kudos: 78
Solutions: 4

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

I have blocked 10001 udp and tcp now for all my ip ranges on my core routers. 10001 udp is bombing from outside, the counter goes grazy up on all public ip ranges. tcp is zero.

 

Out

 

greetings

 

dsich

Ubiquiti Employee
Posts: 10,545
Registered: ‎11-27-2012
Kudos: 3310
Solutions: 671
Contributions: 73

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs


@dsich wrote:

I have blocked 10001 udp and tcp now for all my ip ranges on my core routers. 10001 udp is bombing from outside, the counter goes grazy up on all public ip ranges. tcp is zero.

 

Out

 

greetings

 

dsich


Would it be possible to try and grab a packet capture of this discovery (UDP 10001) traffic hitting your router?

 

Are all of your airMAX radios on 6.0.4+/8.1.3+?

UBNT_Alternate_Logo.png
Ubiquiti Networks airMAX Support Team

Check out our ever-evolving Help Center for answers to many common questions!

FREE UBWA Student Guide-Great RF Primer!

Member
Posts: 193
Registered: ‎06-09-2014
Kudos: 78
Solutions: 4

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

Yes, all my CPEs are up to date. I block the main interface now an we will see. I see this on all my CPEs and not on special ones.
Emerging Member
Posts: 77
Registered: ‎12-30-2010
Kudos: 29

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

I just blocked port 10001 udp on our firewall and I am seeing lots of activity coming from 192.155.71.17 which is owned by alarm.com, anyone else seeing the same IP?

 

--David

Highlighted
Ubiquiti Employee
Posts: 7,522
Registered: ‎04-14-2017
Kudos: 1460
Solutions: 207

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

I think it should only be responding on private addresses, if I recall correctly.
Ubiquiti Employee
Posts: 10,545
Registered: ‎11-27-2012
Kudos: 3310
Solutions: 671
Contributions: 73

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs


@UBNT-SNK wrote:
I think it should only be responding on private addresses, if I recall correctly.

Yes, this was added in 6.0.4/8.1.3 for discovery (UDP).  

UBNT_Alternate_Logo.png
Ubiquiti Networks airMAX Support Team

Check out our ever-evolving Help Center for answers to many common questions!

FREE UBWA Student Guide-Great RF Primer!

Member
Posts: 193
Registered: ‎06-09-2014
Kudos: 78
Solutions: 4

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

This morning (europe) i take a look at my routers and now is on 10001 udp silance. Only some single pings on 10001 tcp from the source 185.143.223.6 and 194.28.115.231. Some of the desination ip addresses are public ips from my routet block they are empty. This ips are not given to customers, they only routet to my core. That says, they scanning the complete range on 10001.
SuperUser
Posts: 5,737
Registered: ‎08-26-2009
Kudos: 1753
Solutions: 55

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

I don't see anything like this on my network, anywhere. So I have plenty of reason to doubt that this is some major issue. But I do follow a number of best practices to lower my attack surface. I've found that in router mode, it's very hard to do this completely, even when all of the "Block" optoins on. Too many daemons bind to 0.0.0.0 and not the management IPs only. Filtering is inferior to separate routing tables in this regard.

 

Whether or not there is an actual exploit happening, and whether or not it's persistent, listening on 10001 is a problem. If answers are restricted to private IPs only, that doesn't solve the attack surface issue. If I assume that everything i'm reading here is the absolute truth, and not a misinterpretation (there is a WIDE amount of room here for misinterpretation.)...Then, the daemons already listened and accepted enough input that infctld was able to hit a major (potentially exploitable) bug and start to consume 100% CPU. 

 

The real solution here, aside from actually fixing the bugs, is to lower the attack surface. airOS should run a modern enough version of the Linux kernel to have multiple routing table support. One routing table for user traffic (in router mode), and one routing table for management traffic (in both router and bridge mode). Management interfaces only participate in the management routing table, period. All other interfaces talk in the user routing table only. Virtually everything running on the box (except the DHCP client and UPNP server) should only participate in the management routing table. 

 

I know these are major changes, but for gear with this much use worldwide, it's worth the effort Man Happy Of course you are still free to be exploitable by putting your management interfaces where the internet or other users can reach them, thus compromising your network...Just like you are free to stand in the middle of highway traffic. This is a bar that each network operator has to learn to meet and exceed, kind of like "you must be this tall to ride" -- except nobody is telling you to get off the ride when you don't do it right. You have to use some technique in your network, perhaps like MPLS IP VPN, to create separate networks which aren't reachable from one another.

Ubiquiti Employee
Posts: 287
Registered: ‎12-21-2017
Kudos: 51
Solutions: 5

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

Our kernel version is quite new to support that. Adding @UBNT-SNK for a discussion on this feature: multiple routing table support for mgmt and client traffic separation.

Ubiquiti Employee
Posts: 287
Registered: ‎12-21-2017
Kudos: 51
Solutions: 5

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

@Wifimax,@efairbairn,@aaronjclark,@dsich,@dthomas

 

thanks for all the reported details - we appreciate it!

 

I would like to ask for some post-mortem details on the issue.

This is a list of questions, that would be helpfull in identifying what went wrong:

 

- Did the issus start suddenly and on many devices (not one)?

- Did the CPU load have an identifyable end - was it the kill of infctld process or rather the blocking the the 10001 port (upd/tcp), anything else?

- Are you using AirControl, if so - which version?

- Which firmware versions were running on the devices that were impacted?

- Any other details / correlations / strage packets in the network?

 

It would be great if you could answer in a list form as well.

Thanks a lot!

Established Member
Posts: 2,552
Registered: ‎09-11-2009
Kudos: 982
Solutions: 55

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

[ Edited ]

@UBNT-Mindas

- Did the issus start suddenly and on many devices (not one)?

Approximately 150 devices all within a few hours of each other. It appeared to be random across the network, the devices were not all in the same VLAN or tower.

 

- Did the CPU load have an identifyable end - was it the kill of infctld process or rather the blocking the the 10001 port (upd/tcp), anything else?

We rebooted the dish and the 100% CPU was resolved. The dishes we cannot manage are still showing offline.

 

- Are you using AirControl, if so - which version?

Server Version: 2.1-180316-1259

 

- Which firmware versions were running on the devices that were impacted?

Various, 6.1.6, 8.5.1 ,8.5.4 WA,XW,XC chipsets for sure impacted

 

- Any other details / correlations / strage packets in the network?

Nope, we have seen this issue once before a few months ago, but never got this far in tracking down what caused it.

Emerging Member
Posts: 46
Registered: ‎11-01-2017
Kudos: 5

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

[ Edited ]

@UBNT-Mindas,

 

- Did the issus start suddenly and on many devices (not one)?

              It started over the course of about 24 hours on the July 3rd and early July 4th - At least 100 devices impacted

 

- Did the CPU load have an identifyable end - was it the kill of infctld process or rather the blocking the the 10001 port (upd/tcp), anything else?

              Killing infctld or rebooting the device stopped the high CPU on the managable devices.  Blocking 10001 did not change the already high CPU devices behavior

 

- Are you using AirControl, if so - which version?

              Yes, v2.1.1-Beta-180525-1045

 

- Which firmware versions were running on the devices that were impacted?

              6.1.6, 6.1.7, 8.5.1, 8.5.4

 

- Any other details / correlations / strage packets in the network?

              We are still trying to get ahold of some customers which have the unmanagable devices to have them power cycle.  These devices are now causing us huge issues with DHCP IP conflicts, to the point we are starting to ban them from APs until the customer calls back.  It seems the unmanagable devices are not responding to the DHCP queries so the IP addresses are getting reissues to other devices. 

              10001 TCP and UDP were open at the time that this started happening, and as far as we can tell only devices with public IP addreses were impacted.  We have since blocked 10001 TCP and UDP, and are not seeing any reoccurence yet.  

 

Thank you

-Aaron

Ubiquiti Employee
Posts: 7,522
Registered: ‎04-14-2017
Kudos: 1460
Solutions: 207

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

Thanks @Wifimax, we will continue investigating.
Ubiquiti Employee
Posts: 287
Registered: ‎12-21-2017
Kudos: 51
Solutions: 5

Re: Possible Exploit - Losing access to SSH and HTTP/HTTPS on CPEs

@aaronjclark @Wifimax - thank you for the details. This will be very helpfull for continuing troubleshooting efforts and in case it happens in the future.

 

Reply