Senior Member
Posts: 2,775
Registered: ‎09-11-2009
Kudos: 1138
Solutions: 64

Losing management access to devices

[ Edited ]

I have HTTP, SSH, and TCP/UDP port 10001 blocked from accessing our CPE's. I still have some CPE's that have lost management access. When the CPE management dies, and I run a UDP port scan I see just a ton of ports showing up on nmap.

 

1. Has there been a fix included in any of the recent firmware releases? I am on 8.5.7/6.1.7.

2. Is there anyway to exploit any of these ports to reboot the device remotely to clear this condition?

 

Capture.JPG

 

@UBNT-James

@UBNT-SNK

@UBNT-AL

Thanks guys!

 

Ubiquiti Employee
Posts: 11,657
Registered: ‎04-14-2017
Kudos: 2165
Solutions: 334

Re: Losing management access to devices

I am not aware of a way to push these ports remotely to reboot and clear this condition. @UBNT-Dalia can clarify on where the fix for this issue is in the release cycle.
Senior Member
Posts: 2,775
Registered: ‎09-11-2009
Kudos: 1138
Solutions: 64

Re: Losing management access to devices

Thanks @UBNT-SNK I look forward to hearing from @UBNT-Dalia to see where things are at.

Ubiquiti Employee
Posts: 173
Registered: ‎03-21-2017
Kudos: 139
Solutions: 16

Re: Losing management access to devices

[ Edited ]

@Wifimax, the "open|filtered" state for UDP scanning is normal and means that nmap doesn't know if the port is open or not because it did not receive any reply. Check out the corresponding section in https://nmap.org/book/man-port-scanning-basics.html

Could you please run a scan on a similarly configured device which works fine and compare open TCP ports?
Are 8080/TCP and 9080/TCP your configured ports or unexpected? 10001/TCP is discovery (TCP discovery only answers to requests from private IP ranges).

1. Has there been a fix included in any of the recent firmware releases? I am on 8.5.7/6.1.7.
No, we were unable to get to the bottom of this issue yet. But v8.5.8 includes discovery lib refactoring (though we haven't got 100% evidence that discovery service is at fault here) and v8.5.11 includes SSH lib update, so I would upgrade to v8.5.11 and watch for symptomps. If you reboot a device, do you lose management access again with 100% reproducability on the same device?

2. Is there anyway to exploit any of these ports to reboot the device remotely to clear this condition?
We can only access airMAX AC devices using HTTP/SSH/telnet/mactelnet using configured credentials.

What do you mean by "I have HTTP, SSH, and TCP/UDP port 10001 blocked from accessing our CPE's"? At which level is it blocked? Can CPEs reach each other in your network?

Senior Member
Posts: 2,775
Registered: ‎09-11-2009
Kudos: 1138
Solutions: 64

Re: Losing management access to devices

[ Edited ]

@UBNT-AL

 
"Could you please run a scan on a similarly configured device which works fine and compare open TCP ports?
Are 8080/TCP and 9080/TCP your configured ports or unexpected? 10001/TCP is discovery (TCP discovery only answers to requests from private IP ranges)"

 

Yes when I run the exact same scan on other devices in my network I only get 8080, 9080 and 10001 open, no UDP ports show up whatsoever. We use 8080 and 9080 for non-standard management ports. Thats why it confused me by showing up all these UDP ports. Perhaps they have IPtables blocking an IP address thats not from the attacker?

 

"If you reboot a device, do you lose management access again with 100% reproducability on the same device?"

 

If port 10001 UDP is open to the internet, yes every device will eventually become compromised.


"What do you mean by "I have HTTP, SSH, and TCP/UDP port 10001 blocked from accessing our CPE's"? At which level is it blocked? Can CPEs reach each other in your network?"

 

We block those ports at our core firewall, but certain CPE's may be able to access other CPE's on the network.

Ubiquiti Employee
Posts: 173
Registered: ‎03-21-2017
Kudos: 139
Solutions: 16

Re: Losing management access to devices


 "Could you please run a scan on a similarly configured device which works fine and compare open TCP ports?
Are 8080/TCP and 9080/TCP your configured ports or unexpected? 10001/TCP is discovery (TCP discovery only answers to requests from private IP ranges)"

 

Yes when I run the exact same scan on other devices in my network I only get 8080, 9080 and 10001 open, no UDP ports show up whatsoever. We use 8080 and 9080 for non-standard management ports. Thats why it confused me by showing up all these UDP ports. Perhaps they have IPtables blocking an IP address thats not from the attacker?


Yes, very possible - if the device is infected, the attacker might be changing firewall rules and nmap could see different/no responses. Do you see any suspicious traffic on an affected device?


"If you reboot a device, do you lose management access again with 100% reproducability on the same device?"

 

If port 10001 UDP is open to the internet, yes every device will eventually become compromised.


We had a pair of exposed honeypot devices (M & AC) for two weeks, but they didn't catch anything. Port 10001 did see some activity, but it was minor (discovery scan / some gas pump exploit / some printer exploit).

Is it possible in your network topology that the attack is coming from customer equipment like PC or smartphone?

Would it be possible for you to try this?

  1. Start packet capture for traffic to a single exposed device with "dst port 10001" filter and wait until SSH/HTTP access is lost. The difficult part is that we need captures from both sides: WAN & LAN. With a packet capture and a timestamp of the exploit we could reproduce this issue locally and solve it.

So far we have received some traffic dumps, but unfortunately they contained general 10001 port traffic (incl. some computer games, which generated quite a bit of bytes) during which no devices where compromised, so it didn't help to reproduce. Also we are 100% assuming that port 10001 is used for the exploit, which seems correct from all the reports, but still unproven.


"What do you mean by "I have HTTP, SSH, and TCP/UDP port 10001 blocked from accessing our CPE's"? At which level is it blocked? Can CPEs reach each other in your network?"

 

We block those ports at our core firewall, but certain CPE's may be able to access other CPE's on the network.


So at some point you had 10001 unblocked and a bigger percentage of devices was affected?

Senior Member
Posts: 2,775
Registered: ‎09-11-2009
Kudos: 1138
Solutions: 64

Re: Losing management access to devices

I'll work on some sniffing and see what I can track.

 

But yes, due to an error in our networking side the 10001 block was erroneously removed temporarily and devices started dropping like flies.