Reply
New Member
Posts: 7
Registered: ‎12-20-2018

UniFi UAP AC Pro Rebooting (Kernel Panics)

[ Edited ]

I have 2 UAP AC Pros (Gen 2) that have been rebooting on their own for several months now. I'm finally getting around to tracking down the reason. As far as I can remember, this started with the 4.x.x firmwares (but maybe 3.9.54), and so far has had the same issues with all versions since and up to the latest version 4.0.21.9965. I have not yet tried downgrading beyond 3.9.54 as I would like to try to figure out the problem and stay on the later firmwares.

 

I have searched for simliar issues, but I have not found one that is really doing what my APs are, or one that was resolved. I think it could be a load issue. At one point I could trigger a kernel panic just by FTPing thousands of smallish files from one computer to another over wifi.

 

Versions

Controller: 5.9.29.0

Current AP Firmware: 4.0.21.9965

 

Max Connected Devices: 10-15

Second AP is a Wireless Uplink

 

I set up syslog and turned on debug a couple of weeks ago to start capturing the issues. See attached for my syslog capture since 2/1/19. This is just for one of the APs, but there were several instances of kernel panics. It is connected to, and powered by, a UniFi Switch 8 POE-60W.

 

You will see 11 instances of kernel panics at the below timestamps:

 

Firmware version 4.0.15.9872

2019-02-07 05:24:22

2019-02-06 21:39:00

2019-02-06 21:38:56

2019-02-06 16:50:35
2019-02-06 05:24:28
2019-02-05 20:09:02
2019-02-05 05:26:35
2019-02-05 05:26:30
2019-02-05 05:25:44
2019-02-02 04:50:32
2019-02-02 04:44:27

 

I have had reboots on the latest firmware, but no kernel panics. Still looking into that. Somewhere around 2019-02-07 14:34:00 and 2019-02-07 14:40:00.

 

I trimmed the syslog since the below 3 items showed up on 2/5 and 2/6 with over 17 millions lines:


kernel: [88593.xxxxxx] BUG: Bad rss-counter state mm:844bf000 idx:0 val:522
kernel: [88593.xxxxxx] BUG: Bad rss-counter state mm:844bf000 idx:1 val:201
kernel: [88593.xxxxxx] BUG: non-zero nr_ptes on freeing mm: 4

 

Let me know if I need to post anything else, or go further back in the syslog. I started the syslog with firmware 4.0.15.9872.

Senior Member
Posts: 16,461
Registered: ‎08-04-2017
Kudos: 3006
Solutions: 810

Re: UniFi UAP AC Pro Rebooting (Kernel Panics)

Hello @icedtrip,

 

Even happening on 4.0.21?

Could you give us some details about your network?

What features are you running?

 

Client count?

 

 

Regards,
Glenn R.

 

 

Regards,
Glenn R.

 

CC @UBNT-BenBuckley 

Cloud Hosted Controllers | Glenn R. | UniFi Installation/Easy Update Scripts | UniFi-VoIP Installation Scripts
USG-4-PRO • USG
USW-48-500W • USW-24-POE-250W 2x • USW-16-POE-150W 3x • USW-24 • USW-8-150W • USW-8
UAP XG • UAP-SHD • UAP-HD • UAP-NanoHD • UAP-AC-PRO 2x • UAP-AC-LITE • UAP-AC-IW • UAP-AC-M
UCK-G2-PLUS • UCK-G2 • UCK
New Member
Posts: 7
Registered: ‎12-20-2018

Re: UniFi UAP AC Pro Rebooting (Kernel Panics)

[ Edited ]

Even happening on 4.0.21?

 - Honestly, it's hard to tell, and may be too early. I upgraded to 4.0.21 around 12:00 today and had a reboot (but no kernel panic) around 2:30-2:40. This really frustrated me, and that reboot is what finally made me put this info together since I felt the problem still existed.

 

Could you give us some details about your network?

What features are you running?

 - Let me know if you need more than this

  • EdgeRouter Lite (firmware v1.10.8)
  • Controller: Server same network (Debian)
  • 2 Wireless Networks (separate VLANs), 1 Guest Network
  • 2 UAP AC Pros (one connected with Wireless Uplink)
  • Base AP
    • 2G Enabled; HT20; Channel 11; Low Power
    • 5G Enabled; VHT80; Channel 157; Med Power
    • Allow Meshing Enabled (1 Wireless Uplink); Band Steering Prefer 5G; Airtime Fairness Disabled
    • Connected to and powered by UniFi Switch 8 POE-60W
  • Wireless Linked AP
    • 2G Disabled
    • 5G Enabled; VHT80; Channel 157; Med Power
    • Allow Meshing Enabled; Band Steering Prefer 5G; Airtime Fairness Disabled
    • Connected to Base AP via Wireless Uplink, Powered by POE injector

 

Client count?

 - 10-15 max (mix of 2.4 and 5G devices)

New Member
Posts: 7
Registered: ‎12-20-2018

Re: UniFi UAP AC Pro Rebooting (Kernel Panics)

Even happening on 4.0.21?

 - Looks like it still is. There was nothing in the log for close to 20 minutes, then I saw a reboot, and pulled this. Again, this is just the one AP. This AP has 10 clients currently connected to it, the other only has 1.

 

See attached log.

Ubiquiti Employee
Posts: 1,348
Registered: ‎12-16-2015
Kudos: 300
Solutions: 10

Re: UniFi UAP AC Pro Rebooting (Kernel Panics)

@icedtrip  even though you want to stay on the latest FW (which I support fully), can you downgrade to 3.9.54 as a diagnostic step?  If you can isolated it to a specific FW version then we can get to work on figuring out what is causing the crashes.

 

Also, do you have log's of the kernel panic?

If you'd like to try our latest features, consider signing up for beta: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Sign-Up-for-Beta-Access

If you're trying to debug a network issue, take a look at this help article: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
New Member
Posts: 7
Registered: ‎12-20-2018

Re: UniFi UAP AC Pro Rebooting (Kernel Panics)

[ Edited ]

I just tried downgrading, and it's failing each time. I've tried through the controller and I tried to SSH and downgrade on the AP itself, and both methods are failing. Thoughts?

 

As for the kernel panics, I posted a log in my first post. The timestamps in the post are for the kernel panics in the log.

 

EDIT: User error on my end. Downgraded fine. I'll do some testing to see if I can trigger a reboot.

New Member
Posts: 7
Registered: ‎12-20-2018

Re: UniFi UAP AC Pro Rebooting (Kernel Panics)

@UBNT-BenBuckley

 

I downgraded to 3.9.54.9373 and ran some tests that would normally cause a kernel panic with the more recent firmware. It held up fine with minimal logging. See attached log "NO_KernelPanic_unifi_syslog_3.9.54.9373.txt".

 

I then upgraded to 4.0.9.9636 and ran the same tests, and although it took longer than I expected, I did get a kernel panic and a reboot. It was also immediatly apparent that a lot more was getting logged. See attached log "YES_KernelPanic_unifi_syslog_4.0.9.9636.txt".

 

Let me know if I can run any more tests. For now, I've downgraded back to 3.9.54.9373.

New Member
Posts: 8
Registered: ‎07-31-2018
Kudos: 4
Solutions: 1

Re: UniFi UAP AC Pro Rebooting (Kernel Panics)

@icedtrip 

We have reboots (kernel panics) on all kinds of APs (Pro Gen2, Mesh, Mesh-Pro, Lite, nanoHD) and every firmware version, distributed over about 180 APs. We try to investigate the problem for some months now. I recently opened a ticket. But until today we were not able to _trigger_ a kernel panic. So I am very interested in your test environment. Do you have a script or something?

My feeling is, that mainly those APs restart with lots of short connects and disconnects. So not exactly a load problem, because all the clients just pass the APs by walking to their course rooms. No massive data transfer at all. No CPU or RAM bootleneck.

New Member
Posts: 7
Registered: ‎12-20-2018

Re: UniFi UAP AC Pro Rebooting (Kernel Panics)

@litauer

I have actually seen my APs reboot under a couple circumstances. On firmwares >= 4.0.9, I tend to be able to trigger one of the ways. I'll get to that in a minute. The other way I have seen a reboot occur is similar to what you are describing, with a possible connect / disconnect. I have seen a reboot occur just as a new device hops online (could be a phone, could be a game device, or a raspberry pi). I can't reproduce this as it appears to be random. At those times, the AP would have no more than 10 devices connected to it.

As for what I can do to trigger it, it's not a script, but the idea could be scriptable. This is my home environment, so we're not talking the scale you're working with. I first noticed that I could trigger a reboot when I was FTP'ing a lot of files to a raspberry pi connected to my network (2.4 GHz) with this usb wifi adaptor ( https://www.amazon.com/Edimax-EW-7811Un-150Mbps-Raspberry-Supports/dp/B003MTTJOY ). The adaptor may have nothing to do with it. I have not gotten around to testing anything like this on the 5 GHz side just yet.

As for what is being transferred, see below. This is why I lean towards some king of load issue; however, CPU and Memory load on the AP don't seem to spike. On firmware 4.0.9 or above, I have been able to trigger a reboot just by initiating this transfer and sometime before it completes, I'll get a kernel panic. Last night was an exception since it took several transfers for it to happen (about 1-2 hours vs within the first 30 minutes). This was on 4.0.9 which I had really not tested until then. On 3.9.54, I have run the same for 4+ hours without issue. I'm going to stay on this version for now to see if I ever get a reboot, but I would really like to get this resolved.

FTP Client: Cyberduck; Allowing multiple connections
Total Size: 7.7GB
Number of Folder Levels: No more than 5-6
Number of Files: 5,600
File Sizes: 50 kB - 20 MB (average 1.3-1.5MB)

Transfer takes 30-45 minutes, in many cases causing a reboot prior to completion.

Highlighted
New Member
Posts: 7
Registered: ‎12-20-2018

Re: UniFi UAP AC Pro Rebooting (Kernel Panics)

@UBNT-BenBuckley

 

I just wanted to check in to see I needed to post anything else. So far I have had no issues on 3.9.54, but I'd really like to resolve this one the later firmwares. Thanks.

Ubiquiti Employee
Posts: 1,348
Registered: ‎12-16-2015
Kudos: 300
Solutions: 10

Re: UniFi UAP AC Pro Rebooting (Kernel Panics)

@icedtrip  thank you for the logs and for following up.  I don't need anything else at the moment, I need to dig into your logs a little bit. 

If you'd like to try our latest features, consider signing up for beta: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Sign-Up-for-Beta-Access

If you're trying to debug a network issue, take a look at this help article: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
Reply