UniFi UAP AC Pro Rebooting (Kernel Panics) [ Edited ]

[icedtrip]

I have 2 UAP AC Pros (Gen 2) that have been rebooting on their own for several months now. I'm finally getting around to tracking down the reason. As far as I can remember, this started with the 4.x.x firmwares (but maybe 3.9.54), and so far has had the same issues with all versions since and up to the latest version 4.0.21.9965. I have not yet tried downgrading beyond 3.9.54 as I would like to try to figure out the problem and stay on the later firmwares.

I have searched for simliar issues, but I have not found one that is really doing what my APs are, or one that was resolved. I think it could be a load issue. At one point I could trigger a kernel panic just by FTPing thousands of smallish files from one computer to another over wifi.

Versions

Controller: 5.9.29.0

Current AP Firmware: 4.0.21.9965

Max Connected Devices: 10-15

Second AP is a Wireless Uplink

I set up syslog and turned on debug a couple of weeks ago to start capturing the issues. See attached for my syslog capture since 2/1/19. This is just for one of the APs, but there were several instances of kernel panics. It is connected to, and powered by, a UniFi Switch 8 POE-60W.

You will see 11 instances of kernel panics at the below timestamps:

Firmware version 4.0.15.9872

2019-02-07 05:24:22

2019-02-06 21:39:00

2019-02-06 21:38:56

2019-02-06 16:50:35
2019-02-06 05:24:28
2019-02-05 20:09:02
2019-02-05 05:26:35
2019-02-05 05:26:30
2019-02-05 05:25:44
2019-02-02 04:50:32
2019-02-02 04:44:27

I have had reboots on the latest firmware, but no kernel panics. Still looking into that. Somewhere around 2019-02-07 14:34:00 and 2019-02-07 14:40:00.

I trimmed the syslog since the below 3 items showed up on 2/5 and 2/6 with over 17 millions lines:

kernel: [88593.xxxxxx] BUG: Bad rss-counter state mm:844bf000 idx:0 val:522
kernel: [88593.xxxxxx] BUG: Bad rss-counter state mm:844bf000 idx:1 val:201
kernel: [88593.xxxxxx] BUG: non-zero nr_ptes on freeing mm: 4

Let me know if I need to post anything else, or go further back in the syslog. I started the syslog with firmware 4.0.15.9872.

unifi_syslog.txt

[AmazedMender16]

Hello @icedtrip,

Even happening on 4.0.21?

Could you give us some details about your network?

What features are you running?

Client count?

Regards,
Glenn R.

Regards,
Glenn R.

CC @UBNT-BenBuckley 

[icedtrip]

Even happening on 4.0.21?

 - Honestly, it's hard to tell, and may be too early. I upgraded to 4.0.21 around 12:00 today and had a reboot (but no kernel panic) around 2:30-2:40. This really frustrated me, and that reboot is what finally made me put this info together since I felt the problem still existed.

Could you give us some details about your network?

What features are you running?

 - Let me know if you need more than this

• EdgeRouter Lite (firmware v1.10.8)
• Controller: Server same network (Debian)
• 2 Wireless Networks (separate VLANs), 1 Guest Network
• 2 UAP AC Pros (one connected with Wireless Uplink)

• Base AP
  • 2G Enabled; HT20; Channel 11; Low Power
  • 5G Enabled; VHT80; Channel 157; Med Power
  • Allow Meshing Enabled (1 Wireless Uplink); Band Steering Prefer 5G; Airtime Fairness Disabled
  • Connected to and powered by UniFi Switch 8 POE-60W
• Wireless Linked AP
  • 2G Disabled
  • 5G Enabled; VHT80; Channel 157; Med Power
  • Allow Meshing Enabled; Band Steering Prefer 5G; Airtime Fairness Disabled
  • Connected to Base AP via Wireless Uplink, Powered by POE injector

Client count?

 - 10-15 max (mix of 2.4 and 5G devices)

[icedtrip]

Even happening on 4.0.21?

 - Looks like it still is. There was nothing in the log for close to 20 minutes, then I saw a reboot, and pulled this. Again, this is just the one AP. This AP has 10 clients currently connected to it, the other only has 1.

See attached log.

unifi_syslog_2.txt

[UBNT-BenBuckley]

@icedtrip  even though you want to stay on the latest FW (which I support fully), can you downgrade to 3.9.54 as a diagnostic step?  If you can isolated it to a specific FW version then we can get to work on figuring out what is causing the crashes.

Also, do you have log's of the kernel panic?

[icedtrip]

I just tried downgrading, and it's failing each time. I've tried through the controller and I tried to SSH and downgrade on the AP itself, and both methods are failing. Thoughts?

As for the kernel panics, I posted a log in my first post. The timestamps in the post are for the kernel panics in the log.

EDIT: User error on my end. Downgraded fine. I'll do some testing to see if I can trigger a reboot.

[icedtrip]

@UBNT-BenBuckley

I downgraded to 3.9.54.9373 and ran some tests that would normally cause a kernel panic with the more recent firmware. It held up fine with minimal logging. See attached log "NO_KernelPanic_unifi_syslog_3.9.54.9373.txt".

I then upgraded to 4.0.9.9636 and ran the same tests, and although it took longer than I expected, I did get a kernel panic and a reboot. It was also immediatly apparent that a lot more was getting logged. See attached log "YES_KernelPanic_unifi_syslog_4.0.9.9636.txt".

Let me know if I can run any more tests. For now, I've downgraded back to 3.9.54.9373.

NO_KernelPanic_unifi_syslog_3.9.54.9373.txt

YES_KernelPanic_unifi_syslog_4.0.9.9636.txt

[litauer]

@icedtrip 

We have reboots (kernel panics) on all kinds of APs (Pro Gen2, Mesh, Mesh-Pro, Lite, nanoHD) and every firmware version, distributed over about 180 APs. We try to investigate the problem for some months now. I recently opened a ticket. But until today we were not able to _trigger_ a kernel panic. So I am very interested in your test environment. Do you have a script or something?

My feeling is, that mainly those APs restart with lots of short connects and disconnects. So not exactly a load problem, because all the clients just pass the APs by walking to their course rooms. No massive data transfer at all. No CPU or RAM bootleneck.

[icedtrip]

@litauer

I have actually seen my APs reboot under a couple circumstances. On firmwares >= 4.0.9, I tend to be able to trigger one of the ways. I'll get to that in a minute. The other way I have seen a reboot occur is similar to what you are describing, with a possible connect / disconnect. I have seen a reboot occur just as a new device hops online (could be a phone, could be a game device, or a raspberry pi). I can't reproduce this as it appears to be random. At those times, the AP would have no more than 10 devices connected to it.

As for what I can do to trigger it, it's not a script, but the idea could be scriptable. This is my home environment, so we're not talking the scale you're working with. I first noticed that I could trigger a reboot when I was FTP'ing a lot of files to a raspberry pi connected to my network (2.4 GHz) with this usb wifi adaptor ( https://www.amazon.com/Edimax-EW-7811Un-150Mbps-Raspberry-Supports/dp/B003MTTJOY ). The adaptor may have nothing to do with it. I have not gotten around to testing anything like this on the 5 GHz side just yet.

As for what is being transferred, see below. This is why I lean towards some king of load issue; however, CPU and Memory load on the AP don't seem to spike. On firmware 4.0.9 or above, I have been able to trigger a reboot just by initiating this transfer and sometime before it completes, I'll get a kernel panic. Last night was an exception since it took several transfers for it to happen (about 1-2 hours vs within the first 30 minutes). This was on 4.0.9 which I had really not tested until then. On 3.9.54, I have run the same for 4+ hours without issue. I'm going to stay on this version for now to see if I ever get a reboot, but I would really like to get this resolved.

FTP Client: Cyberduck; Allowing multiple connections
Total Size: 7.7GB
Number of Folder Levels: No more than 5-6
Number of Files: 5,600
File Sizes: 50 kB - 20 MB (average 1.3-1.5MB)

Transfer takes 30-45 minutes, in many cases causing a reboot prior to completion.

[icedtrip]

@UBNT-BenBuckley

I just wanted to check in to see I needed to post anything else. So far I have had no issues on 3.9.54, but I'd really like to resolve this one the later firmwares. Thanks.

[UBNT-BenBuckley]

@icedtrip  thank you for the logs and for following up.  I don't need anything else at the moment, I need to dig into your logs a little bit.