Upcoming Maintenance Alert:

The UBNT Community will be upgraded at 5pm MDT on April 25th. During this time the community forums will be set to read-only status.

Learn more

×
Reply
Highlighted
New Member
Posts: 19
Registered: ‎04-06-2014
Kudos: 1

Java High CPU usage and will cause https to stop responding

Hi,

 

Anyone encounter https stop responding and top showing java at 700% with 95% CPU usage.

After like 1 day the https will not response and i need to reboot the server for the system to resume as normal.

 

Here is the screenshot when the system is running without issue.

 

top - 20:12:19 up 21:21, 1 user, load average: 2.47, 2.99, 3.21
Tasks: 168 total, 1 running, 167 sleeping, 0 stopped, 0 zombie
%Cpu(s): 29.0 us, 0.4 sy, 0.0 ni, 69.6 id, 0.0 wa, 0.0 hi, 1.0 si, 0.0 st
KiB Mem: 16432336 total, 9793220 used, 6639116 free, 141856 buffers
KiB Swap: 0 total, 0 used, 0 free. 7830692 cached Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1318 root 20 0 6188080 1.351g 22416 S 227.8 8.6 3340:50 java
1343 root 20 0 64.764g 1.550g 1.493g S 9.9 9.9 155:59.05 mongod
1 root 20 0 33636 4196 2740 S 0.0 0.0 0:01.58 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:01.34 ksoftirqd/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:+
7 root 20 0 0 0 0 S 0.0 0.0 1:43.22 rcu_sched
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root rt 0 0 0 0 S 0.0 0.0 0:00.42 migration/0
10 root rt 0 0 0 0 S 0.0 0.0 0:00.17 watchdog/0
11 root rt 0 0 0 0 S 0.0 0.0 0:00.16 watchdog/1
12 root rt 0 0 0 0 S 0.0 0.0 0:00.43 migration/1
13 root 20 0 0 0 0 S 0.0 0.0 0:01.43 ksoftirqd/1
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0
15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:+
16 root rt 0 0 0 0 S 0.0 0.0 0:00.16 watchdog/2

SuperUser
Posts: 14,492
Registered: ‎06-18-2010
Kudos: 4211
Solutions: 1501

Re: Java High CPU usage and will cause https to stop responding

Yes I've experienced that before running Alpha/Beta code.

On the current version I'm using (5.6.3) I don't see this anymore.
Occam was Right; more than two blades is just silly
New Member
Posts: 19
Registered: ‎04-06-2014
Kudos: 1

Re: Java High CPU usage and will cause https to stop responding

Im running 5.4.14 now.

SuperUser
Posts: 14,492
Registered: ‎06-18-2010
Kudos: 4211
Solutions: 1501

Re: Java High CPU usage and will cause https to stop responding

I'd contact UBNT support via the chat. That's a released Unifi version; the out-of-control Java process shouldn't be happening.
Occam was Right; more than two blades is just silly
New Member
Posts: 19
Registered: ‎04-06-2014
Kudos: 1

Re: Java High CPU usage and will cause https to stop responding

ya..it happen to be on prevous 5.4.x version than i upgrade to 5.4.11.

But it still happening.

Im hoping someone able to share some fix for this.

New Member
Posts: 4
Registered: ‎09-08-2016

Re: Java High CPU usage and will cause https to stop responding

To be honest I also have had all sorts of odd Java and DB problems with the controller. Even a clean reinstall wasn't...  I threw in the towel and bought a cloudkey. My thoughts that with a  dedicated CPU with both the OS and Unifi code by UBNT things would run smoother and they are inexpensive.    Sure, I shouldn't have to, but at the end of the day it was a much cheaper solution vs my time.

 

If you go this route, remember there are a few points to note about powering the cloudkey for it to be stable.

I have also heard of a few corruption problems if it loses power, but only rumors online and with unknown f/w versions.  (be sure to keep a current conf backup on your PC, of course)

 

-W

New Member
Posts: 19
Registered: ‎04-06-2014
Kudos: 1

Re: Java High CPU usage and will cause https to stop responding

i have alot of APs more than 300 connected.

Can cloudkey do it?

Established Member
Posts: 956
Registered: ‎11-12-2015
Kudos: 203
Solutions: 20

Re: Java High CPU usage and will cause https to stop responding

@nicholas_saw, did you give java enough memory? Maybe all the time is being spent in garbage collection.

 

New Member
Posts: 19
Registered: ‎04-06-2014
Kudos: 1

Re: Java High CPU usage and will cause https to stop responding

Its running at 16GB.

I will check the memory usage the next time it happens.

It happens yesterday but my colleague just rebooted the server.

New Member
Posts: 8
Registered: ‎07-05-2016
Kudos: 4

Re: Java High CPU usage and will cause https to stop responding

@nicholas_saw are you still having these issues? I am having the same problem ever since we upgraded the controller to 5.14.4. java process is 100% CPU all the time and the controller stops responding. Have to restart the Unifi service numerous times a day. Our controller has ~500 site with total of ~1000 UAPs

New Member
Posts: 8
Registered: ‎07-05-2016
Kudos: 4

Re: Java High CPU usage and will cause https to stop responding

[ Edited ]

Changed this line JVM_MAX_HEAP_SIZE=1024M to JVM_MAX_HEAP_SIZE=2048M on  /usr/lib/unifi/bin/unifi.init. But that did not help either.

New Member
Posts: 19
Registered: ‎04-06-2014
Kudos: 1

Re: Java High CPU usage and will cause https to stop responding

@abin Im still having tjis problem.
Planning to move to centos to try out whether the problem persist.
New Member
Posts: 8
Registered: ‎07-05-2016
Kudos: 4

Re: Java High CPU usage and will cause https to stop responding

Same here. Let me know how CentOS works. Thanks

New Member
Posts: 20
Registered: ‎03-10-2017

Re: Java High CPU usage and will cause https to stop responding

I have also noticed this issue on our UniFi installation - java consistently running between 50% and 100% CPU. We just recently updated from 5.3.8 to 5.4.16 and the issues only started after that upgrade. We noticed this beacuse we run UniFi on an AWS server and it's running out of CPU credits due to the increased CPU.

 

The controller software is running on Ubuntu Server 16.04.2 LTS following the UBNT guide for UniFi installations on Ubuntu. I have also followed the document to increase the unifi.xms and unifi.xmx variables in the system.properties file from 1024Mb to 2048Mb of RAM and restarted UniFi. The problem still remains.

 

At this point it is going to either cost us downtime as we wait for AWS credits to reaccumulate after we run out, or it will cost us additional capital since we'll need to invest in a higher tier AWS instance. It should be noted that we run a guest portal on the controller, so downtime is a real concern.

 

Is there a fix (temporary or permanent) in the works from UBNT to address this?

Established Member
Posts: 956
Registered: ‎11-12-2015
Kudos: 203
Solutions: 20

Re: Java High CPU usage and will cause https to stop responding


bdg_robert wrote:

I have also noticed this issue on our UniFi installation - java consistently running between 50% and 100% CPU. We just recently updated from 5.3.8 to 5.4.16 and the issues only started after that upgrade. We noticed this beacuse we run UniFi on an AWS server and it's running out of CPU credits due to the increased CPU.


Have you checked memory utilization? Perhaps you are running low on memory and running up the CPU doing frequent garbage collection.

 

New Member
Posts: 20
Registered: ‎03-10-2017

Re: Java High CPU usage and will cause https to stop responding

Yeah, its' at 2.7GB/15GB RAM. As I mentioned, I increased the Java memory from 1GB to 2GB. UniFi is the only software running on the server.
Member
Posts: 173
Registered: ‎09-04-2015
Kudos: 45
Solutions: 4

Re: Java High CPU usage and will cause https to stop responding

[ Edited ]

I would like to share our experience with this problem for those who are running a UniFi controller in the cloud (or on a VPS) and a custom captive portal on the same server which interact through API calls (by means of PHP, python, etcetera...). Even with proper security measurements (fail2ban, verification if the AP MAC exists, verify if the client MAC exists in the controller, etcetera...), there's still the possibility that a guest triggers the login page too many times.

 

There are a couple of ways to solve this problem:

- Keeping track of the amount of login attempts in a custom DB like MySQL and issueing the block_sta through the API, which will allow you to unblock afterwards if a human was hitting the refresh button insanely fast

- Creating a custom fail2ban rule which blocks the IP after X amounts of tries. The problem here is that you can't just unblock the client through the unifi controller interface and (most importantly) the IP gets blocked, which will result in all the other clients on that location (and the AP itself) can't connect to the UniFi controller anymore.

- Creating a custom fail2ban rule which issues a block_sta statement through the API after X amounts of tries, based on the MAC of the client instead of the IP. The problem here is that you have to create a custom rule to accomplish this.

We chose the third option because it met our three most important requirements:

- should be able to "undo" a block through the UniFi controller software, which rules out fail2ban just blocking the IP address

- should be able to only block the responsible MAC address, because our AP's are on different IP addresses and connect to the (cloud based) controller through the internet. If we would block the IP address, we lose communication with the AP itself, which is definitely not what we want.
- we would like to prevent integrating a third party software package (like MySQL) to keep track of X number of tries. Also PHP (or Python or whatever server-side code is responsible for the captive portal verification) would still be generating CPU calculations and memory exhaustion while checking. Fail2ban checks requests and can act before the next request hits the actual services.

This problem presents itself when a client has software running that is infected with malware and/or is part of a botnet with frequent outgoing connections. The problem is that the human agreed to the terms and got 3 hours (in our case) of online time. The malware is happy because it can reach the central server. After 3 hours the connection to the internet is blocked and the login page is displayed again. If the human is absent by than, the malware will just keep trying to reach the botnet server X amounts per second. The requests will hit the Apache and serve the login page again and again until the UniFi controller CPU hits 100% and (depending if there is a Varnish running and on the amount of RAM available) the server could go out of memory in the end.

New Member
Posts: 8
Registered: ‎09-10-2014

Re: Java High CPU usage and will cause https to stop responding

any update on this ? having the same problem with Java CPU usage 

New Member
Posts: 8
Registered: ‎09-10-2014

Re: Java High CPU usage and will cause https to stop responding

Hey, did changing the Java memory from 1GB to 2GB make any difference ?
New Member
Posts: 20
Registered: ‎03-10-2017

Re: Java High CPU usage and will cause https to stop responding

Adjusting Java memory did not alleviate any of the CPU issues I've been tracking.

I'm continuing to work a case with UBNT now and the last I heard from them was a request that I upgrade UniFi to 5.6.11 (currently in testing) to see if this resolves the issue.
Reply