Upcoming Maintenance Alert:

The UBNT Community will be upgraded at 5pm MDT on April 25th. During this time the community forums will be set to read-only status.

Learn more

×
Reply
New Member
Posts: 29
Registered: ‎08-02-2016
Accepted Solution

AWS constantly crashing

Hi All,

 

i have an issue with my unifi controller, i'm hosting it on a ec2 t2.small. Most of the time, it's perfectly stable / fast.

From week to week it's CPU average is extreemly high wich causes un reachability. After a while it's back to normal, or after a reboot it's fine.

 

i have approx 100 ap's over 30 sites. Running on 5.5.20

 

greetings


Accepted Solutions
Senior Member
Posts: 3,200
Registered: ‎06-13-2015
Kudos: 851
Solutions: 154

Re: AWS constantly crashing

@walliesss have look at these pages:

http://www.webupd8.org/2012/09/install-oracle-java-8-in-ubuntu-via-ppa.html

 

Best is to stop the controller before the java update and start it again afterward.

 

Your unifi init/systemd script should pick up the updated java package, if not (you can check by looking at the output of "ps -ef | grep unifi") have a read here for instructions how to force it to pick up Oracle java:

http://community.ubnt.com/t5/UniFi-Updates-Blog/UniFi-5-2-9-is-released/ba-p/1694199

Art of WiFi
Check out our UniFi API browser tool on GitHub. The PHP API client which it uses, can be found here on GitHub.

View solution in original post


All Replies
Senior Member
Posts: 3,200
Registered: ‎06-13-2015
Kudos: 851
Solutions: 154

Re: AWS constantly crashing


walliesss wrote:

Hi All,

 

i have an issue with my unifi controller, i'm hosting it on a ec2 t2.small. Most of the time, it's perfectly stable / fast.

From week to week it's CPU average is extreemly high wich causes un reachability. After a while it's back to normal, or after a reboot it's fine.

 

i have approx 100 ap's over 30 sites. Running on 5.5.20

 

greetings


@walliesss did you change the memory settings for java? Also, which OS exactly are you running, which java version and are there any clues in the server.log file?

Art of WiFi
Check out our UniFi API browser tool on GitHub. The PHP API client which it uses, can be found here on GitHub.
New Member
Posts: 29
Registered: ‎08-02-2016

Re: AWS constantly crashing

Hi,

 

i did not make changes to java, should i?

 

i'm running on Ubuntu 16.04.3 LTS, java:

openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

 

can't see any special for now in the server.log, but i just rebooted it to gain acces again.

 

 

Member
Posts: 132
Registered: ‎12-09-2014
Kudos: 20
Solutions: 10

Re: AWS constantly crashing

[ Edited ]
Senior Member
Posts: 3,200
Registered: ‎06-13-2015
Kudos: 851
Solutions: 154

Re: AWS constantly crashing

[ Edited ]

@jperham That java issue specifically applies to 5.6.X, but it surely cannot hurt to update to the Oracle java version.

 

@walliesss Another bit of info to look at is how full your /run mount point is. Simply run this command from the shell on your server:

df -h

Sometimes this fills up completely which can cause issues.

Art of WiFi
Check out our UniFi API browser tool on GitHub. The PHP API client which it uses, can be found here on GitHub.
New Member
Posts: 29
Registered: ‎08-02-2016

Re: AWS constantly crashing

i updated to latest version 5.6.20, not sure it's memory related. I already had the problem before 5.20.

I have enough disk space, only 30% in use right now..

 

For now i upgraded to t2.medium, need to keep it going, hopefully i can find an other solution..

 

Ideas are welcome!

Senior Member
Posts: 3,200
Registered: ‎06-13-2015
Kudos: 851
Solutions: 154

Re: AWS constantly crashing


walliesss wrote:

i updated to latest version 5.6.20, not sure it's memory related. I already had the problem before 5.20.

I have enough disk space, only 30% in use right now..

 

For now i upgraded to t2.medium, need to keep it going, hopefully i can find an other solution..

 

Ideas are welcome!


As I said, please check how full /run is and share the output of the df -h command. /run is a temp mount point which uses tmpfs and normally has a default size of 25% or RAM. In some cases it can get 100% full, causing issues.

 

If you want help, please help back by responding to our requests...Icon Confused

Art of WiFi
Check out our UniFi API browser tool on GitHub. The PHP API client which it uses, can be found here on GitHub.
New Member
Posts: 29
Registered: ‎08-02-2016

Re: AWS constantly crashing

Sorry, i misunderstood:

 

 

Filesystem      Size  Used Avail Use% Mounted on
udev            2.0G     0  2.0G   0% /dev
tmpfs           395M  5.5M  390M   2% /run
/dev/xvda1       99G   37G   58G  39% /
tmpfs           2.0G   16K  2.0G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
tmpfs           395M     0  395M   0% /run/user/1000

 

Highlighted
Senior Member
Posts: 3,200
Registered: ‎06-13-2015
Kudos: 851
Solutions: 154

Re: AWS constantly crashing

No problem. Next time you notice issues, before rebooting, check the output of this command again, just to be sure. /run should normally not be more than a few % full, like it is now.

Art of WiFi
Check out our UniFi API browser tool on GitHub. The PHP API client which it uses, can be found here on GitHub.
SuperUser
Posts: 5,759
Registered: ‎09-03-2013
Kudos: 1957
Solutions: 390

Re: AWS constantly crashing

I say drop OpenJDK. Use Oracle Java instead, I've had much better stability with it over the last two years. My production controller is rebooted once every 6 months or so, just to install updates, and that's it.

Redcon IT Solutions - Florianópolis/SC/Brazil -www.redcon.com.br
To my mind it appears that the difficulty is only imaginary and not real. Rigidity and plasticity are not absolute terms but relative, and all solids are in fact both rigid and plastic. The apparent contrast between the two properties belongs to the laboratory and to those phenomena of nature involving small masses and small forces. When great masses and great forces are involved, as for example in the making of continents and mountain chains, the distinction loses value. - Grove Karl Gilbert
Senior Member
Posts: 3,200
Registered: ‎06-13-2015
Kudos: 851
Solutions: 154

Re: AWS constantly crashing

@R4V3R I couldn't agree more! Rock solid is my experience.

Art of WiFi
Check out our UniFi API browser tool on GitHub. The PHP API client which it uses, can be found here on GitHub.
New Member
Posts: 29
Registered: ‎08-02-2016

Re: AWS constantly crashing

will try that, do you have a suggestion how i can do this?

I made a snapsho of my volume, so i can try it all Man Wink

Senior Member
Posts: 3,200
Registered: ‎06-13-2015
Kudos: 851
Solutions: 154

Re: AWS constantly crashing

@walliesss have look at these pages:

http://www.webupd8.org/2012/09/install-oracle-java-8-in-ubuntu-via-ppa.html

 

Best is to stop the controller before the java update and start it again afterward.

 

Your unifi init/systemd script should pick up the updated java package, if not (you can check by looking at the output of "ps -ef | grep unifi") have a read here for instructions how to force it to pick up Oracle java:

http://community.ubnt.com/t5/UniFi-Updates-Blog/UniFi-5-2-9-is-released/ba-p/1694199

Art of WiFi
Check out our UniFi API browser tool on GitHub. The PHP API client which it uses, can be found here on GitHub.
New Member
Posts: 29
Registered: ‎08-02-2016

Re: AWS constantly crashing

Hi Sloofmaster,

 

looks like this works! my output of ps -ef

 

will try this a couple of weeks and will try to go back to a smaller t2.small instead of medium.

 

Thanks a lot!

 

root      5249     1  0 06:25 ?        00:00:00 unifi -cwd /usr/lib/unifi -home                                                                                                                                                              /usr/lib/jvm/java-8-oracle -cp /usr/share/java/commons-daemon.jar:/usr/lib/unif                                                                                                                                                              /lib/ace.jar -pidfile /var/run/unifi/unifi.pid -procname unifi -outfile SYSLOG -                                                                                                                                                             errfile SYSLOG -user unifi -Dunifi.datadir=/var/lib/unifi -Dunifi.logdir=/var/lo                                                                                                                                                             g/unifi -Dunifi.rundir=/var/run/unifi -Xmx1024M -Djava.awt.headless=true -Dfile.                                                                                                                                                             encoding=UTF-8 com.ubnt.ace.Launcher start
root      5251  5249  0 06:25 ?        00:00:00 unifi -cwd /usr/lib/unifi -home                                                                                                                                                              /usr/lib/jvm/java-8-oracle -cp /usr/share/java/commons-daemon.jar:/usr/lib/unif                                                                                                                                                              /lib/ace.jar -pidfile /var/run/unifi/unifi.pid -procname unifi -outfile SYSLOG -                                                                                                                                                             errfile SYSLOG -user unifi -Dunifi.datadir=/var/lib/unifi -Dunifi.logdir=/var/lo                                                                                                                                                             g/unifi -Dunifi.rundir=/var/run/unifi -Xmx1024M -Djava.awt.headless=true -Dfile.                                                                                                                                                             encoding=UTF-8 com.ubnt.ace.Launcher start
unifi     5252  5249  0 06:25 ?        00:00:01 unifi -cwd /usr/lib/unifi -home                                                                                                                                                              /usr/lib/jvm/java-8-oracle -cp /usr/share/java/commons-daemon.jar:/usr/lib/unif                                                                                                                                                              /lib/ace.jar -pidfile /var/run/unifi/unifi.pid -procname unifi -outfile SYSLOG -                                                                                                                                                             errfile SYSLOG -user unifi -Dunifi.datadir=/var/lib/unifi -Dunifi.logdir=/var/lo                                                                                                                                                             g/unifi -Dunifi.rundir=/var/run/unifi -Xmx1024M -Djava.awt.headless=true -Dfile.                                                                                                                                                             encoding=UTF-8 com.ubnt.ace.Launcher start
unifi     5265  5252 23 06:25 ?        00:04:17 /usr/lib/jvm/java-8-oracle/jre/b                                                                                                                                                             in/java -Xms1024M -Xmx1024M -XX:ErrorFile=/usr/lib/unifi/logs/hs_err_pid%p.log -                                                                                                                                                             Dapple.awt.UIElement=true -jar /usr/lib/unifi/lib/ace.jar start
unifi     6098  5265  4 06:25 ?        00:00:47 bin/mongod --dbpath /usr/lib/uni                                                                                                                                                             fi/data/db --port 27117 --unixSocketPrefix /usr/lib/unifi/run --logappend --logp                                                                                                                                                             ath /usr/lib/unifi/logs/mongod.log --nohttpinterface --bind_ip 127.0.0.1
ubuntu    6298  6283  0 06:44 pts/0    00:00:00 grep --color=auto unifi

o unifi

Senior Member
Posts: 3,200
Registered: ‎06-13-2015
Kudos: 851
Solutions: 154

Re: AWS constantly crashing

You're welcome!

Art of WiFi
Check out our UniFi API browser tool on GitHub. The PHP API client which it uses, can be found here on GitHub.
Reply