Reply
Highlighted
Emerging Member
Posts: 61
Registered: ‎09-21-2014
Kudos: 2
Solutions: 2

Tried upgrading to 4.8.12 and since then no way to reach the unifi controler

Hi,

 

I upgraded yesterday to 4.8.12 and since then can not access the controler anymore... The browsers return "Web page not reachable - ERR_CONNECTION_REFUSED".

 

Any help on where to look the cause of the problem would be appreciated.

 

Note that I think i'm upgrading from 4.7.5...

 

Thanks,

 

PS : Logs & co 

-------------------------

The controler seems to be started :

pi@Raspberrypi ~ $ sudo service unifi status -l
● unifi.service - unifi
   Loaded: loaded (/etc/systemd/system/unifi.service; enabled)
   Active: active (running) since Sat 2016-02-06 19:33:06 CET; 20h ago
  Process: 429 ExecStart=/usr/lib/unifi/bin/unifi.init start (code=exited, status=0/SUCCESS)
 Main PID: 479 (jsvc)
   CGroup: /system.slice/unifi.service
           ├─ 479 unifi -home /usr/lib/jvm/jdk-8-oracle-arm-vfp-hflt -cp /usr/share/java/commons-daemon.jar:/usr/lib/unifi/lib/ace.jar -pidfile /va...
           ├─ 481 unifi -home /usr/lib/jvm/jdk-8-oracle-arm-vfp-hflt -cp /usr/share/java/commons-daemon.jar:/usr/lib/unifi/lib/ace.jar -pidfile /va...
           ├─ 482 unifi -home /usr/lib/jvm/jdk-8-oracle-arm-vfp-hflt -cp /usr/share/java/commons-daemon.jar:/usr/lib/unifi/lib/ace.jar -pidfile /va...
           └─2865 bin/mongod --dbpath /usr/lib/unifi/data/db --port 27117 --logappend --logpath logs/mongod.log --nohttpinterface --bind_ip 127.0.0...

Feb 06 19:33:06 Raspberrypi unifi.init[429]: Starting Ubiquiti UniFi Controller: unifi.
Feb 06 19:33:06 Raspberrypi systemd[1]: Started unifi.

Unifi log file says that :

 

pi@Raspberrypi ~ $ sudo tail /var/log/unifi/server.log
[2016-02-07 11:45:26,929] <localhost-startStop-1> ERROR WebappClassLoader - The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
[2016-02-07 14:00:01,123] <localhost-startStop-1> ERROR WebappClassLoader - The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
[2016-02-07 14:00:01,134] <localhost-startStop-1> ERROR WebappClassLoader - The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
[2016-02-07 14:17:38,350] <localhost-startStop-1> ERROR WebappClassLoader - The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
[2016-02-07 14:17:38,361] <localhost-startStop-1> ERROR WebappClassLoader - The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
[2016-02-07 14:50:07,616] <localhost-startStop-1> ERROR WebappClassLoader - The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
[2016-02-07 14:50:07,627] <localhost-startStop-1> ERROR WebappClassLoader - The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
[2016-02-07 15:03:41,120] <localhost-startStop-1> ERROR WebappClassLoader - The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
[2016-02-07 15:39:07,384] <localhost-startStop-1> ERROR WebappClassLoader - The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
[2016-02-07 15:55:24,489] <localhost-startStop-1> ERROR WebappClassLoader - The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.

Mongodb log :

pi@Raspberrypi ~ $ sudo tail /var/log/mongodb/mongodb.log
Sun Feb  7 16:15:34.943 [initandlisten] db version v2.4.10
Sun Feb  7 16:15:34.943 [initandlisten] git version: nogitversion
Sun Feb  7 16:15:34.944 [initandlisten] build info: Linux bm-wb-03 3.19.0-trunk-armmp #1 SMP Debian 3.19.1-1~exp1+plugwash1 (2015-03-28) armv7l BOOST_LIB_VERSION=1_55
Sun Feb  7 16:15:34.944 [initandlisten] allocator: system
Sun Feb  7 16:15:34.944 [initandlisten] options: { bind_ip: "127.0.0.1", config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", journal: "true", logappend: "true", logpath: "/var/log/mongodb/mongodb.log" }
Sun Feb  7 16:15:35.013 [initandlisten] journal dir=/var/lib/mongodb/journal
Sun Feb  7 16:15:35.017 [initandlisten] recover : no journal files present, no recovery needed
Sun Feb  7 16:15:35.697 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0  reslen:75 133ms
Sun Feb  7 16:15:35.758 [websvr] admin web console waiting for connections on port 28017
Sun Feb  7 16:15:35.759 [initandlisten] waiting for connections on port 27017
pi@Raspberrypi ~ $

 

Member
Posts: 150
Registered: ‎07-14-2015
Kudos: 12
Solutions: 3

Re: Tried upgrading to 4.8.12 and since then no way to reach the unifi controler

Ive posted on here too about a similar problem, mine seems to be caused my mongodb, Ive had to kill the mongo process, then restart unifi service to get it to come back up.

 

 

New Member
Posts: 1
Registered: ‎07-16-2015

Re: Tried upgrading to 4.8.12 and since then no way to reach the unifi controler

[ Edited ]

hi

 

i have same problem, check free space on host

New Member
Posts: 4
Registered: ‎07-05-2014

Re: Tried upgrading to 4.8.12 and since then no way to reach the unifi controler

 
New Member
Posts: 4
Registered: ‎07-05-2014

Re: Tried upgrading to 4.8.12 and since then no way to reach the unifi controler

Thank you for the answers!

@andrewglass3 : i have rebooted several times with no improvement. I think this should have done what you suggest or do you think i should still try this after a fresh boot?

@jskvorc1 : I have 360 MB out of 482 available (says pulseway). This should be OK?

Thanks,
New Member
Posts: 5
Registered: ‎02-18-2016

Re: Tried upgrading to 4.8.12 and since then no way to reach the unifi controler

Had 4.8.12 running just fine.  Associated UAP-AC-Lite with it.  It also worked fine.  Then upgraded AP to latest firmware and controller crashes continuously.

 

[2016-02-18 20:21:59,488] <discover> INFO  dev    -    [Try Adopt] dev[44:d9:e7:90:71:33], dev_ip=172.23.21.110, url=http://172.23.21.106:8080/inform
[2016-02-18 20:22:00,535] <ssh> INFO  event  - [event] AP[44:d9:e7:90:71:33] was automatically readopted
[2016-02-18 20:29:14,906] <31> ERROR WebappClassLoader - The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.

I can apply iptables rule and block port 8080.  Then controller runs just fine after a restart.

 

 

New Member
Posts: 5
Registered: ‎02-18-2016

Re: Tried upgrading to 4.8.12 and since then no way to reach the unifi controler

I downgraded the controller to 4.7.6 and the problem persisted.  Because this is just a test setup at this time it was relatively painless to facotry reset the AP and wipe the contoller configuration.  After that the AP no longer caused the controller to crash.  I then proceeded to downgrade the firmware on the AP to 3.4.8.3291.  This finally resolved the crashing.

 

So at this point I have controller 4.7.6 and AP firmware 3.4.8.3291 running without issues.  

 

BTW, this is the controller running on an old rPI.  I am now hesitant in trying this on our production cloud instance of the UniFi controller.

Ubiquiti Employee
Posts: 679
Registered: ‎03-31-2015
Kudos: 115
Solutions: 57

Re: Tried upgrading to 4.8.12 and since then no way to reach the unifi controler

Hi @adilinden,

 

Do you have any error like "Could not initialize class org.xerial.snappy.Snappy" in your server log? It's a common issue for the AC inform on RPi. If so, you might replace the /usr/lib/unifi/lib/snappy-java-1.0.5.jar with the latest from here (remember to use the same name).

New Member
Posts: 5
Registered: ‎02-18-2016

Re: Tried upgrading to 4.8.12 and since then no way to reach the unifi controler

Hi,

 

No, no such error rpresent in log.

 

pi@raspberrypi:~ $ grep "Could not initialize" /var/log/unifi/server.log 
pi@raspberrypi:~ $ grep org.xerial.snappy /var/log/unifi/server.log
pi@raspberrypi:~ $ 

Here is part of the log. All was well right until the the firmware upgrade on the AP had taken place.

 

[2016-02-18 18:27:50,191] <inform-67> INFO  inform - <<< [upgrade] dev[44:d9:e7:90:71:33] { "_type" : "upgrade" , "server_time_in_utc" : "1455841670186" , "url" : "http://172.23.21.106:8080/dl/firmware/U7PG2/3.4.14.3413/firmware.bin" , "version" : "3.4.14.3413"}
[2016-02-18 18:30:45,771] <launcher> WARN  system - cannot load native lib - ubnt_webrtc_jni
[2016-02-18 18:31:10,691] <discover> INFO  dev    -    [state] dev[44:d9:e7:90:71:33] UNKNOWN->ADOPTING, state_expire=1455841900
[2016-02-18 18:31:10,843] <discover> INFO  dev    -    [Try Adopt] dev[44:d9:e7:90:71:33], dev_ip=172.23.21.110, url=http://172.23.21.106:8080/inform
[2016-02-18 18:32:22,066] <launcher> WARN  system - cannot load native lib - ubnt_webrtc_jni
[2016-02-18 18:32:46,108] <discover> INFO  dev    -    [state] dev[44:d9:e7:90:71:33] UNKNOWN->ADOPTING, state_expire=1455841996
[2016-02-18 18:32:46,241] <discover> INFO  dev    -    [Try Adopt] dev[44:d9:e7:90:71:33], dev_ip=172.23.21.110, url=http://172.23.21.106:8080/inform
[2016-02-18 18:32:51,647] <ssh> INFO  event  - [event] AP[44:d9:e7:90:71:33] was automatically readopted
[2016-02-18 18:37:15,899] <launcher> WARN  system - cannot load native lib - ubnt_webrtc_jni
[2016-02-18 18:37:38,172] <discover> INFO  dev    -    [state] dev[44:d9:e7:90:71:33] UNKNOWN->ADOPTING, state_expire=1455842288
[2016-02-18 18:37:38,282] <discover> INFO  dev    -    [Try Adopt] dev[44:d9:e7:90:71:33], dev_ip=172.23.21.110, url=http://172.23.21.106:8080/inform
[2016-02-18 18:37:43,459] <ssh> INFO  event  - [event] AP[44:d9:e7:90:71:33] was automatically readopted
[2016-02-18 18:40:26,566] <launcher> WARN  system - cannot load native lib - ubnt_webrtc_jni
[2016-02-18 18:40:49,124] <discover> INFO  dev    -    [state] dev[44:d9:e7:90:71:33] UNKNOWN->ADOPTING, state_expire=1455842479
[2016-02-18 18:40:49,244] <discover> INFO  dev    -    [Try Adopt] dev[44:d9:e7:90:71:33], dev_ip=172.23.21.110, url=http://172.23.21.106:8080/inform
[2016-02-18 18:40:52,386] <ssh> INFO  event  - [event] AP[44:d9:e7:90:71:33] was automatically readopted
[2016-02-18 18:44:27,884] <localhost-startStop-2> ERROR WebappClassLoader - The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
[2016-02-18 18:48:42,585] <db-server> ERROR system - [exec] error, rc=100
[2016-02-18 18:48:42,626] <db-server> INFO  db     - DbServer stopped
[2016-02-18 18:48:56,969] <launcher> WARN  system - cannot load native lib - ubnt_webrtc_jni
[2016-02-18 18:49:39,225] <discover> INFO  dev    -    [state] dev[44:d9:e7:90:71:33] UNKNOWN->ADOPTING, state_expire=1455843009
[2016-02-18 18:49:39,346] <discover> INFO  dev    -    [Try Adopt] dev[44:d9:e7:90:71:33], dev_ip=172.23.21.110, url=http://172.23.21.106:8080/inform
[2016-02-18 18:49:44,685] <ssh> INFO  event  - [event] AP[44:d9:e7:90:71:33] was automatically readopted
[2016-02-18 18:51:00,156] <db-server> ERROR system - [exec] error, rc=100
[2016-02-18 18:51:00,194] <db-server> INFO  db     - DbServer stopped
[2016-02-18 18:51:06,034] <launcher> WARN  system - cannot load native lib - ubnt_webrtc_jni
[2016-02-18 18:51:48,796] <discover> INFO  dev    -    [state] dev[44:d9:e7:90:71:33] UNKNOWN->ADOPTING, state_expire=1455843138
[2016-02-18 18:51:53,669] <discover> INFO  dev    -    [Try Adopt] dev[44:d9:e7:90:71:33], dev_ip=172.23.21.110, url=http://172.23.21.106:8080/inform
[2016-02-18 18:53:17,959] <launcher> WARN  system - cannot load native lib - ubnt_webrtc_jni
[2016-02-18 18:53:57,368] <discover> INFO  dev    -    [state] dev[44:d9:e7:90:71:33] UNKNOWN->ADOPTING, state_expire=1455843267
[2016-02-18 18:53:57,809] <discover> INFO  dev    -    [Try Adopt] dev[44:d9:e7:90:71:33], dev_ip=172.23.21.110, url=http://172.23.21.106:8080/inform
[2016-02-18 18:55:58,520] <localhost-startStop-1> ERROR WebappClassLoader - The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
[2016-02-18 18:56:36,368] <launcher> WARN  system - cannot load native lib - ubnt_webrtc_jni
[2016-02-18 18:56:58,442] <discover> INFO  dev    -    [state] dev[44:d9:e7:90:71:33] UNKNOWN->ADOPTING, state_expire=1455843448
[2016-02-18 18:56:58,549] <discover> INFO  dev    -    [Try Adopt] dev[44:d9:e7:90:71:33], dev_ip=172.23.21.110, url=http://172.23.21.106:8080/inform
[2016-02-18 18:57:01,402] <ssh> INFO  event  - [event] AP[44:d9:e7:90:71:33] was automatically readopted
[2016-02-18 18:58:00,567] <localhost-startStop-1> ERROR WebappClassLoader - The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
[2016-02-18 19:00:00,386] <localhost-startStop-2> ERROR WebappClassLoader - The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
[2016-02-18 19:09:28,393] <launcher> WARN  system - cannot load native lib - ubnt_webrtc_jni
[2016-02-18 19:09:31,680] <discover> INFO  dev    -    [state] dev[44:d9:e7:90:71:33] UNKNOWN->ADOPTING, state_expire=1455844201
[2016-02-18 19:09:31,702] <discover> INFO  dev    -    [Try Adopt] dev[44:d9:e7:90:71:33], dev_ip=172.23.21.110, url=http://172.23.21.106:8080/inform
[2016-02-18 19:09:34,257] <ssh> INFO  event  - [event] AP[44:d9:e7:90:71:33] was automatically readopted
[2016-02-18 19:13:03,129] <localhost-startStop-1> ERROR WebappClassLoader - The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.

 

Note that per the install instructions I did remove this file.  With it present the controller would go into a coontinuous cycle of restarting.

 

    sudo rm /usr/lib/unifi/lib/native/Linux/armhf/libubnt_webrtc_jni.so

Thanks,
Adi

Ubiquiti Employee
Posts: 679
Registered: ‎03-31-2015
Kudos: 115
Solutions: 57

Re: Tried upgrading to 4.8.12 and since then no way to reach the unifi controler

Hi @adilinden,

 

Just curious, are you running Raspbian Jessie? If you're not averse to it, I would still try upgrading the snappy since this was known to cause issues with gen2 AC on RPi.

New Member
Posts: 5
Registered: ‎02-18-2016

Re: Tried upgrading to 4.8.12 and since then no way to reach the unifi controler

Hi,

 

Yes, the system is running Raspbian Jesse on an older rPI, not rPI2.

 

I performed the upgrade from 4.7.6 to 4.8.12.  Once updated I removed the file that prevented UniFi controller from starting.

 

sudo rm /usr/lib/unifi/lib/native/Linux/armhf/libubnt_webrtc_jni.so

Upon restart of the entire rPI the controller started successfully.  I was able to login and see the UniFi AP-AC-Lite still running the old firmware.

 

 

I upgraded the firmware and shortly therafter the controller ceased to function.  I stopped the controller and replaced the file per your suggestion.

 

sudo cp snappy-java-1.1.2.jar /usr/lib/unifi/lib/snappy-java-1.0.5.jar

 

Even with this file replaced the controller would not start back up to provide login.  This time I was not seeing any log entries in server.log at all of any kind.  The proper process were in the process list, but looking at the sockets open (using 'sudo lsof -i') I did not see port 8443, 8080, etc, indicating there was still an issue.

 

As a last resort I eventually restarted the entire rPI, instead of just the contoller.  That brought things back to life.  The log excerpt shows when AP upgrade was initiated at 00:20:47, controller 00:27:02 upon finding upgraded AP.  Although I perfromed the replacement of the file and restart of the controller, there were no log messages until a full reboot of the system at 00:36:14 at which time all returned to normal.

 

[2016-02-19 00:20:40,388] <webapi-52> INFO  dev    -    [state] dev[44:d9:e7:90:71:33] CONNECTED->UPGRADING, state_expire=1455862960
[2016-02-19 00:20:40,402] <webapi-52> INFO  event  - [event] AP[44:d9:e7:90:71:33] was scheduled for upgrade by Admin[adi]
[2016-02-19 00:20:47,102] <inform-25> INFO  inform - from [44:d9:e7:90:71:33](, U7LT, 3.4.8.3291): state=UPGRADING, ext/stun_ip=172.23.21.110, dev_ip=172.23.21.110, up=12183
[2016-02-19 00:20:47,141] <inform-25> INFO  inform - <<< [upgrade] dev[44:d9:e7:90:71:33] { "_type" : "upgrade" , "server_time_in_utc" : "1455862847137" , "url" : "http://172.23.21.106:8080/dl/firmware/U7PG2/3.4.14.3413/firmware.bin" , "version" : "3.4.14.3413"}
[2016-02-19 00:26:31,411] <launcher> WARN  system - cannot load native lib - ubnt_webrtc_jni
[2016-02-19 00:26:56,808] <discover> INFO  dev    -    [state] dev[44:d9:e7:90:71:33] UNKNOWN->ADOPTING, state_expire=1455863246
[2016-02-19 00:26:56,932] <discover> INFO  dev    -    [Try Adopt] dev[44:d9:e7:90:71:33], dev_ip=172.23.21.110, url=http://172.23.21.106:8080/inform
[2016-02-19 00:27:02,688] <ssh> INFO  event  - [event] AP[44:d9:e7:90:71:33] was automatically readopted
[2016-02-19 00:36:14,576] <launcher> WARN  system - cannot load native lib - ubnt_webrtc_jni
[2016-02-19 00:36:42,777] <discover> INFO  dev    -    [state] dev[44:d9:e7:90:71:33] UNKNOWN->ADOPTING, state_expire=1455863832
[2016-02-19 00:36:42,942] <discover> INFO  dev    -    [Try Adopt] dev[44:d9:e7:90:71:33], dev_ip=172.23.21.110, url=http://172.23.21.106:8080/inform
[2016-02-19 00:36:47,227] <inform-1> INFO  inform - from [44:d9:e7:90:71:33](, U7LT, 3.4.14.3413): state=ADOPTING, ext_ip=172.23.21.110, dev_ip=172.23.21.110, up=919
[2016-02-19 00:36:47,424] <inform-1> INFO  dev    - saveDeviceAttr(): device[44:d9:e7:90:71:33][ethernet_table]=[{ "mac" : "44:d9:e7:90:71:33" , "name" : "eth0" , "num_port" : 1}] (orig=[ { "mac" : "44:d9:e7:90:71:33" , "num_port" : 2 , "name" : "eth0"}])
[2016-02-19 00:36:47,497] <inform-1> INFO  dev    -    [state] dev[44:d9:e7:90:71:33] ADOPTING->CONNECTED, state_expire=0
[2016-02-19 00:36:47,612] <inform-1> INFO  event  - [event] AP[44:d9:e7:90:71:33] was connected

 

Thanks,
Adi

 

Ubiquiti Employee
Posts: 679
Registered: ‎03-31-2015
Kudos: 115
Solutions: 57

Re: Tried upgrading to 4.8.12 and since then no way to reach the unifi controler

Hi,

 

I'm glad to hear your controller and AP are functioning correctly! Please let us know if this issue starts again.

New Member
Posts: 12
Registered: ‎06-03-2015
Kudos: 3

Re: Tried upgrading to 4.8.12 and since then no way to reach the unifi controler

Hi @UBNT-Amber

 

I just wanted to chime in on this issue.  I also was running into this problem on a Raspberry Pi model B (512MB RAM) running Raspbian Jessie.  I upgraded from UniFi Controller version 4.7.6 to 4.8.14.  I deleted libubnt_webrtc_jni.so file, then restarted the rPi.

 

Just like the original poster and @adilinden, I was unable to load the login page for the UniFi Controller.

 

What solved the problem was upgrading the snappy-java file from 1.0.5 to 1.1.2 as you suggested, and then a restart of the rPi.

 

Just an FYI so maybe in the next hotfix or minor version release you guys can update the dependency for snappy-java to use the 1.1.2 version instead of 1.0.5.

 

Jason

SuperUser
Posts: 4,561
Registered: ‎01-10-2012
Kudos: 1998
Solutions: 224

Re: Tried upgrading to 4.8.12 and since then no way to reach the unifi controler


PhantomTypist wrote:

 

Just an FYI so maybe in the next hotfix or minor version release you guys can update the dependency for snappy-java to use the 1.1.2 version instead of 1.0.5.

 


They may, but I wouldn't bank on it - ARM/rasberri pi is not officially supported.

When you receive a solution to your question/issue, don't forget to mark your thread as solved and to give kudo's to the people who have helped you out!
Reply