Reply
Member
Posts: 257
Registered: ‎05-19-2015
Kudos: 445
Solutions: 2

Can no longer access UNIFI Controller on Cloud Key

Need a little help as I am scrtatching my head.  

 

Cloud Key is online.  I can SSH to it, log into the Key's UI, but all of a sudden the UNIFI side is no tango nada.  It's showing offline in the UBNT cloud as well as I cannot connect to it on port 8443.  Clearly the key is up and running since that side is working but the Unifi controller side all of a sudden is Dead.

 

@UBNT-MikeD or anyone else got any idea what to check?  Tried restarting it from SSH still no dice.  the service says it's running.  Sort of at a loss here it's like it's there but it's not there.

 

root@UniFi-CloudKey:~# service unifi status
* unifi.service - unifi
   Loaded: loaded (/lib/systemd/system/unifi.service; enabled)
   Active: active (running) since Wed 2016-09-07 14:40:06 EDT; 8min ago
  Process: 360 ExecStart=/usr/lib/unifi/bin/unifi.init start (code=exited, status=0/SUCCESS)
 Main PID: 410 (jsvc)
   CGroup: /system.slice/unifi.service
           |-410 unifi -home /usr/lib/jvm/java-7-openjdk-armhf -cp /usr/share/java/commons-daemon.jar:/usr/lib/unifi/lib/ace.jar -pidfile /var/run/...
           |-412 unifi -home /usr/lib/jvm/java-7-openjdk-armhf -cp /usr/share/java/commons-daemon.jar:/usr/lib/unifi/lib/ace.jar -pidfile /var/run/...
           |-413 unifi -home /usr/lib/jvm/java-7-openjdk-armhf -cp /usr/share/java/commons-daemon.jar:/usr/lib/unifi/lib/ace.jar -pidfile /var/run/...
           `-437 /usr/lib/jvm/java-7-openjdk-armhf/jre/bin/java -Xmx768M -XX:ErrorFile=/usr/lib/unifi/data/logs/hs_err_pid<pid>.log -Dapple.awt.UIE...
Member
Posts: 257
Registered: ‎05-19-2015
Kudos: 445
Solutions: 2

Re: Can no longer access UNIFI Controller on Cloud Key

strangly is appears there is no listener on 8443 all of a sudden...

 

root@UniFi-CloudKey:/# netstat -atn
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN     
tcp        0      0 192.168.0.100:443       192.168.100.144:55196   FIN_WAIT2  
tcp        0      0 192.168.0.100:22        192.168.100.144:54545   ESTABLISHED
tcp        0      0 192.168.0.100:443       192.168.100.144:55203   FIN_WAIT2  
tcp6       0      0 :::22                   :::*                    LISTEN 
Member
Posts: 257
Registered: ‎05-19-2015
Kudos: 445
Solutions: 2

Re: Can no longer access UNIFI Controller on Cloud Key

I updated this CK to 0.5.2 last week I have two others still on the previous running fine....so can the CK be rolled back?  I think there is some kind of bug still with this F/W

 

UCK System

  • Fix Java issue with an update which disabled JIT on arm32. This basically caused the controller to be inoperable on UniFi Cloud Key. We are investigating alternative options and should have more to add in the near future. 

I need to get into my switches though so....ideas?

Member
Posts: 257
Registered: ‎05-19-2015
Kudos: 445
Solutions: 2

Re: Can no longer access UNIFI Controller on Cloud Key

found the previous F/W and attempting to roll it back to at least give that a try.

Member
Posts: 257
Registered: ‎05-19-2015
Kudos: 445
Solutions: 2

Re: Can no longer access UNIFI Controller on Cloud Key

Appears the DB is not starting and I cannot figure out why....I just know there is DB errors in the server.log

 

 

Member
Posts: 278
Registered: ‎07-28-2016
Kudos: 47
Solutions: 13

Re: Can no longer access UNIFI Controller on Cloud Key

What errors do you see in server.log? Anything useful there?

 

-KJ

Member
Posts: 257
Registered: ‎05-19-2015
Kudos: 445
Solutions: 2

Re: Can no longer access UNIFI Controller on Cloud Key

[2016-09-07 16:07:40,402] <db-server> ERROR system - [exec] error, rc=100

 

just repeated....tried to run:  

root@UniFi-CloudKey:/usr/lib/unifi/data/db# mongod --dbpath /usr/lib/unifi/data/db --repair

 

but got to the end and shows :

 

Wed Sep  7 16:49:38.784 [initandlisten] warning database /usr/lib/unifi/data/db ace could not be opened

Wed Sep  7 16:49:38.784 [initandlisten] DBException 10446: mmap: can't map area of size 0 file: /usr/lib/unifi/data/db/ace.ns

Wed Sep  7 16:49:38.785 [initandlisten] exception in initAndListen: 10446 mmap: can't map area of size 0 file: /usr/lib/unifi/data/db/ace.ns, terminating

Wed Sep  7 16:49:38.785 dbexit:

 

clearly there is a DB issue.  I saw an error that the DB was not shut down correctly and needs to be repaired...

 

which is what I was trying to do above. 

Member
Posts: 257
Registered: ‎05-19-2015
Kudos: 445
Solutions: 2

Re: Can no longer access UNIFI Controller on Cloud Key

just trying to start it:

 

Wed Sep  7 16:56:19.435 [initandlisten] warning database /usr/lib/unifi/data/db ace could not be opened
Wed Sep  7 16:56:19.436 [initandlisten] DBException 10446: mmap: can't map area of size 0 file: /usr/lib/unifi/data/db/ace.ns
Wed Sep  7 16:56:19.436 [initandlisten] exception in initAndListen: 10446 mmap: can't map area of size 0 file: /usr/lib/unifi/data/db/ace.ns, terminating
Wed Sep  7 16:56:19.436 dbexit: 
Wed Sep  7 16:56:19.436 [initandlisten] shutdown: going to close listening sockets...
Wed Sep  7 16:56:19.437 [initandlisten] shutdown: going to flush diaglog...
Wed Sep  7 16:56:19.437 [initandlisten] shutdown: going to close sockets...
Wed Sep  7 16:56:19.437 [initandlisten] shutdown: waiting for fs preallocator...
Wed Sep  7 16:56:19.437 [initandlisten] shutdown: closing all files...
Wed Sep  7 16:56:19.437 [initandlisten] closeAllFiles() finished
Wed Sep  7 16:56:19.437 [initandlisten] shutdown: removing fs lock...
Wed Sep  7 16:56:19.438 dbexit: really exiting now

somehow this thing got completely jacked up and I'm not sure if I can recover it and last backup was over a month ago....way to much to re-configure on a reset/restore.

Member
Posts: 257
Registered: ‎05-19-2015
Kudos: 445
Solutions: 2

Re: Can no longer access UNIFI Controller on Cloud Key

well then.....looks like this is a known problem....FFS

 

http://community.ubnt.com/t5/UniFi-Wireless/Cloud-Key-Not-Loading/td-p/1595875

 

/CC @UBNT-Brandon and @UBNT-MikeD

 

This is complete crap if I am forced to do a full reset with an old backup to restore from...

Ubiquiti Employee
Posts: 7,528
Registered: ‎03-17-2015
Kudos: 4834
Solutions: 197

Re: Can no longer access UNIFI Controller on Cloud Key

@UBNT-MikeD should be reaching out soon via teamviewer to help here.

 

Thanks,

Brandon

Want to try out new features or fixes before they're released as Stable? Sign up for Beta here: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Signup-for-Beta-Access
Member
Posts: 257
Registered: ‎05-19-2015
Kudos: 445
Solutions: 2

Re: Can no longer access UNIFI Controller on Cloud Key

Still waiting.....

Ubiquiti Employee
Posts: 7,528
Registered: ‎03-17-2015
Kudos: 4834
Solutions: 197

Re: Can no longer access UNIFI Controller on Cloud Key

Sorry about this Chris we had a case of some terrible food poisoning yesterday... should be reaching out now.

Want to try out new features or fixes before they're released as Stable? Sign up for Beta here: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Signup-for-Beta-Access
Established Member
Posts: 871
Registered: ‎12-10-2009
Kudos: 186
Solutions: 16

Re: Can no longer access UNIFI Controller on Cloud Key


UBNT-Brandon wrote:

Sorry about this Chris we had a case of some terrible food poisoning yesterday... should be reaching out now.


Is that reaching or retching?  :-)

Established Member
Posts: 1,103
Registered: ‎08-29-2016
Kudos: 496
Solutions: 50

Re: Can no longer access UNIFI Controller on Cloud Key


UBNT-Brandon wrote:

we had a case of some terrible food poisoning yesterday... 


Food is overrated. Try beer instead. Cheers2

Ubiquiti Employee
Posts: 7,528
Registered: ‎03-17-2015
Kudos: 4834
Solutions: 197

Re: Can no longer access UNIFI Controller on Cloud Key


doktornotor wrote:

UBNT-Brandon wrote:

we had a case of some terrible food poisoning yesterday... 


Food is overrated. Try beer instead. Cheers2


Knew some special forces once... they said stick to fried food and beer - never go wrong.  Also, the Belgian monks actually would live off of beer for long periods of time (here).

 

Anyways, @UBNT-MikeD is now working w/ Chris on this.  Going forward (most of this already implemented) there is nightly auto-backup option in UCK (5.2.X+) and we're working to prompt user to enable this when upgrading to this version.

 

Then we'll also have restore option from UCK UI so that if there is catastrophic power failure, you can just restore from nightly autobackup and be on your merry way.

 

Cheers,

Brandon

Want to try out new features or fixes before they're released as Stable? Sign up for Beta here: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Signup-for-Beta-Access
Emerging Member
Posts: 44
Registered: ‎12-12-2013
Kudos: 3
Solutions: 2

Re: Can no longer access UNIFI Controller on Cloud Key

So what was the answer? I have exactly this issue right down to the repair failing with the same error. I tried contacting support via chat but that person did not even know the correct command to reboot from ssh. I kid you not. I have a case number if that helps.
Ubiquiti Employee
Posts: 7,528
Registered: ‎03-17-2015
Kudos: 4834
Solutions: 197

Re: Can no longer access UNIFI Controller on Cloud Key

We'll reach out.
Want to try out new features or fixes before they're released as Stable? Sign up for Beta here: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Signup-for-Beta-Access
New Member
Posts: 3
Registered: ‎04-26-2016

Re: Can no longer access UNIFI Controller on Cloud Key

Any chance there's a solution that can be provided? I just ran into this problem myself.

Senior Member
Posts: 10,712
Registered: ‎08-04-2017
Kudos: 1742
Solutions: 527

Re: Can no longer access UNIFI Controller on Cloud Key

Hello @danaspiegel,

 

Can you post the server.log and mongod.log?

Have you tried to repair the DB?

 

I can help your through TeamViewer if you want.

 

 

Regards,

Glenn R.

Cloud Hosted Controllers | Glenn R. | UniFi Installation/Easy Update Scripts | UniFi-Video Installation/Easy Update Scripts | UniFi-VoIP Installation Scripts
USG-4-PRO • USG
USW-48-500W • USW-24-POE-250W 2x • USW-16-POE-150W • USW-24 • USW-8-150W • USW-8
UAP XG • UAP-SHD • UAP-HD • UAP-NanoHD • UAP-AC-PRO 2x • UAP-AC-LITE • UAP-AC-IW • UAP-AC-M
UCK-G2 • UCK
New Member
Posts: 3
Registered: ‎04-26-2016

Re: Can no longer access UNIFI Controller on Cloud Key

Glenn, Thanks for the offer of help. Yes, I ran the repair DB process, which is how I came to this thread. Its failing with the following details:

 

 

root@UniFi-CloudKey:~# mongod --dbpath /usr/lib/unifi/data/db --smallfiles --logpath /usr/lib/unifi/logs/server.log --fork
Thu Dec  6 10:45:24.610 
Thu Dec  6 10:45:24.611 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
Thu Dec  6 10:45:24.612 
about to fork child process, waiting until server is ready for connections.
forked process: 17116
all output going to: /usr/lib/unifi/logs/server.log
log file [/usr/lib/unifi/logs/server.log] exists; copied to temporary file [/usr/lib/unifi/logs/server.log.2018-12-06T15-45-24]
ERROR: child process failed, exited with error number 100
root@UniFi-CloudKey:~# cat /usr/lib/unifi/logs/server.log
Thu Dec  6 10:45:24.687 [initandlisten] MongoDB starting : pid=17116 port=27017 dbpath=/usr/lib/unifi/data/db 32-bit host=UniFi-CloudKey
Thu Dec  6 10:45:24.688 [initandlisten] 
Thu Dec  6 10:45:24.688 [initandlisten] ** NOTE: This is a 32 bit MongoDB binary.
Thu Dec  6 10:45:24.688 [initandlisten] **       32 bit builds are limited to less than 2GB of data (or less with --journal).
Thu Dec  6 10:45:24.688 [initandlisten] **       Note that journaling defaults to off for 32 bit and is currently off.
Thu Dec  6 10:45:24.688 [initandlisten] **       See http://dochub.mongodb.org/core/32bit
Thu Dec  6 10:45:24.688 [initandlisten] 
Thu Dec  6 10:45:24.688 [initandlisten] db version v2.4.10
Thu Dec  6 10:45:24.688 [initandlisten] git version: nogitversion
Thu Dec  6 10:45:24.688 [initandlisten] build info: Linux hartmann 3.16.0-4-armmp-lpae #1 SMP Debian 3.16.39-1 (2016-12-30) armv7l BOOST_LIB_VERSION=1_55
Thu Dec  6 10:45:24.688 [initandlisten] allocator: system
Thu Dec  6 10:45:24.689 [initandlisten] options: { dbpath: "/usr/lib/unifi/data/db", fork: true, logpath: "/usr/lib/unifi/logs/server.log", smallfiles: true }
Thu Dec  6 10:45:24.737 [initandlisten] Assertion: 10446:mmap: can't map area of size 0 file: /usr/lib/unifi/data/db/ace.ns
0x48e13c 0x46f83a 0x45d190 0x45d238 0x476636 0x47449c 0x30b1e4 0x30c05c 0x253570 0x253a6e 0x203130 0x20326a 0x16836a 0x1699dc 0x15004c 0x76814632 
 mongod(_ZN5mongo15printStackTraceERSo+0x17) [0x48e13c]
 mongod(_ZN5mongo10logContextEPKc+0xa9) [0x46f83a]
 mongod(_ZN5mongo11msgassertedEiPKc+0x67) [0x45d190]
 mongod(_ZN5mongo11msgassertedEiRKSs+0x7) [0x45d238]
 mongod(_ZN5mongo16MemoryMappedFile3mapEPKcRyi+0x299) [0x476636]
 mongod(_ZN5mongo16MemoryMappedFile14mapWithOptionsEPKci+0x3f) [0x47449c]
 mongod(_ZN5mongo8MongoMMF4openERKSsb+0x27) [0x30b1e4]
 mongod(_ZN5mongo14NamespaceIndex5_initEv+0x53) [0x30c05c]
 mongod(_ZN5mongo8DatabaseC2EPKcRbRKSs+0x133) [0x253570]
 mongod(_ZN5mongo14DatabaseHolder11getOrCreateERKSsS2_Rb+0x33d) [0x253a6e]
 mongod(_ZN5mongo6Client7Context11_finishInitEv+0x23) [0x203130]
 mongod(_ZN5mongo6Client7ContextC1ERKSsS3_b+0x4d) [0x20326a]
 mongod(_ZN5mongo14_initAndListenEi+0x839) [0x16836a]
 mongod(_ZN5mongo13initAndListenEi+0xb) [0x1699dc]
 mongod(main+0x1d3) [0x15004c]
 /lib/arm-linux-gnueabihf/libc.so.6(__libc_start_main+0x99) [0x76814632]
Thu Dec  6 10:45:24.753 [initandlisten] warning database /usr/lib/unifi/data/db ace could not be opened
Thu Dec  6 10:45:24.753 [initandlisten] DBException 10446: mmap: can't map area of size 0 file: /usr/lib/unifi/data/db/ace.ns
Thu Dec  6 10:45:24.754 [initandlisten] exception in initAndListen: 10446 mmap: can't map area of size 0 file: /usr/lib/unifi/data/db/ace.ns, terminating
Thu Dec  6 10:45:24.754 dbexit: 
Thu Dec  6 10:45:24.754 [initandlisten] shutdown: going to close listening sockets...
Thu Dec  6 10:45:24.754 [initandlisten] shutdown: going to flush diaglog...
Thu Dec  6 10:45:24.754 [initandlisten] shutdown: going to close sockets...
Thu Dec  6 10:45:24.754 [initandlisten] shutdown: waiting for fs preallocator...
Thu Dec  6 10:45:24.754 [initandlisten] shutdown: closing all files...
Thu Dec  6 10:45:24.754 [initandlisten] closeAllFiles() finished
Thu Dec  6 10:45:24.754 [initandlisten] shutdown: removing fs lock...
Thu Dec  6 10:45:24.755 dbexit: really exiting now

 Trying to run the unifi service, I see:

 

root@UniFi-CloudKey:~# service unifi start
root@UniFi-CloudKey:~# service unifi status
● unifi.service - unifi
   Loaded: loaded (/lib/systemd/system/unifi.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-12-06 10:50:37 EST; 3s ago
  Process: 17322 ExecStart=/usr/lib/unifi/bin/unifi.init start (code=exited, status=0/SUCCESS)
 Main PID: 17384 (jsvc)
   Status: "{ "configurestate" : "configured" , "port" : 8443}"
   CGroup: /system.slice/unifi.service
           ├─17384 unifi -cwd /usr/lib/unifi -home /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre -cp /usr/share/java/commons-daemon.jar:/usr/lib/unifi/lib/ace.jar -pidfile /var/run/unifi.pid -procname unifi -outfile SYSLOG -errfile SYSLOG -umask 027 -user unifi -Dunifi.datadi
           ├─17386 unifi -cwd /usr/lib/unifi -home /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre -cp /usr/share/java/commons-daemon.jar:/usr/lib/unifi/lib/ace.jar -pidfile /var/run/unifi.pid -procname unifi -outfile SYSLOG -errfile SYSLOG -umask 027 -user unifi -Dunifi.datadi
           ├─17387 unifi -cwd /usr/lib/unifi -home /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre -cp /usr/share/java/commons-daemon.jar:/usr/lib/unifi/lib/ace.jar -pidfile /var/run/unifi.pid -procname unifi -outfile SYSLOG -errfile SYSLOG -umask 027 -user unifi -Dunifi.datadi
           └─17422 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/bin/java -Dfile.encoding=UTF-8 -Djava.awt.headless=true -Dapple.awt.UIElement=true -Xmx768M -XX:ErrorFile=/usr/lib/unifi/logs/hs_err_pid%p.log -jar /usr/lib/unifi/lib/ace.jar start
root@UniFi-CloudKey:~# cat /usr/lib/unifi/logs/server.log
[2018-12-06 10:50:23,005] <launcher> INFO  system - ======================================================================
[2018-12-06 10:50:23,012] <launcher> INFO  system - UniFi 5.9.29 (build atag_5.9.29_11384 - release) is started
[2018-12-06 10:50:23,012] <launcher> INFO  system - ======================================================================
[2018-12-06 10:50:23,026] <launcher> INFO  system - BASE dir:/usr/lib/unifi
[2018-12-06 10:50:23,069] <launcher> INFO  system - Current System IP: 10.1.0.13
[2018-12-06 10:50:23,071] <launcher> INFO  system - Hostname: UniFi-CloudKey
[2018-12-06 10:50:25,853] <launcher> INFO  system - UniFi Cloudkey, UUID = 303a4d55-44ef-5df3-9f07-95903ea31c56
[2018-12-06 10:50:38,304] <launcher> INFO  db     - waiting for db connection...
[2018-12-06 10:50:38,843] <launcher> INFO  db     - Connecting to mongodb://127.0.0.1:27117
[2018-12-06 10:50:41,036] <db-server> ERROR system - [exec] error, rc=100, cmdline=[bin/mongod, --dbpath, /usr/lib/unifi/data/db, --port, 27117, --unixSocketPrefix, /usr/lib/unifi/run, --noprealloc, --nohttpinterface, --smallfiles, --bind_ip, 127.0.0.1]
[2018-12-06 10:50:41,036] <db-server> INFO  db     - DbServer stopped
[2018-12-06 10:50:45,699] <db-server> ERROR system - [exec] error, rc=100, cmdline=[bin/mongod, --dbpath, /usr/lib/unifi/data/db, --port, 27117, --unixSocketPrefix, /usr/lib/unifi/run, --noprealloc, --nohttpinterface, --smallfiles, --bind_ip, 127.0.0.1]
[2018-12-06 10:50:45,699] <db-server> INFO  db     - DbServer stopped
[2018-12-06 10:50:50,152] <db-server> ERROR system - [exec] error, rc=100, cmdline=[bin/mongod, --dbpath, /usr/lib/unifi/data/db, --port, 27117, --unixSocketPrefix, /usr/lib/unifi/run, --noprealloc, --nohttpinterface, --smallfiles, --bind_ip, 127.0.0.1]
[2018-12-06 10:50:50,153] <db-server> INFO  db     - DbServer stopped

Which is where this problem started.

 

I would appreciate any help (hands on or otherwise) you can provide. Note that I'm running a gen1 cloudkey that was recently reprovisioned, upgraded (software) and put into servie about a month or 2 ago.

Reply