Reply
Established Member
Posts: 1,049
Registered: ‎02-23-2008
Kudos: 50

Re: Cloud Key - Unifi Stopped?

Same problem.

Can't backup and reset as can't get into the controller in the first place.

New Member
Posts: 31
Registered: ‎04-25-2016
Kudos: 3

Re: Cloud Key - Unifi Stopped?

Same has just happened on a fresh out the box installation after 24h

Does anyone's login have special characters in it? I have about 30 of these keys deployed and they're all still working. The only thing that sticks in my mind is the ones that are breaking have a ? In the controller password / key config....

Might be totally wrong!
New Member
Posts: 4
Registered: ‎03-22-2017

Re: Cloud Key - Unifi Stopped?

[ Edited ]

Same issue here. I just purchased and installed it a couple of days ago. 

 

 

I have DPI and cloud access enabled if that helps. 

 

Unifi version 5.4.11-9184

Cloud Key Version 0.6.1

 

Disk space: Used = 0.25 GB, Free = 10.05 GB

SD Card: Used = 0.03 GB, Free = 7.15 GB (one supplied in box)

 

Would appreciate a fix for this asap. I don't really want to reset to factory and start again. AP AC lite and USG still working ok. Just the Cloud Key Controller not working. 

 

 @UBNT-APieper Here is the relevant part of the log from /usr/lib/unifi/logs/server.log

 

[2017-03-25 16:03:57,097] <inform-24> INFO  inform - from [xx:aa:aa:aa:aa:aa](Unifi Security Gateway Office, UGW3, 4.3.34.4943823): state=CONNECTED, ext/stun_ip=aaa.bbb.ccc.ddd, dev_ip=xxx.yyy.aaa.bbb, up=201457

[2017-03-25 16:04:12,549] <launcher> INFO  system - ======================================================================

[2017-03-25 16:04:12,557] <launcher> INFO  system - UniFi 5.4.11 (build atag_5.4.11_9184 - release) is started

[2017-03-25 16:04:12,557] <launcher> INFO  system - ======================================================================

[2017-03-25 16:04:12,583] <launcher> INFO  system - BASE dir:/usr/lib/unifi

[2017-03-25 16:04:12,632] <launcher> INFO  system - Current System IP: aaa.bbb.ccc.ddd

[2017-03-25 16:04:12,646] <launcher> INFO  system - Hostname: UniFi-CloudKey

[2017-03-25 16:04:12,838] <launcher> INFO  system - UniFi Cloudkey, UUID = abc123

[2017-03-25 16:04:12,841] <launcher> INFO  system - Setting LED status to INITIALIZING

[2017-03-25 16:04:20,485] <launcher> INFO  system - Setting LED status to READY

[2017-03-25 16:04:21,607] <launcher> INFO  db     - waiting for db connection...

[2017-03-25 16:04:22,109] <launcher> INFO  db     - Connecting to mongodb://127.0.0.1:27117

^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@[2017-03-25 16:05:22,472] <launcher> INFO  system - ======================================================================

[2017-03-25 16:05:22,481] <launcher> INFO  system - UniFi 5.4.11 (build atag_5.4.11_9184 - release) is started

[2017-03-25 16:05:22,482] <launcher> INFO  system - ======================================================================

[2017-03-25 16:05:22,507] <launcher> INFO  system - BASE dir:/usr/lib/unifi

[2017-03-25 16:05:22,564] <launcher> INFO  system - Current System IP: aaa.bbb.ccc.ddd

[2017-03-25 16:05:22,578] <launcher> INFO  system - Hostname: UniFi-CloudKey

[2017-03-25 16:05:22,857] <launcher> INFO  system - UniFi Cloudkey, UUID = abc123

[2017-03-25 16:05:22,860] <launcher> INFO  system - Setting LED status to INITIALIZING

[2017-03-25 16:05:54,698] <launcher> INFO  system - Setting LED status to READY

[2017-03-25 16:05:55,548] <launcher> INFO  db     - waiting for db connection...

[2017-03-25 16:05:56,051] <launcher> INFO  db     - Connecting to mongodb://127.0.0.1:27117

[2017-03-25 16:06:01,305] <db-server> ERROR system - [exec] error, rc=100

[2017-03-25 16:06:01,580] <db-server> ERROR system - [exec] error, rc=100

[2017-03-25 16:06:01,581] <db-server> INFO  db     - DbServer stopped

[2017-03-25 16:06:05,870] <db-server> ERROR system - [exec] error, rc=100

[2017-03-25 16:06:05,870] <db-server> INFO  db     - DbServer stopped

[2017-03-25 16:06:10,160] <db-server> ERROR system - [exec] error, rc=100

[2017-03-25 16:06:10,161] <db-server> INFO  db     - DbServer stopped

[2017-03-25 16:06:14,445] <db-server> ERROR system - [exec] error, rc=100

[2017-03-25 16:06:14,445] <db-server> INFO  db     - DbServer stopped

[2017-03-25 16:06:18,740] <db-server> ERROR system - [exec] error, rc=100

[2017-03-25 16:06:18,740] <db-server> INFO  db     - DbServer stopped

[2017-03-25 16:06:23,033] <db-server> ERROR system - [exec] error, rc=100

 

....

 

 

[2017-03-25 21:52:01,454] <db-server> ERROR system - [exec] error, rc=100

[2017-03-25 21:52:01,455] <db-server> INFO  db     - DbServer stopped

^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@[2017-03-25 21:52:44,444] <launcher> INFO  system - ======================================================================

[2017-03-25 21:52:44,453] <launcher> INFO  system - UniFi 5.4.11 (build atag_5.4.11_9184 - release) is started

[2017-03-25 21:52:44,454] <launcher> INFO  system - ======================================================================

[2017-03-25 21:52:44,479] <launcher> INFO  system - BASE dir:/usr/lib/unifi

[2017-03-25 21:52:44,537] <launcher> INFO  system - Current System IP: aaa.bbb.ccc.ddd

[2017-03-25 21:52:44,554] <launcher> INFO  system - Hostname: UniFi-CloudKey

[2017-03-25 21:52:44,830] <launcher> INFO  system - UniFi Cloudkey, UUID = abc123

[2017-03-25 21:52:44,834] <launcher> INFO  system - Setting LED status to INITIALIZING

[2017-03-25 21:52:46,349] <launcher> INFO  db     - waiting for db connection...

[2017-03-25 21:52:46,851] <launcher> INFO  db     - Connecting to mongodb://127.0.0.1:27117

[2017-03-25 21:52:49,115] <db-server> ERROR system - [exec] error, rc=100

[2017-03-25 21:52:49,115] <db-server> INFO  db     - DbServer stopped

[2017-03-25 21:52:49,116] <db-server> WARN  db     - DbServer not shutdown cleanly and need repairing on next startup

[2017-03-25 21:53:34,971] <db-server> ERROR system - [exec] error, rc=100

[2017-03-25 21:53:35,241] <db-server> ERROR system - [exec] error, rc=100

[2017-03-25 21:53:35,242] <db-server> INFO  db     - DbServer stopped

[2017-03-25 21:53:40,142] <db-server> ERROR system - [exec] error, rc=100

[2017-03-25 21:53:40,143] <db-server> INFO  db     - DbServer stopped

[2017-03-25 21:53:44,424] <db-server> ERROR system - [exec] error, rc=100

 

 

 

New Member
Posts: 11
Registered: ‎01-12-2016

Re: Cloud Key - Unifi Stopped?

Same issue - 4 of 9 total sites so far with this issue.

Unifi service just shows "Stopped" in the Web gui and reboots and firmware updates do not get the service to start again.

 

Emerging Member
Posts: 63
Registered: ‎09-06-2016
Kudos: 11
Solutions: 2

Re: Cloud Key - Unifi Stopped?

[ Edited ]

Assuming you're seeing the following in the server log, it's probably the database corruption issue.

 

<db-server> ERROR system - [exec] error, rc=100

 

Last I read up on this, mongodb can corrupt after an improper shutdown (ex, power loss). Keep a backup handy, factory reset it, and restore it.

 

Other threads will detail how to turn on advanced DB logging, deleting 0k files, and determining which db files hold stats/config/etc.

 

Assuming the problem stems from mongodb, I don't expect much from UBNT to resolve. They might implement some helpful recovery options, but the better approach is to weigh this risk and your use case carefully before using the CK.

New Member
Posts: 2
Registered: ‎07-08-2016

Re: Cloud Key - Unifi Stopped?

ok

What i did

I reseted my cloud key and restored my backup from it

And it helped me

 

Before I updated my Cloud Key to 6.X.xxx version and my UNIFI Service was STOPED.....

 

P.S: After a few day i had some problem,  one of my Unifi Aps was broken. It had 'Provision' status and i reseted it

 

Now OK

New Member
Posts: 4
Registered: ‎04-04-2017
Kudos: 2

Re: Cloud Key - Unifi Stopped?

@UBNT-APieper

 

Same problem here

<db-server> ERROR system - [exec] error, rc=100

Started with firmware upgrade - made the mistake of not doing a backup of the cloud key first. 

Have a POE switch that runs a 24/7 call center that I can't afford downtime. 

Have had a miserable time trying to get help though chat and email support.

Are you able to help debug?

Ubiquiti Employee
Posts: 588
Registered: ‎10-04-2015
Kudos: 634
Solutions: 40
Contributions: 1

Re: Cloud Key - Unifi Stopped?

@thendricks1980 Sure happy to help- can you provide me any further info? Do you have SSH access to the device? Logs?

UBNT_Alternate_Logo.png
Ubiquiti Networks Enterprise Support Team

Check out our ever-evolving Help Center for answers to many common questions!

New Member
Posts: 4
Registered: ‎04-04-2017
Kudos: 2

Re: Cloud Key - Unifi Stopped?

@UBNT-APieper

SSH works no issue.  Going to the LAN IP allows me to manage the cloud key but clicking the link for the Unify section results in an unreachable site. Managing the cloud key I see the Unify status as stopped. I've been able to upgrade the firmware to 0.6.2 so the basic cloud key functionality seems to be just fine. 

 

I have used WinSCP to copy the files from /usr/lib/unifi/data and placed them on a seperate cloud key for testing. The behaviour of the the Unify Service being Stopped followed the configuration files to the new cloud key. 

 

[2017-03-30 18:57:24,208] <inform_stat-35133> ERROR dev - Error in DeviceManager.processStat()
com.mongodb.WriteConcernException: { "serverUsed" : "localhost:27117" , "err" : "bad offset:0 accessing file: /usr/lib/unifi/data/db/ace.0 - consider repairing database" , "code" : 13440 , "n" : 0 , "connectionId" : 9 , "ok" : 1.0}
at com.mongodb.CommandResult.getWriteException(CommandResult.java:90)
at com.mongodb.CommandResult.getException(CommandResult.java:79)
at com.mongodb.CommandResult.throwOnError(CommandResult.java:140)
at com.mongodb.DBTCPConnector._checkWriteError(DBTCPConnector.java:135)
at com.mongodb.DBTCPConnector.access$000(DBTCPConnector.java:40)
at com.mongodb.DBTCPConnector$1.execute(DBTCPConnector.java:186)
at com.mongodb.DBTCPConnector$1.execute(DBTCPConnector.java:181)
at com.mongodb.DBPort.doOperation(DBPort.java:187)
at com.mongodb.DBTCPConnector.doOperation(DBTCPConnector.java:208)
at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:181)
at com.mongodb.DBCollectionImpl.insertWithWriteProtocol(DBCollectionImpl.java:681)
at com.mongodb.DBCollectionImpl.insertImpl(DBCollectionImpl.java:214)
at com.mongodb.DBCollection.insert(DBCollection.java:176)
at com.mongodb.DBCollection.insert(DBCollection.java:93)
at com.mongodb.DBCollection.insert(DBCollection.java:78)
at com.mongodb.DBCollection.insert(DBCollection.java:120)
at com.ubnt.service.OoOO.M.o00000(Unknown Source)
at com.ubnt.service.OoOO.return.o00000(Unknown Source)
at com.ubnt.service.ooOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO.B.?00000(Unknown Source)
at com.ubnt.service.A.O0oOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO$_Oo.o00000(Unknown Source)
at com.ubnt.service.A.K.o00000(Unknown Source)
at com.ubnt.service.A.O0oOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO.o00000(Unknown Source)
at com.ubnt.service.A.O0oOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO.o00000(Unknown Source)
at com.ubnt.service.A.oO0O$_o.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2017-03-30 18:57:28,343] <inform_stat-35121> ERROR dev - Error in DeviceManager.processStat()
com.mongodb.WriteConcernException: { "serverUsed" : "localhost:27117" , "err" : "bad offset:0 accessing file: /usr/lib/unifi/data/db/ace.0 - consider repairing database" , "code" : 13440 , "n" : 0 , "connectionId" : 9 , "ok" : 1.0}
at com.mongodb.CommandResult.getWriteException(CommandResult.java:90)
at com.mongodb.CommandResult.getException(CommandResult.java:79)
at com.mongodb.CommandResult.throwOnError(CommandResult.java:140)
at com.mongodb.DBTCPConnector._checkWriteError(DBTCPConnector.java:135)
at com.mongodb.DBTCPConnector.access$000(DBTCPConnector.java:40)
at com.mongodb.DBTCPConnector$1.execute(DBTCPConnector.java:186)
at com.mongodb.DBTCPConnector$1.execute(DBTCPConnector.java:181)
at com.mongodb.DBPort.doOperation(DBPort.java:187)
at com.mongodb.DBTCPConnector.doOperation(DBTCPConnector.java:208)
at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:181)
at com.mongodb.DBCollectionImpl.insertWithWriteProtocol(DBCollectionImpl.java:681)
at com.mongodb.DBCollectionImpl.insertImpl(DBCollectionImpl.java:214)
at com.mongodb.DBCollection.insert(DBCollection.java:176)
at com.mongodb.DBCollection.insert(DBCollection.java:93)
at com.mongodb.DBCollection.insert(DBCollection.java:78)
at com.mongodb.DBCollection.insert(DBCollection.java:120)
at com.ubnt.service.OoOO.M.o00000(Unknown Source)
at com.ubnt.service.OoOO.return.o00000(Unknown Source)

Member
Posts: 149
Registered: ‎06-08-2014
Kudos: 52
Solutions: 4

Re: Cloud Key - Unifi Stopped?

Had yet another one with a corrupt database today. 

 

Just wondering what Ubiquiti is doing to address the problem permanently, what the ETA is, and what the fix will be fixed... 

 

Is there somewhere we can track the fix. For us, this is becoming a serious issue (as a lot of jobs are in remote areas)

 

 

Established Member
Posts: 1,613
Registered: ‎11-12-2015
Kudos: 443
Solutions: 46

Re: Cloud Key - Unifi Stopped?


@AndrewLuecke wrote:

Had yet another one with a corrupt database today. 

 

Just wondering what Ubiquiti is doing to address the problem permanently, what the ETA is, and what the fix will be fixed... 

 

Is there somewhere we can track the fix. For us, this is becoming a serious issue (as a lot of jobs are in remote areas)


I too would like to hear more about this. I've been lucky - never had a problem with my CK - but figure I am living on borrowed time.

 

ISTM that Ubiquiti could do some simple things that would help with some of the problems that have been reported. For instance, the startup script could look for the zero-lengh files and automatically remove them.

 

Beyond that, I would think that if startup goes poorly the CK could automatically reset to default and reload the most recent automatic backup.

 

But what is really needed is an understanding of the root cause of all these problems. I gather it is DB corruption during a non-graceful shutdown. That would seem to apply to all controller installations, but most of the reported problems seem to be with cloud keys. Is there something about the cloud key that makes it more vulnerable to corruption?

New Member
Posts: 4
Registered: ‎02-27-2017
Kudos: 1

Re: Cloud Key - Unifi Stopped?

@UBNT-APieper

 

Why has this not been addressed/fixed?

New Member
Posts: 16
Registered: ‎10-13-2016

Re: Cloud Key - Unifi Stopped?

I am experiencing the same issues and as far as I can establish there is not a fix for this to date. 

The CK as it stand is not reliable and cannot be used with any level of certainity.

Ubiquiti Employee
Posts: 588
Registered: ‎10-04-2015
Kudos: 634
Solutions: 40
Contributions: 1

Re: Cloud Key - Unifi Stopped?

[ Edited ]

The issues you have noted are definitely priorities for us. We are working on debugging these issues but a number of other customers in this thread have noted db corruption is a frequent cause of the issue- which makes solving this/preventing adverse behavior difficult as we cannot reproduce such corruptions. 

We are looking at ways to make repairing DB errors/issues more easy to identify and fix when encountered with the Cloud Key. One of the problems with the Cloud Key is that if there is a power interruption this easily creates db corruption- and this makes reproducing some issues customers run into very difficult. 

It is recommended to take frequent backups and to use the backups as a way to easily resolve corrupt DBs. 

Other best practices that may prevent these issues from coming up are lessening the degree of data retention as to keep the DB/space dedicated to backups from growing too large in size. 

@brientim Can everyone who has not yet sent me logging send this to me via PM so I can look and compare different causes of this behavior? Thanks!

UBNT_Alternate_Logo.png
Ubiquiti Networks Enterprise Support Team

Check out our ever-evolving Help Center for answers to many common questions!

Ubiquiti Employee
Posts: 588
Registered: ‎10-04-2015
Kudos: 634
Solutions: 40
Contributions: 1

Re: Cloud Key - Unifi Stopped?

@thendricks1980 Can you ssh in and tell me how much disk space is available on the Cloud Key? You can do so by running 'df -h' on the cloud key and sharing what is shown. Thanks! 

UBNT_Alternate_Logo.png
Ubiquiti Networks Enterprise Support Team

Check out our ever-evolving Help Center for answers to many common questions!

Established Member
Posts: 1,613
Registered: ‎11-12-2015
Kudos: 443
Solutions: 46

Re: Cloud Key - Unifi Stopped?


@UBNT-APieper wrote:

The issues you have noted are definitely priorities for us. We are working on debugging these issues but a number of other customers in this thread have noted db corruption is a frequent cause of the issue- which makes solving this/preventing adverse behavior difficult as we cannot reproduce such corruptions. 

We are looking at ways to make repairing DB errors/issues more easy to identify and fix when encountered with the Cloud Key. One of the problems with the Cloud Key is that if there is a power interruption this easily creates db corruption- and this makes reproducing some issues customers run into very difficult. 

It is recommended to take frequent backups and to use the backups as a way to easily resolve corrupt DBs. 

Other best practices that may prevent these issues from coming up are lessening the degree of data retention as to keep the DB/space dedicated to backups from growing too large in size. 

@brientim Can everyone who has not yet sent me logging send this to me via PM so I can look and compare different causes of this behavior? Thanks!


@UBNT-APieper, I'm glad to hear you are investigating.

 

Do you concur that the problem appears to be exclusive to or at least more common on cloud keys than for controllers running on some other platform? That should itself provide some hints to what the problem is.

 

I suspect it may be related to file system caching, and inability to flush the cache when power is lost. I realize that caching is especially important with flash storage, but it does introduce a vulnerability. Perhaps choosing a different file system would help. (I haven't looked at what is being used.) A journaling file system might help.

 

Or, perhaps a different database, with a different storage discipline, might help. (But I recognize that such a change could be extremely difficult.)

 

Other than those things, better recovery heuristics would be an improvement. ISTM the goal ought to be get back up, one way or another, regardless of what went wrong. If it means rolling back to the last auto backup then so be it. That would be better than not coming up at all.

 

And good integration with UPSs would allow people to improve their odds.

 

New Member
Posts: 17
Registered: ‎05-06-2014
Kudos: 16

Re: Cloud Key - Unifi Stopped?

Part of me thinks that the Cloud Key as a PoE device is part of the problem. Cloud keys are used to upgrade switches, which power cycles the switch, which removes power from the Cloud Key temporarily and unexpectedly, which potentially kills your database. Perhaps Ubiquitit should just tell people not to use it as PoE, and to go ahead and always use the USB external power option so that the cloud key doesn't see power interruptions every time the switch gets upgraded.

 

Is my thinking correct here? Alternately, can Ubiquiti change to an internal database that takes frequent (hourly?) snapshots and writes incremental changes since then to a database log. Any time it loses power and sees corruption it can rebuild from the most recent snapshot plus database log as a delta since the snapshot?

New Member
Posts: 12
Registered: ‎08-15-2016
Kudos: 6

Re: Cloud Key - Unifi Stopped?

My Cloud Key is powered by my Switch-24 which is backed up by a rather beefy double conversion UPS.  So in the time I've had the Cloud Key it has never suffered a dirty shutdown other than when it's locked up so tight I have no choice but to power cycle it.

 

It may run for a little more than a week, or less than a day, before it locks up.  And there seems to be differing degrees of "locked up".  Locked up Lite - I can still SSH to the Cloud Key and issue a reboot command.  Locked up Medium - I need to SSH into the switch and PoE cycle the switch port the CK is on.  Locked up Hard - paperclip.  Additionally, since I now keep my CK in the bedroom, rather than two floors down in the basement, I get to see just how often that little thing spontaneously reboots.  It probably averages 3 or 4 times per day, but I would never have known if it wasn't within my view.  To briefly address the issue of PoE vs. microUSB for power - it doesn't matter.

 

I don't know what the issue(s) is/are, but they appear to be terminal with this product.  I'm sure everybody around here is busy, but I just don't have the time nor the desire to be an unpaid bug hunter anymore. It was kind of cool 25 years ago, got to be annoying about 15 years ago, and just isn't even an option now.  I know there had to have been a bit of a scramble to quell the bad press re the 20+ year old PHP bug, but a company this size should have the resources to assign to this issue.

 

This is only my home network, but the spousal acceptance rate is falling like a rock through the sky.  Nevermind that I've been delaying the purchase of around a dozen switches and APs for work.  Until I see something that boosts my confidence level greatly, I'm probably just going to begrudgingly spend many times more for the same kind of equipment with Cisco.

Senior Member
Posts: 4,008
Registered: ‎05-13-2015
Kudos: 1213
Solutions: 99

Re: Cloud Key - Unifi Stopped?

For the people who are affected with their CKs, which one do you have?

 

There is v1 and v2.

Established Member
Posts: 1,613
Registered: ‎11-12-2015
Kudos: 443
Solutions: 46

Re: Cloud Key - Unifi Stopped?


@pvanwylen wrote:

Part of me thinks that the Cloud Key as a PoE device is part of the problem. Cloud keys are used to upgrade switches, which power cycles the switch, which removes power from the Cloud Key temporarily and unexpectedly, which potentially kills your database. Perhaps Ubiquitit should just tell people not to use it as PoE, and to go ahead and always use the USB external power option so that the cloud key doesn't see power interruptions every time the switch gets upgraded.

 

Is my thinking correct here? Alternately, can Ubiquiti change to an internal database that takes frequent (hourly?) snapshots and writes incremental changes since then to a database log. Any time it loses power and sees corruption it can rebuild from the most recent snapshot plus database log as a delta since the snapshot?


My understanding (second hand from reading here) is that when Unifi switches are updated the flow of PoE is not disrupted. That is what I have seen since changing my CK to PoE from a switch.

Reply