Reply
Established Member
Posts: 1,613
Registered: ‎11-12-2015
Kudos: 443
Solutions: 46

Re: Cloud Key - Unifi Stopped?


@mikejr83 wrote:

How do you know if you have a v1 or v2 key?


I'm pretty sure the disk space is different. If you go to the web interface of the CK, on the Main tab check the total disk space. Mine shows 1.45GB. A v2 should have more.

 

Emerging Member
Posts: 44
Registered: ‎08-16-2016
Kudos: 17
Solutions: 2

Re: Cloud Key - Unifi Stopped?

As phk46 said a V2 has 10.3 GB

New Member
Posts: 10
Registered: ‎09-20-2017

Re: Cloud Key - Unifi Stopped?

Update on my key:

 

It sat with the white light (factory defaults) for a day. On a whim I decided to see if I could get back to it. I could! I restored a backup to it. I pulled some logs that showed a bunch of Java errors for some of the process logs in the logs folder and sent them to @UBNT-Corey. From my eye they didn't look too helpful, especially since it just came out of a factory reset.

 

Anyhow, a day ago it went offline, again. No response to the mangement or controller software. Also I was unable to SSH in to get any logs. Great! Well, I cycled the PoE through the router (and then setup the PoE watchdog) and it came online. Kinda. No controller web interface. My thought was that the MongoDB had been corrupted.

 

That was last night. Since then it looks like the watchdog must have cycled the PoE about a half dozen times. When I tired this morning I got the management login! I SSH'd in got the logs and set them on to @UBNT-Corey. I found some stack overflows in the server.log. Hopefully this should help the guys.

 

On the Pi front. I've ordered a PoE splitter so that I can make my Pi very close to how the key operates. Right now both the key have the same Debain distro and appear to have similar arm7 CPUs. The Key has an additional 1Gb of RAM. The other large difference with the two would be that there is onboard stoarge where the Pi has none and the ethernet interface as the Pi's is 10/100 with no PoE. The controller software versions are the same. I haven't looked to see if the versions of Java are the same, but I tend to almost always install OpenJDK/JRE on my *nix machines.

 

The reason I'm using a Pi as comparison is that it appears to be close in specs as the Key. It's not an exact 1:1 comparison but I figure for the cost of the PoE splitter and a Pi 3 kit you'd have a reasonable approximation of a key. I've been mulling over writing a tutorial along with producing some scripts which when run would download the requisite software and provision the Pi with a basic install of the controller software. Would anyone be interested in this? I could just list out the commands to execute or produce a script.

 

I've been trying to come up with a way to isolate the mongo data in a way that would help survive sudden power losses. Without some sort of network attached storage or additional battery backup I'm not sure there is going to be a simple way to modify the setup. A buddy of mine uses his Pi as a car-puter and entertainment system. He has this rather cool switch that hooks in the an ignition wire and the 12v hot. When the car turns off it executes a script on the Pi to shutdown. It turns a GPIO pin "off" which tells the switch that the Pi is off. The result is that the Pi runs on a little lithium batter backup for a few seconds while it shuts down cleanly. That's about the only other alternative I can think of.

Ubiquiti Employee
Posts: 50
Registered: ‎01-30-2017
Kudos: 8

Re: Cloud Key - Unifi Stopped?

[ Edited ]

Hello Community Members,

Our community users have also reached out over PM to gather logs and I am going through all the logs that you have all sent me over the weekend. If I have missed you,  I will circle back around to double check this morning, and gather logs from you as well. If I have missed anyone please send me a PM. 

Best Regards,

New Member
Posts: 10
Registered: ‎09-20-2017

Re: Cloud Key - Unifi Stopped?

[ Edited ]

Hey all,

 

Just an update to the thread:

 

@UBNT-Corey and I got on a web session yesterday to investigate the issue that I was seeing regarding the key stopping. We first upgraded to some beta releases, 0.7.4 for the firmware and 5.6.18 for the controller. The upgrades seemingly went well.

 

As we discussed the issue I was SSH'd into the key and had "tail -f /usr/lib/unifi/logs/server.log" running. As we talked I noticed the key "reboot" several times. The first time I don't think eth0 actually went down because I didn't get a disconnect. However, on the second time it did kick me. I logged in a min or so later. On the third reboot the key died. No SSH or anything.2017-10-03 16_17_01-Logs-pre-crash-cloud-key.png

 

 

The logs dir also has a bunch of hs_error_pidXXX.log files.

 

Interesting!

 

My thought is that these errors which lead to a reboot (or at least a service restart) corrupt the mongo database which in turn is causing the controller service failures. Is anyone else seeing logs similar to this?

 

@UBNT-Corey, I don't recall any identifiable info in those logs I gave you. I haven't gone through in depth to verify, but I'll post them here later to see if anyone else is seeing these same issues after I verify that I am not going to give away the keys to the kingdom.

New Member
Posts: 7
Registered: ‎05-01-2017

Re: Cloud Key - Unifi Stopped?

[ Edited ]

I sent them the debug info requested (via PM) a week or two ago. Hopefully this investigation does lead to them figuring out what is causing the cloud keys to go down.

 

Mine has gone down a few times since but I've ever been able to restart via ssh or via cycling the power. No corruption yet. Hopefully this holds true until they get a fix out.

 

EDIT: Note that I was replying to another message that appears to have been deleted.

New Member
Posts: 2
Registered: ‎03-15-2017

Re: Cloud Key - Unifi Stopped?

You expect a fix from them??? this has been going on for over a year and NO fix.... so, don't get your hopes up...

Senior Member
Posts: 4,008
Registered: ‎05-13-2015
Kudos: 1213
Solutions: 99

Re: Cloud Key - Unifi Stopped?

They have been fixing issues with CK and it will take sometime.

One of the issues I have experienced which is not related and happens randomly on my CK maybe solved soon as they believed they gave pin pointed the cause.
New Member
Posts: 12
Registered: ‎04-05-2017
Kudos: 2

Re: Cloud Key - Unifi Stopped?

[ Edited ]

For a bug to render the item not fit for purpose, should result in you not selling any more Cloud keys to the public, knowing it is not fit for its advertised, and intended purpose.

I think you should also allow people to return to the place of purchase, no questions asked.

It does appear that a lot of us have waited, more than a resonable amount of time , and at this point there is no way for us to get them to work properly.

Perhaps a resolution could be that you offer the above, and should the money be returned to those customers affected, they still can re-purchase if the product is functional sometime in the future, so you could bite the bullet and offer some customers a refund in good faith, as they in turn had good faith that the Cloud Key would work.

Some cusomers like   , myself, would be happy with other Ubiquiti products (we have 12)   , but are not happy with the Cloud key at all.

New Member
Posts: 10
Registered: ‎09-20-2017

Re: Cloud Key - Unifi Stopped?


@iano wrote:

I sent them the debug info requested (via PM) a week or two ago. Hopefully this investigation does lead to them figuring out what is causing the cloud keys to go down.

 

Mine has gone down a few times since but I've ever been able to restart via ssh or via cycling the power. No corruption yet. Hopefully this holds true until they get a fix out.

 

EDIT: Note that I was replying to another message that appears to have been deleted.


Ya. I guess I must have committed some faux pas with the image I posted. Ooops. I thought I redacted anything on it that shouldn't have been there.

 

For anyone wondering, I posted a screen cap of a tail -f /usr/lib/unifi/logs/server.log that showed several "reboots" of either the unifi service or the key itself. The last reboot killed SSH. I was only able to get back to the key after power cycling PoE a couple of times.



@lafleur wrote:

You expect a fix from them??? this has been going on for over a year and NO fix.... so, don't get your hopes up...


While I know Ubiquiti is a company providing a product which goes through dev, testing, and QA before releasing to the end consumer. It's a huge bummer to go out and get a shinny new piece of gear and then have it break or not work. It's a fact that bugs get out and sometimes they suck. 

 

I'm a software developer by day and I've also done a fair share of open source development. I can say that I have had more than I can count of those issues where it cannot be reproduced on my dev machine or in the testing lab but I can see it with my own eyes it fail in a customer environment. Being able to get feedback from a good amount of users can make some of these hard to pinpoint defects a bit easier. From my experience on the dev side talking to customers about an issue I always appreciate every bit of information they can pass along. I'm doing my best to recipricate the karma. Man Happy

 

If I can put a little time at it every now and then and answer a few questions and help along a fix all the better!

New Member
Posts: 12
Registered: ‎04-05-2017
Kudos: 2

Re: Cloud Key - Unifi Stopped?

Hi To Mike and those that are helping Ubiquiti.

Thanks to you guys for helping Ubiquiti.

It is great that you are helpine, and I wish I had time to do so as well, but working too many hours right now, so a thanks to you and the others that are helping Ubiquit debug it, I believe that the key is a good product , but it has been a long time getting it fixed, and too long .

Hope Ubiquiti either fixes the problem, or rewrites  the code so it works.

Either way, they cannot leave loyal customers hanging in indefinitely, it needs a pro-active short term customer relations exercise.  Not every customer has the time to help Ubiquiti fix the bug / s  .

So how about a bit of old fashioned customer service re this issue? 

Cheers.

New Member
Posts: 1
Registered: ‎12-28-2016

Re: Cloud Key - Unifi Stopped?

Same issue here. I'm willing to help debug. PM me.

 

Cloud Key Firmware: v0.5.9
Unifi Version: 5.3.8-8920

New Member
Posts: 4
Registered: ‎11-09-2015
Kudos: 2

Re: Cloud Key - Unifi Stopped?

After upgrade CK to fw 0.7.4 all (i have 10) brick, not totally, ssh worked but gui have that same issue like everyone:

<db-server> ERROR system - [exec] error, rc=100

After fer hours searching in every logs on CK i couldn't find why MongoDB corrupted data.

I'm almost 100% sure that issue is caused by MongoDB (similar issue can be found in other DB when log file was corrupted). Of course this can be cause not only by corrupted logfile but for example, MongoDB was initialized more then once and because of that more then one process try access same file and locked it or lost privileges to modify file and then even after reboot still don't have privileges to modify file.

 

i looked on file structure in  /srv/unifi/data/db like @djallen mention but local.0 and local.ns wasn't empty (have size of 16MB but w8 all ac_stat.* have 0 byte.

After deleting (i deleted after CK back to life, before i only changed name of all file mentioned further) all local.* and ac_stat.*,after restart my CK resurrected. Tried on all my CK and worked on every single one.

New Member
Posts: 27
Registered: ‎03-30-2017
Kudos: 2

Re: Cloud Key - Unifi Stopped?

1. How many of your Cloud Keys are displaying the issue?

Only have one, and it has showed the issue from first use (a few days ago)
2. What Cloud Key firmware and what controller version is the issue being seen on?

Firmware v0.7.5, Controller v5.5.24
3. How are you powering your Cloud Key(s)?

Was PoE from US-8-150W, just switched to try PoE from US-8-60W
4. Was there any interruption of power or power-cycling of the equipment as aligned with the symptoms being displayed?

No, not as far as I am aware.

 

This is a home system with a USG, two switches (a US-8-150W near my NBN modem and a US-8-60W on my desk) and 2 APs. I have been using a laptop running the desktop UniFi app to setup the system and very recently started trying to use the Cloud Key by starting with the configuration saved by the desktop version. It dropped out after less than a day several times before I started trouble shooting it and discovered this discussion.

 

Judging from its own latency records, it seems that after getting it going most recently about 10am this morning, the Cloud Key ran for approx 2.5 hours before losing 20 mins of records, then logged latency for another 2 hours before dropping out twice in half an hour, then ran another 45 minuites before shutting down permanently until I came to look at it this evening. I was running the Cloud Key from the US-8-150W but since it was giving this trouble (going offline after a few hours of running) I decided to relocate it to my desk and run it from the US-8-60W.

 

After unplugging it from the US-8-150W and plugging it into the last (PoE) port on the US-8-60W, I accessed IP address with Chrome browser (Version 61.0.3163.100) and it gave the message:

Outdated Browser
You are using an outdated browser. Please upgrade your browser to improve your experience.
{{toast.message}}

I updated my browser to the latest (Version 62.0.3202.75) and tried again and the controller dashboard came up but appeared to have lost its configuration (all baloons red with no active devices). I thought I would try to reset it by logging in to the "Configure" portal and using the maintenance pages. However when I clicked this and entered username and password (and before doing anything else) I noticed the US-8-60W that the cloud Key was plugged into do a reset (all port link lights flashed orange!) and the browser gave the message:

Device is offline
Your device is offline. After turning it on you will be redirected to login page.

I thought I would look at my UniFi system using the desktop controller app running on my laptop (which I have been using for many months before very recently trying to get the Cloud Key running). Remarkably it seems that the desktop controller has also now lost all its configuration data (all balloons red and no active devices)! But the system is still running and I can access the internet so all the red balloons on the dashboard page and all the "disconnected" statuses on the devices page do not reflect reality. I don't know where the system configuration is stored, but it seems that this malfunction on the Cloud Key wiped out the configuration that was presented by a laptop controller (that was fine a short while ago), but the system configuration is not really wiped out, it just appears to be!


Then I stopped the desktop controller and logged in to the Cloud Key controller again and looked at the dashboard and all had come good again! (3 green balloons, etc). So logged out of that and started the desktop version again but that was still showing all red balloons. Logout of the desktop version and login to the Cloud Key version and all green balloons.


Logged out of the Cloud Key controller software and into its maintenance pages - and this time there was no ill effect (it did not reset the US-8-60W and no unexpected lights flashing orange).

 

I hope that reporting these strange symptoms might help to narrow down the root cause of this problem!

New Member
Posts: 5
Registered: ‎10-14-2017

Re: Cloud Key - Unifi Stopped?

I have also had my Cloud Key stop mysteriously a couple of times in the last few days.  Yesterday it wouldn't allow login from the web page.  I was able to ssh in and do a reset then recover from autobackup.  Today when I tried to login I get the strange message in the web browser "Device is offline  Your device is offline. After turning it on you will be redirected to login page."

 

I ssh'd into the cloud key without problems, and have looked at the log files in both /var/log and /usr/lib/unifi/logs and don't see any obvious errors.  In fact, it appears that the controller is running as server.log is updating with device state information.  "tail -f server.log" shows all sorts of status data being logged, and no apparent errors.

 

I did a reboot via ssh and while again I can ssh in and the logs don't show any obvious errors, attempts to connect via web still result in the offline message.  (It would be interesting to know what is generating this message if the system is offline...)

 

I am running firmware 0.7.5 and the beta controller 5.6.20.

 

Cloud Key is plugged into a US-24-250W switch and the whole system is on a UPS.  I'm not aware of any power issues that would have caused a shutdown.

 

All I can say (along with others) is that this is pretty worrisome behaviour and really makes me question whether the Cloud Key is a worthy product.

 

John

New Member
Posts: 5
Registered: ‎10-14-2017

Re: Cloud Key - Unifi Stopped?

Following up...

 

I just thought to look in /srv/unifi/run and found two zero-byte entries that I suspect might be meaningful:

 

db.needsRepair

launcher.looping

 

I don't see a .sock file for mongodb.

 

So perhaps the database is corrupt.  Is there a way to repair the tables, or do I just restore from the last autobackup file?

 

John

New Member
Posts: 5
Registered: ‎10-14-2017

Re: Cloud Key - Unifi Stopped?

Last night I did a mongodb repair and then a unifi service start.  That did not change anything.  But when I forced a reboot, I was able to login normally.

 

But this morning I am getting the same "Cloud Key is offline" message when I go to the unifi web interface.  Nothing touched the unit overnight, and again it gets POE from a US-24-250 switch plugged into a UPS system.  I can ssh into the unit, but no web GUI.

 

I did a /sbin/halt, unplugged from POE switch, then replugged.  The unit came up and the LED is constant blue.  I can ssh, but the web interface is still down.  server.log does not show any error messages or anything that looks related to the controller; just normal-looking  adoption traffic.

New Member
Posts: 5
Registered: ‎10-14-2017

Re: Cloud Key - Unifi Stopped?

Sorry for all the replies to myself, but I finally discovered something useful.

 

It's only when using Firefox as my web browser that I seem to get the "device is offline" message.  I can login with Chrome or Chromium and it's OK (though with Chromium it complains of an unsupportede browser).

 

New Member
Posts: 27
Registered: ‎03-30-2017
Kudos: 2

Re: Cloud Key - Unifi Stopped?

Hi @N8UR, I think it is helpful to give as much info as possible, hopefully Ubiquiti will keep working on this problem until they have it solved. My Cloud Key has behaved better recently and is till online after 24 hours - a record so far! However I notice that it has dropped ~10mins worth of internet latency logging 4 times during the last 24 hours which I assume happens whenever it crashes and does an automatic restart. But I am not sure what these dropped periods of logging really mean. It would be nice if there was someone who knew more about the Cloud Key system to pass comment.
New Member
Posts: 10
Registered: ‎09-20-2017

Re: Cloud Key - Unifi Stopped?

[ Edited ]

The new cloud key has been online for 14 days without any issues (it would be longer but I've been moving equipment around) and I just restarted to upgrade the firmware to 0.7.5. This is a huge deal as my other one wouldn't stay up for 5 minutes on any firmware or Unifi version.

 

I really think it had something to do with the hardware. 

 

Are y'all seeing Java errors. I had a bunch of hs_err_pidXXXX.log files. I'd get class not found exceptions, etc. @UBNT-Corey and I were working on seeing if those logs could pinpoint the issue. I don't have any updates from my side since the new key has been operating perfect. I wish there was something I could do to help out more I'm not seeing any errors.

 

EDIT:

Looks like my post with the screen shot is back. Are y'all seeing the error then reboot loop in your logs?

Reply