02-11-2019 02:10 PM
Sorry I have had to flatten the server and roll It back to 3.9.12. I really couldn’t wait for a fix it’s a security platform, I need it to be stable...
Please keep me updated and I’ll revisit the update when the problems have been ironed out.
02-11-2019 03:02 PM - edited 02-11-2019 03:03 PM
02-12-2019 03:52 PM
@Johno2518 as per your message, I can see some issues in the older logs with EMS and mongo, but I'm not seeing any errors in the latest ones.
If it creeps up again, take a look and see if mongo aborted again. I'd like a look if so.
@benwilcock let me know if you encounter this going forward.
02-12-2019 04:39 PM
@UBNT-claw yeah since disabling and re-enabling recordings the NVR is working OK now. The FPS was incorrect and like before i made a multi-camera chance to make it work @ 25FPS (i know there is a UI bug but wanted to see if re-changing it would trigger).
I will keep an eye on it as the first time everything ran for a little while before progressively failing, after than it would fail fairly quickly.
I will keep monitoring and post updates. After a day i will reboot the NVR to see if that process does something.
If the NVR is stable no matter what after this point, the only thing that makes sense is that post upgrade something in the camera recording configuration is not right and the workaround (at this stage):
- Reboot to ensure NVR starts up cleanly
- Disable recording on all cameras
- Delete all existing recordings
- Run a DB analysis (it may cleanup stale records etc. from the recordings depending on their state)
- Deleted all the logs via UI (I did this to get a clean start however this is not necessary)
- Re-enable recording on all cameras
If someone is having the same issues and finds rebooting temporarily helps, try the above and post results to see if you get a stable NVR.
Will post more over the next couple of days, in the mean time the older logs have the details of the issues as you mentioned and should be a good point for the engineers to review and try to see if they can pin point potential code problems. At the moment the issue appears to be with recording configuration and recording processing (something seems to get corrupted along the way).
02-13-2019 03:18 PM
I have sent another DM, NVR recording has stopped unfortunately. I am not getting the API error 400 (yet).
I can view live video via a browser (albeit it choppy - should be 25FPS but its showing like its 10-15FPS and this is the same in Microsoft Edge and Google Chrome) - no audio.
The Android app briefly shows 1 frame and then nothing (its just blank, no audio either).
02-15-2019 03:57 PM
@UBNT-claw has there been any progress made? NVR web page completely not available now. I willl reboot and get logs
If i can't get a resolution soon (being a security platform) i will need to rollback as i need a working system. It's been too long already without any resolution.
I'm very surprised at how slow this process has been and would think there would be quite a lot of effort to address major issues with an upcoming release - it's not BETA and while it is a soft release would expect to have more attention on such a major defect for a version essentially a production release.
Let me know either way, need to make a decision on what i will do soon as i need the system working and recording ASAP!
2 weeks ago
I've had the same issue since upgrade to 3.10.1. What I am noticing is evostream begins to climb and eventually runs away with memory usage...the disk can't keep up with it. My server has 32GB ram and 18TB raid 5 hard drive space. It dies with 27 cameras recording motion only after about 1 hour. Switch ports still show traffic flowing, so it is going into server memory. If I turn recordings off, the evostream starts to fall. It will be interesting to see if the recordings make it to the disk after NVR comes back. Will post later.
2 weeks ago
After stopping all cameras from recording, memory eventually went back to normal. Health status still showed red, complaing of NVR off line. Restarted Unifi video service, all cameras came back and health status shows green...but lost all recordings from prior when health status red...i guess the data stream was just going to null. Tried rolling back to 3.9.12, but that version won't recognize cameras because of the camera fimware upgrades I did after upgrading to 3.10. This is a pretty sticky wicket Ubiquiti. You're better than this. Lots of surveillance at risk here. Please fix.
2 weeks ago
a week ago
What I think was happening is that the EMS or other processes were making connections to both instances which was causing some kind of corruption.
@UBNT-claw killed that process and I have since patched / rebooted the server and the rogue MongoDB process has not come back. System has been stable, still seeing some delayed segments ~470 over the two weeks but all segments have been processed, no empty recordings and new status indicators are all green / 100%.
If you are still having issues, check for rogue MongoDB processes (they won't be running under the unifi-video account) and kill them (assuming you are not using MongoDB for other services - if you are, you will need to check the parameters they were started with to see which service they belong to).
Please post what you find and if this fixes the issue for you (assuming you have rogue MongoDB processes).
a week ago - last edited a week ago
have the same issue.
first the upgrade failed, Again and like it usually does.
after a fresh install when recovering the footage it deleted most of them...
after adopting the camera's and setting the settings it deleted all of the remaining recordings...
Now it does not record.
says that the CPU/DISK or network utilization is to high or network is unhealthy.
CPU usage is less then 20% There are constant write activities on the RAID 10 array but no recordings are being made.)this disk has recordings only), network is hardly being used and is definitly healthy, can watch live stream and all 23 cams show streams in the Live View.
Performance counters on the unifi video weblogin are all blank.
I am not able to SSH or login (web) into any of the camera's using the creds i specified.
Is there anything that can fix it so it records again, (what it is supposed to be doing)
a week ago
I am experiencing similar issues with our NVR going offline. Is it possible for me to share a support file with a Ubiquiti employee?
We are running a Ubuntu server 16.04 lts, on Microsoft hyper v. I have it set up with 2TB storage with 16GB ram and 4 cores at 2.5ghz. Running 27 cameras at constant record during business hours and on motion the remainder of the day and night.
I was able to "stabilize" the nvr by disconnecting it from my unifi cloud account. I noticed that each time I logged in via cloud portal the unifi service would go offline. I have attached a screenshot of the alert.