02-05-2019 11:15 AM - edited 02-05-2019 11:22 AM
Strange issue that just popped up on an otherwise stable system. Cameras are working OK and then after a couple/few days the status changes to yellow/managing and none of the cameras are recording. The only alerts logged are "Temporary video cache space is 90% full, some segments were not processed" and "Temporary space was restored. Currant usage: 1%" Rebooting the server brings everything back for a bit.
The only thing that's changed is that we added 4 cameras to the system mid-last year. But this issue seemed to just pop up recently.
UniFi Video: v3.9.12
Camera firmware: v4.4.8
Running on: PowerEdge R330, Xeon E3-1220, 32GB RAM
OS: Debian 9 (stretch)
Storage: 3TB set to store as mucs as possile while leaving 500GB free.
unifi@svr-unifi-01:~$ df -H Filesystem Size Used Avail Use% Mounted on udev 17G 0 17G 0% /dev tmpfs 3.4G 9.0M 3.4G 1% /run /dev/sda2 83G 11G 69G 14% / tmpfs 17G 0 17G 0% /dev/shm tmpfs 5.3M 0 5.3M 0% /run/lock tmpfs 17G 0 17G 0% /sys/fs/cgroup /dev/sda1 536M 148k 536M 1% /boot/efi /dev/sdb 3.0T 2.2T 657G 77% /storage tmpfs 5.1G 3.6M 5.1G 1% /var/cache/unifi-video tmpfs 3.4G 0 3.4G 0% /run/user/1000
Any help in tracking down the issue would be appreciated.
02-05-2019 12:40 PM - edited 02-05-2019 12:47 PM
I am having sudden constant issues with freezes and camera disconnects as well (seemingly out of nowhere). My setup is the latest firmware and software while running the Windows version of the controller on its own physical server with loads of RAM. My store for recording is something like 12TB and is nowhere close to being full. The system has only 4 cameras total, and is monitoring 3 on motion only. I'm also using the recommended RAM disk software as a recording cache.
I've tried a few things - a daily autoreboot of the server, updating from 3.9.12 controller to 3.10.0, checking to make sure the disk array for recording is not degraded, and turning off all recording completely. Nothing is seeming to help at this point. I can upload my support file if that will help.
02-13-2019 10:10 AM
Not with UBNT, but had these same sort of issues. Turning off recording ended up solving my issue. I am awaiting a fix for this, because recording is vital to us.
It was covered alot in this thread - https://community.ubnt.com/t5/UniFi-Video/Cameras-Offline-and-not-recording-after-upgrade-to-3-10-1/...
02-14-2019 07:20 AM
Please send me your support file using this link: https://goo.gl/forms/MYfrkdZHotgoQPRY2
What is your storage architecture?
02-14-2019 10:29 AM
Sent you the support file.
Storrage is simple. Video is stored on a RAID 5 array (using a PERC H730 controller and Seagate ST1000NX0423 drives). OS is on a RAID 1, same controller but using SSDs.
02-14-2019 01:03 PM
The logs (which span the last 3-4 days or so) all indicate that each service restart the media server is unable to bind to the proper port. Has the server been rebooted recently? Can you please reboot the server, let the service come up and run for an hour or so, then send me a new support file?
02-14-2019 01:28 PM
Yeah, I rebooted it this AM before I sent the log as it was inop (camera status was yellow) and rebooting the server is the only way I've found to bring it back. I can reboot it again and send the support file.
02-14-2019 03:41 PM
Just to confirm, are the cameras now recording after the reboot? The logs suggest they are.
If so, can you please monitor the system and take a support file at least once a day until the service fails again? What I'm looking for is logs generated prior to the crash, that might show if there's an issue that is eventually causing the system to fail. It will also help to get the output of `free -h` and `df -h` to see how the system resources are being handled.
The alerts about temporary cache full suggest the system reaches a point where it's failing to write video segments to permanent storage (the video drive), or they aren't being processed correctly. There's a variety of reasons this could be happening. One possibility is IOPS exhaustion as RAID5 is not recommended due to the hefty write performance penalty. The other is that the system is running out of resources caused by running an unsupported OS and database engine.
You may want to upgrade to UniFi Video v3.10.1 - this will provide us with more stats and introduces support for WiredTiger as a storage engine. Prior to performing the upgrade, please be sure to review this thread as it explains what needs to be done to upgrade if the system is already using WiredTiger prior to the official support: https://community.ubnt.com/t5/UniFi-Video/UniFi-Video-3-10-0-and-MongoDB/m-p/2656593#M116394
02-14-2019 03:47 PM
I can do as you suggest. Note that it's been running fine until or about the 3.9.12 update. I was waiting to update to 3.10 to get this issue sorted as I didn't want to compound things. I'll update and let you know from there.
02-15-2019 08:55 AM - edited 02-15-2019 09:30 AM
The 3.10 update seems to have hosed MongoDB... which also hosed the UniFi wireless install.
Guess I know what I'm doing today. Just hoping that I can reimport the old video after reinstalling the OS, darn lucky that I have a recent controller backup.
All I ask is that my UBNT switches work and that the video server work and that updates dont f-over everything. Is that too much to ask?