Member
Posts: 180
Registered: ‎08-08-2013
Kudos: 37
Solutions: 2

Cameras disconnect, recordings stop

[ Edited ]

Strange issue that just popped up on an otherwise stable system. Cameras are working OK and then after a couple/few days the status changes to yellow/managing and none of the cameras are recording. The only alerts logged are "Temporary video cache space is 90% full, some segments were not processed" and "Temporary space was restored. Currant usage: 1%" Rebooting the server brings everything back for a bit. 

 

The only thing that's changed is that we added 4 cameras to the system mid-last year. But this issue seemed to just pop up recently. 

 

UniFi Video: v3.9.12

 

Camera firmware: v4.4.8

 

Running on: PowerEdge R330, Xeon E3-1220, 32GB RAM

 

OS: Debian 9 (stretch)

 

Storage: 3TB set to store as mucs as possile while leaving 500GB free.

 

unifi@svr-unifi-01:~$ df -H
Filesystem      Size  Used Avail Use% Mounted on
udev             17G     0   17G   0% /dev
tmpfs           3.4G  9.0M  3.4G   1% /run
/dev/sda2        83G   11G   69G  14% /
tmpfs            17G     0   17G   0% /dev/shm
tmpfs           5.3M     0  5.3M   0% /run/lock
tmpfs            17G     0   17G   0% /sys/fs/cgroup
/dev/sda1       536M  148k  536M   1% /boot/efi
/dev/sdb        3.0T  2.2T  657G  77% /storage
tmpfs           5.1G  3.6M  5.1G   1% /var/cache/unifi-video
tmpfs           3.4G     0  3.4G   0% /run/user/1000

Any help in tracking down the issue would be appreciated.

 

New Member
Posts: 3
Registered: ‎02-05-2019

Re: Cameras disconnect, recordings stop

[ Edited ]

I am having sudden constant issues with freezes and camera disconnects as well (seemingly out of nowhere). My setup is the latest firmware and software while running the Windows version of the controller on its own physical server with loads of RAM. My store for recording is something like 12TB and is nowhere close to being full. The system has only 4 cameras total, and is monitoring 3 on motion only. I'm also using the recommended RAM disk software as a recording cache. 

 

I've tried a few things - a daily autoreboot of the server, updating from 3.9.12 controller to 3.10.0, checking to make sure the disk array for recording is not degraded,  and turning off all recording completely. Nothing is seeming to help at this point. I can upload my support file if that will help. 

Member
Posts: 180
Registered: ‎08-08-2013
Kudos: 37
Solutions: 2

Re: Cameras disconnect, recordings stop

This issue issue is still occuring. Anyone for UBNT care to offer some help?

New Member
Posts: 3
Registered: ‎02-05-2019

Re: Cameras disconnect, recordings stop

Not with UBNT, but had these same sort of issues. Turning off recording ended up solving my issue. I am awaiting a fix for this, because recording is vital to us.

 

It was covered alot in this thread - https://community.ubnt.com/t5/UniFi-Video/Cameras-Offline-and-not-recording-after-upgrade-to-3-10-1/...

Ubiquiti Employee
Posts: 4,666
Registered: ‎06-18-2015
Kudos: 1425
Solutions: 436

Re: Cameras disconnect, recordings stop

@rocketboy001,

Please send me your support file using this link: https://goo.gl/forms/MYfrkdZHotgoQPRY2

What is your storage architecture?

UBNT_Alternate_Logo.png
Ubiquiti Networks Enterprise Support Team


UniFi Protect: UniFi Protect Help Center | Frequently Asked Questions

UniFi Video 3: UniFi Video Help Center | UFV3 User Guide


Member
Posts: 180
Registered: ‎08-08-2013
Kudos: 37
Solutions: 2

Re: Cameras disconnect, recordings stop

 

@UBNT-Cody 

 

Sent you the support file. 

 

Storrage is simple. Video is stored on a RAID 5 array (using a PERC H730 controller and Seagate ST1000NX0423 drives). OS is on a RAID 1, same controller but using SSDs.

 

Ubiquiti Employee
Posts: 4,666
Registered: ‎06-18-2015
Kudos: 1425
Solutions: 436

Re: Cameras disconnect, recordings stop

@rocketboy001 ,

 

The logs (which span the last 3-4 days or so) all indicate that each service restart the media server is unable to bind to the proper port.  Has the server been rebooted recently?  Can you please reboot the server, let the service come up and run for an hour or so, then send me a new support file?

UBNT_Alternate_Logo.png
Ubiquiti Networks Enterprise Support Team


UniFi Protect: UniFi Protect Help Center | Frequently Asked Questions

UniFi Video 3: UniFi Video Help Center | UFV3 User Guide


Member
Posts: 180
Registered: ‎08-08-2013
Kudos: 37
Solutions: 2

Re: Cameras disconnect, recordings stop

@UBNT-Cody

Yeah, I rebooted it this AM before I sent the log as it was inop (camera status was yellow) and rebooting the server is the only way I've found to bring it back. I can reboot it again and send the support file.
Member
Posts: 180
Registered: ‎08-08-2013
Kudos: 37
Solutions: 2

Re: Cameras disconnect, recordings stop

@UBNT-Cody

 

New support file uploaded. This is after a reboot and running for just over an hour.

Ubiquiti Employee
Posts: 4,666
Registered: ‎06-18-2015
Kudos: 1425
Solutions: 436

Re: Cameras disconnect, recordings stop

@rocketboy001,

Just to confirm, are the cameras now recording after the reboot? The logs suggest they are.

If so, can you please monitor the system and take a support file at least once a day until the service fails again? What I'm looking for is logs generated prior to the crash, that might show if there's an issue that is eventually causing the system to fail. It will also help to get the output of `free -h` and `df -h` to see how the system resources are being handled.

The alerts about temporary cache full suggest the system reaches a point where it's failing to write video segments to permanent storage (the video drive), or they aren't being processed correctly. There's a variety of reasons this could be happening. One possibility is IOPS exhaustion as RAID5 is not recommended due to the hefty write performance penalty. The other is that the system is running out of resources caused by running an unsupported OS and database engine.

You may want to upgrade to UniFi Video v3.10.1 - this will provide us with more stats and introduces support for WiredTiger as a storage engine. Prior to performing the upgrade, please be sure to review this thread as it explains what needs to be done to upgrade if the system is already using WiredTiger prior to the official support: https://community.ubnt.com/t5/UniFi-Video/UniFi-Video-3-10-0-and-MongoDB/m-p/2656593#M116394

UBNT_Alternate_Logo.png
Ubiquiti Networks Enterprise Support Team


UniFi Protect: UniFi Protect Help Center | Frequently Asked Questions

UniFi Video 3: UniFi Video Help Center | UFV3 User Guide


Member
Posts: 180
Registered: ‎08-08-2013
Kudos: 37
Solutions: 2

Re: Cameras disconnect, recordings stop

@UBNT-Cody 

 

I can do as you suggest. Note that it's been running fine until or about the 3.9.12 update. I was waiting to update to 3.10 to get this issue sorted as I didn't want to compound things. I'll update and let you know from there.

Member
Posts: 180
Registered: ‎08-08-2013
Kudos: 37
Solutions: 2

Re: Cameras disconnect, recordings stop

@UBNT-Cody 

 

Oh, yes... after reboot recordings work again for a time. 

 

Also, updated to 3.10.1, how long is typical for this thing to go for?

 

Capture.PNG

Member
Posts: 180
Registered: ‎08-08-2013
Kudos: 37
Solutions: 2

Re: Cameras disconnect, recordings stop

[ Edited ]

This is such a joke. Completly hosed now after upgrading to 3.10. 

 

 

Member
Posts: 180
Registered: ‎08-08-2013
Kudos: 37
Solutions: 2

Re: Cameras disconnect, recordings stop

[ Edited ]

The 3.10 update seems to have hosed MongoDB... which also hosed the UniFi wireless install. 

 

Guess I know what I'm doing today. Just hoping that I can reimport the old video after reinstalling the OS, darn lucky that I have a recent controller backup. 

 

All I ask is that my UBNT switches work and that the video server work and that updates dont f-over everything. Is that too much to ask?