Reply
Highlighted
New Member
Posts: 12
Registered: ‎06-14-2016
Kudos: 2
Accepted Solution

Streaming slows down when storage full

[ Edited ]

I've run into an issue where streaming the recordings becomes almost unusable when the hard drives fill up and the deleting begins. I've currently got 101 cameras adopted, 16 of which not recording. Attached is a webmin screenshot from my server and one from Unifi Video System Configuration. When someone tries to watch recordings, it is hit and miss on if they will be able to without it endlessly buffering. The way I described it to a coworker, it is like the server isn't even breaking a sweat, not even jogging, just having trouble going to and from the filing cabinet, trying to stuff more in while yanking the old ones. It's running a raid 10 of 6x4tb drives to achieve this. Can you help?

2018-10-31 14_22_10-UniFi Video.png

 

2018-10-31 14_21_15-Webmin 1.890 on Cameras710 (Ubuntu Linux 16.04.4).png


Accepted Solutions
Regular Member
Posts: 676
Registered: ‎05-08-2015
Kudos: 217
Solutions: 17

Re: Streaming slows down when storage full

Yup, I think you have hit an I/O bottleneck at the disk level.

 

I assume you must keep 3 months of footage on hand for contractual or legal reasons?

 

My general rule of thumb is to never fill up any disk, RAID array, SSD or any other storage device past 70%.  Your RAID array has likely run out of "play room" in order to get the new recordings onto disk faster than the need to delete the old footage rolling off.  Those drives are crazy busy right now trying to balance that with streaming as well.  As you get nearer to the outer zones of the platters, your I/O is going to suffer as the RPM rating is actually at the spindle itself...things like data density and rotational speed are lower towards the edges of the platters.  Throw in some fragmentation and you have just landed in hell.  Throw in a dash of non-Enterprise level drives and keeping them operating like this will cause you hardware failures.

 

The only solutions I can see are to:

 

Reduce retention time if possible, OR

Find out if some of your cams can be changed to Motion Only if feasible, OR

Determine if FPS or bit rate reduction is appropriate for some or all cams, OR

Failing all of that add more spindles to the array

View solution in original post


All Replies
Regular Member
Posts: 676
Registered: ‎05-08-2015
Kudos: 217
Solutions: 17

Re: Streaming slows down when storage full

Yup, I think you have hit an I/O bottleneck at the disk level.

 

I assume you must keep 3 months of footage on hand for contractual or legal reasons?

 

My general rule of thumb is to never fill up any disk, RAID array, SSD or any other storage device past 70%.  Your RAID array has likely run out of "play room" in order to get the new recordings onto disk faster than the need to delete the old footage rolling off.  Those drives are crazy busy right now trying to balance that with streaming as well.  As you get nearer to the outer zones of the platters, your I/O is going to suffer as the RPM rating is actually at the spindle itself...things like data density and rotational speed are lower towards the edges of the platters.  Throw in some fragmentation and you have just landed in hell.  Throw in a dash of non-Enterprise level drives and keeping them operating like this will cause you hardware failures.

 

The only solutions I can see are to:

 

Reduce retention time if possible, OR

Find out if some of your cams can be changed to Motion Only if feasible, OR

Determine if FPS or bit rate reduction is appropriate for some or all cams, OR

Failing all of that add more spindles to the array

Regular Member
Posts: 676
Registered: ‎05-08-2015
Kudos: 217
Solutions: 17

Re: Streaming slows down when storage full

On a side note, it is too bad we don't have the ability to utilize a two-tier storage system in UFV3.  The first tier would be for, let's say, 31 days of footage and on the 1st or "hot" tier.  You could jam that onto a SSD for extreme speed, for example.  There are SSDs like the Samsung SM861a that are ideally suited for write intensive environments where endurance is measured in Petabytes.  I actually have a site where I make use of two of them in a RAID0 array and boy are they fast.   Never a hiccup with those things.  Even in this scenario I won't let utilization go past 50% space.  The cost of entry, however....

 

Anything from 32 to 93 days would be shuttled off to the 2nd tier or "cold" storage, where access is less likely and you can use less elaborate RAID technology.

Ubiquiti Employee
Posts: 3,772
Registered: ‎06-18-2015
Kudos: 1152
Solutions: 341

Re: Streaming slows down when storage full

You will want far more free space than that for such a large array. The storage needs room to breathe... if the disk is too full, it won't be efficient at allocating new writes. I typically like to retain between 2% to 5% free space for ext4 storage with UFV3 - NTFS requires even often more. Don't forget that tune2fs needs to be set accordingly as well.

UBNT_Alternate_Logo.png
Ubiquiti Networks Enterprise Support Team


UniFi Protect: UniFi Protect Help Center | Frequently Asked Questions

UniFi Video 3: UniFi Video Help Center | UFV3 User Guide


New Member
Posts: 12
Registered: ‎06-14-2016
Kudos: 2

Re: Streaming slows down when storage full

We do want three month, but at the moment we are only getting one out out the storage. They are already all set to motion only at medium. However, at any given moment, at least a quarter will be recording. So yay. Time to see if I can sell both a massive hdd upgrade and a second server to split the io.

Ubiquiti Employee
Posts: 3,772
Registered: ‎06-18-2015
Kudos: 1152
Solutions: 341

Re: Streaming slows down when storage full

@djenkinssmch,

If you haven't already, consider setting Minimum Motion Event Trigger to 1s or 2s, which will greatly reduce false positives, and try enabling adaptive bitrate, which can also reduce storage requirements considerably.

UBNT_Alternate_Logo.png
Ubiquiti Networks Enterprise Support Team


UniFi Protect: UniFi Protect Help Center | Frequently Asked Questions

UniFi Video 3: UniFi Video Help Center | UFV3 User Guide


Reply