Streaming slows down when storage full [ Edited ]

[djenkinssmch]

I've run into an issue where streaming the recordings becomes almost unusable when the hard drives fill up and the deleting begins. I've currently got 101 cameras adopted, 16 of which not recording. Attached is a webmin screenshot from my server and one from Unifi Video System Configuration. When someone tries to watch recordings, it is hit and miss on if they will be able to without it endlessly buffering. The way I described it to a coworker, it is like the server isn't even breaking a sweat, not even jogging, just having trouble going to and from the filing cabinet, trying to stuff more in while yanking the old ones. It's running a raid 10 of 6x4tb drives to achieve this. Can you help?

[donwschultz · Accepted solution]

Yup, I think you have hit an I/O bottleneck at the disk level.

I assume you must keep 3 months of footage on hand for contractual or legal reasons?

My general rule of thumb is to never fill up any disk, RAID array, SSD or any other storage device past 70%.  Your RAID array has likely run out of "play room" in order to get the new recordings onto disk faster than the need to delete the old footage rolling off.  Those drives are crazy busy right now trying to balance that with streaming as well.  As you get nearer to the outer zones of the platters, your I/O is going to suffer as the RPM rating is actually at the spindle itself...things like data density and rotational speed are lower towards the edges of the platters.  Throw in some fragmentation and you have just landed in hell.  Throw in a dash of non-Enterprise level drives and keeping them operating like this will cause you hardware failures.

The only solutions I can see are to:

Reduce retention time if possible, OR

Find out if some of your cams can be changed to Motion Only if feasible, OR

Determine if FPS or bit rate reduction is appropriate for some or all cams, OR

Failing all of that add more spindles to the array

[donwschultz · Accepted solution]

Yup, I think you have hit an I/O bottleneck at the disk level.

I assume you must keep 3 months of footage on hand for contractual or legal reasons?

My general rule of thumb is to never fill up any disk, RAID array, SSD or any other storage device past 70%.  Your RAID array has likely run out of "play room" in order to get the new recordings onto disk faster than the need to delete the old footage rolling off.  Those drives are crazy busy right now trying to balance that with streaming as well.  As you get nearer to the outer zones of the platters, your I/O is going to suffer as the RPM rating is actually at the spindle itself...things like data density and rotational speed are lower towards the edges of the platters.  Throw in some fragmentation and you have just landed in hell.  Throw in a dash of non-Enterprise level drives and keeping them operating like this will cause you hardware failures.

The only solutions I can see are to:

Reduce retention time if possible, OR

Find out if some of your cams can be changed to Motion Only if feasible, OR

Determine if FPS or bit rate reduction is appropriate for some or all cams, OR

Failing all of that add more spindles to the array

[donwschultz]

On a side note, it is too bad we don't have the ability to utilize a two-tier storage system in UFV3.  The first tier would be for, let's say, 31 days of footage and on the 1st or "hot" tier.  You could jam that onto a SSD for extreme speed, for example.  There are SSDs like the Samsung SM861a that are ideally suited for write intensive environments where endurance is measured in Petabytes.  I actually have a site where I make use of two of them in a RAID0 array and boy are they fast.   Never a hiccup with those things.  Even in this scenario I won't let utilization go past 50% space.  The cost of entry, however....

Anything from 32 to 93 days would be shuttled off to the 2nd tier or "cold" storage, where access is less likely and you can use less elaborate RAID technology.

[UBNT-Cody]

You will want far more free space than that for such a large array. The storage needs room to breathe... if the disk is too full, it won't be efficient at allocating new writes. I typically like to retain between 2% to 5% free space for ext4 storage with UFV3 - NTFS requires even often more. Don't forget that tune2fs needs to be set accordingly as well.

[djenkinssmch]

We do want three month, but at the moment we are only getting one out out the storage. They are already all set to motion only at medium. However, at any given moment, at least a quarter will be recording. So yay. Time to see if I can sell both a massive hdd upgrade and a second server to split the io.

[UBNT-Cody]

@djenkinssmch,

If you haven't already, consider setting Minimum Motion Event Trigger to 1s or 2s, which will greatly reduce false positives, and try enabling adaptive bitrate, which can also reduce storage requirements considerably.