Upcoming Maintenance Alert:

The UBNT Community will be upgraded at 5pm MDT on April 25th. During this time the community forums will be set to read-only status.

Learn more

×
Reply
Regular Member
Posts: 351
Registered: ‎07-06-2012
Kudos: 35
Solutions: 8

NVR 3.7.1 Full Time Recordings

I'm having issues with Full Time Recordings, typically they are 10 mins long every 10 mins but that isn't happening. Any ideas on what is causing this?

NVR FullTime Recordings.jpg

SuperUser
Posts: 13,716
Registered: ‎08-07-2008
Kudos: 6030
Solutions: 402
Contributions: 1

Re: NVR 3.7.1 Full Time Recordings

Check the connection log for this camera. 

Walks in, tells it like it is, walks out , milks a cow
Regular Member
Posts: 466
Registered: ‎01-08-2015
Kudos: 64
Solutions: 7

Re: NVR 3.7.1 Full Time Recordings

I would double check the record settings per camera.  Seems like I can see someone walking in the longer recordings, making me believe it's still being held on motion only.

Regular Member
Posts: 351
Registered: ‎07-06-2012
Kudos: 35
Solutions: 8

Re: NVR 3.7.1 Full Time Recordings

[ Edited ]

This site is my largest, with 98 cameras but we have a big server to run the NVR. All the cameras are setup on a Recording Schedule, Motion Only except for three blocks of times which are Always Record. This seems to be happening on all cameras but I'm not seeing any similar issues on other sites running either 3.7.0 or 3.7.1. Here a section of the connection log for that camera from a time when it should be Always Record.

 

1496276984.052 2017-05-31 19:29:44.052/CDT: INFO   Camera[802AA84EE033:4341] Connection closed (Gym) in tomcat-HTTPS-exec-25
1496276984.144 2017-05-31 19:29:44.144/CDT: INFO   Camera[802AA84EE033:439a] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-50
1496277004.431 2017-05-31 19:30:04.431/CDT: INFO   Camera[802AA84EE033:439a] Connection closed (Gym) in tomcat-HTTPS-exec-95
1496277004.519 2017-05-31 19:30:04.519/CDT: INFO   Camera[802AA84EE033:43e6] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-19
1496277047.144 2017-05-31 19:30:47.144/CDT: INFO   Camera[802AA84EE033:43e6] Connection closed (Gym) in tomcat-HTTPS-exec-39
1496277047.235 2017-05-31 19:30:47.235/CDT: INFO   Camera[802AA84EE033:4473] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-68
1496277085.694 2017-05-31 19:31:25.694/CDT: INFO   Camera[802AA84EE033:4473] Connection closed (Gym) in tomcat-HTTPS-exec-27
1496277085.789 2017-05-31 19:31:25.789/CDT: INFO   Camera[802AA84EE033:44c1] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-96
1496277106.080 2017-05-31 19:31:46.080/CDT: INFO   Camera[802AA84EE033:44c1] Connection closed (Gym) in tomcat-HTTPS-exec-36
1496277106.175 2017-05-31 19:31:46.175/CDT: INFO   Camera[802AA84EE033:450f] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-66
1496277126.449 2017-05-31 19:32:06.449/CDT: INFO   Camera[802AA84EE033:450f] Connection closed (Gym) in tomcat-HTTPS-exec-100
1496277126.660 2017-05-31 19:32:06.660/CDT: INFO   Camera[802AA84EE033:456e] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-63
1496277146.944 2017-05-31 19:32:26.944/CDT: INFO   Camera[802AA84EE033:456e] Connection closed (Gym) in tomcat-HTTPS-exec-74

1496277147.037 2017-05-31 19:32:27.037/CDT: INFO   Camera[802AA84EE033:45a9] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-73
1496277167.319 2017-05-31 19:32:47.319/CDT: INFO   Camera[802AA84EE033:45a9] Connection closed (Gym) in tomcat-HTTPS-exec-86
1496277167.410 2017-05-31 19:32:47.410/CDT: INFO   Camera[802AA84EE033:45f5] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-25
1496277233.424 2017-05-31 19:33:53.424/CDT: INFO   Camera[802AA84EE033:45f5] Connection closed (Gym) in tomcat-HTTPS-exec-45
1496277233.513 2017-05-31 19:33:53.513/CDT: INFO   Camera[802AA84EE033:46b9] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-17
1496277254.276 2017-05-31 19:34:14.276/CDT: INFO   Camera[802AA84EE033:46b9] Connection closed (Gym) in tomcat-HTTPS-exec-94
1496277254.345 2017-05-31 19:34:14.345/CDT: INFO   Camera[802AA84EE033:4719] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-94
1496277276.134 2017-05-31 19:34:36.134/CDT: INFO   Camera[802AA84EE033:4736] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-9
1496277276.211 2017-05-31 19:34:36.211/CDT: INFO   Camera[802AA84EE033:4719] Connection closed (Gym) in tomcat-HTTPS-exec-71
1496277296.247 2017-05-31 19:34:56.247/CDT: INFO   Camera[802AA84EE033:4736] Connection closed (Gym) in tomcat-HTTPS-exec-8

1496277296.405 2017-05-31 19:34:56.405/CDT: INFO   Camera[802AA84EE033:476c] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-91
1496277316.706 2017-05-31 19:35:16.706/CDT: INFO   Camera[802AA84EE033:476c] Connection closed (Gym) in tomcat-HTTPS-exec-7
1496277316.796 2017-05-31 19:35:16.796/CDT: INFO   Camera[802AA84EE033:47c1] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-100
1496277337.120 2017-05-31 19:35:37.120/CDT: INFO   Camera[802AA84EE033:47c1] Connection closed (Gym) in tomcat-HTTPS-exec-42
1496277337.216 2017-05-31 19:35:37.216/CDT: INFO   Camera[802AA84EE033:482a] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-71
1496277357.527 2017-05-31 19:35:57.527/CDT: INFO   Camera[802AA84EE033:482a] Connection closed (Gym) in tomcat-HTTPS-exec-27
1496277357.616 2017-05-31 19:35:57.616/CDT: INFO   Camera[802AA84EE033:4865] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-5

 

Regular Member
Posts: 351
Registered: ‎07-06-2012
Kudos: 35
Solutions: 8

Re: NVR 3.7.1 Full Time Recordings

[ Edited ]

Here's a more recent log from the last 24 hours ...

 

1497482122.614 2017-06-14 18:15:22.614/CDT: INFO   Camera[802AA84EE033:1965] Connection closed (Gym) in HeartBeat-Timer
1497485925.676 2017-06-14 19:18:45.676/CDT: INFO   Camera[802AA84EE033:c] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-23
1497735221.439 2017-06-17 16:33:41.439/CDT: INFO   Camera[802AA84EE033:c] Connection closed (Gym) in HeartBeat-Timer
1497735251.886 2017-06-17 16:34:11.886/CDT: INFO   Camera[802AA84EE033:10d] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-77
1497736026.896 2017-06-17 16:47:06.896/CDT: INFO   Camera[802AA84EE033:10d] Connection closed (Gym) in HeartBeat-Timer
1497736057.257 2017-06-17 16:47:37.257/CDT: INFO   Camera[802AA84EE033:2de] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-86
1497736096.074 2017-06-17 16:48:16.074/CDT: INFO   Camera[802AA84EE033:2de] Connection closed (Gym) in tomcat-HTTPS-exec-13
1497736096.154 2017-06-17 16:48:16.154/CDT: INFO   Camera[802AA84EE033:32b] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-16
1497736116.156 2017-06-17 16:48:36.156/CDT: INFO   Camera[802AA84EE033:32b] Connection closed (Gym) in HeartBeat-Timer
1497736127.587 2017-06-17 16:48:47.587/CDT: INFO   Camera[802AA84EE033:370] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-55
1497736147.587 2017-06-17 16:49:07.587/CDT: INFO   Camera[802AA84EE033:370] Connection closed (Gym) in HeartBeat-Timer
1497736157.831 2017-06-17 16:49:17.831/CDT: INFO   Camera[802AA84EE033:3b2] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-90
1497736411.326 2017-06-17 16:53:31.326/CDT: INFO   Camera[802AA84EE033:26] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-8

1497746131.539 2017-06-17 19:35:31.539/CDT: INFO   Camera[802AA84EE033:26] Connection closed (Gym) in HeartBeat-Timer
1497746145.013 2017-06-17 19:35:45.013/CDT: INFO   Camera[802AA84EE033:106] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-76
1497746165.950 2017-06-17 19:36:05.950/CDT: INFO   Camera[802AA84EE033:106] Connection closed (Gym) in tomcat-HTTPS-exec-79
1497746166.043 2017-06-17 19:36:06.043/CDT: INFO   Camera[802AA84EE033:137] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-1
1497746187.319 2017-06-17 19:36:27.319/CDT: INFO   Camera[802AA84EE033:137] Connection closed (Gym) in tomcat-HTTPS-exec-51
1497746187.408 2017-06-17 19:36:27.408/CDT: INFO   Camera[802AA84EE033:167] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-28
1497799768.384 2017-06-18 10:29:28.384/CDT: INFO   Camera[802AA84EE033:167] Connection closed (Gym) in HeartBeat-Timer
1497799799.004 2017-06-18 10:29:59.004/CDT: INFO   Camera[802AA84EE033:1ef] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-30
1497801089.030 2017-06-18 10:51:29.030/CDT: INFO   Camera[802AA84EE033:1ef] Connection closed (Gym) in HeartBeat-Timer
1497801101.122 2017-06-18 10:51:41.122/CDT: INFO   Camera[802AA84EE033:266] Connection from: 192.168.129.180 (Gym) in tomcat-HTTPS-exec-87

Regular Member
Posts: 351
Registered: ‎07-06-2012
Kudos: 35
Solutions: 8

Re: NVR 3.7.1 Full Time Recordings

[ Edited ]

I switched everything over to Always Record from the Schedule but that didn't seem to make any difference. I switch all cameras to Don't Record but one and it worked correctly, so this doesn't look like an issue either the Schedule or firmware but the number of cameras connected? I know that some people had said that 50 was an issue but we haven't had a problem before at this site with 3.6.1 or 3.6.3 (we intentionally skipped 3.6.2). We upgraded to 3.7.1 but it appears like this was an issue before 3.7.1. I'm going to run it with 50 cameras and see what results I get but the machine should easily be able to handle more, it's a Dell PowerEdge T130 with Xeon E3-1230 processor, 32 GB memory, 4x 4 TB 7200 RPM drives in a ZFS array (performance was clocked at 2.5 GB/s).

Regular Member
Posts: 466
Registered: ‎01-08-2015
Kudos: 64
Solutions: 7

Re: NVR 3.7.1 Full Time Recordings

@chmcwill What OS are you running?

 

We have 4 Ubiquiti NVR's, and one Windows Server 2012 instance.  I am thinking about building a large Ubuntu instance or attempting to run on our Synology but not sure how that will work.

Established Member
Posts: 1,348
Registered: ‎07-11-2011
Kudos: 276
Solutions: 42

Re: NVR 3.7.1 Full Time Recordings

I don't have any way to test my theory, but is it possible that you don't have enough IOPS to handle that many cameras recording continuously? I don't think bandwidth should be a problem even though I can't imagine you're actually getting 2.5GB/s out of 4 drives. Surely you have at least one for redundancy. So that that would be maybe 600-750MB/s. Regardless, that should be plenty of bandwidth. However with 98 cameras you'll have 98 roughly 5Mb or less streams trying to all write to the array at the same time. I suspect that's going to require a lot of iops and seeking. I'd have to read up on ZFS to know if adding a cahce drive could mitigate that or not, but I'd look at optimizing the array. Also you might want to check the specs on the drives and the temps they run at to see if any kind of thermal calibration could be taking place. I haven't had to worry about that in a while, but I know that used to be a big issue with video drives in the past.

Regular Member
Posts: 351
Registered: ‎07-06-2012
Kudos: 35
Solutions: 8

Re: NVR 3.7.1 Full Time Recordings

[ Edited ]

@dmkjrWe have switched over almost all of our NVR's to Ubuntu 16.04 LTS. This is the only site where we have multiple drives for the recording and chose ZFS for its write performance. We have 4x 4 TB in a stripe.

Regular Member
Posts: 351
Registered: ‎07-06-2012
Kudos: 35
Solutions: 8

Re: NVR 3.7.1 Full Time Recordings

[ Edited ]

@KevoWhen we built the system, we did some performance benchmarks come up with the following. I had issues 30 years ago with thermal recalibration on a video hard drive trying to write to a 1x CD burner but haven't found that to be an issue since. We used the recommended options, making sure that we optimized the ZFS array for video, aka LZ4 compression and Deduplication disabled.

 

ZFS Performance.jpg

Regular Member
Posts: 351
Registered: ‎07-06-2012
Kudos: 35
Solutions: 8

Re: NVR 3.7.1 Full Time Recordings

I'm guessing that we have maxed out this system ...iostat 5 -x shows the following with all cameras at high always recording.

 

NVR Disk Utilization.jpg

Established Member
Posts: 1,348
Registered: ‎07-11-2011
Kudos: 276
Solutions: 42

Re: NVR 3.7.1 Full Time Recordings

I see that dd reported 2.5GB/s, but that doesn't mean it's actually doing that. Look at specs of the drives you used and see what sort of write speed they claim. I think something in the neighborhood of 200MB/s is reasonable for a 4TB SATA drive. You'd have to look at your specific disk and benchmarks of that disk to see what it is in the real world.

 

Even if it was 500Mb/s which is SSD speed, that would still be 2GB/s if the whole array was striped with no redundancy. 

 

Anyway, I think you are right that your particular setup is maxxed out. It might work properly with more drives in the array to spread the load, but you'd need to really sort out what the actual performance is and how much bandwidth and IOPS you need for 100 cameras. That would require some testing as I don't think you will get that info from UBNT as it will vary quite a bit from system to system.

Regular Member
Posts: 351
Registered: ‎07-06-2012
Kudos: 35
Solutions: 8

Re: NVR 3.7.1 Full Time Recordings

I was able to tweek the array and managed to get it running so far without any issues. Here's the highest writes I've noticed and the %util is not maxing out.

 

NVR Disk Utilization 2.jpg

Highlighted
Established Member
Posts: 1,348
Registered: ‎07-11-2011
Kudos: 276
Solutions: 42

Re: NVR 3.7.1 Full Time Recordings

Nice work. So what kind of tweaks did you apply. Might be helpful for others who would like to use a ZFS or other NAS array for running a lot of cameras. I've thought many times myself about using a FreeNAS server for this sort of thing as well as possibly storing VMs on it. Been looking for a relatively inexpensive way to get something running with 10Gbps ports.
Regular Member
Posts: 351
Registered: ‎07-06-2012
Kudos: 35
Solutions: 8

Re: NVR 3.7.1 Full Time Recordings

I had intentionally used 4x 4 TB HGST Desktor NAS 7200 RPM drives in this server to get better write performance, we chose ZFS in a Stripe with LZ4 compression enabled and Deduplication disabled. It truned out that all the drives has write back caching disabled, so we went ahead to turn that on and well, you can see the difference.

 

Established Member
Posts: 1,348
Registered: ‎07-11-2011
Kudos: 276
Solutions: 42

Re: NVR 3.7.1 Full Time Recordings

That makes sense. I think some sites push turning that off with ZFS unless everything is backed by UPS of some sort and setup for proper power loss shutdown. Since you are only doing video on this array, it's probably not nearly as big a deal although I do wonder what would happen to the NVR db if power were to fail. I haven't had to deal with any db corruption on my 2 systems yet, but I've seen some posts about it in the forums, so it may be a concern.
Regular Member
Posts: 351
Registered: ‎07-06-2012
Kudos: 35
Solutions: 8

Re: NVR 3.7.1 Full Time Recordings

Yes, we have a brand new 1500va batter backup at this location and intentionally used a ZFS stipe for performance and understand that there is no redundancy in the system especially with enabling write back cache on the drives, if power is lost there is the possibility for corruption. I've not had any issues with mogod database corruption, so the plan for this site is to reinstall if we have any failure.

Reply