New Member
Posts: 24
Registered: ‎10-03-2017
Kudos: 2

Re: Cameras Offline and not recording after upgrade to 3.10.1

Also, I believe I still have the logs from before I disabled recordings.

 

I wasn't able to attach them to the DM though. 

New Member
Posts: 16
Registered: ‎04-06-2016

Re: Cameras Offline and not recording after upgrade to 3.10.1

Sorry I have had to flatten the server and roll It back to 3.9.12. I really couldn’t wait for a fix it’s a security platform, I need it to be stable...

 

Please keep me updated and I’ll revisit the update when the problems have been  ironed out.

Ubiquiti Employee
Posts: 557
Registered: ‎03-07-2017
Kudos: 65
Solutions: 31

Re: Cameras Offline and not recording after upgrade to 3.10.1

[ Edited ]

Preferably after some time into recordings at least. Go ahead and send me a link to an online drive like dropbox, that's the simplest method.

UBNT_Alternate_Logo.png
Ubiquiti Networks Enterprise Support Team


Check out our ever-evolving UniFi Video Help Center and UFV3 User Guide for answers to many common questions!

New Member
Posts: 24
Registered: ‎10-03-2017
Kudos: 2

Re: Cameras Offline and not recording after upgrade to 3.10.1

@UBNT-clawdone, DM with details

Ubiquiti Employee
Posts: 557
Registered: ‎03-07-2017
Kudos: 65
Solutions: 31

Re: Cameras Offline and not recording after upgrade to 3.10.1

@Johno2518 as per your message, I can see some issues in the older logs with EMS and mongo, but I'm not seeing any errors in the latest ones. 

If it creeps up again, take a look and see if mongo aborted again. I'd like a look if so.

 

@benwilcock let me know if you encounter this going forward.

UBNT_Alternate_Logo.png
Ubiquiti Networks Enterprise Support Team


Check out our ever-evolving UniFi Video Help Center and UFV3 User Guide for answers to many common questions!

New Member
Posts: 24
Registered: ‎10-03-2017
Kudos: 2

Re: Cameras Offline and not recording after upgrade to 3.10.1

@UBNT-claw  yeah since disabling and re-enabling recordings the NVR is working OK now. The FPS was incorrect and like before i made a multi-camera chance to make it work @ 25FPS (i know there is a UI bug but wanted to see if re-changing it would trigger).

 

I will keep an eye on it as the first time everything ran for a little while before progressively failing, after than it would fail fairly quickly.

 

I will keep monitoring and post updates. After a day i will reboot the NVR to see if that process does something.

 

If the NVR is stable no matter what after this point, the only thing that makes sense is that post upgrade something in the camera recording configuration is not right and the workaround (at this stage):

 

  • Reboot to ensure NVR starts up cleanly
  • Disable recording on all cameras
  • Delete all existing recordings
  • Run a DB analysis (it may cleanup stale records etc. from the recordings depending on their state)
  • Reboot
  • Deleted all the logs via UI (I did this to get a clean start however this is not necessary)
  • Re-enable recording on all cameras

 

If someone is having the same issues and finds rebooting temporarily helps, try the above and post results to see if you get a stable NVR.

 

Will post more over the next couple of days, in the mean time the older logs have the details of the issues as you mentioned and should be a good point for the engineers to review and try to see if they can pin point potential code problems. At the moment the issue appears to be with recording configuration and recording processing (something seems to get corrupted along the way).

New Member
Posts: 24
Registered: ‎10-03-2017
Kudos: 2

Re: Cameras Offline and not recording after upgrade to 3.10.1

@UBNT-claw 

I have sent another DM, NVR recording has stopped unfortunately. I am not getting the API error 400 (yet).

I can view live video via a browser (albeit it choppy - should be 25FPS but its showing like its 10-15FPS and this is the same in Microsoft Edge and Google Chrome) - no audio.

The Android app briefly shows 1 frame and then nothing (its just blank, no audio either).

New Member
Posts: 24
Registered: ‎10-03-2017
Kudos: 2

Re: Cameras Offline and not recording after upgrade to 3.10.1

@UBNT-claw  has there been any progress made? NVR web page completely not available now. I willl reboot and get logs

 

If i can't get a resolution soon (being a security platform) i will need to rollback as i need a working system. It's been too long already without any resolution.

 

I'm very surprised at how slow this process has been and would think there would be quite a lot of effort to address major issues with an upcoming release - it's not BETA and while it is a soft release would expect to have more attention on such a major defect for a version essentially a production release.

 

Let me know either way, need to make a decision on what i will do soon as i need the system working and recording ASAP!

New Member
Posts: 24
Registered: ‎10-03-2017
Kudos: 2

Re: Cameras Offline and not recording after upgrade to 3.10.1

I've also disabled recordings again to keep the NVR stable.

New Member
Posts: 9
Registered: ‎05-17-2011
Kudos: 6

Re: Cameras Offline and not recording after upgrade to 3.10.1

I've had the same issue since upgrade to 3.10.1.  What I am noticing is evostream begins to climb and eventually runs away with memory usage...the disk can't keep up with it.  My server has 32GB ram and 18TB raid 5 hard drive space.  It dies with 27 cameras recording motion only after about 1 hour.  Switch ports still show traffic flowing, so it is going into server memory.  If I turn recordings off, the evostream starts to fall.  It will be interesting to see if the recordings make it to the disk after NVR comes back.  Will post later.

New Member
Posts: 9
Registered: ‎05-17-2011
Kudos: 6

Re: Cameras Offline and not recording after upgrade to 3.10.1

After stopping all cameras from recording, memory eventually went back to normal.  Health status still showed red, complaing of NVR off line.  Restarted Unifi video service, all cameras came back and health status shows green...but lost all recordings from prior when health status red...i guess the data stream was just going to null.  Tried rolling back to 3.9.12, but that version won't recognize cameras because of the camera fimware upgrades I did after upgrading to 3.10.  This is a pretty sticky wicket Ubiquiti.  You're better than this.  Lots of surveillance at risk here.  Please fix.

New Member
Posts: 9
Registered: ‎05-17-2011
Kudos: 6

Re: Cameras Offline and not recording after upgrade to 3.10.1

Any resolve on the 3.10.1 recording issue?

Senior Member
Posts: 3,002
Registered: ‎05-19-2013
Kudos: 1301
Solutions: 28

Re: Cameras Offline and not recording after upgrade to 3.10.1

You may want to explore SSH into NVR and start looking at some of the MongoDB repair, checking if the OS is up-to-date using apt-get, and also 'Recordings DB Analysis with Restore Recordings enabled'. One of the NVR I am supporting had minor issues post-update. After going through this set of maintenance procedures, it fixed the minor issues. It was found that there were few 'video-segment related' issues that I thought newer/later 3.9.x firmwares should not happen.
New Member
Posts: 24
Registered: ‎10-03-2017
Kudos: 2

Re: Cameras Offline and not recording after upgrade to 3.10.1

Yep issue has been resolved, been running for ~2 weeks. It seems that post upgrade from 3.9.12 to 3.10.1 there was a rogue MongoDB process running under root user. The NVR I have is dedicated to UniFi-Video and the only change was the upgrade.

What I think was happening is that the EMS or other processes were making connections to both instances which was causing some kind of corruption.

@UBNT-claw killed that process and I have since patched / rebooted the server and the rogue MongoDB process has not come back. System has been stable, still seeing some delayed segments ~470 over the two weeks but all segments have been processed, no empty recordings and new status indicators are all green / 100%.

If you are still having issues, check for rogue MongoDB processes (they won't be running under the unifi-video account) and kill them (assuming you are not using MongoDB for other services - if you are, you will need to check the parameters they were started with to see which service they belong to).

Please post what you find and if this fixes the issue for you (assuming you have rogue MongoDB processes).
New Member
Posts: 38
Registered: ‎10-03-2016
Kudos: 4

Re: Cameras Offline and not recording after upgrade to 3.10.1

[ Edited ]

Hi,

 

have the same issue.

 

first the upgrade failed, Again and like it usually does.

after a fresh install when recovering the footage it deleted most of them...

after adopting the camera's and setting the settings it deleted all of the remaining recordings...

Now it does not record.

says that the CPU/DISK or network utilization is to high or network is unhealthy.

CPU usage is less then 20% There are constant write activities on the RAID 10 array but no recordings are being made.)this disk has recordings only), network is hardly being used and is definitly healthy, can watch live stream and all 23 cams show streams in the Live View.

Performance counters on the unifi video weblogin are all blank.

I am not able to SSH or login (web) into any of the camera's using the creds i specified.

 

Is there anything that can fix it so it records again, (what it is supposed to be doing)

 

New Member
Posts: 9
Registered: ‎03-06-2015
Kudos: 8

Re: Cameras Offline and not recording after upgrade to 3.10.1

I am experiencing similar issues with our NVR going offline.  Is it possible for me to share a support file with  a Ubiquiti employee?

 

We are running a Ubuntu server 16.04 lts, on  Microsoft hyper v.  I have it set up with 2TB storage with 16GB ram and 4 cores at 2.5ghz.  Running 27 cameras at constant record during business hours and on motion the remainder of the day and night. 

 

Screen Shot 2019-03-14 at 7.41.28 PM.png

Highlighted
New Member
Posts: 38
Registered: ‎10-03-2016
Kudos: 4

Re: Cameras Offline and not recording after upgrade to 3.10.1

Might anyone have a link to the previous version?

i need a working version.

Emerging Member
Posts: 80
Registered: ‎10-24-2013

Re: Cameras Offline and not recording after upgrade to 3.10.1

This is happening to us too since the upgrade, running on windows server 2012r2

 

What's the fix ?

New Member
Posts: 9
Registered: ‎03-06-2015
Kudos: 8

Re: Cameras Offline and not recording after upgrade to 3.10.1

I was able to "stabilize" the nvr by disconnecting it from my unifi cloud account. I noticed that each time I logged in via cloud portal the unifi service would go offline.  I have attached a screenshot of the alert. 

Screen Shot 2019-03-18 at 9.03.32 AM.png
Regular Member
Posts: 660
Registered: ‎10-09-2012
Kudos: 241
Solutions: 8

Re: Cameras Offline and not recording after upgrade to 3.10.1

This is happing to our 2TB UBNT NVR as well, customers are not happy, whats the fix for this?

What does TheHox say