Reply
Highlighted
Regular Member
Posts: 539
Registered: ‎05-02-2014
Kudos: 96
Solutions: 4

Nvr Lost connection to video.ubnt.com

I have a customer that lost connection to video.ubnt.com 

 

after update to 3.9.4 yesterday the customer have to reboot the nvr first before he can access the nvr by remote video.ubnt.com acces. 

 

Local Als acces works fine. 

 

Ubiquiti Employee
Posts: 1,047
Registered: ‎03-03-2016
Kudos: 439
Solutions: 57

Re: Nvr Lost connection to video.ubnt.com

@stekkerdoos The likely cause of this is some network degradation cause the NVR to drop the connection. The cloud has what we call a rate-limit, that allows for an NVR to attempt reconnection up to 3 times within 15 minutes. So if these reconnections failed within the first few minutes, the cloud will only accept NVR to re-connect once that 15 minute window has expired.
We are working on adding feedback to the Web UI so the connection status is known to the users.
Regular Member
Posts: 539
Registered: ‎05-02-2014
Kudos: 96
Solutions: 4

Re: Nvr Lost connection to video.ubnt.com

[ Edited ]

Ok. We just upgrade to 3.9.4 because we have bad internet connections. ( on a island ) and need the variable bit rate for live viewing. 

 

Also the note in the release note did make me hoping things where getting better but now it is worse.

 

Bug Fixes:
Fixed cloud connection failure due to timeouts when UniFi Video is restarted, including after a software upgrade

 

Butt if it is a 3 time lock out protection in the cloud why it is connecting after a reboot ? 

 

New Member
Posts: 26
Registered: ‎10-15-2016
Kudos: 31
Solutions: 1

Re: Nvr Lost connection to video.ubnt.com

[ Edited ]

I experience the same issue here (usually reliable fiber connection) since 3.9.4 beta 1 and now with 3.9.4 still. Reboot of video controller resolves the issue for a few hours (?) I believe. Since release I had this issue a few times.

Regular Member
Posts: 477
Registered: ‎07-14-2014
Kudos: 162
Solutions: 10

Re: Nvr Lost connection to video.ubnt.com

I had the same issue. 

I waited and waited for days for timeout to reset and had no luck. 

I tried to save config, erase Unifi Video and reload with old config. Still having same issues.

Took screenshots of Config on my phone. Completly wiped computer, fresh windows install, drivers, ram partition, etc. 

Reinstalled Unifi video 3.9.3 fresh (instead of 3.9 and then upgrading in the controller) and seems to have fixed issue. 

Still seeing error code 400 upon log in, but issue seems to be resolved. 

Good luck. This was a huge pain for several days

Regular Member
Posts: 579
Registered: ‎01-08-2015
Kudos: 100
Solutions: 7

Re: Nvr Lost connection to video.ubnt.com

We, too, are on a very stable/secure fiber connection in the states.  We have a latentless failover to another provider fiber coming in the DC at a different location.  Just for kicks and giggles, I checked out our stated reporting devices with zero downtime or packet loss in ~4 weeks.  

Emerging Member
Posts: 53
Registered: ‎05-14-2016
Kudos: 26
Solutions: 1

Re: Nvr Lost connection to video.ubnt.com

This is an issue for me as well. Can access the NVR locally using <ip>:8443 but video.ubnt.com shows the NVR as disconnected until I stop and start the unifi video service. Running Windows Server 2008.

 

Any ideas?

Ubiquiti Employee
Posts: 1,047
Registered: ‎03-03-2016
Kudos: 439
Solutions: 57

Re: Nvr Lost connection to video.ubnt.com

[ Edited ]

@charperimg Could you please provide me with the logs. There are several reports of these and I am investigating the root cause.
Disconnections are expected, what is not is the fact it isn't reconnecting in timely manner or until the unifi-video service is restarted.

To everyone here, if you have this issue, I'd like to collect the logs. You can send them direct to me at amir.nathoo<at>ubnt.com

Thanks.

New Member
Posts: 1
Registered: ‎09-02-2016

Re: Nvr Lost connection to video.ubnt.com

I have the same issue. Log sent.

Emerging Member
Posts: 53
Registered: ‎05-14-2016
Kudos: 26
Solutions: 1

Re: Nvr Lost connection to video.ubnt.com

Support file sent. Appreciate the help.

Ubiquiti Employee
Posts: 1,047
Registered: ‎03-03-2016
Kudos: 439
Solutions: 57

Re: Nvr Lost connection to video.ubnt.com

Thanks for the logs. I am going to spend some times looking at them now.
I am trying to find a way to also reproduce this on our end, but so far that cloud connection has been rock solid..I have 7 NVRs currently connected and non of them are showing any signs of failure. So finding a pattern from these logs is critical.
Emerging Member
Posts: 53
Registered: ‎05-14-2016
Kudos: 26
Solutions: 1

Re: Nvr Lost connection to video.ubnt.com

If it helps I’m on a Comcast Business 1Gbps connection

Ubiquiti Employee
Posts: 1,047
Registered: ‎03-03-2016
Kudos: 439
Solutions: 57

Re: Nvr Lost connection to video.ubnt.com

@charperimg So quick look at the logs you sent shows a slow connection to the cloud 

 

1521080371.504 2018-03-14 22:19:31.504/EDT: WARN Slow cloud Websocket connection detected (no pongs after 2 pings - last ping 14 seconds ago) in Cloud-PingPong

1521080376.506 2018-03-14 22:19:36.506/EDT: WARN Slow cloud Websocket connection detected (no pongs after 3 pings - last ping 19 seconds ago) in Cloud-PingPong

 

Camera[44D9E7949DE5] Unable to get stats: Request 2147461851 timed out after 10000ms

Camera[802AA84E9EB8] Unable to get stats: Request 2147461852 timed out after 10000ms 

Camera[802AA84E99B9] Unable to get stats: Request 2147461853 timed out after 10000ms

 

1521080380.419 2018-03-14 22:19:40.419/EDT: INFO Establishing ws connection to wss://device.svc.ubnt.com/device/ufv/v1/7aa7bdca-457f-499c-b97e-0208db7049b7/... in CloudWebsocketClient-Connection

 

CameraService.removeConnection b to 192.168.1.30 - camera now stateMan Very HappyISCONNECTED reason:006.Connection Lost

CameraService.removeConnection 8 to 192.168.1.30 - camera now stateMan Very HappyISCONNECTED reason:006.Connection Lost

 

And this seems to happen right around the time when connections from that NVR to anything it is connected to is failing.
Pretty much all cameras are also disconnecting. And right in the middle of this, Unifi-Video is attempting to re-establish the connection to the cloud.

 

The connections with the cameras are re-established pretty quickly (about a minute after these disconnects)
1521080443.640 2018-03-14 22:20:43.640/EDT: INFO Camera[...] processHelloMessage() - Initialize connection (port 7442)... in tomcat-HTTPS-cam-exec-2
1521080448.876 2018-03-14 22:20:48.876/EDT: INFO Camera[...] processHelloMessage() - Initialize connection (port 7442)... in tomcat-HTTPS-cam-exec-7

But ~8 minutes later cameras are disconnecting again..and at that point the cloud connection is still not established.

1521080917.698 2018-03-14 22:28:37.698/EDT: INFO CameraService::handleCameraDisconnected - camera state DISCONNECTED in AppBus-11


Then again about 1 minute later the cameras are re-connecting on that port 7442, but still controller isn't able to reach the cloud.



The Unifi-Video service is restarted as you mentioned the next day.
1521113675.224 2018-03-15 07:34:35.224/EDT: INFO   >>>> unifi-video v3.9.4+e2c8e9.180312.1753 is starting in main

The network failures appear to be reloved and controller right at start up is establishing the connection again:
1521113686.449 2018-03-15 07:34:46.449/EDT: INFO Establishing ws connection to wss://device.svc.ubnt.com/device/ufv/v1/7aa7bdca-457f-499c-b97e-0208db7049b7/... in CloudWebsocketClient-Connection

All cameras have re-connecting successfully and the cloud connection is re-established sucessfuly within 20 seconds of requesting it.

 

1521113694.263 2018-03-15 07:34:54.263/EDT: INFO Established ws connection to: wss://device.svc.ubnt.com/device/ufv/v1/7aa7bdca-457f-499c-b97e-0208db7049b7/ in CloudWebsocketClient-Connection

 

So in summary:
1. Some kind of network issue caused NVR to disconnect from everything it was connected to (camera, cloud, media streamer,...). There were on and off disconnections. The cloud never reconnected during the network instability
2. Upon restart of the NVR several hours later,  the Network issues were gone and the cameras reconnected and the NVR on it's first attempt to re-connecting to the cloud was also successfully connected.

What we can do on our side is to ensure than when network conditions are impairing Unifi-video functionality, that we report it back to the web application, so users can be aware of issues happening.

We have an in progress task to provide notification for system status and well as providing feedback right in the web application when the cloud connection is about to go down or has gone down. Part of this is already implemented and will be rolled out in 3.10 release, but we are also working on a more complete reporting solution.

Emerging Member
Posts: 53
Registered: ‎05-14-2016
Kudos: 26
Solutions: 1

Re: Nvr Lost connection to video.ubnt.com

Yep so last night I updated all my UniFi networking gear’s firmware based on the new controller firmware coming out and unifi software. Several switches would have rebooted etc. But this AM after all that was done the nvr still wasn’t connected to the cloud. 

Ubiquiti Employee
Posts: 1,047
Registered: ‎03-03-2016
Kudos: 439
Solutions: 57

Re: Nvr Lost connection to video.ubnt.com

[ Edited ]

But it does show the connection was established though, at ~7:34

1521113694.263 2018-03-15 07:34:54.263/EDT: INFO Established ws connection to: wss://device.svc.ubnt.com/device/ufv/v1/7aa7bdca-457f-499c-b97e-0208db7049b7/ in CloudWebsocketClient-Connection

And after that there is no disconnection error messages.

Ancient Member
Posts: 31,609
Registered: ‎05-05-2012
Kudos: 10034
Solutions: 1555

Re: Nvr Lost connection to video.ubnt.com


@charperimgwrote:

Yep so last night I updated all my UniFi networking gear’s firmware based on the new controller firmware coming out and unifi software. Several switches would have rebooted etc. But this AM after all that was done the nvr still wasn’t connected to the cloud. 


When the network bumps during the UniFi Switch upgrade, my NVR loses both DNS and DG which then means it cannot resolve names to IP on the internet.  Whenever that happens, I reboot the NVR to restore full functionality.

 

Note: I don't use the portal but do rely on DNS to resolve the SMTP server IP.

My Topics
Top Solution Authors Leaderboard
"The firings will continue until morale improves." Donald Trump (unconfirmed) 2017.
Ubiquiti Employee
Posts: 1,047
Registered: ‎03-03-2016
Kudos: 439
Solutions: 57

Re: Nvr Lost connection to video.ubnt.com

I do see that we need to improve error/status reporting here to help users diagnose issues on their end, but again we have no control whatsoever over network instability or incidents, cause by equipments external to UniFi-Video
Emerging Member
Posts: 53
Registered: ‎05-14-2016
Kudos: 26
Solutions: 1

Re: Nvr Lost connection to video.ubnt.com

That should have been when I stopped the unifi video service and re started it if I recall correctly. 

Ubiquiti Employee
Posts: 2,537
Registered: ‎02-13-2017
Kudos: 2031
Solutions: 138

Re: Nvr Lost connection to video.ubnt.com

Quick update for everyone having network problems with 3.9.x versions:

 

We identified a cloud connection regression in versions 3.9.0 through 3.9.4 of the UniFi Video 3 software. This regression can eventually cause NVRs running that version to disconnect from the cloud and, in some cases, not reconnect until the service is restarted.

 

We've released an updated 3.9.5 version of the NVR software to beta here: https://community.ubnt.com/t5/UniFi-Video-Beta-Blog/UniFi-Video-3-9-5-Beta-2-Release/ba-p/2285805

 

We will test the beta version as quickly as possible, then promote it to GA so everyone can upgrade their NVRs.

 

My apologies for the trouble this has caused. Although the number of impacted NVRs was low, this is still an unacceptable bug to impact any of our customers. We're increasing our QA testing and QA processes to ensure this won't happen again.

Reply