Reply
Highlighted
Senior Member
Posts: 3,653
Registered: ‎12-21-2013
Kudos: 997
Solutions: 83

All cameras offline since 3.9.6 upgrade

1526508691.661 2018-05-17 00:11:31.661/CEST: INFO   SetControllerHostCheck: Camera[00CAFFEEBABE|Cam 2] not seen in last 364s (state DISCONNECTED) - attempting remanage in CameraService-SetControllerHostTimer
1526508691.663 2018-05-17 00:11:31.663/CEST: INFO   Camera[00CAFFEEBABE|Cam 2] Issuing manage request in CameraService-SetControllerHostTimer
1526508695.668 2018-05-17 00:11:35.668/CEST: INFO   SetControllerHostCheck: Could not remanage Camera[00CAFFEEBABE|Cam 2]: Connect to 1.2.3.4:443 [/1.2.3.4] failed: connect timed out in CameraService-SetControllerHostTimer
1526508695.669 2018-05-17 00:11:35.669/CEST: INFO   SetControllerHostCheck: Camera[00CAFFEEBABF|Cam 1] not seen in last 368s (state DISCONNECTED) - attempting remanage in CameraService-SetControllerHostTimer
1526508695.670 2018-05-17 00:11:35.670/CEST: INFO   Camera[00CAFFEEBABF|Cam 1] Issuing manage request in CameraService-SetControllerHostTimer
1526508699.677 2018-05-17 00:11:39.677/CEST: INFO   SetControllerHostCheck: Could not remanage Camera[00CAFFEEBABF|Cam 1]: Connect to 1.2.3.4:443 [/1.2.3.4] failed: connect timed out in CameraService-SetControllerHostTimer
1526508717.531 2018-05-17 00:11:57.531/CEST: ERROR  Cannot send EMS CLI Request {"command":"listConfig","parameters":{"_messageId":12}}: Unable to establish websocket connection to send message in EmsCliApi-Executor
1526508747.531 2018-05-17 00:12:27.531/CEST: ERROR  Cannot send EMS CLI Request {"command":"listConfig","parameters":{"_messageId":13}}: Unable to establish websocket connection to send message in EmsCliApi-Executor

Is it possible this version has problems with NAT?

When you receive a solution to your question/issue, don't forget to mark your thread as solved and to give kudos to the people who have helped you out!

Senior Member
Posts: 3,653
Registered: ‎12-21-2013
Kudos: 997
Solutions: 83

Re: All cameras offline since 3.9.6 upgrade

I have been using custom certificates and NAT since AirVision2 and so far it has never been any problem.

 

Another part of the server log:

1526508977.938 2018-05-17 00:16:17.938/CEST: INFO   DebugInfo Level:onerror logNone:false logOnError:true in main
1526508977.938 2018-05-17 00:16:17.938/CEST: INFO   Service WebRtcService started in main
1526508977.938 2018-05-17 00:16:17.938/CEST: INFO   Service EventService started in main
1526508977.939 2018-05-17 00:16:17.939/CEST: INFO   Service BackupService started in main
1526508977.939 2018-05-17 00:16:17.939/CEST: INFO   discovery service starting in main
1526508977.942 2018-05-17 00:16:17.942/CEST: INFO   Initializing Discovery Service in main
1526508977.944 2018-05-17 00:16:17.944/CEST: INFO   Service DiscoveryService started in main
1526508977.945 2018-05-17 00:16:17.945/CEST: INFO   Service startup sequence completed successfully in main
1526508978.445 2018-05-17 00:16:18.445/CEST: INFO   Executing EMS binary in ems-service
1526508978.449 2018-05-17 00:16:18.449/CEST: ERROR  ems has quit with rc=0 in ems-service
1526508978.453 2018-05-17 00:16:18.453/CEST: INFO   [EmsApiWebsocketClient] connect - Establishing ws connection to uri:wss://127.0.0.1:7440/ in EmsApiWebsocketClient-Connection
1526508978.497 2018-05-17 00:16:18.497/CEST: ERROR  [EmsApiWebsocketClient] connect - FAILED connection to uri:wss://127.0.0.1:7440/ - DeploymentException - The HTTP request to initiate the WebSocket connection failed in EmsApiWebsocketClient-Connection
1526508978.497 2018-05-17 00:16:18.497/CEST: INFO   [EmsApiWebsocketClient] connect - Retrying ws connection to uri:wss://127.0.0.1:7440/ in 2000ms in EmsApiWebsocketClient-Connection
1526508980.498 2018-05-17 00:16:20.498/CEST: INFO   [EmsApiWebsocketClient] connect - Establishing ws connection to uri:wss://127.0.0.1:7440/ in EmsApiWebsocketClient-Connection
1526508980.517 2018-05-17 00:16:20.517/CEST: ERROR  [EmsApiWebsocketClient] connect - FAILED connection to uri:wss://127.0.0.1:7440/ - DeploymentException - The HTTP request to initiate the WebSocket connection failed in EmsApiWebsocketClient-Connection
1526508980.517 2018-05-17 00:16:20.517/CEST: INFO   [EmsApiWebsocketClient] connect - Retrying ws connection to uri:wss://127.0.0.1:7440/ in 6000ms in EmsApiWebsocketClient-Connection
1526508984.702 2018-05-17 00:16:24.702/CEST: INFO   Bootstrap Manager - generateBootstrapData (Full Bootstrap) for user Marc-André Kolly  in tomcat-HTTPS-exec-4
1526508984.737 2018-05-17 00:16:24.737/CEST: INFO   Bootstrap Manager - newCriticalAlerts for user Marc-André Kolly - 0 in tomcat-HTTPS-exec-4
1526508986.518 2018-05-17 00:16:26.518/CEST: INFO   [EmsApiWebsocketClient] connect - Establishing ws connection to uri:wss://127.0.0.1:7440/ in EmsApiWebsocketClient-Connection
1526508986.534 2018-05-17 00:16:26.534/CEST: ERROR  [EmsApiWebsocketClient] connect - FAILED connection to uri:wss://127.0.0.1:7440/ - DeploymentException - The HTTP request to initiate the WebSocket connection failed in EmsApiWebsocketClient-Connection
1526508986.534 2018-05-17 00:16:26.534/CEST: INFO   [EmsApiWebsocketClient] connect - Retrying ws connection to uri:wss://127.0.0.1:7440/ in 18000ms in EmsApiWebsocketClient-Connection
1526508988.450 2018-05-17 00:16:28.450/CEST: ERROR  Last-chance exception in ems-service
com.ubnt.airvision.service.ems.D: Timeout executing request: 1
	at com.ubnt.airvision.service.ems.request.EmsRequest.getResponseDataAs(Unknown Source) ~[airvision.jar:?]
	at com.ubnt.airvision.service.ems.request.ShutdownServer.getResponse(Unknown Source) ~[airvision.jar:?]
	at com.ubnt.airvision.service.ems.E.Ø00000(Unknown Source) ~[airvision.jar:?]
	at com.ubnt.airvision.service.ems.ooOO.run(Unknown Source) [airvision.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task.
	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:276) ~[guava-14.0.1.jar:?]
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96) ~[guava-14.0.1.jar:?]
	... 5 more
1526509004.534 2018-05-17 00:16:44.534/CEST: INFO   [EmsApiWebsocketClient] connect - Establishing ws connection to uri:wss://127.0.0.1:7440/ in EmsApiWebsocketClient-Connection
1526509004.545 2018-05-17 00:16:44.545/CEST: ERROR  [EmsApiWebsocketClient] connect - FAILED connection to uri:wss://127.0.0.1:7440/ - DeploymentException - The HTTP request to initiate the WebSocket connection failed in EmsApiWebsocketClient-Connection
1526509004.545 2018-05-17 00:16:44.545/CEST: INFO   [EmsApiWebsocketClient] connect - Retrying ws connection to uri:wss://127.0.0.1:7440/ in 54000ms in EmsApiWebsocketClient-Connection
1526509008.453 2018-05-17 00:16:48.453/CEST: ERROR  Cannot send EMS CLI Request {"command":"shutdownServer","parameters":{"_messageId":1}}: Unable to establish websocket connection to send message in EmsCliApi-Executor

When you receive a solution to your question/issue, don't forget to mark your thread as solved and to give kudos to the people who have helped you out!

Regular Member
Posts: 395
Registered: ‎09-25-2014
Kudos: 134
Solutions: 18

Re: All cameras offline since 3.9.6 upgrade

What version did you upgrade from?

Ubiquiti Employee
Posts: 2,975
Registered: ‎06-18-2015
Kudos: 947
Solutions: 261

Re: All cameras offline since 3.9.6 upgrade

Hi @MAKuser,

 

If you have updated from a version prior to v3.8.1, you'll need to review the new certificate requirements as outlined in this article:

 

https://community.ubnt.com/t5/UniFi-Video/Custom-SSL-Certificates-in-3-8-1-and-beyond/m-p/2089043#M8...

 

In addition, if your cameras are behind NAT, you will need to review the camera management ports, as you are likely missing the new port 7442:

 

https://help.ubnt.com/hc/en-us/articles/217875218-UniFi-Video-Ports-Used

UBNT_Alternate_Logo.png
Ubiquiti Networks Enterprise Support Team


Check out our ever-evolving UniFi Video Help Center and UFV3 User Guide for answers to many common questions!

Senior Member
Posts: 3,653
Registered: ‎12-21-2013
Kudos: 997
Solutions: 83

Re: All cameras offline since 3.9.6 upgrade


@UBNT-Cody wrote:

Hi @MAKuser,

 

If you have updated from a version prior to v3.8.1, you'll need to review the new certificate requirements as outlined in this article:

 

https://community.ubnt.com/t5/UniFi-Video/Custom-SSL-Certificates-in-3-8-1-and-beyond/m-p/2089043#M8...

 

In addition, if your cameras are behind NAT, you will need to review the camera management ports, as you are likely missing the new port 7442:

 

https://help.ubnt.com/hc/en-us/articles/217875218-UniFi-Video-Ports-Used


Cody, thanks for the response but you used to know me... I'm always running the latest version. 3.8.1 is ancient, it's from October last year.

I'm sure I upgraded from the version before, which is 4.9.5. Once I'm back on my machine, I'll confirm that with apt logs.

 

No ports are blocked on the server side and have never been blocked either. The server has several public IPs.

The cameras on the other hand do not have port 7442 forwarded to them, in case that's what you mean. Cameras shouldn't need to be reachable by the controller on a fixed port. We have these cool technologies like websockets which allows for bidirectional communication through any NAT or even several NATs behind each other, as long as the server is reachable from the client, just like a regular HTTP call. And I know you're using that already, so I assume it really isn't a requirement now, to forward port 7442 to the cams. In fact it wouldn't even be possible, since many cams are behind the same router with same public IP, eg. Cam 1 and 2 behind 1.2.3.4.

 

I've redone all certificate stuff three times.

 

I had the same problem on another controller, which has mostly local cams though, and there I just had to delete all uvf-keystore, cam-keystore and evostream server files and restart the service to see all cams popping back online one after another. Remote cams behind NAT didn't have a problem there, eventhough this controller is behind a NAT too and only had the required ports forwarded. I have a local test cam (remote to the controller) which I just plugged into the network after the upgrade and certificate fix, it booted, upgraded and came online just like that.

 

No idea what's wrong with the controller I started this thread about.

When you receive a solution to your question/issue, don't forget to mark your thread as solved and to give kudos to the people who have helped you out!

Senior Member
Posts: 3,653
Registered: ‎12-21-2013
Kudos: 997
Solutions: 83

Re: All cameras offline since 3.9.6 upgrade


@MAKuser wrote:

I'm sure I upgraded from the version before, which is 4.9.5. Once I'm back on my machine, I'll confirm that with apt logs.


Ups, just noticed the typo. The version released before 3.9.6 was 3.9.5, not 4.9.5 obviously Man Wink

 

Bildschirmfoto vom 2018-05-17 09-33-25.png

 

So yeah, I upgraded from 3.9.5 to 3.9.6.

When you receive a solution to your question/issue, don't forget to mark your thread as solved and to give kudos to the people who have helped you out!

Senior Member
Posts: 3,653
Registered: ‎12-21-2013
Kudos: 997
Solutions: 83

Re: All cameras offline since 3.9.6 upgrade

Why does the cam-truststore use a random interface IP as CN? I got multiple IPs and interfaces, and it seems to be ignoring the system_ip setting.

What is the ufv-keystore being used for?

 

Here you got a few info on my certs:

14:22:26,78 root@MAKserver06: /var/lib/unifi-video > keytool -list -v -alias airvision -keystore cam-keystore 
Enter keystore password:  
Alias name: airvision
Creation date: May 17, 2018
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=172.19.0.1, OU=airVision, O=ubnt.com, L=San Jose, ST=CA, C=US
Issuer: CN=172.19.0.1, OU=airVision, O=ubnt.com, L=San Jose, ST=CA, C=US
Serial number: 5afd73dd
Valid from: Thu May 17 14:21:49 CEST 2018 until: Sun May 14 14:21:49 CEST 2028
Certificate fingerprints:
	 MD5:  BE:78:AC:00:A5:0E:91:7B:1E:2F:36:EB:C7:6B:05:07
	 SHA1: 28:C8:A1:8D:B7:98:49:6F:30:07:85:E9:FF:90:98:75:2A:B6:EF:04
	 SHA256: EA:02:CC:86:7C:0C:9D:71:FA:07:45:0F:CE:65:51:8D:B4:DF:D6:0A:8D:F4:35:74:8D:16:09:84:8D:A6:FE:E1
Signature algorithm name: SHA1withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore cam-keystore -destkeystore cam-keystore -deststoretype pkcs12".
14:31:25,79 root@MAKserver06: /var/lib/unifi-video > keytool -list -v -alias airvision -keystore ufv-truststore 
Enter keystore password:  
Alias name: airvision
Creation date: May 17, 2018
Entry type: trustedCertEntry

Owner: CN=mydomain.tld
Issuer: CN=My CA
Serial number: 1337
Valid from: Sun Apr 15 00:05:08 CEST 2018 until: Sat Jul 14 00:05:08 CEST 2018
Certificate fingerprints:
	 MD5:  123
	 SHA1: 123
	 SHA256: 123
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 4096-bit RSA key
Version: 3

14:31:46,80 root@MAKserver06: /var/lib/unifi-video > keytool -list -v -alias airvision -keystore keystore 
Enter keystore password:  
Alias name: airvision
Creation date: May 17, 2018
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=mydomain.tld
Issuer: CN=My CA
Serial number: 1337
Valid from: Sun Apr 15 00:05:08 CEST 2018 until: Sat Jul 14 00:05:08 CEST 2018
Certificate fingerprints:
	 MD5:  123
	 SHA1: 123
	 SHA256: 123
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 4096-bit RSA key
Version: 3


Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore keystore -destkeystore keystore -deststoretype pkcs12".
14:32:17,81 root@MAKserver06: /var/lib/unifi-video > 

When you receive a solution to your question/issue, don't forget to mark your thread as solved and to give kudos to the people who have helped you out!

Reply