Emerging Member
Posts: 114
Registered: ‎01-01-2014
Kudos: 31

Protect w/Flex

Has anyone run into an issue where the Flex disconnects and when reviewing the logs it refers to not enough bandwidth available and that frames are being dropped?  It's on a 1G port (switch is vastly under utilized), running on a CKG2+.  Received this error while testing after bumping the FPS to 24 and the bitrate to around 5400.  Installed firmware is v0.8.7.  RAM consumption jumped and is now hovering between 98-99%.  Concerned that more than one camera may be asking 'too much' of the CKG2+.

 

Notion being that 24FPS at a bitrate of around 4000 is optimal for 1080P.  Lowering the bitrate to middle (~3800) and keeping 24FPS (for all cameras - a mix of G3-Flex, G3 Pro and G3 Dome) - does anyone happen to know how many cameras a CKG2+ should be able to amicably support?  Is there any sort of "calculator" that is available?  Would like some means to validate.

 

Thanks!

Ubiquiti Employee
Posts: 3,131
Registered: ‎02-13-2017
Kudos: 2738
Solutions: 177

Re: Protect w/Flex

@no1inparticular The Cloud Key Gen2 Plus handles 20 x G3-FLEX cameras just fine. In fact, most of our QA test racks have a CK+ and 20 cameras per CK+.

 

Can you post the exact log lines you're seeing?

Emerging Member
Posts: 114
Registered: ‎01-01-2014
Kudos: 31

Re: Protect w/Flex

In a bit will change the params and see if it triggers again.  If it does, I'll grab the exact line and post it.

 

re: 20 cameras per CK+ - good to know.  Would 20 cameras at 24FPS be an issue?

Regular Member
Posts: 537
Registered: ‎03-31-2016
Kudos: 152
Solutions: 15

Re: Protect w/Flex

I have an odd issue with my Flex where it will disconnect and stay disconnected until I power cycle the POE port that it is connected to.

 

Once I power cycle the POE port, it'll come back online and stay connected for a period of time before it goes offline again.

 

Haven't had time to diagnose what is going on just yet but I don't have any issue with my G3 camera connected to the same switch and Protect unit.

Emerging Member
Posts: 114
Registered: ‎01-01-2014
Kudos: 31

Re: Protect w/Flex

After having had some issues (in the distant past) with the APs, setup the poe watchdog on the AP ports - although haven't checked to see it "fired" since on an issue:

 

Seem to recall is being on the interface and something to the effect of:

poe watchdog address <ip>
poe watchdog interval 3
poe watchdog off-delay 2
poe watchdog start-delay 1

 

Seem to recall that I may have set a global option of something like "poe powerkeeper" (or similiar) to try to maintain power across reload/reboot for the APs.

 

Not sure if that would help with the Flex issue.  I also noticed as well was that the G3-Flex disconnected for a bit, but then came back online (no watchdog applied to the port though).  I may have missed it, but for diagnostics and historical capture of relevant details, is there a way to get the cameras to use a syslog server?  At least it would make it easier to search and being able to provide diagnostics back as these things come up.

 

IIRC - thought I remember seeing a thread about memory consumption rising substantially when Protect is in use, but should it be as high as 98/99 percent?

 

Thanks!

Regular Member
Posts: 537
Registered: ‎03-31-2016
Kudos: 152
Solutions: 15

Re: Protect w/Flex

The power consumption for the port is exactly what I expect when the camera is online.

 

I do have the Unifi equipment syslog'ing to a server but will need to check logs to see if it outputs anything with that particular port.

Emerging Member
Posts: 114
Registered: ‎01-01-2014
Kudos: 31

Re: Protect w/Flex

@UBNT-TomS

 

Here is one of the lines:

 

2019-01-08 00:54:48 36934,967 P4 streamer[692]: Frames are skipped! The current bandwidth is not enough! [ubnt_encoder:transporttcp.cppMan FrustratedendMessage:151]

 

Settings/Configs:

  • On-Screen Display = Enabled
  • Adaptive Bitrate = Enabled
  • FPS = 24
  • Bitrate = 5400
  • RTSP (HIgh) = Enabled

All other settings disabled or default.  No zones defined

 

- Is there a way to get the cameras to log to a syslog server?  That would be extremely helpful for diagnostic purposes.

Emerging Member
Posts: 114
Registered: ‎01-01-2014
Kudos: 31

Re: Protect w/Flex

After reviewing a few thousand lines of the error, came upon this:

 

NOTE: UniFI IP Address is obscured -

2019-01-08 00:55:23 36969,703 P3 streamer[692]: Unable to send 1445 bytes of data data. Size advertised by network layer was 1445. Permanent error. (110 Connection timed out) [ubnt_utils:iobuffer.cpp:WriteToFd:350]
2019-01-08 00:55:23 36969,704 P4 streamer[692]: Frames are skipped! The current bandwidth is not enough! [ubnt_encoder:transporttcp.cppMan FrustratedendMessage:151]
2019-01-08 00:55:23 36969,712 P3 streamer[692]: Unable to send 787 bytes of data data. Size advertised by network layer was 787. Permanent error. (110 Connection timed out) [ubnt_utils:iobuffer.cpp:WriteToFd:350]
2019-01-08 00:55:23 36969,712 P4 streamer[692]: Frames are skipped! The current bandwidth is not enough! [ubnt_encoder:transporttcp.cppMan FrustratedendMessage:151]
2019-01-08 00:55:23 36969,713 P3 streamer[692]: Unable to send 5049 bytes of data data. Size advertised by network layer was 5049. Permanent error. (110 Connection timed out) [ubnt_utils:iobuffer.cpp:WriteToFd:350]
2019-01-08 00:55:23 36969,736 P3 streamer[692]: Disconnected! URI was tcp://<UNIFI-IP>:7550?retryInterval=1&connectTimeout=5 [ubnt_encoder:transporttcp.cppMan Frustratedervice:117]
2019-01-08 00:55:23 36969,737 P6 streamer[692]: Connected to tcp://<UNIFI-IP>:7550?retryInterval=1&connectTimeout=5 [ubnt_encoder:transporttcp.cppMan Frustratedervice:94]
2019-01-08 00:55:23 36969,737 P3 streamer[692]: Disconnected! URI was tcp://<UNIFI-IP>:7550?retryInterval=1&connectTimeout=5 [ubnt_encoder:transporttcp.cppMan Frustratedervice:117]
2019-01-08 00:55:23 36969,737 P6 streamer[692]: Connected to tcp://<UNIFI-IP>:7550?retryInterval=1&connectTimeout=5 [ubnt_encoder:transporttcp.cppMan Frustratedervice:94]
2019-01-08 00:55:23 36969,738 P3 streamer[692]: Disconnected! URI was tcp://<UNIFI-IP>:7550?retryInterval=1&connectTimeout=5 [ubnt_encoder:transporttcp.cppMan Frustratedervice:117]
2019-01-08 00:55:23 36969,739 P6 streamer[692]: Connected to tcp://<UNIFI-IP>:7550?retryInterval=1&connectTimeout=5 [ubnt_encoder:transporttcp.cppMan Frustratedervice:94

 

Then what appear to be a few hundred+ lines of this:

2019-01-08 00:55:24 36970,136 P3 avclient[749]: Partial write -1 of 369 [ubnt_avclient:connection.cppMan FrustratedendMessageInternal:421]

 

Afterwards:
2019-01-08 00:55:24 36970,136 P6 avclient[749]: Connection closed [ubnt_avclient:connection.cppMan FrustratedecureTransferProtocolHandler:290]
2019-01-08 00:55:24 36970,145 P6 avclient[749]: Other end closed the connection [ubnt_avclient:connection.cppMan FrustratederviceConnection:363]
2019-01-08 00:55:24 36970,146 P4 avclient[749]: Try again later [ubnt_avclient:avclient_app.cppMan Frustratedervice:408]
2019-01-08 00:55:24 36970,151 P6 avclient[749]: Resetting video destinations to /dev/null [ubnt_avclient:avclient_app.cpp:ResetVideoDestinations:297]
2019-01-08 00:55:24 36970,196 P6 avclient[749]: AUTHENTICATION_OK->INITIALIZING [ubnt_avclient:avclient_app.cppMan Frustratedtate:1480]
2019-01-08 00:55:24 36970,198 P6 avclient[749]: Waiting for the network [ubnt_avclient:avclient_app.cpp:Initialize:230]

 

Looks like a reboot/restart occurred as I spotted a line referring to "init: reloading /etc/inittab".

 

Restoring settings to:

  • Display: only "Adaptive Bitrate" = Enabled
  • Video Quality: 24 FPS and Bitrate = 4000
  • All RTSP = disabled

And no errors appear in the log.

 

Camera firmware = UVC.v4.7.29.67.500d203.180929.1132

 

Thanks!

Ubiquiti Employee
Posts: 4,558
Registered: ‎06-18-2015
Kudos: 1374
Solutions: 419

Re: Protect w/Flex

@no1inparticular,

 

Would you mind sending me a support file via PM (Dropbox / Drive / etc public download link) and the date and time your last disconnect email (if different from above)?

 

@wizard155,

 

Please submit an RMA for the camera that is being disconnected, the watchdog shouldn't be necessary.  We'll get it replaced.

UBNT_Alternate_Logo.png
Ubiquiti Networks Enterprise Support Team


UniFi Protect: UniFi Protect Help Center | Frequently Asked Questions

UniFi Video 3: UniFi Video Help Center | UFV3 User Guide


Regular Member
Posts: 537
Registered: ‎03-31-2016
Kudos: 152
Solutions: 15

Re: Protect w/Flex

@UBNT-Cody - Can I do the RMA through the distributor in AU? It'll be much quicker Man Happy

Ubiquiti Employee
Posts: 4,558
Registered: ‎06-18-2015
Kudos: 1374
Solutions: 419

Re: Protect w/Flex

Yes, we encourage customers to process RMAs directly with official distributors, especially when outside the US.

UBNT_Alternate_Logo.png
Ubiquiti Networks Enterprise Support Team


UniFi Protect: UniFi Protect Help Center | Frequently Asked Questions

UniFi Video 3: UniFi Video Help Center | UFV3 User Guide


Emerging Member
Posts: 114
Registered: ‎01-01-2014
Kudos: 31

Re: Protect w/Flex

@UBNT-Cody

Probably shouldn't have, but started thinking that something else may be wrong.  So after posting previous reply, put the camera through a couple reset to defaults, removing it from UniFi Protect and then set some of the camera's configs (via camera UI - such as NTP Server, etc.) then re-joined to Protect.  Playing with settings (bitrate/FPS/RTSP) again and have not been able to get it to generate errors again.  Only difference this time - no tweaking of the camera's brightness, hue, saturation, etc.  Any chance that those combined with FPS, bitrate, OSD or RTSP or other could have been the cause?

 

FYI - The NTP Server appears to only be configurable if standalone.  Once joined, there appears to be no means to configure this value.  What/where/how is time sync handled?  Anything time sensitive needs to be configured to use the local NTP timekeeper - as outbound NTP is blocked, per policy.

 

Granted, a lot of folks may just simply run the camera mostly "AS-IS" without tweaking the video settings and only adjusting FPS/bitrate.  Did switch the IR to sensitivity "high", but didn't make any other changes this round in the video settings of the camera.

 

In a couple days when I have more time, may come back to this and fiddle with the video settings for the camera to see if changing those in combination with higher bitrate/FPS triggers the error.

Ubiquiti Employee
Posts: 4,558
Registered: ‎06-18-2015
Kudos: 1374
Solutions: 419

Re: Protect w/Flex

Time is synchronized with the Protect controller, as it's more important to have the cameras in sync with the controller than NTP. The controller is then of course synced with NTP itself. I'm not aware of any known reproducible bugs that could cause the symptoms you experienced in relation to camera image settings, so I can't give you a definitive answer if that is indeed the issue. If it happens again, please generate a support file, so we can review the configuration and settings to see if the root cause can be identified.

UBNT_Alternate_Logo.png
Ubiquiti Networks Enterprise Support Team


UniFi Protect: UniFi Protect Help Center | Frequently Asked Questions

UniFi Video 3: UniFi Video Help Center | UFV3 User Guide


Emerging Member
Posts: 114
Registered: ‎01-01-2014
Kudos: 31

Re: Protect w/Flex

If it re-surfaces, will download a support file from the camera.

Ubiquiti Employee
Posts: 4,558
Registered: ‎06-18-2015
Kudos: 1374
Solutions: 419

Re: Protect w/Flex

If the cameras restore the connection on their own after the incident, then the camera support files will be included in the controller support file. If they do not, then please log into the camera separately to generate and include it's own camera support file. Thanks!

UBNT_Alternate_Logo.png
Ubiquiti Networks Enterprise Support Team


UniFi Protect: UniFi Protect Help Center | Frequently Asked Questions

UniFi Video 3: UniFi Video Help Center | UFV3 User Guide


Emerging Member
Posts: 114
Registered: ‎01-01-2014
Kudos: 31

Re: Protect w/Flex

@UBNT-Cody

 

To keep this thread updated:  Issue happened again.  Sent support file info.  So, far appears that if there is no interaction with the camera - it won't automatically reconnect.  The camera is still present on the network and you can reach it's web interface.

 

If one switches to browser tab for the camera [itself], do a forced reload and login - the camera appears to automatically re-connect to Protect.  Until a page reload and/or login is performed - the camera does not appear to automatically re-connect.  Uptime does not infer that any reboot has occurred.

 

Notes/Thoughts:

  • Have been leaving up a live view session for this camera, that's how the disconnect was noticed.
  • in another browser tab, have left up the system panel (although all panels in the camera's local interface have a live view), may be material to the issue?

 

Thanks!

Ubiquiti Employee
Posts: 4,558
Registered: ‎06-18-2015
Kudos: 1374
Solutions: 419

Re: Protect w/Flex

@no1inparticular,

I received your camera support file, could you please upload a Protect support file as well?

UBNT_Alternate_Logo.png
Ubiquiti Networks Enterprise Support Team


UniFi Protect: UniFi Protect Help Center | Frequently Asked Questions

UniFi Video 3: UniFi Video Help Center | UFV3 User Guide


Ubiquiti Employee
Posts: 3,131
Registered: ‎02-13-2017
Kudos: 2738
Solutions: 177

Re: Protect w/Flex

Thanks @no1inparticular. Very interested in debugging this case. As soon as we have your Protect support file (Cody will take it) we will dig into it.

Emerging Member
Posts: 114
Registered: ‎01-01-2014
Kudos: 31

Re: Protect w/Flex

@UBNT-Cody

 

After previous message - wanting to make sure that it wasn't a result of any settings, etc. - figured I'd do a factory reset on Protect, only add one camera the G3 Flex and then let it run to see if the errors emerged again.  Given that Protect on the CKG2+ isn't critical - only UniFi, figured it should be easy enough to reconfigure and join the Flex.  Only thing about the 'factory reset' is that it is non-intuitive.  eg: it presents and cycles between two screens:

Screen Shot 2019-01-09 at 23.55.54.jpgScreen Shot 2019-01-09 at 23.56.31.jpg

Suspect that it's just going through video on the drive and deleting the videos.  HOWEVER - it should show something else to let the user know that it's busy working.  After about 10+ minutes, it finally came back and revealed the expected screen.

 

That said, looking to make sure that things are starting from a sane point:

  • Camera had a 'factory reset'
  • Camera's factory config uses a UBNT ntp pool - this would be blocked at the firewall, thus set for local [GPS] NTP server
  • Added Camera to Protect (only one camera)
  • Only changes made:
    • General: LED=off, Display=All Off, FPS=25, Bitrate=4200 (for now), Camera=IR set to High sensitivity
  • Logged out of camera's GUI
  • Live View of Flex will be left open

Other Notes:

  • Camera(s) will be on a dedicated VLAN (separate from UniFI).
  • Camera has to be brought up on UniFi VLAN to 'join'
  • After join, rebooted and moved to another VLAN (critical as it's being done to simulate actual deployment/s scenario)
    • Really wish there was another variant of the CKG2+ that had an additional NIC so that Protect could have an additional interface directly in the "camera VLAN".  Second NIC port would likely be simpler and void potential pitfalls, while still keeping Protect's viewing access on the UniFi IP/port.
  • Started making some notes about settings (made via Protect) that were changed in the camera.  Thus, if issue doesn't re-occur, will start to adjust those settings as well.
  • FPS has been either 24 or 25 and bitrate has been varied between 3800 and 5400.

 

Thanks!

Emerging Member
Posts: 114
Registered: ‎01-01-2014
Kudos: 31

Re: Protect w/Flex

@UBNT-TomS

 

Equally curious too.  From my prior post, noted that a factory reset was done on Protect to make sure that things are starting from a "sane" point.

 

Believe that information on the switch port config was sent to Cody.  Wondering if that has any bearing?

 

From a network path perspective:

 

<camera> => Switch (VLAN "A") => Router => Switch (VLAN "B") => CKG2+.

 

The cameras need to be on a separate VLAN from everything else.  CKG2+ for UniFI and access to RADIUS servers, must be in VLAN B.  Currently, cameras have to be brought up in VLAN B to "join" and then moved to another VLAN to test. IGMP is in play.