Reply
Member
Posts: 251
Registered: ‎10-25-2016
Kudos: 63
Solutions: 7

Latency Spike every 40-50 seconds

[ Edited ]

I have a few customers reporting the same connectivity issue at three different sites where I have the unifi system deployed. Their symptoms are basically network instability. They're complaining of degraded voip applications like facetime and similar, poor internet connection errors during gaming, and buffering during streaming on netflix, youtube, etc. After going on site I didnt really notice any issues other than some basic interference and these weird latency spikes every 40-50 seconds. I'm only seeing these latency spikes at these three buildings. I see the exact same spike if I'm pinging from my laptop, the gateway, or a connected unifi device (CK for example), and it still happens even if I ping the gateway IP vs an IP on the internet. So far I'm only seeing this spike at the three locations that are complaining about issues, all of which are using a USG-PRO as a router, and a mix of unifi and edge switches. I have other apartments that use the same topology that this isnt happening at (or they havent reported it at). It doesnt seem like a huge issue being that it's only 1-2 echos that spike the latency, but it's the only tangible evidence of something going on at these three sites where clearly something is happening (otherwise I wouldnt be getting all these separately relate calls). At these sites, multiple tenants have separately complained about the same issue, reporting it starting around the same timeframe. 

 

Has anyone else seen this recently, or have any ideas why I'm seeing this?

 

Attached is a print of the pings showing the spike in latency every 40-50 seconds, pinging 8.8.8.8 and the gateway. 

 

Note - disabling DPI and IDS did not affect the results. 

Note - i cant make any correlation to a change made, etc, during the time they mentioned this happening. I imagine there was a software update around that time, but I can't say for sure which version. 

Member
Posts: 251
Registered: ‎10-25-2016
Kudos: 63
Solutions: 7

Re: Latency Spike every 40-50 seconds

bump

Member
Posts: 251
Registered: ‎10-25-2016
Kudos: 63
Solutions: 7

Re: Latency Spike every 40-50 seconds

Update:

I had a conversation with a tenant at one of these locations about the issue. He was using an iphone with WiFi Calling turned on, and every time we'd see this latency spike his voice would breakup/delay for 2-3 seconds then return to normal. It happened precisely when we saw the latency spike while pinging continuously out to the internet; which happens every 40-50 seconds.

 

Interference has already been ruled out as we've used the APs RF scan and a stand alone spectrum analyzer to ensure the airwaves were clean. We also see the same latency spike when hardwired. 

 

USG-Pro: v4.4.22.5086057

UAP-LITE: v3.9.42.9152

Edgeswitch: v1.7.3 (used for hardwired jacks)

Unifi 48/24: v3.9.27.8537  (used with unifi APs)

 

 

Has no one else seen this issue? It started in the middle of June sometime. 

Regular Member
Posts: 304
Registered: ‎08-25-2016
Kudos: 70
Solutions: 6

Re: Latency Spike every 40-50 seconds

Back to basics:

  • Turn off RSTP or STP, known to cause some issues and delays in DHCP leasing.
  • Check you haven't done something silly like realy short DHCP lease times.
  • Turn off Auto-AP upgrade.
  • Turn off periodic speed test.
  • Turn off keep-alive ping. 
  • Set the USG to ping ubnt.com, other IP addresses will see the WAN ping plotting stop.
  • Make sure AP's are set to specific channels and not on Auto, it is useless and evil.
  • Sart watching the switch stats for a broadcast storm. 
  • Make sure you know who is connected and how. 
  • Disable unused switch ports. 

Biggest cause of sluggish network is broadcast or slutty prorograms/apps like P2P or Spotify. 

Member
Posts: 251
Registered: ‎10-25-2016
Kudos: 63
Solutions: 7

Re: Latency Spike every 40-50 seconds

Thank you for responding to this.
RSTP is enabled for specific reasons, but it isnt causing this issue. DHCP leasing isn't a problem for any devices I've seen so far. Connection is fine except for this 2-4 second interval every 40-50 seconds. Aside from this interval, the connection is perfectly fine.
Auto-ap upgrade and all other advanced features are disabled already.
USG set to ping ubnt.com by default
Interference has been tested thoroughly and ruled out. Problem exists everywhere on the LAN, wireless, hardwired, and even over my remote access VPN.
I have kept an eye on switchport stats and havent noticed anything unusual. However because the issue exists over L2TP VPN, I dont see a broadcast storm causing this issue.

Again, the network is perfectly fine except for this 2-4 second interval. After noticing the issue over L2TP, I'm becoming more and more convinced this is an issue with the USG-Pro.
Member
Posts: 251
Registered: ‎10-25-2016
Kudos: 63
Solutions: 7

Re: Latency Spike every 40-50 seconds

Additionally:

 

I've been monitoring the processes on the USG during the latency spikes, and while there's no one process that tends to pop up during the spike, I did notice the number of processes actively using the CPU jumps from the average 2-6 processes to 10-13 processes ONLY during the latency spike. I've had a continuous ping from an unused AP and TOP running for almost a half an hour and I only ever see a dozen active processes during the few second spike. 

 

Also, I cant be sure it's related, but 'kworker/0:2[1]' and 'migration' always seem to be at the top of the list (of top) during this spike. 

 

 

Member
Posts: 251
Registered: ‎10-25-2016
Kudos: 63
Solutions: 7

Re: Latency Spike every 40-50 seconds

[ Edited ]

No one has ever run into this issue? I'm seeing the EXACT same thing at two different sites. Here's what it looks like in a ping. 

 

Reply from 8.8.8.8: bytes=32 time=14ms TTL=117
Reply from 8.8.8.8: bytes=32 time=14ms TTL=117
Reply from 8.8.8.8: bytes=32 time=15ms TTL=117
Reply from 8.8.8.8: bytes=32 time=15ms TTL=117
Reply from 8.8.8.8: bytes=32 time=15ms TTL=117
Reply from 8.8.8.8: bytes=32 time=18ms TTL=117
Reply from 8.8.8.8: bytes=32 time=15ms TTL=117
Reply from 8.8.8.8: bytes=32 time=16ms TTL=117
Reply from 8.8.8.8: bytes=32 time=24ms TTL=117
Reply from 8.8.8.8: bytes=32 time=1697ms TTL=117
Reply from 8.8.8.8: bytes=32 time=2449ms TTL=117
Reply from 8.8.8.8: bytes=32 time=1415ms TTL=117
Reply from 8.8.8.8: bytes=32 time=805ms TTL=117
Reply from 8.8.8.8: bytes=32 time=15ms TTL=117
Reply from 8.8.8.8: bytes=32 time=43ms TTL=117
Reply from 8.8.8.8: bytes=32 time=15ms TTL=117
Reply from 8.8.8.8: bytes=32 time=19ms TTL=117
Reply from 8.8.8.8: bytes=32 time=15ms TTL=117
Reply from 8.8.8.8: bytes=32 time=14ms TTL=117

 

 

 

This happens no matter what VLAN/Network I'm attached to at the site. In fact, the above snip is from me pinging while L2TP'd into the site having problems. It's continuous like the above, spiking every 40 seconds. Reboot/reset hasn't changed anything. It does it no matter the time of day or connected users.

Highlighted
New Member
Posts: 1
Registered: Saturday

Re: Latency Spike every 40-50 seconds

Will say that similar lag is happening at my location with a USG router, the throughput is great except for specific spikes of lag exactly on 65 second intervals.

 

New Member
Posts: 3
Registered: ‎02-10-2016

Re: Latency Spike every 40-50 seconds

Im having the same issue.. runnning the latest firmware , usg4p, 100 ap's/

 

Im still struggling with this one.

 

 

Reply