Upcoming Maintenance Alert:

The UBNT Community will be upgraded at 5pm MDT on April 25th. During this time the community forums will be set to read-only status.

Learn more

×
Reply
Highlighted
New Member
Posts: 19
Registered: ‎03-31-2013
Kudos: 9
Accepted Solution

USG's Keep Restarting

I have 4 USG's deployed at client apartment buildings (along with UAPs). These USG's keep randomly restarting. I am running the latest controller version, 4.6.3, and the USG's are on firmware version 4.2.6.4751786.

 

Searching these boards I have seen similar reports of reboots and that clients watching videos/YouTube are thought to be a possible cause, but no definitive answer has been found.

 

Has anyone found any solution or fix? Has Ubiquity accepted there in an issue?  I am about to pull the units and replace them with different routers, but that defeats their purpose of the unified controller for ease of management.

 

I would be happy to help with troubleshooting, but would need some guidance with what to do.

 

Thanks!

 

 


Accepted Solutions
Ubiquiti Employee
Posts: 6,441
Registered: ‎01-28-2013
Kudos: 6218
Solutions: 563
Contributions: 20

Re: USG's Keep Restarting

[ Edited ]

For anyone that reads this, the firmware with the fix for this is included in controller version 4.6.6. It provides stable operation while leaving hardware offload enabled.

 

Cheers,

Mike

 

EDIT: currently the controller is in soaking status (near stable), so it's only avilable in the beta section. If you can't access the above post please see THIS KB article.

UBNT_Alternate_Logo.png

View solution in original post


All Replies
Established Member
Posts: 1,664
Registered: ‎08-27-2010
Kudos: 930
Solutions: 43

Re: USG's Keep Restarting

I have a few USG's deployed and I have not had restart issues lately.  When the USG first came out and I changed the core subnet, it would reboot.  but with the new upgrade, it stopped. 

I can't teach you anything, I can only make you think
Don't forget to mark your thread as solved and to give kudos
New Member
Posts: 19
Registered: ‎03-31-2013
Kudos: 9

Re: USG's Keep Restarting

The USG's I have deployed all have had their main subnet changed. I dont know if that could be the issue? If it was I would think more people would be reporting this issue.

 

I have had this issue for awhile now. It was ocuring on the pervious controller/firmware version and also on this current one.

Ubiquiti Employee
Posts: 6,441
Registered: ‎01-28-2013
Kudos: 6218
Solutions: 563
Contributions: 20

Re: USG's Keep Restarting

My main subnet has always been different and I do not see this issue, even more specifically with YouTube on this firmware. I haven't checked with who was working on the other reported thread that I commented on, but I do know that disabling hardware offload fixes the issue. The issue in that case was classification by the offload engine triggering a kernel panic. I personally can't duplicate that on 4.2.6, but I can on other versions (internal releases), on that specific site and just on YouTube.

 

The above statement is assuming that is the same issue. The first thing I would do is setup a syslog server (just temporarily). It may capture an issue prior to the crash, which is triggering the crash. It's possible that the actual crash won't show though, meaning the only way to determine what is happening would be to connect via a console cable. 

Have you done any testing on any of the units? If so, try YouTube and see if the unit reboots shortly thereafter. If it doesn't appear to try going through a few videos and see if it happens then. Basically by that I mean just click on one along the side bar, let it load a bit, change to another. You can also manually change the quality (which is like a new stream). See if after loading  a few different streams it crashes. In some cases I needed to do that to make it crash (at least on these dev firmwares). 

 

What type of WAN is at these sites?   

 

I wasn't able to find the ticket in trac to look up any info about this (not sure the ticket ID, and likely using the wrong keywords to search), so I'll inquire about this during a meeting later on today. I hope to have something more on that to share later too. 

 

Cheers,

Mike

UBNT_Alternate_Logo.png
Member
Posts: 230
Registered: ‎08-02-2013
Kudos: 119
Solutions: 4

Re: USG's Keep Restarting

I've experienced it on the current firmware/controller, although only once that I know of

New Member
Posts: 19
Registered: ‎03-31-2013
Kudos: 9

Re: USG's Keep Restarting

Thanks for your reply Mike.  I just opened a support ticket to better track the issue, support request # 175238.

 

One of the USG's restarted again today....

 

As I am not at these sites often I have not been able to do what you have suggest as of yet.  All of the sites have Comcast for their WAN's. The USG's connect directly to the Comcast gateway and have a static WAN IP address setup on the USG.

 

I will work on setting up a syslog server when I get some free time this week.

 

Thanks!

Regular Member
Posts: 512
Registered: ‎03-30-2012
Kudos: 269
Solutions: 10

Re: USG's Keep Restarting

I have this issue with only one of my USG's. it's a wierd one.

HNIC - www.thesnowcloud.com
New Member
Posts: 19
Registered: ‎03-31-2013
Kudos: 9

Re: USG's Keep Restarting

I have now setup a syslog server and will wait for a USG to restart to see if it logs anything.

New Member
Posts: 19
Registered: ‎03-31-2013
Kudos: 9

Re: USG's Keep Restarting

[ Edited ]

I am not sure what if there is any helpful info in this, but here is the syslog entries for a restart that occurred 2015-05-03 at approx 23:04pm:

 

2015-05-03 22:53:03	Daemon.Info	x.x.x.73	May  3 22:52:52 (BZ2LR,0418d64e1bd4,v3.2.10.2886) hostapd: ath1: STA x:x:x:x:x:89 IEEE 802.11: disassociated
2015-05-03 22:53:04	Daemon.Info	x.x.x.73	May  3 22:52:52 (BZ2LR,0418d64e1bd4,v3.2.10.2886) hostapd: ath1: STA x:x:x:x:x:89 IEEE 802.11: disassociated
2015-05-03 22:53:33	Daemon.Info	x.x.x.73	May  3 22:53:22 (BZ2LR,0418d64e1bd4,v3.2.10.2886) hostapd: ath1: STA x:x:x:x:x:89 IEEE 802.11: disassociated
2015-05-03 23:02:35	Daemon.Info	x.x.x.73	May  3 23:02:24 (BZ2LR,0418d64620c3,v3.2.10.2886) hostapd: ath1: STA x:x:x:x:x:10 IEEE 802.11: sta_stats
2015-05-03 23:02:35	Daemon.Info	x.x.x.73	May  3 23:02:24 (BZ2LR,0418d64620c3,v3.2.10.2886) hostapd: ath1: STA x:x:x:x:x:10 IEEE 802.11: disassociated
2015-05-03 23:02:35	User.Info	x.x.x.73	May  3 23:02:24 (BZ2LR,0418d64620c3,v3.2.10.2886) syslog: wevent.recv_msg(): EVENT_STA_LEAVE ath1: 1
2015-05-03 23:02:35	Daemon.Info	x.x.x.73	May  3 23:02:24 (BZ2LR,0418d64620c3,v3.2.10.2886) hostapd: ath1: STA x:x:x:x:x:10 IEEE 802.11: associated
2015-05-03 23:02:35	Daemon.Info	x.x.x.73	May  3 23:02:24 (BZ2LR,0418d64620c3,v3.2.10.2886) hostapd: ath1: STA x:x:x:x:x:10 RADIUS: starting accounting session 00000029-00000070
2015-05-03 23:02:35	Daemon.Info	x.x.x.73	May  3 23:02:24 (BZ2LR,0418d64620c3,v3.2.10.2886) hostapd: ath1: STA x:x:x:x:x:10 WPA: pairwise key handshake completed (RSN)
2015-05-03 23:02:36	User.Info	x.x.x.73	May  3 23:02:24 (BZ2LR,0418d64620c3,v3.2.10.2886) syslog: wevent.recv_msg(): EVENT_STA_JOIN ath1: x:x:x:x:x:10 / 1
2015-05-03 23:02:36	Daemon.Error	x.x.x.73	May  3 23:02:25 Greystar360Gateway dhcpd: data: host_decl_name: not available
2015-05-03 23:04:03	Daemon.Error	x.x.x.73	May  3 23:03:52 Greystar360Gateway dhcpd: data: host_decl_name: not available
2015-05-03 23:04:23	User.Info	x.x.x.73	May  3 23:04:12 (BZ2LR,0418d64620c3,v3.2.10.2886) syslog: wevent.recv_msg(): EVENT_STA_LEAVE ath1: 1
2015-05-03 23:04:23	Daemon.Info	x.x.x.73	May  3 23:04:12 (BZ2LR,0418d64620c3,v3.2.10.2886) hostapd: ath1: STA x:x:x:x:x:10 IEEE 802.11: sta_stats
2015-05-03 23:04:23	Daemon.Info	x.x.x.73	May  3 23:04:12 (BZ2LR,0418d64620c3,v3.2.10.2886) hostapd: ath1: STA x:x:x:x:x:10 IEEE 802.11: disassociated
2015-05-03 23:04:25	User.Info	x.x.x.73	May  3 23:04:14 (BZ2LR,0418d64620c3,v3.2.10.2886) syslog: wevent.recv_msg(): EVENT_STA_JOIN ath1: x:x:x:x:x:10 / 1
2015-05-03 23:04:25	Daemon.Info	x.x.x.73	May  3 23:04:14 (BZ2LR,0418d64620c3,v3.2.10.2886) hostapd: ath1: STA x:x:x:x:x:10 IEEE 802.11: associated
2015-05-03 23:04:25	Daemon.Info	x.x.x.73	May  3 23:04:14 (BZ2LR,0418d64620c3,v3.2.10.2886) hostapd: ath1: STA x:x:x:x:x:10 RADIUS: starting accounting session 00000029-00000071
2015-05-03 23:04:25	Daemon.Info	x.x.x.73	May  3 23:04:14 (BZ2LR,0418d64620c3,v3.2.10.2886) hostapd: ath1: STA x:x:x:x:x:10 WPA: pairwise key handshake completed (RSN)
2015-05-03 23:05:11	Daemon.Error	x.x.x.73	May  3 23:05:00 Greystar360Gateway dhcpd: last message repeated 3 times
2015-05-03 23:10:54	User.Info	x.x.x.73	May  3 23:10:43 (BZ2LR,0418d64620c3,v3.2.10.2886) syslog: wevent.recv_msg(): EVENT_STA_LEAVE ath1: 3
2015-05-03 23:10:54	Daemon.Info	x.x.x.73	May  3 23:10:43 (BZ2LR,0418d64620c3,v3.2.10.2886) hostapd: ath1: STA x:x:x:x:x:2e IEEE 802.11: sta_stats
2015-05-03 23:10:54	Daemon.Info	x.x.x.73	May  3 23:10:43 (BZ2LR,0418d64620c3,v3.2.10.2886) hostapd: ath1: STA x:x:x:x:x:2e IEEE 802.11: disassociated
2015-05-03 23:10:54	Daemon.Info	x.x.x.73	May  3 23:10:43 (BZ2LR,0418d64620c3,v3.2.10.2886) hostapd: ath1: STA x:x:x:x:x:2e IEEE 802.11: associated
2015-05-03 23:10:54	Daemon.Info	x.x.x.73	May  3 23:10:43 (BZ2LR,0418d64620c3,v3.2.10.2886) hostapd: ath1: STA x:x:x:x:x:2e RADIUS: starting accounting session 00000029-00000072
2015-05-03 23:10:54	Daemon.Info	x.x.x.73	May  3 23:10:43 (BZ2LR,0418d64620c3,v3.2.10.2886) hostapd: ath1: STA x:x:x:x:x:2e WPA: pairwise key handshake completed (RSN)
2015-05-03 23:10:55	User.Info	x.x.x.73	May  3 23:10:44 (BZ2LR,0418d64620c3,v3.2.10.2886) syslog: wevent.recv_msg(): EVENT_STA_JOIN ath1: x:x:x:x:x:2e / 2

 

Ubiquiti Employee
Posts: 6,441
Registered: ‎01-28-2013
Kudos: 6218
Solutions: 563
Contributions: 20

Re: USG's Keep Restarting

Thanks for the report. It doesn't look like what is triggering the crash, is showing in the log (was afraid of that). I'm wondering if it's related to the hardware offload engine. Can you issue the following to each USG via SSH?

configure
set system offload ipv4 pppoe disable
set system offload ipv4 vlan disable
set system offload ipv4 forwarding disable
set system offload ipv6 vlan disable
set system offload ipv6 forwarding disable
commit;save;exit

It shouldn't have any noticeable impact and I suspect that they will stop the restarting. If you're willing to do this I would be interested to know what is happening either way. This isn't my permanent solution, but it's potentially an immediate solution.

 

 

Cheers,

Mike

UBNT_Alternate_Logo.png
Ubiquiti Employee
Posts: 6,441
Registered: ‎01-28-2013
Kudos: 6218
Solutions: 563
Contributions: 20

Re: USG's Keep Restarting

In case you wanted to know how to revert, you would issue:

configure
set system offload ipv4 forwarding enable
set system offload ipv4 pppoe enable
set system offload ipv4 vlan enable
set system offload ipv6 forwarding enable
set system offload ipv6 vlan enable
commit;save;exit

Slightly different order when enabling compared to disabling..

 

Cheers,

Mike

UBNT_Alternate_Logo.png
New Member
Posts: 12
Registered: ‎01-22-2015
Kudos: 3

Re: USG's Keep Restarting

Looks like very similar behavior to my own USG.

 

I've detailed my constant restart problems in a dedicated post:

thttps://community.ubnt.com/t5/UniFi-Routing-Switching/Unstable-Security-Gateway/m-p/1247637#U1247637

 

I've been in contact with Ubiquiti Support, so I assume my issue is in the ticketing system.

 

Pretty frustrating.

New Member
Posts: 12
Registered: ‎01-22-2015
Kudos: 3

Re: USG's Keep Restarting

I attempted to perform the YouTube crash setup. Opened a lot of tabs, changed settings, had a good amount of traffic, but couldn't force it to restart. It might be a different problem I'm having, but I've disabled the items you had listed.

 

I'll update in a few days, reporting if the USG is still restarting or not.

 

Might be unrelated, but figured it might be helpful to try the suggestion on different setups.

Ubiquiti Employee
Posts: 6,441
Registered: ‎01-28-2013
Kudos: 6218
Solutions: 563
Contributions: 20

Re: USG's Keep Restarting


cjorgensen wrote:

I attempted to perform the YouTube crash setup. Opened a lot of tabs, changed settings, had a good amount of traffic, but couldn't force it to restart. It might be a different problem I'm having, but I've disabled the items you had listed.

 

I'll update in a few days, reporting if the USG is still restarting or not.

 

Might be unrelated, but figured it might be helpful to try the suggestion on different setups.


Thanks for testing and all the details in the other thread. Will be good to know if disabling the offload works. It looks like a panic due to the offload engine. It may be a similar issue, misclassifying the traffic, although I'm not entirely sure myself. That being said, I will get proper attention brought to this. I'm going to look for your ticket too. 

 

There are too many threads about random USG restarts, whether or not they are the same issue (personal opinion).

 

Thanks,

Mike

UBNT_Alternate_Logo.png
Ubiquiti Employee
Posts: 6,441
Registered: ‎01-28-2013
Kudos: 6218
Solutions: 563
Contributions: 20

Re: USG's Keep Restarting

[ Edited ]

swFrick wrote:

 

2015-05-03 23:02:36	Daemon.Error	x.x.x.73	May  3 23:02:25 Greystar360Gateway dhcpd: data: host_decl_name: not available
2015-05-03 23:04:03	Daemon.Error	x.x.x.73	May  3 23:03:52 Greystar360Gateway dhcpd: data: host_decl_name: not available

 


Actually I stand corrected. This is a similar error I saw right before a panic I could recreate in my lab (although I see more info using console cable, and it was on a different firmware version). Just wanted to mention that.. 

 

Cheers,

Mike

 

EDIT: this also appears to be 2 reboots back to back..

UBNT_Alternate_Logo.png
New Member
Posts: 19
Registered: ‎03-31-2013
Kudos: 9

Re: USG's Keep Restarting

Mike,

 

I have updated my ticket (#175238) with the info below and some logs (figured its better then trying to post lines of logs here):

 

Early today one of my USG's (I will call it USG1) restarted 3 times in 10 min. I have attached the syslog entries during that time and a screen shot from my controller listing the times (6:17am, 6:23am, 6:27am). I dont know if it is normal, but it shows it trying to reach the default inform address, http://unifi:8080/inform, when my controller is actually in the cloud with an address FQDN (listed in the ticket).

 

I have another USG (I will call it USG2), that over the last few days has been working fine but generating a growing number of syslog entries. Today at approx 11:10am it crashed and had to be power cycled for it to start working again. The USG's syslog was averaging a couple thousand entries per day but was over 20k today before it crashed.I have attached todays syslog for USG2.

New Member
Posts: 6
Registered: ‎05-08-2015

Re: USG's Keep Restarting

[ Edited ]

I am having the exact same 'random restart' issues, with multiple USG's.

This happens a couple of times a day, and it is really starting to get annoying. Sometimes it restarts 2-3 times within 15 minutes.

Does not seem to happen for days when virtually no one is on the network, load could be a factor perhaps ?

Regular Member
Posts: 510
Registered: ‎03-31-2015
Kudos: 174
Solutions: 5

Re: USG's Keep Restarting

Mine just happened to restart 2 days ago. Didn't make any config changes.

Senior Member
Posts: 3,664
Registered: ‎09-26-2013
Kudos: 910
Solutions: 260

Re: USG's Keep Restarting


JvL wrote:

I am having the exact same 'random restart' issues, with multiple USG's.

This happens a couple of times a day, and it is really starting to get annoying. Sometimes it restarts 2-3 times within 15 minutes.

Does not seem to happen for days when virtually no one is on the network, load could be a factor perhaps ?


There's a possible solution to your problem earlier in the thread here.

 

Cheers,

Andrew

New Member
Posts: 6
Registered: ‎05-08-2015

Re: USG's Keep Restarting

[ Edited ]

Disabling the hardware offloading didn't seem to help for the original poster of the 'restart bug' (see last post):

https://community.ubnt.com/t5/UniFi-Routing-Switching/Unstable-Security-Gateway/m-p/1247637#U1247637

He is still experiencing restarts of his USG after applying the suggested 'fix'.

Reply