Reply
New Member
Posts: 9
Registered: ‎06-19-2018
Kudos: 4

Re: USG-Pro-4 Restarting every 2 days

Add another to the list, mine is rebooting a lot.

 

image.pngimage.png

New Member
Posts: 16
Registered: ‎11-21-2016
Kudos: 3

Re: USG-Pro-4 Restarting every 2 days

And another one to add to the list.  This is a brand new device, only deployed a few weeks ago.  I'm not on site so I don't know what hardware level it is, but it too is running 4.4.22 on a 5.8.24 controller.  

 

I've deployed loads of UAPs, a few switches, and I have a USG at home, and I've never had any issues with any of the kit, but this USG-Pro seems a bit of a pup to me.  The client is a 24 hour business with a 48 port PoE switch and about 12 APs, but it seems that the last 3 restarts have all occured around 96 hours apart.. it lost connection at 7.13am on July 3rd, the client had to do a power reset to get it up at 7.51 on 3rd July.  Then 95 hours later it restarted on 7th July,... then 96 hours later again this morning.  This is not great when they're about to deploy a VoIP system...

Capture.PNG
New Member
Posts: 16
Registered: ‎11-21-2016
Kudos: 3

Re: USG-Pro-4 Restarting every 2 days

Further to last post, I've been looking at the Performance stats (which I didn't really know was there!).

 

Looking at this USG Pro, the memory usage for the last few hours (since I enabled extra storage for 5 minute stats) has been a constant 4.00%.    Compared to my baby USG at home, where the memory constantly fluctuates, so 43.65%, 43.72%, 43..29%, 43.86%...etc etc

 

It seems odd that the memory is a constant 4.00%, even over 1 day it doesn't move from 4.00%

 

Anyone else's like this?

1 day.PNG
hourly.PNG
New Member
Posts: 20
Registered: ‎10-16-2017
Kudos: 4

Re: USG-Pro-4 Restarting every 2 days

Yes, it's the same for except that the value is lower since I upgraded my memory - I don't think that the restarts are caused by OOM errors or leaks - this is something you'd easily spot in this chart - unless it happens so fast that it crashes before it gets reported in the graph.
New Member
Posts: 16
Registered: ‎11-21-2016
Kudos: 3

Re: USG-Pro-4 Restarting every 2 days

I've now enabled 5 minute stats for a week now instead of 1 hour (made sure I upgraded disk space on my controller!) so when it happens again I can look back and see what the stats are saying, but as you say if its that quick, it may not register.

 

From reading this large thread it sounds like Ubiquiti are making some progress in finding the software fault, which I hope will be pushed out soon, as I'm now nervous about using USG Pro in the field until this issue is sorted.

Emerging Member
Posts: 95
Registered: ‎11-07-2016
Kudos: 1

Re: USG-Pro-4 Restarting every 2 days

I'm going to do that with mine once I get it back in sight of the controller. Disappeared randomly from it the other morning; both router and switch. Also, WiFi AP now won't get power all of a sudden, but that might be a separate issue entirely. Also, while not being in touch with the controller is another rather inoptimal way to stabilize the USG; since she fell out of sight, no rebootts. Not sure if this helps anyone, but re-adoption might be a way to mitigate things.
Emerging Member
Posts: 95
Registered: ‎11-07-2016
Kudos: 1

Re: USG-Pro-4 Restarting every 2 days

Going to do the same thing.
Member
Posts: 174
Registered: ‎02-27-2018
Kudos: 150
Solutions: 1

Re: USG-Pro-4 Restarting every 2 days

+1 for seeing random reboots as of late...  5.8.24 controller 4.4.22 USG code

 

Capture.PNG

New Member
Posts: 10
Registered: ‎11-13-2013
Kudos: 2
Solutions: 1

Re: USG-Pro-4 Restarting every 2 days

Same here
New Member
Posts: 21
Registered: ‎12-25-2017
Kudos: 33

Re: USG-Pro-4 Restarting every 2 days

Same thing happens to me... it was rebooting like crazy until I replaced the memory in it. Now it reboots every couple of days.

Established Member
Posts: 1,277
Registered: ‎05-25-2016
Kudos: 225
Solutions: 11

Re: USG-Pro-4 Restarting every 2 days

Swap it out with a regular usg mean while you will be surprised what it can put Man Wink
Ive done the same as a Work around for a coming customer.
Ubiquiti Employee
Posts: 4,927
Registered: ‎08-08-2016
Kudos: 5208
Solutions: 343

Re: USG-Pro-4 Restarting every 2 days


@Cambridgeport90 wrote:
In light of the number of posts, can someone please point me to the correct page where it points out whether the restarts are caused by hardware issues or not?

This thread has become useless because it covers every hardware issue, software issue, and unknown power outage that anyone's had in 2.5 years. We seem to have resolved the last outstanding problem, I'll lock this after folks can weigh in on that so it stops becoming a catch-all mess of unrelated things. 

 

To summarize the issues in this thread: 

1) A number of the early ones were software problems that have been resolved for a while. 

2) Some number of them were legit hardware problems. 

3) Some number of them fall into people unplugging the power cord or other power failures or other human-commanded reboots of which some other person was unaware. 

4) All of the currently-outstanding ones from there look to be attributable to a u-boot problem we resolved last week. We're up to double digit real world impacted systems upgraded, with 0 of them still exhibiting reboots. I'll be posting a follow up reply to this thread momentarily with info on how to apply this change. 

Ubiquiti Employee
Posts: 4,927
Registered: ‎08-08-2016
Kudos: 5208
Solutions: 343

Re: USG-Pro-4 Restarting every 2 days

[ Edited ]

We've identified the cause of what thus far appears to be all the still-outstanding problems here. We're into double-digit real world impacted systems upgraded, with 100% success rate - 0 reboots of upgraded systems, where some were rebooting several times a day, and most of them at least daily. Now up to several days of runtime on those. 

 

The fix is in u-boot, which is kind of like the BIOS on a PC. The new u-boot, and required tools to flash it are included in the following firmware. 

 

Download

  • USG Pro
    • md5:  5576395b1743079e64027d4980367e0d
    • sha256: 4befbb2bcb1f62d9e36e07c674235815ae2f26ce29fd803346f3b40f7e35811e

Note this only applies to USG Pro, the other platforms do not have this problem. 

 

Upgrading u-boot

For now, u-boot must be manually upgraded after upgrading to the above firmware. If you lose power at the wrong point during this process, you'll be left with a bricked system, so don't try it in the middle of a storm with the power flickering and no UPS or anything like that. Man Wink 

 

SSH to USG, and run:

sudo upgrade-uboot.e220

After it finishes, run 'reboot' to restart the system. 

 

The expected output of that is as follows. 

cmb@stable-s2-usg:~$ sudo upgrade-uboot.e220
++ /usr/sbin/ubnt-hal-e getBoardIdE
+ '[' e221 '!=' e221 ']'
+ uboot_file=/etc/ubnt/u-boot.ER-e220.e220.5102844-g4f1145c.bin.1
+ uboot_file_sha256=9e3b75cb248734a41c048ebb24d4af851ec7c6b37a4b29ae6f9504611cc5d1f8
+ flash_erase=/usr/sbin/flash_erase
+ flashcp=/usr/sbin/flashcp
+ mtd_dev=/dev/mtd1
+ required_files=($uboot_file $flash_erase $flashcp)
+ for file in '"${required_files[@]}"'
+ '[' '!' -f /etc/ubnt/u-boot.ER-e220.e220.5102844-g4f1145c.bin.1 ']'
+ for file in '"${required_files[@]}"'
+ '[' '!' -f /usr/sbin/flash_erase ']'
+ for file in '"${required_files[@]}"'
+ '[' '!' -f /usr/sbin/flashcp ']'
++ /usr/bin/sha256sum /dev/mtd1
++ cut -d ' ' -f 1
+ curr_mtd_sha=1bf3f7ffc3a5fe672d3402b2539329e801900e3ac19b3f13b0fe88ab9cbed211
+ '[' 1bf3f7ffc3a5fe672d3402b2539329e801900e3ac19b3f13b0fe88ab9cbed211 == 9e3b75cb248734a41c048ebb24d4af851ec7c6b37a4b29ae6f9504611cc5d1f8 ']'
+ /usr/sbin/flash_erase /dev/mtd1 0 0
Erasing 64 Kibyte @ 90000 -- 100 % complete 
+ /usr/sbin/flashcp /etc/ubnt/u-boot.ER-e220.e220.5102844-g4f1145c.bin.1 /dev/mtd1
cmb@stable-s2-usg:~$

Once it's on the new u-boot, that will exit without making any changes, as the sha256 will already match the new one. 

 

If you want to manually verify its version, after the reboot, run 'sudo sha256sum /dev/mtd1'. It should look like the following on the new version. 

cmb@stable-s2-usg:~$ sudo sha256sum /dev/mtd1
9e3b75cb248734a41c048ebb24d4af851ec7c6b37a4b29ae6f9504611cc5d1f8  /dev/mtd1

If you're still experiencing reboots, please give that a shot and report back. 

New Member
Posts: 11
Registered: ‎03-18-2018
Kudos: 1

Re: USG-Pro-4 Restarting every 2 days

[ Edited ]

Ok, from someone that doesn't play with network equipment all day... what happens between downloading the update and "sudo upgrade-uboot.e220"? Where am I supposed to place this file so that the command actually does something?

 

admin@Gateway:~$ sudo upgrade-uboot.e220

sudo: upgrade-uboot.e220: command not found

Ubiquiti Employee
Posts: 4,927
Registered: ‎08-08-2016
Kudos: 5208
Solutions: 343

Re: USG-Pro-4 Restarting every 2 days


@gnomehole wrote:

Ok, from someone that doesn't play with network equipment all day... what happens between downloading the update and "sudo upgrade-uboot.e220"? Where am I supposed to place this file so that the command actually does something?

 

admin@Gateway:~$ sudo upgrade-uboot.e220

sudo: upgrade-uboot.e220: command not found


You have to install the upgrade first, just like any other upgrade. In controller, Devices, click on USG, go to its Properties panel on the right. Click the Config tag, Manage Device, paste in the URL under Custom Upgrade and click the Custom Upgrade button. 

New Member
Posts: 11
Registered: ‎03-18-2018
Kudos: 1

Re: USG-Pro-4 Restarting every 2 days



You have to install the upgrade first, just like any other upgrade. In controller, Devices, click on USG, go to its Properties panel on the right. Click the Config tag, Manage Device, paste in the URL under Custom Upgrade and click the Custom Upgrade button. 


 

Thanks - easy enough.  Process went as advertised.  Crossing fingers!

Member
Posts: 174
Registered: ‎08-02-2016
Kudos: 110
Solutions: 1

Re: USG-Pro-4 Restarting every 2 days

[ Edited ]

@UBNT-cmb wrote:

 I'll lock this after folks can weigh in on that so it stops becoming a catch-all mess of unrelated things. 

  


Locking the thread will only result in new ones getting opened until (and a bit after) the fix is released as a stable (non-beta) release...  The problem is that anyone who uses the "official" release (currently 4.4.22) will still potentially have the problem, and so will still post threads...

 

 

 

Highlighted
Emerging Member
Posts: 95
Registered: ‎11-07-2016
Kudos: 1

Re: USG-Pro-4 Restarting every 2 days

Agreed.
New Member
Posts: 9
Registered: ‎06-19-2018
Kudos: 4

Re: USG-Pro-4 Restarting every 2 days

Upgraded as well here, everything went fine! Will report back tomorrow , cleared my logs for a fresh look.

New Member
Posts: 21
Registered: ‎10-05-2017
Kudos: 4
Solutions: 1

Re: USG-Pro-4 Restarting every 2 days

[ Edited ]

Huh. I did it but the SHA256 wasn't updated... Should I try again? I'm getting 9e3b75cb248734a41c048ebb24d4af851ec7c6b37a4b29ae6f9504611cc5d1f8

Reply