Reply
Established Member
Posts: 1,543
Registered: ‎03-23-2013
Kudos: 220
Solutions: 52

Mass config failed, all devices showing offline but aren't

So, this is no panic, the network is up, but I scheduled devices for upgrade last night.  When I got up this AM and checked AC2 all devices we're offline.

 

all devices offline.png

 

That's even better than coffee to get a guy out of bed and on his way to work.  Smiley Wink

 

 

Thanks to UNMS, and UCRM I confirmed that the network was in fact working, with the exception of 1 device that did fail to come back up after I rebooted the AC2 VM.


But my question is, what failed and how do I prevent this again?  Did I give AC2 too many devices at once?    I'm just looking for a way to avoid this again without wasting a bunch of device update windows trying to find the the right settings/device count to update at one time.

 

 

Here is the scheduled events I set up:  (the first 3 config changes was just moving 3 devices from HTTP to HTTPS)

 

event log.png

Veteran Member
Posts: 4,659
Registered: ‎03-02-2015
Kudos: 900
Solutions: 212

Re: Mass config failed, all devices showing offline but aren't

How much JVM heapsize (default Xmx1024m)you assigned to AC2?
It's very likely that server run out of memory for JVM. Parallel upgrades consume much RAM. check AC2 server logs.
===================================================
We all work for KUDOs here.
Thx
Established Member
Posts: 1,543
Registered: ‎03-23-2013
Kudos: 220
Solutions: 52

Re: Mass config failed, all devices showing offline but aren't


@Skipper0815 wrote:
How much JVM heapsize (default Xmx1024m)you assigned to AC2?
It's very likely that server run out of memory for JVM. Parallel upgrades consume much RAM. check AC2 server logs.

Not sure how to check the JVM heapsize at the moment, have to look that up but here is the memory chart from that server.  Looks like it's been hogging memory for a while.  After the reboot it's dramatically less.  Maybe there is something for UBNT to look into there.

 

Last ~6 months

 

mem one year.png

 

24 hours showing reboot

 

mem 24 hours.png

Established Member
Posts: 1,543
Registered: ‎03-23-2013
Kudos: 220
Solutions: 52

Re: Mass config failed, all devices showing offline but aren't

Forgot to mention, you can see it in the second chart, but this VM has 10 Gigs of RAM.

Veteran Member
Posts: 4,659
Registered: ‎03-02-2015
Kudos: 900
Solutions: 212

Re: Mass config failed, all devices showing offline but aren't

It effectively doesn't matter how much real or VM RAM you have....
JVM heapsize parameter (Xmx) is in 'airControl2Server.vmoptions'
I would assign 2048 or better 4096.
Howto here:
https://help.ubnt.com/hc/en-us/articles/205233000-airControl-Change-JVM-settings-like-memory-size-an...

Actual consumption in diagpage:
http://server_IP:9081/diag
===================================================
We all work for KUDOs here.
Thx
Established Member
Posts: 1,543
Registered: ‎03-23-2013
Kudos: 220
Solutions: 52

Re: Mass config failed, all devices showing offline but aren't


@Skipper0815 wrote:
It effectively doesn't matter how much real or VM RAM you have....
JVM heapsize parameter (Xmx) is in 'airControl2Server.vmoptions'
I would assign 2048 or better 4096.
Howto here:
https://help.ubnt.com/hc/en-us/articles/205233000-airControl-Change-JVM-settings-like-memory-size-an...

Actual consumption in diagpage:
http://server_IP:9081/diag

Are you meaning this?

mem.png

Veteran Member
Posts: 4,659
Registered: ‎03-02-2015
Kudos: 900
Solutions: 212

Re: Mass config failed, all devices showing offline but aren't

Yes that's default Java heapsize in AC2.
Xmx1024m which means just 1 GB.
parallel fw update tasks ( 50 ...100 or more ) consume much more memory than it's available.

Check AC2 server logs for java heap errors.
===================================================
We all work for KUDOs here.
Thx
Established Member
Posts: 1,543
Registered: ‎03-23-2013
Kudos: 220
Solutions: 52

Re: Mass config failed, all devices showing offline but aren't

Ok I'll resolve that asap. The VM has 10 Gigs and is dedicated to AC2, nothing else running on it. 

 

I hate (strongly dislike) Java.

Veteran Member
Posts: 4,659
Registered: ‎03-02-2015
Kudos: 900
Solutions: 212

Re: Mass config failed, all devices showing offline but aren't

I know. that's a VM issue of Java.
with 10GB ram i would suggest something like Xmx4096. that should solve your issues.

Happily these issues are gone some day with UNMS.
===================================================
We all work for KUDOs here.
Thx
Established Member
Posts: 1,543
Registered: ‎03-23-2013
Kudos: 220
Solutions: 52

Re: Mass config failed, all devices showing offline but aren't

Any particular reason to only use half the RAM?  I understand you have to leave something for the OS to use but there isn't anything else on this VM.

 

I set it at 9000m we'll see how it goes.

Veteran Member
Posts: 4,659
Registered: ‎03-02-2015
Kudos: 900
Solutions: 212

Re: Mass config failed, all devices showing offline but aren't

.....because around half or so... of real RAM is suggested by Oracle. See JVM manuals.
===================================================
We all work for KUDOs here.
Thx
Established Member
Posts: 1,543
Registered: ‎03-23-2013
Kudos: 220
Solutions: 52

Re: Mass config failed, all devices showing offline but aren't

Ok, well I guess we'll see what happens if I push that a bit.

Ubiquiti Employee
Posts: 732
Registered: ‎05-28-2012
Kudos: 374
Solutions: 77

Re: Mass config failed, all devices showing offline but aren't

Hi @Trendal,

 

yes, default of 1GB might not be enough for some operations. As @Skipper0815 suggested, some 4GB is quite enough, but of course you can allocate more, just don't forget to leave some for the OS itself. Can you send me server logs to aircontrol@ubnt.com, so I can inspect what actually happened when AC2 run out of memory.

Established Member
Posts: 1,543
Registered: ‎03-23-2013
Kudos: 220
Solutions: 52

Re: Mass config failed, all devices showing offline but aren't

I emailed the logs to that address.

 

For your interest here is last nights attempt to update 112 and 116 devices in 2 different groups.  1 failure was due to the device being offline, not sure about the other.

 

firmware updates.pngsuccess firmware upgrade.png

Veteran Member
Posts: 4,659
Registered: ‎03-02-2015
Kudos: 900
Solutions: 212

Re: Mass config failed, all devices showing offline but aren't

Check AC2 server logs why it failed, but seems all other updates going through now.
===================================================
We all work for KUDOs here.
Thx
Established Member
Posts: 1,543
Registered: ‎03-23-2013
Kudos: 220
Solutions: 52

Re: Mass config failed, all devices showing offline but aren't

@Skipper0815

1 unknown failure out of 228 devices upgraded remotely on schedule while I'm am blissfully sleeping is a smashing success.

Established Member
Posts: 1,543
Registered: ‎03-23-2013
Kudos: 220
Solutions: 52

Re: Mass config failed, all devices showing offline but aren't

If we were large enough to have hired goons I'd sick a goon on ferriting out the reason for that one failure but I have no goons and better things to do with my time.

Veteran Member
Posts: 4,659
Registered: ‎03-02-2015
Kudos: 900
Solutions: 212

Re: Mass config failed, all devices showing offline but aren't

Was just a hint but I'm glad your fw updates work now.
===================================================
We all work for KUDOs here.
Thx
Member
Posts: 293
Registered: ‎11-13-2015
Kudos: 32
Solutions: 5

Re: Mass config failed, all devices showing offline but aren't

I want to increase my memory also and cant find the "airControl service configuration panel"   what am I missing and where is the panel?

 

John

Highlighted
Veteran Member
Posts: 4,659
Registered: ‎03-02-2015
Kudos: 900
Solutions: 212

Re: Mass config failed, all devices showing offline but aren't

[ Edited ]

@Johnly

There is no service panel. (KB is outdated)

On linux
'airControl2Server.vmoptions' config file is in Aircontrol2 folder

afaik on Windows it should be in:
%USERPATH%\.aircontrol2.....
that usually links to something similar to:
c:\Users\<your_username>\.Aircontrol2 ....\
BUT I would recommend a file search, because I'm not sure about path on Windows.

 


reference:
https://help.ubnt.com/hc/en-us/articles/205233000-airControl-Change-JVM-settings-like-memory-size-an...

===================================================
We all work for KUDOs here.
Thx
Reply