4 weeks ago
Large Scale Unifi Controller
Used Products
×2
×2
×20
×99
×56
×6
×20
×16
×20
×10
×3
×5
×40
×4
×10
×15
×15
Location
Lafayette, LA, USA
Description

Unifi Controller Hardware for large scale deployments.. 

 

Our setup currently has 664 Unifi Devices and 108 Sites... All going back to a central Unifi Controller .. 

 

To be able to handle the load, we chose a supermicro server based on Intexl Xeon D-1541 (8 core 16 thread CPU), 64 GB of ECC RAM, and NVMe based SSD Storage.. 

 

 

With nightly backups to a RAID6 based storage array .. 

 

We are currently growing at a rate of about 1 site per work day.. 

 

On our core WAN side we use Edgemax products for the interconnection traffic between the sites.. These products talk back to our UNMS server.. 

unifi-sites-devices-clients.png

 

We have 13 main office locations inteconnected via fiber wan links... and the remainder of the sites are temporary job sites that move from physical location to physical location month to month .. and these are connected back over private APN LTE connections  that connect back to our datacenters over fiber .. back into our Edgmax setup... 

 

All these locations are interconnected via redundant fiber between 2 of our locations that we have built out datacenters in.. With full redundant power and and air conditioning.. We are currently exploring how to make the unifi and unms servers redundant.. As of right now we will have to restore a backup on a cold backup machine... I am hoping Ubiquiti will come out with a solution for us soon ... 

IMG_1166.jpg

 

IMG_1168.jpg

 

IMG_1169.jpg

 

A lot of the core side is still a work in progress.. But I will try to get more pictures for this post... 

 

 

Large Scale Unifi Controller

by 4 weeks ago

Unifi Controller Hardware for large scale deployments.. 

 

Our setup currently has 664 Unifi Devices and 108 Sites... All going back to a central Unifi Controller .. 

 

To be able to handle the load, we chose a supermicro server based on Intexl Xeon D-1541 (8 core 16 thread CPU), 64 GB of ECC RAM, and NVMe based SSD Storage.. 

 

 

With nightly backups to a RAID6 based storage array .. 

 

We are currently growing at a rate of about 1 site per work day.. 

 

On our core WAN side we use Edgemax products for the interconnection traffic between the sites.. These products talk back to our UNMS server.. 

unifi-sites-devices-clients.png

 

We have 13 main office locations inteconnected via fiber wan links... and the remainder of the sites are temporary job sites that move from physical location to physical location month to month .. and these are connected back over private APN LTE connections  that connect back to our datacenters over fiber .. back into our Edgmax setup... 

 

All these locations are interconnected via redundant fiber between 2 of our locations that we have built out datacenters in.. With full redundant power and and air conditioning.. We are currently exploring how to make the unifi and unms servers redundant.. As of right now we will have to restore a backup on a cold backup machine... I am hoping Ubiquiti will come out with a solution for us soon ... 

IMG_1166.jpg

 

IMG_1168.jpg

 

IMG_1169.jpg

 

A lot of the core side is still a work in progress.. But I will try to get more pictures for this post... 

 

 

{"location":{"title":"Lafayette, LA, USA","placeId":"ChIJm9jFZEOcJIYRE__3kiR1Ssk"},"addedProducts":[{"id":"l2-managed-poe-gigabit-switch-with-sfp","count":2},{"id":"inwall-ap","count":2},{"id":"unifi-ac-lite","count":20},{"id":"unifi-ac-mesh-ap","count":99},{"id":"unifi-ac-mesh-pro-ap-beta","count":56},{"id":"unifi-ac-pro","count":6},{"id":"unifi-direct-attach-copper-cable-10-gbps-1-meter","count":20},{"id":"unifi-switch-8-60w-b","count":16},{"id":"unifi-switch-8-150w","count":20},{"id":"unifiswitch-48-750w","count":10},{"id":"unifi-switch-16-xg-beta","count":3},{"id":"unifi-security-gateway-pro","count":5},{"id":"unifi-security-gateway","count":40},{"id":"unifi-switch-16-150w","count":4},{"id":"unifiswitch-24-250w","count":10},{"id":"unifi-switch-24","count":15},{"id":"unifi-switch-48","count":15}],"solved":"","numbers":"","description":"","mainImage":"166963iF10A067A07519D40"}

Comments
by
4 weeks ago

Looks great!  I see that you have a USG pro in your rack mixed in with the EdgeMAX devices.  What is the USG doing that the EdgeMAX devices aren't?  

by
4 weeks ago

Incredible setup and great use of the UBNT products! I would be interested to know if you are pegging the controller server at all? 

by
4 weeks ago

The USG is on our internernal / corporate network side... The Edgemax is on the core infrastructure side of the network .. 

by
4 weeks ago

verisarioc.. No our utilization is low on the controller .. Usually hovers between 89-97% idle cpu.. But our unifi system.settings is properly optimized for the number of sites / server capabilities..  Keeping the primary storage on NVMe also helps, as the software is not having to wait on the storage... 

 

by
4 weeks ago

Since this post this morning .. 

 

Devices 666
Clients 881
Sites 109
by
4 weeks ago

Incredible! I have dreamt about seeing these types of stats on a server of my own. I really wish my last job did this. Client count well over 28,000. Would have been over 3200 devices and 6 sites. Too bad they went with Cisco. 

 

I am sure the NVMe's are incredibly nice for the controller. No slowdown at all with stats. Those pages must be a blast to watch as it refreshes for that many clients! Does the database get large quickly? How long have you been running the setup? 

Do you have a fiber WAN connection? I see Ethernet CAT 6 possibly, CAT 7,  going to the USG, so possibly not?

 

Sorry for all the questions. What your doing is what I would love to do for that many clients. 

 

Good luck with your endeavours!

 

by
3 weeks ago

Sorry .. verisarioc.. Been a crazy week.. Yes.. we have fiber for our wan connections... 

 

We have been running this setup for a while... The Unifi server was recently upgraded from RAID5 spinning disk setup to the new server with NVMe as we were starting to load... 

 

I have everything going into the switches from the wan and then from the switches to the router... I always have all router ports plugged into the switches.. that way I can make major changes to where packets arrive remotely... So I have a ton a vlans that are for router to router connections that would normally be just a cable between them.. It allows me the freedom to remotely re-assign duties of routers... 

 

 

 

 

by
3 weeks ago

And in the datacenter setup.. we have 20Gbps (2 10Gbps links bonded) going from all major switches to the 10Gbps Unifi switches.. and the 10Gbps unifi switches are connected to each other via 20Gbps as well .. 

by
3 weeks ago

My only wish is that UBNT would finally one day put BGP configuration in the GUI of both Unifi and Edgemax.. I have to confgure on the cli for edgmax and config.gateway.json for the Unifi line.. My entire network is iBGP and eBGP... 

by
3 weeks ago

The main thing that drew me to UBNT is the ability to have a centralized controller ... It has really changed my quality of life... Being able to log into 1 place and troubleshoot / make configuration changes to devices all over the country.. Coming from a mikrotik environment where I had to login and configure each device in the chain... 

 

Now I have a centralized UNIFI controller .. and centralized UNMS controller... Life is good...