3 weeks ago
10G Home Data Center
Used Products
×1
×1
×1
×1
×1
×1
×1
×1
×1
×1
×1
Location
Glendale, OH 45246, USA
Description

I couldn't figure out how to cross-post, so I duplicated (also in video).  If someone can do that, this can be deleted.

 

This is an update to a previous story.  I moved, and seriously upgraded my home network to a full on 10G dual fiber Data Center.  The full story is on my wiki, but I took a spare 17'x11' room in the basement, UniFi'ed the crap out of it, added a bunch of power and AC, and below is the result.

 

It's a biggish house, and I wanted to go all out, no compromises with this build since this is our "forever" home.  I'm also going big into home automation, so I wanted a very reliable network with as many things on Ethernet as I could to free up WiFi spectrum.  I'm very pleased with the outcome, my WiFi is providing 400Mbps download speeds.

 

* Cincinnati Bell 1G down/250M up FTTH primary, 200M/6M Spectrum cable fail-over WAN

* XG-8 10G Router with full IPS/IDS enabled

* 4 XG-16 core switches, dual 10G SFP+ LAG uplinks

* 3 US-48 (1 750W POE, 1 48-L2-POE, 1 US-48) + 1 US-16 POE Distribution switches

* 8 US-8 (some POE, some powered by POE) local switches

* 6 UAP-AC-HD WiFi access points

* 1 UAS-XG UniFi and UVC server

* 5 UVC-Pro

* 4 UVC-G3 Cameras

* 5 UVC-Micro cameras

* 5 Ring Floodlight cameras

* 2 Ring Doorbell cameras

* 7 Sonos Connect Amp for audio

* 3 Tivo Bolt for TV

* 2 HD HomeRun for local TV

* 3 Dell rack mount severs (Proxmox and FreeNAS)

* 1 Rack mount Mac Mini server

* 7 VLAN

* All private services (web, email, wiki, Plex, NextCloud)

* Sonnet 10G copper NIC for local PCs

* Dual 10G fiber to 4 PC locations

* Cat6a cabling throughout (80 drops)

* 1.5 ton mini-split AC

* 6 1500VA CyberPower UPS 

 

Network_top.jpgNetwokring Top

 

 

 

 

 

Network_bottom.jpgNetworking Bottom

 

 

Server_top.jpgServer Top

 

 

Entertain_bottom.jpgEntertainment Bottom

 

Entertain_top.jpgEntertainment top

 

Server_bottom.jpgServer bottom

 

Rack_front.jpgFront of Racks

 

Wire_patch15.jpgCable trough

 

Door6.jpgEntrace to my lair

 

 

 

Cameras_status.jpgCamera StatusCameras_all.jpgUVC Cameras

 

My old setups

https://community.ubnt.com/t5/UniFi-Stories/UniFi-house-network-large-with-FTTH-Fioptics-USG-XG-US-8-and-UAP/cns-p/1801850

https://community.ubnt.com/t5/UniFi-Stories/Ubiquiti-Upgrades-for-my-Fioptics-Installation/cns-p/1705612

 

 

10G Home Data Center

by 3 weeks ago

I couldn't figure out how to cross-post, so I duplicated (also in video).  If someone can do that, this can be deleted.

 

This is an update to a previous story.  I moved, and seriously upgraded my home network to a full on 10G dual fiber Data Center.  The full story is on my wiki, but I took a spare 17'x11' room in the basement, UniFi'ed the crap out of it, added a bunch of power and AC, and below is the result.

 

It's a biggish house, and I wanted to go all out, no compromises with this build since this is our "forever" home.  I'm also going big into home automation, so I wanted a very reliable network with as many things on Ethernet as I could to free up WiFi spectrum.  I'm very pleased with the outcome, my WiFi is providing 400Mbps download speeds.

 

* Cincinnati Bell 1G down/250M up FTTH primary, 200M/6M Spectrum cable fail-over WAN

* XG-8 10G Router with full IPS/IDS enabled

* 4 XG-16 core switches, dual 10G SFP+ LAG uplinks

* 3 US-48 (1 750W POE, 1 48-L2-POE, 1 US-48) + 1 US-16 POE Distribution switches

* 8 US-8 (some POE, some powered by POE) local switches

* 6 UAP-AC-HD WiFi access points

* 1 UAS-XG UniFi and UVC server

* 5 UVC-Pro

* 4 UVC-G3 Cameras

* 5 UVC-Micro cameras

* 5 Ring Floodlight cameras

* 2 Ring Doorbell cameras

* 7 Sonos Connect Amp for audio

* 3 Tivo Bolt for TV

* 2 HD HomeRun for local TV

* 3 Dell rack mount severs (Proxmox and FreeNAS)

* 1 Rack mount Mac Mini server

* 7 VLAN

* All private services (web, email, wiki, Plex, NextCloud)

* Sonnet 10G copper NIC for local PCs

* Dual 10G fiber to 4 PC locations

* Cat6a cabling throughout (80 drops)

* 1.5 ton mini-split AC

* 6 1500VA CyberPower UPS 

 

Network_top.jpgNetwokring Top

 

 

 

 

 

Network_bottom.jpgNetworking Bottom

 

 

Server_top.jpgServer Top

 

 

Entertain_bottom.jpgEntertainment Bottom

 

Entertain_top.jpgEntertainment top

 

Server_bottom.jpgServer bottom

 

Rack_front.jpgFront of Racks

 

Wire_patch15.jpgCable trough

 

Door6.jpgEntrace to my lair

 

 

 

Cameras_status.jpgCamera StatusCameras_all.jpgUVC Cameras

 

My old setups

https://community.ubnt.com/t5/UniFi-Stories/UniFi-house-network-large-with-FTTH-Fioptics-USG-XG-US-8...

https://community.ubnt.com/t5/UniFi-Stories/Ubiquiti-Upgrades-for-my-Fioptics-Installation/cns-p/170...

 

 

{"location":{"title":"Glendale, OH 45246, USA","placeId":"ChIJ5ZpHE-BNQIgROs400sOI628"},"addedProducts":[{"id":"unifi-video-camera-g3","count":1},{"id":"unifi-video-g3-micro","count":1},{"id":"unifi-video-g3-pro-camera","count":1}],"solved":"","numbers":"","description":"","mainImage":"192423i91AA3D85F80BE225"

 

 

{"location":{"title":"Glendale, OH 45246, USA","placeId":"ChIJ5ZpHE-BNQIgROs400sOI628"},"addedProducts":[{"id":"unifi-video-camer

{"location":{"title":"Glendale, OH 45246, USA","placeId":"ChIJ5ZpHE-BNQIgROs400sOI628"},"addedProducts":[{"id":"l2-managed-poe-gigabit-switch-with-sfp","count":1},{"id":"unifi-application-server","count":1},{"id":"unifi-switch-8-60w-b","count":1},{"id":"unifi-switch-8-150w","count":1},{"id":"unifi-security-gateway-xg","count":1},{"id":"unifi-switch-16-150w","count":1},{"id":"unifi-switch-16-xg-beta","count":1},{"id":"unifi-switch-48","count":1},{"id":"unifiswitch-48-750w","count":1},{"id":"unifi-ac-hd","count":1},{"id":"unifi-switch-8-beta","count":1}],"solved":"","numbers":"","description":"","mainImage":"192426i2ED0A8D0118FD447"}

Comments
by
2 weeks ago

Why do you have so many Sonos Connect AMP sitting together? Are you running your audio cables from your racks?

With that much 10GB and POE, how did you decide to handle ground?

by
2 weeks ago

Quite some setup. Well done! Love it.

by
2 weeks ago

What a dream of a setup!

 

Out of interest, which UniFi cameras have you got that run on WiFi? I thought all their solutions were wired?

 

And how are you finding the Security Gateway XG? Do you know if it can handle multuple WAN IPs?

by
2 weeks ago

@ACarneiro The USG-Micro cameras are WiFi only cameras. They do have the option of being powered via POE but they still send there "signal" over WiFi.

 

@bferrell How do you like the Sonos Connect AMPS? I've been thinking about using them in a house that I had wired up.

by
2 weeks ago

@demonmaestro I love the Amps, they're expensive, and they're not getting Airplay2 (at least the classic version, the new ones might), so I picked up some second hand on eBay to get a little savings, but I think it's a really good way (maybe the 'right way' even) to do whole-house audio, at least for me.  They allow for each room to be a zone, and any device (iOS/Mac/whatever) to adjust the source, volume, group and ungroup into as many unique groups as you want.   So, the basement units could all be playing music, and I could have the first floor on the Superbowl from my Tivo (they have optical out, which I route to a TOSLINK to RCA converter to the RCA input of the Amps).  I have a hardwired volume knob in each room in case someone gets punky with the volume control in another part of the house as well.  The grouping feature works really well, and keeps the sound very well sync'ed.  I have another set of 4 from the "existing" part of the house in a separate room from the remodel, so I get pretty full coverage.

 

@ACarneiro I like the USG, it has multiple WAN input, but I haven't gotten that to fully work in the versions I'm running (5.8.30 and firmware 4.4.29 - I'm a little leary of both upgrades to 5.9 and 5.10 based on folk's feedback and how rare the XG-8 is still).  I can't seem to get dyndns to work on both WAN ports if you try to use the same provider for both, and although WAN2 does fail over to the cable backup, it does not properly reroute my incoming port forward rules.  I assume it will come, and other than that I'm pretty happy.  Setting it up was a chore early on but they've dealt with most of that.

 

@BrianAker They're together because I think it's a cleaner install.  Yes, all of the 4 wire audio cables from the indivdiual room volume controls are routed to a patch panel on the wall, and then to the racks.  I also have an RCA cable from a TOSLINK/optical for each from Tivos around the house so I can feed TV audio to the in ceiling speakers for like Superbowl or World Series/Olympics parties.  This keeps me from having to have a separate "audio closet" around the house with power/AC/networking for the Sonos'.

 

I probably haven't thought about grounding as much as I should, but I have Belden 6a cable throughout, and the patches are 6a with grounding straps and the racks have grounding straps as well.  I installed 8 new circuits into a subpanel to power the data center, so I can ground to that common location, and both incoming WANs are connected to APC PNET1GB isolators for surge protection, and everyting in the rack is fed from one of several CyberPower 1500VA UPS/surge protectors.  If there's more I should consider I would be interested in your thoughts.  Right now I don't think the network gear or wire is limiting me on speed, I think my computers are.  On iPerf3 I get a bit under 7Gbps on my SSD based MacOS devices, and my QNAP NAS accepts transfers at about 250MBps, but even copying a movie to it saturates it about 1/2 through and it drops to aboue 120MBps sustained (it as 10GB Seagate Iron Wolf Pro NAS drives, so they're good quality, but not particularly fast).  On the POE, although I can drive a lot of power, I'm not really.  I have 6 US-8 switches, and I'll have 8 AC-HD access points, one POE Ring doorbell and 9 UVC POE cameras, I'm powering the Micros locally.

by
2 weeks ago

@bferrell How are your PowerEdge servers. Do they sound like a jet taking off or did you do the fan mod?

I use to have a couple of the PowerEdge 2950s... Well all I got to say is if I wanted to stay warm all I needed to do was to sit behind the rack for a short period of time.Out

by
2 weeks ago

@demonmaestro Yea, when they boot they are really, really loud.  Or, if I remove the side cover from the rack (or open the case!).  And they generate a lot of heat.  I do have the room conditioned to the mid 60's F, though, so that keeps the fan speed reasonable, and my desk is in the "warm aisle", so it's kind of nice if I'm in there for awhile to get some of the heat.  I actually used to think some of the UBNT gear was a little noisy, but I have to admit the server room isn't as loud as I expected.  At work we're required to wear hearing protection to go into the data center, but I don't think mine is anywhere near noisy enough for that.

by
2 weeks ago

@bferrell heat protection?! Idea

by
2 weeks ago - last edited 2 weeks ago

Hearing protection (ear plugs)... not heat..  but I sometimes take a jacket, so... kind of.  Smiley Happy  The jacket is optional.

by
2 weeks ago

Oh.. Got talking about thermals and I had heat on the mind.. lol

Yea, A jacket in a datacenter can be useful. Although the last datacenter I went into seemed a wee bit warm to me. From what I've hear is that some datacenters are raising the temps to 75F-80F.