3 weeks ago
10G Home Data Center
Used Products
×1
×1
×1
×1
×1
×1
×1
×1
×1
×1
×1
Location
Glendale, OH 45246, USA
Description

I couldn't figure out how to cross-post, so I duplicated (also in video).  If someone can do that, this can be deleted.

 

This is an update to a previous story.  I moved, and seriously upgraded my home network to a full on 10G dual fiber Data Center.  The full story is on my wiki, but I took a spare 17'x11' room in the basement, UniFi'ed the crap out of it, added a bunch of power and AC, and below is the result.

 

It's a biggish house, and I wanted to go all out, no compromises with this build since this is our "forever" home.  I'm also going big into home automation, so I wanted a very reliable network with as many things on Ethernet as I could to free up WiFi spectrum.  I'm very pleased with the outcome, my WiFi is providing 400Mbps download speeds.

 

* Cincinnati Bell 1G down/250M up FTTH primary, 200M/6M Spectrum cable fail-over WAN

* XG-8 10G Router with full IPS/IDS enabled

* 4 XG-16 core switches, dual 10G SFP+ LAG uplinks

* 3 US-48 (1 750W POE, 1 48-L2-POE, 1 US-48) + 1 US-16 POE Distribution switches

* 8 US-8 (some POE, some powered by POE) local switches

* 6 UAP-AC-HD WiFi access points

* 1 UAS-XG UniFi and UVC server

* 5 UVC-Pro

* 4 UVC-G3 Cameras

* 5 UVC-Micro cameras

* 5 Ring Floodlight cameras

* 2 Ring Doorbell cameras

* 7 Sonos Connect Amp for audio

* 3 Tivo Bolt for TV

* 2 HD HomeRun for local TV

* 3 Dell rack mount severs (Proxmox and FreeNAS)

* 1 Rack mount Mac Mini server

* 7 VLAN

* All private services (web, email, wiki, Plex, NextCloud)

* Sonnet 10G copper NIC for local PCs

* Dual 10G fiber to 4 PC locations

* Cat6a cabling throughout (80 drops)

* 1.5 ton mini-split AC

* 6 1500VA CyberPower UPS 

 

Network_top.jpgNetwokring Top

 

 

 

 

 

Network_bottom.jpgNetworking Bottom

 

 

Server_top.jpgServer Top

 

 

Entertain_bottom.jpgEntertainment Bottom

 

Entertain_top.jpgEntertainment top

 

Server_bottom.jpgServer bottom

 

Rack_front.jpgFront of Racks

 

Wire_patch15.jpgCable trough

 

Door6.jpgEntrace to my lair

 

 

 

Cameras_status.jpgCamera StatusCameras_all.jpgUVC Cameras

 

My old setups

https://community.ubnt.com/t5/UniFi-Stories/UniFi-house-network-large-with-FTTH-Fioptics-USG-XG-US-8-and-UAP/cns-p/1801850

https://community.ubnt.com/t5/UniFi-Stories/Ubiquiti-Upgrades-for-my-Fioptics-Installation/cns-p/1705612

 

 

10G Home Data Center

by 3 weeks ago

I couldn't figure out how to cross-post, so I duplicated (also in video).  If someone can do that, this can be deleted.

 

This is an update to a previous story.  I moved, and seriously upgraded my home network to a full on 10G dual fiber Data Center.  The full story is on my wiki, but I took a spare 17'x11' room in the basement, UniFi'ed the crap out of it, added a bunch of power and AC, and below is the result.

 

It's a biggish house, and I wanted to go all out, no compromises with this build since this is our "forever" home.  I'm also going big into home automation, so I wanted a very reliable network with as many things on Ethernet as I could to free up WiFi spectrum.  I'm very pleased with the outcome, my WiFi is providing 400Mbps download speeds.

 

* Cincinnati Bell 1G down/250M up FTTH primary, 200M/6M Spectrum cable fail-over WAN

* XG-8 10G Router with full IPS/IDS enabled

* 4 XG-16 core switches, dual 10G SFP+ LAG uplinks

* 3 US-48 (1 750W POE, 1 48-L2-POE, 1 US-48) + 1 US-16 POE Distribution switches

* 8 US-8 (some POE, some powered by POE) local switches

* 6 UAP-AC-HD WiFi access points

* 1 UAS-XG UniFi and UVC server

* 5 UVC-Pro

* 4 UVC-G3 Cameras

* 5 UVC-Micro cameras

* 5 Ring Floodlight cameras

* 2 Ring Doorbell cameras

* 7 Sonos Connect Amp for audio

* 3 Tivo Bolt for TV

* 2 HD HomeRun for local TV

* 3 Dell rack mount severs (Proxmox and FreeNAS)

* 1 Rack mount Mac Mini server

* 7 VLAN

* All private services (web, email, wiki, Plex, NextCloud)

* Sonnet 10G copper NIC for local PCs

* Dual 10G fiber to 4 PC locations

* Cat6a cabling throughout (80 drops)

* 1.5 ton mini-split AC

* 6 1500VA CyberPower UPS 

 

Network_top.jpgNetwokring Top

 

 

 

 

 

Network_bottom.jpgNetworking Bottom

 

 

Server_top.jpgServer Top

 

 

Entertain_bottom.jpgEntertainment Bottom

 

Entertain_top.jpgEntertainment top

 

Server_bottom.jpgServer bottom

 

Rack_front.jpgFront of Racks

 

Wire_patch15.jpgCable trough

 

Door6.jpgEntrace to my lair

 

 

 

Cameras_status.jpgCamera StatusCameras_all.jpgUVC Cameras

 

My old setups

https://community.ubnt.com/t5/UniFi-Stories/UniFi-house-network-large-with-FTTH-Fioptics-USG-XG-US-8...

https://community.ubnt.com/t5/UniFi-Stories/Ubiquiti-Upgrades-for-my-Fioptics-Installation/cns-p/170...

 

 

{"location":{"title":"Glendale, OH 45246, USA","placeId":"ChIJ5ZpHE-BNQIgROs400sOI628"},"addedProducts":[{"id":"unifi-video-camera-g3","count":1},{"id":"unifi-video-g3-micro","count":1},{"id":"unifi-video-g3-pro-camera","count":1}],"solved":"","numbers":"","description":"","mainImage":"192423i91AA3D85F80BE225"

 

 

{"location":{"title":"Glendale, OH 45246, USA","placeId":"ChIJ5ZpHE-BNQIgROs400sOI628"},"addedProducts":[{"id":"unifi-video-camer

{"location":{"title":"Glendale, OH 45246, USA","placeId":"ChIJ5ZpHE-BNQIgROs400sOI628"},"addedProducts":[{"id":"l2-managed-poe-gigabit-switch-with-sfp","count":1},{"id":"unifi-application-server","count":1},{"id":"unifi-switch-8-60w-b","count":1},{"id":"unifi-switch-8-150w","count":1},{"id":"unifi-security-gateway-xg","count":1},{"id":"unifi-switch-16-150w","count":1},{"id":"unifi-switch-16-xg-beta","count":1},{"id":"unifi-switch-48","count":1},{"id":"unifiswitch-48-750w","count":1},{"id":"unifi-ac-hd","count":1},{"id":"unifi-switch-8-beta","count":1}],"solved":"","numbers":"","description":"","mainImage":"192426i2ED0A8D0118FD447"}

Comments
by
2 weeks ago

Wow, great setup!

 

Not sure if it's too late, but you'll get much better power efficiency if you power your stuff by 240V. Will need different UPSes of course...

You won't see real data centers with 120V PDUs down to racks, it's mostly three phase 240 or 208.

 

If I may suggest something for Home Automation - try UDI ISY + Insteon Man Happy

by
2 weeks ago

Forgive me if this is self evident, but what the is the purpose/need for "Dual 10G fiber to 4 PC locations" ?  

port redundancy? something else?

by
2 weeks ago - last edited 2 weeks ago

@FlyHappy ##Yea, not exactly, the idea is that I would link aggreagate them into a single 20G link to a local distribution switch.  LAG does double the overall pipe to the core switch, but a single stream cannot use both links, but if I have a local PC with a 10G NIC it could use 1/2 of LAG uplink, and the rest of the local users (printers, Tivo, Apple TV, etc.) can share the other 10G link.  Also, it does provide redudancy to that switch so that if either link goes down or is broken, the switch still has a 10G link to the core.

 

So, since the offices are the places where I'm likely to have 10G capable computers, and the latency is lower on fiber versus copper, I've allowed for these fiber links, the rest of the house is just Cat 6a copper.  I also ran dual copper to the offices, and right now have the computers connected to 1 and a 1G copper (US-8 typically) switch for the rest of the users as the Unifi 10G switches are still relatively expensive.

by
2 weeks ago

@apevnev For me it's too late.  I realize that 240V would be preferrble in some significant ways, and although I do have a 'home data center', that was bit further than I wanted to go.

by
a week ago

Man that is some serios setup! Congrats mate. It's a dream come true.

by
a week ago

@bferrell I'm not sure if anyone else has mentioned this or not. But regarding what you asked about differentiating between copper vs fiber, 1G vs 10G, this is what I use on my diagrams at work: black=1G copper, orange=1G MM fiber, yellow=1G SM fiber, red=10G SM fiber, aqua=10G MM. Also usually use violet for 40/100G links and/or just to identify OM3/OM4 fiber runs if I'm doing a physical cable map.

 

With that setup and two really good connections, that i assume aren't cheap, why do you do WAN failover and not some active/active load-balancing? Preference or is the functionality not really there with the router?

 

Did you do any sound proofing to that room? I ran a 1U R610 in my garage for a bit and it was like a small jet that I could hear inside.

 

Also, sorry for multiple post/notifications. I kept getting errors that it couldn't post but apparently was.

by
a week ago

@jasonb113083 No worries, and thanks for the thoughts on diagramming.

 

Sound proofing - that room is kind of unique in that it's in a basement hallway that leads to a garage, so it has mostly to exterior walls that are inuslated and bricked, then one short end is the garage, the other end is a closet, and above is a hallway.  So, there's not much to be bothered by the noise, but yes, the main door side and all of that assoicated wall was fully insulated, mostly since it's conditioned to a different temperature, and an exterior grade door was installed, so I don't note any noise when outside the room.  The servers are quite loud when booting, but I have to say that once running, at least at the temps I keep the room at (mid 60's F) the fans don't run really fast/loud.

 

On the WAN, there are a couple of things.  First, I think I mentioned elsewhere that the current cable download is quite a bit less than the Fiber (200M versus 1,000M), but the main reasons are:

  1. The router doesn't (at least for me, I'm open to the possiblity I'm doing it wrong, but I believe it's a documented bug) doesn't properly handle port forwarding for the servers I host on WAN2
  2. The upload speed on cable is 5M versus 250M on the fiber, and so the cable is not suitable for Plex or some of my other important services outbound

My current cable offering is basically "free" - my family wanted CATV so the internet is in the "package".  When I can get the failover working properly I'll pursue their 1G offering, but the upload on that is still weak, the Cable Company really thinks people only consume data.

 

Brett

 

Note:  to be fair, there seems to be some documentation here on how to properly configure WAN2 that I hadn't seen yet (looks like it was published a couple of weeks ago), so I will try this later.  But the point on the crappy cable upload speeds stands on why I don't load balance versus fail-over.  If the fiber is available, I want to use it.  If I could figure out how to load balance only the download side, I might try that.

by
a week ago

Nice setup!  Standing by for the Visio diagaram!  Can't wait.  (This may be a sign that I need to get out more...).

 

 

by
a week ago

@wrightts You're a little contrained in what you can post here, but you can see my vizio on my blog.

 

https://wiki.bdfserver.com/index.php/Camp_Kiwi_Data_Center#Network_Diagrams

by
Thursday

Awesome setup!

I can see you've used colour-coded patch cables.

Did you follow any standard, or decide yourself what colours to allocate each?