02/01/2019
10G Home Data Center
Used Products
×1
×1
×1
×1
×1
×1
×1
×1
×1
×1
×1
Location
Glendale, OH 45246, USA
Description

I couldn't figure out how to cross-post, so I duplicated (also in video).  If someone can do that, this can be deleted.

 

This is an update to a previous story.  I moved, and seriously upgraded my home network to a full on 10G dual fiber Data Center.  The full story is on my wiki, but I took a spare 17'x11' room in the basement, UniFi'ed the crap out of it, added a bunch of power and AC, and below is the result.

 

It's a biggish house, and I wanted to go all out, no compromises with this build since this is our "forever" home.  I'm also going big into home automation, so I wanted a very reliable network with as many things on Ethernet as I could to free up WiFi spectrum.  I'm very pleased with the outcome, my WiFi is providing 400Mbps download speeds.

 

* Cincinnati Bell 1G down/250M up FTTH primary, 200M/6M Spectrum cable fail-over WAN

* XG-8 10G Router with full IPS/IDS enabled

* 4 XG-16 core switches, dual 10G SFP+ LAG uplinks

* 3 US-48 (1 750W POE, 1 48-L2-POE, 1 US-48) + 1 US-16 POE Distribution switches

* 8 US-8 (some POE, some powered by POE) local switches

* 6 UAP-AC-HD WiFi access points

* 1 UAS-XG UniFi and UVC server

* 5 UVC-Pro

* 4 UVC-G3 Cameras

* 5 UVC-Micro cameras

* 5 Ring Floodlight cameras

* 2 Ring Doorbell cameras

* 7 Sonos Connect Amp for audio

* 3 Tivo Bolt for TV

* 2 HD HomeRun for local TV

* 3 Dell rack mount severs (Proxmox and FreeNAS)

* 1 Rack mount Mac Mini server

* 7 VLAN

* All private services (web, email, wiki, Plex, NextCloud)

* Sonnet 10G copper NIC for local PCs

* Dual 10G fiber to 4 PC locations

* Cat6a cabling throughout (80 drops)

* 1.5 ton mini-split AC

* 6 1500VA CyberPower UPS 

 

Network_top.jpgNetwokring Top

 

 

 

 

 

Network_bottom.jpgNetworking Bottom

 

 

Server_top.jpgServer Top

 

 

Entertain_bottom.jpgEntertainment Bottom

 

Entertain_top.jpgEntertainment top

 

Server_bottom.jpgServer bottom

 

Rack_front.jpgFront of Racks

 

Wire_patch15.jpgCable trough

 

Door6.jpgEntrace to my lair

 

 

 

Cameras_status.jpgCamera StatusCameras_all.jpgUVC Cameras

 

My old setups

https://community.ubnt.com/t5/UniFi-Stories/UniFi-house-network-large-with-FTTH-Fioptics-USG-XG-US-8-and-UAP/cns-p/1801850

https://community.ubnt.com/t5/UniFi-Stories/Ubiquiti-Upgrades-for-my-Fioptics-Installation/cns-p/1705612

 

 

10G Home Data Center

by on ‎02-01-2019 06:35 AM

I couldn't figure out how to cross-post, so I duplicated (also in video).  If someone can do that, this can be deleted.

 

This is an update to a previous story.  I moved, and seriously upgraded my home network to a full on 10G dual fiber Data Center.  The full story is on my wiki, but I took a spare 17'x11' room in the basement, UniFi'ed the crap out of it, added a bunch of power and AC, and below is the result.

 

It's a biggish house, and I wanted to go all out, no compromises with this build since this is our "forever" home.  I'm also going big into home automation, so I wanted a very reliable network with as many things on Ethernet as I could to free up WiFi spectrum.  I'm very pleased with the outcome, my WiFi is providing 400Mbps download speeds.

 

* Cincinnati Bell 1G down/250M up FTTH primary, 200M/6M Spectrum cable fail-over WAN

* XG-8 10G Router with full IPS/IDS enabled

* 4 XG-16 core switches, dual 10G SFP+ LAG uplinks

* 3 US-48 (1 750W POE, 1 48-L2-POE, 1 US-48) + 1 US-16 POE Distribution switches

* 8 US-8 (some POE, some powered by POE) local switches

* 6 UAP-AC-HD WiFi access points

* 1 UAS-XG UniFi and UVC server

* 5 UVC-Pro

* 4 UVC-G3 Cameras

* 5 UVC-Micro cameras

* 5 Ring Floodlight cameras

* 2 Ring Doorbell cameras

* 7 Sonos Connect Amp for audio

* 3 Tivo Bolt for TV

* 2 HD HomeRun for local TV

* 3 Dell rack mount severs (Proxmox and FreeNAS)

* 1 Rack mount Mac Mini server

* 7 VLAN

* All private services (web, email, wiki, Plex, NextCloud)

* Sonnet 10G copper NIC for local PCs

* Dual 10G fiber to 4 PC locations

* Cat6a cabling throughout (80 drops)

* 1.5 ton mini-split AC

* 6 1500VA CyberPower UPS 

 

Network_top.jpgNetwokring Top

 

 

 

 

 

Network_bottom.jpgNetworking Bottom

 

 

Server_top.jpgServer Top

 

 

Entertain_bottom.jpgEntertainment Bottom

 

Entertain_top.jpgEntertainment top

 

Server_bottom.jpgServer bottom

 

Rack_front.jpgFront of Racks

 

Wire_patch15.jpgCable trough

 

Door6.jpgEntrace to my lair

 

 

 

Cameras_status.jpgCamera StatusCameras_all.jpgUVC Cameras

 

My old setups

https://community.ubnt.com/t5/UniFi-Stories/UniFi-house-network-large-with-FTTH-Fioptics-USG-XG-US-8...

https://community.ubnt.com/t5/UniFi-Stories/Ubiquiti-Upgrades-for-my-Fioptics-Installation/cns-p/170...

 

 

{"location":{"title":"Glendale, OH 45246, USA","placeId":"ChIJ5ZpHE-BNQIgROs400sOI628"},"addedProducts":[{"id":"unifi-video-camera-g3","count":1},{"id":"unifi-video-g3-micro","count":1},{"id":"unifi-video-g3-pro-camera","count":1}],"solved":"","numbers":"","description":"","mainImage":"192423i91AA3D85F80BE225"

 

 

{"location":{"title":"Glendale, OH 45246, USA","placeId":"ChIJ5ZpHE-BNQIgROs400sOI628"},"addedProducts":[{"id":"unifi-video-camer

{"location":{"title":"Glendale, OH 45246, USA","placeId":"ChIJ5ZpHE-BNQIgROs400sOI628"},"addedProducts":[{"id":"l2-managed-poe-gigabit-switch-with-sfp","count":1},{"id":"unifi-application-server","count":1},{"id":"unifi-switch-8-60w-b","count":1},{"id":"unifi-switch-8-150w","count":1},{"id":"unifi-security-gateway-xg","count":1},{"id":"unifi-switch-16-150w","count":1},{"id":"unifi-switch-16-xg-beta","count":1},{"id":"unifi-switch-48","count":1},{"id":"unifiswitch-48-750w","count":1},{"id":"unifi-ac-hd","count":1},{"id":"unifi-switch-8-beta","count":1}],"solved":"","numbers":"","description":"","mainImage":"192426i2ED0A8D0118FD447"}

Comments
by
on ‎02-06-2019 10:43 AM

I was relooking at the post and noticed that your fiber connection isn't symmetrical. I thought that all fiber was symmetrical. Are you using there modem?

by
on ‎02-06-2019 11:24 AM

@demonmaestro I think a lot of is, but not Cincinnati Bell.  There isn't really a "modem", it's an ONT (Optical Network Interface) that converts from GPON to Ethernet, and yes, if you look at the Wiki, that's the Alcatel-Lucent box, and its theirs.  They mount them outside now, so they can service or replace them without getting access to the house, or having someone bend the optic, and just punch the Ethernet and a small power wire through the exterior.  I have that on my outside wall, along with the cable modem, the surge isolator, and a small UPS.  I think I didn't post pictures of that, but I will. 

 

But unfortunately 1G/250M is the top tier offering.  Once the WAN2 fail-over works fully/properly I'll upgrade my cable to the top 300M/60M offering there as well, right now I just have 100M/5M with them.  There's talk of a 1G cable offering too, but I don't think I'm eligble either, and they don't do the highspeed upstream.  I think 60M is their best.

 

https://www.cincinnatibell.com/shop-cincinnati-bell/internet/plans-pricing

 

by
on ‎02-06-2019 11:31 AM

Yeah I am familure with the ONT.  I was thinking with your type of setup that you might have fiber to fiber going straight into the USG-XG-8.

Coax uplinks are never fast when talking on a gigabit level.

by
on ‎02-06-2019 11:33 AM

@demonmaestro Yea, I'd like to bring the fiber to my XG-8, and they essentially admitted it was techincally possible, but they don't currently support it.  I meant to mention, the want you to take their Zyxel router/wifi access unit too, which they call "the modem", but I don't use it, I run the ONT output to the surge-surppressor (which is unlikely on the fiber, but can happen on the coax), direct to the USG...

by
on ‎02-06-2019 08:08 PM

Brilliant, I recently discussed having a 10G backbone with the wife for our next house, which, admittedly may be a few years away. Learnt some sweet ideas from your setup, especially the running of OM4 and dual 10G NICs on desktops! Nice work.

 

Some Qs, what were your options for 10G NICs and what made you choose Sonnet?

Did you run into Linux vs Win driver issues that affected your choice of NICs?

Why not Intel - copper or fiber 10G NICs?

Did you test max file transfer speeds with a decent NVMe SSDs in RAID0 at both ends?

by
‎02-07-2019 05:15 AM - edited ‎02-07-2019 05:15 AM

@AviHD On the OM4, I really like the armored type in the wall, much more durable.  I got pre-terminated lengths that were pretty close to my needs so I wouldn't have to mess with the (very expensive) terminating tools.  I looked at the "composite" cable first, with a couple of Ethernet, a couple of fiber, maybe coax too, but you always end up with the wrong number of cables, and I couldn't find any with 6a Ethernet and OM4.

 

I use Macs for my desktops (which have pretty vanilla SSDs), so they don't have regular PCI slots, pushing me to look at NICs that were Thunderbolt comatible.  I'm sure some of them just have Intel PCI cards inside, but it's not always clear what you're really getting.  There may be others, but the only ones that I really found that fit the bill were the Akito Thudner 2 and the Sonnet Twin10G (links below).  Honestly, I tried the Akitio first, because it didn't require a power connection and was smaller/cleaner to install (and cheaper), even though it only has one 10G port.  But, I couldn't get the speed through it.  I filed bug reports and actually got some responses from them, but nothing was able to get me even moderately acceptable speeds.  So, I returned those and got the Sonnets, and have been pretty happy even though they're big and have a power brick.

 

I don't run Windows, except my work laptop which I don't do support for, so I don't really have thoughts there.  I got my Dell Rx20's from Orange Computer, and I have not had any driver issues with those.  I'm running Proxmox, which is basically Debian as I understand it, and FreeNAS and aside from getting one dead card (no lights) on my first order haven't had any issues, and they're all generic Broadcom cards.  The QNAP has SFP+, but honestly that machine has been a disappointment from a performance standpoint from day one.  I used to run Plex and Nextcloud on it, but it just wasn't up to the task (it as a Core i7 and 32GB of ram), and even the raw drive throughput is less than I'd hope for.  Long term I'll probably continue to build out FreeNAS storage on 720XD units and retire that QNAP.

 

And no, all of my tests to date have just been generic iperf3 to/from my Macs, just to make sure I was getting "much better than 1G" speeds, I haven't spent any time tweaking that at all, trying jumbo frame or anything.  I haven't even tried iperf to me VMs yet.  The data center is part of a bigger renovation of the new house, so there's a lot going on and that just hasn't bubbled to the top yet.

 

https://www.amazon.com/Thunder2-10G-network-adapter-T2NA-TLITS-AKT/dp/B014VAMBAQ/ref=sr_1_4?ie=UTF8&...

 

https://www.sonnettech.com/support/kb/kb.php?cat=490#position_content

by
on ‎02-07-2019 05:26 AM

@AviHD Maybe I can poke in this alittle. I use the 10GTek and haven't had any driver issues yet from both a Linux and Windows side. Granted they are using an Intel chipset.

by
‎02-08-2019 06:54 AM - edited ‎02-08-2019 07:00 AM

@bferrell 

 

Have you thought about replacing the QNAP with a Synology NAS?

by
on ‎02-08-2019 07:13 AM

@Aldiallo-Ubt Well, no, not really, if I replace the QNAP I'll just go with a 'real' server at this point. 

 

I'd looked at QNAP and Synology because they provided a "friendlier" management interface on sort of commodity hardware, with good storage sizes.  I'd considered both QNAP and Synology at the time, and they both had pluses and minuses, and maybe Synology would've been a better choice, but the Dell rack hardware I'm now familiar with is so much more powerful, and now that I have experience with FreeNAS and Proxmox, I would personally go that route.  Maybe Synology would've been better, but none of the reviews I saw said there were any significant performance differences between the two, it was mostly the interface, openess, flexibililty, etc.

by
‎02-08-2019 07:21 AM - edited ‎02-08-2019 07:22 AM

@bferrell also check out UnRaid by lime-technology for your NAS. You still have the option for VMs and you can mix and match your HDDs for storage. It has the option for NFS file storage if that's your thing.

 

*Edit spelling*