2 weeks ago
DC powered server cluster
Used Products
×1
×1
Location
Denver, CO, USA
Description

We currently run our management tools (AC2 and many other things) on a Ganeti Cluster of large HP servers - DL360 G7 - in our main Data Center.   If you aren't familiar with Ganeti, it's the software that Google developed to allow them to run servers in multiple locations across the globe and have them automatically move running processes/VMs between servers and locations as needed with disk replication - it's basically what runs Google worldwide aln many things for them internally.   It's Open Source and therefore free, so we decided to use it for all our Linux Server centerred tasks several years ago.   However, the HP servers are AC powered only, unless you want to purchase their DC power supplies, which even on the secondary market cost more than the servers do.   We chose not to do that ;-)

 

So for some time I have been looking for a DC powered alrternative we could use.   I tried small SBC solutions several years ago and built a group of Odroid machines that we still use for out authoratative DNS, remote syslog and several other things.   But they are limited in whast they can do both CPU wise and as far as RAM/Disk are concerned (all flash so lifespan issues there too...)

 

Then about 6 months ago I stumbled across the Lenovo ThinkCenter M92P Tiny - an Intel I5 based micro PC designed to be a desktop machine, but it's DC powered.   No it doesn't do RAID or have multiple disks, but with a Cluster you don't care - if a disk or machine fails everything automatically moves to another machine.   And the best part is they are DC powered - designed to use a regular 19V laptop power brick.

 

So I've designed a simple rack mount setup to both house up to 10 of them in a singe unit 5 RU high, but also to provide the regulated power they need in a redundant manner which lets us use the 54VDC power already available at the Data Center, and also at all our large sites if we need that as well.

DCcluster.jpg

 

 

This is what it looks like from the rear - 4 of the M92Ps are in the upper left. with the power regulators in the center foreground, the 54V Powr Distribution unit on the right and a diode combiner on the left - more about that later...DCclusterFront.jpg

 

 

This is what they look like from the front.   A single rack shelf is big enough to hold 10 of the PCs, which each have 12GB of RAM, a 1TB disk, and a QuadCore 2.9GHz CPU.   Plenty pumpy to run just about anything we need in a number of VMs.   Ane the fans in each unit pull in from the front and exhaust to the rear, so having them all stacked like this doesn't interfere with the cooling at all.DCclusterScreen.jpg

 

Here's the screen of one of them just running Ubuntu 18.04 LTS so you can see them actually running off of 54VDC in to theDC-DC regulators, which buck that down to 19.0 volts.   Each regulator is rated at 120 watts, but I have 5 of them each powerig 2 computers, so they never are run hard - each computer pulls about 12 watts at idle, and maxes at around 35 watts .   Plus this adds redundancy - if a regulator fails, only 2 computers drop out of the cluster.DCclusterPower.jpg

 

 The power to each regulator is delivered through a PDU with a 3A fuse for each.   We buy these direct from the manufacturer in China, but you can get them on ebay as well for $25 each.   They output 10 individual fused connections, and include an LED for each which light up if the fuse blows.DCclusterTop.jpg

 

This shows more detail of the power system.   All the units have screw terminals so it's easy to connect them or replace one in the advent of a failure.   We color code all the wiring too - makes it easier to keep things straight when we have 54V unregulated, 48V regulated, 24V and possibly -48V for the phone company's stuff...DCclusterQuintDiode.jpg

 

I mentioned the QuintDiode isolator earlier.   This is made by Phoenix Contact, a manufactureer of many types of Industrial plugs and connectors (hence the name) but they also make DIN rail mounted devices like power supplies.   These units (ignore the strange sounding name ;-) are diode isolators which allow you to combine two DC power sources (the A and B 54VDC power rails in our Data Center and in our large sites) into a single output for redundancy.   They actually use MOSFETs internally rather than diodes to give even less forward voltage drop than a Schottkey Diode would.   They are also rated at 40 amps (2 x 20) at 60 volts so they can handle a lot of power, and the entire unit is built on a large aluminum extrusion for a heatsink.   They are pricey but they just work, and I use the DIN rail mount in other large power systems - I'll be doing a story on some of that later this year.

 

I know this isn't directly EdgeMax related, but the first thing we are going to be doing with this new cluster once it's installed is hosting UNMS/UCRM for our entire ISP.   So I thought this might be of interest to others needin to do domething similar.

Jim 

 

 

 

DC powered server cluster

by 2 weeks ago - last edited 2 weeks ago

We currently run our management tools (AC2 and many other things) on a Ganeti Cluster of large HP servers - DL360 G7 - in our main Data Center.   If you aren't familiar with Ganeti, it's the software that Google developed to allow them to run servers in multiple locations across the globe and have them automatically move running processes/VMs between servers and locations as needed with disk replication - it's basically what runs Google worldwide aln many things for them internally.   It's Open Source and therefore free, so we decided to use it for all our Linux Server centerred tasks several years ago.   However, the HP servers are AC powered only, unless you want to purchase their DC power supplies, which even on the secondary market cost more than the servers do.   We chose not to do that ;-)

 

So for some time I have been looking for a DC powered alrternative we could use.   I tried small SBC solutions several years ago and built a group of Odroid machines that we still use for out authoratative DNS, remote syslog and several other things.   But they are limited in whast they can do both CPU wise and as far as RAM/Disk are concerned (all flash so lifespan issues there too...)

 

Then about 6 months ago I stumbled across the Lenovo ThinkCenter M92P Tiny - an Intel I5 based micro PC designed to be a desktop machine, but it's DC powered.   No it doesn't do RAID or have multiple disks, but with a Cluster you don't care - if a disk or machine fails everything automatically moves to another machine.   And the best part is they are DC powered - designed to use a regular 19V laptop power brick.

 

So I've designed a simple rack mount setup to both house up to 10 of them in a singe unit 5 RU high, but also to provide the regulated power they need in a redundant manner which lets us use the 54VDC power already available at the Data Center, and also at all our large sites if we need that as well.

DCcluster.jpg

 

 

This is what it looks like from the rear - 4 of the M92Ps are in the upper left. with the power regulators in the center foreground, the 54V Powr Distribution unit on the right and a diode combiner on the left - more about that later...DCclusterFront.jpg

 

 

This is what they look like from the front.   A single rack shelf is big enough to hold 10 of the PCs, which each have 12GB of RAM, a 1TB disk, and a QuadCore 2.9GHz CPU.   Plenty pumpy to run just about anything we need in a number of VMs.   Ane the fans in each unit pull in from the front and exhaust to the rear, so having them all stacked like this doesn't interfere with the cooling at all.DCclusterScreen.jpg

 

Here's the screen of one of them just running Ubuntu 18.04 LTS so you can see them actually running off of 54VDC in to theDC-DC regulators, which buck that down to 19.0 volts.   Each regulator is rated at 120 watts, but I have 5 of them each powerig 2 computers, so they never are run hard - each computer pulls about 12 watts at idle, and maxes at around 35 watts .   Plus this adds redundancy - if a regulator fails, only 2 computers drop out of the cluster.DCclusterPower.jpg

 

 The power to each regulator is delivered through a PDU with a 3A fuse for each.   We buy these direct from the manufacturer in China, but you can get them on ebay as well for $25 each.   They output 10 individual fused connections, and include an LED for each which light up if the fuse blows.DCclusterTop.jpg

 

This shows more detail of the power system.   All the units have screw terminals so it's easy to connect them or replace one in the advent of a failure.   We color code all the wiring too - makes it easier to keep things straight when we have 54V unregulated, 48V regulated, 24V and possibly -48V for the phone company's stuff...DCclusterQuintDiode.jpg

 

I mentioned the QuintDiode isolator earlier.   This is made by Phoenix Contact, a manufactureer of many types of Industrial plugs and connectors (hence the name) but they also make DIN rail mounted devices like power supplies.   These units (ignore the strange sounding name ;-) are diode isolators which allow you to combine two DC power sources (the A and B 54VDC power rails in our Data Center and in our large sites) into a single output for redundancy.   They actually use MOSFETs internally rather than diodes to give even less forward voltage drop than a Schottkey Diode would.   They are also rated at 40 amps (2 x 20) at 60 volts so they can handle a lot of power, and the entire unit is built on a large aluminum extrusion for a heatsink.   They are pricey but they just work, and I use the DIN rail mount in other large power systems - I'll be doing a story on some of that later this year.

 

I know this isn't directly EdgeMax related, but the first thing we are going to be doing with this new cluster once it's installed is hosting UNMS/UCRM for our entire ISP.   So I thought this might be of interest to others needin to do domething similar.

Jim 

 

 

 

{"location":{"title":"Denver, CO, USA","placeId":"ChIJzxcfI6qAa4cR1jaKJ_j0jhE"},"addedProducts":[{"id":"unms","count":1},{"id":"ucrm","count":1}],"solved":"","numbers":"","description":"","mainImage":"188603i00C5C767425EAC41"}

" How can anyone trust Scientists? If new evidence comes along, they change their minds! " Politician's joke (sort of...)
"Humans are allergic to change..They love to say, ‘We’ve always done it this way.’ I try to fight that. "Admiral Grace Hopper, USN, Computer Scientist
":It's not Rocket Science! - Oh wait, Actually it is... "NASA bumper sticker
":The biggest problem in tech I see right now is that most users don't want to do things that are hard. That doesn't bode well for the industry or the society.": (me. actually ;-)
Comments
by
2 weeks ago

Thanx @eejimm!

PS: Loves your tidy desk ;-)

by
2 weeks ago - last edited 2 weeks ago

jjonsson

Thanx @eejimm!

PS: Loves your tidy desk ;-)

 

Guess I fit more with the Artist type...   ;-)

ArtistVsProgrammer.jpg

by
2 weeks ago

Amazing work Jim!  If you ever find yourself in the Omaha area, ping me.  We will grab lunch and show you our stuff.

by
2 weeks ago

Very clean setup. We're doing something similar with NUCs.

 

Any chance you could send a link to those DC-DC converters? I've been looking for one for a while that does 19V.

by
2 weeks ago

Wish i could hang out with ya for a week, i would learn so much.

 

We use thoes in our sites but we got the monitor that they dock in to so every thing is in one place.

by
2 weeks ago

@danf8121

Very clean setup. We're doing something similar with NUCs.

 

Any chance you could send a link to those DC-DC converters? I've been looking for one for a while that does 19V

 

Here's an ebay link

https://www.ebay.com/itm/DC-DC-Step-down-Converter-Buck-7-60V-to-12V-1-35V-MAX-15A-150W-Voltage-Regu...

Jim

by
2 weeks ago

I don´t understand. Why is it important for your working with DC Power?

by
2 weeks ago

@sgmelin

I don´t understand. Why is it important for your working with DC Power?

 

Two basic reasons - backup power during mains outages and efficiency.

 

Using DC power for everything means that you automatically get UPS capability by adding battereis to the system.   And you can add as many as you want to increase the run-time during a power outage to anything you can afford (money and space-wise).   As an ISP this is critical - our customers demand that our service be working all the time, so we need to meet their expectations.   We even use Solar Panels to run some sites where there's no AC power available - hard to do that with a standard AC UPS to run your Dell Servers where there's no AC power to start with ;-)

 

And using DC power means less energy waste - there's a ton of power lost in converting AC to DC to AC with a normal UPS system.   Since all electronics run on DC power internally, why not cut out the middleman?

Jim

by
2 weeks ago

Yup - lots of waste in traditional AC to DC to AC back to DC in many UPS/Computer combos.  Nice stuff!   It would be nice if the UPS vendors and server vendors would pick up on this more and just offer more native DC solutions designed to work together as currently as the AC mains voltage stuff does.  Even at my larger site where I only have a handful of equipment, I could still probably get some pretty decent effeciency and savings over time - as well as ot having to pay for a bunch of power supplies I don't otherwise need.  

by
2 weeks ago

One other advantage is you don't get power surges or sags that can damage your equipment - battery stacks are amazingly good at sucking up any of that - think of giant supercapacitors here - so the damage due to that kind of thing has dropped to nearly zero in the years since we switched to DC power.

Jim