Emerging Member
Posts: 54
Registered: ‎03-10-2017
Kudos: 8

Load Balanced LAN Routing

[ Edited ]

I'm testing load balancing for the first time with UBNT and I'm curious what the best approach would be for this particular scenario:

 

  • Customer Site A has an AF24HD link back to our Hub Site to provide internet access.
  • An AF5 will be added for redudancy and increased capacity.
  • The two router interfaces on the Site A side will be placed in a load balancing configuration and routed back to the same router on the Hub Site.
  • Site A has a single customer subnet, so the Hub Site router will need to be able to route over both links to a single subnet.

 

I'm unclear as to what configuration I should implement on the Hub Site router in order to route traffic over both the AF24HD link and the AF5 link, both to increase the aggregate capacity to Site A and for failover in the event one link fails. I've tried putting the route in twice and pointed it to the two different Site A interfaces, but when I kill one of the links I get packet loss - I assume because the Hub Site router is attempting to send packets evenly over the two routes.

 

The two methods I can think of, but have no experience with, would be OSPF and source based routing Any suggestions would be most appreciated.

 

 

EDIT: Added bullet point about single subnet at Site A.

SuperUser
Posts: 20,402
Registered: ‎09-17-2013
Kudos: 5144
Solutions: 1458

Re: Load Balanced LAN Routing

Shouldn't need to do anything "special" at the upstream router -- just have the two interfaces in different subnets.

Emerging Member
Posts: 54
Registered: ‎03-10-2017
Kudos: 8

Re: Load Balanced LAN Routing

Site A has a single subscriber subnet behind it, so on the upstream router I'd need a way to split the traffic between the two radio links.
SuperUser
Posts: 20,402
Registered: ‎09-17-2013
Kudos: 5144
Solutions: 1458

Re: Load Balanced LAN Routing

You end up with

 

SiteA_Range1 (AF link) --> radio --> hub router portA (or VLAN)
SiteA_Range2 (backup link) --> radio --> hub router portB (or VLAN)

SiteA_LAN

 

If they're paying for one(1) public IP or something, then there's no harm in the backup link being RFC6598 (CGNAT)...

Emerging Member
Posts: 54
Registered: ‎03-10-2017
Kudos: 8

Re: Load Balanced LAN Routing

It's an entire duplex community served by a PtMP setup; we'd really need the aggregation and link failover.
SuperUser
Posts: 20,402
Registered: ‎09-17-2013
Kudos: 5144
Solutions: 1458

Re: Load Balanced LAN Routing

[ Edited ]

... so your tower has a router and two radios, right?

 

You take those two radios, put them on different ports in the tower router, put those ports on different subnets, and shoot the signal to the customer site.

 

At the customer site, you've got two more radios, and another router.

 

You take "incoming connection A" and pipe it into one port on the site router
You take "incoming connection B" and pipe it into a second port on the site router.

 

You take a third port, and designate it for their LAN.

 

 

Incoming connection A and B on the site router are thrown into a loadbalance group, and you tell it to balance 50/50 or 60/40 or whatever represents the two links.   You don't do anything "special" to the upstream router ...

Emerging Member
Posts: 54
Registered: ‎03-10-2017
Kudos: 8

Re: Load Balanced LAN Routing

Correct! I've set all of this up in a lab environment and tested killing one link, however when one link is down there is packet loss. This indicates to me that the upstream router is sending data down both paths even though one of the downstream radios has failed.
SuperUser
Posts: 20,402
Registered: ‎09-17-2013
Kudos: 5144
Solutions: 1458

Re: Load Balanced LAN Routing

If upstream is sending to "portA" of the downstream router, and you disconnect the link mid stream, you will lose packets.  There is no way around that ...

Veteran Member
Posts: 8,074
Registered: ‎03-24-2016
Kudos: 2124
Solutions: 929

Re: Load Balanced LAN Routing

But if you detect a broken midstraem link, you can stop using that link.

 

The ER loadbalance can do ping test for remote peer.  Alternatively, you can use OSPF, so you'll also detect a broken link.

 

Note: others have reported problems when using OSPF for loadsharing:  Even packets belonging to a single tcp session are balanced, so prepare for packets arriving out-of-order.

SuperUser
Posts: 20,402
Registered: ‎09-17-2013
Kudos: 5144
Solutions: 1458

Re: Load Balanced LAN Routing


@16again wrote:

But if you detect a broken midstraem link, you can stop using that link.

 

The ER loadbalance can do ping test for remote peer.  Alternatively, you can use OSPF, so you'll also detect a broken link.

 


Which is why I've been telling him to set the "siteA" router up with loadbalance / failover.

 

It seems the problem is that even with it, he doesn't like that the upstream router will still try sending data across a purposely killed connection when he unplugs the cable mid-transfer.  I've yet to see networking gear that can seamlessly handle that scenario ... 

Emerging Member
Posts: 54
Registered: ‎03-10-2017
Kudos: 8

Re: Load Balanced LAN Routing

[ Edited ]

@dpurgertI understand what you're saying - the disconnection of the radio is meerely meant to simulate a radio failure. What I'm seeing in the lab is a far different behavior than lost packets at the moment of the disconnect - with two static routes set on the upstream router to point back to a single subnet on the downstream router via two different hops, and one of the two links is disabled, I see this:

 

PING 10.2.0.1 (10.2.0.1): 56 data bytes
64 bytes from 10.2.0.1: icmp_seq=0 ttl=63 time=1.327 ms
Request timeout for icmp_seq 1
64 bytes from 10.2.0.1: icmp_seq=2 ttl=63 time=1.378 ms
Request timeout for icmp_seq 3
64 bytes from 10.2.0.1: icmp_seq=4 ttl=63 time=1.430 ms
64 bytes from 10.2.0.1: icmp_seq=5 ttl=63 time=1.320 ms
Request timeout for icmp_seq 6
Request timeout for icmp_seq 7
Request timeout for icmp_seq 8
Request timeout for icmp_seq 9
64 bytes from 10.2.0.1: icmp_seq=10 ttl=63 time=1.537 ms
64 bytes from 10.2.0.1: icmp_seq=11 ttl=63 time=1.471 ms
Request timeout for icmp_seq 12
64 bytes from 10.2.0.1: icmp_seq=13 ttl=63 time=1.454 ms
64 bytes from 10.2.0.1: icmp_seq=14 ttl=63 time=1.258 ms
64 bytes from 10.2.0.1: icmp_seq=15 ttl=63 time=1.233 ms
64 bytes from 10.2.0.1: icmp_seq=16 ttl=63 time=1.447 ms
Request timeout for icmp_seq 17
Request timeout for icmp_seq 18
64 bytes from 10.2.0.1: icmp_seq=19 ttl=63 time=1.355 ms
^C
--- 10.2.0.1 ping statistics ---
20 packets transmitted, 11 packets received, 45.0% packet loss
round-trip min/avg/max/stddev = 1.233/1.383/1.537/0.090 ms

 So simply setting two routes on the upstream router will not suffice.

 

I had hoped that Bonding would be the solution here, but I see now that it is done in software on ERs so that's a nonstarter. I'm stuck with either finding a routing solution to the load balancing or inserting a switch and doing bonding on Layer 2.

SuperUser
Posts: 20,402
Registered: ‎09-17-2013
Kudos: 5144
Solutions: 1458

Re: Load Balanced LAN Routing

I never said to set routes on the upstream router.

 

I said to set two subnets on two interfaces (or two VLANs), and to set the "site" router to loadbalance against those two incoming connections. Your network should have no concept of their network -- the siteA router should be performing NAT back to your network.

 

Now, if both sites are "your network" (and not "your network at tower, customer's network at site" -- which it sounds like), then a routing protocol can be used, as the tower router needs to know information about the site's networks ...

 

 

No matter what though, you're going to get some packet loss from tower -> site if a link goes down mid transfer.  You're going to continue to have some loss until either

  • Requests start coming from "site router, IP_B", and other requests finally time out.
  • your routing protocol re-converges and tells the tower router that it's got to use "route B" to get to the site.
Emerging Member
Posts: 54
Registered: ‎03-10-2017
Kudos: 8

Re: Load Balanced LAN Routing

Ah yes, I see where our disconnect is now.

 

The entire network is mine. Everything is managed down to the user level, so it's on me to route the traffic from end to end.

SuperUser
Posts: 20,402
Registered: ‎09-17-2013
Kudos: 5144
Solutions: 1458

Re: Load Balanced LAN Routing

In that case, OSPF or another routing protocol is the way you go ... not using the loadbalancer.

 

Kind of weird that you (as the ISP) would be managing the customer's network ... but meh.

Emerging Member
Posts: 54
Registered: ‎03-10-2017
Kudos: 8

Re: Load Balanced LAN Routing

We provide WiFi and a wired drop, with DHCP provided by a router that we manage.

I'm hoping I can find another way. Layer 2 may be it though.
Highlighted
SuperUser
Posts: 20,402
Registered: ‎09-17-2013
Kudos: 5144
Solutions: 1458

Re: Load Balanced LAN Routing

[ Edited ]

Barring you meaning "customer siteA" is "one of your customer access towers" ...

 

 

The router there does NAT between the customer's RFC1918 space, and your public IP space (or RFC6598 space if you're using CGNAT).  Your network only cares about the two IP addresses you've assigned "your side" of their router (i.e. WAN_1 and WAN_2).  You don't route from your edge to mary sue in accounting...

 

Or at least this is how it's normally done -- "your responsibiilties" (at least "as the ISP") end at the customer's gateway. The "LAN" side is theirs (or "you as their managed service provider" -- but in either event, your 'ISP network' has no real concept of what's behind the router).

 

Or perhaps I'm way off base on what you mean and should probably just go get more coffee.