Reply
Emerging Member
Posts: 44
Registered: ‎02-02-2015
Kudos: 6
Accepted Solution

multi-WAN for beginner

[ Edited ]

Hi all

 

I'm waiting for delivery of an ER and installation of two more cable connections from my local ISP with the intention of setting up 3 WAN connections and sharing them in a hotel environment.  My aim is to aggregate the connections to provide a seamless "triple-wide" internet connection to my guests. While I wait I thought I would spend some time trying to wrap my head around how I'm going to set this up but I'm finding it a little daunting.

 

I understand there's a wizard now for setting up Dual WAN but not triple, is that correct?  If I use the wizard, will it show me what it's changing, so to speak, so that I can muddle through adding a third myself, or will I have to start from scratch?  Does anyone have any recomendations on some beginner-level reading so I can get my feet wet over the weekend before delivery next week?

 

EDIT: here's my final working configuration.   https://community.ubnt.com/t5/EdgeMAX/multi-WAN-for-beginner/m-p/1183246#M57425


Accepted Solutions
Previous Employee
Posts: 10,504
Registered: ‎06-09-2011
Kudos: 3113
Solutions: 945
Contributions: 16

Re: multi-WAN for beginner

Well the address on eth1 is 192.168.0.1/24 so if for example the gateway was 192.168.0.254 then you would do:

configure
set protocols static route 0.0.0.0/0 next-hop 192.168.0.254
commit
save
exit

 Do you know what the gateway for eth1 is?

EdgeMAX Router Software Development

View solution in original post

Previous Employee
Posts: 10,504
Registered: ‎06-09-2011
Kudos: 3113
Solutions: 945
Contributions: 16

Re: multi-WAN for beginner

eth0 and eth2 are in the same subnet - that causes routing problems.  Each routed interface need to be on a different subnet.

EdgeMAX Router Software Development

View solution in original post


All Replies
Previous Employee
Posts: 10,504
Registered: ‎06-09-2011
Kudos: 3113
Solutions: 945
Contributions: 16

Re: multi-WAN for beginner

[ Edited ]

This post may answer some of your questions - LINK.

 

Some best practices for multiple WANs:

1) If possible use different ISP.  Using the same ISP may mean that when 1 link goes down, so do the others.

 

2) Static IP address/gateway are easier to work with than dynamic address/gateways that may change (dhcp, pppoe).  If dynamic is necessary, pppoe is easier to deal with since it is a point-to-point link where the gateway doesn't change (it's always the other side of the link).

 

3) Make sure each WAN address is in a different subnet with a different gateway.  If all the ISP are different then this usually isn't an issue.

 

4) Make sure every name-server in /etc/resolv.conf can be used by all WAN interfaces.  If ISP-A gives you a name-server that only works on WAN-A, then dns will fail if WAN-A goes down.  For dhcp and pppoe it's often best to not update the name-server and instead use a global name-server.

  • For dhcp:
configure
set interfaces ethernet ethX dhcp-options name-server no-update
set system name-server 8.8.8.8
commit
save
exit

- Then ask for a new address:

release dhcp interface ethX
renew dhcp interface ethx
  •  For pppoe
configure
set interfaces ethernet eth0 pppoe 0 name-server none
commit
save
exit

- Then disconnect/re-connect the pppoe session

disconnect interface pppoeX
connect interface pppoeX

 

EdgeMAX Router Software Development
Emerging Member
Posts: 44
Registered: ‎02-02-2015
Kudos: 6

Re: multi-WAN for beginner

Thanks

 

1 - can't, unfortunately.  There are no other options.

 

2 - static IPs aren't possible but the WAN IPs provided rarely change, if ever.  Not sure if pppoe is an option (or what exactly it is, actually!) but I don't think so.

 

3 - how do i do this?  Is it likely possible if the ISPs are the same?  Do I need the ISP to configure this part (as I understand it, the modem gets all this via DHCP).

 

4 - same ISP so not likely a problem?  I'm learning just enough to see what you're doing in configuring the ER via the CLI there; is this the "obvious step" i'm missing (see next post)?

Emerging Member
Posts: 44
Registered: ‎02-02-2015
Kudos: 6

Re: multi-WAN for beginner

Well, I think I've officially bitten off more than I can chew.  I would love some help please!  Some background - I am very much a beginner at this; I know nothing more than how little I know...  

Here's my layout.  I have three cisco 3825 cable modem/routers, all provided by my cable ISP with 100mbps service.  The original is in "router mode", and is currently providing internet to my hotel guests through a handful of UAPs.  The router is in use on this modem and wifi is disabled.  DHCP is turned on with a range of 192.168.1.11 to 192.168.1.150.  The router is on 192.168.1.1/24. Ideally I'll leave this in place until everything else is set up and working to minimize disruption. As such I won't refer to this modem again, for now.

I have a second 3825 in essentially the same configuration.  "Router" mode, DHCP and wifi OFF in this case, IP of 192.168.0.1 (default).  I'm advised by the ISP that these two modems must remain in "router" mode.

My third 3825 is in "bridge" mode and is (as I understand it) purely a modem.  All routing functionality is turned off.  It does not have a private IP as a result.

When I connect my laptop to either modem, I get an IP (either public on the bridged one or private with a static IP config in my laptop) and can browse the internet.

I have added the ER-8Pro with the aim of improving the internet service to my guests.  I frequently have 100+ clients with demanding expectations connecting simultaneously and want to give them each 2+ mbps of service, and have limited each user accordingly in the Unifi controller.  I do not need to give any single user excessive service (there are a handful of users in an Admin group with unlimited service but they are an aside).

I have successfully managed to log into the ER and update the firmware to 1.6.  I then ran the wizard to Load Balance the two new modems, on Eth0 and Eth1, with my laptop plugged into Eth2.  The wizard completes without complaint, however, I cannot connect past the ER.

I suspect I've missed something obvious and fundamental to you, but I'm at a loss to figure out what that may be, haha.  For what its worth, I tried the basic setup wizard too, with only 1 modem connected, and still wasn't able to connect past the router.

Emerging Member
Posts: 44
Registered: ‎02-02-2015
Kudos: 6

Re: multi-WAN for beginner

Okay, babysteps.

 

I rebooted the modems after running the wizard again and I'm now online.  Not sure how to check that both connections are being used, though.  Speedtest leads me to believe only the one is but I'm sure there's a better way.

Emerging Member
Posts: 44
Registered: ‎02-02-2015
Kudos: 6

Re: multi-WAN for beginner

I'm pretty sure that only the "bridge mode" modem is working properly.  The "router mode" says it's connected in the dashboard but there's nothing about it in the routing tables and if I take the other connection down, I lose internet access.

 

More reading and experimenting to do Man Happy  Thank god for testbeds.

Previous Employee
Posts: 10,504
Registered: ‎06-09-2011
Kudos: 3113
Solutions: 945
Contributions: 16

Re: multi-WAN for beginner

Woundn't hurt to post your config file.

EdgeMAX Router Software Development
Emerging Member
Posts: 44
Registered: ‎02-02-2015
Kudos: 6

Re: multi-WAN for beginner

Previous Employee
Posts: 10,504
Registered: ‎06-09-2011
Kudos: 3113
Solutions: 945
Contributions: 16

Re: multi-WAN for beginner

What's the output of "show load-balance status" and "show load-balance watchdog".

EdgeMAX Router Software Development
Previous Employee
Posts: 10,504
Registered: ‎06-09-2011
Kudos: 3113
Solutions: 945
Contributions: 16

Re: multi-WAN for beginner

Unrelated, but if you plan do add any debian packages then change your repos to wheezy.  If not just delete the package repos.

EdgeMAX Router Software Development
Established Member
Posts: 1,208
Registered: ‎10-18-2014
Kudos: 338
Solutions: 33

Re: multi-WAN for beginner


jward wrote:

Thanks

 

2 - static IPs aren't possible but the WAN IPs provided rarely change, if ever.  Not sure if pppoe is an option (or what exactly it is, actually!) but I don't think so.

 


If it is a modem/router. Yuo may be able to set up the modem as a router. Assign a static IP to the edgerouter, and tell the modem to DMZ to that static IP. This will normally work fine. 

Emerging Member
Posts: 44
Registered: ‎02-02-2015
Kudos: 6

Re: multi-WAN for beginner

show load-balance status
Group G
interface : eth0
carrier : up
status : active
gateway : 24.66.208.1
weight : 100
flows
WAN Out : 1447
WAN In : 4
Local Out : 6181
 
interface : eth1
carrier : up
status : inactive
gateway : unknown
weight : 0
flows
WAN Out : 818
WAN In : 0
Local Out : 1459
 
 
 
show load-balance watchdog
Group G
eth0
status: Running
pings: 29
fails: 0
run fails: 0/3
route drops: 5
ping gateway: ping.ubnt.com - REACHABLE
last route drop : Fri Feb 13 03:59:17 2015
last route recover: Fri Feb 13 03:59:58 2015
 
eth1
status: Waiting on recovery (0/3)
pings: 16
fails: 4
run fails: 3/3
route drops: 4
ping gateway: ping.ubnt.com - DOWN
last route drop : Fri Feb 13 00:39:26 2015
last route recover: Fri Feb 13 00:33:38 2015
Previous Employee
Posts: 10,504
Registered: ‎06-09-2011
Kudos: 3113
Solutions: 945
Contributions: 16

Re: multi-WAN for beginner

Looks like the problem is that you haven't configured a default gateway for eth1.  For eth0 it get isn't gateway from dhcp.

EdgeMAX Router Software Development
Emerging Member
Posts: 44
Registered: ‎02-02-2015
Kudos: 6

Re: multi-WAN for beginner

Yeah I saw that too. Haven't figured out how to configure that part of the interface yet. Doesn't seem to be in the cli primer that's stickied?
Previous Employee
Posts: 10,504
Registered: ‎06-09-2011
Kudos: 3113
Solutions: 945
Contributions: 16

Re: multi-WAN for beginner

Well the address on eth1 is 192.168.0.1/24 so if for example the gateway was 192.168.0.254 then you would do:

configure
set protocols static route 0.0.0.0/0 next-hop 192.168.0.254
commit
save
exit

 Do you know what the gateway for eth1 is?

EdgeMAX Router Software Development
Emerging Member
Posts: 44
Registered: ‎02-02-2015
Kudos: 6

Re: multi-WAN for beginner

Pretty sure it's the ip of the router? 192.168.0.3 iirc, not in front of it anymore. I think there's a wan ip listed in the webui of the router too, but you don't mean that right?
Emerging Member
Posts: 44
Registered: ‎02-02-2015
Kudos: 6

Re: multi-WAN for beginner

It's working Man Happy

dualwan speedtest.PNG

 

Now I just need to figure out how to add a third.

 

I think I have the idea down on how to set up the interfaces.  Now I just need to determine how to do a three way load balance.  I've watched the video in the KB for dual wan; it seems like i should modify the balance firewall rules (I thought these were set up by the wizard for me?) to use three routing tables, and to mark packets 1/2/3 - 1 @ 33%, then if not already marked, mark 2 @ 50%?  However, when I view the firewall rules now, I don't see the same types of rules with the packet marking, so I assume the wizard does this somehow differently?

 

Also, before i move onto setting up the third ISP/WAN interface, is there anything more I need to do?

Emerging Member
Posts: 44
Registered: ‎02-02-2015
Kudos: 6

Re: multi-WAN for beginner

Based on the info here it seems like there's two ways to do multi-WAN; one is somewhat automatic (ECMP) and another less so but offers failover (policy based).  As all my WAN connections are from the same ISP and failover is not too likely to help, should I just be using this ECMP method?

 

My show load-balance status has both WAN interfaces at 50 weight, and I didn't explicitly set that up, so does the wizard do that or is this the ECMP method (or both)?  If I configure another WAN interface, will they all just go to 33 weight?

I will experiment later today.

Emerging Member
Posts: 44
Registered: ‎02-02-2015
Kudos: 6

Re: multi-WAN for beginner

[ Edited ]

OK, so I've been reading and trying to understand how this all works.  I've modified a dual-wan config file I found to the code below to try and use 3-wan; does this look right?

modified version posted below

 

Emerging Member
Posts: 44
Registered: ‎02-02-2015
Kudos: 6

Re: multi-WAN for beginner

OK that last version had some small things I tried changing, so here's the version I tried.  It doesn't work.  I also tried modifing the existing dual wan config from the router to reflect a third WAN interface but it didn't work either; the load-balance group wouldn't grow to 3.

 

How can I do a triple-WAN setup?

 

firewall {
    modify ISP1_IN {
        rule 1 {
            action modify
            description "use mark 1 for new ISP1 connections"
            log enable
            modify {
                connmark {
                    set-mark 1
                }
            }
            protocol tcp_udp
            state {
                new enable
            }
        }
    }
    modify ISP2_IN {
        rule 1 {
            action modify
            description "use mark 2 for new ISP2 connections"
            log enable
            modify {
                connmark {
                    set-mark 2
                }
            }
            protocol tcp_udp
            state {
                new enable
            }
        }
    }
    modify ISP3_IN {
        rule 1 {
            action modify
            description "use mark 3 for new ISP3 connections"
            log enable
            modify {
                connmark {
                    set-mark 3
                }
            }
            protocol tcp_udp
            state {
                new enable
            }
        }
    }
    modify balance {
        enable-default-log
        rule 10 {
            action modify
            description "restore mark from connection"
            modify {
                connmark {
                    restore-mark
                }
            }
        }
        rule 20 {
            action accept
            description "accept the packet if the mark isn't zero"
            mark !0
        }
        rule 30 {
            action modify
            description "for new connections mark 33% with mark 1"
            modify {
                mark 1
            }
            protocol tcp_udp
            state {
                new enable
            }
            statistic {
                probability 33%
            }
        }
        rule 40 {
            action modify
            description "for packets still with mark zero, mark 50% with mark 2"
            mark 0
            modify {
                mark 2
            }
            protocol tcp_udp
            state {
                new enable
            }
			            statistic {
                probability 50%
            }
        }
		        rule 50 {
            action modify
            description "for packets still with mark zero, mark with 3"
            mark 0
            modify {
                mark 3
            }
            protocol tcp_udp
            state {
                new enable
            }
        }
        rule 60 {
            action modify
            description "save the packet mark to the connection mark"
            modify {
                connmark {
                    save-mark
                }
            }
        }
    }
 }
 interfaces {
    ethernet eth0 {
        address 192.168.0.1/24
        duplex auto
        firewall {
            in {
                modify ISP1_IN
            }
        }
        speed auto
    }
    ethernet eth1 {
        address 24.66.208.150/22
        duplex auto
        firewall {
            in {
                modify ISP2_IN
            }
        }
        speed auto
    }    
	ethernet eth2 {
        address 192.168.0.2/24
        duplex auto
        firewall {
            in {
                modify ISP3_IN
            }
        }
        speed auto
    }
    ethernet eth3 {
        address 192.168.1.1/24
        duplex auto
        firewall {
            in {
                modify balance
            }
        }
        speed auto
    }
    loopback lo {
    }
 }
 protocols {
    static {
        route 0.0.0.0/0 {
            next-hop 192.168.0.1 {
            }
            next-hop 24.66.204.1 {
            }
            next-hop 192.168.0.2 {
            }
        }
        table 1 {
            mark 1
            route 0.0.0.0/0 {
                next-hop 192.168.0.1 {
                }
            }
        }
        table 2 {
            mark 2
            route 0.0.0.0/0 {
                next-hop 24.66.204.1 {
                }
            }
        }
        table 3 {
            mark 3
            route 0.0.0.0/0 {
                next-hop 192.168.0.2 {
                }
            }
        }
    }
 }
 service {
    dhcp-server {
        disabled false
        shared-network-name LAN {
            authoritative disable
            subnet 192.168.0.0/24 {
                default-router 192.168.1.1
                dns-server 8.8.8.8
                lease 86400
                start 192.168.1.11 {
                    stop 192.168.1.255
                }
            }
        }
    }
    gui {
        https-port 443
    }
    nat {
        rule 5000 {
            outbound-interface eth0
            type masquerade
        }
        rule 6000 {
            outbound-interface eth1
            type masquerade
        }
        rule 7000 {
            outbound-interface eth2
            type masquerade
        }		
    }
    ssh {
        port 22
        protocol-version v2
    }
 }
 system {
    host-name WLB
    login {
        user ubnt {
            authentication {
                encrypted-password $1$zKNoUbAo$gomzUbYvgyUMcD436Wo66.
            }
            level admin
        }
    }
    name-server 8.8.8.8
    ntp {
        server 0.ubnt.pool.ntp.org {
        }
        server 1.ubnt.pool.ntp.org {
        }
        server 2.ubnt.pool.ntp.org {
        }
        server 3.ubnt.pool.ntp.org {
        }
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone UTC
 }


 /* Warning: Do not remove the following line. */
 /* === vyatta-config-version: "config-management@1:dhcp-relay@1:dhcp-server@4:firewall@4:ipsec@3:nat@3:qos@1:quagga@2:system@4:ubnt-pptp@1:vrrp@1:webgui@1:webproxy@1:zone-policy@1" === */
 /* Release version: v1.1.0beta3dev.4539852.130227.0102 */

 

Reply