Reply
New Member
Posts: 6
Registered: ‎11-04-2015
Kudos: 2

USG, Multi-WAN, Policy-based routing.

Hello,

 

I have a USG that I would like to have DUAL WAN connectivity, with policy based routing.  I have configured the USG through UNIFI (WAN1 is DHCP and receives a private address, WAN2 has a public /29).

 

My intention is to apply specific source_nat rules to different corporate subnets, eg

172.16.1.0/24 -> xxx.xxx.xxx.59/29

172.16.2.0/24 -> xxx.xxx.xxx.60/29

172.16.3.0/24 -> xxx.xxx.xxx.61/29

 

(I have this arrangement working with a pfsense based firewall, and we're looking to replace with USG devices).

 

I have been reading https://help.ubnt.com/hc/en-us/articles/204952274-EdgeMAX-Policy-based-routing-source-address-based- and while helpful, I receive errors when performing a commit

 

Firewall config error: Action 'modify' requires more specific configuration under the 'modify' node

 

I attempt the commit after entering the commands below (I'm literally following the guide as I am new to EdgeOS).

ubnt@RTR# set firewall modify SOURCE_ROUTE rule 10 description 'traffic from eth2.100 to ISP1'
ubnt@RTR# set firewall modify SOURCE_ROUTE rule 10 source address 172.16.1.0/24
ubnt@RTR# set firewall modify SOURCE_ROUTE rule 10 modify table 1

Below is my interface configuration for the two WAN ports:

ethernet eth2 {
     address dhcp
     dhcp-options {
         client-option "retry 60;"
     }
     firewall {
         in {
             name WAN_IN
         }
         local {
             name WAN_LOCAL
         }
     }
 }
 ethernet eth3 {
     address xxx.xxx.xxx.58/29
     firewall {
         in {
             name WAN_IN
         }
         local {
             name WAN_LOCAL
         }
     }
 }

Appreciate any assistance please.  

New Member
Posts: 19
Registered: ‎06-08-2016
Kudos: 162

Re: USG, Multi-WAN, Policy-based routing.

Hey did you have any Solution for your Problem now? i want to do the same thing with 6 vlan´s but it is not working


thank you
New Member
Posts: 15
Registered: ‎02-12-2017

Re: USG, Multi-WAN, Policy-based routing.

[ Edited ]

Hi - I'm also very curious if this is possible, I have a similar existing configuration with pfSense performing policy routing of certain subnets out of a specific gateway (which happens to be an OpenVPN client connection). I'd love to replicate this on a USG.

 

If anyone has gotten this working on a USG please let us know Man Happy

 

EDIT: It seems that with some manual configuration it is possible:

 

Ubiquiti Employee
Posts: 783
Registered: ‎02-28-2017
Kudos: 231
Solutions: 79

Re: USG, Multi-WAN, Policy-based routing.

[ Edited ]

Yes, policy routing is possible via CLI and configurable to your hearts desire, here are some examples:
If you have a WAN1/WAN2 load balanced or configured for failover, you can force vlans out WAN2 if you like:

configure
set protocols static table 1 route 0.0.0.0/0 next-hop <wan2gateway> set firewall modify LOAD_BALANCE rule 3000 action modify set firewall modify LOAD_BALANCE rule 3000 modify table 1 set firewall modify LOAD_BALANCE rule 3000 source address <vlan_network> set firewall modify LOAD_BALANCE rule 3000 protocol all
commit;exit

You can do this, or even create another LB-group with WAN2 and primary and WAN1 as failover, and policy route it that way too.

Here's an example to get a LAN/VLAN or multiple networks policy routed out a VPN tunnel (using VTI):

configure
set firewall modify VPN_Gateway rule 3000 action modify set firewall modify VPN_Gateway rule 3000 modify table 1 set firewall modify VPN_Gateway rule 3000 source address <vlan_network> set firewall modify VPN_Gateway rule 3000 protocol all set protocols static interface-route 0.0.0.0/0 next-hop-interface vtix set interfaces ethernet ethx vif x firewall in modify VPN_Gateway
commit;exit


 

Brandon Jaffe | UniFi Routing & Switching | Austin, TX
New Member
Posts: 36
Registered: ‎06-12-2017
Kudos: 3
Solutions: 1

Re: USG, Multi-WAN, Policy-based routing.

This Is such a great feature that a lot of people would use, please implement a setting in the usg that allows a specific wan route per network 

 

New Member
Posts: 36
Registered: ‎06-12-2017
Kudos: 3
Solutions: 1

Re: USG, Multi-WAN, Policy-based routing.

@UBNT-jaffe  Will this work on the small USG 3 unit or will it only work on the bigger ones ?

Ubiquiti Employee
Posts: 783
Registered: ‎02-28-2017
Kudos: 231
Solutions: 79

Re: USG, Multi-WAN, Policy-based routing.

@dwayneohara Policy routing is on the roadmap for implementation in the GUI, and it's possible right now via CLI on the USG3 and USG Pro 4.

Brandon Jaffe | UniFi Routing & Switching | Austin, TX
New Member
Posts: 18
Registered: ‎12-22-2015

Re: USG, Multi-WAN, Policy-based routing.

Rather than routing a whole VLAN over a VPN (e.g. pptpc0) is it possible to specify a single LAN address (e.g. 192.168.1.13) to use the VPN gateway while all other LAN addresses use the default route?

Ubiquiti Employee
Posts: 783
Registered: ‎02-28-2017
Kudos: 231
Solutions: 79

Re: USG, Multi-WAN, Policy-based routing.

@tmr0 Sure, just take note of the commands I posted above, and in the Source Address portion, just fill in the address rather than the entire network.

Brandon Jaffe | UniFi Routing & Switching | Austin, TX
New Member
Posts: 2
Registered: ‎11-12-2016
Kudos: 2

Re: USG, Multi-WAN, Policy-based routing.

@UBNT-jaffe is it possible to configure this for <wan2gateway> that I get from my ISP as DHCP?

 

New Member
Posts: 36
Registered: ‎11-21-2016
Kudos: 11

Re: USG, Multi-WAN, Policy-based routing.

[ Edited ]

YSo, I’m looking for a bit of a sanity check before I apply this configuration.

 

Capture111.JPG

I have 2 WAN connections connected to a USG with Load Balancing (Failover) configured.
While running through WAN 1, I need management access to the router on WAN 2, and same vise versa.
From what I gather, its policy routing that is required to achieve this and not a static route??

With that thought, is the following config the correct way to achieve this:

 

set protocols static table 1 route 10.11.11.0/24 next-hop 10.11.11.11
set firewall modify LOAD_BALANCE rule 3000 action modify
set firewall modify LOAD_BALANCE rule 3000 modify table 1
set firewall modify LOAD_BALANCE rule 3000 destination address 10.11.11.0/24
set firewall modify LOAD_BALANCE rule 3000 protocol all
set protocols static table 2 route 10.22.22.0/24 next-hop 10.22.22.22
set firewall modify LOAD_BALANCE rule 2999 action modify
set firewall modify LOAD_BALANCE rule 2999 modify table 2
set firewall modify LOAD_BALANCE rule 2999 destination address 10.22.22.0/24
set firewall modify LOAD_BALANCE rule 2999 protocol all
Ubiquiti Employee
Posts: 783
Registered: ‎02-28-2017
Kudos: 231
Solutions: 79

Re: USG, Multi-WAN, Policy-based routing.

Yes Gary that looks correct so long as you have the default masquarade NAT rule enabled. When 192.168.0.x clients leave either WAN or WAN2 they must be NATted to the 10.11.11.1 or 10.22.22.2 IP if you want return traffic from the upstream routers to return correctly.

If you for any reason have NAT disabled on the USG, you could just add a static route on the upstream routers pointing 192.168.0.0/24 to 10.11.11.1 or 10.22.22.2 depending on the router.
Brandon Jaffe | UniFi Routing & Switching | Austin, TX
New Member
Posts: 36
Registered: ‎11-21-2016
Kudos: 11

Re: USG, Multi-WAN, Policy-based routing.

Thanks,

 

I can confirm it works. I also have NAT disabled on the USG WAN Interfaces with Static Routes on the ISP Routers/Firewalls pointing the LAN subnets back towards the USG.

 

Would have liked to have added this config to my config.gateway.json file .

However after running the commands above, then commiting and saving them; mca-ctrl -t dump-cfg fails to show the changes, yet i can see them in /config/config.boot .

New Member
Posts: 7
Registered: ‎06-26-2017

Re: USG, Multi-WAN, Policy-based routing.

@UBNT-jaffe I guess routing VLANs out WAN2 is still supposed to work using CLI and USG v4.3.49?
It's not working for me somehow. At least nothing happens and the specified VLAN still goes out WAN1.

This is my console output:

reepje123@UniFiSecurityGateway:~$ configure
[edit]
reepje123@UniFiSecurityGateway# set protocols static table 1 route 0.0.0.0/0 next-hop 10.0.0.138
[edit]
reepje123@UniFiSecurityGateway# set firewall modify LOAD_BALANCE rule 2990 action modify
[edit]
reepje123@UniFiSecurityGateway# set firewall modify LOAD_BALANCE rule 2990 modify table 1
[edit]
reepje123@UniFiSecurityGateway# set firewall modify LOAD_BALANCE rule 2990 source address 192.168.2.1/24
[edit]
reepje123@UniFiSecurityGateway# set firewall modify LOAD_BALANCE rule 2990 protocol all
[edit]
reepje123@UniFiSecurityGateway# commit;exit
Warning: configuration changes have not been saved.
exit
reepje123@UniFiSecurityGateway:~$

I have a config.gateway.json to route one IP out WAN2 which works fine when I additionally enable a "next hop" static route for that IP. It is based on a file from another Ubnt employee from another thread. Could this conflict somehow?

Member
Posts: 213
Registered: ‎01-10-2017
Kudos: 28
Solutions: 11

Re: USG, Multi-WAN, Policy-based routing.

@UBNT-jaffe  is it possible to specify the outbound IP for those of us with multiple statics?

New Member
Posts: 36
Registered: ‎11-21-2016
Kudos: 11

Re: USG, Multi-WAN, Policy-based routing.


reepje123 wrote:

@UBNT-jaffe I guess routing VLANs out WAN2 is still supposed to work using CLI and USG v4.3.49?
It's not working for me somehow. At least nothing happens and the specified VLAN still goes out WAN1.

This is my console output:

reepje123@UniFiSecurityGateway:~$ configure
[edit]
reepje123@UniFiSecurityGateway# set protocols static table 1 route 0.0.0.0/0 next-hop 10.0.0.138
[edit]
reepje123@UniFiSecurityGateway# set firewall modify LOAD_BALANCE rule 2990 action modify
[edit]
reepje123@UniFiSecurityGateway# set firewall modify LOAD_BALANCE rule 2990 modify table 1
[edit]
reepje123@UniFiSecurityGateway# set firewall modify LOAD_BALANCE rule 2990 source address 192.168.2.1/24
[edit]
reepje123@UniFiSecurityGateway# set firewall modify LOAD_BALANCE rule 2990 protocol all
[edit]
reepje123@UniFiSecurityGateway# commit;exit
Warning: configuration changes have not been saved.
exit
reepje123@UniFiSecurityGateway:~$

I have a config.gateway.json to route one IP out WAN2 which works fine when I additionally enable a "next hop" static route for that IP. It is based on a file from another Ubnt employee from another thread. Could this conflict somehow?


 

Could it be that your source address is set to "192.168.2.1/24" and not "192.168.2.0/24" ?

 

Ubiquiti Employee
Posts: 783
Registered: ‎02-28-2017
Kudos: 231
Solutions: 79

Re: USG, Multi-WAN, Policy-based routing.

@reepje123 I would try doing what @Gary_L mentioned and define the source network as 192.168.2.0/24.

You can "show firewall modify statistics" to see if rule 2990 is actually getting matched on (packet,byte counter). 
You also want to "clear connection-tracking" after committing that rule to reset the state table.


@kb9gxk shouldn't be a problem if you add a source NAT rule into the mix:
set service nat rule 5500 type source

set service nat rule 5500 outside-address x.x.x.x
set service nat rule 5500 outbound-interface ethx
set service nat rule 5500 source address <vlan_network>
set service nat rule 5500 protocol all

Brandon Jaffe | UniFi Routing & Switching | Austin, TX
New Member
Posts: 7
Registered: ‎06-26-2017

Re: USG, Multi-WAN, Policy-based routing.

@UBNT-jaffe I don't exactly know why, but it works now. I first retried using 192.168.2.0/24 but that didn't work. It didn't even show up in "show firewall modify statistics" for some reason.

I then reset the state table and tried everything again with another rule number. That finally worked. Thanks!

New Member
Posts: 7
Registered: ‎06-26-2017

Re: USG, Multi-WAN, Policy-based routing.

How can I make these rules persistent? After any provision all changes are lost. They're actually lost everytime I commit the rules because it starts provisioning afterwards. I tried the save command but that didn't help.

Ubiquiti Employee
Posts: 783
Registered: ‎02-28-2017
Kudos: 231
Solutions: 79

Re: USG, Multi-WAN, Policy-based routing.

@reepje123 Sorry, I almost always leave that part out now since I assume everyone knows about it. 
You can make them persistent with a file named config.gateway.json 
(more on that here: https://help.ubnt.com/hc/en-us/articles/215458888-UniFi-How-to-further-customize-USG-configuration-w...)

Basically, you make the changes on the USG, commit the changes, then type mca-ctrl -t dump-cfg
That will spit out the current configuration in JSON format, then you take the new parts you want and throw them into a file called config.gateway.json and put that file in the correct directory on your controller (/srv/unifi/data/sites/<site_name>/config.gateway.json for the cloud key. Paths for other OS's should be in the article I linked above.

You can validate the JSON on jsonlint.com and find other examples all throughout google / the forum for things like this.

The only one issue I've ran into is that certain parts of the configuration are "omitted" when running mca-ctrl -t dump-cfg, so the new changes won't always display in a JSON format (depending on what you configured). You can "cat /config/config.boot" or "show configuration all" - which will show all of your updated changes, but it won't show them in JSON formatting. I usually just look at examples of config.gateway.json files posted online to get it right if that's the case.

Brandon Jaffe | UniFi Routing & Switching | Austin, TX
Reply