Reply
Member
Posts: 105
Registered: ‎07-23-2015
Kudos: 16
Solutions: 3

PBR and NAT WAN DHCP

All configuration on an EdgeRouter X. eth0 is WAN. eth1 is LAN1 (10.92.9.0/30). eth2 disabled. eth3 disabled. eth4 is LAN2 and LAN3 (VLAN subnets 10.92.10.0/30 and 10.92.11.0/30 respectively). I've configured two psuedo interfaces on WAN to pull two DHCP address from ISP (they provide me with two WAN addresses over DHCP - they MUST use DHCP due to DHCP snooping/binding, DARP inspection, etc on their side). These are peth0 and peth1. The phyiscal eth0 itself is set to "no address". The WAN interface(s) is working fine and pulling IP addresses on the psuedo interfaces. I need to set the LAN1 address of 10.92.9.2 to go out through the peth0 interface via a 1:1 NAT (by using peth0 as an interface delcaration instead of WAN address). I need to set inbound traffic from peth0 to go directly to LAN1 address of 10.92.9.2. I need to set the LAN2 and LAN3 addresses of 10.92.10.2 and 10.92.11.2 to go out through peth1 (by using peth1 as an interface delcation instead of WAN address). I hope I'm making sense here. I've been pulling my hair out with NAT and policy-based routing. Both WAN address on DHCP point to the same provider router.

 

Essentially, provider's network is 192.168.1.0/24. Provider router is 192.168.1.1. I'm pulling 192.168.1.100 (peth0) and 192.168.1.101 (peth1). I need traffic to/from 10.92.9.2 to use peth0. I need traffic going out from 10.92.10.2 and 10.92.11.2 to use peth1.

 

I've been searching around the forums like crazy for an existing thread that answers my question. I'm finding issues similar to mine but I'm unable to piece them together to come to a complete solution. Thank you in advance for any help anybody can provide - it's greatly appreciated!

Veteran Member
Posts: 7,050
Registered: ‎03-24-2016
Kudos: 1824
Solutions: 803

Re: PBR and NAT WAN DHCP

You only have a single WAN connection.  With multiple IPs.

 

I'd try to use only a single interface for sending out traffic.  (assign 1st IP to interface itself)

The extra IPs  shouldn't alter routing table:   ".....dhcp-options default-route no-update"

 

Now use sNAT rules in single interface instead of masquerade, use ADDRv4_peth0 or 1 to use extra addresses.

This way, no PBR is required

Member
Posts: 105
Registered: ‎07-23-2015
Kudos: 16
Solutions: 3

Re: PBR and NAT WAN DHCP

[ Edited ]

I seem to be having a heck of a time trying to get this set up properly. I'd prefer it if I could use peth0 and peth1 for both addresses instead of obtaining one of my available WAN addresses to the physical port as I want to be able to set the MAC addresses for each interface on the WAN which I can do with pseudo interfaces.

 

Below is my current interface configuration. I'm playing with the 192.168.6.2 address to go outbound on peth1 (PAT) in addition ot the /30 networks mentioned in the original post. Traffic to/from 10.92.9.2 should go in/out of peth0 (1:1 NAT).

 

interfaces {
    ethernet eth0 {
        description WAN
        duplex auto
        firewall {
            in {
                name WAN_IN
            }
            local {
                name WAN_LOCAL
            }
        }
        speed auto
    }
    ethernet eth1 {
        description STROTZ
        duplex auto
        speed auto
    }
    ethernet eth2 {
        description Local
        disable
        duplex auto
        speed auto
    }
    ethernet eth3 {
        address 192.168.6.1/24
        description Local
        duplex auto
        firewall {
            in {
                name ALLOW-TEST
            }
        }
        speed auto
    }
    ethernet eth4 {
        description FARM
        duplex auto
        poe {
            output off
        }
        speed auto
        vif 2 {
            address 10.92.2.1/24
            description MGMT
            mtu 1500
        }
        vif 10 {
            address 10.92.10.1/30
            description FARM-BIGHOUSE
            firewall {
                in {
                    name CUSTOMER-BLOCK-OUTBOUND
                }
            }
            mtu 1500
        }
        vif 11 {
            address 10.92.11.1/30
            description FARM-LITTLEHOUSE
            firewall {
                in {
                    name CUSTOMER-BLOCK-OUTBOUND
                }
            }
            mtu 1500
        }
    }
    loopback lo {
    }
    pseudo-ethernet peth0 {
        address dhcp
        firewall {
            in {
                name WAN_IN
            }
            local {
                name WAN_LOCAL
            }
        }
        link eth0
        mac 00:01:22:aa:bb:2c
    }
    pseudo-ethernet peth1 {
        address dhcp
        dhcp-options {
            default-route no-update
        }
        firewall {
            in {
                name WAN_IN
            }
            local {
                name WAN_LOCAL
            }
        }
        link eth0
        mac 00:01:22:aa:bb:23
    }
    switch switch0 {
        address 10.92.9.1/30
        description STROTZ-SW
        switch-port {
            interface eth1 {
            }
        }
    }
}

 

 

And here is my NAT configuration. I can't seem to set sNAT rules to use the interface IP address, only dNAT rules.

nat {
        rule 1 {
            destination {
                group {
                    address-group ADDRv4_peth0
                }
            }
            inbound-interface peth0
            inside-address {
                address 10.92.9.2
            }
            log disable
            type destination
        }
        rule 5000 {
            description STROTZ
            log disable
            outbound-interface peth0
            protocol all
            source {
                address 10.92.9.2
            }
            type masquerade
        }
        rule 5001 {
            description FARM
            log disable
            outbound-interface peth1
            outside-address {
            }
            protocol all
            source {
                group {
                    address-group FARM
                }
            }
            type masquerade
        }

 

Member
Posts: 105
Registered: ‎07-23-2015
Kudos: 16
Solutions: 3

Re: PBR and NAT WAN DHCP

[ Edited ]

So I was searching through the forums some more and found this thread. Sounds like the same exact thing I am trying to accomplish, just with less IP addresses. It doesn't look like there was actually a solution to this thread at that time. Is there any way to accomplish this?

 

https://community.ubnt.com/t5/EdgeMAX/source-based-routing-to-pseudo-ethernet/td-p/998775

 

Veteran Member
Posts: 7,050
Registered: ‎03-24-2016
Kudos: 1824
Solutions: 803

Re: PBR and NAT WAN DHCP

What is the error message on the sNAT rule?

If your IP addresses are "sort-of" static, use those in sNAT rules

Member
Posts: 105
Registered: ‎07-23-2015
Kudos: 16
Solutions: 3

Re: PBR and NAT WAN DHCP

[ Edited ]

Currently have this plugged into a test "WAN" network - same scenario as what my ISP is going to provide. Test WAN network is 10.2.10.0/24. Router is 10.2.10.1. Currently am obtaining 10.2.10.35 and 10.2.10.36 through DHCP on the peth0-1 interfaces. Went ahead and tried manually entering 10.2.10.35 and 10.2.10.36 in the sNAT rules with no luck. One translation works on one IP, but the other doesn't. If I move the "dhcp-options default-route no-update" to the other peth interface, the other translation starts working. I believe I'm having a routing problem. See route table below. Thanks again for all the help on this - I truly appreciate it.

 

capture 2017-11-06 at 11.23.52 PM.png

Member
Posts: 105
Registered: ‎07-23-2015
Kudos: 16
Solutions: 3

Re: PBR and NAT WAN DHCP

I realize that these forums are active and there are a lot of individuals that need assistance but I'm hoping that I'll be able to get this up and running before this weekend. It looks as though I won't be able to make this work without PBR of some sort since the WAN addresses are DHCP and reside within the same network with the same upstream gateway. Once I get a good kick start on the possible routing issue that I have, I should be able to handle the NAT, firewall rules, etc. Thank you all again in advance for any assistance you can provide!

Member
Posts: 105
Registered: ‎07-23-2015
Kudos: 16
Solutions: 3

Re: PBR and NAT WAN DHCP

I found another post where the individual is attempting to do the same thing I am. Been digging as hard as I can! Hopefully somebody can assist me.

https://community.ubnt.com/t5/EdgeMAX/specify-interface-for-default-gw-in-static-route/td-p/1032359
Member
Posts: 105
Registered: ‎07-23-2015
Kudos: 16
Solutions: 3

Re: PBR and NAT WAN DHCP

[ Edited ]

@cabsil thank you for the reply but I think you are missing the point. The solution simply is not working. Below is an updated configuration showing the exact steps as mentioned by @16again. The problem lies in the fact that the WAN addresses are in the same subnet on different interfaces, whether they're pseudo interfaces or not. I have verified that this is a routing problem by changing the default route to reside on a different WAN interface while leaving NAT alone. It's not multiple static WAN IP addresses on a single interface with that interface having the default route otherwise this wouldn't be a problem because I have personally configured that before in a different scenario with zero problems. This is WAN IP addresses on DIFFERENT interfaces over DHCP that are still in the same subnet. I have to do it this way because it doesn't appear possible to have multiple DHCP addresses on a single interface because there is only one MAC per interface and I can't get assigned two DHCP addresses on a single interface with only a single MAC address on that interface. How is the router going to know how to send traffic over peth0 when eth0 and peth0 have IP addresses in the same network and eth0 is marked as the default route? I don't want to create an argument out of this but if you don't think I haven't tried @16again's methods 16 different ways, that's simply incorrect. I've already attempted to do what he has mentioned. Thank you for the valuable input regarding changing the MAC of a non-pseudo interface but the rest of your post seems to be a preach that really was unnecessary. I appreciate all valuable input but I am not helpless and I'm not simply giving up after one command that doesn’t work and saying "didn't work!" after a single configuration change. My ISP uses ip-source-guard and all of those security measures. If I obtain an IP on peth0, I can’t use eth0 to send traffic out using that address that was assigned to peth0 because the source MAC in the frame won’t match the assigned MAC provided through DHCP to peth0. I must say I found your post slightly offensive. If anybody has valuable input in regards to assist me in solving this problem, it would be greatly appreciated.

 

WAN DHCP leases shown (10.2.10.36 and 10.2.10.40) with 10.2.10.40 being directly on eth0:

ubnt@ubnt:~$ show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface    IP Address                        S/L  Description                 
---------    ----------                        ---  -----------                 
eth0         10.2.10.40/24                     u/u  WAN                         
eth1         -                                 u/D  STROTZ                      
eth2         -                                 A/D  Local                       
eth3         192.168.6.1/24                    u/u  Local                       
eth4         -                                 u/D  FARM                        
eth4.2       10.92.2.1/24                      u/D  MGMT                        
eth4.10      10.92.10.1/30                     u/D  FARM-BIGHOUSE               
eth4.11      10.92.11.1/30                     u/D  FARM-LITTLEHOUSE            
lo           127.0.0.1/8                       u/u                              
             ::1/128                          
peth0        10.2.10.36/24                     u/u                              
switch0      10.92.9.1/30                      u/u  STROTZ-SW                   

Route table shown (default route to 10.2.10.1 over eth0):

ubnt@ubnt:~$ show ip route
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       > - selected route, * - FIB route, p - stale info
IP Route Table for VRF "default"
S    *> 0.0.0.0/0 [210/0] via 10.2.10.1, eth0
C    *> 10.2.10.0/24 is directly connected, eth0
C       10.2.10.0/24 is directly connected, peth0
C    *> 10.92.9.0/30 is directly connected, switch0
C    *> 127.0.0.0/8 is directly connected, lo
C    *> 192.168.6.0/24 is directly connected, eth3

Address group configured:

ubnt@ubnt:~$ show firewall group FARM
Name       : FARM
Type       : address
Family     : IPv4
Description: 
References : none
Members    :
             10.92.10.2
             10.92.11.2
             192.168.6.2

sNAT rules configured (I'm not even worried about dNAT rules at this point):

rule 5000 {
            description STROTZ
            log disable
            outbound-interface eth0
            outside-address {
                address 10.2.10.40
            }
            protocol all
            source {
                address 10.92.9.2
            }
            type source
        }
        rule 5001 {
            description FARM
            log disable
            outbound-interface eth0
            outside-address {
                address 10.2.10.36
            }
            protocol all
            source {
                group {
                    address-group FARM
                }
            }
            type source
        }

EDIT: Even without ip source guard in my test WAN environment, I still can't get this to work. 

Senior Member
Posts: 5,558
Registered: ‎01-04-2017
Kudos: 767
Solutions: 272

Re: PBR and NAT WAN DHCP

Create two static route tables one with peth0 one with peth1. Then create a firewall modify rule forcing"farm" our of peth1 and your other group out of Peth0. This is the same concept as load balancing without the load balancing part. Nat rules should stay as masquerade.
Member
Posts: 105
Registered: ‎07-23-2015
Kudos: 16
Solutions: 3

Re: PBR and NAT WAN DHCP

[ Edited ]

I believe I was muddling around with this the other day in my efforts to get this working but I’ll give that a shot tomorrow and post back the results! Thank you all again for the help so far. I really appreciate it.

Veteran Member
Posts: 7,050
Registered: ‎03-24-2016
Kudos: 1824
Solutions: 803

Re: PBR and NAT WAN DHCP

@tstrotz wrote:

My ISP uses ip-source-guard and all of those security measures. If I obtain an IP on 
peth0, I can’t use eth0 to send traffic out using that address that was assigned to
peth0 because the source MAC in the frame won’t match the assigned MAC provided
through DHCP to peth0.

That really complicates matters, and will break my suggested solution.

Note it's your ISP giving you a hard time.

I also have been in cases to get configs working in a single device, and after many hours I started thinking: Why didn't I start with two ER-X devices to begin with?  At the price point, it's worth considering.

Senior Member
Posts: 5,558
Registered: ‎01-04-2017
Kudos: 767
Solutions: 272

Re: PBR and NAT WAN DHCP


tstrotz wrote:

I believe I was muddling around with this the other day in my efforts to get this working but I’ll give that a shot tomorrow and post back the results! Thank you all again for the help so far. I really appreciate it.


For dhcp you may need to make two scripts for each interface (or you should be able to combine them together_ and 2 scripts total in /config/scripts/

This should 

/config/scripts/enter-dhcp-fix-peth1

 # enter hooks script:

 case $reason in

 BOUND|RENEW|REBIND|REBOOT)
    # Do not set default gw, but create a copy to use it in exit hooks
    # scripts:
if [ "$interface" = "peth1" -a -n "$new_routers" ] then export new_routers_copy=$new_routers # Avoid gw in default route table, but preserve copy for # policy based routing: unset new_routers
fi ;; FAIL|NBI|STOP|TIMEOUT|RELEASE|EXPIRE)
if [ "$interface" = "peth1"] then ip route del $old_network_number/$old_subnet_mask \ dev $interface table 2 ip route del default via $old_routers table 2 ip rule del from $old_network_number/$old_subnet_mask lookup 2
fi ;; esac

/config/scripts/exit-dhcp-fix-peth1

 # exit hooks script:

 case $reason in

 BOUND|RENEW|REBIND|REBOOT)
if [ "$interface" = "peth1"] then ip route add $new_network_number/$new_subnet_mask \ dev $interface table 2 ip route add default via $new_routers_copy table 2 ip rule add from $new_network_number/$new_subnet_mask lookup 2
fi ;; esac

you would symbolic link these scripts to the enter/exit hookd in /etc/dhcp3/

 

 

These scripts just take the defaulte routes and place them on a seperate route table.  In the example we used table 2.

Member
Posts: 105
Registered: ‎07-23-2015
Kudos: 16
Solutions: 3

Re: PBR and NAT WAN DHCP

[ Edited ]

@smyers119 I've went ahead and attempted the following with no luck. Outbound pings to 8.8.8.8 don't succeeed from 10.92.9.2 on the client on eth1 (switch0) or from 192.168.6.2 on the client on eth3. I've followed this guide and below is the configuration. I may be missing something but I think I followed the guide fairly closely. It was definitely worth a shot - any other ideas that may involve a simple config setup? I'm going to start working on your script next - thank you VERY much for taking the time to provide that. I'll post back results although it may be a bit as I'm not too sharp with bash (if that's even what it is). @16again it definitely does complicate matters and my apologies - I really should have mentioned this aspect earlier on but I didn't think it would become a factor until I started thinking about it. I'm starting to think I may need to result to pfSense (or the two EdgeRouter X's that @16again suggested) but I'm REALLY hoping I can get it working with EdgeMax.

 

Firewall config:

firewall {
    all-ping enable
    broadcast-ping disable
    group {
        address-group FARM {
            address 10.92.10.2
            address 10.92.11.2
            address 192.168.6.2
            description ""
        }
    }
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians enable
    modify SOURCE_ROUTE {
        rule 10 {
            modify {
                table 10
            }
            source {
                address 10.92.9.2/30
            }
        }
        rule 20 {
            modify {
                table 20
            }
            source {
                address 192.168.6.2/24
            }
        }
    }

 Static route (two tables) config:

protocols {
    static {
        table 10 {
            interface-route 0.0.0.0/0 {
                next-hop-interface eth0 {
                }
            }
        }
        table 20 {
            interface-route 0.0.0.0/0 {
                next-hop-interface peth0 {
                }
            }
        }
    }
}

Interface firewall inbound rule config:

interfaces {
    ethernet eth0 {
        address dhcp
        description WAN
        duplex auto
        firewall {
            in {
                name WAN_IN
            }
            local {
                name WAN_LOCAL
            }
        }
        speed auto
    }
    ethernet eth1 {
        description STROTZ
        duplex auto
        firewall {
            in {
                modify SOURCE_ROUTE
            }
        }
        speed auto
    }
    ethernet eth2 {
        description Local
        disable
        duplex auto
        speed auto
    }
    ethernet eth3 {
        address 192.168.6.1/24
        description Local
        duplex auto
        firewall {
            in {
                modify SOURCE_ROUTE
                name ALLOW-TEST
            }
        }
        speed auto
    }
    ethernet eth4 {
        description FARM
        duplex auto
        poe {
            output off
        }
        speed auto
        vif 2 {
            address 10.92.2.1/24
            description MGMT
            mtu 1500
        }
        vif 10 {
            address 10.92.10.1/30
            description FARM-BIGHOUSE
            firewall {
                in {
                    name CUSTOMER-BLOCK-OUTBOUND
                }
            }
            mtu 1500
        }
        vif 11 {
            address 10.92.11.1/30
            description FARM-LITTLEHOUSE
            firewall {
                in {
                    name CUSTOMER-BLOCK-OUTBOUND
                }
            }
            mtu 1500
        }
    }
    loopback lo {
    }
    pseudo-ethernet peth0 {
        address dhcp
        dhcp-options {
            default-route no-update
            default-route-distance 210
            name-server update
        }
        firewall {
            in {
                name WAN_IN
            }
            local {
                name WAN_LOCAL
            }
        }
        link eth0
        mac 00:01:22:aa:bb:2c
    }
    switch switch0 {
        address 10.92.9.1/30
        description STROTZ-SW
        firewall {
            in {
                modify SOURCE_ROUTE
            }
        }
        mtu 1500
        switch-port {
            interface eth1 {
            }
            vlan-aware disable
        }
    }
}

And the NAT rules haven't changed:

nat {
        rule 1 {
            destination {
                group {
                    address-group ADDRv4_eth0
                }
            }
            inbound-interface eth0
            inside-address {
                address 10.92.9.2
            }
            log disable
            protocol all
            type destination
        }
        rule 5000 {
            description STROTZ
            log disable
            outbound-interface eth0
            outside-address {
                address 10.2.10.40
            }
            protocol all
            source {
                address 10.92.9.2
            }
            type source
        }
        rule 5001 {
            description FARM
            log disable
            outbound-interface eth0
            outside-address {
                address 10.2.10.36
            }
            protocol all
            source {
                group {
                    address-group FARM
                }
            }
            type source
        }
    }

 

Member
Posts: 182
Registered: ‎10-19-2016
Kudos: 50
Solutions: 19

Re: PBR and NAT WAN DHCP

@tstrotz, did you get this working? 

 

 

Member
Posts: 105
Registered: ‎07-23-2015
Kudos: 16
Solutions: 3

Re: PBR and NAT WAN DHCP

I gave up on the EdgeMax platform and got this working on pfSense in a VM on my on-site server in under an hour. The pfSense platform is far superior in featureset compared to what Ubiquiti has to offer in EdgeMax. Hopefully the simple task I was attempting to perform will be available in a later release. Thank you everyone for the help thus far - it is greatly appreciated and it’s nobody’s fault but Ubiquiti’s that I was unable to get this working. It should be available directly in the CLI without requiring a script (by the way @smyers119, thank you very much on developing that script for me, unfortunately, I was hoping it would be directly available in the CLI).
Reply