New Member
Posts: 10
Registered: ‎06-26-2017
Kudos: 1
Solutions: 1

Advice/Help configuring multiple subnets with ER-X

Hi, I'm totally new to Ubiquti and would appreciate some advice/guidance on setting up a router with multiple subnets (and also whether the ER-X I purchased is a reasonable choice for the job).

 

I have a small OpenStack cluster for hosting virtual machines on an isolated development subnet ('devnet' in the diagram below). There are 3 ports/networks of interest:

 

1) one port for management via a web UI. This is on the development subnet (10.10.11.0/24).

 

2) and 3) are the data networks ("provider" or "tenant" networks in OpenStack speak) used to communicate with the VMs. I want the router to provide a separate DHCP service for each subnet. Also, the VMs should be able to access the outside network via NAT.

 

Some co-workers with similar setups just use a "jump machine" with 3 NICs and simply remote log-in to to the jump machine to talk to devices on each network. I thought it would be nice to have a router that would allow me to access VMs or external devices on the two (or more in future) separate tentant networks from the  development subnet.

 

I took a stab at configuring things, but clearly there's a lot I don't understand, and again, just wanted a sanity check that what I'm trying to do isn't off base or better done another way.

 

Here's a diagram of my setup. Assume that I have a switch attached to each ER-X port and that all my networks are flat (no vlans):

 

ER-X-as-Multi-lan-router.png

 

Here is my config.boot from my attempt to get this working by reading forums, etc. Note that I really do mean to use '/23' for the network mask on one of the networks.. it's dictated by something out of my control.

 

interfaces {
    ethernet eth0 {
        address dhcp
        description devnet
        duplex auto
        ip {
            enable-proxy-arp
        }
        speed auto
    }
    ethernet eth1 {
        address 10.10.12.1/24
        description tic-flat-net0
        duplex auto
        ip {
            enable-proxy-arp
        }
        speed auto
    }
   ethernet eth2 {
        address 172.28.124.1/23
        description tic-provider-net0
        duplex auto
        speed auto
    }
    ethernet eth3 {
        description Local
        disable
        duplex auto
        speed auto
    }
    ethernet eth4 {
        description Local
        disable
        duplex auto
        speed auto
    }
    loopback lo {
    }
 switch switch0 {
        mtu 1500
    }
}
port-forward {
    auto-firewall disable
    hairpin-nat enable
    lan-interface eth2
    lan-interface eth3
    wan-interface eth0
}
protocols {
    static {
    }
}
service {
    dhcp-server {
        disabled false
        hostfile-update disable
        shared-network-name tic-flat-net {
            authoritative disable
            subnet 10.10.12.0/24 {
                dns-server 8.8.8.8
                dns-server 8.8.4.4
                lease 86400
                start 10.10.12.50 {
                    stop 10.10.12.100
                }
            }
        }
        shared-network-name tic-provider-net {
            authoritative disable
subnet 172.28.124.0/23 {
                default-router 172.28.124.1
                dns-server 8.8.8.8
                dns-server 8.8.4.4
                lease 86400
                start 172.28.124.50 {
                    stop 172.28.124.100
                }
            }
        }
        use-dnsmasq disable
    }
    dns {
        forwarding {
            cache-size 150
            listen-on eth0
            listen-on switch0
        }
    }
    gui {
        http-port 80
        https-port 443
        older-ciphers enable
    }
    nat {
        rule 5010 {
            log disable
            outbound-interface eth0
            protocol all
            type masquerade
        }
    }
ssh {
        port 22
        protocol-version v2
    }
}
system {
    domain-name devnet
    gateway-address 10.10.11.1
    host-name ubnterx01
    login {
        user admin {
            authentication {
                encrypted-password <redacted!>
                plaintext-password ""
            }
            level admin
        }
    }
    ntp {
        server 0.ubnt.pool.ntp.org {
        }
        server 1.ubnt.pool.ntp.org {
        }
        server 2.ubnt.pool.ntp.org {
        }
        server 3.ubnt.pool.ntp.org {
        }
    }
    offload {
        hwnat enable
        ipsec enable
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone America/New_York
    traffic-analysis {
        dpi enable
        export enable
    }
}


/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "config-management@1:conntrack@1:cron@1:dhcp-relay@1:dhcp-server@4:firewall@5:ipsec@5:nat@3:qos@1:quagga@2:system@4:ubnt-pptp@1:ubnt-util@1:vrrp@1:webgui@1:webproxy@1:zone-policy@1" === */
/* Release version: v1.9.1.1.4977602.170427.0113 */


 

I can ping/ssh to VMs, etc when logged into the ER-X CLI, but when I try to set up a route on a laptop on the devnet via the ER-X (eth0, 10.10.11.15), clearly traffic isn't getting through. Any help or pointers appreciated.

 

Thanks,


Rich

Senior Member
Posts: 2,737
Registered: ‎04-21-2015
Kudos: 404
Solutions: 108

Re: Advice/Help configuring multiple subnets with ER-X

Any security policies in place for your traffic to get through the router?
Thanks,
Myky
CWNA
--------------------------------------------------------------------------------------------------------------------------------------------------
Don`t blame the device as it`s always doing what you have asked it to do, this is not always the same as what you want.
Veteran Member
Posts: 8,076
Registered: ‎03-24-2016
Kudos: 2124
Solutions: 929

Re: Advice/Help configuring multiple subnets with ER-X

You have masquerade in place.  So if you try to access a VM through the ER-X, reply gets masqueraded to IP address of eth0

 

to get around this:

-you can use portmappings, but that's not scalable if VMs come and go

Or

-get rid of NAT altogether on ER-X.  The 10.10.11.1 gateway must have route back to VM networks, and do NAT for them

Or

-Exclude traffic to 10.10.11.0/24 from being masqueraded,  On masquerade rule, add destination match ! 10.10.11.0/24   , or add seperate nat exclude rule.

New Member
Posts: 10
Registered: ‎06-26-2017
Kudos: 1
Solutions: 1

Re: Advice/Help configuring multiple subnets with ER-X

> Any security policies in place for your traffic to get through the router?

 

I don't think so ... not intentionally anyway. This is a development network that's totally under my control. The gateway router at 10.10.11.1 is just a  home-grade Asus router with some alternate firmware.

 

I'm not sure I understand all the implications of your question,but since this is all internal networking, I don't think I need (or want) a firewall and it would be nice if devices on any of the 3 networks could reach each other. But is that really what you're asking?

 

I know that if I were setting up a Linux box with 3 NICs as a router, I'd need to enable port forwarding. I also have a vague notion that some IPtables rules would be required, but this is where I get a bit fuzzy.

 

Senior Member
Posts: 2,737
Registered: ‎04-21-2015
Kudos: 404
Solutions: 108

Re: Advice/Help configuring multiple subnets with ER-X

Sorry, no default firewall rules are in place (l assumed you used wizard for this set up). If all traffic is allowed, when @16again is right. You have a source ip (your PC) and destination ip (one of the VMs). Lets say you run ping from PC ====> VM. Traffic passing through the router. All networks are directly connected, so Edge knows how to get to the VM. The router forwards your ping request. VM receives it (hopefull ping is not disable on the VM and it has a default gateway ip address of the EdgeRouter interface. It sends a reply to the PC but router applies NAT on the way out changing the source ip of the VM. This most likely your issue. As per advice exclude either one of the VMs ip address from the NAT rule for the test or whole subnet and test again.  Assuming your test PC has a default gateway mgmt ip address of the Edge

Thanks,
Myky
CWNA
--------------------------------------------------------------------------------------------------------------------------------------------------
Don`t blame the device as it`s always doing what you have asked it to do, this is not always the same as what you want.
New Member
Posts: 10
Registered: ‎06-26-2017
Kudos: 1
Solutions: 1

Re: Advice/Help configuring multiple subnets with ER-X

[ Edited ]

@16again: Thanks for the response.

VMs will come and go, so I really just want any host on 10.10.11.0/24 to be capable of routing to any arbitrary host on the other 2 subnets. I don't care that much about routing from outside of the gateway router at this point (I get access to 10.10.11.0/24 either directly or through OpenVPN).

So if I got rid of NAT altogether on the ER-X, would I just add static routes between the subnets?

As you can probably tell, I'm used to working with typical home routers and am trying to understand what I can and can't do with the ER-X. For example, without NAT, can you still use DHCP services and somehow limit them to DHCP requests coming in on a particular port?

Also, the 10.10.11.1 gateway is just an ASUS RT-AC68R router that I started using to set up my isolated development subnet -- would it be better to just use the ER-X as the router for all 3 subnets?

Thanks again for taking the time to respond.

Senior Member
Posts: 2,737
Registered: ‎04-21-2015
Kudos: 404
Solutions: 108

Re: Advice/Help configuring multiple subnets with ER-X

[ Edited ]

So if I got rid of NAT altogether on the ER-X, would I just add static routes between the subnets? - All networks are directly connected, no need a static routing between the subnets. You may have an issue for Asus to be able to access the VMs if you disable NAT on Edge. If this is the case then yes you need a static routes on Asus

As you can probably tell, I'm used to working with typical home routers and am trying to understand what I can and can't do with the ER-X. For example, without NAT, can you still use DHCP services and somehow limit them to DHCP requests coming in on a particular port? - Not sure who is providing a DHCP for your VMs now, but usually each subnet has its own DHCP server (most likely your edge router interface). Limit the scope if you want but l don't understand why you need it.

Also, the 10.10.11.1 gateway is just an ASUS RT-AC68R router that I started using to set up my isolated development subnet -- would it be better to just use the ER-X as the router for all 3 subnets? - Absolutely. Even will make things work easier. You can simply use Edge without the Asus router

Thanks,
Myky
CWNA
--------------------------------------------------------------------------------------------------------------------------------------------------
Don`t blame the device as it`s always doing what you have asked it to do, this is not always the same as what you want.
Veteran Member
Posts: 8,076
Registered: ‎03-24-2016
Kudos: 2124
Solutions: 929

Re: Advice/Help configuring multiple subnets with ER-X

For routing without NAT being used, focus on devnet, as it has 2 routers present....and you can only configure a single DG on client.  If you configure both routers with proper static routes, you can get things working.

 

The ER is not a normal home router. The GUI already has more to offer, and way more enterprise stuff is in CLI. NAT is unrelated to DHCP.  DHCP needs a pool defined, and an interface having an IP address in that pool.

 

I don't know the ASUS RT-AC68R, if you can't manually add routes to it, add NAT to ER-X , and have your 3 LANs behind the ER-X.

Highlighted
New Member
Posts: 10
Registered: ‎06-26-2017
Kudos: 1
Solutions: 1

Re: Advice/Help configuring multiple subnets with ER-X

OK, I'll try using the ER-X for all 3 subnets (and configure the ASUS router as a wifi AP on devnet) when I get a chance.

 

 

> DHCP needs a pool defined, and an interface having an IP address in that pool.

 

That's useful to know, I was wondering how to specify which DHCP pool went with which interface. I need to sit down and read the manual, but just too much going on right now.Thanks to both 16again and mlskrypka for your replies.