Reply
New Member
Posts: 33
Registered: ‎11-03-2018
Accepted Solution

PIA questions with ERLite

Hello, 

I have a few questions that I am hoping the community might be able to help me out with, or at least provide some advice. I am wanting to have PIA for my home network so that all computers and phones will be masked from the ISP. I understand that this could easily be done with my ERLite, but I’m thinking there must be another way.. I built an overkill home server with plenty of resources recently that’s running Ubuntu 16.04 LTS for file sharing and NVR purposes, could this also be used for OpenVPN with PIA. Essentially I am wondering if I could route all traffic from my ERLite to my server to handle the VPN traffic, in hopes that I get some better throughput than with the ERLite by itself. I’d love to hear some suggestions if my idea is completely off base, or is possible.

 

Thanks


Accepted Solutions
Established Member
Posts: 1,615
Registered: ‎05-03-2016
Kudos: 554
Solutions: 155

Re: PIA questions with ERLite

@kwr41230

 

As for seeing how many instances of openvpn you have running, do:

 

ps aux | grep -v grep | grep openvpn

It looks like you added the routes for 2 an 10 as well as 0 (for 192.168.X.0/24) to the server. Good.

 

No, you do not want any push routes. That is generally for the case where you are running an openvpn server on the erl for granting remote access and wanting to send the local routes to the remote client.

 

Take a look at Enable IP Forwarding on Ubuntu 13.04. It tells you how to check and setup ip forwarding with permanence on Ubuntu.

 

In your us-texas ovpn, you can change dev tun to dev tun0 so that you get a consistent interface name. Also, I add user nobody and group nobody to the ovpn. This reduces the privs of the running openvpn once it is up and running. Check to make sure nobody is defined as both a user and a group on your server by doing grep nobody /etc/passwd and grep nobody /etc/group.

 

One other thing I thought of is that I add a masquerade rule to the vpn tun interface . I am not entirely clear if this is necessary as the vpn service should handle this. But it does not hurt and I feel it is cleaner. And I add accept rules for established/related for the tun device on both FORWARD and INPUT chains followed by drop all rules. Below is a copy of the saved iptables rules I use and note that they are set for tun0.

 

Spoiler
# Generated by iptables-save v1.6.0 on Mon Sep 12 17:32:50 2016
*filter
:INPUT ACCEPT [417:105209]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [263:57107]
-A INPUT -i tun0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i tun0 -j DROP
-A FORWARD -i tun0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i tun0 -j DROP
COMMIT
# Completed on Mon Sep 12 17:32:50 2016
# Generated by iptables-save v1.6.0 on Mon Sep 12 17:32:50 2016
*nat
:PREROUTING ACCEPT [88:11967]
:INPUT ACCEPT [88:11967]
:OUTPUT ACCEPT [525:50669]
:POSTROUTING ACCEPT [525:50669]
-A POSTROUTING -o tun0 -j MASQUERADE
COMMIT
# Completed on Mon Sep 12 17:32:50 2016

If after all of this, the vpn is woring but not other hosts, specify which subnet the failing hosts are on. Try traceroutes to 8.8.8.8 from the failing hosts to see where things go awry. Also can try tcpdump.

View solution in original post


All Replies
Established Member
Posts: 1,615
Registered: ‎05-03-2016
Kudos: 554
Solutions: 155

Re: PIA questions with ERLite

@kwr41230 Sure, you can do this.

 

The basic idea is to set up the OpenVPN client on Ubuntu using the one of the client files from PIA adjusted for AUTH. Do NOT add route-nopull to the ovpn file as you want (almost) all traffic to go via PIA. You also need to turn on ip forwarding so that the Ubuntu can act as a router. I do this on Arch linux. Don't know how to do it on Ubuntu but should be easy to google. Make sure to do it in a way that persists. On Arch you can set this in /etc/sysctl.d/ipforwarding.conf with a line that contains "net.ipv4.ip_forward=1"

 

To the Ubuntu, you need to add static routes for all of your local subnets other than the one that Ubuntu is on and point them to the ip addr of the ERL on the same subnet as Ubuntu. This handles local traffic.

 

On the ERL, you need two static routes for 0.0.0.0/1 and 128.0.0.0/1 which point to the ip address of Ubuntu as this will override the static route 0.0.0.0/0 but not override the more specific static routes for the local subnets.

 

There is one final problem. You need routing on the ERL that will send the packets to PIA for the OpenVPN. Say you want to use foo.privateinternetaccess.com as the PIA server. You need to route all packets to any of its addresses out your WAN port. You cannot do this with a standard static route. So you need Policy Based Routing (PBR). Then you can define an address group that has all of the addresses for the PIA server and use PBR to route them out WAN. Since these addresses might change from time to time, rather than define the addresses in the group via the config, just define an empty group in the config and them populate that group with a script that runs say once a night in the wee hours that uses "host -t A foo.privateinternetaccess.com" and /sbin/ipset to fill the group. Be sure to flush the group before you repopulate it.

 

First thing I would do is set up the PBR routing to PIA. Next set ip forwarding on Ubuntu. Then get PIA running on Ubuntu. Set up Ubuntu routes. Finally, set up ERL routes.

 

Note that if your PIA connection on Ubuntu goes down, you will lose internet access until you repair that.

 

I think that should about do it. Make sure you understand all the various parts of the above before you try to do anything.

New Member
Posts: 33
Registered: ‎11-03-2018

Re: PIA questions with ERLite

@karog

Thank you for the reply! I was hoping that this configuration was going to be possible, I just didn’t know how it would work. It sounds like I have my homework to do before I get started, but should be a great learning experience for a weekend project. I’ll post some configurations as I go if I get stuck, and I’m sure there are more people in the community that would benefit from learning this. 

 

I have a basic understanding of most of the steps you listed as I was playing around with OpenVPN for the past couple weeks on a VM, but still confused on a few topics. For starters, I am not sure what you mean by adjusting the PIA client files for AUTH. Are you saying the first line of the .opvn file should be AUTH instead of client or server? Or just referring to the auth-user-pass line? I can setup the IP forward without a problem, but where would I set the Config for setting static routes that point to my ERL ip? I also plan on creating a new subnet to group all the devices that I want to pass through the VPN, so that should make things a bit easier to define VPN traffic. Other than that, I can’t wait to get started. Thanks for the help!

Established Member
Posts: 1,615
Registered: ‎05-03-2016
Kudos: 554
Solutions: 155

Re: PIA questions with ERLite

[ Edited ]

@kwr41230  By AUTH, I meant the path for auth-user-pass as well as paths to the ca and crl in the ovpn ie all the AUTH related stuff which will reside locally on the PIA client.

 

I assume you can set up routes on Ubuntu with "ip route whatever" but don't know how you make that happen on boot in Ubuntu.

 

The scheme I described was for sending everything from all local machines to ip addrs outside the local network via PIA. If you want it just for a particular subnet, that would be a bit different. You don't really have to organize subnets to make some subset of machines use PIA. You just have to get the routing right and probably need PBR for some of it. And in this case, you may after all want to add route-nopull to the PIA client ovpn so you have complete control of routing.

New Member
Posts: 33
Registered: ‎11-03-2018

Re: PIA questions with ERLite

[ Edited ]

@karog

So I have been trying to do as much research as I can to make this work, but I think I hit a point where I am confused. So far Ive got PIA up and running on my ubuntu server, and on the ERL Ive started to try and define my firewall modify policies. The part I am stuck at is setting up the interface for vtun0. Do I need to transfer the client, ca, and crt files over to the ERLite as well so that vtun0 has a config to run off of? Without a vtun0 interface, I have not been able to do much else with setting static protocols or anything, but Ill attach a copy of my ERL config so you can take a look at it, but its a probably a bit of a mess right now.

 

Spoiler
firewall {
    all-ping enable
    broadcast-ping disable
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians enable
    modify OPENVPN_ROUTE {
        rule 5 {
            action modify
            modify {
                table main
            }
            source {
                address 192.168.1.0/24
            }
        }
        rule 10 {
            action modify
            description PIA
            modify {
                table 1
            }
            source {
                address 192.168.1.0/24
            }
        }
    }
    name WAN_IN {
        default-action drop
        description "WAN to internal"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
        rule 21 {
            action accept
            description PIA
            destination {
                port 1198
            }
            log disable
            protocol udp
        }
    }
    name WAN_LOCAL {
        default-action drop
        description "WAN to router"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
        rule 50 {
            action accept
            description PIA
            destination {
                port 1198
            }
            log enable
            protocol udp
        }
    }
    receive-redirects disable
    send-redirects enable
    source-validation disable
    syn-cookies enable
}
interfaces {
    ethernet eth0 {
        address dhcp
        description Internet
        duplex auto
        firewall {
            in {
                name WAN_IN
            }
            local {
                name WAN_LOCAL
            }
        }
        speed auto
    }
    ethernet eth1 {
        address 192.168.1.1/24
        description Local
        duplex auto
        firewall {
            in {
                modify OPENVPN_ROUTE
            }
        }
        speed auto
        vif 10 {
            address 192.168.10.1/24
            description "Guest VLAN"
        }
    }
    ethernet eth2 {
        address 192.168.2.1/24
        description "Local 2"
        duplex auto
        speed auto
    }
    loopback lo {
    }
}
service {
    dhcp-server {
        disabled false
        hostfile-update disable
        shared-network-name Guest_VLAN {
            authoritative disable
            subnet 192.168.10.0/24 {
                default-router 192.168.10.1
                dns-server 192.168.10.1
                dns-server 1.1.1.1
                lease 14000
                start 192.168.10.2 {
                    stop 192.168.10.154
                }
            }
        }
        shared-network-name LAN1 {
            authoritative enable
            subnet 192.168.1.0/24 {
                default-router 192.168.1.1
                dns-server 192.168.1.121
                dns-server 1.1.1.1
                lease 86400
                start 192.168.1.100 {
                    stop 192.168.1.254
                }
                static-mapping home-server {
                    ip-address 192.168.1.111
                    mac-address 70:85:c2:86:06:c0
                }
            }
        }
        shared-network-name LAN2 {
            authoritative enable
            subnet 192.168.2.0/24 {
                default-router 192.168.2.1
                dns-server 192.168.2.1
                dns-server 1.1.1.1
                lease 86400
                start 192.168.2.100 {
                    stop 192.168.2.254
                }
            }
        }
        static-arp disable
        use-dnsmasq disable
    }
    dns {
        forwarding {
            cache-size 150
            listen-on eth1
            listen-on eth2
            listen-on vtun0
        }
    }
    gui {
        http-port 80
        https-port 443
        older-ciphers enable
    }
    nat {
        rule 5000 {
            description WAN
            log disable
            outbound-interface eth0
            protocol all
            source {
            }
            type masquerade
        }
    }
    ssh {
        port 22
        protocol-version v2
    }
    unms {
        disable
    }
}
system {
    host-name ubnt
    login {
        user kwr41230 {
            authentication {
                encrypted-password ****************
            }
            level admin
        }
    }
    ntp {
        server 0.ubnt.pool.ntp.org {
        }
        server 1.ubnt.pool.ntp.org {
        }
        server 2.ubnt.pool.ntp.org {
        }
        server 3.ubnt.pool.ntp.org {
        }
    }
    offload {
        hwnat disable
        ipsec enable
        ipv4 {
            forwarding enable
            gre enable
            pppoe enable
            vlan enable
        }
        ipv6 {
            forwarding enable
            pppoe enable
        }
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone America/Chicago
    traffic-analysis {
        dpi disable
        export disable

 

Thanks

Established Member
Posts: 1,615
Registered: ‎05-03-2016
Kudos: 554
Solutions: 155

Re: PIA questions with ERLite

[ Edited ]

@kwr41230

 

First, there is to be NO vtun0 on the ERL. So there is nothing to configure in that respect. The OpenVPN runs strictly between your Ubuntu machine and PIA.

 

The PBR I mentioned before is for the scenario where you want everything to go thru PIA which I understood you to specify in your first post. In your second post, you mentioned creating a subnet for a subset of hosts to use PIA and I followed that with a post saying that would change things a bit. So first you need to figure out which scenario you want.

 

In the first case, I specified that you would create default routing rules to send everything to Ubuntu from the ERL but would then need an execption for the underlying packet stream that is the VPN itselft; namely, the packets coming from the Ubuntu headed for PIA. This is where PBR would come in so that such packets would not get redirected back to Ubuntu by the default routing rules. You could filter in the modify rule on port 1198 or whatever port on which you are sending to PIA (they vary). You would do this in your modify rule 10 where you should remove the source clause. And you do not need modify rule 5. You still need to define table 1 in protocols static to do a route next-hop to your WAN gateway address - which will need updating should it ever change. You should not use interface-route in this case.

 

I will also mention since you are just beginning, that it is a bad idea to use the very common subnets like 192.168.XXX.0/24 for XXX < 5. If you ever want to set up remote access to your home net with a vpn, you are very likely to find yourself on such a subnet at the remote location (it is common) and then things would not work. Choose from RFC 1918 subnets where XXX is some more obscure number like 20 <= XXX <= 250 but better something in 10.XXX.YYY.0/24 or 172.16-31.XXX.0/24.

 

Finally, when you post something long like a config, put it and the code tag inside a spoiler tag. The spolier tag starts out collapsed and can be expanded if desired by the viewer. This cuts down on long scrolling. You can edit your post to add that.

New Member
Posts: 33
Registered: ‎11-03-2018

Re: PIA questions with ERLite

[ Edited ]

@karogSo decided to go with the first scenario that you suggested where everything will go through the VPN. I havent changed the IP on my server yet, but that is on my to-do list, I am just trying to figure out how to make this all work first. So far I have enable IP forwarding on the ubuntu server and added a route that points to the IP of my ERLite. I have removed rule 5 from firewall modify, removed the source address, and added destination address port 1198. I have also added some protocol static IP's with table 1 next-hop to my WAN ip. Only problem is, now I broke my internet.. lol.. I am not sure exactly which command stopped everything from working, but I'll attach some spoilers for you to take a look at, I may have messed up on either the static protocols, or the IP route from my server. Thanks for the help!!

 

ERLite config

 

Spoiler
firewall {
    all-ping enable
    broadcast-ping disable
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians enable
    modify OPENVPN_ROUTE {
        rule 10 {
            action modify
            description PIA
            destination {
                port 1198
            }
            modify {
                table 1
            }
            protocol udp
        }
    }
    name WAN_IN {
        default-action drop
        description "WAN to internal"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
        rule 21 {
            action accept
            description PIA
            destination {
                port 1198
            }
            log disable
            protocol udp
        }
    }
    name WAN_LOCAL {
        default-action drop
        description "WAN to router"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
        rule 50 {
            action accept
            description PIA
            destination {
                port 1198
            }
            log enable
            protocol udp
        }
    }
    receive-redirects disable
    send-redirects enable
    source-validation disable
    syn-cookies enable
}
interfaces {
    ethernet eth0 {
        address dhcp
        description Internet
        duplex auto
        firewall {
            in {
                name WAN_IN
            }
            local {
                name WAN_LOCAL
            }
        }
        speed auto
    }
    ethernet eth1 {
        address 192.168.1.1/24
        description Local
        duplex auto
        firewall {
            in {
                modify OPENVPN_ROUTE
            }
        }
        speed auto
        vif 10 {
            address 192.168.10.1/24
            description "Guest VLAN"
        }
    }
    ethernet eth2 {
        address 192.168.2.1/24
        description "Local 2"
        duplex auto
        speed auto
    }
    loopback lo {
    }
}
protocols {
    static {
        route 0.0.0.0/0 {
            next-hop 192.168.1.111 {
            }
        }
        route 0.0.0.0/1 {
            next-hop 192.168.1.111 {
            }
        }
        route 128.0.0.0/1 {
            next-hop 192.168.1.111 {
            }
        }
        table 1 {
            route 0.0.0.0/0 {
                next-hop 50.26.116.1 {
                }
            }
        }
    }
}
service {
    dhcp-server {
        disabled false
        hostfile-update disable
        shared-network-name Guest_VLAN {
            authoritative disable
            subnet 192.168.10.0/24 {
                default-router 192.168.10.1
                dns-server 192.168.10.1
                dns-server 1.1.1.1
                lease 14000
                start 192.168.10.2 {
                    stop 192.168.10.154
                }
            }
        }
        shared-network-name LAN1 {
            authoritative enable
            subnet 192.168.1.0/24 {
                default-router 192.168.1.1
                dns-server 192.168.1.121
                dns-server 1.1.1.1
                lease 86400
                start 192.168.1.100 {
                    stop 192.168.1.254
                }
                static-mapping home-server {
                    ip-address 192.168.1.111
                    mac-address 70:85:c2:86:06:c0
                }
            }
        }
        shared-network-name LAN2 {
            authoritative enable
            subnet 192.168.2.0/24 {
                default-router 192.168.2.1
                dns-server 192.168.2.1
                dns-server 1.1.1.1
                lease 86400
                start 192.168.2.100 {
                    stop 192.168.2.254
                }
            }
        }
        static-arp disable
        use-dnsmasq disable
    }
    dns {
        forwarding {
            cache-size 150
            listen-on eth1
            listen-on eth2
            listen-on vtun0
        }
    }
    gui {
        http-port 80
        https-port 443
        older-ciphers enable
    }
    nat {
        rule 5000 {
            description WAN
            log disable
            outbound-interface eth0
            protocol all
            source {
            }
            type masquerade

network interfaces config from ubuntu

Spoiler
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto enp0s31f6
iface enp0s31f6 inet static
        address 192.168.1.111
        netmask 255.255.255.0
        gateway 192.168.1.1
dns-nameservers 1.1.1.1 8.8.8.8
        up route add -net 192.168.0.0 netmask 255.255.255.0 gw 192.168.1.1

show ip route from ubuntu

Spoiler
kevin@home-server:~$ ip route show
0.0.0.0/1 via 10.41.10.5 dev tun0
default via 192.168.1.1 dev enp0s31f6 onlink
10.41.10.1 via 10.41.10.5 dev tun0
10.41.10.5 dev tun0  proto kernel  scope link  src 10.41.10.6
128.0.0.0/1 via 10.41.10.5 dev tun0
162.216.46.116 via 192.168.1.1 dev enp0s31f6
192.168.1.0/24 dev enp0s31f6  proto kernel  scope link  src 192.168.1.111

Any ideas what I did wrong?

New Member
Posts: 33
Registered: ‎11-03-2018

Re: PIA questions with ERLite

@karog

I tried to reply earlier, but for some reason my post was removed and listed as spam. But here I go again..

So I decided to go with your first scenario you suggested in which everything will be routed through the VPN. I have not chnaged the IP of my server yet, but it is on my to-do list once I figure all of this out and get everything working. So far I have removed rule 5 on the ERLite, removed source address from rule 10, and added destination port 1198. I have setup some static protocols and tried to follow along with your suggestions of each address for them. I have also enabled IPV4 forwarding on my server, with an add route to the IP of the ERLite gateway. Only problem is, now my internet is dead.. lol, lucky for a wifi hotspot! I am not sure which command shut everything down, but I am thinking it has to be something on either the ip route address on the server, or the static protocols on the ERLite. Hopefully you can take a look and see what you think. Thank you!

 

ERlite config

 

Spoiler
firewall {
    all-ping enable
    broadcast-ping disable
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians enable
    modify OPENVPN_ROUTE {
        rule 10 {
            action modify
            description PIA
            destination {
                port 1198
            }
            modify {
                table 1
            }
            protocol udp
        }
    }
    name WAN_IN {
        default-action drop
        description "WAN to internal"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
        rule 21 {
            action accept
            description PIA
            destination {
                port 1198
            }
            log disable
            protocol udp
        }
    }
    name WAN_LOCAL {
        default-action drop
        description "WAN to router"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
        rule 50 {
            action accept
            description PIA
            destination {
                port 1198
            }
            log enable
            protocol udp
        }
    }
    receive-redirects disable
    send-redirects enable
    source-validation disable
    syn-cookies enable
}
interfaces {
    ethernet eth0 {
        address dhcp
        description Internet
        duplex auto
        firewall {
            in {
                name WAN_IN
            }
            local {
                name WAN_LOCAL
            }
        }
        speed auto
    }
    ethernet eth1 {
        address 192.168.1.1/24
        description Local
        duplex auto
        firewall {
            in {
                modify OPENVPN_ROUTE
            }
        }
        speed auto
        vif 10 {
            address 192.168.10.1/24
            description "Guest VLAN"
        }
    }
    ethernet eth2 {
        address 192.168.2.1/24
        description "Local 2"
        duplex auto
        speed auto
    }
    loopback lo {
    }
}
protocols {
    static {
        route 0.0.0.0/0 {
            next-hop 192.168.1.111 {
            }
        }
        route 0.0.0.0/1 {
            next-hop 192.168.1.111 {
            }
        }
        route 128.0.0.0/1 {
            next-hop 192.168.1.111 {
            }
        }
        table 1 {
            route 0.0.0.0/0 {
                next-hop 50.26.116.1 {
                }
            }
        }
    }
}
service {
    dhcp-server {
        disabled false
        hostfile-update disable
        shared-network-name Guest_VLAN {
            authoritative disable
            subnet 192.168.10.0/24 {
                default-router 192.168.10.1
                dns-server 192.168.10.1
                dns-server 1.1.1.1
                lease 14000
                start 192.168.10.2 {
                    stop 192.168.10.154
                }
            }
        }
        shared-network-name LAN1 {
            authoritative enable
            subnet 192.168.1.0/24 {
                default-router 192.168.1.1
                dns-server 192.168.1.121
                dns-server 1.1.1.1
                lease 86400
                start 192.168.1.100 {
                    stop 192.168.1.254
                }
                static-mapping home-server {
                    ip-address 192.168.1.111
                    mac-address 70:85:c2:86:06:c0
                }
            }
        }
        shared-network-name LAN2 {
            authoritative enable
            subnet 192.168.2.0/24 {
                default-router 192.168.2.1
                dns-server 192.168.2.1
                dns-server 1.1.1.1
                lease 86400
                start 192.168.2.100 {
                    stop 192.168.2.254
                }
            }
        }
        static-arp disable
        use-dnsmasq disable
    }
    dns {
        forwarding {
            cache-size 150
            listen-on eth1
            listen-on eth2
            listen-on vtun0
        }
    }
    gui {
        http-port 80
        https-port 443
        older-ciphers enable
    }
    nat {
        rule 5000 {
            description WAN
            log disable
            outbound-interface eth0
            protocol all
            source {
            }
            type masquerade

network interfaces on server

Spoiler
 This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto enp0s31f6
iface enp0s31f6 inet static
        address 192.168.1.111
        netmask 255.255.255.0
        gateway 192.168.1.1
dns-nameservers 1.1.1.1 8.8.8.8
        up route add -net 192.168.0.0 netmask 255.255.255.0 gw 192.168.1.1

show ip route from server

Spoiler
kevin@home-server:~$ ip route show
0.0.0.0/1 via 10.41.10.5 dev tun0
default via 192.168.1.1 dev enp0s31f6 onlink
10.41.10.1 via 10.41.10.5 dev tun0
10.41.10.5 dev tun0  proto kernel  scope link  src 10.41.10.6
128.0.0.0/1 via 10.41.10.5 dev tun0
162.216.46.116 via 192.168.1.1 dev enp0s31f6
192.168.1.0/24 dev enp0s31f6  proto kernel  scope link  src 192.168.1.111

Anything stand out that looks completely wrong?

Established Member
Posts: 1,615
Registered: ‎05-03-2016
Kudos: 554
Solutions: 155

Re: PIA questions with ERLite

@kwr41230

 

Is it the case that even though the internet is not accessbile, you are still able to access devices on your local net?

 

The Ubuntu part looks mostly ok to me. You have the 0.0.0.0/1 and 128.0.0.0/1 going thru tun0 which is the vpn. 162.216.46.116 is an exception and I am guessing is the ip of the foo.privateinternetaccess.com you are using. Which PIA server are you using? A reverse DNS did not help but if I knew which server, I could get its ip addresses. What is missing are routes for 192.168.2..0/24 and 192.168.10.0/24 to go to 192.168.1.1.

 

On the ERL, you do not need the protocols static route 0.0.0.0/0 as it is overridden by the 0.0.0.0/1 and 128.0.0.0/1 anyway. I am assuming the table 1 ip 50.26.116.1 is your WAN gateway address. In your text you say enabled IPv4 forwarding with an add route to the IP of ERL gateway. That kind of sounds like you did that on the Ubuntu but I think you meant in table 1 on ERL, right?

 

I would also like to see from the ERL ip route show or better route -n which is better columnated.

 

To see why things are not working, I would start by running a ping and traceroute on Ubuntu to say 8.8.8.8, the google dns. This will tell you whether with the current overall config the vpn is still working. If not, then need to figure out why probably using tcpdump on the ERL If that is working, then you need to debug routes. Which makes me ask, when you say internet is no longer working, what exactly do you mean by  that? From what machine on what subnet? Is it a connectivity problem or a DNS problem? You can decide that by ping 8.8.8.8 or try some DNS lookup of some external domain name like ibm.com.

New Member
Posts: 33
Registered: ‎11-03-2018

Re: PIA questions with ERLite

@karog

Yes I am stil able to ping all devices on my network, and access the shared drives from my home server. But, by no internet I mean that I cannot access anything outside of my LAN. The only way I am able to get online is from a mobile hot spot off of my phone. All wifi and internet connections off my router are dead. The PIA server that I am using is us-texas.privateinternetaccess.com. Even without the routes for 192.168.2.0 and 10.0 being forwarded, the 192.168.1.0 should work still right? 

So on my dashboard for eth0 it says the address is 50.26.119.9/22, but on the routing tab it had next hop to 50.26.116.1, so I thought that was my WAN. 

Hmm yes I enabled net.ipv4.ip_forward=1 on my ubuntu server, Is that wrong? Table 1 on my ERL now has route 0.0.0.0/0 next-hop 50.26.119.9. 

Heres the ip route from ERL

Spoiler
0.0.0.0/1 via 192.168.1.111 dev eth1 proto zebra
default via 50.26.116.1 dev eth0 proto zebra
50.26.116.0/22 dev eth0 proto kernel scope link src 50.26.119.9
128.0.0.0/1 via 192.168.1.111 dev eth1 proto zebra
192.168.1.0/24 dev eth1 proto kernel scope link
192.168.2.0/24 dev eth2 proto kernel scope link src 192.168.2.1
192.168.10.0/24 dev eth1.10 proto kernel scope link

I was able to ping to 8.8.8.8 earlier today from my ubuntu machine, but now that it is not getting any internet connection, I cannot ping outside of my LAN. 

Established Member
Posts: 1,615
Registered: ‎05-03-2016
Kudos: 554
Solutions: 155

Re: PIA questions with ERLite

@kwr41230

 

Curiously, when I dig us-texas I get many 162.216.46.XXX addresses but none that end in 116 which is what your ip route showed before. I did the dig at both 8.8.8.8 and 1.1.1.1 and got the same result without 116.

 

 

Yes, 192.168.1.0/24 should work without 2 and 10 routed but you probably cannot get to those subnets from your main subnet.

 

In the last ip route you posted, the line starting default via 50 is your gateway. This is a bit more obvious with route -n which has a gateway column.

 

Have you tried to restart PIA on your Ubuntu? If not, that is the first thing I would do. If it fails to start, look at the log to see what the error is.

 

Also, are you sure 1198 is the right port listed in the PIA ovpn?

New Member
Posts: 33
Registered: ‎11-03-2018

Re: PIA questions with ERLite

@karog

So last night I wound up having to disable the static IP protocols where 0.0.0.0/1 and 128.0.0.0/1 where set for next-hop to 192.168.1.111, and disabled IP forwarding on my ubuntu server (not sure if this was really needed, but did it anyway), just to get internet back on in the house (family was getting a bit fed up with not having wi-fi all day! lol). Anyways, I wound up getting back connected with the VPN this morning on the ubuntu server and ran route -n, here are the results..

Spoiler
kevin@home-server:~$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.94.10.9      128.0.0.0       UG    0      0        0 tun0
0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 enp0s31f            6
10.53.10.1      10.53.10.5      255.255.255.255 UGH   0      0        0 tun1
10.53.10.5      0.0.0.0         255.255.255.255 UH    0      0        0 tun1
10.94.10.1      10.94.10.9      255.255.255.255 UGH   0      0        0 tun0
10.94.10.9      0.0.0.0         255.255.255.255 UH    0      0        0 tun0
128.0.0.0       10.94.10.9      128.0.0.0       UG    0      0        0 tun0
162.216.46.115  192.168.1.1     255.255.255.255 UGH   0      0        0 enp0s31f            6
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 enp0s31f            6

Its interesting though when I run the command curl ipinfo.io/ip that the output is address 173.239.232.115. I am sure that port 1198 is correct, as I was originally trying 1194, as this is usually default for udp but I was getting TLS handshake failure errors. Once I did some research and found to use 1198, everything initialized perfectly. Another interesting thing I noticed last night before removing the statis protocols is that when I pinged 8.8.8.8 from a VM on my computer it showed me this output..

Spoiler
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data
from 192.168.1.1: icmp_seq=2 Redirect Host(New nexthop: 192.168.1.111)

So I would interpret this as meaning that the packets were succesfully being routed from my LAN devices, to the ERL, out to the ubuntu server, but was either not going back out the WAN after returning from the server, or something along those lines. I must be missing something in my PBR or something wrong in my table 1 route. What address should I be routing for "set protocols static table 1 route ?x?x? next-hop  50.26.116.1? Should this be the 162.216.46.115 address that I see from the routing table on my ubuntu server? Again, thank you for your help, not sure what I would do without it!

Established Member
Posts: 1,615
Registered: ‎05-03-2016
Kudos: 554
Solutions: 155

Re: PIA questions with ERLite

@kwr41230

 

The way I work is that for a multicomponent effort like this, I first try to thoroughly understand all of the steps and then do one step at a time and test to make sure that the things that should continue to work still work. I do not do all of the steps and then test. It is much easier to diagnose one step at a time. Then when something breaks, you know it is related to the last step.

 

Looking at your route -n, I see both tun0 and tun1. It looks like tun0 is now in use and tun1 are remnants of something past. You should clean that up.

 

For PIA, the way to find the proper port is to look in the us-texas.privateinternetaccess.ovpn config file. It varies depending on the level of encryption being used.

 

That 173.239.232.115 ip address is the public ip address upon exiting the PIA tunnel to the internet. Part of using a vpn service is to hide your actual WAN ip address.

 

The table 1 route next-hop should be to 50.26.116.1 as shown in the default line of ip route show on the ERL.

 

The ping 8.8.8.8 with redirect looks to me like you did this from the server when the vpn was down and its route were missing so that the ping went to the ERL rather than thru the vpn and then hit the routes on the ERL to send it back. Should not happen if the PIA is properly up with its routes.

 

So I would change the order of my steps from a previous post. I would start by removing the static routes on the ERL. You can leave table 1. Disable the modify rule so that it will not fire but you won't later have to reenter it. Stop OpenVPN on the server and get all the tun interfaces gone unless tun1 is in fact for a second instance of OpenVPN.

 

Make sure ip forwarding is set on the Ubuntu server. Remember that the line I gave was for Arch linux. I do not know if that is what Ubuntu also uses. You need to verify that.

 

Then bring up OpenVPN to PIA. You should then be able to traceroute to some internet host and verify that it is going thru the vpn. Once that is working you can add back the PBR by deleting the modify rule disable. Then test the vpn again. It should still be working. traceroutes from hosts other than your server should go thru the ERL at this point.

 

Next I would set the static routes on the ERL after which all hosts on 192.168.1.0/24 should go thru the vpn. Test that.

 

Finally add the static routes to the server for the 2 and 10 local subnets to go to the ERL at 192.168.1.1. Test that.

 

If test fail at any point, stop there and identify the problem.

New Member
Posts: 33
Registered: ‎11-03-2018

Re: PIA questions with ERLite

@karog

I followed through your steps testing each one along the way to make sure everything still worked. First, I deleted the tun1 from the server, as I am not sure why this was even there. After disabling table 1 action modify and removing the 2 static routes from the ERL, I verified a traceroute that packets were going throught the router and all is good. I verified a traceroute on my server and got the following.. 

Spoiler
kevin@home-server:~$ traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  10.10.10.1 (10.10.10.1)  47.346 ms  47.347 ms  47.340 ms
 2  ip-253-232-239-173.texas.us.northamericancoax.com (173.239.232.253)  47.344                        ms  47.338 ms  49.914 ms
 3  199.36.220.176 (199.36.220.176)  49.933 ms  49.927 ms 104.200.142.38 (104.20                       0.142.38)  49.861 ms
 4  104.200.142.36 (104.200.142.36)  49.891 ms eqix-da1.google.com (206.223.118.                       137)  51.024 ms  51.006 ms
 5  108.170.240.193 (108.170.240.193)  51.048 ms  51.009 ms eqix-da1.google.com                        (206.223.118.137)  50.913 ms
 6  72.14.234.103 (72.14.234.103)  50.902 ms 108.170.252.129 (108.170.252.129)                         48.763 ms 108.170.229.131 (108.170.229.131)  51.201 ms
 7  google-public-dns-a.google.com (8.8.8.8)  50.413 ms 108.170.238.113 (108.170                       .238.113)  50.923 ms google-public-dns-a.google.com (8.8.8.8)  50.530 ms

So it looks like the vpn is working on the server. I set my protocol for table 1 as..

Spoiler
protocols {
    static {
        route 0.0.0.0/1 {
            next-hop 192.168.1.111 {
                disable
            }
        }
        route 128.0.0.0/1 {
            next-hop 192.168.1.111 {
                disable
            }
        }
        table 1 {
            route 162.216.46.115/32 {
                next-hop 50.26.116.1 {

I then retested a traceroute on everything, and all was still working, packets from other hosts on my LAN were still going through the router and the server still going through the vpn. I deleted the disable for action modify table 1, and still the same, all was working. As soon as I enabled the static routes on the ERL for 0.0.0.0/1 and 128.0.0.0/1 for next-hop to 192.168.1.111 my internet went down. I was unablle to run a traceroute from my server to 8.8.8.8 or from a local host. I received a request timed out response. Am I using the wrong address route in my table 1 static protocols that points to my WAN? I know the WAN address must be correct as this is what we determined from the route -n but I still wasnt sure if the destination address of 162.216.46.115/32 is correct. Thanks!

Established Member
Posts: 1,615
Registered: ‎05-03-2016
Kudos: 554
Solutions: 155

Re: PIA questions with ERLite

[ Edited ]

@kwr41230

 

The route in table 1 should be for 0.0.0.0/0, not 162.216.46.115/32. The last post I found above with protocols had that correct. Your most recent post is wrong. Don't know why you changed that.

 

ETA: you will most likely need to restart vpn after fixing that as the existing one is broken.

New Member
Posts: 33
Registered: ‎11-03-2018

Re: PIA questions with ERLite

@karog

Yes, you are right, sorry about that I must have gotten confused on the table 1 route. Anyways, I corrected that issue and restarted openvpn but still have the same issue. As soon as I enable the static routes to 192.168.1.111 I lose all internet connections. On top of that, I checked my route -n output after restarting and now I have not only a tun0, but also a tun1 and tun2!

Spoiler
kevin@home-server:~$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.87.10.5      128.0.0.0       UG    0      0        0 tun0
0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 enp0s31f6
10.27.10.1      10.27.10.5      255.255.255.255 UGH   0      0        0 tun2
10.27.10.5      0.0.0.0         255.255.255.255 UH    0      0        0 tun2
10.56.10.1      10.56.10.5      255.255.255.255 UGH   0      0        0 tun1
10.56.10.5      0.0.0.0         255.255.255.255 UH    0      0        0 tun1
10.87.10.1      10.87.10.5      255.255.255.255 UGH   0      0        0 tun0
10.87.10.5      0.0.0.0         255.255.255.255 UH    0      0        0 tun0
128.0.0.0       10.87.10.5      128.0.0.0       UG    0      0        0 tun0
162.216.46.7    192.168.1.1     255.255.255.255 UGH   0      0        0 enp0s31f6
162.216.46.54   192.168.1.1     255.255.255.255 UGH   0      0        0 enp0s31f6
162.216.46.84   192.168.1.1     255.255.255.255 UGH   0      0        0 enp0s31f6
192.168.0.0     192.168.1.1     255.255.255.0   UG    0      0        0 enp0s31f6
192.168.1.0     192.168.1.1     255.255.255.0   UG    0      0        0 enp0s31f6
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 enp0s31f6
192.168.2.0     192.168.1.1     255.255.255.0   UG    0      0        0 enp0s31f6
192.168.10.0    192.168.1.1     255.255.255.0   UG    0      0        0 enp0s31f6

It seems that every time I delete tun1, it just keeps coming back after I either reboot the server or restart openvpn. I must be doing something wrong on my server's network/interfaces for adding static routes.

Established Member
Posts: 1,615
Registered: ‎05-03-2016
Kudos: 554
Solutions: 155

Re: PIA questions with ERLite

@kwr41230

 

Looking at the route -n, not only do you see many tuns, you also see many Destinations 16.216.46.XXX which are ip addrs for us-texas. All of this is the remnants of many attempts to start the tunnel and it not being cleaned up. Do you even know if you have multiple instances of openvpn running? They may fail to connect or work properly yet still be running.

 

Post your us-texas ovpn config file.

 

Let's not talk yet about losing internet connections. What matters at first step is whether you lose the vpn connection which you test by doing ping and traceroute to 8.8.8.8 from the server. If that goes down when you add routes on the ERL, then that has messed up the underlying packet flow for the vpn. That is what the PBR is supposed to protect for packets with port 1198. Note 1198 is for the weaker vpn encryption while port 1197 is for the stronger encryption. That would be the problem to solve.

 

Looking at the ip addrs for us-texas, I noticed that they all are subsumed by the subnet 162.216.46.0/24. Not all of their servers have such consistent ip addrs. But you can take advantage of that by using a normal static route for that subnet to the WAN gateway rather than using PBR. Of course, if they ever add additional ip addrs for that domain name not in that subnet and you happened to get such an ip addr on connection, the route would then fail. You could write a script wachdog to look out for that case and notify you.

New Member
Posts: 33
Registered: ‎11-03-2018

Re: PIA questions with ERLite

@karog

After doing another restart on the ubuntu server and restarting openvpn, the routes seemed to clear up from last night and went back to only tun0, although I have no idea why.. but here is the route -n as of right now with openvpn running PIA.

Spoiler
kevin@home-server:~$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.44.10.5      128.0.0.0       UG    0      0        0 tun0
0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 enp0s31f6
10.44.10.1      10.44.10.5      255.255.255.255 UGH   0      0        0 tun0
10.44.10.5      0.0.0.0         255.255.255.255 UH    0      0        0 tun0
128.0.0.0       10.44.10.5      128.0.0.0       UG    0      0        0 tun0
162.216.46.119  192.168.1.1     255.255.255.255 UGH   0      0        0 enp0s31f6
192.168.0.0     192.168.1.1     255.255.255.0   UG    0      0        0 enp0s31f6
192.168.1.0     192.168.1.1     255.255.255.0   UG    0      0        0 enp0s31f6
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 enp0s31f6
192.168.2.0     192.168.1.1     255.255.255.0   UG    0      0        0 enp0s31f6
192.168.10.0    192.168.1.1     255.255.255.0   UG    0      0        0 enp0s31f6

As for the question of if I have multiple instances of openvpn running, I dont think that I do (I know that sounds dumb), but let me explain. I was following this tutorial when installing openvpn on my server https://medium.com/@cubxi/connect-to-private-internet-access-pia-vpn-with-openvpn-on-ubuntu-3f8091d6... and the first time I downloaded it I couldnt get things working right so I used the command sudo apt-get purge openvpn to remove the old instance and re-install to start over. If there were any remnants left over from the first intstall than I suppose its possible. Following the tutorial, I created the following client file..

Spoiler
client

dev tun

proto udp

remote us-texas.privateinternetaccess.com 1198 udp

resolv-retry infinite

nobind

persist-key

persist-tun

cipher aes-128-cbc

auth sha1

remote-cert-tls server

tls-client

crl-verify /etc/openvpn/crl.rsa.2048.pem

ca /etc/openvpn/ca.rsa.2048.crt

auth-user-pass /etc/openvpn/creds.conf

auth-nocache

comp-lzo

verb 1

reneg-sec 0

disable-occ

I just tried to test a traceroute to 8.8.8.8 after re-enabling the routes for 0.0.0.0/1 and 128.0.0.0/1 to 192.168.1.111 on the ERL, and to my surprise the server still was connected to PIA, check it out..

Spoiler
kevin@home-server:~$ traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  10.44.10.1 (10.44.10.1)  48.143 ms  48.133 ms  48.127 ms
 2  ip-253-232-239-173.texas.us.northamericancoax.com (173.239.232.253)  48.280 ms  48.275 ms           48.270 ms
 3  104.200.142.38 (104.200.142.38)  48.054 ms  48.241 ms 199.36.220.176 (199.36.220.176)  48.         137 ms
 4  104.200.142.36 (104.200.142.36)  48.214 ms  48.209 ms  48.548 ms
 5  108.170.240.193 (108.170.240.193)  48.947 ms eqix-da1.google.com (206.223.118.137)  48.886          ms  48.919 ms
 6  108.170.231.15 (108.170.231.15)  48.915 ms 72.14.234.225 (72.14.234.225)  46.578 ms 108.17         0.240.193 (108.170.240.193)  47.552 ms
 7  google-public-dns-a.google.com (8.8.8.8)  47.462 ms 108.170.226.107 (108.170.226.107)  48.         429 ms google-public-dns-a.google.com (8.8.8.8)  47.449 ms

Meanwhile, all other devices on my LAN lost internet access. I noticed in some other forums that people have added "push routes" into their ERL config under the port forwarding section I believe, not sure if this would apply to my situation. 

 

If all the us-texas addresses have the same subnet, that sounds like it could be a win situation to make things a little bit easier!

Again, thank you for taking the time to help me along with this project, it is becoming quite a great learning experience.

Established Member
Posts: 1,615
Registered: ‎05-03-2016
Kudos: 554
Solutions: 155

Re: PIA questions with ERLite

@kwr41230

 

As for seeing how many instances of openvpn you have running, do:

 

ps aux | grep -v grep | grep openvpn

It looks like you added the routes for 2 an 10 as well as 0 (for 192.168.X.0/24) to the server. Good.

 

No, you do not want any push routes. That is generally for the case where you are running an openvpn server on the erl for granting remote access and wanting to send the local routes to the remote client.

 

Take a look at Enable IP Forwarding on Ubuntu 13.04. It tells you how to check and setup ip forwarding with permanence on Ubuntu.

 

In your us-texas ovpn, you can change dev tun to dev tun0 so that you get a consistent interface name. Also, I add user nobody and group nobody to the ovpn. This reduces the privs of the running openvpn once it is up and running. Check to make sure nobody is defined as both a user and a group on your server by doing grep nobody /etc/passwd and grep nobody /etc/group.

 

One other thing I thought of is that I add a masquerade rule to the vpn tun interface . I am not entirely clear if this is necessary as the vpn service should handle this. But it does not hurt and I feel it is cleaner. And I add accept rules for established/related for the tun device on both FORWARD and INPUT chains followed by drop all rules. Below is a copy of the saved iptables rules I use and note that they are set for tun0.

 

Spoiler
# Generated by iptables-save v1.6.0 on Mon Sep 12 17:32:50 2016
*filter
:INPUT ACCEPT [417:105209]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [263:57107]
-A INPUT -i tun0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i tun0 -j DROP
-A FORWARD -i tun0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i tun0 -j DROP
COMMIT
# Completed on Mon Sep 12 17:32:50 2016
# Generated by iptables-save v1.6.0 on Mon Sep 12 17:32:50 2016
*nat
:PREROUTING ACCEPT [88:11967]
:INPUT ACCEPT [88:11967]
:OUTPUT ACCEPT [525:50669]
:POSTROUTING ACCEPT [525:50669]
-A POSTROUTING -o tun0 -j MASQUERADE
COMMIT
# Completed on Mon Sep 12 17:32:50 2016

If after all of this, the vpn is woring but not other hosts, specify which subnet the failing hosts are on. Try traceroutes to 8.8.8.8 from the failing hosts to see where things go awry. Also can try tcpdump.

New Member
Posts: 33
Registered: ‎11-03-2018

Re: PIA questions with ERLite

@karog

I ran the command to check for multiple instances of openvpn, and here is what I got..

Spoiler
kevin@home-server:~$ ps aux | grep -v grep | grep openvpn
root      2192  0.0  0.0  41924  5800 ?        Ss   Nov28   0:09 /usr/sbin/openvpn --daemon ovpn-pia --          status /run/openvpn/pia.status 10 --cd /etc/openvpn --script-security 2 --config /etc/openvpn/pia.conf           --writepid /run/openvpn/pia.pid

I checked out the link regarding IP forwarding on my ubuntu server and this is actually what I already did to enable forwarding on boot, so thats a plus. I changed the client file to tun0 and added the user and group nobody as well. After running the command for grep nobody /etc/passwd I got an output of

Spoiler
kevin@home-server:~$ grep nobody /etc/passwd
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin

But after entering the command  grep nobody /etc/group, I did not receive an output, it just went to the next line with my server name and user name. 

 

I am looking at your IP tables and I know I have seen a tutorial on how to add these rules, but for the life of me I cannot find it right now on google. How did you get that output screen for iptables? And to add these rules, am I just to use configure on my ERL and then enter

Spoiler
-A INPUT -i tun0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i tun0 -j DROP
-A FORWARD -i tun0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i tun0 -j DROP
Reply