Emerging Member
Posts: 49
Registered: ‎04-14-2014
Kudos: 6

PPPoE client MTU RFC4638

Hi All

Ive just bought a new EdgeRouter Lite and so far im very impressed!  I work in the ISP industry on cisco and Juipers a lot so this is great for the home with all the features Man Happy

I am trying to setup the Edgerouter to be a PPPoE client over and ATM connection (ADSL).  See the below diagram for network setup.

Screen Shot 2014-06-19 at 22.54.02.png

In the UK BT Wholesale and my ISP will support RFC4638 and thus can use the PPPoEoA.  I seem to be hitting a brick wall which I think may be a bug as shown below.

Cisco router is all set to allow MTU of 1508 across ATM and Fa0 interfaces.  These 2 interfaces are bridged together:

quavey-dsl-bridge#sh run int ATM0
interface ATM0
 mtu 1508
 no ip address
 no ip route-cache
 atm ilmi-keepalive 20
 bridge-group 1
 pvc 0/38
  encapsulation aal5snap
 !
end

quavey-dsl-bridge#sh int ATM0
ATM0 is up, line protocol is up
  Hardware is MPC ATMSAR, address is 649e.f352.03b8 (bia 649e.f352.03b8)
  MTU 1508 bytes, sub MTU 1508, BW 1091 Kbit/sec, DLY 330 usec,
     reliability 255/255, txload 1/255, rxload 1/255

quavey-dsl-bridge#sh run int Fa0
interface FastEthernet0
 description Edgerouter eth0
 mtu 1508
 no ip address
end

quavey-dsl-bridge#sh int Fa0
FastEthernet0 is up, line protocol is up
  Hardware is Fast Ethernet, address is 649e.f352.03b8 (bia 649e.f352.03b8)
  Description: Edgerouter eth0
  MTU 1508 bytes, BW 100000 Kbit/sec, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255


quavey-dsl-bridge#sh run int vlan 1
interface Vlan1
 no ip address
 no ip route-cache
 bridge-group 1
end

quavey-dsl-bridge#sh int vlan1
Vlan1 is up, line protocol is up
  Hardware is EtherSVI, address is 649e.f352.03b8 (bia 649e.f352.03b8)
  MTU 1508 bytes, BW 100000 Kbit/sec, DLY 100 usec,
     reliability 255/255, txload 12/255, rxload 1/255

 As you can see all the necessary interfaces are at 1508 MTU.

I have configured the EdgeRouter Lite to also have an outside interface (eth0) of MTU 1508 and thus the pppoe 0 interface can have an MTU of 1500:

interfaces {
    ethernet eth0 {
        description WAN
        duplex auto
        firewall {
            in {
                name WAN-LAN
            }
            local {
                name WAN-ME
            }
        }
        mtu 1508
        pppoe 0 {
            default-route auto
            firewall {
                in {
                    name WAN-LAN
                }
                local {
                    name WAN-ME
                }
            }
            mtu 1500
            name-server auto
            password ****************
            user-id ***************
        }
        speed auto

 This reflects in a "show interfaces detail:

eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1508 qdisc noqueue state UNKNOWN
    link/ether dc:9f:db:29:9b:10 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::de9f:dbff:fe29:9b10/64 scope link
       valid_lft forever preferred_lft forever
    Description: WAN

pppoe0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 100
    link/ppp
    inet xxx.xxx.xxx.xxx peer xxx.xxx.xxx.xxx/32 scope global pppoe0

 So all looks good and I have a PPPoE session to my ISP up with an MTU of 1500 rather than a crappy 1492 MTU.

This is where the problem occurs.  Traffic cannot be passed and there are major symptoms of an MTU issue (pages not loading at all, loading very slow etc....)

I have pinged from an internal server (bosun) to an external server that accepts full 1500 byte packets and this is the results:

rich@bosun:~$ ping -D -s 1469 213.131.170.14
PING 213.131.170.14 (213.131.170.14) 1469(1497) bytes of data.
^C
--- 213.131.170.14 ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 6046ms

rich@bosun:~$ ping -D -s 1468 213.131.170.14
PING 213.131.170.14 (213.131.170.14) 1468(1496) bytes of data.
[1403215411.464478] 1476 bytes from 213.131.170.14: icmp_req=1 ttl=57 time=30.9 ms
[1403215412.464705] 1476 bytes from 213.131.170.14: icmp_req=2 ttl=57 time=29.9 ms
[1403215413.466167] 1476 bytes from 213.131.170.14: icmp_req=3 ttl=57 time=30.1 ms
[1403215414.468607] 1476 bytes from 213.131.170.14: icmp_req=4 ttl=57 time=31.1 ms
[1403215415.469053] 1476 bytes from 213.131.170.14: icmp_req=5 ttl=57 time=30.1 ms
^C
--- 213.131.170.14 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 29.903/30.447/31.113/0.489 ms

 So it seems from the above that the max MTU I can pass is 1496 and not the 1500 as established.....  WIERD Man Frustrated

To get round this issue I have had to use TCP MSS clamping, but I want to remove this and use a full 1500 byte packet.

I know that the Cisco 887 is setup correctly as I have replaced the Edge Router Lite with another cisco performing the PPPoE client duties and this works as expected and can pass full 1500 bytes.  So unfortuantly points to an issue with the Edge Router Lite.

Any ideas or suggestions on how to get this working would be greatfully recieved, or if its a bug, perhaps it can be filed somewhere?

Im happy to provide any further details as needed, so please feel free to ask if you need some further info.

Thanks in advance.

Rich

 

Highlighted
Veteran Member
Posts: 5,460
Registered: ‎03-12-2011
Kudos: 2749
Solutions: 129

Re: PPPoE client MTU RFC4638

[ Edited ]

Odd, normally without RFC4638 you would hit the MTU limit at 1492, so it's working partially...

Are you sure it's not an issue somewhere else in the network? 1496 is 4 bytes off 1500, the same size as a VLAN tag (and I believe some types of MPLS consume 4 bytes as well). Strikes me as suspicious. Perhaps try bumping the MTU on all of the the Cisco 887 interfaces by 4 bytes to see if that makes a difference.

In your pings, have you verified that the full packet size is being sent and received in tact when you were testing with the second cisco as the PPPoE client? I've seen some devices do funky things before that only showed up when looking at the packets coming back in via tcpdump. It would also be interesting to ping to and from a remote host that has known working 1500 byte MTUs to see if 1500 byte packets can flow in one direction but not the other which could indicate a MPLS provider network with penultimate hop popping enabled and insufficient MTU sizes causing an asymetric MTU. I had this exact issue with a provider here in Australia (in this case I knew my EdgeRouter worked because I had the same setup at my old house, moved house and suddenly the issue appeared) which they eventually tracked down to a misconfiguration on their end.

tcpdump is your friend (just because ping said it received a packet of a certain length, it may have well been truncated silently with no error thrown depending on the implementation!), as is netalyzr.