Reply
New Member
Posts: 5
Registered: ‎08-11-2017
Kudos: 4

Re: Comcast IPv6 issues when hwnat enabled on ER-X

I've only seen speculation that the root cause is hardware, no evidence of it however. 

 

 

Member
Posts: 240
Registered: ‎12-22-2015
Kudos: 46
Solutions: 3

Re: Comcast IPv6 issues when hwnat enabled on ER-X

There is some logic behind it
when you turn on hwnat, it skips the kernel when doing nat, doing hardware...

Member
Posts: 240
Registered: ‎12-22-2015
Kudos: 46
Solutions: 3

Re: Comcast IPv6 issues when hwnat enabled on ER-X

@UBNT-sandisn This is now a year old
Can we get an update on it?

Ubiquiti Employee
Posts: 486
Registered: ‎01-06-2017
Kudos: 181
Solutions: 20

Re: Comcast IPv6 issues when hwnat enabled on ER-X

 

We haven't been able to reproduce the issue in our lab and the root cause is currently unknown.

New Member
Posts: 11
Registered: ‎11-20-2016
Kudos: 5

Re: Comcast IPv6 issues when hwnat enabled on ER-X

I think there are many of us that can reproduce this on demand that would be happy to allow you access to see the issue.   I don't have out of band access so you would be interrupted when rebooting to apply the configuration.   My LTE service is not good enough to provide that as an out of band connection.

Regular Member
Posts: 342
Registered: ‎04-26-2016
Kudos: 85
Solutions: 19

Re: Comcast IPv6 issues when hwnat enabled on ER-X

@UBNT-sandisn

 

You can have access to my home ERX. It's behind Spectrum service. Same problem exists across two different modems I've had. PM me and I can set you up with locked down SSH.

Ubiquiti Employee
Posts: 486
Registered: ‎01-06-2017
Kudos: 181
Solutions: 20

Re: Comcast IPv6 issues when hwnat enabled on ER-X

@chewychewbacca

 

Can you set up a switch with port mirroring on the WAN side of ER and grant access to that as well?

Regular Member
Posts: 342
Registered: ‎04-26-2016
Kudos: 85
Solutions: 19

Re: Comcast IPv6 issues when hwnat enabled on ER-X

Can't do the switch, at least not today as I'm not at home. Just wondering why that's needed... the only thing connected to my modem is the ERX so you can do a capture from the WAN and see everything I think you would with a port mirror.

New Member
Posts: 31
Registered: ‎05-28-2017
Kudos: 12

Re: Comcast IPv6 issues when hwnat enabled on ER-X

I'm guessing that @UBNT-sandisn wants to be able to see if anything is getting dropped between the wire and the network stack on the ER-X (like with the whole hop limit thing), or if something is getting dropped further upstream (before it reaches the wire).

 

New Member
Posts: 16
Registered: ‎06-17-2016
Kudos: 7

Re: Comcast IPv6 issues when hwnat enabled on ER-X

[ Edited ]

Hardware port mirroring or SPAN was never officially supported. (https://community.ubnt.com/t5/EdgeMAX/Mirror-SPAN-on-Edge-Router-X/m-p/1974640/highlight/true#M16592...)

 

However it can be done in software so it would be a good reference against the offloaded side. 

 

@chewychewbacca What's your MTU coming up as for IPv6 locally? Also can you by chance find out if the provider is pushing an MTU value on IPv6 RA?

New Member
Posts: 10
Registered: ‎12-29-2016
Kudos: 7

Re: Comcast IPv6 issues when hwnat enabled on ER-X

[ Edited ]

Joyn wrote:
3. Can you post the MAC address of your IPv4 default gateway?

Not sure if it will help, but the MAC address of my gateway is 00:01:5c:6a:c8:46

 


UBNT-sandisn wrote:

@chewychewbacca

 

Can you set up a switch with port mirroring on the WAN side of ER and grant access to that as well?


Now that you mention port mirroring, I did a little search on it (I've never heard of or dealt with it before).  So from what I see, I should be able to set up OpenWRT to do it?

Actually, now that I've looked at it, it appears I don't even have to do the port mirroring.  I can just plug the modem and router into the switch on the device running OpenWRT, and then I can run tcpdump (or whatever other software needed) on that interface and see everything.

New Member
Posts: 10
Registered: ‎12-29-2016
Kudos: 7

Re: Comcast IPv6 issues when hwnat enabled on ER-X

@UBNT-sandisn

 

Attached are two pcaps, one with hwnat on and one with it off, captured after rebooting the router and modem.  The hwnat off one stops once the ER-X has an IPv6 address.  I let the hwnat on one run for a bit longer, but it seemed to be in a loop and the ER-X never got an IPv6 address.

Attachment
New Member
Posts: 31
Registered: ‎05-28-2017
Kudos: 12

Re: Comcast IPv6 issues when hwnat enabled on ER-X

Interesting.

 

One question - in the hwnat off case, I see the DHCPv6 solicit, advertise, and request packets (21, 24, and 25), but not a reply. Just lost in the packet capture maybe?

 

And one observation - in the hwnat off case, the single request packet (25) is 205 bytes. ALL of the request packets in the hwnat on case (e.g., 49, 63, 70, ...) are only 156 bytes. They all seem to get replies, but I'm not enough of a guru to pick the packets apart.

 

 

New Member
Posts: 5
Registered: ‎08-11-2017
Kudos: 4

Re: Comcast IPv6 issues when hwnat enabled on ER-X

Interesting observation.  Perhaps the CMTS engineers in the forum can weigh in on this.  In the one Request packet in hwnat-off, the Server Identifier DUID type is link layer plus time, and a (incorrect) DUID time stamp is included.  In hwnat-on, none of the Request packets contain the a timestamp in the Server Identiifer.  

 

Perhaps with hwnat on, the Requests are not including a time stamp in Server Identifier and the result is that the upstream servers are ignoring them?  RFC 3315 states that DUID plsu time stamp is to prevent client identifier collision.  Maybe some ISPs ignore requests that do not use DUID+time?

 

Maybe this is a rabbit hole and not relevant to the problem.  My understanding of how DHCPv6 should work is very cliff notes.

New Member
Posts: 10
Registered: ‎12-29-2016
Kudos: 7

Re: Comcast IPv6 issues when hwnat enabled on ER-X


kf-kenobi wrote:

In the one Request packet in hwnat-off, the Server Identifier DUID type is link layer plus time, and a (incorrect) DUID time stamp is included.


Hmm... that's interesting.  I have both devices set to use ntp and their times are at the moment are both correct. Only difference is the ER-X is set to UTC and the OpenWRT is set to PST.  But the DUID time stamp isn't even for the right day, so it can't just be the timezone difference.

New Member
Posts: 5
Registered: ‎08-11-2017
Kudos: 4

Re: Comcast IPv6 issues when hwnat enabled on ER-X

[ Edited ]

Actually, the timestamp being wrongisn't that strange.  The timestamp is just there to ensure DUID uniquness, it doesn't actually have to be accurate.  For instance, in TLS, browsers are supposed to include a timestamp as part of the handshake sequence, but most modern browsers don't use real time stamps (e.g. Chrome).  They just generate a unqiue (random) value here.

 

So it's possible the same thing is going on here.  However I still wonder if the lack of the timestamp when one is expected by the upstream server is causing the DHCPv6 request to be ignored.  Perhpas @UBNT-sandisn or another UBNT dev can comment on this.

New Member
Posts: 16
Registered: ‎06-17-2016
Kudos: 7

Re: Comcast IPv6 issues when hwnat enabled on ER-X

 

Few questions to ubnt.

 

Is there an official way to disable SMT for debugging purposes?

 

Can we check the debug the hardware backed tables?

 

What version SDK is this hw nat module from or forked from?

New Member
Posts: 6
Registered: ‎01-30-2018
Kudos: 4

Re: Comcast IPv6 issues when hwnat enabled on ER-X

I'm having this issue. I don't know yet whether disabling hwnat fixes it, as I just made that change today, but assume it will.

Location: Bethlehem, GA
Modem SB6141
ER X Release version: v1.9.7+hotfix.4.5024279.171006.0255

IPv6 initially works fine with hwnat enabled, but after some time I lose my WAN IPv6 address and all delegated addresses. The only way I've been able to get IPv6 working after this happens is to reboot, which so far has worked every time, giving me working IPv6 for the next 1-3 days. I've been using IPv6 with /60 PD for years without issue until I installed this router.

New Member
Posts: 6
Registered: ‎01-30-2018
Kudos: 4

Re: Comcast IPv6 issues when hwnat enabled on ER-X

I've had no PD/IPv6 issues after about a week with hwnat off.

 

This thread will be a year old in 3 weeks. It doesn't seems like this is getting fixed if they haven't figured it out by now.

New Member
Posts: 10
Registered: ‎12-29-2016
Kudos: 7

Re: Comcast IPv6 issues when hwnat enabled on ER-X

I was really hoping they'd follow up with something after I posted the packet capture. 

Reply