01-08-2018 08:28 AM
I think there are many of us that can reproduce this on demand that would be happy to allow you access to see the issue. I don't have out of band access so you would be interrupted when rebooting to apply the configuration. My LTE service is not good enough to provide that as an out of band connection.
01-08-2018 08:50 AM
You can have access to my home ERX. It's behind Spectrum service. Same problem exists across two different modems I've had. PM me and I can set you up with locked down SSH.
01-08-2018 09:03 AM
Can't do the switch, at least not today as I'm not at home. Just wondering why that's needed... the only thing connected to my modem is the ERX so you can do a capture from the WAN and see everything I think you would with a port mirror.
01-08-2018 09:29 AM
I'm guessing that @UBNT-sandisn wants to be able to see if anything is getting dropped between the wire and the network stack on the ER-X (like with the whole hop limit thing), or if something is getting dropped further upstream (before it reaches the wire).
01-08-2018 11:07 AM - edited 01-08-2018 01:26 PM
Hardware port mirroring or SPAN was never officially supported. (https://community.ubnt.com/t5/EdgeMAX/Mirror-SPAN-on-Edge-Router-X/m-p/1974640/highlight/true#M16592...)
However it can be done in software so it would be a good reference against the offloaded side.
@chewychewbacca What's your MTU coming up as for IPv6 locally? Also can you by chance find out if the provider is pushing an MTU value on IPv6 RA?
01-08-2018 06:12 PM - edited 01-08-2018 09:31 PM
3. Can you post the MAC address of your IPv4 default gateway?
Not sure if it will help, but the MAC address of my gateway is 00:01:5c:6a:c8:46
Can you set up a switch with port mirroring on the WAN side of ER and grant access to that as well?
Now that you mention port mirroring, I did a little search on it (I've never heard of or dealt with it before). So from what I see, I should be able to set up OpenWRT to do it?
Actually, now that I've looked at it, it appears I don't even have to do the port mirroring. I can just plug the modem and router into the switch on the device running OpenWRT, and then I can run tcpdump (or whatever other software needed) on that interface and see everything.
01-08-2018 10:26 PM
Attached are two pcaps, one with hwnat on and one with it off, captured after rebooting the router and modem. The hwnat off one stops once the ER-X has an IPv6 address. I let the hwnat on one run for a bit longer, but it seemed to be in a loop and the ER-X never got an IPv6 address.
01-09-2018 08:28 AM
One question - in the hwnat off case, I see the DHCPv6 solicit, advertise, and request packets (21, 24, and 25), but not a reply. Just lost in the packet capture maybe?
And one observation - in the hwnat off case, the single request packet (25) is 205 bytes. ALL of the request packets in the hwnat on case (e.g., 49, 63, 70, ...) are only 156 bytes. They all seem to get replies, but I'm not enough of a guru to pick the packets apart.
01-13-2018 12:06 PM
Interesting observation. Perhaps the CMTS engineers in the forum can weigh in on this. In the one Request packet in hwnat-off, the Server Identifier DUID type is link layer plus time, and a (incorrect) DUID time stamp is included. In hwnat-on, none of the Request packets contain the a timestamp in the Server Identiifer.
Perhaps with hwnat on, the Requests are not including a time stamp in Server Identifier and the result is that the upstream servers are ignoring them? RFC 3315 states that DUID plsu time stamp is to prevent client identifier collision. Maybe some ISPs ignore requests that do not use DUID+time?
Maybe this is a rabbit hole and not relevant to the problem. My understanding of how DHCPv6 should work is very cliff notes.
01-13-2018 07:04 PM
In the one Request packet in hwnat-off, the Server Identifier DUID type is link layer plus time, and a (incorrect) DUID time stamp is included.
Hmm... that's interesting. I have both devices set to use ntp and their times are at the moment are both correct. Only difference is the ER-X is set to UTC and the OpenWRT is set to PST. But the DUID time stamp isn't even for the right day, so it can't just be the timezone difference.
01-13-2018 07:11 PM - edited 01-13-2018 07:11 PM
Actually, the timestamp being wrongisn't that strange. The timestamp is just there to ensure DUID uniquness, it doesn't actually have to be accurate. For instance, in TLS, browsers are supposed to include a timestamp as part of the handshake sequence, but most modern browsers don't use real time stamps (e.g. Chrome). They just generate a unqiue (random) value here.
So it's possible the same thing is going on here. However I still wonder if the lack of the timestamp when one is expected by the upstream server is causing the DHCPv6 request to be ignored. Perhpas @UBNT-sandisn or another UBNT dev can comment on this.
01-15-2018 02:24 AM
Few questions to ubnt.
Is there an official way to disable SMT for debugging purposes?
Can we check the debug the hardware backed tables?
What version SDK is this hw nat module from or forked from?
3 weeks ago
I'm having this issue. I don't know yet whether disabling hwnat fixes it, as I just made that change today, but assume it will.
Location: Bethlehem, GA
ER X Release version: v1.9.7+hotfix.4.5024279.171006.0255
IPv6 initially works fine with hwnat enabled, but after some time I lose my WAN IPv6 address and all delegated addresses. The only way I've been able to get IPv6 working after this happens is to reboot, which so far has worked every time, giving me working IPv6 for the next 1-3 days. I've been using IPv6 with /60 PD for years without issue until I installed this router.
2 weeks ago
I've had no PD/IPv6 issues after about a week with hwnat off.
This thread will be a year old in 3 weeks. It doesn't seems like this is getting fixed if they haven't figured it out by now.