Reply
Ubiquiti Employee
Posts: 528
Registered: ‎01-06-2017
Kudos: 190
Solutions: 20

Re: Comcast IPv6 issues when hwnat enabled on ER-X


dbirkhead wrote:

I think the post talking about Hop Limit on DHCPv6 packets was on to something.   I have observed (both with Charter and Comcast) in two different geographies that when HWNAT is disabled the Hop Limit is at 1 from upstream.  When it is enabled I still see Hop Limit of 1 but it is not being seen or processed by the ERX.

 

This leads me to think that something in the offload process is decrementing HLIM before it is given to the processor which then dutifully discards it since it is now zero.

 

Obviously this is a theory, but am hoping that UBNT might investigate this as a possibility.


Thanks for the pointer!

 

Set the hop limit to 1 on the outgoing interface of the PD server.

 

echo 1 > /proc/sys/net/ipv6/conf/eth1/hop_limit

 

On ER-X I can see that it was really set to 1. PD works as expected though.

 

 

New Member
Posts: 31
Registered: ‎03-06-2017
Kudos: 2

Re: Comcast IPv6 issues when hwnat enabled on ER-X

Does ipv6 with hwnat work with is mod?

Tnx
Ubiquiti Employee
Posts: 528
Registered: ‎01-06-2017
Kudos: 190
Solutions: 20

Re: Comcast IPv6 issues when hwnat enabled on ER-X

I set the hop limit to 1 on the PD server. On the PD client (ER-X) PD still works. So hop limit alone is not causing the issue.

New Member
Posts: 28
Registered: ‎11-01-2016
Kudos: 2
Solutions: 2

Re: Comcast IPv6 issues when hwnat enabled on ER-X

As of two days ago, I now have the same issue with similar story as @kifl described a few posts back, Spectrum either did maintenance to their system or my 6183 modem and I lost ipv6 connectivity.  This morning, disabled hwnat and ipv6 returned.  Thx.

Member
Posts: 202
Registered: ‎06-19-2015
Kudos: 37
Solutions: 4

Re: Comcast IPv6 issues when hwnat enabled on ER-X

[ Edited ]

ER-X 1.9.7 hotfix 3

Motorola Surfboad SB6190 300mbps/20mpbs Spectrum aka Recked'um aka Rectum

Cat6a gigabit certified cable to clients

1 UAP AC LR

 

I got dhcpv6 PD working with HWNAT offload disabled, turn it on and reboot and its a no go, the wan port eth0 gets an IPv6 /128 address but no prefix delegation happens. I can post a config if need be.


!!! ERX is now dead, can not SSH, GUI login, no DHCP, !!! Going to RMA it and use my old ASUS in the mean time. I really liked having the ERX at home. Man Sad Even did a factory reset, no bueno

 

Regular Member
Posts: 447
Registered: ‎04-26-2016
Kudos: 119
Solutions: 24

Re: Comcast IPv6 issues when hwnat enabled on ER-X

@ncscomm

 

You can always buy an $8 USB-TTL cable and restore the ER-X yourself. I've done this successfully with a bricked ER-X... it's about 5 minutes of work... although if you are within warranty I understand.

 

https://community.ubnt.com/t5/EdgeMAX/ERX-ERX-SFP-System-Recovery/td-p/2056921

New Member
Posts: 31
Registered: ‎05-28-2017
Kudos: 12

Re: Comcast IPv6 issues when hwnat enabled on ER-X

[ Edited ]

For the record, within the past month I've started to see this same issue on a Comcast home network with an SB6183 and a TP-Link Archer C7 V2 router. TP-Link's recommendation? Wait for it ....

 

Turn off hardware NAT offload.

 

Just to clarify, what I'm seeing is that there is still a prefix delegation, but a disappearing IPV6 WAN address. That would suggest that the router is not receiving the DHCPv6 reply to the client's lease renewal request, wouldn't it?

 

New Member
Posts: 16
Registered: ‎06-17-2016
Kudos: 8

Re: Comcast IPv6 issues when hwnat enabled on ER-X

[ Edited ]

You mentioned Spectrum also, this issue is starting to sound more like a UBNT side related defect. 

 

What could help is if someone had this issue with one ERX, if they had another one to test with, a hardware defect would be far less likely. 

Emerging Member
Posts: 78
Registered: ‎07-18-2016
Kudos: 29
Solutions: 3

Re: Comcast IPv6 issues when hwnat enabled on ER-X

archer c7 is a qualcomm design so probably not relevant to the er-x issue

New Member
Posts: 16
Registered: ‎06-17-2016
Kudos: 8

Re: Comcast IPv6 issues when hwnat enabled on ER-X

[ Edited ]

I was going to say before that it was a bit off the topic. But a similarity is the offload process causing issues. Perhaps pointing to something in the linux networking stack where the offload process is happening.

New Member
Posts: 31
Registered: ‎05-28-2017
Kudos: 12

Re: Comcast IPv6 issues when hwnat enabled on ER-X

I would be very surprised if the router hardware platform had anything at all to do with it, as opposed to the firmware/software of the router or modem or the data on the wire from the ISP. In this case, nothing had changed for months regarding the router or the modem when this (familiar) problem suddenly appeared. That leaves the ISP.

 

Now maybe what the ISP is sending is perfectly fine, but different enough to exploit a latent problem in the modem or router firmware/software. Or maybe what the ISP is sending is not fine and is therefore unanticipated by the downstream devices and software.

 

New Member
Posts: 31
Registered: ‎05-28-2017
Kudos: 12

Re: Comcast IPv6 issues when hwnat enabled on ER-X

Two additional things:

1) Here's what's in my SB6183 modem log. The last actual timestamp corresponds
to a regression in the QoS version from 1.1 to 1.0 and the disappearance of a
CMTS MAC address. Problems appeared after that, to the best of my recollection.
I've also seen a reduction in my uplink channels from 4 to 3. Maybe there is a
also a signal strength issue involved here.

Time     Priority     Description
Time Not Established     Notice (6)     Honoring MDD; IP provisioning mode = IPv6
Time Not Established     Critical (3)     SYNC Timing Synchronization failure - Failed to acquire QAM/QPSK symbol timing;;CM-MAC=78:96:84:e7:0a:e7;CMTS-MAC=00:00:00:00:00:00;CM-QOS=1.0;CM-VER=3.0;
Time Not Established     Critical (3)     SYNC Timing Synchronization failure - Failed to acquire FEC framing;CM-MAC=78:96:84:e7:0a:e7;CMTS-MAC=00:00:00:00:00:00;CM-QOS=1.0;CM-VER=3.0;
Time Not Established     Critical (3)     SYNC Timing Synchronization failure - Failed to acquire QAM/QPSK symbol timing;;CM-MAC=78:96:84:e7:0a:e7;CMTS-MAC=00:00:00:00:00:00;CM-QOS=1.0;CM-VER=3.0;
Thu Sep 28 09:33:04 2017     Critical (3)     SYNC Timing Synchronization failure - Failed to acquire FEC framing;CM-MAC=78:96:84:e7:0a:e7;CMTS-MAC=00:00:00:00:00:00;CM-QOS=1.0;CM-VER=3.0;
Thu Sep 28 09:33:01 2017     Critical (3)     Received Response to Broadcast Maintenance Request, But no Unicast Maintenance opportunities received - T4 time out;CM-MAC=78:96:84:e7:0a:e7;CMTS-MAC=00:9e:1e:59:69:5d;CM-QOS=1.1;CM-VER=3.0;
Thu Sep 28 09:33:01 2017     Critical (3)     SYNC Timing Synchronization failure - Failed to acquire QAM/QPSK symbol timing;;CM-MAC=78:96:84:e7:0a:e7;CMTS-MAC=00:9e:1e:59:69:5d;CM-QOS=1.1;CM-VER=3.0;
Thu Sep 28 09:32:48 2017     Critical (3)     SYNC Timing Synchronization failure - Failed to acquire QAM/QPSK symbol timing;;CM-MAC=78:96:84:e7:0a:e7;CMTS-MAC=00:9e:1e:59:69:5d;CM-QOS=1.1;CM-VER=3.0;
Thu Sep 28 09:32:41 2017     Critical (3)     SYNC Timing Synchronization failure - Failed to acquire QAM/QPSK symbol timing;;CM-MAC=78:96:84:e7:0a:e7;CMTS-MAC=00:9e:1e:59:69:5d;CM-QOS=1.1;CM-VER=3.0;
Thu Sep 28 09:32:38 2017     Critical (3)     SYNC Timing Synchronization failure - Failed to acquire FEC framing;CM-MAC=78:96:84:e7:0a:e7;CMTS-MAC=00:9e:1e:59:69:5d;CM-QOS=1.1;CM-VER=3.0;

2) See this (long) thread on the Time-Warner (Spectrum) forum.
http://forums.timewarnercable.com/t5/forums/v3_1/forumtopicpage/board-id/IPv6/thread-id/650/page/1

If you read far enough in, these IPv6 problems with the SB6183 (and 6141?) go back
several firmware revisions. They have affected UNIX-like systems (such as Linux,
Android, and Mac OS X), but not Windows. Problems are still being reported with
the current firmware (D30CM-OSPREY-1.5.2.3-GA-01-NOSH) on systems that were previously
working: http://forums.timewarnercable.com/t5/IPv6/Arris-SB6183-and-ipv6/td-p/128668

 

New Member
Posts: 16
Registered: ‎06-17-2016
Kudos: 8

Re: Comcast IPv6 issues when hwnat enabled on ER-X

Did a little digging into the hwnat module it appears there were issues with the offload routes for IPv6 not being updated properly.

 

Is there anyway to view the hwnat table and compare it against the kernel routes?

New Member
Posts: 1
Registered: ‎03-16-2014

Re: Comcast IPv6 issues when hwnat enabled on ER-X


trs3 wrote:

For the record, within the past month I've started to see this same issue on a Comcast home network with an SB6183 and a TP-Link Archer C7 V2 router. TP-Link's recommendation? Wait for it ....

 

Turn off hardware NAT offload.

 

Just to clarify, what I'm seeing is that there is still a prefix delegation, but a disappearing IPV6 WAN address. That would suggest that the router is not receiving the DHCPv6 reply to the client's lease renewal request, wouldn't it?

 


I believe that I am experiencing the exact same issue, but with a totally different ISP and modem. I have a Pace/Arris 5268AC from AT&T Uverse. When I first got my ERX several months ago it was running some really old firmware (1.8.0 , maybe 1.9.0?). Once AT&T updated the 5268AC to add DHCPv6-PD, I manually setup PD on the ERX using the CLI and it worked great. I had working IPv6 on all my devices. Some time later, I updated the ERX to the 1.9.7 firmware and messed with some other settings to fix my VoIP ATA. Then, I noticed that IPv6 addresses were no longer showing up on the WAN interface and my devices had lost v6 connectivity. Unfortunately I did not save a backup of my working config. I have been going crazy the past few months trying to get v6 working again without any success. I've done multiple factory resets on both the ERX and 5268. I've tried using the wizards, cli, and config tree on the ERX to setup my network. I've looked at every post I can find on this forum and the broader internet about setting up IPv6 on the ERX and tried their instructions. Still, I cannot get the ERX to acquire a delegated prefix. If I set the ERX to use SLAAC for WAN IPv6, it can acquire that just fine (but then I don't get PD). When I go into the interface and logs on the 5268, I can always see that the DHCPv6-PD request reaches the 5268, and a prefix and WAN address are assigned. However, it appears that the ERX is not receiving/processing the DHCPv6 reply. I'm now running 1.9.7+hotfix.4 on my ERX. 

New Member
Posts: 31
Registered: ‎05-28-2017
Kudos: 12

Re: Comcast IPv6 issues when hwnat enabled on ER-X

I had a chance today to check my Comcast Business installation (ER-X V1.9.7 (no hotfix) with hwnat enabled and an SB6121). That modem has been up for almost 4 months and that router for almost 3. Everything is still fine. However, I did note that the modem logs are still showing QoS V1.1 and a non-zero CMTS MAC address, unlike my Comcast home modem, where HW NAT offload suddenly stopped working after an apparent Comcast modem and/or server reconfiguration.

 

Meanwhile, the home network (TP-Link Archer C7 V2 and SB6131 with QoS 1.0 and a zero CMTS MAC address) has now been up for 5 days with no further problem after disabling HW NAT offload.

 

 

 

Established Member
Posts: 1,758
Registered: ‎03-02-2016
Kudos: 407
Solutions: 134

Re: Comcast IPv6 issues when hwnat enabled on ER-X

[ Edited ]

FWIW, I'm experiencing this same issue. I have an ERX at my parents' house outside Chicago. They have Comcast. With hwnat enabled, I can't get an IPv6 address. With it disabled, IPv6 works fine.

 

Running 1.9.7 hotfix 4

 

Here is the output of the requested commands in the first post.

 

 

New Member
Posts: 31
Registered: ‎05-28-2017
Kudos: 12

Re: Comcast IPv6 issues when hwnat enabled on ER-X


unknownzer wrote:

Did a little digging into the hwnat module it appears there were issues with the offload routes for IPv6 not being updated properly.

 


Now that you mention it, I vaguely recall while navigating the config tree in the GUI that below the hwnat option were three(?) suboptions, one of which was IPv6 routes. If my memory is correct maybe someone who is having the problem could try enabling hwnat and disabling only that suboption to further narrow down the problem..

 

Established Member
Posts: 1,758
Registered: ‎03-02-2016
Kudos: 407
Solutions: 134

Re: Comcast IPv6 issues when hwnat enabled on ER-X

 


trs3 wrote:

unknownzer wrote:

Did a little digging into the hwnat module it appears there were issues with the offload routes for IPv6 not being updated properly.

 


Now that you mention it, I vaguely recall while navigating the config tree in the GUI that below the hwnat option were three(?) suboptions, one of which was IPv6 routes. If my memory is correct maybe someone who is having the problem could try enabling hwnat and disabling only that suboption to further narrow down the problem..

 


 

You are thinking of the "IPv6" offload option? That's for the ERL and its ilk. ERX can only use hwnat and ipsec, which just take an "enable" or "disable".

New Member
Posts: 31
Registered: ‎05-28-2017
Kudos: 12

Re: Comcast IPv6 issues when hwnat enabled on ER-X


gfunkdave wrote:

 


trs3 wrote:

unknownzer wrote:

Did a little digging into the hwnat module it appears there were issues with the offload routes for IPv6 not being updated properly.

 


Now that you mention it, I vaguely recall while navigating the config tree in the GUI that below the hwnat option were three(?) suboptions, one of which was IPv6 routes. If my memory is correct maybe someone who is having the problem could try enabling hwnat and disabling only that suboption to further narrow down the problem..

 


 

You are thinking of the "IPv6" offload option? That's for the ERL and its ilk. ERX can only use hwnat and ipsec, which just take an "enable" or "disable".


As I said, I might be mistaken, but I recall suboptions BELOW hwnat on the ER-X (if you expand hwnat in the config tree GUI). I'm not physically present at my ER-X site (and there's no reference manual), or I'd be less vague about it.

 

Senior Member
Posts: 5,692
Registered: ‎01-04-2017
Kudos: 795
Solutions: 288

Re: Comcast IPv6 issues when hwnat enabled on ER-X

There is no "sub-options" under hwnat. You can enable or disable. That's it.
Reply