Reply
New Member
Posts: 5
Registered: ‎08-11-2017
Kudos: 4

Re: Comcast IPv6 issues when hwnat enabled on ER-X

Awesome work!  This is a great investigation, and writeup of what you found.  Did you find that this fixed the issue on your ER-X?

 

I tried updating /opt/vyatta/sbin/dhcpv6-pd-duid.pl with your script, then from the CLI:

 

release dhcpv6-pd interface eth0
delete dhcpv6-pd duid
renew dhcpv6-pd interface eth0

 

And rebooting.  With hwnat enabled, I still experienced the same results - no IPv6 address on eth0.  So it would appear that while you did fix a bug, at least for some of us in this boat there is more going on.

 

Thanks again for investigating.

New Member
Posts: 20
Registered: ‎09-06-2015
Kudos: 39

Re: Comcast IPv6 issues when hwnat enabled on ER-X

OK... call me a glutton for punishment if you want, but I tried digging into this even further.  I'm still working on a write up which I'm going to do as a blog post, but I've hit a new roadblock... 

 

So the good news is that it seems the kernel module used in hardware nat offloading ("hw_nat.ko") includes all of it's debugging symbols:

 

root@panic:/# objdump --syms /lib/modules/3.10.107-UBNT/kernel/net/nat/hw_nat/hw_nat.ko 

/lib/modules/3.10.107-UBNT/kernel/net/nat/hw_nat/hw_nat.ko:     file format elf32-tradlittlemips

SYMBOL TABLE:
00000000 l    d  .text	00000000 .text
00000000 l    d  .text.unlikely	00000000 .text.unlikely
00000000 l    d  .rodata	00000000 .rodata
00000000 l    d  .bss	00000000 .bss
000001ac l     F .text	0000010c PpeSetFoeGloCfgEbl
0000223c l     F .text	000003f4 PpeInitMod
00002630 l     F .text	000000b0 PpeCleanupMod
00000000 l     O .modinfo	00000027 __UNIQUE_ID_description2
00000027 l     O .modinfo	0000000c __UNIQUE_ID_license1
00000033 l     O .modinfo	0000001c __UNIQUE_ID_author0
00000000 l       .rodata.str1.4	00000000 $LC0
0000002c l       .rodata.str1.4	00000000 $LC1
000000a4 l       .rodata.str1.4	00000000 $LC3
00000068 l       .rodata.str1.4	00000000 $LC2
00000124 l       .rodata.str1.4	00000000 $LC10
00000134 l       .rodata.str1.4	00000000 $LC11
00000158 l       .rodata.str1.4	00000000 $LC12
0000017c l       .rodata.str1.4	00000000 $LC13
00000110 l       .rodata.str1.4	00000000 $LC8
000001d0 l       .rodata.str1.4	00000000 $LC18
000001e0 l       .rodata.str1.4	00000000 $LC19
000001e8 l       .rodata.str1.4	00000000 $LC20
00000238 l       .rodata.str1.4	00000000 $LC26
00000248 l       .rodata.str1.4	00000000 $LC27
00000254 l       .rodata.str1.4	00000000 $LC28
000001f0 l       .rodata.str1.4	00000000 $LC21
000000ec l       .rodata.str1.4	00000000 $LC5
000001ac l       .rodata.str1.4	00000000 $LC16
000001c4 l       .rodata.str1.4	00000000 $LC17
00000218 l       .rodata.str1.4	00000000 $LC24
00000228 l       .rodata.str1.4	00000000 $LC25
000001f8 l       .rodata.str1.4	00000000 $LC22
00000208 l       .rodata.str1.4	00000000 $LC23
0000019c l       .rodata.str1.4	00000000 $LC15
0000018c l       .rodata.str1.4	00000000 $LC14
00000118 l       .rodata.str1.4	00000000 $LC9
00000104 l       .rodata.str1.4	00000000 $LC7
000000f8 l       .rodata.str1.4	00000000 $LC6
000000e0 l       .rodata.str1.4	00000000 $LC4
0000025c l       .rodata.str1.4	00000000 $LC29
00000280 l       .rodata.str1.4	00000000 $LC30
0000028c l       .rodata.str1.4	00000000 $LC31
00000290 l       .rodata.str1.4	00000000 $LC32
00000294 l       .rodata.str1.4	00000000 $LC33
0000029c l       .rodata.str1.4	00000000 $LC34
000002a4 l       .rodata.str1.4	00000000 $LC35
00000000 l     F .text.unlikely	0000006c is_request_done
00000000 l     O .rodata	00000012 __func__.35194
000002cc l       .rodata.str1.4	00000000 $LC4
000002e4 l       .rodata.str1.4	00000000 $LC5
00000308 l       .rodata.str1.4	00000000 $LC6
00000340 l       .rodata.str1.4	00000000 $LC8
000002b4 l       .rodata.str1.4	00000000 $LC1
000002bc l       .rodata.str1.4	00000000 $LC2
0000032c l       .rodata.str1.4	00000000 $LC7
00000354 l       .rodata.str1.4	00000000 $LC9
00000388 l       .rodata.str1.4	00000000 $LC10
00000394 l       .rodata.str1.4	00000000 $LC11
000002c4 l       .rodata.str1.4	00000000 $LC3
000002ac l       .rodata.str1.4	00000000 $LC0
000003c0 l       .rodata.str1.4	00000000 $LC12
000009a0 l     O .bss	00000020 Buf.35208
000003cc l       .rodata.str1.4	00000000 $LC0
000003d8 l       .rodata.str1.4	00000000 $LC0
000000b0 l     O .rodata	00000014 __func__.31060
000000c4 l     O .rodata	00000014 __func__.31071
000000d8 l     O .rodata	00000014 __func__.31082
000003e0 l       .rodata.str1.4	00000000 $LC0
00000420 l       .rodata.str1.4	00000000 $LC1
00000450 l       .rodata.str1.4	00000000 $LC2
0000049c l       .rodata.str1.4	00000000 $LC3
000004c4 l       .rodata.str1.4	00000000 $LC4
00000050 l     O .modinfo	00000009 __module_depends
00000059 l     O .modinfo	00000009 __UNIQUE_ID_intree1
00000062 l     O .modinfo	00000037 __UNIQUE_ID_vermagic0
00000000 l    d  .note.gnu.build-id	00000000 .note.gnu.build-id
00000000 l    d  .reginfo	00000000 .reginfo
00000000 l    d  .rodata.str1.4	00000000 .rodata.str1.4
00000000 l    d  .modinfo	00000000 .modinfo
00000000 l    d  .data	00000000 .data
00000000 l    d  .gnu.linkonce.this_module	00000000 .gnu.linkonce.this_module
00000000 l    d  .pdr	00000000 .pdr
00000000 l    d  .comment	00000000 .comment
00000000 l    d  .gnu.attributes	00000000 .gnu.attributes
00000000 l    d  .mdebug.abi32	00000000 .mdebug.abi32
000016a4 g     F .text	00000330 PpeFillInL3Info
00002fd8 g     F .text	000000ac GetNext
00000000 g     F .text	000001ac update_foe_ac_timer_handler
000039a8 g     F .text	00000088 PpeSetBindLifetime
00000000         *UND*	00000000 __udelay
00000704 g     F .text	000000b0 PpeHitBindForceToCpuHandler
00000c00 g     F .text	00000008 isHwVlanTx
000021a8 g     F .text	00000018 ra_dev_get_by_name
00003654 g     F .text	000000a0 FoeToOrgIpHdr
00000000         *UND*	00000000 ra_sw_nat_hook_tx
00000000         *UND*	00000000 dev_queue_xmit
000000c2 g     O .data	00000002 lan_vid
00000000         *UND*	00000000 csum_partial
00000000 g     O .gnu.linkonce.this_module	00000140 __this_module
00000000         *UND*	00000000 memmove
000002b8 g     F .text	00000178 RemoveVlanTag
00000990 g     O .bss	00000004 PpeFoeBase
00000808 g     O .bss	00000084 PpeParseResult
00003700 g     F .text	00000048 PpeRegIoctlHandler
00003fe8 g     F .text	000001a0 foe_mcast_entry_del
00002704 g     F .text	00000014 FoeSetMacLoInfo
0000386c g     F .text	00000010 PpeSetBindThreshold
00003a30 g     F .text	0000023c HwNatIoctl
00002630 g     F .text	000000b0 cleanup_module
00000000         *UND*	00000000 memcpy
00001b20 g     F .text	00000098 PpeSetMtrPktInfo
00002f3c g     F .text	00000044 FoeDelEntryByNum
000008ac g     F .text	000002bc PpeRxHandler
00002160 g     F .text	00000048 PpeSetFoeEbl
000032a4 g     F .text	00000060 RegModifyBits
0000325c g     F .text	00000048 CalIpRange
0000223c g     F .text	000003f4 init_module
00003c70 g     F .text	000000b0 mcast_entry_get
000000d0 g     O .data	00000014 mtbl_lock
000004e4 g     F .text	00000008 PpeExtIfRxHandler
00000000         *UND*	00000000 kfree_skb
000021e8 g     F .text	00000054 foe_ac_update_ebl
00000000         *UND*	00000000 is_switch0_member
00001f34 g     F .text	0000022c PpeTxHandler
0000387c g     F .text	00000088 PpeSetMaxEntryLimit
00000000 g     O .bss	00000001 queue_number
00001510 g     F .text	00000194 PpeFillInL2Info
00000000         *UND*	00000000 boot_tvec_bases
00001ed0 g     F .text	00000064 PpeSetEntryBind
00002b84 g     F .text	00000338 FoeGetAllEntries
00004308 g     F .text	0000009c foe_mcast_entry_del_all
0000001c g     O .data	0000002a ag_map
00000000         *UND*	00000000 init_net
00000050 g     O .data	0000006c hw_nat_fops
00000000         *UND*	00000000 skb_pull
000004ec g     F .text	00000008 PpeExtIfPingPongHandler
00000b68 g     F .text	00000068 GetPppoeSid
00003748 g     F .text	00000020 PpeUnRegIoctlHandler
00000000         *UND*	00000000 add_timer
00002efc g     F .text	00000040 FoeUnBindEntry
00003768 g     F .text	00000104 PpeGetAGCnt
0000088c g     O .bss	00000100 DstPort
00000000         *UND*	00000000 mutex_lock
00000000         *UND*	00000000 skb_push
00003904 g     F .text	0000006c PpeSetKaInterval
000026e0 g     F .text	00000024 FoeSetMacHiInfo
00000000         *UND*	00000000 printk
000021c0 g     F .text	00000028 SetGdmaFwd
000000c0 g     O .data	00000002 wan_vid
00000430 g     F .text	000000b4 FoeDumpPkt
000004f4 g     F .text	00000210 PpeKeepAliveHandler
00000000         *UND*	00000000 _ctype
00001bb8 g     F .text	00000310 PpeSetForcePortInfo
00001ec8 g     F .text	00000008 PpeSetExtIfNum
000000bc g     O .data	00000004 DebugLevel
000034ac g     F .text	000001a8 FoeToOrgUdpHdr
00003ed0 g     F .text	00000118 foe_mcast_entry_qid
00000000         *UND*	00000000 memset
00004188 g     F .text	00000180 foe_mcast_entry_dump
00000000         *UND*	00000000 del_timer_sync
000007b4 g     F .text	000000f8 PpeHitBindForceMcastToWiFiHandler
00003304 g     F .text	000001a8 FoeToOrgTcpHdr
00000008 g     O .bss	00000800 ac_info
00000000         *UND*	00000000 jiffies
00003084 g     F .text	00000058 Ip2Str
00000000         *UND*	00000000 sprintf
00000000         *UND*	00000000 ra_sw_nat_hook_rx
00002ebc g     F .text	00000040 FoeBindEntry
00000000         *UND*	00000000 mutex_unlock
000019d4 g     F .text	00000094 PpeFillInL4Info
00002718 g     F .text	0000035c FoeDumpCacheEntry
000030dc g     F .text	00000180 Str2Ip
00000000         *UND*	00000000 __register_chrdev
00000000         *UND*	00000000 mips_dma_map_ops
00000bd0 g     F .text	00000008 isSpecialTag
00000000         *UND*	00000000 skb_copy
00000000         *UND*	00000000 skb_clone
00000000         *UND*	00000000 dev_get_by_name
00002a74 g     F .text	00000110 FoeDumpEntry
00002f80 g     F .text	0000001c FoeTblClean
00000000         *UND*	00000000 upd_eth_stats
00001a68 g     F .text	000000b8 PpeSetMtrByteInfo
0000098c g     O .bss	00000004 PpePhyFoeBase
00002fa0 g     F .text	00000038 MacReverse
00003970 g     F .text	00000038 PpeSetUnbindLifeTime
00000000         *UND*	00000000 strchr
00000000 g     O .data	0000001c update_foe_ac_timer
000000c4 g     O .data	00000001 bind_dir
00000c08 g     F .text	00000908 PpeParseLayerInfo
00003d20 g     F .text	000001b0 foe_mcast_entry_ins
00000000         *UND*	00000000 __unregister_chrdev
00000bd8 g     F .text	00000028 is8021Q

The bad news is that it seems the kernel does not:

 

root@panic:/# gdb vmlinux /proc/kcore
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "mipsel-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
vmlinux: No such file or directory.
[New process 1]
Core was generated by `console=ttyS1,57600n8 ubi.mtd=7 root=ubi0_0 rootfstype=ubifs rootsqimg=squashfs.'.
#0  0x00000000 in ?? ()
(gdb) p jiffies_64
No symbol table is loaded.  Use the "file" command.

Are the debug symbols available for the kernel?

New Member
Posts: 6
Registered: ‎01-30-2018
Kudos: 4

Re: Comcast IPv6 issues when hwnat enabled on ER-X

Just checking in to say, using an updated `dhcpv6-pd-duid.p`l my IPv6 connectivity died after 4 days.

New Member
Posts: 20
Registered: ‎09-06-2015
Kudos: 39

Re: Comcast IPv6 issues when hwnat enabled on ER-X

(NOTE: I had to split this post in two because of message limits)

 

Well, along those lines I realized a few things. 

 

1) In all of this I'd left hwnat off because of the problems (the previous issue was causing my ipv6 addressing to keep changing which for me was even worse than unavailability).  

 

2) The bad good news is that I'm still experiencing the same issue.  I went through and did some more exhaustive testing.  I hooked up one of my network taps on the eth0 of the Edgerouter so that I wouldn't have to kill/restart dhcp6c and (potentially) see behaviour which was different from the manual execution.  This has meant that I have a growing number of pcapng files which I have been digging through.

 

To begin, let's define two states:

 

   "hw_nat disabled" - ipv6 DHCPv6 completes successfully

   "hw_nat enabled" - ipv6 DHCPv6 does not complete successfully

 

from here on out I'm going to simplify these as "disabled" and "enabled" for brevity.

 

So in a disabled state we see a total of 4 packets (as expected) in the DHCPv6 handshake - Solicit, Advertise, Request, Reply.  Everything looks great.

 

In an enabled state I see 2 + N*2 packets where the first two packets are Solicit & Advertise and then there is a repeated number of Request/Reply pairs working in an an expotential backoff.

 

Unfortunately, while I have the published source, my brain is melting at reading a unified diff because the lions share of the changes against both the upstream WIDE-DHCPv6 (https://sourceforge.net/p/wide-dhcpv6/git/ci/master/tree/dhcp6c.c) and other mirrors doesn't seem to show their changes (and thus the commit history to show why things changed).

 

I've run through this test a few times now, but here's the newest of what I've seen.


Start off in a powered off state and reset my PCAPNG capture on my desktop machine, then power the EdgeRouter-X on.  Once it's booted, we're off to the races.

 

Check for  enabled mode:

 

$ show configuration commands | grep hwnat 
set system offload hwnat enable

 

Confirm this state:

 

$ lsmod | grep hw_nat
hw_nat                 24220  0 

Look for IPv6 Addressing:

 

 

$ ip -o -6 addr show dev eth0
4: eth0    inet6 fe80::618:d6ff:fe06:6105/64 scope link \       valid_lft forever preferred_lft forever

Stop dhcp6c:

 

 

$ sudo pkill dhcp6c

 

 

Behind the scenes I reset the PCAP on my tap

 

Start it manually:

 

$ sudo /usr/sbin/dhcp6c -c /var/run/dhcp6c-eth0-pd.conf -p /var/run/dhcp6c-eth0-pd.pid -D -f eth0
Apr/03/2018 04:22:49: get_duid: extracted an existing DUID from /var/lib/dhcpv6/dhcp6c_duid: 00:01:06:00:5a:bd:e7:f2:04:18:d6:06:61:05
Apr/03/2018 04:22:49: cfdebug_print: <3>comment [# This file was auto-generated by /opt/vyatta/sbin/dhcpv6-pd-client.pl] (70)
Apr/03/2018 04:22:49: cfdebug_print: <3>comment [# configuration sub-system.  Do not edit it.] (44)
Apr/03/2018 04:22:49: cfdebug_print: <3>[interface] (9)
Apr/03/2018 04:22:49: cfdebug_print: <5>[eth0] (4)
Apr/03/2018 04:22:49: cfdebug_print: <3>begin of closure [{] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>[send] (4)
Apr/03/2018 04:22:49: cfdebug_print: <3>[ia-na] (5)
Apr/03/2018 04:22:49: cfdebug_print: <3>[0] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>[request] (7)
Apr/03/2018 04:22:49: cfdebug_print: <3>[domain-name-servers] (19)
Apr/03/2018 04:22:49: cfdebug_print: <3>[,] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>[domain-name] (11)
Apr/03/2018 04:22:49: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>[send] (4)
Apr/03/2018 04:22:49: cfdebug_print: <3>[rapid-commit] (12)
Apr/03/2018 04:22:49: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>[send] (4)
Apr/03/2018 04:22:49: cfdebug_print: <3>[ia-pd] (5)
Apr/03/2018 04:22:49: cfdebug_print: <3>[0] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>[script] (6)
Apr/03/2018 04:22:49: cfdebug_print: <3>["/opt/vyatta/sbin/ubnt-dhcp6c-script"] (37)
Apr/03/2018 04:22:49: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>end of closure [}] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>[id-assoc] (8)
Apr/03/2018 04:22:49: cfdebug_print: <15>[na] (2)
Apr/03/2018 04:22:49: cfdebug_print: <15>[0] (1)
Apr/03/2018 04:22:49: cfdebug_print: <15>begin of closure [{] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>end of closure [}] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>[id-assoc] (8)
Apr/03/2018 04:22:49: cfdebug_print: <15>[pd] (2)
Apr/03/2018 04:22:49: cfdebug_print: <15>[0] (1)
Apr/03/2018 04:22:49: cfdebug_print: <15>begin of closure [{] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>[prefix] (6)
Apr/03/2018 04:22:49: cfdebug_print: <3>[::] (2)
Apr/03/2018 04:22:49: cfdebug_print: <3>[/] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>[60] (2)
Apr/03/2018 04:22:49: cfdebug_print: <3>[infinity] (8)
Apr/03/2018 04:22:49: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>[prefix-interface] (16)
Apr/03/2018 04:22:49: cfdebug_print: <5>[switch0] (7)
Apr/03/2018 04:22:49: cfdebug_print: <3>begin of closure [{] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>[sla-id] (6)
Apr/03/2018 04:22:49: cfdebug_print: <3>[0] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>[sla-len] (7)
Apr/03/2018 04:22:49: cfdebug_print: <3>[4] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>[ifid] (4)
Apr/03/2018 04:22:49: cfdebug_print: <3>[1] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>end of closure [}] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>end of closure [}] (1)
Apr/03/2018 04:22:49: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:22:49: configure_pool: called
Apr/03/2018 04:22:49: clear_poolconf: called
Apr/03/2018 04:22:49: dhcp6_reset_timer: reset a timer on eth0, state=INIT, timeo=0, retrans=677
Apr/03/2018 04:22:50: client6_send: a new XID (10068b) is generated
Apr/03/2018 04:22:50: copy_option: set client ID (len 14)
Apr/03/2018 04:22:50: copyout_option: set identity association
Apr/03/2018 04:22:50: copy_option: set rapid commit (len 0)
Apr/03/2018 04:22:50: copy_option: set elapsed time (len 2)
Apr/03/2018 04:22:50: copy_option: set option request (len 4)
Apr/03/2018 04:22:50: copyout_option: set IA_PD prefix
Apr/03/2018 04:22:50: copyout_option: set IA_PD
Apr/03/2018 04:22:50: client6_send: send solicit to ff02::1:2%eth0
Apr/03/2018 04:22:50: dhcp6_reset_timer: reset a timer on eth0, state=SOLICIT, timeo=0, retrans=1049
Apr/03/2018 04:22:51: copy_option: set client ID (len 14)
Apr/03/2018 04:22:51: copyout_option: set identity association
Apr/03/2018 04:22:51: copy_option: set rapid commit (len 0)
Apr/03/2018 04:22:51: copy_option: set elapsed time (len 2)
Apr/03/2018 04:22:51: copy_option: set option request (len 4)
Apr/03/2018 04:22:51: copyout_option: set IA_PD prefix
Apr/03/2018 04:22:51: copyout_option: set IA_PD
Apr/03/2018 04:22:51: client6_send: send solicit to ff02::1:2%eth0
Apr/03/2018 04:22:51: dhcp6_reset_timer: reset a timer on eth0, state=SOLICIT, timeo=1, retrans=2173
Apr/03/2018 04:22:53: copy_option: set client ID (len 14)
Apr/03/2018 04:22:53: copyout_option: set identity association
Apr/03/2018 04:22:53: copy_option: set rapid commit (len 0)
Apr/03/2018 04:22:53: copy_option: set elapsed time (len 2)
Apr/03/2018 04:22:53: copy_option: set option request (len 4)
Apr/03/2018 04:22:53: copyout_option: set IA_PD prefix
Apr/03/2018 04:22:53: copyout_option: set IA_PD
Apr/03/2018 04:22:53: client6_send: send solicit to ff02::1:2%eth0
Apr/03/2018 04:22:53: dhcp6_reset_timer: reset a timer on eth0, state=SOLICIT, timeo=2, retrans=4219
Apr/03/2018 04:22:57: copy_option: set client ID (len 14)
Apr/03/2018 04:22:57: copyout_option: set identity association
Apr/03/2018 04:22:57: copy_option: set rapid commit (len 0)
Apr/03/2018 04:22:57: copy_option: set elapsed time (len 2)
Apr/03/2018 04:22:57: copy_option: set option request (len 4)
Apr/03/2018 04:22:57: copyout_option: set IA_PD prefix
Apr/03/2018 04:22:57: copyout_option: set IA_PD
Apr/03/2018 04:22:57: client6_send: send solicit to ff02::1:2%eth0
Apr/03/2018 04:22:57: dhcp6_reset_timer: reset a timer on eth0, state=SOLICIT, timeo=3, retrans=8573
^C

 

 

Stop my packet capture.

 

And check out the state of eth0:

 

$ ip -o -6 addr show dev eth0
4: eth0    inet6 fe80::618:d6ff:fe06:6105/64 scope link \       valid_lft forever preferred_lft forever

 

And compare my pcap files, first looking at boot:

 

 

$ tshark -r dhcpv6-boot.pcap dhcpv6
13610 120.887084336 fe80::618:d6ff:fe06:6105 → ff02::1:2    DHCPv6 163 Solicit XID: 0xce1ea9 CID: 000106005abde7f20418d6066105 
13624 120.942684414 fe80::201:5cff:fe6d:9c46 → fe80::618:d6ff:fe06:6105 DHCPv6 227 Advertise XID: 0xce1ea9 CID: 000106005abde7f20418d6066105 IAA: 2001:558:6045:61:e468:ec18:1892:322a 
13795 121.889983471 fe80::618:d6ff:fe06:6105 → ff02::1:2    DHCPv6 205 Request XID: 0x8c635a CID: 000106005abde7f20418d6066105 IAA: 2001:558:6045:61:e468:ec18:1892:322a 
13799 121.917675071 fe80::201:5cff:fe6d:9c46 → fe80::618:d6ff:fe06:6105 DHCPv6 227 Reply XID: 0x8c635a CID: 000106005abde7f20418d6066105 IAA: 2001:558:6045:61:e468:ec18:1892:322a 
13919 122.908736113 fe80::618:d6ff:fe06:6105 → ff02::1:2    DHCPv6 205 Request XID: 0x8c635a CID: 000106005abde7f20418d6066105 IAA: 2001:558:6045:61:e468:ec18:1892:322a 
13922 122.939235366 fe80::201:5cff:fe6d:9c46 → fe80::618:d6ff:fe06:6105 DHCPv6 227 Reply XID: 0x8c635a CID: 000106005abde7f20418d6066105 IAA: 2001:558:6045:61:e468:ec18:1892:322a 
14102 124.850359832 fe80::618:d6ff:fe06:6105 → ff02::1:2    DHCPv6 205 Request XID: 0x8c635a CID: 000106005abde7f20418d6066105 IAA: 2001:558:6045:61:e468:ec18:1892:322a 
14104 124.881100512 fe80::201:5cff:fe6d:9c46 → fe80::618:d6ff:fe06:6105 DHCPv6 227 Reply XID: 0x8c635a CID: 000106005abde7f20418d6066105 IAA: 2001:558:6045:61:e468:ec18:1892:322a 
14532 128.774355292 fe80::618:d6ff:fe06:6105 → ff02::1:2    DHCPv6 205 Request XID: 0x8c635a CID: 000106005abde7f20418d6066105 IAA: 2001:558:6045:61:e468:ec18:1892:322a 
14536 128.831952043 fe80::201:5cff:fe6d:9c46 → fe80::618:d6ff:fe06:6105 DHCPv6 227 Reply XID: 0x8c635a CID: 000106005abde7f20418d6066105 IAA: 2001:558:6045:61:e468:ec18:1892:322a 
15182 136.402075613 fe80::618:d6ff:fe06:6105 → ff02::1:2    DHCPv6 205 Request XID: 0x8c635a CID: 000106005abde7f20418d6066105 IAA: 2001:558:6045:61:e468:ec18:1892:322a 
15189 136.462648373 fe80::201:5cff:fe6d:9c46 → fe80::618:d6ff:fe06:6105 DHCPv6 227 Reply XID: 0x8c635a CID: 000106005abde7f20418d6066105 IAA: 2001:558:6045:61:e468:ec18:1892:322a 
16921 151.818034349 fe80::618:d6ff:fe06:6105 → ff02::1:2    DHCPv6 205 Request XID: 0x8c635a CID: 000106005abde7f20418d6066105 IAA: 2001:558:6045:61:e468:ec18:1892:322a 
16924 151.875715413 fe80::201:5cff:fe6d:9c46 → fe80::618:d6ff:fe06:6105 DHCPv6 227 Reply XID: 0x8c635a CID: 000106005abde7f20418d6066105 IAA: 2001:558:6045:61:e468:ec18:1892:322a 
18268 184.209968769 fe80::618:d6ff:fe06:6105 → ff02::1:2    DHCPv6 205 Request XID: 0x8c635a CID: 000106005abde7f20418d6066105 IAA: 2001:558:6045:61:e468:ec18:1892:322a 
18273 184.265890431 fe80::201:5cff:fe6d:9c46 → fe80::618:d6ff:fe06:6105 DHCPv6 227 Reply XID: 0x8c635a CID: 000106005abde7f20418d6066105 IAA: 2001:558:6045:61:e468:ec18:1892:322a 
20375 213.174006343 fe80::618:d6ff:fe06:6105 → ff02::1:2    DHCPv6 205 Request XID: 0x8c635a CID: 000106005abde7f20418d6066105 IAA: 2001:558:6045:61:e468:ec18:1892:322a 
20376 213.229691955 fe80::201:5cff:fe6d:9c46 → fe80::618:d6ff:fe06:6105 DHCPv6 227 Reply XID: 0x8c635a CID: 000106005abde7f20418d6066105 IAA: 2001:558:6045:61:e468:ec18:1892:322a 

 

Then checking my manual run:

 

$ tshark -r dhcpv6-manual.pcap dhcpv6
  394 13.949859802 fe80::618:d6ff:fe06:6105 → ff02::1:2    DHCPv6 163 Solicit XID: 0x10068b CID: 000106005abde7f20418d6066105 
  395 14.006477532 fe80::201:5cff:fe6d:9c46 → fe80::618:d6ff:fe06:6105 DHCPv6 227 Advertise XID: 0x10068b CID: 000106005abde7f20418d6066105 IAA: 2001:558:6045:61:e468:ec18:1892:322a 
  397 15.005555803 fe80::618:d6ff:fe06:6105 → ff02::1:2    DHCPv6 163 Solicit XID: 0x10068b CID: 000106005abde7f20418d6066105 
  398 15.032525211 fe80::201:5cff:fe6d:9c46 → fe80::618:d6ff:fe06:6105 DHCPv6 227 Advertise XID: 0x10068b CID: 000106005abde7f20418d6066105 IAA: 2001:558:6045:61:e468:ec18:1892:322a 
  426 17.184783931 fe80::618:d6ff:fe06:6105 → ff02::1:2    DHCPv6 163 Solicit XID: 0x10068b CID: 000106005abde7f20418d6066105 
  427 17.212408974 fe80::201:5cff:fe6d:9c46 → fe80::618:d6ff:fe06:6105 DHCPv6 227 Advertise XID: 0x10068b CID: 000106005abde7f20418d6066105 IAA: 2001:558:6045:61:e468:ec18:1892:322a 
  562 21.411082513 fe80::618:d6ff:fe06:6105 → ff02::1:2    DHCPv6 163 Solicit XID: 0x10068b CID: 000106005abde7f20418d6066105 
  563 21.465649631 fe80::201:5cff:fe6d:9c46 → fe80::618:d6ff:fe06:6105 DHCPv6 227 Advertise XID: 0x10068b CID: 000106005abde7f20418d6066105 IAA: 2001:558:6045:61:e468:ec18:1892:322a

Low and behold... different behavior when run via boot versus being run manually.  Likely this is due to some state change in the system.  Thinking I have an idea as to what this may be (::cough cough:: the hw_nat kernel module ::cough cough::) I decide to check this out:

 

Ensure that DHCP6c isn't running and double check that I didn't magically get an IPv6 address:

 

$ pgrep dhcp6c
$ ip -o -6 addr show dev eth0
4: eth0    inet6 fe80::618:d6ff:fe06:6105/64 scope link \       valid_lft forever preferred_lft forever

 

New Member
Posts: 20
Registered: ‎09-06-2015
Kudos: 39

Re: Comcast IPv6 issues when hwnat enabled on ER-X

[ Edited ]

(NOTE: Continued from the last post)

 

Now I reset my next pcap, unload hw_nat, and then try again:

 

$ sudo modprobe -r hw_nat 
$ lsmod | grep hw_nat
$ sudo /usr/sbin/dhcp6c -c /var/run/dhcp6c-eth0-pd.conf -p /var/run/dhcp6c-eth0-pd.pid -D -f eth0
Apr/03/2018 04:33:01: get_duid: extracted an existing DUID from /var/lib/dhcpv6/dhcp6c_duid: 00:01:06:00:5a:bd:e7:f2:04:18:d6:06:61:05
Apr/03/2018 04:33:01: cfdebug_print: <3>comment [# This file was auto-generated by /opt/vyatta/sbin/dhcpv6-pd-client.pl] (70)
Apr/03/2018 04:33:01: cfdebug_print: <3>comment [# configuration sub-system.  Do not edit it.] (44)
Apr/03/2018 04:33:01: cfdebug_print: <3>[interface] (9)
Apr/03/2018 04:33:01: cfdebug_print: <5>[eth0] (4)
Apr/03/2018 04:33:01: cfdebug_print: <3>begin of closure [{] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>[send] (4)
Apr/03/2018 04:33:01: cfdebug_print: <3>[ia-na] (5)
Apr/03/2018 04:33:01: cfdebug_print: <3>[0] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>[request] (7)
Apr/03/2018 04:33:01: cfdebug_print: <3>[domain-name-servers] (19)
Apr/03/2018 04:33:01: cfdebug_print: <3>[,] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>[domain-name] (11)
Apr/03/2018 04:33:01: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>[send] (4)
Apr/03/2018 04:33:01: cfdebug_print: <3>[rapid-commit] (12)
Apr/03/2018 04:33:01: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>[send] (4)
Apr/03/2018 04:33:01: cfdebug_print: <3>[ia-pd] (5)
Apr/03/2018 04:33:01: cfdebug_print: <3>[0] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>[script] (6)
Apr/03/2018 04:33:01: cfdebug_print: <3>["/opt/vyatta/sbin/ubnt-dhcp6c-script"] (37)
Apr/03/2018 04:33:01: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>end of closure [}] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>[id-assoc] (8)
Apr/03/2018 04:33:01: cfdebug_print: <15>[na] (2)
Apr/03/2018 04:33:01: cfdebug_print: <15>[0] (1)
Apr/03/2018 04:33:01: cfdebug_print: <15>begin of closure [{] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>end of closure [}] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>[id-assoc] (8)
Apr/03/2018 04:33:01: cfdebug_print: <15>[pd] (2)
Apr/03/2018 04:33:01: cfdebug_print: <15>[0] (1)
Apr/03/2018 04:33:01: cfdebug_print: <15>begin of closure [{] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>[prefix] (6)
Apr/03/2018 04:33:01: cfdebug_print: <3>[::] (2)
Apr/03/2018 04:33:01: cfdebug_print: <3>[/] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>[60] (2)
Apr/03/2018 04:33:01: cfdebug_print: <3>[infinity] (8)
Apr/03/2018 04:33:01: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>[prefix-interface] (16)
Apr/03/2018 04:33:01: cfdebug_print: <5>[switch0] (7)
Apr/03/2018 04:33:01: cfdebug_print: <3>begin of closure [{] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>[sla-id] (6)
Apr/03/2018 04:33:01: cfdebug_print: <3>[0] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>[sla-len] (7)
Apr/03/2018 04:33:01: cfdebug_print: <3>[4] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>[ifid] (4)
Apr/03/2018 04:33:01: cfdebug_print: <3>[1] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>end of closure [}] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>end of closure [}] (1)
Apr/03/2018 04:33:01: cfdebug_print: <3>end of sentence [;] (1)
Apr/03/2018 04:33:01: configure_pool: called
Apr/03/2018 04:33:01: clear_poolconf: called
Apr/03/2018 04:33:01: dhcp6_reset_timer: reset a timer on eth0, state=INIT, timeo=0, retrans=132
Apr/03/2018 04:33:01: client6_send: a new XID (deb0f2) is generated
Apr/03/2018 04:33:01: copy_option: set client ID (len 14)
Apr/03/2018 04:33:01: copyout_option: set identity association
Apr/03/2018 04:33:01: copy_option: set rapid commit (len 0)
Apr/03/2018 04:33:01: copy_option: set elapsed time (len 2)
Apr/03/2018 04:33:01: copy_option: set option request (len 4)
Apr/03/2018 04:33:01: copyout_option: set IA_PD prefix
Apr/03/2018 04:33:01: copyout_option: set IA_PD
Apr/03/2018 04:33:01: client6_send: send solicit to ff02::1:2%eth0
Apr/03/2018 04:33:01: dhcp6_reset_timer: reset a timer on eth0, state=SOLICIT, timeo=0, retrans=1022
Apr/03/2018 04:33:01: client6_recv: receive advertise from fe80::201:5cff:fe6d:9c46%eth0 on eth0
Apr/03/2018 04:33:01: dhcp6_get_options: get DHCP option client ID, len 14
Apr/03/2018 04:33:01:   DUID: 00:01:06:00:5a:bd:e7:f2:04:18:d6:06:61:05
Apr/03/2018 04:33:01: dhcp6_get_options: get DHCP option server ID, len 14
Apr/03/2018 04:33:01:   DUID: 00:01:00:01:1c:e3:7b:f8:40:a8:f0:2e:87:34
Apr/03/2018 04:33:01: dhcp6_get_options: get DHCP option identity association, len 40
Apr/03/2018 04:33:01:   IA_NA: ID=0, T1=92753, T2=148404
Apr/03/2018 04:33:01: copyin_option: get DHCP option IA address, len 24
Apr/03/2018 04:33:01: copyin_option:   IA_NA address: 2001:558:6045:61:e468:ec18:1892:322a pltime=185506 vltime=185506
Apr/03/2018 04:33:01: dhcp6_get_options: get DHCP option IA_PD, len 41
Apr/03/2018 04:33:01:   IA_PD: ID=0, T1=92755, T2=148408
Apr/03/2018 04:33:01: copyin_option: get DHCP option IA_PD prefix, len 25
Apr/03/2018 04:33:01: copyin_option:   IA_PD prefix: 2601:643:8400:ecf0::/60 pltime=185511 vltime=185511
Apr/03/2018 04:33:01: dhcp6_get_options: get DHCP option DNS, len 32
Apr/03/2018 04:33:01: client6_recvadvert: server ID: 00:01:00:01:1c:e3:7b:f8:40:a8:f0:2e:87:34, pref=-1
Apr/03/2018 04:33:01: client6_recvadvert: unexpected advertise
Apr/03/2018 04:33:01: client6_recvadvert: reset timer for eth0 to 0.939701
Apr/03/2018 04:33:02: select_server: picked a server (ID: 00:01:00:01:1c:e3:7b:f8:40:a8:f0:2e:87:34)
Apr/03/2018 04:33:02: client6_send: a new XID (3f1911) is generated
Apr/03/2018 04:33:02: copy_option: set client ID (len 14)
Apr/03/2018 04:33:02: copy_option: set server ID (len 14)
Apr/03/2018 04:33:02: copyout_option: set IA address
Apr/03/2018 04:33:02: copyout_option: set identity association
Apr/03/2018 04:33:02: copy_option: set elapsed time (len 2)
Apr/03/2018 04:33:02: copy_option: set option request (len 4)
Apr/03/2018 04:33:02: copyout_option: set IA_PD prefix
Apr/03/2018 04:33:02: copyout_option: set IA_PD
Apr/03/2018 04:33:02: client6_send: send request to ff02::1:2%eth0
Apr/03/2018 04:33:02: dhcp6_reset_timer: reset a timer on eth0, state=REQUEST, timeo=0, retrans=1063
Apr/03/2018 04:33:02: client6_recv: receive reply from fe80::201:5cff:fe6d:9c46%eth0 on eth0
Apr/03/2018 04:33:02: dhcp6_get_options: get DHCP option client ID, len 14
Apr/03/2018 04:33:02:   DUID: 00:01:06:00:5a:bd:e7:f2:04:18:d6:06:61:05
Apr/03/2018 04:33:02: dhcp6_get_options: get DHCP option server ID, len 14
Apr/03/2018 04:33:02:   DUID: 00:01:00:01:1c:e3:7b:f8:40:a8:f0:2e:87:34
Apr/03/2018 04:33:02: dhcp6_get_options: get DHCP option identity association, len 40
Apr/03/2018 04:33:02:   IA_NA: ID=0, T1=92752, T2=148404
Apr/03/2018 04:33:02: copyin_option: get DHCP option IA address, len 24
Apr/03/2018 04:33:02: copyin_option:   IA_NA address: 2001:558:6045:61:e468:ec18:1892:322a pltime=185505 vltime=185505
Apr/03/2018 04:33:02: dhcp6_get_options: get DHCP option IA_PD, len 41
Apr/03/2018 04:33:02:   IA_PD: ID=0, T1=92755, T2=148408
Apr/03/2018 04:33:02: copyin_option: get DHCP option IA_PD prefix, len 25
Apr/03/2018 04:33:02: copyin_option:   IA_PD prefix: 2601:643:8400:ecf0::/60 pltime=185510 vltime=185510
Apr/03/2018 04:33:02: dhcp6_get_options: get DHCP option DNS, len 32
Apr/03/2018 04:33:02: info_printf: nameserver[0] 2001:558:feed::1
Apr/03/2018 04:33:02: info_printf: nameserver[1] 2001:558:feed::2
Apr/03/2018 04:33:02: get_ia: make an IA: PD-0
Apr/03/2018 04:33:02: update_prefix: create a prefix 2601:643:8400:ecf0::/60 pltime=185510, vltime=185510
Apr/03/2018 04:33:02: ifaddrconf: add an address 2601:643:8400:ecf0::1/64 on switch0
Apr/03/2018 04:33:02: get_ia: make an IA: NA-0
Apr/03/2018 04:33:02: update_address: create an address 2001:558:6045:61:e468:ec18:1892:322a pltime=185505, vltime=185505
Apr/03/2018 04:33:02: ifaddrconf: add an address 2001:558:6045:61:e468:ec18:1892:322a/128 on eth0
Apr/03/2018 04:33:02: dhcp6_remove_event: removing an event on eth0, state=REQUEST
Apr/03/2018 04:33:02: dhcp6_remove_event: removing server (ID: 00:01:00:01:1c:e3:7b:f8:40:a8:f0:2e:87:34)
Apr/03/2018 04:33:02: client6_recvreply: executes /opt/vyatta/sbin/ubnt-dhcp6c-script
Apr/03/2018 04:33:05: client6_script: script "/opt/vyatta/sbin/ubnt-dhcp6c-script" terminated
Apr/03/2018 04:33:05: client6_recvreply: got an expected reply, sleeping.
^C

And to confirm:

$ ip -o -6 addr show dev eth0
4: eth0    inet6 2001:558:6045:61:e468:ec18:1892:322a/128 scope global \       valid_lft forever preferred_lft forever
4: eth0    inet6 fe80::618:d6ff:fe06:6105/64 scope link \       valid_lft forever preferred_lft forever

And just like that, we confirm that this kernel module isn't playing nicely with the IPv6 stack.

 

So here's where we're at:

 

  - In "disabled" mode, this just works for folks

  - In "enabled" mode, the RALink/MediaTek hw_nat kernel module is causing undesired behavior with IPv6 

    - If dhcp6c makes a request when the module is not loaded, it succeeds.

    - If dhcp6c makes a request when the module is loaded, it fails 

  - When dhcp6c makes a request it's state is different at boot time versus later in the boot.

  - The RALink/MediaTek hw_nat kernel module is difficult to track due to the fact that the MediaTek patches have been merged rather than being maintained as patch files.

 

...So, about those debug symbols.  Icon Wink

 

(Edited: Included content lost from taking too long to post and lost since the last AutoSave) 

Senior Member
Posts: 5,146
Registered: ‎01-04-2017
Kudos: 715
Solutions: 256

Re: Comcast IPv6 issues when hwnat enabled on ER-X

Member
Posts: 174
Registered: ‎05-24-2014
Kudos: 64
Solutions: 4

Re: Comcast IPv6 issues when hwnat enabled on ER-X

Yeah, I'm the one with the Spectrum issue in Los Angeles. 

 

Thanks @smyers119 for your help, but what an infuriating problem to track down! Ubiquiti should really add a note to their hwnat help article describing this problem. 

New Member
Posts: 10
Registered: ‎12-29-2016
Kudos: 7

Re: Comcast IPv6 issues when hwnat enabled on ER-X


@smyers119 wrote:

@UBNT-sandisnlooks like this is no longer just a comcast issue

 

Check here:

 

https://community.ubnt.com/t5/EdgeRouter/IPv6-on-Spectrum-with-EdgeRouter-and-Netgear-CM500-modem/m-...


Uh... It's already been reported multiple times that it's not just Comcast.  They really should update the thread title.

New Member
Posts: 31
Registered: ‎03-06-2017
Kudos: 2

Re: Comcast IPv6 issues when hwnat enabled on ER-X

I started this thread about a year ago. I gave up but always believed there was a soulution that

didn't involve Comcast. Looks like you have proved there is. Congratulations. I hope

Ubiquity is able to incorporate  a fix. Thanks to everyone involved.

New Member
Posts: 7
Registered: ‎08-04-2017

Re: Comcast IPv6 issues when hwnat enabled on ER-X

Just reporting that I am on Comcast w/ HWNAT enabled with an uptime of 1 week & 2 days with no IPV6 issues. I am located in Florida. We are DOCSIS 3.1 ready & my CMTS is a Cisco CBR. My cable modem is a NETGEAR CM600 hookup up to eth0 on my ERX.

 

09:09:35 up 8 days, 23:22,  1 user,  load average: 1.10, 1.11, 1.12

firewall {
    all-ping enable
    broadcast-ping disable
    group {
        network-group PROTECT_NETWORKS {
            description "Protected Networks"
            network 192.168.0.0/16
            network 172.16.0.0/12
            network 10.0.0.0/8
        }
        port-group DHCP/DNS {
            description DHCP/DNS
            port 53
            port 67
            port 546
        }
    }
    ipv6-name WANv6_IN {
        default-action drop
        description "WAN inbound traffic forwarded to LAN"
        enable-default-log
        rule 10 {
            action accept
            description "Allow established/related sessions"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
        rule 30 {
            action accept
            description "Allow IPv6 icmp"
            protocol ipv6-icmp
        }
    }
    ipv6-name WANv6_LOCAL {
        default-action drop
        description "WAN inbound traffic to the router"
        enable-default-log
        rule 10 {
            action accept
            description "Allow established/related sessions"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
        rule 30 {
            action accept
            description "Allow IPv6 icmp"
            protocol ipv6-icmp
        }
        rule 40 {
            action accept
            description "allow dhcpv6"
            destination {
                port 546
            }
            protocol udp
            source {
                port 547
            }
        }
    }
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians enable
    name BLOCK_IN {
        default-action accept
        description ""
        rule 1 {
            action accept
            description "Accept Established/Related"
            log disable
            protocol all
            state {
                established enable
                invalid disable
                new disable
                related enable
            }
        }
        rule 2 {
            action accept
            description "Allow DNS"
            destination {
                address 192.168.3.3
                port 53
            }
            log disable
            protocol tcp_udp
        }
        rule 3 {
            action drop
            description "Drop PROTECT_NETWORKS"
            destination {
                group {
                    network-group PROTECT_NETWORKS
                }
            }
            log disable
            protocol all
        }
    }
    name BLOCK_LOCAL {
        default-action drop
        description ""
        rule 1 {
            action accept
            description "Accept Established/Related"
            log disable
            protocol all
            state {
                established enable
                invalid disable
                new disable
                related enable
            }
        }
        rule 2 {
            action accept
            description "Accept DHCP/DNS"
            destination {
                group {
                    port-group DHCP/DNS
                }
            }
            log disable
            protocol tcp_udp
        }
    }
    name WAN_IN {
        default-action drop
        description "WAN to internal"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
    }
    name WAN_LOCAL {
        default-action drop
        description "WAN to router"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
    }
    receive-redirects disable
    send-redirects enable
    source-validation disable
    syn-cookies enable
}
interfaces {
    ethernet eth0 {
        address dhcp
        description Internet
        dhcp-options {
            default-route update
            default-route-distance 210
            name-server no-update
        }
        dhcpv6-pd {
            no-dns
            pd 0 {
                interface eth1 {
                    host-address ::1
                    no-dns
                    prefix-id :1
                    service slaac
                }
                interface eth1.10 {
                    host-address ::1
                    no-dns
                    prefix-id :4
                    service slaac
                }
                interface eth1.11 {
                    host-address ::1
                    no-dns
                    prefix-id :5
                    service slaac
                }
                interface eth2 {
                    host-address ::1
                    no-dns
                    prefix-id :3
                    service slaac
                }
                interface switch0 {
                    host-address ::1
                    no-dns
                    prefix-id :2
                    service slaac
                }
                prefix-length /60
            }
            rapid-commit enable
        }
        duplex auto
        firewall {
            in {
                ipv6-name WANv6_IN
                name WAN_IN
            }
            local {
                ipv6-name WANv6_LOCAL
                name WAN_LOCAL
            }
        }
        speed auto
    }
    ethernet eth1 {
        address 192.168.1.1/24
        description Lan
        duplex auto
        speed auto
        vif 10 {
            address 192.168.4.1/24
            description Gucci
            firewall {
                in {
                    name BLOCK_IN
                }
                local {
                    name BLOCK_LOCAL
                }
            }
            mtu 1500
        }
        vif 11 {
            address 192.168.5.1/24
            description Virus
            firewall {
                in {
                    name BLOCK_IN
                }
                local {
                    name BLOCK_LOCAL
                }
            }
            mtu 1500
        }
    }
    ethernet eth2 {
        address 192.168.3.1/24
        description Raspi
        duplex auto
        firewall {
            in {
                name BLOCK_IN
            }
            local {
                name BLOCK_LOCAL
            }
        }
        speed auto
    }
    ethernet eth3 {
        description Mj
        duplex auto
        speed auto
    }
    ethernet eth4 {
        description Xh
        duplex auto
        poe {
            output off
        }
        speed auto
    }
    loopback lo {
    }
    switch switch0 {
        address 192.168.2.1/24
        description Local
        firewall {
            in {
                name BLOCK_IN
            }
            local {
                name BLOCK_LOCAL
            }
        }
        mtu 1500
        switch-port {
            interface eth3 {
            }
            interface eth4 {
            }
            vlan-aware disable
        }
    }
}
service {
    dhcp-server {
        disabled false
        hostfile-update disable
        shared-network-name LAN1 {
            authoritative enable
            subnet 192.168.1.0/24 {
                default-router 192.168.1.1
                dns-server 192.168.3.3
                lease 86400
                start 192.168.1.38 {
                    stop 192.168.1.243
                }
                static-mapping Gucci {
                    ip-address 192.168.1.6
                    mac-address 
                }
                static-mapping Hackr {
                    ip-address 192.168.1.5
                    mac-address 
                }
                static-mapping RT-AC1900P-E8A0 {
                    ip-address 192.168.1.4
                    mac-address 
                }
                static-mapping Switch {
                    ip-address 192.168.1.2
                    mac-address 
                }
                static-mapping Virus {
                    ip-address 192.168.1.7
                    mac-address 
                }
                unifi-controller 192.168.3.3
            }
        }
        shared-network-name LAN2 {
            authoritative enable
            subnet 192.168.2.0/24 {
                default-router 192.168.2.1
                dns-server 192.168.3.3
                lease 86400
                start 192.168.2.38 {
                    stop 192.168.2.243
                }
            }
        }
        shared-network-name LAN3 {
            authoritative enable
            subnet 192.168.3.0/24 {
                default-router 192.168.3.1
                dns-server 192.168.3.3
                lease 86400
                start 192.168.3.38 {
                    stop 192.168.3.243
                }
                static-mapping raspberrypi {
                    ip-address 192.168.3.3
                    mac-address 
                }
            }
        }
        shared-network-name LAN4 {
            authoritative enable
            subnet 192.168.4.0/24 {
                default-router 192.168.4.1
                dns-server 192.168.3.3
                lease 86400
                start 192.168.4.38 {
                    stop 192.168.4.243
                }
            }
        }
        shared-network-name LAN5 {
            authoritative enable
            subnet 192.168.5.0/24 {
                default-router 192.168.5.1
                dns-server 192.168.3.3
                lease 86400
                start 192.168.5.38 {
                    stop 192.168.5.243
                }
            }
        }
        static-arp disable
        use-dnsmasq disable
    }
    dns {
        forwarding {
            cache-size 150
            listen-on eth1
            listen-on switch0
            listen-on eth1.10
            listen-on eth1.11
            listen-on eth2
        }
    }
    gui {
        http-port 80
        https-port 443
        older-ciphers enable
    }
    nat {
        rule 5010 {
            description "masquerade for WAN"
            outbound-interface eth0
            type masquerade
        }
    }
    ssh {
        port 22
        protocol-version v2
    }
    ubnt-discover {
        disable
    }
    ubnt-discover-server {
        disable
    }
    unms {
        disable
    }
}
system {
    host-name ubnt
    login {
        user  {
            authentication {
                encrypted-password 
            }
            level admin
        }
    }
    name-server 192.168.3.3
    name-server 
    ntp {
        server 0.ubnt.pool.ntp.org {
        }
        server 1.ubnt.pool.ntp.org {
        }
        server 2.ubnt.pool.ntp.org {
        }
        server 3.ubnt.pool.ntp.org {
        }
    }
    offload {
        hwnat enable
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone America/New_York
}


/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "config-management@1:conntrack@1:cron@1:dhcp-relay@1:dhcp-server@4:firewall@5:ipsec@5:nat@3:qos@1:quagga@2:system@4:ubnt-pptp@1:ubnt-udapi-server@1:ubnt-unms@1:ubnt-util@1:vrrp@1:webgui@1:webproxy@1:zone-policy@1" === */
/* Release version: v1.10.1.5067582.180305.1832 */

 

Highlighted
New Member
Posts: 31
Registered: ‎05-28-2017
Kudos: 12

Re: Comcast IPv6 issues when hwnat enabled on ER-X


@kifl wrote:

   It's already been reported multiple times that it's not just Comcast.  They really should update the thread title.


Yes, it's been reported with multiple ISPs, multiple router vendors, and multiple modem vendors. It DOES seem to be dependent on the head-end equipment and/or modem provisioning in different geographies, possibly using different or mis-matched standards.

 

 

New Member
Posts: 4
Registered: ‎04-15-2018
Kudos: 5

Re: Comcast IPv6 issues when hwnat enabled on ER-X

When I initially configured offloading my ipv6 settings were gone even after a couple of reboots. When I changed the prefix-length from /64 to /60 everything seemed to work again, no idea why.  I'm on Comcast in the SF Bay Area with a NETGEAR CM500-1AZNAS modem.

 

I have a much simpler configuration than @Taxed1, mine's basically out of the box, I'll post the relevant bits:

...
interfaces {
    ethernet eth0 {
        address dhcp
        description Internet
        dhcp-options {
            default-route update
            default-route-distance 210
            name-server no-update
        }
        dhcpv6-pd {
            no-dns
            pd 0 {
                interface switch0 {
                    host-address ::1
                    no-dns
                    service slaac
                }
                prefix-length /60
            }
            rapid-commit enable
        }
        duplex auto
        firewall {
            in {
                ipv6-name WANv6_IN
                name WAN_IN
            }
            local {
                ipv6-name WANv6_LOCAL
                name WAN_LOCAL
            }
        }
        speed auto
    }
...
    switch switch0 {
        address 192.168.1.1/24
        description Local
        mtu 1500
        switch-port {
            interface eth1 {
            }
            interface eth2 {
            }
            interface eth3 {
            }
            interface eth4 {
            }
            vlan-aware disable
        }
    }
}
...
    offload {
        hwnat enable
        ipsec enable
    }
...
New Member
Posts: 5
Registered: ‎01-18-2017
Kudos: 1

Re: Comcast IPv6 issues when hwnat enabled on ER-X

[ Edited ]

@beastie29a wrote:

When I initially configured offloading my ipv6 settings were gone even after a couple of reboots. When I changed the prefix-length from /64 to /60 everything seemed to work again, no idea why.  I'm on Comcast in the SF Bay Area with a NETGEAR CM500-1AZNAS modem.

 

I have a much simpler configuration than @Taxed1, mine's basically out of the box, I'll post the relevant bits:

...
interfaces {
    ethernet eth0 {
        address dhcp
        description Internet
        dhcp-options {
            default-route update
            default-route-distance 210
            name-server no-update
        }
        dhcpv6-pd {
            no-dns
            pd 0 {
                interface switch0 {
                    host-address ::1
                    no-dns
                    service slaac
                }
                prefix-length /60
            }
            rapid-commit enable
        }
        duplex auto
        firewall {
            in {
                ipv6-name WANv6_IN
                name WAN_IN
            }
            local {
                ipv6-name WANv6_LOCAL
                name WAN_LOCAL
            }
        }
        speed auto
    }
...
    switch switch0 {
        address 192.168.1.1/24
        description Local
        mtu 1500
        switch-port {
            interface eth1 {
            }
            interface eth2 {
            }
            interface eth3 {
            }
            interface eth4 {
            }
            vlan-aware disable
        }
    }
}
...
    offload {
        hwnat enable
        ipsec enable
    }
...

I'm on Comcast in the North Bay Area with an ARRIS 8200 modem. My ipv6 has always worked at /60 but after reading your post I tried /64 which did not work. Then I notice that my what my router is pulling

 

Router.png

 So with my router set to /60 (the only way it works) its pulling a /128? Either way hwnat has never worked for me.

Established Member
Posts: 1,672
Registered: ‎03-02-2016
Kudos: 380
Solutions: 126

Re: Comcast IPv6 issues when hwnat enabled on ER-X

The prefix size you request is the prefix that Comcast will assign you for your own internal needs. You care getting two /64s on your LAN interfaces, which is the standard. If you are requesting a /60, then Comcast is giving you 16 /64 prefixes for you to do with as you like.

The /128 assigned to your WAN interface is just a routing address to route between your /60 and Comcast.
New Member
Posts: 8
Registered: ‎04-21-2018

Re: Comcast IPv6 issues when hwnat enabled on ER-X

I checked a couple of months ago in Michigan and IPv6 did not work. Has something changed in the Michigan area that is now letting us to use Acceleration with IPv6 Comcast?

 

Or for residential use is it better to ditch IPv6 and go IPv4 for performance, 

 

Thanks,

Steve.

New Member
Posts: 20
Registered: ‎09-06-2015
Kudos: 39

Re: Comcast IPv6 issues when hwnat enabled on ER-X

As a followup to my previous work, I'm happy to contribute more but this requires somebody from Ubiquity responding acknowledging this....

 

Without debug symbols etc I'm squarely in a situation of being unable to fix the root cause.  That being said, I've found an acceptable breakfix which I'm classifying as "WORKSFORME".

 

After thinking about this during the week with a colleage he first tried pointing out "if it's an issue of the order the kernel modules are loading in, you can always force the ordering."  While that was a good idea it's less an issue of the order the modules are loading in and more the interaction between hw_nat.ko and dhcp6c, then it hit me.

 

Just make the hardware offloading module run later.

 

To do that, it was a matter of taking the following file:

 

$ cat /opt/vyatta/share/vyatta-cfg/templates/system/offload/node.def
help: Setting for hardware offload
priority: 999

and increasing the priority by 1 to 1000.

 

After doing that it will prioritize the loading of the hardware offloading kernel modules after dhcp6c (and a whole bunch of other stuff because it seems 999 became a dumping ground for being unwilling to make a decision of when to load configs.  The only thing "lower" in priority on my host is the wireguard bits built by @Lochnair).

 

After running with this change in place, rebooting, and watching my packet captures it seems to be working well enough.

New Member
Posts: 11
Registered: ‎11-20-2016
Kudos: 5

Re: Comcast IPv6 issues when hwnat enabled on ER-X

I tried this as well in two locations (One Comcast/One Charter) and it seems to be working in both cases.

 

Thanks!

New Member
Posts: 2
Registered: ‎09-11-2017

Re: Comcast IPv6 issues when hwnat enabled on ER-X

I've got from an hour after the workaround post to present on an ER-Lite-3 on Comcast SF Bay Area and have made it a couple hours on an EdgeRouter X on Comcast SF Bay Area and offload is behaving correctly on both with full IPv6 support. Both are running 1.10.1.

 

Thanks again @brianredbeard!

 

-Gene

Member
Posts: 174
Registered: ‎05-24-2014
Kudos: 64
Solutions: 4

Re: Comcast IPv6 issues when hwnat enabled on ER-X

It's awesome that the community seems to have devised a fix, but where is Ubiquiti on this matter.

 

And my worry is that this won't last through a prefix renewal request. If I understand correctly, the prefix is requested during bootup, then the offload module is loaded. But during the next prefix renewal request, if the EdgeRouter is not rebooted, then the offload module, which is now running, will break the request?

 

Has someone tried:

 

release dhcpv6-pd interface eth0
delete dhcpv6-pd duid
renew dhcpv6-pd interface eth0

 

New Member
Posts: 11
Registered: ‎11-20-2016
Kudos: 5

Re: Comcast IPv6 issues when hwnat enabled on ER-X

Unfortunately your suspicions turn out to be correct...the fix does not survive a renewal cycle.   I echo your sentiments that this seems to be something that UBNT should be able to address.

Reply