09-21-2014 05:54 AM
I've been using MAC address cloning on my ERL for about a year without an issue on Firmware 1.2. i.e.,
set interfaces ethernet eth0 MAC xx:xx:xx:xx:xx:xx
Yesterday, I updated to 1.5 and discovered a "bug."
When my ERL and my cable modem both boot at the same time, it appears that my ERL uses the built-in MAC address during the boot sequence. This causes my Cable Modem to lock to that MAC. Once the ERL finishes booting and starts using the new MAC address I've programmed, the Cable Modem refuses to recognize it. I can work around the bug by rebooting my cable modem a second time (or deleting the MAC clone).
This bug will seem to occur whenever both units boot together, i.e. after a power outage, or in my case, after a scheduled reboot.
01-06-2015 12:16 PM
Well I set up a quick back to back ERL setup with dhcp-server on router A and dhcp-client on router B with override mac address. Then I rebooted router B and did a packet capture on router A. Below are the 1st few packets we see:
No. Time Source Destination Protocol Info New Column 1 0.000000 :: ff02::16 ICMPv6 Multicast Listener Report Message v2 110 Frame 1 (110 bytes on wire, 110 bytes captured) Ethernet II, Src: 04:18:d6:31:95:6f (04:18:d6:31:95:6f), Dst: IPv6mcast_00:00:00:16 (33:33:00:00:00:16) Internet Protocol Version 6 Internet Control Message Protocol v6 No. Time Source Destination Protocol Info New Column 2 0.659961 :: ff02::1:ff31:956f ICMPv6 Neighbor solicitation 78 Frame 2 (78 bytes on wire, 78 bytes captured) Ethernet II, Src: 04:18:d6:31:95:6f (04:18:d6:31:95:6f), Dst: IPv6mcast_ff:31:95:6f (33:33:ff:31:95:6f) Internet Protocol Version 6 Internet Control Message Protocol v6 No. Time Source Destination Protocol Info New Column 3 1.679942 fe80::618:d6ff:fe31:956f ff02::16 ICMPv6 Multicast Listener Report Message v2 130 Frame 3 (130 bytes on wire, 130 bytes captured) Ethernet II, Src: 04:18:d6:31:95:6f (04:18:d6:31:95:6f), Dst: IPv6mcast_00:00:00:16 (33:33:00:00:00:16) Internet Protocol Version 6 Internet Control Message Protocol v6 No. Time Source Destination Protocol Info New Column 4 8.539951 :: ff02::16 ICMPv6 Multicast Listener Report Message v2 110 Frame 4 (110 bytes on wire, 110 bytes captured) Ethernet II, Src: 3com_03:04:05 (00:01:02:03:04:05), Dst: IPv6mcast_00:00:00:16 (33:33:00:00:00:16) Internet Protocol Version 6 Internet Control Message Protocol v6 No. Time Source Destination Protocol Info New Column 5 8.959945 :: ff02::1:ff03:405 ICMPv6 Neighbor solicitation 78 Frame 5 (78 bytes on wire, 78 bytes captured) Ethernet II, Src: 3com_03:04:05 (00:01:02:03:04:05), Dst: IPv6mcast_ff:03:04:05 (33:33:ff:03:04:05) Internet Protocol Version 6 Internet Control Message Protocol v6 No. Time Source Destination Protocol Info New Column 6 9.979932 fe80::201:2ff:fe03:405 ff02::16 ICMPv6 Multicast Listener Report Message v2 130 Frame 6 (130 bytes on wire, 130 bytes captured) Ethernet II, Src: 3com_03:04:05 (00:01:02:03:04:05), Dst: IPv6mcast_00:00:00:16 (33:33:00:00:00:16) Internet Protocol Version 6 Internet Control Message Protocol v6 No. Time Source Destination Protocol Info New Column 7 11.231290 0.0.0.0 255.255.255.255 DHCP DHCP Discover - Transaction ID 0x52de48f6 342 Frame 7 (342 bytes on wire, 342 bytes captured) Ethernet II, Src: 3com_03:04:05 (00:01:02:03:04:05), Dst: Broadcast (ff:ff:ff:ff:ff:ff) Internet Protocol User Datagram Protocol Bootstrap Protocol No. Time Source Destination Protocol Info New Column 8 11.231636 220.127.116.11 18.104.22.168 ICMP Echo (ping) request 62 Frame 8 (62 bytes on wire, 62 bytes captured) Ethernet II, Src: 24:a4:3c:3d:53:ac (24:a4:3c:3d:53:ac), Dst: 3com_03:04:05 (00:01:02:03:04:05) Internet Protocol Internet Control Message Protocol No. Time Source Destination Protocol Info New Column 9 12.232519 22.214.171.124 126.96.36.199 DHCP DHCP Offer - Transaction ID 0x52de48f6 342 Frame 9 (342 bytes on wire, 342 bytes captured) Ethernet II, Src: 24:a4:3c:3d:53:ac (24:a4:3c:3d:53:ac), Dst: 3com_03:04:05 (00:01:02:03:04:05) Internet Protocol User Datagram Protocol Bootstrap Protocol
The first 3 IPv6 packets have the original mac adress, but by frame 4 the new mac address is used. By frame 7 the dhcp request goes out and does use the new mac address. I don't know if those IPv6 packets would confuse you're modem, but it might be worth disabling IPv6 to see if the problem goes away.
configure set system ipv6 disable commit save exit reboot
Of course it would be better to do a packet capture on the wire between the modem & router to see if the modem is doing something different.
09-21-2014 09:16 AM
Could you clarify if the ERL is doing DHCP to get IP address from cable modem for example? Also could you confirm if you see the actual "lock" on the cable modem (e.g., see the original MAC associated with the router IP address in the modem's UI)?
09-21-2014 11:19 AM - edited 09-21-2014 11:20 AM
Yes, the ERL is set-up using DHCP with eth0 (internet).
I confirmed the lock with the lights on the cable modem.
I also replicated the issue about 15x on Firmware 1.2 (everything works) and then about 15x on Firmware 1.5 (never obtains a DHCP). I then deleted the MAC address clone and replicated that everything works again on Firmware 1.5.
Not sure, but I'm guessing that during the boot sequence, Firmware 1.5 must address the cable modem in some respect using its default MAC address, which the cable modem locks on to, and then when the ERL finally processes the command to update the MAC, the cable modem won't respond to the device with a different MAC than it expected. I can get around that by rebooting the cable modem by itself, but doing so may not be an option after a power failure.
Also, I have both the cable modem and ERL on a timer switch that powers them off for one minute each week. I discovered the problem when they kept booting up together each week and I'd have no internet until I rebooted the cable modem again (or removed the MAC assignment from the ERL).
09-22-2014 11:49 AM
It may help you to know that I'm on Time Warner Cable in Southern California, using a Motorola Surfboard modem.
If you have any trouble duplicating the issue, please feel free to IM me and I'll work with you.
10-14-2014 05:19 PM
I was looking at something similar. The question is, when does the ERL changes the MAC?
Cablemodems config files usually have 1 CPE set (customer premise equipment), and this looks to be the case with Advocate99.
I have 2 CMs with different tiers, one with 1 CPE, and another one with 2 CPE. That means that if the ERL request a DHCP to the CM with 1 CPE at boot time, and THEN the MAC change is applied, it won't get another IP. That is an expected scenario 'cause of the CM configuration file won't allow the ERL to request another IP.
If the CM has 2 CPE configured in the .cfg, after the MAC change on the ERL it will get a new DHCP IP. Tried this, and it works. Change the MAC on the fly, or add a pseudo-ethernet linked to the WAN eth, and you'll get the second IP for that MAC.
The problem resides on -when- does the MAC change takes place.
10-14-2014 05:25 PM
I think you're right. In earlier firmware versions (1.2 in my case), it happened so quickly that my Cable Modem never got the first MAC.
In any case, it needs to be fixed, as many people like me only get 1 IP...
10-15-2014 06:20 PM
Actually we did test this and could not replicate the issue (only the new MAC is used), which does match the implementation ordering (MAC is set first). Could you verify (e.g., packet capture) if for example the ER does send a DHCP request with the old MAC before using the new MAC? If so, maybe there's some corner case that is not covered yet and we'll need to find and address it.
10-15-2014 06:24 PM
I'd be happy to do that for you, but I'll need instructions on how to accomplish that.
10-16-2014 11:29 AM
One possibility is for example connect the router's WAN port to a PC, start packet capture on the PC and power up the router. Then check if the router sends DHCP requests using the original MAC.
10-16-2014 11:30 AM
I can do that.
Can you recommend an open source packet capture program that I could install on a machine running Windows 7?
10-31-2014 10:13 AM
I've used wireshark in the past. Given it's complexity, I doubt that I'm going to be able to find the DHCP request. How about if I do a capture and then send the details to you guys for analysis.
Also, isn't it possible that my cable modem is locking onto something that occurs prior to the DHCP request and which contains the old MAC??
10-31-2014 11:44 AM
It's even simpler to just use tcpdump from the ERL. Connecto to the ERL CLI and type:
sudo tcpdump -i eth# port 67 or port 68 -w dhcp.pcap -vvv
After the packets are captured, copy the .pcap to your PC (WinSCP for example) and open it with Wireshark.
10-31-2014 12:00 PM
I like the idea of using TCPDUMP from the ERL.
Remember, though that we need to capture what's happening during a bootup. Is there a way to get that command executed as the very first command upon bootup, so we can capture what's going on during the boot cycle?
11-05-2014 08:45 PM
I'm still waiting on instructions on how to get Wireshark to capture the information that you need, and how to get it to you..
Have you been able to duplicate this? It seems like it should be fairly easy to replicate, given that I was able to consistently replicate it after using the MAC Clone feature...
11-06-2014 09:06 AM
One more thing:
I don't have a vanilla installation. I'm running an OpenVPN site-to-site server and an L2TP server. Is there a way I can get my configuration file to UBNT so you can test it the same way that I'm running mine?
01-05-2015 04:25 PM - edited 01-05-2015 04:51 PM
I justed tested this again and the bug is definitely still present. If you have a cloned MAC address on eth0, and you boot your cable modem first and then your ERL, the ERL will never get internet access. If you delete the cloned mac address and use the default hardware address, the bug does not occur.
This time, I tested it by booting my cable modem completely, and then booting the ERL. If I have a cloned MAC, I am never able to get a DHCP. However, if I reverse the order with a cloned MAC, i.e. boot the ERL first and then the cable modem, everything works just fine.
If I remove the MAC clone, then everything works no matter what the order.
01-05-2015 04:59 PM
Hmm this is an area were we did thing a bit different from Vyatta. I added some echo statements in the node.def file and rebooted. Any output should go into /var/log/vyatta/vyatta-commit.log and I see:
[ interfaces ethernet eth1 ] about set mac [ interfaces ethernet eth1 address dhcp ] Starting DHCP client on eth1 ...
So it appears to be in the right order, but maybe it arp'ed first???
01-05-2015 06:23 PM - edited 01-05-2015 07:16 PM
I just want to clarify some things. I'm not sure why the issue is occurring. My assumption is that the issue is a DHCP and MAC issue, but it may be something else. Here's what I know:
1. The problem occurs if you set a custom MAC address while running firmware version 1.5, boot your cable modem (using an ISP that provides an IP address using DHCP and only allows one IP Address) FIRST and then boot the ERL. If you do that, your WAN port will never obtain a DHCP from your cable modem. If you reboot the cable modem, the WAN port will then be able to obtain a DHCP assignment from your cable modem.
2. The problem DOES NOT occur if you boot the ERL First and then the cable modem, or if you reboot the cable modem after the failure has manifested itself.
3. I haven't tested the issue with a static IP or a provider that allows multiple IP addresses (because that situation is not available to me).
4. The problem did not occur at all in firmware 1.2. I have not tested 1.3 or 1.4. When I was running 1.2, I often booted the ERL and the Cable Modem at the same time and never saw this issue. However, I never booted the cable modem completely and then booted the ERL when running 1.2.
5. The problem DOES occur in firmware 1.5. I first noticed this issue when I booted the ERL and Cable Modem at the same time and the failure occurred. Since then, I've run extensive tests booting the ERL first and the Cable Modem second, and the other way around.
My assumption is that you changed something between 1.2 and 1.5 relating to the boot sequence and that the non-custom MAC address is seen by the modem and locked on, so it ignores any contact from the other MAC. But that's only an assumption. However, if I delete the custom MAC and commit, I still am unable to obtain an IP address by DHCP. I also see a possibility that there's something going on related to the MAC address and DHCP that I simply don't understand. When I used 1.2, I never experienced the problem, but I always booted the modem AND the ERL at the same time. It may be that the boot sequence on 1.5 simpy takes longer and so the problem is manifesting itself for that reason.
I just completed some additional testing. I defaulted my ERL, used the wizard to configure ETH1 for internet, and ETH0/2 for internal using the default wizard settings. I then configured a static MAC of 00:08:5d:01:01:01. Then I shutdown the ERL. I powered off my Cable Modem. I powered up the cable modem and waited for it to boot completely. Then I booted the ERL. The ERL completed its boot but was never able to get an IP assignment from the cable modem. I then rebooted the cable modem and the ERL was able to get an assignment. So, it appears that there's nothing unique about my configuration that's causing the issue.
This is an important feature to me and many others, and given the nature of the failure, it's likely to cause a lot of headaches and support issues so I hope you'll figure out the issue and fix it.