Reply
Member
Posts: 264
Registered: ‎01-15-2011
Kudos: 275
Solutions: 2
Contributions: 2
Accepted Solution

MAC Address Cloning Bug in Firmware 1.5

I've been using MAC address cloning on my ERL for about a year without an issue on Firmware 1.2.  i.e.,

 

set interfaces ethernet eth0 MAC xx:xx:xx:xx:xx:xx

 

Yesterday, I updated to 1.5 and discovered a "bug."

 

When my ERL and my cable modem both boot at the same time, it appears that my ERL uses the built-in MAC address during the boot sequence.  This causes my Cable Modem to lock to that MAC.  Once the ERL finishes booting and starts using the new MAC address I've programmed, the Cable Modem refuses to recognize it.  I can work around the bug by rebooting my cable modem a second time (or deleting the MAC clone).

 

This bug will seem to occur whenever both units boot together, i.e. after a power outage, or in my case, after a scheduled reboot.

 

 

Other stuff I've written:

Prepaid Wireless FAQ: goo.gl/CkZcf
Straight Talk Wireless FAQ!: goo.gl/Sn0Pk

How to choose the best Wifi channel: goo.gl/0s3Rj

Installing FreePBX and Asterisk: goo.gl/tDs5U
Configuring FreePBX: goo.gl/omJnf

Accepted Solutions
Previous Employee
Posts: 10,504
Registered: ‎06-09-2011
Kudos: 3092
Solutions: 945
Contributions: 16

Re: MAC Address Cloning Bug in Firmware 1.5

Well I set up a quick back to back ERL setup with dhcp-server on router A and dhcp-client on router B with override mac address.  Then I rebooted router B and did a packet capture on router A.  Below are the 1st few packets we see:

 

No.     Time        Source                Destination           Protocol Info                                                            New Column
      1 0.000000    ::                    ff02::16              ICMPv6   Multicast Listener Report Message v2                            110

Frame 1 (110 bytes on wire, 110 bytes captured)
Ethernet II, Src: 04:18:d6:31:95:6f (04:18:d6:31:95:6f), Dst: IPv6mcast_00:00:00:16 (33:33:00:00:00:16)
Internet Protocol Version 6
Internet Control Message Protocol v6

No.     Time        Source                Destination           Protocol Info                                                            New Column
      2 0.659961    ::                    ff02::1:ff31:956f     ICMPv6   Neighbor solicitation                                           78

Frame 2 (78 bytes on wire, 78 bytes captured)
Ethernet II, Src: 04:18:d6:31:95:6f (04:18:d6:31:95:6f), Dst: IPv6mcast_ff:31:95:6f (33:33:ff:31:95:6f)
Internet Protocol Version 6
Internet Control Message Protocol v6

No.     Time        Source                Destination           Protocol Info                                                            New Column
      3 1.679942    fe80::618:d6ff:fe31:956f ff02::16              ICMPv6   Multicast Listener Report Message v2                            130

Frame 3 (130 bytes on wire, 130 bytes captured)
Ethernet II, Src: 04:18:d6:31:95:6f (04:18:d6:31:95:6f), Dst: IPv6mcast_00:00:00:16 (33:33:00:00:00:16)
Internet Protocol Version 6
Internet Control Message Protocol v6

No.     Time        Source                Destination           Protocol Info                                                            New Column
      4 8.539951    ::                    ff02::16              ICMPv6   Multicast Listener Report Message v2                            110

Frame 4 (110 bytes on wire, 110 bytes captured)
Ethernet II, Src: 3com_03:04:05 (00:01:02:03:04:05), Dst: IPv6mcast_00:00:00:16 (33:33:00:00:00:16)
Internet Protocol Version 6
Internet Control Message Protocol v6

No.     Time        Source                Destination           Protocol Info                                                            New Column
      5 8.959945    ::                    ff02::1:ff03:405      ICMPv6   Neighbor solicitation                                           78

Frame 5 (78 bytes on wire, 78 bytes captured)
Ethernet II, Src: 3com_03:04:05 (00:01:02:03:04:05), Dst: IPv6mcast_ff:03:04:05 (33:33:ff:03:04:05)
Internet Protocol Version 6
Internet Control Message Protocol v6

No.     Time        Source                Destination           Protocol Info                                                            New Column
      6 9.979932    fe80::201:2ff:fe03:405 ff02::16              ICMPv6   Multicast Listener Report Message v2                            130

Frame 6 (130 bytes on wire, 130 bytes captured)
Ethernet II, Src: 3com_03:04:05 (00:01:02:03:04:05), Dst: IPv6mcast_00:00:00:16 (33:33:00:00:00:16)
Internet Protocol Version 6
Internet Control Message Protocol v6

No.     Time        Source                Destination           Protocol Info                                                            New Column
      7 11.231290   0.0.0.0               255.255.255.255       DHCP     DHCP Discover - Transaction ID 0x52de48f6                       342

Frame 7 (342 bytes on wire, 342 bytes captured)
Ethernet II, Src: 3com_03:04:05 (00:01:02:03:04:05), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
Internet Protocol
User Datagram Protocol
Bootstrap Protocol

No.     Time        Source                Destination           Protocol Info                                                            New Column
      8 11.231636   77.0.0.1              77.0.0.14             ICMP     Echo (ping) request                                             62

Frame 8 (62 bytes on wire, 62 bytes captured)
Ethernet II, Src: 24:a4:3c:3d:53:ac (24:a4:3c:3d:53:ac), Dst: 3com_03:04:05 (00:01:02:03:04:05)
Internet Protocol
Internet Control Message Protocol

No.     Time        Source                Destination           Protocol Info                                                            New Column
      9 12.232519   77.0.0.1              77.0.0.14             DHCP     DHCP Offer    - Transaction ID 0x52de48f6                       342

Frame 9 (342 bytes on wire, 342 bytes captured)
Ethernet II, Src: 24:a4:3c:3d:53:ac (24:a4:3c:3d:53:ac), Dst: 3com_03:04:05 (00:01:02:03:04:05)
Internet Protocol
User Datagram Protocol
Bootstrap Protocol

 The first 3 IPv6 packets have the original mac adress, but by frame 4 the new mac address is used.  By frame 7 the dhcp request goes out and does use the new mac address.  I don't know if those IPv6 packets would confuse you're modem, but it might be worth disabling IPv6 to see if the problem goes away.

configure
set system ipv6 disable
commit
save
exit

reboot

 Of course it would be better to do a packet capture on the wire between the modem & router to see if the modem is doing something different.

EdgeMAX Router Software Development

View solution in original post


All Replies
Previous Employee
Posts: 13,551
Registered: ‎06-10-2011
Kudos: 5429
Solutions: 1656
Contributions: 2

Re: MAC Address Cloning Bug in Firmware 1.5

Could you clarify if the ERL is doing DHCP to get IP address from cable modem for example? Also could you confirm if you see the actual "lock" on the cable modem (e.g., see the original MAC associated with the router IP address in the modem's UI)?

Member
Posts: 264
Registered: ‎01-15-2011
Kudos: 275
Solutions: 2
Contributions: 2

Re: MAC Address Cloning Bug in Firmware 1.5

[ Edited ]

Yes, the ERL is set-up using DHCP with eth0 (internet).

 

I confirmed the lock with the lights on the cable modem.

 

I also replicated the issue about 15x on Firmware 1.2 (everything works) and then about 15x on Firmware 1.5 (never obtains a DHCP).  I then deleted the MAC address clone and replicated that everything works again on Firmware 1.5.

 

Not sure, but I'm guessing that during the boot sequence, Firmware 1.5 must address the cable modem in some respect using its default MAC address, which the cable modem locks on to, and then when the ERL finally processes the command to update the MAC, the cable modem won't respond to the device with a different MAC than it expected.  I can get around that by rebooting the cable modem by itself, but doing so may not be an option after a power failure.  

 

Also, I have both the cable modem and ERL on a timer switch that powers them off for one minute each week. I discovered the problem when they kept booting up together each week and I'd have no internet until I rebooted the cable modem again (or removed the MAC assignment from the ERL).

Other stuff I've written:

Prepaid Wireless FAQ: goo.gl/CkZcf
Straight Talk Wireless FAQ!: goo.gl/Sn0Pk

How to choose the best Wifi channel: goo.gl/0s3Rj

Installing FreePBX and Asterisk: goo.gl/tDs5U
Configuring FreePBX: goo.gl/omJnf
Previous Employee
Posts: 13,551
Registered: ‎06-10-2011
Kudos: 5429
Solutions: 1656
Contributions: 2

Re: MAC Address Cloning Bug in Firmware 1.5

OK we'll need to do some testing to try and replicate the issue. Thanks for reporting this.

Member
Posts: 264
Registered: ‎01-15-2011
Kudos: 275
Solutions: 2
Contributions: 2

Re: MAC Address Cloning Bug in Firmware 1.5

It may help you to know that I'm on Time Warner Cable in Southern California, using a Motorola Surfboard modem.

 

If you have any trouble duplicating the issue, please feel free to IM me and I'll work with you.

Other stuff I've written:

Prepaid Wireless FAQ: goo.gl/CkZcf
Straight Talk Wireless FAQ!: goo.gl/Sn0Pk

How to choose the best Wifi channel: goo.gl/0s3Rj

Installing FreePBX and Asterisk: goo.gl/tDs5U
Configuring FreePBX: goo.gl/omJnf
Emerging Member
Posts: 64
Registered: ‎09-21-2014
Kudos: 5

Re: MAC Address Cloning Bug in Firmware 1.5

 

I was looking at something similar. The question is, when does the ERL changes the MAC?

 

Cablemodems config files usually have 1 CPE set (customer premise equipment), and this looks to be the case with Advocate99.

 

I have 2 CMs with different tiers, one with 1 CPE, and another one with 2 CPE. That means that if the ERL request a DHCP to the CM with 1 CPE at boot time, and THEN the MAC change is applied, it won't get another IP. That is an expected scenario 'cause of the CM configuration file won't allow the ERL to request another IP.

 

If the CM has 2 CPE configured in the .cfg, after the MAC change on the ERL it will get a new DHCP IP. Tried this, and it works. Change the MAC on the fly, or add a pseudo-ethernet linked to the WAN eth, and you'll get the second IP for that MAC.

 

The problem resides on -when- does the MAC change takes place.

Member
Posts: 264
Registered: ‎01-15-2011
Kudos: 275
Solutions: 2
Contributions: 2

Re: MAC Address Cloning Bug in Firmware 1.5

I think you're right.  In earlier firmware versions (1.2 in my case), it happened so quickly that my Cable Modem never got the first MAC.

 

In any case, it needs to be fixed, as many people like me only get 1 IP...

Other stuff I've written:

Prepaid Wireless FAQ: goo.gl/CkZcf
Straight Talk Wireless FAQ!: goo.gl/Sn0Pk

How to choose the best Wifi channel: goo.gl/0s3Rj

Installing FreePBX and Asterisk: goo.gl/tDs5U
Configuring FreePBX: goo.gl/omJnf
Previous Employee
Posts: 13,551
Registered: ‎06-10-2011
Kudos: 5429
Solutions: 1656
Contributions: 2

Re: MAC Address Cloning Bug in Firmware 1.5

Actually we did test this and could not replicate the issue (only the new MAC is used), which does match the implementation ordering (MAC is set first). Could you verify (e.g., packet capture) if for example the ER does send a DHCP request with the old MAC before using the new MAC? If so, maybe there's some corner case that is not covered yet and we'll need to find and address it.

Member
Posts: 264
Registered: ‎01-15-2011
Kudos: 275
Solutions: 2
Contributions: 2

Re: MAC Address Cloning Bug in Firmware 1.5

I'd be happy to do that for you, but I'll need instructions on how to accomplish that.

Other stuff I've written:

Prepaid Wireless FAQ: goo.gl/CkZcf
Straight Talk Wireless FAQ!: goo.gl/Sn0Pk

How to choose the best Wifi channel: goo.gl/0s3Rj

Installing FreePBX and Asterisk: goo.gl/tDs5U
Configuring FreePBX: goo.gl/omJnf
Previous Employee
Posts: 13,551
Registered: ‎06-10-2011
Kudos: 5429
Solutions: 1656
Contributions: 2

Re: MAC Address Cloning Bug in Firmware 1.5

One possibility is for example connect the router's WAN port to a PC, start packet capture on the PC and power up the router. Then check if the router sends DHCP requests using the original MAC.

Member
Posts: 264
Registered: ‎01-15-2011
Kudos: 275
Solutions: 2
Contributions: 2

Re: MAC Address Cloning Bug in Firmware 1.5

I can do that.  

 

Can you recommend an open source packet capture program that I could install on a machine running Windows 7?

Other stuff I've written:

Prepaid Wireless FAQ: goo.gl/CkZcf
Straight Talk Wireless FAQ!: goo.gl/Sn0Pk

How to choose the best Wifi channel: goo.gl/0s3Rj

Installing FreePBX and Asterisk: goo.gl/tDs5U
Configuring FreePBX: goo.gl/omJnf
Previous Employee
Posts: 13,551
Registered: ‎06-10-2011
Kudos: 5429
Solutions: 1656
Contributions: 2

Re: MAC Address Cloning Bug in Firmware 1.5

Yeah Wireshark is probably the most commonly used one.

Member
Posts: 264
Registered: ‎01-15-2011
Kudos: 275
Solutions: 2
Contributions: 2

Re: MAC Address Cloning Bug in Firmware 1.5

I've used wireshark in the past.  Given it's complexity, I doubt that I'm going to be able to find the DHCP request.  How about if I do a capture and then send the details to you guys for analysis.

 

Also, isn't it possible that my cable modem is locking onto something that occurs prior to the DHCP request and which contains the old MAC??

Other stuff I've written:

Prepaid Wireless FAQ: goo.gl/CkZcf
Straight Talk Wireless FAQ!: goo.gl/Sn0Pk

How to choose the best Wifi channel: goo.gl/0s3Rj

Installing FreePBX and Asterisk: goo.gl/tDs5U
Configuring FreePBX: goo.gl/omJnf
Emerging Member
Posts: 64
Registered: ‎09-21-2014
Kudos: 5

Re: MAC Address Cloning Bug in Firmware 1.5

Capture with Wireshark and filter port 67 and 68, protocol UDP. It's simple.

It's even simpler to just use tcpdump from the ERL. Connecto to the ERL CLI and type:

sudo tcpdump -i eth# port 67 or port 68 -w dhcp.pcap -vvv

After the packets are captured, copy the .pcap to your PC (WinSCP for example) and open it with Wireshark.
Member
Posts: 264
Registered: ‎01-15-2011
Kudos: 275
Solutions: 2
Contributions: 2

Re: MAC Address Cloning Bug in Firmware 1.5

I like the idea of using TCPDUMP from the ERL.  

 

Remember, though that we need to capture what's happening during a bootup.  Is there a way to get that command executed as the very first command upon bootup, so we can capture what's going on during the boot cycle?

Other stuff I've written:

Prepaid Wireless FAQ: goo.gl/CkZcf
Straight Talk Wireless FAQ!: goo.gl/Sn0Pk

How to choose the best Wifi channel: goo.gl/0s3Rj

Installing FreePBX and Asterisk: goo.gl/tDs5U
Configuring FreePBX: goo.gl/omJnf
Member
Posts: 264
Registered: ‎01-15-2011
Kudos: 275
Solutions: 2
Contributions: 2

Re: MAC Address Cloning Bug in Firmware 1.5

I'm still waiting on instructions on how to get Wireshark to capture the information that you need, and how to get it to you..  

 

Have you been able to duplicate this?  It seems like it should be fairly easy to replicate, given that I was able to consistently replicate it after using the MAC Clone feature...

Other stuff I've written:

Prepaid Wireless FAQ: goo.gl/CkZcf
Straight Talk Wireless FAQ!: goo.gl/Sn0Pk

How to choose the best Wifi channel: goo.gl/0s3Rj

Installing FreePBX and Asterisk: goo.gl/tDs5U
Configuring FreePBX: goo.gl/omJnf
Member
Posts: 264
Registered: ‎01-15-2011
Kudos: 275
Solutions: 2
Contributions: 2

Re: MAC Address Cloning Bug in Firmware 1.5

One more thing:

 

I don't have a vanilla installation.  I'm running an OpenVPN site-to-site server and an L2TP server.  Is there a way I can get my configuration file to UBNT so you can test it the same way that I'm running mine?

Other stuff I've written:

Prepaid Wireless FAQ: goo.gl/CkZcf
Straight Talk Wireless FAQ!: goo.gl/Sn0Pk

How to choose the best Wifi channel: goo.gl/0s3Rj

Installing FreePBX and Asterisk: goo.gl/tDs5U
Configuring FreePBX: goo.gl/omJnf
Member
Posts: 264
Registered: ‎01-15-2011
Kudos: 275
Solutions: 2
Contributions: 2

Re: MAC Address Cloning Bug in Firmware 1.5

[ Edited ]

I justed tested this again and the bug is definitely still present.  If you have a cloned MAC address on eth0, and you boot your cable modem first and then your ERL, the ERL will never get internet access.  If you delete the cloned mac address and use the default hardware address, the bug does not occur.

 

This time, I tested it by booting my cable modem completely, and then booting the ERL.  If I have a cloned MAC, I am never able to get a DHCP.  However, if I reverse the order with a cloned MAC, i.e. boot the ERL first and then the cable modem, everything works just fine.  

 

If I remove the MAC clone, then everything works no matter what the order.

Other stuff I've written:

Prepaid Wireless FAQ: goo.gl/CkZcf
Straight Talk Wireless FAQ!: goo.gl/Sn0Pk

How to choose the best Wifi channel: goo.gl/0s3Rj

Installing FreePBX and Asterisk: goo.gl/tDs5U
Configuring FreePBX: goo.gl/omJnf
Previous Employee
Posts: 10,504
Registered: ‎06-09-2011
Kudos: 3092
Solutions: 945
Contributions: 16

Re: MAC Address Cloning Bug in Firmware 1.5

Hmm this is an area were we did thing a bit different from Vyatta.  I added some echo statements in the node.def file and rebooted.  Any output should go into /var/log/vyatta/vyatta-commit.log and I see:

 

[ interfaces ethernet eth1 ]
about set mac

[ interfaces ethernet eth1 address dhcp ]
Starting DHCP client on eth1 ...

 So it appears to be in the right order, but maybe it arp'ed first???

EdgeMAX Router Software Development
Member
Posts: 264
Registered: ‎01-15-2011
Kudos: 275
Solutions: 2
Contributions: 2

Re: MAC Address Cloning Bug in Firmware 1.5

[ Edited ]

I just want to clarify some things.  I'm not sure why the issue is occurring.  My assumption is that the issue is a DHCP and MAC issue, but it may be something else.  Here's what I know:

 

1.  The problem occurs if you set a custom MAC address while running firmware version 1.5, boot your cable modem (using an ISP that provides an IP address using DHCP and only allows one IP Address) FIRST and then boot the ERL.  If you do that, your WAN port will never obtain a DHCP from your cable modem.  If you reboot the cable modem, the WAN port will then be able to obtain a DHCP assignment from your cable modem.

 

2.  The problem DOES NOT occur if you boot the ERL First and then the cable modem, or if you reboot the cable modem after the failure has manifested itself.

 

3.  I haven't tested the issue with a static IP or a provider that allows multiple IP addresses (because that situation is not available to me).

 

4.  The problem did not occur at all in firmware 1.2.  I have not tested 1.3 or 1.4.  When I was running 1.2, I often booted the ERL and the Cable Modem at the same time and never saw this issue.  However, I never booted the cable modem completely and then booted the ERL when running 1.2.

 

5.  The problem DOES occur in firmware 1.5.  I first noticed this issue when I booted the ERL and Cable Modem at the same time and the failure occurred.  Since then, I've run extensive tests booting the ERL first and the Cable Modem second, and the other way around.


My assumption is that you changed something between 1.2 and 1.5 relating to the boot sequence and that the non-custom MAC address is seen by the modem and locked on, so it ignores any contact from the other MAC.  But that's only an assumption. However, if I delete the custom MAC and commit, I still am unable to obtain an IP address by DHCP.  I also see a possibility that there's something going on related to the MAC address and DHCP that I simply don't understand.  When I used 1.2, I never experienced the problem, but I always booted the modem AND the ERL at the same time.   It may be that the boot sequence on 1.5 simpy takes longer and so the problem is manifesting itself for that reason. 

 

I just completed some additional testing.  I defaulted my ERL, used the wizard to configure ETH1 for internet, and ETH0/2 for internal using the default wizard settings.  I then configured a static MAC of 00:08:5d:01:01:01.  Then I shutdown the ERL.  I powered off my Cable Modem.  I powered up the cable modem and waited for it to boot completely.  Then I booted the ERL.  The ERL completed its boot but was never able to get an IP assignment from the cable modem.  I then rebooted the cable modem and the ERL was able to get an assignment.  So, it appears that there's nothing unique about my configuration that's causing the issue.

 

This is an important feature to me and many others, and given the nature of the failure, it's likely to cause a lot of headaches and support issues so I hope you'll figure out the issue and fix it.  Man Happy

Other stuff I've written:

Prepaid Wireless FAQ: goo.gl/CkZcf
Straight Talk Wireless FAQ!: goo.gl/Sn0Pk

How to choose the best Wifi channel: goo.gl/0s3Rj

Installing FreePBX and Asterisk: goo.gl/tDs5U
Configuring FreePBX: goo.gl/omJnf
Reply