Reply
Regular Member
Posts: 725
Registered: ‎01-29-2014
Kudos: 289
Solutions: 33

Re: EdgeMAX EdgeRouter software version v1.9.7+hotfix.2 has been released!

Why are these hotfixes not going through beta forum first?  It would seem that hotfix 2 is incomplete - something which has been picked up within 12 hours of release, and would, presumably, have been found that fast in beta - by people who volunteer to take the risk of beta software.    GA is NOT the place to test fixes.  

Senior Member
Posts: 3,870
Registered: ‎05-15-2014
Kudos: 1379
Solutions: 265

Re: EdgeMAX EdgeRouter software version v1.9.7+hotfix.2 has been released!

I've tested the hotfix 2 fail-back wiht PPPoE (primary) and eth (failover) scenario. It's working. Tomorrow I'll test eth/eth scenario and will report back.

New Member
Posts: 5
Registered: ‎07-18-2017
Kudos: 1

Re: EdgeMAX EdgeRouter software version v1.9.7+hotfix.2 has been released!

[ Edited ]

I was thinking the exact same thing.


OzPHB wrote:

Why are these hotfixes not going through beta forum first?  It would seem that hotfix 2 is incomplete - something which has been picked up within 12 hours of release, and would, presumably, have been found that fast in beta - by people who volunteer to take the risk of beta software.    GA is NOT the place to test fixes.  


 

Senior Member
Posts: 2,599
Registered: ‎05-19-2013
Kudos: 1120
Solutions: 24

Re: EdgeMAX EdgeRouter software version v1.9.7+hotfix.2 has been released!


ethan_rose wrote:

I upgraded my ERPoe-5 to both v1.9.7+hotfix.1 and v1.9.7+hotfix.2 and both eventually stop responding to IPv6 connections (icmpv6 and others) from my monitoring platforms after a few hours despite no other config changes and icmpv6 allowed on the WAN interface. 

 

Anyone else seeing this behaviour?


@ethan_rose

 

I am not able to test this because I do not have a secondary site with IPv4 & IPv6 dual stack. Possible to PM u my IPv6 address and see if you are seeing the same issue?

New Member
Posts: 7
Registered: ‎02-20-2017

Re: EdgeMAX EdgeRouter software version v1.9.7+hotfix.2 has been released!

thanks for tips, tried all sorts, complains insuffcient space on 2x POE units.

Defaulted one and upgrade was ok

The other is up a huge hill in the rain so i tried harder!

 

Ran:

sudo apt-get autoclean
 
sudo du -hcd 2  / |more
 
sudo rm -R /var/lib/apt/lists
sudo rm -R /var/cache/apt/archives

 Upgraded via GUI ok wiyth config intact=Dry happy user, will monitor the LB now, cheers UBNT

Regular Member
Posts: 340
Registered: ‎06-04-2015
Kudos: 57
Solutions: 6

Re: EdgeMAX EdgeRouter software version v1.9.7+hotfix.2 has been released!

[ Edited ]

@ethan_rose There is a related post re IPv6 problem in the Edgemax ER4/6P forum. It is in the beta-2 thread.

 

EdgeRouter beta-2

Ubiquiti Employee
Posts: 1,145
Registered: ‎07-20-2015
Kudos: 1244
Solutions: 75

Re: EdgeMAX EdgeRouter software version v1.9.7+hotfix.2 has been released!

@torankusu
> I just upgraded my ERL to hotfix2 and LB for me still broken, actually i have LB with eth0/eth1 - eth1 is failover only

@UBNT-Fenng is investigating this issue right now

 

> If i restart my ERL while isp1 is not coming back up, the load balance config disappears on next reboot.
Could you please post LoadBalancing config that disappears after reboot?

 

@rrobinet
> The LB service messes up the route table when eth1 is plugged in by immediately setting the default route through the eth1 GW:
Is lb-local-metric-change enabled? Please show your load-balancing configuration.

 

@phillipmcmahon
> Could you update the release notes to qualify which elements of LB are fixed and which are still broken.
Yes, as soon as we find out which LoadBalancing scenario is not working.

 

> Will you future regression tests include these specific cases?
Until now LB regression testing included only eth/pppoe scenario. Now we will add more scenarios.

 

@OzPHB
> Why are these hotfixes not going through beta forum first?
We do alpha/beta releases before doing first major release which changes a lot of features.
But hotfix release is intended to fix particular issues ASAP (e.g. OpenVPN security bugs or LB regression). That's why doing beta testing for hotfixes would unnecessary lengthen release cycle.

 

Emerging Member
Posts: 47
Registered: ‎04-13-2017
Kudos: 14
Solutions: 1

Re: EdgeMAX EdgeRouter software version v1.9.7+hotfix.2 has been released!

I'd advise you to revisit this approach.

Hotfixes still require production level assurances. Multiple releases to fix the same issue isn't helpful to your customer bases and reveals a lack of in depth testing for the issue at hand.

I'd wager your user base would prefer stable and complete releases and hotfixes over multiple releases never quite hitting the mark and complicating release decisions and testing...
Emerging Member
Posts: 62
Registered: ‎08-09-2015
Kudos: 7
Solutions: 3

Re: EdgeMAX EdgeRouter software version v1.9.7+hotfix.2 has been released!

I volunteered to test new builds, so I accept the risk.
The range of configurations for your product is so wide that I doubt that your internal QA can cover many customer configurations,.
So sharing 'test builds' with willing customers like me is good practice in my opinion.
Emerging Member
Posts: 47
Registered: ‎04-13-2017
Kudos: 14
Solutions: 1

Re: EdgeMAX EdgeRouter software version v1.9.7+hotfix.2 has been released!

@rrobinet - through general release versions posted publically or through the beta forum?

There's a whole host of willing participants wanting to throw test code on their setups. Making a general release available should come with some assurances that code has been through more than a quick internal process.

Unless there is an external unauthenticated security issue affecting the firmware I see no reason to release without thorough testing including using the beta firmware process.

People pushing firmware to their production without proper testing is clearly outside of ubiquity's responsibility and unfortunately clouds the issue when these releases hit the public forum.
Emerging Member
Posts: 47
Registered: ‎04-13-2017
Kudos: 14
Solutions: 1

Re: EdgeMAX EdgeRouter software version v1.9.7+hotfix.2 has been released!

A quick addition to my last post. I love the products and also happy to provide testing on my setup when given the opportunity.

The intention isn't to knock Ubiquiti, expecting all production configuration issues to be found pre release is not realistic, however the quality of the 1.9.7 release branch appears to have dropped below the normal quality or previous releases.
New Member
Posts: 23
Registered: ‎10-20-2015
Kudos: 7

Re: EdgeMAX EdgeRouter software version v1.9.7+hotfix.2 has been released!

[ Edited ]

@UBNT-afomins

> If i restart my ERL while isp1 is not coming back up, the load balance config disappears on next reboot.
>>Could you please post LoadBalancing config that disappears after reboot?

 

Sure thing here it is:

 

load-balance {
    group G {
        interface eth0 {
            route-test {
                initial-delay 60
                interval 10
                type {
                    ping {
                        target 8.8.8.8
                    }
                }
            }
            weight 70
        }
        interface eth1 {
            failover-only
            route-test {
                initial-delay 60
                interval 10
                type {
                    ping {
                        target 208.67.220.220
                    }
                }
            }
            weight 30
        }
        lb-local enable
        lb-local-metric-change disable
        transition-script /config/scripts/wlb-transition

it gets wiped when i reboot since it never fails back to eth0. after reboot i have to restore config to get eth0-eth1 failover config back online even though it fails when eth0 goes offline and comes back up, as long as eth0 doesnt go offline the router keeps the loadbalance/failover config.

Highlighted
Senior Member
Posts: 3,870
Registered: ‎05-15-2014
Kudos: 1379
Solutions: 265

Re: EdgeMAX EdgeRouter software version v1.9.7+hotfix.2 has been released!

@UBNT-afomins

Just re-tested hotfix2 with eth/eth combo and it's still not failing back.

 

My tests:

  1. Primary PPPoE, fail-over eth ... fail-back now works
  2. Primary eth, fail-over eth ... fail-back does not work
Regular Member
Posts: 725
Registered: ‎01-29-2014
Kudos: 289
Solutions: 33

Re: EdgeMAX EdgeRouter software version v1.9.7+hotfix.2 has been released!

[ Edited ]

BranoB wrote:

@UBNT-afomins

Just re-tested hotfix2 with eth/eth combo and it's still not failing back.

 

Primary eth, fail-over eth ... fail-back does not work


@BranoB in your tests were the IPs on eth statically assigned, or dynamic? From earlier posts it seems lb may work ok if the ip is statically assigned, just not when assigned by DHCP?

Senior Member
Posts: 3,870
Registered: ‎05-15-2014
Kudos: 1379
Solutions: 265

Re: EdgeMAX EdgeRouter software version v1.9.7+hotfix.2 has been released!

With the eth/eth scenario, primary interface was DHCP and fail-over interface was STATIC.

 

However, in my test I've pulled out the COAX from cable modem and the primary interface never lost link/IP. The primary interface was always up, the IP stayed on, but the network was unreachable. ...I think this one of very common scenarios that needs to be tested. Pulling cable from WAN port is also valid scenario, but IMO less common occurrence in real life.

Established Member
Posts: 1,095
Registered: ‎10-31-2016
Kudos: 288
Solutions: 35

Re: EdgeMAX EdgeRouter software version v1.9.7+hotfix.2 has been released!

successful upgrade via UNMS running in AWS.

 

(no failover tests, this ERL supports two LAN interfaces...)

 

Controllers: UCK, UAS-XG, UCKG2, UCKG2+; Routers: USG-3P, USG-XG-8, ERL, ER-4, ER-6, EP-R6, ER-10X, ER-12; Switches: USW-8, USW-8-60W, USW-8-150W, USW-16-150W, USW-24-L2-500W, USW-48-L2-500W, USW-24-AT-250W, USW-48-AT-500W, USW-16-XG, TS-5, TS-8, ES-10XP, NanoSwitch; APs: AC-LR, AC-HD, AC-SHD, AC-Pro, AC-IW-Pro, HD-IW, AC-Mesh, nanoHD; UNMS: 0.13.0; AirMax: IsoStationM5, AirCubeISP, AirCubeAC; Video: NVR, UAS-XG, UCKG2+, G3, G3-Dome, G3-Micro, G3-Pro, G3-Flex; EtherMagic: EM-S, EM-EP; SolarBeam: SB-700-1; EoT: ULED-AT, ULED-AC, UDIM-AT; mFi: mPort, mFi-THS, mFi-CS, mFi-MPW, mPower-Pro, mPower-Mini
Ubiquiti Employee
Posts: 1,145
Registered: ‎07-20-2015
Kudos: 1244
Solutions: 75

Re: EdgeMAX EdgeRouter software version v1.9.7+hotfix.2 has been released!

@phillipmcmahon
> I'd advise you to revisit this approach.
> Hotfixes still require production level assurances. Multiple releases to fix the same issue isn't helpful to your customer bases and reveals a lack of in depth testing for the issue at hand.
I suppose you are right - from now on we will publish hotfixes on beta forum and few days later on main forum.

 

> however the quality of the 1.9.7 release branch appears to have dropped below the normal quality or previous releases.
I appologise for all regressions in 1.9.7 release. They were caused by QA refactoring that EdgeRouter is going through right now.

 

@BranoB
> Primary PPPoE, fail-over eth ... fail-back now works
> Primary eth, fail-over eth ... fail-back does not work
@UBNT-Feng found root cause - eth-to-eth failover does not work if link on failover interface is lost and then restored during single LB poll interval

 

@torankusu

> it gets wiped when i reboot since it never fails back to eth0. after reboot i have to restore config to get eth0-eth1 failover config

I was not able to reproduce situation when LB config disappears after reboot.

  1. Are you sure that you save your configuration after commit?
  2. Could you please show output of cat /var/log/vyatta/vyatta-commit.log when LB config is lost after reboot
Regular Member
Posts: 641
Registered: ‎01-16-2011
Kudos: 398
Solutions: 12

Re: EdgeMAX EdgeRouter software version v1.9.7+hotfix.2 has been released!

@UBNT-afomins

I think given the circumstances all of the devs on the ER side of things have been doing a great job. Given how different 1.9.7 is from previous releases, I think most of us are pretty appreciative at everything that is going on to try and improve the overall quality of the releases.

And for what its worth, I do like the hotfix releases since I can make a judgment call at how important the fixes are to apply right away vs waiting for the next major release.
New Member
Posts: 10
Registered: ‎06-26-2016

Re: EdgeMAX EdgeRouter software version v1.9.7+hotfix.2 has been released!

[ Edited ]

I can confirm eth0/eth1 failover does not work. Specifically, if one interface fails, and you have a flap on the second interface, you get stuck in a weird state where everything is getting dropped - permanently - *even when the interface comes back again*.

 

When this happens, both local and remote traffic seems dropped with 'no route':

 

root@ER-WDS-LON3:/home/noc# ip route show
default via 192.168.1.1 dev eth1  proto zebra 
192.168.1.0/24 dev eth1  proto kernel  scope link  src 192.168.1.39 
192.168.17.0/24 dev switch0  proto kernel  scope link  src 192.168.17.1 
root@ER-WDS-LON3:/home/noc# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
ping: sendmsg: Network is unreachable
ping: sendmsg: Network is unreachable
^Cping: sendmsg: Network is unreachable
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2010ms

root@ER-WDS-LON3:/home/noc# 

This is terrible, and caused already two outages since we upgraded to 1.9.7.

If you can suggest any steps that may help you narrow down the issue, I will try and go through them the next time it happens.

 

Saverio

 

Senior Member
Posts: 3,870
Registered: ‎05-15-2014
Kudos: 1379
Solutions: 265

Re: EdgeMAX EdgeRouter software version v1.9.7+hotfix.2 has been released!

[ Edited ]

UBNT-afomins wrote:

 

 

@BranoB
> Primary PPPoE, fail-over eth ... fail-back now works
> Primary eth, fail-over eth ... fail-back does not work
@UBNT-Feng found root cause - eth-to-eth failover does not work if link on failover interface is lost and then restored during single LB poll interval

 


@UBNT-afomins @UBNT-Fenng

In my test the link was never lost. I've pulled out coax from the cable modem, but the link between router and modem never went down. Yet the fail-back, upon plugging back coax, never happend. ... the root cause you've found maybe differnet from what I'm seeing.

 

That all said, this is all working fine in 1.9.7b2 ...see what changes you've made between 1.9.7b2 and 1.9.7 GA ...it should give you some clues.

Reply