EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)


  • Beta - EdgeRouter

    EDIT by UBNT-stig:  NOTE this rescue kit will only works on ER-Lite and ER-PoE.

    This forum topic is the home of the EdgeMax Rescue Kit—a network bootable rescue image for EdgeOS-based routers.

    Basically, it's a minimal Linux kernel with recovery tools embedded in initramfs. You connect to your router via serial, boot it via TFTP and get into a minimal rescue system.

    The following scripts for automated recovery procedures are included:

    1. emrk-factory-reset — removes the config.boot, optionally making a backup copy inside the config dir so you can access it easily after reboot.

    2. emrk-remove-user-data — removes any data from writable dirs, getting the system back to clean EdgeOS (all config data, user installed packages on so on are removed). It also removes copies of user data left after resets, as pressing reset button doesn't really remove the data but renames the old writable dir to some random name and creates a new one. Useful if you are going to give away your ERL to an untrusted third party.

    3. emrk-reinstall — downloads release tarball from URL and reinstalls EdgeOS from scratch .

    **NOTE: **emrk-remove-user-data and emrk-reinstall are highly disruprive! The former will remove everything but the stock EdgeOS image. The latter will remove and re-create all the partitions, so all your data, if you still have any, will be lost irrecoverably. Use it only if there is no hope to recover your installation in a less disruptive way.

    I did my best to get it to work, but I can't guarantee there are no bugs, so use at your own risk.

    How to use

    Download the latest version of the EMRK image from http://packages.vyos.net/tools/emrk/ (or use mirror director link: http://mirror.vyos.net/tools/emrk/)

    Install a TFTP server. On UNIX systems some TFTP implementation is always available from ports/repos, Windows users may use this one: http://tftpd32.jounin.net/

    Copy the image to your TFTP server directory.

    It's better to set up a DHCP server. If you are using ISC DHCPD, use these options in your subnet declaration:

    option tftp-server-name "x.x.x.x"; # Your TFTP server address
    next-server x.x.x.x; # Your TFTP server address
    option bootfile-name "emrk-0.9c.bin";
    filename "emrk-0.9c.bin
    

    Connect to your router via serial. Power on or reboot the router and, before EdgeOS kernel loads, press any key (I usually keep pressing some key repeatedly from the start). This will get you to the bootloader console where you can boot your router manually.

    Connect router eth0 interface to your network.

    If you use a DHCP server, issue the following command in U-Boot CLI:

    dhcp;tftpboot;bootoctlinux $loadaddr
    

    If you don't have a DHCP server, you may configure network manually:

    set ipaddr x.x.x.x (where x.x.x.x is desired router IP address)
    
    set netmask 255.255.255.0 (or whatever mask you want)
    
    set serverip y.y.y.y (where y.y.y.y is your TFTP server address)
    
    set bootfile <emrk image="" file="" name="">
    
    Optionally,
    set gatewayip z.z.z.z (where z.z.z.z is your default gateway)
    
    tftpboot
    bootoctlinux $loadaddr</emrk>
    

    If everything goes right, after bootoctlinux command you will see the kernel booting and will be asked if you want to continue after a scary disclaimer. Answer "yes" (case insensitive) or "no".

    Next you will be asked if you want to use DHCP or configure the network interface statically.

    To reinstall EdgeOS from scratch, you will need to download the tarball and put it on your HTTP/FTP/SSH server.

    Here is an example reinstall log (kernel boot messages omitted).

    Loading EMRK 0.9a
    Mounting filesystems
    Bringing up eth0
    
    Checking boot partition
    Boot partition looks intact
    Attempting to mount boot partition
    Boot partition successfully mounted
    Looking for kernel file
    Found a kernel
    Checking kernel MD5 sum file
    Found kernel MD5 sum file
    Checking kernel MD5 sum
    Kernel MD5 sum is correct
    
    Checking root partition
    Root partition looks intact
    Attempting to mount root partition
    kjournald starting.  Commit interval 5 seconds
    EXT3 FS on sda2, internal journal
    EXT3-fs: mounted filesystem with writeback data mode.
    Root partition successfully mounted
    Looking for system image file
    Found a system image file
    Checking system image MD5 sum file
    Found system image MD5 sum file
    Checking system image MD5 sum
    System image MD5 sum is correct
    
    **********************************************
    Welcome to EdgeMax Rescue Kit!
    
    This tool is distributed under the terms of
    GNU General Public License and other licenses
    
    Brought to you by SO3 Group
    
    WARNING: This tool is not developed, officially
    supported or endorsed by Ubiquiti Networks!
    
    Using it may lead to destroying your router
    configuration or operating system
    
    Ubiquiti Networks support will not help you
    with using it or fixing consequences of
    using it.
    
    This tool itself is distributed without any
    warranty and authors are not liable for
    any damage it may cause
    
    By using this tool you agree you are doing
    it at your own risk and understand what
    you are doing
    
    *********************************************
    
    Enter 'Yes' to proceed, 'No' to reboot
    yes or no: yes
    
    Do you want to configure network via DHCP?
    yes or no: yes
    udhcpc (v1.17.1) started
    Sending discover…
    Sending select for 10.91.19.175...
    Lease of 10.91.19.175 obtained, lease time 86400
    /usr/share/udhcpc/default.script: Resetting default routes
    route: SIOCDELRT: No such process
    /usr/share/udhcpc/default.script: Adding DNS 10.91.19.1
    
    EMRK provides some scripts for automated
    recovery procedures:
    
    emrk-factory-reset -- reset config to factory default
    emrk-remove-user-data -- remove all the user data including
        config and everything
    emrk-reinstall -- reinstall EdgeOS from scratch
        (wipes any user data too)
    
    Enter 'reboot' to reboot your router
    
    BusyBox v1.17.1 (Debian 1:1.17.1-8) built-in shell (ash)
    Enter 'help' for a list of built-in commands.
    
    /bin/ash: can't access tty; job control turned off
    EMRK>
    EMRK>emrk-reinstall 
    WARNING: This script will reinstall EdgeOS from scratch
    If you have any usable data on your router storage,
    it will be irrecoverably destroyed!
    Do you want to continue?
    yes or no: yes
    Unmounting boot partition
    Unmounting root partition
    Re-creating partition table
    Creating boot partition
    Formatting boot partition
    mkfs.vfat 3.0.9 (31 Jan 2010)
    Creating root partition
    Formatting root partition
    Mounting boot parition
    Mounting root partition
    kjournald starting.  Commit interval 5 seconds
    EXT3 FS on sda2, internal journal
    EXT3-fs: mounted filesystem with writeback data mode.
    Enter EdgeOS image url: [ftp://10.91.19.5/pub/ER-e100.v1.2.0.4574253.tar](ftp://10.91.19.5/pub/ER-e100.v1.2.0.4574253.tar)
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100 68.3M  100 68.3M    0     0  8319k      0  0:00:08  0:00:08 --:--:-- 6318k
    Unpacking EdgeOS release image
    Verifying EdgeOS kernel
    Copying EdgeOS kernel to boot partition
    Verifying EdgeOS system image
    Copying EdgeOS system image to root partition
    Creating EdgeOS writable data directory
    Cleaning up
    Installation finished
    Please reboot your router
    

    If you want to do things manually, there are busybox with symlinks to its applets (cat/grep/rm/cp/mv/tar/etc., ping, traceroute, nc etc.), curl, parted, and mkfs.ext3/mkfs.vfat.

    I used it on my on ERL when I replaced the USB stick with a 8GB one yesterday. Hope you find it useful, any bug reports or suggestions are welcome.

    Here are the scripts it uses: https://github.com/vyos/emrk


  • U
    Previous Employees

    Thanks for the effort and for including the warning! :icon_wink:


  • Beta - EdgeRouter

    This is pretty cool.  One of my test ERL's has been bricked for a while I hadn't gotten around to running our factory install script, so I gave your build a try instead.  It worked and rebooted fine.

    One thing you may want to add is the version file since the normal upgrade script looks for that.  Without that the next upgrade failed for me:

    ubnt@ubnt:~$ add system image upgrade-v1.2.0dev.4578432.130715.0102.tar
    Checking upgrade image… Done
    Preparing to upgrade... Done
    Copying upgrade image... /usr/bin/ubnt-upgrade: line 449: [: ==: unary operator expected
    Done
    Removing old image... Done
    Checking upgrade image... Done
    cp: can't stat '/root.dev/version': No such file or directory
    Finishing upgrade... Failed to move current image
    ubnt@ubnt:~$ 
    
    

    After the failed upgrade, I was left with an upbootable system.  So I used the rescue imagine again to recover and then did a "sudo touch /root.dev/version" so that the next upgrade would work.  I suppose we should be the upgrade script more robost so that it doesn't leave you with an unbootable system if the upgrade goes sideways.


  • L
    Beta Testers

    I might have to try this. My 8GB upgrade is not going so well. After the first dd the router booted, then after a reboot the router refused to boot from the drive ever again. Even after zeroing it out and starting dd over…

    [EDIT]

    This script didnt work for my 8GB drive. I'm starting to think either something is corrupted on it or the router just doesnt like the drive. I'm going to RMA the drive.


  • Beta - EdgeRouter

    Yeah, good point, I overlooked it. Just copy the version file to the root partition?


  • Beta - EdgeRouter

    linuxwarz: Hhm, interesting. What messages do you get from U-Boot when you connect via serial after running the reinstall script?

    Could you also boot to the rescue image once more and post "ls /mnt/root" and "ls /mnt/boot"?

    Some say there are incomptible USB sticks and it may be the case, but I want to make sure if it was not bug in the script.


  • L
    Beta Testers

    All of your answers are in the attached file. I wonder why I have DHCP issues in the bootloader too… your script works fine. I use the DHCP in the tftpd32 application.

    out.txt


  • Beta - EdgeRouter

    Interesting. To me it looks like the ERL doesn't like the flash drive, but the fact parted and mkfs didn't throw any errors confuses me. You may look at the bin/emrk-reinstall script from the github link above and reproduce the steps manually, maybe it will give some useful information.

    Also I updated the image, fixed the bug with missing version file. http://baturin.org/files/edgeos/emrk-0.9b.bin


  • Beta Testers

    Thanks for this it looks like It's unbricked my ERL :hurray:

    I also ran the command to fix the missing file, so hopefully an update won't re-brick it.


  • Beta - EdgeRouter

    With 0.9b the version file should get copied correctly already, so in the future it should not be a trouble. Would be nice if someone tried to install 1.1.0 and upgrade to 1.2.0 for verification. (:


  • N
    Beta - EdgeRouter

    notbad.jpg

    Will have to remember this for if/when I brick an EdgeMAX (so far I've been lucky! :P)


  • Super Users

    This makes me want to brick one of mine to test!!! I have been lucky with 80+ in the network and none failed yet minus the one that found lightning and let its smoke out :)


  • Beta Testers

    Magician wrote:

    This makes me want to brick one of mine to test!!! I have been lucky with 80+ in the network and none failed yet minus the one that found lightning and let its smoke out :)


    I've got a fix for that. https://www.sparkfun.com/products/retired/10622 :lol:

    http://youtu.be/Ex8oY24zycQ?t=8m42s


  • H
    Beta Testers

    Thanks for the great tutorial!

    Just wanted to say that I had no luck with TFTPD on windows 8 x64, it would start the connection but then never transfer any data and time out.

    But Open TFTP server worked first try to serve the file: http://sourceforge.net/projects/tftp-server/

    Because I didn't want to setup my own FTP/HTTP server for the final upload of tarball I just used the direct link to the current beta image: http://dl.ubnt.com/firmwares/edgemax/v1.3.0/ER-e100.v1.3.0beta1.4599951.tar
    Now it works again :)


  • Beta - EdgeRouter

    Hey guys,

    I've heard from Stig yesterday some folks couldn't download the image.

    We were having multiple problems with hardware, L3 switch, and DoS attacks yesterday. Now we resolved that,  EMRK images and other stuff we host should be available again.

    Sorry for the inconvenience.


  • H
    Beta Testers

    I am running version "emrk-0.9a.bin" instead of the newest "emrk-0.9c.bin"

    Are there any big changes that I should know about between them?


  • Beta - EdgeRouter

    hilo90mhz wrote:

    I am running version "emrk-0.9a.bin" instead of the newest "emrk-0.9c.bin"

    Are there any big changes that I should know about between them?


    There was an issue with the version file missing which would cause a problem the NEXT time you upgrade.  It should be ok if you do:

    sudo touch /root.dev/version
    

    Not sure what changed in version "c".


  • Beta - EdgeRouter

    0.9c has some more tools built-in, and some random bugfixes/enhancements to the scripts. Generally I'd recommend it over 0.9b as some bugs were easy to hit, AFAIR.

    I should start keeping proper changelogs I guess.


  • K
    Beta Testers

    I have a ER-Lite that failed the firmware upgrade from 1.3 to 1.4.  From the console port, it appears to boot but there are a ton on SQUASHFS errors that fly by about read failures.  These errors even occur after defaulting the unit.

    So I've tried running the rescue kit to reinstall the image but I'm running into an issue at the final boot step.  The emrk-0.9c.bin transfers fine from my TFTP server.  But when I type bootoctlinux $loadaddr, the boot process hangs at a point.  The attached pic shows where it hangs.

    Any thoughts?

    Will

    image.jpeg


  • U
    Previous Employees

    If it's stopping there, it could be a hardware issue, so please contact support@ubnt.com for help with verification and RMA if necessary.


recovery6 bricked4 rescue1 Posts 422Views 232
Log in to reply

Looks like your connection to Ubiquiti Networks Community was lost, please wait while we try to reconnect.