Upcoming Maintenance Alert:

The UBNT Community will be upgraded at 5pm MDT on April 25th. During this time the community forums will be set to read-only status.

Learn more

×
Reply
Member
Posts: 209
Registered: ‎05-21-2013
Kudos: 256
Solutions: 8
Contributions: 3

EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)

[ Edited ]

EDIT by UBNT-stig:  NOTE this rescue kit will only works on ER-Lite and ER-PoE.

 

This forum topic is the home of the EdgeMax Rescue Kit—a network bootable rescue image for EdgeOS-based routers.

 

Basically, it's a minimal Linux kernel with recovery tools embedded in initramfs. You connect to your router via serial, boot it via TFTP and get into a minimal rescue system.

 

The following scripts for automated recovery procedures are included:

  1. emrk-factory-reset — removes the config.boot, optionally making a backup copy inside the config dir so you can access it easily after reboot.
  2. emrk-remove-user-data — removes any data from writable dirs, getting the system back to clean EdgeOS (all config data, user installed packages on so on are removed). It also removes copies of user data left after resets, as pressing reset button doesn't really remove the data but renames the old writable dir to some random name and creates a new one. Useful if you are going to give away your ERL to an untrusted third party.
  3. emrk-reinstall — downloads release tarball from URL and reinstalls EdgeOS from scratch .

NOTE: emrk-remove-user-data and emrk-reinstall are highly disruprive! The former will remove everything but the stock EdgeOS image. The latter will remove and re-create all the partitions, so all your data, if you still have any, will be lost irrecoverably. Use it only if there is no hope to recover your installation in a less disruptive way.

 

I did my best to get it to work, but I can't guarantee there are no bugs, so use at your own risk.

 

How to use

Download the latest version of the EMRK image from http://packages.vyos.net/tools/emrk/ (or use mirror director link: http://mirror.vyos.net/tools/emrk/)

Install a TFTP server. On UNIX systems some TFTP implementation is always available from ports/repos, Windows users may use this one: http://tftpd32.jounin.net/

Copy the image to your TFTP server directory.

 

It's better to set up a DHCP server. If you are using ISC DHCPD, use these options in your subnet declaration:

option tftp-server-name "x.x.x.x"; # Your TFTP server address
next-server x.x.x.x; # Your TFTP server address
option bootfile-name "emrk-0.9c.bin";
filename "emrk-0.9c.bin

Connect to your router via serial. Power on or reboot the router and, before EdgeOS kernel loads, press any key (I usually keep pressing some key repeatedly from the start). This will get you to the bootloader console where you can boot your router manually.

 

Connect router eth0 interface to your network.

 

If you use a DHCP server, issue the following command in U-Boot CLI:

dhcp;tftpboot;bootoctlinux $loadaddr

 If you don't have a DHCP server, you may configure network manually:

set ipaddr x.x.x.x (where x.x.x.x is desired router IP address)

set netmask 255.255.255.0 (or whatever mask you want)

set serverip y.y.y.y (where y.y.y.y is your TFTP server address)

set bootfile <EMRK image file name>

Optionally,
set gatewayip z.z.z.z (where z.z.z.z is your default gateway)

tftpboot
bootoctlinux $loadaddr

 

If everything goes right, after bootoctlinux command you will see the kernel booting and will be asked if you want to continue after a scary disclaimer. Answer "yes" (case insensitive) or "no".

Next you will be asked if you want to use DHCP or configure the network interface statically.

 

To reinstall EdgeOS from scratch, you will need to download the tarball and put it on your HTTP/FTP/SSH server.

 

Here is an example reinstall log (kernel boot messages omitted).

Loading EMRK 0.9a
Mounting filesystems
Bringing up eth0

Checking boot partition
Boot partition looks intact
Attempting to mount boot partition
Boot partition successfully mounted
Looking for kernel file
Found a kernel
Checking kernel MD5 sum file
Found kernel MD5 sum file
Checking kernel MD5 sum
Kernel MD5 sum is correct

Checking root partition
Root partition looks intact
Attempting to mount root partition
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda2, internal journal
EXT3-fs: mounted filesystem with writeback data mode.
Root partition successfully mounted
Looking for system image file
Found a system image file
Checking system image MD5 sum file
Found system image MD5 sum file
Checking system image MD5 sum
System image MD5 sum is correct

**********************************************
Welcome to EdgeMax Rescue Kit!

This tool is distributed under the terms of
GNU General Public License and other licenses

Brought to you by SO3 Group

WARNING: This tool is not developed, officially
supported or endorsed by Ubiquiti Networks!

Using it may lead to destroying your router
configuration or operating system

Ubiquiti Networks support will not help you
with using it or fixing consequences of
using it.

This tool itself is distributed without any
warranty and authors are not liable for
any damage it may cause

By using this tool you agree you are doing
it at your own risk and understand what
you are doing

*********************************************

Enter 'Yes' to proceed, 'No' to reboot
yes or no: yes

Do you want to configure network via DHCP?
yes or no: yes
udhcpc (v1.17.1) started
Sending discover...
Sending select for 10.91.19.175...
Lease of 10.91.19.175 obtained, lease time 86400
/usr/share/udhcpc/default.script: Resetting default routes
route: SIOCDELRT: No such process
/usr/share/udhcpc/default.script: Adding DNS 10.91.19.1

EMRK provides some scripts for automated
recovery procedures:

emrk-factory-reset -- reset config to factory default
emrk-remove-user-data -- remove all the user data including
    config and everything
emrk-reinstall -- reinstall EdgeOS from scratch
    (wipes any user data too)

Enter 'reboot' to reboot your router


BusyBox v1.17.1 (Debian 1:1.17.1-8) built-in shell (ash)
Enter 'help' for a list of built-in commands.

/bin/ash: can't access tty; job control turned off
EMRK>
EMRK>emrk-reinstall 
WARNING: This script will reinstall EdgeOS from scratch
If you have any usable data on your router storage,
it will be irrecoverably destroyed!
Do you want to continue?
yes or no: yes
Unmounting boot partition
Unmounting root partition
Re-creating partition table
Creating boot partition
Formatting boot partition
mkfs.vfat 3.0.9 (31 Jan 2010)
Creating root partition
Formatting root partition
Mounting boot parition
Mounting root partition
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda2, internal journal
EXT3-fs: mounted filesystem with writeback data mode.
Enter EdgeOS image url: ftp://10.91.19.5/pub/ER-e100.v1.2.0.4574253.tar
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 68.3M  100 68.3M    0     0  8319k      0  0:00:08  0:00:08 --:--:-- 6318k
Unpacking EdgeOS release image
Verifying EdgeOS kernel
Copying EdgeOS kernel to boot partition
Verifying EdgeOS system image
Copying EdgeOS system image to root partition
Creating EdgeOS writable data directory
Cleaning up
Installation finished
Please reboot your router

If you want to do things manually, there are busybox with symlinks to its applets (cat/grep/rm/cp/mv/tar/etc., ping, traceroute, nc etc.), curl, parted, and mkfs.ext3/mkfs.vfat.

 

I used it on my on ERL when I replaced the USB stick with a 8GB one yesterday. Hope you find it useful, any bug reports or suggestions are welcome.

 

Here are the scripts it uses: https://github.com/vyos/emrk

Previous Employee
Posts: 13,551
Registered: ‎06-10-2011
Kudos: 5359
Solutions: 1654
Contributions: 2

Re: EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)

Thanks for the effort and for including the warning! Icon Wink

Previous Employee
Posts: 10,504
Registered: ‎06-09-2011
Kudos: 3028
Solutions: 945
Contributions: 16

Re: EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)

[ Edited ]

This is pretty cool.  One of my test ERL's has been bricked for a while I hadn't gotten around to running our factory install script, so I gave your build a try instead.  It worked and rebooted fine.  

 

One thing you may want to add is the version file since the normal upgrade script looks for that.  Without that the next upgrade failed for me:

ubnt@ubnt:~$ add system image upgrade-v1.2.0dev.4578432.130715.0102.tar
Checking upgrade image... Done
Preparing to upgrade... Done
Copying upgrade image... /usr/bin/ubnt-upgrade: line 449: [: ==: unary operator expected
Done
Removing old image... Done
Checking upgrade image... Done
cp: can't stat '/root.dev/version': No such file or directory
Finishing upgrade... Failed to move current image
ubnt@ubnt:~$ 

 After the failed upgrade, I was left with an upbootable system.  So I used the rescue imagine again to recover and then did a "sudo touch /root.dev/version" so that the next upgrade would work.  I suppose we should be the upgrade script more robost so that it doesn't leave you with an unbootable system if the upgrade goes sideways.

EdgeMAX Router Software Development
New Member
Posts: 17
Registered: ‎07-06-2013
Kudos: 8

Re: EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)

[ Edited ]

I might have to try this. My 8GB upgrade is not going so well. After the first dd the router booted, then after a reboot the router refused to boot from the drive ever again. Even after zeroing it out and starting dd over...

 

[EDIT]

 

This script didnt work for my 8GB drive. I'm starting to think either something is corrupted on it or the router just doesnt like the drive. I'm going to RMA the drive.

Member
Posts: 209
Registered: ‎05-21-2013
Kudos: 256
Solutions: 8
Contributions: 3

Re: EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)

Yeah, good point, I overlooked it. Just copy the version file to the root partition?

Member
Posts: 209
Registered: ‎05-21-2013
Kudos: 256
Solutions: 8
Contributions: 3

Re: EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)

linuxwarz: Hhm, interesting. What messages do you get from U-Boot when you connect via serial after running the reinstall script?

 

Could you also boot to the rescue image once more and post "ls /mnt/root" and "ls /mnt/boot"?

 

Some say there are incomptible USB sticks and it may be the case, but I want to make sure if it was not bug in the script.

New Member
Posts: 17
Registered: ‎07-06-2013
Kudos: 8

Re: EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)

All of your answers are in the attached file. I wonder why I have DHCP issues in the bootloader too... your script works fine. I use the DHCP in the tftpd32 application.

Member
Posts: 209
Registered: ‎05-21-2013
Kudos: 256
Solutions: 8
Contributions: 3

Re: EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)

Interesting. To me it looks like the ERL doesn't like the flash drive, but the fact parted and mkfs didn't throw any errors confuses me. You may look at the bin/emrk-reinstall script from the github link above and reproduce the steps manually, maybe it will give some useful information.

 

Also I updated the image, fixed the bug with missing version file. http://baturin.org/files/edgeos/emrk-0.9b.bin

Established Member
Posts: 920
Registered: ‎05-28-2012
Kudos: 184
Solutions: 6

Re: EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)

Thanks for this it looks like It's unbricked my ERL Hurray

 

I also ran the command to fix the missing file, so hopefully an update won't re-brick it.

 

Member
Posts: 209
Registered: ‎05-21-2013
Kudos: 256
Solutions: 8
Contributions: 3

Re: EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)

With 0.9b the version file should get copied correctly already, so in the future it should not be a trouble. Would be nice if someone tried to install 1.1.0 and upgrade to 1.2.0 for verification. (:

Veteran Member
Posts: 4,720
Registered: ‎03-12-2011
Kudos: 2324
Solutions: 113

Re: EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)

notbad.jpg

 

Will have to remember this for if/when I brick an EdgeMAX (so far I've been lucky! Man Tongue)

SuperUser
Posts: 3,435
Registered: ‎10-12-2010
Kudos: 1879
Solutions: 27

Re: EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)

This makes me want to brick one of mine to test!!! I have been lucky with 80+ in the network and none failed yet minus the one that found lightning and let its smoke out Man Happy

UBRSS, UBWA, UEWA - Ubiquiti Certified Trainer
Member
Posts: 277
Registered: ‎03-08-2009
Kudos: 71
Solutions: 3

Re: EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)

[ Edited ]

Magician wrote:

This makes me want to brick one of mine to test!!! I have been lucky with 80+ in the network and none failed yet minus the one that found lightning and let its smoke out Man Happy


I've got a fix for that. https://www.sparkfun.com/products/retired/10622 Lol

http://youtu.be/Ex8oY24zycQ?t=8m42s

All your WiFi are belong to us!
New Member
Posts: 26
Registered: ‎09-22-2013
Kudos: 5

Re: EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)

Thanks for the great tutorial! 

Just wanted to say that I had no luck with TFTPD on windows 8 x64, it would start the connection but then never transfer any data and time out. 

But Open TFTP server worked first try to serve the file: http://sourceforge.net/projects/tftp-server/

Because I didn't want to setup my own FTP/HTTP server for the final upload of tarball I just used the direct link to the current beta image: http://dl.ubnt.com/firmwares/edgemax/v1.3.0/ER-e100.v1.3.0beta1.4599951.tar

Now it works again Man Happy 

 

Highlighted
Member
Posts: 209
Registered: ‎05-21-2013
Kudos: 256
Solutions: 8
Contributions: 3

Re: EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)

Hey guys,

I've heard from Stig yesterday some folks couldn't download the image.

We were having multiple problems with hardware, L3 switch, and DoS attacks yesterday. Now we resolved that,  EMRK images and other stuff we host should be available again.

Sorry for the inconvenience.

New Member
Posts: 26
Registered: ‎09-22-2013
Kudos: 5

Re: EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)

I am running version "emrk-0.9a.bin" instead of the newest "emrk-0.9c.bin"

Are there any big changes that I should know about between them?

Previous Employee
Posts: 10,504
Registered: ‎06-09-2011
Kudos: 3028
Solutions: 945
Contributions: 16

Re: EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)


hilo90mhz wrote:

I am running version "emrk-0.9a.bin" instead of the newest "emrk-0.9c.bin"

Are there any big changes that I should know about between them?


There was an issue with the version file missing which would cause a problem the NEXT time you upgrade.  It should be ok if you do:

sudo touch /root.dev/version

 Not sure what changed in version "c".

EdgeMAX Router Software Development
Member
Posts: 209
Registered: ‎05-21-2013
Kudos: 256
Solutions: 8
Contributions: 3

Re: EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)

0.9c has some more tools built-in, and some random bugfixes/enhancements to the scripts. Generally I'd recommend it over 0.9b as some bugs were easy to hit, AFAIR.

I should start keeping proper changelogs I guess.
New Member
Posts: 27
Registered: ‎07-02-2013
Kudos: 2

Re: EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)

I have a ER-Lite that failed the firmware upgrade from 1.3 to 1.4.  From the console port, it appears to boot but there are a ton on SQUASHFS errors that fly by about read failures.  These errors even occur after defaulting the unit.

So I've tried running the rescue kit to reinstall the image but I'm running into an issue at the final boot step.  The emrk-0.9c.bin transfers fine from my TFTP server.  But when I type bootoctlinux $loadaddr, the boot process hangs at a point.  The attached pic shows where it hangs.

Any thoughts?

Will

image.jpeg
Previous Employee
Posts: 13,551
Registered: ‎06-10-2011
Kudos: 5359
Solutions: 1654
Contributions: 2

Re: EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)

If it's stopping there, it could be a hardware issue, so please contact support@ubnt.com for help with verification and RMA if necessary.

Reply