Cloud Key firmware update runs out of disk space.

[jinie]

I've noticed this twice now, when updating the CK firmware, the download goes to /var/tmp, causing the root fs to run out of space.

And meanwhile, /data sits there with ~8 GB free space.

I'm running a beta version of UniFi on it, if that makes any difference.

A quick fix would be to simply download the firmware to /data instead of /var/tmp, or simply linking /var/tmp to /data.

[danz331]

same here, current firmware version v0.3.4. I tried to update to latest. Update via "Update Now" or "Update Manualy" failed both.

Root filesystem has only 376 MB and current update firmware file (UCK.mtk7623.v0.4.2.6c2ebce.160105.1604.bin) has approx. 400MB

Filesystem Size Used Avail Use% Mounted on
aufs-root 1.1G 678M 376M 65% /

It actually seems that firmware upload goes via nginx process to /var/lib/nginx/body/ which - part of the root fs - and obiosly it cannot save update fw file

here is err from nginx process

2016/01/19 18:01:02 [crit] 383#0: *2196 pwrite() "/var/lib/nginx/body/0000000002" failed (28: No space left on device), client: 10.0.0.6, server: localhost, request: "POST /api/upload-upgrade HTTP/1.1", host: "10.0.0.9", referrer: "https://10.0.0.9/"

What can be done here to fix issue?

Dan

[jinie]

I "ninjaed" the URL from the running wget process, and then manually downloaded it to /data, and manually started the upgrade

i.e.

ubnt-systool fwupdate /data/UCK.mtk7623.v0.3.6.9510e85.151112.1544.bin

[danz331]

unfortuntely it didn't help either

root@UniFi-CloudKey:/data# ubnt-systool fwupdate /data/UCK.mtk7623.v0.4.2.6c2ebce.160105.1604.bin
Firmware file: '/data/UCK.mtk7623.v0.4.2.6c2ebce.160105.1604.bin'
Firmware: UCK.mtk7623.v0.4.2.6c2ebce.160105.1604
mv: error writing '/boot/fwupdate.bin': No space left on device
mv: failed to extend '/boot/fwupdate.bin': No space left on device
ERROR: Failed moving /data/UCK.mtk7623.v0.4.2.6c2ebce.160105.1604.bin to /boot/fwupdate.bin

root@UniFi-CloudKey:/data# ls -l /boot/
total 395656
-rw------- 1 root root 405147648 Jan 19 19:04 fwupdate.bin

now can I safely delete /boot/fwupdate.bin file to release some space?  

[danz331]

ok, here is what I did

"apt-get clean" to clean deb packages cache and free some space on / 

then fwupdate went ok

is this cloud key some kind of beta?

[Capt_Greenbean]

The problem with disk space is usually because the DB is growing too large for the original install location. Since the Cloud Key unit ships with an 8GB SD card which mounts as /data (as you pointed out) I was able to solve this problem by moving the db folder to the SD card and then setting up a symbolic link to the new location. This provides much more storage space for the DB to grow and should free up space to handle future firmware updates. Here are the steps to do this:

## STOP SERVICES
sudo /etc/init.d/unifi stop
sudo /etc/init.d/cloudkey-webui stop

## MOVE DB AND SETUP SYMBOLIC LINK
sudo mv /srv/unifi/data/db /data
sudo ln -s /data/db/ /srv/unifi/data/db

## START SERVICES
sudo /etc/init.d/unifi start
sudo /etc/init.d/cloudkey-webui start

I then also ran the script located here to clean out older data from the DB:
https://help.ubnt.com/hc/en-us/articles/204911424-UniFi-Remove-prune-older-data-and-adjust-mongo-dat...

I am thinking about setting this script as a cron job to run periodically on its own, but I haven't done that yet. So far everything seems stable but your milage may vary. Be sure to check the log files for any errors.

[soloithz23]

Help, I am tryng to do this and the "sudo In" is not a valid command it says.

[soloithz23]

nevermind it was an L not an i

[nilesf]

Still no love. Still not enough free space. Ran through this post and nothing let me upgrade the CK firmware.

Please advise.

[scyto]

same issue as nilesf - this didn't resolve my space issues.

[scyto]

---

@nilesf wrote:

Still no love. Still not enough free space. Ran through this post and nothing let me upgrade the CK firmware.

Please advise.

---

This doesn't work, you rpobably have same issue as me that aufs-root is too full, none of the stuff listed in this thread is stored on aufs-root; i have a thread open in the beta forum you can track if you like?

[outie]

What I did was to remove the bundled firmwares on the cloud key. Once the new cloud key firmware installs the new bundled firmwares will be back.

[scyto]

i just got told to do this, this gave me space i needed (now just need to figure out how to move logs back to the main unit as having them on the SD card will destroy the SD card eventually).

## CLEAN AUFS 

apt-get update

apt-get install aufs-tools

/sbin/auplink / flush

Note if when you df -h your aufs-root is at 100% try deleting the fwupdate.bin from /var/tmp to have enough space to do the commands above.

I figured out how to back out the DBs from the SD card back to main unit:

## STOP SERVICES
sudo /etc/init.d/unifi stop
sudo /etc/init.d/cloudkey-webui stop

## REMOVE SYMBOLIC LINK AND MOVE DB BACK TO NORMAL STORAGE

cd /srv/unifi/data

unlink db
sudo mv /data/db /srv/unifi/data
 

## START SERVICES
sudo /etc/init.d/unifi start
sudo /etc/init.d/cloudkey-webui start