0 Kudos

OverlayFS - Use it to make all writes to memory except those that we want to be persistent

Submitted by -
Status: New Idea

Extend the lifespan of the flash!


Writes that occur directly to the disk may, over time, wear out the flash.   If all writes take place in a tmpfs layer on top of an OverlayFS, then a flash-write overlay can be used for only the super important logs, DHCP lease info, etc.   Or, better yet, just do a scheduled sync to flash to minimize the number of writes (acknowledging a potential maximum amount of seconds of lost data if a crash occurred).   Normal shutdown and restart procedures could trigger a sync, so routine reboots would not cause any data loss.

on ‎07-28-2017 06:34 PM

From what I know of the current EdgeOS/VyOS codebase, it relies on some behavior of UnionFS to achieve the image-based boot and config tree. Unfortunately this behavior is not the same in OverlayFS which would be a much better choice. Now, I can't see them using OverlayFS anywhere else if they can't also replace UnionFS which at the moment is...Not easily done. 

on ‎08-03-2017 07:40 PM

Could you provide an example of a filesystem where this would be of value on EdgeOS?


/tmp, /run, and /var/log are already mounted as tmpfs and these are where the majority of writes would occur.


Based on long-term monitoring via SNMP there really are minimal writes happening to the persistent storage already.

on ‎08-11-2017 12:53 PM

The minimal writes that happen are typically constant and will burn out flash drive eventually.   The challenge is that some data needs to persist across reboots or system crashes such as the DHCP service database.   A periodic sync to flash for that data is a bitter-sweet solution because it's all but guaranteed to be inconsistent on reboot, but it would minimize the routine writes.   A second "data" flash device might be a better solution because then the OS remains on the primary flash that has zero writes except when updating the flash or config.   Then these routers could last 20 years and the worst-case scenario is that you have to replace the secondary flash device.   


I am not saying that I have the answer.  I'm just putting some thoughts out there to start a conversation that may improve this product.   I used to run VyOS on USB flash drives and I used OverlayFS to mount tmpfs over the /var/log and /var/spool folders.   That extended the lifespan of the USB flash drives by 10x.  I wish I could do even more, though.   These things should last forever.

on ‎08-11-2017 02:12 PM
I think the newest EdgeRouters coming now all have SD card slots and/or USB ports. Secondary flash issue sorted.
on ‎08-14-2017 07:18 AM

Based on my monitoring (via SNMP) of many ER devices the writes are definitely not constant and are actually infrequent.  A scheduled 'sync' would actually add more overhead and complexity without reducing writes at all.  Similarly, adding support for another FS type also would add complexity and increase kernel footprint (memory use).


The ER is not like a normal linux system and has diverged from Vyatta quite a bit (as has VyOS) and there is actually very little that needs to be persistent.


As already noted /var/log is already tmpfs-mounted and there are virtually no writes to /var/spool after boot.


Only the ER-Lite-3 uses a USB drive.  There are no issue with writes on other platforms.