Reply
New Member
Posts: 20
Registered: ‎11-27-2017

Diskspace filling up

Our experimentel UCRM server (2.14.0-beta4) is filling up its diskspace.  It has 8GB of disk (plus 2GB swap) and after some days this is slowly eaten up.  It appears to stop using the space before it is completely full, but the steady state is insufficient to do an upgrade of the software.

Attempting it (with e.g. 800MB of free space) destroys it because it fills the disk to 100% and gets stuck, apparently there is no error handling or free space checking in the upgrade process.

Fortunately we run it as a VM and have a disk-image backup solution, so after a failed upgrade I can put it back to that night's snapshot and try other things.

 

"docker system prune -f" does not recover the space.  I have seen it recover space in the past, probably after a successful upgrade, but it does not recover space in the leak scenario.

 

The sequence:

docker-compose stop

docker-compose rm

docker-compose up -d

 

fixes the problem at that time, and then it is e.g. possible to do an upgrade, but the problem just comes back as the space leaks away again.  After a day or two it already has lost some 1GB of space (now 2.2GB free) so I know that when I look in a week or two weeks time it again is almost full and upgrade would kill it.

 

Ubiquiti Employee
Posts: 3,528
Registered: ‎12-10-2015
Kudos: 1255
Solutions: 272

Re: Diskspace filling up

Hi @pe1chl
How big is your UCRM - How many clients do you have? If you have big data and the traffic is high in your UCRM, the system backups and system logs can grow. You can lower the log retention and the number of backups and exclude some data from the backups.
See
1) System > Tools > Backup
2) System > Settings > Logs
New Member
Posts: 20
Registered: ‎11-27-2017

Re: Diskspace filling up

At the moment the system is just monitoring ~25 access points but there are no client records defined at all.

In the logs there are regular messages about connecting and synchronizing but not a message flood.

There are 30 backups of ~400KB each, so that does not explain a full disk either.

The majority of used space is in /var/lib/docker/overlay2

Ubiquiti Employee
Posts: 3,528
Registered: ‎12-10-2015
Kudos: 1255
Solutions: 272

Re: Diskspace filling up

Ok, it sounds like some docker issue, not UCRM issue.
Try this to debug the disk usage:

docker system df
or
docker system df -v
New Member
Posts: 20
Registered: ‎11-27-2017

Re: Diskspace filling up

docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              6                   4                   2.091GB             489MB (23%)
Containers          4                   4                   18.65MB             0B (0%)
Local Volumes       6                   0                   0B                  0B
Build Cache         0                   0                   0B                  0B

docker system prune -f
Total reclaimed space: 0B

docker system df -v
Images space usage:

REPOSITORY                                          TAG                 IMAGE ID            CREATED             SIZE                SHARED SIZE         UNIQUE SIZE         CONTAINERS
postgres                                            9.5                 1dbed2567d30        4 days ago          225.5MB             0B                  225.5MB             1
ubnt/ucrm-billing                                   2.14.0-beta4        28def925d2ee        5 days ago          736.8MB             0B                  736.8MB             1
rabbitmq                                            3                   e8261c2af9fe        3 weeks ago         124.8MB             0B                  124.8MB             1
elastic/elasticsearch                               6.2.4               7cb69da7148d        7 months ago        514.5MB             0B                  514.5MB             1
docker.elastic.co/elasticsearch/elasticsearch-oss   6.2.2               0453814a47b3        9 months ago        483.4MB             0B                  483.4MB             0
ubnt/ucrm-conntrack                                 latest              1142f53df4c4        20 months ago       5.621MB             0B                  5.621MB             0

Containers space usage:

CONTAINER ID        IMAGE                            COMMAND                  LOCAL VOLUMES       SIZE                CREATED             STATUS              NAMES
029fb148c062        ubnt/ucrm-billing:2.14.0-beta4   "make server"            0                   17.3MB              44 hours ago        Up 7 hours          ucrm_web_app_1
27901af24b1a        elastic/elasticsearch:6.2.4      "/usr/local/bin/dock…"   0                   1.37MB              44 hours ago        Up 7 hours          ucrm_elastic_1
57726ba652da        rabbitmq:3                       "docker-entrypoint.s…"   0                   79B                 44 hours ago        Up 7 hours          ucrm_rabbitmq_1
abb460724f28        postgres:9.5                     "docker-entrypoint.s…"   0                   63B                 44 hours ago        Up 7 hours          ucrm_postgresql_1

Local Volumes space usage:

VOLUME NAME                                                        LINKS               SIZE
389236904be00c9781e9be7d0fbfb2ba59cc57c290fb37597253722e41562d70   0                   0B
428d6736e4d2ab2402a01d3fde6a3475845ff032ae372fb4084f87d411a865c7   0                   0B
7acf7d45c9f4e738f7c41501a7a9937423186b3665c3d0dd95fdad4b3c93e455   0                   0B
923a1e4704416fdf611808851d263f6a6d07c95664f1e48936eaf1c4d1993003   0                   0B
9b2571f2dd3bca2e5b56d052bbee344bd344ae46c14353ea1dcd9c564b1b1943   0                   0B
d7e5a0e99ae87cc3a1f578fbdd2c48ed842e1ce4ba9f846856573de073ee6da5   0                   0B

Build cache usage: 0B

CACHE ID            CACHE TYPE          SIZE                CREATED             LAST USED           USAGE               SHARED

The system is running Debian stretch (9) with the repository from download.docker.com for stretch, all uptodate.

Ubiquiti Employee
Posts: 3,528
Registered: ‎12-10-2015
Kudos: 1255
Solutions: 272

Re: Diskspace filling up

I see no issue here. Please, run "docker info" and send us the output.
New Member
Posts: 20
Registered: ‎11-27-2017

Re: Diskspace filling up

docker info
Containers: 4
Running: 4
Paused: 0
Stopped: 0
Images: 6
Server Version: 18.09.0
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: c4446665cb9c30056f4998ed953e6d4ff22c7c39
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.0-8-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.957GiB
Name: ucrm
ID: TN5M:3TAQ:6FPE:4AURMan FrustratedD3J:VSUA:5ZWI:VOSS:QQLRMan TongueADI:3QRSMan SurprisedWZW
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: No swap limit support

Ubiquiti Employee
Posts: 3,528
Registered: ‎12-10-2015
Kudos: 1255
Solutions: 272

Re: Diskspace filling up

New Member
Posts: 20
Registered: ‎11-27-2017

Re: Diskspace filling up

docker system prune -a
WARNING! This will remove:
- all stopped containers
- all networks not used by at least one container
- all images without at least one container associated to them
- all build cache
Are you sure you want to continue? [y/N] y
Deleted Images:
untagged: docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.2
untagged: docker.elastic.co/elasticsearch/elasticsearch-oss@sha256:2340904dc75d43373b6cc81935b43eda459dbabec83455f6cee6fd24269abe18
deleted: sha256:0453814a47b3e83f1112cfd739b043c5f33d1e4c0972d212409c2b18a45dc451
deleted: sha256:f2991b9864e96723f3fc9ded78cd3376c043bb5353fc8f6b9b7ce83ac884ea59
deleted: sha256:16c2a3040f2361e620009da34ecc1f48a10124464fe068c7d1fa80dab3d4d6b3
deleted: sha256:c400a5e29c85eddea0bfe5a8a01c42a8eb38190809e5068e7e832bd542d6672d
deleted: sha256:fdc8bbaa329db72f80bc7ac7cbfe651ddf45380ffac82ba9bdbf62f023a84d4d
deleted: sha256:6384a1335cc50890131df5b609d594b011a37a30975e2571dc9dd8de24fd9d59
deleted: sha256:e15afa4858b655f8a5da4c4a41e05b908229f6fab8543434db79207478511ff7
untagged: ubnt/ucrm-conntrack:latest
untagged: ubnt/ucrm-conntrack@sha256:5ed724152a396c413db84cdb412146d9f2f655a2ef96ed45b26e4d2e2c4ebee3
deleted: sha256:1142f53df4c41b53e8e63680657eb3f77fb78558e4ed2db2818b2060b8d04156
deleted: sha256:511ae78209468e2e0ff72c54fc3b785ceec4f9684268761026a24ab3f27c6a54
deleted: sha256:23b9c7b43573dd164619ad59e9d51eda4095926729f59d5f22803bcbe9ab24c2

Total reclaimed space: 489MB

 

Ok that reclaims a good deal of space.  I will keep checking if running this will keep the space usage stable.

I put the following in /etc/cron.weekly/dsprune:

#!/bin/sh
docker system prune -a -f >/tmp/dsprune.log

 

Maybe it is  a good idea to do that from the program itself as some timed job?

(it already issues the advise to run "docker system prune" in the upgrade log but does not do that automatically)

Ubiquiti Employee
Posts: 3,528
Registered: ‎12-10-2015
Kudos: 1255
Solutions: 272

Re: Diskspace filling up

it's risky to delete something automatically from user's server (check the warning above), but we can consider this.
you can also monitor the size of /home/ucrm/data
New Member
Posts: 20
Registered: ‎11-27-2017

Re: Diskspace filling up

[ Edited ]

Ok in my case it does not matter because we make a separate virtual server for every task...

(but that is also the reason we don't have terabytes of space on the file system)

 

I don't know if docker supports some way to limit the scope of the command to only the UCRM

related containers, if so it would be a good idea to use that.

 

For me it is no problem to add a script like the above to work around issues, once I know them.

But it could not be the same for every user.

 

BTW, it would be good to at least check for remaining space before the upgrade is started.

New Member
Posts: 31
Registered: ‎11-20-2016
Kudos: 4

Re: Diskspace filling up

we also see this as a major issue. afaik its impossible to install in a specified location other than /home/ucrm. running a droplet on digitalocean with a max 50gb storage is not sustainable and would love to see where all the space is going!

Highlighted
Ubiquiti Employee
Posts: 3,528
Registered: ‎12-10-2015
Kudos: 1255
Solutions: 272

Re: Diskspace filling up

Hi, the images and containers related to UCRM are automatically pruned.
We will also improve the update process to check the available disk space before the update starts.
Reply