Reply
New Member
Posts: 1
Registered: ‎09-20-2013

UniFi mongo database corrupt - site purge

Hi guys,

 

My controller ran out of disk space and started going into a loop... not focusing I upgraded my 5.6.39 controller to 5.9.29 in hopes that it would solve my issue, it did not as you have guest already.

Once I realized the storage crisis, I increased the storage and extended the partition successfully but the controller is still in a loop.

I believe one of the sites are causing this loop due to corupted databse.

 

I have tried with no success the unifi-repair.sh scipt from: HOW-TO-Repair-MongoDB-on-Linux

I can successfully load my latest backup, but it is 2 months old and missing 9 sites!

 

Is there a way to purge specific sites cause I believe maybe one of the newer sites are causing the issue?

Or are there other troubleshooting steps I can take?

 

Here is the log details:

[2018-10-29 13:53:49,403] <launcher> INFO  system - ======================================================================
[2018-10-29 13:53:49,404] <launcher> INFO  system - UniFi 5.9.29 (build atag_5.9.29_11384 - release) is started
[2018-10-29 13:53:49,404] <launcher> INFO  system - ======================================================================
[2018-10-29 13:53:49,405] <launcher> INFO  system - BASE dir:/usr/lib/unifi
[2018-10-29 13:53:49,410] <launcher> INFO  system - Current System IP: 10.0.10.150
[2018-10-29 13:53:49,410] <launcher> INFO  system - Hostname: unisrvhs1
[2018-10-29 13:54:38,628] <localhost-startStop-1> WARN  SessionIdGeneratorBase - Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [47,996] milliseconds.
[2018-10-29 13:54:39,046] <launcher> INFO  db     - waiting for db connection...
[2018-10-29 13:54:39,548] <launcher> INFO  db     - Connecting to mongodb://127.0.0.1:27117
[2018-10-29 13:54:40,110] <launcher> INFO  db     - DB version (5.6.39) different from Runtime Version(5.9.29), migrating...
[2018-10-29 13:54:40,116] <launcher> INFO  system - [server backup] started
[2018-10-29 13:54:40,117] <launcher> INFO  db     - Connecting to mongodb://127.0.0.1:27117
[2018-10-29 13:54:41,270] <launcher> INFO  db     - Connecting to mongodb://127.0.0.1:27117
[2018-10-29 13:54:41,375] <launcher> ERROR system - Fail to create a backup for version '5.6.39'
[2018-10-29 13:54:41,376] <launcher> ERROR system - Info on backup error is...
java.lang.NullPointerException
        at com.ubnt.ace.C.o00000(Unknown Source)
        at com.ubnt.ace.C.o00000(Unknown Source)
        at com.ubnt.service.OoOO.O0oO.o00000(Unknown Source)
        at com.ubnt.service.OoOO.interfacesuper.new(Unknown Source)
        at com.ubnt.service.OoOO.oooo.void(Unknown Source)
        at com.ubnt.service.OoOO.W.afterPropertiesSet(Unknown Source)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1758)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1695)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:573)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.resolveBeanReference(ConfigurationClassEnhancer.java:392)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:364)
        at com.ubnt.service.AppContext$$EnhancerBySpringCGLIB$$849691fd.dbService(<generated>)
        at com.ubnt.service.AppContext.statService(Unknown Source)
        at com.ubnt.service.AppContext$$EnhancerBySpringCGLIB$$849691fd.CGLIB$statService$10(<generated>)
        at com.ubnt.service.AppContext$$EnhancerBySpringCGLIB$$849691fd$$FastClassBySpringCGLIB$$85380ab3.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:361)
        at com.ubnt.service.AppContext$$EnhancerBySpringCGLIB$$849691fd.statService(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:582)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1247)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1096)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:535)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.resolveBeanReference(ConfigurationClassEnhancer.java:392)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:364)
        at com.ubnt.service.AppContext$$EnhancerBySpringCGLIB$$849691fd.statService(<generated>)
        at com.ubnt.service.AppContext.houseKeeper(Unknown Source)
        at com.ubnt.service.AppContext$$EnhancerBySpringCGLIB$$849691fd.CGLIB$houseKeeper$19(<generated>)
        at com.ubnt.service.AppContext$$EnhancerBySpringCGLIB$$849691fd$$FastClassBySpringCGLIB$$85380ab3.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:361)
        at com.ubnt.service.AppContext$$EnhancerBySpringCGLIB$$849691fd.houseKeeper(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:582)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1247)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1096)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:535)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:759)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
        at org.springframework.context.annotation.AnnotationConfigApplicationContext.<init>(AnnotationConfigApplicationContext.java:88)
        at com.ubnt.service.B.Oo0000(Unknown Source)
        at com.ubnt.service.B.Õ00000(Unknown Source)
        at com.ubnt.ace.Launcher.main(Unknown Source)
[2018-10-29 13:54:41,379] <launcher> INFO  system - [server backup][BACKUP] end
[2018-10-29 13:54:42,102] <launcher> WARN  AnnotationConfigApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'houseKeeper' defined in com.ubnt.service.AppContext: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.ubnt.service.OoOO.OoO0OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO]: Factory method 'houseKeeper' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'statService' defined in com.ubnt.service.AppContext: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.ubnt.service.OoOO.d]: Factory method 'statService' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dbService' defined in com.ubnt.service.AppContext: Invocation of init method failed; nested exception is java.lang.NullPointerException
[2018-10-29 13:54:42,705] <UniFi> ERROR system - [exec] error, rc=1, cmdline=[/usr/lib/jvm/java-8-oracle/jre/bin/java, -Dfile.encoding=UTF-8, -Djava.awt.headless=true, -Dapple.awt.UIElement=true, -Xmx1024M, -XX:ErrorFile=/usr/lib/unifi/logs/hs_err_pid%p.log, -jar, /usr/lib/unifi/lib/ace.jar, start]
2018-10-29T14:31:44.387+0200 ***** SERVER RESTARTED *****
2018-10-29T14:31:44.389+0200 [initandlisten] MongoDB starting : pid=59404 port=27117 dbpath=/usr/lib/unifi/data/db 64-bit host=unisrvhs1
2018-10-29T14:31:44.389+0200 [initandlisten] db version v2.6.10
2018-10-29T14:31:44.389+0200 [initandlisten] git version: nogitversion
2018-10-29T14:31:44.389+0200 [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2018-10-29T14:31:44.389+0200 [initandlisten] build info: Linux lgw01-12 3.19.0-25-generic #26~14.04.1-Ubuntu SMP Fri Jul 24 21:16:20 UTC 2015 x86_64 BOOST_LIB_VERSION=1_58
2018-10-29T14:31:44.389+0200 [initandlisten] allocator: tcmalloc
2018-10-29T14:31:44.389+0200 [initandlisten] options: { net: { port: 27117, unixDomainSocket: { pathPrefix: "/usr/lib/unifi/run" } }, repair: true, storage: { dbPath: "/usr/lib/unifi/data/db" }, systemLog: { destination: "file", logAppend: true, path: "/usr/lib/unifi/logs/mongod.log" } }
2018-10-29T14:31:44.395+0200 [initandlisten] repairDatabase ace
2018-10-29T14:31:44.395+0200 [initandlisten] allocating new ns file /usr/lib/unifi/data/db/_tmp_repairDatabase_0/ace.ns, filling with zeroes...
2018-10-29T14:31:44.423+0200 [FileAllocator] allocating new datafile /usr/lib/unifi/data/db/_tmp_repairDatabase_0/ace.0, filling with zeroes...
2018-10-29T14:31:44.423+0200 [FileAllocator] creating directory /usr/lib/unifi/data/db/_tmp_repairDatabase_0/_tmp
2018-10-29T14:31:44.428+0200 [FileAllocator] done allocating datafile /usr/lib/unifi/data/db/_tmp_repairDatabase_0/ace.0, size: 64MB,  took 0.002 secs
2018-10-29T14:31:44.452+0200 [initandlisten] warning Listener::getElapsedTimeMillis returning 0ms
2018-10-29T14:31:44.505+0200 [initandlisten] warning Listener::getElapsedTimeMillis returning 0ms
2018-10-29T14:31:44.658+0200 [initandlisten] repairDatabase local
2018-10-29T14:31:44.658+0200 [initandlisten] allocating new ns file /usr/lib/unifi/data/db/_tmp_repairDatabase_0/local.ns, filling with zeroes...
2018-10-29T14:31:44.686+0200 [FileAllocator] allocating new datafile /usr/lib/unifi/data/db/_tmp_repairDatabase_0/local.0, filling with zeroes...
2018-10-29T14:31:44.686+0200 [FileAllocator] creating directory /usr/lib/unifi/data/db/_tmp_repairDatabase_0/_tmp
2018-10-29T14:31:44.698+0200 [FileAllocator] done allocating datafile /usr/lib/unifi/data/db/_tmp_repairDatabase_0/local.0, size: 64MB,  took 0.005 secs
2018-10-29T14:31:44.709+0200 [initandlisten] finished checking dbs
2018-10-29T14:31:44.709+0200 [initandlisten] dbexit:
2018-10-29T14:31:44.709+0200 [initandlisten] shutdown: going to close listening sockets...
2018-10-29T14:31:44.709+0200 [initandlisten] shutdown: going to flush diaglog...
2018-10-29T14:31:44.709+0200 [initandlisten] shutdown: going to close sockets...
2018-10-29T14:31:44.709+0200 [initandlisten] shutdown: waiting for fs preallocator...
2018-10-29T14:31:44.709+0200 [initandlisten] shutdown: closing all files...
2018-10-29T14:31:44.709+0200 [initandlisten] closeAllFiles() finished
2018-10-29T14:31:44.709+0200 [initandlisten] shutdown: removing fs lock...
2018-10-29T14:31:44.710+0200 [initandlisten] dbexit: really exiting now

Any help would be greatly appreciated.

New Member
Posts: 1
Registered: ‎05-22-2018

Re: UniFi mongo database corrupt - site purge

Just a brief update:

I'm under going the prcess of recreating the 9 sites that weren't backed up, going through call sheets and communications to redo the configurations. Sending resources out to the sites to reset the units, all while trying to explain to a customer how I could have let this happen... Icon Confused

Still hoping someone out there could at least help me to export the Device Authentication information for a site from the ace database? Aureola

Senior Member
Posts: 21,846
Registered: ‎08-04-2017
Kudos: 4124
Solutions: 1062

Re: UniFi mongo database corrupt - site purge

[ Edited ]

Hello @fleroy,

 

Tried following this to repair the DB?

 

Downgrade to 5.6.40. ( hope the migration didn't start yet. )

After that upgrade to 5.8.30 and then 5.9.29.

 

You can use my easy update scirpt if you have Debian 8/9 or Ubuntu 16.04/18.04

 

 

Regards,

Glenn R.

Cloud Hosted Controllers | Glenn R. | UniFi Installation/Easy Update Scripts | UniFi-Video Installation Scripts | UniFi-VoIP Installation Scripts
USG-XG-8 • USG-4-PRO • USG
US-XG-16 • US-48-500W • US-24-POE-250W 2x • US-16-POE-150W 3x • US-24 • US-8-150W • US-8
UAP XG • UAP-SHD • UAP-HD • UAP-NanoHD 2x • UAP-AC-PRO 2x • UAP-AC-LITE • UAP-AC-IW • UAP-AC-M • UAP-AC-M-PRO 2x
UAS-XG • UCK-G2-PLUS • UCK-G2 • UCK
Highlighted
Senior Member
Posts: 3,086
Registered: ‎04-26-2016
Kudos: 1230
Solutions: 314

Re: UniFi mongo database corrupt - site purge

His database was already botched before he tried to update the controller. He tried to repair things by trying an update. So that has no use.

 

Only lesson to learn from this: Backup your controller and backup often. For example every day. And especially when you add a new site.

Reply