Reply
Emerging Member
Posts: 77
Registered: ‎03-04-2015
Kudos: 37
Solutions: 5

Issues Starting MFi + MongoDB on Ubuntu 14.04

[ Edited ]

I currently have MFi 2.1.11 installed on Ubuntu 14.04.1 x64. I was previously using MFi 2.1.8 and was experiencing these issues there as well.

 

Frequency, when I start the mfi service, mongodb fails to correctly startup, leading to a spewing of error messages in the MFi logs. This happens after a system reboot, after an mfi service restart, etc.

 

Upon starting, MFi seems to launch it's own instance of mongo, which then dies after a moment or two as shown in the "/var/log/mfi/mongod.log" mongo log:

 

Tue Apr 21 23:57:31.597 [initandlisten] MongoDB starting : pid=2101 port=37017 dbpath=/usr/lib/mfi/data/db 64-bit host=mfi-controller
Tue Apr 21 23:57:31.597 [initandlisten] db version v2.4.9
Tue Apr 21 23:57:31.597 [initandlisten] git version: nogitversion
Tue Apr 21 23:57:31.597 [initandlisten] build info: Linux orlo 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86_64 BOOST_LIB_VERSION=1_54
Tue Apr 21 23:57:31.597 [initandlisten] allocator: tcmalloc
Tue Apr 21 23:57:31.597 [initandlisten] options: { bind_ip: "127.0.0.1", dbpath: "/usr/lib/mfi/data/db", logappend: true, logpath: "logs/mongod.log", nohttpinterface: true, port: 37017 }
Tue Apr 21 23:57:32.629 [initandlisten] journal dir=/usr/lib/mfi/data/db/journal
Tue Apr 21 23:57:32.629 [initandlisten] recover : no journal files present, no recovery needed
Tue Apr 21 23:57:34.010 [initandlisten] ERROR: listen(): bind() failed errno:98 Address already in use for socket: 127.0.0.1:37017
Tue Apr 21 23:57:34.010 [initandlisten] ERROR:   addr already in use
Tue Apr 21 23:57:34.010 [initandlisten] now exiting
Tue Apr 21 23:57:34.010 dbexit: 
Tue Apr 21 23:57:34.010 [initandlisten] shutdown: going to close listening sockets...
Tue Apr 21 23:57:34.010 [initandlisten] shutdown: going to flush diaglog...
Tue Apr 21 23:57:34.010 [initandlisten] shutdown: going to close sockets...
Tue Apr 21 23:57:34.011 [initandlisten] shutdown: waiting for fs preallocator...
Tue Apr 21 23:57:34.011 [initandlisten] shutdown: lock for final commit...
Tue Apr 21 23:57:34.011 [initandlisten] shutdown: final commit...
Tue Apr 21 23:57:35.377 [initandlisten] shutdown: closing all files...
Tue Apr 21 23:57:35.377 [initandlisten] closeAllFiles() finished
Tue Apr 21 23:57:35.377 [initandlisten] journalCleanup...
Tue Apr 21 23:57:35.377 [initandlisten] removeJournalFiles
Tue Apr 21 23:57:36.143 [initandlisten] shutdown: removing fs lock...
Tue Apr 21 23:57:36.143 dbexit: really exiting now

Despite the error messages, there does not seem to be anything else using the address/port in question.

 

Mongo crashing leads to the repetition of the following error in the "/var/log/mfi/server.log" mfi log:

 

[2015-04-22 00:00:22,795] <localhost-startStop-1> INFO  DbService  - Mongo is attached to 127.0.0.1:37017
[2015-04-22 00:00:22,795] <22> WARN  DynamicConnectionStatus  - Server seen down: /127.0.0.1:37017
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at com.mongodb.DBPort._open(DBPort.java:223)
        at com.mongodb.DBPort.go(DBPort.java:125)
        at com.mongodb.DBPort.go(DBPort.java:106)
        at com.mongodb.DBPort.findOne(DBPort.java:162)
        at com.mongodb.DBPort.runCommand(DBPort.java:170)
        at com.mongodb.ConnectionStatus$UpdatableNode.update(ConnectionStatus.java:196)
        at com.mongodb.DynamicConnectionStatus$DynamicNode.update(DynamicConnectionStatus.java:167)
        at com.mongodb.DynamicConnectionStatus$1.run(DynamicConnectionStatus.java:114)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

 Generally, the only way to resolve the error is to shutdown mfi (which often requires killing a few java jobs since 'sudo service mfi stop' doesn't handle everything) and manually start up mongo in the same manner that mfi does. E.g.:

 

sudo /usr/bin/mongod --dbpath /usr/lib/mfi/data/db --port 37017 --logappend --logpath /var/log/mfi/mongod.log --nohttpinterface --bind_ip 127.0.0.1

I can then gracefully kill this instance of mongo (via ctrl-c). Afterwards, I'm generally able restart mfi via 'sudo service mfi start', which kicks off a now-working mongo instance after which everything is good to go.

 

As far as I can tell, this has been happening for some time and on 2.1.8. Maybe some sort of race in the way mfi spins up mongo? Or some kind of bad state left over by mongo not being shut down properly on reboot? I'm happy to do more debugging if it would be helpful. Also, it's probably worth noting that I have ~18GB of MFi mongo data stored at this point, so maybe it's a race condition that only shows up with large DBs? And I'm running this version of Ubuntu 14.04 in a VM, so maybe there's an issue with slower IO speeds?

SuperUser
Posts: 15,105
Registered: ‎08-07-2008
Kudos: 7313
Solutions: 415
Contributions: 1

Re: Issues Starting MFi + MongoDB on Ubuntu 14.04

Mongo needs a lot of ram and space, even swap space.

 

This is the first time I have read about it starting twice.  Any chance you have an entry in the startup config twice for mongo? or mFi ? 

 

Did you install mongo on the system then mFi afterwards?

Walks in, tells it like it is, walks out , milks a cow
Emerging Member
Posts: 77
Registered: ‎03-04-2015
Kudos: 37
Solutions: 5

Re: Issues Starting MFi + MongoDB on Ubuntu 14.04

I don't think this is a resource issue - I havn't seen the machine come close to topping out the available resources (e.g. 4GB of RAM).

 

I also don't think multiple copies of Mongo are ever running. When I watch the process list during the mfi startup process, it only ever launches a single copy of mongo (which the dies as described above). I've disabled the default mongo instance that normally starts at system boot, so the mfi instance is the only one ever running. And even when it complains about the port and IP being is use, all of the normal network tools show that not to be the case - it seems the error message itself is incorrect (or at least misleading).

 

I installed mfi via the apt package, and it pulled in mongo as a dependency during the install process.

Emerging Member
Posts: 41
Registered: ‎01-22-2015
Kudos: 16

Re: Issues Starting MFi + MongoDB on Ubuntu 14.04

I could be way off but your symptoms look similar to the problem I had when I tried to upgrade Mongodb to version 2.6.X which is not supported by mFi

http://community.ubnt.com/t5/mFi/mongodb-2-4x-to-2-6x-upgrade/m-p/1177400/highlight/true#M11382

Emerging Member
Posts: 77
Registered: ‎03-04-2015
Kudos: 37
Solutions: 5

Re: Issues Starting MFi + MongoDB on Ubuntu 14.04

[ Edited ]

I considered that, but I'm still running Mongo 2.4.9:

 

ubuntu@mfi-controller:~$ mongo --version
MongoDB shell version: 2.4.9
ubuntu@mfi-controller:~$ mongod --version
db version v2.4.9
Thu Apr 23 09:28:15.962 git version: nogitversion

 

New Member
Posts: 20
Registered: ‎06-16-2015
Kudos: 2

Re: Issues Starting MFi + MongoDB on Ubuntu 14.04

I have a similar problem ... here the mfi-Software starts; but doesn't work after a while. Before I could always run a Backup but now even this doesn't work any more. Is there a solution for the problem?
SuperUser
Posts: 15,105
Registered: ‎08-07-2008
Kudos: 7313
Solutions: 415
Contributions: 1

Re: Issues Starting MFi + MongoDB on Ubuntu 14.04

2 things  for a ubuntu mfi system.

1 is it a 64 bit version of Ubuntu?

2 if you manually installed mongo , it puts on the wrong version.

mfi will install the version of mongo it requires all by it's self in ubuntu 12 or 14.

Walks in, tells it like it is, walks out , milks a cow
Emerging Member
Posts: 77
Registered: ‎03-04-2015
Kudos: 37
Solutions: 5

Re: Issues Starting MFi + MongoDB on Ubuntu 14.04

I havn't had much time to dig into this further, but I'm still having the originally reported issue on Ubuntu 14.04 x64 running Mongo 2.4.

 

I'm pretty sure there is a race condition in the mFi startup code where it doesn't wait for Mongo to reach state where it's ready to accept incoming requests prior to trying to send such requests. The issue is most noticeable in situations where Mongo can take a while to start because:

 

+ The Mongo DB is large (e.g. 10s of GB), or

+ The disk speed is slow (e.g. not SSD, heavily fragmented, etc)

 

On my machine, the combination of these two factors seems to slow down the Mongo stratup cycle enough such that the mFi app starts sending requests to Mongo before Mongo is ready to accept them, causing the mFi app to error out and close. It seems mFi needs to monitor Mongo's startup process and not try to send any data until Mongo has fully started, even when these takes 10s of seconds or longer. Alternatively, mFi should just have an option to specify a separate Mongo server instead of trying to spin one up itself so that this issue could be resolved via the normal init/systemd startup dependency management tools.

 

I've been working around the issue by manually starting the mFi Mongo DB instance prior to starting the mFi app, allowing me to wait for Mongo to fully start prior to mFi trying to use it.

New Member
Posts: 20
Registered: ‎06-16-2015
Kudos: 2

Re: Issues Starting MFi + MongoDB on Ubuntu 14.04

[ Edited ]
I installed via aptitude -- mongo is Version 2.4.9 And Yes -- it is 64 Bit!
As I told before: the mfi-Software runs -- for a while... but then something breaks: If I try it with Firefox on https://localhost:6443 i have to wait very long and finally get a timeout for the mfi-Login

I run the Unifi-Controller + mfi-Software on the same machine and I virtualized everything with Proxmox.

When I run ps aux |grep mongo everything seems to be ok!? The only strange thing is this message in /var/log/mfi/mongod.log: Please make at least 3379MB available in /usr/lib/mfi/data/db/journal or use --smallfiles
15926 Insufficient free space for journals, terminating.

So: What to do?

I used 4GB RAM with this VM and the HDD isn't nearly full...
SuperUser
Posts: 15,105
Registered: ‎08-07-2008
Kudos: 7313
Solutions: 415
Contributions: 1

Re: Issues Starting MFi + MongoDB on Ubuntu 14.04

That's the issue

MONGO needs a huge amount of disk space.

It calculates what it thinks it needs for your DB and auto expands.

If possible add more space or another drive and change the db path to use it.

With another drive. on 12.04 /var/lib/mfi/system.properties

This is the line from my system

unifi.db.dir=/media/mfi/db

The folder /media and all folders inside need to have root perms. So run as the root user if you set one up.

You can also copy your existing DB files into this folder. Shut down mfi before doing so.

Walks in, tells it like it is, walks out , milks a cow
Emerging Member
Posts: 77
Registered: ‎03-04-2015
Kudos: 37
Solutions: 5

Re: Issues Starting MFi + MongoDB on Ubuntu 14.04

Just commenting that whiterabbit's disk-space issue and it's solution seems unrelated to the OP, race-condition startup issue. The latter still has not been solved.

New Member
Posts: 20
Registered: ‎06-16-2015
Kudos: 2

Re: Issues Starting MFi + MongoDB on Ubuntu 14.04

Ok, sorry --- you are right! There were ~ 3 GB free which was not enough. I enlarged the partition with Proxmox and gparted and mfi runs again.

Thanks for help ... my problem is solved ... not so the OPs problem...

Emerging Member
Posts: 80
Registered: ‎04-07-2015
Kudos: 24
Solutions: 1

Re: Issues Starting MFi + MongoDB on Ubuntu 14.04

used the following command

 

netstat -lnptu

 

and did see witch process using 6443 and kill that and the used

 

service mfi start

 

and it worked.

Highlighted
Emerging Member
Posts: 40
Registered: ‎03-25-2013
Kudos: 10

Re: Issues Starting MFi + MongoDB on Ubuntu 14.04

Interesting ... same issue, different cause and solution.

 

I think if your /var/log/unifi/mongodb.log has got too big, mongodb takes too long to start (or just fails), leading to the same symptom. In my case I had to rename it so a new smaller logfile took its place, and mFi then started.

Reply