We recently had quite some issues with my a customers icinga2 installation, where icingaweb2 was constantly complaining about the IDO Backend is not running.
The setup was as follows:
icinga2 + icingaweb2 was running on server „webnode.local“
icinga2 database and icingaweb2 databases were on „database.local“
The customer complained about the error message, since it looked like a „false error“ since all reports showed up in icinga2 db and in the webinterface.
Only that annoying message was popping up red on the bottom of icingaweb2 and in the monitoring/info page:
„Backend icinga2-ido is not running“
After some investigations that led nowhere, we came back to the original IDO Health page and found the following query.
That one brought no results at all, so we modified the query into different steps:
SELECT status_update_time FROM icinga_programstatus –> result!
SELECT status_update_time FROM icinga_programstatus ps JOIN icinga_instances i ON ps.instance_id=i.instance_id –> result!
SELECT status_update_time FROM icinga_programstatus ps JOIN icinga_instances i ON ps.instance_id=i.instance_id WHERE (UNIX_TIMESTAMP(ps.status_update_time) > UNIX_TIMESTAMP(NOW())-60) –> no result !!!!!
We modified the „Difftime“ from 60 to 5000 and we got a result. That pointed us to a possible time gap which led to an analysis of both servers time and ntp settings.
And in fact – the server had a timegap of more than 70 seconds (since ntp seemed to be dead on the database server).
Fixing NTP and re-syncing the times on both boxes finally cleared the error message out.
