Today I was going through and updating our Sensu install to version 0.17.2-1. Everything was going smoothly until, one host complained about the sensu-client failing to start.
Tailing the sensu-client.log file, I found this at the end.
/opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:859:in 'open_udp_socket': no datagram socket (RuntimeError)
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:859:in 'open_datagram_socket'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-0.17.2/lib/sensu/client/process.rb:329:in `setup_sockets'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-0.17.2/lib/sensu/client/process.rb:370:in `start'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-0.17.2/lib/sensu/client/process.rb:19:in `block in run'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `call'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run_machine'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-0.17.2/lib/sensu/client/process.rb:18:in `run'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-0.17.2/bin/sensu-client:10:in `<top (required)>'
from /opt/sensu/bin/sensu-client:23:in `load'
from /opt/sensu/bin/sensu-client:23:in `<main>'
Figured that the eventmachine gem was messed up. Tried to run /opt/sensu/embedded/bin/gem install eventmachine resulted in:
Building native extensions. This could take a while...
ERROR: Error installing eventmachine:
ERROR: Failed to build gem native extension.
/opt/sensu/embedded/bin/ruby extconf.rb
checking for main() in -lssl... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers. Check the mkmf.log file for more details. You may
need configuration options.
Provided configuration options:
--with-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=.
--curdir
--ruby=/opt/sensu/embedded/bin/ruby
--with-openssl-config
--without-openssl-config
--with-pkg-config
--without-pkg-config
--with-ssl-dir
--without-ssl-dir
--with-ssl-include
--without-ssl-include=${ssl-dir}/include
--with-ssl-lib
--without-ssl-lib=${ssl-dir}/
--with-ssllib
--without-ssllib
/opt/sensu/embedded/lib/ruby/2.0.0/mkmf.rb:434:in `try_do': The compiler failed to generate an executable file. (RuntimeError)
You have to install development tools first.
from /opt/sensu/embedded/lib/ruby/2.0.0/mkmf.rb:519:in `try_link0'
from /opt/sensu/embedded/lib/ruby/2.0.0/mkmf.rb:534:in `try_link'
from /opt/sensu/embedded/lib/ruby/2.0.0/mkmf.rb:720:in `try_func'
from /opt/sensu/embedded/lib/ruby/2.0.0/mkmf.rb:950:in `block in have_library'
from /opt/sensu/embedded/lib/ruby/2.0.0/mkmf.rb:895:in `block in checking_for'
from /opt/sensu/embedded/lib/ruby/2.0.0/mkmf.rb:340:in `block (2 levels) in postpone'
from /opt/sensu/embedded/lib/ruby/2.0.0/mkmf.rb:310:in `open'
from /opt/sensu/embedded/lib/ruby/2.0.0/mkmf.rb:340:in `block in postpone'
from /opt/sensu/embedded/lib/ruby/2.0.0/mkmf.rb:310:in `open'
from /opt/sensu/embedded/lib/ruby/2.0.0/mkmf.rb:336:in `postpone'
from /opt/sensu/embedded/lib/ruby/2.0.0/mkmf.rb:894:in `checking_for'
from /opt/sensu/embedded/lib/ruby/2.0.0/mkmf.rb:945:in `have_library'
from extconf.rb:5:in `block in check_libs'
from extconf.rb:5:in `each'
from extconf.rb:5:in `all?'
from extconf.rb:5:in `check_libs'
from extconf.rb:39:in `manual_ssl_config'
from extconf.rb:64:in `<main>'
Gem files will remain installed in /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.7 for inspection.
Results logged to /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.7/ext/gem_make.out
Welp. After some searching, it seems that eventmachine needs a compiler. Simple fix on Debian, aptitude install build-essential
Boom! /opt/sensu/embedded/bin/gem install eventmachine works perfectly. Attempting to start the sensu-client and it was still failing. Whhhyyyyyyy...
Back to the searching. Looks like the issue was SUPER simple. The error that pointed me in the right directions was from the sensu-client log, ``open_udp_socket': no datagram socket (RuntimeError). Running netstat -tulnp | grep 3030on the problem host quickly showed me that there was a sensu check just sitting on the port. One simple kill command, double check the port, and anotherservice sensu-client restart` and presto back online.
Hope this saves someone a headache.