Jetpack Flight Log » sysadmin

Making monit, delayed_job, and bundler play nice together

Brian Racer — Wed, 19 May 2010 22:21:53 +0000

Recently I was having a heck of a time getting monit to start my delayed_job instances. I was using the monit template that came with delayed job, it looks something like this:

check process delayed_job_bandwith_prod 
  with pidfile /home/bandwith/apps/production/shared/pids/delayed_job.pid
  start program = "/usr/bin/env RAILS_ENV=production /home/bandwith/apps/production/current/script/delayed_job start" as uid bandwith and gid bandwith 
  stop program  = "/usr/bin/env RAILS_ENV=production /home/bandwith/apps/production/current/script/delayed_job stop" as uid bandwith and gid bandwith

This did not work however, and after quite a bit of debugging I found there are a couple of issues you might need to be aware of:

1. Your $PATH

monit starts things with a ‘spartan path‘ of:

/bin:/usr/bin:/sbin:/usr/sbin

My ruby happens to be in /usr/local/bin, so we will need to pass that in too:

start program = "/usr/bin/env PATH=/usr/local/bin:PATH RAILS_ENV=production /var/www/apps/{app_name}/current/script/delayed_job start"

2. monit doesn’t define a $HOME environment variable

Even though we are starting these processes with uids and guids specified, that doesn’t actually load the users shell. With no $HOME env variable, bundler can’t find where your gems are installed and thinks they are all missing. I ended up just putting the variable in the monit command, another option might be running su -c ‘{command}’ (I didn’t test that).

So putting that all together you get the following which should make everything work:

check process delayed_job_bandwith_prod 
  with pidfile /home/bandwith/apps/production/shared/pids/delayed_job.pid
  start program = "/usr/bin/env HOME=/home/bandwith PATH=/usr/local/bin:$PATH RAILS_ENV=production /home/bandwith/apps/production/current/script/delayed_job start" as uid bandwith and gid bandwith 
  stop program  = "/usr/bin/env HOME=/home/bandwith PATH=/usr/local/bin:$PATH RAILS_ENV=production /home/bandwith/apps/production/current/script/delayed_job stop" as uid bandwith and gid bandwith

I would be interested to know if anyone has any better suggestions, but this seems to be working nicely.

Monitoring delayed_job with god on CentOS

Brian Racer — Mon, 02 Nov 2009 22:24:51 +0000

I recently started using god rather than monit for process monitoring. god lets me be a bit more expressive with how I want processes monitored using the the power of Ruby.

The current project I am working on has a number of tasks that I want processed asynchronously so I will setup god to monitor my delayed_jobs. If you are not familiar with awesome delayed_job gem, watch the excellent Railscast tutorial.

First install the god gem:

$ sudo gem install god

Next we will create a Redhat compatible init script for god:

$ sudo vi /etc/init.d/god
 
#!/bin/bash
#
# God
#
# chkconfig: - 85 15
# description: start, stop, restart God (bet you feel powerful)
#
 
RETVAL=0
 
case "$1" in
    start)
      /usr/bin/god -P /var/run/god.pid -l /var/log/god.log
      /usr/bin/god load /etc/god.conf
      RETVAL=$?
      ;;
    stop)
      kill `cat /var/run/god.pid`
      RETVAL=$?
      ;;
    restart)
      kill `cat /var/run/god.pid`
      /usr/bin/god -P /var/run/god.pid -l /var/log/god.log
      /usr/bin/god load /etc/god.conf
      RETVAL=$?
      ;;
    status)      
      /usr/bin/god status
      RETVAL=$?
      ;;
    *)
      echo "Usage: god {start|stop|restart|status}"
      exit 1
  ;;
esac
 
exit $RETVAL

(adapted from debian version at http://mylescarrick.com/articles/simple_delayed_job_with_god)

Now adjust the permissions, and set the init script to start on system boot:

$ sudo chmod a+x /etc/init.d/god
$ sudo /sbin/chkconfig --add god
$ sudo /sbin/chkconfig --level 345 god on

Before we start god up, we need to create a configuration file that tells it what configuration files to load:

$ sudo vi /etc/god.conf
 
God.load "/srv/apps/your_app/current/config/god/*.god"

You will need to adjust the above depending on how you have your app setup. When working in a Rails project I like to put my god scripts in config/god.

We will use a script from the guys at github to monitor our job daemon. I tweaked it slightly to have less workers, and to set the environment properly.

RAILS_ROOT = "/srv/apps/your_app/current"
 
1.times do |num|
  God.watch do |w|
    w.name = "dj-#{num}"
    w.group = 'dj'
    w.interval = 30.seconds
    w.start = "rake -f #{RAILS_ROOT}/Rakefile RAILS_ENV=production jobs:work"
 
    w.uid = 'your_app_user'
    w.gid = 'your_app_user'
 
    # retart if memory gets too high
    w.transition(:up, :restart) do |on|
      on.condition(:memory_usage) do |c|
        c.above = 300.megabytes
        c.times = 2
      end
    end
 
    # determine the state on startup
    w.transition(:init, { true => :up, false => :start }) do |on|
      on.condition(:process_running) do |c|
        c.running = true
      end
    end
 
    # determine when process has finished starting
    w.transition([:start, :restart], :up) do |on|
      on.condition(:process_running) do |c|
        c.running = true
        c.interval = 5.seconds
      end
 
      # failsafe
      on.condition(:tries) do |c|
        c.times = 5
        c.transition = :start
        c.interval = 5.seconds
      end
    end
 
    # start if process is not running
    w.transition(:up, :start) do |on|
      on.condition(:process_running) do |c|
        c.running = false
      end
    end
  end
end

It’s now time to start the daemon:

$ sudo /etc/init.d/god start
$ sudo /etc/init.d/god status
dj:
  dj-0: up

Looks good! If you want to make sure it’s working, kill the rake task running jobs:work. god will see that it is stopped and automatically restart it!

Unobtrusive viewing of MySQL queries with tcpdump

Brian Racer — Wed, 16 Sep 2009 17:33:32 +0000

There are times when you need to monitor the queries coming in to MySQL, but turning on query logging would create too much of a disk I/O hit, or you can’t restart the server to setup MySQL Proxy. Instead we can just monitor the network traffic and extract data that might be interesting using tcpdump and an inline perl script:

sudo tcpdump -i lo -s 0 -l -w - dst port 3306 | strings | perl -e '
while(<>) { chomp; next if /^[^ ]+[ ]*$/;
  if(/^(SELECT|UPDATE|DELETE|INSERT|SET|COMMIT|ROLLBACK|CREATE|DROP|ALTER)/i) {
    if (defined $q) { print "$q\n"; }
    $q=$_;
  } else {
    $_ =~ s/^[ \t]+//; $q.=" $_";
  }
}'

This will only work for clients communicating via TCP – if you are connecting through ‘localhost’ you will be going through a unix socket instead. If you switch ‘localhost’ to ’127.0.0.1′ then your queries will go through the network stack.

If you just want to dump the traffic to a file for a little bit and analyze it later, do this instead:

sudo tcpdump -i lo port 3306 -s 65535 -x -n -q -tttt> tcpdump.out

You can then use mk-query-digest from Maatkit with–type=tcpdump. See more about this at the MySQL Performance Blog.

Linux Tip: Keep track of packages you have installed

Brian Racer — Tue, 15 Sep 2009 01:12:01 +0000

During development on a linux system, you probably install many packages using your favorite package manager. When you have to use a new system, or reimage your current one, it can be a pain to remember all the packages you had setup. One solution is to keep a list of the packages installed after the OS load, and then periodically generate a list of what has been added since.

On a freshly installed system, create the starting baseline list of packages:

# On Debian based systems(Ubuntu):
dpkg --get-selections > packages-alpha.txt
 
# Or CentOS/Fedora:
yum list installed > packages-alpha.txt

You can run the command again at a later time, concatenating the output into a different file so you can view what has changed since the original system setup. Use a diff tool like diff3, vimdiff, or meld:

meld packages-alpha.txt packages-omega.txt

On Debian systems, once you have that file you can use it in a new or different system to mark packages to install using the –set-selection parameter:

dpkg --set-selections < packages-omega.txt
sudo apt-get upgrade

Determining linux distribution

Brian Racer — Wed, 19 Aug 2009 18:00:40 +0000

I work with a wide variety of client server deployments, and sometimes it isn’t obvious(via uname) what distribution and version a server is running. Here is a quick list of common files which contain that information:

Debian          /etc/debian_release, /etc/debian_version,
Fedora          /etc/fedora-release
Gentoo          /etc/gentoo-release
Mandrake        /etc/mandrake-release
Novell SUSE     /etc/SUSE-release
Red Hat         /etc/redhat-release, /etc/redhat_version
Slackware       /etc/slackware-release, /etc/slackware-version
Solaris/Sparc   /etc/release
Sun JDS         /etc/sun-release
Ubuntu          /etc/lsb-release
UnitedLinux     /etc/UnitedLinux-release
Yellow dog      /etc/yellowdog-release

Subversion repository creation script

Brian Racer — Wed, 22 Jul 2009 18:12:03 +0000

Here is a simple script I use to create subversion repositories and setup common hook scripts. This assumes you are making svn available through an HTTPS/Apache/DAV.

My directory svn structure looks like the following:

$ tree -L 2 /home/svn
 
/home/svn
|-- conf
|   |-- permissions.conf
|   `-- mailer.conf
|-- repos
|   |-- repo1
|   |-- repo2
|   `-- repo3 
`-- scripts
    |-- post-commit
    |-- pre-commit
    `-- svncreate.sh

There are 3 main directories:

repos – This is where the actual subversion repositories are stored
scripts – Various scripts, including the following creation script, and also various hook scripts that repositories can symlink to
conf – Various configuration files that might contain mailer configuration or repository permissions(both outside the scope of this article)

The following script will create the new repository, set proper user and permissions, symlink common pre and pos commit scripts, and then make the initial import that contains the trunk/branches/tags structure.

#!/bin/bash
 
REPOS_URL=https://svn.example.com
REPOS_PATH=/home/svn/repos
SCRIPTS_PATH=/home/svn/scripts
APACHE_USER=www-data
 
SVNADMIN=`which svnadmin`
EXPECTED_ARGS=1
E_BADARGS=65
REPO=$1
 
if [ $# -ne $EXPECTED_ARGS ]
then
  echo "Usage: $0 reponame"
  exit $E_BADARGS
fi
 
$SVNADMIN create --fs-type fsfs $REPOS_PATH/$REPO
 
rm -rf /tmp/subversion-layout/
mkdir -pv /tmp/subversion-layout/{trunk,branches,tags}
 
ln -s $SCRIPTS_PATH/pre-commit $REPOS_PATH/$1/hooks/pre-commit
ln -s $SCRIPTS_PATH/post-commit $REPOS_PATH/$1/hooks/post-commit
 
chown $APACHE_USER:$APACHE_USER -R $REPOS_PATH/$1
chmod -R 2775 $REPOS_PATH/$1
 
svn import -m "Initial Import" /tmp/subversion-layout/ $REPOS_URL/$REPO
rm -rf /tmp/subversion-layout/

As you root you just run it as /home/svn/scripts/svncreate.sh reponame

Useful PHP Subversion Commit Hook

Brian Racer — Sun, 19 Jul 2009 18:02:26 +0000

Here is a subversion pre-commit hook script we use on PHP projects to make sure the developer making the commit is providing a meaningful description, and then PHP lint is run on each PHP script to make sure it will compile correctly.

#!/bin/bash
 
REPOS="$1"
TXN="$2"
 
PHP="/usr/bin/php"
SVNLOOK="/usr/bin/svnlook"
AWK="/usr/bin/awk"
GREP="/bin/egrep"
SED="/bin/sed"
 
CHANGED=`$SVNLOOK changed -t "$TXN" "$REPOS" | $AWK '{print $2}' | $GREP \\.php$`
 
for FILE in $CHANGED
do
    MESSAGE=`$SVNLOOK cat -t "$TXN" "$REPOS" "$FILE" | $PHP -l`
    if [ $? -ne 0 ]
    then
        echo 1>&2
        echo "***********************************" 1>&2
        echo "PHP error in: $FILE:" 1>&2
        echo `echo "$MESSAGE" | $SED "s| -| $FILE|g"` 1>&2
        echo "***********************************" 1>&2
        exit 1
    fi
done
 
# Make sure that the log message contains some text.
SVNLOOKOK=1
SVNLOOK=/usr/bin/svnlook
$SVNLOOK log -t "$TXN" "$REPOS" | \
   grep "[a-zA-Z0-9]" > /dev/null || SVNLOOKOK=0
if [ $SVNLOOKOK = 0 ]; then
  echo Empty log messages are not allowed. Please provide a proper log message. 1>&2
  exit 1
fi
 
# Make sure text might be meaningful
LOGMSGLEN=$($SVNLOOK log -t "$TXN" "$REPOS" | grep [a-zA-Z0-9] | wc -c)
if [ "$LOGMSGLEN" -lt 6 ]; then
  echo -e "Please provide a meaningful comment when committing changes." 1>&2
  exit 1
fi
 
# All checks passed, so allow the commit.
exit 0

Useful Linux Trick: cron @reboot

Brian Racer — Fri, 17 Jul 2009 17:19:03 +0000

There are various ways to make sure something is run at system startup – Redhat has /etc/rc.local script, and it and many others have /etc/init.d/* scripts – but many times you might not have access to those files or creating init scripts might be overkill for your needs.

People are always amazed when I tell them they can achieve this basic functionality by using cron. Many of our websites use Sphinx, the excellent full text indexer, to allow site searches. Should the server ever reboot, we need to make multiple search daemons start back up. Take the following line from a crontab:

crontab -l
 
@reboot /usr/local/bin/searchd --config ~/conf/sphinx.conf

This will make sure the searchd daemon starts on bootup.

Also there are a few other shortcuts you can use:

@yearly        Run once a year, "0 0 1 1 *".
@annually      (same as @yearly)
@monthly       Run once a month, "0 0 1 * *".
@weekly        Run once a week, "0 0 * * 0".
@daily         Run once a day, "0 0 * * *".
@midnight      (same as @daily)
@hourly        Run once an hour, "0 * * * *".

See man 5 cron for more information.

(And to be pedantic, @reboot is run when cron is started or restarted, not necessarily the OS itself. So /etc/init.d/cron restart would trigger that line to be run. You may want to keep that in mind.)

How to fix Munin’s MySQL Graph on cPanel

Brian Racer — Thu, 16 Jul 2009 16:54:51 +0000

We have a few cPanel servers deployed and to visually monitor performance we use the Munin monitoring tool. We were having issues where the MySQL graphs would stop updating. The bug has to do with one of the perl libraries the script uses, which causes the MySQL plugin to have problems location the mysqladmin program.

To fix these, create or edit the following file:

vi /etc/munin/plugin-conf.d/cpanel.conf

It should contain the following:

[mysql*]
user root
group wheel
env.mysqladmin /usr/bin/mysqladmin
env.mysqlopts --defaults-extra-file=/root/.my.cnf

This would load the username and password from root’s mysql config. If that file doesn’t exist you can create it:

sudo vi /root/.my.cnf

[client]
user="root"
pass="secret!password"

Now just restart Munin, wait a a little bit, and the graphs should start populating again!

sudo /etc/init.d/munin-node restart

chkdsk and Grub

Brian Racer — Mon, 13 Jul 2009 23:41:28 +0000

Recently I used GParted to resize an NTFS disk to dual boot Ubuntu and Windows XP. I finished the Ubuntu installation and everything seemed to be working fine until I tried to boot back into XP. Windows reported there might be disk corruption, ran chkdsk, and chkdsk ended up freezing. I rebooted again and saw that the grub bootloader was now also freezing. Delightful. Although I didn’t think chkdsk modified the MBR, upon further research in some cases it does(when run with the /r switch) and in this case ends up corrupting the MBR. To fix this issue you can perform the following:

Boot with an Ubuntu Live CD and open up a terminal.

sudo grub
 
grub> find /boot/grub/stage1

Note the hdx number and partition number to it’s right. Now type the following commands into the grub prompt:

grub> root (hdx, y)
 
grub> setup (hdx)
 
grub> quit

Note whitespace is important here. Now you can reboot. When Windows asks to run chkdsk hit a key to cancel. Once in Windows, open a command prompt and type:

chkntfs /c c:

This will schedule a disk check on reboot that will not alter the MBR. Reboot and allow the disk check to complete. It may automatically reboot your system again, but once that is finished both OS’s should boot fine from now on.