I recently started using god rather than monit for process monitoring. god lets me be a bit more expressive with how I want processes monitored using the the power of Ruby.
The current project I am working on has a number of tasks that I want processed asynchronously so I will setup god to monitor my delayed_jobs. If you are not familiar with awesome delayed_job gem, watch the excellent Railscast tutorial.
First install the god gem:
Next we will create a Redhat compatible init script for god:
$ sudo vi /etc/init.d/god
#!/bin/bash
#
# God
#
# chkconfig: - 85 15
# description: start, stop, restart God (bet you feel powerful)
#
RETVAL=0
case "$1" in
start)
/usr/bin/god -P /var/run/god.pid -l /var/log/god.log
/usr/bin/god load /etc/god.conf
RETVAL=$?
;;
stop)
kill `cat /var/run/god.pid`
RETVAL=$?
;;
restart)
kill `cat /var/run/god.pid`
/usr/bin/god -P /var/run/god.pid -l /var/log/god.log
/usr/bin/god load /etc/god.conf
RETVAL=$?
;;
status)
/usr/bin/god status
RETVAL=$?
;;
*)
echo "Usage: god {start|stop|restart|status}"
exit 1
;;
esac
exit $RETVALNow adjust the permissions, and set the init script to start on system boot:
$ sudo chmod a+x /etc/init.d/god
$ sudo /sbin/chkconfig --add god
$ sudo /sbin/chkconfig --level 345 god on
Before we start god up, we need to create a configuration file that tells it what configuration files to load:
$ sudo vi /etc/god.conf
God.load "/srv/apps/your_app/current/config/god/*.god"
You will need to adjust the above depending on how you have your app setup. When working in a Rails project I like to put my god scripts in config/god.
We will use a script from the guys at github to monitor our job daemon. I tweaked it slightly to have less workers, and to set the environment properly.
RAILS_ROOT = "/srv/apps/your_app/current"
1.times do |num|
God.watch do |w|
w.name = "dj-#{num}"
w.group = 'dj'
w.interval = 30.seconds
w.start = "rake -f #{RAILS_ROOT}/Rakefile RAILS_ENV=production jobs:work"
w.uid = 'your_app_user'
w.gid = 'your_app_user'
# retart if memory gets too high
w.transition(:up, :restart) do |on|
on.condition(:memory_usage) do |c|
c.above = 300.megabytes
c.times = 2
end
end
# determine the state on startup
w.transition(:init, { true => :up, false => :start }) do |on|
on.condition(:process_running) do |c|
c.running = true
end
end
# determine when process has finished starting
w.transition([:start, :restart], :up) do |on|
on.condition(:process_running) do |c|
c.running = true
c.interval = 5.seconds
end
# failsafe
on.condition(:tries) do |c|
c.times = 5
c.transition = :start
c.interval = 5.seconds
end
end
# start if process is not running
w.transition(:up, :start) do |on|
on.condition(:process_running) do |c|
c.running = false
end
end
end
endIt’s now time to start the daemon:
$ sudo /etc/init.d/god start
$ sudo /etc/init.d/god status
dj:
dj-0: up
Looks good! If you want to make sure it’s working, kill the rake task running jobs:work. god will see that it is stopped and automatically restart it!
Tags: centos, delayed_job, god
There are times when you need to monitor the queries coming in to MySQL, but turning on query logging would create too much of a disk I/O hit, or you can’t restart the server to setup MySQL Proxy. Instead we can just monitor the network traffic and extract data that might be interesting using tcpdump and an inline perl script:
sudo tcpdump -i lo -s 0 -l -w - dst port 3306 | strings | perl -e '
while(<>) { chomp; next if /^[^ ]+[ ]*$/;
if(/^(SELECT|UPDATE|DELETE|INSERT|SET|COMMIT|ROLLBACK|CREATE|DROP|ALTER)/i) {
if (defined $q) { print "$q\n"; }
$q=$_;
} else {
$_ =~ s/^[ \t]+//; $q.=" $_";
}
}'This will only work for clients communicating via TCP – if you are connecting through ‘localhost’ you will be going through a unix socket instead. If you switch ‘localhost’ to ‘127.0.0.1′ then your queries will go through the network stack.
If you just want to dump the traffic to a file for a little bit and analyze it later, do this instead:
sudo tcpdump -i lo port 3306 -s 65535 -x -n -q -tttt> tcpdump.out
You can then use mk-query-digest from Maatkit with–type=tcpdump. See more about this at the MySQL Performance Blog.
Tags: tcpdump
During development on a linux system, you probably install many packages using your favorite package manager. When you have to use a new system, or reimage your current one, it can be a pain to remember all the packages you had setup. One solution is to keep a list of the packages installed after the OS load, and then periodically generate a list of what has been added since.
On a freshly installed system, create the starting baseline list of packages:
# On Debian based systems(Ubuntu):
dpkg --get-selections > packages-alpha.txt
# Or CentOS/Fedora:
yum list installed > packages-alpha.txt
You can run the command again at a later time, concatenating the output into a different file so you can view what has changed since the original system setup. Use a diff tool like diff3, vimdiff, or meld:
meld packages-alpha.txt packages-omega.txt
On Debian systems, once you have that file you can use it in a new or different system to mark packages to install using the –set-selection parameter:
dpkg --set-selections < packages-omega.txt
sudo apt-get upgrade
Tags: aptitude, yum
I work with a wide variety of client server deployments, and sometimes it isn’t obvious(via uname) what distribution and version a server is running. Here is a quick list of common files which contain that information:
Debian /etc/debian_release, /etc/debian_version,
Fedora /etc/fedora-release
Gentoo /etc/gentoo-release
Mandrake /etc/mandrake-release
Novell SUSE /etc/SUSE-release
Red Hat /etc/redhat-release, /etc/redhat_version
Slackware /etc/slackware-release, /etc/slackware-version
Solaris/Sparc /etc/release
Sun JDS /etc/sun-release
Ubuntu /etc/lsb-release
UnitedLinux /etc/UnitedLinux-release
Yellow dog /etc/yellowdog-release
Here is a simple script I use to create subversion repositories and setup common hook scripts. This assumes you are making svn available through an HTTPS/Apache/DAV.
My directory svn structure looks like the following:
$ tree -L 2 /home/svn
/home/svn
|-- conf
| |-- permissions.conf
| `-- mailer.conf
|-- repos
| |-- repo1
| |-- repo2
| `-- repo3
`-- scripts
|-- post-commit
|-- pre-commit
`-- svncreate.shThere are 3 main directories:
repos – This is where the actual subversion repositories are stored
scripts – Various scripts, including the following creation script, and also various hook scripts that repositories can symlink to
conf – Various configuration files that might contain mailer configuration or repository permissions(both outside the scope of this article)
The following script will create the new repository, set proper user and permissions, symlink common pre and pos commit scripts, and then make the initial import that contains the trunk/branches/tags structure.
#!/bin/bash
REPOS_URL=https://svn.example.com
REPOS_PATH=/home/svn/repos
SCRIPTS_PATH=/home/svn/scripts
APACHE_USER=www-data
SVNADMIN=`which svnadmin`
EXPECTED_ARGS=1
E_BADARGS=65
REPO=$1
if [ $# -ne $EXPECTED_ARGS ]
then
echo "Usage: $0 reponame"
exit $E_BADARGS
fi
$SVNADMIN create --fs-type fsfs $REPOS_PATH/$REPO
rm -rf /tmp/subversion-layout/
mkdir -pv /tmp/subversion-layout/{trunk,branches,tags}
ln -s $SCRIPTS_PATH/pre-commit $REPOS_PATH/$1/hooks/pre-commit
ln -s $SCRIPTS_PATH/post-commit $REPOS_PATH/$1/hooks/post-commit
chown $APACHE_USER:$APACHE_USER -R $REPOS_PATH/$1
chmod -R 2775 $REPOS_PATH/$1
svn import -m "Initial Import" /tmp/subversion-layout/ $REPOS_URL/$REPO
rm -rf /tmp/subversion-layout/As you root you just run it as /home/svn/scripts/svncreate.sh reponame
Here is a subversion pre-commit hook script we use on PHP projects to make sure the developer making the commit is providing a meaningful description, and then PHP lint is run on each PHP script to make sure it will compile correctly.
#!/bin/bash
REPOS="$1"
TXN="$2"
PHP="/usr/bin/php"
SVNLOOK="/usr/bin/svnlook"
AWK="/usr/bin/awk"
GREP="/bin/egrep"
SED="/bin/sed"
CHANGED=`$SVNLOOK changed -t "$TXN" "$REPOS" | $AWK '{print $2}' | $GREP \\.php$`
for FILE in $CHANGED
do
MESSAGE=`$SVNLOOK cat -t "$TXN" "$REPOS" "$FILE" | $PHP -l`
if [ $? -ne 0 ]
then
echo 1>&2
echo "***********************************" 1>&2
echo "PHP error in: $FILE:" 1>&2
echo `echo "$MESSAGE" | $SED "s| -| $FILE|g"` 1>&2
echo "***********************************" 1>&2
exit 1
fi
done
# Make sure that the log message contains some text.
SVNLOOKOK=1
SVNLOOK=/usr/bin/svnlook
$SVNLOOK log -t "$TXN" "$REPOS" | \
grep "[a-zA-Z0-9]" > /dev/null || SVNLOOKOK=0
if [ $SVNLOOKOK = 0 ]; then
echo Empty log messages are not allowed. Please provide a proper log message. 1>&2
exit 1
fi
# Make sure text might be meaningful
LOGMSGLEN=$($SVNLOOK log -t "$TXN" "$REPOS" | grep [a-zA-Z0-9] | wc -c)
if [ "$LOGMSGLEN" -lt 6 ]; then
echo -e "Please provide a meaningful comment when committing changes." 1>&2
exit 1
fi
# All checks passed, so allow the commit.
exit 0Tags: php, svn
There are various ways to make sure something is run at system startup – Redhat has /etc/rc.local script, and it and many others have /etc/init.d/* scripts – but many times you might not have access to those files or creating init scripts might be overkill for your needs.
People are always amazed when I tell them they can achieve this basic functionality by using cron. Many of our websites use Sphinx, the excellent full text indexer, to allow site searches. Should the server ever reboot, we need to make multiple search daemons start back up. Take the following line from a crontab:
crontab -l
@reboot /usr/local/bin/searchd --config ~/conf/sphinx.conf
This will make sure the searchd daemon starts on bootup.
Also there are a few other shortcuts you can use:
@yearly Run once a year, "0 0 1 1 *".
@annually (same as @yearly)
@monthly Run once a month, "0 0 1 * *".
@weekly Run once a week, "0 0 * * 0".
@daily Run once a day, "0 0 * * *".
@midnight (same as @daily)
@hourly Run once an hour, "0 * * * *".
See man 5 cron for more information.
(And to be pedantic, @reboot is run when cron is started or restarted, not necessarily the OS itself. So /etc/init.d/cron restart would trigger that line to be run. You may want to keep that in mind.)
Tags: cron, linux
We have a few cPanel servers deployed and to visually monitor performance we use the Munin monitoring tool. We were having issues where the MySQL graphs would stop updating. The bug has to do with one of the perl libraries the script uses, which causes the MySQL plugin to have problems location the mysqladmin program.
To fix these, create or edit the following file:
vi /etc/munin/plugin-conf.d/cpanel.conf
It should contain the following:
[mysql*]
user root
group wheel
env.mysqladmin /usr/bin/mysqladmin
env.mysqlopts --defaults-extra-file=/root/.my.cnf
This would load the username and password from root’s mysql config. If that file doesn’t exist you can create it:
[client]
user="root"
pass="secret!password"
Now just restart Munin, wait a a little bit, and the graphs should start populating again!
sudo /etc/init.d/munin-node restart
Tags: cpanel, munin, mysql
Recently I used GParted to resize an NTFS disk to dual boot Ubuntu and Windows XP. I finished the Ubuntu installation and everything seemed to be working fine until I tried to boot back into XP. Windows reported there might be disk corruption, ran chkdsk, and chkdsk ended up freezing. I rebooted again and saw that the grub bootloader was now also freezing. Delightful. Although I didn’t think chkdsk modified the MBR, upon further research in some cases it does(when run with the /r switch) and in this case ends up corrupting the MBR. To fix this issue you can perform the following:
Boot with an Ubuntu Live CD and open up a terminal.
sudo grub
grub> find /boot/grub/stage1
Note the hdx number and partition number to it’s right. Now type the following commands into the grub prompt:
grub> root (hdx, y)
grub> setup (hdx)
grub> quit
Note whitespace is important here. Now you can reboot. When Windows asks to run chkdsk hit a key to cancel. Once in Windows, open a command prompt and type:
This will schedule a disk check on reboot that will not alter the MBR. Reboot and allow the disk check to complete. It may automatically reboot your system again, but once that is finished both OS’s should boot fine from now on.
Tags: chkdsk, grub, linux, ubuntu
I have been using Doug Hellman’s useful svnautobackup script for our subversion backups. I think it is a pretty useful script, but it only focuses on backups and not restoration.
During a recent server upgrade I needed to restore a large amount of repositories from our dumpfiles. I made the following shell script in php that will loop through all of the backup dumps and restore them. There is a similar tool that uses ruby, but it can only restore one repository and I do not know enough ruby to modify it 
#!/usr/bin/php -q
<?php
// edit this
$repo_dir = '/svn/';
$backup_dir = '/home/backup/svn/';
// don't edit this
foreach(new DirectoryIterator($backup_dir) as $file)
{
if(isDir() && !$file->isDot())
{
// get files
$dumpfiles = glob($backup_dir.$file.'/dumpfile*.bzip2');
// sort them
natsort($dumpfiles);
// create the repo if not exists
if(!is_dir($repo_dir.$file))
{
$cmd = "svnadmin create {$repo_dir}{$file}";
system(escapeshellcmd($cmd));
}
// restore dump files
foreach($dumpfiles as $dumpfile)
{
$cmd = "bzcat {$dumpfile} | svnadmin load {$repo_dir}{$file}";
system(escapeshellcmd($cmd));
}
}
}For reference, here is the script that runs nightly though cron
#!/bin/bash
files=`ls /svn`
for file in $files
do
if [ -d /svn/$file ]
then
mkdir -p $file
/home/backup/scripts/svnautobackup/svnbackup.sh -v -i 100 --history-file /home/backup/svn/$file-hist --out-dir /home/backup/svn/$file /svn/$file
fi
doneTags: backups, linux, php, subversion, svn
