Pretty Paging in Rails Console

words by Brian Racer

When using irb or Rails console I use the awesome_print gem to get nicer colorized output. I also like to use looksee to examine method lookup paths which the gem colorizes nicely. For large objects looksee can produce a lot of output and if there is more output than your terminal can display at once it will get handed off to your system’s pager (probably less). I was having an issue while in Rails console when the output got paged it would show the ANSI color escape codes rather than colorized text (this didn’t happen in irb for whatever reason).

Luckily less has a flag that will repaint the screen when paging. To make it a default you need to export a LESS variable to your shell’s environment. Something like this:

export LESS="-R"

Just throw that in your ~/.bashrc or dotfiles and you’re all set!


Making monit, delayed_job, and bundler play nice together

words by Brian Racer

Recently I was having a heck of a time getting monit to start my delayed_job instances. I was using the monit template that came with delayed job, it looks something like this:

check process delayed_job_bandwith_prod 
  with pidfile /home/bandwith/apps/production/shared/pids/delayed_job.pid
  start program = "/usr/bin/env RAILS_ENV=production /home/bandwith/apps/production/current/script/delayed_job start" as uid bandwith and gid bandwith 
  stop program  = "/usr/bin/env RAILS_ENV=production /home/bandwith/apps/production/current/script/delayed_job stop" as uid bandwith and gid bandwith

This did not work however, and after quite a bit of debugging I found there are a couple of issues you might need to be aware of:

1. Your $PATH

monit starts things with a ‘spartan path‘ of:

/bin:/usr/bin:/sbin:/usr/sbin

My ruby happens to be in /usr/local/bin, so we will need to pass that in too:

start program = "/usr/bin/env PATH=/usr/local/bin:PATH RAILS_ENV=production /var/www/apps/{app_name}/current/script/delayed_job start"

2. monit doesn’t define a $HOME environment variable

Even though we are starting these processes with uids and guids specified, that doesn’t actually load the users shell. With no $HOME env variable, bundler can’t find where your gems are installed and thinks they are all missing. I ended up just putting the variable in the monit command, another option might be running su -c ‘{command}’ (I didn’t test that).

So putting that all together you get the following which should make everything work:

check process delayed_job_bandwith_prod 
  with pidfile /home/bandwith/apps/production/shared/pids/delayed_job.pid
  start program = "/usr/bin/env HOME=/home/bandwith PATH=/usr/local/bin:$PATH RAILS_ENV=production /home/bandwith/apps/production/current/script/delayed_job start" as uid bandwith and gid bandwith 
  stop program  = "/usr/bin/env HOME=/home/bandwith PATH=/usr/local/bin:$PATH RAILS_ENV=production /home/bandwith/apps/production/current/script/delayed_job stop" as uid bandwith and gid bandwith

I would be interested to know if anyone has any better suggestions, but this seems to be working nicely.


Netopia Routers, Slow DNS, and IPv6

words by Brian Racer

We have been having issues for a while where DNS lookups in Ubuntu setups have been taking significant amounts of time. We learned turning off IPv6 support in Firefox would fix the issue for that application, but we needed a system wide fix.

It turns out the Netopia router(3347) seems to have issues with IPv6 AAAA requests when acting as a DNS proxy to our ISP. Any linux computer that was getting DNS info via DHCP was generally being affected. There is no way to fix this in the Netopia web-gui, but we can telnet into the router and twiddle some things. Run the following commands once you log in:

configure
set dns proxy-enable off
save
exit
restart

Once the router restarts DNS look-ups will be quite snappy!


Ubuntu 10.4 (Lucid Lynx) and Broadcom BCM4312

words by Brian Racer

This is basically just a repost of the same issue I had when Karmic Koala launched

Ubuntu 10.4 (Lucid Lynx) launched today and I figured it was time to do an install from scratch onto my Dell D830 Latitude laptop. Everything went quite smoothly but when it started up I noticed two issues:

Problem 1: No wireless

I know the Broadcom card inside the laptop isn’t the greatest, but the last two Ubuntu releases it has worked out of the box. The following command enabled the card after a reboot:

sudo apt-get install bcmwl-kernel-source

Problem 2: Really slow DNS lookups (because of IPV6)

As documented on Launchpad, there still doesn’t seem to be an official fix. Strangely disabling IPV6 in /etc/sysctl.conf didn’t solve anything, however disabling it in Firefox at least fixes the issue in the browser. Just type about:config in the address bar, and set network.dns.disableIPv6 to false.


Pimp your $PS1 with source control information

words by Brian Racer

I recently found a useful tip of appending source control information of the current working directory to your shell’s $PS1 line. It might look something like the following image:

The method I saw suggested using vcprompt, a small program that outputs a short string basic version control info. Although it worked well, there were a couple issues I had with it. First, it’s subversion support was somewhat lacking. Second, it was written in C which made it more of a pain to modify, and I wasn’t a huge fan of keeping the binary in version control.

After a little searching I stumbled across vcprompt.py, a python script that did the same thing. This version also had wider support for source control systems, and being a standard python text script it was something I could easily modify and put into my dotfiles git repo. I wasn’t happy with how this one displayed subversion information either(just a revision number which I didn’t find very helpful), so I made my own modification to display the branch you are in. Please pardon my lacking Python skills.

Anyway, on to pimping your prompt. Before we modify the PS1 variable, we need to make sure the vcprompt.py is in your $PATH. I like to put scripts like this in a custom bin directory in my homedir. One way to accomplish that might be the following:

$ mkdir -p ~/bin
$ cd ~/bin
$ wget http://github.com/anveo/dotfiles/raw/master/scripts/vcprompt.py
$ chmod +x vcprompt.py
$ export PATH=~/bin:$PATH

Displayed next is the PS1 line I use – it takes up two lines: the first line contains the current user, hostname, current working directory, and possibly version control info, and the second is just a nice looking arrow for my input. Your terminal will need to support Unicode if you want to use that symbol.

# .bashrc
PS1="\n\[email protected]\h:\w \[\e[1;30m\]$(vcprompt)\[\e[0m\] \n→"
 
# If you are using zsh you may also need the following in .zshrc
setopt prompt_subst

If you use the colors specified, you may need to define those too.

You should now have a prompt similar to the image above! For more shell customizations checkout the rest of my dotfiles, and consider buying Peepcode’s Advanced Command Line screencasts for more productive tips!


Monitoring delayed_job with god on CentOS

words by Brian Racer

I recently started using god rather than monit for process monitoring. god lets me be a bit more expressive with how I want processes monitored using the the power of Ruby.

The current project I am working on has a number of tasks that I want processed asynchronously so I will setup god to monitor my delayed_jobs. If you are not familiar with awesome delayed_job gem, watch the excellent Railscast tutorial.

First install the god gem:

$ sudo gem install god

Next we will create a Redhat compatible init script for god:

$ sudo vi /etc/init.d/god
 
#!/bin/bash
#
# God
#
# chkconfig: - 85 15
# description: start, stop, restart God (bet you feel powerful)
#
 
RETVAL=0
 
case "$1" in
    start)
      /usr/bin/god -P /var/run/god.pid -l /var/log/god.log
      /usr/bin/god load /etc/god.conf
      RETVAL=$?
      ;;
    stop)
      kill `cat /var/run/god.pid`
      RETVAL=$?
      ;;
    restart)
      kill `cat /var/run/god.pid`
      /usr/bin/god -P /var/run/god.pid -l /var/log/god.log
      /usr/bin/god load /etc/god.conf
      RETVAL=$?
      ;;
    status)      
      /usr/bin/god status
      RETVAL=$?
      ;;
    *)
      echo "Usage: god {start|stop|restart|status}"
      exit 1
  ;;
esac
 
exit $RETVAL
(adapted from debian version at http://mylescarrick.com/articles/simple_delayed_job_with_god)

Now adjust the permissions, and set the init script to start on system boot:

$ sudo chmod a+x /etc/init.d/god
$ sudo /sbin/chkconfig --add god
$ sudo /sbin/chkconfig --level 345 god on

Before we start god up, we need to create a configuration file that tells it what configuration files to load:

$ sudo vi /etc/god.conf
 
God.load "/srv/apps/your_app/current/config/god/*.god"

You will need to adjust the above depending on how you have your app setup. When working in a Rails project I like to put my god scripts in config/god.

We will use a script from the guys at github to monitor our job daemon. I tweaked it slightly to have less workers, and to set the environment properly.

RAILS_ROOT = "/srv/apps/your_app/current"
 
1.times do |num|
  God.watch do |w|
    w.name = "dj-#{num}"
    w.group = 'dj'
    w.interval = 30.seconds
    w.start = "rake -f #{RAILS_ROOT}/Rakefile RAILS_ENV=production jobs:work"
 
    w.uid = 'your_app_user'
    w.gid = 'your_app_user'
 
    # retart if memory gets too high
    w.transition(:up, :restart) do |on|
      on.condition(:memory_usage) do |c|
        c.above = 300.megabytes
        c.times = 2
      end
    end
 
    # determine the state on startup
    w.transition(:init, { true => :up, false => :start }) do |on|
      on.condition(:process_running) do |c|
        c.running = true
      end
    end
 
    # determine when process has finished starting
    w.transition([:start, :restart], :up) do |on|
      on.condition(:process_running) do |c|
        c.running = true
        c.interval = 5.seconds
      end
 
      # failsafe
      on.condition(:tries) do |c|
        c.times = 5
        c.transition = :start
        c.interval = 5.seconds
      end
    end
 
    # start if process is not running
    w.transition(:up, :start) do |on|
      on.condition(:process_running) do |c|
        c.running = false
      end
    end
  end
end

It’s now time to start the daemon:

$ sudo /etc/init.d/god start
$ sudo /etc/init.d/god status
dj:
  dj-0: up

Looks good! If you want to make sure it’s working, kill the rake task running jobs:work. god will see that it is stopped and automatically restart it!


Ubuntu 9.10 (Karmic Koala) and Broadcom BCM4312

words by Brian Racer

Ubuntu 9.10 (Karmic Koala) launched today and I figured it was time to do an install from scratch onto my Dell D830 Latitude laptop. Everything went quite smoothly but when it started up I noticed two issues:

Problem 1: No wireless

I know the Broadcom card inside the laptop isn’t the greatest, but the last two Ubuntu releases it has worked out of the box. The following command enabled the card after a reboot:

sudo apt-get install bcmwl-kernel-source

Problem 2: Really slow DNS lookups (because of IPV6)

As documented on Launchpad, there still doesn’t seem to be an official fix. Strangely disabling IPV6 in /etc/sysctl.conf didn’t solve anything, however disabling it in Firefox at least fixes the issue in the browser. Just type about:config in the address bar, and set network.dns.disableIPv6 to false.

Otherwise things seem to be working well, although I don’t understand why they stick with a color scheme that looks like mud.


pbcopy / pbpaste in Ubuntu (command line clipboard)

words by Brian Racer

OS X has a neat command-line tool called pbcopy which takes the standard input and places it in the clipboard to paste into other applications.

In Ubuntu(or any Linux distro with Xwindows), a similar tool is xclip. I like to make this alias:

alias pbcopy='xclip -selection clipboard'
alias pbpaste='xclip -selection clipboard -o'

or the following also works if you would rather use xsel:

alias pbcopy='xsel --clipboard --input'
alias pbpaste='xsel --clipboard --output'

Now you can pipe any text to pbcopy

$ cat ~/.ssh/id_dsa.pub | pbcopy

Your public ssh key is transferred to your clipboard and is ready to be pasted(perhaps with pbpaste).


Unobtrusive viewing of MySQL queries with tcpdump

words by Brian Racer

There are times when you need to monitor the queries coming in to MySQL, but turning on query logging would create too much of a disk I/O hit, or you can’t restart the server to setup MySQL Proxy. Instead we can just monitor the network traffic and extract data that might be interesting using tcpdump and an inline perl script:

sudo tcpdump -i lo -s 0 -l -w - dst port 3306 | strings | perl -e '
while(<>) { chomp; next if /^[^ ]+[ ]*$/;
  if(/^(SELECT|UPDATE|DELETE|INSERT|SET|COMMIT|ROLLBACK|CREATE|DROP|ALTER)/i) {
    if (defined $q) { print "$q\n"; }
    $q=$_;
  } else {
    $_ =~ s/^[ \t]+//; $q.=" $_";
  }
}'

This will only work for clients communicating via TCP – if you are connecting through ‘localhost’ you will be going through a unix socket instead. If you switch ‘localhost’ to ‘127.0.0.1’ then your queries will go through the network stack.

If you just want to dump the traffic to a file for a little bit and analyze it later, do this instead:

sudo tcpdump -i lo port 3306 -s 65535 -x -n -q -tttt> tcpdump.out

You can then use mk-query-digest from Maatkit with–type=tcpdump. See more about this at the MySQL Performance Blog.


Linux Tip: Keep track of packages you have installed

words by Brian Racer

During development on a linux system, you probably install many packages using your favorite package manager. When you have to use a new system, or reimage your current one, it can be a pain to remember all the packages you had setup. One solution is to keep a list of the packages installed after the OS load, and then periodically generate a list of what has been added since.

On a freshly installed system, create the starting baseline list of packages:

# On Debian based systems(Ubuntu):
dpkg --get-selections > packages-alpha.txt
 
# Or CentOS/Fedora:
yum list installed > packages-alpha.txt

You can run the command again at a later time, concatenating the output into a different file so you can view what has changed since the original system setup. Use a diff tool like diff3, vimdiff, or meld:

meld packages-alpha.txt packages-omega.txt

On Debian systems, once you have that file you can use it in a new or different system to mark packages to install using the –set-selection parameter:

dpkg --set-selections < packages-omega.txt
sudo apt-get upgrade