Munin Graphs for Phusion Passenger (a.k.a. mod_rails)

words by Brian Racer

The goal of this article is to get fairly nice looking graphs of Phusion Passenger’s performance and memory metrics:


This specific setup focuses on CentOS (on cPanel none the less) – but instructions should apply for most linux distros. It assumes you already have Passenger and Munin successfully setup. See my previous article on getting Phusion Passenger setup if you have not already.

First we need to allow the munin user sudo privileges for a few passenger related commands:

$ sudo /usr/sbin/visudo
 
# Add the following line to the file
munin ALL=(ALL) NOPASSWD:/usr/bin/passenger-status, /usr/bin/passenger-memory-stats
 
# Depending on your setup, you may also have to comment out the following line:
Defaults requiretty

If you see the error sorry, you must have a tty to run sudo in /var/log/munin/munin-node.log, comment out the final line shown above.

The following two files will glean some performance and memory statistics.

Passenger Status:

sudo vi /usr/share/munin/plugins/passenger_status
#!/usr/bin/env ruby
 
def output_config
  puts <<-END
graph_category App
graph_title passenger status
graph_vlabel count
 
sessions.label sessions
max.label max processes
running.label running processes
active.label active processes
END
  exit 0
end
 
def output_values
  status = `sudo /usr/bin/passenger-status`
  unless $?.success?
    $stderr.puts "failed executing passenger-status"
    exit 1
  end
  status =~ /max\s+=\s+(\d+)/
  puts "max.value #{$1}"
 
  status =~ /count\s+=\s+(\d+)/
  puts "running.value #{$1}"
 
  status =~ /active\s+=\s+(\d+)/
  puts "active.value #{$1}"
 
  total_sessions = 0
  status.scan(/Sessions: (\d+)/).flatten.each { |count| total_sessions += count.to_i }
  puts "sessions.value #{total_sessions}"
end
 
if ARGV[0] == "config"
  output_config
else
  output_values
end

Memory Stats:

sudo vi /usr/share/munin/plugins/passenger_memory_status
#!/usr/bin/env ruby
# put in /etc/munin/plugins and restart munin-node
# by Dan Manges, http://www.dcmanges.com/blog/rails-application-visualization-with-munin
# NOTE: you might need to add munin to allow passwordless sudo for passenger-memory-stats
 
def output_config
  puts <<-END
graph_category App
graph_title Passenger memory stats
graph_vlabel count
 
memory.label memory
END
  exit 0
end
 
def output_values
  status = `sudo /usr/bin/passenger-memory-stats | tail -1`
  unless $?.success?
    $stderr.puts "failed executing passenger-memory-stats"
    exit 1
  end
  status =~ /(\d+\.\d+)/
  puts "memory.value #{$1}"
end
 
if ARGV[0] == "config"
  output_config
else
  output_values
end

Now we will link these to the active plugins, and make them executable:

sudo chmod +x /usr/share/munin/plugins/passenger_status
sudo chmod +x /usr/share/munin/plugins/passenger_memory_status
sudo ln -s /usr/share/munin/plugins/passenger_status /etc/munin/plugins/passenger_status
sudo ln -s /usr/share/munin/plugins/passenger_memory_status /etc/munin/plugins/passenger_memory_status

Last thing we need to do is make sure those scripts run as the munin user:

sudo vi /etc/munin/plugin-conf.d/munin-node
[passenger_*]
user munin
command ruby %c

Restart the munin node, and wait and you should see the graphs start to propagate.

sudo /etc/init.d/munin-node restart

For even more detailed performance analytics, checkout NewRelic monitoring. And thanks to Dan Mange for the munin scripts.


pbcopy / pbpaste in Ubuntu (command line clipboard)

words by Brian Racer

OS X has a neat command-line tool called pbcopy which takes the standard input and places it in the clipboard to paste into other applications.

In Ubuntu(or any Linux distro with Xwindows), a similar tool is xclip. I like to make this alias:

alias pbcopy='xclip -selection clipboard'
alias pbpaste='xclip -selection clipboard -o'

or the following also works if you would rather use xsel:

alias pbcopy='xsel --clipboard --input'
alias pbpaste='xsel --clipboard --output'

Now you can pipe any text to pbcopy

$ cat ~/.ssh/id_dsa.pub | pbcopy

Your public ssh key is transferred to your clipboard and is ready to be pasted(perhaps with pbpaste).


Clean Webalizer Referrer Stats in cPanel

words by Brian Racer

I find it useful to override some of cPanel’s default webalizer configurations. There can be a lot of misleading or unnecessary data when you don’t ignore traffic that is being referred to from your own site.

Create the following file:

vi /home/username/tmp/webalizer/webalizer.conf
HideSite *example.com
HideSite localhost
HideReferrer example.com/
MangleAgents 4
 
TopSites        10
TopKSites       10
TopURLs         10
TopKURLs        50
TopReferrers    250
TopAgents       15
TopCountries    10
TopEntry        10
TopExit         10
TopSearch       20
TopUsers        20
 
# Usually you want to hide these
HideURL         *.gif
HideURL         *.GIF
HideURL         *.jpg
HideURL         *.JPG
HideURL         *.png
HideURL         *.PNG
HideURL         *.bmp
HideURL         *.BMP

Replace example.com with the domain you are working on. The next time webalizer runs it will not be clogged with useless referrers generated by your site. To force webalizer to run(in a cPanel environment), execute:

/scripts/runweblogs username

Textmate and GNU screen error in Leopard

words by Brian Racer

When running mate from inside a GNU screen session in Leopard, I was getting this error:

$ mate .
mate: failed to establish connection with TextMate.

I read reports of falling back to a previous version in macports, but a simple solution is to put this line in your .profile config:

alias mate='open -a TextMate.app'

That will fix the problem.


Unobtrusive viewing of MySQL queries with tcpdump

words by Brian Racer

There are times when you need to monitor the queries coming in to MySQL, but turning on query logging would create too much of a disk I/O hit, or you can’t restart the server to setup MySQL Proxy. Instead we can just monitor the network traffic and extract data that might be interesting using tcpdump and an inline perl script:

sudo tcpdump -i lo -s 0 -l -w - dst port 3306 | strings | perl -e '
while(<>) { chomp; next if /^[^ ]+[ ]*$/;
  if(/^(SELECT|UPDATE|DELETE|INSERT|SET|COMMIT|ROLLBACK|CREATE|DROP|ALTER)/i) {
    if (defined $q) { print "$q\n"; }
    $q=$_;
  } else {
    $_ =~ s/^[ \t]+//; $q.=" $_";
  }
}'

This will only work for clients communicating via TCP – if you are connecting through ‘localhost’ you will be going through a unix socket instead. If you switch ‘localhost’ to ‘127.0.0.1’ then your queries will go through the network stack.

If you just want to dump the traffic to a file for a little bit and analyze it later, do this instead:

sudo tcpdump -i lo port 3306 -s 65535 -x -n -q -tttt> tcpdump.out

You can then use mk-query-digest from Maatkit with–type=tcpdump. See more about this at the MySQL Performance Blog.


Linux Tip: Keep track of packages you have installed

words by Brian Racer

During development on a linux system, you probably install many packages using your favorite package manager. When you have to use a new system, or reimage your current one, it can be a pain to remember all the packages you had setup. One solution is to keep a list of the packages installed after the OS load, and then periodically generate a list of what has been added since.

On a freshly installed system, create the starting baseline list of packages:

# On Debian based systems(Ubuntu):
dpkg --get-selections > packages-alpha.txt
 
# Or CentOS/Fedora:
yum list installed > packages-alpha.txt

You can run the command again at a later time, concatenating the output into a different file so you can view what has changed since the original system setup. Use a diff tool like diff3, vimdiff, or meld:

meld packages-alpha.txt packages-omega.txt

On Debian systems, once you have that file you can use it in a new or different system to mark packages to install using the –set-selection parameter:

dpkg --set-selections < packages-omega.txt
sudo apt-get upgrade

Installing MySQL Ruby Gem on CentOS 5 (and an explanation about getopt’s double hyphens)

words by Brian Racer

I had an issue trying to build the mysql gem on CentOS 5.x.

ERROR: While executing gem … (Gem::Installer::ExtensionBuildError)
ERROR: Failed to build gem native extension.
 
ruby extconf.rb update
checking for mysql_query() in -lmysqlclient… no
...

After much hairpulling, two hyphens solved my problem:

sudo gem install mysql -- \
--with-mysql-include=/usr/include/mysql \
--with-mysql-lib=/usr/lib/mysql

A blog comment I found explains the double hyphens:

The double hyphens (–) tells the option parser that there is no more options on the command line. This special syntax comes from GNU getopt. Everything after ‘–’ is treated as non-options. This is useful if you want to write something on the command line that looks like an option but is not, or if it should be parsed though as an option to another program called by the one you are calling.

The two hyphens in this particular command string is important since the gem binary must not confuse the ‘–with-mysql-dir’ option as an option for gem it self. Instead this option should be passed on to the make command called in the gem internals.

Thomas Watson

And if you are on a 64-bit system, be sure to use –-with-mysql-lib=/usr/lib64/mysql instead.


Unobtrusive RESTful jQuery

words by Brian Racer

Many Rails(and non-Rails) web applications these days strive to create RESTful interfaces for their application design. When dealing with Ajax updates through jQuery, this can become somewhat tricky. Since most browsers only implement GET and POST requests, we have to fake it in Rails by sending a parameter called _methd for all PUT and DELETE requests. To make things even more complicated, we need to include a token to prevent CSRF attacks.

In Rails 2.x, the default way it generates the javascript to send a a RESTful command via POST request looks something like this:

<a href="/foos/1" onclick="if (confirm('Are you sure?')) { var f = document.createElement('form'); f.style.display = 'none'; this.parentNode.appendChild(f); f.method = 'POST'; f.action = this.href;var m = document.createElement('input'); m.setAttribute('type', 'hidden'); m.setAttribute('name', '_method'); m.setAttribute('value', 'delete'); f.appendChild(m);var s = document.createElement('input'); s.setAttribute('type', 'hidden'); s.setAttribute('name', 'authenticity_token'); s.setAttribute('value', 'VGcSbbdenz7ZCMDWl7LugKC2KFldp7oKdgdvjGyb4Zo='); f.appendChild(s);f.submit(); };return false;">Destroy</a> |

Yuck. It’s creating a new form, setting a hidden input fields for the submission method and csrf token, and then submitting it. Not only is it obtrusive, that gets inserted at every location there is a delete or update link.

First lets extend jquery with PUT and DELETE methods. Well call this jquery.rest.js:

;(function($){
  $.put = $.update = function(uri, data, callback, type = 'json') {
    if ($.isFunction(data)) callback = data, data = {}
    return $.post(uri, $.extend(data, { _method: 'put' }), callback, type)
  }
 
  $.delete = $.del = $.destroy = function(uri, data, callback, type = 'json') {
    if ($.isFunction(data)) callback = data, data = {}
    return $.post(uri, $.extend(data, { _method: 'delete' }), callback, type)
  }
{)(jQuery)

The previous code will POST data while always including a ‘_method’ parameter. Using this code is as simple as a normal jQuery ajax call:

$('.deletable').click(function() {
  $.delete('/videos/delete', {
    'video_id': $(this).attr('id');
  });
});

That example might iterate through all elements with the deletable class, and then sent the DELETE method when clicked.

Now in a Rails app we also need to include the CSRF token in all POST, PUT, and DELETE requests. The way I go about this is to put this at the bottom of my applications layout:

$(document).ready(function() {
  window.AUTH_TOKEN = '#{form_authenticity_token}';
  $(document).ajaxSend(function(event, request, settings) {
    if (typeof(window.AUTH_TOKEN) == "undefined") return;
    // IE6 fix for http://dev.jquery.com/ticket/3155
    if (settings.type == 'GET' || settings.type == 'get') return;
    settings.data = settings.data || "";
    settings.data += (settings.data ? "&" : "") + "authenticity_token=" + encodeURIComponent(window.AUTH_TOKEN);
  });
});

This will extend the parameters of any ajax request with the authenticity token. I hope this short guide gives you a better idea how to do REST in an unobtrusive way.


jQuery Sortables: Getting DOM element position for an efficient ajax update in Rails

words by Brian Racer

The jQuery UI library has some excellent interaction functionality, especially ‘sortables’ to make cool things like rearrangeable lists. Although there are lots of tutorials on sortable lists, one problem I have with them is the amount of database queries a single update can generate. They generally make use of the Sortable.serialize method, send *all* the elements back to the server, and update each element with something like ActiveRecord’s update_all(which can be smart), or worse, separate SQL queries for each list element.

What we can do instead is just send the id and position of the single element that has moved, and use acts_as_list to adjust the positions in the database. Lets say we have an unordered list of Video models (I am using HAML in this example):

%h3= "Videos"
%ul(class="sortable")
  - @videos.each do |video|
    %li{ :id => "video_#{video.id}" }= video.title

That might output something like this:

<h3>Videos</h3>
<ul class='sortable'>
  <li id='video_5'>Batman Begins</li>
  <li id='video_6'>Ghostbusters</li>
  <li id='video_7'>Indiana Jones and the Temple of Doom</li>
 </ul>

We have the video’s database id’s in each of the element id’s, and we have given the ul element the sortable class so we can select it later. Now lets select that ul element and make it ‘sortable’:

$(function() {
  $('.sortable').sortable();
}

With that we can now drag each list item around. Now comes the important part. When we finish dragging a single list element we will send a single ajax request to the server that contains the numeric value of the element’s id, and it’s position in the list:

  $(function() {
    $('.sortable').sortable({
      stop: function(event, ui) {
        $(ui.item).effect("highlight");
        var video_id = $(ui.item).attr('id').replace(/[^\d]+/g, ''))
        var position = ui.item.prevAll().length;
        $.post('/videos/update_position', {
          'video_id': video_id,
          'position': position
         });
         $(ui.item).effect("highlight");
      }
    });
  })

Couple of notes:

ui.item is the DOM element we are dragging

$(ui.item).attr(‘id’).replace(/[^\d+]+/g, ”)) is pulling out the list item’s DOM id and removing anything that isn’t numeric, so we are left with the model’s ID

ui.item.prevAll().length is what gives us the list item’s position in relation to it’s parent ul

Now our controller action can be as simple as:

Video.find(params[:video_id]).insert_at(params[:position])

Again this requires acts_as_list. I believe this should never do more that 4 queries: One to find the model, one to update it’s position, and possibly two more to shift what was above and below it previously. Hopefully this saves you some SQL queries on larger lists.


Autotest notifications on Ubuntu using lib-notify

words by Brian Racer

At Jetpack we are big fans of Test Driven Development(TDD), and specifically Behavior Driven Development(BDD) using excellent testing frameworks like RSpec and Cucumber. When you are using ZenTest’s autotest or autospec, it’s nice to have unobtrusive notifications of passing/failing tests while you work. Mac users can integrate autotest with the excellent Growl, but what about us Linux users? Well be can use a library called lib-notify.

First lets play with the lib-notify binary. Install it:

sudo apt-get install libnotify-bin

To make sure it is working, try this:

notify-send "Hello World!"

You should see the notification appear! Now lets edit our ~/.autotest file to use that notification:

vi ~/.autotest
module Autotest::GnomeNotify
 
  # Time notification will be displayed before disappearing automatically
  EXPIRATION_IN_SECONDS = 2
  ERROR_STOCK_ICON = "gtk-dialog-error"
  SUCCESS_STOCK_ICON = "gtk-dialog-info"
 
  # Convenience method to send an error notification message
  #
  # [stock_icon]   Stock icon name of icon to display
  # [title]        Notification message title
  # [message]      Core message for the notification
  def self.notify stock_icon, title, message
    options = "-t #{EXPIRATION_IN_SECONDS * 1000} -i #{stock_icon}"
    system "notify-send #{options} '#{title}' '#{message}'"
  end
 
  Autotest.add_hook :red do |at|
    notify ERROR_STOCK_ICON, "Tests failed", "#{at.files_to_test.size} tests failed"
  end
 
  Autotest.add_hook :green do |at|
    notify SUCCESS_STOCK_ICON, "All tests passed, good job!", ""
  end
 
end

Although not quite as pretty as Growl yet, it works! If you are happy with that you can stop there. If you want something that looks a little bit nicer, read on!

The next few steps are going to require a few development libraries and gems, so install the following packages to make sure we are on the same page:

apt-get install  libinotify-ruby libgtk2-ruby  libnotify-dev
sudo gem install zentest autotest autotest-rails red-green launchy

Unfortunately the libnotify ruby library isn’t included in Ubuntu 9.04 Jaunty, so we will have to build that ourselves:

wget http://rubyforge.org/frs/download.php/27134/ruby-libnotify-0.3.3.tar.bz2
tar -xvjpf ruby-libnotify-0.3.3.tar.bz2
cd  ruby-libnotify-0.3.3
ruby extconf.rb
make
sudo make install

Next put the following pretty images in your ~./autotest_images directory:

Now we can use a modified version of Linden LAN’s .autotest:

require 'rnotify'
require 'launchy'
require 'gtk2'
 
module Autotest::RNotify
  class Notification
    attr_accessor :verbose, :image_root, :tray_icon, :notification,
                  :image_pass, :image_pending, :image_fail,
                  :image_file_pass, :image_file_pending, :image_file_fail,
                  :status_image_pass, :status_image_pending, :status_image_fail
 
    def initialize(timeout = 5000,
                   image_root = "#{ENV['HOME']}/.autotest_images" ,
                   report_url = "doc/spec/report.html",
                   verbose = false)
      self.verbose = verbose
      self.image_root = image_root
      self.image_file_pass = "#{image_root}/pass.png"
      self.image_file_pending = "#{image_root}/pending.png"
      self.image_file_fail = "#{image_root}/fail.png"
 
      raise("#{image_file_pass} not found") unless File.exists?(image_file_pass)
      raise("#{image_file_pending} not found") unless File.exists?(image_file_pending)
      raise("#{image_file_fail} not found") unless File.exists?(image_file_fail)
 
      puts 'Autotest Hook: loading Notify' if verbose
      Notify.init('Autotest') || raise('Failed to initialize Notify')
 
      puts 'Autotest Hook: initializing tray icon' if verbose
      self.tray_icon = Gtk::StatusIcon.new
      tray_icon.pixbuf = Gdk::Pixbuf.new(image_file_pending,22,22)
      tray_icon.tooltip = 'RSpec Autotest'
 
      puts 'Autotest Hook: Creating Notifier' if verbose
      self.notification = Notify::Notification.new('X', nil, nil, tray_icon)
 
      notification.timeout = timeout
 
      puts 'Autotest Hook: Connecting mouse click event' if verbose
      tray_icon.signal_connect("activate") do
        Launchy::Browser.new.visit(report_url)
      end
 
      Thread.new { Gtk.main }
      sleep 1
      tray_icon.embedded? || raise('Failed to set up tray icon')
    end
 
    def notify(icon, tray, title, message)
      notification.update(title, message, nil)
      notification.pixbuf_icon = icon
      tray_icon.tooltip = "Last Result: #{message}"
      tray_icon.pixbuf = tray
      notification.show
    end
 
    def passed(title, message)
      self.image_pass ||= Gdk::Pixbuf.new(image_file_pass, 48, 48)
      self.status_image_pass ||= Gdk::Pixbuf.new(image_file_pass, 22, 22)
      notify(image_pass, status_image_pass, title, message)
    end
 
    def pending(title, message)
      self.image_pending ||= Gdk::Pixbuf.new(image_file_pending, 48, 48)
      self.status_image_pending ||= Gdk::Pixbuf.new(image_file_pending, 22, 22)
      notify(image_pending, status_image_pending, title, message)
    end
 
    def failed(title, message)
      self.image_fail ||= Gdk::Pixbuf.new(image_file_fail, 48, 48)
      self.status_image_fail ||= Gdk::Pixbuf.new(image_file_fail, 22, 22)
      notify(image_fail, status_image_fail, title, message)
    end
 
    def quit
      puts 'Autotest Hook: Shutting Down...' if verbose
      #Notify.uninit
      Gtk.main_quit
    end
  end
 
  Autotest.add_hook :initialize do |at|
    @notify = Notification.new
  end
 
  Autotest.add_hook :ran_command do |at|
    results = at.results.last
 
    unless results.nil?
      output = results[/(\d+)\s+examples?,\s*(\d+)\s+failures?(,\s*(\d+)\s+pending)?/]
      if output
        failures = $~[2].to_i
        pending = $~[4].to_i
      end
 
      if failures > 0
        @notify.failed("Tests Failed", output)
      elsif pending > 0
        @notify.pending("Tests Pending", output)
      else
        unless at.tainted
          @notify.passed("All Tests Passed", output)
        else
          @notify.passed("Tests Passed", output)
        end
      end
    end
  end
 
  Autotest.add_hook :quit do |at|
    @notify.quit
  end
end

This should leave you with a pretty good base if you want to further customize the script!