For those using chef to automate your server infrastructure you probably find managing third-party cookbooks to be a pain. Ideally I want to make custom changes to a cookbook while still being able to track for upstream enhancements.

A few techniques I see being used are:

no tracking: Manually download an archive from github or opscode community and drop it in your cookbooks/ directory. Easy to make custom changes but you have no automated way to check for updates.

git submodules: This tracks upstream well, but unless you own the repo you can’t make changes.

fork it: Since pretty much all cookbooks reside on github, so you can fork a copy. This works, but now you might have dozens of different repos to manage. And checking for updates means going into each repo and manually merging in enhancements from the upstream.

knife vendor: Now we are getting somewhere. Chef’s knife command has functionality for dealing with third-party cookbooks. It looks something like this:

knife cookbook site vendor nginx

This downloads the nginx cookbook from the opscode community site, puts an unmodified copy in a chef-vendor-nginx git branch, and then puts a copy in your cookbooks/nginx dir in your master branch. When you run the install again it will download the updated version into the chef-vendor-nginx branch, and then merge that into master.

This is a good start, but it has a number of problems. First you are restricted to using what is available on the opscode community site. Second, although this seems like a git-centric model, knife is actually downloading a .tar.gz file. In fact if you visited the nginx cookbook page you would see it only offers an archive download, no way to inspect what this cookbook actually provides before installing.

There is a sea of great high-quality cookbooks on github. Since we all know and love git it would be great if we could get the previous functionality but using git repositories as a source instead.

A Solution

Enter knife-github-cookbooks. This gem enhances the knife command and lets us get the above functionality by pulling from github rather than downloading from opscode. To use it just install the gem and run:

knife cookbook github install cookbooks/nginx

By default it assumes a username/repo from github. So for each cookbook you install you will have a chef-vendor-cookbookname branch storing the upstream copy and a cookbooks/cookbook-name directory in your master branch to make local changes to.

If you want to check for upstream changes:

knife cookbook github compare nginx

That will launch a github compare view. You can even pass this command a different user who has forked the repo and see or merge in changes from that fork! Read more about it on the github page.

One thing to keep in mind is this gem doesn’t pull in required dependencies automatically, so you will have to make sure you download any requirements a cookbook might have. You can check what dependencies a cookbook requires by inspecting the metadata files.

Bonus Tip!

Opscode has a github repository full of recipes you probably want to use (opscode/cookbooks). Unfortunately using this repository would mean pulling in *all* of those cookbooks. That just clutters up your chef project with cookbooks you don’t need. Luckily there is https://github.com/cookbooks! This repository get updated daily and separates each opscode/cookbook cookbook into a separate git repository.

Now you can cherry-pick the cookbooks you want, and manage them with knife-github-cookbooks!

I’ll go through a couple of addons in this article, but if you just want to make sure you have as many dependencies as possible to run the addons here is a line you can paste:

$ sudo apt-get install build-essential cmake libreadline-dev libssl-dev ncurses-dev libffi-dev zlib1g-dev libpcre3-dev libpng-dev libtiff4-dev libjpeg62-dev python-dev libpng-dev libtiff4-dev libjpeg62-dev libmysqlclient-dev libmemcached-dev libtokyocabinet-dev libsqlite3-dev libdbi0-dev libpq-dev libgmp3-dev libogg-dev libvorbis-dev libtaglib-cil-dev libtag1-dev libtheora-dev libsamplerate0-dev libloudmouth1-dev libsndfile1-dev libflac-dev libgl1-mesa-dev libglu1-mesa-dev freeglut3-dev libxmu-dev libxi-dev libxml2-dev libyajl-dev uuid-dev liblzo2-dev zlib1g-dev

You will need to rebuild Io once these are all installed.

I would encourage you to browse the addons/* directory in the Io source tree. There are many good useful addons and samples, although unfortunately there are few that do not seem to currently work or are missing samples, so dust off that book on C

Sockets

sudo apt-get install libevent-dev

Here is a minimal webserver using sockets:

WebRequest := Object clone do(
    cache := Map clone
    handleSocket := method(socket, server,
        socket streamReadNextChunk
        if(socket isOpen == false, return)
        request := socket readBuffer betweenSeq("GET ", " HTTP")         
 
        data := cache atIfAbsentPut(request,
            writeln("caching ", request)
            f := File clone with(request)
            if(f exists, f contents, nil)
        )                                                                
 
        if(data,
            socket streamWrite("HTTP/1.0 200 OK\n\n")
            socket streamWrite(data)
        ,
            socket streamWrite("Not Found")
        )                                                                
 
        socket close
        server requests append(self)
    )
)                                                                        
 
WebServer := Server clone do(
    setPort(7777)
    socket setHost("127.0.0.1")
    requests := List clone
    handleSocket := method(socket,
        WebRequest handleSocket(socket, self)
    )
) start

Lots of other good socket based examples in addons/Socket/samples.

Regex

sudo apt-get install libpcre3-dev

That will install Perl Compatible Regular Expression support for Io. You can use it like:

regex := Regex with("(?\\d+)([ \t]+)?(?\\w+)")
match := "73noises" matchesOfRegex(regex) next

CFFI

During the configure process you might have noticed a message saying Could NOT find FFI  (missing:  FFI_INCLUDE_DIRS).  FFI (foreign function interface) is basically a system that lets us call functions in different programming languages. First make sure you have the development libraries:

$ sudo apt-get install libffi-dev

How FFI functions is very architecture and compiler dependent, and it seems debian places the includes in a location the cmake scripts aren’t looking. I’m not that familiar with cmake and couldn’t find a very elegant solution, so just place the following line in the modules/FindFFI.cmake script:

$ vim modules/FindFFI.cmake
 
# Add the following line
set(FFI_INCLUDE_DIRS /usr/include/x86_64-linux-gnu)
# Above these two
include(FindPackageHandleStandardArgs)
FIND_PACKAGE_HANDLE_STANDARD_ARGS(FFI DEFAULT_MSG FFI_INCLUDE_DIRS FFI_LIBRARIES)

Here is a small program that gets us direct access to libc’s puts(3) function:

CFFI
 
lib := Library clone setName("libc.so.6")
puts := Function with(Types CString) setLibrary(lib) setName("puts")
 
puts "Hello Io!"

Python

sudo apt-get install python-dev

Want to access Python from Io?

# Import a module
sys := Python import("sys")
 
"Which version of python are we running?" println
sys version println
 
"Split a string" println
str := "Brave brave Sir Robin"
str println
string split(str) println
 
"Load a C module (.so)" println
t := Python import("time")
 
writeln("Current time is: ", t time)

Databases

sudo apt-get install libmysqlclient-dev libmemcache-dev libtokyocabinet-dev libsqlite3-dev libdbi0-dev

Io has addons for MySQL, PostgresQL, memcached, Tokyo Cabinet, SQLite and a few others.

Sound

sudo apt-get install libgmp3-dev libogg-dev libvorbis-dev libtaglib-cil-dev libtag1-dev libtheora-dev libsamplerate0-dev libloudmouth1-dev libsndfile1-dev libflac-dev

Various sound processing libraries.

Images

$ sudo apt-get install libpng-dev libtiff4-dev libjpeg62-dev

Various image loading libraries.

GUI

$ sudo apt-get install x11proto-xf86misc-dev xutils-dev libxpm-dev libpango1.0-dev libcairo2-dev libfreetype6-dev 
 
$ sudo apt-get install libclutter-1.0-dev libatk1.0-dev

There is also a GUI called Flux that requires OpenGL support. I wasn’t able to get it working however.

OpenGL

$ sudo apt-get install libgl1-mesa-dev libglu1-mesa-dev freeglut3-dev libxmu-dev libxi-dev

Lots of great examples in addons/OpenGL/samples.

XML and JSON

$ sudo apt-get install libxml2-dev libyajl-dev

If you need to do any XML or JSON parsing.

UUID

$ sudo apt-get install uuid-dev

Support for UUID generator. Seems to be broken however.

Misc

$ sudo apt-get install libreadline-dev libssl-dev ncurses-dev libffi-dev zlib1g-dev liblzo2-dev zlib1g-dev

SSL, archives, REPL history, curses GUI.

Generally on my development machine when I compile from source I like to install locally to my home directory rather than system wide. This way sudo privileges are not needed plus I just like the idea of keeping these items close to home.

First Io requires the cmake build system so make sure that is available.

$ sudo apt-get install cmake

Next download and extract the source code.

$ wget --no-check-certificate http://github.com/stevedekorte/io/zipball/master -O io-lang.zip
$ unzip io-lang.zip
$ cd stevedekorte-io-[hash]

Io provides a build script, however it is setup to install the language to /usr/local. Since I want it to go in $HOME/local you just need to modify that file. Here is a quick one liner:

$ sed -i -e 's/^INSTALL_PREFIX="\/usr\/local/INSTALL_PREFIX="$HOME\/local/' build.sh

Now build and install.

$ ./build.sh
$ ./build.sh install

Since we are installing into a location our OS doesn’t really know about, we need to configure a few paths.

$ vim ~/.bashrc
export PATH="${HOME}/local/bin:${PATH}"
export LD_LIBRARY_PATH="${HOME}/local/lib:${LD_LIBRARY_PATH}"
 
# You might want these too
export LD_RUN_PATH=$LD_LIBRARY_PATH
export CPPFLAGS="-I${HOME}/local/include"
export CXXFLAGS=$CPPFLAGS
export CFLAGS=$CPPFLAGS
export MANPATH="${HOME}/local/share/man:${MANPATH}"

Lastly restart your shell and type ‘io’ and you should be dropped into Io’s REPL!

A side benefit to this method is you can install anything you build into $HOME/local. Usually you just need to pass the –prefix=$HOME/local parameter when you run a ./configure script.

Luckily less has a flag that will repaint the screen when paging. To make it a default you need to export a LESS variable to your shell’s environment. Something like this:

export LESS="-R"

Just throw that in your ~/.bashrc or dotfiles and you’re all set!

check process delayed_job_bandwith_prod 
  with pidfile /home/bandwith/apps/production/shared/pids/delayed_job.pid
  start program = "/usr/bin/env RAILS_ENV=production /home/bandwith/apps/production/current/script/delayed_job start" as uid bandwith and gid bandwith 
  stop program  = "/usr/bin/env RAILS_ENV=production /home/bandwith/apps/production/current/script/delayed_job stop" as uid bandwith and gid bandwith

This did not work however, and after quite a bit of debugging I found there are a couple of issues you might need to be aware of:

1. Your $PATH

monit starts things with a ‘spartan path‘ of:

/bin:/usr/bin:/sbin:/usr/sbin

My ruby happens to be in /usr/local/bin, so we will need to pass that in too:

start program = "/usr/bin/env PATH=/usr/local/bin:PATH RAILS_ENV=production /var/www/apps/{app_name}/current/script/delayed_job start"

2. monit doesn’t define a $HOME environment variable

Even though we are starting these processes with uids and guids specified, that doesn’t actually load the users shell. With no $HOME env variable, bundler can’t find where your gems are installed and thinks they are all missing. I ended up just putting the variable in the monit command, another option might be running su -c ‘{command}’ (I didn’t test that).

So putting that all together you get the following which should make everything work:

check process delayed_job_bandwith_prod 
  with pidfile /home/bandwith/apps/production/shared/pids/delayed_job.pid
  start program = "/usr/bin/env HOME=/home/bandwith PATH=/usr/local/bin:$PATH RAILS_ENV=production /home/bandwith/apps/production/current/script/delayed_job start" as uid bandwith and gid bandwith 
  stop program  = "/usr/bin/env HOME=/home/bandwith PATH=/usr/local/bin:$PATH RAILS_ENV=production /home/bandwith/apps/production/current/script/delayed_job stop" as uid bandwith and gid bandwith

I would be interested to know if anyone has any better suggestions, but this seems to be working nicely.

It turns out the Netopia router(3347) seems to have issues with IPv6 AAAA requests when acting as a DNS proxy to our ISP. Any linux computer that was getting DNS info via DHCP was generally being affected. There is no way to fix this in the Netopia web-gui, but we can telnet into the router and twiddle some things. Run the following commands once you log in:

configure
set dns proxy-enable off
save
exit
restart

Once the router restarts DNS look-ups will be quite snappy!

Ubuntu 10.4 (Lucid Lynx) launched today and I figured it was time to do an install from scratch onto my Dell D830 Latitude laptop. Everything went quite smoothly but when it started up I noticed two issues:

Problem 1: No wireless

I know the Broadcom card inside the laptop isn’t the greatest, but the last two Ubuntu releases it has worked out of the box. The following command enabled the card after a reboot:

sudo apt-get install bcmwl-kernel-source

Problem 2: Really slow DNS lookups (because of IPV6)

As documented on Launchpad, there still doesn’t seem to be an official fix. Strangely disabling IPV6 in /etc/sysctl.conf didn’t solve anything, however disabling it in Firefox at least fixes the issue in the browser. Just type about:config in the address bar, and set network.dns.disableIPv6 to false.

Much of the screencast focuses on the excellent rails.vim plugin, primarily using it to move around your project in incredibly efficient ways. He also covers snipMate, ctags, and searching with ack. I found his advice on using ctags quite useful for jumping around the rails source code.

Ben is well spoken and well paced, and I hope to see more screencasts from him(especially ones focusing on vim). rails.vim in particular has much more functionality to offer that I would love to see covered. If you enjoy the visual/audible style of learning be sure to also checkout vimcasts.org for more vim related screencasts.

This screencast probably best for late-beginner to moderately skilled vim users, but even advanced users will probably learn a new technique or two. It might be a bit overwhelming if you are incredibly new to vim as it’s focus is on usage and not getting things setup or configured.

Overall I feel if you are looking to get serious with vim the $9 will quickly pay for itself.

More resources:

The Screencast: Vim for Rails Developers
My vimfiles
Ben’s vimfiles
Ben’s twitter

Selector Expansion

Selector expansion is the plug-in’s primary purpose. It lets you write in a CSS selector syntax that get expanded to full HTML:

#album.photo

If you type that into vim and then press Ctrl-e on that line it will be expanded to:

<div id="album" class="photo">|</div>

This next one is a bit more complicated, I’ll explain what’s going on:

#container > #nav > ul > li.first {Home} + li*2 + li.last {About Us} < <  #content > p*2 < #footer > span.copyright {(c) 2010 Jetpack LLC}

1. Creating a div with the id of “container”
2. Creating a div with an id of “nav” that is a child of the “container” div. The > specifies we are creating a child element in the DOM tree.
3. Creating a ul tag that is a child of “nav“
4. Creating an li tag with the class name “first“. Text in brackets will be plain text placed in-between the tag we are creating.
5. Creating two sibling li elements. The + denotes what comes next will be a sibling. The * is a multiplier to create any number of similar elements.
6. Creating an li tag with the class name “last” as a sibling to the previous li elements we have created. It will contain the text ‘About Us‘.
7. Next now use the < symbol to go up two levels of the DOM tree.
8. Next we create a div with the id of “content” that will contain two paragraph tags.
9. We go back up a level and add a footer div that will have a span with the class of “copyright” that contains some boilerplate copyright text.

If we press Ctrl-e on that line, it will be expanded to the following:

  <div id="container">
    <div id="nav">
      <ul>
        <li class="first">Home</li>
        <li>|</li>
        <li></li>
        <li class="last">About Us</li>
      </ul>
    </div>
    <div id="content">
      <p></p>
      <p></p>
    </div>
    <div id="footer">
      <span class="copyright">(c) 2010 Jetpack LLC</span>
    </div>
  </div>

It will also place your cursor in the first empty tag denoted by the |. You can jump around to other empty tags with Ctrl-n (I change this mapping, more on that in a second).

If you are a fan of HAML but forced to use standard HTML in your projects this plug-in might make you a bit happier.

Shortcuts

The other piece of functionality this plug-in provides is a snipMate like shortcuts feature. These act much like snipMate snippets except they are hard-coded into the python script. However most are still quite useful and you can view the python script to review them.

Issues

After installing this plugin I was having tabbing/tab-expansion issues. I believe it’s Ctrl-n mapping which allows you to jump around to empty tags was conflicting with some other plug-in I had installed, possibly SuperTab. I remapped it by putting the following in my .vimrc:

let g:sparkupNextMapping = '<c-x>'

That doesn’t seem to conflict with anything for me, and since it only gets used in normal mode it doesn’t conflict with Tim Pope’s excellent ragtag plug-in (read more on that here).

Video Demonstration

Some of this might make more sense when you see it in action, so watch the following YouTube video(make sure to switch it to 720p for crisper text):

Conclusion

Have fun writing faster HTML! You can see more examples on sparkup’s Github page.

sudo apt-get install compizconfig-settings-manager

Next navigate the system menu to System -> Preferences -> CompizConfig Settings Manager. The app should start so next scroll down to the Window Management section and click on Static Application Switcher. Next click the Behavior tab and the last option will be ‘Allow Mouse Selection‘. Enable that check-box and exit the application.

It’s not quite as nice as OS X – you can only select by clicking, just mousing over won’t make each item the active target, but it’s better than nothing!