Manage your third-party Chef cookbooks with knife-github-cookbooks

words by Brian Racer

The Problem

For those using chef to automate your server infrastructure you probably find managing third-party cookbooks to be a pain. Ideally I want to make custom changes to a cookbook while still being able to track for upstream enhancements.

A few techniques I see being used are:

no tracking: Manually download an archive from github or opscode community and drop it in your cookbooks/ directory. Easy to make custom changes but you have no automated way to check for updates.

git submodules: This tracks upstream well, but unless you own the repo you can’t make changes.

fork it: Since pretty much all cookbooks reside on github, so you can fork a copy. This works, but now you might have dozens of different repos to manage. And checking for updates means going into each repo and manually merging in enhancements from the upstream.

knife vendor: Now we are getting somewhere. Chef’s knife command has functionality for dealing with third-party cookbooks. It looks something like this:

knife cookbook site vendor nginx

This downloads the nginx cookbook from the opscode community site, puts an unmodified copy in a chef-vendor-nginx git branch, and then puts a copy in your cookbooks/nginx dir in your master branch. When you run the install again it will download the updated version into the chef-vendor-nginx branch, and then merge that into master.

This is a good start, but it has a number of problems. First you are restricted to using what is available on the opscode community site. Second, although this seems like a git-centric model, knife is actually downloading a .tar.gz file. In fact if you visited the nginx cookbook page you would see it only offers an archive download, no way to inspect what this cookbook actually provides before installing.

There is a sea of great high-quality cookbooks on github. Since we all know and love git it would be great if we could get the previous functionality but using git repositories as a source instead.

A Solution

Enter knife-github-cookbooks. This gem enhances the knife command and lets us get the above functionality by pulling from github rather than downloading from opscode. To use it just install the gem and run:

knife cookbook github install cookbooks/nginx

By default it assumes a username/repo from github. So for each cookbook you install you will have a chef-vendor-cookbookname branch storing the upstream copy and a cookbooks/cookbook-name directory in your master branch to make local changes to.

If you want to check for upstream changes:

knife cookbook github compare nginx

That will launch a github compare view. You can even pass this command a different user who has forked the repo and see or merge in changes from that fork! Read more about it on the github page.

One thing to keep in mind is this gem doesn’t pull in required dependencies automatically, so you will have to make sure you download any requirements a cookbook might have. You can check what dependencies a cookbook requires by inspecting the metadata files.

Bonus Tip!

Opscode has a github repository full of recipes you probably want to use (opscode/cookbooks). Unfortunately using this repository would mean pulling in *all* of those cookbooks. That just clutters up your chef project with cookbooks you don’t need. Luckily there is https://github.com/cookbooks! This repository get updated daily and separates each opscode/cookbook cookbook into a separate git repository.

Now you can cherry-pick the cookbooks you want, and manage them with knife-github-cookbooks!

  • Nicolae Darie

    I am glad i ran over this post!I was looking for something like this:)

  • http://www.simplyexcited.co.uk Alastair Brunton

    Excellent work, going to start using right away.

  • http://twitter.com/samumbach Sam Umbach

    Definitely enjoying knife-github-cookbooks.  Thank you!

  • eric.dp

    Thank you very much !

    PS: you might like the source code tab on the opscode community thingie

  • Pingback: nodejs, npm, lessc, django, workflow to production | madteckhead's weblog

  • http://twitter.com/spazm Mr. Spaz

    bonus tip #2: opscode maintains a split version of their cookbooks at http://github.com/opscode-cookbooks/  . Caveat that these are the development versions before they get released to the community site.

  • Pingback: Provisioning a LAMP stack with Chef, Vagrant, and EC2 (2 of 3) | The Blog of Jason Grimes

  • http://dwradcliffe.com David Radcliffe

    librarian-chef works pretty well for managing cookbook dependencies: https://github.com/applicationsonline/librarian#readme

  • http://twitter.com/fujin_ AJ Christensen

    Recently the Opscode Cookbooks were all split into individual repositories.

    The https://github.com/cookbooks repositories are community maintained and official Opscode cookbooks can be fetched (using the methods detailed above) from https://github.com/opscode-cookbooks/ – the Opscode account for public cookbooks.

  • http://twitter.com/fujin_ AJ Christensen

    Berkshelf (from the guys at Riot Games) is in the same vein as Librarian Chef, it allows one to maintain a variety of cookbooks sourced from all kinds of sources (cookbooks, git, paths, standard Bundler shit)..

    http://berkshelf.com/ 

  • Pingback: Installing Chef Repos Direct from Github | Burgerstuff