Tuesday, April 15, 2014

Cookbook Development Process

UPDATE: We've migrated away from VirtualBox and Minitest in favor of Docker, Chefspec, and Serverspec.  Most of the workflow below is still relevant, but I'll be posting a more detailed update at a future date.

After a few recent conversations I've had, I realized that it may be helpful to share our current Chef cookbook workflow with the community.  It works well for us and I'm pretty proud of how far it's enabled us to come.  That said, if you have any detailed suggestions on how to make it better, let me know.  

Source Code Management

We use Git (Gitolite fronted by GitList) to house all of our cookbooks.  Each cookbook has its own dedicated repository.  We create branches for all new development, merge to master after peer code reviews are completed, and use tags to designate successful codelines (more on this later).

Development Suite

Our cookbook development suite includes CentOS, Test-Kitchen, Vagrant, VirtualBox, Berkshelf, Foodcritic, chef-minitest handler, Veewee to create custom box images, and RVM with dedicated gemsets for each cookbook.

Development Methodology

Integration testing, code promotion
Our testing suite is based on the fail-fast methodology, which means we execute the cheapest tests first (time wise and computationally).  This reduces the feedback loop on the easy stuff and frees up our test systems to spend more time in the Test-Kitchen phase, which is the most costly.

We opted to use Jenkins to perform the testing because it's free, and frankly, it's awesome.  Each Jenkins job checks out the code for the target cookbook and executes the same build job.  It does this via a bash script that we wrote.  The script is kept in its own Git repo and is checked out during each Jenkins build. This allows us to change the testing logic across all of the jobs from a single place, and allows us to track changes to the script over time.  Feel free to check out the script and use it if you like.  You can get a copy here.  The script does the following:

Cookbook linting:
  • Cookbook grammar checking, coding standards.  (Spaces vs Tabs, etc)
  • Verify that each cookbook dependency is version locked in metadata.rb
  • README.md formatting (for example: Jenkins build job URI)
  • Berkshelf Berksfile opscode reference
  • Foodcritic validation
  • Execute the "berks" command to verify that all cookbook dependencies are indeed pulled from our local Git repo instead of the Internet

  • This is where we see if the cookbook executes cleanly and if the minitest suite passes

This completes the testing phases.  At this point, we know that the cookbook passes.  What we do next is my favorite part.

Code Promotion

We now retrieve the version of the cookbook we are testing from the metadata.rb file.  With that version number, we check to see if there is currently a tag in the related Git repo with that version as the name.  If the answer is yes, then that concludes the Jenkins job.  

However, if the tag does NOT exist, we create a new tag based on the current code, and upload the cookbook and all of its dependencies to the Chef Server (note that when Berkshelf uploads the cookbooks, it only uploads cookbook versions that do not already exist on the Chef Server).  And, since we version lock all of our cookbook dependencies, there is no risk of the new cookbook being accidentally rolled out to nodes.  It's just ready for future use.
All Jenkins cookbook jobs are triggered on code commit, and are also scheduled to build every Tuesday morning.  The scheduled build on Tuesdays helps us catch changes that occur in the Ruby world that could negatively affect our build system.  We implemented the scheduled builds after one day when we needed to make a change in our infrastructure and found out that cookbook builds have been broken for two weeks because of a broken Ruby dependency/environment issue.

Development Process

Below are the specific steps we use to develop cookbooks.  I've included a flowchart at the end to give you a visual on the process.

Identify new cookbook requirements

Review the list of requirements for this cookbook.  If this cookbook is replacing an existing configuration, document each requirement that will need to be migrated.  Verify that all proposed requirements are achievable.

Check community site and local repo for existing cookbook

Visit the Opscode Chef Community website and local Git repo and search for an existing cookbook that may already cover some or all of the documented requirements.  If such a cookbook exists, use that one in conjunction with a wrapper cookbook to apply our company-specific settings, if any.

Run the automated cookbook initialization script

You must pass the cookbook name at the end $COOKBOOK\_NAME.  This script performs the following:
  • Accepts a single argument (cookbook name)
  • Creates the cookbook via the berks cookbook command
  • Creates a ruby gemset with the same name as the cookbook (this isolates each cookbook and allows us to experiment with new features without compromising the integrity of the entire development environment)
  • Creates a default Gemfile
  • Executes bundle install
  • Creates a default .kitchen.yml
  • Creates a default README.md, based on our standard format
  • Initializes a new local Git repo, adds all the new cookbook files to the repo, and then performs an initial commit
For more details about what this script does, please analyze the source.

curl -L https://raw.githubusercontent.com/jmauntel/cookbook-init/master/cookbook-init.sh | bash -s -l $COOKBOOK_NAME

Create the new cookbook repo on the Git server so you have a place to store your code

Update README.md

Include description, dependencies, requirements, test cases, and author sections.

Commit your code and push to the Git server

git add .
git commit -a -m "First pass at README.md"
git remote add origin git@git.acme.com:chef-cookbooks/${COOKBOOK_NAME}.git
git push origin master

If the repo does not yet exist, perform this step as soon as possible.

Create minitest logic to test the first requirement

In your cookbook directory, update `files/default/tests/minitest/default_test.rb`.  Examples can be found here.

Boot the first kitchen instance / validate test failure

In most cases you should use the default-centos-6.3 image as your initial testing system, however the full list of available images can be listed by executing `kitchen list`.

kitchen test default-centos-6.3 -d never

Update the cookbook recipe to satisfy the first requirement

Resource documentation and examples can be found on the Opscode documentation site. You can also reference other cookbooks in your Git repo.

Retest cookbook with Test Kitchen

In your cookbook directory, execute the following:

kitchen converge default-centos-6.3

In your cookbook directory, update the appropriate recipe (default is `recipes/default.rb`)

Lint cookbook with Foodcritic

In your cookbook directory, execute the following:

foodcritic -f any . && echo PASS || echo FAIL

If any rules fail, you can look up the meaning on the Foodcritic website.

Rebuild & test cookbook from scratch

In your cookbook directory, execute the following:

kitchen destroy && kitchen test default-centos-6.3 -d never

Destroy Test Kitchen instance

If all tests pass, destroy the instance executing the following:

kitchen destroy

Commit code, push to Git server

git add .
git commit –a –m "New feature passed"
git push origin master

Create a Jenkins job for the cookbook

I can go into this step in more detail, if anyone is interested.

Update the version number in metadata.rb

In your cookbook directory, update `metadata.rb` with a new version number, following SEMVER standards.

Add all files to local git repo, perform commit, push to Git server

In your cookbook directory, execute the following:

git add .
git commit –a –m 'First functional version of cookbook'
git push origin master

Monitor the Jenkins job

If failures are found, refactor the cookbook.


  1. Great article, thanks for sharing! I was wondering about your comment "Execute the 'berks' command to verify that all cookbook dependencies are indeed pulled from our local Git repo instead of the Internet". Do you use a Berksfile that has a source for your Git repo? If so, when pulling in new community cookbooks, do you just pull directly into your Git repo?

  2. Thanks for the feedback, Curtis! You are correct. When we use community cookbooks, we first pull them down and import them into out own Git server. Then we update our berksfiles to point to the git repo to resolve the dependency. Eventually we plan to upgrade to Berkshelf 3, which should make the maintenance of the berksfiles easier.

  3. If I'm not mistaken, `source .rvmrc` could be used instead of `cd ..; cd -`.
    Although, if you don't need multiple ruby versions, you could be better off with just bundler itself (run bundle install with the --path flag).
    Great article, btw! This work is under-appreciated more often than not, yet it makes every engineers' life easier.

  4. Could you please advice on the jenkins job for chef cookbook? I would like to know the approach you took to finally upload the cookbook to the chef server.

  5. Mahesh, if you look at the source of the jenkins-automation job you can see what we did.


    To summarize it up, we fetch the version number from the matadata.rb file, then check and see if the Got repo already has a tag for that version. If there isn't an existing tag, we upload all of the cookbooks used for this run to the Chef server and then create an push the tag to the Git server. We're using the 'berks upload" command to upload the cookbooks.

    Hope this helps.

  6. Thank you. Saves much iteration and dev headache!

  7. Thank you. Saves much iteration and dev headache!