Configuration Management with Chef on Debian, Part 2

09 Jan 2010

At Harvest we are working on putting Chef into production use rapidly. In part 1 I gave you an overview of how to get your Chef server and client installed. There wasn't anything really Chefy in that post. In fact, some of the things in that post were done in a decidedly un-Cheflike manor. Seven lashings for me. Let's look now at getting a first basic configuration task done with Chef.

No handsPreviously we installed Chef from the Opscode Debian packages and then replaced the Chef server URL in the Chef client configuration with sed so that we could register and validate the Chef client with the Chef server for the first time. This obviously won't do, we need to manage all configuration files properly without little post-installation hacks in them. That is a core idea of the Chef. So let's now take control of the Chef client configuration files themselves with Chef.

To do this we need to create a Chef cookbook called "chef" in our repo, in this cookbook we will create a recipe called "client" to handle the client stuff (as opposed to the Chef server stuff which can live happily side by side in the same cookbook but in different recipes), then we need to assign the Chef client recipe to a role (which we will call something like "base" because we are going to add it to all servers) and lastly we need to add that "base" role to our 2 nodes we have. Let's make it work, people.

Rake it like it's 1990My workflow right now is to have my Chef Git repo cloned on my local Mac and to work in there, then commit to this repo and push it to our origin. Then I login to the Chef server, which has it's own clone of the same repo on it, and then use Rake to pull the latest Git changes down and install them into the Chef server directories.

So on my local Mac, lets create the skeleton dir structure for this new cookbook of ours. You could use Rake for this (rake new_cookbook COOKBOOK=chef) but where's the fun in letting someone else do all the menial labor for you?

git clone git@yourgitserver.yourdomain.com:your_chef_repo.git
cd your_chef_repo
mkdir -p cookbooks/chef/attributes
mkdir -p cookbooks/chef/templates/default
mkdir -p cookbooks/chef/recipes

Now let's first define the attributes of our Chef client configuration. These are basically the values which we want in our config files. What we will now do is create some default values, which can be overridden in other places if we ever needed specific Chef client settings to be different on different servers for some reason.

Here is our barebones cookbooks/chef/attributes/chef.rb

set_unless[:chef][:server_fqdn] = "chef.yourdomain.com"
set_unless[:chef][:server_port] = "4000"
set_unless[:chef][:openid_port] = "4001"
set_unless[:chef][:log] = "/var/log/chef/client.log"
set_unless[:chef][:log_level] = "debug"
set_unless[:chef][:filecache] = "/var/cache/chef"
set_unless[:chef][:pid] = "/var/run/chef/client.pid"

Now, we need a template file, which is an ERB template, which our recipe will use to make these dreamy attributes appear on the filesystem somewhere in a fully cooked configuration file.

This is what our cookbooks/chef/templates/default/client.erb looks like:

log_level          <%= @node[:chef][:log_level] %>
log_location       "<%= @node[:chef][:log] %>"

ssl_verify_mode    :verify_none

registration_url   "http://<%= @node[:chef][:server_fqdn] %>:<%= @node[:chef][:server_port] %>"
openid_url         "http://<%= @node[:chef][:server_fqdn] %>:<%= @node[:chef][:openid_port] %>"
template_url       "http://<%= @node[:chef][:server_fqdn] %>:<%= @node[:chef][:server_port] %>"
remotefile_url     "http://<%= @node[:chef][:server_fqdn] %>:<%= @node[:chef][:server_port] %>"
search_url         "http://<%= @node[:chef][:server_fqdn] %>:<%= @node[:chef][:server_port] %>"
role_url           "http://<%= @node[:chef][:server_fqdn] %>:<%= @node[:chef][:server_port] %>"

file_cache_path    "<%= @node[:chef][:filecache] %>"
pid_file           "<%= @node[:chef][:pid] %>"

OK, now let's create a chef::client recipe which will take the attributes and the template and combine them into cold hard configuration files. This is our cookbooks/chef/recipes/client.rb

 
case node[:platform]
 when "debian","ubuntu"

 template "/etc/chef/client.rb" do
    source "client.erb"
    mode 0644
    owner "root"
    group "root"
    backup false
  end

end

OK, so now we have the attributes, the template and the recipe. Doesn't do us much good unless we somehow tell the Chef server that our nodes need to execute the chef::client recipe when the Chef client runs. So let's create our role, called "base". Here is roles/base.rb

name "base"
description "All systems to get this role"
recipes "chef::client"

override_attributes "chef" => { 
    "server_fqdn" => "chef.yourdomain.com",
    "log_level" => "info"
}

Something to notice here. In our attributes file earlier, we used set_unless to define a few attributes of this cookbook. But here in the role we are overriding them (with the same values in this case). The utility here is that using the exact same chef cookbook, we could create a new role, called something like base_london and use override_attributes in the Role definition to give those nodes a different value for server_fqdn.

Now let's commit all of this new code we have written to our Chef git repo.

git add *
git commit -a -m "adding our chef cookbook and role, first pass"
git push

Now, let's login to our Chef server, pull down all these new changes in our git repo and actually get this stuff into our running Chef server.

ssh chef.yourdomain.com
git clone git@yourgitserver.yourdomain.com:your_chef_repo.git
cd your_chef_repo
rake install
sudo /etc/init.d/chef-server restart

(The last step, the restart of the Chef server, is needed because we have added a new role. Your role you defined in the Ruby file above gets compiled to JSON and then is stored in CouchDB, and we need to bounce the CouchDB server in the 0.7 Chef series for your changes to become effective).

KnifeOK. So, now we have everything in place, except one. We haven't told the Chef server to actually apply the "base" role to our nodes. Now you would expect to use an elegant CLI tool which would use the Chef API to perform these types of operations on the Chef server. The good news is that Knife is that tool. The bad news is that I haven't tested Knife yet, and because you are on my blog you are SOL. So we'll be using the Chef web UI for this task until I get to try the Knife.

So, login to http://chef.yourdomain.com:4000 using your OpenID login. Go to Nodes and click on "edit" next to one of your nodes listed there if both chef.yourdomain.com and web1.yourdomain.com are registered on the server, or whichever node you have been working on. What you need to do now is drag "base" from the Available Roles into the Run List and Save Node.

So the final step you need to do is actually run chef-client on your server to actually fetch and apply the base role to it.

sudo chef-client -l debug

If all went according to plan, then you should see that Chef is now managing the contents of /etc/chef/client.rb.

This was a very simple quick example of how a Chef cookbook, recipe, role and runlist work together to Get Things Done. There is a whole lot more to a Chef cookbook than this. The next thing to do is to forget everything I just told you, thank the heavens for jtimberman and kallistec and all the other fine Chefs who hang out in #chef on freenode. Take a look at some real Cookbooks from these guys which really start to leverage the power of Chef.

In part 3 of this series, coming out to coincide with Duke Nukem Forever, I'll talk about a more advanced cookbook, and hopefully Knife. Thanks for staying this long.