warwick poole, blog.

Is UCEPROTECT running a protection racket?

19 Aug 2010

There may be a bulkmail flyer business operating out of your suburb somewhere. Assume there is, just humor me. Imagine then if a mail carrier standing in front of a mailbox in Alaska is about to deposit the Christmas card you wrote to your pal in Anchorage into your pal's mailbox. But before the mail carrier opens the mailbox, he looks at your return address and then he takes out his cellphone and calls someone who we'll call Scumbag Bob. He asks Scumbag Bob if there has ever been a report of junkmail originating from your zipcode. Turns out there has been. In fact, Scumbag Bob tells him, your zipcode is a source of lots of junkmail because remember you've got a junkmail business down the road from you. The mail carrier then refuses to deliver your postcard to your pal, instead he returns it to you with a note to contact Scumbag Bob.

All is not lost though! You can easily fix the situation. You can complain to your city council. Once they have had some meetings, eaten some donuts, maybe had a few fist fights and finally closed down all junkmail businesses in your zipcode, they can then talk to Scumbag Bob and get your zipcode removed from his list. Then your cookie recipes will be zipping around the world again.

OR... If YOU PAY Scumbag Bob $20/month, he will remember to tell mail carriers that even though your zipcode is a known source of junkmail, they should allow your (and only yours) letters through because you are a decent enough fellow.

Of course everyone knows that a scam like this would never fly. First of all, why should your entire zipcode suffer because of the junkmail business down the road from you? Secondly, no self respecting mail carrier would subscribe to that kind of service, right? And thirdly, Scumbag Bob would be put out of business because it's just a ridiculous (and borderline illegal) idea anyway.

Well, ladies and gentlemen, I present to you Scumbag Bob. He is known as UCEPROTECT-network and he sells his services as whitelisted.org.

Here is UCEPROTECT trying to get me to pay them $18/month to whitelist my own specific IPs because they have blacklisted my entire ISP. That's every IP address at my entire datacenter apparently.

Based on the above, I have to urge reasonable email server administrators to NOT subscribe to UCEPROTECT Level-3, which seems like a massive scam to me.

My ISP reports endless problems trying to get removed from this UCEPROTECT RBL. In fact, given the shady extortion fee system they have going here, I suspect UCEPROTECT are in the business of keeping innocent bystander IPs on their blacklists, and getting paid to remove them.

Never Work For Free

12 Aug 2010

Never work for free. I don't do consulting work anymore. However, about an hour ago I got a little email...

From: Martin Family
Date: Thu, Aug 12, 2010 at 2:17 PM
To: Warwick Poole
Subject: (no subject)

I need a plugin that does EXACTLY what the calendar here does: www.iwearyourshirt.com/calendar

I am also interested in possibly another simple plugin that uses innerHTML, and your SEO services.

This is a start-up business, so there is not a lot of $$ to go around as of now. I have done all the web design so far, but modifying/writing plugins is something that will be a bit too time consuming for someone with limited experience, like myself. I am really looking for someone, or some company, who is interested in doing this project for me in exchange for a one year spot as a partner on our homepage (this is valued at $12,000...that is what we are charging for these spots, just like our competitor), and the pick of any day of the year for advertising (Up to a $800 value depending on the day). If you want to see what we do (i know the site is not too informative yet), go to www.iwearyourshirt.com . They do the same thing we do, but we are an entire family, and we are more visible than they are overall. I can say that people will see your work on our site first hand, and you will get a lot of exposure from this project. Let me know what you think. Thanks.

Carl Martin | www.billboardfamily.com

Follow Us On Twitter: http://www.twitter.com/billboardfamily
Become A Fan On Facebook: http://www.facebook.com/BillboardFamily
Facebook (Carl): http://www.facebook.com/CarlMartin.BillboardFamily
Facebook (Amy): http://www.facebook.com/AmyMartin.BillboardFamily
Follow Us On Tumblr: http://billboardfamily.tumblr.com/

From: Warwick Poole
Date: Thu, Aug 12, 2010 at 2:17 PM
To: Martin Family
Subject: re: (no subject)

Hi Carl Martin (AND FAMILY!)

Yes, I'd love to do some work for you for a promise of exposure on your site, especially since your idea is so original! $12,000 value? Count me in.

Thanks for the unsolicited mail. You make the Internet a much better place.

Best,
Warwick Poole

From: Martin Family
Date: Thu, Aug 12, 2010 at 2:17 PM
To: Warwick Poole
Subject: re: (no subject)

Are you being sarcastic?

From: Warwick Poole
Date: Thu, Aug 12, 2010 at 2:17 PM
To: Martin Family
Subject: re: (no subject)

No.

From: Martin Family
Date: Thu, Aug 12, 2010 at 2:17 PM
To: Warwick Poole
Subject: re: (no subject)

The message sounded a bit sarcastic. Sorry if I misunderstood you. If you are really interested in designing these plugins for us, let me know how you want to proceed. Have you developed any other Wordpress plugins?

Thanks

From: Warwick Poole
Date: Thu, Aug 12, 2010 at 2:17 PM
To: Martin Family
Subject: re: (no subject)

I did a plugin which is featured on this page: http://www.27bslash6.com/p2p2.html

Have you ever seen it?

From: Martin Family
Date: Thu, Aug 12, 2010 at 2:17 PM
To: Warwick Poole
Subject: re: (no subject)

No....what is the plugin called, and what does it do?

From: Warwick Poole
Date: Thu, Aug 12, 2010 at 2:17 PM
To: Martin Family
Subject: re: (no subject)

The plugin is called OneSmallChange.
Essentially you take an existing original idea, make one small change, and wait for success.

From: Martin Family
Date: Thu, Aug 12, 2010 at 2:17 PM
To: Warwick Poole
Subject: re: (no subject)

ok, so when can you have the plugins I need finished?

From: Warwick Poole
Date: Thu, Aug 12, 2010 at 2:17 PM
To: Martin Family
Subject: re: (no subject)

Sorry I misspoke earlier. When you asked:

> Are you being sarcastic?

I meant to say "yes". But I typed "no" for some reason. My mistake.

From: Martin Family
Date: Thu, Aug 12, 2010 at 2:17 PM
To: Warwick Poole
Subject: re: (no subject)

Way to run a business...what a douche bag

The Martin Family
Carl, Amy, Carrera, Layne, and Kaitlyn
martinfamily2005@yahoo.com

From: Warwick Poole
Date: Thu, Aug 12, 2010 at 2:17 PM
To: Martin Family
Subject: re: (no subject)

Good luck Martin Family!

Navarro, the cat with Buffalo Butterfly wings

12 Aug 2010

The Butterfly effect. Small differences in the initial condition of a dynamical system may produce large variations in the long term behavior of the system.

To illustrate: A clearly insane person in Buffalo, and his unfortunate (but ultimately lucky) cat Navarro, conspire together to flap their butterfly wings through Google indices directly to my browser's search bar.

Of course, that's not really how the the Butterfly effect was actually first described by Edward Lorenz. Neat, correlating pairs of cause and effect are not implied.

Regardless, perhaps the person who let Mr. Korkuc adopt little Navarro from the SPCA should have a quick bath in Dave's Insanity Sauce?

Calling out the spammers

10 Aug 2010

It's not going to help. Not even one bit. Regardless, I'm going to maintain a little list here of the companies who call me at home, unsolicited, or who spam me in other obvious ways.

I've had Comcast voice for 5 months, never really use it. I don't even know the number, so I've never given it to anyone, or signed up for any service using it. I didn't pay the Comcast extortion fee to not go into their list they sell to the charlatans of the world. The price I pay is around 8 spam calls a week.

Without further ado, I present to you the bottom feeders of the world. Here are the companies our world would be a better place without.

Synovate

Oh my God, just the name. You can feel the lizardy skin. Seriously, this is the name that Lumbergh would have called Initech had he founded it. They called me to ask my opinion on some bullshit for their global opinion poll. I was off the phone before I could hear the Earth shattering topic of the poll. My opinion: Life is short and you, Synovate employees, spend your time making the world a worse place FOR ALL OF US. Do you get calls at home? Go and do something else. And Synovate customers: It occurs to me you may not know the source of the data in those pretty charts Synovate hand you, in the same way you may not know the source of the "meat" in your hotdog. Well, I'll tell you: The data comes from people in their homes at 8:30pm, trying to put their 11 month old son to sleep, and after finally getting him to sleep after two hours, guess who calls to wake him up? Nope, not Grandpa Joe, it's Bob from Synovate to ask me if I use some kind of dish detergent you make. Well guess what, I do now, and it's to make bombs. (Side note: I don't know if detergent can make bombs. Please don't make bombs, I don't make bombs. FBI: I don't make bombs.)

LocalEdge Media

Most people have heard of the disappearing rainforests. Let me present to you the people who don't give a shit: The owners of LocalEdge, Hearst Corporation. These people dropped a massive deadtree "Yellow Pages" (I don't care to know the various types of spambooks) directory on every driveway in the whole Charleston area. Even if you ask them not to. Near as I can tell, they then get some call center in India to call you up and GET YOU TO PAY THEM FOR IT. Wow. Soak that in. Holy crap, these people walk among us. And so thousands of companies must be paying these people to print these books. And so when your grandchild sits on your knee and says "Grandpa, what did you put gifts under before there were Christmas carbon fiber poles?", I hope you, dear tax accountant in Mount Pleasant, treasure that ad you paid the devil to chop down a tree and print in the blood of our planet. I am building a retaliatory slingshot which will propel deadtree books back at the vehicle which tossed them on my driveway, but at 400 feet per second. PS: You are reading this on the Internet. The Internet can be used to advertise your business.

To be continued...

Debugging Rails processes under Phusion Passenger with nginx.

14 Jun 2010

If you are involved in web operations you'll no doubt be familiar with this problem: Everything is running along smoothly and suddenly an anomalous load spike occurs which does not correlate to a traffic spike or any problem on which you have any data. Last week this very thing occurred in a mature Rails application which runs under Phusion Passenger on nginx. The initial symptom was a rapid CPU spike (Rails processes in user space) and then Rails processes becoming stuck on a single request. This rapidly brought on a typical web server death spiral: Increasing CPU usage, increasing memory usage, eventually swapping occurs and/or the application server runs out of available processes in the pool. Frustrated users hit refresh and the issue compounds.

In my experience the order of likely causes of a problem goes like this: Buggy code recently deployed, large and expensive reports being run, or network (or firewall) problems causing processes to hang on IO operations. How to troubleshoot this problem on Passenger?

Checking the output of passenger-status we were seeing Rails processes not completing requests, the "Processed" column for certain Rails processes were not incrementing, indicating these processes were somehow stuck, thereby decreasing the pool of available processes for subsequent requests. Attaching to these stuck processes with strace revealed no activity in the process at all. Similarly, using gdb.rb revealed nothing immediately useful. Strangely we were seeing some Rails processes continuing to run even after they were no longer being actively managed by Passenger, at least according to the output of passenger-status, in which these processes no longer appeared.

First I upgraded Passenger (using Chef, of course) to the latest version on one server, to see if there was a Passenger bug causing this problem. The issue persisted. Making things a bit more complicated we had recently moved a memcached instance to a different part of the network, and all the stuck Rails processes had open network connections to this memcached instance, leading me to wonder if it was this network IO which was somehow causing the Rails process to hang. I could find no problems between the application servers and the memcached port and no logs of any network issues there. So what are the stuck Rails processes stuck on?

The way to troubleshoot this problem on Passenger is to  issue a kill -SIGABRT to the stuck Rails processes, and find their backtrace in the Rails log. Looking at the backtraces we were able to track the issue down to a bug in a Ruby gem which was being used to generate certain types of reports. Disabling this reporting prevented new occurrences of the problem until the underlying issue can be resolved. An interesting combination of two usual suspects: Reporting and code library bugs.

Vodafone / Vodacom 3G in South Africa

17 May 2010

I’m in South Africa for 2 weeks and am using a Vodafone Huawei USB 3G card on my Macbook Pro with Snow Leopard. Here are a few tips I found useful:

Brandon West’s post was very useful in getting started.

Summary: Download and install VodafoneMCInstaller then add the 3G card as a network interface and change the settings as mentioned in Brandon’s post.

To get online you’ll need to open the Vodafone application, and use the 3G card’s pin number to Activate your modem every time you connect. Once activated, you need to then dial up the modem using the Network panel in System Preferences.

Because your Vodacom 3G account is a metered account, and I’m not used to conserving bandwidth in any way, I found SurplusMeter very useful to watch my usage.

Ruby self-throttling Nagios alerts

19 Mar 2010

If you use Nagios no doubt you've experienced a situation where a network issue has triggered an alert on every service on every server. There are a few ways to deal with this in Nagios, but after spending 10 minutes deleting SMS messages on my iPhone I decided to implement an alert system which will self throttle and not allow a gigantic flood of alerts.

In Nagios I created 2 notify commands, one for host notifications and another for service notifications:

Then throttle_alert.rb uses a control file to determine how long it has been since the previous alert, and will redirect alerts to a secondary (non-SMS) email address (and note the throttling in the alert itself) if an alert occurs inside the throttled window (90 seconds in this example):

Not rocket science and not perfect but it works to prevent draining my iPhone battery in a single incident.

Configuration Management with Chef on Debian, Part 2

09 Jan 2010

At Harvest we are working on putting Chef into production use rapidly. In part 1 I gave you an overview of how to get your Chef server and client installed. There wasn't anything really Chefy in that post. In fact, some of the things in that post were done in a decidedly un-Cheflike manor. Seven lashings for me. Let's look now at getting a first basic configuration task done with Chef.

Previously we installed Chef from the Opscode Debian packages and then replaced the Chef server URL in the Chef client configuration with sed so that we could register and validate the Chef client with the Chef server for the first time. This obviously won't do, we need to manage all configuration files properly without little post-installation hacks in them. That is a core idea of the Chef. So let's now take control of the Chef client configuration files themselves with Chef.

To do this we need to create a Chef cookbook called "chef" in our repo, in this cookbook we will create a recipe called "client" to handle the client stuff (as opposed to the Chef server stuff which can live happily side by side in the same cookbook but in different recipes), then we need to assign the Chef client recipe to a role (which we will call something like "base" because we are going to add it to all servers) and lastly we need to add that "base" role to our 2 nodes we have. Let's make it work, people.

My workflow right now is to have my Chef Git repo cloned on my local Mac and to work in there, then commit to this repo and push it to our origin. Then I login to the Chef server, which has it's own clone of the same repo on it, and then use Rake to pull the latest Git changes down and install them into the Chef server directories.

So on my local Mac, lets create the skeleton dir structure for this new cookbook of ours. You could use Rake for this (rake new_cookbook COOKBOOK=chef) but where's the fun in letting someone else do all the menial labor for you?

git clone git@yourgitserver.yourdomain.com:your_chef_repo.git
cd your_chef_repo
mkdir -p cookbooks/chef/attributes
mkdir -p cookbooks/chef/templates/default
mkdir -p cookbooks/chef/recipes

Now let's first define the attributes of our Chef client configuration. These are basically the values which we want in our config files. What we will now do is create some default values, which can be overridden in other places if we ever needed specific Chef client settings to be different on different servers for some reason.

Here is our barebones cookbooks/chef/attributes/chef.rb

set_unless[:chef][:server_fqdn] = "chef.yourdomain.com"
set_unless[:chef][:server_port] = "4000"
set_unless[:chef][:openid_port] = "4001"
set_unless[:chef][:log] = "/var/log/chef/client.log"
set_unless[:chef][:log_level] = "debug"
set_unless[:chef][:filecache] = "/var/cache/chef"
set_unless[:chef][:pid] = "/var/run/chef/client.pid"

Now, we need a template file, which is an ERB template, which our recipe will use to make these dreamy attributes appear on the filesystem somewhere in a fully cooked configuration file.

This is what our cookbooks/chef/templates/default/client.erb looks like:

log_level          <%= @node[:chef][:log_level] %>
log_location       "<%= @node[:chef][:log] %>"

ssl_verify_mode    :verify_none

registration_url   "http://<%= @node[:chef][:server_fqdn] %>:<%= @node[:chef][:server_port] %>"
openid_url         "http://<%= @node[:chef][:server_fqdn] %>:<%= @node[:chef][:openid_port] %>"
template_url       "http://<%= @node[:chef][:server_fqdn] %>:<%= @node[:chef][:server_port] %>"
remotefile_url     "http://<%= @node[:chef][:server_fqdn] %>:<%= @node[:chef][:server_port] %>"
search_url         "http://<%= @node[:chef][:server_fqdn] %>:<%= @node[:chef][:server_port] %>"
role_url           "http://<%= @node[:chef][:server_fqdn] %>:<%= @node[:chef][:server_port] %>"

file_cache_path    "<%= @node[:chef][:filecache] %>"
pid_file           "<%= @node[:chef][:pid] %>"

OK, now let's create a chef::client recipe which will take the attributes and the template and combine them into cold hard configuration files. This is our cookbooks/chef/recipes/client.rb

 
case node[:platform]
 when "debian","ubuntu"

 template "/etc/chef/client.rb" do
    source "client.erb"
    mode 0644
    owner "root"
    group "root"
    backup false
  end

end

OK, so now we have the attributes, the template and the recipe. Doesn't do us much good unless we somehow tell the Chef server that our nodes need to execute the chef::client recipe when the Chef client runs. So let's create our role, called "base". Here is roles/base.rb

name "base"
description "All systems to get this role"
recipes "chef::client"

override_attributes "chef" => { 
    "server_fqdn" => "chef.yourdomain.com",
    "log_level" => "info"
}

Something to notice here. In our attributes file earlier, we used set_unless to define a few attributes of this cookbook. But here in the role we are overriding them (with the same values in this case). The utility here is that using the exact same chef cookbook, we could create a new role, called something like base_london and use override_attributes in the Role definition to give those nodes a different value for server_fqdn.

Now let's commit all of this new code we have written to our Chef git repo.

git add *
git commit -a -m "adding our chef cookbook and role, first pass"
git push

Now, let's login to our Chef server, pull down all these new changes in our git repo and actually get this stuff into our running Chef server.

ssh chef.yourdomain.com
git clone git@yourgitserver.yourdomain.com:your_chef_repo.git
cd your_chef_repo
rake install
sudo /etc/init.d/chef-server restart

(The last step, the restart of the Chef server, is needed because we have added a new role. Your role you defined in the Ruby file above gets compiled to JSON and then is stored in CouchDB, and we need to bounce the CouchDB server in the 0.7 Chef series for your changes to become effective).

OK. So, now we have everything in place, except one. We haven't told the Chef server to actually apply the "base" role to our nodes. Now you would expect to use an elegant CLI tool which would use the Chef API to perform these types of operations on the Chef server. The good news is that Knife is that tool. The bad news is that I haven't tested Knife yet, and because you are on my blog you are SOL. So we'll be using the Chef web UI for this task until I get to try the Knife.

So, login to http://chef.yourdomain.com:4000 using your OpenID login. Go to Nodes and click on "edit" next to one of your nodes listed there if both chef.yourdomain.com and web1.yourdomain.com are registered on the server, or whichever node you have been working on. What you need to do now is drag "base" from the Available Roles into the Run List and Save Node.

So the final step you need to do is actually run chef-client on your server to actually fetch and apply the base role to it.

sudo chef-client -l debug

If all went according to plan, then you should see that Chef is now managing the contents of /etc/chef/client.rb.

This was a very simple quick example of how a Chef cookbook, recipe, role and runlist work together to Get Things Done. There is a whole lot more to a Chef cookbook than this. The next thing to do is to forget everything I just told you, thank the heavens for jtimberman and kallistec and all the other fine Chefs who hang out in #chef on freenode. Take a look at some real Cookbooks from these guys which really start to leverage the power of Chef.

In part 3 of this series, coming out to coincide with Duke Nukem Forever, I'll talk about a more advanced cookbook, and hopefully Knife. Thanks for staying this long.

Using multiple EC2 accounts with the EC2 API tools

04 Jan 2010

Various projects I am involved with each have their own Amazon EC2 accounts. This means I have a few sets of certificates/keys to access and manage these different instances. My previous solution was a different user account on my Macbook Pro for each project. That is not very elegant at all. This is my new solution, which involves a set of dynamic bash aliases and a script to create them.

To start, create a directory structure like this (preferably in a Git or SVN repo)

base/
accounts/

Then in accounts/, create a subdirectory for each project/client/ec2 account, and add 3 things to each subdirectory: the EC2 X.509 certificate, the EC2 private key and a single SSH private key to use for SSH access. So end up with a directory structure something like this, separated by client/project/account:

base/
accounts/project1/cert-TYTYTHJHDJHDJHDJHDJH.pem
accounts/project1/pk-DNKJNFKDJNFFKJNDFKNSDFJ.pem
accounts/project1/id_dsa_project1
accounts/client55/cert-SASDASDASDASDASDASD.pem
accounts/client55/pk-UYWYWUYWUYNNSNSNS.pem
accounts/client55/mykey_client55
accounts/myappx/cert-XCMLKMLKMLKMLKM.pem
accounts/myappx/pk-CCJKJDKPOPOPOPPOP.pem
accounts/myappx/awskey_appx

Now we'll download the latest EC2 API tools and expand them into base/, so we will have a directory structure like this:

base/THIRDPARTYLICENSE.TXT
base/bin/ec2-add-group
base/bin/ec2-add-group.cmd
base/bin/ec2-add-keypair
base/bin/ec2-add-keypair.cmd
-- ETC -- OMITTING THE REST OF bin/
base/lib/activation-1.1.jar
base/lib/bcprov.jar
-- ETC -- OMITTING THE REST OF lib/
base/license.txt
base/notice.txt
accounts/project1/cert-TYTYTHJHDJHDJHDJHDJH.pem
accounts/project1/pk-DNKJNFKDJNFFKJNDFKNSDFJ.pem
accounts/project1/id_dsa_project1
-- ETC -- OMITTING THE REST OF accounts/

Now, we will use a simple bash script to generate a bash alias for each EC2 operation for each EC2 account we have. It essentially maps each EC2 command to the appropriate key and cert for each of our projects/clients.

NOTE: This version sets up some environment variables which are specific to a Mac OS X > Leopard environment. Yours may need different JAVA_HOME, if you are on a different OS, etc.

#!/bin/bash
export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Home/
export EC2_TOP=$( dirname $BASH_SOURCE )
export EC2_HOME=$EC2_TOP/base

ALL_ACCOUNTS=$( ls $EC2_TOP/accounts )
EC2_TOOLS=$( ls $EC2_HOME/bin | grep -v .cmd )

THIS_KEY=''
THIS_CERT=''

for a in $ALL_ACCOUNTS; do
        THIS_KEY=$( ls $EC2_TOP/accounts/$a/pk-*.pem )
        THIS_CERT=$( ls $EC2_TOP/accounts/$a/cert-*.pem )
        THIS_SSH=$( ls $EC2_TOP/accounts/$a/* | grep -v .pem )

        alias ssh-ec2-$a="ssh -i $THIS_SSH"

        for e in $EC2_TOOLS; do
                alias ec2-$a-$e="$EC2_HOME/bin/$e -K $THIS_KEY -C $THIS_CERT"
        done

        THIS_KEY=''
        THIS_CERT=''

done

This base script (I've called mine setup_env.sh) needs to live at the top level, so again, we have a structure like this:

base/
accounts/
setup_env.sh

Now, simply source setup_env.sh as part of your login procedure, by putting something like this in .bash_profile:

NOTE: My top level directory lives at ~/Tools/Amazon

source ~/Tools/Amazon/setup_env.sh

Now, when I login, I can create an instance on the correct EC2 account with something like:

ec2-project1-ec2run <options>

And I can SSH to a running EC2 instance with something like:

ssh-ec2-project1 root@myinstance.address.com

Simple, possibly inelegant, but very functional. Do you have anything which works better than this? Please share..

Side Project. I need one. Help me decide, idea 1: HPIT

30 Dec 2009

I want to write a another web application in early 2010. Here is the first idea I had. Rain some criticism down or heap some praise. I haven't seen this service out there, so if I missed it somewhere, please tell me.

Idea 1: The Hosting Provider Incident Tracker (HPIT)

A real time database where we track major (and minor) incidents at hosting providers around the world. Too many times I have asked hosting providers about their history of incidents only to receive vague assertions about 100% uptime and heroic stories surviving the great blackouts of our time. This always fails to impress me. What would REALLY impress me is a provider who would give me a link to their incident history with detailed technical explanations around the root causes (ugh, hate that word) and mitigation steps. Sadly, providers still want to sell the snakeoil of 100% uptime. This isn't the point, because there isn't 100% uptime. There is only honest and timely response to inevitable outages. This is what matters. This, however, opens the door for an independent tracker of these incidents. This service will vet incident reports, seek and post responses from hosting providers, or simply link to their own responses, and be your source for evaluating how a hosting company responds to critical issues.

BTW: I don't know why Pingdom isn't providing this service today. It has most of the data required already.

Configuration Management with Chef on Debian, Part 1

17 Dec 2009

UPDATE: Part 2 now online

I'm familiar with configuration management tools like Cfengine, Puppet and slack. I am also familiar with using homebrewed tools which leverage trees of shell scripts to apply and manage a bunch of configuration files stored in version control. I am going to teach myself to use Chef on Debian Lenny. Follow along.

Firstly, idempotent. Chef is idempotent. Wha? This means that if you run the chef client on a node multiple times, you should never get a gradually changing state. Given the same recipes, the system should always return to an identical state after the Chef tools run on the system. Think about those shell scripts you have written in the past. Sometimes they are appending strings to files, or removing strings from files, or moving files around based on some algorithm. Well, Chef doesn't perform a single action out of context. It does every action, every time. And the actions Chef performs are defined in recipes. Collections of recipes are called cookbooks. These recipes are written in Ruby. Recipes can be grouped together in roles, so when you apply a role to a node that node will receive all the recipes in that role. Recipes can reference attributes in other recipes, and also apply other recipes.

The bulk of the Chef work is done by the chef-client which runs on the host you are managing with chef. It may or may not contact a chef-server. Using chef-solo you can do a bunch of useful things without needing a central chef-server.

There are quite a few different tools in use in the complete Chef stack. Wonderful tools like couchdb, Stomp, Merb. Luckily, thanks to great packages from Opscode, you can get all of this stuff installed without too much trouble.

Note on versions: Chef 0.8 branch is currently getting some very interesting sounding features. In fact, I remarked in IRC today that I fully expect Chef 0.8 to file my taxes and walk my dog. However, I am not going there yet. This post will be all about Chef 0.7 branch, specifically the version of Chef provided in the Opscode apt repositories. By the way, you should totally hang out in #chef on Freenode. The people in there are awesome, helpful and very very good looking.

This is it. Let's do things.

Set up your Git repository

The various pieces which end up becoming files on the filesystem are stored in version control. You can apparently use SVN, but we'll go with Git. You'll need a nice fresh Git repo. To this repo you'll add your Chef cookbooks and roles, etc. So you could either host your Git repo with GitHub (you'll probably want to use a private repo, the repo will most likely contain some private things) or you could set up you own Git repo using Gitosis.

Once you have your Chef repo setup, get the contents of the Opscode Chef Repository into your own repo. I'm not going to go into the details of how to do this using Git, mostly because you don't want Git advice from me. Nevertheless, you want to grab the contents of this repo, and merge them in your new repo:

git clone git://github.com/opscode/chef-repo.git

When all is said and done, your repo should have a structure similar to this:

certificates/
config/
config/client.rb.example
config/server.rb.example
config/rake.rb
config/solo.rb.example
cookbooks/
Rakefile
roles/
site-cookbooks/

More detailed info on the Chef Repository

Setup your 2 servers and your DNS

For the purposes of this tutorial, we'll assume you have 2 servers. You could have 17 servers, that would be fine too. We'll call server 1 chef.yourdomain.com and server 2 will be web1.yourdomain.com. Make sure these servers have Debian Lenny installed and you have created proper working DNS entries for the hostnames chef.yourdomain.com and web1.yourdomain.com

We'll be installing Chef server AND a Chef client on chef.yourdomain.com We'll be installing only a Chef client on web1.yourdomain.com

Setup your Chef server

There are Chef cookbooks which will allow Chef to bootstrap itself, which is a fairly cool concept. This isn't necessary, however, when using the Opscode chef packages for Debian, since they will provide everything we need for our Chef server in the packages. So we'll simply install the packages and be on our way.

First, we'll need these:

apt-get update
apt-get install -y curl git-core sudo

Now let's set up the Opscode repo and install Chef:

echo "deb http://apt.opscode.com/ debian contrib" > /etc/apt/sources.list.d/opscode.list
curl http://apt.opscode.com/packages@opscode.com.gpg.key | apt-key add -
apt-get update
apt-get install -y rubygems libshadow-ruby1.8 ohai chef chef-server

Done. Buy Opscode some beer! OK, now let's do some configuration of the Chef server. Firstly, Chef uses OpenID extensively for authentication. We need to have an OpenID provider, which we trust defined in the chef server config. If you don't have an OpenID provider that you use, head over to myOpenID and sign up for an account. You'll get an OpenID URL. Now we'll put that into the chef config at /etc/chef/server.rb by adding the following line at the bottom of that file:

authorized_openid_providers [ "https://chef.localdomain", "https://chef", "https://whateveryourswas.myopenid.com"]

Now we are going to generate a validation token for our Chef clients to use to auto-register themselves against the Chef server. You need a nice random string. If you lack the creativity to make one up, try this:

thedate=$( date ); thehost=$( hostname ); echo $thedate$thehost | sha256sum

Now put the string you came up with above into /etc/chef/server.rb by finding the validation_token line, uncommenting it and giving it a value:

validation_token   "2d926b251e18cb39b00350bb956d44ef8639b4ed33809194a7e2ed60ac5d772c"

Restart Chef server:

/etc/init.d/chef-server restart

Setup your Chef Repository

Now we need our Git repository created earlier. We are going to use this repository to store our recipes, and we are going to use tools inside this repository to manage Chef itself. How meta. So on your newly installed chef server, in your home directory, or whereever else, clone your git repo:

git clone git@yourgitserver.yourdomain.com:your_chef_repo.git
cd your_chef_repo

We are now going to use Rake to install the initial (blank) Chef cookbooks into place, and also to set up the SSL certificates which Chef uses when communicating between components. Obviously you'll want to use your Chef server's proper FQDN here.

rake install
rake ssl_cert FQDN='chef.yourdomain.com'

Now, try logging into the Chef web UI, at http://chef.yourdomain.com:4000

In the login field, enter the openID URL you configured earlier, eg: https://whateveryourswas.myopenid.com you'll be forwarded to the OpenID provider, asked to authenticate and sent back to Chef. You should now be logged in.

Set up your Chef client

OK, now that we have a working, albeit currently fairly useless, Chef server let's install our first Chef client. On your other host, web1.yourdomain.com, you could use this script I quickly put together to go from a freshly installed server to a working Chef client. Obviously you should substitute your auth_token you created earlier and your chef's DNS location.

#!/bin/bash
### VARIABLES ##########################################################
chef_server="chef.yourdomain.com"
chef_client=/usr/bin/chef-client
chef_client_config="/etc/chef/client.rb"
chef_auth_token="2d926b251e18cb39b00350bb956d44ef8639b4ed33809194a7e2ed60ac5d772c"

opscode_apt="deb http://apt.opscode.com/ debian contrib"

### ACTIONS ############################################################
apt-get update
apt-get install -y curl git-core sudo

echo $opscode_apt > /etc/apt/sources.list.d/opscode.list
curl http://apt.opscode.com/packages@opscode.com.gpg.key | apt-key add -
apt-get update
apt-get install -y rubygems ohai chef libshadow-ruby1.8

### SETUP CHEF CLIENT AND REGISTER #####################################
sed -i "s/localhost/$chef_server/g" $chef_client_config
$chef_client -t $chef_auth_token

### REMOVE THYSELF #####################################################
rm $0

If all went according to plan, this should now be a working chef client which knows where the chef server is, and has auto registered itself with the Chef server. Have a look at http://chef.yourdomain.com:4000/registrations and see if web1.yourdomain.com is listed there. If so you are halfway to Nirvana. So now we actually want to start doing things by creating cookbooks, roles, ie: actually doing shit.

Stay tuned for part 2 in this enthralling series. Contact me for movie rights.

UPDATE: Part 2 now online

Cliff Notes from Founder Factory 2009

20 Nov 2009

Yesterday the Founder Factory took place at the World Cafe Live in Philadelphia. I love the venue and there were some interesting speakers and discussions. Here are the things I found most interesting from the talks:

Doug Alexander, President - Internet Capital Group

• If you have a B2B product or service, figure out which budget you come out of. Don't come out of the IT budget, which is heavily pressurized and quickly trimmed. Come out of the marketing, or COGS budget, which are generally more sacrosanct.
• Again with a B2B product or service, you want to be solving a problem that keeps a senior executive (at your target client) up at night. By inference (if it were indeed a b2b product), I declare sildenafil citrate the greatest product of all time.
• Avoid missionary selling, which is to say, avoid spending money to educate influencers in your market that they need your product or service. This missionary work usually has the effect of educating the market on behalf of your competitors as well.
• Advice to a fishbowl company: If seeking outside funding, ensure you have a concrete vision (with detail) of what success looks like for your company. Instead of relying on the potential investor to create their own vision of this scenario, paint this picture in detail and sell this vision to the investor.

Dr Jeremy Siegel, Professor - Wharton School

• While the dollar has slumped recently, it slumped after a spectacular and unnatural rise, setting it's current value still above the levels it was in 2007 (I believe, I need references, since I do not follow currencies at all). Essentially, view current value in a broader context before declaring disaster. I've had this sentiment expressed to me before about the stock market in general.
• Focusing on the deficit and foreign debt as threats to the US economy ignores far larger threats, such as social security and medicare, which represent buckets orders of magnitude larger than the deficit.

David Brussin, CEO - Monetate

• Use external QA resources. It's a triple win of fresh eyes with more perspective and no emotional ties to the product.
• Cloud, cloud, cloud. And CDN. With cloud.
• Interesting tactic: don't upgrade your software in the cloud. Deploy new infrastructure and move traffic over, your rollback being the previous setup. (IMHO this is only practical to a threshold. What happens where your 'setup' is hundreds/thousands of nodes?)
• If your product imposes management overhead on your customer, becoming "full service", ie: providing a product as well as management of the product as a combined offering, might be a solid commercial solution and an easier sell. "The work on your part will be limited to banking all this money we create for you".

The Fishbowl

In the fishbowl, three companies in various phases of development, asked a panel of experts (and the audience) for advice on certain issues.

Kendra Gaeta of Kidzillions was unfortunate enough to be the person who taught some of the speakers (and me) a valuable lesson, which is: If you cannot do your presentation without your presentation, then you might want to rethink your appearance. Nothing cooperated with her, yet she soldiered on and won the support of the audience, sans slides. It may have been better without slides. The most valuable advice I took from her feedback is while certain products may lend themselves to white-labeling, sometimes a strong brand is more valuable to the business, and should remain in tact through all markets.

Ryan Meinzer of PlaySay asked for advice, mainly on a strategy to deal with potential threat from the very large players in markets adjacent to his. The consensus among the panel seemed to (to me anyway) be that when you can profitably serve multiple smaller niches with a predictable formula, you don't need to be overly concerned about the Walmarts of your space. I appreciate the candor of an entrepreneur who tells you that a business "landed in his lap" and he ran with it. Also, there is dogfooding going on here where PlaySay evolved out of his own need to learn Japanese. There was discussion of patents, and all I could hear was Paul Graham talking about how the value of patents for a startup really is "as an element of the mating dance with acquirers". I was hoping one of the panelists would say (warning, my $0.02 ahead) that should PlaySay's immediate strategy be acquisition by one of those neighboring 800 pound gorillas, then by all means invest in those patents, which can be line items in the inventory to sell an acquirer. Alternatively, focus resources on product development.

The three guys from RevZilla have built an impressive business. They bootstrapped heavily, are passionate about their products and the space, and were seeking advice on growth. I was (pleasantly) surprised to hear a panel of VCs suggest to these guys not take take external funding, and thereby introduce a whole new set of circumstances into their business, when they have such demonstrable growth. It seems that the powersports industry suffers from a common problem (is Ludditry a word?) which is that equipment distributors and manufacturers generally will not sell to Internet-only retailers. I found this in the golf equipment industry too, when I looked into it. The Revzilla solution is an impressive looking store front/warehouse in South Philly. In general, it seemed that the advice to RevZilla was that the curve they are on seems very promising, so ride (oh no a motorcycle pun) it as 3 founders to a great life, rather than try to engineer it as 3 founders + VC money and complex priorities. This may be a serious over simplification, of course.

During the course of the day, the following diagram began to form in my mind:

At every conference I have been to, the best speakers are the ones who provide a few select nuggets of advice, for problems they assume the audience has. If, as a speaker, you need to turn around and read a bullet point off your own PowerPoint slide, then you may as well be part of the audience yourself. Also, if you are simply detailing the a story about something you have done, then a book or a blog post is a better format, since you hold a different type of attention from the audience when up on stage. It's generally not a very patient type of attention, either. Unscripted-ness, or even the illusion of it, goes a long way in winning attention. Speakers are getting better and better at this, but we are still not there. If you've ever delivered a presentation which included more than 6 PowerPoint slides, each with more than 2 bullet points per page (that group includes me, by the way), I can't recommend an evening at The Moth enough.

A big thank you to Philly Startup Leaders for doing this important work in the Philly area. I personally enjoyed the day, and took home some interesting advice and thoughts to write rambling blog posts about.

WP Memcached Manager

18 Nov 2009

If you use Memcached and WordPress you may find this little plugin handy. Version 0.1 of wp-memcached-manager allows you to test your connection to a Memcached server, see some basic Memcached server stats and perform a cache flush. There is also a very rudimentary tool to see some of the cached data stored, however, it is super basic and not massively useful. In future releases you will be able to walk through data and perform updates on specific keys etc.

Plugin home: http://warwickp.com/projects/wp-memcached-manager/

On WordPress.org repository: http://wordpress.org/extend/plugins/wp-memcached-manager/

Getting to know our Amazon Kindle

12 Nov 2009

I'm always sneaking cool gadgets in as birthday gifts to my wife. "Happy birthday, honey, here is your electronic grill scrubber." But this time she actually asked for a Kindle, so I didn't need to be too sneaky. The idea is that she can read while trying to hold junior. She reads a lot, so theoretically this thing should pay for itself through the lower book costs. Right?

First thoughts, as I unwrap this thing.

• So Amazon hired someone from the Apple packaging team then? Nicely done.
• I thought it had a paper screen covering on in the box. Turns out that's the screen. Amazing display. Looks like paper. Amazing! I said that.
• There is significant lag on most operations, even typing on the keypad. The screen refresh rate is low. I guess that's the very idea of this device.
• The keypad is a little finicky (used that word in two consecutive posts. Need a prize). Particularly the enter/return key which needs to be bigger IMHO.
• There is a browser, and most pages render like it's 1995 and you just fired up Mosaic. Nostalgia.
• The Amazon wireless network is called Whispernet. I immediately thought of Blue Thunder.
• The first book I wanted to buy is Atlas Shrugged by Ayn Rand. Doesn't appear to be in the Kindle Store.
• How about Neil Stephenson's Anathem? Found it. OK, let that be my first Kindle purchase then.
• OK, I can subscribe to TechCrunch. Oh, wait, that will cost me $1.99/month? Arrington, you'll stay on my Mac.
• Great, there is a blog called "Hey, It's Free!" so I'll subscribe to that, whatever it is. Oh, Hey It's Free costs $0.99/month? Hmm, OK.

The Kindle User Agent string is:

"Mozilla/4.0 (compatible; Linux 2.6.22) NetFront/3.4 Kindle/2.1 (screen 600x800)"

The main challenge this Kindle is going to face is drool and repeated punching by my son, which will no doubt occur while my wife is trying to read "Patient Parenting Tips" on it. I'll let you know how it holds up.