Roll out API Umbrella v0.7 to servers

[GUI]

We upgraded to API Umbrella v0.7 last night, which fixed some important edge-cases around DNS handling and analytics gathering. I just wanted to document the fact that this upgrade happened here and also document the steps taken on the servers for the upgrade (in case it helps with future upgrades--although, there are pieces of this upgrade process and how we manage these servers that could probably be streamlined):

• Update Chef cookbook on each server: cd /opt/chef/solo/api-umbrella && git pull && rm -rf /opt/chef/solo/cookbooks && /opt/chef/embedded/bin/berks vendor /opt/chef/solo/cookbooks
• Upgrade DB servers first, one at a time.
  • Ensure there are at least 3 members of the replicaset: mongo --eval "printjson(rs.status())"
  • Operate on the mongo secondary nodes first.
  • Stop API Umbrella: /etc/init.d/api-umbrella stop
  • Run chef on secondaries: chef-solo -c /etc/chef/solo.rb -j /etc/chef/solo.json
  • Force a mongo secondary to be primary (use the priority based approach to prevent downtime).
  • After primary is switched over, stop API Umbrella on old primary and run chef on that server.
• Upgrade web servers
  • Remove one server from all the ELBs. Wait a couple minutes to be sure traffic stops to that server.
  • Stop API Umbrella: /etc/init.d/api-umbrella stop
  • Run chef: chef-solo -c /etc/chef/solo.rb -j /etc/chef/solo.json
  • Verify that a localhost connection works: curl -v "http://localhost/nrel/alt-fuel-stations/v1.json?limit=1&api_key=DEMO_KEY"
  • Add server back to ELBs. Wait a couple minutes for it to start seeing traffic.
  • Repeat for each web server.

[GUI]

Oh, and just an extra bonus of rolling out this release: we've seen a pretty good improvement in our average CPU usage and system load on our servers after rolling out this update (I think it's related to the changes and improvements in how we gather log data):

Here are some charts of load and CPU use before and after the flip at approximately midnight:

At our average system state, the CPU use has gone down from around 15% to less than 5%. Hooray for efficiency!