The Shining Path of Least Resistance

LeastResistance.Net

Posts Tagged ‘spiceweasel’

Spiceweasel 2.0

Posted by mattray on January 14, 2013

One year after the release of Spiceweasel 1.0, Spiceweasel 2.0 is now available! For those of you unfamiliar with Spiceweasel, it is a command-line tool that helps manage loading Chef infrastructure from a manifest file. The JSON or YAML manifest is a simplified representation of the Chef repository contents that you may use to recreate and share how to build the application or infrastructure. This file is validated to ensure that all the components listed are present and the correct versions are available, and can (and should) be managed in version control with the rest of your repository.

There are minor updates to the manifest syntax in 2.0, with a focus on clarity. Spiceweasel has 3 major new features since the 1.0 release :

Execute
Spiceweasel now has the ability to directly execute the knife commands, creating (or deleting or rebuilding) the infrastructure described in the manifest when you use the -e/--execute flag.

Extract
Rather than write your manifest from scratch, Spiceweasel may generate the knife commands or manifests for you. Running spiceweasel --extractlocal generates the knife commands required to upload all the existing cookbooks, roles, environments and data bags found in your Chef repository with validation. spiceweasel --extractjson or spiceweasel --extractyaml will generate the JSON or YAML manifest for your Chef repository, which may be captured and then tracked in version control.

Clusters
Clusters are not a type supported by Chef, this is a logical construct added by Spiceweasel to enable managing sets of infrastructure together. The clusters section is a special case of nodes, where each member of the named cluster in the manifest will be tagged to ensure that the entire cluster may be created in sync (refresh and delete coming soon). The node syntax is the same as that under nodes, the only addition is the cluster name.

clusters:
- mycloud:
  - openstack 1:
      run_list: role[db]
      options: -S mray -i ~/.ssh/local.pem -Ia0b27dd9-e008-4a13-8cda-a2c74a28b7b1 -f 3
  - openstack 2:
      run_list: role[web] recipe[mysql::client]
      options: -S mray -i ~/.ssh/local.pem -Ia0b27dd9-e008-4a13-8cda-a2c74a28b7b1 -f 2
- amazon:
  - ec2 1:
      run_list: role[db]
      options: -S mray -i ~/.ssh/mray.pem -x ubuntu -I ami-8af0f326 -f m1.medium
  - ec2 2:
      run_list: role[web] recipe[mysql::client]
      options: -S mray -i ~/.ssh/mray.pem -x ubuntu -I ami-7000f019 -f m1.small

This would generate the following knife commands:
knife openstack server create -S mray -i ~/.ssh/local.pem -Ia0b27dd9-e008-4a13-8cda-a2c74a28b7b1 -f 3 -j '{"tags":["mycloud+roledb"]}' -r 'role[db]'
knife openstack server create -S mray -i ~/.ssh/local.pem -Ia0b27dd9-e008-4a13-8cda-a2c74a28b7b1 -f 2 -j '{"tags":["mycloud+rolewebrecipemysqlclient"]}' -r 'role[web],recipe[mysql::client]'
knife openstack server create -S mray -i ~/.ssh/local.pem -Ia0b27dd9-e008-4a13-8cda-a2c74a28b7b1 -f 2 -j '{"tags":["mycloud+rolewebrecipemysqlclient"]}' -r 'role[web],recipe[mysql::client]'
knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -I ami-8af0f326 -f m1.medium -j '{"tags":["amazon+roledb"]}' -r 'role[db]'
knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -I ami-7000f019 -f m1.small -j '{"tags":["amazon+rolewebrecipemysqlclient"]}' -r 'role[web],recipe[mysql::client]'
knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -I ami-7000f019 -f m1.small -j '{"tags":["amazon+rolewebrecipemysqlclient"]}' -r 'role[web],recipe[mysql::client]'

What’s important to note here is the use of tags to identify them, we’ll be coming back to this in future releases.

Roadmap
As always the code is available at https://github.com/mattray/spiceweasel with a detailed changelog. The code has been heavily refactored into discrete libraries and the next few releases should add additional cluster features and integration with Berkshelf and Librarian.

Feature suggestions and patches are always welcome, thanks for using Spiceweasel!

Posted in chef, opschef, spiceweasel | Tagged: , , , | 6 Comments »

Spiceweasel 1.0

Posted by mattray on January 3, 2012

One of the more useful things I’ve written since I’ve been at Opscode (over a year now) is a tool called Spiceweasel. Spiceweasel processes a simple YAML (or JSON) manifest that describes how to deploy Chef-managed infrastructure with the command-line tool knife. It is fairly simple, but it fills a useful niche by making it easy to document your infrastructure’s dependencies and how to deploy in a file that may be managed with version control. Spiceweasel also attempts to validate that the dependencies you list in your YAML manifest all exist in the repository and that all of their dependencies are included as well.

Examples

There is the https://github.com/mattray/ravel-repo which provides a working example for bootstrapping a Chef repository with Spiceweasel. The examples directory in GitHub is slowly getting more examples based on the Chef Quick Starts.

Given the example YAML file example.yml:

cookbooks:
- apache2:
- apt:
 - 1.2.0
- mysql:

environments:
- development:
- qa:
- production:

roles:
- base:
- iisserver:
- monitoring:
- webserver:

data bags:
- users:
 - alice
 - bob
 - chuck
- data:
 - *
- passwords:
 - secret secret_key
 - mysql
 - rabbitmq

nodes:
- serverA:
 - role[base]
 - -i ~/.ssh/mray.pem -x user --sudo -d ubuntu10.04-gems
- serverB serverC:
 - role[base]
 - -i ~/.ssh/mray.pem -x user --sudo -d ubuntu10.04-gems -E production
- ec2 4:
 - role[webserver] recipe[mysql::client]
 - -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small
- rackspace 3:
 - recipe[mysql],role[monitoring]
 - --image 49 --flavor 2
- windows_winrm winboxA:
 - role[base],role[iisserver]
 - -x Administrator -P 'super_secret_password'
- windows_ssh winboxB winboxC:
 - role[base],role[iisserver]
 - -x Administrator -P 'super_secret_password'

Spiceweasel generates the following knife commands:

knife cookbook upload apache2
knife cookbook upload apt
knife cookbook upload mysql
knife environment from file development.rb
knife environment from file qa.rb
knife environment from file production.rb
knife role from file base.rb
knife role from file iisserver.rb
knife role from file monitoring.rb
knife role from file webserver.rb
knife data bag create users
knife data bag from file users alice.json
knife data bag from file users bob.json
knife data bag from file users chuck.json
knife data bag create data
knife data bag create passwords
knife data bag from file passwords mysql.json --secret-file secret_key
knife data bag from file passwords rabbitmq.json --secret-file secret_key
knife bootstrap serverA -i ~/.ssh/mray.pem -x user --sudo -d ubuntu10.04-gems -r 'role[base]'
knife bootstrap serverB -i ~/.ssh/mray.pem -x user --sudo -d ubuntu10.04-gems -E production -r 'role[base]'
knife bootstrap serverC -i ~/.ssh/mray.pem -x user --sudo -d ubuntu10.04-gems -E production -r 'role[base]'
knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -r 'role[webserver],recipe[mysql::client]'
knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -r 'role[webserver],recipe[mysql::client]'
knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -r 'role[webserver],recipe[mysql::client]'
knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -r 'role[webserver],recipe[mysql::client]'
knife rackspace server create --image 49 --flavor 2 -r 'recipe[mysql],role[monitoring]'
knife rackspace server create --image 49 --flavor 2 -r 'recipe[mysql],role[monitoring]'
knife rackspace server create --image 49 --flavor 2 -r 'recipe[mysql],role[monitoring]'
knife bootstrap windows winrm winboxA -x Administrator -P 'super_secret_password' -r 'role[base],role[iisserver]'
knife bootstrap windows ssh winboxB -x Administrator -P 'super_secret_password' -r 'role[base],role[iisserver]'
knife bootstrap windows ssh winboxC -x Administrator -P 'super_secret_password' -r 'role[base],role[iisserver]'

Cookbooks

The `cookbooks` section of the manifest currently supports `knife cookbook upload FOO` where `FOO` is the name of the cookbook in the `cookbooks` directory. The default behavior is to download the cookbook as a tarball, untar it and remove the tarball. The `–siteinstall` option will allow for use of `knife cookbook site install` with the cookbook and the creation of a vendor branch if git is the underlying version control. If a version is passed, it is validated against the existing cookbook `metadata.rb` and it must match the `metadata.rb` string exactly. Validation is also done to ensure dependencies listed in the metadata for the cookbooks exists.

Environments

The `environments` section of the manifest currently supports `knife environment from file FOO` where `FOO` is the name of the environment file ending in `.rb` or `.json` in the `environments` directory. Validation is done to ensure the filename matches the environment and that any cookbooks referenced are listed in the manifest.

Roles

The `roles` section of the manifest currently supports `knife role from file FOO` where `FOO` is the name of the role file ending in `.rb` or `.json` in the `roles` directory. Validation is done to ensure the filename matches the role name and that any cookbooks or roles referenced are listed in the manifest.

Data Bags

The `data bags` section of the manifest currently creates the data bags listed with `knife data bag create FOO` where `FOO` is the name of the data bag. Individual items may be added to the data bag as part of a JSON or YAML sequence, the assumption is made that they `.json` files and in the proper `data_bags/FOO` directory. You may also pass a wildcard as an entry to load all matching data bags (ie. `*`). Encrypted data bags are supported by listing `secret filename` as the first item (where `filename` is the secret key to be used). Validation is done to ensure the JSON is properly formatted, the id matches and any secret keys are in the correct locations.

Nodes

The `nodes` section of the manifest bootstraps a node for each entry where the entry is a hostname or provider and count. A shortcut syntax for bulk-creating nodes with various providers where the line starts with the provider and ends with the number of nodes to be provisioned. Windows nodes need to specify either `windows_winrm` or `windows_ssh` depending on the protocol used, followed by the name of the node(s). Each node requires 2 items after it in a sequence. You may also use the `–parallel` flag from the command line, allowing provider commands to run simultaneously for faster deployment.

The first item after the node is the run_list and the second are the CLI options used. The run_list may be space or comma-delimited. Validation is performed on the run_list components to ensure that only cookbooks and roles listed in the manifest are used. Validation on the options ensures that any Environments referenced are also listed. You may specify multiple nodes to have the same configuration by listing them separated by a space.

Status and Roadmap

Spiceweasel has now hit version 1.0, it’s fairly complete and there are no open issues. I’ll continue to track changes in Chef and fix any issues that arise. If I start feeling ambitious (or someone sends patches), I may turn it into a knife plugin. I’ve also considered having Spiceweasel “extract” existing infrastructure, by parsing a list of nodes and documenting their cookbooks, environments, roles and run lists. I’ll write another post soon about using it in the “real world”.

Posted in chef, opschef, opscode, spiceweasel | Tagged: , , , | 1 Comment »