Paul's Wibblings

Puppet Camp 2009

Sun, 04 Oct 2009 01:17:07 GMT

I have to say I've really enjoyed Puppet Camp. It's a reinvigorating experience, despite the jetlag, to meet up with smart, passionate and engaged operations people from all walks of life. My highlight really has been the people, and that's what I love about smaller unconferences is that you can meet everyone and everyone has something to contribute. Agile 2009 was great to go to, but I personally find I get more from smaller events.

I was happy to see the wide range of companies both in terms of size and domains who are using puppet and thinking about how to improve operations and systems engineering. San Francisco is an awesome city, and I've enjoyed finding some little havens of coffee drinking bliss, good food, fantastic cocktails and shopping heaven in my limited explorations so far.

Unfortunately I was a little nervous during my talk, which I'm pretty sure comes down to tiredness. I need to write up in more depth. For now here are my slides, video should be up shortly - I was really impressed by time to put up initial video by SFSU, and then them cleaning up the audio for the re-push.

Get Your Facts Right

View more presentations from pnasrat.

Agile Operations - Thoughts on Operations Testing

Wed, 19 Aug 2009 20:04:20 GMT

When we talk about agile systems practices the subject of testing often comes up. In the development world practices such as Test Driven Development and Behaviour Driven Development provide a supporting practice to enable iterative development, make change safe (enabling refactoring), and provide a way to do just enough to deliver what is asked for.

It's important to look at why developers test in order to think about what the enabling practices in systems and operations is. For those not familiar with test-driven development Nat and Steve have a great introduction in their forthcoming book: http://www.mockobjects.com/book/tdd-introduction.html

I think about tests for systems administration in terms of what they provide both in terms of quality and confidence. Coming up with the right language to describe testing for operations is hard.

I've seen a lot of development teams internally struggle with the naming of what Nat and Steve call Integration Tests and Acceptance Tests (Functional Tests, Customer Tests, System Tests). If we agree with the premise that testing types a certain level is desirable for operations teams then we should focus on what we achieve by that and a first step approach. It's also important to have a clear language that everyone can understand what you mean - so if your team is using Integration Tests to mean something specific, then don't use that language to mean something different in operational testing.

It's important to think about what the different test levels give you and how that maps to operations/systems engineering. Also we're hitting the fact you illustrated that testing and monitoring are conflated. For development internal quality of code driven by unit tests enables ease of understanding and ease of change. If we want to have a practice in operations that makes our intent clear and enables ease of change this is not monitoring.

Agile Systems Administration Slides from Manchester Geek Night

Fri, 07 Aug 2009 13:15:15 GMT

I spoke in Manchester last night, it was nice to be back in the city I spent five years failing to get a degree in. It's the place I first started working in systems at the Faculty of Arts helpdesk, first exposure to gopher, lynx, usenet and the web, and where I installed my first Linux systems.

The venue was the Kilburn building and the area outside the room had displays of Baby which was nice.

The group was small and mostly dominated by sytems people, I felt it went better than my previous talk on this at skills matter - I was more focussed and I think the presentation was more polished. Following we went on to the SandBar for good beer and further discussion.

Thanks to ThoughtWorks for inviting me up and putting me up in Manchester.

Agile Systems Administration Geek Night

View more presentations from pnasrat.

Talking in Manchester

Sat, 01 Aug 2009 10:25:40 GMT

I'm giving a talk about Agile Systems Administration in Manchester on Thursday August 6th.

http://manchestergeeknights.wetpaint.com/page/Agile+Systems+Administration

If you're in the area and interested in agile practices, developer and operations relationships, automated infrastructure and continual deployment you can sign up on upcoming.

Event: London Talk and Discussion on Agile Systems

Tue, 14 Apr 2009 15:37:32 GMT

I'm giving a talk and discussion on Agile Systems Administration and Infrastructure at Skills Matter on April 23rd.

http://skillsmatter.com/event/agile-scrum/agile-pragmatic-systems-administration

Fosdem 2009

Sat, 17 Jan 2009 10:36:08 GMT

I finally got organinsed and booked going to FOSDEM, I didn't get there last year and am looking forward to it. Hopefully get a chance to catch up with some people.

A new entry into the state driven config management field

Fri, 16 Jan 2009 10:02:17 GMT

Whilst reading my daily blog roll I noticed that Ezra posted about a new configuration management tool chef and fact gatherer project ohai from the guys as opscode.

There is more details on the wiki, and I'm interested in finding out design differences to puppet and facter. An initial look at ohai looks good - JSON output of facts leads to clearer ability to collect nested facts than namespacing ipaddress_eth0.

chef looks to use ruby rather than an external dsl for it's cookbooks (recipes).

Puppet gets Continuous Integration

Mon, 01 Dec 2008 20:45:09 GMT

Whilst I was caught up with the day job (long, long commutes ...) Puppet seems to have got itself a CI setup:

http://reductivelabs.com/trac/puppet/wiki/PuppetContinuousIntegration

If you have a weird and wonderful os you want hosted in it for functional/integration testing follow the instructions.

puppetdoc goodness

Thu, 20 Nov 2008 13:46:39 GMT

More puppet goodenss from masterzen in the form of improved puppetdoc allowing inline manifest documentation:

http://reductivelabs.com/trac/puppet/wiki/PuppetManifestDocumentation

Sample output here:

http://www.masterzen.fr/puppet/rdoc/index.html

puppetdoc goodness

Thu, 20 Nov 2008 10:21:05 GMT

More puppet goodenss from masterzen in the form of improved puppetdoc:

http://reductivelabs.com/trac/puppet/wiki/PuppetManifestDocumentation

Scribe

Fri, 07 Nov 2008 18:05:48 GMT

I've just come across Facebook's open sourced logging aggregator scribe, which looks interesting:

http://www.facebook.com/notes.php?id=9445547199#/note.php?note_id=32008268919&id=9445547199&index=0
http://developers.facebook.com/scribe/

Capturing the intent of system changes

Tue, 04 Nov 2008 21:05:24 GMT

There has been some discussion on the puppet lists about potential language shortcomings, something lak said echoed with some thoughts I've been having recently:

No one starts with code, even when they think they do -- you always
start with intent. Why are you building this node? What services
should it offer? Why?

In Test Driven/Behaviour Driven Development projects I see intent and goal extensively within the language of the tests.

Time and again when I see setups with systems in a state where their configuration is divergent and it's impossible to tell what is the desired state (if you haven't heard of puppet - check it out) and why the change was needed.

Take for an example an organisation using NFS or Samba as a backend store for documents for a distributed workflow. Tweaks to the config (kernel parameters, mount options, config changes) may be apparent from the system or puppet, but it may not be clear the underlying value that you're trying to achieve. It may be throughput, reliability, access - systems changes are usually made for a specific reason - I'd like to be able to more easily capture that (preferably in the format of executable tests to validate) but make it easy for Jo, the Sysadmin, to easily do as change is driven out either in reaction to a situation or as part of a planned change.

Puppet 0.24.6

Sat, 01 Nov 2008 11:08:45 GMT

I'm really excited to get back from holiday and see the release of puppet 0.24.6.

http://groups.google.com/group/puppet-users/browse_thread/thread/d3752a90ef20323f#

Some of the new language features from Brice are really useful and should allow for much richer expression in manifests.

I'm hoping to get back to working on the win32 port, and looking at improving facter again.

Systems Administration must die, die, die

Sun, 01 Jun 2008 14:46:54 GMT

I really think that the term "Systems Administration" needs to go away. Use Systems Integrator, Systems Engineer (or Systems Reliability Engineer if you're Google) or some other term that more accurately represents the role.

I know there has been a lot of discussion about DSL's and Polyglot Programming around (eg http://ola-bini.blogspot.com/2008/05/new-hope-polyglotism.html), but I know that good Systems Engineers regularly context switch between different DSLs (config file syntax, puppet) languages (shell, batch, sed, awk, perl, python, ruby, powershell).

From my own experience I might be debugging a core dumping application using gdb, then swapping puppet update something, followed by a dash of shell, a sprinkling of graphing in gnuplot (or gruff and ruby), then some Java. I don't think the word administration even comes close to explaining that.

I don't hear the term "Operator" in common usage for Systems Engineer, and I'd like to see Administrator go the same way. If you're recruiting for people think about the fact that words are powerful, think about the role and what it involves, then see if you feel that Administrator is the word you'd choose to fit that.

If you don't know what your friendly neighbourhood Systems Integrator does, ask her if she would mind you shadowing for a day, if you're developing you probably will learn a lot about how to write applications that are supportable just from understanding the pain points of debugging with just a log file!

Thoughtworks Geek Night - Mocking 28th May

Tue, 20 May 2008 05:49:27 GMT

May 28th will be a Mocking focussed Geek Night at the ThoughtWorks office:

Mocking can make code more reliable, more comprehensible and allow you
to spot problems in design long before your code is deployed. Speakers
will include Steve Freeman and Nat Pryce, developers of JMock and
Felix Leipold of ThoughtWorks.

Time permitting there will also be a mocking dojo to allow you to get

See:

http://londongeeknights.wetpaint.com
http://upcoming.yahoo.com/event/498162/

Shell history meme

Sat, 12 Apr 2008 17:36:12 GMT

history|awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -rn|head
96 ls
47 vi
37 sudo
35 cd
30 make
21 vim
17 svn
16 curl
15 gdb
12 grep

OS X and dtrace

Wed, 12 Mar 2008 08:19:59 GMT

Played a bit with trying to get mod_trace the dtrace module for apache working with Leopard yesterday evening.

gcc -o foo -arch x86_64 foo.c
/var/folders/rV/rV1x2DafFr0R6tGG+1bbk++++TM/-Tmp-//ccnykQ1o.s:11:bad
register name `%%esi)'

I didn't have time to fully investigate so fired off a reproducer to dtrace-discuss. It turns out that DTRACE_PROBE, DTRACE_PROBE1 and friends are not supported on OS X despite being exported. A little further investigation this morning showed that the root cause is actually a bad define in /usr/include/mach/i386/sdt_isa.h:


#ifdef __x86_64__

#define DTRACE_NOPS                     \
       "nop"                   "\n\t"  \
       "leal 0(%%esi), %%esi"  "\n\t"

So rather than use the DTRACE_PROBE# family the correct way to do this on OS X is to use the generated macros (and probe tests) from dtrace -h:

dtrace -h -arch x86_64 -s foo.d
cat foo.d
provider testDtraceProbe {
    probe probe__noargs();
};

Then include the generated foo.h in your and use the generated probe, eg: TESTDTRACEPROBE_PROBE_NOARGS. A good example of this is the patch for dtrace on postgresql for OS X from Lee Packham.

Hopefully tonight I can get a working mod_trace on OS X and write some dtrace scripts for apache and in the future maybe be able to look at uprobes systemtap on Linux as well. It looks as if systemtap user space probes are progressing

Tool of the day

Tue, 19 Feb 2008 17:45:40 GMT

I was looking for something to simulate delay and latency on linux using netfilter or iptables and discovered Netem and the wonderful lagfactory script that did just what I wanted.

Puppet and ctags

Mon, 11 Feb 2008 21:06:53 GMT

Whilst finding my way around puppet, I realised that some of the tools I'm used to when developing in other languages weren't quite there.

A quick read through the ctags documentation and I knocked up a couple of simple regular expressions that enable me to quick navigate through a large, split out puppet configuration using vim. Adding the following to ~/.ctags will enable you to run ctags -R at the top of your puppet manifests and navigate through tags. I'll probably need to actually go through the language definition and ctags docs some more to provide more comprehensive functionality but this works for me right now so I wanted to share it.

--langdef=puppet
--langmap=puppet:.pp
--regex-puppet=/^class[ \t]*([:a-zA-Z0-9_\-]+)[ \t]*/\1/d,definition/
--regex-puppet=/^site[ \t]*([a-zA-Z0-9_\-]+)[ \t]*/\1/d,definition/
--regex-puppet=/^node[ \t]*([a-zA-Z0-9_\-]+)[ \t]*/\1/d,definition/
--regex-puppet=/^define[ \t]*([:a-zA-Z0-9_\-]+)[ \t]*/\1/d,definition/

Virtualisation libraries

Mon, 04 Feb 2008 10:11:16 GMT

Whilst looking into the progress of libvirt and use with VMWare and OS X, I discovered ivi which is a java library around VMWare, Xen, KVM and OpenVZ. Hopefully VMWare in libvirt won't be far off.

Yaboot update

Fri, 04 Jan 2008 13:33:40 GMT

It's taken me some time since I moved in to the new place to setup my test environment again, it's not ideal as I don't have as much ppc hardware as I used to but it's a start. The morning was spent getting dhcp setup, and ensuring neboot worked end to end with the iBook. Once that was done I could test yaboot git HEAD using netboot.

Some of the recent features are really useful, I love being able to netboot to test the netboot code, then do


device=hd partition=2 filename=yaboot.conf

To boot off the hard disk. The use of the initrd= to load an alternate initrd from the command line is an improvement I've wanted for a long time and is great. It's also a good start to adding multiple initramfs images.

There is also the bootonce feature which I've not really played with in anger, as I tend to do most of my yaboot testing via netboot, which is even easier now we support larger images.

yaboot

Fri, 17 Aug 2007 15:36:14 GMT

yaboot 1.3.14 is released

http://yaboot.ozlabs.org/

Once my Fedora account is sorted I'll update there.

Many nice things for enterprise ppc users - bootonce, pSeries netbooting, user confs, pxelinux style netbooting.

Time for a change

Fri, 17 Aug 2007 14:32:35 GMT

After much consideration - I've decided to pursue other opportunities outside of Red Hat. My last day is today.

It's been a tough call, Red Hat have been a fantastic employer but it's time for me to do something different with my days.

I'm still looking forward to being active in Fedora, RPM, yaboot and other communities in my free time.

RPM 4.4.2.1

Mon, 23 Jul 2007 13:52:33 GMT

As Panu discusses and the announcement details. rpm.org has released rpm-4.4.2.1 maintenance release.

The hg branch for 4.4.x is http://hg.rpm.org/rpm-4.4.x
The hg branch for HEAD and new development is http://hg.rpm.org/

Mailing list for discussion is here https://lists.rpm.org/mailman/listinfo/rpm-maint

Fedora users should see it hitting rawhide shortly.

Perl/Python Module Finding

Fri, 09 Feb 2007 10:21:42 GMT

I had a look at Peter Gordon's python module finding code to work with rpmbuild.

I'd rather something use the modulefinder module.

Whilst we're at it'd also be nice if we could use Module::ScanDeps for the perl module finding. We really shouldn't be writing our own language specific dependency generators where good ones already exist.