Paul's Wibblings

Event: London Talk and Discussion on Agile Systems

2009-04-14T15:37:32Z

I'm giving a talk and discussion on Agile Systems Administration and Infrastructure at Skills Matter on April 23rd.

http://skillsmatter.com/event/agile-scrum/agile-pragmatic-systems-administration

Fosdem 2009

2009-01-17T10:36:08Z

I finally got organinsed and booked going to FOSDEM, I didn't get there last year and am looking forward to it. Hopefully get a chance to catch up with some people.

A new entry into the state driven config management field

2009-01-16T10:02:17Z

Whilst reading my daily blog roll I noticed that Ezra posted about a new configuration management tool chef and fact gatherer project ohai from the guys as opscode.

There is more details on the wiki, and I'm interested in finding out design differences to puppet and facter. An initial look at ohai looks good - JSON output of facts leads to clearer ability to collect nested facts than namespacing ipaddress_eth0.

chef looks to use ruby rather than an external dsl for it's cookbooks (recipes).

Puppet gets Continuous Integration

2008-12-01T20:45:09Z

Whilst I was caught up with the day job (long, long commutes ...) Puppet seems to have got itself a CI setup:

http://reductivelabs.com/trac/puppet/wiki/PuppetContinuousIntegration

If you have a weird and wonderful os you want hosted in it for functional/integration testing follow the instructions.

puppetdoc goodness

2008-11-20T13:46:39Z

More puppet goodenss from masterzen in the form of improved puppetdoc allowing inline manifest documentation:

http://reductivelabs.com/trac/puppet/wiki/PuppetManifestDocumentation

Sample output here:

http://www.masterzen.fr/puppet/rdoc/index.html

puppetdoc goodness

2008-11-20T10:21:05Z

More puppet goodenss from masterzen in the form of improved puppetdoc:

http://reductivelabs.com/trac/puppet/wiki/PuppetManifestDocumentation

Scribe

2008-11-07T18:05:48Z

I've just come across Facebook's open sourced logging aggregator scribe, which looks interesting:

http://www.facebook.com/notes.php?id=9445547199#/note.php?note_id=32008268919&id=9445547199&index=0
http://developers.facebook.com/scribe/

Capturing the intent of system changes

2008-11-04T21:05:24Z

There has been some discussion on the puppet lists about potential language shortcomings, something lak said echoed with some thoughts I've been having recently:

No one starts with code, even when they think they do -- you always
start with intent. Why are you building this node? What services
should it offer? Why?

In Test Driven/Behaviour Driven Development projects I see intent and goal extensively within the language of the tests.

Time and again when I see setups with systems in a state where their configuration is divergent and it's impossible to tell what is the desired state (if you haven't heard of puppet - check it out) and why the change was needed.

Take for an example an organisation using NFS or Samba as a backend store for documents for a distributed workflow. Tweaks to the config (kernel parameters, mount options, config changes) may be apparent from the system or puppet, but it may not be clear the underlying value that you're trying to achieve. It may be throughput, reliability, access - systems changes are usually made for a specific reason - I'd like to be able to more easily capture that (preferably in the format of executable tests to validate) but make it easy for Jo, the Sysadmin, to easily do as change is driven out either in reaction to a situation or as part of a planned change.

Puppet 0.24.6

2008-11-01T11:08:45Z

I'm really excited to get back from holiday and see the release of puppet 0.24.6.

http://groups.google.com/group/puppet-users/browse_thread/thread/d3752a90ef20323f#

Some of the new language features from Brice are really useful and should allow for much richer expression in manifests.

I'm hoping to get back to working on the win32 port, and looking at improving facter again.

Systems Administration must die, die, die

2008-06-01T14:46:54Z

I really think that the term "Systems Administration" needs to go away. Use Systems Integrator, Systems Engineer (or Systems Reliability Engineer if you're Google) or some other term that more accurately represents the role.

I know there has been a lot of discussion about DSL's and Polyglot Programming around (eg http://ola-bini.blogspot.com/2008/05/new-hope-polyglotism.html), but I know that good Systems Engineers regularly context switch between different DSLs (config file syntax, puppet) languages (shell, batch, sed, awk, perl, python, ruby, powershell).

From my own experience I might be debugging a core dumping application using gdb, then swapping puppet update something, followed by a dash of shell, a sprinkling of graphing in gnuplot (or gruff and ruby), then some Java. I don't think the word administration even comes close to explaining that.

I don't hear the term "Operator" in common usage for Systems Engineer, and I'd like to see Administrator go the same way. If you're recruiting for people think about the fact that words are powerful, think about the role and what it involves, then see if you feel that Administrator is the word you'd choose to fit that.

If you don't know what your friendly neighbourhood Systems Integrator does, ask her if she would mind you shadowing for a day, if you're developing you probably will learn a lot about how to write applications that are supportable just from understanding the pain points of debugging with just a log file!

Thoughtworks Geek Night - Mocking 28th May

2008-05-20T05:49:27Z

May 28th will be a Mocking focussed Geek Night at the ThoughtWorks office:

Mocking can make code more reliable, more comprehensible and allow you
to spot problems in design long before your code is deployed. Speakers
will include Steve Freeman and Nat Pryce, developers of JMock and
Felix Leipold of ThoughtWorks.

Time permitting there will also be a mocking dojo to allow you to get

See:

http://londongeeknights.wetpaint.com
http://upcoming.yahoo.com/event/498162/

Shell history meme

2008-04-12T17:36:12Z

history|awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -rn|head
96 ls
47 vi
37 sudo
35 cd
30 make
21 vim
17 svn
16 curl
15 gdb
12 grep

OS X and dtrace

2008-03-12T08:19:59Z

Played a bit with trying to get mod_trace the dtrace module for apache working with Leopard yesterday evening.

gcc -o foo -arch x86_64 foo.c
/var/folders/rV/rV1x2DafFr0R6tGG+1bbk++++TM/-Tmp-//ccnykQ1o.s:11:bad
register name `%%esi)'

I didn't have time to fully investigate so fired off a reproducer to dtrace-discuss. It turns out that DTRACE_PROBE, DTRACE_PROBE1 and friends are not supported on OS X despite being exported. A little further investigation this morning showed that the root cause is actually a bad define in /usr/include/mach/i386/sdt_isa.h:


#ifdef __x86_64__

#define DTRACE_NOPS                     \
       "nop"                   "\n\t"  \
       "leal 0(%%esi), %%esi"  "\n\t"

So rather than use the DTRACE_PROBE# family the correct way to do this on OS X is to use the generated macros (and probe tests) from dtrace -h:

dtrace -h -arch x86_64 -s foo.d
cat foo.d
provider testDtraceProbe {
    probe probe__noargs();
};

Then include the generated foo.h in your and use the generated probe, eg: TESTDTRACEPROBE_PROBE_NOARGS. A good example of this is the patch for dtrace on postgresql for OS X from Lee Packham.

Hopefully tonight I can get a working mod_trace on OS X and write some dtrace scripts for apache and in the future maybe be able to look at uprobes systemtap on Linux as well. It looks as if systemtap user space probes are progressing

Tool of the day

2008-02-19T17:45:40Z

I was looking for something to simulate delay and latency on linux using netfilter or iptables and discovered Netem and the wonderful lagfactory script that did just what I wanted.

Puppet and ctags

2008-02-11T21:06:53Z

Whilst finding my way around puppet, I realised that some of the tools I'm used to when developing in other languages weren't quite there.

A quick read through the ctags documentation and I knocked up a couple of simple regular expressions that enable me to quick navigate through a large, split out puppet configuration using vim. Adding the following to ~/.ctags will enable you to run ctags -R at the top of your puppet manifests and navigate through tags. I'll probably need to actually go through the language definition and ctags docs some more to provide more comprehensive functionality but this works for me right now so I wanted to share it.

--langdef=puppet
--langmap=puppet:.pp
--regex-puppet=/^class[ \t]*([:a-zA-Z0-9_\-]+)[ \t]*/\1/d,definition/
--regex-puppet=/^site[ \t]*([a-zA-Z0-9_\-]+)[ \t]*/\1/d,definition/
--regex-puppet=/^node[ \t]*([a-zA-Z0-9_\-]+)[ \t]*/\1/d,definition/
--regex-puppet=/^define[ \t]*([:a-zA-Z0-9_\-]+)[ \t]*/\1/d,definition/

Virtualisation libraries

2008-02-04T10:11:16Z

Whilst looking into the progress of libvirt and use with VMWare and OS X, I discovered ivi which is a java library around VMWare, Xen, KVM and OpenVZ. Hopefully VMWare in libvirt won't be far off.

Yaboot update

2008-01-04T13:33:40Z

It's taken me some time since I moved in to the new place to setup my test environment again, it's not ideal as I don't have as much ppc hardware as I used to but it's a start. The morning was spent getting dhcp setup, and ensuring neboot worked end to end with the iBook. Once that was done I could test yaboot git HEAD using netboot.

Some of the recent features are really useful, I love being able to netboot to test the netboot code, then do


device=hd partition=2 filename=yaboot.conf

To boot off the hard disk. The use of the initrd= to load an alternate initrd from the command line is an improvement I've wanted for a long time and is great. It's also a good start to adding multiple initramfs images.

There is also the bootonce feature which I've not really played with in anger, as I tend to do most of my yaboot testing via netboot, which is even easier now we support larger images.

yaboot

2007-08-17T15:36:14Z

yaboot 1.3.14 is released

http://yaboot.ozlabs.org/

Once my Fedora account is sorted I'll update there.

Many nice things for enterprise ppc users - bootonce, pSeries netbooting, user confs, pxelinux style netbooting.

Time for a change

2007-08-17T14:32:35Z

After much consideration - I've decided to pursue other opportunities outside of Red Hat. My last day is today.

It's been a tough call, Red Hat have been a fantastic employer but it's time for me to do something different with my days.

I'm still looking forward to being active in Fedora, RPM, yaboot and other communities in my free time.

RPM 4.4.2.1

2007-07-23T13:52:33Z

As Panu discusses and the announcement details. rpm.org has released rpm-4.4.2.1 maintenance release.

The hg branch for 4.4.x is http://hg.rpm.org/rpm-4.4.x
The hg branch for HEAD and new development is http://hg.rpm.org/

Mailing list for discussion is here https://lists.rpm.org/mailman/listinfo/rpm-maint

Fedora users should see it hitting rawhide shortly.

Perl/Python Module Finding

2007-02-09T10:21:42Z

I had a look at Peter Gordon's python module finding code to work with rpmbuild.

I'd rather something use the modulefinder module.

Whilst we're at it'd also be nice if we could use Module::ScanDeps for the perl module finding. We really shouldn't be writing our own language specific dependency generators where good ones already exist.

RPM developer position

2007-02-08T09:54:33Z

There's a job at Red Hat for someone who wants to work with RPM, we want to build a bigger team around RPM and this should help

http://redhat.hrdpt.com/cgi-bin/a/highlightjob.cgi?jobid=1949

If you're interested drop me a line to let me know as well as applying online.

Off on holiday

2006-05-31T10:43:17Z

Just to let people know I'm away on holiday over the next week and a half and will be out of contact as I'll not be taking a computer and will be enjoying hiking, camping, etc.

pysqlite memory fun

2006-02-01T19:57:55Z

Today got put on hold a bit due to the yum eating all the memory on the box issues on rawhide. I thought I'd share how I tracked it down. I noticed the issue doing a test install on an IBM OpenPower LPAR. Thinking it was the lpar, I boosted the memory - but noted that I got OOM killer regardless of memory. In car hypothesis - probably sqlite upgrade, confirmed on getting to office on cube mates machine, whilst mine booted up. Start tracking down.

Let's see what changed:

diff -ur sqlite-3.3.2 sqlite-3.3.3 | diffstat
 62 files changed, 663 insertions(+), 279 deletions(-)

OK, great - now if they were using git I could use git bisect I think. I look what source control they are using and discover it's CVS. A little bit of poking reveals they use trac so I can get a timeline: http://www.sqlite.org/cvstrac/timeline?d=30&e=2006-Feb-01&c=2&px=&s=3&dm=1&x=1&m=1 From that I see that changeset 3020 to 3045 are relevant. I see that I can individually pull them without using CVS:

for patch in $(seq 3020 3045); do curl -o ${patch}.patch http://www.sqlite.org/cvstrac/patchset?cn=$patch ; done

All well and good, a quick test shows that they are CVS patch, never mind patchutils comes to the rescue with fixcvsdiff. Now I have a starting tarball, a whole bunch of patches. I set up a local git (cg-init) tree of 3.3.2 and ct-tag it, then I loop over and apply each patches (ignoring some rejects I either don't care about or were applied already) and cg-commit with a comment on which patch number it came from. The behaviour only exhibits itself in python using pysqlite, so something changed that broke table creation. I create a simple test case.

Now I whip out git bisect - it's the first time I've used it and it along with the test case really sped up the process. A few bisections later and I'm at the patch where the issue exhibits itself #3040. Time to look it up, http://www.sqlite.org/cvstrac/tktview?tn=1554. OK so now I can head over to pysqlite source and see what it does with sqlite3_prepare.

rc = sqlite3_prepare(db, tail, 0, &statement, &tail);

That really doesn't look like a good plan, but never bit us before as it was unimplemented. Checking the API docs I decide to just use a negative value. Test it out and all seems good.

I'm very grateful to have such tools as git bisect and patchutils at my disposal, and certainly for my first foray with git in anger it really worked well. I'm also grateful that sqlite had something that I could create patchsets from easily, else I'd have had to do a lot more by hand.

RPM Guide Available

2005-11-03T22:22:50Z

I'm very pleased to announce that the Red Hat RPM Guide has been made
available under the Open Publication Licence, Version 1.0.

http://fedora.redhat.com/docs/drafts/rpm-guide-en/

This is much more recent than Maximum RPM and gives a good starting
point to work towards living documentation for RPM. Although some
sections - notably Chapter 7 requires a complete rewrite. However I
wanted to get it out in it's current form (basically as published plus
errata).

Participation through normal Fedora Documentation process:

http://fedora.redhat.com/projects/docs/

I'll be attempting to form a team to help edit/update content. If you
want to help please follow up to fedora-docs-list.