tag:blogger.com,1999:blog-78548922904378832162024-03-08T17:12:39.423+13:00How BazaarTim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.comBlogger55125tag:blogger.com,1999:blog-7854892290437883216.post-78965170018802382852016-04-27T09:36:00.000+12:002016-04-27T09:36:05.757+12:00It has been too longWell, it has certainly been a lot longer since I wrote a post than I thought.<br />
<br />
My work at Canonical still has me on the Juju team. Juju has come a long way in the last few years, and we are on the final push for the 2.0 version. This was initially intended to come out with the Xenial release, but unfortunately was not ready. Xenial has 2.0-beta4 right now, soon to be beta 6. Hoping that real soon now we'll step through the release candidates to a final release. This will be SRU'ed into both Xenial and Trusty.<br />
<br />
I plan to do some more detailed posts on some of the Go utility libraries that have come out of the Juju work. In particular, talking again about <a href="http://github.com/juju/loggo">loggo</a> which I moved under the "<a href="http://github.com/juju">github.com/juju</a>" banner, and the <a href="http://github.com/juju/errors">errors</a> package.<br />
<br />
Recent work has had me look at the database agnostic model representations for migrating models from one controller to another, and also at <a href="http://github.com/juju/gomaasapi">gomaasapi</a> - the Go library for talking with <a href="http://maas.io/">MAAS</a>. Perhaps more on that later.Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com0tag:blogger.com,1999:blog-7854892290437883216.post-8513566626044065742013-12-28T20:40:00.000+13:002013-12-28T20:40:29.711+13:002013 in review2013 started with what felt like a failure, but in the end, I believe that the<br />best decision was made. During 2011 and 2012 I worked on and then managed<br />the Unity desktop team. This was a C++ project that brought me back to my<br />hard-core hacker side after four and a half years on Launchpad. The Unity<br />desktop was a C++ project using glib, nux, and Compiz. After bringing Unity to<br />be the default desktop in 12.04 and ushering in the stability and performance<br />improvements, the decision was made to not use it as the way to bring the<br />Ubuntu convergence story forward. At the time I was very close tho the Unity 7<br />codebase and I had an enthusiastic capable team working on it. The decision<br />was to move forwards with a QML based user interface. I can see now that this<br />was the correct decision, and in fact I could see it back in January, but that<br />didn't make it any easier to swallow.<br /><br />I felt that I was at a juncture and I had to move on. Either I stayed with<br />Canonical and took another position or I found something else to do. I do like<br />the vision that Mark has for Ubuntu and the convergence story and I wanted to<br />hang around for it even if I wasn't going to actively work on the story itself. For a while I was interested in learning a new programming language, and Go was considered the new hotness, so I looked for a position working on Juju. I was lucky to be able to join the the juju-core team.<br /><br />After a two weak break in January to go to a family wedding, I came back to<br />work and started reading around Go. I started with the language specification<br />and then read around and started with the Go playground. Then started with the<br />Juju source.<br /><br />Go was a very interesting language to move to from C++ and Python. No<br />inheritance, no exceptions, no generics. I found this quite a change. I even<br />blogged about some of these frustrations.<br /><br />As much as I love the C++ language, it is a huge and complex language. One<br />where you are extremely lucky if you are working with other really competent<br />developers. C++ is the sort of language where you have a huge amount of power and control, but you pay other costs for that power and control. Most C++ code is pretty terrible.<br /><br />Go, as a contrast, is a much smaller, more compact, language. You can keep the<br />entire language specification in your head relatively easily. Some of this is<br />due to specific decisions to keep the language tight and small, and others I'm<br />sure are due to the language being young and immature. I still hope for<br />generics of some form to make it into the language because I feel that they<br />
are a core building block that is missing.<br /><br />I cut my teeth in Juju on small things. Refactoring here, tweaking<br />there. Moving on to more substantial changes. The biggest bit that leaps to<br />mind is working with Ian to bring LXC containers and the local provider to the<br />Go version of Juju. Other smaller things were adding much more infrastructure<br />around the help mechanism, adding plugin support, refactoring the provisioner,<br />extending the logging, and recently, adding KVM container support.<br /><br />Now for the obligatory 2014 predictions...<br /><br />I will continue working on the core Juju product bringing new and wonderful<br />features that will only be beneficial to that very small percentage of<br />developers in the world who actually deal with cloud deployments.<br /><br />Juju will gain more industry support outside just Canonical, and will be seen<br />as the easiest way to OpenStack clouds.<br /><br />I will become more proficient in Go, but will most likely still be complaining<br />about the lack of generics at the end of 2014.<br /><br />Ubuntu phone will ship. I'm guessing on more than just one device and with<br />more than one carrier. Now I do have to say that these are just personal<br />predictions and I have no more insight into the Ubuntu phone process than<br />anyone outside Canonical.<br /><br />The tablet form-factor will become more mature and all the core applications,<br />both those developed by Canonical and all the community contributed core<br />applications will support the form-factor switching on the fly.<br /><br />The Unity 8 desktop that will be based on the same codebase as the phone and<br />tablet will be available on the desktop, and will become the way that people<br />work with the new very high resolution laptops.<br />Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com0tag:blogger.com,1999:blog-7854892290437883216.post-8345692356408205082013-10-31T10:24:00.000+13:002013-10-31T10:24:04.460+13:00loggo - hierarchical loggers for GoSome readers of this blog will just think of me as that guy that complains about the Go language a lot. I complain because I care.<br />
<br />
I am working on the <a href="https://juju.ubuntu.com/">Juju</a> project. Juju is all about orchestration of cloud services. Getting workloads running on clouds, and making sure they communicate with other workloads that they need to communicate with. Juju currently works with <a href="http://aws.amazon.com/ec2/">Amazon EC2</a>, <a href="http://www.hpcloud.com/">HP Cloud</a>, <a href="http://www.windowsazure.com/en-us/">Microsoft Azure</a>, local <a href="http://linuxcontainers.org/">LXC</a> containers for testing, and Ubuntu's <a href="http://maas.ubuntu.com/">MAAS</a>. More cloud providers are in development. Juju is also written in <a href="http://golang.org/">Go</a>, so that was my entry point to the language.<br />
<br />
My background is from Python and C++. I have written several logging libraries in the past, but always in C++ and with reasonably specific performance characteristics. One thing I really felt was missing with the standard library in Go was a good logging library. Features that I felt were pretty necessary were:<br />
<ul>
<li>A hierarchy of loggers</li>
<li>Able to specify different logging levels for different loggers</li>
<li>Loggers inherited the level of their parent if not explicitly set</li>
<li>Multiple writers could be attached</li>
<li>Defaults should "just work" for most cases</li>
<li>Logging levels should be configurable easily</li>
<li>The user shouldn't have to care about synchronization</li>
</ul>
Initially this project was hosted on Launchpad. I am trialing moving the trunk of this branch to github. I have been quite isolated from the git world for some time, and this is my first foray in git, and specifically git and go. If I have done something wrong, please let me know.<br />
<br />
<h3>
Basics</h3>
There is an <a href="https://github.com/howbazaar/loggo/tree/master/example">example</a> directory which demonstrates using loggo (albeit relatively trivially). <br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">import "<a href="http://godoc.org/github.com/howbazaar/loggo">github.com/howbazaar/loggo</a>"</span><br />
<span style="font-family: "Courier New",Courier,monospace;">...</span><br />
<span style="font-family: "Courier New",Courier,monospace;">logger = loggo.<a href="http://godoc.org/github.com/howbazaar/loggo#GetLogger">GetLogger</a>("project.area")</span><br />
<span style="font-family: "Courier New",Courier,monospace;">logger.Debugf("This is debug output.")</span><br />
<span style="font-family: "Courier New",Courier,monospace;">logger.Warningf("Some error: %v", err) </span><br />
<br />
In juju, we normally create one logger for the module, and the dotted name normally reflects the module. This logger is then used by the other files in the module. Personally I would have preferred file local variables, but Go doesn't support that, not where they are private to the file, and as a convention, we use the variable name "<span style="font-family: "Courier New",Courier,monospace;">logger</span>".<br />
<br />
<h3>
Specifying logging levels</h3>
There are two main ways to set the logging levels. The first is explicitly for a particular logger:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">logger.<a href="http://godoc.org/github.com/howbazaar/loggo#Logger.SetLogLevel">SetLogLevel</a>(loggo.DEBUG)</span><br />
<br />
or chained calls:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">loggo.GetLogger("some.logger").SetLogLevel(loggo.TRACE)</span><br />
<br />
Alternatively you can use a function to specify levels based on a string.<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">loggo.<a href="http://godoc.org/github.com/howbazaar/loggo#ConfigureLoggers">ConfigureLoggers</a>("<root>=INFO; project=DEBUG; project.some.area=TRACE")</span><br />
<br />
The <span style="font-family: "Courier New",Courier,monospace;">ConfigureLoggers</span> function parses the string and sets the logging levels for the loggers specified. This is an additive function. To reset logging back to the default (which happens to be "<span style="font-family: "Courier New",Courier,monospace;"><root>=WARNING</span>", you call<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">loggo.<a href="http://godoc.org/github.com/howbazaar/loggo#ResetLoggers">ResetLoggers</a>()</span><br />
<br />
You can see a summary of the current logging levels with<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">loggo.<a href="http://godoc.org/github.com/howbazaar/loggo#LoggerInfo">LoggerInfo</a>()</span><br />
<br />
<h3>
Adding Writers</h3>
A <a href="http://godoc.org/github.com/howbazaar/loggo#Writer">writer</a> is defined using an interface. The default configuration is to have a "default" writer that writes to Stderr using the default formatter. Additional writers can be added using <span style="font-family: "Courier New",Courier,monospace;">loggo.<a href="http://godoc.org/github.com/howbazaar/loggo#RegisterWriter">RegisterWriter</a></span> and reset using <span style="font-family: "Courier New",Courier,monospace;">loggo.<a href="http://godoc.org/github.com/howbazaar/loggo#ResetWriters">ResetWriters</a></span>. Named writers can be removed using <span style="font-family: "Courier New",Courier,monospace;">loggo.<a href="http://godoc.org/github.com/howbazaar/loggo#RemoveWriter">RemoveWriter</a></span>. Writers are registered with a severity level. Logging below that severity level are not written to that writer.<br />
<br />
<h3>
<span style="font-family: inherit;">More to do</span></h3>
I want to add a syslog writer, but the default syslog package for Go doesn't give the formatting I want. It has been suggested to me to just take a copy of the library implementation and make it work how I want.<br />
<br />
I also want to add some filter-ability to the writers, both on the inclusive and exclusive, so you could say when registering a writer, "only show me messages from these modules", or "don't show messages from these other modules".<br />
<br />
This library has been used in Juju for some time now, and fits with most our needs. For now at least.<br />
<br />
<br />Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com2tag:blogger.com,1999:blog-7854892290437883216.post-41912805788749145352013-07-29T11:42:00.002+12:002013-07-29T13:08:48.748+12:00A personal thank youYesterday evening I had a wonderful IM conversation with a previous team member. He moved on from Canonical to new challenges last year and he was just getting in touch to let me know that I had been a significant positive influence in his professional development. This gave me nice warm fuzzies, but also made me think of those that had helped me over the years. This post is dedicated to those people, who I am going to attempt to recall in roughly historical order. I am however going to try to keep this limited to significant remembered events otherwise this list may get too huge (it may well anyway).<br />
<br />
Firstly I'd like to thank <a href="http://au.linkedin.com/pub/jason-butler/a/b1/761">Jason Butler</a>. You taught me an important lesson very early on. Jason and I worked together as interns (as close a term as I can work out) while at university. Jason taught me me this:<br />
<br />
Just because someone talks slowly, doesn't mean that they think slowly.<br />
<br />
I'd like to thank Jason Ngaio for my first real exposure to C++. Jason was the instructor of the C++ course that my first employers sent me on. This was my first real job, and the first time that I think I really got object oriented programming.<br />
<br />
I'd like to thank Derrick and Pam Finlayson, <a href="http://nz.linkedin.com/in/arranfinlayson">Arran Finlayson</a>, <a href="http://nz.linkedin.com/in/blaircrookston">Blair Crookston</a>, Jenny Cohen, Mathew Downes and <a href="http://nz.linkedin.com/pub/rachel-saunders/32/b59/374">Rachel Saunders</a>. You guys helped me develop personally. The confidence and people skills that I learnt while around you has undoubtedly helped me in my professional career in software development.<br />
<br />
David Cittadini from the then Sapphire Technology company based in Wellington really expanded my vision and understanding of developing complex systems. David also got me back into reading around the programming topic. My technical library started there. Working with <a href="http://nz.linkedin.com/pub/chris-double/3/513/607">Chris Double</a> helped me understand what it is like to work with someone else in synergy. Our joint output I'm sure was a lot more than what we would have both produced independently added together.<br />
<br />
David Ewing made a significant impression on me around knowing my worth and helped in contract negotiations. David has a wonderful way of dealing with people. <br />
<br />
Moving over to London gave me the opportunity to meet up with some truly awesome people. Getting involved with <a href="http://accu.org/">ACCU</a> was great for me. I worked briefly with <a href="http://www.allankelly.net/">Allan Kelly</a> at Reuters, but learned a lot in a brief time. I also had the opportunity to work with <a href="http://uk.linkedin.com/in/gasproni">Giovanni Asproni</a> and <a href="http://uk.linkedin.com/in/alangriffiths">Alan Griffiths</a> at Barclays Capital. Working with you two really helped me understand the power that the developers hold when talking to the business. A few other people I'd like to make a personal note of from this time in the UK are <a href="http://uk.linkedin.com/in/kevlin">Kevlin Henney</a>, <a href="http://uk.linkedin.com/pub/roger-orr/19/245/522">Roger Orr</a> and <a href="http://uk.linkedin.com/in/petegoodliffe">Pete Goodliffe</a>.<br />
<br />
From my early time at Canonical, I'd like to personally thank <a href="http://uk.linkedin.com/in/jonathanlange">Jonathan Lange</a>, <a href="http://nz.linkedin.com/pub/robert-collins/0/b68/403">Robert Collins</a> and <a href="http://nz.linkedin.com/pub/michael-hudson-doyle/0/722/3a6">Michael Hudson-Doyle</a>. You guys really helped me understand the importance of writing good tests, and test driven development. Also the hammering in the code reviews teaching me how to write those tests well.<br />
<br />
There are so many other people that I have had great connections with over my professional career and I'd like to thank you all. Work is more than just what you produce, but the friendships and connections you make with the people you are creating things with.Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com2tag:blogger.com,1999:blog-7854892290437883216.post-23298852773689346202013-07-17T18:03:00.002+12:002013-07-17T18:03:46.240+12:00Stunned by Go<p>
The original working title for this post was "Go is hostile to developers".
This was named at a time of extreme frustration, and it didn't quite seem
right in the cooler light of days later. Instead I've settled on the term
"stunned", because I really was. I felt like the built-in standard library
had really let me down.
</p>
<p>
Let's take a small step back in time to the end of last week as I was debugging
a problem. In our codebase, we had an open file that we would read from, seek
back to the start, and re-read, sometimes several times. This file was passed
as an <tt>io.Reader</tt> into another of our interfaces which had a <tt>Put</tt> method. This
stored the content of the <tt>io.Reader</tt> in a remote location. I was getting this
succeeding the first time, but then erroring out with "bad file descriptor".
</p>
<p>
The more irritating bit was that the same code worked perfectly as expected
with one of our interface implementations but not another. The one that failed was
our "simple" one. All it used was the built-in <tt>http</tt> library to serve a
directory using <tt>GET</tt> and <tt>PUT</tt> http commands.
</p>
<p>
<a href="https://twitter.com/themue">@TheMue</a> suggested that our simple storage provider must be closing the file
somehow. Some digging ensued. What I found had me a little exasperated. The
standard http library was calling <tt>Close</tt> on my <tt>io.Reader</tt>. This is not expected
behaviour when the interface clearly just takes an <tt>io.Reader</tt> (which exposes
one and only one method <tt>Read</tt>).
</p>
<p>
This clearly breaks the <a
href="https://en.wikipedia.org/wiki/Principle_of_least_astonishment">"Principle
of Least Astonishment"</a></p>
<blockquote>
People are part of the system. The design should match the user's
experience, expectations, and mental models.
</blockquote>
<p>
Developers and maintainers are users of the development language.
As an experienced developer, it is my expectation that if a method says it
takes an interface that exposes only <tt>Read</tt>, then only <tt>Read</tt> will be called.
This is not the case in Go standard library.</p>
<p>While I have found just one case, I have been informed that this is common in
Go, and that interfaces are just a "minimum" requirement.</p>
<blockquote class="twitter-tweet"><p><a href="https://twitter.com/howbazaar">@howbazaar</a> you'll also find it's pretty common. io.Copy() will call ReadFrom and WriteTo methods, WriteString() is called to avoid copy.</p>— Jesse McNelis (@jessemcnelis) <a href="https://twitter.com/jessemcnelis/statuses/355991223212834818">July 13, 2013</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>
It seems to me that Go uses the interface casting mechanism as a way to allow
the function implementation to see if the underlying structure supports other
methods, or to check for actual concrete implementation types so the function
can take advantage of extra knowledge. It is one thing to call methods that
don't modify state, however calling a mutating function that the original
function did not express an intent to call is so much more than just
unexpected, but astonishing.</p>
<p>The types of the parameters being passed into a function form a contract.
This has been <a
href="http://en.wikipedia.org/wiki/Design_by_contract">formalized</a> in a
number of languages, particularly <a href="http://dlang.org/dbc.html">D</a> and
<a href="http://en.wikipedia.org/wiki/Eiffel_(programming_language)">Eiffel</a>.</p>
<p>I found myself asking the question "Why do they do this?" The answer I came
up with two things:</p>
<ol>
<li>To take advantage of extra information about the underlying object to make
the execution of the function more efficient</li>
<li>To work around the lack of function overloading</li>
</ol>
<p>Now the second point is tightly coupled to the first point, because if there
was function overloading, then you could clearly have another function that
took a <tt>ReaderCloser</tt> and it would be clear that the <tt>Close</tt> method may well be
called.</p>
<p>My fundamental issue here is that the contract between the function and the caller has been broken. There was not even any documentation to suggest that the contract may be broken. In this case, the calling of the <tt>Close</tt> method on my <tt>io.Reader</tt> broke our code in unexpected ways. As a language that is supposed to be used for systems programming, this just seems crazy.</p>
Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com16tag:blogger.com,1999:blog-7854892290437883216.post-88013400735503866152013-05-10T07:04:00.000+12:002013-05-10T07:08:18.479+12:00juju switchThe <tt>switch</tt> command has recently landed in trunk, and will be included in the next juju-core release.<br />
<br />
<tt>juju switch</tt> is another way to specify the current working environment. Current precedence for environment lookup still holds, but this now sits between the <tt>JUJU_ENV</tt> environment variable and the default value in <tt>environments.yaml</tt>.<br />
<br />
If you have multiple environments defined, there are several different ways to tell juju which environment you mean when executing commands.<br />
<br />
Prior to <tt>switch</tt>, there were three ways to specify the environment.<br />
<br />
The first and default way to specify the environment is to use the <tt>default</tt> value in the <tt>environments.yaml</tt> file. This was always the fallback position if one of the other ways was not specified.<br />
<br />
Another way was to be explicit for some commands, and use the <tt>-e</tt> or <tt>--environment</tt> command line argument.<br />
<br />
<code><br />$ juju bootstrap -e hpcloud</code><br />
<br />
<br />
There is also an environment variable that can be set which will override the default specified in the <tt>environments.yaml</tt> file.<br />
<br />
<code><br />$ export JUJU_ENV=hpcloud<br />$ juju bootstrap # now bootstraps hpcloud<br />$ juju deploy wordpress # deploys to hpcloud</code><br />
<br />
<br />
The <tt>switch</tt> option effectively overrides what the default is for the environments.yaml file without actually changing the environments.yaml file. This means that <tt>-e</tt> and the <tt>JUJU_ENV</tt> options still override the environment defined by switch.<br />
<br />
<code><br />$ juju help switch<br />usage: juju switch [options] [environment name]<br />purpose: show or change the default juju environment name<br /> <br />options:<br />-l, --list (= false)<br /> list the environment names<br /> <br />Show or change the default juju environment name.<br /> <br />If no command line parameters are passed, switch will output the current<br />environment as defined by the file $JUJU_HOME/current-environment.<br /> <br />If a command line parameter is passed in, that value will is stored in the<br />current environment file if it represents a valid environment name as<br />specified in the environments.yaml file.<br /> <br />aliases: env</code><br />
<br />
<br />
It works something like this:<br />
<br />
<code><br />$ juju env<br />Current environment: "amazon-ap"<br />$ juju switch<br />Current environment: "amazon-ap"<br />$ juju switch -l<br />Current environment: "amazon-ap"<br /><br />Environments:<br /> amazon<br /> amazon-ap<br /> hpcloud<br /> openstack<br />$ juju switch amazon<br />Changed default environment from "amazon-ap" to "amazon"<br />$ juju switch amazon<br />Current environment: "amazon"<br />$ juju switch<br />Current environment: "amazon"</code><br />
<br />
<br />
If you have <tt>JUJU_ENV</tt> set, then you get told that the current environment is defined by this. Also if you try to use switch to change the current environment when the environment is defined by <tt>JUJU_ENV</tt>, you will get an error.<br />
<br />
<code><br />$ export JUJU_ENV="amazon-ap"<br />$ juju switch<br />Current environment: "amazon-ap" (from JUJU_ENV)<br />$ juju switch amazon<br />error: Cannot switch when JUJU_ENV is overriding the environment (set to "amazon-ap")</code><br />
<br />
Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com3tag:blogger.com,1999:blog-7854892290437883216.post-59974909201590115642013-04-17T15:51:00.000+12:002013-04-17T15:51:18.450+12:00The Go Language - My thoughtsI've been using the <a href="http://golang.org/">Go</a> programming language
for almost three months now pretty much full time. I have moved from the
Desktop Experience team working on <a href="http://www.ubuntu.com/desktop">Unity</a>, into the <a href="https://juju.ubuntu.com/">Juju</a> team. One of the main reasons I
moved was to learn Go. It had been too long since I had learned another
language, and I felt it was better to dive in, than to just mess with it on my
own time.<br />
<br />
A friend of mine had poked around with Go during a hack fest and <a href="http://corte.si/posts/code/go/go-rant.html">blogged</a> about his
thoughts. This was just before I really started poking around. Interestingly
the main issues that Aldo found frustrating with the errors for unused
variables and unused imports, I have found not to be such a big deal.
Passingly griping, sure, but not a big issue. Having the language enforce
what is often a lint checker in other languages I see as an overall
benefit. Also, even though I don't agree with the Go formatting rules,
enforced by <tt>gofmt</tt>, it doesn't matter. It doesn't matter because all code is
formatted by the tool prior to commit. As an emacs user, I found the <a href="http://golang.org/misc/emacs/go-mode-load.el">go-mode</a> to be
extremely helpful, as I have it formatting all my code using <tt>gofmt</tt> before
saving. I never have to think about it. One thing I couldn't handle though,
was the eight character tabs. Luckily emacs can hide this from me.<br />
<br />
<code>
;; Go bits. </code><br />
<code>(require 'go-mode-load)</code><br />
<code>(add-hook 'before-save-hook #'gofmt-before-save)</code><br />
<code>(add-hook 'go-mode-hook (lambda () (setq tab-width 4)))</code><br />
<code> </code>
<br />
There are some nice bits to Go. I very much approve of channels being first
class objects, and the use of channels to communicate between concurrently
executing code. Go routines are also nifty, although I've not used them too
much myself yet. Our codebase does, but I've not poked into all the nooks and
crannies yet.<br />
<br />
However there are several things which irritate the crap out of me with Go.<br />
<br />
<h2>
Error handling</h2>
The first one I guess is a fundamental design decision which I don't really
agree with. That is around error handling being in your face so you have to
deal with it, as opposed to exceptions, which are all to often not thought
about. Now if our codebase is in any way representative of Go code out there,
this is just flat out wrong. The most repeated lines of code in the codebase
would have to be:<br />
<br />
<code>
if err != nil {</code><br />
<code> return nil</code><br />
<code>}
</code>
<br />
<br />
This isn't error handling. This is just passing it up to the chain, which is
exactly what exception propagation does, only Go makes your codebase two to
three times larger due to needing these three lines after every line of code
that calls into another function. This is one thing I really dislike, but
unlikely to change.<br />
<br />
As a user of a language though, there are other things that could be added at
the language level to make things slightly nicer. Syntactic sugar, as it is
often known, makes the code easier to read.<br />
<br />
If the language is wanting to keep the explicit handling of errors in the
current way, how about some sugar with that.<br />
<br />
Instead of<br />
<br />
<code>func magic() (*type, error) {</code><br />
<code> something, err := somefunc("blah")</code><br />
<code> if err == nil {</code><br />
<code> return nil, err</code><br />
<code> }</code><br />
<code> otherThing, err := otherfunc("blah")</code><br />
<code> if err == nil {</code><br />
<code> return nil, err</code><br />
<code> }</code><br />
<code> return foo(something, otherThing), nil </code><br />
<code>} </code><br />
<code> </code>
<br />
we had some magic sugar, say a built-in method like <tt>raise_error</tt>, which
interrogated the function signature, and returned zeroed values for all
non-error types, and the error, and returned only non-error values, we could
have this<br />
<br />
<code>
func magic() (*type, error) {</code><br />
<code> something := raise_error(somefunc("blah"))</code><br />
<code> otherThing := raise_error(otherfunc("blah"))</code><br />
<code> return foo(something, otherThing), nil </code><br />
<code>}</code><br />
<br />
<h2>
The range function</h2>
There are several different issues I have with the <tt>range</tt> function.<br />
<ol>
<li>range returns one or two parameters, but the language doesn't allow any user defined
functions to return one or two parameters, range is super special</li>
<li>using range with a slice or array and getting a single value, doesn't give
you the value, but instead the index - I never want this</li>
<li>there is no way to define range behaviour for a user defined type</li>
</ol>
These three things are mostly equal in annoyance factor. I'd love to see this
change.<br />
<br />
<h2>
No generics</h2>
Initially I accepted this as a general part of the language. Shouldn't be a
big deal right? C doesn't have generics. I guess I spent too long with C++
then.<br />
My first real annoyance was when I had two integers, and I wanted to find the
maximum value of the two. I go to look in the standard library and find
<tt>math.max</tt>. However that is just for the <tt>float64</tt> type. The standard response
from the team was "it is only a two line function". My response is "that's not
the point".<br />
<br />
Since there is no function overloading, nor generics, there is no way with the
language at this stage to make a standard library function that determines the
maximum value of two or more numeric types, and return that maximum in the
same type as the parameters. Generics would help here.<br />
<br />
A second case for generics is standard containers. The primary container in
Go at this stage is the map. So many places in our codebase we have
<tt>map[string]interface{}</tt>. The problem with this is that you have to
cast all values retrieved from the map. There is also no <tt>set</tt>,
<tt>multi_map</tt>, or <tt>multi_set</tt>. Since there is no way to provide
simple iteration for user defined types, you can't easily define your own
<tt>set</tt> type and have simple iteration using <tt>range</tt>.<br />
<br />
Interfaces that aren't explicitly marked as being implemented help in some
ways to provide features provided by generic types and functions, but it is a
poor substitute.<br />
<br />
<h2>
So far...</h2>
Learning Go has been an interesting experience so far. I like learning new
things, and I'm going to be using Go for some time now with the current
project. No doubt I'll have more to write about later.Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com11tag:blogger.com,1999:blog-7854892290437883216.post-43937843239950554392012-03-27T16:55:00.002+13:002012-03-27T16:58:06.810+13:00Unity 5.8 issues and workaroundsWell... with the release of Unity 5.8 and associated dependencies, we got the extra testing we were after in precise, and with it a number of bugs. The positive side to this is that with the extra information from our wonderful beta-testers we have been able to work out how to reproduce a number of the issues. As any developer would tell you, being able to reproduce your user's problems is often the biggest hurdle.<br />
<br />
Over the weekend I noticed a number of issues around the release of Unity 5.8, and this morning while going through the bug reports, I was happy to notice that we had some way to work around most of them.<br />
<br />
<a href="https://bugs.launchpad.net/unity/+bug/963093">Unity 5.8: Flickering and corruption on Unity UI elements </a> - a fix for many is "unity --reset". The cause appears to be how compiz is dealing with plug-ins that are no longer around. For some there have been plug-ins that existed with Oneiric that are no longer around in Precise, and the reset caused them to be removed from the list to load.<br />
<br />
<a href="https://bugs.launchpad.net/ubuntu/+source/unity/+bug/963633">Unity 5.8: Login to blank screen (all black or just wallpaper) </a> - some have been fixed by "unity --reset", but the underlying cause of this one is still a bit of a mystery.<br />
<br />
<a href="https://bugs.launchpad.net/ubuntu/+source/unity/+bug/963465">Unity 5.8: Can't login to Unity since upgrade to 5.8 </a> - some have found that disabling "Unity MT Grab Handles" compiz plug-in fixes this issue. We still need to work out what the underlying problem is.<br />
<br />
<a href="https://bugs.launchpad.net/compiz-core/+bug/940603">white box randomly shows up at top left corner blocking applications from using stuff under it </a> - this one appears to be triggered by chromium desktop notifications. There have been reports that disabling the animations plug-in in compiz, and then re-enabling it fixes this. We are still investigating why.<br />
<br />
If you are getting these issues, you can try the workarounds suggested here.Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com0tag:blogger.com,1999:blog-7854892290437883216.post-3726139282044772672012-02-20T14:23:00.000+13:002012-02-20T14:23:39.312+13:00Guilt reductionSo it is now Monday morning and I'm sitting next to <a href="http://tech-foo.blogspot.com/">Thomi</a>. We are going to pair program on this test stuff. Partly because I think that pair programming is really cool, and partly due to Thomi knowing the autopilot test infrastructure really well, and that'll make this go much faster.<br />
<br />
The bug in question related to the launcher getting into a very confused state where it thought there were multiple active applications. And clicking on a launcher icon that was in this confused state caused a new application to be started rather than switching to the one that was running.<br />
<br />
The first step in making all this work then, is to create a branch that is based off a revision that was before the fix. This way we can write a test that fails first. A key part of tests is to make sure they fail first. Then when they start passing, you know it isn't by mistake, and that you have tested what you think, not just created something that passes.<br />
<br />
Firstly, find that revision...<br />
<br />
<code>$ bzr log | less</code><br />
<br />
The fix is revision 1977, so lets make a branch of trunk from revision 1976.<br />
<br />
<code>$ bzr cbranch trunk -r 1976 hud-ap-test<br />$ cd hud-ap-test/<br />$ bzr revno<br />1976</code><br />
<br />
I use light weight checkouts for the unity repo, hence cbranch rather than branch.<br />
<br />
At this revision, there is a HUD test that really just checks the reveal. Lets make sure it passes...<br />
<br />
<code>$ cd tests/autopilot/<br />$ python -m testtools.run autopilot.tests.test_hud<br />Tests running...<br />No handlers could be found for logger "autopilot.emulators.X11"<br /><br />Ran 1 test in 4.238s<br />OK</code><br />
<br />
I deleted a bunch of gtk warnings, they don't add any value for what I'm trying to show here. Would be great if someone fixed them though :-)<br />
<br />
Now I need to actually build and run my local unity (and test the autopilot test again).<br />
<br />
Found out that my machine was failing to build for other reasons, so we switched to Thomi's. The existing test still passed (of course it did), so the next step was to write a test that encapsulated the broken behaviour that we had found during the many hours of analysis.<br />
<br />
That can be found at <a href="https://code.launchpad.net/%7Ethomir/unity/autopilot-hud-triple-hit">lp:~thomir/unity/autopilot-hud-triple-hit</a>.<br />
<br />
The test failed with the old revision, we then merged trunk, rebuilt, and ran the test again. Test passed. Job done.Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com1tag:blogger.com,1999:blog-7854892290437883216.post-26746598617882789452012-02-18T21:34:00.000+13:002012-02-18T21:34:23.854+13:00That guilty feelingToday had been a frustrating day. I had been quick to anger and my family bore the brunt of that. It wasn't until I was confronted with this that I actually took a minute to think why I was feeling this way. It came back to something I read on IRC this morning, where I read that some people I deeply respect were disappointed with the test coverage with <a href="https://launchpad.net/unity/5.0/5.4.0">Unity 5.4</a>.<br />
<div>
<br /></div>
<div>
I took this disappointment the way people often take it from their parents. Remember when as a child, one of the worst things you could feel was the disappointment of your parents. Well I guess that is how I felt.</div>
<div>
<br /></div>
<div>
I took over the engineering manager position of the unity team at the end of last year, and I tend to take criticism of the project and team personally.</div>
<div>
<br /></div>
<div>
So... why the guilty feeling?</div>
<div>
<br /></div>
<div>
Well, back around the time I took over managing the team, the general acceptance criteria for getting <a href="http://canonical.com/">Canonical</a> projects into <a href="http://ubuntu.com/">Ubuntu</a> changed. This includes Unity. There were a number of automated tests for Unity, and a series of distro acceptance tests that were manually executed. What we needed to do was to really change the team culture to one where tests were not only written, but expected. New features needed test coverage, bug fixes needed test coverage. The idea here, for all those that understand test driven development, and automated testing, was to make sure that bugs that were fixed, and new features, didn't get broken accidentally by new changes.</div>
<div>
<br /></div>
<div>
The guilt really came from knowing that I had allowed code reviews through the process without enforcing the need for tests. And that as a senior person on the team, others took a lead from what I did. If I was letting things through, so would others. This is where the feeling really came from.</div>
<div>
<br /></div>
<div>
It is very easy to land fixes to crashes quickly when under pressure. Especially when you've spent the last eight hours debugging in gdb, and auditing all the recently landed code looking for that change that would contribute to the broken behaviour that you have been trying to fix. When you finally find that one line fix, it is so tempting to just commit the one line. You know it works, you've just spent the last freaking eight hours looking at the weird behaviour. What you haven't done however, is stopped it from happening again, by encapsulating the behaviour in an automated test.</div>
<div>
<br /></div>
<div>
I plan to spend some of Monday going back and adding an automated test to cover the particular behaviour that we fixed the other day. I'll also write up what, and how this test gets written. Hopefully by writing this, not only will Unity get better test coverage, but I'll personally feel better knowing that I've done the right thing.</div>Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com1tag:blogger.com,1999:blog-7854892290437883216.post-64606594155207010572011-10-30T10:37:00.001+13:002011-10-30T10:37:54.381+13:006 months on UnityWe have just finished another design sprint prior to UDS-P.<br />
<br />
While talking with some others I realise that I have worked on Unity for six months, and not changed a single pixel on the output. No graphical changes, no moving widgets, no changing colours.<br />
<br />
So what have I been doing?<br />
<br />
First step was getting some new coding standards accepted by the team, which was much easier than I was expecting.<br />
<br />
I added some property classes to nux, and did some general clean up in the code of nux and unity.<br />
<br />
Refactored the indicator internals for the panel service which started off the shared unity core library for sharing code between the 2D and 3D code-bases.<br />
<br />
Then I focused primarily on fixing memory leaks and crashes.<br />
<br />
Once we hit final freeze, I did a little more refactoring internally, and now we are on to Precise Pangolin.Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com2tag:blogger.com,1999:blog-7854892290437883216.post-74012981505786628392011-07-11T14:46:00.001+12:002011-07-11T14:46:46.297+12:00Properties in C++<div xmlns='http://www.w3.org/1999/xhtml'><p>Once you have done any development in a language that natively supports properties, like Python or C#, going back to C++ and not having them often feels like a real pain. I've just <a href='https://code.launchpad.net/~thumper/nux/properties/+merge/67479'>proposed</a> my second attempt at C++ properties for the <a href='https://launchpad.net/nux'>nux</a> library.</p><p>This change leans heavily on a paper written by Lois Goldthwaite: <a title='C++ Properties - a Library Solution' href='http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2004/n1615.pdf'>SC22/WG21/N1615</a> - C++ Properties -- a Library Solution. I added change notifications using <a href='http://developer.gnome.org/libsigc++/stable/modules.html'>sigc++</a>. I found that using sigc::slot was nicer than templating the properties on the class and member function pointer. This also meant that I could provide a way for a simple property to get its own custom setter method while still having a sensible default.</p><p>Compiling C++ templates still gives absolutely horrendous error message that can take a while to mentally parse. I guess one advantage of having done a lot of template programming in the past is that I don't get too phased by copious quantities of error messages, especially for templates, as for one example today, I had just forgotten to change a template arg in a test function, and got way too many lines to sensibly look at. One benefit of that was it caused me to look at what I was doing, and I ended up simplifying my tests a little more.</p><p>Thank you Lois for the time you spent writing up the C++ properties proposal, it was a fantastic starting point for me.</p></div>Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com1tag:blogger.com,1999:blog-7854892290437883216.post-79994930417532164872011-05-23T10:28:00.001+12:002011-05-23T10:28:17.022+12:00Getting back into C++<div xmlns='http://www.w3.org/1999/xhtml'><p>I have to admit that unit testing in C++, even with <a href='http://code.google.com/p/googletest/wiki/Documentation'>google-test</a>, is so much more of a <a href='http://www.urbandictionary.com/define.php?term=PITA'>PITA</a> than in python. Especially when checking string output. Simple string matching using split and regular expressions has really spoiled me.</p><p>Another thing that I've noticed is that I spend more time thinking about object design, and what should that object really be able to do, and what should it allow others to do to it.</p><p>It is an interesting time as I realise how much I still have to learn for our current domain. Most of my previous C++ experience has been on server side processing. Drawing stuff on the screen, real end user stuff, is still relatively new for me.</p></div>Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com0tag:blogger.com,1999:blog-7854892290437883216.post-4892016381781562712011-05-08T03:02:00.001+12:002011-05-08T03:02:25.830+12:00DX Sprint<div xmlns='http://www.w3.org/1999/xhtml'><p>What a week. I've spent the last week in Budapest sprinting with the rest of the Desktop Experience (DX) team. This week was also my first official week with the DX team as I have moved now from the Launchpad team to the DX team. This was a good week. I had met some of the DX team before at other company get togethers, but not really talked to them much. A really important part of any new job is meeting the people that you are working with. This is always ends up happening when you work in the same office with them, but for a distributed company like Canonical, you can end up working on the same team with people that you don't get to meet for months.</p><p>It was great to meet different sub-teams of DX, especially those that I'll be working closely with. I'll be hanging out in the #ayatana irc channel now, but I'll also still be in #launchpad and #launchpad-dev. There are some very interesting plans for oneiric, and it will be interesting to see how much we can end up getting done. In the normal way we have "too much to do" and the gauntlet has been thrown.</p><p>So... I'll be hacking on the unity stack. Please don't ask me to fix any particular bugs yet as it'll no doubt take me time to find my way through the code :-)</p><p>Now for the 40+ hour journey home.</p><p/></div>Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com0tag:blogger.com,1999:blog-7854892290437883216.post-3026673013859833142011-04-21T13:38:00.001+12:002011-04-21T13:38:02.092+12:00Launchpad and stacked branches<div xmlns='http://www.w3.org/1999/xhtml'><p>As I'm sure most of you are aware, <a href='https://launchpad.net'>Launchpad</a> hosts <a href='http://bazaar.canonical.com'>Bazaar</a> branches. One early design decision that we had on Launchpad was that branches should be able to be renamed, moved between people and projects without limitation. This is one reason why each branch that was pushed to Launchpad had a complete history. We wanted to make sure that there weren't any problems where one person was blocking another pushing branches, or that people weren't able to get at revisions that they shouldn't be able to.</p><p>The Bazaar library, bzrlib, gives you awesome power to manipulate the internals giving you access to the repository of revisions through the branch. This can be a blessing and a curse, as if you have private revisions, then they can't be in a public repository.</p><p>Having a complete copy of the data for every branch became a severe limitation, especially for larger projects, of which Launchpad itself is one. A solution to this was a change in Bazaar itself that allowed a fallback repository which contained some of the revisions. This is what we call <em>stacked</em> branches. The repository for the branch on Launchpad has a fallback to another repository, which is linked to a different Launchpad branch. We ideally wanted all of this to be entirely transparent to the users of Launchpad. What it means is that when you are pushing a new branch to Launchpad, the bzr client <em>asks</em> for a stacked location. If there is a development focus branch specified for the project, this is then offered back to the client. The new branch then only adds revisions to its repository that don't exist in the development focus branch's repository. This makes for faster pushes, and smaller server side repositories.</p><p>The problem though was what do we specify the stacked on location to be? When we created the feature, we used absolute paths from the transport root. What the mean was that we stored the path aspect of the branch. For example, <span style=' font-family:'Courier New,courier';'>lp:wikkid</span> gets translated to <span style=' font-family:'Courier New,courier';'>bzr+ssh://bazaar.launchpad.net/~wikkid/wikkid/trunk</span> or <span style=' font-family:'Courier New,courier';'>http://bazaar.launchpad.net/~wikkid/wikkid/trunk</span> depending on whether the bzr client knows your Launchpad id. The absolute path stored would be <span style=' font-family:'Courier New,courier';'>/~wikkid/wikkid/trunk</span>. This information was then stored in the branch data on the file system.</p><p>The problem however was that the web interface allows you to rename branches. The actual branch itself on disk is referred to using a database id, which is hidden from the user using a virtual file system which has rewrite rules for http and at the bazaar transport level. However since the stacked on location refers to a full branch path, changing any part of that, whether it is the branch owner, branch name, or the project or package that the branch is for, would cause any branches stacked on that changed branch to break, <a title='Stacked on location breaks if the stacked upon branch is renamed' href='https://bugs.launchpad.net/bugs/377519'>bug 377519</a>.</p><p>In order to fix this we had to change the location that the branch is stacked on to be independent of the branch path. The best solution here is to use the database id. I really didn't want to expose the user to this opaque id, but one opaque id is as good as another. Now when pushing branches to Launchpad, when it is creating a stacked branch you'll see a message like:</p><p><span style=' font-family:'Courier New,courier';'>Created new stacked branch referring to /+branch-id/317141.</span></p><p>Existing branches still have their old branch paths saved for now. We'll run a migration script early next week to fix all these up, and hopefully we'll have seen the last of this bug.</p></div>Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com1tag:blogger.com,1999:blog-7854892290437883216.post-59269303690212581062011-03-11T11:17:00.001+13:002011-03-11T11:17:27.703+13:00Blueprint magic<div xmlns='http://www.w3.org/1999/xhtml'><p>Just landed in <a href='https://qastaging.launchpad.net'>qastaging</a> is some itch-scratching work I did adding AJAX widgets to the main blueprint page. This has passed QA and will end up in production with the next no-downtime rollout (which should be real soon now).</p><p>This work was adding a bunch of the lazr-js wrapped widgets. Now we can update the following without reloading the primary page:</p><ul><li>title - the H1 heading</li><li>summary</li><li>whiteboard</li><li>assignee</li><li>drafter</li><li>approver</li><li>priority</li><li>implementation status</li><li>definition status</li></ul><p>Using the new custom events that the page raises when the context object changes (using YUI magic and API PATCH requests), when you change the title of the blueprint, the document title (title bar) and the breadcrumbs also change. When the implementation status is updated, the overall status updates, and the "started by" and "completed by" are shown or hidden as appropriate.</p><p>This is work that I've wanted to see done for almost a year, and recent other changes I've done adding more widget wrappers and javascript goodness have made this possible without adding copious amounts of custom javascript.</p><p>A side-effect of these changes is that there are now more fields exported over the API for blueprints.</p></div>Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com0tag:blogger.com,1999:blog-7854892290437883216.post-50432302432476503022011-03-11T10:19:00.001+13:002011-03-11T10:19:16.264+13:00Announcing sloecode<div xmlns='http://www.w3.org/1999/xhtml'><p><a href='https://launchpad.net/sloecode'>Sloecode</a> is a simple <a href='http://bazaar.canonical.com'>Bazaar</a> hosting project.</p><p>Last year I tried to set up my home server to offer a place for three people to have shared, private access to a bazaar repository for a project. I found it really ackward. I felt that there had to be a simpler way.</p><p>Launchpad is an awesome place to host Bazaar branches. However <a href='htps://launchpad.net'>Launchpad</a> is for open source projects and personal branches are public.</p><p>I was approached towards the end of last year by <a href='http://tech-foo.blogspot.com'>Thomi</a> who suggested we create a simple Bazaar hosting project for <a href='http://op.ac.nz'>Otago Polytechnic</a> to provide a place for the students to host their senior year projects. Since it was something I also cared about, and hadn't found a reasonable solution elsewhere, I agreed to help.</p><p>Our initial requirements went something like this:</p><ul><li>Users defined in a database and no local login needed on the hosting server</li><li>Private repositories for the students to host personal branches</li><li>Lecturers should be able to see the repositories of the students</li><li>Projects have private repositories only visible to the members of the project team, and the Lecturers</li><li>Simple URLs for getting access to the branches</li><li>Scalability isn't a priority</li></ul><p>In the end we went with the new pyramid libraries for the web application. We tried briefly with django, but I found the framework blocked me whenever I tried to do something. I had worked a lot with zope, and repoze.bfg was something we looked at. When repoze.bfg and turbogears merged into pyramid we felt that we had found a good match for us.</p><p>Installing and running the sloecode server is still a bit messy. We'd love to get it to the stage we you can just run an installer and magic happens. But we are not there yet.</p><p>The application server runs on the same machine as the filesystem hosting the branches and repositories. Shared repositories are created for each user as they are added. When a project is created so is its shared repository, and trunk branch which is set to append only.</p><p>Users log in and add their public SSH key. Users should also install the <a href='https://launchpad.net/bzr-sloecode'>bzr-sloecode</a> client plugin. Initially the plugin had hard coded site names for the Otago Polytechnic. But looking forward I decided that we should just use an environment variable. This allows access to the sloecode server using a short hand url.</p><ul><li>sc:my-project - gets access to the <em>trunk</em> branch of the project called <em>my-project</em></li><li>sc:my-project/trunk - also gets access to the <em>trunk</em> branch</li><li>sc:my-project/some-branch - gets access to the branch <em>some-branch </em>of <em>my-project</em></li><li>sc:~myid/personal-branch - gets access to the branch <em>personal-branch</em> on my repository</li></ul><p>The <strong>sc </strong>translation is done client-side, like the <strong>lp </strong>expansion for Launchpad.</p><p>We run a custom <a href='http://twistedmatrix.com'>twisted</a> SSH server on the hosting machine. This does the SSH key lookup against our user database, and doesn't allow password login. It also restricts the commands that can be executed on the server side to what it expects Bazaar to ask for. No shells are given. The server then launches a subprocess that uses a wrapped smart server that has a virtual filesystem to translate the requested paths to the underlying filesystem. This operates in a similar way to Launchpad, but much more trivially. The repository hosting configuration just needs two paths: one for the project repositories, and one for the personal repositories. The smart server code also handles the privacy aspect, not allowing unauthorized users access to repositories they shouldn't see.</p><h2>The future</h2><p>We'd like to have some form of code browsing functionality added. Whether we use <a href='https://launchpad.net/loggerhead'>loggerhead</a>, or something else is still up in the air. We'd also like to integrate wikis for the projects. <a href='https://launchpad.net/wikkid'>Wikkid</a> would be a good fit (another of my personal projects) and could spur on some of the stalled development.</p><p>Ease of installation and configuration is another aspect we'd like to tidy up.</p><p>If you are interested in getting involved join the <a href='https://launchpad.net/~sloecode-dev'>sloecode-dev</a> team on Launchpad as it handles the development mailing list.</p></div>Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com2tag:blogger.com,1999:blog-7854892290437883216.post-26866073077725422852011-02-15T18:01:00.001+13:002011-02-15T18:01:31.431+13:00More responsive recipe builds<div xmlns='http://www.w3.org/1999/xhtml'><p><a href='https://launchpad.net/~stevenk'>SteveK</a> has recently landed a change that at the same time makes our admins happy, and should give our recipe users more responsive builds.</p><p>Daily builds were previously kicked off at 23:00 UTC. A rather arbitrary time that put quite a load on the build farm as recipes became more popular.</p><p>The new change has our script to kick of daily recipe builds run much more often, and uses a cunning mix of magic and smarts to run the jobs (ok, not so much magic).</p><p>When new changes are pushed to Launchpad for branches that are used in source package recipes, the recipe is marked as <em>dirty</em>. Dirty recipes are candidates for daily builds. If a recipe has not been built into the daily build PPA within the last 24 hours a new recipe build job is created very soon after Launchpad notices the new changes. If there has been a build, then the new build job isn't created until 24 hours has past since the last build. Manual builds into other PPAs do not affect the daily build time check.</p><p>What this means is that if you have a daily build recipe, and tend to change the branch less often than every day, then when you do change it, the package is built much more quickly.</p></div>Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com0tag:blogger.com,1999:blog-7854892290437883216.post-78561638512500064072011-01-31T14:57:00.001+13:002011-01-31T14:57:11.794+13:00Refactoring Launchpad's lazrjs widgets<div xmlns='http://www.w3.org/1999/xhtml'><p>Just landed (<a href='http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/revision/12285'>r12285 on devel</a>) is some <a href='https://code.launchpad.net/~thumper/launchpad/refactor-lazrjs-text-widgets/+merge/47634'>refactoring</a> I've been doing on the <a href='https://launchpad.net/lazr-js'>lazr-js</a> widget wrappers. After hacking on the picker at Dallas, I felt the other editor widgets needed some attention as well. The primary documentation can be found in the source tree at lib/lp/app/doc/lazr-js-widgets.txt, in fact this is probably the best thing to read.</p><p>This new changes did several things:</p><ul><li><a href='https://bugs.launchpad.net/launchpad/+bug/708267'>fixed</a> the multi-line editor so you don't need extra HTML tags surrounding the widget in the page template</li><li>changed the widgets to actually use page templates to render the content instead of fiddling with string substitution</li><li>refactored the class inheritance - the multi-line editor no longer inherits from the single line editor, but instead they both inherit from a common base</li><li>it changed the way the editable attribute is defined - from being the name of the attribute, to being the field from the interface</li><li>the initial text for the multi-line editor is now determined from the object itself, not a parameter passed through to the constructor</li><li>the permission checks were unified, which fixed a problem with the text editors using mutator functions exposed through <a href='https://launchpad.net/lazr.restful'>lazr.restful</a></li><li>you now have to be explicit about the id used for the HTML tag surrounding the editor - this wasn't much of an ask as all call sites were already doing this</li></ul><p>All in all I'm pretty happy with this change.</p></div>Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com0tag:blogger.com,1999:blog-7854892290437883216.post-30052122538886657192011-01-23T05:39:00.001+13:002011-01-23T05:39:25.262+13:00Code Blue at the Thunderdome<div xmlns='http://www.w3.org/1999/xhtml'><p>I am sitting in the hotel at the end of the <a href='https://launchpad.net'>Launchpad</a> Thunderdome. Really this was just a sprint for all of the Launchpad developers, but someone decided it needed a neat name, and Thunderdome stuck.</p><p>It is at this sprint that the Launchpad team is transitioning away from the old application based teams to "squads". There are five squads, each given a colour for now: Blue, Green, Orange, Yellow, and Red. So my "Code" team has been split up and I have a new squad "Blue". Before we even switched to the squads, on of my new members resigned to move to something new, so I have an <a href='http://webapps.ubuntu.com/employment/canonical_CDO-LPC-SE/'>opening</a>. Actually right now we have <a href='http://webapps.ubuntu.com/employment/canonical_CDO-LPB-SE/'>two</a> openings. One squad is primarily based in Australasia (with one in New York) and the other has people from North America and Europe.</p><p>The majority of our time has been spent getting to know our new squads. Three of the squads are working on features, and two are on maintenance. The Blue squad (called "Code Blue" from now) is finishing of the <a href='https://help.launchpad.net/Packaging/SourceBuilds'>recipe feature</a>. This allows packages to be built automatically directly from a source package assembled from one or more branches on Launchpad. This feature was started over a year ago and needs to be finished off. Most of what we are doing now is just polishing the user experience and interface. The feature has been available since around the middle of the year, and we have almost <a href='https://code.launchpad.net/+daily-builds'>250 recipes</a> that are configured to build packages automatically. Once we finish this off Code Blue will move on to maintenance until some other feature teams finish up.</p><p align='left'>We have done some pretty impressive work on the <a href='https://help.launchpad.net/API'>webservice</a> this week. Primarily due to trying to make nice AJAX popups for changing the owner and <a href='https://help.launchpad.net/Packaging/PPA'>PPA</a> of a recipe. This resulted in much <a title='Duplicate adapters for webservice XHTML representation' href='https://bugs.launchpad.net/launchpad/+bug/704641'>yak</a> <a title='Person xhtml representation adds a <span>' href='https://bugs.launchpad.net/launchpad/+bug/704651'>shaving</a>, and <a title='Changing a bug's affected project or description doesn't ever finish' href='https://bugs.launchpad.net/launchpad/+bug/443217'>refactoring</a>. The work we did make it trivially easy to add an AJAX popup to any place on Launchpad where a single value is chosen from a list of options. We also fixed <a title='Stop allowing export of Object, just Reference' href='https://code.launchpad.net/~leonardr/lazr.restful/stop-using-object/+merge/47106'>a</a> <a title='IReferenceChoice should inherit from IReference' href='https://code.launchpad.net/~leonardr/lazr.restful/ireferencechoice-is-a-reference/+merge/46850'>number</a> <a title='Request an XHTML representation, get redirected to a JSON representation' href='https://bugs.launchpad.net/lazr.restful/+bug/704214'>of</a> <a title='Wacky error when using the inline edit for bug task project in Chromium' href='https://bugs.launchpad.net/launchpad/+bug/423149'>bugs</a> in <a title='Publish data model objects as RESTful web services by annotating their Zope interfaces.' href='https://launchpad.net/lazr.restful'>lazr.restful</a> and very soon there will be a release that will add a '<a title='Add web_link property to resources' href='https://bugs.launchpad.net/launchpad/+bug/316694'>web_link</a>' property to the entries in <a title='A free Python library for scripting Launchpad through its web services interface.' href='https://launchpad.net/launchpadlib'>launchpadlib</a> that will refer to the actual object on Launchpad.</p></div>Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com0tag:blogger.com,1999:blog-7854892290437883216.post-61272414446761703842010-11-03T13:27:00.001+13:002010-11-03T13:27:47.239+13:00Coworking<div xmlns='http://www.w3.org/1999/xhtml'><p>I've been working for Canonical for around four years now, and for the vast majority of that time I've worked out of a room in my house. You need to have a certain style of discipline to work effectively from home and I like to think that I have that.</p><p>More recently I've been feeling a little down about work in general. I think it is because I am insulated and tend to get a lot of information from blogs, a number of feeds, twitter, identi.ca, facebook and email lists. A lot of the information that I then get exposed to is often from the vocal minority, and this can be a little draining.</p><p>Last Friday I started spending some of my work day at <a href='http://thedistiller.org/'>the distiller</a> lab. I have to say it has been great for my general state of mind. Today is my forth day in a row spending some time here and so far, so good.</p></div>Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com2tag:blogger.com,1999:blog-7854892290437883216.post-73499006883584168502010-05-26T21:35:00.002+12:002010-05-26T21:53:03.720+12:00Wikkid WikiAs with any creation being released, I'm writing this with some trepidation. I'd like to announce the first release, 0.1, of <a href="http://wikkid.info">Wikkid Wiki</a>.<br /><br /><strong>What is it?</strong><br /><br />Wikkid is a wiki that uses the <a href="http://bazaar.canonical.com">Bazaar DVCS</a> as an underlying storage model. Wiki pages are text files in a branch.<br /><br /><strong>Why another wiki, surely there are enough already?</strong><br /><br />There is the obvious reason, because I felt like it. But this is not the primary reason. Since I started working for <a href="http://canonical.com">Canonical</a> I've come to appreciate the whole culture of free, libre, and open source software more. During the day I work on <a href="https://launchpad.net">Launchpad</a>. Primarily in the area of integrating the Bazaar DVCS. Launchpad doesn't have a wiki integrated, and it is my plan to see wikkid be the wiki that is integrated.<br /><br />I have to admit to having very strong opinions myself on what I wanted for wikis in Launchpad. I've tried to encapsulate that in the vision below. Since no one else was looking at it, I took it upon myself. Discussions started last year between a small group of Launchpad developers, but there was no traction. At the start of March I started writing the Bazaar backed wiki. It needed a name though. Thankfully after trying several I got the perfect name from Aaron Bentley - wikkid.<br /><br /><strong>The wikkid vision</strong><br /><br /><ul><br /><li>Any wikkid wiki can be branched locally for off line editing.</li><br /><li>Any branch can be viewed using wikkid - not limited to branches created through wikkid</li><br /><li>A local wikkid server can be run using a Bazaar plugin</li><br /><li>Local commits use the local users Bazaar identity</li><br /><li>Wikkid can be configured to operate in a stand alone, public facing mode where it has a database of users</li><br /><li>Wikkid can be integrated as a library into other python applications</li><br /><li>Wikkid uses standard wiki markup languages - not inventing its own</li><br /></ul><br /><br /><strong>What does Wikkid Wiki offer?</strong><br /><br />Right now, wikkid offers basic page editing, rendering and browsing.<br /><ul><br /><li><a href="http://docutils.sourceforge.net/rst.html">ReST</a> is the default wiki format</li><br /><li><a href="http://www.wikicreole.org/wiki/Creole1.0">Creole 1.0</a> is also supported (as long as the first line is "# creole")</li><br /><li>source files are syntax highlighted using pygments</li><br /><li>you get to see your gravatar for your local Bazaar identity</li><br /><li>no page locks are used, but instead a three way merge</li><br /><li>conflicts due to concurrent editing are shown for the user to resolve</li><br /></ul><br /><br /><strong>Where to from here?</strong><br /><br />Well here is just the beginning. The TODO file is quite long already, and<br />that is just a simple brain dump.<br /><br />Things that I want to have done for 0.2 include:<br /><ul><br /><li>Change the underlying server from twisted.web to WSGI</li><br /><li>Change the generated URL format</li><br /><li>Add the stand-alone user database code, along with sessions and logins and email validation</li><br /><li>Add a view to show changes for a page</li><br /><li>Allow the reverting of any historical change</li><br /><li>Daily build of trunk into the wikkid developer's PPA</li><br /></ul><br /><br />Ideally for the 0.2 release I'd like to provide everything that is needed for<br />wikkid to be deployed as a stand-alone, public facing, wiki.<br /><br /><strong>Wow, how can I help?</strong><br /><br />Wikkid uses Launchpad for colaboration and project tracking -<br />https://launchpad.net/wikkid.<br /><ul><br /><li>You can get a copy of trunk using 'bzr branch lp:wikkid'</li><br /><li><a href="https://bugs.launchpad.net/wikkid/+filebug">File bugs</a> for any issues you find playing with it</li><br /><li>Join the <a href="https://launchpad.net/~wikkid-dev">development and discussion mailing list</a></li><br /><li>If you feel so inclined, you could implement a feature or fix a bug, push the branch to Launchpad and propose the merge</li><br /><li>Come and chat in #wikkid on freenode, nothing fires developers up like having encouraging users</li><br /></ul>Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com4tag:blogger.com,1999:blog-7854892290437883216.post-81525145030332540692010-05-06T12:00:00.002+12:002010-05-06T12:04:23.231+12:00Launchpad code updateWe've been very busy over the last couple of months with lots of changes that most people will never notice.<br /><br /><h3>Reduced latency</h3><br />Branches pushed to Launchpad are now immediately available over http to anonymous readers of the branch, which includes the loggerhead code browser.<br /><br /><h3>Code review email changes</h3><br />When proposing a branch for review the initial emails and subsequent comments will now come in order. Previously if someone commented before the script that generated the diff was run, the comment would be emailed out first, now it isn't.<br /><br /><h3>Teams requested to review now get email</h3><br />Everyone in the team that is requested to review will get email now. This is a blessing for those that want it, and almost a curse for those that aren't interested. Launchpad adds a number of email headers to help users with filtering of email. Here is an example from an email I received:<br /><br /><tt><br /><strong>X-Launchpad-Message-Rationale:</strong> Reviewer @drizzle-developers<br /><strong>X-Launchpad-Notification-Type:</strong> code-review<br /><strong>X-Launchpad-Project:</strong> drizzle<br /></tt><br /><br />Since it was a team that was requested to review, there is the <tt>@drizzle-developers</tt> added to the X-Launchpad-Message-Rationale. If I was personally asked to review, the header would just say <strong>Reviewer</strong>.<br /><br /><h3>Build branch to archive</h3><br />This was the original name of the feature, but it is more about recipes now. A recipe is a collection of instructions on how to build a source package. We are still testing this internally, but I'm hoping to get this enabled on edge very soon. This will be extended to add daily builds.<br /><br />What does this really mean?<br /><br />Lets say you want to have a daily build of a project, like <a href="https://launchpad.net/gwibber">gwibber</a>. You would then create a recipe that uses trunk as a base branch, merge in the packaging info, and say <em>"Please build this every day into my PPA"</em>. And Launchpad will.Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com0tag:blogger.com,1999:blog-7854892290437883216.post-46355646127051237412010-02-22T20:30:00.004+13:002010-02-22T20:35:15.215+13:00Trivial bugsThis is just a quick note really. One thing I've been trying to do more and more is to address simple bugs in a more timely manner.<br /><br />I use the tag "<a href="https://bugs.edge.launchpad.net/launchpad-code/+bugs?field.tag=trivial">trivial</a>" to indicate to me that the bug is very simple to fix. By this I mean that I should be able to have the fix and the test all written in under an hour, and normally under 30 minutes.<br /><br />Personally I'm (hopefully) fixing one trivial bug a day in addition to other work. This way the simple bugs get some attention, and I get the feeling of accomplishing something when other things are in the pipeline that take longer to get completed.<br /><br />My scheduling of trivial bugs is somewhat arbitrary. Often the most recently commented on trivial bug will get my attention.Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com0tag:blogger.com,1999:blog-7854892290437883216.post-143118980706418272009-11-13T21:32:00.002+13:002009-11-13T21:59:51.508+13:00Don't wait for perfectionWay back in July I was thinking of writing a post about the new branch listings in Launchpad. I was working on making branch listings for distributions and distroseries work, for example <a href="https://code.launchpad.net/ubuntu">Package branches for Ubuntu</a> and <a href="https://code.launchpad.net/ubuntu/lucid">Packages branches for Ubuntu Lucid</a>. But the code wasn't entirely optimised. Then as things happen, the optimisation got pushed back... and back. And finally when I did get the optimisation in, I didn't feel it was worthy of talking about.<br /><br />I guess the thing to remember is: <em>don't wait for perfection</em>. Sure it wasn't perfect, but if more people were accessing the pages, the optimisation may have happened sooner.<br /><br />One thing going on at the moment is more integration of the <a href="https://launchpad.net/lazr-js">lazr-js</a> widgets. The main merge proposal page now has in in-page multi-line editor for the commit message. Sure, it needs <a href="https://bugs.launchpad.net/launchpad-code/+bug/479526">tweaking</a>, but the main functionality is there. More ajaxy goodness is finding its way into Launchpad.<br /><br />One of the things that I'm thinking about right now is splitting the concepts of code reviews and merge proposals. At the moment we almost use the term interchangeably, which does cause some confusion. I'd like to have the merge proposal reflect the meta-data and information around the intent to have work from one branch be landed or merged into another branch (normally the trunk branch), and the code review the conversation that goes on around the merge proposal. Merge proposals may have an associated code review, but right now, a code review must be associated with a merge proposal.<br /><br />Associated with this, I'd like to extract some state information. Currently merge proposals have only one status, which really reflects two things. I'd like to break this out into two: review status; and merge status. Review status would be one of: work in progress; needs review; approved; rejected; superseded; and maybe withdrawn. Merge status would be one of: proposed; merged; queued; and merge failed. Queued relates to the merge queues which are currently partially implemented in the Launchpad codebase, and merge failed is the state that a proposal would be set to when a landing robot like <a href="https://launchpad.net/pqm">PQM</a> or <a href="https://launchpad.net/tarmac">Tarmac</a> attempt to land the branch but it fails due to either merge conflicts or test failures.<br /><br />My goal for the next six months it to write more often, talk about ideas more, and not wait for perfection.Tim Penhey (thumper)http://www.blogger.com/profile/09872376538882273724noreply@blogger.com2