2013 in review

2013 started with what felt like a failure, but in the end, I believe that the
best decision was made.  During 2011 and 2012 I worked on and then managed
the Unity desktop team.  This was a C++ project that brought me back to my
hard-core hacker side after four and a half years on Launchpad.  The Unity
desktop was a C++ project using glib, nux, and Compiz. After bringing Unity to
be the default desktop in 12.04 and ushering in the stability and performance
improvements, the decision was made to not use it as the way to bring the
Ubuntu convergence story forward. At the time I was very close tho the Unity 7
codebase and I had an enthusiastic capable team working on it. The decision
was to move forwards with a QML based user interface.  I can see now that this
was the correct decision, and in fact I could see it back in January, but that
didn't make it any easier to swallow.

I felt that I was at a juncture and I had to move on.  Either I stayed with
Canonical and took another position or I found something else to do. I do like
the vision that Mark has for Ubuntu and the convergence story and I wanted to
hang around for it even if I wasn't going to actively work on the story itself.  For a while I was interested in learning a new programming language, and Go was considered the new hotness, so I looked for a position working on Juju. I was lucky to be able to join the the juju-core team.

After a two weak break in January to go to a family wedding, I came back to
work and started reading around Go. I started with the language specification
and then read around and started with the Go playground. Then started with the
Juju source.

Go was a very interesting language to move to from C++ and Python. No
inheritance, no exceptions, no generics. I found this quite a change.  I even
blogged about some of these frustrations.

As much as I love the C++ language, it is a huge and complex language. One
where you are extremely lucky if you are working with other really competent
developers. C++ is the sort of language where you have a huge amount of power and control, but you pay other costs for that power and control. Most C++ code is pretty terrible.

Go, as a contrast, is a much smaller, more compact, language. You can keep the
entire language specification in your head relatively easily. Some of this is
due to specific decisions to keep the language tight and small, and others I'm
sure are due to the language being young and immature. I still hope for
generics of some form to make it into the language because I feel that they
are a core building block that is missing.

I cut my teeth in Juju on small things. Refactoring here, tweaking
there. Moving on to more substantial changes.  The biggest bit that leaps to
mind is working with Ian to bring LXC containers and the local provider to the
Go version of Juju.  Other smaller things were adding much more infrastructure
around the help mechanism, adding plugin support, refactoring the provisioner,
extending the logging, and recently, adding KVM container support.

Now for the obligatory 2014 predictions...

I will continue working on the core Juju product bringing new and wonderful
features that will only be beneficial to that very small percentage of
developers in the world who actually deal with cloud deployments.

Juju will gain more industry support outside just Canonical, and will be seen
as the easiest way to OpenStack clouds.

I will become more proficient in Go, but will most likely still be complaining
about the lack of generics at the end of 2014.

Ubuntu phone will ship.  I'm guessing on more than just one device and with
more than one carrier. Now I do have to say that these are just personal
predictions and I have no more insight into the Ubuntu phone process than
anyone outside Canonical.

The tablet form-factor will become more mature and all the core applications,
both those developed by Canonical and all the community contributed core
applications will support the form-factor switching on the fly.

The Unity 8 desktop that will be based on the same codebase as the phone and
tablet will be available on the desktop, and will become the way that people
work with the new very high resolution laptops.

loggo - hierarchical loggers for Go

Some readers of this blog will just think of me as that guy that complains about the Go language a lot.  I complain because I care.

I am working on the Juju project.  Juju is all about orchestration of cloud services.  Getting workloads running on clouds, and making sure they communicate with other workloads that they need to communicate with. Juju currently works with Amazon EC2, HP Cloud, Microsoft Azure, local LXC containers for testing, and Ubuntu's MAAS. More cloud providers are in development. Juju is also written in Go, so that was my entry point to the language.

My background is from Python and C++.  I have written several logging libraries in the past, but always in C++ and with reasonably specific performance characteristics.  One thing I really felt was missing with the standard library in Go was a good logging library. Features that I felt were pretty necessary were:

• A hierarchy of loggers
• Able to specify different logging levels for different loggers
• Loggers inherited the level of their parent if not explicitly set
• Multiple writers could be attached
• Defaults should "just work" for most cases
• Logging levels should be configurable easily
• The user shouldn't have to care about synchronization

Initially this project was hosted on Launchpad.  I am trialing moving the trunk of this branch to github.  I have been quite isolated from the git world for some time, and this is my first foray in git, and specifically git and go.  If I have done something wrong, please let me know.

Basics

There is an example directory which demonstrates using loggo (albeit relatively trivially).

import "github.com/howbazaar/loggo"
...
logger = loggo.GetLogger("project.area")
logger.Debugf("This is debug output.")
logger.Warningf("Some error: %v", err)

In juju, we normally create one logger for the module, and the dotted name normally reflects the module. This logger is then used by the other files in the module.  Personally I would have preferred file local variables, but Go doesn't support that, not where they are private to the file, and as a convention, we use the variable name "logger".

Specifying logging levels

There are two main ways to set the logging levels. The first is explicitly for a particular logger:

logger.SetLogLevel(loggo.DEBUG)

or chained calls:

loggo.GetLogger("some.logger").SetLogLevel(loggo.TRACE)

Alternatively you can use a function to specify levels based on a string.

loggo.ConfigureLoggers("<root>=INFO; project=DEBUG; project.some.area=TRACE")

The ConfigureLoggers function parses the string and sets the logging levels for the loggers specified.  This is an additive function.  To reset logging back to the default (which happens to be "<root>=WARNING", you call

loggo.ResetLoggers()

You can see a summary of the current logging levels with

loggo.LoggerInfo()

Adding Writers

A writer is defined using an interface. The default configuration is to have a "default" writer that writes to Stderr using the default formatter.  Additional writers can be added using loggo.RegisterWriter and reset using loggo.ResetWriters. Named writers can be removed using loggo.RemoveWriter.  Writers are registered with a severity level. Logging below that severity level are not written to that writer.

More to do

I want to add a syslog writer, but the default syslog package for Go doesn't give the formatting I want. It has been suggested to me to just take a copy of the library implementation and make it work how I want.

I also want to add some filter-ability to the writers, both on the inclusive and exclusive, so you could say when registering a writer, "only show me messages from these modules", or "don't show messages from these other modules".

This library has been used in Juju for some time now, and fits with most our needs.  For now at least.

A personal thank you

Yesterday evening I had a wonderful IM conversation with a previous team member.  He moved on from Canonical to new challenges last year and he was just getting in touch to let me know that I had been a significant positive influence in his professional development.  This gave me nice warm fuzzies, but also made me think of those that had helped me over the years.  This post is dedicated to those people, who I am going to attempt to recall in roughly historical order.  I am however going to try to keep this limited to significant remembered events otherwise this list may get too huge (it may well anyway).

Firstly I'd like to thank Jason Butler.  You taught me an important lesson very early on.  Jason and I worked together as interns (as close a term as I can work out) while at university.  Jason taught me me this:

Just because someone talks slowly, doesn't mean that they think slowly.

I'd like to thank Jason Ngaio for my first real exposure to C++.  Jason was the instructor of the C++ course that my first employers sent me on.  This was my first real job, and the first time that I think I really got object oriented programming.

I'd like to thank Derrick and Pam Finlayson, Arran Finlayson, Blair Crookston, Jenny Cohen, Mathew Downes and Rachel Saunders.  You guys helped me develop personally.  The confidence and people skills that I learnt while around you has undoubtedly helped me in my professional career in software development.

David Cittadini from the then Sapphire Technology company based in Wellington really expanded my vision and understanding of developing complex systems. David also got me back into reading around the programming topic.  My technical library started there.  Working with Chris Double helped me understand what it is like to work with someone else in synergy.  Our joint output I'm sure was a lot more than what we would have both produced independently added together.

David Ewing made a significant impression on me around knowing my worth and helped in contract negotiations.  David has a wonderful way of dealing with people.

Moving over to London gave me the opportunity to meet up with some truly awesome people.  Getting involved with ACCU was great for me.  I worked briefly with Allan Kelly at Reuters, but learned a lot in a brief time. I also had the opportunity to work with Giovanni Asproni and Alan Griffiths at Barclays Capital.  Working with you two really helped me understand the power that the developers hold when talking to the business.  A few other people I'd like to make a personal note of from this time in the UK are Kevlin Henney, Roger Orr and Pete Goodliffe.

From my early time at Canonical, I'd like to personally thank Jonathan Lange, Robert Collins and Michael Hudson-Doyle.  You guys really helped me understand the importance of writing good tests, and test driven development.  Also the hammering in the code reviews teaching me how to write those tests well.

There are so many other people that I have had great connections with over my professional career and I'd like to thank you all.  Work is more than just what you produce, but the friendships and connections you make with the people you are creating things with.

Stunned by Go

The original working title for this post was "Go is hostile to developers". This was named at a time of extreme frustration, and it didn't quite seem right in the cooler light of days later. Instead I've settled on the term "stunned", because I really was. I felt like the built-in standard library had really let me down.

Let's take a small step back in time to the end of last week as I was debugging a problem. In our codebase, we had an open file that we would read from, seek back to the start, and re-read, sometimes several times. This file was passed as an io.Reader into another of our interfaces which had a Put method. This stored the content of the io.Reader in a remote location. I was getting this succeeding the first time, but then erroring out with "bad file descriptor".

The more irritating bit was that the same code worked perfectly as expected with one of our interface implementations but not another. The one that failed was our "simple" one. All it used was the built-in http library to serve a directory using GET and PUT http commands.

@TheMue suggested that our simple storage provider must be closing the file somehow. Some digging ensued. What I found had me a little exasperated. The standard http library was calling Close on my io.Reader. This is not expected behaviour when the interface clearly just takes an io.Reader (which exposes one and only one method Read).

This clearly breaks the "Principle of Least Astonishment"

People are part of the system. The design should match the user's experience, expectations, and mental models.

Developers and maintainers are users of the development language. As an experienced developer, it is my expectation that if a method says it takes an interface that exposes only Read, then only Read will be called. This is not the case in Go standard library.

While I have found just one case, I have been informed that this is common in Go, and that interfaces are just a "minimum" requirement.

@howbazaar you'll also find it's pretty common. io.Copy() will call ReadFrom and WriteTo methods, WriteString() is called to avoid copy.

— Jesse McNelis (@jessemcnelis) July 13, 2013

It seems to me that Go uses the interface casting mechanism as a way to allow the function implementation to see if the underlying structure supports other methods, or to check for actual concrete implementation types so the function can take advantage of extra knowledge. It is one thing to call methods that don't modify state, however calling a mutating function that the original function did not express an intent to call is so much more than just unexpected, but astonishing.

The types of the parameters being passed into a function form a contract. This has been formalized in a number of languages, particularly D and Eiffel.

I found myself asking the question "Why do they do this?" The answer I came up with two things:

1. To take advantage of extra information about the underlying object to make the execution of the function more efficient
2. To work around the lack of function overloading

Now the second point is tightly coupled to the first point, because if there was function overloading, then you could clearly have another function that took a ReaderCloser and it would be clear that the Close method may well be called.

My fundamental issue here is that the contract between the function and the caller has been broken. There was not even any documentation to suggest that the contract may be broken. In this case, the calling of the Close method on my io.Reader broke our code in unexpected ways. As a language that is supposed to be used for systems programming, this just seems crazy.

juju switch

The switch command has recently landed in trunk, and will be included in the next juju-core release.

juju switch is another way to specify the current working environment. Current precedence for environment lookup still holds, but this now sits between the JUJU_ENV environment variable and the default value in environments.yaml.

If you have multiple environments defined, there are several different ways to tell juju which environment you mean when executing commands.

Prior to switch, there were three ways to specify the environment.

The first and default way to specify the environment is to use the default value in the environments.yaml file.  This was always the fallback position if one of the other ways was not specified.

Another way was to be explicit for some commands, and use the -e or --environment command line argument.


$ juju bootstrap -e hpcloud


There is also an environment variable that can be set which will override the default specified in the environments.yaml file.


$ export JUJU_ENV=hpcloud
$ juju bootstrap          # now bootstraps hpcloud
$ juju deploy wordpress   # deploys to hpcloud


The switch option effectively overrides what the default is for the environments.yaml file without actually changing the environments.yaml file. This means that -e and the JUJU_ENV options still override the environment defined by switch.


$ juju help switch
usage: juju switch [options] [environment name]
purpose: show or change the default juju environment name
 
options:
-l, --list  (= false)
    list the environment names
 
Show or change the default juju environment name.
 
If no command line parameters are passed, switch will output the current
environment as defined by the file $JUJU_HOME/current-environment.
 
If a command line parameter is passed in, that value will is stored in the
current environment file if it represents a valid environment name as
specified in the environments.yaml file.
 
aliases: env


It works something like this:


$ juju env
Current environment: "amazon-ap"
$ juju switch
Current environment: "amazon-ap"
$ juju switch -l
Current environment: "amazon-ap"

Environments:
        amazon
        amazon-ap
        hpcloud
        openstack
$ juju switch amazon
Changed default environment from "amazon-ap" to "amazon"
$ juju switch amazon
Current environment: "amazon"
$ juju switch
Current environment: "amazon"


If you have JUJU_ENV set, then you get told that the current environment is defined by this.  Also if you try to use switch to change the current environment when the environment is defined by JUJU_ENV, you will get an error.


$ export JUJU_ENV="amazon-ap"
$ juju switch
Current environment: "amazon-ap" (from JUJU_ENV)
$ juju switch amazon
error: Cannot switch when JUJU_ENV is overriding the environment (set to "amazon-ap")

 

The Go Language - My thoughts

I've been using the Go programming language for almost three months now pretty much full time. I have moved from the Desktop Experience team working on Unity, into the Juju team. One of the main reasons I moved was to learn Go. It had been too long since I had learned another language, and I felt it was better to dive in, than to just mess with it on my own time.

A friend of mine had poked around with Go during a hack fest and blogged about his thoughts. This was just before I really started poking around. Interestingly the main issues that Aldo found frustrating with the errors for unused variables and unused imports, I have found not to be such a big deal. Passingly griping, sure, but not a big issue. Having the language enforce what is often a lint checker in other languages I see as an overall benefit. Also, even though I don't agree with the Go formatting rules, enforced by gofmt, it doesn't matter. It doesn't matter because all code is formatted by the tool prior to commit. As an emacs user, I found the go-mode to be extremely helpful, as I have it formatting all my code using gofmt before saving. I never have to think about it. One thing I couldn't handle though, was the eight character tabs. Luckily emacs can hide this from me.

;; Go bits. 
(require 'go-mode-load)
(add-hook 'before-save-hook #'gofmt-before-save)
(add-hook 'go-mode-hook (lambda () (setq tab-width 4)))
 
There are some nice bits to Go. I very much approve of channels being first class objects, and the use of channels to communicate between concurrently executing code. Go routines are also nifty, although I've not used them too much myself yet. Our codebase does, but I've not poked into all the nooks and crannies yet.

However there are several things which irritate the crap out of me with Go.

Error handling

The first one I guess is a fundamental design decision which I don't really agree with. That is around error handling being in your face so you have to deal with it, as opposed to exceptions, which are all to often not thought about. Now if our codebase is in any way representative of Go code out there, this is just flat out wrong. The most repeated lines of code in the codebase would have to be:

if err != nil {
  return nil
}

This isn't error handling. This is just passing it up to the chain, which is exactly what exception propagation does, only Go makes your codebase two to three times larger due to needing these three lines after every line of code that calls into another function. This is one thing I really dislike, but unlikely to change.

As a user of a language though, there are other things that could be added at the language level to make things slightly nicer. Syntactic sugar, as it is often known, makes the code easier to read.

If the language is wanting to keep the explicit handling of errors in the current way, how about some sugar with that.

Instead of

func magic() (*type, error) {
    something, err := somefunc("blah")
    if err == nil {
        return nil, err
    }
    otherThing, err := otherfunc("blah")
    if err == nil {
        return nil, err
    }
    return foo(something, otherThing), nil 
} 
 
we had some magic sugar, say a built-in method like raise_error, which interrogated the function signature, and returned zeroed values for all non-error types, and the error, and returned only non-error values, we could have this

func magic() (*type, error) {
    something := raise_error(somefunc("blah"))
    otherThing := raise_error(otherfunc("blah"))
    return foo(something, otherThing), nil 
}

The range function

There are several different issues I have with the range function.

1. range returns one or two parameters, but the language doesn't allow any user defined functions to return one or two parameters, range is super special
2. using range with a slice or array and getting a single value, doesn't give you the value, but instead the index - I never want this
3. there is no way to define range behaviour for a user defined type

These three things are mostly equal in annoyance factor. I'd love to see this change.

No generics

Initially I accepted this as a general part of the language. Shouldn't be a big deal right? C doesn't have generics. I guess I spent too long with C++ then.
My first real annoyance was when I had two integers, and I wanted to find the maximum value of the two. I go to look in the standard library and find math.max. However that is just for the float64 type. The standard response from the team was "it is only a two line function". My response is "that's not the point".

Since there is no function overloading, nor generics, there is no way with the language at this stage to make a standard library function that determines the maximum value of two or more numeric types, and return that maximum in the same type as the parameters. Generics would help here.

A second case for generics is standard containers. The primary container in Go at this stage is the map. So many places in our codebase we have map[string]interface{}. The problem with this is that you have to cast all values retrieved from the map. There is also no set, multi_map, or multi_set. Since there is no way to provide simple iteration for user defined types, you can't easily define your own set type and have simple iteration using range.

Interfaces that aren't explicitly marked as being implemented help in some ways to provide features provided by generic types and functions, but it is a poor substitute.

So far...

Learning Go has been an interesting experience so far. I like learning new things, and I'm going to be using Go for some time now with the current project. No doubt I'll have more to write about later.