6 months on Unity

We have just finished another design sprint prior to UDS-P.

While talking with some others I realise that I have worked on Unity for six months, and not changed a single pixel on the output.  No graphical changes, no moving widgets, no changing colours.

So what have I been doing?

First step was getting some new coding standards accepted by the team, which was much easier than I was expecting.

I added some property classes to nux, and did some general clean up in the code of nux and unity.

Refactored the indicator internals for the panel service which started off the shared unity core library for sharing code between the 2D and 3D code-bases.

Then I focused primarily on fixing memory leaks and crashes.

Once we hit final freeze, I did a little more refactoring internally, and now we are on to Precise Pangolin.

Properties in C++

Once you have done any development in a language that natively supports properties, like Python or C#, going back to C++ and not having them often feels like a real pain. I've just proposed my second attempt at C++ properties for the nux library.

This change leans heavily on a paper written by Lois Goldthwaite: SC22/WG21/N1615 - C++ Properties -- a Library Solution. I added change notifications using sigc++. I found that using sigc::slot was nicer than templating the properties on the class and member function pointer. This also meant that I could provide a way for a simple property to get its own custom setter method while still having a sensible default.

Compiling C++ templates still gives absolutely horrendous error message that can take a while to mentally parse. I guess one advantage of having done a lot of template programming in the past is that I don't get too phased by copious quantities of error messages, especially for templates, as for one example today, I had just forgotten to change a template arg in a test function, and got way too many lines to sensibly look at. One benefit of that was it caused me to look at what I was doing, and I ended up simplifying my tests a little more.

Thank you Lois for the time you spent writing up the C++ properties proposal, it was a fantastic starting point for me.

Getting back into C++

I have to admit that unit testing in C++, even with google-test, is so much more of a PITA than in python. Especially when checking string output. Simple string matching using split and regular expressions has really spoiled me.

Another thing that I've noticed is that I spend more time thinking about object design, and what should that object really be able to do, and what should it allow others to do to it.

It is an interesting time as I realise how much I still have to learn for our current domain. Most of my previous C++ experience has been on server side processing. Drawing stuff on the screen, real end user stuff, is still relatively new for me.

DX Sprint

What a week. I've spent the last week in Budapest sprinting with the rest of the Desktop Experience (DX) team. This week was also my first official week with the DX team as I have moved now from the Launchpad team to the DX team. This was a good week. I had met some of the DX team before at other company get togethers, but not really talked to them much. A really important part of any new job is meeting the people that you are working with. This is always ends up happening when you work in the same office with them, but for a distributed company like Canonical, you can end up working on the same team with people that you don't get to meet for months.

It was great to meet different sub-teams of DX, especially those that I'll be working closely with. I'll be hanging out in the #ayatana irc channel now, but I'll also still be in #launchpad and #launchpad-dev. There are some very interesting plans for oneiric, and it will be interesting to see how much we can end up getting done. In the normal way we have "too much to do" and the gauntlet has been thrown.

So... I'll be hacking on the unity stack. Please don't ask me to fix any particular bugs yet as it'll no doubt take me time to find my way through the code :-)

Now for the 40+ hour journey home.

Launchpad and stacked branches

As I'm sure most of you are aware, Launchpad hosts Bazaar branches. One early design decision that we had on Launchpad was that branches should be able to be renamed, moved between people and projects without limitation. This is one reason why each branch that was pushed to Launchpad had a complete history. We wanted to make sure that there weren't any problems where one person was blocking another pushing branches, or that people weren't able to get at revisions that they shouldn't be able to.

The Bazaar library, bzrlib, gives you awesome power to manipulate the internals giving you access to the repository of revisions through the branch. This can be a blessing and a curse, as if you have private revisions, then they can't be in a public repository.

Having a complete copy of the data for every branch became a severe limitation, especially for larger projects, of which Launchpad itself is one. A solution to this was a change in Bazaar itself that allowed a fallback repository which contained some of the revisions. This is what we call stacked branches. The repository for the branch on Launchpad has a fallback to another repository, which is linked to a different Launchpad branch. We ideally wanted all of this to be entirely transparent to the users of Launchpad. What it means is that when you are pushing a new branch to Launchpad, the bzr client asks for a stacked location. If there is a development focus branch specified for the project, this is then offered back to the client. The new branch then only adds revisions to its repository that don't exist in the development focus branch's repository. This makes for faster pushes, and smaller server side repositories.

The problem though was what do we specify the stacked on location to be? When we created the feature, we used absolute paths from the transport root. What the mean was that we stored the path aspect of the branch. For example, lp:wikkid gets translated to bzr+ssh://bazaar.launchpad.net/~wikkid/wikkid/trunk or http://bazaar.launchpad.net/~wikkid/wikkid/trunk depending on whether the bzr client knows your Launchpad id. The absolute path stored would be /~wikkid/wikkid/trunk. This information was then stored in the branch data on the file system.

The problem however was that the web interface allows you to rename branches. The actual branch itself on disk is referred to using a database id, which is hidden from the user using a virtual file system which has rewrite rules for http and at the bazaar transport level. However since the stacked on location refers to a full branch path, changing any part of that, whether it is the branch owner, branch name, or the project or package that the branch is for, would cause any branches stacked on that changed branch to break, bug 377519.

In order to fix this we had to change the location that the branch is stacked on to be independent of the branch path. The best solution here is to use the database id. I really didn't want to expose the user to this opaque id, but one opaque id is as good as another. Now when pushing branches to Launchpad, when it is creating a stacked branch you'll see a message like:

Created new stacked branch referring to /+branch-id/317141.

Existing branches still have their old branch paths saved for now. We'll run a migration script early next week to fix all these up, and hopefully we'll have seen the last of this bug.

Blueprint magic

Just landed in qastaging is some itch-scratching work I did adding AJAX widgets to the main blueprint page. This has passed QA and will end up in production with the next no-downtime rollout (which should be real soon now).

This work was adding a bunch of the lazr-js wrapped widgets. Now we can update the following without reloading the primary page:

• title - the H1 heading
• summary
• whiteboard
• assignee
• drafter
• approver
• priority
• implementation status
• definition status

Using the new custom events that the page raises when the context object changes (using YUI magic and API PATCH requests), when you change the title of the blueprint, the document title (title bar) and the breadcrumbs also change. When the implementation status is updated, the overall status updates, and the "started by" and "completed by" are shown or hidden as appropriate.

This is work that I've wanted to see done for almost a year, and recent other changes I've done adding more widget wrappers and javascript goodness have made this possible without adding copious amounts of custom javascript.

A side-effect of these changes is that there are now more fields exported over the API for blueprints.

Announcing sloecode

Sloecode is a simple Bazaar hosting project.

Last year I tried to set up my home server to offer a place for three people to have shared, private access to a bazaar repository for a project. I found it really ackward. I felt that there had to be a simpler way.

Launchpad is an awesome place to host Bazaar branches. However Launchpad is for open source projects and personal branches are public.

I was approached towards the end of last year by Thomi who suggested we create a simple Bazaar hosting project for Otago Polytechnic to provide a place for the students to host their senior year projects. Since it was something I also cared about, and hadn't found a reasonable solution elsewhere, I agreed to help.

Our initial requirements went something like this:

• Users defined in a database and no local login needed on the hosting server
• Private repositories for the students to host personal branches
• Lecturers should be able to see the repositories of the students
• Projects have private repositories only visible to the members of the project team, and the Lecturers
• Simple URLs for getting access to the branches
• Scalability isn't a priority

In the end we went with the new pyramid libraries for the web application. We tried briefly with django, but I found the framework blocked me whenever I tried to do something. I had worked a lot with zope, and repoze.bfg was something we looked at. When repoze.bfg and turbogears merged into pyramid we felt that we had found a good match for us.

Installing and running the sloecode server is still a bit messy. We'd love to get it to the stage we you can just run an installer and magic happens. But we are not there yet.

The application server runs on the same machine as the filesystem hosting the branches and repositories. Shared repositories are created for each user as they are added. When a project is created so is its shared repository, and trunk branch which is set to append only.

Users log in and add their public SSH key. Users should also install the bzr-sloecode client plugin. Initially the plugin had hard coded site names for the Otago Polytechnic. But looking forward I decided that we should just use an environment variable. This allows access to the sloecode server using a short hand url.

• sc:my-project - gets access to the trunk branch of the project called my-project
• sc:my-project/trunk - also gets access to the trunk branch
• sc:my-project/some-branch - gets access to the branch some-branch of my-project
• sc:~myid/personal-branch - gets access to the branch personal-branch on my repository

The sc translation is done client-side, like the lp expansion for Launchpad.

We run a custom twisted SSH server on the hosting machine. This does the SSH key lookup against our user database, and doesn't allow password login. It also restricts the commands that can be executed on the server side to what it expects Bazaar to ask for. No shells are given. The server then launches a subprocess that uses a wrapped smart server that has a virtual filesystem to translate the requested paths to the underlying filesystem. This operates in a similar way to Launchpad, but much more trivially. The repository hosting configuration just needs two paths: one for the project repositories, and one for the personal repositories. The smart server code also handles the privacy aspect, not allowing unauthorized users access to repositories they shouldn't see.

The future

We'd like to have some form of code browsing functionality added. Whether we use loggerhead, or something else is still up in the air. We'd also like to integrate wikis for the projects. Wikkid would be a good fit (another of my personal projects) and could spur on some of the stalled development.

Ease of installation and configuration is another aspect we'd like to tidy up.

If you are interested in getting involved join the sloecode-dev team on Launchpad as it handles the development mailing list.

More responsive recipe builds

SteveK has recently landed a change that at the same time makes our admins happy, and should give our recipe users more responsive builds.

Daily builds were previously kicked off at 23:00 UTC. A rather arbitrary time that put quite a load on the build farm as recipes became more popular.

The new change has our script to kick of daily recipe builds run much more often, and uses a cunning mix of magic and smarts to run the jobs (ok, not so much magic).

When new changes are pushed to Launchpad for branches that are used in source package recipes, the recipe is marked as dirty. Dirty recipes are candidates for daily builds. If a recipe has not been built into the daily build PPA within the last 24 hours a new recipe build job is created very soon after Launchpad notices the new changes. If there has been a build, then the new build job isn't created until 24 hours has past since the last build. Manual builds into other PPAs do not affect the daily build time check.

What this means is that if you have a daily build recipe, and tend to change the branch less often than every day, then when you do change it, the package is built much more quickly.

Refactoring Launchpad's lazrjs widgets

Just landed (r12285 on devel) is some refactoring I've been doing on the lazr-js widget wrappers. After hacking on the picker at Dallas, I felt the other editor widgets needed some attention as well. The primary documentation can be found in the source tree at lib/lp/app/doc/lazr-js-widgets.txt, in fact this is probably the best thing to read.

This new changes did several things:

• fixed the multi-line editor so you don't need extra HTML tags surrounding the widget in the page template
• changed the widgets to actually use page templates to render the content instead of fiddling with string substitution
• refactored the class inheritance - the multi-line editor no longer inherits from the single line editor, but instead they both inherit from a common base
• it changed the way the editable attribute is defined - from being the name of the attribute, to being the field from the interface
• the initial text for the multi-line editor is now determined from the object itself, not a parameter passed through to the constructor
• the permission checks were unified, which fixed a problem with the text editors using mutator functions exposed through lazr.restful
• you now have to be explicit about the id used for the HTML tag surrounding the editor - this wasn't much of an ask as all call sites were already doing this

All in all I'm pretty happy with this change.

Code Blue at the Thunderdome

I am sitting in the hotel at the end of the Launchpad Thunderdome. Really this was just a sprint for all of the Launchpad developers, but someone decided it needed a neat name, and Thunderdome stuck.

It is at this sprint that the Launchpad team is transitioning away from the old application based teams to "squads". There are five squads, each given a colour for now: Blue, Green, Orange, Yellow, and Red. So my "Code" team has been split up and I have a new squad "Blue". Before we even switched to the squads, on of my new members resigned to move to something new, so I have an opening. Actually right now we have two openings. One squad is primarily based in Australasia (with one in New York) and the other has people from North America and Europe.

The majority of our time has been spent getting to know our new squads. Three of the squads are working on features, and two are on maintenance. The Blue squad (called "Code Blue" from now) is finishing of the recipe feature. This allows packages to be built automatically directly from a source package assembled from one or more branches on Launchpad. This feature was started over a year ago and needs to be finished off. Most of what we are doing now is just polishing the user experience and interface. The feature has been available since around the middle of the year, and we have almost 250 recipes that are configured to build packages automatically. Once we finish this off Code Blue will move on to maintenance until some other feature teams finish up.

We have done some pretty impressive work on the webservice this week. Primarily due to trying to make nice AJAX popups for changing the owner and PPA of a recipe. This resulted in much yak shaving, and refactoring. The work we did make it trivially easy to add an AJAX popup to any place on Launchpad where a single value is chosen from a list of options. We also fixed a number of bugs in lazr.restful and very soon there will be a release that will add a 'web_link' property to the entries in launchpadlib that will refer to the actual object on Launchpad.