6 months on Unity

We have just finished another design sprint prior to UDS-P.

While talking with some others I realise that I have worked on Unity for six months, and not changed a single pixel on the output.  No graphical changes, no moving widgets, no changing colours.

So what have I been doing?

First step was getting some new coding standards accepted by the team, which was much easier than I was expecting.

I added some property classes to nux, and did some general clean up in the code of nux and unity.

Refactored the indicator internals for the panel service which started off the shared unity core library for sharing code between the 2D and 3D code-bases.

Then I focused primarily on fixing memory leaks and crashes.

Once we hit final freeze, I did a little more refactoring internally, and now we are on to Precise Pangolin.

Properties in C++

Once you have done any development in a language that natively supports properties, like Python or C#, going back to C++ and not having them often feels like a real pain. I've just proposed my second attempt at C++ properties for the nux library.

This change leans heavily on a paper written by Lois Goldthwaite: SC22/WG21/N1615 - C++ Properties -- a Library Solution. I added change notifications using sigc++. I found that using sigc::slot was nicer than templating the properties on the class and member function pointer. This also meant that I could provide a way for a simple property to get its own custom setter method while still having a sensible default.

Compiling C++ templates still gives absolutely horrendous error message that can take a while to mentally parse. I guess one advantage of having done a lot of template programming in the past is that I don't get too phased by copious quantities of error messages, especially for templates, as for one example today, I had just forgotten to change a template arg in a test function, and got way too many lines to sensibly look at. One benefit of that was it caused me to look at what I was doing, and I ended up simplifying my tests a little more.

Thank you Lois for the time you spent writing up the C++ properties proposal, it was a fantastic starting point for me.

Getting back into C++

I have to admit that unit testing in C++, even with google-test, is so much more of a PITA than in python. Especially when checking string output. Simple string matching using split and regular expressions has really spoiled me.

Another thing that I've noticed is that I spend more time thinking about object design, and what should that object really be able to do, and what should it allow others to do to it.

It is an interesting time as I realise how much I still have to learn for our current domain. Most of my previous C++ experience has been on server side processing. Drawing stuff on the screen, real end user stuff, is still relatively new for me.

DX Sprint

What a week. I've spent the last week in Budapest sprinting with the rest of the Desktop Experience (DX) team. This week was also my first official week with the DX team as I have moved now from the Launchpad team to the DX team. This was a good week. I had met some of the DX team before at other company get togethers, but not really talked to them much. A really important part of any new job is meeting the people that you are working with. This is always ends up happening when you work in the same office with them, but for a distributed company like Canonical, you can end up working on the same team with people that you don't get to meet for months.

It was great to meet different sub-teams of DX, especially those that I'll be working closely with. I'll be hanging out in the #ayatana irc channel now, but I'll also still be in #launchpad and #launchpad-dev. There are some very interesting plans for oneiric, and it will be interesting to see how much we can end up getting done. In the normal way we have "too much to do" and the gauntlet has been thrown.

So... I'll be hacking on the unity stack. Please don't ask me to fix any particular bugs yet as it'll no doubt take me time to find my way through the code :-)

Now for the 40+ hour journey home.

Launchpad and stacked branches

As I'm sure most of you are aware, Launchpad hosts Bazaar branches. One early design decision that we had on Launchpad was that branches should be able to be renamed, moved between people and projects without limitation. This is one reason why each branch that was pushed to Launchpad had a complete history. We wanted to make sure that there weren't any problems where one person was blocking another pushing branches, or that people weren't able to get at revisions that they shouldn't be able to.

The Bazaar library, bzrlib, gives you awesome power to manipulate the internals giving you access to the repository of revisions through the branch. This can be a blessing and a curse, as if you have private revisions, then they can't be in a public repository.

Having a complete copy of the data for every branch became a severe limitation, especially for larger projects, of which Launchpad itself is one. A solution to this was a change in Bazaar itself that allowed a fallback repository which contained some of the revisions. This is what we call stacked branches. The repository for the branch on Launchpad has a fallback to another repository, which is linked to a different Launchpad branch. We ideally wanted all of this to be entirely transparent to the users of Launchpad. What it means is that when you are pushing a new branch to Launchpad, the bzr client asks for a stacked location. If there is a development focus branch specified for the project, this is then offered back to the client. The new branch then only adds revisions to its repository that don't exist in the development focus branch's repository. This makes for faster pushes, and smaller server side repositories.

The problem though was what do we specify the stacked on location to be? When we created the feature, we used absolute paths from the transport root. What the mean was that we stored the path aspect of the branch. For example, lp:wikkid gets translated to bzr+ssh://bazaar.launchpad.net/~wikkid/wikkid/trunk or http://bazaar.launchpad.net/~wikkid/wikkid/trunk depending on whether the bzr client knows your Launchpad id. The absolute path stored would be /~wikkid/wikkid/trunk. This information was then stored in the branch data on the file system.

The problem however was that the web interface allows you to rename branches. The actual branch itself on disk is referred to using a database id, which is hidden from the user using a virtual file system which has rewrite rules for http and at the bazaar transport level. However since the stacked on location refers to a full branch path, changing any part of that, whether it is the branch owner, branch name, or the project or package that the branch is for, would cause any branches stacked on that changed branch to break, bug 377519.

In order to fix this we had to change the location that the branch is stacked on to be independent of the branch path. The best solution here is to use the database id. I really didn't want to expose the user to this opaque id, but one opaque id is as good as another. Now when pushing branches to Launchpad, when it is creating a stacked branch you'll see a message like:

Created new stacked branch referring to /+branch-id/317141.

Existing branches still have their old branch paths saved for now. We'll run a migration script early next week to fix all these up, and hopefully we'll have seen the last of this bug.

Blueprint magic

Just landed in qastaging is some itch-scratching work I did adding AJAX widgets to the main blueprint page. This has passed QA and will end up in production with the next no-downtime rollout (which should be real soon now).

This work was adding a bunch of the lazr-js wrapped widgets. Now we can update the following without reloading the primary page:

• title - the H1 heading
• summary
• whiteboard
• assignee
• drafter
• approver
• priority
• implementation status
• definition status

Using the new custom events that the page raises when the context object changes (using YUI magic and API PATCH requests), when you change the title of the blueprint, the document title (title bar) and the breadcrumbs also change. When the implementation status is updated, the overall status updates, and the "started by" and "completed by" are shown or hidden as appropriate.

This is work that I've wanted to see done for almost a year, and recent other changes I've done adding more widget wrappers and javascript goodness have made this possible without adding copious amounts of custom javascript.

A side-effect of these changes is that there are now more fields exported over the API for blueprints.

Announcing sloecode

Sloecode is a simple Bazaar hosting project.

Last year I tried to set up my home server to offer a place for three people to have shared, private access to a bazaar repository for a project. I found it really ackward. I felt that there had to be a simpler way.

Launchpad is an awesome place to host Bazaar branches. However Launchpad is for open source projects and personal branches are public.

I was approached towards the end of last year by Thomi who suggested we create a simple Bazaar hosting project for Otago Polytechnic to provide a place for the students to host their senior year projects. Since it was something I also cared about, and hadn't found a reasonable solution elsewhere, I agreed to help.

Our initial requirements went something like this:

• Users defined in a database and no local login needed on the hosting server
• Private repositories for the students to host personal branches
• Lecturers should be able to see the repositories of the students
• Projects have private repositories only visible to the members of the project team, and the Lecturers
• Simple URLs for getting access to the branches
• Scalability isn't a priority

In the end we went with the new pyramid libraries for the web application. We tried briefly with django, but I found the framework blocked me whenever I tried to do something. I had worked a lot with zope, and repoze.bfg was something we looked at. When repoze.bfg and turbogears merged into pyramid we felt that we had found a good match for us.

Installing and running the sloecode server is still a bit messy. We'd love to get it to the stage we you can just run an installer and magic happens. But we are not there yet.

The application server runs on the same machine as the filesystem hosting the branches and repositories. Shared repositories are created for each user as they are added. When a project is created so is its shared repository, and trunk branch which is set to append only.

Users log in and add their public SSH key. Users should also install the bzr-sloecode client plugin. Initially the plugin had hard coded site names for the Otago Polytechnic. But looking forward I decided that we should just use an environment variable. This allows access to the sloecode server using a short hand url.

• sc:my-project - gets access to the trunk branch of the project called my-project
• sc:my-project/trunk - also gets access to the trunk branch
• sc:my-project/some-branch - gets access to the branch some-branch of my-project
• sc:~myid/personal-branch - gets access to the branch personal-branch on my repository

The sc translation is done client-side, like the lp expansion for Launchpad.

We run a custom twisted SSH server on the hosting machine. This does the SSH key lookup against our user database, and doesn't allow password login. It also restricts the commands that can be executed on the server side to what it expects Bazaar to ask for. No shells are given. The server then launches a subprocess that uses a wrapped smart server that has a virtual filesystem to translate the requested paths to the underlying filesystem. This operates in a similar way to Launchpad, but much more trivially. The repository hosting configuration just needs two paths: one for the project repositories, and one for the personal repositories. The smart server code also handles the privacy aspect, not allowing unauthorized users access to repositories they shouldn't see.

The future

We'd like to have some form of code browsing functionality added. Whether we use loggerhead, or something else is still up in the air. We'd also like to integrate wikis for the projects. Wikkid would be a good fit (another of my personal projects) and could spur on some of the stalled development.

Ease of installation and configuration is another aspect we'd like to tidy up.

If you are interested in getting involved join the sloecode-dev team on Launchpad as it handles the development mailing list.