Originally posted on the Technology at GDS blog.

Over the course of the past few years many teams across government have begun publishing their code under open source licenses. That’s a change that’s been pushed by the Digital by Default Service Standard but it’s just as much a result of the change in development culture that’s permeating the civil service, with teams eager to share their work.

Despite that, releasing code is still a challenge for many teams and it can be one of the trickier areas of service assessments for many. Recently, Alex wrote about the work that the GOV.UK team did to open up their infrastructure code. Another project GDS has been deeply involved in that has been looking at opening up its code is the Register to Vote service.

Register to Vote started out in private because it was designed to support a policy that hadn’t yet been announced and between the need to move at pace and the complexity of connections with some other parts of government the team haven’t to date been able to go back and really address that.

The service is now maintained outside of GDS but we continue to liaise with the team and in order to support the opening of the code we set up a workshop with representatives of the service team, GDS and CESG to work through piece by piece and think through what needed to be considered.

The approach

We deliberately had a mixture of people in the room, some with deep knowledge of the service, others who were new to it but who have a solid understanding of software development, architecture and security. That mixture made sure there were people who could answer detailed, specific questions and had been thinking about this for a while, and others who could challenge assumptions and bring fresh perspectives.

  • We started out by discussing some broad considerations that apply to most projects. That helped us set the scene and also to understand each others’ perspectives.
  • We then went step by step through the architecture of the system, looking for any particular issues around specific components, making notes as we went along.
  • Finally we went back through and looked at a few issues that had come up repeatedly. In this case the main thing that came up here was around infrastructure and test suites.

This post is based on notes I took through the session, slightly reformatted for clarity.

Why do we want to open the code?

We started by reviewing the reasons that we would want to open the code in the first place. There’s a clear requirement to do so to meet the service standard but it was important to us that everyone in the room had more understanding of why that requirement is there. We quickly listed out four reasons for opening up as much of our code as possible (in no particular order)

  • Clarity about intellectual property. Opening our code forces us to think about the licenses applied to it and to know exactly who has what rights. This helps us reduce vendor lock-in as we have flexibility to change suppliers or move to in-house teams without needing to pay to move the code.
  • Transparency and audit-ability. Where the work is being done in public it’s immediately clear what work is being done, and also makes it much easier to consult with a wider community for input, whether that’s experts in forums like StackOverflow, vulnerability researchers, or others.
  • Provide examples for other teams. Whether or not our code is directly reusable it’s helpful for teams to have examples of how other people have solved common problems. Sometimes that’s government-specific, but often it’s also just about developing the knowledge across the development community.
  • Reusability for other people. Some of the code we use will be directly useful to other people. It can be hard to invest the time in extracting that and packaging it, but working in public gets us thinking about that more clearly, and gives other teams the opportunity to say which bits they’d like to make use of.

It was interesting to note that for two of these, being able to see the process you’ve followed is useful as well as the end result.

It’s much easier to start in public

As we discussed the particular challenges for Register To Vote we kept coming back to the fact that it’s much harder to make something public after the fact than to start out that way. When you’re working in private it’s easy to be a little lazy about what you disclose in the code and the commit messages you make. You can often make a set of assumptions about who will be reading it and fall into jargon that makes sense to them rather than to a wider group. Those assumptions aren’t safe even when working in private as you never know who will be working on the code in future, but working in the open should challenge them more quickly.

Taking an existing repository of code and going back through the history to review every past commit can be a hugely time consuming challenge and it’s easy to miss something you didn’t mean to make public. As the GOV.UK team explained, careful use of our tools can help with that but it can still be very time consuming. With all that work to do in one go, it can be difficult to prioritise the effort when you also have changes to make based on what you’re then learning about your users (or to address technical debt).

If you start out in public then you can think about that as you go along. If not, you have to make the choice between investing the time in sanitising the history or taking a snapshot and losing all that history, which may also lose some of the value. It won’t always be possible to start in public (as it wasn’t for Register to Vote) but we’d encourage all teams to at least try and think as if their code was public and conduct code reviews accordingly to avoid problems later on.

Considering security

Security is often the word used to caution against opening up code. As we worked through the Register to Vote architecture we kept coming back to what security issues could be introduced or exacerbated by opening up the code.

GDS has blogged before about when it’s okay not to open source code but the areas we particularly focussed on here were:

  • Code that you are not confident in the quality of (eg code that is yet to be reviewed). This should be the easiest to manage as code that hasn’t been reviewed shouldn’t be deployed, but it’s worth checking that your processes prevent that
  • Bespoke functions that are designed to reduce fraud or mitigate attacks, though quite what that means will vary from service to service
  • Configuration rules for security appliances. It may be worth sharing heavily used rules for the wider community but generally you won’t want to share it all.
  • Code that is known to bring your infrastructure down (eg certain types of performance testing scripts)
  • And of course, our keys, usernames and passwords

There is certainly a chance that opening up your code, particularly if you open it all, can make potential attackers’ lives easier - they can conduct their reconnaissance and craft their attacks on local copies of the system rather than having to test against your live system and potentially triggering your monitoring to give you advance warning. Hiding your code isn’t a strong defense and our aspiration should always be for the system as a whole to be solid enough that this isn’t a serious threat, but this is very much an area of risk that should be flagged and considered carefully.

We talked at length about protective security (monitoring, fraud detection, etc) and approaches to performance and security testing. When working through these things it’s very important to put yourself in the mindset of an attacker and ask as an attacker is there anything I’d learn from the open code that would let me bypass triggers/fraud detection/etc? Tests are a particularly thorny issue as good tests will push systems to their extremes and may well reveal the elements you consider fragile. While your development focus should be to remove that fragility, that does take time. As with everything else we see value in making tests public, but sometimes it’s better to open source the framework you use for testing and a few examples, and allow for holding back some of the more invasive tests.

And when, as with Register to Vote, you’re calling out to third-party systems that have private interfaces it’s polite to discuss with those third-parties about what they’re comfortable with you opening up. If they’re not, it may be worth talking to other teams who are likely to be using the same interfaces and either working together to change the approach, or working out some other way of collaborating on the code.

Infrastructure as code

We talked for quite a while about “infrastructure code”, by which we meant everything from puppet modules to firewall configuration.

The general sense here, as with a lot else, was that we could open up most of the code but that we’d be more deliberate about extracting specific modules. Partly that’s because we could more immediately see the potential for re-use of some of the puppet modules we’re looking at, but it also seemed like releasing the puppet code as a single lump would expose a particularly large amount of information about the configuration. Clear, strong separation between code and configuration is always good practice, but absolutely vital when you’re planning to make that code open. Careful attention to making your puppet code modular is really helpful for making sure you think that through. It’s worth noting that while GOV.UK opted to open up their main puppet repository, they’d extracted and released a number of modules beforehand.

Firewall configuration, and in particular the rules used in a Web Application Firewall, also seemed like an area to be particularly cautious as they may well contain details of specific attacks (and sometimes attackers) that would affect our ability to defend the system. That said, most of these tools have sets of common default rules and if we discover useful new rules that aren’t deeply specific to our systems there’s no reason that we shouldn’t contribute them back to the community in some way.

The wider context

Things that sit outside your specific code also have an impact on how to approach opening up your code.

We need to be confident that our code will be managed with integrity, particularly if we decide to make the public version of the code the definitive one. If we’re using code hosting and management tools that are run by other people, are we confident of their processes and that only agreed changes will make it in? The Cloud Security Principles apply here as much as anywhere.

Working in the open also raises the importance of being careful about how you manage your dependencies and keep them up to date. If a vulnerability appears in one of the third-party libraries you use, will you know about it and be able to update your code before someone can discover that you’re exposed via simple use of a search engine?

This is another area where being open doesn’t create a new threat but it does potentially exacerbate it. In this case we’re confident that our patching policies are responsive enough that we’re covered, but raising it did lead to a useful conversation about safe dependency management, when we should be keeping our own repository of common packages with code review before we accept new versions, etc.

Similarly, if you find a vulnerability in your own open code, you’ll need a way to make sure you’ve deployed that code into the production service as quickly as possible and don’t accidentally disclose the details via commits to your open repositories until you’ve fully dealt with them. For Register to Vote the approach to handle that by doing their primary work in a private copy with the changes pushed out to a public version after a period of time, for most GDS projects we work completely in public but have scripts available that let us switch to working in private copies for a period of time should the need arise. There are trade-offs involved in either approach and it’s important to carefully consider priorities when making those decisions.

Learning from the experience

The session felt like a helpful one and is something we’d encourage all teams to do, probably at specific points in the evolution of their architecture such as when new applications or services are being added, or aligned with the Service Standard stages of alpha/beta/live.

As teams get more experienced at running the sessions they should get shorter and more focussed. This one took us quite a while as we wanted to start from first principles and were looking at an already running service piece by piece. If it were part of a regular rhythm we could focus more specifically on what had changed since last time we’d met.

The external input was really helpful and well worth pulling in from time to time, but not necessarily essential every time you do a review of this sort. In this case we actually spotted a couple of issues that the team went away to fix.

What are Register To Vote doing next?

In the short-term the IER team are planning to open source the core Register to Vote API code and associated documentation.

Once that’s done, they’ll move on to making as much of the rest of the code public as possible. That won’t happen overnight given the amount of due diligence required to ensure that only ‘appropriate’ code is made available but following the workshop it’s much clearer what they need to look out for and they will work to do this alongside the ongoing responsibility of supporting the live service.