Release management is a relatively new but rapidly growing discipline within software engineering. As systems, software development processes and resources become more distributed, they invariably become more specialized and complex.
Additionally, software products (especially web applications) are normally “never done” and often in a continual cycle of development, testing, and release, often running on evolving platforms with growing complexity.
Shipping products from development to system test and user acceptance test often require dedicated resources to oversee the integration and flow of development, testing, deployment, and support.
In this article, aimed at professionals involved in software development projects, we explore Release Management Best Practices and some of these processes. This article will be of interest to developers, test analysts, development managers, CTOs and teams responsible for shipping software products and will cover the following topics:
- A definition of productivity
- Coding Standards and Checklists
- Source Control
- Continuous Integration
- Promoting through Release Paths and Change Management
- Test Automation
Why Release Management
Let’s explore just some of the reasons why you might want to adopt a more formal release management process for your software projects.
Makes shipping product more predictable and reliable
Manually deploying releases is painful and always contains an element of risk. Introducing a formal release management removes a lot of the manual overheads associated with publishing and releasing software manually.
By introducing tools such as Visual Studio Team Services or Jenkins, you can automate your releases thereby making them more predictable and reliable.
Consider for a minute you’re deploying an ASP.NET website, you may have environment specific application settings in the web.config file (connection string and so on). Naturally, these need to change each time a release is published. Configuration changes like these can be a pain to track manually as you promote your website through the release path.
Release Management removes this pain and can handle this for you by parameterizing configuration values at runtime when the release is published.
Improves Software Quality
We’ll explore this is more detail in a subsequent section but incorporating Continuous Integration into your Release Management process can improve the quality of your software product. Each time code is checked in by a developer, the CI suite can invoke a suite of unit tests, execute and analyze the code quality (as per standards defined by the Team) and report this back, almost immediately, to the developer that performed the code check-in.
Focussing on what matters
Release Management allows each function to focus on the work that matters, i.e. developers can write code, test analysts can execute and write automated test plans. The near instant feedback loop that release management affords teams is hard to ignore. Developers can be notified if a change has broken something and test analysts can find bugs before software releases are formally shipped.
Quicker time to market
Automated, predictable builds, improved code quality, configuration management, automated unit tests all add to time savings and therefore quicker time to market. Of course, implementing formal release management and continuous integration processes into your development lifecycle can take time initially to set up but it can pay off massive dividends in the long run.
The first thing to decide on the environment architecture you want to adopt. Deployment architectures can vary somewhat, a common 4 tier architecture is DEV, TEST, STAGING, and PROD. Or to give them their full names, development, system testing, staging, and production.
Your software products should transition in order from DEV through to PROD. You may wish to add a User Acceptance Testing environment that sits between STAGE and PROD. Doing so lets the client validate the release post system test by your team and allows you to release software only after the client has deeded it “production ready”.
The development environment (dev) is the environment in which changes to software are developed, most simply an individual developer’s workstation. This differs from the ultimate target environment in various ways – the target may not be a desktop computer (it may be a smartphone, embedded system, headless machine in a data centre, etc.), and even if otherwise similar, the developer’s environment will include development tools like a compiler, integrated development environment, different or additional versions of libraries and support software, etc., which are not present in a user’s environment.
The purpose of the test environment is to allow either automated tests or human testers to exercise the new and changed code. After the developer accepts the new code and configurations through unit testing in the development environment, the items are moved to one or more test environments. Upon test failure, the test environment can remove the faulty code from the test platforms, contact the responsible developer, and provide detailed test and result logs. If all tests pass, the test environment or a continuous integration framework controlling the tests can automatically promote the code to the next deployment environment.
Staging is an environment for final testing immediately prior to deploying to production. It seeks to mirror the actual production environment as closely as possible and may connect to other production services and data, such as databases. For example, servers will be run on remote machines, rather than locally (as on a developer’s workstation during dev, or on a single test machine during the test), which tests the effect of networking on the system.
The primary use of a staging environment is to test all installation/configuration/migration scripts and procedures before they are applied to the production environment. This ensures that all major and minor upgrades to the production environment will be completed reliably without errors, in minimum time.
The production environment is also known as live, particularly for servers, as it is the environment that users directly interact with. Deploying to production is the most sensitive step; it may be done by deploying new code directly (overwriting old code, so only one copy is present at a time), or by deploying a configuration change. This can take various forms: deploying a parallel installation of a new version of code, and switching between them with a configuration change; deploying a new version of code with the old behavior and a feature flag, and switching to the new behavior with a configuration change that performs a flag flip; or by deploying separate servers (one running the old code, one the new) and redirecting traffic from old to new with a configuration change at the traffic routing level. These, in turn, may be done all at once or gradually, in phases.
Coding Standards and Checklists
Now that you understand the key environments that you should configure to support release management it’s time to turn your attention to the code itself. Coding standards and conventions serve several purposes:
- They help create a consistent “look” to code thereby allowing developers to focus on the content and not the layout.
- They allow developers to understand the source code more quickly as assumptions can be made based on previous experience. For example, in a service-oriented application, you’d have stored procedures, data access layers, logic layers and a service layer – all passing distinct objects such as DTOs and Data Contracts.
- Promotes code reuse and maintenance.
Ensuring your code is neatly formatted makes it easier to read. From a developer’s perspective, there’s nothing worse than being assigned a defect, “lifting the hood” on the code to find out you’re looking at a brownfield mess. What follows are some pointers that will ensure your code is nearly formatted and standardized in terms of look and feel.
- Use the default Code Editor settings (four-character indents for tabs etc).
- Only one statement per line.
- Only one declaration per line.
- At least one blank line between method definitions and property definitions.
- Use parentheses to make clauses in an expression obvious. For example, in calculations like this:
Some argue that your code shouldn’t need comments, I disagree. A few choice words or XML comments can give new developers an overview of what the underlying class or method is trying to achieve.
The only things to be mindful of are:
- Comments should be placed on a separate line, not at the end of a line of code.
- If you have too many comments, your code is too complex. Refactor your code.
- Create get only properties if said property is not be changed by the caller.
- Provide reasonable names for all properties.
- Provide default values for properties.
- Don’t throw Exceptions from property getters.
- Use a noun or noun phrase to name a class.
- Use Pascal case.
- Use abbreviations sparingly.
- Do not use a type prefix, such as C for class, on a class name. For example, use the class name FTPStream rather than CFTPStream.
- Do not use the underscore character (_).
- Use Pascal case
- Provide descriptive names, think about what it’s wider purpose is
- Use verbs or verb phrases
- Reveal the intent of the implementation, e.g. SendJSONData should be renamed to PostTweet
You can’t have an effective release management strategy without implementing a source control product. How else are you going to easily maintain numerous iterations of your product, implement features, maintain previous releases and merge code between concurrent development streams?
There are many source control products to choose from such as Visual Studio Team Services and Git.
Source control allows you to manage the workflow of code files as they pass from one developer to another. Depending on your preferences, source control products allow you to define both shared an exclusive file access to code files. If for example, you configure your source control product to enable “exclusive access”, then only on developer can work on that file at a time.
As developers check code in and out of the source control product, the data store builds an archive and snapshot of each file with an accompanying changeset id (assuming Visual Studio Team Services or TFS for the time being).
If required, it’s easy enough to restore any version-controlled file from this archive should you need to undo any breaking changes.
Depending on the number of developers and size of your software project, introducing a Branching strategy can help accelerate the build process and make maintenance tasks and feature development easier to manage.
The Branching mechanism allows developers or teams to work in isolation from the “main source code branch” and there are many strategies you can adopt.
A deep dive into branch planning is beyond the scope of this blog post but if you’re considering it, it’s important to consider the following:
- How big is the team?
- How often do you need to ship the product?
- How stable does the build need to be?
It’s important to only branch when you need to as Branching introduces additional tasks such as source repository maintenance and merging tasks. If you’re the only developer in your team, it’s unlikely you’ll need to implement a branching strategy, that said, you might decide to maintain one branch for hotfixes and another for features.
Some common branching scenarios and their definitions:
- Release – Branch to stabilize code you want to release. You can branch before release, avoiding the need to lock down the main branch.
- Maintenance – Branch to maintain a previously released build.
- Feature – Branch to isolate feature development that might make the rest of the project unstable.
- Team – Branch to isolate sub-teams so that they can work without being subject to breaking changes, or that they can work towards unique milestones. These are very like feature branches.
Once you have your application code safely stored in your source control provider, another important element of in release management is continuous integration (CI). CI is the automated process of integrating code changes and merging all developer code in the source control repository. It provides near-instant feedback as to the integrity of the build and gives development and QA teams confidence in the current build.
Developers should strive to check in code daily, no code should be left checked out overnight – it may break the build!
Cloud platforms such as Windows Azure allow you to configure workflows which integrate with source control products like BitBucket, GitHub and Visual Studio Team Services. Azure will then pull the most recent updates from your project once committed to any of these source control providers.
Your CI strategy should be self-testing. Developers should write unit tests that automatically get invoked during an automated build.
Database updates naturally have to be synchronized with code deployments and deployed at the same time. These can be hard to maintain, rollback scripts should be considered in the event that a build/release initiated by the CI platform fails, otherwise you could invalidate the integrity of the database schema or data itself.
Promoting through Release Paths and Change Management
This deserves a series of blog posts but let us cover the basics. With the environments mentioned earlier (DEV, SYSTEM TEST, STAGING, and PROD) in mind, your software should only transition through each of the environments when “signed off” by the respective function such as development or system test.
As a team, you need to decide on a workflow that best suits your project. At a high- level, a typical set of release management processes include:
Each function will naturally have its own set internal checkpoints. Your developers will manually test code behaves as expected. They can also supplement this with automated unit tests that can be run on demand. Test Analysts will execute test plans and may also write automate test plans using products such as Selenium. Processes within each function serve to act a checkpoint prior to transitioning your software from one environment to the next. Ignore this phased approach at your peril!
Test Automation and Continuous Integration go hand in hand. Always strive to automate as many unit tests as possible. If your team find themselves routinely testing a suite of use cases then automate these. Yes, it may take some time to initially setup your project to adopt TDD approach but the benefits outweigh this initial overhead. By automating unit tests, you can test an entire application with just a few clicks and get almost instant feedback thereby allowing you to pinpoint code failures in your application.
From a developer’s perspective, automated unit tests save you from waiting until the product has been promoted to a system test environment for QA analysts. They also serve to prevent your product from regressing when change requests are implemented. If you’re working on a legacy or “brownfield” application that doesn’t contain any automated unit tests, tackle the high traffic risk areas first. Don’t worry about having 100% test coverage initially, otherwise, the task may be too overwhelming.
After your product has passed all developer and system tests (and UAT if you have that environment too), you need to release to production.If this development has been done in a separate branch out-with the root/master branch, then these changes must be merged back into the root.
Most popular source control tools will prompt you to resolve any conflicts in a difference review window, this lets you amalgamate code from all developers. You need to take care during this exercise as it’s this new merged version that will be promoted into your production environment.
- Only merge the code from the development branch to the master branch when you want to deploy, don’t try to merge anything in advance otherwise you’ll give yourself unnecessary work.
- Schedule major releases to production on scheduled date/time which everyone is aware of, with the aid of analytics tools, you can establish which time of the day is best. You’ll want to pick a time when there aren’t many active users.
- If you can, present a “down for maintenance page” for the user community.
- Ensure you have key personnel around for a few hours post upgrade should you need to apply a hotfix.
- Use your source control provider to generate release notes that contain all new features and share with the team and user community
- Circulate documentation or implementation guides across each function (test/dev ops etc.) that lets them know exactly what needs deployed and how.
- Delete any additional branches after you have completed your merge to keep the source control repository clean.
- After the deployment is complete, perform key smoke tests to check everything is working as expected in the production environment.
- As a rule of thumb, don’t perform upgrades on a Monday or Friday!
With your software now out in the wild its over to the users, and you can be assured that users will use your product in ways you never anticipated. They will no doubt raise issues or defects and you need to manage them accordingly.
For example, with VSTS, you can raise defects on your Kanban Board against the original User Story or Product Feature. This makes it easier for your team to track the original change requests and lets them identify any code changes and respective test scripts that were written and executed.
So there you have it, release management and some best practices. There’s no silver bullet or one size fits all approach but by applying at some of the techniques and ideas we’ve explored it will help smoothen your software development release cycle.
We hope you’ve enjoyed reading this blog post, feel free to leave any comments or suggestions.