We work in a time where the lines between products and services are merging fast. Almost anything in technology has a drive to be “X As A Service“. Whether we are talking about Software as A Service, Infrastructure as A Service, or whatever flavor of the week is popular, we need to look at adopting traditional Service Release Management Strategies to support our projects. As more organizations adopt DevOps methodologies across their teams, it amplifies this need, as the barriers between development and the customer come down, and schedules compress.
Feature teams push product strategy, but doesn’t DevOps Release need its own strategy as well? If we are integrating across large projects, then we need goals and actions that support that as well. But from a DevOps Release Management perspective, our strategic goals will align with the longer term needs of the business, and not center on any specific product. More than that, these goals may frequently be at odds with any specific team, as release tends to act as a voice of the wider team, partner, and customer.
Generally, the following six strategic goals keep us moving our projects to release:
Defend the Customer
Everyone working on a project wants to build features that delight the customer. But that isn’t the same thing as being the voice of the customer. Unintended consequences of a new feature, downtime from servicing or release, or a lack of stability all happen from time to time. Everyone should always take the needs of the customer first, but it is the job of the Release Manager to make sure that the customer voice is heard. This doesn’t just mean reviewing feedback or design requests, this means proactively looking at what is coming on the schedule and asking hard questions of the team. Does this really improve the experience, or does it fulfill a buzzword? Do we need more QA before shipping, and do we need to move our date?
When something does go wrong, Planning ahead makes all the difference. Whether it is having a rollback, being able to quickly patch a specific issue, of messaging out to customers the status of what is happening. These plans need to be made ahead of changes, as it may be too late once a change has been pushed.
- Have a High but Solid “No Ship” Bar
Halting a release is a big move. Have a clear bar that says that if quality and experience conditions aren’t met, the train stops. Have teams and leadership sign off beforehand, and enforce that decision.
- Mitigate Impact of System Failures
Planning ahead for Business Continuity in the event of failures is the best way to preserve the customer experience. Automated failover of datacenters, in-app error handling, and proper offline experiences need to be built and tested so users can continue while system repair is in action.
- Have a Plan for Rollbacks
Changes need to be reversible. In rare cases, errors will only surface out in the real world, and won’t have a fix. Last Known Good code and service configurations need to be preserved. Plans to deploy need to be in place and verified before they are needed.
Close the Dev/Ops loop
The concept of “DevOps” isn’t a set of specific actions, but a mindset that the team drives towards. It is the close engagement of the whole team, from Concept to Customer and back again. We naturally set our project flow to take ideas from spec, to development, QA, the out to the customer. But we don’t always have good processes to capture the needed feedback from the customer. Or to capture changes in environments, tools, or the industry that might be outside the view of you developer teams. Release needs to capture, organize and present these to action can be taken.
- Implement Escalation and Monitoring
Directly Responsible Individuals (DRI’s) should be assigned (typically from dev) with clear automated alerting from within a service. Ownership needs to be clear so action can be taken.
- Capture Orphaned Features
As teams quickly move between features, DevOps breaks down. DRIs vanish, and support can get lost. Automated tracking of alerting must catch when there isn’t someone to respond. Plan to deprecate or get new owners as appropriate.
- Drive Postmortems
Fixing an issue on production is a Band-Aid. We heal by finding the root cause and correcting it. Related teams need to engage so the same issue doesn’t appear elsewhere when solved once.
Evolve the Product
As many of us work in iterative sprint cycles, we are in a state of constant feedback on the product as we build it. We want to continuously review what we are building, and refine and correct it as we go. This is the same if we are working towards a single large release, or small feature iterations. We want to question our assumptions about what we build, then back up our assertions with solid data. This continuous re-evaluation helps to keep us from fooling ourselves when our product is going astray.
- Hold SHIPROOM Meetings
A Shiproom meeting is focused on the product. Periodic meetings for teams to hear how features are working. Determine if everything is moving in the right direction.
- Cross-Feature Status and Feedback
Are features working together? Let teams give feedback to each other.
- Capture Usage and Quality Metrics
Understand the usage of your services. Are users giving feedback through tools or social media? Focus on the greatest impact.
Drive The Dates
The foundation for any project. Each team has their own internal sprint schedule, but they all need to coordinate across a master schedule where projects intersect. Having release managers track cross team is critical when it might not be obvious to an individual sprint team that, even though they are hitting their schedules, they may already be tracking to miss a critical cross-team junction or deliverable. Dates can also be impacted when requirements cross teams, Release managers should be the bridge to make sure that happens smoothly.
- Hold WAR Meetings
War meetings bridge the gaps between feature team Agile meetings. Frequently held to endue that tactical needs don’t fall into the gaps.
- Daily Issue and Bug Drives
Be the forum to get issues raised. Keep bugs from falling between teams. Make sure issues have positive handoff.
- Create Leading Indicators for Problems
Expose problems in process early. If a team can’t follow process, it may indicate issues (overload, delays, etc) . Get out in front of problems before they arise.
Align Deliverables Across Partners
When you have external partners that depend on your project, or if you depend on deliverables from them, constant communication is needed to avoid conflicts. This becomes more difficult when you have a series of partners that conflict with each other, either needing linked deliverables, or requiring the same resources from your team simultaneously. We need to fairly balance everyone’s needs.
- Hold GLIDE Meetings
Glide meetings are to keep partners aligned on a path to release. Partners should be able to hear each other’s issues. Open lines for cross-communication. Build relationships between partners.
- Have Partners Communicate their Most Critical Issues
Partners will always tell what they want, but hard dates and mission critical issues need to be broadcast loudly. Identify conflicts early so partners can adjust to tough decisions about priority.
- Plan To Over Communicate
Whether it is deliverables, schedules, or commitments, you need to over communicate. Internal teams have multiple channels of communication to depend on. Partners are depending on the Release Manager to keep them informed.
Make Metrics Valuable
The adage we hear repeated frequently is, “If it isn’t worth measuring, it isn’t worth doing.” While that is true, if you have to compile data across several teams, you quickly learn that there are many ways to measure an app or a service. While each team usually has a good idea of what data is valuable to them, it is still necessary to make sure that when it comes time to compile metrics for a service or product as a whole, we are comparing apples to apples.
- Natural vs. Synthetic Transactions
Are you measuring real users or automated tools? Know when each is appropriate, and what they are trying to tell you.
- Heartbeat vs. Core Metrics
A Heartbeat tells you that your service or app is running. Core metrics tell you how it is being used. Both have value, depending on what is being investigated. These should alight to a features goals and user stories.
- Use Metrics Consistently Across Teams
If you want to be able to roll up measurements across teams, they all have to speak the same language for the result to have meaning. Having a Metrics Standards spec that everyone uses helps make this happen.
Every team and project has different needs and goals, but the above strategies should be a great start for most projects with a DevOps Release cadence. When going through the process design for your project, take time to explicitly state your Goals, just like the product teams do for features . It will help to guide you in determining next steps when driving to the big release. Ask yourself what you think your goals should be. The only wrong answer is not asking questions in the first place.