Kingsmen Software

What is DevOps? Scaling (Part 2)

Jun 14, 2017

Chris Nowak, Chief Transformation Officer | Kingsmen FlowStates


In my last article, I promised you I’d dive deeper into two things – Our Kingsmen DevOps Reference Model, and the secret scaling sauce of DevOps.

Since that first article, there have been exciting revelations from the 2017 State of DevOps report that corroborate our approach (published on June 6th by Puppet Labs and DORA (DevOps Research Assessment), led by Dr. Nicole Forsgren, Gene Kim and Jez Humble). For the first time, Transformational Leadership is included in the survey and found to be a significant factor in differentiating between High and Low performing teams. The report is in its 6th year of publishing, and reflects approximately 27,000 total survey responses.

Get it here - 2017 State of DevOps Report

I’ll break this up into several articles to help introduce our concepts and how we got to our model. This article will address our observations, beliefs and experiences with scaled transformations that prompted us to think more deeply about CALMS and how to tie it back to success. In following articles, I’ll discuss our model more deeply, with a heavy emphasis on culture and leadership, and tie it back to the new 2017 State of DevOps Report by Puppet and DORA.

Today, let’s talk about our observations with the CALMS model in scaled transformations and our adaptations based on experience.

The foundation of our model is well established in the DevOps community. The most important DevOps elements are represented by a CALMS mnemonic. Culture, Automation, Lean (process & interaction), Measures (Metrics) and Sharing (collaboration). Our model is derived from CALMS, but we’ve extended it.

In our view, CALMS is a great foundation, but since it is a high-level mnemonic, it doesn’t capture the complexities of transformations in larger, older organizations. First, in practice, it is IT Team focused without enough detail on business and culture, essentially the holistic systems view. Second, the cultural component (which includes leadership) is much broader than a bullet point can capture. Third, we’ve seen that dev teams naturally gravitate towards CALMS, but operations (any post-production activity) is often left behind. For this reason, we include an Operations dimension to call attention to post-production teams. I know, it’s supposed to be DevOps but we think most organizations still need a nudge here.

With that in mind, let’s examine CALMS from our perspective.

In General:

Every company is different. There are principles that everyone can follow, but different skills, size, corporate age, maturity levels, and cultures make the implementation look a little different for each company.

In a large company, we often see the equivalent of many smaller “companies” or teams which are tied to organizational boundaries. They have their own rules, subcultures, and customer / product characteristics.

Most teams / companies only focus on automation and tooling. They achieve some level of Continuous Integration (CI) and sometimes a bit of Continuous Delivery (CD). They may do some agile and talk a little about the sharing aspects of culture, but they never really unite cross-functional teams behind the effort and create common incentives.

This will work for a while, but it begins to fall apart when they have a need to scale or cross organizational silos. This is where having a strong leader who can transcend organizational barriers is crucial to success.

On Automation:

Everyone in IT starts with tooling. We are technologists after all, and we feel safe working with things we know – tools. Unfortunately, most DevOps initiative also end here. Is tooling or CICD a form of DevOps? Yes. If all you do is automate, do you generate durable services with the best ROI possible? Probably not, and definitely not if you need to do DevOps at Scale.

Technologists often get tangled up in tools. The goal of DevOps isn’t tooling, it’s results.

If a DevOps transformation is struggling, 9 times out of 10 there is a tooling argument occurring. This is just the symptom. The root cause is more likely your cultural component: turf wars, lack of leadership or lack of Systems Thinking. Systems Thinking is taking into account the entire delivery process from Business to Customer. It is Gene Kim’s “First Way” in his book “The Phoenix Project”, and is what Eli Goldratt, author of “The Goal” talks about in his Theory of Constraints.

Don’t get me wrong – tooling is critically important and you need it as the foundation of all other efforts, but it only gets you so far. You can’t be successful with your Agile, Lean or DevOps transformation without good automation, but you need more than automation to anchor and scale.

To borrow another phrase from Eli Goldratt, Automation is “necessary but not sufficient” for scaled transformations. When tooling is localized without thinking of high-level Lean Process or Cultural aspects, you end up with pockets of unconnected excellence and no executive champions. Your ‘product’ has no flow. If you don’t anchor DevOps behaviors in your culture, it will eventually regress to an ineffective state.

On Lean:

In many DevOps efforts, there is no effort to lean out existing processes during automation. I once had an executive tell me he wanted a “one day onboard” and then later a “one hour onboard”. What he meant was take an existing manual build and deploy process and wire it up in the DevOps tool of the day.

There is some small incremental value in porting your existing manual POM or Solutions file into Jenkins, but the real exponential value is if you examine the entire delivery process to eliminate steps, scripts and handoffs. You also begin to consider architecture or delivery decisions such as how to handle configuration files and parameters more effectively. You should become familiar with Value Stream Mapping for your more complex delivery processes.

The process engineering, or Lean Value Stream, is not a one hour or one day exercise. In a larger organization, it typically spans 3-4 teams, 3-4 tools and 3-4 processes. Our fastest onboard time of 4 hours was a simple .Net app with a homogeneous pattern. We averaged 2-3 weeks for an onboard, and we were working around the client team delivery schedules. Our longest was many months as we unraveled years of bad process wrapped in even worse process. An “Automation Onboard” is really a Process Re-Engineering Onboard, and often a Behavior and Culture Re-Engineering effort.

On Measures (or Metrics):

Many teams aren’t measuring anything, or if they are, it is done poorly. We often work with Build / Release Engineering teams who have no real handle on their metrics. Even if their tools tracked it, they aren’t watching the metrics or trending. When you start an initiative, the baseline is important. You can’t know how far you came if you don’t know where you started.

I tell my teams that from the beginning, build metrics into what you do. If you don’t have metrics, you can’t defend your services or ask for more funding. What gets measured gets done, and you can’t improve what you don’t measure. There will never be enough funding for everything you want to do, so good metrics let you know where to focus your efforts for quick, meaningful wins. The DevOps world is an 80/20 world – explore and exploit.

If you’re scaling services, you should also know your gearing ratios, meaning how many applications can each of your automation engineers support, or your ops teams support. If groups have the data and are honest about it, about 4:1 is common, meaning 4 apps per engineer. A high performing team ratio is 40:1 or better.

On Culture:

One thing is clear – the bigger you are, the more culture will matter for success or failure of your transformation. This translates into needing higher quality leadership as your transformation ambitions grow.

Here’s an eye-opening chart from a 2016 Continuous Delivery Survey by DZone.

dzone 720

Interestingly, the point at which culture becomes the biggest barrier is Dunbar’s Number, ~150. (Robin Dunbar, Anthropologist). Here’s Wikipedia on Dunbar’s Number: “Dunbar’s number is a suggested cognitive limit to the number of people with whom one can maintain stable social relationships”

It isn’t surprising that you need talented leadership at many levels to pull off a successful transformation. Most studies show that upwards of 70% of business, digital and IT transformations fail or do not achieve their stated outcomes. From our DevOps perspective, we attribute 100% of the failed or substandard transformations we’ve seen to culture and leadership.

Think of it this way: companies of every size have roughly the same access to the same DevOps tools. There are great open source tools in almost every category, and if you want commercial products, you can easily get what you need for under $1MM. So, tools aren’t the issue. How about Lean process? Everyone has the same access to YouTube, blogs, webinars and books. The knowledge is freely available. Knowing about lean isn’t the problem. What differentiates you from your competition is your people, and by extension, your culture.

Here’s a bit of bad news. Unless you’re a very small company or the CEO, you really can’t change overall company Culture directly. Now for the good news: you can change it locally and you do this through behaviors, goal alignment and proper incentivization. You can then broaden your sphere of influence as your DevOps success grows, very much like ripples in a pond. Think of it like sphere of control and sphere of influence. You can grow it and influence the broader company culture, but it takes time, and it takes some transformational leaders willing to take the risks of going against the grain almost every day.

Here’s how it works. Get clear on the results you’re after. These should be business goals. Then decide what behaviors you need to achieve those results. Put Process (Lean) and Technology (Automation) guardrails in place to reinforce the behavior you want. Now your behaviors influence culture by delivering results against business objectives, which further anchors desired behavior in your culture.

Lead with behavior, not with tools. In our model, we include behavior in the Culture component.

With all this in mind, the DevOps idea of Culture is big and we’ve made it even bigger. Once you make it past your initial tooling automation, the exponential results come from cultural dynamics. We’ll need to do an entire article series on Culture & Leadership, but here’s our quick view. We lump “Sharing (collaboration)” into culture since it is part of a healthy DevOps environment. We also expand it with discussions of Behavior, Trust, Empathy, Ownership, Accountability, Systems Thinking, and answering “Why”. We add Leadership in various forms: Transformational, Servant, and Intent-Based.

Here’s what we’ve done to CALMS:

• We start and end with Culture

• We’ve broadened how we approach culture by also adding leadership, ownership and trust

            David Marquet on Intent-Based Leadership

            Navy Seals Jocko Willink & Leif Babin on Extreme Ownership

            Stephen M. R. Covey on Trust and Culture

• We folded Sharing & Collaboration into Culture

• We elevate Operations as an explicit component, and not just buried somewhere in the “L” or “S” of CALMS

• The beginning of our culture conversation includes the “Why”
            Simon Sinek’s TED talk on Why

I’ll leave you with this as a teaser into our next discussion. This is our view of CALMS. Notice that Culture (Leadership) wraps everything, and we include Operations as a dimension. The outer Culture Wheel isn’t really a linear progression, but it does approximate the awareness and experience growth that emerges as individuals begin to truly operate as trusting, empathetic, results-focused teams.

Culture is the driver for high-octane DevOps and organizational results. It is also the most difficult to change.

Most teams start in the green and stay there. They give up the exponential returns available to those who fuse behaviors and teamwork into their culture. It is hard, but worth it.

The path to scale runs through culture.

ks calms 720

About Kingsmen Software

Kingsmen Software is a software development company that crafts high-quality software products and empowers their clients to do the same. Their exclusive approach, The Kingsmen Way, combines processes, tooling, automation, and frameworks to enable scalability, efficiency, and business agility. Kingsmen’s craftsmen can be found collaborating in the historic Camden Cotton Mill at the corner of 5th and Graham in Charlotte, NC.

Back to Blog