Your DevOps loop is broken; Solving the continuous feedback puzzle – Part 1
With DevOps, we have come a long way from it’s the code or it’s the environment puzzles. As the concept became familiar, thought leaders like Qentelli are giving emphasis on democratizing feedback loop (listening and responding) across the entire DevOps. Further course of CI/CD requires near real-time detailed feedback for each phase to make finer improvements in the application delivery. Properly administered feedback loop results in offering new dimensions to act upon such as new events, feedback, data, customer demands or variables for businesses.
In this 2-blog series of the DevOps Feedback loop we will talk about signs for identifying missing feedback loop and how-to plug-in feedback at all stages of DevOps. The first article in the series talks about signs of missing feedback and characteristics of the feedback loop. Let’s get started –
Observe the obvious
Organizations assume few things are obvious, but the assumptions bust when things go wrong. Observing the smallest of changes and communicating at the right moment completes the feedback to foster proper improvements in DevOps. For instance, a top game development company assessed that industry will witness the mobile revolution in second decade of 2000s, but it came early. Though they were the market leaders, they witnessed a major fall as none of their games were ready for mobiles. There was a gap to identify market forces and how it would impact internal business plans. The application development space is changing so rapidly that what holds relevance a year ago or even five minutes before is irrelevant now. Teams not evolving rapidly are leaving behind in the race.
Organizations should look for such obvious gaps to identify their broken feedback loop. These signs can serve as a beacon to navigate the path of DevOps and establish a feedback loop to avoid mediocre software deliveries.
#1 – Treating DevOps and Business separately
The meaning of DevOps doesn’t convey properly because of its formation. It says Dev and Ops–combining development and operations, but in reality, it expands to integrate business into engineering. If your business teams are not part of your next feature release or prioritizing tickets, it is a red flag for DevOps program. Business teams can translate Developed versus Released features, Automation, Security, Compliance & Governance integration into costs resulting into right rankings for development.
It’s the software economy and application releases decide how the balance sheet and P&L statements of organizations look like. Early feedback from management helps in tailoring application delivery as per the business goals.
#2 – Unidentified communication barriers
Communication and collaboration are the true enablers of DevOps. In DevOps, prescriptive rules do not drive communication, instead it is a philosophy for cross-functional teams to work together for software deployment, support and maintenance in production and beyond that. Organizations constantly reinforce different modes of communication such as stand up meetings, smiley boards, chat rooms and open spaces. But does it create transparent and open communication culture? The answer is No! As per the Pulse of Profession Report 2018, inadequate or poor communication accounts for the top 5 reasons for the project failure in any organization.
The goal of DevOps is to have development and operations work together collaboratively for better application releases. But if your developers still throw surprises to Operations about the releases, this is a sign of unidentified communication barriers. If there are many instances like this, it is an alarm to fix DevOps feedback loop around communication.
The only way of removing these communication barriers is to learn, educate and promote DevOps. Teams questioning about “What is DevOps?”… “Why we keep on changing priorities in every sprint”… “We are doing well, why we need it?” are not mindset challenges, they are just poor communication about the benefits of DevOps. Management and other technology leaders must proactively act to establish communication rhythm and sell DevOps across the organization.
#3 – Late identification of issues
Organizations are trying hard to identify and resolve errors in the early stage of development lifecycle but are achieving limited success. Teams identifying bugs at later stages of development is a sign of missing DevOps feedback loop.
It is unavoidable to commit mistakes but spotting, preventing or resolving them in lower environments is possible using feedback. Teams need to stop looking at defects as an unaccepted practice, rather look for better and stronger feedback loops and quicker processes to fix them. DevOps teams should align testing with the development to catch errors in an early stage and avoid any mismatches downstream.
The formal feedback loop should reflect customer perception of poor quality. Say, for example, customers perceive hundred defects that doesn’t impact user-experience as great rather ten issues impacting usage of important features as poor quality. Tracking customer perception of quality and duly developed testing strategies exploits the feedback around perceived quality. This helps in resolving bugs as per the customer perception of quality. Above all customer experience is Emotion, Convenience & Outcomes and not underlying application technology or infrastructure performance.
#4 – Lack of automation
For most of the organizations, starting automation initiative is a fine mess. If your most of the development processes are still manual, this is a sign of missing DevOps loop.
Automation is heavily associated with DevOps. DevOps stresses on creating a highly automated environment from planning to production and post-production monitoring. Lack of automation in DevOps breaks the feedback loop as–
- Errors are detected in later stages
- Increased risk of a deployment causing downtime affecting application
- Delayed roll-back in case of issues
Non-automated processes cannot be put under a continuous frame of capturing feedback from delivery pipelines and set up notification triggers. Based on the processes, applications and architectures, organizations need to identify critical steps from where important and timely feedback distributes to the team.
#5 – Faster releases but no insights about critical KPIs
DevOps adoption is increasing, but DevOps improvement has reached a plateau stage. If the only improvement area in your releases is deployment speed, then this is a sign of missing DevOps feedback. Faster releases with compromised quality are giving sub-optimal benefits to teams. Improvements in Customer centric and quality metrics such as Number of production defects, Mean time between failures, Mean time to recover and Cycle time show about the efficient processes and time taken to provide actionable feedback to developers about their codes.
The more, the time taken for feedback, the more time is wasted in delivering value to customers. The feedback can be faster if developers are running unit tests early in the pipeline saving time for them.
The feedback should include customer experience metrics tied to improve overall DevOps metrics. Customer experience metrics need to be shared across the organization, reviewed by a set of identified cross-functional team members to discuss how organizations can work on them to meet most of the end-user expectations. Cross-functional teams should create a product roadmap to break user stories small enough to release them fast and provide value to users. Feedback and involvement from business teams avoid backlogs of non-value-added features later in the development.
#6 – Monitoring should extend to feedback loops
In DevOps, just monitoring without taking relevant actions is a sign of missing feedback loop. Feedback is the extension of Monitoring. Monitoring infrastructure, applications and logs do not give a clear and comprehensive picture of DevOps as a whole.
In current times, DevOps teams are monitoring different areas of DevOps pipeline resulting into a siloed picture of the DevOps pipeline. The way to DevOps feedback starts with alerts and monitoring extending into feedback for all the areas of DevOps pipeline and recommended improvement areas. Some areas to track for creating feedback in a simple delivery pipeline are–
Commit Stage–Commit notifications, build fail/pass results, unit test results and code metrics
Testing Stage–Performance testing results, having threshold criteria for performance and comparing them
Deploy Stage–Production performance monitoring and deployment reports.
Qentelli’s tool like TED covers the entire DevOps lifecycle and provides feedback on how each of the process are doing and actionable insights for potential flaws. TED is technology and tool agnostic and can be integrated with lots of DevOps tools used by organizations.
The nature of DevOps feedback loop
The right feedback loop must bear these characteristics–Faster, Relevant, Actionable and Accessible. Engineering teams need to set rules for acting on different feedback and own the complete code quality checked in by engineering teams.
We all remember the classic example of SlideShare DevOps failure where one small reorganization in database brought whole site down for over 60,000 users. Had it been a timely and properly targeted feedback at the developer level they would have avoided the whole situation?
Another example comes from IBM in early 2000s. IBM started their DevOps journey to speed up their software releases. But unable to achieve so, they assumed that the problem is with code deployment and automated it. Still, didn’t achieve results. Later with the help of experts they discovered that the problem was with operational and development environment. Again, a relevant feedback would have given the expected results.
This is the early 2000s scenario, companies failed, changed strategies and now, they are maturing in DevOps. But still things that seem obvious go wrong with the broken feedback loop. Some scenarios we still encounter are failed smoke tests in production, insufficient test code coverage, etc. The feedback loop places a set of rules to roll back to the original state or not to introduce any new changes without reaching the threshold of the test code coverage.
Feedback is fundamental not only to DevOps practice but throughout the SDLC process. A customized feedback loop and process is necessary for every organization to act as a control center to alter the course early when things go wrong.
In the next blog of this 2-blog series we will talk about the integrating feedback at each stage and how the future of DevOps with feedback looks like. Stay tuned!