The word “Automation” sounds very exciting. It is modern, realistic… even a little dramatic in its tone. If it had a color, it would be Bright Red – the color of boldness, excitement, courage, and speed. To draw an analogy, any organization goes through similar feelings when implementing ‘DevOps Automation’. And why not! Automation saves time, increases quality, improves employee satisfaction, and optimizes costs for any organization.
But what if certain DevOps processes are Over-Automated or Wrongly Automated? Whereas the right automation magnifies efficiencies, over-automation can conceal the innovation element (Human Intellect) in the DevOps program. In this article, we are highlighting some common areas where organizations fall prey to DevOps Over-Automation and how to avoid them. In the end, we will also propose a comprehensive DevOps Automation approach.
Automating all the tests
Test Automation is an excellent achievement for DevOps teams. Test Automation means upholstering the quality of your working product and move it more quickly from one phase to another. But there is more to this story of Test Automation.
The availability of Test Automation success stories and the ever-growing tools landscape makes organizations believe that they found their magic bullet. As a result, they rush to automate everything in testing. While they expect green CI-CD pipelines, increasing test coverage, a growing number of automated test cases, and empowered testing teams; The results are opposite – Choked CI-CD pipelines, Delayed feedbacks, Bug slippages, Increased testing inefficiencies, and Casual approach of testers due to over-dependency on Automation.
World’s largest quick-service restaurant chain with a legacy monolithic POS windows-based application faced multiple challenges while automating their tests. The application was deployed across 15000+ stores around the globe and a single testing cycle was 6-8 weeks. With multiple languages, currency, and a multitude of configurations, their automation initiatives went in vain.
They were trying to automate everything including Usability and Exploratory Testing. No doubt, with the rapid release cycles and continuous UI/UX changes, automation results were underwhelming. Usability and Exploratory Testing require human instincts and critical thinking and thus forcing them into automation is not advisable.
Simplified testing workflows, adopting shorter iterations, the whole-team approach, and the Quality Engineering mindset bring the right balance in Test Automation.
Overplaying Build Automation adds to Developers’ Inefficiencies
A rightly automated build compiles computer source code into binary code, package it, & run automated tests. The build then gets merged with the main code. Does that sound like a perfect CI story? Wait!
The initial set of build workflows were working fine, but then the team added multiple tests to existing workflows. With multiple alerts and notifications for each check-in, a simple workflow becomes over-complicated and time-consuming.
Automation smitten teams add all tests and notifications to build pipelines. They overlook the fact that adding unnecessary tests will add to build compilation and running time. Also, configuring multiple notifications inundate developers and they start overlooking even the important alerts. Imagine, a ten-hour build process against a half an hour build process. Such bloated builds delay feedback adding to the developers’ inefficiencies. The Build Automation instance points to the habit of using Automation superfluously.
Instead, evaluate your code check-in frequencies, branching strategy, and the team’s comfort level in Build Automation. This ensures correct automation of Build Processes.
Overwhelming Infrastructure Automation
Infrastructure Automation serves as a force multiplier for DevOps. However, indiscriminate automation comes at a premium of unmanaged environments and IT inefficiencies & costs. Several scenarios where organizations should avoid Over-Automation are:
- Dealing with sensitive and business-critical data. For example, dealing with environment deployment is different than dealing with data automation. A critical database can be at risk while using automation tools, there is a risk of losing it if provision conditions start from the clean slate approach. Similarly, a different set of data may be encountered as new by automation tools and can be accidentally deleted, if they are not able to identify it.
- Lack of Technical Resources to manage code – More Automation means More Code to Manage
- No weekly or daily releases (Not everyone is Netflix or Amazon)
- Highly inconsistent and changing environments
The overriding lesson is to use Infrastructure Automation to decrease downtimes, increase self-service capabilities, empower Ops, and maintaining a solid (99.9999%) uptime.
Over-Automation in Deployment leads to acute risks
Smooth Deployment is like a dream come true for DevOps teams. It acts as a bridge between Value Generated and Value Delivered. But with exaggerated Automation, the whole release can be botched turning a dream into a nightmare. The inflated Deployment Automation can slow-down the release, put operations on firefighting mode after every release, and make roll-back processes time-taking and tedious.
Deployment Automation is for teams with daily or weekly release cadence. In the absence of weekly or daily releases, Hybrid Deployment Automation (Manual + Automated) works best. Go for it when you have:
- One-time Projects like Customer Pilot Deployments, Internal Deliveries
- Projects with infrequent releases
- Sufficient technical skills for Manual Deployments
Follow a manual check-gate to ensure security, reliability, and compliance considerations are taken care of before the deployment.
Security cannot be completely Automated
With the rise of cyber-security threats, this is the most dreaded aspect of DevOps Automation. Security is integrated within DevOps pipelines, but a single automation error can put a company’s existence at risk. Automating Routine security checks and identifying potential breach attempts is a good practice. However, there are several areas, where automation can create more security problems than it solves.
Automating security systems like Network Firewalls, Intrusion Detection, Prevention systems, and Patch Management Systems open doors for cybercrimes in any organization. Similarly, Compliance as code increases the risks of infusing vulnerabilities in the application.
For Security Automation, it is highly advisable to involve human counterparts in critical tasks. Also validating every automated change can enhance security posture.
Towards a Balanced DevOps Automation
At Qentelli, we understand the risks of DevOps Over-Automation. Our DevOps Automation practices add value to clients with existing DevOps processes. We understand the key to Agile Operations, Faster Development, Top-notch Quality, and Uncompromised Security is well-balanced DevOps Automation. Here are 5 steps for making DevOps Automation work for you as expected:
1. DevOps Vision – The success of DevOps is not dependent on tools and level of Automation but a defined vision and strategy to achieve business value. Bring stakeholders together to understand the business outcomes expected from DevOps. The clear vision helps in determining the level of Automation and timelines.
2. DevOps Assessment and Strategy – In step two take your DevOps vision and match it against current DevOps capabilities and maturity level. Get all stakeholders together to determine the future state of DevOps and existing gaps. Additionally, determine DevOps Automation Areas, Strategy, and Roadmap. Initially, keep the Automation scope constrained so it does not block the vision of the overall DevOps program.
3. Pilot Framework and Tools – Building a DevOps pilot framework helps in predicting the success and failure of the program. While building a DevOps pilot, generate an Automation blueprint that outlines tools and technologies, industry-specific solutions, and integration and implementation timelines. Start with limited Automation in these areas:
- Repetitive tests (Smoke, Sanity, & Regression)
- Testing Multiple configurations (OS & Browser combinations)
- Tests with large volumes of data
- Performance tests like stress and load tests
- Automate deployment for repetitive infrastructure
- Self-service capabilities for a limited set of developers
Streamlined development and deployment
- Simple code integration and build processes
- Automated testing against builds
4. DevOps Implementation – With the DevOps pilot up and running, the next step is to start scaling DevOps organization-wide. DevOps implementation and Automation will go together across all stages to align standard practices, orchestrated workflows, and foster culture change. Here are a few best practices to apply right amount of Automation:
Identify Quick-Win Areas
With DevOps, define an overarching Automation strategy, tools, skill gaps and tie them to organizational objectives. Be it Build Automation, Infrastructure Automation, or Deployment select a single process to automate and showcase business outcomes.
Build pilots that get noticed and document them. The documented Automation use cases serve as a source of truth and learning tools to scale Automation capabilities.
Create a center of excellence with the DevOps program to apply a methodical approach towards Automation rather than rushing into it.
Drive Culture Change
Continuously communicate with your teams about the pros and cons of Automation as well as over-dependence on Automation.
5. DevOps Evolution – The DevOps process is all about improving the speed and quality of outcomes and enabling continuous learning. There is no finish line but there is always room for improving your DevOps Automation practices. Culture and communication play the key role in DevOps Evolution.
How Qentelli can help
Our DevOps Assessments or Benchmarks, Automation guidance, and be-spoke Implementation services with periodic assessments help clients apply even and level-headed Automation. Skeptical about your DevOps Automation Approach, we can help! Reach out at firstname.lastname@example.org