As I started writing this piece, I talked to Quality Engineers and DevOps teams’ members with varied years of experience in industry. This gave rise to two school of thoughts for what to measure in QE and which model fits the bill. The first school stresses on collecting every data possible and then analyse for the correlation while the second talks about collecting only relevant metrics.
I believe both of them have their pros and cons–as the first school can be helpful to find innovative metrics or new correlations, the second can be the base to remove the monotonous patterns leading to same defects and downtime.
As a leader with responsibility to build ‘world-class’ digital solutions, how many times did you daydream about getting those magical metrics in your dashboard? Quite often, Right! While leaders are caught with big visions, engineering teams are consumed in metrics and models discussions for measuring Software Quality.
This article will draw reference from the second school of thought and help IT leaders understand what are the ‘hot metrics’ in 2020 and models to implement in their Software Engineering.
Before we move onto identifying right metrics and models of Quality Engineering, it is important to understand how you choose right metrics and implement them across your organization.
Build consistency in QE Skeleton – Applying models and metrics rely on bringing consistencies in processes, tools, and technologies. Common definitions of Defects, Issues, Acceptable Criteria, Scope Changes in testing and Coding, establishes a common line of communications for teams to adopt Metrics-Driven processes.
Automate Data Collection Process – Your organization can apply relevant model and capture metrics based on how much data you have over the period to discover trends, drive improvements, and uncover underlying causes of defects. There are various tools available for automating the data collection across the entire software lifecycle. Understanding these data trends helps in identifying right metrics and improving them to achieve the higher quality products.
Data for Distinguishing Defects – Do not treat all defects equally. So how do your teams decide which defects are worth their first-hand time and efforts? The data collected in your automated systems can build correlations and find out which defects can cost more to the company (in terms of money or brand value). The automated systems give faster feedback for teams to tweak their QE Practices and achieve threshold metrics to get the best quality product out in the market.
QE Models and Metrics
Measuring in digital ecosystems requires quality models and related metrics system to give right value about the processes, projects, and products. Our Premium Business Networking client began to focus on quality of applications and developed one stop dashboard to give insights about on-demand environments for Testing and Build Promotion. A finance & consulting firm is spending 50% lesser time on obtaining Test data, improved QA cycles by 10 times with 5 times faster test execution.
Check out similar Client Stories.
There are tons of QE models. We will discuss some of the models and approaches that are still relevant and used by organizations.
1. Quality Management System (QMS) – As per definition, QMS is defined as a formalized system that documents processes, procedures, and responsibilities for achieving quality policies and objectives.
2. Total Quality Management (TQM) – TQM brings in the new facet of customer satisfaction along with managing processes, procedures, and responsibilities. The eight pillars of TQM are Customer-focused, Total Employee Involvement, Process-Centered, Integrated System, Strategic and Systematic Approach, Continual Improvement, Fact-Based Decision Making and Communications.
3. Kaizen Model – It relies on the Continuous Improvement in Continuous Integration and Development cycles. This applied to projects, products, employees, and organizations to improve business practices.
4. Six Sigma – It talks about removing the causes of defects and minimizing variability to establish consistent processes without any deviations or errors.
5. Voice of Customer (VOC) – VOC is a process to capture the customer feedbacks, requirements and integrate them in existing processes to start and scale the improvement journey.
There are various models for QE initiatives. You need to figure out what works best for your teams, products, and organization. It can be a single model or a combination of two or more models. We will discuss three major buckets of Metrics in this article – Project, Product, and In-Process.
Best Practices alone will not guarantee quality in your applications and can even result into late-identified and high costs issues if not applied correctly. The good idea is to catch the defects as early as possible and capturing right metrics at the Project level ensures Quality Assurance from the beginning.
Right Project level metrics captured from beginning can help testers, managers, and executives to measure and improve day-to-day QA operations. Some popular Project-level metrics are:
1. Requirements and requirement coverage: Requirement metrics measure the requirements laid by your business teams and what your developers need to achieve at the outset of production. Outlining these guidelines at the beginning help teams in setting their course for Development and Testing. Additionally, QA teams get clear direction about testing requirements to maintain their trajectory as the development picks pace.
2. Cycle Time: It is a metric borrowed from lean thinking and manufacturing disciplines, but it holds a lot of importance to measure the efficiency of the development process. Cycle time is the time taken from when an item is accepted and picked by the dev team through till go-live; Lead time also includes the time that an item is pushed into backlog and is waiting to be picked.
Cycle time says a lot about your team coordination and collaboration, code reviews, tools used, CI-CD integration, automated tests, and deployments etc. An optimized cycle time reflects optimized software development process, a team with shorter cycle time can be termed as more efficient than one with the longer cycle time.
3. Productivity: Productivity metric considers two factors 1. Size of the software you deliver and 2. Efforts to build the software (Cost, time size, time taken and other factors to be considered). Measuring productivity metric helps in calculating the right team size for your project, finalizing cost-effective tools and techniques. And this metric can go a long way for small and medium-sized companies in costs-savings and large organizations can form right team size to be nimble and act agile.
4. Rate of Requirements Change: Requirements framed at an early stage are bound to change and it is a part of the development process. Most of the teams are attuned to the process but ever-changing requirements impacts the estimation, accuracy, release time, cycle time, and quality of the product. Small changes do not create much disturbance in the ongoing release cycles, frequent changes can be frustrating and time-consuming for developers leading to bad quality product. Measure your rate of requirements change and aim for the highest stability possible in the requirements.
There are few other metrics that are considered crucial depending on business/project objectives, teams can measure and monitor them. Some of them are – Estimation Accuracy, Staffing Change Pattern, Cost, Scope, Risk.
Product Quality is directly associated with Customer Satisfaction and Adoption, metrics are a crosscutting dimension in terms of making a good product based on continuous feedback, improvement, and iteration. Due to DevOps and other technologies, Product Development is catching up faster than usual, making it imperative for teams to measure Product Quality continuously.
Here are some Product Metrics that you can measure to 'build-better' products:
1. Reliability (Defect Density) : Reliability or Defect Density metric helps your team to understand the code quality per unit. The lesser defects per unit, the higher the product quality. Measuring reliability metric gives the confidence of better quality from the beginning as developers can receive immediate feedback on how their codes are functioning. The defect density in per thousand lines of code shows how effectively your developers are coding and how processes like peer code reviews and unit testing are working.
I would like to briefly touch some important Reliability Metrics:
Mean Time to Failure (MTTF)
MTTF is the time interval between the two-consecutive system/application failures. MTTF is consistent for systems with large transactions. Teams with reduced or consistent MTTF are designated to have stable systems, meaning stable code and new changes.
Mean Time to Repair (MTTR)MTTR is the average time to track the error, identify causes and fix it. MTTR shows agility and maturity of teams. It is an effective measure of Operational Resilience or how quickly systems bounce back to normal state.
Mean Time Between Failures (MTBF)This is the average time between repairable failures of an application/system/product. This can be used to track both the availability and reliability of a product. The longer the time between failures, the higher the system reliability.
2. Availability : In some studies, Availability is considered as the most important Customer Satisfaction Metric. And we can understand why, No one wants a system to crash when a user is in the middle of filing the new financial reports on last day of Financial year or process payment of the next insurance premium on the due date.
Reliability metric can be collected using internal systems and resources for the latter one, your team can adopt three approaches: collect the data directly from small set of customers, collect the data via your customer service process, and conduct special customer surveys. Availability metrics is directly proportional to your Customer Satisfaction and hence measuring it can give teams real break-through if they are looking to achieve that magic number of 99.9999%.
At Qentelli, we have achieved this magic number for some of our clients by employing Availability Engineering in building digital products and applications. Talk to us about your next Quality Engineering initiative.
3. Usability : Usability metric measures the degree to which end-users use product effectively, efficiently, and satisfactorily.
As you continue to release new features in digital products and applications, there is an increased need to assess the Product Usability. If as a technical leader, you want to present numbers about how many customers are liking a newly released feature, this is your metric.
This is linked to your User Interface and how quickly users can adapt to its usage. For example, if a social media tool allows a user to schedule its posts with 7 clicks, the software is simply not optimized for usability. Similar tool lacking in showing consolidated clicks, views, and reactions on social media post lacks user friendliness in presenting information.
You can track Usability by collecting data such as User Analysis, Site Visits, Task Analysis, Usability Benchmarks and Designs. The right approach for your team is to define Usability targets from Day 0 and work towards achieving it.
4. Overall Customer Satisfaction : Your Product Metric dashboard is not complete without Overall Customer Satisfaction metric. This metric gives the collective picture of your product, all other metrics, your team development efforts, uptime, usability, etc.
The ideal mix and match of data sources lie in your product user-base and nature of users. Your team can determine customer satisfaction by customer surveys, customer reported defects/month and resolution time, nature of defects, collecting data from customer service team and social channels.
A good customer satisfaction rate means customers trust your product. For a Digital Organization operating in a competitive market, failure to address falling customer satisfaction rate can lead to a loss of market share. Metric can help your team in pinpointing exact point of failures and fall in customer satisfaction and resolve the issues to alleviate satisfaction graphs. Faster the feedback, faster the resolution leading to an increase in satisfaction rates.
Becoming ‘Metrics-Driven at the Core’ can create more sustainable value for your Engineering initiatives. Customers are updating expectations for organizations everywhere. Any organization that wants to sustain the digital race must accommodate the new rules of engagement with digital solutions. Companies like yours need to be metrics-driven, proactively plan for acting on metrics to be more profitable that peers.
Talk to us about what Metrics your teams are tracking and we can help you in measuring new metrics relevant for your projects and teams.
In-Process metrics are not well-defined across software development industry owning to the lack of standardized and common processes of software development. Teams claim that they are doing Agile, but Agile practices vary within the same organization.
But we believe tracking in-process provides a strong monitoring and control to ensure that your teams react and respond to deviations at an early stage. Additionally, teams can alter processes producing regular defects to bring more control over software quality.
1. Reliability Growth Pattern : You have new CI CD process with integrated testing and security tools. As you are confident about these processes and tools, your team reports longer system testing time and more failures. So, what has gone wrong? Reliability Growth Pattern can pinpoint the flaw in the process.
This metric measure and predicts the improvement of reliability programs in the testing process. The growth model represents the reliability or failure rate of a system as a function of time or the number of test cases.
Your team should benchmark plotted versus testing time with the magnitude of spike related to significance and volume of changes. If there is an increase in plotted time versus actual time, as an IT leader you might want to revisit the testing processes and tools.
2. Defect Patterns During Testing : The type of defects plotted versus actual found during testing is an important metric to further evaluate your release time and thus it is an important in-process metric.
These defects must decrease as the product moves towards release. If it remains same, re-look into the testing coverage, is it sufficient for the current release or does your team need more testing?
If the defects have gone substantially high, your testing effectiveness must increase. Bring in Automation or other sophisticated tools based on team, project, and budget. This metric will help you win the buy-in from the management team for Test Automation.
3. Backlog Management Index : Backlog Management Index is the accumulated difference between defects closing and defects arriving. Tracking Backlogs are important for assessing testing progress and customer rediscoveries of defect.
A lot of factors influence Backlog Management Index like if your team is working on a legacy system, there are high chances of discovering more defects and taking more time to resolve them. With the pressure of new releases, this backlog can increase over time.
Your process management approach should be three-pronged: With an effective and strong testing plan in place, ramp-up your testing processes to achieve early and faster feedback. Aim for targeted improvements by analysing data from the trends and their timings. Manage backlog reduction by prioritising the defects based on severity and priority. Try to get the values within the pre-determined levels.
These are in-process metrics to bridge the long-term gap between processes. Every project is unique, and we recommend working with QE specialists in figuring out right metrics to track your development process.
One more SLA metric is Percent Delinquent Fixes that determine how long does it take to fix the defect by the number of fixes delivered.
A note for Management Team
With your consensus about the company-wide metrics program, your team is already exploring various models and metrics. As a positive push to their efforts, you can help them in implementing metrics using piecemeal rather than the wholesome approach. Ensure your leaders or champions driving the program foster transparency and achieve buy-in from Developers. Empowering your team with automated tools and learning about the latest technologies will go a long way. Do not fall for ‘Flight Magazine Syndrome’ and expect results to flow in, always look for opportunities to reward team rather than punishing approach (a usual mindset with Metric Implementation).
After working with more than 100 clients, our QE consultants and Innovation team developed a product, The Engineering Dashboard (TED) to bring right metrics together for better Quality Management. Know more about TED. TED tracks all metrics including Process, Product, and Project and presents a consolidated view for stakeholders enabling faster decision making.
Want to discuss more on Metrics with respect to your development process and Quality Engineering plan. Feel free to write to us. firstname.lastname@example.org