Skip to main content

Fighting unavailability with Performance Engineering

Category
QA Transformation
Posted On 29 Jun 2022

One of the biggest club management software providers in USA is associated with Qentelli to up their game of Performance Engineering.

Business Vertical:

Club Management Software Provider

Region:

USA

Size of the firm:

Partnered with 16,000+ health clubs and 150,000+ fitness professionals

Main Challenge:

Application Performance

Challenge

Contemplating the data history, we noticed there are maximum number of transactions during November to February due to increased yearly subscription during that period every year.  As the users increased, the client started getting feedback from their stake holders about slow response rates and frequent application crashing incidents during these peak business hours. The in-house application team did not have a choice but to restart the entire farm in sequential order every week to restore the system and it became habit over time.

Slow response, unavailability and crashing of the application in production environment are nightmares, especially when it happens during peak business hours. It not only impact end user/business user but also hit the revenue very hard.

challenge
solution

Solutions Proposed

The client’s club management system was configured to be highly customizable to an intricate detail such as the button colour, text, design, etc. for every client. Besides, having inventory management system, order management system and many other features in one place makes it a great choice for the club owners. To keep the client retain their top-player position in the market, it was high time to fix the performance issues in their applications. Here’s what the team of quality experts suggested the client in a nutshell.

What Qentelli Did

During the initial investigation and discovery phases, there was a lot of uncertainty about the key bottleneck. After carefully studying the server logs, we suspected memory leak on the code. As It was legacy application, the logging and alerting system was not intact.

We teamed up with the in-house application team to understand all the critical business scenarios and started creating a common configuration for the business flows. Because every club that’s using the application had a different configuration and it was difficult to build the scripts to test each.

Strong and dynamic scripts were created:

As we learned that the application is highly configurable, hence we decided to make the performance test scripts more robust to ensure there are no scripts issues going further. This initiation of Script enhancement helped us to add custom logic, enable basic parameters and dynamic value handling functions etc.

 

Fully automated test data pipeline setup:

Manging test data was one of the critical aspects as a lot of test data was not reusable. The application data were stored in the Oracle and another two Databases. So, before every test, team has to request for the fresh data which was not ideal. Qentelli’s Performance Engineering team proposed to remove manual and data dependency completely and decided to automate this process without compromising the users’ access and data security.

Our Performance team streamlined the entire data pipeline so that all the test data from multiple DBs would go through PostgreSQL (Performance test DB) and Load testing tool (JMeter) shall test the data from PostgreSQL DB without disturbing the main application DB. The benefit of having dedicated performance DB is that we can perform Read and Write operations more conveniently.

 

Tuning the system to improve Response time:

Performance Engineering team has simulated multiple load & stress tests and tuned many transactions which were designed to breach Service Level Agreement (SLA). Team were mostly focusing on:

 

Investigating the memory leak:

Our Performance Engineering team has executed multiple rounds of endurance tests, spending over 72 hours for each test. It helped engineering team to drill down and get to the root cause of memory leak. Application has multiple functionalities. Tt’s performing just fine when the user request required less data however, it was becoming a logjam when the user request needed a lot of data to display on the screen. The application was consuming a lot of memory not only from server side but from the end-user as well. Qentelli’s team suggested Pagination to reduce the load on JVM. The code was also identified as one of the reasons for the memory leak in the system apart from user enrolment business flow.

Tech Stack

Spring
Java
Oracle
Apache Tomacat
JBoss
MySQL
DB2
Active MQ

Outcomes

60%

increased sign-ups

60%

reduced risk usage

75%

improved SQL query performance

60-98%

better Transaction response time