Performance Engineering - An Architect's Point of View

Yes, you read it right! There are many more examples where the server got crashed due to high user traffic, it does not mean it was not tested before going live. However, these servers received heavy traffic in a very short span which led to the failure on the user screen. This phenomenon either leads the user to repetitively reload resulting in increased load on the system or driving them away to your competitor.

Performance Testing/Engineering is one of the key areas that all projects must consider before going live. Performance Testing will help you find out how your product is serving its users in terms of response time and failure percentage when you have peak load on the system & huge data volume on DB/Storage.

Let me ask one question- Have you ever noticed, how much time it takes to display the number/character on your device when you type something on a keyboard or mobile? Well, I would say it’s very quick. Now, let’s imagine if you are typing and it takes 1-2 seconds to display on the screen. I am sure you would not prefer using that product for long and start looking for alternatives.

Let me give you one more example- Let’s say you are interested to buy a product which is available on “Website - A” and “Website - B” for the same price. Website A is working slower compare to Website B. Where you would prefer to buy the product from? Of course, the latter one because it serves you faster and gives you a nice user experience. Does not matter how good you build your product but if it takes a lot of time, then you might start losing your customer.

Performance Testing is not only to help us understand how good a product is in terms of handling a greater number of users, but it also determines the end user’s experience in terms of response time with a greater load on the system. It also helps to forecast how your system will handle the load for the next 5-10 years down the line when user data keeps growing every year with the business.

Now that you have understood the importance of performance testing, the next challenge is to find a tool that is good for your product/project to implement Performance Testing.

I would recommend keeping the following points in mind while deciding on the Performance Testing tool.

• Does the tool support your product?
• Do you have skilled resources to handle the tool?
• How much does it cost? (Personally, I would rather save money on load test tools and invest in a good application performance monitoring tool for reliable results.)

Here are a few Load testing tools

Of course, there are many more tools in the market, but the most popular ones are JMeter (open Source), Microfocus load runner (License), Neoload, Gatling, etc.

All these tools perform the load test by simulating virtual users. 

When is the right time to start Performance Testing/Engineering activity?

Most of the companies do consider Performance Testing as an important factor however the sad part is, they choose to do it just before they go live. Involve the Performance Engineer just 3-4 weeks before going live won’t do much good and in fact, would bring in a lot of risks. For example,

1. Changing the scheduled go-live date based on issues
2. Cutting down the scope of Performance Engineering to meet the go-live date
3. Compromising with the real-life scenarios and real-time data
4. Inability to arrange separate performance environments because of time constraints 

Here are some key recommendations to enhance the performance of an application:

1. Involve the performance engineer right from the designing phase of the project. It’s good to have performance testing on each component that is involved in the project.
2. It’s good to keep the performance environment as identical as the possible production environment.
3. Keep the realistic and futuristic data in mind. Testing on empty database or database with fewer data in it vs anticipated data in the next 5-10 years down the line.
4. It’s important to know how many times a functionality is likely to be used in production in X hours instead of just focusing on the number of users.

Key recommendations to consider the timeline

The most critical part of Performance Engineering is to write the correct and reliable test scripts. This could be a time-taking job based on application complexity.

The best way to do a POC with a correct view of the application is:

1. First, identify the critical scenarios you have
2. If you haven’t gotten a chance to do the POC, understand the technology architecture behind the application as it will give you an approximation for the timeline?
3. When we decide the timeline for scripting, it is important to understand your team’s individual capabilities as not everyone has the same pace of understanding.
4. Understand your test data. Your script is very much dependent on the test data. Your 1^st test script output becomes the input of the next script and so on. That means, at some point, you have to manage the test data well in advance to run multiple tests back to back.
5. Data dependency can be avoided by using correlation or any other method however it also involves the risk of primary script failure due to various reasons.

 Commonly used Performance Test types

Based on your test result and various KPIs, you might need to modify your test and execute it multiple times to get more closer to the bottleneck. 

What is the ideal number of Performance Tests?

I would not recommend a predefined number of tests as performance test is always a cyclic hit and learn process. Once we find the bottleneck, we should fix it and retest it so we can keep discovering new bottlenecks until we reach the ultimate satisfaction level.

If we are starting the Performance Test with a pre-defined number of tests, it might discover some bottlenecks but there are high chances that critical bottlenecks might not get discovered due to it. 

How do we define KPIs that matter?

In addition to your business and market demands, do not forget to keep an eye on the following.

• Expected health of all your servers
• End-user response time for all kinds of transactions, you might want to divide the transactions into multiple groups like simple transactions and complex transactions
• Expected response time of Database query
• Expected response time of middle layer APIs
• Transactions per second for all the requests including APIs of the middle layer
• Number of users from different geo locations

Before running the test, you want to know how many load generators we should be using to test a particular application. We might have a very well configured machine that can generate and handle all the load however it is still a good idea to go for horizontal scaling rather than depending on one machine.

If you are using only one load generator and it has low bandwidth of NIC (Network Interface connector), it might impact your response time. 

Conclusion

Performance Testing/Engineering is a very critical activity that every software development team must consider. Using the right tool, having the right KPIs in place and the process will defiantly help us improve the product quality. Everything that is mentioned above is just peripheral but getting started with these would help you get familiar with your product or application’s strengths and vulnerabilities. The core aim for Performance Engineering is to reduce re-work and the need for refactoring. I hope you find the article useful. If you want to get expert advice on specific performance issues of your application, drop an email sometime. We’d love to brainstorm with you. info@qentelli.com