So many test cases, so little resources! Which test cases should be inventoried and which should be executed? To do any job efficiently you must learn the art of test case prioritization. This subjective and difficult part of testing is about risk management, test planning, cost, value, and being analytical about which tests to run in the context of your specific project. Testing is one of the most critically important phases of the software development life cycle and consumes significant resources in terms of effort, time and cost. In this article, I will share the criteria I use to reduce the number of test cases, while maintaining quality and customer satisfaction when faced with the challenge of testing complex applications with limited resources.
For complex applications it is impractical or impossible to exhaustively test every scenario and all data variables. For example, let’s say you were given an application that has 20 screens, an average of 4 menus on each screen, 3 options on each menu, 10 fields on each screen, 2 types of input per field and around 100 possible input values. That’s a lot, see below:
Approximate total for exhaustive testing
20 x 4 x 3 x 10 x 2 x 100 = 480,000 tests
“A lot” isn’t a very precise description, so if you agree that the number of tests is large, also consider the length of time to run them per the test time variables below:
Test length = 1 sec then test duration = 17.7 days
Test length = 10 sec then test duration = 34 weeks
Test length = 1 min then test duration = 4 years
Test length = 10 min then test duration = 40 years
Unless you’re working with a simple application, it’s impossible to test every systemic interaction and data combination. If your test cases take a minute to run, then you’re probably not going to have a 4 year testing window (don’t even think about the 40 year one). To attempt an exhaustive testing approach in these situations would cost a tremendous amount of time and money. Regardless of application size, resource constraints can limit the amount of testing you can execute for each application build. Prioritizing test cases based on perceived risks and customer expressed needs can efficiently reduce the number of test cases necessary for comprehensive testing of a software application, to meet project milestones while ensuring customer’s requirements and expectations have been met. Below are some factors to consider in prioritizing test cases:
- Mission-critical components(take advice from Requirements/Customer)
- Complex features (take advice from development team/Requirements)
- Where failures would be most visible (take advice from development team)
- Features that undergo frequent changes (take advice from development team)
- Areas with past histories of problems (take advice from development team)
- Areas with complex coding (where were the developers most challenged)
- Areas of most frequent use
- Major functionalities rather than going into detail
- New functionality