Prioritization

  • 1
  • Question
  • Updated 9 years ago
  • Answered
In the Create Tests, the test cases are numbered. Are these arranged according to priority? If so would that mean when it says 80% at 12 tests (out of 35 tests) in the Analyze Coverage for example,
I would have achieved 80% coverage for the first 12 test cases?

Or there’s no prioritization in the Create Tests? I can choose arbitrarily and run any of the 12 tests out of 35 tests.
Photo of Ed

Ed

  • 25 Posts
  • 0 Reply Likes
  • happy

Posted 9 years ago

  • 1
Photo of Justin Hunter, Hexawise Founder

Justin Hunter, Hexawise Founder, Founder and CEO

  • 246 Posts
  • 15 Reply Likes
Prioritization is important. The test cases are put in priority order so that you will achieve the highest amount of coverage possible in the first few tests. These are "front-loaded" tests.

So your first interpretation is correct. In your example, the first 12 tests (from the Create Tests screen), would achieve 80% coverage of all the possible pairs of values in the plan. E.g., if there were 100 pairs of values that could be tested in the SUT, after 12 tests, 80 of those pairs of values would have been covered. If you were to "take any 12" tests and random and execute those instead, you would achieve significantly less coverage.

Let's look at the chart below to illustrate that point further:



This chart is for a 2-way (or "pairwise") test plan. The chart shows that, for the total possible number of pairs of values that could be tested, 100% of them have been tested by the final test case (#46). They are intentionally ordered so that the earlier tests cover as many pairs as possible. As the test plan proceeds, it becomes more and more difficult for a test to cover as many new, "yet to be tested" pairs as it was in the early tests of the plan. So over time, fewer and fewer new pairs can be "covered off" with each test case.

Why is this? This screenshot, showing the "any values" in the 46th test case, makes the point. (Click on it to see a larger version):



Thanks again for your excellent questions and suggestions, Ed. We're lucky to have you as a customer.