Hexawise delivers Awesome large volume handling!!! Of a large quantity of tests, which are the most important ones?

  • 1
  • Praise
  • Updated 6 years ago
We have tried some of our many parameter inputs into a Hexawise test model and it generated 8,806 tests of a total possible 2,275,463,135,232,000 tests. That's pretty amazing. Congratulations!!!
Now if we have time for only the first 50% of coverage, can we use the analysis coverage chart to know how many tests get us to 50% coverage and the grid in Create Tests to pick the first nn tests? How do we know which are the most important tests and which are the least important ones?
Photo of Porcupine

Porcupine

  • 5 Posts
  • 0 Reply Likes

Posted 6 years ago

  • 1
Photo of Justin Hunter, Hexawise Founder

Justin Hunter, Hexawise Founder, Founder and CEO

  • 246 Posts
  • 15 Reply Likes
Karen,

Thank you for your kind words!

You wrote:

>>>>>"Now if we have time for only the first 50% of coverage, can we use the analysis coverage chart to know how many tests get us to 50% coverage and the grid in Create Tests to pick the first nn tests?"

Simplified answer: yes.

You asked:

>>>>>"How do we know which are the most important tests and which are the least important ones?"

Simplified answer: the first tests are generally the most important ones.

For a more complete and nuanced answer, please see:

http://help.hexawise.com/s/help/m/743...

Pay particular attention to the caveats / disclaimers at the bottom of that explanation (which are cut and pasted here...)

A few things are worth pointing out about the coverage information in these charts:

First, when used correctly (and thoughtfully), it can be extremely useful. It gives you a quick method for objectively assessing: "how much extra testing coverage are you achieving with each new test?" and "how much testing is enough?"

Many testing teams have a rule of thumb, for example, to stop executing the Hexawise-generated tests after they have achieved 80% coverage because they can clearly see diminishing marginal returns to further testing after that point.

The second thing to keep in mind is cautionary. As George Box says, "All models are wrong. Some models are useful". It would be a mistake to look at the graph, see "100% coverage" has been achieved after the final Hexawise-generated test and conclude that the tests cover everything that should be tested.

An "Analyze Tests" chart generated by Hexawise, like all software testing coverage reports, is an imperfect model of what should be covered (which is itself based on an imperfect model of the System Under Test). There could be significant aspects of the System Under Test that were not entered into the "Define Inputs" screen. It is important to remember that one or more of those excluded aspects (a hardware configuration? a software configuration or plug-in? the order in which actions are executed? whether a user is navigating with mouse or a keyboard? whether or not "submit" buttons are clicked multiple times quickly? etc.) could potentially cause a defect that might not be identified in your current set tests.