Why is Hexawise creating tests with no added coverage?

  • 1
  • Question
  • Updated 3 years ago
I noticed that, when I generated tests for a scenario, the last 2 tests in the list provide no new coverage and are redundant with earlier tests. What I noticed is, there were only two parameters in "black" and are identical across those tests, and all the others are purple italic, meaning they do not provide new coverage. If that is the case, why do I have 2 tests with identical black entries in the test list?
Photo of Ilhan Dilber

Ilhan Dilber

  • 4 Posts
  • 1 Reply Like

Posted 3 years ago

  • 1
Photo of Sean Johnson

Sean Johnson, CTO

  • 270 Posts
  • 26 Reply Likes
Ilhan,

Taking your description of the scenario at face value, I have no explanation as to why you'd have 2 final tests w/ the same coverage achieved in each (black value tuples). I'll look into this a bit more and get back to you.

Cheers,
Sean
Photo of Sean Johnson

Sean Johnson, CTO

  • 270 Posts
  • 26 Reply Likes
Ilhan,

Do you by chance have multiple parameter values of the same parameter defined with the same name?

e.g. State: FL, GA, SC, NY, NY, NJ, MD

This could make it look like Hexawise is giving you 2 final tests that achieve the same coverage, both pairing NY with the same other parameter value.

Hexawise generates tests using the mathematical model of your test plan, not the English (or French!) parameter names and values. So, in the case above, the first instance of State: "NY" and the second instance of State: "NY" are different parameter values, even though they happen to get the same label.

I suspect this is the scenario you're seeing.

Cheers,
Sean
Photo of Justin Hunter, Hexawise Founder

Justin Hunter, Hexawise Founder, Founder and CEO

  • 246 Posts
  • 15 Reply Likes
Ilhan,

Good question. That should not normally happen.

One "edge case" that could explain this unusual behavior: If you include the same parameter value multiple times for a given parameter (e.g., Gender = Male, Male, Male, Female), a "human"-reading of two test cases which both say Gender = Male and Driving History = Average would see no difference between the 1st or 2nd; they're both for all intents and purposes identical.

But from a combinatorial coverage standpoint, the Hexawise algorithm will pair up the first "Male" value with every other value in the plan, and will pair up the second "Male" value with every other value in the plan, and will pair up the third "Male" with every other value in the plan.

That would explain what you appear to be seeing. If this is in fact what you are seeing, (a) you would be able to see an added "green" pair being covered in the final test in your test set when you look at your coverage chart, and (b) you could safely decide NOT to execute the final test in your plan (and you would still have the confidence that the 2-way coverage you're seeking to achieve has, in fact, been substantively achieved even without executing that final test).

Justin
Photo of Ilhan Dilber

Ilhan Dilber

  • 4 Posts
  • 1 Reply Like
That was indeed the case.  Unfortunately our brains read in English not in math? :) Getting rusty from not using it for several weeks :)