What is keeping inputs at the same level of detail mean?

  • 1
  • Question
  • Updated 10 years ago
  • Answered
Under hints, what does this statement mean exactly "Keep inputs at the same level of detail"? Can you please provide a good and bad example to illutsrate this point?
Photo of Ed

Ed

  • 25 Posts
  • 0 Reply Likes
  • happy

Posted 10 years ago

  • 1
Photo of Justin Hunter, Hexawise Founder

Justin Hunter, Hexawise Founder, Founder and CEO

  • 246 Posts
  • 15 Reply Likes
Official Response
Ed,

Thank you for your questions! They are very good ones.

In this question, you ask about "same level of detail." In another question, you ask about "realistic weightings."

I believe the example below provides a partial answers to both of these questions. I'll focus on your "same level of detail" question with this answer.

Imagine you're testing the photo editing capabilities of Google's (cool and free) online photo album & photo editing application, Picasa.

Please see the test design consideration comments here: http://skitch.com/justinh/nq4nk/hexawise-picasa-photo-editing-sample-plan

The "level of detail" point is addressed in the first couple paragraphs which state:



"For demonstration purposes, we are intentionally limiting the amount of detail included in the sample Hexawise test plan.

In practice, testers should explore each feature in more detailed ways. Instead of simply instructing a tester to test warmify and the crop features together, it could be useful to specify approximately how much to warmify and approximately how much to crop and/or what kind of cropping to be used (3X5 vs. 8X10, etc.). The relatively simple example in the sample plan included in the Hexawise tool does not specify those details but a plan could easily be created that includes much more detail."



There are several points that are relevant here:

1) When you are considering how much detail to include for each value, this is an important consideration that will drive your test cases:

a) Are you comfortable instructing the tester to simply "include a test for this feature" to see if it interacts appropriately with other features? If so, a simple "Yes / No" for each feature could be appropriate for the output of the Hexawise tool.

b) Sometimes, though, you'll want to provide more explicit guidance to testers (e.g., instead of simply "yeah, crop the picture", you might want to specify "crop the picture to approximately 25% of it's current size and do so with a 3X5 shape.").

c) Which way is better for a given test will depend upon the context of the test plan. Do you and your team value Exploratory Testing principles (in which tester curiosity and ongoing questioning like "hmmm, that's interesting and not what I expected to happen, I wonder what would happen if I pushed that a little further...?" are encouraged), then you might be more comfortable with leaving instructions at a little bit higher level of detail (Y / N). If you are working in a firm that really values highly detailed test scripts, you would probably want to add additional details to not only one parameter value but all parameter values.

d) Testers who are getting started using pairwise (and more sophisticated n-wise combinatorial test case design tools like Hexawise) often don't have a good understanding of the impact of # of parameters and the # of values have on the number of tests required to execute. (Hint: the impacts are often very different - see "point 2" immediately below).

2) The number of values per parameter will heavily influence the number of test cases required for you to execute (often much more than the number of parameters in a plan). In other words, if you have a test plan with 40 parameters with only 2 values per parameter, you would only need 12 test cases to achieve complete pairwise coverage. This means that at least one of those 12 test cases would test for every possible imaginable pair of values (7 = Y and 14 = N? Covered; 18 = Y and 20 = Y? Covered; 2 = N and 13 = N? Covered; 2 = Y and 13 = Y? Covered; 2 = Y and 13 = N? Covered; 2 = N and 13 = Y? Covered; any first value and any second value? Covered). This 12 tests are from over 1 million possible tests. Hexawise can create extraordinary efficiency gains in this type of test plan.

If you have a test plan with only 2 parameters, you would need to execute every possible combination to achieve pairwise coverage. Therefore, if you had a test plan with 2 parameters that each had 20 values each, all of the 400 possible test cases would be required to execute. In (rare) test plans such as these, Hexawise can quickly generate test cases and ensure no combination is forgotten, but cannot create savings during the test execution phase.



Takeaway from this: it would be a bad idea to create a test plan that had 18 parameters at the Y/N level of detail and 2 parameters with 20 values each worth of detail. What should you do to avoid this? Several strategies (which can be used simultaneously) are worth considering.

A) Reduce the number of values for the parameters that have the highest number of values associated with them. This can be done by grouping values into equivalence classes, for example, and/or by using the Extend Values feature. Instead of listing 20 zip codes as 20 separate values, list 4 appropriate categories of zip codes and with the Extend Values feature, identify 5 values per category.

B) Where the maximum number of values per parameter in a test plan cannot be reduced beyond, say 8 values for 2 of the parameters in a test plan, you'll have to execute at least 64 test cases to achieve complete 2-way (or pairwise) coverage. If you're in a situation like this, feel free to play with the value descriptions that only have 2 or 3 values per parameter. Could it be useful to add more values? (e.g., instead of "warmify" = Y or N (2 values), you might as well include additional detail, like, "warmify" = "way to the left" "a little left" "slightly right" "way to the right" (4 values)); chances are you won't add to the number of tests you'll need to execute and yet you'll add additional variability to the tests you're executing (which will increase the odds that you'll uncover more defects). It is worth iterating test plans to consciously examine the tradeoffs between, e.g., (more vs. fewer tests, more vs. fewer values per parameter, values grouped by equivalence class (that can reduce the number of tests) vs. risks associated with erroneously grouping two "different" values into the same equivalence class (and potentially missing a defect because of it), etc.

C) Because of (A) and (B), "all else being equal," (a) I like to try to find ways to expand value descriptions from 2 to 3 or 4 where possible, and (b) reduce the number of values from 8 or more to 4 or 5 where possible (often using the Expand Values feature to mitigate the risk of "dropping values").

I hope this answer helps. Thanks again for your question. Please keep them coming.

- Justin Hunter
Founder and CEO of Hexawise