Sam Ng's dashboard

Sam Ng set one of Sam Ng's replies as an official response to "Required number of participants for "significant" Chalkmark results?" in Optimal Workshop

Sam Ng replied on December 18, 2009 07:35 to the question "Required number of participants for "significant" Chalkmark results?" in Optimal Workshop:

Hi Amy,

Great question. The short answer is at least 30.

The long answer is it depends. We ordinarily would suggest that you aim for about 30 per discreet group. That is, if there are known differences between say participants living in New York and participants living in New Delhi then you would want 30 from each city.

Having said this, we did build this tool with the intention that some feedback is better than no feedback. So even if you got 10, I would hope that it gives you some insight.

Lastly, for some interesting reading, our current stats show that:

- The number of participants for Chalkmark survey averages at 119 participants (mind you there are some BIG surveys running!.
The median number of participants is at 33.

Hope that helps! As for the upper limit, no there isn't really one if you wanted to go crazy, you could.

Sam Ng set one of Sam Ng's replies as an official response to "Sales please!" in Optimal Workshop

Sam Ng replied on December 18, 2009 03:53 to the question "Sales please!" in Optimal Workshop:

Hi Ben,

Thanks for your enquiry. In short, no we do not currently have a reseller program though we are considering a partner network of high quality user experience companies and an affiliate program.

The best way to purchase is by signing up for an Optimal Workshop account and paying by credit card after logging into your account.

You can sign up here:
https://www.optimalworkshop.com/works...

Thanks!
Sam

Sam Ng posted an update in Optimal Workshop on December 15, 2009 02:53:

How are Treejack scores calculated?

Treejack provides high level indicators of participants' success with your information architecture.

These indicators are in the form of scores ranging from 1 (bad!) to 10 (great!). The scores are calcuated using 3 different measures:

1. Success is the percentage of participants who chose an item in the tree which matches a 'correct answer' as specified by the survey originator.
2. Speed is the percentage of participants who completed a task within 2 standard deviations of their average task time for the survey.
3. Directness is the percentage of participants who did not backtrack (i.e. who went back up the tree to follow a different path) when completing a task.

Speed and directness combine to give an impression of the participants' confidence.

The score out of ten for a task (and for the overall score of the project) is determined as follows. We turn the success percentage, and a composite 'confidence' score (made up from the speed and directness percentages 50/50 weighted), into a score out of ten each.

These correspond to the row and column of a cell in a ten-by-ten grid, which contains the overall score of the task (or project). This grid stipulates that if a participant was unsuccessful but was extremely "confident", that is a bad score. Conversely, if the participant was successful and confident, that is a very good score.

We will regularly monitor the accuracy of this scoring algorithm and continue to improve it as we collect more data.

Sam Ng posted an update in Optimal Workshop on August 19, 2009 04:57:

OptimalSort results spreadsheet - known issue with data import

There is a known issue with the OptimalSort results spreadsheet if you have opted for "Anonymous" participation.

The issue is that the Excel template cannot import data from the "Raw data CSV" file because the headers in the CSV file are inconsistent with what the template is expecting.

We are working on identifying a fix for this. Meanwhile, to work around this:
- Rename column B in the CSV file to "email"
- Rename column C in the CSV file to "email_confirmed"
- Import the new CSV file into the Excel template.

A comment on the question "Participant Email Addresses" in Optimal Workshop:

Hi Tim,

As William has commented below, we are working on this because we recognise how difficult this is for some people. It's definitely in our top 3!

Sam – Sam Ng, on July 13, 2009 05:07

Sam Ng set one of Sam Ng's replies as an official response to "Do not include the words 'duplicate' at the beginning" in Optimal Workshop

Sam Ng replied on July 07, 2009 04:54 to the idea "Do not include the words 'duplicate' at the beginning" in Optimal Workshop:

Hi Eleanor,

We've just deployed functionality that implements this idea. Thanks for the suggestion!

Sam

Sam Ng replied on June 25, 2009 23:40 to the idea "Chalkmark: how many questions?" in Optimal Workshop:

Gotcha. We'll mull over this.

Sam Ng replied on June 25, 2009 11:56 to the praise "Endorsement of Treejack" in Optimal Workshop:

@lynseystruthers - Thanks for your encouraging words of praise for Treejack. It's a great feeling to know that our work is helping people like you build a more usable world!

Sam Ng gave praise in Optimal Workshop on June 25, 2009 11:56:

Endorsement of Treejack

Another rousing endorsement for @optimalworkshop's TreeJack: Results easy to interpret & allow for nav changes based on real user feedback.

Sam Ng replied on June 25, 2009 11:52 to the idea "Chalkmark: how many questions?" in Optimal Workshop:

Thanks Deborah for this idea. To clarify, I presume you mean you would like to see the number of tasks for each Chalkmark survey on the Chalkmark dashboard page?

In other words, when you log in to Optimal Workshop and click on the Chalkmark tab, you would like the ability to see the number of tasks for each Chalkmark survey? (In the same way you can currently see the number of survey responses if the survey is active.)

Thanks,
Sam

Sam Ng replied on June 25, 2009 11:50 to the idea "Review/Edit Existing" in Optimal Workshop:

Hi Rebecca,

It'd be great to learn a little more about what you are suggesting as an idea. Is it to enable the ability to edit an existing task once a survey has been set to "Active"?

Thanks,
Sam

Sam Ng replied on June 25, 2009 11:48 to the question "Participant Email Addresses" in Optimal Workshop:

Hi Deborah,

This is a bit of an oversight on our part. We currently don't report on the email addresses that are collected in Chalkmark. Chalkmark and Treejack use the same components for identifying participants (eg email address) and we report on individual participants within Treejack but not (currently) within Chalkmark.

Email addresses are used simply as the easiest way to uniquely identify survey responses. This is currently used in the results for Treejack, but not (yet) in Chalkmark.

In the coming weeks we will be working on allowing for anonymous survey participation (so that participants do not need to provide an email address) and a number of enhancements to Chalkmark.

If you need the email addresses, for example to do a prize draw, contact us directly and we will provide run a manual report for you.

Once again, sorry for the run-around. We hope this will make you feel a little happier.

Thanks,
Sam

Sam Ng set one of Sam Ng's replies as an official response to "Participant Email Addresses" in Optimal Workshop

Sam Ng replied on June 25, 2009 11:32 to the praise "Value for money - Treejack and Chalkmark" in Optimal Workshop:

@chiladelphia - Thanks for your kind words. We hope to make user experience affordable to the masses.

Sam Ng gave praise in Optimal Workshop on June 25, 2009 11:32:

Value for money - Treejack and Chalkmark

Quanatative automated user testing tools:Keynote too expensive and clunky, Userzoom too expensive, Treejack & chalkmark totally worth it and

Sam Ng replied on June 22, 2009 01:26 to the problem "Overall success average is averaging incorrectly" in Optimal Workshop:

Hi Ruth,

Thanks for reporting this problem. Sorry we missed this simple but critical calculation!

We will get onto this straight away.

Thanks!
Sam

Sam Ng replied on June 22, 2009 01:25 to the idea "Do not include the words 'duplicate' at the beginning" in Optimal Workshop:

Hi Eleanor,

Thanks for pointing that out to us. Sorry that this is clearly a bit frustrating for you.

When you duplicate just a single task/question it appends "Duplicate" in front of the task. Unfortunately it appears that when duplicating the entire survey, the same method has been used to duplicate the tasks.

We will get onto this pronto!

Cheers,
Sam