If You Can't Average the Answers, You Don't Have a Crowd

Skip this step, and you'll make everything upstream worthless. Why, you ask? Because the crowd's wisdom you seek depends entirely on averaging. Which means your question has to produce answers you can average. No average, no mission-critical intel. For example, "What's your favorite color?" doesn't work because there's no meaningful average of blue and green. Here's what does work:

• Percentages — "What percent chance does this campaign hit its awareness target?"
• Likert scales — strongly disagree to strongly agree, mapped to numbers (this is easy to screw up, so be sure you know how to build a valid and reliable scale).
• Numeric ratings — "Rate this logo's memorability, 1 to 10"
• Binary questions — encoded as 1s and 0s, then averaged

Some good news is that mean or median both work fine as your aggregation method. However, don't make the mistake of skipping the scale entirely and just reading open-ended comments, because now you're not aggregating a crowd. Nope, you're just impressionistically reacting to whoever wrote the most vivid comment. Or, the loudest voice in a thread starts to look like consensus even when it isn't one.

So, my fellow marketers, whether you're testing subject lines, hero images, or landing pages, put a number on clarity, relevance, and appeal, collect it from your properly-sourced crowd, and average it.

That's the whole method, and it's not complicated. It's just a discipline most people don't actually follow.

PS Never cross a river if it is on average four feet deep, as localized deep spots will drown you even if the average is safe.