I was just reading through an interview about Jonathon Blow’s new game called The Witness when I ran into a very nice quote regarding focus testing.

“When you’re too reactive to playtests, says Blow, “You get this totally featureless game out of doing that. No game ever made by man provides a perfect experience to anybody. And attempting to do so can easily — speaking for me — carry me away from what’s really important about the game. So I try to rein myself in about that.”

This is an excellent point that I try to champion myself in my professional career. Focus testing is important to get a general idea for how people are “reacting” to your overall game concept and then to point out the very large flaws that you may have missed when creating smaller/tinier experiences within that constructed game. Here, let me break it down for you.

1. Focus test at the right time, with the right people. What does your core audience think about your game concept without even playing the game? Is the subject material interesting for them? Does the concept excite them and get them engaged? I think this is a very tough set of questions to answer when you sit a 49 year old female down and ask them what they think of your first person shooter concept. You are mismatching your hard core audience with random, potentially un-interested parties (and by un-interested I mean uninterested in games in general). It doesn’t work. Focus testing with lots of different people outside of your core demographic is good when you are unsure of who your target audience is to begin with. Often times, you as a design team in a commercial setting will have a pretty good idea of who would want to play your game and who would not from the get go.

2. Focus testing for existing software should be used to point out the GIANT mistakes/holes in your original thinking/logic. IF you have missed something, it will become apparent and you as the creator will know. It will take the form of your “a-ha!” moments in your puzzles, only as a “designer of software for a player” instead of the “player of the designers software”. It will be something more like “Wow, I never thought that players would do that!” And the most important reaction to that is you AS THE DESIGNER will agree with the focus testers/players experience. Then and only then should you consider changing that interactivity based on that information.

Often times, when a player struggles with a game or a sequence in a game, most studios take those experiences as gospel and change the very heart of the interactivity to fit only fifty, maybe 100 peoples experiences with the software. In my math brain, this isn’t a large enough sampling to get an accurate reading of what to change in your software. If you guess that you will sell around 650,000 units, that’s only .015 percent!!! So whats the solution? Make changes only with certainty. Gamers aren’t as dumb as you think they are, even if they look like bumbling idiots in front of you during the focus test. Always have your designers make these calls. Those “a-ha” moments in realizing what your players are doing/not doing revolve around who designed it, not who’s running the focus test.

So what about the niggling little details, or the grey area? For example, let’s say you give focus testers about an hour to complete the first two levels of your game, and only 50% of your focus testers finished both levels, the rest didn’t get past level 1. What’s the take away here? Depends on the specific sticking points (puzzle, enemy, boss, navigation, etc.) but for the most part, you can’t get forget about the 50 percent who DID complete it! It’s very disappointing when I see broad sweeping changes made to software because someone says that 50% “isn’t good enough” for selling more units of a video game. Sometimes, when players struggle, it’s considered a good thing because they are actually being challenged and are learning how to play your game. And this is easy to tell, so long as you provide them with an outlet where they can express their experience in terms of difficulty/frustration and difficulty/boredom on the same scale. So often, I see the experiences of focus testers only weighed against half of the testing variables! There are FOUR things to look for, and the sweet spot is in the middle. Just because something is difficult, doesn’t mean it is frustrating! People can still enjoy an experience if it’s difficult, the idea is that it doesn’t skew into frustrating territory for too long. And of course the flip side to that is that if a game is too easy, people may become bored and lose interest. The sweet spot is in the middle.

Games are about overcoming challenges, and sometimes challenges require more perseverance than others. Difficulty is generally a hard thing to match to EVERY player.

Another thing to remember, too, is that completion is not necessarily the equation for happiness from a gamer. There are many cases where I have not finished a game or gotten very far only to say “that’s an awesome game, I really liked it”. And the opposite is true, I’ve completed games that I have found to be uninteresting, terribly designed and boring. Crackdown and LA Noire respectively, in case you’re interested. 🙂

As for The Witness, I look forward to playing it. Braid was a triumphant success in my mind and I eagerly await the new efforts from Mr. Blow and his newly formed company. I highly encourage them to release on the consoles, it will do much for the state of gaming on those platforms.