Recently, the developers of World of Warcraft announced the creation of a “Community Council”–a means of gathering feedback from players and allowing select players to interact with the development team. This is a bad move on Blizzard’s part and I should know; I have first-hand experience being on a community council for a large, rabid, devoted fanbase: Back in the early 2000s, I was made a permanent member of the ToolArmy community council (ToolArmy is the official fan club for the eccentric prog-metal band Tool). I left the forums before I could even take my seat because I saw what a devolved shitshow the “council” would become–and did become–and why the fan club is now not what it was initially designed to be.
The truth is, this “council” of elite WoW players will fail to help guide the game’s growth because of one simple factor:
Yes, bias is real and we all have it. Bias basically means a set of personal motivators and desires that guide our behavior. If you tell me my ideas are good and you love them all no matter what I do, I’ll probably agree with you because I’m biased to believe I make good decisions (as are we all). However, if there are one hundred people and twenty of them love my ideas, 20 of them hate my ideas, and the rest are indifferent, I’ll probably assume indifference is close enough to approval to declare I have a majority of support and the 20 who voice dissent are just crabby. I’ll think that the 20 who agree with me “really get the big picture” and give greater weight to their opinions in future; the 20 who voiced dissent will be largely ignored.
In most psychology doctoral programs there’s a class entitled “Tests & Measures” or “Metrics & Scales” or something along those lines. In these classes (which, along with stats and research methods are considered “make or break” courses), psychologists are taught that a good way to gather qualitative data is by forming a randomly selected focus group (usually 7 to 12 people) and then interviewing them using open-ended questions that allow the participants to generate rich narratives. The narratives are transcribed and then mined for themes. After a point, a number of themes emerge and dominate the discussion and this is called thematic saturation. Once you have saturation of themes, you’re ready to design an instrument that can explore these themes.
After the initial instrument is designed, it is deployed to a large sample of the target population. Once the data are retrieved from this initial instrument, a number of statistical tests are employed to analyze the data. From this phase, the instrument is refined and latent variables called factors emerge. Performing factor analysis is akin to magic because it really does reveal the hidden dimensions in human motivation, behavior, and preference; items that emerged under different themes in the scale suddenly load on different factors. The initial instrument is redesigned with the factor model in mind (often stripped of unnecessary items) and redeployed to the target population.
Once data are gathered from the redesigned instrument, confirmatory factor analysis is performed to make sure that the desired factors emerge once again. If the factors emerge and the items load as anticipated, you can be pretty much be certain that the instrument is reliable and valid–that is, that the instrument works and it measures what it purports to measure.
If Blizzard was serious about improving game design, they would follow this method as opposed to the bias-laden “community council” model. A single PhD student given the right access and right funding could achieve better results from this method than 100 bickering voices on a forum. An actual PhD in psychology with experience in instrument design could have the hidden dimensions of game enjoyment identified in a fortnight with solid recommendations what not to touch, what to expand, and what needs a radical redesign.
Basically, what I see with the promise of a “community council” is a green-light and a scapegoat for bad design. “The Council Has Spoken!” will be the narrative for every awful change and the players will take the heat–much as the Blizzard devs have already done by blaming in-game toxicity on Asmongold.