Article on beta testing

[Don]

Saw this article and I think it's dead on http://www.tentonhammer.com/node/66885

Unfortunately it also is kind of sad that you're beta something but people really aren't interested in helping the game succeed. All the recent betas I've been to are just filled with people who are trying to get a leg-up on others when the game goes live, as opposed to actually trying to help the game become better.

[SineSwiper]

Good stuff. I think ultimately the key is to use the betas as a marketing tool AND to listen to your testers. If the game is truly a test, then listen to your fans and change the game according to how it needs to be fixed.

[Don]

Well the problem is who do you listen to. It's easy to release a game and see it flop and then say see you should've taken more time, but as the article mention you got money deadline. Plus, most people in your beta tests are morons and listening to them can be harmful. There are people who said WoW was not ready and needed more polish and that is clearly wrong. People who say that are just guys who wanted to play WoW for free for longer (WoW is essentially unchanged at the group level between beta and release). So the question is who are you supposed to listen to?

I don't think you'll ever get a time like when EverQuest originally betaed where you actually have a bunch of beta testers that are truly trying to improve the game. For example look at WoW, pretty much all the guys betaed just wanted to play it for free, so they'll just pretend the game has problems to keep the beta going for longer. If WoW actually listened to this guy it'd be defintely detrimental because WoW isn't going to get any better, and each day it is not released is one less day to be making money.

Then you go to a game like WAR maybe it was not quite finished in beta, but it's not really clear if taking more time would have helped. Then you have games like Vanguard where it's pretty clear it would not have gone anywhere no matter what.

I participated in a few EverQuest expansion betas and the quality of the players just suck. I think like half of the players just go buy the uberest gear ever and then try to solo something with beta godmode buffs. Like the article said, they're not people intersted in helping your game. They're just guys from to play it for free/get head start before the other guys. I know one of the guild that beta tested the latest EverQuest expansion said whenever they tuned an encounter they wanted to make sure nobody can possibly beat it besides them so it keeps the gimps out. How can you possibly have a meaningful beta when you're basically working with some of the biggest scums as your beta testers? But somebody still has to test this stuff.

[Julius Seeker]

They don't listen to complaints about quality and such of graphics, etc... The major thing they are looking for is final balancing, as well as usability issues (the game may have some components that may make sense to the dev team, but not to the uninitiated user who doesn't have access to the design doc.

[SineSwiper]

There are people who said WoW was not ready and needed more polish and that is clearly wrong. People who say that are just guys who wanted to play WoW for free for longer (WoW is essentially unchanged at the group level between beta and release). So the question is who are you supposed to listen to?

See, here's the problem. If you advertise your "beta" as a free demo, then it's a free demo. Instead, tell your beta testers that if they're not going to test the game and submit bug reports, then they can STFU/GTFO and use the slot for somebody who's actually going to test.

You HAVE to have a group of (relatively) unbiased testers that haven't had 5 years buried in the game already. They can tell you if something is out of whack with the balance, or that bugs with the interface.

Even with good development houses like Turbine, I'm still puzzled at the type of things that go through the patch like normal, even though the beta testers of the patch are screaming that some glaring bugs need to be fixed. For example, there was a major delay with the Riddle skill, and people saw it in the beta server. Many people reported the bug, but the patch went through anyway. So, now, a skill that was fine is now bugged, despite the fact that the company knew it was a problem. Instead, fix the problem, then submit the patch. Also, there have been a few Book patches (major patches with new areas) where they have to spend a lot of hours on the production server to fix rubberbanding issues. Why wasn't this caught on the beta server, which was designed for this sort of thing?

[Don]

The problem is that from a dev point of view it is very hard to distinguish with someone who want something made easier or changed for his personal gain or for the good of the game. It's easy to say 'listen/get the right guys' but you really can't tell who are the right guys easily.

[SineSwiper]

Well, the right guys are the ones who don't want to submit bug reports. It's like a job at work. If you don't do your work, you get fired.

[Don]

But Beta today is more like a PR stunt, so you don't really want to 'fire' the beta testers so to speak because that gets you bad PR. It's a tough situation to be in.

[SineSwiper]

Well, the betas probably shouldn't be a PR stunt. Just tell them ahead of time in the initial welcome email that "This is not a PR beta, this is a real beta, and we'd expect you to actually test the game."