Call of Duty Study Finally Settles One of the Biggest Debates in Online Gaming

Competitive shooters stir up a lot of strong opinions from their players — from tier-list debates to whether camping out near a spawn point is cheating or just annoying — but one of the biggest debates raging around games like Call of Duty is whether skill-based matchmaking is a good tool for balance or a game-destroying plague. Activision recently weighed in on the question with a study of its own game showing that, despite how unpopular skill-based matchmaking (or SBMM) is in the community, players may actually appreciate it more than they think.

In games that use SBMM, player performance is considered when teams are formed before a match, with an automated system doing its best to keep the average skill level of players about the same. The theory behind this is that players don’t necessarily want to play impossible matches against other players who are way better than them, while playing against competitors who they can easily stomp is no fun at all. On the flip side, opponents of SBMM say that this balance makes games boring, since you can never really test your skills against someone who’s out of your league or enjoy crushing an easy opponent.

Activision recently put that to the test, running a study in early 2024 designed to see how SBMM affects player retention. In a report published last week, the studio detailed how it progressively lowered the amount of influence SBMM had in picking teams for 50% of Call of Duty: Modern Warfare 3 players in North America. The results were either shocking or extremely predictable, depending on which side of the SBMM debate you fall on.

According to Activision, lowering the effect of SBMM increased the rate at which players quit matches by 80% and reduced the rate of returning players. In other words, with SBMM toned down, most people were less likely to see matches through to the end and more likely to stop playing altogether. Only the top 10% of highly skilled players kept playing the same amount. That means fewer people were around to play the game, making wait times for matches longer and all but ensuring that remaining low-skill players would be matched with the high-skill players who largely stuck around. Activision suspects that the effect of fully removing SBMM would be disastrous for the game.

“This test only deprioritized skill in the matching rules,” Activision said in the report. “If it were completely removed, we would expect to see the player population erode rapidly in the span of a few months, resulting in a negative outcome for all our players."

The most striking thing about the paper is how much it’s at odds with commonly expressed player sentiment. SBMM is by no means universally hated, but there’s a large contingent of players who view it as essentially ruining the game. But with such a dramatic drop in player retention with SBMM reduced, it’s almost guaranteed that plenty of players who profess to hate SBMM actually had a worse time with the game when it was turned off. More importantly, the drop in the player pool the study saw means that even if some players would theoretically be happier with SBMM turned off, its removal would still ruin the game since there simply wouldn’t be enough people to play against.

Activision’s report is also interesting in just how rare it is to see a large developer be this transparent about how it balances its games and works to retain players. Game studios are notoriously secretive, leaving the gaming public without a clear idea of how almost anything that goes into making games works. The technical details of the report aren’t exactly scintillating on their own, but the sheer fact that Activision is revealing information about how it tunes its games to keep people playing makes it worth a look.