← All posts

Understanding Matchups with Reinforcement Learning

One of the challenges of playing any TCG is learning to play against all of the likely matchups you'll encounter. In Star Wars: Unlimited for example, even if you've settled on a specific deck you're going to play with, your resourcing decisions, sequencing, and overall strategy is going to depend on your opponent's deck and its strengths and weaknesses.

The Opponent

In order to learn how to play a matchup well, you need an opponent that is going to play the opposing deck effectively. For SwooLab, that means an AI opponent that learns to play through reinforcement learning. By playing against itself across millions of games, with millions of different randomly generated decks, the AI can learn to use all the same basic principles that a human player would learn.

The AI consists of a single neural net model that outputs

1. a probability distribution over the legal moves at the current state (the policy), and
2. an estimate of the probability of winning the game (the value).

The policy tells you what actions you should take, and the value tells you how good each position is for you.

Training the model

The model is trained in iterations that involve first collecting thousands of games of data, and then training on that data.

One common way to compare the skill level of two players is to use Elo ratings, which assigns a rating to each player based on their win-loss record. While there are some issues with applying Elo to a game with random events (like a shuffled deck) and non-transitivity (the rock, paper, scissors effect applied to decks), we can still use it to show how the rate of learning changes over time during the training process.

The chart below shows the Elo rating of the SwooLab model over the first 3,600 training iterations.

linear log

It's interesting to see what the model learns early on vs later in the process. For example, early in training, moves are mostly random, and the model learns to not value upgrades or most events. An upgrade played randomly is just as likely to benefit your opponent as it is you, and an event aimed at the wrong unit could be detrimental. It's only after it learns to correctly target units with each event and upgrade that it begins to learn the value of these cards. We'll cover this in a future article on the site.

Simulating the Matchups

Practicing against the AI model is helpful preparation for playing the real thing, but we can do more than that. I ran an experiment, simulating some matchups over thousands of games and observed how the model plays it from both sides.

First thing to to was decide which matchups to simulate. SwooLab's AI uses a custom-built rules engine that currently only supports the first set, Spark of Rebellion. The other sets will be added soon, but for the purpose of demonstration, I've selected a set of 12 decks from the Set 1 meta as a rough stand-in for what the true meta once was.

The experiment consisted of

• Twelve decks. Taken from near the top of SWUDB's top decks list, plus the two starter decks (Luke and Vader) as a baseline to measure against.
• 132,000 games. All 66 pairings, 2,000 games each. That gives us roughly ±2% for a margin of error on a single cell of the matrix — enough to distinguish a 60/40 from a 50/50, not enough to distinguish a 50% from a 52% on its own. Of all these games, just 47 were draws.
• Lookahead search. The models searched 16 positions per move in this experiment. This improves the AI opponent's Elo rating significantly by about 70 points. This also required 16x more compute to run all the games, which is why the experiment was limited to 16 searched positions and no more.

The matchup table

Each cell is the row deck's win rate against the column deck, over every game the two played in either seat, so a cell and its mirror across the diagonal always sum to 100%. Decks are ordered by overall Elo rating.

Hover any cell for that matchup's details:

• How long the games ran, and which deck won the shorter ones
• Which opening hand cards helped each deck the most
• Which card hurt each deck's chances most, if it never got drawn
• How much of an edge going first gives each deck

A simulated metagame

A matchup table is only half the story. What matters competitively is how a deck does against the field it will actually face. And players are going to attempt to anticipate this field and plan to bring a deck that performs well against it. So it's not just a question of "which deck is best", it is about predicting which decks will actually get played, which is influenced by the win rates in the matrix above.

We can try to estimate the share of each deck in the field by treating the metagame as its own self-contained game. Each player selects a deck from the field with a certain probability, and then outcomes are determined by the estimated win rates in the table. Games like that have a Nash equilibrium: a single mix of decks that nobody can beat by switching to something else. Find the mix and you have found the field the format is pulling toward.

To find the Nash equilibrium, we find the deck mix that maximizes the worst case across every possible opponent. The solution has a property that makes it easy to verify: every deck that appears in the mix wins exactly 50% of the time against it, and every deck left out wins less than 50% of the time. Trying to pick a dark horse only leads to suboptimal results, which is what makes it an equilibrium.

The result, and what happens without Boba

I solved for the Nash equilibrium in a field consisting of only these 12 decks. The format collapses to just three viable decks of the 12, with two thirds of the field playing Boba Cunning:

How to read the numbers

The twelve-deck field is not a healthy format. Boba Cunning is dominant, only three decks are playable and nine are not. The best of the nine remaining decks only manages 49.3% against the three.

Now drop both Boba Fett decks and rerun it on the remaining ten. The format that emerges is much healthier: five decks in the mix, the largest at 38.8% rather than 64.3%. The intransitivity gets richer too: Palpatine crushes Iden and Krennic but folds to Sabine, Sabine loses to Han and Krennic, Han loses to Krennic.

It is also a much more robust result. Re-solving the equilibrium two thousand times over resampled matchup data, all five decks appear in at least 99% of the resamples. In the twelve-deck field, 27% of the time the meta collapses to just Boba Cunning. Krennic's place in the meta rests on a 2-point matchup edge, which is not much more than the noise from resampling. And without Krennic in the mix, Palpatine simply loses to Boba.

What this does and doesn't tell us

What this demonstrates is that reinforcement learning and simulations can be used to analyze a meta without playing a single game by hand. We can predict ahead of time whether a deck is likely to become too dominant and we can explore how banning specific cards will affect the new meta that forms in its wake.

However, these are not predictions of what people would have brought with them to a tournament on any given weekend. They are the answer to a narrower question: what would a field have to look like for no counter-pick to exist?

To answer this, in addition, I made a number of notable simplifications:

• There are mroe than twelve decks. These twelve were picked off the top of a list; plenty of real archetypes are missing entirely. The 9 decks sitting at 0% share are only unplayable against this pool, not against Spark of Rebellion. Add a thirteenth deck that performs well against Boba and the whole equilibrium moves.
• Every list is fixed. No variations on deck lists, no tech cards, no sideboarding, no adapting between games of a match. Players answer a bad matchup by changing a few cards, and a couple of well-chosen cards can move a matchup further than anything on this page. Measuring that properly is its own post.
• Skill varies a lot. A real win rate depends on who is holding the deck and how prepared they are for the matchup. Win rates and meta games are also influenced by variances in player skill. Some decks are more difficult to pilot correctly. Other decks become popular for being easy to pilot. Popularity also brings attention, meaning opponents are more likely to be prepared. None of this is accounted for because the model only played using a single policy.

Conclusions

Despite the simplifications, we have the benefit of hindsight and can see that this approach supports the same conclusion that FFG ultimately came to (albeit only after Shadow of the Galaxy), which is that Boba Fett was overtuned. Simulations powered by reinforcement learning can reveal how a player can approach a given matchup with ease.

Questions about our findings? Email admin@swoolab.com.

Play against the model