From xkcd:

Admittedly, I’ve never seen *A Beautiful Mind, *although I’ve come across the clip parodied in the cartoon above which, as the New York Times also points out, is not a great example of John Nash’s contributions to game theory. But I’ve always liked the way game theory, like other forms of modeling, builds upward from simple premises, and Nash’s work features in a good chunk of what is familiar to me in game theory. So, in tribute to Professor Nash (and wishing to learn a little more about game theory), I’ve put together a little tutorial on building games into an agent-based model.

Game theory considers how strategic decisions are made; these are usually conceived of in terms of the relative payoffs for each choice within a realm of possible choices for a given decision. Many of the games studied by game theoreticians are models and have a limited number of options, but depending on how the payoffs are structured and what knowledge is attributed to the players, these elementary games may produce non-intuitive outcomes and insights into behavior.

This description is perhaps better served by an example: imagine you find yourself in the produce section at the grocery store, and you have to choose between buying a delicious pumpkin that requires two people to carry and buying asparagus that are less enjoyable. If you and an acquaintance work together, you can bring the pumpkin to the register and both benefit from eating it. On the other hand, both of you could just buy asparagus separately and get less enjoyment. But there’s also another option: one of you could decide to buy the gigantic pumpkin, while the other could take asparagus. This would leave one person with a decent amount of asparagus and the other without any pumpkin because they are unable to carry it. How does one decide which strategy to choose?

On its face, it might seem like choosing to cooperate and taking the tasty pumpkin makes the most sense as this would provide the best payoff for everyone, but this is a risky strategy if you don’t know what the other person will do; you could be left trying to drag an immense pumpkin to the checkout all by yourself while the other person makes off with heaps of asparagus (the horror!). In that case, it might make the most sense to take the sure thing. This can be visualized as a decision matrix:

For this game (which is really the famous Prisoner’s Dilemma game), the asparagus-asparagus strategy represents a Nash equilibrium: a strategy under which both players would not benefit from changing strategies, even when the equilibrium strategies of other players is known. If Player 1 initially chose pumpkin, but knew it would benefit the other player if they switched to asparagus, then Player 1 would rationally change their strategy to asparagus as well to avoid being left to drag the pumpkin alone. But if Player 1 initially chose asparagus, they would not benefit from changing strategies, no matter whether Player 2 chose asparagus or pumpkin; therefore, the rational choice would always be asparagus. Under some games, there may be multiple Nash equilibria. For instance, we could change the values so that the benefit of defecting were the same no matter what the other player is doing, with a decision matrix that looks something like this:

In this case, there is no rational reason for a player to switch strategies knowing what the other has chosen, so the game has two Nash equilibria: both players choosing the pumpkin, or both choosing the asparagus. Both of these games are kinds of *non-cooperative games*, which are a class of games in which the players make decisions without consulting one another (and the type where Nash made his most well-known contributions).

Games explored by game theorists are usually conceptual, such that players are not typically constrained by any physical distance between them. Agent-based models, on the other hand, have the ability to incorporate space explicitly within the framework of the model, making it possible to inhibit or enhance agent interactions based on proximity. While the incorporation of space is not necessary for all models or games, many real world processes and entities are affected by space, so it might be useful (and dare I say fun?) to see how space might affect the outcomes of games. In that spirit, we’ll use NetLogo to build non-cooperative games within a spatial framework.

**Building iterated games in an agent-based model**

In this model, agents perform a random walk, selecting a direction at random and taking one step forward. Following each step, agents search for other agents within a given radius, **search-radius**. If any agents are present, one of them is chosen, and together they play a coordination game not unlike our produce aisle example above, receiving different payoffs depending on whether they choose to cooperate or defect. The model will be flexible enough so that the payoffs for both cooperating, both defecting, or losing and winning from a cooperate-defect scenario can be variables capable of being tuned by the user. We’ll call these, respectively, **cooperate-cooperate**, **defect-defect**, **lone-cooperator**, and **lone-defector**. These will be made into sliders at the front end of the model.

Before we do anything, we’ll want to figure out how we want agents to make decisions. We could assume that agents know the equilibrium strategies and will thus act accordingly. But what if our agents don’t know what the payoffs will be? In the absence of any information, agents could make their decisions randomly, but this isn’t likely to remain a reasonable course of action once the agent has played the game a few times (this concept probably has a name, but for now we’ll call it the “this ain’t my first rodeo” effect). An alternative is to let agents make decisions about whether to cooperate or not based on their past experiences with other agents. We can do this by asking each agent to keep track of the payoffs in a set of lists for each strategy. The relative numbers of choices of strategy in each time step can then be used to gauge how the population is making decisions.

`globals [ decisions ]`

turtles-own [ cooperate-history defect-history ]

Here, the **cooperate-history **and **defect-history** variables will be lists used to keep track of payoffs received when a particular strategy is employed. For example, if two agents cooperate and the **cooperate-cooperate** payoff is 5, then both agents will add 5 to their **cooperate-history **lists; however, if one agent defects and the other cooperates, and the **lone-defector** payoff is 6 and the **lone-cooperator** payoff is 0, then the defecting agent will add 6 to their **defect-history** and the cooperating agent will add 0 to their **cooperate-history**. The **decisions** variable will keep track of all agent strategy decisions for each time step, with 1 representing a cooperation and 0 representing a defection. We can use this as a gauge on population-level decision-making as the model progresses.

Now we can create the agents. What we need to consider carefully is how agents will make their initial decisions. We could start them out with no history whatsoever, and let them guess at first until they figure out a good strategy, but unless agents try both strategies at least once, they will unwittingly choose a random strategy and then stick with that strategy simply because the outcome of the other strategy is unknown. To get around this, we’ll seed the **cooperate-history** and **defect-history** of each agent with ten random outcomes from each of those choices.

`to setup`

clear-all

set decisions [0 0 0 0 0 1 1 1 1 1]

crt 100 [

setxy random-xcor random-ycor

set cooperate-history []

set defect-history []

repeat 10 [

set cooperate-history lput (one-of (list lone-cooperator cooperate-cooperate )) cooperate-history

set defect-history lput (one-of (list lone-defector defect-defect )) defect-history

]

]

reset-ticks

end

Here, we’ve taken the **decisions** variable and seeded it with an equal number of 0s and 1s, starting off with a mean value of 0.5; we do this so that the plot we create for this later has some values to use when the model is set up. Next, we create (**crt**) 100 agents, distribute them to random coordinates with the **setxy** command, and then give them **cooperate-history** and **defect-history** lists with randomly chosen outcomes for cooperation (either **lone-cooperator** or **cooperate-cooperate**) and defection (either **lone-defector** or **defect-defect**) respectively.

Next, we create our **go** command, where the model is controlled. Here, we want to make our turtles move, and, if any agents are within the **search-radius**, they’ll play a coordination game. It should look something like this:

`to go`

set decisions []

ask turtles [

set heading random 360

fd 1

if any? other turtles in-radius search-radius [

check-game

]

]

tick

end

In this code, the **decisions** variable, which is a list, is cleared in order to get fresh decisions from the new time step. This is done by converting it to an empty list **[]**. Next, each turtle is asked first to pick a random direction by using **set heading random 360**, and then asked to move forward (**fd**) one step. The agent then looks for other players. The **any?** command checks to see whether any of a specified group of agents meets a given criteria; in this case, we want to know whether there are any agents within the **search-radius**. If there are, we’ll run a routine called **check-game.**

The **check-game** routine has three objectives: first, it determines what the preferred choice of our focal agent is based on their **cooperate-history **and **defect-history **lists; second, it identifies the second agent playing the game and determines their preferred choice; and finally, it determines the payoffs for both agents based on the combination of their decisions. This routine is a bit long, so we’ll break it up based into a few parts, using ellipses (…) to indicate where code leaves off and picks up.

to check-game

let p1choice 0

ifelse sum cooperate-history > sum defect-history [

set p1choice 1

]

[

ifelse sum cooperate-history = sum defect-history [

set p1choice one-of [ 0 1 ]

]

[

set p1choice 0

]

]

...

This code first declares a local variable for the choice of the focal agent (hereafter referred to as Player 1), called **p1choice**, and sets it to a default of 0 (defect). Then, it runs through a series of checks to see whether the p. First, it uses the **ifelse** command to check whether the **mean** value of the **cooperate-history** is greater then that of the **defect-history**, indicating this agent has historically received greater benefit from cooperating than defecting. If this is the case, it sets **p1choice** to 1 (cooperate). If this is not the case, then the agent uses **if** to check whether the mean **cooperate-history** is equal to the mean **defect-history**. If this is true, then the agent randomly sets its **p1choice** to **one-of** two values (0 or 1); otherwise, the agent retains its original **p1choice** of 0.

Now that Player 1 has made a choice, we can determine who Player 2 is and make their choice:

`...`

let p2 one-of other turtles in-radius search-radius

let p2choice 0

ifelse [ sum cooperate-history ] of p2 > [ sum defect-history ] of p2 [

set p2choice 1

]

[

ifelse [ sum cooperate-history ] of p2 = [ sum defect-history ] of p2 [

set p2choice one-of [ 0 1 ]

]

[

set p2choice 0

]

]

...

Here, we set a local variable called **p2** which holds the identity of Player 2, and associates that variable with an agent selected randomly from within the** search-radius**. We do this because the call of **check-game **is being made by a particular agent, and so any commands made to Player 2 are made through Player 1. So while the choice algorithm is effectively the same as that used for Player 1, variables held by Player 2 are referred to using the **of** command (for example, **[ cooperation-history ] of p2**).

Now that this is done, we can compare the two decisions and establish the payoffs to each player. We’ll begin with scenarios where Player 1 chooses to cooperate.

`...`

ifelse p1choice = 1 [

ifelse p2choice = 1 [

set cooperate-history lput cooperate-cooperate cooperate-history

ask p2 [

set cooperate-history lput cooperate-cooperate cooperate-history

]

]

[

set cooperate-history lput lone-cooperator cooperate-history

ask p2 [

set defect-history lput lone-defector defect-history

]

]

]

...

The first **ifelse** here is used to establish if Player 1 has chosen to cooperate or to defect, with cooperate being **p1choice = 1**. We use a second ifelse to establish if Player 2 has chosen to cooperate or to defect. In the first case, where Player 2 has chosen to cooperate (**p2choice = 1**), then each player adds the **cooperate-cooperate** payoff to their respective **cooperate-history** lists. In the second case (following the open bracket **[** indicating the alternative condition of the second **ifelse** statement), where Player 2 has chosen to defect, Player 1 adds the **lone-cooperator** payoff to their **cooperate-history** list, while Player 2 adds the **lone-defector** payoff to their **cooperate-history**.

Next, we’ll look at the alternative condition of that first **ifelse **statement, in which Player 1 chooses to defect.

`...`

[

ifelse p2choice = 1 [

set defect-history lput lone-defector defect-history

ask p2 [

set cooperate-history lput lone-cooperator cooperate-history

]

]

[

set defect-history lput defect-defect defect-history

ask p2 [

set defect-history lput defect-defect defect-history

]

]

]

...

Under this scenario, Player 1 has chosen to defect (**p1choice** **= 0**). Again, we’ll use **ifelse** to establish if Player 2 has chosen to cooperate or to **defect. In the first case, where Player 2 has chosen to cooperate (p2choice = 1)**, then Player 1 adds the **lone-defector** payoff to their **defect-history** while Player 2 adds the **lone-cooperator** payoff to their **cooperate-history**. In the second case, where Player 2 has also chosen to defect, both players add the **defect-defect** payoff to their respective **defect-history** lists.

Now that all potential outcomes from the game have been covered, we’ll finish up the **check-game** routine by first maintaining the **cooperate-history** and **defect-history** lists and then updating the **decisions** variable.

`...`

if length cooperate-history > 10 [

set cooperate-history remove-item 0 cooperate-history

]

if length defect-history > 10 [

set defect-history remove-item 0 defect-history

]

ask p2 [

if length cooperate-history > 10 [

set cooperate-history remove-item 0 cooperate-history

]

if length defect-history > 10 [

set defect-history remove-item 0 defect-history

]

]

set decisions lput p1choice decisions

set decisions lput p2choice decisions

end

The first part of this code asks agents to check the **length** their respective **cooperate-history** and **defect-history** lists. We’ll cap these lists at 10 interactions by asking agents whose lists are longer than 10 to drop the oldest item from those lists. We can do this using the **remove-item** command. Because we were updating the lists with **lput**, which adds items to the end of a list, the oldest item in each list is item **0**, so that will be the item which gets dropped. Finally, the global **decisions** list is updated with the choices from both players.

**Exploring games in this model**

We’ll use a plot to keep track of two variables within the model: 1) the mean of the decisions variable, which should show us the balance of strategies being employed, and 2) the proportion of agents likely to cooperate on their next move. We can do this by going to the interface tab, right-clicking on some open space, and selecting **Plot**. This brings up a dialog box which we can enter in the following:

- Set the
**Y min**value to 0 and**Y max**value to 1, as both of our measures will fall somewhere between 0 and 1 (make sure the**Auto scale?**box is ticked). - Under the
**Pen update**commands for the default pen, enter this code to get the mean of the**decisions**variable :**plot mean decisions** - Click the
**Add pen**button, and for this new pen (pen-1), enter this code under the**Pen update**to track the percentage of “cooperative” agents:**plot ((count turtles with [ sum cooperate-history > sum defect-history ]) + round ( 0.5 * count turtles with [ sum cooperate-history = sum defect-history ] )) / count turtles** - The examples below will plot
**mean decisions**in black and**percent cooperating**in grey. You can change the colors of the pen to suit your needs.

First, let’s start with something simple: a game where the payoffs for cooperate-cooperate and defect-defect are exactly the same (also called the “choosing sides” game). The decision-matrix should look like this:

This game has two Nash equilibria: **cooperate-cooperate** and **defect-defect**. To start, we’ll run the model with a search-radius of 5. Doing so produces two basic types of outcomes:

In one instance, the overall strategy trends toward 1 (cooperation), while in the other the overall strategy moves toward 0 (defection). Why does this happen? At the outset, each agent has a cooperate-history and defect-history seeded with 0’s and 4’s; the relative proportion of these for each agent will determine whether they will cooperate or defect on their next turn. At first, these will fluctuate around even, but over time, these random fluctuations reach a point where the proportions of cooperators versus defectors drives the system toward fixation at one end of the spectrum or the other.

Next, we’ll try decreasing the **search-radius** to 1 :

The increased variability in the** mean decisions** is due to the fact that, because some agents may not have any other agents within the **search-radius **during a given time step, there will be fewer games being played overall, and the outcomes from those that are played can swing the **mean decisions** to a greater degree. However, this also affects the time it takes to reach fixation, as the homogenizing effects of the process don’t spread as quickly when the number of potential opponents for any given agent is small.

We can also see how the Prisoner’s Dilemma plays out in our current model. As a reminder, the decision matrix looks like this:

Based on the outcomes of the previous simulation, and what we know about the Nash equilibrium of the Prisoner’s Dilemma (**defect-defect**), the outcomes are pretty predictable:

The plot on the left uses a **search-radius** of 5, while the plot on the right uses a **search-radius** of 1. As you can see, both move toward defection fairly rapidly; however, the time until defection becomes the sole strategy can change based on the **search-radius**, with smaller radii taking longer to reach fixation.

The last game we’ll examine probably has a name in the game theory literature, but I can’t find it. The decision matrix for this game looks a little something like this:

This game, like the Prisoner’s Dilemma, only has one Nash equilibrium (in this case, **cooperate-cooperate**). However, when we run the scenario using the two **search-radius** settings we’ve used so far, we see some strange behavior:

In both scenarios, agents begin for the most part in a cooperative mood, which makes sense since this is the optimal strategy. Over time, however, they move toward the center, eventually settling into an uneasy equilibrium around 0.5, never reaching fixation. As earlier, the **search-radius** of 5 (left) is less variable in terms of its **mean decision** value than that with a **search-radius** of 1 (right). But the **search-radius** of 5 takes longer to reach the center equilibrium state than does the **search-radius** of 1. Why might this be? We might get some insight by further increasing our **search-radius** to 10.

In this setting, the mean decisions and percent cooperating both hover near 1, but don’t quite reach it, persisting in this state for 5000 times steps. What on earth is causing this to happen? Turns out it’s all because of this guy:

Remember how the memories of all of our agents are seeded with random values from the relevant strategies? This means that some agents will start out more cooperative and some will start out more prone to defection. Under the decision-matrix used for this game, very few agents will be prone toward defection (in the last case, just one), but those that are will persist in defecting because nearly everyone they encounter will be cooperating with them. They are free to be jerks with absolute impunity.

In this scenario, if these defectors are interacting with a large number of potential players (**search-radius = 10**), then the ill-effects of their bad behavior get spread around, having little effect on others. This allows a few defectors to persist within a general climate of cooperation. But if the **search-radius** is smaller, then there is a good chance that a defecting agent will interact with the same opponent multiple times in succession. This has the effect of wearing down the cooperating agents, eventually encouraging them to defect. As more and more are brought over to the dark side of defecting, the population eventually reaches a threshold at which the number of cooperating and defecting agents balances out, producing shifts around a **mean decision** of 0.5. Under such conditions, even though the optimal strategy is cooperation and agents generally start out cooperative, a few bad apples can spoil the party.

**Going further**

The way this model is built allows us to explore a multitude of non-cooperative games and the effects of interaction distances on them. But this is not the only way to build game theory ideas into an agent-based model. The NetLogo Models Library has several examples of Prisoner’s Dilemma, spatial and non-spatial, which can be used to assess different ways the game might play out.

This tutorial is meant to be a thought provoker on how game theory ideas might be incorporated into an agent-based model, rather than a thorough treatment of spatially-explicit games. Spatially-explicit and iterated games have been explored using simulation in much greater detail elsewhere. A good place to start is Robert Axelrod’s book The Evolution of Cooperation, as well as the work of Nowak and May. There are several articles in the Journal of Artificial Societies and Social Simulation dealing with this subject.

Finally, while this tutorial has focused on non-cooperative games, there are other kinds of games which can be explored. Cooperative games are aimed at the strategic formation of coalitions. Some games involve stochastic mechanisms for deciding strategies: mixed strategy games, for example, involve a probabilistic determination of strategies rather than a strict choice of the most optimal strategies.

Code for the model can be found here.

*Featured image: A non-cooperating Super Mario*