That's Right, I Game 😎

JS disabled, defaulting to: Mind of a Thief | Mother 3

Making an AI to play Mario Golf

Mario Golf is the first of six current entries in the Mario Golf series (no, NES Open Tournament Golf doesn't count). It was developed by Camelot software and published in 1999, and received positive reviews as both a Mario game and a golf simulator. I own the japanese release of the game, both because it's cheaper and because the box art rivals "All your base are belong to us".

mariogolf.jpg

For all players hoping to touch the true entertainment

I've been wanting a reason to gain more exposure to machine learning and neural networks, and teaching a computer to golf seemed like a great opportunity. Golf has a small number of clearly defined inputs and outputs, which in theory makes training a machine learning model simple and effective. In practice, getting good data was less straightforward than I imagined, but I ended up with an AI that I'm alright with. I'll be going through the details of both creating an AI and linking it to an emulator in this post, but if you're only interested in the results you can skip to the end.

I'll be emulating the game using Project64 and mostly writing the code in Perl because I need to learn the language. As a result, I'd highly discourage studying my code too much.

Inputs & Outputs

The first step of the project was figuring out how to read data from an emulator to process, and then control the game based on the results. Fortunately, Perl makes reading data from a seperate process in Windows very easy through the Win32 module. Finding which data to read is also a well-documented process. Project64, like many emulators, has a memory searching tool which allows you to find the ram addresses of values. Finding the values you need is then a matter of guessing how the game's variables are stored in memory, and narrowing your search based on how you believe those variables have changed. Here's a table of the memory addresses I found (some may be a bit off because of weird <4 byte differences between addresses according to the emulator and Win32 that I can't be bothered to recheck):

Table of addresses

Address	Data type	Description
0x802225F4	Float	X, Y, and Z positions of the ball
0x802225F8
0x802225FC
0x800DAC58	int16	X, Y, and Z positions of the hole
0x800DAC5A
0x800DAC5C
0x800DEAC0	int16	Player's position (don't know why it's an int)
0x800DEAC2
0x800DEAC4
0x801B553C	Float	Final X, Y, and Z positions of the camera(?), which usually reflect where a shot will land
0x801B5540
0x801B5544
0x80104F44	Float	Wind strengths in the X and Z directions
0x80104F4C	Float	Wind strengths in the X and Z directions
0x800FBE73	int8	Surface type the ball is on.
0x800C310C	?	Some sort of timer that increases once per frame
0x800B7736	?	Another timer, because the first one doesn't increment when the game is paused
0x800FBE5B	int8	Club (1W is 0, putter is 13)
0x800FBE57	int8	Shot type (power, approach, etc)
0x801F4438	float	Facing angle (0-2pi)
0x80250B60	int8	Surface type the ball will land on
0x800BA9F8	int8	Is it raining?

And here's the Perl code to read them:

input.pm


use strict; use warnings; use Switch;
use Win32;
use Win32::Process;
use Win32::Process::Memory;

our %addresses = (
    0x202225F4 => {
        type => "float",
        desc => "X position of ball",
        value => 0
    },
    # ...
    0x20223E5C => {
        type => "uint8",
        desc => "Game state (0 when can hit, 1 when can't)",
        value => 10
    }
);

my $mem = Win32::Process::Memory->new({
    pid => Process ID of the emulator,
    access => 'read/query'
});

sub updateValues {
    foreach my $key (keys %addresses){
        switch ($addresses{$key}{type}){
            case "float" {$addresses{$key}{value} = $mem->get_float($key, 255)}
            case "uint8" {$addresses{$key}{value} = $mem->get_u8($key, 255)}
            case "int16" {$addresses{$key}{value} = $mem->get_i16($key, 255)}
            case "int32" {$addresses{$key}{value} = $mem->get_i32($key, 255)}
            else {$addresses{$key}{value} = 255}
        }
    }
}

sub updateValue {
    my $key = $_[0];
    switch ($addresses{$key}{type}){
        case "float" {$addresses{$key}{value} = $mem->get_float($key, 255)}
        case "uint8" {$addresses{$key}{value} = $mem->get_u8($key, 255)}
        case "int16" {$addresses{$key}{value} = $mem->get_i16($key, 255)}
        case "int32" {$addresses{$key}{value} = $mem->get_i32($key, 255)}
        else {$addresses{$key}{value} = 255}
    }
}

sub printValues {
    foreach my $key (keys %addresses){
        printf "$key\n    $addresses{$key}{desc}\n    $addresses{$key}{value}\n";
    }
}

There were two values I was hoping to find that I couldn't. The first is some sort of register that controls controller input. While I found a lot of addresses that mirrored controller input, writing values to them didn't seem to affect the game. It's possible that I have to write to the address at the start of the frame for it to have an effect, but regardless it doesn't seem like there's a straightforward way to control controller input through memory.

The next value is lie, or the slope of the ground the ball is on. This affects the trajectory of the shot and needs to be adjusted for. But while I couldn't find the lie, I did find the position of the player, who circles around the ball as you angle your shot. Using the positions of the player and the ball, I can get a good enough value for the slope.

output.pm


sub findLieVectors {
    updateValues();
    my $facingAngle = $addresses{0x201F4438}{value};
    my @xVect = (cos($facingAngle - 3.1415 / 2), 0, sin($facingAngle - 3.1415 / 2));
    my @yVect = (cos($facingAngle), 0, sin($facingAngle));
    my @zVect = (0, 1, 0);
    my @playerPos = ($addresses{0x200DEAC2}{value}, $addresses{0x200DEAC0}{value}, $addresses{0x200DEAC6}{value});
    my @ballPos = ($addresses{0x202225F4}{value}, $addresses{0x202225F8}{value}, $addresses{0x202225FC}{value});
    my @v1 = ($playerPos[0] - $ballPos[0], $playerPos[1] - $ballPos[1], $playerPos[2] - $ballPos[2]);
    my $mag = sqrt($v1[0] ** 2 + $v1[1] ** 2 + $v1[2] ** 2);
    $v1[0] /= $mag;
    $v1[1] /= $mag;
    $v1[2] /= $mag;
    rotate(68, 0x200C310C);
    updateValues();
    @playerPos = ($addresses{0x200DEAC2}{value}, $addresses{0x200DEAC0}{value}, $addresses{0x200DEAC6}{value});
    @ballPos = ($addresses{0x202225F4}{value}, $addresses{0x202225F8}{value}, $addresses{0x202225FC}{value});
    my @v2 = ($playerPos[0] - $ballPos[0], $playerPos[1] - $ballPos[1], $playerPos[2] - $ballPos[2]);
    my $mag2 = sqrt($v2[0] ** 2 + $v2[1] ** 2 + $v2[2] ** 2);
    $v2[0] /= $mag2;
    $v2[1] /= $mag2;
    $v2[2] /= $mag2;
    rotate(-68, 0x200C310C);
    my @v3 = ($v1[2] * $v2[1] - $v1[1] * $v2[2], $v1[0] * $v2[2] - $v1[2] * $v2[0], $v1[1] * $v2[0] - $v1[0] * $v2[1]);
    if($v3[1] < 0){
        $v3[0] *= -1;
        $v3[1] *= -1;
        $v3[2] *= -1;
    }
    return ($v1[0], $v1[1], $v1[2], $v3[0], $v3[1], $v3[2], $v2[0], $v2[1], $v2[2]);
}

Output is pretty simple. Because I can't modify memory to simulate key presses, I used a Perl module called Win32::GuiTest, which has a subroutine called SendRawKey to simulate pressing or releasing a certain key. I can't simulate joystick inputs using this method which limits my control of the contact point of each stroke, but apart from that it works well. The code is messy and repetitive so I won't include it here, but if you really want to see it it's in my Github profile.

Gathering Training Data

My initial method of gathering training data was to go to various points in different holes and determine the shot which landed closest to the flag. I made a vaguely binary search-y algorithm to find an optimal rotation, power level, and contact point for each stroke and collected data from 64 different save states. This failed miserably. After days of troubleshooting, I had figured out a few flaws in my method.

While gathering data in actual courses is realistic, there's way too much variation in conditions I don't record to determine meaningful trends, like the random decrease in power based on the surface I hit from or the slope the ball landed on.
The method I was using to find the optimal shot required about 100 strokes. This probably could have been optimized, but I would still be taking tens of strokes per data point.
Because I couldn't simulate joystick input, there were only nine contact points available (-1, 0, or 1 in both dimensions). I think this was too limited to effectively predict.

I scrapped the contact point, but the other two problems were more difficult to address. To solve the second problem, I could just take a random shot and then pretend that the hole was wherever it landed. This would still be limited by the issue of random course conditions, though. If only there were a way to take shots from a tee on a completely flat course.

drivingRange.png

Using the driving range does come with its own problems. There won't be any elevation change between the tee and the ground and the surface will always be flat, meaning that the only parameters about the course that I can change are the distance to the "hole" and the wind. The only other parameters I'd really need to consider are the lie and the elevation change to the hole, though, and at this point I was happy to adjust for those outside of the neural network. I ended up running a script to have Wario whack the ball around like a maniac and gathered over 5000 data points overnight.

testDrives.pl


use input;
use output;
use Math::Round qw( round );
use 5.010;

sub testDrive {
    state $currentState = "";
    my $power = round(5 + rand(27));
    my $rotation = round(rand(40) - 20);
    my $club = int(rand(13));
    my $shotType = 1;
    my $contactX = 0;
    my $contactY = 0;
    my $windDir = round(rand(150));
    my $windStrength = round(rand(21));
    $rotation = round(cos($windDir * 3.1415 / 75 + 3.1415 / 2) * $windStrength * 5/21);
    if($club >= 5){
        $rotation = 0;
    }
    print("$power, $rotation, $club, $shotType, $contactX, $contactY, $windDir, $windStrength\n");
    # Adjust for a divot in the range
    if($powerLevels[$club][$shotType] * $power / 31 > 40){
        if($currentState ne "d"){
            loadState("d");
            $currentState = "d";
            sleep(1);
            writeKey2(0x67);
            sleep(1);
        } else {
            writeKey2(0x65);
            sleep(1);
        }
    } else {
        if($currentState ne "e"){
            loadState("e");
            $currentState = "e";
            sleep(1);
            writeKey2(0x67);
            sleep(1);
        } else {
            writeKey2(0x65);
            sleep(1);
        }
        $rotation = 0;
        # $windDir -= 38;
    }
    sleep(2);
    updateValues();
    my @initialPos = ($addresses{0x202225F4}{value}, $addresses{0x202225F8}{value}, $addresses{0x202225FC}{value});
    my $initialAngle = $addresses{0x201F4438}{value};

    takeDrivingRangeStroke($rotation, $power, $contactX, $contactY, $windDir, $windStrength, $club);
    waitTicks(50);

    updateValues();
    my @finalPos = ($addresses{0x201B553C}{value} + 15, $addresses{0x201B5540}{value} + 30, $addresses{0x201B5544}{value});
    my @deltaPos = ($finalPos[0] - $initialPos[0], $finalPos[1] - $initialPos[1], $finalPos[2] - $initialPos[2]);
    if($finalPos[1] > -299){
        print("Landed off course");
        print($addresses{0x201B5540}{value});
        print("\n");
        return;
    } else {
        my $angle = atan2(-$deltaPos[2], $deltaPos[0]);
        # print("Initial Angle: $initialAngle\n");
        # print("Angle: $angle\n");
        # print deltaPos
        # print("$deltaPos[0], $deltaPos[1], $deltaPos[2]\n");
        my $deltaAngle = $initialAngle - $angle;
        if($deltaAngle < -3.1415){
            $deltaAngle += 2 * 3.1415;
        } elsif ($deltaAngle > 3.1415){
            $deltaAngle -= 2 * 3.1415;
        }
        $deltaAngle += $rotation * 3.1415 * 2 / 271;
        my $unitsPerYard = 24 / 1.75;
        my $dist = sqrt($deltaPos[0] ** 2 + $deltaPos[2] ** 2);
        my $clubPower = $powerLevels[$club][$shotType];
        my $distanceRatio = $dist / ($clubPower * $unitsPerYard);
        my @xVect = (cos($angle - 3.1415 / 2), -sin($angle - 3.1415 / 2));
        my @yVect = (cos($angle), -sin($angle));
        my $windAngle = $windDir * 3.1415 / 75 - 3.1415 / 2;
        my @windVect = (-cos($windAngle) * $windStrength, -sin($windAngle) * $windStrength);
        my $revisedWindX = $xVect[0] * $windVect[0] + $xVect[1] * $windVect[1];
        my $revisedWindY = $yVect[0] * $windVect[0] + $yVect[1] * $windVect[1];
        
        my $inData = "$distanceRatio, $club, $revisedWindX, $revisedWindY";
        my $outData = "$deltaAngle, $power";
        my $rawData = "$power, $rotation, $club, $shotType, $contactX, $contactY, $windDir, $windStrength, $initialPos[0], $initialPos[1], $initialPos[2], $initialAngle, $finalPos[0], $finalPos[1], $finalPos[2]";
        
        open my $fh, '>>', './data/drives.csv';
        print $fh ($inData.", ".$outData."\n");
        close $fh;

        open my $fh, '>>', './data/rawdrives.csv';
        print $fh ($rawData."\n");
        close $fh;
    }
}

sleep(3);
while(1){
    testDrive();
}

The Basics of a Neural Network

There are a lot of methods that fall under the "AI" umbrella, but I'll be using a feed-forward neural network. If you're unfamiliar with neural networks, I'll briefly describe them here. If you'd like to learn more about them, there are plenty of other resources more qualified to explain them than I am.

Many of the problems you run into in computing are straightforward. You have inputs and a function, and need to produce outputs.

Input	Function	Output
n = 2	f(x) = 3x + 1	f(n) = ?

But sometimes, you'll have inputs and outputs, but no function. Let's say you're doing a study on toothpaste brands, and you know the concentrations of flouride and sorbitol in each brand, and the number of dentists that recommend it.

Flouride (ppm)	Sorbitol (%)	Dentists (/10)
1130	70	2
1185	30	3
1250	20	5
1350	30	9
1550	10	7
1590	40	8
1675	50	6
1715	50	4

You want to make a toothpaste that 10/10 dentists recommend, so you need to figure out the relationship between ingredients and reception. The tricky part is that this relation could take any form. It could be linear, quadratic, cubic, exponential, logarithmic, or some combination of types. How can you teach a program to learn any arbitrary relationship? The answer is neural networks.

neuralnet.webp

An image of a simple neural network with two inputs, four hidden nodes, and one output.

Image by Sabrina Jiang © Investopedia 2020

In a paragraph, neural networks work like this:
A neural network is made of layers of nodes, with the first layer providing the inputs and the last layer producing the outputs. To get the value of each node, you perform algebra on the nodes in the previous layer and then pass the result through an activation function, like tanh or ReLU. To train the network, you can compare the values of the output nodes to their expected values and adjust the connections in the network accordingly.

The actual implementation of a neural network is more complicated, but that's the gist. Applying a neural network to our toothpaste problem (2 inputs -> 2 hidden nodes (sigmoid) -> 1 output node (sigmoid)), we get the following results:

network.py


import torch
import torch.nn as nn
import matplotlib.pyplot as plt

data = [
[[0, 0.7],[0.2]],
[[0.096, 0.3],[0.3]],
[[0.21, 0.2],[0.5]],
[[0.38, 0.3],[0.9]],
[[0.73, 0.1],[0.7]],
[[0.8, 0.4],[0.8]],
[[0.95, 0.5],[0.6]],
[[1, 0.5],[0.4]]
]
dataX = torch.FloatTensor([i[0] for i in data])
dataY = torch.FloatTensor([i[1] for i in data])

model = nn.Sequential(nn.Linear(2, 2),nn.Sigmoid(),nn.Linear(2, 1),nn.Sigmoid())

loss_function = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
losses = []

for epoch in range(150000):
    pred_y = model(dataX)
    loss = loss_function(pred_y, dataY)
    losses.append(loss.item())
    model.zero_grad()
    loss.backward()
    optimizer.step()

plt.plot(losses)
plt.ylabel('loss')
plt.xlabel('epoch')
plt.show()

torch.save(model, './toothpaste.pt')

Figure_1.png

We can see that dentists prefer toothpaste with an amount of flouride around 1440 ppm and high amounts of sorbitol. Rather than bothering them with a flood of toothpastes to rate, you can use past data to approximate what they like.

Teaching the AI

And now, it's time for the moment you've all been waiting for...

funnygarlicman.png

To train the AI, I used a pretty simple feed-forward neural network. It has:

7 inputs

Distance to hole / club's max distance
Distance to hole / 1W max distance (yes, that's probably slightly redundant)
Wind's x direction relative to the player
Wind's z direction relative to the player
Lateral component of club's launch vector (high for driver, low for sand wedge)
Vertical component of club's launch vector (low for driver, high for sand wedge)
Club's draw/fade (how much the ball hooks after being hit)

One hidden layer with 6 nodes using ReLU as an activation function

2 outputs again using ReLU

Rotation (mapped from [-7, 7] to [0, 1])
Power (mapped from [0, 31] to [0, 1])

After training the network for around 30000 epochs, I reached a mean squared error of just over 0.0025. This means that on average, the power level was off by 1.6 and the rotation was off by 0.7, although one or the other could have dominated the error.

This problem was pretty simple, so I only needed one hidden layer. I probably could have gotten away with using fewer hidden nodes as well, but this is what I settled on. I mainly tested the network with sigmoid and ReLU as activation functions, and ReLU performed faster. I'd guess this means that relationship between the inputs and outputs is pretty linear, although that could be entirely wrong. Regardless, the network performs at least well enough to take logical shots.

To account for elevation changes, I just added the change in elevation to the distance so that the AI would hit softer shots if the hole is below it and longer shots if it's above it. To account for the lie, I figured out how much the ground sloped in the x direction relative to the player and rotated a little to compensate. I tried adjusting the parameters dealing with the launch vector based on the slope, but that seemed to perform poorly for whatever reason. This network was trained on all clubs except for the putter. The putter is different enough that it needs its own logic, and rather than figure out how to collect meaningful data to train another network I just coded it myself.

I trained the neural network in Python because I couldn't get any good modules working in Perl, so I have my Perl script send a request to a Python script with the inputs to the network. To avoid slow startup times from both Python and Perl, both scripts run continuously and requests and responses are sent by updating txt files. The Python code is a mess and the neural network part is covered earlier in this post, so I'll just share the Perl script:

stroke.pm


package stroke;

use lib './';

use Win32::GuiTest qw(FindWindowLike GetWindowText SetForegroundWindow SendKeys SendRawKey KEYEVENTF_KEYUP KEYEVENTF_EXTENDEDKEY);
use input;
use output;
use Time::HiRes qw( usleep );

our @ISA = qw( Exporter );

our @EXPORT = qw( stroke );

sub sendInputs {
    my $line = $_[0] || "";
    my @inputs = split(", ", $line);
    updateValues();
    my $club = $addresses{0x200FBE58}{value};
    if($club <= 12){
        my @keys = (0x202225F4, 0x202225F8, 0x202225FC, 0x20104F44, 0x20104F4C, 0x200FBE70, 0x200DAC5A, 0x200DAC5E, 0x200DAC58, 0x200FBE58, 0x200FBE54, 0x201F4438, 0x20106243, 0x200BA9F8);
        my $inputStr = "0, 0, ";
        for (my $key = 0; $key < scalar @keys; $key ++){
            $inputStr = $inputStr.$addresses{$keys[$key]}{value}.", ";
        }
        # my @lie = findLie();
        my $numInputs = scalar @inputs;
        print("Num inputs: ".$numInputs."\n");
        if($numInputs == 0){
            SendRawKey(0x63, 0);
            usleep(50000);
            SendRawKey(0x63, KEYEVENTF_KEYUP);
            my @lie = findLieVectors();
            SendRawKey(0x63, 0);
            usleep(50000);
            SendRawKey(0x63, KEYEVENTF_KEYUP);
            $inputStr = $inputStr."$lie[0], $lie[1], $lie[2], $lie[3], $lie[4], $lie[5], $lie[6], $lie[7], $lie[8]";
        } else {
            for(my $i = 9; $i > 0; $i--){
                $inputStr = $inputStr.$inputs[$numInputs - $i].", ";
            }
        }
        $inputStr = substr $inputStr, 0, -2;
        print("Normal stroke\n");
        open my $fh, '>>', './input/stroke.txt';
        print $fh ($inputStr);
        close $fh;
    } else {
        # putting
        # Network doesn't adjust for rain
        updateValue(0x200BA9F8);
        my $raining = $addresses{0x200BA9F8}{value};
        my $rainingMultiplier = 1 + $raining * 0.3;
    
        # find power based on distance
        my $dist = getDistance();
        updateValue(0x200FBE54);
        my $shotType = $addresses{0x200FBE54}{value};
        my $unitsPerYard = 24 / 1.75;
        updateValue(0x200DAC58);
        updateValue(0x202225F8);
        my $elevationChange = $addresses{0x200DAC58}{value} - $addresses{0x202225F8}{value};
        if($elevationChange > 0){
            $dist -= $elevationChange * 1.5;
        } else {
            $dist -= $elevationChange * 4;
        }
        my $power = int($dist / ($powerLevels[13][$shotType] * $unitsPerYard) * 31 * $rainingMultiplier + 3);
        if($power > 31){
            SendRawKey(0x43, 0);
            usleep(50000);
            waitTicks(2);
            SendRawKey(0x43, KEYEVENTF_KEYUP);
            usleep(50000);
            updateValue(0x200FBE54);
            $shotType = $addresses{0x200FBE54}{value};
            $power = int($dist / ($powerLevels[13][$shotType % 3] * $unitsPerYard) * 31 * $rainingMultiplier + 3);
        }

        # find rotation based on slope of ground
        updateValue(0x201F4438);
        my $facingAngle = $addresses{0x201F4438}{value};
        SendRawKey(0x63, 0);
        usleep(50000);
        SendRawKey(0x63, KEYEVENTF_KEYUP);
        my @lie = findLieVectors();
        SendRawKey(0x63, 0);
        usleep(50000);
        SendRawKey(0x63, KEYEVENTF_KEYUP);
        my @lieY = ($lie[3], $lie[4], $lie[5]);
        my @xVect = (cos($facingAngle -3.1415 / 2), 0, -sin(facingAngle - 3.1415 / 2));
        my @yVect = (cos($facingAngle), 0, -sin($facingAngle));
        
        my $rotation = $xVect[0] * $lieY[0] + $xVect[2] * $lieY[2];
        $rotation *= $dist / $unitsPerYard * 3 * 0.2;
        $rotation = int($rotation);
        print("Rotation: ".$rotation.", Power: ".$power."\n");

        takeStroke($rotation, $power, 0, 0);

        open my $fh, '>>', './input/response.txt';
        print $fh ("ok");
        close $fh;
    }
}

sub stroke {
    # Network doesn't adjust for rain
    updateValue(0x200BA9F8);
    my $raining = $addresses{0x200BA9F8}{value};
    my $rainingMultiplier = 1 + $raining * 0.05;

    my $numClubChanges = 0;
    truncate './input/response.txt', 0;
    # Either putt or send data to Python script with network
    sendInputs();

    # While not done
    while (1) {
        my $firstLine = "";
        my $secondLine = "";
        # Wait for a response
        while($firstLine eq ""){
            usleep(200000);
            open my $fh, '<', './input/response.txt';
            $firstLine = <$fh>;
            $secondLine = <$fh>;
            close $fh;
        }
        # Truncate file for the next message
        truncate './input/response.txt', 0;
        if($firstLine ne "ok"){
            updateValue(0x200FBE58);
            my @args = split(",", $firstLine);
            # Github Copilot just predicted that I'd want to set the maximum number of club changes to 2. That's crazy.
            # If returned power is above power gauge max, change club
            if(int(int($args[1]) * $rainingMultiplier) > 31 && $addresses{0x200FBE58}{value} >= 2 && $numClubChanges < 2){
                SendRawKey(0x68, 0);
                waitTicks(2);
                SendRawKey(0x68, KEYEVENTF_KEYUP);
                sendInputs($secondLine);
                $numClubChanges++;
            # else, just take the stroke
            } else {
                takeStroke(int($args[0]), int(int($args[1]) * $rainingMultiplier), int($args[2]), int($args[3]));
                last;
            }
        # And if the status is "ok" just end
        } else {
            last;
        }
    }
}

1;

Is a neural network overkill for such a simple problem? Probably. But would it be fun to slog through developing an accurate model otherwise? No. And if I graph the error of the AI over the course of creating it, it probably looks something like this:

error.png

Maybe the real neural network was the knowledge we gained along the way.