Published on: November 29, 2022

Using Computer Vision to make €6,147,455 Overnight in In-Game Currency

Table of Contents

Introduction
Creating a Python script to collect the valuables for me
1. Capture the live game feed
2. Identify the valuables in the screenshot
3. Collect the valuables by clicking on them
4. Close any popups that may appear
Full code. It works yay! 🔥🔥
Results after running overnight
Regrets
Conclusion

Introduction

I've been playing strategy + city building + business simulation? games like TownsMen 6, Clash of the Clans, SimCity, Transport Fever 2 and my favourite, OpenTTD (that's also open source!) for the last 10 years.

On trying out City Island 5 I found it mildly irritating that my collectables could not accumulate while I was outside the game. I might have had the best businesses, strategy, etc but I had to be in the game to ensure I collect the cash/keys/gold over time. For example, if my bakery makes €100 per minute, I would only earn €100 after leaving the game and coming back 24 hours later.

This became especially tiresome while trying to accumulate the €5,000,000 required to buy the island shown below. This would take me roughly two weeks of gameplay if I don't spend any money - it's not worth it.

Disclaimer - This content is for educational purposes only!

Creating a Python script to collect the valuables for me

This is a problem that can be solved using one of the greatest tools in my tool belt - programming.

1. Capture the live game feed

I needed a way to capture the live game feed.

The easiest way is to capture an in-game screenshot and pass it to the next steps in the script.

For the screenshot, I use the Python MSS library. It's a simple library that allows you to capture the screen and save it to a file. We can also use the library to select a monitor and get its properties like its width and height.

We will be using OpenCv (cv2) for the computer vision part of the script. It is a library that allows us to perform image processing and computer vision tasks. Here we use the cv2.imread() method to load an image from the specified file.

import cv2 
import mss

sct = mss.mss() 
default_monitor = sct.monitors[1]

def click_template_image(monitor=default_monitor):

    # Screenshot
    game_screenshot_path = "sct_{width}x{height}.png".format(**monitor)
    sct_img = sct.grab((0, 0, monitor["width"], monitor["height"]))
    mss.tools.to_png(sct_img.rgb, sct_img.size, output=game_screenshot_path)

    game_screenshot = cv2.imread(game_screenshot_path, 1)

2. Identify the valuables in the screenshot

We need a way to detect a valuable in the game's feed and then return its coordinates.

OpenCv's Template Matching algorithms are perfect for this.

They are used for searching and finding the location of a template image (like a valuable) in a larger image (like the game's feed). It simply slides the template image over the input image (as in 2D convolution) and compares the template and patch of the input image under the template image. Several comparison methods are implemented in OpenCV. (You can check docs for more details). We use it in the method: cv2.matchTemplate(... )

To achieve this, I needed the template images. I took screenshots by hand and then cropped off the cash, star and key:

In the code sample below we are using cash.

import cv2
import mss
import numpy as np

sct = mss.mss()
default_monitor = sct.monitors[1]

def click_template_image(monitor=default_monitor):

    # 1. Screenshot
    game_screenshot_path = "sct_{width}x{height}.png".format(**monitor)
    sct_img = sct.grab((0, 0, monitor["width"], monitor["height"]))
    mss.tools.to_png(sct_img.rgb, sct_img.size, output=game_screenshot_path)
    game_screenshot = cv2.imread(game_screenshot_path, 1)


    # 2. Find a way to identify the valuables in the screenshot
    template_image = cv2.imread("images/cash.png", 1)

    search_result = cv2.matchTemplate(game_screenshot, template_image, cv2.TM_CCOEFF_NORMED)

    y_coords, x_coords = np.where(search_result >= threshold)

    for idx in range(len(x_coords)):
        x, y = x_coords[idx], y_coords[idx]

3. Collect the valuables by clicking on them

Once we have the coordinates of an item we can try to click on it.

pyautogui's .click(x,y) function works like magic for this. It clicks the screen on the coordinates x and y where our valuable is lying. Learn more about it here.

Note:

We can decide to pick up coordinates that meet a certain confidence score/threshold. A confidence score is a number between 0 and 1 that represents the likelihood that the output of a model is correct and will satisfy a user’s request. For example, we can pick up coordinates that have a confidence level of 0.7 or higher. That's what we are using the threshold variable for. The matchTemplate() algorithm gives us several points in the map that match our query. I then decided to filter out the points that are below the threshold: y_coords, x_coords = np.where(search_result >= threshold)
After several trials, I realised that clicking multiple times on the map per algorithm run results in errors and inaccuracies. For example, before clicking on a moving car, it might have moved a bit. I decided to experiment with a number of clicks every time the click_template_image() function is called using the number_of_clicks variable and settled with one click per run.
I found out that clicking on the centre of an image works better than clicking on the top left, that is the coordinates given to us by our template matching function. We can use the template image's height and width to calculate the centre coordinates: x_c = int((x + x + w) // 2) & y_c = int((y + y + h) // 2)

import cv2
import mss
import numpy as np
import pyautogui

pyautogui.FAILSAFE = False

sct = mss.mss()
default_monitor = sct.monitors[1]

def click_template_image(monitor=default_monitor, number_of_clicks=1, threshold=0.7):

    # 1. Screenshot
    game_screenshot_path = "sct_{width}x{height}.png".format(**monitor)
    sct_img = sct.grab((0, 0, monitor["width"], monitor["height"]))
    mss.tools.to_png(sct_img.rgb, sct_img.size, output=game_screenshot_path)
    game_screenshot = cv2.imread(game_screenshot_path, 1)

    # 2. Find a way to identify the valuables in the screenshot
    template_image = cv2.imread("images/cash.png", 1)
    search_result = cv2.matchTemplate(game_screenshot, template_image, cv2.TM_CCOEFF_NORMED)
    y_coords, x_coords = np.where(search_result >= threshold)

    # get the width and height of the template image
    w, h = template_image.shape[1], template_image.shape[0]
    for idx in range(number_of_clicks):
        if idx + 1 > len(x_coords):
            continue
        x, y = x_coords[idx], y_coords[idx]

        # 3. Collect the valuables by clicking on them
        # get centres
        x_c = int((x + x + w) // 2)
        y_c = int((y + y + h) // 2)

        pyautogui.click(x=x_c, y=y_c)

Collecting cash

Collecting a star

Collecting a key

4. Close any popups that may appear

Our clicks above may result in popups when we are being given a reward, levelling up, etc.

We need to close it before attempting to collect valuables again. We use the same logic used in finding and clicking on valuables.

To achieve this, I needed the template images for the popups' close buttons so that they can be clicked. I took screenshots by hand and then cropped off the various close buttons:

It uses the same code as the one required for clicking on valuables.

Closing a popup

Full code. It works yay! 🔥🔥

We do the steps above repeatedly to collect the valuables while the script is running.

game.py


import cv2  # https://docs.opencv.org/4.x/
import numpy as np
import pyautogui
import mss  # https://python-mss.readthedocs.io/index.html
from time import sleep

pyautogui.FAILSAFE = False

sct = mss.mss()
default_monitor = sct.monitors[1]  # https://python-mss.readthedocs.io/api.html#mss.tools.mss.base.MSSBase.monitors


def click_template_image(
    template_image_path: str,
    monitor=default_monitor,
    threshold: float = 0.7,
    number_of_clicks: int = 1,
):
    print(f"{template_image_path} search")
    template_image = cv2.imread(template_image_path, 1)

    # Screenshot
    # game_screenshot_path = "sct_{width}x{height}.png".format(**monitor)
    # sct_img = sct.grab((0, 0, monitor["width"], monitor["height"]))
    # mss.tools.to_png(sct_img.rgb, sct_img.size, output=game_screenshot_path)  # type:ignore
    # game_screenshot = cv2.imread(game_screenshot_path, 1)
    game_screenshot = np.array(sct.grab((0, 0, monitor["width"], monitor["height"])))
    game_screenshot = game_screenshot[:, :, :3]  # remove alpha

    # https://docs.opencv.org/master/d4/dc6/tutorial_py_template_matching.html
    search_result = cv2.matchTemplate(
        game_screenshot, template_image, cv2.TM_CCOEFF_NORMED
    )

    y_coords, x_coords = np.where(search_result >= threshold)  # type:ignore
    # the screenshot might have a different resolution/dimensions form the actual screen. 
    # the width & height reset multipliers are used to reset the w & h (screenshot's dimensions) to the actual screen's dimensions
    width_reset_multiplier = game_screenshot.shape[1] / monitor["width"]
    height_reset_multiplier = game_screenshot.shape[0] / monitor["height"]
    w = template_image.shape[1]
    h = template_image.shape[0]
    for idx in range(number_of_clicks):
        if idx + 1 > len(x_coords):
            continue

        x, y = x_coords[idx], y_coords[idx]

        x /= width_reset_multiplier
        y /= height_reset_multiplier

        x_c = int((x + x + w) // 2)
        y_c = int((y + y + h) // 2)

        pyautogui.click(x=x_c, y=y_c)  # type:ignore

        sleep(0.3) # wait for popups to appear

close_buttons = [
    "close.png",
    "close_big.png",
    "continue_level.png",
    "yes_close_offer.png",
]

valuables = [
    "cash.png",
    "star.png",
    "key.png",
]

while True:

    for valuable_image in valuables:
        click_template_image("images/" + valuable_image)

        for close_button_image in close_buttons:
            click_template_image("images/" + close_button_image)

Access the full source code, the images and the videos used in this article here: https://github.com/paulonteri/play-game-with-computer-vision

Results after running overnight

I started the game with €316,415.

The following morning I had €6,463,870.

I made €6,147,455 overnight!

I then proceeded to buy the Island I wanted:

Regrets

This is cheating.
Why get a game if you're not the one playing it?

Conclusion

Access the full source code, the images and the videos used in this article here: https://github.com/paulonteri/play-game-with-computer-vision

This was fun!