Dev Journal: Embedding-Based Steam Search

Published Project URL: https://storage.danieltperry.me/share/steamvibes/build
Project Details Sheet: Steam Embedding Search

Foreword

I recently had the idea of using machine learning text embedding models to generate embeddings for Steam game descriptions and their reviews. Steam game discovery has been a notoriously difficult problem to solve, and I think embeddings could be a really useful tool for helping games become more discoverable.

I’ve been working on this project for a little while now, and I’ve gotten to the point where I have a web UI used for querying a database of embeddings. I think that as a first attempt using an off-the-shelf model, it shows some promise for the future. However, search results so far have not been spectacular, but I’m hoping that performance might improve with some slight changes to my implementation, or better future models.

Since I’m going to be waiting a while for the whole database to be populated (it’s going to take a few days due to the Steam store’s API rate limiting), I thought I would write up a series of blog posts all about how each piece was implemented, and the challenges I faced along the way.

Important: I’m by no means an expert in any of this. I’m just a guy who made a thing and is sharing how he made that thing. I’m sure there are lots of mistakes and optimizations that I could have made all over the place. Feel free to e-mail me about them!
Under no circumstance should you be under the impression that what I’ve written here is the ‘best’ way of implementing all or any part of this project. I’m simply documenting how I made the thing.

I believe that embeddings are a really cool tool that are going to become more and more commonplace as AI models become more advanced, so I hope that this can be useful to anyone who wants to do something similar, or just wants to learn more about how embeddings work.

To check out the final product, head over to the search page to test it out for yourself. Maybe by the time you’re reading this, I’ll have managed to get the fully populated database deployed.

You can also see the complete source code for every part of this project on GitHub. Each individual post also contains a link to its specific source code.

Check out the individual parts below for all the details of how it was made.

To send me a comment about this post, send an e-mail to: steam_search@danieltperry.me

Posts

Steam Game Search Overview

A brief implementation overview of the entire project and how it was put together, including definitions and background information needed to understand how it works.

Part 1: Obtaining Data

Covering how to query the Steam API to get game data, and how to store it in a database from Python. It also covers some issues that arose along the way.

Part 2: Creating the Embeddings

This post covers how I used Instructor to create embeddings for my database. Details involved include how I implemented chunking, and how I stored embeddings in sqlite3.

Part 3: Querying the Embeddings

Covering the math behind embedding comparisons, how to create an index for the embeddings, and providing some examples of the results from my testing database.

Part 4: Create an API

The first step in sharing the project with others. In this post, we’ll create a simple API that can be used to query the embeddings we’ve created.

Part 5: Create a Webapp

How to write a React app when you don’t know React, using Copilot to write most of the code for you.

Foreword#

Posts#

Foreword

Posts