About RL Garage
RL Garage is an online community focused on enabling quick and easy item trading between players of the Psyonix/Epic Games video game Rocket League. RL Garage is home to over 4 million Rocket League players who use it to list and search for items to trade with other players.
RL Garage provides both a website and a mobile app which both provide Rocket League news, Rocket League daily item shop information, trade searching and posting, a bespoke Rocket League car designer as well as premium accounts for additional site/app features.
How I helped
I was initially engaged by RL Garage through a mutual contact to assist with recruiting a DevOps Engineer. Due to a lack of good candidates as well as an urgency to the project RL Garage requested I do an assessment on their database server and help to identify solutions to the load based outages they were encountering.
Shortly after I began this work the existing infrastructure hosted at OVH was lost in a serious datacenter fire. Working with the information about the infrastructure I had already gathered, as well as a set of partial backups from the previous weeks and a set of Ansible playbooks which configured some of the services RL Garage relied on, we were able to piece together a working solution in AWS within about 40 hours of the site and app going offline.
Following on from this event I was tasked with building a fresh solution within AWS that took advantage of spot services to automatically scale available resources to meet the exact demand. I based the new solution on Puppet Community Edition, developing Puppet classes for the various services and server types which RL Garage required to operate.
One of the secondary projects that presented the largest challenge and required a serious amount of research and experimentation was my involvement with the RL Garage Car Designer. I was responsible for building the underlying servers that spin up copies of the Rocket League as well as a Node.js microservice written by the RL Garage development team. The current solution launches a collection of spot services running standard Ubuntu 20.04, Puppet is used to install and configure the Nvidia graphics drivers, the Rocket League game, configuration and mod files, Node.js and some remote monitoring tools. While the Puppet configuration is in progress, CodePipeline is responsible for deploying the Node.js microservice. Once the microservice is running it will ping Rocket League waiting for it to come online, once online it will pull jobs from an SQS queue and forward them to Rocket League while forwarding completed design images to the main RL Garage site for further processing.
While working on the new AWS environment I was also able to assist with several other key areas of the site, such as:
- The adoption of Socket.io to provide real time information to both the website and the app
- Improving the security of the admin site and server administration by introducing a VPN
- Developing some additional malicious user identification systems
- Integrating real time monitoring and alerts of the site and underlying services
- Assisting with the development of key components to the RL Garage car designer
- Separation of site and API traffic to allow for independent resource scaling
- Centralised logging through Fluentd with live searching provided by Kibana and ElasticSearch and long term storage of logs on S3
- Migration of site assets away from EC2 resources to a lower cost object storage based solution
While I stopped working full-time for RL Garage in February 2022, I continue to provide as-needed assistance due to the small size of their team and the scale of the site they manage.
Technologies I worked with
- AWS (EC2, ALB, ASG, CodeBuild, CodeDeploy, S3)
- Ubuntu / Windows Server
- Puppet
- Cloudflare
- Nginx
- MySQL / Redis / ElasticSearch
- PHP / Node.js
- Websockets / Socket.io
- Docker / Docker Compose
- Jira Cloud / Confluence Cloud
- Pritunl
A reference from RL Garage is available by request, please contact me for further details.
