If you’ve been following my channel, you’ve noticed that every year I help MAGFest, a
gamer convention, with their swadges (a sort of hand-held video game system) and I also
enjoy VRChat. So, this year, I asked the bizarre question: Can I run VRChat, a VR game that
takes a $3,000+ investment in hardware to really enjoy, on a hand-held game system powered
by ~$1 processor, the ESP32S2?
I’ve gotta level with you. I probably wouldn’t be making this video right now if the answer
wasn’t at least a little bit of a “yes.”
Swadges have been a mainstay of my involvement with MAGFest since they were powered by an
ESP8266, in 2017. Every year we make 2,000 to 3,000 of these hand-held game systems and
every year they have a different IO complement and games written specifically for that year’s
MAGFest by people in the MAGFest community. Previous years we would in some cases only
have LEDs, sometimes OLED displays, various button configurations and processors but this
year, we moved to an ESP32-S2 microcontroller, a processor designed to operate little more
than a lightbulb. The most important addition this year, however, was a beautiful 280x240
pixel RGB display!
Starting last year, I helped run the VRChat Portal at MAGFest. It’s where there’s
a camera in real life that streams audio and video data from the real world into a screen
on the wall in the virtual harbor where players can join and see everyone at MAGFest walking
by. And, through the power of a screen and some speakers, people at MAGFest can see them.
It has been awesome to see people’s reactions and a witness the connections people could
make across the digital boundary. Friends who couldn’t make it to MAGFest could see
each other, chat or even take selfies together! Memes that started in VRChat but got lost
in reality looking for da queen and asking people native to their meme if they knew de
wei. It only was logical I would want to find a way of merging these two interests!
For the swadges I’ve been maintaining “flight sim”, a really basic game mode where you
fly through a { similar but legally distinct } 3D model of the Gaylord National Resort,
where MAGFest is held every year. GPLord originally designed this 3D model for our MAGFest VRChat
world 2 years ago and once done he then made a really lo-fi version of it with only 3,728
vertices, 5,830 edges and only 560 triangles so that it could be rendered on the swadge!
Normally in the flight sim the mode, you just fly around going through 14 rings and collecting
69 baked beans humming along on an ESP8266 on its 128x96 pixel OLED display.
This year, we had a small boost to the CPU power, to an ESP32-S2 and we got a BEAUTIFUL
high(er) res color LCDs. We wanted to give the flight game a huge boost to really make
it a more fun so people would play it. So this year we added a flashing color feature!
And fixed a long-running bug in the flight code. Surely that’s enough, right?
But, there were 3 months left before MAGFest and with the final touches on the version
of the game for the 1.0 firmware in… With 3 months to sit on my hands before I’d see
everyone at MAGFest, I had to do something.
I couldn’t shake it. Maybe this dream. No, vision. Of VRChat running on a swadge was
a direction instead of just a hairbrained fever dream from when I had COVID.
First of all, I would need some way to render more than just a wireframe of the atrium on
the Swadge. I’d need to be able to draw people, guns, and boolets.
All of the rendering for the flight game on the swadge is done with fixed-point math instead
of floating point. The positions for all the objects are held in units of 1/48 of a meter.
That way there are no decimals. Then, all of the objects are sorted, from farthest to
closest. That way, we can blast through all the objects, plastering them on top of each
other, just like an artist draws a picture. This technique is called the painter’s algorithm.
It wasn’t hard to just add some more object types. Maybe a basic set of interconnected
lines that could be a unique color. I don’t know what my player’s skeletons would be,
so I just made up a brand new model format and YOLO’ed it out to the screen.
I also wanted to render the other ships flying around on the swadges so players could see
each other. For this I needed a new model. I messaged GPLord, our artist and he was like
gotchu fam. And 15 minutes later I had these super stripped down M-wings.
I’d need some basic info, like objects’ positions, velocities, maybe some relationship
between players and ”boolets” since everyone loves flying around and blasting each other
with lasers.
All of the data needed to render all of this would need to be small since there was only
around 80kB of fast memory left and more importantly, everything we would need to send and receive
would have to fit into very small ESP-NOW wifi packets.
—----------------
Adding special rendering code to render all of these new objects actually was pretty straightforward.
The rendering basically uses regular 3D matrices and vectors which are all fixed point. It’s
actually the same code I copied from when I broadcasted analog television with an ESP8266.
For the networking, ESP-NOW made it so the swadges could talk to each other directly
and at the same time, a central swadge that would be plugged into the computer running
the VRChat portal. ESP-NOW is a wireless protocol that lets Espressif chips send small (<256
byte) messages to one another using regular WiFi packets, but needs no IPs or even a connection
to a network. Any swadge running the free-play version of the flight sim within the range
of another would let the users see and shoot each other. It even supported up to 80 or
so swadges all in the same space. I had to do some tricks, like using virtual clocks,
allowing each swadge have its own timebase and generating events, like blowing up or
shooting boolets be synchronized in spite of dropped packets. Using virtual clocks lets
you determine when an event happened on a separate swadge in a local swadge’s timebase
and without the need for pings. The timestamps were just 32-bit numbers in microseconds.
One fun thing about 32-bit timestamps is when you store them in 32 bit numbers, wrap around
and overflow errors aren’t a problem if you just put in a little bit of care and mind
the sign.
There’s one thing that remains a big concern and that’s RF interference. From experiments
in previous years, I knew that MAGFest, with its convention halls full of people make it
very hard for wifi devices to communicate. To make matters even more disastrous, the
VRChat portal was to be set up right next to the LAN room.
If a swadge begins to hear the transmission from any wifi device on its channel, it will
begin decoding that transmission and any other transmissions that come in will be lost. For
instance if the sender didn’t know to wait one swadge could send a message to another,
oblivious that the other was already receiving a message (also called the “Hidden Node
Problem”).
Another issue is if there was a transmission detected, a sender would “hold off” because
of “Carrier Sense Multiple Access.” So, sometimes packets transmissions could be delayed
by 100ms or more! With these issues, neither the portal or competitive fast-paced head-to-head
sci-fi flight free-for-all dualing would be any fun.
We were going to have to put earmuffs on the swadges. If they can’t t hear distant wireless
devices, then those distant wireless devices can’t cause problems.
We actually did this previous years, by accident, by putting a metal plane immediately behind
the built-in antenna on the ESP module.
No such luck this year, since we had actually followed the design guidelines, and with thousands
of swadges already in production, there was no hope of changing that now. Not that putting
copper in the wrong place was an efficient, reproducible or reliable solution anyway.
We’d have to hack the ESP to put those earmuffs on in software.
Espressif doesn’t document their wifi silicon, or provide source to their libraries that
use it. I knew this would take several hundred or even thousands of experiments to figure
out how to give the ESPs a little hearing loss. I had another trick up my sleeve. When
I was optimizing the drawing code for the swadges, I wrote a tool that could help me
rapidly prototype code. Any time a source file was saved, the tool automatically compiled
and ran code on the device all in about ¼ of a second. This meant I could recompile
and test all sorts of wacky ideas quickly. I was able to shoot through 10 or even 20
development cycles in just one minute.
Instead of being strategic with what registers I wanted to explore, I just sprayed and prayed.
A coffee-fueled morning, forgotten afternoon, sleepless night, one breakfast, one lunch,
and a sad girlfriend later, I had some cursed code that did the trick. The virtual earmuffs
were created! We could convince the hardware to reduce its “IGI” or “Input Gain”
by several dB. The Swadges could now talk much more reliably with each other, at the
cost of range. Also, to prevent other interference and save battery, I significantly reduced
the TX power. I just made sure they could still talk ~25 feet (or 8 meters) apart. It’s
not the 1500 feet (or 450 meters) that they normally get, but for swadges, this is exactly
what we wanted.
We added multiplayer to a 2rd tier Magfest Swadge game which is nothing new, since the
headliner game, Swadge Bros was doing this already. So far these changes only let people
play in meat-space with each other. No one in VRChat would be able to participate.
The Swadges have a USB port. So, using hidapi (a very simple cross-platform library able
to send messages to/from USB devices) and a special USB descriptor that tells windows
“allow apps to use the USB device without administrator access” We could now connect
a swadge to a desktop PC that could act as a bridge. Hearing all of these updates from
the swadges in the area.
Once the events were on the desktop, I needed a way of getting the data to everyone in the
MAGFest VRChat world. One technique was to use video players. In VRChat you can use video
players to play video streams, like of live concerts, DJs spinning, or whatever else.
But, we needed to send in game data. Not video. Both Lox and AcChosen have already used video
players to stream data into VRChat worlds with data riding along. Lox wrote ShaderMotion
to enable streaming, recording and playback of people dancing and AcChosen who use video
players to stream in DMX lighting data for the most off-the-hook club worlds.
The only issue is when using colors to encoded video data, it is really inefficient. For
80+ ships flying around and shooting boolets it would be infeasible to encode all that
data as colors with enough reliability.
This is where I had an ace up my sleeve. About 2 years ago I wanted to teach myself how the
video encoding format H.264 worked. I learned how to encode binary data into video streams,
sidestepping all compression. Using h264fun I was able to take the raw data, all of the
ships flying around and the boolets they shot, and then broadcast that data though RTMP into
VRCDN, a cloud content delivery network. VRCDN then broadcasts RTSP to all the players in
the VRChat instance from around the world, most of whom, with less than 80 milliseconds
of latency.
Because there no was no encoding going on we could save time and not worry about encoding
artifacts in addition to freeing up the encoder and OBS to stream the video data coming in
the webcam at the portal. There were some subtle restrictions about 0-bit runs so I
needed to encode the data a little creatively. But, I was able to get the data through!
We ran into some subtle issues on a few people’s systems with unusual video settings that made
the raw data not have a 1:1 relationship. In future projects instead of asking users
to turn off settings in Windows, we could probably resolve our issues by encoding 6
bits of data per component instead of 8.
Once the raw data was in the world, the in-game video players would place the video data on
a texture. That texture could then be “exported” to other things in-world. One of those being
a “CRT” or “Custom Render Texture” that could parse out the data from the video
stream into a format that was easier to deal with. That data could then be loaded into
a pile of triangles with a “geometry shader” to take the data from the CRT and rotate and
translate the source triangles to twist it into rendering the M-Wing ships and bananas
in the VRChat player’s worlds. This allowed them to see high-fidelity versions of the
M-Wing ships and boolets the swadges were shooting out.
This was fun but players in VRChat would need a way to defend themselves from the swadge
players flying around and shooting them with bananas. Thankfully Draken Stark had been
working on a system of banana guns for the VRChat users, so that they too could shoot
bananas. Once Draken was done, he gave me a wonderfully basic interface so that I could
add some code to handle things like when bananas were fired, or informing when a banana gun
is picked up, so I could render the bananas and the guns on all the swadges.
I just wrote some glue code that got this data into a constant buffer that was accessible
by a small strip of pixels that lived on the left side of the VRChat window. I also output
the data from AudioLink, a tool in VRChat that does sound reactive lighting so that
the lights around the portal or the lights on my computer case for the mobile portal
would match the visual effects in-world and on people’s avatars.
I found an API in VRChat that allowed me to “export” skeletal data for all the players,
so I piled that in the exported strip of pixels as well.
To prevent resource issues or the annoying yellow border, or having to write thousands
of lines of code to implement windows streaming, I found out I could just use the Windows 3.1
GDI from 1992 to grab a handle to the VRChat window and and “print” it out to a buffer
in my application. From there I sifted through the pixels to reconstruct data.
First I stripped off the color data from the music. I sent that to the swadge flashed as
a DMX lighting controller which lit up the light tubes around the portal or on my motherboard
to be in sync with the DJs DJing and other lighting effects in-world.
Then, I used the skeletal data to create small stick-people skeletons of everyone in VRChat.
Because it was using the actual skeletal data, for all the players in VRC who use full-body
tracking, you could see them dancing around on the swadge. Yes, even this potato of a
game system supported legs.
I took these stick figures, gun locations and bananas as they were being fired, and
sent out all the data through the bridge swadge swadge that took in the ESP-NOW data. This
way the players in VRChat could actually shoot back at the swadge players to blow them up.
People had a lot of fun and players ran around having a ball firing bananas at each other,
at ships and ships blowing each other up, too!
You might be wondering, what happens if the people playing on the swadges shoot back at
the players in VRChat? Nothing. That’s what happens. In VRChat data from the GPU can’t
go back to the CPU without extreme performance implications. It’s like a trap door. Data
can flow from the CPU to the GPU but not from the GPU back into the game logic itself. At
least data can’t go back until we get async readback.
We had our main portal, and sometimes a mobile portal we set up in the main atrium.
We had a great DJ lineup. Tons of people taking pictures on both sides of the portal. People
meming off each other, dancing, trading up hanging out and chatting.
Jazzy Sempai, our last DJ of the night, asked us if she could stay on after her set to play
some jackbox games. Perplexed, we said sure, since they’re also the Project Lead of VRCDN,
that little service that we were using, so we weren’t going to stand in the way of
one of their ideas.
Jazzy started a game, and people on their cell phones in the real world and in VRChat
joined the game. It was incredible to see how engaged people across this digital divide
played all together.
Looking at this, I had to ask myself. Did Jazzy’s popping open a session of Jackbox
Games accomplish everything that we spent months reverse engineering, designing and
executing?
You can tell me what you think in the comment section.
Thanks for hanging on for this long of a video. I wanted to thank VRChat for this great platform,
The Swadge team, VRChat Portal team, MAGFest for giving us these opportunities, the Portal
moderators and VRChat players who came out, my girlfriend for editing the script, and,
you. Thanks for watching.