I've been working really hard over the last few weeks on this little tiny RISC-V emulator. The
really tiny part about is is that it doesn't have an MMU which is something that virtually
all desktop modern processors have. The reason I wanted to do this was I wanted to see if I could
run Linux on it. Something that was close to but not as simple as an ESP32-C3. And well the answer
was. Yes. In fact I was able to write a really tiny RISC-V emulator. The actual emulator part
all exists in this one function in this header file and it was only around 350, 400 lines of
code. And it's able to run Linux and I'm able to have executables and whatever on it. Like I can
run duktapetest with fizzbuzz.js and you can see that it's able to run these application in Linux.
Now the big part of this is that no one would be able to expect to run Linux without an MMU,
a Memory Management Unit. But there's been a lot of efforts by people like ReggyMM and the people
who are working on the K210 Linux efforts. To be able to make it so you can actually run Linux
on these really primitive processors. My primary inspiration for this project was by Pi. Pi wrote
this pixel shader that actually emulated a full RISC-V RV32IMA+SU CPU. Now this had everything.
It had the full memory management unit. It had full SU. It had all the ZiCSR's. And it ran Linux.
Full-fledge Linux inside of the pixel shader. Which was wild. Because inside of the world that
this computer ran in, there was a texture on the wall that was literally the system memory of the
system that was booting Linux when you loaded into the world. That was just crazy. I wanted to see if
I could make it a little bit faster, but mostly, I just kinda wanted to explore this just for fun.
I've been throwing around a lot of acronyms. But that's because RISC-V isn't just one basic
architecture. It's a very flexible architecture. Able to be 32, 64, or 128 bit. And it has a myriad
of different configurations of core processor and different components that can be placed around
it. Everything from core processors that don't have integer divide and multiply features to
ones that have all sorts of crazy features like ones that have quad floating point extensions.
While you can build the whole system from scratch, I included a build step that does all
the hard work you're about to hear about called `make testdlimage` that downloads a pre-built
system image and compile the emulator and runs it so you can get it up and running in under 2
seconds. Or if you're on Winodws and you don't want to install the Windows Subsystem for Linux,
you can easily download tinycc with a powershell script or an installer. And I have a powershell
script that downloads the image and compiles the emulator with TCC and runs it. Neat thing
is the resulting EXE in Windows is only about 16kB and has no dependencies. The emulator,
itself has platform-specific code to handle things like the Windows terminal being put into VT100
mode that Linux expects, handling keyboard and timers. Going deeper though, I'd like to tell the
story of how I did this project. I barely knew about RISC-V and I had no idea about any of the
more advanced features in it. Or if it was even possible to build an RV32-NOMMU Linux image. So,
the first thing I did was google it. And I came across this tweet from August. So I reached out
and RegyMM stated heping me. What this actually was was just a buildroot configuration and a
set of patches. So, I was able to take those patches and put them into my project. Now,
buildroot is a sort of Makefile sort of thing. But it's enough of a Makefile to build an entire Linux
system. If you're going to work on this project one of the first things you're going to want to do
is clone this project and say `make everything`. `make everything` is actually going to get a copy
of buildroot, and use the configuration that regymm and I put together for the kernel and
for buildroot to be able to put together a system. Now, buildroot is realy cool because
it builds EVERYTHING. It builds a compiler. And it builds the compiler that's going to use that
compiler. That's going to be your host compiler. Now your host compiler is also going to build the
compiler which is going to compile the OS on your target device which is a lot of fun. Because you
can watch the text streaming by, it really feels like you're gaining a couple IQ every single time.
Now, this is not just compiling the base system. It's compiling things like Busybox, and Busybox
contains all the important functionality for you to be able to do on your system. Things like
`ls` and deleting and moving files and things like that. But it's also compiling the Kernel,
which is the core OS, and on top of that it can even compile things like QEMU. Now,
the nice thing about that is it was able to get me to the point where I could verify that the
version of what I was working on actually did work. Which was great, because it's
so much easier to start from a known working condition of things and kinda work backwards.
Now I do want to spend a quick moment complaining about that. Because for some reason QEMU uses
something in their toolchain which causes python to get built which requires ninja
which requires cmake. Now, the annoying thing is buildroot would just be done if not needing
CMAKE. CMAKE is one of the longest legs in all of this and Python right behind. And QEMU last. Gah.
Now, it's realy important that you start with something that works because you want the thing
that you're testing to actually work so when you're writing your own emulator so when you're
met with this. This black screen of doom with nothing coming out the other side you can start
to figure out what's going on. There's realy two pieces that I had to go on. One was that I could
make my kernel and I could test with QEMU and I could verify that whatever I was trying to test
really did work. The other was I could go and I could look at the code that pi maker wrote for
their system and I'd be able to see how they were able to get it working. I've also included a bare
metal project. And what this does is it runs without any OS or any bootloader or anything
else. So that you can write C or assembly code and see how it gets linked together in one flat binary
file. Now this includes examples of how to write inline assembly and how to write GNU assembly
right there that can be linked together with your C code. It's kind of an educational tool for the
most part but it's really helpful when I was first getting the processor off the ground just to be
able to step through everything individually to make sure that the processor was behaving itself.
One of the things that I wanted to touch on was makefiles. Makefiles are a really
powerful tool that are actually a lot simpler than you'd expect. All they are is just a set
of rules of how to make things. So, to the left of the colon you have the thing,
and to the right you have all of the things that that thing depends on and underneath of it,
you have all of the commands that you need to be able to run to get that thing. And this is really
great for chaining together whatever kind of complicated tasks that you'd want to do. You can
just tell make exactly how you want to go build what you want to build. It doesn't have to be
complicated. One of the things I did a lot of was printf debugging. And that was one of the things I
left into the final copy here. So, with -s being passed into the flags, you can then
see what the processor is doing at every single time step here. It's starting at address 8000000,
moving up to 5C, 60, 64 and so on. And I can go catch it when it's in some weird failure mode.
You might wonder, that's great, but what do those crazy numbers mean? Well, what I can do is I can
say `make dumpkern` and this is using one of the tricks I talked about in my talk. Where you can
use `objdump -S` to dump the raw binary of the thing into a sort of easier to read assembly
file. And that's what this is right here. And what I can see on the far left is the address that each
one of these assembly instructions is located at and the assembly instructions assocated with that
command and because there's debugging symbols included with the kernel I can also get the
functions that everything is located within. So, for instance where I'm stuck bouncing between b4,
b8, b0, b4, b8 and so on, I can go see where is that. And I can do that by ctrl+f and looking
for it and seeing oh, it's actually inside this section called clearbss and that's great because
that's going to take a while for that section to finish. But let's let it run a little bit longer.
So, now, we're in some other strange location. Let's see where we are right now. I'm gonna go
take this address and paste it in here and see what it's doing. And so I can see that this is
inside this little save context thing and I can go up and look around and see that it's inside of an
interrupt. And I can start searching these things out and trying to understand why is my processor
not working. Now the one thing that's really depressing is I did find out what it was. But
it took over 7 or 8 hours for me, in this case, one of the most frustrating bugs that I had was
that I had to go flip the direction of the sign there. So, if I were to do that, and I were to say
testkern, now all of a sudden everything did work. Now, of course in reality, it wasn't everything
just worked. But I would do one step at a time and you could just imagine the party I had when I was
finally able to see it print out Linux version 5.18.0. For now, though, let's go change this
back and I wanted to show you another technique that I used to debug this. Inside of here is the
entry code for the kenerl. And conveniently everything goes through the one handler. So,
it's very, very basic. So I can go insert assembly code in here to do whatever I'd like.
Now, one of the interesting parts about being able to write your own emulator is you can do
whatever you want. So, for instance if I go write to a CSR, I can go execute a printf
right there. So, I can printf debug my code from inside my kernel outside my kernel. Now,
some of the things I had before were 138 would go print a string from wherever the
pointer was. Is this safe? Absolutely not! Because it means that something inside of
the emulator could run code outside the emulator. Was it fun? Oh boy was it!
So, for instance, here, let's go add something just to be able to see what the registers are
or something like that. If say, for instance here, I wanted to be able to investigate what
is the value of SP, I could say csrw 0x137, sp. Now, I might have the code wrong, but let's see.
Now, inside of the main thing I can say `make toolchain` again. And it's
going to go copy the configs over and it's going to go buildroot. Buildroot
is then going to recompile the kernel and get it ready for output. And, from there,
I can say `make testkern` and from there I can see that it's saying a lot of 80001bc's. So,
I can now know what that value is right there in assembly. So that whole process was pretty quick.
Just because from here you can go investigate all of these different things and be able to
pause hte code effectively or even have it so the code would exit in a certain situation if a
certain value is a certain thing and that was just one of the techniques I used when I was exploring
all of this. If I wanted to say investigate x1 I could do that and you can see this again.
I can say `make toolchain` and toolchain goes pulls all of the configs then starts compiling,
recompiles the kernel has it ready for use and there all of that wasin real time. So I
was recompiling the Kernel. And rebuilding the system image to be able to do that. Now
one of the interesting parts that I'd like to dig into now is exactly how the system image works.
The system image is part of buildroot. And inside buildroot you have the output folder. And inside
the output folder you have Images. And here we have a bunch of different things. We have a
rootfs, which is a sort of tar version of our target right here and target can be whatever
you want to put in here. Now, mind you this is not a root filesystem because you can't run it
from here. But you can put files that you want to end up in there. Now what happens is the tar
is made. The tar is turned into a cpio. The Linux kernel has to be compiled once to get that going,
but then the CPIO can be embedded in the image. So, the image here is literally RISC-V code,
from the start and it contains the entire root filesystem. So, if I say `hexdump -C image`
you can actually see that this code here contains the raw binary blob and it can just start
executing. In fact this first thing here is a jump. Which takes me to another interesting point.
That interesting point is this tool here. This rvcodec.js. Now it's a little bit annoying
because they represent the bytes backwards for me. But, we can just go reorder the bytes inside
of here and we can see what the command is. So I can look at the very first line,
the very first 4 bytes of the kernel image and we can see that it's a jump and link x0, 92. So
that means it jumps forward 92 instructions. 92 is 5C hex. So if we go to 5C right here
we can go see what the next instruction that the kernel executes is. And that next instruction is
csrrw x0, mie, x0. I don't off the top of my head know what that is. I think what that is doing is
making it so no interrupts can fire. Which is a pretty grand thing to do as your first operation.
Let's see how that compares to what the emulator says is going on. So, if I go now instead of
running it with that. Let's go and undo the change that we just made. And rebuild.
Ok, and now what we're going to do is step through each instruction. One at a time.
And let's see. We're going to 8000000 and then we jump to 5C and then the
next instruction is 60. And you can see this image that is being produced by Buildroot is
literally just a binary executable blob and that's what represents the kernel.
Let's go get this done with and fix the code again so this actually runs.
Once I was able to get my code working once I was able to then enter a mode of what's
called test-driven-development. Now, normally what people would do is start with really small tests
and get bigger from there. This was a little bit weird because really what I was trying to
find out was what is the bare minimum of what you can get away with and run the kernel. So,
I kinda did this backwards to get to the kernel. Once I got something that mostly ran,
a system that was up and running, I was able to go then and convert the code. This code here
is realy rough. Like very very poorly written, lots of junk everywhere and I was able to start
changing things and cleaning things up to the more modern code that I have here. And because
of test-driven-developemnt I had a really good test and I could run the Linux kernel and run
applications inside of the Linux kernel and I could make really small changes like that one
there and see if the Kernel still works. And I was able to shave off a lot of size
and complexity from my emulator over time just by being able to go edit it and and be able to
go test the emulator and be able to see that it still worked, because I was using it with Linux.
Normally when doing a project like this,
you'd want to use these RISC-V tests. These are very granular tests, the best way to do it,
but unfortunately the RISC-V tests here were particularly difficult to
get going in my infrastructure and they test huge portions of the processor that
Linux doesn't use and I really don't want to implement those portions of the processor.
While there's a lot more to do to make things better and faster and everything else. There are
some directions I would like to head with this. I would like to see if I can find a way I can make
Linux boot without the RV32A extension. For instance if you don't have multiple processors,
I think you should be able to get Linux to run without the RV32A extension. I also want to be
able to see if we can support relocatable ELFs. And there has been a lot of work done recently
on non-MMU systems. Additionally, one of the really annoying parts that I find is that when
I run it it runs really fast at boot, so like everything prints out immediately. But when
the final UART driver gets loaded, it makes it go really slow. And it's because the UART
driver that actually gets loaded for the real system is more complicated. But I don't really
get it? It looks like it's waiting on the baud rate basically? Verses earlycon. If you watch,
it just boots up instantly, so it'd be realy cool if I could find a way of making the boot
console be the thing that actually runs. You can see here, it's disabling the boot console. And
I have tried running it with `keep_bootcon` that doesn't seem to work for some reason.
I do hope to run this in a pixelshader. It'd be great if I could get OpenSBI working on
this because then I could just write some of the drivers in OpenSBI instead. This was the
route that Pi went, and that was a really good way to go it but I have not been able to get
OpenSBI working on this yet. Additionally, I think it would be really fun to be able
to embed this RV32 emulator in other random projects. It's like a scripting language,
except a scripting language you can compile to with GCC. There are other questions,
like can I allocate more memory contiguously? Any rate. This has been a really fun project. And I
might keep going with it. I might drop it on the floor. I'm not sure. I've really been surprised
by how fast it is, how great everything around it has worked. I wanted to give give regymm for
helping with a lot of this and their project. This QuaziSOC, which is intended to be something that
could be potentially something like a desktop system. If you're interested in this you can
check out mini-rv32ima. Project link is in the description. And thanks for watching.