Weekend Update: Ubuntu and OSX

During a “final push” this week to get LuaRousr 1.0 out, I delved into the land of CMake, Ubuntu, and macOS. Learned how to build my C++ extensions cross-platform, and also exporting from GameMaker Studio 2 with both the macOS and Ubuntu modules! While I’m still working on LuaRousr’s export (I ran into some issues that seem to be on the [GMS2] side), I did add support for Ubuntu to outsideTheBox as well as fix a couple lingering bugs I didn’t realize existed. ;)

!pitfall

Also, I’ve been working on a game with @net8floz, the newest member of rousr!

Crossing the Platforms

I’m pretty pleased with LuaRousr’s feature set now, having finished writing the ‘sync’ code to make sure both Lua and GML see instances the same, that I decided it was finally time to get the Ubuntu / macOS versions working. While I’ve got a tiny bit of experience messing with both platforms, I’ve never actually built anything for all 3 desktop platforms in C++, so it was a learning experience. First things first though, I had to get up and running on the other platforms to make sure I could even test to begin with.

Virtual Mac

I wrote a little bit about the process for Ubuntu earlier this week, and I’ll most likely do another one for macOS. Since I don’t actually own a Mac anymore, I sought an alternative method for working with it and tried both HostMyApple and MacInCloud. Having used MacInCloud previously, I knew what I was getting, but had hoped HostMyApple might provide a slightly less expensive experience, or maybe more performant. Ultimately, since HostMyApple is virtualized, I couldn’t test my GMS2 games with it - not enough OpenGL support… so back to MacInCloud for me.

Learning CMake ‘cause who needs to learn Make?

With both platforms setup - I needed to learn how to get my extensions actually compiling. I previously had some experience using CMake, so I gave it a shot which turned out to be very simple to configure builds how I need ‘em. The main issue I ended up having on the other platforms was not realizing the LuaRousr demo actually made use of outsideTheBox… and that’s how we’ve ended up with an outsideTheBox update. There’s some slight platform differences in implementing the executable path and file operations between Windows and the others, so it was nice to learn something new… easy, but new. After fiddling around a bit to make sure Lua was compiled in C++ and everything was compiled for the x86 architecture, it was pretty simple to get the extensions built. Upon building and running though, the Ubuntu build would hit some deadlocks or throw exceptions from futex I soon realized that I’d have to rethink how the threads were working…

Also, vscode

It’s worth a mention that [vscode] is fricken’ awesome for cross-platform C++ dev work like this. Something to possibly touch upon in the future is getting it up and going on Ubuntu and macOS, but thanks to [vscode’s][vscode] ease of use, running with gdb/lldb in a visual environment was absolutely simple. I had also set it up with the C++ extension, but since I was using CMake to build Makefiles, I didn’t build through it. However, running make from a vscode terminal does allow you to CTRL+Click error messages and vscode will hop right to the file with it… so it’s totally worth running through vscode still.

Found Work: LuaRosur Threading Rewrite

Unfortunately, I was having issues with the concurrentqueue library I was using to manage the threads in LuaRousr, and ultimately decided to rethink and rewrite how LuaRousr uses threads.

Note: We’re about to get really technical… perhaps you wanna skip to the game

During the first iteration of LuaRousr, I initially thought it’d be super cool to run Lua states on their own physical hardware threads, communicating with GML on an as needed basis. This actually turned out to be a lot of work for a pretty terrible idea: since GML is single threaded, the multiple threads would always be stalled waiting for GML to be able to answer any questions Lua may have (like calling GML functions, or getting a GML value). There was absolutely no benefit to running Lua this way as a result, and the communication nightmares and extra work wouldn’t be worth it.

But then…

So the second iteration of LuaRousr, I had the concept of a thread that runs Lua, but it was more a worker thread - it had a message queue and mostly waited to be told if it should execute some Lua script or resume Lua threads. Using buffers, GML would write what is basically a network packet, and pass it along to Lua via the C++ extension. Since you can pass buffer addresses to extensions, you can use buffers to pass around a lot of data. This helps with the fact that you’re limited to a maximum of 4 arguments on extension functions (unless they’re all Reals).

In addition to the Lua worker thread, there’s also the concept of a GML Command Queue that also ran on its own thread. These are associated with a command id which is an index the GML passes to the extension when it requests a new Lua action - I just increment it on any Lua Execute / Calls. It would then make its own thread, with the goal of sending data through a specific buffer. The necessity of the queue is because of how complicated things get once you have GML that calls Lua that calls GML that calls more Lua (into a theoretical infinity), and I wanted to ensure I supported it. This too used the model of a message queue, waiting for GML commands to run, then waiting for GML to give the buffer back to the extension.

And Hey, it’s faster too!

But after finally getting everything all setup on Ubuntu and building the demo, it quickly became apparent something was wrong. I’d hit uncaught futex exceptions, which I boiled down to some of the functions I was using from the concurrentQueue class which left me with two options: try to fix it or try without it. Obviously, more options include reporting the issue and waiting for a fix but I’m not so sure I wasn’t misusing it to begin with… so I opted to try to remove them. The first realization that I had was ultimately, there are far too many threads hanging out and doing things. The only necessary thread is for each call executed by something GML DID and not both to Lua and to GML. I ended up pretty much eliminating the GML Command Queue system, replacing it with more of a layer system that the working buffers are pushed and popped on to as the calls from GML->Lua->GML->Lua stack up. Each new GML call takes a thread from a worker pull, and rather than using a concurrentQueue I simply use the built-in std::condition_variable. I also adapted this excellently succicnt spin lock to use for checking when Lua/GML are ready to proceed.

The old architecture from the first implementation still lingered in some ways, which is why we still were left with this two thread source nightmare. I now was intimiately familiar with the concept that since GML is single threaded, only one “execution path” is ever proceeding, and it’s always the last began one (a stack of commands, basically). This realization allowed me some confidence on where and when things would need to block and wait, and ultimately, reducing the threads down to the single thread per call and not using a queue of messages (we’d never wait on more than 1 at a time anyway), I was able to beat the Ubuntu build issues and gained ~130FPS on my runs of the demo (on my machine it was 220->350FPS, results vary of course).

Pitfall

pitfall

@net8floz and I are also working on a game! It’s loosely based on what we think Pitfall! is without really looking at what it really is. @net8floz set up the platforming physics and just about all the gameplay we’ve got so far, while I added the verlet rope and bloom shader.

We’ve both delved a bit into the madness that is networking, and after a failed attempt to TCP Hole Punching using GMS2 + node.js, @net8floz switched over to UDP and is implementing RUDP. We’ve both done a fair bit of studying the Gaffer On Games article on Reliable Ordered Messaging which is an excellent guide to this stuff. After RUDP is up and running, I’ll make another attempt at a lobby system using UDP Punching.

It’s also of note that the sprites used currently are from didigameboy’s free asset pack which has a bunch of excellent looking sprites to use… for FREE!

In the quite near future, there should be some more info and animations pouring out of us over this little project, but for now that’s all I’ve got for this week!