Love2D vs Allegro vs ClanLib vs LibGDX vs Cocos2D-X vs MonoGame vs XNA vs SFML
// October 12th, 2012 // Robofish
Hopefully you’ve read my previous post on the importance of choosing the right game framework. Taking what we learned from Robofish we decided that our next game would be built upon a framework that had good performance, documentation, and ease-of-use. If we stopped at those requirements then we probably would’ve just stuck with XNA since we used that on Robofish, and it meets those requirements. However we decided to throw one more requirement into the mix: cross-platform support.
We’re targetting the PC for our next game, and we hope to get it released through Steam. When we read about Steam Greenlight we were even more excited! Getting your game on Steam greatly increases the amount of people you can reach, which can lead to great success. Steam recently started supporting other platforms like Mac OS X and soon Linux as well. I’ve also read this article on how releasing on multiple platforms can affect your sales. We still haven’t 100% decided to go cross-platform, but it couldn’t hurt to make it as easy as possible to do that if we decide to, right?
Now that we have identified what to look for in a framework we can start evaluating our options. I guess I should also mention that we’re sticking to 2D for now, so we’re only considering frameworks that have strong 2D support. That helps narrow it down a bit. Here’s the list of frameworks I came up with to evaluate:
Love2D v0.8.0: Code your game in Lua with this framework. It supports Windows, Mac, and Linux. It doesn’t have a huge following yet, but the forums are fairly active. The documentation seems fine to me, and it was pretty painless to get a benchmark going.
Allegro v5.1: Started in the mid-90s by Shawn Hargreaves, this C/C++ framework has since evolved through the hard work of hundreds of people over the net. They have support for Windows, Mac, Linux, iPhone, and Android. The documentation is pretty standard, and there are lots of tutorials. The forums have hundreds of thousands of posts. Unfortunately after looking through the tutorials I decided it just looked too difficult to use.
ClanLib v2.3.6: This C++ framework is supported fully on Windows and Linux; it’s only partially supported on Mac. The online documentation was incomplete when I viewed it. The forums only had a few thousand posts, and they didn’t look that active. Since this framework doesn’t completely support Mac and looks relatively immature I decided not to try it out.
LibGDX v0.9.6: A framework for Java, it supports Windows, Mac, Linux, Android, and HTML5/WebGL with iPhone support in the pipeline. The documentation is there, but it’s a little unorganized. Also the wiki doesn’t seem to be complete yet. The forums are active with 10s of thousands of posts, and it looks like the earliest posts are from 2010, so it’s been around a couple years at least.
Cocos2D-X v2.0.1: You may have heard of Cocos2D, the ios objective-c framework. The Cocos2D-X C++ framework is based on the same API but designed to be cross-platform; it supports Windows, Mac (somewhat), Linux, iPhone, and Android. The documentation is good with some helpful tutorials. There were about 15k posts on the forums when I looked, and there was at least one active topic each day.
MonoGame v2.5.1: This is a C# framework based on XNA’s API. It uses Mono instead of .NET so that it can run on Windows, Mac, Linux, iPhone, and Android. There doesn’t seem to be much MonoGame-specific documentation, but the API is identical to XNA, and there is a ton of XNA documentation. It doesn’t seem to have taken off yet though, with less than 10k posts on the forums.
XNA v4.0: We used this C# framework for Robofish, so this is our control group. XNA is easy to use and extensively documented with tons of tutorials and example code around. The biggest drawback is that it is for Windows only since it runs on the .NET framework. As I mentioned earlier a bit, if XNA had built-in cross-platform support I think it would’ve been a no-brainer for us.
SFML v2.0 RC: The Simple and Fast Multimedia Library is a C++ framework that looks really promising. I love the documentation and tutorials on the website. The forums are active with posts numbering in the 10s of thousands. I looked at the API, and it looked very simple to use. I was really looking forward to trying this one out until I discovered that it doesn’t support sprite batching. That is a deal-breaker for us, unfortunately. Hopefully it will be supported in future releases!
SDL v2.0 with OpenGL v4.3: I considered just using C++ SDL and OpenGL. I figured that it would take a lot longer to get going as I learned how to access the video card at a lower level. I also thought that maybe it would still be worth it if the performance was there. However after spending a few hours looking through documentation and tutorials on various websites I decided that I’m just not ready to develop a game engine from this low of a level yet. I had a hard enough time just trying to get some sort of sprite batching going, which I don’t think I ever really accomplished. I want to focus more on making our game and less on telling the video card exactly what it has to do.
Whew, that’s quite a list of frameworks to investigate! Fortunately we were able to knock four out of the running before we even got started, which leaves us with five frameworks to benchmark. I didn’t want to spend a ton of time writing throwaway benchmark code, but I did want to see how each framework handled sprite batching and how easy it was to write a simple demo in each framework. I decided the goal of the demo was to draw as many sprites as possible to the screen while translating, rotating, and scaling them at the same time. I defined “as many sprites as possible” to mean that the demo had to maintain a steady 60 FPS.
So I grabbed a 64×64 PNG we had lying around (actually an old blowfish sprite from Robofish) and got to coding. Developing the benchmark took different amounts of time for each framework. I started in XNA because I was most familiar with that framework, and it was probably the easiest. In the benchmark I simply create an array of sprites, and then in the update loop I loop through each one and set its rotation, scale, and location. Next, in the draw function, I loop through all the sprites again and use XNA’s SpriteBatch class to draw all of them to the screen. XNA performs great, and I was able to draw about 38,000 sprites to the screen at 60 FPS.
Note that this screenshot is from the LibGDX benchmark. I used FRAPS to measure the FPS on Windows, but I later added code to print the FPS for the benchmarks for Linux.
If you’re interested in the code I’ve included a download link for the benchmarks on Windows here. Once I had the code working as I wanted it to in XNA I began porting it to each of the other frameworks.
Porting the benchmark to Love2D was really quick. I think it may have been the easiest to use after XNA. It started off with decent performance as well with lots of sprites on the screen; that is, until I started rotating, moving, and scaling those sprites. I ended up only being able to draw about 6,000 sprites and maintain 60 FPS. Disappointing, but I wasn’t too surprised considering that Lua is an interpreted language.
LibGDX was up next. I had a bit of a rough time getting the development environment set up. They have a download tool you can use to get the framework jar files and set up a new project in Eclipse, but I didn’t know about that at first and ended up wasting some time. The scaling didn’t default to a nice texture filter like XNA, so I also spent some time looking into getting that working so it didn’t look so pixelated when sprites were scaled up. I also spent time trying out their built in spritesheet tool, which they call a texture atlas. One other thing that tripped me up was even after I used their tool the generated Eclipse project wouldn’t compile until I created some directories in the project directory manually. So there was definitely a learning curve on this one. However, I was astounded by the peformance I got: about 33,000 sprites at 60 FPS.
I spent a lot of time looking at all of the features that Cocos2D-X provides, and I thought it looked like a really nice framework. I was starting to regret looking at it after I spent a few hours banging my head against my desk trying to get a simple benchmark going. I will admit that it is probably just me and my inexperience with the Cocos2D API, but I thought it was a pain to get working. The code for this benchmark is also different from the other benchmarks. I tried using some of the built-in framework functions for scaling, rotating, and translating on a timer; maybe that had something to do with how long it took. Performance was pretty good with 24,000 sprites at 60 FPS.
That just leaves us with MonoGame. I didn’t have to change my code at all from the XNA framework since it directly supports the XNA API. That was definitely nice. It was relatively painless setting up a new Visual Studio project and getting my code to compile and link with MonoGame instead of XNA. MonoGame delivered good performance with 30,000 sprites on the screen at 60 FPS. One benefit to using MonoGame is that we wouldn’t even have to use MonoGame on the Windows platform. We could stick with XNA for Windows and use MonoGame for everything else.
Endless paragraphs of text are fun and all, but here’s a graph to show Windows performance for the different frameworks I measured:
Well sweet, we’ve got our benchmarks rolling and we know which one was the easiest to use with the best performance… XNA! Wait, this was only Windows. We should at least try the same benchmark on a different platform. What if all this cross-platform talk is just shenanigans? Let’s make sure that these frameworks actually do what they say! I don’t have a Mac, so I decided to install Ubuntu 12.04 on my second hard drive and try getting the benchmarks to run on there.
I’m familiar with Linux, but I’ve never done any gaming on it. About all I knew I needed to do was make sure there were some decent drivers installed. Ubuntu makes it pretty easy to install drivers for your video card if you know where to look. Once I had the drivers installed I copied over my benchmarks and tried to get them to run.
I decided to set up LibGDX first. I used their tool to download the framework, launched Eclipse, imported my project, and hit the run button. It actually ran! At about 5 or 10 FPS. Wow, that doesn’t seem very cross-platform to me! Something has to be wrong. I decided to cut back the number of sprites. After dropping the number of sprites to less than 1000 without seeing a noticeably performance increase I knew something was wrong. Maybe I hadn’t installed my drivers correctly after all? I’ll skip over describing a couple painful hours and instead inform you that disabling Compiz on Ubuntu can do wonders for your FPS when running games. After I learned that trick I used it for every other benchmark I ran on Ubuntu (it was a necessity). The great part is that I got about 40,000 sprites on the screen at 60 FPS; that’s right, even more than I had in Windows! The even better part is that it required absolutely zero code changes to port it to Linux!
Well alright, these frameworks actually are cross-platform! Let’s try some more! I downloaded Love2D next, hoping to see a performance increase there as well. Unfortunately I actually saw a decrease. It was only able to do about 4,000 sprites at 60 FPS on Linux. I’m not sure why. Since the performance wasn’t that great on Windows I didn’t really spend much time looking into optimizing it. This one also didn’t require any code changes!
Next I downloaded Cocos2D-X and fired up my benchmark using that framework! Or rather, I tried to. Instead I got compile errors. Linux support seems to be of secondary importance for Cocos2D-X. I was less than impressed when I had to spend a half hour or so tweaking the code to get it to compile. Once I had it running the performance was about 20,000 sprites at 60 FPS, which was less than on Windows.
Last we have MonoGame. I figured it would be pretty easy to get this one to work on Linux. That was unfortunately not the case. I probably spent the most time trying to get my benchmark running with MonoGame. One problem was that it required some extra setup in the solution that wasn’t documented that well. A bigger problem was that the content pipeline isn’t completely implemented yet for Linux. What that means is that you are dependent on a Windows machine with XNA installed in order to generate your resource files (like a sprite texture) for the game. The MonoGame website states that they are in the process of developing a content pipeline that isn’t dependent on XNA, so hopefully they get that going soon. Once I got over that hurdle performance was about 29,000 sprites at 60 FPS, which is pretty similar to Windows. The bad news here is that it did not look good. This is a bit subjective, but the MonoGame benchmark was noticeably more “choppy”. The movement, rotation, and scaling of the sprites was not “smooth” even though it was supposedly 60 FPS. I’m not sure what was going on here.
Here are a couple more graphs for you:
This is the performance of the benchmarks on Linux. This graph looks pretty similar to the first one except the numbers are a bit lower for most of the benchmarks.
This is the performance of the benchmarks in both Windows and Linux. You can see that all the benchmarks performed slightly worse except for LibGDX, which actually improved!
So which one do you think we chose? Which one would you choose?
Despite some of the initial trouble I had setting it up, we were really impressed with the performance of LibGDX. It also couldn’t be any easier to port to Linux. You may be surprised that we’re going with a Java-based framework since most AAA games are written in C++. Well we’re an indie studio, and indie studios have experienced success using Java before. Even so, the benchmarks don’t lie, LibGDX is a beast when it comes to performance so we’re not too worried about it.
Hopefully LibGDX will work out for us, but there are a few other things we are taking into consideration that could lead to problems down the road. The benchmarks showed that LibGDX performs well on Windows and Linux when doing simple sprite batching. However there is a lot more to a game than just drawing sprites. What about audio? Or collision detection? Don’t forget particles and shaders! There are a lot of features we haven’t tested in LibGDX that may or may not work well in both Windows and Linux. Now that we’ve chosen a specific framework it’s probably a good time to test the more advanced features that we know we’ll use on both Windows and Linux and make sure everything works as we expect!
Aside from untested features to worry about we also have to consider the install process. Whoever wants to play our game will have to have Java on their computer in order for it to run. Should we bundle the JRE in with our game? If we do bundle it with our game should we do it behind the scenes or should we require them to actually install it? There is some precedent for this on Steam: lots of games come bundled with DirectX and the DirectX installer runs when the game is installing. I’ve also seen games come bundled with XNA. If we don’t bundle it in then we need to inform the user that they need to find a specific version of Java and install it before installing our game. We will likely bundle the JRE in with the game somehow so the user doesn’t have to worry about it.
Hopefully these two articles help you decide on a framework for your game, and if anyone ports my code to any other framework I’d love to see your results!