Love2D vs Allegro vs ClanLib vs LibGDX vs Cocos2D-X vs MonoGame vs XNA vs SFML

// October 12th, 2012 // Robofish

Hopefully you’ve read my previous post on the importance of choosing the right game framework.  Taking what we learned from Robofish we decided that our next game would be built upon a framework that had good performance, documentation, and ease-of-use.  If we stopped at those requirements then we probably would’ve just stuck with XNA since we used that on Robofish, and it meets those requirements.  However we decided to throw one more requirement into the mix: cross-platform support.

We’re targetting the PC for our next game, and we hope to get it released through Steam.  When we read about Steam Greenlight we were even more excited!  Getting your game on Steam greatly increases the amount of people you can reach, which can lead to great success.  Steam recently started supporting other platforms like Mac OS X and soon Linux as well.  I’ve also read this article on how releasing on multiple platforms can affect your sales.  We still haven’t 100% decided to go cross-platform, but it couldn’t hurt to make it as easy as possible to do that if we decide to, right?

Now that we have identified what to look for in a framework we can start evaluating our options.  I guess I should also mention that we’re sticking to 2D for now, so we’re only considering frameworks that have strong 2D support.  That helps narrow it down a bit.  Here’s the list of frameworks I came up with to evaluate:

Love2D v0.8.0: Code your game in Lua with this framework.  It supports Windows, Mac, and Linux.  It doesn’t have a huge following yet, but the forums are fairly active.  The documentation seems fine to me, and it was pretty painless to get a benchmark going.

Allegro v5.1: Started in the mid-90s by Shawn Hargreaves, this C/C++ framework has since evolved through the hard work of hundreds of people over the net.  They have support for Windows, Mac, Linux, iPhone, and Android.  The documentation is pretty standard, and there are lots of tutorials.  The forums have hundreds of thousands of posts.  Unfortunately after looking through the tutorials I decided it just looked too difficult to use.

ClanLib v2.3.6: This C++ framework is supported fully on Windows and Linux; it’s only partially supported on Mac.  The online documentation was incomplete when I viewed it.  The forums only had a few thousand posts, and they didn’t look that active.  Since this framework doesn’t completely support Mac and looks relatively immature I decided not to try it out.

 

LibGDX v0.9.6: A framework for Java, it supports Windows, Mac, Linux, Android, and HTML5/WebGL with iPhone support in the pipeline.  The documentation is there, but it’s a little unorganized.  Also the wiki doesn’t seem to be complete yet.  The forums are active with 10s of thousands of posts, and it looks like the earliest posts are from 2010, so it’s been around a couple years at least.

Cocos2D-X v2.0.1: You may have heard of Cocos2D, the ios objective-c framework.  The Cocos2D-X C++ framework is based on the same API but designed to be cross-platform; it supports Windows, Mac (somewhat), Linux, iPhone, and Android.  The documentation is good with some helpful tutorials.  There were about 15k posts on the forums when I looked, and there was at least one active topic each day.

MonoGame v2.5.1: This is a C# framework based on XNA’s API.  It uses Mono instead of .NET so that it can run on Windows, Mac, Linux, iPhone, and Android.  There doesn’t seem to be much MonoGame-specific documentation, but the API is identical to XNA, and there is a ton of XNA documentation.  It doesn’t seem to have taken off yet though, with less than 10k posts on the forums.

XNA v4.0: We used this C# framework for Robofish, so this is our control group.  XNA is easy to use and extensively documented with tons of tutorials and example code around.  The biggest drawback is that it is for Windows only since it runs on the .NET framework.  As I mentioned earlier a bit, if XNA had built-in cross-platform support I think it would’ve been a no-brainer for us.

 

SFML v2.0 RC: The Simple and Fast Multimedia Library is a C++ framework that looks really promising.  I love the documentation and tutorials on the website.  The forums are active with posts numbering in the 10s of thousands.  I looked at the API, and it looked very simple to use.  I was really looking forward to trying this one out until I discovered that it doesn’t support sprite batching.  That is a deal-breaker for us, unfortunately.  Hopefully it will be supported in future releases!

SDL v2.0 with OpenGL v4.3: I considered just using C++ SDL and OpenGL.  I figured that it would take a lot longer to get going as I learned how to access the video card at a lower level.  I also thought that maybe it would still be worth it if the performance was there.  However after spending a few hours looking through documentation and tutorials on various websites I decided that I’m just not ready to develop a game engine from this low of a level yet.  I had a hard enough time just trying to get some sort of sprite batching going, which I don’t think I ever really accomplished.  I want to focus more on making our game and less on telling the video card exactly what it has to do.

Whew, that’s quite a list of frameworks to investigate!  Fortunately we were able to knock four out of the running before we even got started, which leaves us with five frameworks to benchmark.  I didn’t want to spend a ton of time writing throwaway benchmark code, but I did want to see how each framework handled sprite batching and how easy it was to write a simple demo in each framework.  I decided the goal of the demo was to draw as many sprites as possible to the screen while translating, rotating, and scaling them at the same time.  I defined “as many sprites as possible” to mean that the demo had to maintain a steady 60 FPS.

So I grabbed a 64×64 PNG we had lying around (actually an old blowfish sprite from Robofish) and got to coding.  Developing the benchmark took different amounts of time for each framework.  I started in XNA because I was most familiar with that framework, and it was probably the easiest.  In the benchmark I simply create an array of sprites, and then in the update loop I loop through each one and set its rotation, scale, and location.  Next, in the draw function, I loop through all the sprites again and use XNA’s SpriteBatch class to draw all of them to the screen.  XNA performs great, and I was able to draw about 38,000 sprites to the screen at 60 FPS.

Note that this screenshot is from the LibGDX benchmark.  I used FRAPS to measure the FPS on Windows, but I later added code to print the FPS for the benchmarks for Linux.

If you’re interested in the code I’ve included a download link for the benchmarks on Windows here.  Once I had the code working as I wanted it to in XNA I began porting it to each of the other frameworks.

Porting the benchmark to Love2D was really quick.  I think it may have been the easiest to use after XNA.  It started off with decent performance as well with lots of sprites on the screen; that is, until I started rotating, moving, and scaling those sprites.  I ended up only being able to draw about 6,000 sprites and maintain 60 FPS.  Disappointing, but I wasn’t too surprised considering that Lua is an interpreted language.

LibGDX was up next.  I had a bit of a rough time getting the development environment set up.  They have a download tool you can use to get the framework jar files and set up a new project in Eclipse, but I didn’t know about that at first and ended up wasting some time.  The scaling didn’t default to a nice texture filter like XNA, so I also spent some time looking into getting that working so it didn’t look so pixelated when sprites were scaled up.  I also spent time trying out their built in spritesheet tool, which they call a texture atlas.  One other thing that tripped me up was even after I used their tool the generated Eclipse project wouldn’t compile until I created some directories in the project directory manually.  So there was definitely a learning curve on this one.  However, I was astounded by the peformance I got: about 33,000 sprites at 60 FPS.

I spent a lot of time looking at all of the features that Cocos2D-X provides, and I thought it looked like a really nice framework.  I was starting to regret looking at it after I spent a few hours banging my head against my desk trying to get a simple benchmark going.  I will admit that it is probably just me and my inexperience with the Cocos2D API, but I thought it was a pain to get working.  The code for this benchmark is also different from the other benchmarks.  I tried using some of the built-in framework functions for scaling, rotating, and translating on a timer; maybe that had something to do with how long it took.  Performance was pretty good with 24,000 sprites at 60 FPS.

That just leaves us with MonoGame.  I didn’t have to change my code at all from the XNA framework since it directly supports the XNA API.  That was definitely nice.  It was relatively painless setting up a new Visual Studio project and getting my code to compile and link with MonoGame instead of XNA.  MonoGame delivered good performance with 30,000 sprites on the screen at 60 FPS.  One benefit to using MonoGame is that we wouldn’t even have to use MonoGame on the Windows platform.  We could stick with XNA for Windows and use MonoGame for everything else.

Endless paragraphs of text are fun and all, but here’s a graph to show Windows performance for the different frameworks I measured:

 

Well sweet, we’ve got our benchmarks rolling and we know which one was the easiest to use with the best performance… XNA!  Wait, this was only Windows.  We should at least try the same benchmark on a different platform.  What if all this cross-platform talk is just shenanigans?  Let’s make sure that these frameworks actually do what they say!  I don’t have a Mac, so I decided to install Ubuntu 12.04 on my second hard drive and try getting the benchmarks to run on there.

I’m familiar with Linux, but I’ve never done any gaming on it.  About all I knew I needed to do was make sure there were some decent drivers installed.  Ubuntu makes it pretty easy to install drivers for your video card if you know where to look.  Once I had the drivers installed I copied over my benchmarks and tried to get them to run.

I decided to set up LibGDX first.  I used their tool to download the framework, launched Eclipse, imported my project, and hit the run button.  It actually ran!  At about 5 or 10 FPS.  Wow, that doesn’t seem very cross-platform to me!  Something has to be wrong.  I decided to cut back the number of sprites.  After dropping the number of sprites to less than 1000 without seeing a noticeably performance increase I knew something was wrong.  Maybe I hadn’t installed my drivers correctly after all?  I’ll skip over describing a couple painful hours and instead inform you that disabling Compiz on Ubuntu can do wonders for your FPS when running games.  After I learned that trick I used it for every other benchmark I ran on Ubuntu (it was a necessity).  The great part is that I got about 40,000 sprites on the screen at 60 FPS; that’s right, even more than I had in Windows!  The even better part is that it required absolutely zero code changes to port it to Linux!

Well alright, these frameworks actually are cross-platform!  Let’s try some more!  I downloaded Love2D next, hoping to see a performance increase there as well.  Unfortunately I actually saw a decrease.  It was only able to do about 4,000 sprites at 60 FPS on Linux.  I’m not sure why.  Since the performance wasn’t that great on Windows I didn’t really spend much time looking into optimizing it.  This one also didn’t require any code changes!

Next I downloaded Cocos2D-X and fired up my benchmark using that framework!  Or rather, I tried to.  Instead I got compile errors.  Linux support seems to be of secondary importance for Cocos2D-X.  I was less than impressed when I had to spend a half hour or so tweaking the code to get it to compile.  Once I had it running the performance was about 20,000 sprites at 60 FPS, which was less than on Windows.

Last we have MonoGame.  I figured it would be pretty easy to get this one to work on Linux.  That was unfortunately not the case.  I probably spent the most time trying to get my benchmark running with MonoGame.  One problem was that it required some extra setup in the solution that wasn’t documented that well.  A bigger problem was that the content pipeline isn’t completely implemented yet for Linux.  What that means is that you are dependent on a Windows machine with XNA installed in order to generate your resource files (like a sprite texture) for the game.  The MonoGame website states that they are in the process of developing a content pipeline that isn’t dependent on XNA, so hopefully they get that going soon.  Once I got over that hurdle performance was about 29,000 sprites at 60 FPS, which is pretty similar to Windows.  The bad news here is that it did not look good.  This is a bit subjective, but the MonoGame benchmark was noticeably more “choppy”.  The movement, rotation, and scaling of the sprites was not “smooth” even though it was supposedly 60 FPS.  I’m not sure what was going on here.

Here are a couple more graphs for you:


This is the performance of the benchmarks on Linux.  This graph looks pretty similar to the first one except the numbers are a bit lower for most of the benchmarks.


This is the performance of the benchmarks in both Windows and Linux.  You can see that all the benchmarks performed slightly worse except for LibGDX, which actually improved!

So which one do you think we chose?  Which one would you choose?

Despite some of the initial trouble I had setting it up, we were really impressed with the performance of LibGDX.  It also couldn’t be any easier to port to Linux.  You may be surprised that we’re going with a Java-based framework since most AAA games are written in C++.  Well we’re an indie studio, and indie studios have experienced success using Java before.  Even so, the benchmarks don’t lie, LibGDX is a beast when it comes to performance so we’re not too worried about it.

Hopefully LibGDX will work out for us, but there are a few other things we are taking into consideration that could lead to problems down the road.  The benchmarks showed that LibGDX performs well on Windows and Linux when doing simple sprite batching.  However there is a lot more to a game than just drawing sprites.  What about audio?  Or collision detection?  Don’t forget particles and shaders!  There are a lot of features we haven’t tested in LibGDX that may or may not work well in both Windows and Linux.  Now that we’ve chosen a specific framework it’s probably a good time to test the more advanced features that we know we’ll use on both Windows and Linux and make sure everything works as we expect!

Aside from untested features to worry about we also have to consider the install process.  Whoever wants to play our game will have to have Java on their computer in order for it to run.  Should we bundle the JRE in with our game?  If we do bundle it with our game should we do it behind the scenes or should we require them to actually install it?  There is some precedent for this on Steam: lots of games come bundled with DirectX and the DirectX installer runs when the game is installing.  I’ve also seen games come bundled with XNA.  If we don’t bundle it in then we need to inform the user that they need to find a specific version of Java and install it before installing our game.   We will likely bundle the JRE in with the game somehow so the user doesn’t have to worry about it.

Hopefully these two articles help you decide on a framework for your game, and if anyone ports my code to any other framework I’d love to see your results!

50 Responses to “Love2D vs Allegro vs ClanLib vs LibGDX vs Cocos2D-X vs MonoGame vs XNA vs SFML”

  1. gabin says:

    LibGXD rules anyway!

  2. Toni Ruža says:

    Revisit alegro, I like it because it uses DirectX on windows and OpenGL on other platforms. OpenGL on windows can be trouble even with 2.0 on many intel video adapters.

  3. venesectrixzero says:

    That’s a good tip! I didn’t know that Allegro used DirectX on Windows, thanks!

  4. Solerman Kaplon says:

    Dunno how steam works, but on windows you should bundle de JRE in the installer, it will work without registry changes, just call the java(w).exe runtime and be done. Steam could have a list of more common dependencies and install them instead of each application installing their own everytime (see ubuntu package manager)

  5. Laszlo says:

    Have you take a look at MOAI?
    Would’ve loved to see how it compares with these frameworks.
    Maybe a follow up post? :D

  6. venesectrixzero says:

    Thanks for asking! I actually downloaded it, but I decided not to benchmark it. I think I may have seen the pricing page and not realized it was completely free to use. Or maybe I was disappointed that they didn’t support Linux yet. I’ll definitely keep an eye on it though!

  7. Haxe Games says:

    HaxeNME is the best cross-platform game framework if you’re interested in supporting Native, Flash, and HTML5.

    http://www.haxenme.org/documentation/features/

    http://esdot.ca/site/2012/performance-showdown-starling-vs-nd2d-vs-genome2d-vs-haxe-nme

  8. Sphair says:

    For those interested, here is a version of the benchmark coded in ClanLib 2.3 (OpenGL) and ClanLib 3.0 (D3D): http://esoteric.clanlib.org/~sphair/ClanLibBenchmark.zip Contained in the zip is the full sourcecode and precompiled Windows binaries for both ClanLib versions. Sorry didn’t have time to compile it on Linux, but should compile without any code changes if a Makefile was created.

  9. Mika says:

    Moai (http://www.getmoai.com) is both free to use and it also says in their FAQ that “The current release of Moai SDK is designed for mobile games on iOS and Android. Games built with Moai can also run natively on Chrome browsers, Windows, Macintosh and Linux.”.

    They don’t list Linux as recommended development platform currently though.

  10. Steve says:

    Hi Robofish, I performed your love benchmark again using a LuaJIT version of love, which improved the performance by ~80%, bringing it up to just better than 1/3rd the number of sprites that XNA could manage on my machine.

    @60fps
    Love – 5000 sprites
    Love (LuaJIT) – 9000 sprites
    XNA – 25000 sprites

    http://dl.dropbox.com/u/4214717/love-luajit-win.zip

  11. venesectrixzero says:

    @Steve, Hey, thanks for trying that out. That’s great to know about!

  12. venesectrixzero says:

    I wasn’t aware of this framework. Thanks for posting, I’ll check it out!

  13. Adrian says:

    I just stumbled accross this post on Reddit.

    I changed the window size to 1024×768 ( using an old 17” monitor) and got 60 fps on:
    libgdx – 60000 sprites ( I also got 60k using 1920×1080)
    XNA – 38000
    ClanLIB – 28000
    luaJIT – 12000

    Cocos 2D-X was a bit weird. I got 25000 sprites for the Release build and 200000+ for the Debug build but there’s this bit in the that must be causing the weirdness.
    // set FPS. the default value is 1.0/60 if you don’t call this
    pDirector->setAnimationInterval(1.0 / 85);
    It’s 1 AM so I’m not gonna bother computing the FPS manually.

  14. Max says:

    While I can’t speak for MonoGame specically, the choppiness you experienced might be a common quirk with the XNA framework that can be remedied by setting Game.IsFixedTimeStep to false.

  15. David says:

    SFML 2.0 does support sprite batching in the form of vertex arrays. There are also entries in the wiki regarding sprite batching.

  16. mr eel says:

    Thanks for posting this. Very valuable stuff. I hope more devs shared their experiences in this way.

    Perhaps add a link to more advanced tests if you made them already.

    Btw are you sure you can run java on windows 8?

  17. Edu says:

    Great article. I’m also choosing a framework for game development, I’m also looking to AndEngine. Any reason for you not considering it?

  18. Edu says:

    Forget it! Not cross platform…

  19. venesectrixzero says:

    Yep, looks like it is just for Android unfortunately!

  20. venesectrixzero says:

    @mr eel, I hadn’t thought about that actually. There is an official Windows 8 faq that says it is supported though!

  21. venesectrixzero says:

    @David, Thanks for pointing that out. I actually have an old SFML project where I experimented with vertex arrays a bit, but I wasn’t sure if I was setting everything up correctly so I didn’t think it would make for a very good benchmark. Which wiki page were you talking about? I don’t think I was aware of the wiki when I wrote this article.

  22. krzat says:

    I wrote a SFML.Net spritebatch a while ago:
    https://bitbucket.org/krzat/sfml.utils/src/5529b08fa64b62ccb9359ea8754b273c6acfb2b6/SpriteBatch.cs?at=master

    I will try to benchmark it later.

  23. Shell32 says:

    After optimizing the love2d code and modifying the love2d source. Using luajit FFI i got 80.000sprites at 35fps and 50.000 sprites at 60fps.

  24. venesectrixzero says:

    @Shell32, that’s awesome that you got love2d to perform so well; thanks for posting! Would love to see any code you want to share on how you did it.

  25. Shell32 says:

    @venesectrixzero Sure, https://dl.dropboxusercontent.com/u/70131373/Love090_Spritebatch.zip

    Just run RUNME.BAT
    It start with 80.000 sprites on screen. You may change it on the main.lua (NUM_SPRITES)

  26. venesectrixzero says:

    @Shell32 That’s really cool! It runs great on my machine as well. It’s great to know that Love2D can be on the same level as other engines as far as performance goes. How much did you have to modify the Love2D source?

  27. johan says:

    It looks to me like he’s using the FFI-branch from https://bitbucket.org/bartbes/love-experiments/

    That does make a huge amount of difference on my machine too.

  28. johan says:

    PS you can host the rad(deg(val.r) + val.rotSpeed * dt) to the love.init function. Not that it’ll help much but that’s 160,000 less function calls per frame.

  29. Shell32 says:

    @johan Now slime released it on experimental. The first comment I test by my own. After a discussion I convinced slime to try FFI.
    Btw, nice point. I didnt write this sprite script. I just modified it a bit.

  30. Crossie says:

    Hey, first of all, thank you very mucgh for this great article! I’m in the process of deciding which engine to use for my game as well.
    Do you guys still use LibGdx? What are your experiences with it? Would you recommend it?
    Did you have any problem with garbage collection (I’m always a little bit paranoid about GC ;)) ?
    Thank you very much in advance!

  31. venesectrixzero says:

    @Crossie Thanks for checking out the site! Yep, we’re still using LibGDX. I continue to like it, and it keeps getting more updates! I would definitely recommend it so far; it’s very easy to use. I think garbage collection worries may depend more on the platform you’re targeting. We’re not planning on releasing on Android at the moment, so we haven’t worried about the GC as much as we would if that were the case. I haven’t noticed any issues with it yet.

  32. Crossie says:

    Thanks for the quick reply! That’s good to hear. Thanks for sharing your experiences. It’s really hard to decide on an engine/framework, there are just so many things to take into account. Having the possibility to ask somebody who actually works with it and also knows other engines/frameworks is really great. Cheers!

  33. […] this. Here is a benchmark test for libgdx and is where I saw it reaching 40k sprites at 60fps. http://www.sparkrift.com/2012/1/love2d-vs-allegro-vs-clanlib-vs-libgdx-vs-cocos2d-x-vs-monogame-vs-x… . It seems libgdx barely goes over 30k actually. But still seems amazing. This is on the same level […]

  34. Dude K says:

    Hey dude!
    I wanted to know if you’re using libGDX for iOS game development also. I know that libGDX’s capabilities to cross compile code for iOS are limited now, but I’ll love to hear from you if you’ve done it, and how complicated or what limitations you’ve had, because I’m tired of cocos2d-x, and want to try out libGDX.
    For iOS.

  35. venesectrixzero says:

    Sorry, I haven’t tried out LibGDX for iOS yet. It still requires a Mac and the $100/year developer license to work with it. I know they just switched to the RoboVM for iOS, which is supposed to work a lot better than what they were doing before.

  36. Martin Felis says:

    Thanks for your benchmark. Do you still have the sources for the libgdx and the Love2d benchmark? Would love to perform checks with the newer versions. Especially since the newest Love2d version uses LuaJIT instead of stock Lua.

    If not: how did you move the sprites around? Randomly?

    Cheers,
    Martin

  37. Igor Silva says:

    When talking about Cocos2D-X you said it had a pretty good performance with 24K sprites on the screen, but I’m pretty sure that you meant 34K.
    Just a little correction :)
    Anyway, excellent article, thanks and gratz!

  38. venesectrixzero says:

    @Martin Yes, I do still have the source code. The link I posted is still working: . That’s good to hear about the performance improvements. I’ll have to take another look sometime!

  39. venesectrixzero says:

    @Igor Thanks for the comment! The Cocos2D-X bar (red) in the Windows performance graph looks like 24k sprites, right? Did I misunderstand you?

  40. minggo says:

    @Dude K
    Why did you say that you are tired with cocos2d-x?

  41. Hello! I could have sworn I’ve been to this website before
    but after browsing through some of the post I realized it’s new to me.
    Anyways, I’m definitely delighted I found it and
    I’ll be bookmarking and checking back frequently!

    Here is my site location bungalow Tahiti (http://www.hotel-bungalow-moorea.com/)

  42. Undeniably believe that that you said.Your favourite
    justification seemed to be on the web the easiest factor to understand of.
    I say to you, I definitely get irked at the ame time as other folks think about worries that they plainly don’t knoww about.
    You managed to hit the nail upon the highest aas well as outlined oout tthe entire thing without
    having side-effects , otuer folks can take a signal.Will probably be back to geet more.
    Thank you

  43. Very shortly this websit will be famous among all blog users,
    due to it’s pleasant articles

  44. SEO companies inside Middle East can handle keeping up with your small business expectations.
    So, a hardcore competition is on within the internet among the companies to increase
    their sales. This implies that it is imperative for all those business owners to
    operate a successful website.

    Also visit my website … consultant seo Belgique

  45. lenora8436 says:

    Hey there. Are you aware of making a web site cell phone helpful?
    I’m trying to find a theme or wordpress tool that might be able to handle this matter. Please share if you have any suggestions.
    Brief question that’s totally away matter.
    My web page appears odd when exploring.

  46. google.com says:

    Way cool! Some very valid points! I appreciate you writing this post
    and also the rest of the site is also very good.

  47. If you might have a very good credit standing, check around for any low rate
    of interest personal line of credit. In by doing this, anyone can fix any erroneous
    information that concerns his worthiness as being a credit card holder.
    t know how to discover if there is certainly wrong information on his or her
    credit profile, which impacts their total credit
    rating negatively.

  48. It’s appropriate time to make some plans for the future and it’s time to be happy.
    I have read this submit and if I may I wish to counsel you some
    attention-grabbing issues or suggestions. Perhaps you could write next articles referring to this article.
    I want to learn more things approximately it!

  49. Great post. I was checking continuously this blog and I’m
    impressed! Very useful information specially the last
    part :) I care for such info a lot. I was looking for this certain information for a very long time.
    Thank you and good luck.

  50. Avis says:

    I read a lot of interesting content here. Probably
    you spend a lot of time writing, i know how to save you a lot
    of work, there is an online tool that creates unique, google friendly posts in seconds, just search in google – laranitas free content source

Leave a Reply