Sparkrift

Love2D vs Allegro vs ClanLib vs LibGDX vs Cocos2D-X vs MonoGame vs XNA vs SFML

// October 12th, 2012 // 23 Comments » // Robofish

Hopefully you’ve read my previous post on the importance of choosing the right game framework.  Taking what we learned from Robofish we decided that our next game would be built upon a framework that had good performance, documentation, and ease-of-use.  If we stopped at those requirements then we probably would’ve just stuck with XNA since we used that on Robofish, and it meets those requirements.  However we decided to throw one more requirement into the mix: cross-platform support.

We’re targetting the PC for our next game, and we hope to get it released through Steam.  When we read about Steam Greenlight we were even more excited!  Getting your game on Steam greatly increases the amount of people you can reach, which can lead to great success.  Steam recently started supporting other platforms like Mac OS X and soon Linux as well.  I’ve also read this article on how releasing on multiple platforms can affect your sales.  We still haven’t 100% decided to go cross-platform, but it couldn’t hurt to make it as easy as possible to do that if we decide to, right?

Now that we have identified what to look for in a framework we can start evaluating our options.  I guess I should also mention that we’re sticking to 2D for now, so we’re only considering frameworks that have strong 2D support.  That helps narrow it down a bit.  Here’s the list of frameworks I came up with to evaluate:

Love2D v0.8.0: Code your game in Lua with this framework.  It supports Windows, Mac, and Linux.  It doesn’t have a huge following yet, but the forums are fairly active.  The documentation seems fine to me, and it was pretty painless to get a benchmark going.

Allegro v5.1: Started in the mid-90s by Shawn Hargreaves, this C/C++ framework has since evolved through the hard work of hundreds of people over the net.  They have support for Windows, Mac, Linux, iPhone, and Android.  The documentation is pretty standard, and there are lots of tutorials.  The forums have hundreds of thousands of posts.  Unfortunately after looking through the tutorials I decided it just looked too difficult to use.

ClanLib v2.3.6: This C++ framework is supported fully on Windows and Linux; it’s only partially supported on Mac.  The online documentation was incomplete when I viewed it.  The forums only had a few thousand posts, and they didn’t look that active.  Since this framework doesn’t completely support Mac and looks relatively immature I decided not to try it out.

LibGDX v0.9.6: A framework for Java, it supports Windows, Mac, Linux, Android, and HTML5/WebGL with iPhone support in the pipeline.  The documentation is there, but it’s a little unorganized.  Also the wiki doesn’t seem to be complete yet.  The forums are active with 10s of thousands of posts, and it looks like the earliest posts are from 2010, so it’s been around a couple years at least.

Cocos2D-X v2.0.1: You may have heard of Cocos2D, the ios objective-c framework.  The Cocos2D-X C++ framework is based on the same API but designed to be cross-platform; it supports Windows, Mac (somewhat), Linux, iPhone, and Android.  The documentation is good with some helpful tutorials.  There were about 15k posts on the forums when I looked, and there was at least one active topic each day.

MonoGame v2.5.1: This is a C# framework based on XNA’s API.  It uses Mono instead of .NET so that it can run on Windows, Mac, Linux, iPhone, and Android.  There doesn’t seem to be much MonoGame-specific documentation, but the API is identical to XNA, and there is a ton of XNA documentation.  It doesn’t seem to have taken off yet though, with less than 10k posts on the forums.

XNA v4.0: We used this C# framework for Robofish, so this is our control group.  XNA is easy to use and extensively documented with tons of tutorials and example code around.  The biggest drawback is that it is for Windows only since it runs on the .NET framework.  As I mentioned earlier a bit, if XNA had built-in cross-platform support I think it would’ve been a no-brainer for us.

SFML v2.0 RC: The Simple and Fast Multimedia Library is a C++ framework that looks really promising.  I love the documentation and tutorials on the website.  The forums are active with posts numbering in the 10s of thousands.  I looked at the API, and it looked very simple to use.  I was really looking forward to trying this one out until I discovered that it doesn’t support sprite batching.  That is a deal-breaker for us, unfortunately.  Hopefully it will be supported in future releases!

SDL v2.0 with OpenGL v4.3: I considered just using C++ SDL and OpenGL.  I figured that it would take a lot longer to get going as I learned how to access the video card at a lower level.  I also thought that maybe it would still be worth it if the performance was there.  However after spending a few hours looking through documentation and tutorials on various websites I decided that I’m just not ready to develop a game engine from this low of a level yet.  I had a hard enough time just trying to get some sort of sprite batching going, which I don’t think I ever really accomplished.  I want to focus more on making our game and less on telling the video card exactly what it has to do.

Whew, that’s quite a list of frameworks to investigate!  Fortunately we were able to knock four out of the running before we even got started, which leaves us with five frameworks to benchmark.  I didn’t want to spend a ton of time writing throwaway benchmark code, but I did want to see how each framework handled sprite batching and how easy it was to write a simple demo in each framework.  I decided the goal of the demo was to draw as many sprites as possible to the screen while translating, rotating, and scaling them at the same time.  I defined “as many sprites as possible” to mean that the demo had to maintain a steady 60 FPS.

So I grabbed a 64×64 PNG we had lying around (actually an old blowfish sprite from Robofish) and got to coding.  Developing the benchmark took different amounts of time for each framework.  I started in XNA because I was most familiar with that framework, and it was probably the easiest.  In the benchmark I simply create an array of sprites, and then in the update loop I loop through each one and set its rotation, scale, and location.  Next, in the draw function, I loop through all the sprites again and use XNA’s SpriteBatch class to draw all of them to the screen.  XNA performs great, and I was able to draw about 38,000 sprites to the screen at 60 FPS.

Note that this screenshot is from the LibGDX benchmark.  I used FRAPS to measure the FPS on Windows, but I later added code to print the FPS for the benchmarks for Linux.

If you’re interested in the code I’ve included a download link for the benchmarks on Windows here.  Once I had the code working as I wanted it to in XNA I began porting it to each of the other frameworks.

Porting the benchmark to Love2D was really quick.  I think it may have been the easiest to use after XNA.  It started off with decent performance as well with lots of sprites on the screen; that is, until I started rotating, moving, and scaling those sprites.  I ended up only being able to draw about 6,000 sprites and maintain 60 FPS.  Disappointing, but I wasn’t too surprised considering that Lua is an interpreted language.

LibGDX was up next.  I had a bit of a rough time getting the development environment set up.  They have a download tool you can use to get the framework jar files and set up a new project in Eclipse, but I didn’t know about that at first and ended up wasting some time.  The scaling didn’t default to a nice texture filter like XNA, so I also spent some time looking into getting that working so it didn’t look so pixelated when sprites were scaled up.  I also spent time trying out their built in spritesheet tool, which they call a texture atlas.  One other thing that tripped me up was even after I used their tool the generated Eclipse project wouldn’t compile until I created some directories in the project directory manually.  So there was definitely a learning curve on this one.  However, I was astounded by the peformance I got: about 33,000 sprites at 60 FPS.

I spent a lot of time looking at all of the features that Cocos2D-X provides, and I thought it looked like a really nice framework.  I was starting to regret looking at it after I spent a few hours banging my head against my desk trying to get a simple benchmark going.  I will admit that it is probably just me and my inexperience with the Cocos2D API, but I thought it was a pain to get working.  The code for this benchmark is also different from the other benchmarks.  I tried using some of the built-in framework functions for scaling, rotating, and translating on a timer; maybe that had something to do with how long it took.  Performance was pretty good with 24,000 sprites at 60 FPS.

That just leaves us with MonoGame.  I didn’t have to change my code at all from the XNA framework since it directly supports the XNA API.  That was definitely nice.  It was relatively painless setting up a new Visual Studio project and getting my code to compile and link with MonoGame instead of XNA.  MonoGame delivered good performance with 30,000 sprites on the screen at 60 FPS.  One benefit to using MonoGame is that we wouldn’t even have to use MonoGame on the Windows platform.  We could stick with XNA for Windows and use MonoGame for everything else.

Endless paragraphs of text are fun and all, but here’s a graph to show Windows performance for the different frameworks I measured:

Well sweet, we’ve got our benchmarks rolling and we know which one was the easiest to use with the best performance… XNA!  Wait, this was only Windows.  We should at least try the same benchmark on a different platform.  What if all this cross-platform talk is just shenanigans?  Let’s make sure that these frameworks actually do what they say!  I don’t have a Mac, so I decided to install Ubuntu 12.04 on my second hard drive and try getting the benchmarks to run on there.

I’m familiar with Linux, but I’ve never done any gaming on it.  About all I knew I needed to do was make sure there were some decent drivers installed.  Ubuntu makes it pretty easy to install drivers for your video card if you know where to look.  Once I had the drivers installed I copied over my benchmarks and tried to get them to run.

I decided to set up LibGDX first.  I used their tool to download the framework, launched Eclipse, imported my project, and hit the run button.  It actually ran!  At about 5 or 10 FPS.  Wow, that doesn’t seem very cross-platform to me!  Something has to be wrong.  I decided to cut back the number of sprites.  After dropping the number of sprites to less than 1000 without seeing a noticeably performance increase I knew something was wrong.  Maybe I hadn’t installed my drivers correctly after all?  I’ll skip over describing a couple painful hours and instead inform you that disabling Compiz on Ubuntu can do wonders for your FPS when running games.  After I learned that trick I used it for every other benchmark I ran on Ubuntu (it was a necessity).  The great part is that I got about 40,000 sprites on the screen at 60 FPS; that’s right, even more than I had in Windows!  The even better part is that it required absolutely zero code changes to port it to Linux!

Well alright, these frameworks actually are cross-platform!  Let’s try some more!  I downloaded Love2D next, hoping to see a performance increase there as well.  Unfortunately I actually saw a decrease.  It was only able to do about 4,000 sprites at 60 FPS on Linux.  I’m not sure why.  Since the performance wasn’t that great on Windows I didn’t really spend much time looking into optimizing it.  This one also didn’t require any code changes!

Next I downloaded Cocos2D-X and fired up my benchmark using that framework!  Or rather, I tried to.  Instead I got compile errors.  Linux support seems to be of secondary importance for Cocos2D-X.  I was less than impressed when I had to spend a half hour or so tweaking the code to get it to compile.  Once I had it running the performance was about 20,000 sprites at 60 FPS, which was less than on Windows.

Last we have MonoGame.  I figured it would be pretty easy to get this one to work on Linux.  That was unfortunately not the case.  I probably spent the most time trying to get my benchmark running with MonoGame.  One problem was that it required some extra setup in the solution that wasn’t documented that well.  A bigger problem was that the content pipeline isn’t completely implemented yet for Linux.  What that means is that you are dependent on a Windows machine with XNA installed in order to generate your resource files (like a sprite texture) for the game.  The MonoGame website states that they are in the process of developing a content pipeline that isn’t dependent on XNA, so hopefully they get that going soon.  Once I got over that hurdle performance was about 29,000 sprites at 60 FPS, which is pretty similar to Windows.  The bad news here is that it did not look good.  This is a bit subjective, but the MonoGame benchmark was noticeably more “choppy”.  The movement, rotation, and scaling of the sprites was not “smooth” even though it was supposedly 60 FPS.  I’m not sure what was going on here.

Here are a couple more graphs for you:

This is the performance of the benchmarks on Linux.  This graph looks pretty similar to the first one except the numbers are a bit lower for most of the benchmarks.

This is the performance of the benchmarks in both Windows and Linux.  You can see that all the benchmarks performed slightly worse except for LibGDX, which actually improved!

So which one do you think we chose?  Which one would you choose?

Despite some of the initial trouble I had setting it up, we were really impressed with the performance of LibGDX.  It also couldn’t be any easier to port to Linux.  You may be surprised that we’re going with a Java-based framework since most AAA games are written in C++.  Well we’re an indie studio, and indie studios have experienced success using Java before.  Even so, the benchmarks don’t lie, LibGDX is a beast when it comes to performance so we’re not too worried about it.

Hopefully LibGDX will work out for us, but there are a few other things we are taking into consideration that could lead to problems down the road.  The benchmarks showed that LibGDX performs well on Windows and Linux when doing simple sprite batching.  However there is a lot more to a game than just drawing sprites.  What about audio?  Or collision detection?  Don’t forget particles and shaders!  There are a lot of features we haven’t tested in LibGDX that may or may not work well in both Windows and Linux.  Now that we’ve chosen a specific framework it’s probably a good time to test the more advanced features that we know we’ll use on both Windows and Linux and make sure everything works as we expect!

Aside from untested features to worry about we also have to consider the install process.  Whoever wants to play our game will have to have Java on their computer in order for it to run.  Should we bundle the JRE in with our game?  If we do bundle it with our game should we do it behind the scenes or should we require them to actually install it?  There is some precedent for this on Steam: lots of games come bundled with DirectX and the DirectX installer runs when the game is installing.  I’ve also seen games come bundled with XNA.  If we don’t bundle it in then we need to inform the user that they need to find a specific version of Java and install it before installing our game.   We will likely bundle the JRE in with the game somehow so the user doesn’t have to worry about it.

Hopefully these two articles help you decide on a framework for your game, and if anyone ports my code to any other framework I’d love to see your results!

How to Choose a Game Engine

// October 6th, 2012 // 2 Comments » // Robofish

Good news: Jaret and I have begun work on Sparkrift’s next game! Bad news: it’s going to be a while before you can play it. Good news: the game is going to be awesome! We’ve been using the past couple of months to start laying the foundation for our next game. Jaret is doing a lot of concept art, and I’ve spent a lot of time looking into game engines and graphics frameworks, which brings us to to the topic of this post.

There are a lot of game engines and graphics frameworks to choose from, so how can you pick the right one? Maybe you shouldn’t even pick one; why not just write your own from scratch? That way you’ll have complete control and can tune it to match your needs exactly! Only you can decide what’s right for you, and I won’t go into writing an engine versus writing a game in this post. Here I’m going to walk you through our experience with Robofish.

Our goal for Robofish was pretty simple: prove that we work together and make a game that runs on the Xbox 360. Short of paying thousands of dollars for an official Xbox 360 developer kit that leaves you with XNA. That was fine with me because I’m a fan of C#, and since Robofish was our first game I didn’t think I’d be able to code it in OpenGL or Direct3D directly without it taking an extremely long time. After looking into XNA I realized that it’s really just a framework. That was good, but I knew if we used a game engine we’d probably finish the game even sooner.

Cue Torque X 2D. For a while XNA had a partnership with Torque where they would provide XNA developers with free access to the Torque X 2D game engine binaries. So I checked it out, read through the tutorials, coded up a little test to see if I could be productive, and it looked good! In only a few days I had gone from never creating a game to being able to move a little box around the screen and “shoot” smaller boxes. This was going to be cake!

I showed Jaret my fantastic demo and he was sold! After working on Robofish for a few months, I became a little frustrated with the documentation though. The tutorials were pretty decent, but there were only half a dozen or so, and they didn’t cover all the things I wanted to do in Robofish. Worse though was the fact that engine API documentation was practically nonexistent. I had no idea what features were available to me through the Torque engine. Posting on the Torque forums would lead to a handful of responses from a few dedicated Torque users if you were lucky.

Well I really needed to know how to use this engine, so we decided to buy the source for the engine, which was an option at the time. Once I had the source I was able to divine some of the meanings of the API calls. The source was decently documented, so that was how I proceeded. It was slow, and I was very unfamiliar with game engine frameworks and 2D graphics in general. Luckily I found a few things that seemed to work for the graphics and stuck with them while working on things like weapons and fish AI which didn’t need much from the engine. Things were going fairly smoothly until I realized XNA 4 was coming out soon.

Hurray, an upgrade to XNA! That should bring better performance and lots of cool new features and bug fixes, right? Well sure, it did some of that but there was a problem. Microsoft stated that you wouldn’t be able to publish XNA games on Xbox 360 unless they were using XNA 4 after a certain date. Robofish was never going to be done by that date. That meant we needed to upgrade to XNA 4. Torque X 2D was based on XNA 3, and I knew it wasn’t going to compile if I linked it with XNA 4. Well crap, to the Torque forums! Bad news led to worse news as I discovered that Torque was discontinuing support for the Torque X 2D engine. Holy crap, this game that we’ve been working on for over a year isn’t ever going to make it to Xbox 360 now!

You know that wasn’t the case though, so how did we do it? Not alone, that’s for sure! Torque X 2D was ported to XNA 4 by some very dedicated and generous Torque X 2D users. They named the port Torque X 2D CEV, which stands for “Community Enhanced Version”. Without their dedication I don’t know what would’ve happened. I doubt I would’ve been able to port the engine to XNA 4 by myself. Unfortunately the problems with Torque didn’t end there.

Performance was the largest problem we ran into with Torque. Maybe this isn’t an actual Torque problem because I don’t think they advertised blazing speeds with their engine. As an amateur I just assumed that if they were making this engine available to develop games for Xbox 360 that it would perform well. I did not benchmark it, which was my own fault. If I had benchmarked it at the start I would’ve discovered two things: the number of sprites that can be drawn to the screen is severely limited and games deployed to the Xbox 360 encounter massive stuttering periodically.

I was, and still am, extremely angry about the sprite drawing performance with Torque X 2D. It turned out that we were limited to somewhere around 200 to 300 sprites on the screen at once if we wanted to stay close to 60 FPS. That may be more than enough for a lot of games; it is not nearly enough for a SHMUPS. The number of sprites shown on the screen can add up quickly. Robofish had enemy sprites, weapon projectile sprites, effect sprites, and clam sprites on the screen all at once. The largest number of sprites comes from the weapon projectiles.

Let’s say you have a weapon that shoots 8 projectiles at once. Each time the weapon fires that’s 8 new sprites to draw to the screen. Depending on the weapon rate of fire and the movement speed of the projectiles there could be up to or more than 200 projectiles, and their corresponding sprites, on the screen at once. That’s pretty much all of available sprites right there. So we had to be really careful about limiting exactly how many weapon projectile sprites could be on the screen at once. Don’t forget that some of the enemies shoot projectiles as well. Get a few of those on the screen and that’s dozens more projectiles right there as well.

We also wanted to add particle effects to the weapons, which we did have in there at one point. The weapons looked awesome with particle effects, and it killed us to take them out, but we just couldn’t maintain a playable framerate with them in there. So why couldn’t Torque X 2D handle very many sprites on the screen at once? After investigating the engine source code I discovered that they weren’t using any kind of sprite batching. This means that each sprite gets its own GPU draw call, which is extremely inefficient. The difference in the number of sprites that can be drawn one at a time versus batching them together is a couple orders of magnitude; it’s that big of a deal. So if we were limited to 200 sprites before, if we had been using sprite batching we could’ve potentially been drawing 20,000 sprites at once with similar performance.

The second major performance problem in the Torque X 2D engine was the stuttering we encountered whenever we ran Robofish on Xbox 360. Anyone who has ever developed a game for the Xbox 360 using XNA knows what’s coming next: the garbage collector. Other people have written on the Xbox 360 .NET garbage collector before, so you can read about it from them if you like. The short version is you have two options when it comes to garbage collections: don’t create any garbage so the garbage collector doesn’t have to run or limit the number of objects on the heap so that the garbage collector can run quickly.

Running Robofish on top of the Torque X 2D engine meant the second option wasn’t really available to us. There were already tens of thousands of objects in the heap, so whenever the garbage collector ran it took a few milliseconds, sometimes up to 40 or 50 milliseconds, which is around 3 frames. Not drawing anything to the screen for 3 frames or more would definitely cause noticeable stutter for us. So if a lot of garbage was being generated by the game the garbage collector would run to free up memory and cause Robofish to stutter quite frequently. It wasn’t fun to play in that state, so I began to dig into the engine to find out where this garbage was coming from.

After digging into the Torque X 2D engine for weeks and months I basically came to the conclusion that whoever developed the engine hadn’t tested it very well on the Xbox 360. Maybe not at all. Maybe it was still a work in progress; I don’t know. What I do know is that I ended up making dozens of fixes to the engine source itself to keep it from creating garbage. One of the biggest improvements was the object pooling system.

Since Robofish’s weapons shoot so many projectiles, a good way to save memory is reuse the projectiles after they’re off screen. Instead of creating a new projectile when the weapon is fired and destroying the projectile when it hits an enemy or goes off screen you can turn the projectile into a new projectile. Recycling! Great idea in practice, but it wasn’t implemented very well in the Torque X 2D engine, and it would still leak garbage during the recycling process.

Even after all the time I spent trying to improve the engine I never truly got rid of the garbage allocation. I just got it into a state that was good enough to release. I’d like to note that both of the performance issues were not caused by Torque X 2D alone. There were many things I personally did wrong that caused performance issues. However, after I had fixed those issues I still had to go into the engine to fix what was left before the game was playable.

The point of this post is that game engines can be a blessing and a curse. I’m not sure if Torque X 2D saved us work or delayed our game release after everything is said and done. I suspect, even with all the issues we had, that we still came out ahead by using the engine… barely. That’s probably due more to me being a complete newbie to game programming at the start of the project, so it would’ve been really rough to create an engine for Robofish from scratch.

After thinking about it a lot, there are many things we could’ve considered to prevent the issues we ran into:

• Performance: We should’ve benchmarked the engine to find out if it would meet our needs. If we had done that at the beginning we would’ve caught most of the problems we ran into, and we could’ve avoided them.

• Community: The size of the user base and how active they are makes a big difference. If no one is using the game engine it’s probably for a reason. Having a large number of users also means quicker answers to questions you post on the forums.

• Documentation: If a game engine isn’t well documented it’s not a good sign. If you’re designing a game engine to be used easily you want to provide good docs to make it easy for the developer. I don’t want to use a game engine, or any library, that doesn’t have good docs; it’s just too much of a pain.

• Source: When the docs don’t provide enough information you can always fall back on the source… unless it’s closed. If you have the source you can also implement bug fixes yourself if the engine developer is too slow to provide fixes. Although if the developer is slow to provide fixes that’s a bad sign in itself.

• Maturity: If a game engine is brand new it’s probably still going to be changing, full of bugs, and not well documented. So just keep in mind that the newer something is, the less tested it will be and the more likely you’ll be doing the testing yourself.

• Features: The whole point of using a game engine is to save time. Verify that the game engine provides the features you will need for your game and test them to see that they work well.

Keep these factors in mind when choosing a game engine, and I think you’ll avoid most of the problems we ran into with Robofish. One final note on Torque X 2D, though: I tried to specifically call out the fact that we used Torque X 2D because there are lots of other Torque engines out there for other platforms. I have no comments on other Torque engines because I haven’t used them.

Stay tuned for my follow up post where I post some benchmarks on current game frameworks to show how we determined what framework to use for our next game!

Creating Sprites for Robofish

// July 31st, 2011 // No Comments » // Robofish

Robofish is my first real game project and when I started working on it I knew that using sprite sheets was the preferred method for exporting graphics to the engine but I didn’t have a clue how to set them up efficiently.  Sprite sheets are basically large image files that tile out each graphic or animation’s frame so that the game engine can read the images in order according to the sheet and display them on the screen when called.  That sounds confusing, I am not even sure I understand what I just typed!  Simply, a game engine runs faster if it can see a bunch of graphics on one image file rather than thousands of individual images.  Jacob, my Brother and the programmer behind our games, and I knew this but our first attempt at sprite sheets wasn’t quite the most efficient.  Continue reading to see how first discovered how to make sprite sheets and how we had to modify our entire sprite sheet process about 60% through production, also pictures below!

(more…)