My Writings. My Thoughts.

Multiplayer coding and the lockstep network model

// July 30th, 2013 // No Comments » // Robofish


Robofish was fun to play, but if it had been multiplayer it would’ve been easily twice as fun. Hanging out with your friends and playing the same game, cooperatively or competitively, adds a whole new dimension to gameplay. I loved playing Doom and Warcraft when I was a kid, but those few times I was able to get a network set up to take on my friends were always my favorite gaming sessions. Most of the game ideas my brother and I come up with are multiplayer because that’s where we have the most fun, which is why we decided to make our next game multiplayer. The problem with that is it’s only our second game, and we have never created a multiplayer game before. Time to do some research!


Luckily there have been hundreds of multiplayer games created to learn from. Unforunately only a small portion of them have their multiplayer algorithms documented or source code available. I have found information on the Source multiplayer engine, Quake 3, Unreal engine, Halo Reach, Tribes, and Age of Empires to name a few. All of these are great resources, but it can be more than a little overwhelming to try to come up with your own multiplayer networking implementation based on only these articles. And don’t forget that if you want to have networked physics there’s a whole other series of articles for that!  So how do you determine what networking architecture your game needs?


There are dozens (or probably hundreds) of solid ways to make a multiplayer game, but not every game has the same requirements. If you have a fast-paced first person shooter then obviously the Source/Unreal/Halo/Tribes architecture is going to be very appropriate for your game. If you are making an RTS with hundreds or thousands of units then you’re probably going to want to look more closely at the Age of Empires example. That’s basically what it came down to for us: is our game going to be fast paced and require instant feedback or can we live with a little delay? Will we have a few dozen units or a few hundred? Since our game is going to be a bit more relaxed with hundreds of units we decided to follow the RTS architecture and use a lockstep model.

In a nutshell, the lockstep model is each player’s computer simulating the game the exact same way as everyone else’s computer and receiving the exact same input as everyone else. Input is executed at the same time in each simulation on each computer by using turns. A turn is normally a short period of time, 100ms for example, and each time a new turn begins you handle any player input that occurs on that turn. So a game client will simulate the game for 100ms, and by the end of that turn the client will normally have received the data for the next turn. The client begins a new turn by handling any external changes to the simulation (this is a multiplayer game so other players will be changing the world as well), and then the client executes the turn just like the last one by simulating the world until it’s time to handle more external input.


The great thing about the lockstep networking model is it can support hundreds or thousands of units in a game. The downside to this is that it requires each client simulate the game exactly the same way as all of the other clients. This is different from a multiplayer first person shooter like Team Fortress 2 where the game server sends out continuous position/velocity/rotation/state updates for each unit in the game. The Team Fortress 2 model is workable when you are limited to a few dozen entities, which is the case for most first person shooters, but you quickly run out of bandwidth when you try to do that for hundreds or thousands of units in an RTS. The way we avoid sending all this data to each player all the time is by sending each player the initial state of the simulation and letting their computers simulate it. Also don’t forget that each player is interacting with the simulation; their interactions must be sent to all the other players so that all of the simulations can be kept in sync. If you know the initial state of the simulation, exactly how to simulate it, and all of the changes in the system from other players you can simulate the whole game on each client.

The trickiest part of simulating the game on each player’s computer is that each computer has different hardware, which can result in slightly different calculations. Over time these small differences can accumulate into big differences in the simulation state. When two simulations differ on two machines it is said that the two simulations are out of sync with each other. When a game is out of sync player A may think his warrior just killed Player B’s rogue, but Player B’s simulation shows his rogue handily dodging the warrior’s attack before stabbing him with his dagger. Bugs like this can be very difficult to track down, so think carefully about your game architecture before proceeding with the lockstep model. Since we’re using Java with LibGDX and JBox2D for physics it makes it easier for us to keep everything in sync.


The Age of Empires article I’ve linked a couple times already has a great explanation of the lockstep algorithm. However there was a major downside to their approach: “Since the simulations must always have the exact same input, the game can really only run as fast as the slowest machine can process the communications, render the turn, and send out new commands.” We didn’t want player A in our game to be slowed down by player B’s bad connection to the game server, so we make a slight variation of the lockstep model that goes like this:

Client:
Accept any available turn information from the server
Accept player input
Update physics and world state
Send any player commands to the server
Render the world

Server:
Accept player input from clients and queue it to be sent out in the next turn
Update physics and world state
Send next turn information to all clients if the turn period has elapsed


This is a client server model with the server relaying all players’ input to the other players. The server periodically sends out turn information, which the clients require in order to keep the game going. Even when players have not submitted any input for a turn the server will still send out “empty turns” which keep the clients processing. When the server receives player input from a client it queues it up to be sent out with the next turn packet. This is slightly different than how it was done in Age of Empires.  Age of Empires would queue up input to be sent out a couple turns later instead of the next turn. This was done to give computers with higher latency a chance to get their input into the same execution turn as other players. The price for this is that all other players have to wait longer to receive the turn information. Again, we didn’t want to sacrifice everyone’s connection for the sake of one bad connection.

There are advantages and disadvantages to running the lockstep model with a client and servert architecture. We have the input relayed through the server for three reasons. First, our game is going to have a persistent world that will be up 24/7. In a peer-to-peer architecture you can’t guarantee a peer is always going to be up, so we use a server. Next, the server can check player input and prevent cheating; the server is authoritative and the clients can always trust the data coming from it. Last, the server can also send out events that cannot be generated on the client, such as random enemy invasions. If we allowed something like that to be generated by a client it could potentially lead to one player cheating to generate a ‘random’ invasion into an enemy player’s base. The cost of this is some additional latency. Routing all packets to the server to be processed and then back out to the client adds a bit of overhead, but the tradeoff is worth it to us.


So what happens if one player lags in this lockstep model? Let’s say player A jumps from a 100ms latency to a 1 second latency to the server. That means that when player A sends input to the server it will take 1 second to get there. From that point, the server will send out player A’s input in the next turn to the other players. All of the other players will most likely receive player A’s input before the turn information containing that input from the server reaches player A. So player A will have to wait about 2 seconds before the client responds to his commands. However, everyone else will be completely unaffected by player A’s latency: to everyone else it will appear as if player A is sluggish. Once player A’s latency returns to normal his client will receive all the updates it was waiting on in rapid succession and catch up in a burst of activity. After that everything will be running smoothly again.

This type of lockstep networking most likely will not work for a lot of games. It is not very forgiving to high latencies. The input lag is essentially latency*2 + Max(0,turn time). You have the time to send the command to the server (latency), plus the time to get a response from the server (latency), plus how long it is until the next turn. Worst case it’s going to be the whole turn time and best case it’ll be no time. So if your ping to the server is 150ms, then the input lag will be 300-400ms. We can simulate latency in our game, and I consider it playable without annoyance up to about 200ms latency. After that point it becomes more difficult to ignore the input lag when you tell your ship to move, and it doesn’t move for half a second. Even at high latencies the game is still completely playable for the person with high latency, it just does not feel responsive. Our game isn’t twitch based and doesn’t need to feel immediately responsive, but input lag can be masked somewhat with client-side prediction.


I have a couple ideas of how some basic client-side prediction could be done to help mask lag. First, when a player tells their ship to move it could immediately begin rotating toward its destination. If the ship begins moving immediately after interacting with it then it’s going to feel more responsive. Second, play an animation when moving. So a player would tell their ship to move and it would play an animation showing the engines firing up and the exhaust trail pluming out. Another idea is to play sound effects that give the player immediate feedback. This can almost always be seen in RTS games when you give a unit an order and they will respond with an acknowledgement of some sort (zug zug!).  The nice thing about these modifications is that they would not affect the simulation and could be added on top of everything else relatively easily.  I have seen other, more complicated, ways suggested which may work as well. With any combination of these enhancements I think you could mask some of the input lag pretty effectively.

That’s our plan for our network model. If anyone has any suggestions or questions I’d love to hear them!

The Art of Concept Art

// November 9th, 2012 // No Comments » // Graphic Design

My brother and I have started working on our new project and I have been spending a lot of time researching different methods of producing concept art.  In this post I will be discussing my experience so far on the topic and the steps I have been taking to create my concepts.

When I first started working on Robofish I told myself I need to make some concept art before I do anything else.  I had seen plenty of concept art on the internet from other developers and came to the conclusion it would be a good idea, however, I had no clue what I was doing.  I didn’t do any research and just started drawing, while this helps to get some of your creative juices flowing it doesn’t end up being to productive in the long run.  When I went to create a new enemy or weapon asset in Robofish I would skip the conceptual phase and move straight to the production phase of the asset.  This lead to faster “in game” visuals but over time we kept noticing things about the graphics we didn’t like and I ended up re-creating nearly all the assets two or three times before the game was released.  If you take the time in the beginning to hash out some concepts and agree upon something before hand it will save you time in the production phase of your project.

So that is what I decided to do this time around and the first thing I did was head out into the internet and look at what other, more experienced, graphic designers have done.  I did quite a bit of research on different methods of concept art but decided to alter the steps a little to my style.  I do not consider myself a fine illustrator but more of a cartoonist or caricature kind of guy and a lot of the tutorials I read start out with sketches so I decided to do my sketches in Adobe Illustrator instead of on paper.  Nearly all of the artwork that you see in Robofish was designed and animated using Adobe Flash, so the transition to illustrator, while still a vector based program, has been pretty time consuming.  The main reason I am switching from Flash to Illustrator is “level of detail”.  Flash is an excellent tool for animation and interface graphics but when you want to get down and dirty into the cracks of your artwork and really polish the details it leaves a lot to be desired.  Illustrator has many powerful tools that give you the ability to manipulate your pixels to exact points.  Now that I know what program I want to be designing in I started the concept process.

  The first step I took was to acquire reference materials.   In my case, we have the idea of a space simulation and we are leaning towards 2D graphics at the moment; so I immediately downloaded a bunch of other developer’s screenshots of their 2D space games as reference material.  I also grabbed a few photos of stars, galaxies, and other realistic space imagery.  The reference material you use should be a guideline or sort of like a scrap book of past iterations of the type of project you are working on.  As you develop concept art these images will allow you track your progress as you evolve and improve on existing ideas currently being used, assuming that is your goal.  After I got my reference materials I started thinking of my initial ideas for the project.

Reference Material

 

At this point I opened up Illustrator and started working on my first sketches. This is one of the most difficult parts of the process for me as I am never satisfied with how my sketches look, they always seem to lack the detail that I am going for. While frustrating at first I learned that this is normal. Your first ideas should be rough and quick so you can produce a handful of ideas before deciding on one you like.  I started out by making the mistake of spending hours on detailed concept art which ended up looking to much like the reference material I was using as a guideline.  I had to take a step back and first think of ways that would make my designs look unique compared to existing artwork.  I did this by creating very rough and quick drawings of conceptual ideas I had, each one taking ten to thirty minutes.  I experimented from high detail artwork to more abstract looking designs.  I still have not decided on a specific look I am going for yet but now I have some concepts to display and discuss with my partner about.

At this point I am still working on fresh ideas and more concepts.  I really want to narrow down our options on this project to avoid having to remake graphics down the road.  Once I get a few concepts my Brother and I agree upon I move into the next phase of higher quality concepts.  Keep an eye out for the next post regarding more on my design process and some pics of my concept art.

Love2D vs Allegro vs ClanLib vs LibGDX vs Cocos2D-X vs MonoGame vs XNA vs SFML

// October 12th, 2012 // 41 Comments » // Robofish

Hopefully you’ve read my previous post on the importance of choosing the right game framework.  Taking what we learned from Robofish we decided that our next game would be built upon a framework that had good performance, documentation, and ease-of-use.  If we stopped at those requirements then we probably would’ve just stuck with XNA since we used that on Robofish, and it meets those requirements.  However we decided to throw one more requirement into the mix: cross-platform support.

We’re targetting the PC for our next game, and we hope to get it released through Steam.  When we read about Steam Greenlight we were even more excited!  Getting your game on Steam greatly increases the amount of people you can reach, which can lead to great success.  Steam recently started supporting other platforms like Mac OS X and soon Linux as well.  I’ve also read this article on how releasing on multiple platforms can affect your sales.  We still haven’t 100% decided to go cross-platform, but it couldn’t hurt to make it as easy as possible to do that if we decide to, right?

Now that we have identified what to look for in a framework we can start evaluating our options.  I guess I should also mention that we’re sticking to 2D for now, so we’re only considering frameworks that have strong 2D support.  That helps narrow it down a bit.  Here’s the list of frameworks I came up with to evaluate:

Love2D v0.8.0: Code your game in Lua with this framework.  It supports Windows, Mac, and Linux.  It doesn’t have a huge following yet, but the forums are fairly active.  The documentation seems fine to me, and it was pretty painless to get a benchmark going.

Allegro v5.1: Started in the mid-90s by Shawn Hargreaves, this C/C++ framework has since evolved through the hard work of hundreds of people over the net.  They have support for Windows, Mac, Linux, iPhone, and Android.  The documentation is pretty standard, and there are lots of tutorials.  The forums have hundreds of thousands of posts.  Unfortunately after looking through the tutorials I decided it just looked too difficult to use.

ClanLib v2.3.6: This C++ framework is supported fully on Windows and Linux; it’s only partially supported on Mac.  The online documentation was incomplete when I viewed it.  The forums only had a few thousand posts, and they didn’t look that active.  Since this framework doesn’t completely support Mac and looks relatively immature I decided not to try it out.

 

LibGDX v0.9.6: A framework for Java, it supports Windows, Mac, Linux, Android, and HTML5/WebGL with iPhone support in the pipeline.  The documentation is there, but it’s a little unorganized.  Also the wiki doesn’t seem to be complete yet.  The forums are active with 10s of thousands of posts, and it looks like the earliest posts are from 2010, so it’s been around a couple years at least.

Cocos2D-X v2.0.1: You may have heard of Cocos2D, the ios objective-c framework.  The Cocos2D-X C++ framework is based on the same API but designed to be cross-platform; it supports Windows, Mac (somewhat), Linux, iPhone, and Android.  The documentation is good with some helpful tutorials.  There were about 15k posts on the forums when I looked, and there was at least one active topic each day.

MonoGame v2.5.1: This is a C# framework based on XNA’s API.  It uses Mono instead of .NET so that it can run on Windows, Mac, Linux, iPhone, and Android.  There doesn’t seem to be much MonoGame-specific documentation, but the API is identical to XNA, and there is a ton of XNA documentation.  It doesn’t seem to have taken off yet though, with less than 10k posts on the forums.

XNA v4.0: We used this C# framework for Robofish, so this is our control group.  XNA is easy to use and extensively documented with tons of tutorials and example code around.  The biggest drawback is that it is for Windows only since it runs on the .NET framework.  As I mentioned earlier a bit, if XNA had built-in cross-platform support I think it would’ve been a no-brainer for us.

 

SFML v2.0 RC: The Simple and Fast Multimedia Library is a C++ framework that looks really promising.  I love the documentation and tutorials on the website.  The forums are active with posts numbering in the 10s of thousands.  I looked at the API, and it looked very simple to use.  I was really looking forward to trying this one out until I discovered that it doesn’t support sprite batching.  That is a deal-breaker for us, unfortunately.  Hopefully it will be supported in future releases!

SDL v2.0 with OpenGL v4.3: I considered just using C++ SDL and OpenGL.  I figured that it would take a lot longer to get going as I learned how to access the video card at a lower level.  I also thought that maybe it would still be worth it if the performance was there.  However after spending a few hours looking through documentation and tutorials on various websites I decided that I’m just not ready to develop a game engine from this low of a level yet.  I had a hard enough time just trying to get some sort of sprite batching going, which I don’t think I ever really accomplished.  I want to focus more on making our game and less on telling the video card exactly what it has to do.

Whew, that’s quite a list of frameworks to investigate!  Fortunately we were able to knock four out of the running before we even got started, which leaves us with five frameworks to benchmark.  I didn’t want to spend a ton of time writing throwaway benchmark code, but I did want to see how each framework handled sprite batching and how easy it was to write a simple demo in each framework.  I decided the goal of the demo was to draw as many sprites as possible to the screen while translating, rotating, and scaling them at the same time.  I defined “as many sprites as possible” to mean that the demo had to maintain a steady 60 FPS.

So I grabbed a 64×64 PNG we had lying around (actually an old blowfish sprite from Robofish) and got to coding.  Developing the benchmark took different amounts of time for each framework.  I started in XNA because I was most familiar with that framework, and it was probably the easiest.  In the benchmark I simply create an array of sprites, and then in the update loop I loop through each one and set its rotation, scale, and location.  Next, in the draw function, I loop through all the sprites again and use XNA’s SpriteBatch class to draw all of them to the screen.  XNA performs great, and I was able to draw about 38,000 sprites to the screen at 60 FPS.

Note that this screenshot is from the LibGDX benchmark.  I used FRAPS to measure the FPS on Windows, but I later added code to print the FPS for the benchmarks for Linux.

If you’re interested in the code I’ve included a download link for the benchmarks on Windows here.  Once I had the code working as I wanted it to in XNA I began porting it to each of the other frameworks.

Porting the benchmark to Love2D was really quick.  I think it may have been the easiest to use after XNA.  It started off with decent performance as well with lots of sprites on the screen; that is, until I started rotating, moving, and scaling those sprites.  I ended up only being able to draw about 6,000 sprites and maintain 60 FPS.  Disappointing, but I wasn’t too surprised considering that Lua is an interpreted language.

LibGDX was up next.  I had a bit of a rough time getting the development environment set up.  They have a download tool you can use to get the framework jar files and set up a new project in Eclipse, but I didn’t know about that at first and ended up wasting some time.  The scaling didn’t default to a nice texture filter like XNA, so I also spent some time looking into getting that working so it didn’t look so pixelated when sprites were scaled up.  I also spent time trying out their built in spritesheet tool, which they call a texture atlas.  One other thing that tripped me up was even after I used their tool the generated Eclipse project wouldn’t compile until I created some directories in the project directory manually.  So there was definitely a learning curve on this one.  However, I was astounded by the peformance I got: about 33,000 sprites at 60 FPS.

I spent a lot of time looking at all of the features that Cocos2D-X provides, and I thought it looked like a really nice framework.  I was starting to regret looking at it after I spent a few hours banging my head against my desk trying to get a simple benchmark going.  I will admit that it is probably just me and my inexperience with the Cocos2D API, but I thought it was a pain to get working.  The code for this benchmark is also different from the other benchmarks.  I tried using some of the built-in framework functions for scaling, rotating, and translating on a timer; maybe that had something to do with how long it took.  Performance was pretty good with 24,000 sprites at 60 FPS.

That just leaves us with MonoGame.  I didn’t have to change my code at all from the XNA framework since it directly supports the XNA API.  That was definitely nice.  It was relatively painless setting up a new Visual Studio project and getting my code to compile and link with MonoGame instead of XNA.  MonoGame delivered good performance with 30,000 sprites on the screen at 60 FPS.  One benefit to using MonoGame is that we wouldn’t even have to use MonoGame on the Windows platform.  We could stick with XNA for Windows and use MonoGame for everything else.

Endless paragraphs of text are fun and all, but here’s a graph to show Windows performance for the different frameworks I measured:

 

Well sweet, we’ve got our benchmarks rolling and we know which one was the easiest to use with the best performance… XNA!  Wait, this was only Windows.  We should at least try the same benchmark on a different platform.  What if all this cross-platform talk is just shenanigans?  Let’s make sure that these frameworks actually do what they say!  I don’t have a Mac, so I decided to install Ubuntu 12.04 on my second hard drive and try getting the benchmarks to run on there.

I’m familiar with Linux, but I’ve never done any gaming on it.  About all I knew I needed to do was make sure there were some decent drivers installed.  Ubuntu makes it pretty easy to install drivers for your video card if you know where to look.  Once I had the drivers installed I copied over my benchmarks and tried to get them to run.

I decided to set up LibGDX first.  I used their tool to download the framework, launched Eclipse, imported my project, and hit the run button.  It actually ran!  At about 5 or 10 FPS.  Wow, that doesn’t seem very cross-platform to me!  Something has to be wrong.  I decided to cut back the number of sprites.  After dropping the number of sprites to less than 1000 without seeing a noticeably performance increase I knew something was wrong.  Maybe I hadn’t installed my drivers correctly after all?  I’ll skip over describing a couple painful hours and instead inform you that disabling Compiz on Ubuntu can do wonders for your FPS when running games.  After I learned that trick I used it for every other benchmark I ran on Ubuntu (it was a necessity).  The great part is that I got about 40,000 sprites on the screen at 60 FPS; that’s right, even more than I had in Windows!  The even better part is that it required absolutely zero code changes to port it to Linux!

Well alright, these frameworks actually are cross-platform!  Let’s try some more!  I downloaded Love2D next, hoping to see a performance increase there as well.  Unfortunately I actually saw a decrease.  It was only able to do about 4,000 sprites at 60 FPS on Linux.  I’m not sure why.  Since the performance wasn’t that great on Windows I didn’t really spend much time looking into optimizing it.  This one also didn’t require any code changes!

Next I downloaded Cocos2D-X and fired up my benchmark using that framework!  Or rather, I tried to.  Instead I got compile errors.  Linux support seems to be of secondary importance for Cocos2D-X.  I was less than impressed when I had to spend a half hour or so tweaking the code to get it to compile.  Once I had it running the performance was about 20,000 sprites at 60 FPS, which was less than on Windows.

Last we have MonoGame.  I figured it would be pretty easy to get this one to work on Linux.  That was unfortunately not the case.  I probably spent the most time trying to get my benchmark running with MonoGame.  One problem was that it required some extra setup in the solution that wasn’t documented that well.  A bigger problem was that the content pipeline isn’t completely implemented yet for Linux.  What that means is that you are dependent on a Windows machine with XNA installed in order to generate your resource files (like a sprite texture) for the game.  The MonoGame website states that they are in the process of developing a content pipeline that isn’t dependent on XNA, so hopefully they get that going soon.  Once I got over that hurdle performance was about 29,000 sprites at 60 FPS, which is pretty similar to Windows.  The bad news here is that it did not look good.  This is a bit subjective, but the MonoGame benchmark was noticeably more “choppy”.  The movement, rotation, and scaling of the sprites was not “smooth” even though it was supposedly 60 FPS.  I’m not sure what was going on here.

Here are a couple more graphs for you:


This is the performance of the benchmarks on Linux.  This graph looks pretty similar to the first one except the numbers are a bit lower for most of the benchmarks.


This is the performance of the benchmarks in both Windows and Linux.  You can see that all the benchmarks performed slightly worse except for LibGDX, which actually improved!

So which one do you think we chose?  Which one would you choose?

Despite some of the initial trouble I had setting it up, we were really impressed with the performance of LibGDX.  It also couldn’t be any easier to port to Linux.  You may be surprised that we’re going with a Java-based framework since most AAA games are written in C++.  Well we’re an indie studio, and indie studios have experienced success using Java before.  Even so, the benchmarks don’t lie, LibGDX is a beast when it comes to performance so we’re not too worried about it.

Hopefully LibGDX will work out for us, but there are a few other things we are taking into consideration that could lead to problems down the road.  The benchmarks showed that LibGDX performs well on Windows and Linux when doing simple sprite batching.  However there is a lot more to a game than just drawing sprites.  What about audio?  Or collision detection?  Don’t forget particles and shaders!  There are a lot of features we haven’t tested in LibGDX that may or may not work well in both Windows and Linux.  Now that we’ve chosen a specific framework it’s probably a good time to test the more advanced features that we know we’ll use on both Windows and Linux and make sure everything works as we expect!

Aside from untested features to worry about we also have to consider the install process.  Whoever wants to play our game will have to have Java on their computer in order for it to run.  Should we bundle the JRE in with our game?  If we do bundle it with our game should we do it behind the scenes or should we require them to actually install it?  There is some precedent for this on Steam: lots of games come bundled with DirectX and the DirectX installer runs when the game is installing.  I’ve also seen games come bundled with XNA.  If we don’t bundle it in then we need to inform the user that they need to find a specific version of Java and install it before installing our game.   We will likely bundle the JRE in with the game somehow so the user doesn’t have to worry about it.

Hopefully these two articles help you decide on a framework for your game, and if anyone ports my code to any other framework I’d love to see your results!

How to Choose a Game Engine

// October 6th, 2012 // 2 Comments » // Robofish

Good news: Jaret and I have begun work on Sparkrift’s next game! Bad news: it’s going to be a while before you can play it. Good news: the game is going to be awesome! We’ve been using the past couple of months to start laying the foundation for our next game. Jaret is doing a lot of concept art, and I’ve spent a lot of time looking into game engines and graphics frameworks, which brings us to to the topic of this post.

There are a lot of game engines and graphics frameworks to choose from, so how can you pick the right one? Maybe you shouldn’t even pick one; why not just write your own from scratch? That way you’ll have complete control and can tune it to match your needs exactly! Only you can decide what’s right for you, and I won’t go into writing an engine versus writing a game in this post. Here I’m going to walk you through our experience with Robofish.

 

Our goal for Robofish was pretty simple: prove that we work together and make a game that runs on the Xbox 360. Short of paying thousands of dollars for an official Xbox 360 developer kit that leaves you with XNA. That was fine with me because I’m a fan of C#, and since Robofish was our first game I didn’t think I’d be able to code it in OpenGL or Direct3D directly without it taking an extremely long time. After looking into XNA I realized that it’s really just a framework. That was good, but I knew if we used a game engine we’d probably finish the game even sooner.

Cue Torque X 2D. For a while XNA had a partnership with Torque where they would provide XNA developers with free access to the Torque X 2D game engine binaries. So I checked it out, read through the tutorials, coded up a little test to see if I could be productive, and it looked good! In only a few days I had gone from never creating a game to being able to move a little box around the screen and “shoot” smaller boxes. This was going to be cake!

 

I showed Jaret my fantastic demo and he was sold! After working on Robofish for a few months, I became a little frustrated with the documentation though. The tutorials were pretty decent, but there were only half a dozen or so, and they didn’t cover all the things I wanted to do in Robofish. Worse though was the fact that engine API documentation was practically nonexistent. I had no idea what features were available to me through the Torque engine. Posting on the Torque forums would lead to a handful of responses from a few dedicated Torque users if you were lucky.

Well I really needed to know how to use this engine, so we decided to buy the source for the engine, which was an option at the time. Once I had the source I was able to divine some of the meanings of the API calls. The source was decently documented, so that was how I proceeded. It was slow, and I was very unfamiliar with game engine frameworks and 2D graphics in general. Luckily I found a few things that seemed to work for the graphics and stuck with them while working on things like weapons and fish AI which didn’t need much from the engine. Things were going fairly smoothly until I realized XNA 4 was coming out soon.

Hurray, an upgrade to XNA! That should bring better performance and lots of cool new features and bug fixes, right? Well sure, it did some of that but there was a problem. Microsoft stated that you wouldn’t be able to publish XNA games on Xbox 360 unless they were using XNA 4 after a certain date. Robofish was never going to be done by that date. That meant we needed to upgrade to XNA 4. Torque X 2D was based on XNA 3, and I knew it wasn’t going to compile if I linked it with XNA 4. Well crap, to the Torque forums! Bad news led to worse news as I discovered that Torque was discontinuing support for the Torque X 2D engine. Holy crap, this game that we’ve been working on for over a year isn’t ever going to make it to Xbox 360 now!

You know that wasn’t the case though, so how did we do it? Not alone, that’s for sure! Torque X 2D was ported to XNA 4 by some very dedicated and generous Torque X 2D users. They named the port Torque X 2D CEV, which stands for “Community Enhanced Version”. Without their dedication I don’t know what would’ve happened. I doubt I would’ve been able to port the engine to XNA 4 by myself. Unfortunately the problems with Torque didn’t end there.

 

Performance was the largest problem we ran into with Torque. Maybe this isn’t an actual Torque problem because I don’t think they advertised blazing speeds with their engine. As an amateur I just assumed that if they were making this engine available to develop games for Xbox 360 that it would perform well. I did not benchmark it, which was my own fault. If I had benchmarked it at the start I would’ve discovered two things: the number of sprites that can be drawn to the screen is severely limited and games deployed to the Xbox 360 encounter massive stuttering periodically.

I was, and still am, extremely angry about the sprite drawing performance with Torque X 2D. It turned out that we were limited to somewhere around 200 to 300 sprites on the screen at once if we wanted to stay close to 60 FPS. That may be more than enough for a lot of games; it is not nearly enough for a SHMUPS. The number of sprites shown on the screen can add up quickly. Robofish had enemy sprites, weapon projectile sprites, effect sprites, and clam sprites on the screen all at once. The largest number of sprites comes from the weapon projectiles.

Let’s say you have a weapon that shoots 8 projectiles at once. Each time the weapon fires that’s 8 new sprites to draw to the screen. Depending on the weapon rate of fire and the movement speed of the projectiles there could be up to or more than 200 projectiles, and their corresponding sprites, on the screen at once. That’s pretty much all of available sprites right there. So we had to be really careful about limiting exactly how many weapon projectile sprites could be on the screen at once. Don’t forget that some of the enemies shoot projectiles as well. Get a few of those on the screen and that’s dozens more projectiles right there as well.

 

We also wanted to add particle effects to the weapons, which we did have in there at one point. The weapons looked awesome with particle effects, and it killed us to take them out, but we just couldn’t maintain a playable framerate with them in there. So why couldn’t Torque X 2D handle very many sprites on the screen at once? After investigating the engine source code I discovered that they weren’t using any kind of sprite batching. This means that each sprite gets its own GPU draw call, which is extremely inefficient. The difference in the number of sprites that can be drawn one at a time versus batching them together is a couple orders of magnitude; it’s that big of a deal. So if we were limited to 200 sprites before, if we had been using sprite batching we could’ve potentially been drawing 20,000 sprites at once with similar performance.

The second major performance problem in the Torque X 2D engine was the stuttering we encountered whenever we ran Robofish on Xbox 360. Anyone who has ever developed a game for the Xbox 360 using XNA knows what’s coming next: the garbage collector. Other people have written on the Xbox 360 .NET garbage collector before, so you can read about it from them if you like. The short version is you have two options when it comes to garbage collections: don’t create any garbage so the garbage collector doesn’t have to run or limit the number of objects on the heap so that the garbage collector can run quickly.

Running Robofish on top of the Torque X 2D engine meant the second option wasn’t really available to us. There were already tens of thousands of objects in the heap, so whenever the garbage collector ran it took a few milliseconds, sometimes up to 40 or 50 milliseconds, which is around 3 frames. Not drawing anything to the screen for 3 frames or more would definitely cause noticeable stutter for us. So if a lot of garbage was being generated by the game the garbage collector would run to free up memory and cause Robofish to stutter quite frequently. It wasn’t fun to play in that state, so I began to dig into the engine to find out where this garbage was coming from.

 

After digging into the Torque X 2D engine for weeks and months I basically came to the conclusion that whoever developed the engine hadn’t tested it very well on the Xbox 360. Maybe not at all. Maybe it was still a work in progress; I don’t know. What I do know is that I ended up making dozens of fixes to the engine source itself to keep it from creating garbage. One of the biggest improvements was the object pooling system.

Since Robofish’s weapons shoot so many projectiles, a good way to save memory is reuse the projectiles after they’re off screen. Instead of creating a new projectile when the weapon is fired and destroying the projectile when it hits an enemy or goes off screen you can turn the projectile into a new projectile. Recycling! Great idea in practice, but it wasn’t implemented very well in the Torque X 2D engine, and it would still leak garbage during the recycling process.

Even after all the time I spent trying to improve the engine I never truly got rid of the garbage allocation. I just got it into a state that was good enough to release. I’d like to note that both of the performance issues were not caused by Torque X 2D alone. There were many things I personally did wrong that caused performance issues. However, after I had fixed those issues I still had to go into the engine to fix what was left before the game was playable.

The point of this post is that game engines can be a blessing and a curse. I’m not sure if Torque X 2D saved us work or delayed our game release after everything is said and done. I suspect, even with all the issues we had, that we still came out ahead by using the engine… barely. That’s probably due more to me being a complete newbie to game programming at the start of the project, so it would’ve been really rough to create an engine for Robofish from scratch.

After thinking about it a lot, there are many things we could’ve considered to prevent the issues we ran into:

  • Performance: We should’ve benchmarked the engine to find out if it would meet our needs. If we had done that at the beginning we would’ve caught most of the problems we ran into, and we could’ve avoided them.
  • Community: The size of the user base and how active they are makes a big difference. If no one is using the game engine it’s probably for a reason. Having a large number of users also means quicker answers to questions you post on the forums.
  • Documentation: If a game engine isn’t well documented it’s not a good sign. If you’re designing a game engine to be used easily you want to provide good docs to make it easy for the developer. I don’t want to use a game engine, or any library, that doesn’t have good docs; it’s just too much of a pain.
  • Source: When the docs don’t provide enough information you can always fall back on the source… unless it’s closed. If you have the source you can also implement bug fixes yourself if the engine developer is too slow to provide fixes. Although if the developer is slow to provide fixes that’s a bad sign in itself.
  • Maturity: If a game engine is brand new it’s probably still going to be changing, full of bugs, and not well documented. So just keep in mind that the newer something is, the less tested it will be and the more likely you’ll be doing the testing yourself.
  • Features: The whole point of using a game engine is to save time. Verify that the game engine provides the features you will need for your game and test them to see that they work well.

Keep these factors in mind when choosing a game engine, and I think you’ll avoid most of the problems we ran into with Robofish. One final note on Torque X 2D, though: I tried to specifically call out the fact that we used Torque X 2D because there are lots of other Torque engines out there for other platforms. I have no comments on other Torque engines because I haven’t used them.

Stay tuned for my follow up post where I post some benchmarks on current game frameworks to show how we determined what framework to use for our next game!

Reviews

// March 16th, 2012 // No Comments » // Robofish

Some very positive reviews are coming in!

OtakuDante’s review
Piki Geek’s review

It’s out!

// March 15th, 2012 // No Comments » // Robofish

Robofish is on Xbox Live marketplace right now, stop reading and go check it out!

No big deal, just a release date

// March 6th, 2012 // No Comments » // Robofish

Robofish is invading the Xbox Live Indie Games channel next Thursday, March 15! Prepare for awesomeness!

Creating Sprites for Robofish

// July 31st, 2011 // No Comments » // Robofish

Ice Blaster Sheet

Robofish is my first real game project and when I started working on it I knew that using sprite sheets was the preferred method for exporting graphics to the engine but I didn’t have a clue how to set them up efficiently.  Sprite sheets are basically large image files that tile out each graphic or animation’s frame so that the game engine can read the images in order according to the sheet and display them on the screen when called.  That sounds confusing, I am not even sure I understand what I just typed!  Simply, a game engine runs faster if it can see a bunch of graphics on one image file rather than thousands of individual images.  Jacob, my Brother and the programmer behind our games, and I knew this but our first attempt at sprite sheets wasn’t quite the most efficient.  Continue reading to see how first discovered how to make sprite sheets and how we had to modify our entire sprite sheet process about 60% through production, also pictures below!

Continue Reading

Robofish Hit Effects

// July 23rd, 2011 // No Comments » // Robofish

Check out the latest video displaying a few of the weapon hit effects available in the game!

Hit effects are special abilities that you can apply to projectiles when you create a weapon in the weapon factory.  Some of the effects include freezing, life leech, explosion, teleport, and many others.  Keep checking back to see footage on other hit effects!

The Evolution of Robofish

// July 21st, 2011 // No Comments » // Robofish

As the graphic designer on Robofish I thought it would be interesting to go over the different concepts of Robofish from the beginning of the project up until now where we have pretty much a solid idea of what Robofish is going to look like when we release the game. Robofish is my first indie game and I have learned a lot during the process, pretty much starting with zero knowledge on how to effectively make 2D graphics in a game environment.  I used Adobe Flash for the design and animation of Robofish.

Robofish Concept 1

Above you will see the very first few concepts of Robofish. Pretty rough around the edges! My main goal at the start of the project was to try and draw something humorous and interesting at the same time. I wasn’t taking our “hero” too seriously and didn’t want the player too either. After the initial concept I proceeded to create a more refined design with some shading and different colors and cleaner detail. I would say my favorite part of these concept designs is the jet-pack like thing on the right side of him. However the ugly “roundness” of the fin and the head had to go, and with a little help from my brother we slimmed him up and gave him a fresher look below:

Continue Reading