Serializing Shit

So I was looking at serialization yesterday. Here’s what I have found out.

What Is Serialization

Saving your classes to some format that you can rebuild them from. So, say for example in Rust you have a box filled with stuff and you want to save your game.. you need some way to save that information to disk. Or if you have an item and its health has changed, you need some way to send that data from the server to the client.

Because we’re talking about a game, we’re not talking XML or JSON. We’re talking binary.

The Lowest Level, Simplest Way

I don’t know that you can get any simpler than this in C# – I’d be surprised if you can.

2014-04-09_17-01-07

I think it’s safe to assume that you can’t really get any faster than this.

Reflection

Reflection is slow. We all know this.. right? So yep. Iterating fields, checking for attributes, and using GetValue( obj ) is about 4-5x slower than the above method. Which isn’t crazy slow (because above is quite fast).

There’s ways to make this faster. By caching the fields and only calling GetValue( obj ) I got it to about 2x slower.

I found some code that created a function on the fly to GetValue.. and then it was only about 1.5x slower. Which was acceptable.

At this point I decided that even though this worked, and the speed was acceptable.. the amount of code wasn’t. The fact that it didn’t handle compatibility properly, all that had to be custom coded. That was a turn off.

Google Proto Buffers

So then there’s Protocol Buffers. We use this c# flavour. Protocol buffers solve a lot of problems we’re likely going to come up against. They are future compatible, so we can add new fields to them and old saves will still be compatible.

So now the process is to create a proto file, which looks like this..

2014-04-10_12-38-43

Which, when we run our batch file.. generates a bunch of c# classes.. which allow us to make our new save/load to this:

2014-04-10_12-34-43

And.. they’re fast. Obviously not as fast as the low level mode but not far off – certainly faster than anything I could make happen with reflection.

I experimented with a few things that did make them faster. For example, using the ProtoBuffer itself to store the variables, rather than creating a builder every time. I couldn’t justify the code mess it made, and speed increases by doing this were minimal.

It’s not made obvious in the code, but the item returned by Save can be used to create a byte[] array, or a json string, or an xml string. This is another great reason to use Protobuffers.. because at any time you can call Save() and save that item to JSON.. to find out what’s going on.

Also – the same proto files can be used to generate code in c++ and PHP. So you can share the data between pretty much anything and it becomes loadable and usable instantly.

Conclusion

Use Proto buffers if you’re saving/loading/sending network messages.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

8 thoughts on “Serializing Shit

  1. The other option would be to use unsafe C# and just blit the objects to disk, which I *think* is possible. Just, y’know, an incredibly horrible idea.

    I’d personally go with Protobuf too, they’re pretty awesome. Getting language agnosticism is a really nice side-benefit too, and they’re damned fast. I know Valve uses Protobuf pretty extensively, the network protocol in Steam is all Protobuf, and I think CS:GO even uses it for the game’s network protocol now.

    1. Yep, blitting out to disk is probably a horrible idea. Also no good for realtime networking – you’ll probably end up with data bloat (a lot more data than you actually need, not to mention if you’re using Unity some/most of that data is going to be meaningless on another machine or next time you start up the game)
      I shall have to look into this whole protobuf thing. Don’t think I need it at the moment, but it could be very useful for later games…

  2. Minor thing, but I think the item container should be responsible for knowing which slot an item is in, not the item itself.

  3. Also, you might want to look into a library called Automapper… It lets you map your proto classes with your data models automatically (you can even perform logic in those mappings). We’ve found it to be a incredibly useful tool so you don’t have to muck around with your proto classes.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s