The Rust backend

After Rust took off we spent a week improving the back-end. Previously, deploying new versions was slow and bug ridden. It was compiled on someone’s computer (usually Pat’s). The whole thing was an ordeal – and a lot of the time Pat was the only one that could actually do it.

So, improvements were needed. Here’s what happens when we change something now.

Something Is Committed

Someone commits something to SVN. This can be code, or a model/texture/material/sound/etc change.

A build is triggered

Jenkins polls the SVN every minute. If a change is detected it tells the Windows build server to download the changes and build.

We have different jobs for the server and client. This means that the server and client can be built at the same time – halving the time it takes. Building was taking 40 minutes before on Pat’s PC. On our windows build server it’s taking less than 10.

A lot of this time is spent constructing bundles. We don’t want the player to have to download 300mb each time. So we create 60+ bundles – and try to keep the size under 5mb. We have found that if the size is under 5mb the browser will cache them – like it would an image – so the client doesn’t have to download them. These bundles are created and a manifest file is created – to signify which bundles are needed, and which order they should be loaded.

When it builds it commits the binaries to a different SVN.

Upload Client

Another Jenkins job. When it detects changes on the client SVN, it runs a php script. This script loads the manifest file and turns the renames the bundles to be crc hashes. Then it uploads the files to Amazon S3, so we can serve them nice and fast. We use CRC hashes because sometimes the bundles don’t change – so the client can keep using that bundle. But if it does change we want to be sure as shit that the client is getting the updated version. This was a fault we had previously – and causes a whole bunch of problems.

Once the bundles are uploaded, the PHP script sends the manifest file to our playrust server – and playrust creates a new version.

Testing New Version

Admins can test the new version immediately on the website. They have this box visible under the webplayer.

It shows the current ‘live’ version (green flag) and clicking on one of the versions allows them to switch to that version.

Make It Live

Once that’s done we can make it live in the admin section on the website.

The Future

Hopefully this whole system will mostly become redundant. It’s our hope to get on Steam eventually – and what I’m doing here pretty much re-creates the steampipe system. We don’t have a system for updating servers yet – we just use SVN – which has worked OK so far.. but probably won’t work so great once we let other people host servers.

With the Steam stuff – I think we’re going to go through the greenlight system. This is for two reasons. First of all I’d feel kinda like I was stepping on all the other people on greenlight if we went around it, and second of all – if it doesn’t get greenlit then it shouldn’t be on Steam in the first place.

Alpha Papa

The demand for alpha keys has been immense. Totally totally immense. We want to start selling access as soon as possible.. but we obviously can’t open that up for everyone or we will be swamped. So we will probably be putting a limit on it, so only so many keys a day will be available.. Hopefully that limit will increase over time as we optimize and make the servers support more people, get more regional servers of our own online, and start supporting third party servers.

77 thoughts on “The Rust backend

  1. I’m interested to hear why you preferred Subversion to Git for a project like this – I realise that Steampipe does a large amount of versioning but it’s still not really a replacement for proper management (IMO anyway)

  2. GIT doesn’t have broad support. I know you can get GIT support blah blah – but it doesn’t have the same support SVN has.

    We’re also uploading huge binary files to SVN, it checks out to 2GB. That’s not great for GIT.

    The real answer is.. why wouldn’t we?

    1. Garry you have done a great job developing games, i love gmod and i hope to buy rust when it comes out or in alpha/beta (if there is a beta). Keep up the good work Garry!!

    2. GIT is more widely used at this point than SVN is so it has greater support imo. Even Microsoft are starting to use GIT and as a result are hosting code on github now. GIT also works fine using large binary files, you are best using submodules for this so you don’t end up with a huge bloated mess (http://git-scm.com/docs/git-submodule). Using submodules you will end up with less bloat and a much faster SCM system than SVN could ever manage.

      Given you have already set up the SVN there is no point changing it now though.

    3. Hello garrys im nervous just writing to you because your a game founder but i would like to know if i can have a beta key please i would be very pleased

    4. GIT is awful. Support on windows is still a joke outside of commercial project management. Anything can work with svn and the advantages newer versioning systems have don’t really make a difference on one man to tiny team projects, they’d only add complications unless everyone was already versed with their use.
      After my god years I kept using svn till I was forever to use mercurial. HG is OK once I got used to it, tortoisehg v1.1 is easy enough to transition to from tortoisesvn. V2 of it is junk though, the annotation tool is worthless. Mercurial seems like a decent step up from svn if you find yourself spending more time resolving merges than actual coding.

      Jinkens, I really should switch my build servers over to that same day…

      Sup Garry, its your old annoyance TAD2020 :3

      1. Except there is native git support in visual studio now, as well as tortoisegit that works exactly like the other things you mentioned. There is also a github application that works well too as well as gitextensions.

      2. I have used all of those.

        tortoisegit has lackings and some issues. I’ve had it cause serious damage to remote git repo when it pushed to completely different branches than what was being worked with or specified to push to. Its disorganized, feels like a random assortment of tools thrown together. That all said, its the one I do still use.

        Githubs app, not sure how to go about this one. For one it just would not work at all, the uninstaller didn’t work, it broke all git on the couple comps i put it on, and worst of all, it uses a strange ms application framework system (I forget what’s its called) that the dev of the github app tolds me he did not how the install or uninstall process worked. Took a week to get it off and have an install of mysysgit actually work again.

        I can’t stand using MSVS as an IDE. Its even worse to use if you’re targeting more than just windows. I only use it for building and debugging. Also note that MSVS is a commercial product and the license of it is stupidly high.

        For a short period last year I was getting back into gmod addons again, but I rage quit after all this shitty problems that git caused.

        PS, this last update to the android keyboard on 4.2 is killing me, key the right work and on space it autocorrect s it to junk.

    5. Is it possible to even get a beta/alpha key anymore D: i’l have been looking everywhere! no 1 is selling them anymore/giving D:

    6. ive been looking and your new game named rust and it realy looks good and id love to get a chance to try it out so i would love it if i could get a key thanks garry love g mod love you m8 keep up the great work on the game

  3. I don’t think it would be stepping on indies if you went around it, Valve themselves have said it is a mess. As you have an established relationship with Valve you should probably use it.

    Also wouldn’t running a Greenlight campaign divert attention away from other indie devs that may need the system to get their game through.

    Just my opinion though, I’m obviously not an expert on this but I have been following many developers issues with greenlight.

    1. I have to agree with you, judging by the immense response it has received before it has even hit Greenlight, it definitely has the hype and capacity to be on Steam. I think putting it on Greenlight would do more harm rather than helping the indie developers.

    2. Out of curiosity, where did Valve say Greenlight is a mess? I think recently (past two years or so) Valve has been making ALL OF STEAM a huge mess. But I never thought I’d see them admit it.

  4. I really liked reading this blog post, in order to keep the servers unswamped you could put a login limit, like maximum 1000 players online and the rest will be put in a queue. Since it’s still Alpha they should not complain.

    Also I don’t think 3rd party servers will be a good idea since all the fun is coming from that we all are put in 1 server and PVP and play.

    Edit: Key please :3

  5. So obviously this game is in early development but how long have you been working on the game and how much money have you put toward the game? Playing your previous game it seems you may have a handful of money on this game it looks to have really good potential as i have seen in game play you are combining the key factors that make popular games good although i am yet to have access i will definitely be buying this game with such a creator like you as the brains of the operation.

  6. I don’t understand, people who have now the Alpha Keys will keep it in the future of the game or all the people will need to buy it ?

    1. Most likely all the accounts will be wiped and you will have to purchase the game through steam or their website.

      1. Garry said himself on twitter that all alpha users will be upgraded to the full version (for free)

  7. I see some real good ideas , I hope everything will turn out good .Selling the keys in waves seems like a good idea . Going to be crazy trying to get one though . Im still hoping on getting a key right now :D .
    Keep up the good work , and good luck with rust !

  8. Garry, can you elaborate on “once we let other people host servers.” Is the plan to let the public host servers or do you mean, people as in, your team?

    Also on this point, will developers be able to customize the server to their liking similar to how we’re able to do with GMod?

    Thank you!

    1. Garry have said in the past that he wants to make it kinda like how bf3 does it, license out the server software. This is because it will all have a globalized save, stats, etc. Which wouldn’t work if people could modify the server binaries.

  9. Do you host rust servers on AWS instances? What platform are you planning to expand to when creating more global servers to play on? Or… are you going to let it turn to DayZ where server owners are abusive and spawn shit for their own benefit…

    If Facepunch could handle the servers and inter-connectivity it’d be a lot better of a play experience.

    Maybe release a separate stand-alone server for people to mess around on but keep everything per-server on that end.

  10. Is SVN really the best technology for storing binaries? At my company we have a strict “no binaries in SVn, EVER” policy because they inflate repository to ridiculous sizes.

    Also, you don’t have to let Jenkins poll SVN every minute, you can simply set a up a post-commit hook on your SVN server and let that trigger the build. It’s faster AND doesn’t cause unnecessary load on your servers.
    https://wiki.jenkins-ci.org/display/JENKINS/Subversion+Plugin (scroll down to post-commit-hook)

    1. Robert is right, the correct way to do this is to use a post-commit hook not to poll the SVN constantly. Using CRC hashes for the file names so they match/don’t match when uploaded seems a very hacky way of doing that part. You should do a hash of the file and then compare hashes and only upload if the hash has changed, this means you can keep meaningful files names instead of just having a clusterfuck of hash named files on your server. Other than that your automated build/distribution system is pretty good nice work.

  11. It doesn’t matter what or when he sells something, people are going to buy it in droves simply because of his one hit wonder (100% success rate in the past).

      1. What I’m saying is that something similar to what happened when the second left 4 dead version got released could probably happen. People expected the second version to break new grounds like the first version did and got their hopes slammed but bought it without knowing what they were even buying.

        It’s just an asset you have and what I wrote could be interpreted harshly but not certainly not meant to be, at least not towards Rust but rather the irrational expectations that people could have on it based on your previous commercial work.

        By the way, could you answer the question I posted in the office thread about the comfort of sitting for long hours in the skruvsta chairs you have from ikea? Is that comfortable for office/computer work or does it make you miss other chairs with back and lumbar support?

  12. Garry you legend… I have been tryin to build such a game in unity for several years now… Painfully slow process with the minimal amount of knowledge I have. Still fun though. I cant wait to see a beta release of Rust. Minecraft meets DayZ FTW

    1. wait…you planed to make a game like that too? Whats the name of it bro?

      I tryed to code games myself and its hard as fuck , but if you already made a 3D game (even if its not working well) I need to give you my respect sir…Nice one :)

  13. Hi Garry I read your twitter replied to Will Rust not only be on windows, but Mac OS X and Linux?
    and you said OSX yeah
    Please could you considered it for linux? If not please could you tell us some reasons? Thanks for your time :)

      1. Ah i see so the game engine is unity right? But isn’t that webplayer and I thought Garry want to get it in Steam? Or will it be a standalone game in future?

      2. Unity is an engine, and it has a browser plugin so that games using Unity can be played in your browser.

  14. Hey Gary my name is will. I’ve done some research and I’m pretty sure you and your teams reasons for limiting the keys out there is due to the ability of Jenkins to perform well at the moment with to great of a capacity. If tht being so, is there a way u could release an offline version of Rust for people to test it out them selves. My only concern with this would be people trying to claim it as theirs or how we would get back to u with bugs and errors (face punch maybe?) maybe if u consider this you might implement a way for offline users to play via LAN. If u read this plz get back to me if u have any thoughts I’m very interested in this project. To record errors each launcher could have a console or terminal tht monitored the activities and had a button where one could paste the login to a website such as pastebin.com My account can be found on facepunch under coldairkiller as my username. Thank you for your many great years or developing and I hope for many more for you.
    -Will
    This took a great deal of time and though to find the right words. Hope u read it.

    1. No u don’t quit asking u whiny little fracker. Didn’t u read the forums on facepunch.com. No asking for handouts
      -Will
      Ps why the frack is this a comment on Gary’s blog giving us a heads up at what’s happening. No where did if ever say giveaway…

    1. He sayd the following :

      (I will not post the name of that person)
      Will rust be playable with an intel graphics card eventually

      Garry Newman ‏@garrynewman 9 Jul
      Should be!

  15. Being a developer myself I wouldn’t invest in an alpha version, but I can understand why people of the gaming mentality would. Paying for an alpha release is nothing that would’ve ever happened 10-15 years ago, but in this age it looks like people would rather put their dollars in products with potential rather than proven results.

    The game its self does look like it’s in good shape though. I’m a frequent viewer of several Twitch channels streaming Rust. Props to your creativity with both the game and marketing.

  16. hi can I have a beta key please I got one but it said I cant have it because it was too long :( so a new one would be lovely Thanks

  17. Great work on the game! I’ve only seen videos and stuff but it actually looks great, cant wait till the game or keys are available

  18. I think that this is the type of game that a lot of people have been waiting on for quite some time. I love the fps, crafting and landscape modification aspects and hope that you expand upon them greatly over time.

    Rust could potentially be one of the greatest FPS / MMO games ever ( aside form AC1 of course ).

    I once dreamed of making a game of some sort. I’ve read countless books including but not limited to many of Andre Lamothe’s books. I even accepted a position with Wolfpack Studios when they premiered ShadowBane in Austin, TX only days before Ubisoft bought them out. Needless to say I never got the job. Blessing in disguise, I suppose.

    People don’t realize the sacrifices and hardship involved. It’s not all fun and games, hehe. I envy game developers, and truly appreciate the hard work that goes into their games.

    It takes dedication, countless hours, resources, and patience. I applaud your effort, vision, and hard work to bring us a flexible world to explore and spend countless hours in.

    In closing, thank you for Rust. May your game become a great success.

  19. Just wanted to say I’ve been banned on face punch for telling people it was my birthday and wanted a few people to wish me a happy one, a ban for this? Seriously? Bit over the top moderators on there…

  20. Garry i love Gmod its the only game i really play on steam i saw it at a frineds house and bought it for my self i love your work and think its great how interactive you are with the players and the game. Im haveing a problem i keep getting error sings for a lot of the things how do i updae if thats the prob??

  21. Hi Garry great job so far on Rust ! I am having difficulties locating where to make propositions as id like to help out as I can. So yeah here it goes :
    1) Crafting Levels, sure you can drop BP (and its easy but kinda kills the life of the game), instead have some bp unlocked by crafting levels.
    2) Craftable Compass (it shoes where your friends are but as a simple dot on the compass so that it doesnt get too unrealistic but still useful) Maybe it has limited range of 100m or so ? your the boss
    3) Missions (this is to give an objective to the game and give it with 1) a longer game lifespan. Missions could be obtained as various ways : message in a bottle (maybe if water and fishing is implemented), Sos message carved on random trees,
    4) Perhaps in the manner of Anarchy Online, dungeons that can either be discovered at random open locations that close when the dungeon has been completed (to avoid Player killers to camp static dungeon entrances), some of those dungeons could be mission related. Entering a dungeon would mean loading a different environment (cave, secret lab, prison etc….). Check out Anarchy Online because it was pretty awesome !
    5) Someway has to be found to limit the size (rather the amount) of buildings one player can create at a time, specially when one player is forced to create 20 fortress because his previous home was raided. perhaps it can be limited to a number of house components… based on the craft lvl, or the number of players gathered together to build (say i can only craft one floor, my friend crafts the second floor). Finally a building could be destroyed by killing its core (bf2142 style), it would look like a fireplace. The fireplace is responsible for controlling indoor lights.
    5bis) implement indoor lights, candles with animal fat, very slow burning, put inside the fireplace. or straight wood.
    6) Regarding combo lock doors, if you put door down you’re its admin you give a code that friends type in once and from then on they don’t need to input it every time unless you (the door admin press E for a long time and select “change door code”). Or pressing E creates a consumable called Key that you throw to your friend and they consume it. Pressing E on a door can either Create new key or Change lock.
    7) Buyable (yeah you need to earn money on this project for your team to keep working on it) skins (realistic skins not funky sh***) color taints for clothes, houses etc…

    Good luck and thanks for this awesome game base.

  22. Hi garry, Rust is great … realy !!! we are a french linux community who want to play on a private server and it’s hard to have some … so if you have 1 minutes … after you ;)

    SYA and realy ggod game … it’s hard , it’s fun and it’s cool …

    kill or not to kill that’s the surviving garry’s mod :)

  23. Hi Garry, I know you guys are away for the Steam Dev Days but I read some interesting things for the Zombie controversy / replacement. As I mentioned in a previous post on this thread, the game requires higher level objectives (i think caves/dungeons could be interesting in the long term).

    Regarding Zombies, each radtown could have its own occupying faction :
    – a military group
    – animals (wolf pack perhaps)
    – primitive tribe

    Killing one of these factions gives you point in the other factions ex : killing military gives points for animal faction, killing primitive tribe members gives military group appreciation and killing animals gives primitive tribes respect.

    Each faction has different combat aspects skills and stats, military group is slow (armored) but strong at a distance. Animals are super fast and very high damage, perhaps they come two by two, primitive tribe is a mix “in between” of the two previous, average speed, average armor and average damage.

    When you reach certain faction levels you are able to learn competencies (which are craft objects) :

    Military can teach you Sentry-Gun, hard to build but super lethal (like the robots in Portal 2) they can really protect your base. maybe even wall upgrades to really protect your base.

    Animals can teach you an object called perhaps “colar” which allows you to tame the wildlife. Having a wolf or a bear with you allows you to attack environment or players from a distance or track an enemy across a distance. You can bring it in your house and it stays in to protect you when you log out. (so it gives good offense advantages, but not as good as a defense as the sentry when it comes to protecting your house). maybe you could even control it in first or 3rd person and run around fast speed with it to explore your surroundings (this would make your body exposed to environment and players). The death of the animal forces you to retam an animal to make sure nobody spams you.

    Primitive tribe teaches you camouflage and traps. Camouflage allows you to lower your heartbeat(in reality you are hungry not as fast so you can camp long hours) and appear as perhaps a rock or maybe for now just being invisible (but impossible to move until you attack) Only bow can be used when in camouflage and breaking camouflage has a delay of a few minutes after you broke it. Traps are low tech traps (like vietnam war kind of stuff) and don’t require too much resources but limited in term of how many you can place in an area (perhaps an algorithm : the more you place traps close together, the further away you can then place new traps. like another region or something).

    On the longterm factions can teach you more and more stuff craft different armor, ex Kevlar, wolf armor (makes you run a bit faster, and makes animals unaware of you), ritual armor (medium armor but fast life regeneration or perhaps immune to bleeding)

    Perhaps it is through the factions that you can get new crafts and missions to access loadable dungeons (like Anarchy Online, missions have different kind of objectives, kill a named NPC, find a named NPC and escort him out to your bed…because it may be easier to do a trigger at your bed…houses could be anybodies lol) or retrieve an object layed on the ground. Each missions has stuff to kill to put challenges and different loading environments like prison, cave, canibal/bloody horror themed lair, advanced research lab etc…

    You could learn up to two factions out of the three and the more you kill a faction the harder it is to get back to a respectable level with them. Each faction crafts and items give you a different playstyle in the world of rust.

    Faction Houses : as you learn different traps/defense/ability objects from faction, you could learn building types, say the tribals teach you to make tree houses, military teaches you metal houses for extra defense, animals teach you to do a simple weed hut that you can move with you (the crates you put in stay in the hut and it all unfolds wherever you put it) or perhaps its a rock house with lots of animals that pop around to protect it…. perhaps you start building a small rock Igloo, and then place somewhere around a wolf dent which if you are attacked or someone is near your house will randomly send wolves chasing the invader’s arse. It would be awesome for a friend and me to play different factions, have different role in the gathering of resources and the defense of our kingdom because or difference in abilities through faction crafts.

    Anyways I just think that at this stage, the game is getting boring once you know how to do explosives, please don’t allow people to craft it because it kills the game and makes building houses useless. We can’t really justify glitches (like puting a crate on a spike wall) to be player creativity because it is a bit unrealistic and not fair (30 wood for a create – 200 for a large spike wall). Or make houses harder to build but close to impossible to destroy unless you get explosive from air drop. We want to have one base that we defend till the end, reconstructible if necessary. The way the game is : as soon as someone knows where your house is and raided it you have to move elsewhere or build s***loads of doors.

    Hope you can contact me id be happy to skype if you wanted to discuss further in detail objectives and end game possibilities.

    Have a great trip.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s