This is about two weeks of searching for the suitable way to implement multi-player games on the internet. I discuss different approaches with their strengths and weaknesses.
(This is a revived old article from NOMone’s old website that only exists on the Wayback Machine now. In fact, it actually belonged to our older website which exists nowhere now!)
Plain Sockets
The very first thing I thought of was “Sockets”. Somehow I’ll get the users to know the addresses of each other and let them do the game management on their own. This will require only a trivial server that establishes the connection between the end points. Sockets are fast and have very small overhead. Two things did stand in the way of such implementation … NAT (Network Address Translation) and Firewalls.
Users connecting to the internet using wifi will have to setup the virtual server (port forwarding) on their routers before they can connect using sockets. They might also need to turn off any firewall software they are running on their machines or add an exception for your application. This is too much to ask from a casual user. The ideal case would be a one click configuration!
That’s not all. Not all users are using their own routers. Some ISPs don’t provide their customers with real IP addresses. Instead, they put them on a local network and use NATing to differentiate the traffic of different users. That simply means that many users are going to share the same IP address and none of them has administrator access to the router so they can setup port forwarding. Luckily, there are solutions.
The first solution is using UPnP (Universal Plug and Play). It allows you to configure the port forwarding programatically without needing user interference. I’m not going to dive into details of it, but here are some links:
http://en.wikipedia.org/wiki/Universal_Plug_and_Play
http://www.codeproject.com/Articles/13285/Using-UPnP-for-Programmatic-Port-Forwardings-and-N
The second solution is using UDP hole punching. Check this out:
http://en.wikipedia.org/wiki/UDP_hole_punching
I didn’t use any of these techniques though. I moved on and made some more research.
PHP Sockets Server
Sometimes plain sockets and peer to peer communication is simply not enough. They may be perfect for point to point communication or simply two-player games, but when the number of communicating peers increase, they start to become less useful. Imagine having 100 users playing the same game, let it be a fast paced game that need to transfer the state of every player to every player 10 times per second. Using the peer to peer approach, each player will have to transfer its state to 99 other players 10 times every second. This can cost a lot of bandwidth, stuff that users (specially mobile ones) may not be able to provide. In this case, you may need a centralized point of communication, a server that has large bandwidth and is dedicated to transferring the whole state (not individual players’ states). The server would listen to all the updates, combine them and create a consistent global state and then publish it to all the players. Voila! We have reduced the clients’ outgoing traffic by 99%.
Now to the real work. We need a server. There are paid services and free shared webhosting services. By all means go for the paid ones! At the time of writing this article, I was totally new to this server stuff, so I just used a free account at:
These people are amazing. They provide a good service for free. I used 000webhosts to host nomone.com. Unfortunately, you can’t run your native sockets application on the managed shared server, you can only run PHP. That seemed ok to me, for a PHP sockets API exists and they claim that they support it (and they do. The catch is coming soon :D). The problem appeared when I considered deployment options. Even if your server application is written in PHP and the sockets API is supported, that doesn’t mean that you are good to go. You can’t run your PHP server script! Using the free account, you don’t have SSH access (Secure Shell, a way to connect to a remote server and issue unix commands), so you can’t just run your script. You may try to run it using an HTTP request, but the scripts have a timeout. They are automatically closed after a few minutes. Some people thought of creating a cron job to automatically start the server after every timeout, then an administrator on their forum said that this violates the TOS (Terms Of Service) and thus your account could be terminated (note that this is the case with most shared servers, even if you are on a paid plan).
Ok, so does moving to a paid plan with SSH (even on another service provide) solve the issues? No. There is another limitation, the CPU. Most shared webhosts force a limit on the CPU usage. For instance, Just Host mentions that if your server scripts ran for more than 6 minutes every hour, they’ll suspend your process. They call it 10% CPU thrashing policy.
So, unless you are using a VPS or a dedicated server, PHP sockets is not the way to go.
HTTP Requests
Now that I can’t do it, it’s time to deny needing it :D. I wasn’t planning to do real-time games anyway :P. All I was thinking of was some board/turn based games. An HTTP requests approach should work.
It’s very simple, I wrote two PHP scripts, one to write data at the server and one to read it. The procedure was simple.
– Clients should poll the server using the read script to find if they received any data.
– Clients use the write script to write data for the other clients.
Using this approach, I made a simple chat program that supports any number of concurrent users where every user polls the server once every second for new messages. Guess what, it worked perfectly… for a few minutes!
Before I get to why it stopped working, I have to write down how I implemented the client or I would forget really soon. It was an Android application. There are two normal approaches to send an HTTP request on Android. Using HttpClient (which is blocking, so you may want to wrap it in an AsyncTask) or using URL and HttpURLConnection. It seems that Google is in favor of using the URL method for any device which has Gingerbread or higher. They say it’s lighter and faster. One more thing to note. My free webhost has the habit of attaching three lines of javascript to the response of every HTTP request I issue. They do this to track statistics of my website. The client program takes care of these three lines and gets rid of them. Normally you wouldn’t have to do this.
So, what happens after a few minutes of awesomeness? The server stops responding completely. At first I though I might have broken something, but whenever I restarted my router and tried it worked again for a few more minutes! So what was happening?
After a bit of searching, I found that 000webhosts limits the number of requests that any specific IP can issue to your website to 2000 requests per day, after which the IP gets banned for 24 hours (note that even if you didn’t reach the requests limit, you do get a “Server busy” message occasionally). This doesn’t affect other users, only the user who exceeded the limit. This ruined everything. I contacted them and they said that this limit doesn’t exist in the paid plans. So I decided to get a paid plan, which they claim to have SSH access as well.
Warning: (many years old rant ahead! Things might have changed now.)
Ok so I went on and started registering for the paid service, which according to their website, costs $4.84 per month. And guess what :D, you have to subscribe for 3 years prepaid in order to get this price. If you want a single month, that costs $7.84, and to make it worse, you don’t get to use SSH on this plan! To get SSH, you have to subscribe for “Gold” plan, which costs $10.84 per month.
….
I began to think negatively :D. This HTTP requests technique is sub-optimal for my requirements :D. It incurs large overhead for the HTTP header, can’t send less than 256 bytes at a time, requires a complex PHP request to run at the server side with every read or write and will get the clients and the server occupied most of the time handling the data available to read polling.
$7.84 monthly to serve HTTP requests on a shared server and being subject for suspension or termination whenever they think you are overloading their server? $7.84 to get this sub-optimal solution? No thanks! Lets face it. The shared webhosting planes were meant for personal and small business websites. They are perfect for hosting your limited traffic blog, forum or WordPress website, but they weren’t meant to handle large masses of traffic. I can raise the bar a little and get a more suitable solution.
VPS (Virtual Private Server)
A virtual private server is a virtual machine hosted at some powerful physical machine (hopefully). The term virtual means that you don’t get a real server, but you get to act as if you really had one. This can have several advantages over dedicated servers. First, being virtual means that your server can be easily backup-ed and transferred to another physical machine at any given time. It also means that you are not alone on the server, you have many other virtual machines running next to you. This is good, for this way you only pay for a tiny fraction of the price of renting the server, yet, if the other machines are idle you can harness the full powers of the CPU to serve your own process. If they are not idle, at least you can rest assured that you get an equal share of the CPU whenever you need it. Virtual machines are also great because you can scale so easily. You can upgrade your server at any time with minimum effort. You don’t have to have somebody physically install extra memory or more hard-disks.
VPSs are great also because most of the time you get root access. That is, you get to play the role of the system administrator, installing the software of your choice and running it as you see appropriate. You should never get the hosting company screaming due to overload, or terminating your processes (but they can charge you for extra traffic beyond your plan limits, which is usually measured in terabytes). You get to utilize your server share the way you see fit without complaints. And the best part is, you get to implement your sockets server in C/C++ instead of php 😉
Now about which service provider to use. I’ve used Linode and DigitalOcean. Both are very good. If you are a loyal Windows user that knows nothing about Linux, you may want to try managing a VPS for free before you actually rent one. This is very easy:
– Install a ubuntu linux on a virtual machine (using VirtualBox) or on a physical machine.
– Connect the machine to your local network (If you are on VirtualBox, you can change the network adapter mode of the machine to “Bridged” to attach it to the physical network. You can also use “Host only” adapter, this way the virtual machine will only be accessible from the host machine).
– Get your server IP. Go to shell (press Ctrl+Alt+F1. You can get back to GUI by pressing Ctrl+Alt+F7) and type: ifconfig
– From shell, type: sudo apt-get install openssh-server
– Follow the tutorials on Linode. You can find them here.
Thus you can implement your server and test it before you actually buy anything. One of the things that you should pay attention to is, if you are going to host a PHP page on your server that is going to be accessed by masses of people (large concurrent traffic), you should make sure that no deadlocks occur. The Apache server can service a limited number of requests concurrently in an efficient way. If more clients connect to the server when it has already reached its limit, they are put into a pending state until they can be served. This shouldn’t take more than a few milliseconds though, but can result into your entire server hanging up if a deadlock occurs. A deadlock is when the maximum number of request is in action and these requests won’t terminate because they are trying to issue other requests that can’t be run (remember that the maximum requests was already reached). This results in the server waiting forever (or till timeout or till pending buffers are full) trying to finish requests that will simply never get finished. Solve this issue by setting time-outs for requests, either globally or in your scripts. Also consider using Nginx webserver, which is event based instead of process based, hence overcoming this problem automatically.
From here, lets get into the technical details of implementing your sockets server…
TCP vs UDP
You have to make up your mind which protocol you are going to use for your sockets connections. TCP is reliable, which means that your packets should never get lost and should always be delivered in order, but in order to do so it has to incur a little overhead. UDP is unreliable, meaning that your data may or may not arrive and if they do arrive, they may not be in the correct order. It’s important to notice that, packet losses are very rare, they almost never occur, but you can never be sure. If a single packet is very important to your application, meaning that it would affect the future of the game (or make a player state inconsistent with the others) and you chose to use UDP, then you have to handle this situation carefully. UDP is better suited to games where the clients constantly share their entire state, thus if a packet is dropped it wouldn’t affect the future of game play and the clients should remain consistent.
It’s also important to note that since packets getting lost is a rare situation, the TCP protocol doesn’t get to practice it’s overhead too often. Another point worth noting is that modern network infrastructure can optimize TCP traffic, so the overhead is even lower. Finally, if you need reliability on UDP, you’ll have to implement it on your own in the software. This could be a hassle. While if you choose TCP, reliability is achieved in hardware layers.
Here is a heated debate about which is more suitable to what. To me, I prefer to test both and see what better suits my specific application. For now, my board games can be very well built on top of TCP, so I choose TCP.
This tutorial shows you how to implement sockets on Linux. If you want to build sockets on Windows, you should look for Winsocks.
Minimizing forks
Sockets are blocking by default (i.e: socket function calls don’t return until the event they are waiting for takes place). The easiest way to implement sockets server is to listen until a client wants to connect. When one does, fork another process to handle that client from now on. This have the advantage of a more readable code and one that assumes that the entire world is just one client. It has the disadvantage of consuming your resources very quickly. Forking a new process isn’t lightweight, it costs CPU and memory. A better approach could be turning sockets into non-blocking mode (it’s possible) and having one thread handle all clients. This solution might not be the best one as well, since it doesn’t benefit from the multi-core nature of most modern servers. The best approach in my opinion would be a central process and a limited number of forks (maybe number of CPU cores * 4). The central process should accept connections and do a bit of load balancing and assign the client to the fork of the least number of active clients. This way we get to use the CPUs to their fullest extent and at the same time we are not forking too much.
Here are two tutorials of how to implement non-blocking sockets:
http://www.lowtek.com/sockets/select.html
http://public.dhe.ibm.com/systems/power/docs/systemi/v5r3/en_US/rzab6mst.pdf#page=149
Read the first one first. It has better documentation and will get you to understand the sockets API. Then read the second one, it has better code which you can build upon.
Final notes:
– The file descriptor set (fd_set) can hold a maximum number of sockets equal to FD_SETSIZE (default is 64?). You can increase that limit, see: select().
– The send() function will block if the sending buffers are full. Make sure to check if a socket can be written to before trying to write. One way of doing so is by providing an fd_set containing the sockets you intend to write to – to the select method as the “writefds” (the 3rd parameter).
Other techniques – The Cloud
Cloud services are just like normal web-hosting services, but on a cloud :D. A cloud is large set of interconnected servers that appear to your server program as a single server. Clouds have the advantage of being very reliable and very scalable. If a server happens to die, your server program can still be run from other servers. In fact, you don’t know how many servers are hosting your application at the same time. Clouds are very scalable because all the resources are dynamic in nature. CPU, memory and storage are all handled in a transparent manner so that you can adjust any of them to exactly what you need. Cloud services can also reduce the latency of your requests, for the server machines could be scattered anywhere in the world. Finally, on clouds you pay for what you get. Cloud hosting companies will charge you for the CPU cycles, for the memory and for every megabyte of storage accessed, not at a fixed rate.
But .. you don’t get to be the root user ;). You’ll have to use technologies supported by the cloud. Three major companies provide cloud hosting:
– Google. The Google Apps Engine. They will host your server for free up to a certain usage limit (quite a good one though, you may never need to buy resources). The bad thing is, you’ll have to use Google tools. I HATE google tools. They are buggy and frustrating and will cause you hours of frustration every now and then. With Google tools, you can never be sure if the problem is in your code or in their tools. I declare Google to be the worst option when it comes to developer tools.
– Amazon AWS (Amazon Web Services). There are two flavors of web-hosting at Amazon that I’m going to discuss, Amazon S3 and Amazon EC2. The S3 services are web-storage services. They host your data (not your applications) very reliably for a very little cost. Many people choose to have their storage hosted on S3, while their server applications hosted somewhere else. It works very well and saves a lot of money. The other service is the EC2 (Elastic Compute Cloud), which can host your server applications. You can sign up for Amazon EC2 service for free for the first year, where you get the free-tier service package. After the first year, you’ll have to pay something like a minimum charge in advance for resources you may use. Amazon tools are know to be better than Google tools 😀
– Microsoft Windows Azure. I don’t know much about this, but I saw their advertisement mentioning 90 days for free. I believe Microsoft developer tools to be the best of All.
Thank you very much for reading that far. I hope I saved you some time searching for answers 🙂