1500 Archers on a 28.8: Age of Empires Promo Network Programming - PDF document

1500 Archers on a 28.8: Age of Empires – Promo Network Programming in Age of Empires and Beyond J. Smed, T. Kaukoranta and H. Hakonen Gamasutra ( First appeared at the Game Developer’s Conference, 2001) http://www.microsoft.com/games/empires/multimedia.htm March 22, 2001 Age of Empires – Real Time Strategy Introduction • This page explains Age of Empires (AoE) 1 • Build • Combat and 2 multiplayer implementation. • Explore • Explains: – Design Architecture – Implementation – Lessons learned • Also for the “future” RTS by Ensemble – (Age of Mythology, AoM) Outline AoE: Multiplayer Design Goals • Support for 8 players • Introduction (done) • Smooth simulation over modem, Internet, • Implementation (next) • Lessons Learned LAN • Target platform: 16 MB P-90, 28.8 modem • Improvements for AoE 2 – (AoM is PX-450, 56.6 modem) • RTS3 – At 15 frames per second (one frame every • Summary 67 ms) • Use (existing) Genie engine – next 1

AoE in Early Stages (1 of 2) AoE in Early Stages (2 of 2) • Wanted: army on army, large supporting structure, • Game engine was running … (“1500 archers on a …”) – 2d, single-threaded game loop • Time to complete each simulation step varied: – Sprites in 256 colors – Reasonably stable – Render time changes with number of units • Large map, thousands of objects, trees could be – When scrolling chopped, animals ran around … – AI computation time varied with units or time • Breakdown: – As much as 200 ms (larger than a frame time!) – 30% graphic rendering • Bandwidth a critical resource: – 30% AI – Passing x,y coordinates, status, action, facing – 30% simulation • “Compelling” single-player game already damage … limit of 250 moving units at most – (MLC: 1 bytes each � 6 actions x 250 units x 15 – (ie- “Don’t ruin it!”) updates per second ~ 160 Kbps) Communication Turns Simultaneous Simulations • Separate communications turns from frame rendering • Schedule commands for later time • Each PC ran exact same simulation – Allows for some variance in network and turn – Synchronized game time processing • Turns typically 200 ms in length – Synchronized random number generators • Still – Send all commands entered that turn, but schedule them for 2 turns later – Internet latency from 20 to 1000 – Process any scheduled turns milliseconds – Variable time to process each step • Needed a more responsive approach The Need for Speed Control Speed Control • Each client calculates frame rate • Since all machines in – Since varies with game state, use moving average “lock step”, can only – Send with “Turn Done” message run as fast as slowest – Use to achieve “minimum” frame rate machine • Each client measures round-trip “ping” time – Process – Since varies with Internet traffic, use largest for communications, all players (MLC: assume moving average) render turn, send out • fps + rtt � 2-bytes total overhead new commands • After getting “Turn Done” messages • “Lag” if – Adjust target frame rate (based on local PC render – One machine slows rate) down and others wait – Adjust communication turn (based on ping-times + remote PC render rates) – Delayed or lost – Weighted, so only “laggy” during worst spikes Internet data • (Examples next) 2

Speed Control Transport Protocol - UDP • Unreliable, so each client handles command ordering, drop detection and re-sending 1) Typical communication turn – “When in doubt, assume it dropped” • Messages arriving from past turns are discarded • If out of order message received, request 2) High latency, normal machine a resend of supposedly “missing” messages – Note, if really out of order, will get duplicate so must account for • If ack is “late”, then assume lost so resend 3) High latency, slow machine Side Problems – Out of Synch Side Benefit - Cheat Prevention “In every project, there is one stubborn bug that goes all the way to the wire…” • Simultaneous simulations means games are – Microsoft product manager • Subtle, since small errors multiply identical • If there is a discrepancy, game stopped – Example – a deer slightly out of alignment, • Prevents cheaters from using hacked client causes villager to “miss” so no meat, causing • But there still could be cheating via different food amounts • Checksums (objects, pathing, targeting …), information exposure but always something – Wade through 50 MB of message traces – (MLC: different game states when commands are lost or are too late?) Lesson: Know Your User Outline • Each genre is different • Introduction (done) – RTS • Implementation (done) – FPS • Lessons Learned – Sports (next) • Improvements for AoE 2 – MMORPG • RTS3 – … 1) Know latency expectations • Summary 2) Prototype multiplayer aspects early 3) Watch for behavior that hurts multiplayer performance 3

Know Your User - Expectations Know Your User - Actions • Simulated latency with single-player engine • Users clicked once per 1.5 – 2 seconds – Look for: good, sluggish, jerky, horrible • During battle, spikes of 3-4 clicks per • For AoE second – 250 milliseconds (ms) not noticed • Many of the commands are repeats – 250-500 ms playable – 500+ ms noticeable – Turn is longer than command • Consistent slow (500 ms) better than – Unit takes several turns to process • Add “filter” to remove repeat commands “jerky” (80 ms – 500 ms) – Suggested picking conservative turn length – Make change to new turn length gradually Lesson: Educating the Developers Lesson: Metering is King • Get programmers to think about • Make performance meters human readable multiplayer (distributed systems!) and understood by testers • Multiple, independent processing – Need to educate testers – Command request might happen later (or not – Testers can notice differences, help at all) determine where problems are • Additional calls to random numbers can • Keep running all the time throw off synchronization – Low impact – Random sounds or animations on high-perf – Early development measurements may computers need to save and re-seed to keep change in later game random in-synch Misc Lessons Outline • Verify 3rd party code (AoE used Microsoft’s • Introduction DirectPlay ) (done) • Implementation – Is “guaranteed delivery” guaranteed? (done) – Does product have hidden bottlenecks? • Lessons Learned • Create simulations and stress tests (done) • Improvements for AoE 2 – Isolated connection flooding, simultaneous (next) connections, dropping of guaranteed packets… • RTS3 • Test with modems as early as possible • Summary – Hard to isolate network problems (ex: could be ISP) – Helps make sure not the networking part – Although tests harder (and not as fun), do as many modem tests as LAN tests 4

Improvements for Age of Empires 2 Outline – the Age of Kings • New multiplayer features • Introduction (done) • Implementation – Recorded games (done) • Lessons Learned – File transfer (custom maps) (done) • Improvements for AoE 2 – Statistics tracking (on “The Zone”) (done) • Recorded games helps in debugging • RTS3 (next) – Can replay exactly to spot problems – Overview – New features and tools • Summary RTS3 RTS3 – Beyond AoE • Forget DirectPlay � created own library • RTS3 is “codename” for next generation – Use in subsequent games Ensemble RTS (probably Age of Mythology) – Integrated with BANG! • Add 3-d capability (used BANG!) • Fully 3-d world meant frame rate maybe an • Multiplayer requirements issue – Again, complex maps, thousands of units, – Overall smoothness from frame rate Internet play impacted – More than 8 players (AoM allows 12) – Also more variation – Still modem, but 56.6k • Realized play-testing was iterative, so – May be firewalls and NAT boxes so peer-to- wanted multiplayer running earlier peer harder An OO Approach Peer-to-Peer Topology • Strengths – Reduces latency – No central point of failure (can continue game if one client leaves) • Weaknesses – More active connections (n) – Impossible to support some NATs (Make protocol specific parts as small as possible) 5

Net.lib (1 of 2) Net.lib (2 of 2) • Level 1: Socks • Level 3: Multiplayer – Fundamental C API, – Client, Session, Channel Berkeley sockets (ordered or unordered), Time • Level 2: Link Sync • Level 4: Game Communications – Transport layer services – Packet, Link, Listener, – RTS functionality (could Data stream, Network define others for different Address, Ping genres) New Tools and Features Summary • Improved synch • Peer-to-Peer for responsiveness – Geared towards rapid turn around time from out-of-synch bugs • Synchronous simulation for scalability – Compile-out extra synch debugging code • Compensation for heterogeneity in clients upon release • Console commands and config and variability in networking • Overall – Simple text “hooks” to game engine – In multiplayer, passed to other clients and – Multiplayer “feels” like single player executed there – Success! – Testing without writing additional code 6