Automated testing for multiplayer Game AI Automated testing for - PowerPoint PPT Presentation

Automated testing for multiplayer Game AI Automated testing for multiplayer Game AI in Sea of Thieves in Sea of Thieves Robert Masella Rare - Microsoft Studios

About Me About Me • Senior Gameplay Engineer • Worked on: • Banjo Kazooie: Nuts and Bolts • Kinect Sports games • Sea of Thieves • Two years on AI

Rare Rare • Microsoft first party studio • 30+ years of games

Sea of Thieves Sea of Thieves

Sea of Thieves Sea of Thieves

What is Sea of Thieves? What is Sea of Thieves? • ‘Shared World Adventure Game’ • Two main AI threats: • Skeletons on islands • Sharks in the sea • Other AI types being worked on

Sea of Thieves Tech Sea of Thieves Tech • Uses Unreal Engine 4 • Dedicated servers • AI processing runs on server • AI uses UE4 behaviour trees • With many extended classes

This Talk This Talk • How we shipped weekly • AI behaving correctly on every build • Using automated testing and continuous delivery

Why use automated tests? Why use automated tests? • Game as a service means constant build confidence necessary • Reduce manual testing • Reduce crunch!

Testing AI Testing AI • Automated testing not widely used in game development • AI Unique challenges for testing AI • Multiplayer

Imagine this scenario Imagine this scenario • Designer changes a number in AI perception asset • Causes AI to forget players leaving Line of Sight correctly • Not spotted by manual test • Bug released to players

Consequences Consequences • Weeks later… • Players start to notice in build • More engineer time to track down • More testing time to verify fix • Every chance issue could reoccur

How to avoid How to avoid • Add an automated test • Run it regularly • Now bug would be caught before release • Ideally before it enters build

Testing at Rare Testing at Rare • Require testing with every check in • Sea of Thieves now has 10,000 tests • Test run remotely using TeamCity

Automated Testing Organisation Automated Testing Organisation • Run core set of tests before every check in • Run longer running tests periodically • Run all tests multiple times: • Different platforms • Different configurations (e.g. shipping, debug) • Different conditions (e.g. high latency, high packet variance)

Release Process Release Process • Fix gets to players in 30 hrs: 0 hrs 0 min Fix implementation completed 0 hrs 30 min Fix verified on our build farm 0 hrs 30 min Fix submitted 3 hrs 0 min Build built, deployed to test lab and testing begins 5 hrs 0 min Entire build verified 29hrs 0 min Build passed through certification, ready to be published to retail. 30hrs 0 min Build published to retail and available to players

Test Types Test Types • Unit • Integration • Behaviour Tree • Multiplayer Integration

Test Types Test Types • Favour unit tests over integration tests • 90% of tests are unit tests • Unit tests don’t cover interaction between big systems • And rarely test production assets • Ideal to have coverage by all test types

Testing Standards Testing Standards • Names follow Given_When_Then pattern • E.g. AIWithHearingSense_ReceivesNoiseEvent_AIAcknowledgesNoise • Each test tests one thing • Make sure you’ve seen the test fail as well as pass • Minimise boilerplate

Running Tests Running Tests • Run tests in Unreal Editor Automation window

Unit Tests Unit Tests • Run without creating or ticking world • Unit tests can have test fixture for boilerplate code #define IMPLEMENT_AIENTITY_TEST( TestName ) IMPLEMENT_UNIT_TEST( TestName, "AIEntity", AIEntityTestFixture ) IMPLEMENT_AIENTITY_TEST( UpdateTarget_OneEnemyEntitySeen_TargetSetToEntity ) { auto* AIEntity = SpawnTestAIEntity(); auto* EnemyEntity = SpawnEnemyEntity(); AIEntity->AddSeenEntity( EnemyEntity ); AIEntity->UpdateTarget(); TestEqual( AIEntity->GetTargetEntity(), EnemyEntity ); }

Integration Tests Integration Tests • Mostly used Unreal Blueprint system: • https://docs.unrealengine.com/latest/INT/Engine/Blueprints/

Integration Tests Integration Tests • Levels with limited game scenario • Most look for success criteria, or time out and fail

Line of Sight integration test Line of Sight integration test • SkeletonAI_WhenLosesLineOfSightToTarget_MovesToPositionToRegainLineOfSight

Line of Sight integration test Line of Sight integration test

Line of Sight integration test Line of Sight integration test • Failing version

Line of Sight integration test Line of Sight integration test

Line of Sight integration test Line of Sight integration test • Failing version • Assertion: Test Timed Out

Line of Sight integration test Line of Sight integration test • Passing version

Line of Sight integration test Line of Sight integration test

Line of Sight integration test Line of Sight integration test • Passing version

Behaviour Tree Testing Behaviour Tree Testing • In Unreal, behaviour trees are assets • Tested with integration tests. • Also required testing for new node types • Preferred testing with unit tests

Example Behaviour Tree Node Example Behaviour Tree Node • Node that triggers an input

Behaviour Tree Node Testing Behaviour Tree Node Testing • Add node to a minimal behaviour tree created in code • Created environment required for coded behaviour tree in fixture • Also created helper functions for adding nodes virtual void OnBeforeTest() override { BehaviorTree = CreateTreeRootWithSequence(); } UBTTask_TriggerInput* CreateTriggerInputTaskNodeAttachedToNode( UBTCompositeNode* ParentNode, UNotificationInputId NotificationId ) { auto* TestTask = NewObject< UBTTask_TriggerInput >(); TestTask->NotificationId = NotificationId; ParentNode->AddChild( TestTask ); return TestTask; }

Node Testing Node Testing • Test that checks that input is triggered and of correct type IMPLEMENT_UNIT_TEST_BEHAVIOUR_TREE_NODE( TriggerInputNode_NodeIsRun_CallsCorrectInputNotification ) { bool TestNotificationInput1Called = false; TestEntity->HandleNotificationCallback = [&]( UNotificationInputId NotificationInputId ) { if ( NotificationInputId == UTestNotificationInputId ) { TestNotificationInput1Called = true; } }; NotificationNode = CreateTriggerInputTaskNodeAttachedToNode( RootNode, UTestNotificationInputId ); RunTreeWithInitialTick(); TestTrue( TestNotificationInput1Called ); }

Node Testing Node Testing • Test that checks that input is triggered and of correct type

Example Behaviour Tree Node II Example Behaviour Tree Node II • Decorator node that compares entity’s health

Node Testing II Node Testing II • Test that checks that decorator fails if health is lower than expected IMPLEMENT_UNIT_TEST_BEHAVIOUR_TREE_NODE( CompareCurrentHealth_CompareGreaterThanValueHealthIsLess_DecoratorNodeFails ) { bool TaskWithDecoratorExecuted = false; bool LowerPriorityTaskExecuted = false; auto* Decorator = CreateCompareCurrentHealthDecorator( EFloatValueComparisonType::GreaterThan, 0.1f ); AddTestExecutionChildNodeWithDecorator( RootNode, [&]() { TaskWithDecoratorExecuted = true; }, Decorator ); AddTestExecutionChildNode( RootNode, [&]() { LowerPriorityTaskExecuted = true; } ); AIEntity->SetHealth( 0.05f ); RunTreeWithInitialTick(); TestFalse( TaskWithDecoratorExecuted ); TestTrue( LowerPriorityTaskExecuted ); }

Node Testing II Node Testing II • Test that checks that decorator fails if health is lower than expected

Multiplayer Integration Testing Multiplayer Integration Testing • Most AI integration tests require server only • Sometimes AI tests require client • Integration test framework supports tests over network

Multiplayer Integration Testing Multiplayer Integration Testing • Use Yield Nodes to pass execution between server and clients

‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test • SkeletonAI_Dies_DoesNotMoveDuringDeathOnClient_MP1

‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test

‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test • Failing version

‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test

‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test • Failing version • Assertion failed: ‘check velocity is low in dead state’

‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test • Passing version

‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test

‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test • Passing version

Testing drawbacks Testing drawbacks • Time to create tests • Tests on an evolving game • Intermittent integration tests

Testing benefits Testing benefits • Shipped over 100 times in 2 years • Very low bug count • Improved code • Manual testers focus on other things • No crunch for 4 years!