Running Unit Test on Serverless Infrastructure Adinata Thayib @dieend
source: xkcd.com
85% less
Running Unit Test on Serverless Infrastructure Adinata Thayib @dieend
About Pocket Gems ● Mobile Game Company ○ War Dragons & Episode ● 10 years old (and hiring!) ● > 4K tests
Frustration
What is the problem? Running full unit tests suite is slow Is it worth solving? ● Productivity ● Deployment Speed
How to make test suite complete faster? 1. Have fewer unit tests
How to make test suite complete faster? 1. Have fewer unit tests 2. Divide & Conquer a.k.a Parallelize the unit tests execution
How do we parallelize test execution? 1. Run it locally with multiple CPUs 2. Run it in multiple machine 3. Run it in serverless infrastructure
Running locally with multiple CPUs Nosetests Multiprocess plugin: ● Collect tests into a Queue object ● Configure the child process ● Start process which poll from the test Queue
Running locally with multiple CPUs Collect:
Running locally with multiple CPUs Collect: The plugin can't know (unless you tell it) if a context fixture can be called many times concurrently (is re-entrant), or if it can be shared among tests running in different processes. Therefore, if a context has fixtures, the default behavior is to dispatch the entire suite to a worker as a unit.
Running with Multiple Machine No out of the box python solution. Ruby Alternative (Knapsack Pro - for Ruby and JS). Python WIP
Running with Multiple Machine
Running with Multiple Machine
Designing the solution Constraint: ● We won't have engineer resources to manage the worker machine full time Opportunity: ● We use GAE: all the tests don't have external fixtures ● We use GAE: 100% python code and aren't using many binary dependencies Proposed Solution ● Use AWS Lambda to remove the necessity on managing worker machine
Running with serverless infrastructure
Running with serverless infrastructure
Running with serverless infrastructure nosetest plugin with custom selector
Running with serverless infrastructure Running Test... thread.join() Parse output
What we learned
What we learned 1. AWS Lambda Source File Limits 2. Concurrent Executions 3. Network Bandwidth 4. Longest running unit tests is the weakest link 5. Binary Dependencies 6. No re-entrant fixture support
After 2 years...
~21min faster
> 1000 executions/month
> 350hr engineering hours are saved each month
Monthly Impact Assuming engineering hour cost Minute saved each 21 mins execution $50/month (Bay Area): Monthly Execution 1000 execution Hours 350 hr Monthly Saved ~$17,500 Engineering Cost
~$120 monthly cost
> $17K Saving per months
Assuming engineering hour cost Monthly Saved ~$17,500 Engineering Cost $50/month (Bay Area): Monthly Cost ~$(120) Total ~$17,380 Initial Investment: Engineering Cost to Build ~$8,000 4 engineering weeks Break Even Point < 1 month
>70 engineers adopted it
Frustration Others might have the same frustration with you. You might solve everyone's problem.
Q&A Channel Discussion: #talk-unit-test-serverless Send me a tweet: @dieend
Channel Discussion: #talk-unit-test-serverless Send me a tweet: @dieend Thank you
Recommend
More recommend
