Continuous Integration/Testing and why you should assume every - PowerPoint PPT Presentation

W ESTFÄLISCHE W ILHELMS -U NIVERSITÄT M ÜNSTER Continuous Integration/Testing and why you should assume every change breaks your code living knowledge WWU Münster René Milk 4th November 2015

W ESTFÄLISCHE W ILHELMS -U NIVERSITÄT M ÜNSTER Continuous Integration/Testing 2 /16 Not talking about C ∞ functions Continuous ◮ completely automatic triggering (nightly, on branch updates, ...) ◮ automatic reporting ◮ simple visualization living knowledge ◮ wide accessibility WWU Münster ◮ metrics , , rene.milk@wwu.de

W ESTFÄLISCHE W ILHELMS -U NIVERSITÄT M ÜNSTER Continuous Integration/Testing 3 /16 No quadrature rules contained in this talk either Integration ◮ does your software stack still build after changes? ◮ compile, run, link, work as a dependency for other software? living knowledge ◮ how about on system X ? WWU Münster , , rene.milk@wwu.de

W ESTFÄLISCHE W ILHELMS -U NIVERSITÄT M ÜNSTER Continuous Integration/Testing 4 /16 and ofc no weak solutions for PDEs Testing ◮ automated, repeatable, deterministic, reliable ◮ Unit Testing ◮ System Testing living knowledge ◮ Performance Testing WWU Münster , , rene.milk@wwu.de

W ESTFÄLISCHE W ILHELMS -U NIVERSITÄT M ÜNSTER Continuous Integration/Testing 5 /16 assume your every change breaks everybody’s code ◮ saves time, no need to argue about possible effects of changes ◮ It’s better to err on the side of caution. You will break code from time to time. ◮ software is modular, codebases are huge, dependency graphs are complex ◮ distributed development: multiple people, branches, features/fixes, ... living knowledge ◮ http://whatever.scalzi.com/2010/06/16/the-failure-state-of-clever/ WWU Münster , , rene.milk@wwu.de

W ESTFÄLISCHE W ILHELMS -U NIVERSITÄT M ÜNSTER Continuous Integration/Testing 6 /16 Tools for CI: Travis ◮ free for FOSS ◮ integrates with GitHub repositories ◮ explicitly supports C ++ , python, ruby, but if your test/software runs on a standard-ish Ubuntu 12.04 you’re good living knowledge ◮ triggers on branch push, visual feedback in pull requests and badges ◮ very easy to create different configurations WWU Münster , , rene.milk@wwu.de

W ESTFÄLISCHE W ILHELMS -U NIVERSITÄT M ÜNSTER Continuous Integration/Testing 7 /16 Tools for CI: Travis language: python python: - "2.7" script: - DISPLAY=:99.0 py.test -k "${PYTEST_MARKER}" install: - python setup.py build_ext -i notifications: email: living knowledge on_success: change on_failure: change after_success: WWU Münster - coveralls branches: except: - gh-pages , , rene.milk@wwu.de

W ESTFÄLISCHE W ILHELMS -U NIVERSITÄT M ÜNSTER Continuous Integration/Testing 8 /16 Tools for CI: Travis sudo: false language: cpp matrix: include: - os: linux compiler: gcc addons: &gcc49 apt: sources: [’ubuntu-toolchain-r-test’] living knowledge packages: [’g++-4.9’, ’gcc-4.9’] env: CXX_COMPILER=g++-4.9 COMPILER=gcc-4.9 WWU Münster - os: linux compiler: gcc addons: *gcc49 env: CXX_COMPILER=g++-4.9 COMPILER=gcc-4.9 DELETE="dune-istl" , , rene.milk@wwu.de

W ESTFÄLISCHE W ILHELMS -U NIVERSITÄT M ÜNSTER Continuous Integration/Testing 9 /16 Tools for CI: Travis living knowledge WWU Münster , , rene.milk@wwu.de

W ESTFÄLISCHE W ILHELMS -U NIVERSITÄT M ÜNSTER Continuous Integration/Testing 10 /16 Tools for CI: Travis living knowledge WWU Münster , , rene.milk@wwu.de

W ESTFÄLISCHE W ILHELMS -U NIVERSITÄT M ÜNSTER Continuous Integration/Testing 11 /16 Tools for CI: Buildbot ◮ Python framework to build CI systems, not a ready made solution ◮ Master/Slave setup ◮ good for testing complex stacks on diverse architectures living knowledge ◮ much more freedom than on Travis, at a much higher maintenance and setup cost WWU Münster , , rene.milk@wwu.de

W ESTFÄLISCHE W ILHELMS -U NIVERSITÄT M ÜNSTER Continuous Integration/Testing 12 /16 Tools for CI: Buildbot - Example ◮ setup to deploy new masters with only 5 LOC ◮ buildbot code for that to work is 2300 LOC ◮ git repository tracks individual DUNE-modules´ repositories -> dynamic build steps -> make test, result log living knowledge ◮ tied into redmine WWU Münster ◮ email on buildstatus change, filter by user in change , , rene.milk@wwu.de

W ESTFÄLISCHE W ILHELMS -U NIVERSITÄT M ÜNSTER Continuous Integration/Testing 13 /16 Tools for CI: Buildbot - Example (master config) from common import config, passwords, genconfig config.CURRENT_DB = passwords.dune_db_url import redmine.db.connection redmine.db.connection.setup(config.CURRENT_DB) BuildmasterConfig = genconfig.genconfig( living knowledge "http://users.../dune-stuff-demos.git", WWU Münster project="dune-stuff", base_port=11020, use_cmake=True) , , rene.milk@wwu.de

W ESTFÄLISCHE W ILHELMS -U NIVERSITÄT M ÜNSTER Continuous Integration/Testing 14 /16 Tools for CI: Buildbot - Example (buildbot.opts) [clang_3.5] file = config.opts/clang-3.5 cc = clang-3.5 [gcc_5.0] file = config.opts/gcc-snapshot cc = gcc-snapshot [variants] living knowledge # name = semicolon seperated list of directory names WWU Münster # to delete prior to buildbot config no_fem = dune-fem minimal = dune-grid;dune-localfunctions;dune-fem; dune-typetree;dune-pdelab;dune-istl , , rene.milk@wwu.de

W ESTFÄLISCHE W ILHELMS -U NIVERSITÄT M ÜNSTER Continuous Integration/Testing 15 /16 Buildbot living knowledge WWU Münster , , rene.milk@wwu.de

W ESTFÄLISCHE W ILHELMS -U NIVERSITÄT M ÜNSTER Continuous Integration/Testing 16 /16 Testing: What? utopic Every line of code is unit tested, with function level granularity. System tests cover all possible software configurations and inputs. Performance testing is run on a wide variety of benchmark problems, in a restricted, possibly virtualized, environment. All of that on a very diverse set of computer architectures, operating systems, support library versions. living knowledge Not viable in the real world. Configuration space is just too big. Compute time, and WWU Münster available hardware is limited. Test cycles need to be short to not impede development. , , rene.milk@wwu.de

Continuous Integration/Testing 17 /16 Testing: What? realistic Every non-trivial piece of code is unit tested, with algorithm level granularity. System tests cover the most common configuration options. Benchmark runs are recorded. , , rene.milk@wwu.de

Continuous Integration/Testing 18 /16 Testing: What? minimal The code is run for a benchmark problem with known/analytic solution. , , rene.milk@wwu.de

Continuous Integration/Testing 19 /16 Testing: How? ◮ Use a test harness: google-test, Boost.Test, py.test Simplifies selective test running, result gathering, behavior expectation ◮ Writing a DUNE module? Use dune-testtools for simple test parameterization ◮ record algorithm results, error norms and compare in system tests ◮ Make sure exceptions get raised on invalid data input ◮ integrate into build system (ie make test ) , , rene.milk@wwu.de

Continuous Integration/Testing 20 /16 Testing: How? - Gtest Example typedef testing::Types<YaspGrid<1>, YaspGrid<4> Grids; template <class T> struct CornerRangeTest : public ::testing::Test { CornerRangeTest() : grid_prv(0., 1., level) {} void check() { for (auto it : grid_prv.grid().leafGridView()) check_range(it, cornerRange(it->geometry())); for (auto v : DSC::valueRange(T::dimensionworld)) EXPECT_GE(v, 0); } }; TYPED_TEST_CASE(CornerRangeTest, Grids); TYPED_TEST(CornerRangeTest, Misc) { this->check(); } , , rene.milk@wwu.de

Continuous Integration/Testing 21 /16 Testing: What tests are allowed to fail NONE except those where the platform requirements cannot be met , , rene.milk@wwu.de

Continuous Integration/Testing 22 /16 Testing: What else is there? ◮ Using cmake? Make ctest submit test results to cdash , , rene.milk@wwu.de

Continuous Integration/Testing 23 /16 Thank you for your attention. Slides available at http://work.milk.pm , , rene.milk@wwu.de