BUIP052: Set up dedicated Continuous Integration
Proposer: freetrader
Submitted on: 2017-04-15
Status: draft

Summary
=======

Get a paid-for dedicated Continuous Integration (CI) service to do 
daily / nightly builds which cover the full test suite.

This would be an addition to the existing free service provided by 
Travis.


Motivation
==========

Right now, BU is running on free instance of Travis CI [1], which is 
a great service for open source projects. However, this is limited 
in the build minutes, which means that BU cannot run the full suite 
of software tests (which can take up to a few hours to run through 
depending on hardware).

Instead, it runs a limited subset of quicker tests (a kind of "smoke 
testing").

Practically, individual developers are limited in the build 
platforms available to them, and do not always run the extended 
tests, which results in a lack of awareness of whether changes have 
broken some parts of the software on some supported platforms.

Indeed, it is difficult if not impossible to currently obtain an 
accurate picture of the overall project test status (all applicable 
tests on supported platforms). A dedicated CI service could run the 
full test suite once a day (testing 'dev' and 'release' branch heads 
for various build platforms).


Benefits
========

Contributing developers need information on how reliably a test is 
expected to perform - when you develop a feature and the test fails, 
you want to know whether the test was occasionally known to fail 
before you started, or whether it's likely due to changes you've 
made, etc.

Acts as a 'health check' for users who want to run BU. They can look 
at the test status to gauge whether they are satisfied with the 
software's health and prepared to take the risk of running it.

Helps the project obtain better quality by making sure they don't 
break the software accidentally (daily feedback on merged code 'dev' 
and 'release' branches).


Implementation
==============

This BUIP does not intend to prescribe a particular service or 
provider, but strongly recommends going with an existing, well known 
service with good reputation and public interface.

A 'Startup' Travis instance (described as 'best for small teams') 
costs $129 / month, i.e. ~$1548 / year, and allows 2 concurrent jobs 
and unlimited build minutes / repositories / collaborators.

A 'Small Business' Travis instance (described as 'best for growing 
teams') costs $249 / month, i.e. ~$2988 / year, and allows 5 
concurrent jobs and unlimited build minutes / repositories / 
collaborators.

$3K / year for reliable full-spectrum daily test status information 
accessible to all project members would be a good investment in the 
author's view.


Notes
=====

The existing free Travis service which we are using to test PRs and 
merge commits should remain in operation unchanged by this BUIP 
(this is a recommendation). The free service is useful as-is, a lot 
of effort has been invested and there is no major benefit to 
changing it / moving everything to another service.

The daily / nightly tests can also be run quite differently, 
requiring a different CI configuration. For example, for tests that 
fail, one could re-run them several times to gauge whether it's a 
random failure and what the probability of failure is. One could 
also set up repetitions of passing tests with different execution 
orders, various random seeds etc. There's a lot that can be done to 
get more mileage out of the existing tests.

The author is unaware of Bitcoin Core or other Satoshi-based clients 
publishing full test suite run data to the public. This is certainly 
not optimal, and a good chance for BU to improve upon current 
practice and provide a good example.


Optional Item 1: upgrade existing Travis for speedier builds
============================================================

As mentioned by @solex in the bitco.in forum thread [2], Travis has 
made an offer of upgrading the standard CI service that BU is 
currently, giving developers access to faster build times for the 
regular builds. This is offered at $2750 p.a.


Alternatives
============

Developers could set up dedicated test boxes at home / work and feed 
back test results into a common Git repository.

This would lack the nice interface etc., and would be more costly 
overall as people might have to acquire dedicated test hardware 
(although some of this could be obtained through sponsorship or 
donations).

One could make a separate BUIP for a project to aggregate this 
"ragtag" test data into something that's easily digestible for 
project members. I'm assuming this would end up costing more and not 
performing as well as using an experienced CI provider. The 
advantage is that it would be more decentralized.


Additional information
======================

[1] https://travis-ci.com/plans
[2] https://bitco.in/forum/threads/buip052-set-up-dedicated-continuous-integration.2034/