changeset 1132:f0a1b88367b0

coding_style: Looked into feasibility of forcing developers to test their code
author Olivier Delalleau <delallea@iro>
date Wed, 15 Sep 2010 14:02:45 -0400
parents d9550c27a192
children 9baa47482ccc
files doc/v2_planning/coding_style.txt
diffstat 1 files changed, 31 insertions(+), 0 deletions(-) [+]
line wrap: on
line diff
--- a/doc/v2_planning/coding_style.txt	Wed Sep 15 12:23:27 2010 -0400
+++ b/doc/v2_planning/coding_style.txt	Wed Sep 15 14:02:45 2010 -0400
@@ -269,6 +269,7 @@
 automatically warn the user when he is using untested stuff (and to remind
 ourselves we should add a test).
 Task: See feasibility. (OD)
+Result: See section 'Enforcing strict testing policy'.
 
    * VIM / Emacs plugins / config files
 To enforce good coding style automatically.
@@ -417,3 +418,33 @@
 here - the main goal would be for everyone to use the same inf / nan to make
 the code consistent).
 
+Enforcing strict testing policy
+-------------------------------
+
+The `coverage` third-party module provides a way to gather code coverage
+statistics in the test suite. `nosetests` has a plugin that can be activated
+with the --with-coverage option to use this module.
+It is possible to know which lines specifically lack coverage. However, we
+will probably want to post-process this data to do more than a simple report
+(which noone will care about). This could be done either by parsing nosetests'
+coverage output, or modifying its coverage plugin, or writing our own version
+of it. The main goal would be to identify who is responsible for writing lines
+that are not currently covered (using 'hg annotate'), in order to send email
+notifications.
+
+We should aim at 100% code coverage in tests. This is realistic because
+`coverage` offers ways to ignore coverage for lines we explicitely do not want
+to cover (typically debug code, or AssertionError / NotImplementedError that
+are not supposed to be triggered during normal usage).
+We may need to do some advanced processing though to e.g. collect results from
+multiple build bots, if for instance some bot is running tests without GPU
+support, and another one is taking care of the GPU tests.
+
+Code that should be tested but for which no test is currently written would
+also require some decorator / helper function that would trigger a warning at
+run-time (only once / execution). This could be enforced by adopting a
+different policy about lack-of-coverage notification emails, depending on
+whether or not the warning is present:
+- if there is no warning, daily email notification (ADD A WARNING!!!)
+- if there is a warning, weekly email notification (ADD A TEST!!!)
+