Mercurial > pylearn
changeset 1132:f0a1b88367b0
coding_style: Looked into feasibility of forcing developers to test their code
author | Olivier Delalleau <delallea@iro> |
---|---|
date | Wed, 15 Sep 2010 14:02:45 -0400 |
parents | d9550c27a192 |
children | 9baa47482ccc |
files | doc/v2_planning/coding_style.txt |
diffstat | 1 files changed, 31 insertions(+), 0 deletions(-) [+] |
line wrap: on
line diff
--- a/doc/v2_planning/coding_style.txt Wed Sep 15 12:23:27 2010 -0400 +++ b/doc/v2_planning/coding_style.txt Wed Sep 15 14:02:45 2010 -0400 @@ -269,6 +269,7 @@ automatically warn the user when he is using untested stuff (and to remind ourselves we should add a test). Task: See feasibility. (OD) +Result: See section 'Enforcing strict testing policy'. * VIM / Emacs plugins / config files To enforce good coding style automatically. @@ -417,3 +418,33 @@ here - the main goal would be for everyone to use the same inf / nan to make the code consistent). +Enforcing strict testing policy +------------------------------- + +The `coverage` third-party module provides a way to gather code coverage +statistics in the test suite. `nosetests` has a plugin that can be activated +with the --with-coverage option to use this module. +It is possible to know which lines specifically lack coverage. However, we +will probably want to post-process this data to do more than a simple report +(which noone will care about). This could be done either by parsing nosetests' +coverage output, or modifying its coverage plugin, or writing our own version +of it. The main goal would be to identify who is responsible for writing lines +that are not currently covered (using 'hg annotate'), in order to send email +notifications. + +We should aim at 100% code coverage in tests. This is realistic because +`coverage` offers ways to ignore coverage for lines we explicitely do not want +to cover (typically debug code, or AssertionError / NotImplementedError that +are not supposed to be triggered during normal usage). +We may need to do some advanced processing though to e.g. collect results from +multiple build bots, if for instance some bot is running tests without GPU +support, and another one is taking care of the GPU tests. + +Code that should be tested but for which no test is currently written would +also require some decorator / helper function that would trigger a warning at +run-time (only once / execution). This could be enforced by adopting a +different policy about lack-of-coverage notification emails, depending on +whether or not the warning is present: +- if there is no warning, daily email notification (ADD A WARNING!!!) +- if there is a warning, weekly email notification (ADD A TEST!!!) +