[PD] regression testing WAS Re: [psql] object hand-holding

[IOhannes m zmoelnig]

Hans-Christoph Steiner wrote:
> On Dec 21, 2007, at 10:30 AM, IOhannes m zmölnig wrote:
> 
>> Mathieu Bouchard wrote:
>>> On Mon, 10 Dec 2007, Hans-Christoph Steiner wrote:
>>>> What we really need is regression testing, I think that would  
>>>> make it
>>>> easier to convince Miller to accept patches, and would be a great
>>>> tool for finding bugs.  Matju started that project, any word on that
>>>> Matju?
>>> No, I haven't done much progress. The problem with that thing is  
>>> that no-one wants to do it, not even me. It's hard to consider it  
>>> a priority as much as it should be.
>>
>> at least i am using "kind-of" regression tests in zexy and iemlib.
>> the "suite" is rather simplistic, but works well for my purposes.
>>
>> see zexy/tests for how it works
> 
> Any interest in making this a general purpose testing framework?  I  

have you had a look at it?

> would like to come up with a suite that is run every night  
> automatically, like the nightly builds.  It sounds very similar to  
> what I was thinking.
> 
> If we just come up with a standard set of inlets, outlets, and a  
> standard output value, 

well, i think i ahve documented this in the README.
since i am using the same framework on several projects (not just the 2 
mentioned), i would prefer to keep it rather fixed like it is.
writing tests is not that much fun for me. right now there are more than 
300 tests in iemmatrix...

> then it will be easy to write a script to run  
> thru them all and report on them.  Sounds like you might already have  
> that working to some degree.

have you had a look at it?
i am using my suites for >2years now.

> 
> I would like to make a set that tests for the strict aliasing bugs so  
> I can safely use gcc optimizations.  This means testing tilde objects.


while there are no explicit tests for ~-objects in any of my suites (i 
think), they are thought of.


> I think it might make sense to start a section in CVS for this called  
> 'tests', then we can start gathering tests there into subfolders  
> organized by library.  Another option would be a standard folder in  
> each library called 'tests' like you have.

why would you keep them apart?

> 
> That's my two bits, but we really should be doing a lot more of this.


i agree.

and despite of saying how great my regression-test-suite is, in reality 
it is not.
and i am using several different versions of it out there [1]

my advice:
- keep the tests as simple as possible, but not simpler
one should be able to write a test fairly easily; i think shortcuts are 
most likely not needed most of the times (e.g. iirc, matju introduced 
several shortcuts in his unity-test frameworks to test for 
mathematically important concepts (like assosiativity of operations); i 
think it is way simpler to just program 3 tests manually instead)

- as for results i have found it sufficient to have the following 
return-states: SUCCESS, FAIL, WAIT
i found it makes writing tests simpler if all of them implicitely fail 
if they do not respond with a return-state _immediately_ (this might 
make the FAIL-state unnecessary; but again it makes tests simpler if you 
can provide an explicit FAIL)
if the test takes some time (e.g. for signal-objects or timing objects), 
i return the WAIT state which halts the testing-framework until the 
currently tested object returns a value.

i usually use "1" for success, "0" for fail, "-1" for wait, the rest is 
fail too.

- write tests to test the framework

- allow fail-tests which have to return FAIL-state in order to success

- handle crash-tests separately

- allow "manual" tests (not "scriptable" (as in "as fast as possible"))

- i found that tests-patches should not have arguments (especially: 
default-arguments, like "$0"), as this sometimes makes testing of 
special features unneccessarily complicated.

- ...


fmga.sdr
IOhannes

[1] the most current incarnation of my framework is currently used in 
http://ftm.cvs.sourceforge.net/ftm/ftm/externals/pd/regression/