Hi All,

I am noticing a lot of tests and development where we're testing
locally with CSD's.  It would be great that while we're doing our
development and debugging of bugs to keep adding more tests to unit
tests in the tests folder so that we keep building up more and more
examples to use for regression testing.

I am thinking though, maybe something that would make it easier is if
we modify the testing script to not require manually adding them to
the python script but just have it walk the tests folder for csd's and
run them.  We could modify the testing so that the CSD itself could
have in the first few lines some info about expected results, whether
a test is disabled by default, etc.  Also, we could add a csd testing
target to CMake so that we could do "make testcsds".

I think to it'd be nice to add API tests with python scripts, and
possible also start using CTest
(http://www.cmake.org/cmake/help/ctest-2-8-docs.html).  I don't have
experience with CTest however, so if anyone has any experience and
could comment on usefulness that'd be great.

One other nice thing, with more tests, it'd add some more
self-documentation, as developers unfamiliar with a feature could look
at the tests to see what is expected.

Thoughts?

Thanks!
steven

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Csound-devel mailing list
Csound-devel@lists.sourceforge.net

Hi Steven,

I'm all for unit tests!

I've used CTest, and it works quite well. But for Csound where most of
the results are not in memory, but in external files, it might become
cumbersome, and a python script might be better suited.

Maybe files could automatically be added if they are not already
there, but I think adding test rules to the csd file itself might not
be the best idea as we would have to define a whole new syntax,
parsing, etc for it.

I think it's just a matter of bugging everyone a bit to do add the
tests until it becomes a habit. I had to do this for a while with the
manual pages... :)

Cheers,
Andrés

On Mon, Apr 30, 2012 at 12:26 AM, Steven Yi  wrote:
> Hi All,
>
> I am noticing a lot of tests and development where we're testing
> locally with CSD's.  It would be great that while we're doing our
> development and debugging of bugs to keep adding more tests to unit
> tests in the tests folder so that we keep building up more and more
> examples to use for regression testing.
>
> I am thinking though, maybe something that would make it easier is if
> we modify the testing script to not require manually adding them to
> the python script but just have it walk the tests folder for csd's and
> run them.  We could modify the testing so that the CSD itself could
> have in the first few lines some info about expected results, whether
> a test is disabled by default, etc.  Also, we could add a csd testing
> target to CMake so that we could do "make testcsds".
>
> I think to it'd be nice to add API tests with python scripts, and
> possible also start using CTest
> (http://www.cmake.org/cmake/help/ctest-2-8-docs.html).  I don't have
> experience with CTest however, so if anyone has any experience and
> could comment on usefulness that'd be great.
>
> One other nice thing, with more tests, it'd add some more
> self-documentation, as developers unfamiliar with a feature could look
> at the tests to see what is expected.
>
> Thoughts?
>
> Thanks!
> steven
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Csound-devel mailing list
> Csound-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/csound-devel

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Csound-devel mailing list
Csound-devel@lists.sourceforge.net
https://lists.sourceforge

Are you going to incorporate the old tests suite into your python script? 
It used to check md5 hash of the audio file output as well as not crashing


> Hi Steven,
>
> I'm all for unit tests!
>
> I've used CTest, and it works quite well. But for Csound where most of
> the results are not in memory, but in external files, it might become
> cumbersome, and a python script might be better suited.
>
> Maybe files could automatically be added if they are not already
> there, but I think adding test rules to the csd file itself might not
> be the best idea as we would have to define a whole new syntax,
> parsing, etc for it.
>
> I think it's just a matter of bugging everyone a bit to do add the
> tests until it becomes a habit. I had to do this for a while with the
> manual pages... :)
>
> Cheers,
> Andrés
>
> On Mon, Apr 30, 2012 at 12:26 AM, Steven Yi  wrote:
>> Hi All,
>>
>> I am noticing a lot of tests and development where we're testing
>> locally with CSD's.  It would be great that while we're doing our
>> development and debugging of bugs to keep adding more tests to unit
>> tests in the tests folder so that we keep building up more and more
>> examples to use for regression testing.
>>
>> I am thinking though, maybe something that would make it easier is if
>> we modify the testing script to not require manually adding them to
>> the python script but just have it walk the tests folder for csd's and
>> run them.  We could modify the testing so that the CSD itself could
>> have in the first few lines some info about expected results, whether
>> a test is disabled by default, etc.  Also, we could add a csd testing
>> target to CMake so that we could do "make testcsds".
>>
>> I think to it'd be nice to add API tests with python scripts, and
>> possible also start using CTest
>> (http://www.cmake.org/cmake/help/ctest-2-8-docs.html).  I don't have
>> experience with CTest however, so if anyone has any experience and
>> could comment on usefulness that'd be great.
>>
>> One other nice thing, with more tests, it'd add some more
>> self-documentation, as developers unfamiliar with a feature could look
>> at the tests to see what is expected.
>>
>> Thoughts?
>>
>> Thanks!
>> steven
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond.
>> Discussions
>> will include endpoint security, mobile security and the latest in
>> malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Csound-devel mailing list
>> Csound-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/csound-devel
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Csound-devel mailing list
> Csound-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/csound-devel
>



------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Csound-devel mailing list
Csound-devel@lists.sourceforge.net

Hi John,

I'm not thinking of changing anything for now... I assumed the tests
were already a python script, but if they're not, that's fine. My
comment was more towards CTests, which might be more complicated for
our case than using some kind of script.

Cheers,
Andrés

On Mon, Apr 30, 2012 at 9:22 AM,   wrote:
> Are you going to incorporate the old tests suite into your python script?
> It used to check md5 hash of the audio file output as well as not crashing
>
>
>> Hi Steven,
>>
>> I'm all for unit tests!
>>
>> I've used CTest, and it works quite well. But for Csound where most of
>> the results are not in memory, but in external files, it might become
>> cumbersome, and a python script might be better suited.
>>
>> Maybe files could automatically be added if they are not already
>> there, but I think adding test rules to the csd file itself might not
>> be the best idea as we would have to define a whole new syntax,
>> parsing, etc for it.
>>
>> I think it's just a matter of bugging everyone a bit to do add the
>> tests until it becomes a habit. I had to do this for a while with the
>> manual pages... :)
>>
>> Cheers,
>> Andrés
>>
>> On Mon, Apr 30, 2012 at 12:26 AM, Steven Yi  wrote:
>>> Hi All,
>>>
>>> I am noticing a lot of tests and development where we're testing
>>> locally with CSD's.  It would be great that while we're doing our
>>> development and debugging of bugs to keep adding more tests to unit
>>> tests in the tests folder so that we keep building up more and more
>>> examples to use for regression testing.
>>>
>>> I am thinking though, maybe something that would make it easier is if
>>> we modify the testing script to not require manually adding them to
>>> the python script but just have it walk the tests folder for csd's and
>>> run them.  We could modify the testing so that the CSD itself could
>>> have in the first few lines some info about expected results, whether
>>> a test is disabled by default, etc.  Also, we could add a csd testing
>>> target to CMake so that we could do "make testcsds".
>>>
>>> I think to it'd be nice to add API tests with python scripts, and
>>> possible also start using CTest
>>> (http://www.cmake.org/cmake/help/ctest-2-8-docs.html).  I don't have
>>> experience with CTest however, so if anyone has any experience and
>>> could comment on usefulness that'd be great.
>>>
>>> One other nice thing, with more tests, it'd add some more
>>> self-documentation, as developers unfamiliar with a feature could look
>>> at the tests to see what is expected.
>>>
>>> Thoughts?
>>>
>>> Thanks!
>>> steven
>>>
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond.
>>> Discussions
>>> will include endpoint security, mobile security and the latest in
>>> malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> Csound-devel mailing list
>>> Csound-devel@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/csound-devel
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Csound-devel mailing list
>> Csound-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/csound-devel
>>
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Csound-devel mailing list
> Csound-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/csound-devel

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Csound-devel mailing list
Csound-devel@lists.sourceforge.net
https://lists.sour

Which tests are these?  The only ones I'm aware of are the ones in the
tests folder and there is no md5 hash checking in those (though that
could be added).

I think we may also want to categorize tests.  Some may be for API,
others for regression against a bug, others for opcodes.  We may want
to have long running and quick tests and separate them, so that some
tests can be run more regularly and others as time permits.

On Mon, Apr 30, 2012 at 9:22 AM,   wrote:
> Are you going to incorporate the old tests suite into your python script?
> It used to check md5 hash of the audio file output as well as not crashing
>
>
>> Hi Steven,
>>
>> I'm all for unit tests!
>>
>> I've used CTest, and it works quite well. But for Csound where most of
>> the results are not in memory, but in external files, it might become
>> cumbersome, and a python script might be better suited.
>>
>> Maybe files could automatically be added if they are not already
>> there, but I think adding test rules to the csd file itself might not
>> be the best idea as we would have to define a whole new syntax,
>> parsing, etc for it.
>>
>> I think it's just a matter of bugging everyone a bit to do add the
>> tests until it becomes a habit. I had to do this for a while with the
>> manual pages... :)
>>
>> Cheers,
>> Andrés
>>
>> On Mon, Apr 30, 2012 at 12:26 AM, Steven Yi  wrote:
>>> Hi All,
>>>
>>> I am noticing a lot of tests and development where we're testing
>>> locally with CSD's.  It would be great that while we're doing our
>>> development and debugging of bugs to keep adding more tests to unit
>>> tests in the tests folder so that we keep building up more and more
>>> examples to use for regression testing.
>>>
>>> I am thinking though, maybe something that would make it easier is if
>>> we modify the testing script to not require manually adding them to
>>> the python script but just have it walk the tests folder for csd's and
>>> run them.  We could modify the testing so that the CSD itself could
>>> have in the first few lines some info about expected results, whether
>>> a test is disabled by default, etc.  Also, we could add a csd testing
>>> target to CMake so that we could do "make testcsds".
>>>
>>> I think to it'd be nice to add API tests with python scripts, and
>>> possible also start using CTest
>>> (http://www.cmake.org/cmake/help/ctest-2-8-docs.html).  I don't have
>>> experience with CTest however, so if anyone has any experience and
>>> could comment on usefulness that'd be great.
>>>
>>> One other nice thing, with more tests, it'd add some more
>>> self-documentation, as developers unfamiliar with a feature could look
>>> at the tests to see what is expected.
>>>
>>> Thoughts?
>>>
>>> Thanks!
>>> steven
>>>
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond.
>>> Discussions
>>> will include endpoint security, mobile security and the latest in
>>> malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> Csound-devel mailing list
>>> Csound-devel@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/csound-devel
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Csound-devel mailing list
>> Csound-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/csound-devel
>>
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Csound-devel mailing list
> Csound-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/csound-devel

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Csound-devel mailing list
Csound-devel@lists.sourceforge.net