[Csnd-dev] Unit, CSD Testing
| Date | 2019-10-12 19:56 |
| From | Steven Yi |
| Subject | [Csnd-dev] Unit, CSD Testing |
Hi All, I thought I'd start a thread about automated testing for Csound. We have in the past done some work on this but it seems like this would be an area which can and should be improved. Right now we have: 1. CSD tests - Blackbox testing for the system. These seem to be categorized into different commands to run regression and standard tests. I think we need may need to reorganize our tests and have just one test running script run all of these. We may also need a better testing system; the python script I wrote years back has been serviceable but could certainly be improved. Any thoughts on improvements? Some of the tests I wrote early on were essentially testing language features as they were written when we moved to CS6 and adding to the language. These kinds of tests are in CS7 for the new language features too. We also have tests that involve specific opcodes and are part of the regression tests too. 2. Writing CSD tests - We should probably have documentation on how to do this, particularly with the goal to make it easier to contribute tests. Perhaps a collaborative test writing marathon one weekend would be a way to get people on the same page and give us a way to move forward? 3. Unit tests - These are currently written in C using CUnit and are for testing internal code. There's been attempts to move away from CUnit to other frameworks. We should revisit this soon. (I have a student who may be able to assist in the move to a different testing system, we just need to choose what to move to.) 4. Writing unit tests - We certainly have a lot of the system uncovered by tests at this time. It would be great if we used the test writing process as a way to help document the system for new developers as well as help ensure the system remains as expected, especially as we move to CS7 development. 5. Integration tests - We have some of this as part of our C tests but not so formally. We might consider these our "API tests" that help ensure the API is functioning as expected through our future changes. Also could serve as documentation for new API users. 6. Opcode tests - we don't have this now at the C level. I think John had a test suite from the past that compared outputs from previous runs to detect changes but I don't remember any tests for particular opcodes. Perhaps this would be something that would be easier in CS7 with the UGen API. Okay, just some ideas from me above. Would love to hear other feedback on suggested changes that could lead to action items. Thanks, Steven |
| Date | 2019-10-13 13:16 |
| From | Stephen Kyne |
| Subject | Re: [Csnd-dev] Unit, CSD Testing |
|
Hi Steven,
Yeah I think it’s a great idea to expand the automated testing. For the unit testing specifically, I was going to try and reimplement them using Doctest. https://github.com/onqtam/doctest
It would slot in pretty well with what is there already. Just a matter of switching the C API and minor CTest changes for the CMakeFiles.
Stephen
From: Csound-developers <CSOUND-DEV@LISTSERV.HEANET.IE> on behalf of Steven Yi <stevenyi@GMAIL.COM>
Sent: Saturday, October 12, 2019 8:56:22 PM To: CSOUND-DEV@LISTSERV.HEANET.IE <CSOUND-DEV@LISTSERV.HEANET.IE> Subject: [Csnd-dev] Unit, CSD Testing Hi All,
I thought I'd start a thread about automated testing for Csound. We have in the past done some work on this but it seems like this would be an area which can and should be improved. Right now we have: 1. CSD tests - Blackbox testing for the system. These seem to be categorized into different commands to run regression and standard tests. I think we need may need to reorganize our tests and have just one test running script run all of these. We may also need a better testing system; the python script I wrote years back has been serviceable but could certainly be improved. Any thoughts on improvements? Some of the tests I wrote early on were essentially testing language features as they were written when we moved to CS6 and adding to the language. These kinds of tests are in CS7 for the new language features too. We also have tests that involve specific opcodes and are part of the regression tests too. 2. Writing CSD tests - We should probably have documentation on how to do this, particularly with the goal to make it easier to contribute tests. Perhaps a collaborative test writing marathon one weekend would be a way to get people on the same page and give us a way to move forward? 3. Unit tests - These are currently written in C using CUnit and are for testing internal code. There's been attempts to move away from CUnit to other frameworks. We should revisit this soon. (I have a student who may be able to assist in the move to a different testing system, we just need to choose what to move to.) 4. Writing unit tests - We certainly have a lot of the system uncovered by tests at this time. It would be great if we used the test writing process as a way to help document the system for new developers as well as help ensure the system remains as expected, especially as we move to CS7 development. 5. Integration tests - We have some of this as part of our C tests but not so formally. We might consider these our "API tests" that help ensure the API is functioning as expected through our future changes. Also could serve as documentation for new API users. 6. Opcode tests - we don't have this now at the C level. I think John had a test suite from the past that compared outputs from previous runs to detect changes but I don't remember any tests for particular opcodes. Perhaps this would be something that would be easier in CS7 with the UGen API. Okay, just some ideas from me above. Would love to hear other feedback on suggested changes that could lead to action items. Thanks, Steven |
| Date | 2019-10-15 02:57 |
| From | Hlöðver Sigurðsson |
| Subject | Re: [Csnd-dev] Unit, CSD Testing |
To echo what we discussed, I'd be especially excited to see built-in test library in csound. This would be useful even for composers as well as internals testing, if there was the possibility of giRecorded ftgen ... giExpected ftgen ... test "expected-audio" assert-tab-eq giRecorded, giExpected endtest event_i("i", "record-to-table", 0, 1) event_i("test", "expected-audio", 1, 0) On Sun, 13 Oct 2019 at 14:16, Stephen Kyne <stevek@outlook.ie> wrote:
|