I too am interested in this idea, though I don't think the cross-platform
issues should be under-estimated (different flavours of lex and yacc -this
comment also applies to the dynamic library question), both in the coding,
and in  the matter of ongoing maintenance and development thereafter. It is,
of course, an enormous task, as no doubt most, if not all, of the current
Csound data structures and access routines might need to be recast to suit
the parser. Just how incrementally could such  change be made?  What would
be the minumun number of opcodes needed to excercise the parser?

I can imagine that the creation of a reasonably self-contained parser might
lead to a new range of Csound facilities - perhaps even a source-file
debugger to step through an orc file?

I would be interested, to the extent that my un-schooled knowledge  permits,
to contribute to some work on this; apart from anythig else, I am keen to
learn to use flex and bison on a real project. The SAOL/MPEG-4 team at MIT:
http://sound.media.mit.edu/~eds/mpeg4
have included yacc and lex scripts in the source distribution for saol -  it
might be a useful model.

Richard Dobson


Paul Winkler wrote:

> This sounds like a worthwhile project to me. I would certainly be
> interested in the results. I have to work pretty hard just to get csound
> instruments working, though-- no programming experience-- so that puts
> me out of the development team.
>
> --PW
>
> >Date: Fri, 13 Feb 1998 17:02:48 +0100 (MET)
> >From: Nicola Bernardini 
> >To: Csound mailing list 
> >Subject: Re: entry.c and dynamic linking
>
> > Can you imagine what csound would be like with these
> >constructs? It would have the same impact as switching from
> >fortran-like Music V to csound itself!
> >
> >I am available to put some effort into this, but I would like to
> >understand from this list if this is something effectively useful
> >to the community (I won't do it for the sake of doing it, I don't
> care).
>
> ______________________________________________________
> Get Your Private, Free Email at http://www.hotmail.com

> 
> 2) I don't think it is such an enormous task: I think that, for the sake of
>    making everything work, we should not *touch* even in the smallest detail
>    csound data structures (if needed, these can be enclosed in other data
>    structures); I think this is mandatory to be backward compatible. I think

I agree that getting rid of the parsing code in csound in favor of a
more flexible bison/flex approach is a good idea.  I also agree that
this can be done with very little change to the existing data
structures.  However, I think that in the (near) future, it would be a
good idea to consider dramatic structural changes to csound.

Problems with current csound implementation:

1) It's organic code written by many people over a long period of
time.  Little documentation is available to make things clearer.

2) It's written in a style conducive to saving memory and could
perhaps better utilize the speed of current processors if it paid less
attention to memory management.  

3) It lacks extensibility.  Adding opcodes seems prohibitively
esoteric, and the messy implementation is the major reason why it is
so difficult to extend the csound language itself (for instruments to
turn on other instruments, to include local state variables within
instruments, to support opcode arrays, etc.)  It seems the creators of
the MPEG4 standard have thought this through.  The orc/sco files I've
seen for SAOL/MPEG4 seems to have a sort of "csound++" flavor.  I
especially like the ability to define new opcodes in the orc file.
However, they have made a number of design decisions that have made
realtime performance implausible.

4) It's written in C (ugh..)  I am considering undertaking a total
rewrite in C++ that would alleviate many of the current problems.  I'd
be interested to hear ideas from other people as far as object
hierarchy, etc..  A good rewrite *should* have future extensibility as
a primary goal.

>    the parser could and should access the entry data structure just as it is
>    (right after, we could implement a fast hash search of the entry array
>    to speed up things - but this can be postponed, and if somebody can help
>    there: I am not good at these things)
> 

This is trivial to do with the builtin hashtable class in the libg++
library.

> > I think that all these are very good things, of course only if backward
> > compatiblity with old orc/sco will be completely kept back.
> 
> of course: this is absolutely essential *BEFORE* writing any extension.
> 

For the time being I would agree with this.  In the future, if any
major changes are made to the csound language, it would be relatively
trivial to maintain backward compatibility simply by providing a
translator.

> 1) I will try to produce a testing parser for orcs (let's start from orcs,
>    if you agree) *as quickly as I can*. I'll try to include everything,
>    but I think it is mandatory now to be very very quick, otherwise
>    this thing will go stale... The testing parser will be a separate
>    application that just groks orchestras and if it works, it should never barf.
>    It should just report on standard output what it is reading on
>    standard input (that would be enough to understand wheter it works
>    or not).
> 

Just remember that people in the future may be building on whatever
code is written.

> produce an intelligent parser for sco files, I need some informed
> advice on whether it would be useful to run the sco parser on the
> sorted sco file or on the original sco file: I believe the first
> option poses limits
>    on what could be the future types included in sco files (would it be
>    able to handle string constants for example, an interesting extension);
>    while the second option would imply re-building sort and extract
>    features, a useless task IMHO (if it's not broken, don't
>    fix it, an old programming adagio).
> 

Unfortunately, right now csound reads the sorted sco file line by line
during actual processing.  This would have to be changed before you
get fancy with pre-processing.

__________________________________________________________________________
David Ratajczak 
Web:    http://web.mit.edu/dratajcz/
E-mail: dratajcz@mit.edu

Dear csound list,

thank you for your kind feedback. I was very pleased that these ideas
sound interesting to people that are csound old-timers and that have
already contributed in many ways to the growth of csound. I will try
to reply below to individual questions, and then I'll make a proposal.

On Fri, 13 Feb 1998, Paul Winkler wrote:

[snip]
> interested in the results. I have to work pretty hard just to get csound 
> instruments working, though-- no programming experience-- so that puts 
> me out of the development team.

well, you could be part of the testing team (if such thing existed)...

On Sat, 14 Feb 1998, Richard Dobson wrote:

[snip]
> I too am interested in this idea, though I don't think the cross-platform
> issues should be under-estimated (different flavours of lex and yacc -this
> comment also applies to the dynamic library question), both in the coding,
> and in  the matter of ongoing maintenance and development thereafter. It is,
> of course, an enormous task, as no doubt most, if not all, of the current
> Csound data structures and access routines might need to be recast to suit
> the parser. Just how incrementally could such  change be made?  What would
> be the minumun number of opcodes needed to excercise the parser?

1) I don't under-estimate different platforms, I am ready to discuss to
   any depth how they should be handled (among other things, porting software
   is what I do for a living...); besides, the different flavours of lex
   and yacc is easy to handle: let's just use flex and bison which are good
   and quite standard lex and yaccs. These are ported under any known platform
   (even mac, I believe...), and even if they were'nt, the output of these
   tools is plain c code: those who don't have lex and yacc won't be able
   to make changes to the syntax and grammar of csound, but will still be
   able to compile the parser with a standard C compiler

2) I don't think it is such an enormous task: I think that, for the sake of
   making everything work, we should not *touch* even in the smallest detail
   csound data structures (if needed, these can be enclosed in other data
   structures); I think this is mandatory to be backward compatible. I think
   the parser could and should access the entry data structure just as it is
   (right after, we could implement a fast hash search of the entry array
   to speed up things - but this can be postponed, and if somebody can help
   there: I am not good at these things)

3) thus, we can very well include all the opcodes right away, because
   we just look in the entry table, right?

> 
> I can imagine that the creation of a reasonably self-contained parser might
> lead to a new range of Csound facilities - perhaps even a source-file
> debugger to step through an orc file?

very nice idea indeed. Now that is a fairly difficult thing to do, at least
for my abilities. But once somebody wrote something like that,
it would be very easy to make a separate utility (one of those -U things).
One thing that is certain is that error checking could be made to be
much better and informative. (It is also true that error checking is
one of the most difficult parsing activities: now that should not be
underestimated).
Another nice thing would be to use the parsers for visual display,
or to go to/from nice programs like Miller Puckette's PD.

> 
> I would be interested, to the extent that my un-schooled knowledge  permits,
> to contribute to some work on this; apart from anythig else, I am keen to
> learn to use flex and bison on a real project. The SAOL/MPEG-4 team at MIT:
> http://sound.media.mit.edu/~eds/mpeg4
> have included yacc and lex scripts in the source distribution for saol -  it
> might be a useful model.

Actually, that was exactly what I had in mind: let's pick up the lex and
yacc files from saol as an inspiration and let's produce an initial csound
parser. I have already studied that code, and to tell you frankly, I
don't like it very much: I don't want to be presumptous, but I think we
should/could do much better and I'll try to demonstrate that. It is a
useful model, though, and we'll certainly make use of it.
Thanks, Mr. Dobson, for your words.

On Sat, 14 Feb 1998, Gabriel Maldonado wrote:

[snip]
> I think that all these are very good things, of course only if backward
> compatiblity with old orc/sco will be completely kept back.

of course: this is absolutely essential *BEFORE* writing any extension.

> I think that all these are very good things, of course only if backward
> compatiblity with old orc/sco will be completely kept back.
> BtW what are exactly lex, yacc, flex and bison? Do they work  with all
> platforms and with all C-C++ compilers?

lex is a lexical analyzer generator, a finite-state automaton generator
which is able to produce c code that does lexical scanning on standard
input. Flex is the GNU public domain lex, available all over. yacc
is a grammar generator, and bison is its GNU counterpart. yacc too
produces c code, so as I said, these tools produce code that can
be compiled with standard c compilers. I believe the GNU extensions
would allow to englobe lexer and parser in C++ objects, a nifty feature
to have multiple parsers, and very useful for us in order to produce
different parsers for the orc and the sco files (I'll look into that).
These tools come out of the best intelligence products back in the seventies
in the creation of unix platforms (you know: Kernighan, Bentley, Aho,
Ullman, all these geniuses which were working at AT&T along with our
guru Max Mathews). If you don't know them, be ready to meet some
very very powerful tools. Besides, you might be able to find Kernighan
documentation on the net (the famous papers) and there is a very
good book on lex/yacc from O'Reilly (called "lex & yacc", what would you
know).

Ok: I think all these nice answers are a good enough stimulus to start
working. Let me tell you what I am going to do:

1) I will try to produce a testing parser for orcs (let's start from orcs,
   if you agree) *as quickly as I can*. I'll try to include everything,
   but I think it is mandatory now to be very very quick, otherwise
   this thing will go stale... The testing parser will be a separate
   application that just groks orchestras and if it works, it should never barf.
   It should just report on standard output what it is reading on
   standard input (that would be enough to understand wheter it works
   or not).

2) All the interested people should compile and test the parser with
   their orchestra and report any errors and bugs. What especially
   worries me is error reporting: the parser should be tested against
   anything: wrong syntaxes, random files, binary input, everything.

3) then we'll attack the score part (as a matter of fact, if somebody
   would like to attack that in parallel, it's fine...).
   To produce an intelligent parser for sco files, I need some informed
   advice on whether it would be useful to run the sco parser on the sorted
   sco file or on the original sco file: I believe the first option poses limits
   on what could be the future types included in sco files (would it be
   able to handle string constants for example, an interesting extension);
   while the second option would imply re-building sort and extract
   features, a useless task IMHO (if it's not broken, don't
   fix it, an old programming adagio).

Sorry for the very long post.

I'll be right back with the goods

Nicola
------------------------------------------------------------------------
Nicola Bernardini
E-mail: nicb@axnet.it
 
Re graphics: A picture is worth 10K words -- but only those to describe
the picture.  Hardly any sets of 10K words can be adequately described
with pictures.

seems like we had a similar thread a few months back...
ive sort of skimmed the recent posts, so i might have not caught
everything, but:

	here are my thoughts and *personal* wishes for a "new" csound:

> > 2) I don't think it is such an enormous task: I think that, for the sake of
> >    making everything work, we should not *touch* even in the smallest detail
> >    csound data structures (if needed, these can be enclosed in other data
> >    structures); I think this is mandatory to be backward compatible. I think

> good idea to consider dramatic structural changes to csound.

> rewrite in C++ that would alleviate many of the current problems.  I'd
> be interested to hear ideas from other people as far as object

	i would propose the exact opposite:  start from the ground up by
rewriting the (at least basic) opcodes as a platform independant Shared
Library in C (if this is done right, will secure the longevity of the code
-
how much can an oscilator change? - plus open-ended for many types of
applications).  then can write a synthesis engine that will make the
opcode calls - a csound orc/sco parser can sit on that.

	yes having orc/sco backwards compatibility is essential,
but i usually want something else:	there are
so many users of CSound that are programmers too, there
must be alot of you who share my experience: wanting to at times just use
CSound (or even something of a higher level - maybe a GUI) and some times
wanting to either temporarily drop down into C/C++ or just write your own
code that makes certain calls to CSound (so far
the only thing we can really do is write score generators and write our
own opcodes)

 > the MPEG4 standard have thought this through. The orc/sco files
I've > seen for SAOL/MPEG4 seems to have a sort of "csound++" flavor.  I
> However, they have made a number of design decisions that have made
> realtime performance implausible.

	Can someone explain to me what is the *intent* of SAOL/MPEG4??  Is
it indeed a "new Csound"?  i was thinking it was intended as sort of a
real-time java for sound?  is SAOL designed to be a composer's tool or for
(commercial) developers? please explain..

 > 
> >    on what could be the future types included in sco files (would it be
> >    able to handle string constants for example, an interesting extension);

	i actually just last night was looking into this (for existing
CSound) i was going to try something a bit simpler by making a GEN02-kind
of thing but with strings for values.  (this is to have a lookup table for
a soundfile so i dont have to rename all my files to soundin.xx)  anybody
ever do anything like this?

-matt

Dear csounders,

let me pick up the last mail I got in this thread to make some quick
replies (it won't be long as last time, I promise).


On Sat, 14 Feb 1998, Matt J. Ingalls wrote:
(this replies to David Ratajczak too)

[snip]
> 
> > good idea to consider dramatic structural changes to csound.
> 
> > rewrite in C++ that would alleviate many of the current problems.  I'd
> > be interested to hear ideas from other people as far as object
> 
> 	i would propose the exact opposite:  start from the ground up by
> rewriting the (at least basic) opcodes as a platform independant Shared
> Library in C (if this is done right, will secure the longevity of the code

my personal experience in programming tells me exactely the opposite
of these two opposites: as I already said, you should'nt touch code
that works. Even if it's ugly. Several years ago I picked up the csound
code and tried to clean up the warnings cropping up with gcc -Wall:
tons and tons of correct warnings (that is, badly written conditionals,
etc.). Result: I broke the code and had to throw everything away because
I did not have enough time to debug it properly. We must remember that
we are all working for free here, so we are paying with our time the
need for these nicely-featured tools that we all get. Personally, I
cannot afford a rewrite: the only thing I can afford right now (and I
am not even sure about that) is a new parser, and that's what I am doing
right now. Initially, this new parser won't change the language *even
in the smallest detail*: that's my target. But if/when it'll be available,
it will allow other people to add new syntactic features to csound.
Be aware that grammars are not an easy thing to deal with: it is extremely
easy to produce "personalized" features which are not really useful
in a general way and may even make the language less clear than it
is now (a hard endeavour I must admit). A language, in order to be useful,
must be diffused (read/spoken) by a community, is'nt it?

[snip]
> so many users of CSound that are programmers too, there
> must be alot of you who share my experience: wanting to at times just use
> CSound (or even something of a higher level - maybe a GUI) and some times
> wanting to either temporarily drop down into C/C++ or just write your own
> code that makes certain calls to CSound (so far
> the only thing we can really do is write score generators and write our
> own opcodes)

I usually use text filtering programs (awk, perl, etc.) to produce
final orc/sco files and that works for me very well. Another thing that
could be done is (as I proposed in another mail) to keep the opcodes
in dinamically linked libraries and to access them via the csound main
*OR* with other applications (it would be nice, for example, to use a
tcl/tk interface to the library; another idea would be to build the
correct translation interface for PD, the public domain MAX replacement
from Miller Puckette that works also with audio, to the csound opcodes,
etc. etc.). The lightweight approach of using interpreters allow to
do disposable little programs that can, if need be, written for single
musical pieces or requirements and then thrown away (that is, rest unused
for the rest of their/my life).

> 
>  > the MPEG4 standard have thought this through. The orc/sco files
> I've > seen for SAOL/MPEG4 seems to have a sort of "csound++" flavor.  I
> > However, they have made a number of design decisions that have made
> > realtime performance implausible.
> 
> 	Can someone explain to me what is the *intent* of SAOL/MPEG4??  Is
> it indeed a "new Csound"?  i was thinking it was intended as sort of a
> real-time java for sound?  is SAOL designed to be a composer's tool or for
> (commercial) developers? please explain..

>From what I understand, SAOL is an attempt to write a protocol (there's
no specification for 'real-time', AFAIK) so that you don't have to send
synthesized samples through the network, you just send the algorithms.
It's the same idea than X11, basically. The nice thing is that they picked
up quite a lot of lessons from csound and made a better language. There
are many other nice languages for sound synthesis: nyquist, common music,
etc. They have many advantages over csound, because they came later.
But: csound still has the biggest following, and thus more people that
write opcodes, etc. etc. So: let's talk about adding features and
easyness to csound - other languages will come and we'll switch to
them when times will be ripe.

[snip]
> 	i actually just last night was looking into this (for existing
> CSound) i was going to try something a bit simpler by making a GEN02-kind
> of thing but with strings for values.  (this is to have a lookup table for
> a soundfile so i dont have to rename all my files to soundin.xx)  anybody
> ever do anything like this?

I pick up this example as a typical case of an extension that is connected
to a platform-specific problem. If you have a platform that *truly*
support symbolic links, you don't have to rename your files to soundin.xx,
and it is actually very convenient to use symbolic links between
files and soundins, so that you don't have to hardwire in your orc or
sco the names that may change in the future. For those diminished platforms
that do not fully support links, maybe we (somebody else, not me) could
work out a solution like this: why not providing a configuration file
(loaded with an option, for example, like the -x files) that gives
the correspondence between soundins and file names? This is:

1) easier to do
2) perhaps even useful to everybody else

Sorry, it again took very long.

Nicola

------------------------------------------------------------------------
Nicola Bernardini
E-mail: nicb@axnet.it
 
Re graphics: A picture is worth 10K words -- but only those to describe
the picture.  Hardly any sets of 10K words can be adequately described
with pictures.

> right now. Initially, this new parser won't change the language *even
> in the smallest detail*: that's my target. But if/when it'll be available,

	im suggesting leave CSound as is and start "porting" it into a
more open-ended platform independant synthesis library.   but does
something like this already exist?
> 
> I pick up this example as a typical case of an extension that is connected
> to a platform-specific problem. If you have a platform that *truly*
> support symbolic links, you don't have to rename your files to soundin.xx,

	im just a stupid mac-user - could you explain this more?

-matt

On Sun, 15 Feb 1998, Matt J. Ingalls wrote:

[snip]
> > I pick up this example as a typical case of an extension that is connected
> > to a platform-specific problem. If you have a platform that *truly*
> > support symbolic links, you don't have to rename your files to soundin.xx,
> 
> 	im just a stupid mac-user - could you explain this more?

please don't take my words as offensive: that is absolutely not
what I mean. What I meant is that on some systems, namely Windows 95
or Mac OS, there are things like "connections" and "aliases" which
at first sight may look like unix's symbolic links. I do not know
about macintoshes, but unfortunately the connections on Win95 do not
work as symbolic links, that is, csound is not able to read files
that are just connections (or aliases, if you wish) to sound files
(at least that was the case the last time I looked at it).
On unix systems these things work, that's all: you can apply a symbolic
link to a sound file, name it soundin.xx and csound will read it
as a sound file.

Nicola

------------------------------------------------------------------------
Nicola Bernardini
E-mail: nicb@axnet.it
 
Re graphics: A picture is worth 10K words -- but only those to describe
the picture.  Hardly any sets of 10K words can be adequately described
with pictures.