Hi

I am working on a set of opcodes implementing a hashtable. These
hashtables can be either local to a note or global. I have been looking
to other opcodes which access global data, such as the OSC opcodes.
These use a mutex to synchronize access to shared data. On the other
hand, the channel opcodes use spinlocks, but these are not implemented
in the opcode api. What should an opcode use? At the moment my implemen-
tation uses mutexes, as in this example below.

static int32_t
hashtab_set_int_float(CSOUND *csound, HASHTAB_SET_if *p) {
     KHASH_GLOBALS *g = p->globals;
     int32_t idx = (int32_t)*p->ihandle;
     KHASH_HANDLE *handle = &(g->handles[idx]);
     khash_t(khIntFloat) *h = handle->khashptr;
     int32_t key = *p->key;
     khint_t k;
     csound->LockMutex(handle->mutex_);     // <---------------------
     kh_key(h, k) = key;
     kh_value(h, k) = *p->value;
     csound->UnlockMutex(handle->mutex_);   // <----------------------
     return OK;
}

Best regards,
Eduardo

Csound mailing list
Csound@listserv.heanet.ie
https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
Send bugs reports to
        https://github.com/csound/csound/issues
Discussions of bugs and features can be posted here

Before we could look at the best solution, I just wanted to know why do you need to synchronise access, is that for multicore Csound?

---

From: A discussion list for users of Csound <CSOUND@LISTSERV.HEANET.IE> on behalf of Eduardo Moguillansky <eduardo.moguillansky@GMAIL.COM>
Sent: Friday 8 February 2019 00:38:43
To: CSOUND@LISTSERV.HEANET.IE
Subject: [Csnd] mutex / spinlock on opcodes

 

Hi

I am working on a set of opcodes implementing a hashtable. These
hashtables can be either local to a note or global. I have been looking
to other opcodes which access global data, such as the OSC opcodes.
These use a mutex to synchronize access to shared data. On the other
hand, the channel opcodes use spinlocks, but these are not implemented
in the opcode api. What should an opcode use? At the moment my implemen-
tation uses mutexes, as in this example below.

static int32_t
hashtab_set_int_float(CSOUND *csound, HASHTAB_SET_if *p) {
     KHASH_GLOBALS *g = p->globals;
     int32_t idx = (int32_t)*p->ihandle;
     KHASH_HANDLE *handle = &(g->handles[idx]);
     khash_t(khIntFloat) *h = handle->khashptr;
     int32_t key = *p->key;
     khint_t k;
     csound->LockMutex(handle->mutex_);     // <---------------------
     kh_key(h, k) = key;
     kh_value(h, k) = *p->value;
     csound->UnlockMutex(handle->mutex_);   // <----------------------
     return OK;
}

Best regards,
Eduardo

Csound mailing list
Csound@listserv.heanet.ie
https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
Send bugs reports to
        https://github.com/csound/csound/issues
Discussions of bugs and features can be posted here

Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here

The spinlocks are there to synchronise talk to hosts so that csound processing can run on its own thread outside the UI/message processing. Inside a Csound orchestra they have no use unless multicore is used, but I have also to look whether there are is also other synchronisation involved in that case.

Are you planning to replace the bus channels? That does not sound like a good

idea. Maybe we need to think a little more

about it.

Victor Lazzarini

Dean of Arts, Celtic Studies, and Philosophy

Maynooth University

Ireland

On 8 Feb 2019, at 09:56, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote:

The goal of these opcodes is to replace channels for the use case of inter-note communication. Looking at the code for channels I noticed that they have a lock per channel so I assumed that some sort of synchronisation was necessary to prevent data being corrupted if multiple notes are writing to the same hashtable.

On 08.02.19 10:28, Victor Lazzarini wrote:

Before we could look at the best solution, I just wanted to know why do you need to synchronise access, is that for multicore Csound?

---

From: A discussion list for users of Csound <CSOUND@LISTSERV.HEANET.IE> on behalf of Eduardo Moguillansky <eduardo.moguillansky@GMAIL.COM>
Sent: Friday 8 February 2019 00:38:43
To: CSOUND@LISTSERV.HEANET.IE
Subject: [Csnd] mutex / spinlock on opcodes

 

Hi

I am working on a set of opcodes implementing a hashtable. These
hashtables can be either local to a note or global. I have been looking
to other opcodes which access global data, such as the OSC opcodes.
These use a mutex to synchronize access to shared data. On the other
hand, the channel opcodes use spinlocks, but these are not implemented
in the opcode api. What should an opcode use? At the moment my implemen-
tation uses mutexes, as in this example below.

static int32_t
hashtab_set_int_float(CSOUND *csound, HASHTAB_SET_if *p) {
     KHASH_GLOBALS *g = p->globals;
     int32_t idx = (int32_t)*p->ihandle;
     KHASH_HANDLE *handle = &(g->handles[idx]);
     khash_t(khIntFloat) *h = handle->khashptr;
     int32_t key = *p->key;
     khint_t k;
     csound->LockMutex(handle->mutex_);     // <---------------------
     kh_key(h, k) = key;
     kh_value(h, k) = *p->value;
     csound->UnlockMutex(handle->mutex_);   // <----------------------
     return OK;
}

Best regards,
Eduardo

Csound mailing list
Csound@listserv.heanet.ie
https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
Send bugs reports to
        https://github.com/csound/csound/issues
Discussions of bugs and features can be posted here

Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here

Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here

Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here

+1 We should identify the cause of the slowdown in channels and
address that (hence why I asked about naming pattern in the problem
project so that we can test and profile).  If a problem is identified
then it'd benefit everyone to get that addressed in the channel
system.

There's also a longstanding proposal to have generic lists, sets, and
maps for CS7 as data types.

On Fri, Feb 8, 2019 at 8:47 AM Victor Lazzarini  wrote:
>
>  Just a thought, can you use global arrays?
>
> Also the zak opcodes were designed for this kind of thing as well.
>
> It would be better to fix the chn opcodes rather than add similar but not quite similar
> functionality. This happened often in the past and is a known historical problem of
> Csound development.
>
> Steven, John, do you have any thoughts on this?
>
> Victor Lazzarini
> Dean of Arts, Celtic Studies, and Philosophy
> Maynooth University
> Ireland
>
> On 8 Feb 2019, at 13:42, Eduardo Moguillansky  wrote:
>
> On 08.02.19 12:39, Victor Lazzarini wrote:
>
> The spinlocks are there to synchronise talk to hosts so that csound processing can run on its own thread outside the UI/message processing. Inside a Csound orchestra they have no use unless multicore is used, but I have also to look whether there are is also other synchronisation involved in that case.
>
> ok. Is there a #define I can test against to check if a multicore version is being built?
>
>
> Are you planning to replace the bus channels? That does not sound like a good
> idea. Maybe we need to think a little more
> about it.
>
> No, I am not planning to replace the bus channels. I need an efficient / flexible way to communicate between notes. The bus channel opcodes don't allow to remove key value pairs and their efficiency degrades with size of the global hashtable. At the same time they are slow for the case where the string key changes frequently. This set of opcodes try to provide a hashtable which for these use cases is ~ 30x faster, for integer keys as fast as builtin arrays, for constant string keys as fast as chnget/set and offers the flexibility of being able to create and free hashtables, pass them around, delete key pairs, print the contents, etc. The implementation uses klib's khash (https://github.com/attractivechaos/klib). It is a proof of concept, and some testing is still needed:
>
> https://github.com/gesellkammer/ugens/blob/master/klib/klib.c
>
>
> Victor Lazzarini
> Dean of Arts, Celtic Studies, and Philosophy
> Maynooth University
> Ireland
>
> On 8 Feb 2019, at 09:56, Eduardo Moguillansky  wrote:
>
> The goal of these opcodes is to replace channels for the use case of inter-note communication. Looking at the code for channels I noticed that they have a lock per channel so I assumed that some sort of synchronisation was necessary to prevent data being corrupted if multiple notes are writing to the same hashtable.
>
> On 08.02.19 10:28, Victor Lazzarini wrote:
>
> Before we could look at the best solution, I just wanted to know why do you need to synchronise access, is that for multicore Csound?
>
> ________________________________
> From: A discussion list for users of Csound  on behalf of Eduardo Moguillansky 
> Sent: Friday 8 February 2019 00:38:43
> To: CSOUND@LISTSERV.HEANET.IE
> Subject: [Csnd] mutex / spinlock on opcodes
>
> Hi
>
> I am working on a set of opcodes implementing a hashtable. These
> hashtables can be either local to a note or global. I have been looking
> to other opcodes which access global data, such as the OSC opcodes.
> These use a mutex to synchronize access to shared data. On the other
> hand, the channel opcodes use spinlocks, but these are not implemented
> in the opcode api. What should an opcode use? At the moment my implemen-
> tation uses mutexes, as in this example below.
>
> static int32_t
> hashtab_set_int_float(CSOUND *csound, HASHTAB_SET_if *p) {
>      KHASH_GLOBALS *g = p->globals;
>      int32_t idx = (int32_t)*p->ihandle;
>      KHASH_HANDLE *handle = &(g->handles[idx]);
>      khash_t(khIntFloat) *h = handle->khashptr;
>      int32_t key = *p->key;
>      khint_t k;
>      csound->LockMutex(handle->mutex_);     // <---------------------
>      kh_key(h, k) = key;
>      kh_value(h, k) = *p->value;
>      csound->UnlockMutex(handle->mutex_);   // <----------------------
>      return OK;
> }
>
> Best regards,
> Eduardo
>
> Csound mailing list
> Csound@listserv.heanet.ie
> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
> Send bugs reports to
>         https://github.com/csound/csound/issues
> Discussions of bugs and features can be posted here
> Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here
>
> Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here
>
> Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here
>
> Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here
>
> Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here

Csound mailing list
Csound@listserv.heanet.ie
https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
Send bugs reports to
        https://github.com/csound/csound/issues
Discussions of bugs and features can be posted here

Regarding slowness and memory usage of chnset, you can look at this:

https://github.com/gesellkammer/ugens/blob/master/klib/examples/dict_bench_set.csd

Both instruments do the same, but instr 2 (chnset) takes more than 30x 
longer and uses significantly more memory. Regarding possible 
optimizations you can take a look at the implementation of dict_set 
(https://github.com/gesellkammer/ugens/blob/master/klib/klib.c, 
beginning at line 523) which makes some efforts to avoid a lookup if it can.
It can also be that the underlying hashtable implementation in csound is 
not so efficient.

Another problem with the chn opcodes is the impossibility of removing a 
key:value pair, or of creating a local/global hashtable which is 
independent of the global table used by all chn opcodes.

Independently of how/if the chn opcodes are improved, maybe it would be 
possible to address the issue of having plugins outside of the main 
tree, similar to externals in pd or sc-plugins in supercollider or 
similar to a stdlib and external libraries in any other language.

On 08.02.19 16:43, Steven Yi wrote:
> +1 We should identify the cause of the slowdown in channels and
> address that (hence why I asked about naming pattern in the problem
> project so that we can test and profile).  If a problem is identified
> then it'd benefit everyone to get that addressed in the channel
> system.
>
> There's also a longstanding proposal to have generic lists, sets, and
> maps for CS7 as data types.
>
> On Fri, Feb 8, 2019 at 8:47 AM Victor Lazzarini  wrote:
>>   Just a thought, can you use global arrays?
>>
>> Also the zak opcodes were designed for this kind of thing as well.
>>
>> It would be better to fix the chn opcodes rather than add similar but not quite similar
>> functionality. This happened often in the past and is a known historical problem of
>> Csound development.
>>
>> Steven, John, do you have any thoughts on this?
>>
>> Victor Lazzarini
>> Dean of Arts, Celtic Studies, and Philosophy
>> Maynooth University
>> Ireland
>>
>> On 8 Feb 2019, at 13:42, Eduardo Moguillansky  wrote:
>>
>> On 08.02.19 12:39, Victor Lazzarini wrote:
>>
>> The spinlocks are there to synchronise talk to hosts so that csound processing can run on its own thread outside the UI/message processing. Inside a Csound orchestra they have no use unless multicore is used, but I have also to look whether there are is also other synchronisation involved in that case.
>>
>> ok. Is there a #define I can test against to check if a multicore version is being built?
>>
>>
>> Are you planning to replace the bus channels? That does not sound like a good
>> idea. Maybe we need to think a little more
>> about it.
>>
>> No, I am not planning to replace the bus channels. I need an efficient / flexible way to communicate between notes. The bus channel opcodes don't allow to remove key value pairs and their efficiency degrades with size of the global hashtable. At the same time they are slow for the case where the string key changes frequently. This set of opcodes try to provide a hashtable which for these use cases is ~ 30x faster, for integer keys as fast as builtin arrays, for constant string keys as fast as chnget/set and offers the flexibility of being able to create and free hashtables, pass them around, delete key pairs, print the contents, etc. The implementation uses klib's khash (https://github.com/attractivechaos/klib). It is a proof of concept, and some testing is still needed:
>>
>> https://github.com/gesellkammer/ugens/blob/master/klib/klib.c
>>
>>
>> Victor Lazzarini
>> Dean of Arts, Celtic Studies, and Philosophy
>> Maynooth University
>> Ireland
>>
>> On 8 Feb 2019, at 09:56, Eduardo Moguillansky  wrote:
>>
>> The goal of these opcodes is to replace channels for the use case of inter-note communication. Looking at the code for channels I noticed that they have a lock per channel so I assumed that some sort of synchronisation was necessary to prevent data being corrupted if multiple notes are writing to the same hashtable.
>>
>> On 08.02.19 10:28, Victor Lazzarini wrote:
>>
>> Before we could look at the best solution, I just wanted to know why do you need to synchronise access, is that for multicore Csound?
>>
>> ________________________________
>> From: A discussion list for users of Csound  on behalf of Eduardo Moguillansky 
>> Sent: Friday 8 February 2019 00:38:43
>> To: CSOUND@LISTSERV.HEANET.IE
>> Subject: [Csnd] mutex / spinlock on opcodes
>>
>> Hi
>>
>> I am working on a set of opcodes implementing a hashtable. These
>> hashtables can be either local to a note or global. I have been looking
>> to other opcodes which access global data, such as the OSC opcodes.
>> These use a mutex to synchronize access to shared data. On the other
>> hand, the channel opcodes use spinlocks, but these are not implemented
>> in the opcode api. What should an opcode use? At the moment my implemen-
>> tation uses mutexes, as in this example below.
>>
>> static int32_t
>> hashtab_set_int_float(CSOUND *csound, HASHTAB_SET_if *p) {
>>       KHASH_GLOBALS *g = p->globals;
>>       int32_t idx = (int32_t)*p->ihandle;
>>       KHASH_HANDLE *handle = &(g->handles[idx]);
>>       khash_t(khIntFloat) *h = handle->khashptr;
>>       int32_t key = *p->key;
>>       khint_t k;
>>       csound->LockMutex(handle->mutex_);     // <---------------------
>>       kh_key(h, k) = key;
>>       kh_value(h, k) = *p->value;
>>       csound->UnlockMutex(handle->mutex_);   // <----------------------
>>       return OK;
>> }
>>
>> Best regards,
>> Eduardo
>>
>> Csound mailing list
>> Csound@listserv.heanet.ie
>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
>> Send bugs reports to
>>          https://github.com/csound/csound/issues
>> Discussions of bugs and features can be posted here
>> Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here
>>
>> Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here
>>
>> Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here
>>
>> Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here
>>
>> Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here
> Csound mailing list
> Csound@listserv.heanet.ie
> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
> Send bugs reports to
>          https://github.com/csound/csound/issues
> Discussions of bugs and features can be posted here

Csound mailing list
Csound@listserv.heanet.ie
https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
Send bugs reports to
        https://github.com/csound/csound/issues
Discussions of bugs and features can be posted here

i am not sure i got everything right from this discussion, but i 
sometimes miss some dictonary-like possibility in csound.  so eduardo's 
attempt seems to cover something which is not there yet.

but should this discussion not move to the developer list?

	joachim



On 08/02/19 18:07, Eduardo Moguillansky wrote:
> Regarding slowness and memory usage of chnset, you can look at this:
>
> https://github.com/gesellkammer/ugens/blob/master/klib/examples/dict_bench_set.csd
>
>
> Both instruments do the same, but instr 2 (chnset) takes more than 30x
> longer and uses significantly more memory. Regarding possible
> optimizations you can take a look at the implementation of dict_set
> (https://github.com/gesellkammer/ugens/blob/master/klib/klib.c,
> beginning at line 523) which makes some efforts to avoid a lookup if it
> can.
> It can also be that the underlying hashtable implementation in csound is
> not so efficient.
>
> Another problem with the chn opcodes is the impossibility of removing a
> key:value pair, or of creating a local/global hashtable which is
> independent of the global table used by all chn opcodes.
>
> Independently of how/if the chn opcodes are improved, maybe it would be
> possible to address the issue of having plugins outside of the main
> tree, similar to externals in pd or sc-plugins in supercollider or
> similar to a stdlib and external libraries in any other language.
>
> On 08.02.19 16:43, Steven Yi wrote:
>> +1 We should identify the cause of the slowdown in channels and
>> address that (hence why I asked about naming pattern in the problem
>> project so that we can test and profile).  If a problem is identified
>> then it'd benefit everyone to get that addressed in the channel
>> system.
>>
>> There's also a longstanding proposal to have generic lists, sets, and
>> maps for CS7 as data types.
>>
>> On Fri, Feb 8, 2019 at 8:47 AM Victor Lazzarini
>>  wrote:
>>>   Just a thought, can you use global arrays?
>>>
>>> Also the zak opcodes were designed for this kind of thing as well.
>>>
>>> It would be better to fix the chn opcodes rather than add similar but
>>> not quite similar
>>> functionality. This happened often in the past and is a known
>>> historical problem of
>>> Csound development.
>>>
>>> Steven, John, do you have any thoughts on this?
>>>
>>> Victor Lazzarini
>>> Dean of Arts, Celtic Studies, and Philosophy
>>> Maynooth University
>>> Ireland
>>>
>>> On 8 Feb 2019, at 13:42, Eduardo Moguillansky
>>>  wrote:
>>>
>>> On 08.02.19 12:39, Victor Lazzarini wrote:
>>>
>>> The spinlocks are there to synchronise talk to hosts so that csound
>>> processing can run on its own thread outside the UI/message
>>> processing. Inside a Csound orchestra they have no use unless
>>> multicore is used, but I have also to look whether there are is also
>>> other synchronisation involved in that case.
>>>
>>> ok. Is there a #define I can test against to check if a multicore
>>> version is being built?
>>>
>>>
>>> Are you planning to replace the bus channels? That does not sound
>>> like a good
>>> idea. Maybe we need to think a little more
>>> about it.
>>>
>>> No, I am not planning to replace the bus channels. I need an
>>> efficient / flexible way to communicate between notes. The bus
>>> channel opcodes don't allow to remove key value pairs and their
>>> efficiency degrades with size of the global hashtable. At the same
>>> time they are slow for the case where the string key changes
>>> frequently. This set of opcodes try to provide a hashtable which for
>>> these use cases is ~ 30x faster, for integer keys as fast as builtin
>>> arrays, for constant string keys as fast as chnget/set and offers the
>>> flexibility of being able to create and free hashtables, pass them
>>> around, delete key pairs, print the contents, etc. The implementation
>>> uses klib's khash (https://github.com/attractivechaos/klib). It is a
>>> proof of concept, and some testing is still needed:
>>>
>>> https://github.com/gesellkammer/ugens/blob/master/klib/klib.c
>>>
>>>
>>> Victor Lazzarini
>>> Dean of Arts, Celtic Studies, and Philosophy
>>> Maynooth University
>>> Ireland
>>>
>>> On 8 Feb 2019, at 09:56, Eduardo Moguillansky
>>>  wrote:
>>>
>>> The goal of these opcodes is to replace channels for the use case of
>>> inter-note communication. Looking at the code for channels I noticed
>>> that they have a lock per channel so I assumed that some sort of
>>> synchronisation was necessary to prevent data being corrupted if
>>> multiple notes are writing to the same hashtable.
>>>
>>> On 08.02.19 10:28, Victor Lazzarini wrote:
>>>
>>> Before we could look at the best solution, I just wanted to know why
>>> do you need to synchronise access, is that for multicore Csound?
>>>
>>> ________________________________
>>> From: A discussion list for users of Csound
>>>  on behalf of Eduardo Moguillansky
>>> 
>>> Sent: Friday 8 February 2019 00:38:43
>>> To: CSOUND@LISTSERV.HEANET.IE
>>> Subject: [Csnd] mutex / spinlock on opcodes
>>>
>>> Hi
>>>
>>> I am working on a set of opcodes implementing a hashtable. These
>>> hashtables can be either local to a note or global. I have been looking
>>> to other opcodes which access global data, such as the OSC opcodes.
>>> These use a mutex to synchronize access to shared data. On the other
>>> hand, the channel opcodes use spinlocks, but these are not implemented
>>> in the opcode api. What should an opcode use? At the moment my implemen-
>>> tation uses mutexes, as in this example below.
>>>
>>> static int32_t
>>> hashtab_set_int_float(CSOUND *csound, HASHTAB_SET_if *p) {
>>>       KHASH_GLOBALS *g = p->globals;
>>>       int32_t idx = (int32_t)*p->ihandle;
>>>       KHASH_HANDLE *handle = &(g->handles[idx]);
>>>       khash_t(khIntFloat) *h = handle->khashptr;
>>>       int32_t key = *p->key;
>>>       khint_t k;
>>>       csound->LockMutex(handle->mutex_);     // <---------------------
>>>       kh_key(h, k) = key;
>>>       kh_value(h, k) = *p->value;
>>>       csound->UnlockMutex(handle->mutex_);   // <----------------------
>>>       return OK;
>>> }
>>>
>>> Best regards,
>>> Eduardo
>>>
>>> Csound mailing list
>>> Csound@listserv.heanet.ie
>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
>>> Send bugs reports to
>>>          https://github.com/csound/csound/issues
>>> Discussions of bugs and features can be posted here
>>> Csound mailing list Csound@listserv.heanet.ie
>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
>>> https://github.com/csound/csound/issues Discussions of bugs and
>>> features can be posted here
>>>
>>> Csound mailing list Csound@listserv.heanet.ie
>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
>>> https://github.com/csound/csound/issues Discussions of bugs and
>>> features can be posted here
>>>
>>> Csound mailing list Csound@listserv.heanet.ie
>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
>>> https://github.com/csound/csound/issues Discussions of bugs and
>>> features can be posted here
>>>
>>> Csound mailing list Csound@listserv.heanet.ie
>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
>>> https://github.com/csound/csound/issues Discussions of bugs and
>>> features can be posted here
>>>
>>> Csound mailing list Csound@listserv.heanet.ie
>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
>>> https://github.com/csound/csound/issues Discussions of bugs and
>>> features can be posted here
>> Csound mailing list
>> Csound@listserv.heanet.ie
>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
>> Send bugs reports to
>>          https://github.com/csound/csound/issues
>> Discussions of bugs and features can be posted here
>
> Csound mailing list
> Csound@listserv.heanet.ie
> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
> Send bugs reports to
>        https://github.com/csound/csound/issues
> Discussions of bugs and features can be posted here
>

Csound mailing list
Csound@listserv.heanet.ie
https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
Send bugs reports to
        https://github.com/csound/csound/issues
Discussions of bugs and features can be posted here

yes, moving it here is a good idea. 

Steven has this planned for 7. The issue we have to face is to stop ad-hoc solutions to
local problems multiplying the number of
similar but not quite similar functionality.

To be clear: users and developers are free
to duplicate/customise/add anything they
want for their needs, that is why we have a
plugin mechanism. 

However, we are being very careful in curating the software, so such solutions 
may not be mainstreamed.

Victor Lazzarini
Dean of Arts, Celtic Studies, and Philosophy
Maynooth University
Ireland

> On 9 Feb 2019, at 09:48, joachim heintz  wrote:
> 
> i am not sure i got everything right from this discussion, but i sometimes miss some dictonary-like possibility in csound.  so eduardo's attempt seems to cover something which is not there yet.
> 
> but should this discussion not move to the developer list?
> 
>    joachim
> 
> 
> 
>> On 08/02/19 18:07, Eduardo Moguillansky wrote:
>> Regarding slowness and memory usage of chnset, you can look at this:
>> 
>> https://github.com/gesellkammer/ugens/blob/master/klib/examples/dict_bench_set.csd
>> 
>> 
>> Both instruments do the same, but instr 2 (chnset) takes more than 30x
>> longer and uses significantly more memory. Regarding possible
>> optimizations you can take a look at the implementation of dict_set
>> (https://github.com/gesellkammer/ugens/blob/master/klib/klib.c,
>> beginning at line 523) which makes some efforts to avoid a lookup if it
>> can.
>> It can also be that the underlying hashtable implementation in csound is
>> not so efficient.
>> 
>> Another problem with the chn opcodes is the impossibility of removing a
>> key:value pair, or of creating a local/global hashtable which is
>> independent of the global table used by all chn opcodes.
>> 
>> Independently of how/if the chn opcodes are improved, maybe it would be
>> possible to address the issue of having plugins outside of the main
>> tree, similar to externals in pd or sc-plugins in supercollider or
>> similar to a stdlib and external libraries in any other language.
>> 
>>> On 08.02.19 16:43, Steven Yi wrote:
>>> +1 We should identify the cause of the slowdown in channels and
>>> address that (hence why I asked about naming pattern in the problem
>>> project so that we can test and profile).  If a problem is identified
>>> then it'd benefit everyone to get that addressed in the channel
>>> system.
>>> 
>>> There's also a longstanding proposal to have generic lists, sets, and
>>> maps for CS7 as data types.
>>> 
>>> On Fri, Feb 8, 2019 at 8:47 AM Victor Lazzarini
>>>  wrote:
>>>>  Just a thought, can you use global arrays?
>>>> 
>>>> Also the zak opcodes were designed for this kind of thing as well.
>>>> 
>>>> It would be better to fix the chn opcodes rather than add similar but
>>>> not quite similar
>>>> functionality. This happened often in the past and is a known
>>>> historical problem of
>>>> Csound development.
>>>> 
>>>> Steven, John, do you have any thoughts on this?
>>>> 
>>>> Victor Lazzarini
>>>> Dean of Arts, Celtic Studies, and Philosophy
>>>> Maynooth University
>>>> Ireland
>>>> 
>>>> On 8 Feb 2019, at 13:42, Eduardo Moguillansky
>>>>  wrote:
>>>> 
>>>> On 08.02.19 12:39, Victor Lazzarini wrote:
>>>> 
>>>> The spinlocks are there to synchronise talk to hosts so that csound
>>>> processing can run on its own thread outside the UI/message
>>>> processing. Inside a Csound orchestra they have no use unless
>>>> multicore is used, but I have also to look whether there are is also
>>>> other synchronisation involved in that case.
>>>> 
>>>> ok. Is there a #define I can test against to check if a multicore
>>>> version is being built?
>>>> 
>>>> 
>>>> Are you planning to replace the bus channels? That does not sound
>>>> like a good
>>>> idea. Maybe we need to think a little more
>>>> about it.
>>>> 
>>>> No, I am not planning to replace the bus channels. I need an
>>>> efficient / flexible way to communicate between notes. The bus
>>>> channel opcodes don't allow to remove key value pairs and their
>>>> efficiency degrades with size of the global hashtable. At the same
>>>> time they are slow for the case where the string key changes
>>>> frequently. This set of opcodes try to provide a hashtable which for
>>>> these use cases is ~ 30x faster, for integer keys as fast as builtin
>>>> arrays, for constant string keys as fast as chnget/set and offers the
>>>> flexibility of being able to create and free hashtables, pass them
>>>> around, delete key pairs, print the contents, etc. The implementation
>>>> uses klib's khash (https://github.com/attractivechaos/klib). It is a
>>>> proof of concept, and some testing is still needed:
>>>> 
>>>> https://github.com/gesellkammer/ugens/blob/master/klib/klib.c
>>>> 
>>>> 
>>>> Victor Lazzarini
>>>> Dean of Arts, Celtic Studies, and Philosophy
>>>> Maynooth University
>>>> Ireland
>>>> 
>>>> On 8 Feb 2019, at 09:56, Eduardo Moguillansky
>>>>  wrote:
>>>> 
>>>> The goal of these opcodes is to replace channels for the use case of
>>>> inter-note communication. Looking at the code for channels I noticed
>>>> that they have a lock per channel so I assumed that some sort of
>>>> synchronisation was necessary to prevent data being corrupted if
>>>> multiple notes are writing to the same hashtable.
>>>> 
>>>> On 08.02.19 10:28, Victor Lazzarini wrote:
>>>> 
>>>> Before we could look at the best solution, I just wanted to know why
>>>> do you need to synchronise access, is that for multicore Csound?
>>>> 
>>>> ________________________________
>>>> From: A discussion list for users of Csound
>>>>  on behalf of Eduardo Moguillansky
>>>> 
>>>> Sent: Friday 8 February 2019 00:38:43
>>>> To: CSOUND@LISTSERV.HEANET.IE
>>>> Subject: [Csnd] mutex / spinlock on opcodes
>>>> 
>>>> Hi
>>>> 
>>>> I am working on a set of opcodes implementing a hashtable. These
>>>> hashtables can be either local to a note or global. I have been looking
>>>> to other opcodes which access global data, such as the OSC opcodes.
>>>> These use a mutex to synchronize access to shared data. On the other
>>>> hand, the channel opcodes use spinlocks, but these are not implemented
>>>> in the opcode api. What should an opcode use? At the moment my implemen-
>>>> tation uses mutexes, as in this example below.
>>>> 
>>>> static int32_t
>>>> hashtab_set_int_float(CSOUND *csound, HASHTAB_SET_if *p) {
>>>>      KHASH_GLOBALS *g = p->globals;
>>>>      int32_t idx = (int32_t)*p->ihandle;
>>>>      KHASH_HANDLE *handle = &(g->handles[idx]);
>>>>      khash_t(khIntFloat) *h = handle->khashptr;
>>>>      int32_t key = *p->key;
>>>>      khint_t k;
>>>>      csound->LockMutex(handle->mutex_);     // <---------------------
>>>>      kh_key(h, k) = key;
>>>>      kh_value(h, k) = *p->value;
>>>>      csound->UnlockMutex(handle->mutex_);   // <----------------------
>>>>      return OK;
>>>> }
>>>> 
>>>> Best regards,
>>>> Eduardo
>>>> 
>>>> Csound mailing list
>>>> Csound@listserv.heanet.ie
>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
>>>> Send bugs reports to
>>>>         https://github.com/csound/csound/issues
>>>> Discussions of bugs and features can be posted here
>>>> Csound mailing list Csound@listserv.heanet.ie
>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
>>>> https://github.com/csound/csound/issues Discussions of bugs and
>>>> features can be posted here
>>>> 
>>>> Csound mailing list Csound@listserv.heanet.ie
>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
>>>> https://github.com/csound/csound/issues Discussions of bugs and
>>>> features can be posted here
>>>> 
>>>> Csound mailing list Csound@listserv.heanet.ie
>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
>>>> https://github.com/csound/csound/issues Discussions of bugs and
>>>> features can be posted here
>>>> 
>>>> Csound mailing list Csound@listserv.heanet.ie
>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
>>>> https://github.com/csound/csound/issues Discussions of bugs and
>>>> features can be posted here
>>>> 
>>>> Csound mailing list Csound@listserv.heanet.ie
>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
>>>> https://github.com/csound/csound/issues Discussions of bugs and
>>>> features can be posted here
>>> Csound mailing list
>>> Csound@listserv.heanet.ie
>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
>>> Send bugs reports to
>>>         https://github.com/csound/csound/issues
>>> Discussions of bugs and features can be posted here
>> 
>> Csound mailing list
>> Csound@listserv.heanet.ie
>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
>> Send bugs reports to
>>       https://github.com/csound/csound/issues
>> Discussions of bugs and features can be posted here
>> 
> 
> Csound mailing list
> Csound@listserv.heanet.ie
> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
> Send bugs reports to
>       https://github.com/csound/csound/issues

I totally agree with the reluctance to include everything within the 
main tree. It would be great to have a second repository under the aegis 
of "csound" in github, into which plugins can be aggregated, with a more 
liberal policy regarding commit rights. There is still the problem of 
documentation, because external plugins could not be included in the 
docbook manual, and the manual, in its current version, can only be 
built as a monolith.

Anyway, this thread started only as a question regarding thread 
synchronisation - I never intended to ask for integration of these 
opcodes, much less to replace the chn ones!

Eduardo

On 09.02.19 11:15, Victor Lazzarini wrote:
> yes, moving it here is a good idea.
>
> Steven has this planned for 7. The issue we have to face is to stop ad-hoc solutions to
> local problems multiplying the number of
> similar but not quite similar functionality.
>
> To be clear: users and developers are free
> to duplicate/customise/add anything they
> want for their needs, that is why we have a
> plugin mechanism.
>
> However, we are being very careful in curating the software, so such solutions
> may not be mainstreamed.
>
> Victor Lazzarini
> Dean of Arts, Celtic Studies, and Philosophy
> Maynooth University
> Ireland
>
>> On 9 Feb 2019, at 09:48, joachim heintz  wrote:
>>
>> i am not sure i got everything right from this discussion, but i sometimes miss some dictonary-like possibility in csound.  so eduardo's attempt seems to cover something which is not there yet.
>>
>> but should this discussion not move to the developer list?
>>
>>     joachim
>>
>>
>>
>>> On 08/02/19 18:07, Eduardo Moguillansky wrote:
>>> Regarding slowness and memory usage of chnset, you can look at this:
>>>
>>> https://github.com/gesellkammer/ugens/blob/master/klib/examples/dict_bench_set.csd
>>>
>>>
>>> Both instruments do the same, but instr 2 (chnset) takes more than 30x
>>> longer and uses significantly more memory. Regarding possible
>>> optimizations you can take a look at the implementation of dict_set
>>> (https://github.com/gesellkammer/ugens/blob/master/klib/klib.c,
>>> beginning at line 523) which makes some efforts to avoid a lookup if it
>>> can.
>>> It can also be that the underlying hashtable implementation in csound is
>>> not so efficient.
>>>
>>> Another problem with the chn opcodes is the impossibility of removing a
>>> key:value pair, or of creating a local/global hashtable which is
>>> independent of the global table used by all chn opcodes.
>>>
>>> Independently of how/if the chn opcodes are improved, maybe it would be
>>> possible to address the issue of having plugins outside of the main
>>> tree, similar to externals in pd or sc-plugins in supercollider or
>>> similar to a stdlib and external libraries in any other language.
>>>
>>>> On 08.02.19 16:43, Steven Yi wrote:
>>>> +1 We should identify the cause of the slowdown in channels and
>>>> address that (hence why I asked about naming pattern in the problem
>>>> project so that we can test and profile).  If a problem is identified
>>>> then it'd benefit everyone to get that addressed in the channel
>>>> system.
>>>>
>>>> There's also a longstanding proposal to have generic lists, sets, and
>>>> maps for CS7 as data types.
>>>>
>>>> On Fri, Feb 8, 2019 at 8:47 AM Victor Lazzarini
>>>>  wrote:
>>>>>   Just a thought, can you use global arrays?
>>>>>
>>>>> Also the zak opcodes were designed for this kind of thing as well.
>>>>>
>>>>> It would be better to fix the chn opcodes rather than add similar but
>>>>> not quite similar
>>>>> functionality. This happened often in the past and is a known
>>>>> historical problem of
>>>>> Csound development.
>>>>>
>>>>> Steven, John, do you have any thoughts on this?
>>>>>
>>>>> Victor Lazzarini
>>>>> Dean of Arts, Celtic Studies, and Philosophy
>>>>> Maynooth University
>>>>> Ireland
>>>>>
>>>>> On 8 Feb 2019, at 13:42, Eduardo Moguillansky
>>>>>  wrote:
>>>>>
>>>>> On 08.02.19 12:39, Victor Lazzarini wrote:
>>>>>
>>>>> The spinlocks are there to synchronise talk to hosts so that csound
>>>>> processing can run on its own thread outside the UI/message
>>>>> processing. Inside a Csound orchestra they have no use unless
>>>>> multicore is used, but I have also to look whether there are is also
>>>>> other synchronisation involved in that case.
>>>>>
>>>>> ok. Is there a #define I can test against to check if a multicore
>>>>> version is being built?
>>>>>
>>>>>
>>>>> Are you planning to replace the bus channels? That does not sound
>>>>> like a good
>>>>> idea. Maybe we need to think a little more
>>>>> about it.
>>>>>
>>>>> No, I am not planning to replace the bus channels. I need an
>>>>> efficient / flexible way to communicate between notes. The bus
>>>>> channel opcodes don't allow to remove key value pairs and their
>>>>> efficiency degrades with size of the global hashtable. At the same
>>>>> time they are slow for the case where the string key changes
>>>>> frequently. This set of opcodes try to provide a hashtable which for
>>>>> these use cases is ~ 30x faster, for integer keys as fast as builtin
>>>>> arrays, for constant string keys as fast as chnget/set and offers the
>>>>> flexibility of being able to create and free hashtables, pass them
>>>>> around, delete key pairs, print the contents, etc. The implementation
>>>>> uses klib's khash (https://github.com/attractivechaos/klib). It is a
>>>>> proof of concept, and some testing is still needed:
>>>>>
>>>>> https://github.com/gesellkammer/ugens/blob/master/klib/klib.c
>>>>>
>>>>>
>>>>> Victor Lazzarini
>>>>> Dean of Arts, Celtic Studies, and Philosophy
>>>>> Maynooth University
>>>>> Ireland
>>>>>
>>>>> On 8 Feb 2019, at 09:56, Eduardo Moguillansky
>>>>>  wrote:
>>>>>
>>>>> The goal of these opcodes is to replace channels for the use case of
>>>>> inter-note communication. Looking at the code for channels I noticed
>>>>> that they have a lock per channel so I assumed that some sort of
>>>>> synchronisation was necessary to prevent data being corrupted if
>>>>> multiple notes are writing to the same hashtable.
>>>>>
>>>>> On 08.02.19 10:28, Victor Lazzarini wrote:
>>>>>
>>>>> Before we could look at the best solution, I just wanted to know why
>>>>> do you need to synchronise access, is that for multicore Csound?
>>>>>
>>>>> ________________________________
>>>>> From: A discussion list for users of Csound
>>>>>  on behalf of Eduardo Moguillansky
>>>>> 
>>>>> Sent: Friday 8 February 2019 00:38:43
>>>>> To: CSOUND@LISTSERV.HEANET.IE
>>>>> Subject: [Csnd] mutex / spinlock on opcodes
>>>>>
>>>>> Hi
>>>>>
>>>>> I am working on a set of opcodes implementing a hashtable. These
>>>>> hashtables can be either local to a note or global. I have been looking
>>>>> to other opcodes which access global data, such as the OSC opcodes.
>>>>> These use a mutex to synchronize access to shared data. On the other
>>>>> hand, the channel opcodes use spinlocks, but these are not implemented
>>>>> in the opcode api. What should an opcode use? At the moment my implemen-
>>>>> tation uses mutexes, as in this example below.
>>>>>
>>>>> static int32_t
>>>>> hashtab_set_int_float(CSOUND *csound, HASHTAB_SET_if *p) {
>>>>>       KHASH_GLOBALS *g = p->globals;
>>>>>       int32_t idx = (int32_t)*p->ihandle;
>>>>>       KHASH_HANDLE *handle = &(g->handles[idx]);
>>>>>       khash_t(khIntFloat) *h = handle->khashptr;
>>>>>       int32_t key = *p->key;
>>>>>       khint_t k;
>>>>>       csound->LockMutex(handle->mutex_);     // <---------------------
>>>>>       kh_key(h, k) = key;
>>>>>       kh_value(h, k) = *p->value;
>>>>>       csound->UnlockMutex(handle->mutex_);   // <----------------------
>>>>>       return OK;
>>>>> }
>>>>>
>>>>> Best regards,
>>>>> Eduardo
>>>>>
>>>>> Csound mailing list
>>>>> Csound@listserv.heanet.ie
>>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
>>>>> Send bugs reports to
>>>>>          https://github.com/csound/csound/issues
>>>>> Discussions of bugs and features can be posted here
>>>>> Csound mailing list Csound@listserv.heanet.ie
>>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
>>>>> https://github.com/csound/csound/issues Discussions of bugs and
>>>>> features can be posted here
>>>>>
>>>>> Csound mailing list Csound@listserv.heanet.ie
>>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
>>>>> https://github.com/csound/csound/issues Discussions of bugs and
>>>>> features can be posted here
>>>>>
>>>>> Csound mailing list Csound@listserv.heanet.ie
>>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
>>>>> https://github.com/csound/csound/issues Discussions of bugs and
>>>>> features can be posted here
>>>>>
>>>>> Csound mailing list Csound@listserv.heanet.ie
>>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
>>>>> https://github.com/csound/csound/issues Discussions of bugs and
>>>>> features can be posted here
>>>>>
>>>>> Csound mailing list Csound@listserv.heanet.ie
>>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
>>>>> https://github.com/csound/csound/issues Discussions of bugs and
>>>>> features can be posted here
>>>> Csound mailing list
>>>> Csound@listserv.heanet.ie
>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
>>>> Send bugs reports to
>>>>          https://github.com/csound/csound/issues
>>>> Discussions of bugs and features can be posted here
>>> Csound mailing list
>>> Csound@listserv.heanet.ie
>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
>>> Send bugs reports to
>>>        https://github.com/csound/csound/issues
>>> Discussions of bugs and features can be posted here
>>>
>> Csound mailing list
>> Csound@listserv.heanet.ie
>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
>> Send bugs reports to
>>        https://github.com/csound/csound/issues

(Answering prior email from user list before it got moved here)

Interesting, I don't think many many thousands of channels was a
condition ever factored into the design. I took a look and the channel
system code uses the cs hash table code which in turn has two main
deficiencies:

1. Hash function could be improved to be more uniform.
2. Hash table has a fixed size and does not resize.

The hash table has an array of buckets with length 4099. The code for
cs_hash_table was extracted by me from existing hash table code when
we moved to CS6  (there was a couple of separate hash table
implementations sprinkled into the code at that time) so that there
would be just one implementation to maintain.

The hash table we use in Csound should be redone/replaced to address
both of those issues.

Regarding 3rd party plugins: We had a long discussion about this after
the 2017 Csound Conference to have a package manager similar to PD's
deken and SC3's Quark system.  There was some buy-in but not much
volunteer time to implement since that discussion.

As for the plugins on their own, I think Rory recently setup an opcode
library github project that could build itself on major desktop
platforms using Appveyor and Travis.  (Rory: Is that right? Or did I
misunderstand?)


On Sat, Feb 9, 2019 at 8:59 AM Eduardo Moguillansky
 wrote:
>
> I totally agree with the reluctance to include everything within the
> main tree. It would be great to have a second repository under the aegis
> of "csound" in github, into which plugins can be aggregated, with a more
> liberal policy regarding commit rights. There is still the problem of
> documentation, because external plugins could not be included in the
> docbook manual, and the manual, in its current version, can only be
> built as a monolith.
>
> Anyway, this thread started only as a question regarding thread
> synchronisation - I never intended to ask for integration of these
> opcodes, much less to replace the chn ones!
>
> Eduardo
>
> On 09.02.19 11:15, Victor Lazzarini wrote:
> > yes, moving it here is a good idea.
> >
> > Steven has this planned for 7. The issue we have to face is to stop ad-hoc solutions to
> > local problems multiplying the number of
> > similar but not quite similar functionality.
> >
> > To be clear: users and developers are free
> > to duplicate/customise/add anything they
> > want for their needs, that is why we have a
> > plugin mechanism.
> >
> > However, we are being very careful in curating the software, so such solutions
> > may not be mainstreamed.
> >
> > Victor Lazzarini
> > Dean of Arts, Celtic Studies, and Philosophy
> > Maynooth University
> > Ireland
> >
> >> On 9 Feb 2019, at 09:48, joachim heintz  wrote:
> >>
> >> i am not sure i got everything right from this discussion, but i sometimes miss some dictonary-like possibility in csound.  so eduardo's attempt seems to cover something which is not there yet.
> >>
> >> but should this discussion not move to the developer list?
> >>
> >>     joachim
> >>
> >>
> >>
> >>> On 08/02/19 18:07, Eduardo Moguillansky wrote:
> >>> Regarding slowness and memory usage of chnset, you can look at this:
> >>>
> >>> https://github.com/gesellkammer/ugens/blob/master/klib/examples/dict_bench_set.csd
> >>>
> >>>
> >>> Both instruments do the same, but instr 2 (chnset) takes more than 30x
> >>> longer and uses significantly more memory. Regarding possible
> >>> optimizations you can take a look at the implementation of dict_set
> >>> (https://github.com/gesellkammer/ugens/blob/master/klib/klib.c,
> >>> beginning at line 523) which makes some efforts to avoid a lookup if it
> >>> can.
> >>> It can also be that the underlying hashtable implementation in csound is
> >>> not so efficient.
> >>>
> >>> Another problem with the chn opcodes is the impossibility of removing a
> >>> key:value pair, or of creating a local/global hashtable which is
> >>> independent of the global table used by all chn opcodes.
> >>>
> >>> Independently of how/if the chn opcodes are improved, maybe it would be
> >>> possible to address the issue of having plugins outside of the main
> >>> tree, similar to externals in pd or sc-plugins in supercollider or
> >>> similar to a stdlib and external libraries in any other language.
> >>>
> >>>> On 08.02.19 16:43, Steven Yi wrote:
> >>>> +1 We should identify the cause of the slowdown in channels and
> >>>> address that (hence why I asked about naming pattern in the problem
> >>>> project so that we can test and profile).  If a problem is identified
> >>>> then it'd benefit everyone to get that addressed in the channel
> >>>> system.
> >>>>
> >>>> There's also a longstanding proposal to have generic lists, sets, and
> >>>> maps for CS7 as data types.
> >>>>
> >>>> On Fri, Feb 8, 2019 at 8:47 AM Victor Lazzarini
> >>>>  wrote:
> >>>>>   Just a thought, can you use global arrays?
> >>>>>
> >>>>> Also the zak opcodes were designed for this kind of thing as well.
> >>>>>
> >>>>> It would be better to fix the chn opcodes rather than add similar but
> >>>>> not quite similar
> >>>>> functionality. This happened often in the past and is a known
> >>>>> historical problem of
> >>>>> Csound development.
> >>>>>
> >>>>> Steven, John, do you have any thoughts on this?
> >>>>>
> >>>>> Victor Lazzarini
> >>>>> Dean of Arts, Celtic Studies, and Philosophy
> >>>>> Maynooth University
> >>>>> Ireland
> >>>>>
> >>>>> On 8 Feb 2019, at 13:42, Eduardo Moguillansky
> >>>>>  wrote:
> >>>>>
> >>>>> On 08.02.19 12:39, Victor Lazzarini wrote:
> >>>>>
> >>>>> The spinlocks are there to synchronise talk to hosts so that csound
> >>>>> processing can run on its own thread outside the UI/message
> >>>>> processing. Inside a Csound orchestra they have no use unless
> >>>>> multicore is used, but I have also to look whether there are is also
> >>>>> other synchronisation involved in that case.
> >>>>>
> >>>>> ok. Is there a #define I can test against to check if a multicore
> >>>>> version is being built?
> >>>>>
> >>>>>
> >>>>> Are you planning to replace the bus channels? That does not sound
> >>>>> like a good
> >>>>> idea. Maybe we need to think a little more
> >>>>> about it.
> >>>>>
> >>>>> No, I am not planning to replace the bus channels. I need an
> >>>>> efficient / flexible way to communicate between notes. The bus
> >>>>> channel opcodes don't allow to remove key value pairs and their
> >>>>> efficiency degrades with size of the global hashtable. At the same
> >>>>> time they are slow for the case where the string key changes
> >>>>> frequently. This set of opcodes try to provide a hashtable which for
> >>>>> these use cases is ~ 30x faster, for integer keys as fast as builtin
> >>>>> arrays, for constant string keys as fast as chnget/set and offers the
> >>>>> flexibility of being able to create and free hashtables, pass them
> >>>>> around, delete key pairs, print the contents, etc. The implementation
> >>>>> uses klib's khash (https://github.com/attractivechaos/klib). It is a
> >>>>> proof of concept, and some testing is still needed:
> >>>>>
> >>>>> https://github.com/gesellkammer/ugens/blob/master/klib/klib.c
> >>>>>
> >>>>>
> >>>>> Victor Lazzarini
> >>>>> Dean of Arts, Celtic Studies, and Philosophy
> >>>>> Maynooth University
> >>>>> Ireland
> >>>>>
> >>>>> On 8 Feb 2019, at 09:56, Eduardo Moguillansky
> >>>>>  wrote:
> >>>>>
> >>>>> The goal of these opcodes is to replace channels for the use case of
> >>>>> inter-note communication. Looking at the code for channels I noticed
> >>>>> that they have a lock per channel so I assumed that some sort of
> >>>>> synchronisation was necessary to prevent data being corrupted if
> >>>>> multiple notes are writing to the same hashtable.
> >>>>>
> >>>>> On 08.02.19 10:28, Victor Lazzarini wrote:
> >>>>>
> >>>>> Before we could look at the best solution, I just wanted to know why
> >>>>> do you need to synchronise access, is that for multicore Csound?
> >>>>>
> >>>>> ________________________________
> >>>>> From: A discussion list for users of Csound
> >>>>>  on behalf of Eduardo Moguillansky
> >>>>> 
> >>>>> Sent: Friday 8 February 2019 00:38:43
> >>>>> To: CSOUND@LISTSERV.HEANET.IE
> >>>>> Subject: [Csnd] mutex / spinlock on opcodes
> >>>>>
> >>>>> Hi
> >>>>>
> >>>>> I am working on a set of opcodes implementing a hashtable. These
> >>>>> hashtables can be either local to a note or global. I have been looking
> >>>>> to other opcodes which access global data, such as the OSC opcodes.
> >>>>> These use a mutex to synchronize access to shared data. On the other
> >>>>> hand, the channel opcodes use spinlocks, but these are not implemented
> >>>>> in the opcode api. What should an opcode use? At the moment my implemen-
> >>>>> tation uses mutexes, as in this example below.
> >>>>>
> >>>>> static int32_t
> >>>>> hashtab_set_int_float(CSOUND *csound, HASHTAB_SET_if *p) {
> >>>>>       KHASH_GLOBALS *g = p->globals;
> >>>>>       int32_t idx = (int32_t)*p->ihandle;
> >>>>>       KHASH_HANDLE *handle = &(g->handles[idx]);
> >>>>>       khash_t(khIntFloat) *h = handle->khashptr;
> >>>>>       int32_t key = *p->key;
> >>>>>       khint_t k;
> >>>>>       csound->LockMutex(handle->mutex_);     // <---------------------
> >>>>>       kh_key(h, k) = key;
> >>>>>       kh_value(h, k) = *p->value;
> >>>>>       csound->UnlockMutex(handle->mutex_);   // <----------------------
> >>>>>       return OK;
> >>>>> }
> >>>>>
> >>>>> Best regards,
> >>>>> Eduardo
> >>>>>
> >>>>> Csound mailing list
> >>>>> Csound@listserv.heanet.ie
> >>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
> >>>>> Send bugs reports to
> >>>>>          https://github.com/csound/csound/issues
> >>>>> Discussions of bugs and features can be posted here
> >>>>> Csound mailing list Csound@listserv.heanet.ie
> >>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
> >>>>> https://github.com/csound/csound/issues Discussions of bugs and
> >>>>> features can be posted here
> >>>>>
> >>>>> Csound mailing list Csound@listserv.heanet.ie
> >>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
> >>>>> https://github.com/csound/csound/issues Discussions of bugs and
> >>>>> features can be posted here
> >>>>>
> >>>>> Csound mailing list Csound@listserv.heanet.ie
> >>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
> >>>>> https://github.com/csound/csound/issues Discussions of bugs and
> >>>>> features can be posted here
> >>>>>
> >>>>> Csound mailing list Csound@listserv.heanet.ie
> >>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
> >>>>> https://github.com/csound/csound/issues Discussions of bugs and
> >>>>> features can be posted here
> >>>>>
> >>>>> Csound mailing list Csound@listserv.heanet.ie
> >>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
> >>>>> https://github.com/csound/csound/issues Discussions of bugs and
> >>>>> features can be posted here
> >>>> Csound mailing list
> >>>> Csound@listserv.heanet.ie
> >>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
> >>>> Send bugs reports to
> >>>>          https://github.com/csound/csound/issues
> >>>> Discussions of bugs and features can be posted here
> >>> Csound mailing list
> >>> Csound@listserv.heanet.ie
> >>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
> >>> Send bugs reports to
> >>>        https://github.com/csound/csound/issues
> >>> Discussions of bugs and features can be posted here
> >>>
> >> Csound mailing list
> >> Csound@listserv.heanet.ie
> >> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
> >> Send bugs reports to
> >>        https://github.com/csound/csound/issues

> As for the plugins on their own, I think Rory recently setup an opcode
> library github project that could build itself on major desktop
> platforms using Appveyor and Travis.  (Rory: Is that right? Or did I
> misunderstand?)
I could not find such a project. Do you mean this?

https://github.com/rorywalsh/opcode_sdk

>
>
> On Sat, Feb 9, 2019 at 8:59 AM Eduardo Moguillansky
>  wrote:
>> I totally agree with the reluctance to include everything within the
>> main tree. It would be great to have a second repository under the aegis
>> of "csound" in github, into which plugins can be aggregated, with a more
>> liberal policy regarding commit rights. There is still the problem of
>> documentation, because external plugins could not be included in the
>> docbook manual, and the manual, in its current version, can only be
>> built as a monolith.
>>
>> Anyway, this thread started only as a question regarding thread
>> synchronisation - I never intended to ask for integration of these
>> opcodes, much less to replace the chn ones!
>>
>> Eduardo
>>
>> On 09.02.19 11:15, Victor Lazzarini wrote:
>>> yes, moving it here is a good idea.
>>>
>>> Steven has this planned for 7. The issue we have to face is to stop ad-hoc solutions to
>>> local problems multiplying the number of
>>> similar but not quite similar functionality.
>>>
>>> To be clear: users and developers are free
>>> to duplicate/customise/add anything they
>>> want for their needs, that is why we have a
>>> plugin mechanism.
>>>
>>> However, we are being very careful in curating the software, so such solutions
>>> may not be mainstreamed.
>>>
>>> Victor Lazzarini
>>> Dean of Arts, Celtic Studies, and Philosophy
>>> Maynooth University
>>> Ireland
>>>
>>>> On 9 Feb 2019, at 09:48, joachim heintz  wrote:
>>>>
>>>> i am not sure i got everything right from this discussion, but i sometimes miss some dictonary-like possibility in csound.  so eduardo's attempt seems to cover something which is not there yet.
>>>>
>>>> but should this discussion not move to the developer list?
>>>>
>>>>      joachim
>>>>
>>>>
>>>>
>>>>> On 08/02/19 18:07, Eduardo Moguillansky wrote:
>>>>> Regarding slowness and memory usage of chnset, you can look at this:
>>>>>
>>>>> https://github.com/gesellkammer/ugens/blob/master/klib/examples/dict_bench_set.csd
>>>>>
>>>>>
>>>>> Both instruments do the same, but instr 2 (chnset) takes more than 30x
>>>>> longer and uses significantly more memory. Regarding possible
>>>>> optimizations you can take a look at the implementation of dict_set
>>>>> (https://github.com/gesellkammer/ugens/blob/master/klib/klib.c,
>>>>> beginning at line 523) which makes some efforts to avoid a lookup if it
>>>>> can.
>>>>> It can also be that the underlying hashtable implementation in csound is
>>>>> not so efficient.
>>>>>
>>>>> Another problem with the chn opcodes is the impossibility of removing a
>>>>> key:value pair, or of creating a local/global hashtable which is
>>>>> independent of the global table used by all chn opcodes.
>>>>>
>>>>> Independently of how/if the chn opcodes are improved, maybe it would be
>>>>> possible to address the issue of having plugins outside of the main
>>>>> tree, similar to externals in pd or sc-plugins in supercollider or
>>>>> similar to a stdlib and external libraries in any other language.
>>>>>
>>>>>> On 08.02.19 16:43, Steven Yi wrote:
>>>>>> +1 We should identify the cause of the slowdown in channels and
>>>>>> address that (hence why I asked about naming pattern in the problem
>>>>>> project so that we can test and profile).  If a problem is identified
>>>>>> then it'd benefit everyone to get that addressed in the channel
>>>>>> system.
>>>>>>
>>>>>> There's also a longstanding proposal to have generic lists, sets, and
>>>>>> maps for CS7 as data types.
>>>>>>
>>>>>> On Fri, Feb 8, 2019 at 8:47 AM Victor Lazzarini
>>>>>>  wrote:
>>>>>>>    Just a thought, can you use global arrays?
>>>>>>>
>>>>>>> Also the zak opcodes were designed for this kind of thing as well.
>>>>>>>
>>>>>>> It would be better to fix the chn opcodes rather than add similar but
>>>>>>> not quite similar
>>>>>>> functionality. This happened often in the past and is a known
>>>>>>> historical problem of
>>>>>>> Csound development.
>>>>>>>
>>>>>>> Steven, John, do you have any thoughts on this?
>>>>>>>
>>>>>>> Victor Lazzarini
>>>>>>> Dean of Arts, Celtic Studies, and Philosophy
>>>>>>> Maynooth University
>>>>>>> Ireland
>>>>>>>
>>>>>>> On 8 Feb 2019, at 13:42, Eduardo Moguillansky
>>>>>>>  wrote:
>>>>>>>
>>>>>>> On 08.02.19 12:39, Victor Lazzarini wrote:
>>>>>>>
>>>>>>> The spinlocks are there to synchronise talk to hosts so that csound
>>>>>>> processing can run on its own thread outside the UI/message
>>>>>>> processing. Inside a Csound orchestra they have no use unless
>>>>>>> multicore is used, but I have also to look whether there are is also
>>>>>>> other synchronisation involved in that case.
>>>>>>>
>>>>>>> ok. Is there a #define I can test against to check if a multicore
>>>>>>> version is being built?
>>>>>>>
>>>>>>>
>>>>>>> Are you planning to replace the bus channels? That does not sound
>>>>>>> like a good
>>>>>>> idea. Maybe we need to think a little more
>>>>>>> about it.
>>>>>>>
>>>>>>> No, I am not planning to replace the bus channels. I need an
>>>>>>> efficient / flexible way to communicate between notes. The bus
>>>>>>> channel opcodes don't allow to remove key value pairs and their
>>>>>>> efficiency degrades with size of the global hashtable. At the same
>>>>>>> time they are slow for the case where the string key changes
>>>>>>> frequently. This set of opcodes try to provide a hashtable which for
>>>>>>> these use cases is ~ 30x faster, for integer keys as fast as builtin
>>>>>>> arrays, for constant string keys as fast as chnget/set and offers the
>>>>>>> flexibility of being able to create and free hashtables, pass them
>>>>>>> around, delete key pairs, print the contents, etc. The implementation
>>>>>>> uses klib's khash (https://github.com/attractivechaos/klib). It is a
>>>>>>> proof of concept, and some testing is still needed:
>>>>>>>
>>>>>>> https://github.com/gesellkammer/ugens/blob/master/klib/klib.c
>>>>>>>
>>>>>>>
>>>>>>> Victor Lazzarini
>>>>>>> Dean of Arts, Celtic Studies, and Philosophy
>>>>>>> Maynooth University
>>>>>>> Ireland
>>>>>>>
>>>>>>> On 8 Feb 2019, at 09:56, Eduardo Moguillansky
>>>>>>>  wrote:
>>>>>>>
>>>>>>> The goal of these opcodes is to replace channels for the use case of
>>>>>>> inter-note communication. Looking at the code for channels I noticed
>>>>>>> that they have a lock per channel so I assumed that some sort of
>>>>>>> synchronisation was necessary to prevent data being corrupted if
>>>>>>> multiple notes are writing to the same hashtable.
>>>>>>>
>>>>>>> On 08.02.19 10:28, Victor Lazzarini wrote:
>>>>>>>
>>>>>>> Before we could look at the best solution, I just wanted to know why
>>>>>>> do you need to synchronise access, is that for multicore Csound?
>>>>>>>
>>>>>>> ________________________________
>>>>>>> From: A discussion list for users of Csound
>>>>>>>  on behalf of Eduardo Moguillansky
>>>>>>> 
>>>>>>> Sent: Friday 8 February 2019 00:38:43
>>>>>>> To: CSOUND@LISTSERV.HEANET.IE
>>>>>>> Subject: [Csnd] mutex / spinlock on opcodes
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> I am working on a set of opcodes implementing a hashtable. These
>>>>>>> hashtables can be either local to a note or global. I have been looking
>>>>>>> to other opcodes which access global data, such as the OSC opcodes.
>>>>>>> These use a mutex to synchronize access to shared data. On the other
>>>>>>> hand, the channel opcodes use spinlocks, but these are not implemented
>>>>>>> in the opcode api. What should an opcode use? At the moment my implemen-
>>>>>>> tation uses mutexes, as in this example below.
>>>>>>>
>>>>>>> static int32_t
>>>>>>> hashtab_set_int_float(CSOUND *csound, HASHTAB_SET_if *p) {
>>>>>>>        KHASH_GLOBALS *g = p->globals;
>>>>>>>        int32_t idx = (int32_t)*p->ihandle;
>>>>>>>        KHASH_HANDLE *handle = &(g->handles[idx]);
>>>>>>>        khash_t(khIntFloat) *h = handle->khashptr;
>>>>>>>        int32_t key = *p->key;
>>>>>>>        khint_t k;
>>>>>>>        csound->LockMutex(handle->mutex_);     // <---------------------
>>>>>>>        kh_key(h, k) = key;
>>>>>>>        kh_value(h, k) = *p->value;
>>>>>>>        csound->UnlockMutex(handle->mutex_);   // <----------------------
>>>>>>>        return OK;
>>>>>>> }
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Eduardo
>>>>>>>
>>>>>>> Csound mailing list
>>>>>>> Csound@listserv.heanet.ie
>>>>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
>>>>>>> Send bugs reports to
>>>>>>>           https://github.com/csound/csound/issues
>>>>>>> Discussions of bugs and features can be posted here
>>>>>>> Csound mailing list Csound@listserv.heanet.ie
>>>>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
>>>>>>> https://github.com/csound/csound/issues Discussions of bugs and
>>>>>>> features can be posted here
>>>>>>>
>>>>>>> Csound mailing list Csound@listserv.heanet.ie
>>>>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
>>>>>>> https://github.com/csound/csound/issues Discussions of bugs and
>>>>>>> features can be posted here
>>>>>>>
>>>>>>> Csound mailing list Csound@listserv.heanet.ie
>>>>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
>>>>>>> https://github.com/csound/csound/issues Discussions of bugs and
>>>>>>> features can be posted here
>>>>>>>
>>>>>>> Csound mailing list Csound@listserv.heanet.ie
>>>>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
>>>>>>> https://github.com/csound/csound/issues Discussions of bugs and
>>>>>>> features can be posted here
>>>>>>>
>>>>>>> Csound mailing list Csound@listserv.heanet.ie
>>>>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to
>>>>>>> https://github.com/csound/csound/issues Discussions of bugs and
>>>>>>> features can be posted here
>>>>>> Csound mailing list
>>>>>> Csound@listserv.heanet.ie
>>>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
>>>>>> Send bugs reports to
>>>>>>           https://github.com/csound/csound/issues
>>>>>> Discussions of bugs and features can be posted here
>>>>> Csound mailing list
>>>>> Csound@listserv.heanet.ie
>>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
>>>>> Send bugs reports to
>>>>>         https://github.com/csound/csound/issues
>>>>> Discussions of bugs and features can be posted here
>>>>>
>>>> Csound mailing list
>>>> Csound@listserv.heanet.ie
>>>> https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
>>>> Send bugs reports to
>>>>         https://github.com/csound/csound/issues