Hi All,

I managed to get through an initial implementation of pass-by-ref for
UDOs. Took a while to do analyses of the code to get to this point,
but hopes are that it should be quicker to iterate upon from here with
this base implementation.

If you're interested to try it out, the branch is:

feature/pass-by-ref

Some notes on this implementation:

1. I moved UDO-related code out from insert.c/.h to a separate udo.c/.h file.
2. I implemented this only for new-style UDOs. There was discussion
about making this something we could control on a per-argument basis,
but starting with this simplified what had to get figured out. I also
wonder if it wouldn't be a good idea to simply go with this, since it
makes UDOs have the same characteristics of pass-by-ref as native
opcodes. It would also be safe to do this for backwards compatibility
and keep pass-by-value for old-style UDOs.
3. I added a new ##userOpcode2 entry that gets used for new-style
UDOs. It does not have any perf-time code. useropcdset_newstyle
handles ref finding and setting with the following algorithm:

* 1. Iterate through the init chain to find xin/xout opcodes.
* 2. When found, lookup names of input and output variables for this
current UDO instance.
* 3. Setup a map of input/output variable names to passed-in arg pointers
* 4. Iterate through the init chain to find references to xin/xout
vars and set the pointers to the input/out variable argument pointer
addresses.
* 5. Iterate through the perf chain to find references to xin/xout
vars and set the pointers to the input/out variable argument pointer
addresses.

xinset and xoutset were also modified to skip copying of UDO in/out
args to/from IOBUFS when inside a new-style UDO.

At the moment, all csd tests are passing including the new test I
added while doing dev. There's a number of additional tests to add to
ensure this works with UDOs calling UDOs, structs, etc. There's also
performance testing to do to see how well this affects things like
passing f-sigs and arrays into UDOs. There's also some known issues
with returning constants from UDOs and some other arg types I have to
check to see if they are working or not. Also, there is a bigger issue
of what to do for local ksmps (will likely revert to pass-by-value for
short-term) as well as how to handle oversampling (i.e., local sr).

I'm pretty excited to get past this initial implementation. Feedback
and any tests are welcome and appreciated.

Thanks!
Steven



*Warning*

This email originated from outside of Maynooth University's Mail System. Do not reply, click links or open attachments unless you recognise the sender and know the content is safe.

This is great. Regarding passing by ref for all arguments, does that not change the semantics of opcodes? I for sure would not expect that modifying a k- variable inside an opcode could change the value outside it

Cheers, Eduardo 

On Mon, Feb 19, 2024, 21:43 Steven Yi <stevenyi@gmail.com> wrote:

Hi All,

I managed to get through an initial implementation of pass-by-ref for
UDOs. Took a while to do analyses of the code to get to this point,
but hopes are that it should be quicker to iterate upon from here with
this base implementation.

If you're interested to try it out, the branch is:

feature/pass-by-ref

Some notes on this implementation:

1. I moved UDO-related code out from insert.c/.h to a separate udo.c/.h file.
2. I implemented this only for new-style UDOs. There was discussion
about making this something we could control on a per-argument basis,
but starting with this simplified what had to get figured out. I also
wonder if it wouldn't be a good idea to simply go with this, since it
makes UDOs have the same characteristics of pass-by-ref as native
opcodes. It would also be safe to do this for backwards compatibility
and keep pass-by-value for old-style UDOs.
3. I added a new ##userOpcode2 entry that gets used for new-style
UDOs. It does not have any perf-time code. useropcdset_newstyle
handles ref finding and setting with the following algorithm:

* 1. Iterate through the init chain to find xin/xout opcodes.
* 2. When found, lookup names of input and output variables for this
current UDO instance.
* 3. Setup a map of input/output variable names to passed-in arg pointers
* 4. Iterate through the init chain to find references to xin/xout
vars and set the pointers to the input/out variable argument pointer
addresses.
* 5. Iterate through the perf chain to find references to xin/xout
vars and set the pointers to the input/out variable argument pointer
addresses.

xinset and xoutset were also modified to skip copying of UDO in/out
args to/from IOBUFS when inside a new-style UDO.

At the moment, all csd tests are passing including the new test I
added while doing dev. There's a number of additional tests to add to
ensure this works with UDOs calling UDOs, structs, etc. There's also
performance testing to do to see how well this affects things like
passing f-sigs and arrays into UDOs. There's also some known issues
with returning constants from UDOs and some other arg types I have to
check to see if they are working or not. Also, there is a bigger issue
of what to do for local ksmps (will likely revert to pass-by-value for
short-term) as well as how to handle oversampling (i.e., local sr).

I'm pretty excited to get past this initial implementation. Feedback
and any tests are welcome and appreciated.

Thanks!
Steven

It's a good question, but currently native opcodes can change values
for both input and output arguments, so this puts it on par with
native opcodes. It's by convention that most native opcodes do not
modify input args, though there are examples of some that do such as
many array and table operations, or vaset (which is used to write into
asigs with array notation). Users would certainly have to be careful,
but I think it's the price to pay for performance. We can try to help
the situation upon initial release by documenting heavily and making
sure early code examples have clear notes in comments.

On Mon, Feb 19, 2024 at 4:22 PM Eduardo Moguillansky
 wrote:
>
> This is great. Regarding passing by ref for all arguments, does that not change the semantics of opcodes? I for sure would not expect that modifying a k- variable inside an opcode could change the value outside it
>
> Cheers, Eduardo
>
> On Mon, Feb 19, 2024, 21:43 Steven Yi  wrote:
>>
>> Hi All,
>>
>> I managed to get through an initial implementation of pass-by-ref for
>> UDOs. Took a while to do analyses of the code to get to this point,
>> but hopes are that it should be quicker to iterate upon from here with
>> this base implementation.
>>
>> If you're interested to try it out, the branch is:
>>
>> feature/pass-by-ref
>>
>> Some notes on this implementation:
>>
>> 1. I moved UDO-related code out from insert.c/.h to a separate udo.c/.h file.
>> 2. I implemented this only for new-style UDOs. There was discussion
>> about making this something we could control on a per-argument basis,
>> but starting with this simplified what had to get figured out. I also
>> wonder if it wouldn't be a good idea to simply go with this, since it
>> makes UDOs have the same characteristics of pass-by-ref as native
>> opcodes. It would also be safe to do this for backwards compatibility
>> and keep pass-by-value for old-style UDOs.
>> 3. I added a new ##userOpcode2 entry that gets used for new-style
>> UDOs. It does not have any perf-time code. useropcdset_newstyle
>> handles ref finding and setting with the following algorithm:
>>
>> * 1. Iterate through the init chain to find xin/xout opcodes.
>> * 2. When found, lookup names of input and output variables for this
>> current UDO instance.
>> * 3. Setup a map of input/output variable names to passed-in arg pointers
>> * 4. Iterate through the init chain to find references to xin/xout
>> vars and set the pointers to the input/out variable argument pointer
>> addresses.
>> * 5. Iterate through the perf chain to find references to xin/xout
>> vars and set the pointers to the input/out variable argument pointer
>> addresses.
>>
>> xinset and xoutset were also modified to skip copying of UDO in/out
>> args to/from IOBUFS when inside a new-style UDO.
>>
>> At the moment, all csd tests are passing including the new test I
>> added while doing dev. There's a number of additional tests to add to
>> ensure this works with UDOs calling UDOs, structs, etc. There's also
>> performance testing to do to see how well this affects things like
>> passing f-sigs and arrays into UDOs. There's also some known issues
>> with returning constants from UDOs and some other arg types I have to
>> check to see if they are working or not. Also, there is a bigger issue
>> of what to do for local ksmps (will likely revert to pass-by-value for
>> short-term) as well as how to handle oversampling (i.e., local sr).
>>
>> I'm pretty excited to get past this initial implementation. Feedback
>> and any tests are welcome and appreciated.
>>
>> Thanks!
>> Steven

The difficulty with read-only modifiers for pass-by-reference is we
don't always know what native opcodes do with input arguments. We
could catch lines of code that attempt to use read-only signals as
outputs at the syntax level though.

My initial reaction is that this might take a lot of work, perhaps
adding a const or readonly field to CS_VARIABLE, integrating this into
the type system and checking, and figuring out how to deal with this
in native opcodes. However, maybe it won't be as much work as I think?
For the moment, I'm going to focus on getting the pass-by-ref
implementation working and I think we can discuss and track
readonly/const as a separate feature to implement as they're really
dependent upon each other. (For example, even without UDOs, users
might want to mark certain variables readonly or const within an
instrument.)

On Mon, Feb 19, 2024 at 4:34 PM Victor Lazzarini  wrote:
>
> I was thinking about this earlier, and I guess there's two ways to take this
>
> (1) opcodes can modify left hand side variables (to use them as outputs). So allowing this would make sense for udos.
>
> (2) it may be dangerous, so might want to enforce read-only pass by reference
>
>
> Prof. Victor Lazzarini
> Maynooth University
> Ireland
>
> On 19 Feb 2024, at 21:22, Eduardo Moguillansky  wrote:
>
> 
> *Warning* This email originated from outside of Maynooth University's Mail System. Do not reply, click links or open attachments unless you recognise the sender and know the content is safe.
> This is great. Regarding passing by ref for all arguments, does that not change the semantics of opcodes? I for sure would not expect that modifying a k- variable inside an opcode could change the value outside it
>
> Cheers, Eduardo
>
> On Mon, Feb 19, 2024, 21:43 Steven Yi  wrote:
>>
>> Hi All,
>>
>> I managed to get through an initial implementation of pass-by-ref for
>> UDOs. Took a while to do analyses of the code to get to this point,
>> but hopes are that it should be quicker to iterate upon from here with
>> this base implementation.
>>
>> If you're interested to try it out, the branch is:
>>
>> feature/pass-by-ref
>>
>> Some notes on this implementation:
>>
>> 1. I moved UDO-related code out from insert.c/.h to a separate udo.c/.h file.
>> 2. I implemented this only for new-style UDOs. There was discussion
>> about making this something we could control on a per-argument basis,
>> but starting with this simplified what had to get figured out. I also
>> wonder if it wouldn't be a good idea to simply go with this, since it
>> makes UDOs have the same characteristics of pass-by-ref as native
>> opcodes. It would also be safe to do this for backwards compatibility
>> and keep pass-by-value for old-style UDOs.
>> 3. I added a new ##userOpcode2 entry that gets used for new-style
>> UDOs. It does not have any perf-time code. useropcdset_newstyle
>> handles ref finding and setting with the following algorithm:
>>
>> * 1. Iterate through the init chain to find xin/xout opcodes.
>> * 2. When found, lookup names of input and output variables for this
>> current UDO instance.
>> * 3. Setup a map of input/output variable names to passed-in arg pointers
>> * 4. Iterate through the init chain to find references to xin/xout
>> vars and set the pointers to the input/out variable argument pointer
>> addresses.
>> * 5. Iterate through the perf chain to find references to xin/xout
>> vars and set the pointers to the input/out variable argument pointer
>> addresses.
>>
>> xinset and xoutset were also modified to skip copying of UDO in/out
>> args to/from IOBUFS when inside a new-style UDO.
>>
>> At the moment, all csd tests are passing including the new test I
>> added while doing dev. There's a number of additional tests to add to
>> ensure this works with UDOs calling UDOs, structs, etc. There's also
>> performance testing to do to see how well this affects things like
>> passing f-sigs and arrays into UDOs. There's also some known issues
>> with returning constants from UDOs and some other arg types I have to
>> check to see if they are working or not. Also, there is a bigger issue
>> of what to do for local ksmps (will likely revert to pass-by-value for
>> short-term) as well as how to handle oversampling (i.e., local sr).
>>
>> I'm pretty excited to get past this initial implementation. Feedback
>> and any tests are welcome and appreciated.
>>
>> Thanks!
>> Steven