[Csnd-dev] Array corrections

[Csnd-dev] Array corrections

Date	2019-04-25 19:38
From	Eduardo Moguillansky
Subject	[Csnd-dev] Array corrections
	Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. OSClisten kk[] iSS OSCraw S[]k i abs k[] k[] bformdec1 a[] ia[] bformenc1 a[] akk bpf k[] k[]M bpfcos k[] k[]M c2r k[] k[] cbrt k[] k[] ceil k[] k[] ceps k[] k[]k cepsinv k[] k[] cmp k[] kSk[]Sk cmp k[] k[]Sk cmp k[] k[]Sk[] cmplxprod k[] k[]k[] cos k[] k[] cosh k[] k[] cosinv k[] k[] dct k[] k[] dctinv k[] k[] deinterleave k[]k[] k[] directory S[] SN diskin a[] SPooooooo diskin a[] iPooooooo diskin2 a[] SPooooooo diskin2 a[] iPooooooo exp k[] k[] fft k[] k[] fftinv k[] k[] fillarray S[] W fillarray k[] S fillarray k[] m fillarray k[] z floor k[] k[] fluidInfo S[] i fmax k[] k[]k fmax k[] k[]k[] fmin k[] k[]k fmin k[] k[]k[] fmod k[] k[]k fmod k[] k[]k[] frac k[] k[] ftom k[] k[]o genarray k[] kkp genarray_i k[] iip getcol k[] k[]k getrow k[] k[]k getrowlin k[] k[]kOOP getrowlin k[] kiiooop hypot k[] k[]k[] in a[] (null) in a[] (null) init .[] m inletv a[] S int k[] k[] interleave k[] k[]k[] limit k[] k[]kk limit1 k[] k[] linlin k[] k[]kkOP linlin k[] kk[]k[]OP log k[] k[] log k[] k[]i log10 k[] k[] log2 k[] k[] loscilx a[] xkioojjoo mags k[] k[] maparray k[] k[]S mfb k[] k[]kki monitor a[] (null) mtof k[] k[] passign k[] po phs k[] k[] pol2rect k[] k[] pol2rect k[] k[]k[] pow k[] k[]k pow k[] k[]k[] powoftwo k[] k[] pows k[] k[] pvsceps k[] fo pvstrace fk[] fko r2c k[] k[] rect2pol k[] k[] rfft k[] k[] rifft k[] k[] round k[] k[] setcol k[] k[]k setrow k[] k[]k shiftin k[] a sin k[] k[] sinh k[] k[] sininv k[] k[] slicearray S[] S[]iip slicearray a[] a[]iip slicearray k[] k[]iip sorta k[] k[] sortd k[] k[] sqrt k[] k[] string2array k[] S tab2array k[] iOOP tan k[] k[] tanh k[] k[] taninv k[] k[] taninv k[] k[]k[] unwrap k[] k[] vbap a[] akOOo vbapg k[] kOOo vbapgmove k[] iiim vbapmove a[] aiiim window k[] k[]Op

Date	2019-04-25 20:50
From	Victor Lazzarini
Subject	Re: [Csnd-dev] Array corrections
	I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe operation where memory allocation can only happen at init-time on a separate thread. If this is a solution to a problem, then it’s not the right solution. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 > On 25 Apr 2019, at 19:38, Eduardo Moguillansky wrote: > > Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. > > OSClisten kk[] iSS > OSCraw S[]k i > abs k[] k[] > bformdec1 a[] ia[] > bformenc1 a[] akk > bpf k[] k[]M > bpfcos k[] k[]M > c2r k[] k[] > cbrt k[] k[] > ceil k[] k[] > ceps k[] k[]k > cepsinv k[] k[] > cmp k[] kSk[]Sk > cmp k[] k[]Sk > cmp k[] k[]Sk[] > cmplxprod k[] k[]k[] > cos k[] k[] > cosh k[] k[] > cosinv k[] k[] > dct k[] k[] > dctinv k[] k[] > deinterleave k[]k[] k[] > directory S[] SN > diskin a[] SPooooooo > diskin a[] iPooooooo > diskin2 a[] SPooooooo > diskin2 a[] iPooooooo > exp k[] k[] > fft k[] k[] > fftinv k[] k[] > fillarray S[] W > fillarray k[] S > fillarray k[] m > fillarray k[] z > floor k[] k[] > fluidInfo S[] i > fmax k[] k[]k > fmax k[] k[]k[] > fmin k[] k[]k > fmin k[] k[]k[] > fmod k[] k[]k > fmod k[] k[]k[] > frac k[] k[] > ftom k[] k[]o > genarray k[] kkp > genarray_i k[] iip > getcol k[] k[]k > getrow k[] k[]k > getrowlin k[] k[]kOOP > getrowlin k[] kiiooop > hypot k[] k[]k[] > in a[] (null) > in a[] (null) > init .[] m > inletv a[] S > int k[] k[] > interleave k[] k[]k[] > limit k[] k[]kk > limit1 k[] k[] > linlin k[] k[]kkOP > linlin k[] kk[]k[]OP > log k[] k[] > log k[] k[]i > log10 k[] k[] > log2 k[] k[] > loscilx a[] xkioojjoo > mags k[] k[] > maparray k[] k[]S > mfb k[] k[]kki > monitor a[] (null) > mtof k[] k[] > passign k[] po > phs k[] k[] > pol2rect k[] k[] > pol2rect k[] k[]k[] > pow k[] k[]k > pow k[] k[]k[] > powoftwo k[] k[] > pows k[] k[] > pvsceps k[] fo > pvstrace fk[] fko > r2c k[] k[] > rect2pol k[] k[] > rfft k[] k[] > rifft k[] k[] > round k[] k[] > setcol k[] k[]k > setrow k[] k[]k > shiftin k[] a > sin k[] k[] > sinh k[] k[] > sininv k[] k[] > slicearray S[] S[]iip > slicearray a[] a[]iip > slicearray k[] k[]iip > sorta k[] k[] > sortd k[] k[] > sqrt k[] k[] > string2array k[] S > tab2array k[] iOOP > tan k[] k[] > tanh k[] k[] > taninv k[] k[] > taninv k[] k[]k[] > unwrap k[] k[] > vbap a[] akOOo > vbapg k[] kOOo > vbapgmove k[] iiim > vbapmove a[] aiiim > window k[]

Date	2019-04-25 22:20
From	Eduardo Moguillansky
Subject	Re: [Csnd-dev] Array corrections
	If this is a requirement, then the array semantics need to be redefined and made clear. There can't be any operation at perf time which can grow an array. Even further, all perf time array operation need to operate with initialized arrays, since that would mean a perf time allocation. Following this semantics, we could go back to two tabensure functions in the form: // this should be called at init time by any opcode returning an array // even if the opcode only works at perf-time. static inline void tabensure_init(CSOUND csound, ARRAYDAT p, int size) { size_t ss = p->arrayMemberSizesize; if (p->data == NULL) { CS_VARIABLE var = p->arrayType->createVariable(csound, NULL); p->arrayMemberSize = var->memBlockSize; p->data = (MYFLT)csound->Calloc(csound, ss); p->allocated = ss; } else if (ss > p->allocated) { p->data = (MYFLT)csound->ReAlloc(csound, p->data, ss); p->allocated = ss; } p->sizes[0] = size; p->dimensions = 1; } // this should be called at perf time static inline void tabensure_perf(CSOUND csound, ARRAYDAT p, int size) { if(p->data == NULL) { return csound->PerformanceError(...); } size_t ss = p->arrayMemberSize*size; if(ss > p->allocated) { return csound->PerformanceError(...); } p->sizes[0] = size; } With this scheme, all allocation and resizing would happen at init time and any attempt to resize at perf time would result in an error. The user then can preallocate arrays as big as needed: kMyArray[] init i_largestsize possible Would that be an acceptable solution? On 25.04.19 21:50, Victor Lazzarini wrote: I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe operation where memory allocation can only happen at init-time on a separate thread. If this is a solution to a problem, then it’s not the right solution. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 On 25 Apr 2019, at 19:38, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. OSClisten kk[] iSS OSCraw S[]k i abs k[] k[] bformdec1 a[] ia[] bformenc1 a[] akk bpf k[] k[]M bpfcos k[] k[]M c2r k[] k[] cbrt k[] k[] ceil k[] k[] ceps k[] k[]k cepsinv k[] k[] cmp k[] kSk[]Sk cmp k[] k[]Sk cmp k[] k[]Sk[] cmplxprod k[] k[]k[] cos k[] k[] cosh k[] k[] cosinv k[] k[] dct k[] k[] dctinv k[] k[] deinterleave k[]k[] k[] directory S[] SN diskin a[] SPooooooo diskin a[] iPooooooo diskin2 a[] SPooooooo diskin2 a[] iPooooooo exp k[] k[] fft k[] k[] fftinv k[] k[] fillarray S[] W fillarray k[] S fillarray k[] m fillarray k[] z floor k[] k[] fluidInfo S[] i fmax k[] k[]k fmax k[] k[]k[] fmin k[] k[]k fmin k[] k[]k[] fmod k[] k[]k fmod k[] k[]k[] frac k[] k[] ftom k[] k[]o genarray k[] kkp genarray_i k[] iip getcol k[] k[]k getrow k[] k[]k getrowlin k[] k[]kOOP getrowlin k[] kiiooop hypot k[] k[]k[] in a[] (null) in a[] (null) init .[] m inletv a[] S int k[] k[] interleave k[] k[]k[] limit k[] k[]kk limit1 k[] k[] linlin k[] k[]kkOP linlin k[] kk[]k[]OP log k[] k[] log k[] k[]i log10 k[] k[] log2 k[] k[] loscilx a[] xkioojjoo mags k[] k[] maparray k[] k[]S mfb k[] k[]kki monitor a[] (null) mtof k[] k[] passign k[] po phs k[] k[] pol2rect k[] k[] pol2rect k[] k[]k[] pow k[] k[]k pow k[] k[]k[] powoftwo k[] k[] pows k[] k[] pvsceps k[] fo pvstrace fk[] fko r2c k[] k[] rect2pol k[] k[] rfft k[] k[] rifft k[] k[] round k[] k[] setcol k[] k[]k setrow k[] k[]k shiftin k[] a sin k[] k[] sinh k[] k[] sininv k[] k[] slicearray S[] S[]iip slicearray a[] a[]iip slicearray k[] k[]iip sorta k[] k[] sortd k[] k[] sqrt k[] k[] string2array k[] S tab2array k[] iOOP tan k[] k[] tanh k[] k[] taninv k[] k[] taninv k[] k[]k[] unwrap k[] k[] vbap a[] akOOo vbapg k[] kOOo vbapgmove k[] iiim vbapmove a[] aiiim window k[] k[]Op

Date	2019-04-25 22:33
From	Victor Lazzarini
Subject	Re: [Csnd-dev] Array corrections
	Sounds like the correct thing to me. As with other dynamic memory uses by opcodes, we only allocate or reallocate at init-time (e.g. we don't change delay line max sizes at perf time). Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 25 Apr 2019, at 22:21, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: If this is a requirement, then the array semantics need to be redefined and made clear. There can't be any operation at perf time which can grow an array. Even further, all perf time array operation need to operate with initialized arrays, since that would mean a perf time allocation. Following this semantics, we could go back to two tabensure functions in the form: // this should be called at init time by any opcode returning an array // even if the opcode only works at perf-time. static inline void tabensure_init(CSOUND csound, ARRAYDAT p, int size) { size_t ss = p->arrayMemberSizesize; if (p->data == NULL) { CS_VARIABLE var = p->arrayType->createVariable(csound, NULL); p->arrayMemberSize = var->memBlockSize; p->data = (MYFLT)csound->Calloc(csound, ss); p->allocated = ss; } else if (ss > p->allocated) { p->data = (MYFLT)csound->ReAlloc(csound, p->data, ss); p->allocated = ss; } p->sizes[0] = size; p->dimensions = 1; } // this should be called at perf time static inline void tabensure_perf(CSOUND csound, ARRAYDAT p, int size) { if(p->data == NULL) { return csound->PerformanceError(...); } size_t ss = p->arrayMemberSize*size; if(ss > p->allocated) { return csound->PerformanceError(...); } p->sizes[0] = size; } With this scheme, all allocation and resizing would happen at init time and any attempt to resize at perf time would result in an error. The user then can preallocate arrays as big as needed: kMyArray[] init i_largestsize possible Would that be an acceptable solution? On 25.04.19 21:50, Victor Lazzarini wrote: I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe operation where memory allocation can only happen at init-time on a separate thread. If this is a solution to a problem, then it’s not the right solution. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 On 25 Apr 2019, at 19:38, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. OSClisten kk[] iSS OSCraw S[]k i abs k[] k[] bformdec1 a[] ia[] bformenc1 a[] akk bpf k[] k[]M bpfcos k[] k[]M c2r k[] k[] cbrt k[] k[] ceil k[] k[] ceps k[] k[]k cepsinv k[] k[] cmp k[] kSk[]Sk cmp k[] k[]Sk cmp k[] k[]Sk[] cmplxprod k[] k[]k[] cos k[] k[] cosh k[] k[] cosinv k[] k[] dct k[] k[] dctinv k[] k[] deinterleave k[]k[] k[] directory S[] SN diskin a[] SPooooooo diskin a[] iPooooooo diskin2 a[] SPooooooo diskin2 a[] iPooooooo exp k[] k[] fft k[] k[] fftinv k[] k[] fillarray S[] W fillarray k[] S fillarray k[] m fillarray k[] z floor k[] k[] fluidInfo S[] i fmax k[] k[]k fmax k[] k[]k[] fmin k[] k[]k fmin k[] k[]k[] fmod k[] k[]k fmod k[] k[]k[] frac k[] k[] ftom k[] k[]o genarray k[] kkp genarray_i k[] iip getcol k[] k[]k getrow k[] k[]k getrowlin k[] k[]kOOP getrowlin k[] kiiooop hypot k[] k[]k[] in a[] (null) in a[] (null) init .[] m inletv a[] S int k[] k[] interleave k[] k[]k[] limit k[] k[]kk limit1 k[] k[] linlin k[] k[]kkOP linlin k[] kk[]k[]OP log k[] k[] log k[] k[]i log10 k[] k[] log2 k[] k[] loscilx a[] xkioojjoo mags k[] k[] maparray k[] k[]S mfb k[] k[]kki monitor a[] (null) mtof k[] k[] passign k[] po phs k[] k[] pol2rect k[] k[] pol2rect k[] k[]k[] pow k[] k[]k pow k[] k[]k[] powoftwo k[] k[] pows k[] k[] pvsceps k[] fo pvstrace fk[] fko r2c k[] k[] rect2pol k[] k[] rfft k[] k[] rifft k[] k[] round k[] k[] setcol k[] k[]k setrow k[] k[]k shiftin k[] a sin k[] k[] sinh k[] k[] sininv k[] k[] slicearray S[] S[]iip slicearray a[] a[]iip slicearray k[] k[]iip sorta k[] k[] sortd k[] k[] sqrt k[] k[] string2array k[] S tab2array k[] iOOP tan k[] k[] tanh k[] k[] taninv k[] k[] taninv k[] k[]k[] unwrap k[] k[] vbap a[] akOOo vbapg k[] kOOo vbapgmove k[] iiim vbapmove a[] aiiim window k[] k[]Op

Date	2019-04-25 23:13
From	Steven Yi
Subject	Re: [Csnd-dev] Array corrections
	The code change for tabensure (the one in the array-init-change branch) seems good to me. If the problem is that it should not be called at perf-time, then existing opcodes have problems and need to be rewritten. I'd rather we not have tabensure_perf and just document "only use at init-time". On Thu, Apr 25, 2019 at 5:33 PM Victor Lazzarini <Victor.Lazzarini@mu.ie> wrote: Sounds like the correct thing to me. As with other dynamic memory uses by opcodes, we only allocate or reallocate at init-time (e.g. we don't change delay line max sizes at perf time). Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 25 Apr 2019, at 22:21, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: If this is a requirement, then the array semantics need to be redefined and made clear. There can't be any operation at perf time which can grow an array. Even further, all perf time array operation need to operate with initialized arrays, since that would mean a perf time allocation. Following this semantics, we could go back to two tabensure functions in the form: // this should be called at init time by any opcode returning an array // even if the opcode only works at perf-time. static inline void tabensure_init(CSOUND csound, ARRAYDAT p, int size) { size_t ss = p->arrayMemberSizesize; if (p->data == NULL) { CS_VARIABLE var = p->arrayType->createVariable(csound, NULL); p->arrayMemberSize = var->memBlockSize; p->data = (MYFLT)csound->Calloc(csound, ss); p->allocated = ss; } else if (ss > p->allocated) { p->data = (MYFLT)csound->ReAlloc(csound, p->data, ss); p->allocated = ss; } p->sizes[0] = size; p->dimensions = 1; } // this should be called at perf time static inline void tabensure_perf(CSOUND csound, ARRAYDAT p, int size) { if(p->data == NULL) { return csound->PerformanceError(...); } size_t ss = p->arrayMemberSize*size; if(ss > p->allocated) { return csound->PerformanceError(...); } p->sizes[0] = size; } With this scheme, all allocation and resizing would happen at init time and any attempt to resize at perf time would result in an error. The user then can preallocate arrays as big as needed: kMyArray[] init i_largestsize possible Would that be an acceptable solution? On 25.04.19 21:50, Victor Lazzarini wrote: I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe operation where memory allocation can only happen at init-time on a separate thread. If this is a solution to a problem, then it’s not the right solution. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 On 25 Apr 2019, at 19:38, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. OSClisten kk[] iSS OSCraw S[]k i abs k[] k[] bformdec1 a[] ia[] bformenc1 a[] akk bpf k[] k[]M bpfcos k[] k[]M c2r k[] k[] cbrt k[] k[] ceil k[] k[] ceps k[] k[]k cepsinv k[] k[] cmp k[] kSk[]Sk cmp k[] k[]Sk cmp k[] k[]Sk[] cmplxprod k[] k[]k[] cos k[] k[] cosh k[] k[] cosinv k[] k[] dct k[] k[] dctinv k[] k[] deinterleave k[]k[] k[] directory S[] SN diskin a[] SPooooooo diskin a[] iPooooooo diskin2 a[] SPooooooo diskin2 a[] iPooooooo exp k[] k[] fft k[] k[] fftinv k[] k[] fillarray S[] W fillarray k[] S fillarray k[] m fillarray k[] z floor k[] k[] fluidInfo S[] i fmax k[] k[]k fmax k[] k[]k[] fmin k[] k[]k fmin k[] k[]k[] fmod k[] k[]k fmod k[] k[]k[] frac k[] k[] ftom k[] k[]o genarray k[] kkp genarray_i k[] iip getcol k[] k[]k getrow k[] k[]k getrowlin k[] k[]kOOP getrowlin k[] kiiooop hypot k[] k[]k[] in a[] (null) in a[] (null) init .[] m inletv a[] S int k[] k[] interleave k[] k[]k[] limit k[] k[]kk limit1 k[] k[] linlin k[] k[]kkOP linlin k[] kk[]k[]OP log k[] k[] log k[] k[]i log10 k[] k[] log2 k[] k[] loscilx a[] xkioojjoo mags k[] k[] maparray k[] k[]S mfb k[] k[]kki monitor a[] (null) mtof k[] k[] passign k[] po phs k[] k[] pol2rect k[] k[] pol2rect k[] k[]k[] pow k[] k[]k pow k[] k[]k[] powoftwo k[] k[] pows k[] k[] pvsceps k[] fo pvstrace fk[] fko r2c k[] k[] rect2pol k[] k[] rfft k[] k[] rifft k[] k[] round k[] k[] setcol k[] k[]k setrow k[] k[]k shiftin k[] a sin k[] k[] sinh k[] k[] sininv k[] k[] slicearray S[] S[]iip slicearray a[] a[]iip slicearray k[] k[]iip sorta k[] k[] sortd k[] k[] sqrt k[] k[] string2array k[] S tab2array k[] iOOP tan k[] k[] tanh k[] k[] taninv k[] k[] taninv k[] k[]k[] unwrap k[] k[] vbap a[] akOOo vbapg k[] kOOo vbapgmove k[] iiim vbapmove a[] aiiim window k[] k[]Op

Date	2019-04-25 23:30
From	Victor Lazzarini
Subject	Re: [Csnd-dev] Array corrections
	As long as calloc and friends are not called at perf time, I am happy with any fix you feel is the right one. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 25 Apr 2019, at 23:14, Steven Yi <stevenyi@gmail.com> wrote: The code change for tabensure (the one in the array-init-change branch) seems good to me. If the problem is that it should not be called at perf-time, then existing opcodes have problems and need to be rewritten. I'd rather we not have tabensure_perf and just document "only use at init-time". On Thu, Apr 25, 2019 at 5:33 PM Victor Lazzarini <Victor.Lazzarini@mu.ie> wrote: Sounds like the correct thing to me. As with other dynamic memory uses by opcodes, we only allocate or reallocate at init-time (e.g. we don't change delay line max sizes at perf time). Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 25 Apr 2019, at 22:21, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: If this is a requirement, then the array semantics need to be redefined and made clear. There can't be any operation at perf time which can grow an array. Even further, all perf time array operation need to operate with initialized arrays, since that would mean a perf time allocation. Following this semantics, we could go back to two tabensure functions in the form: // this should be called at init time by any opcode returning an array // even if the opcode only works at perf-time. static inline void tabensure_init(CSOUND csound, ARRAYDAT p, int size) { size_t ss = p->arrayMemberSizesize; if (p->data == NULL) { CS_VARIABLE var = p->arrayType->createVariable(csound, NULL); p->arrayMemberSize = var->memBlockSize; p->data = (MYFLT)csound->Calloc(csound, ss); p->allocated = ss; } else if (ss > p->allocated) { p->data = (MYFLT)csound->ReAlloc(csound, p->data, ss); p->allocated = ss; } p->sizes[0] = size; p->dimensions = 1; } // this should be called at perf time static inline void tabensure_perf(CSOUND csound, ARRAYDAT p, int size) { if(p->data == NULL) { return csound->PerformanceError(...); } size_t ss = p->arrayMemberSize*size; if(ss > p->allocated) { return csound->PerformanceError(...); } p->sizes[0] = size; } With this scheme, all allocation and resizing would happen at init time and any attempt to resize at perf time would result in an error. The user then can preallocate arrays as big as needed: kMyArray[] init i_largestsize possible Would that be an acceptable solution? On 25.04.19 21:50, Victor Lazzarini wrote: I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe operation where memory allocation can only happen at init-time on a separate thread. If this is a solution to a problem, then it’s not the right solution. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 On 25 Apr 2019, at 19:38, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. OSClisten kk[] iSS OSCraw S[]k i abs k[] k[] bformdec1 a[] ia[] bformenc1 a[] akk bpf k[] k[]M bpfcos k[] k[]M c2r k[] k[] cbrt k[] k[] ceil k[] k[] ceps k[] k[]k cepsinv k[] k[] cmp k[] kSk[]Sk cmp k[] k[]Sk cmp k[] k[]Sk[] cmplxprod k[] k[]k[] cos k[] k[] cosh k[] k[] cosinv k[] k[] dct k[] k[] dctinv k[] k[] deinterleave k[]k[] k[] directory S[] SN diskin a[] SPooooooo diskin a[] iPooooooo diskin2 a[] SPooooooo diskin2 a[] iPooooooo exp k[] k[] fft k[] k[] fftinv k[] k[] fillarray S[] W fillarray k[] S fillarray k[] m fillarray k[] z floor k[] k[] fluidInfo S[] i fmax k[] k[]k fmax k[] k[]k[] fmin k[] k[]k fmin k[] k[]k[] fmod k[] k[]k fmod k[] k[]k[] frac k[] k[] ftom k[] k[]o genarray k[] kkp genarray_i k[] iip getcol k[] k[]k getrow k[] k[]k getrowlin k[] k[]kOOP getrowlin k[] kiiooop hypot k[] k[]k[] in a[] (null) in a[] (null) init .[] m inletv a[] S int k[] k[] interleave k[] k[]k[] limit k[] k[]kk limit1 k[] k[] linlin k[] k[]kkOP linlin k[] kk[]k[]OP log k[] k[] log k[] k[]i log10 k[] k[] log2 k[] k[] loscilx a[] xkioojjoo mags k[] k[] maparray k[] k[]S mfb k[] k[]kki monitor a[] (null) mtof k[] k[] passign k[] po phs k[] k[] pol2rect k[] k[] pol2rect k[] k[]k[] pow k[] k[]k pow k[] k[]k[] powoftwo k[] k[] pows k[] k[] pvsceps k[] fo pvstrace fk[] fko r2c k[] k[] rect2pol k[] k[] rfft k[] k[] rifft k[] k[] round k[] k[] setcol k[] k[]k setrow k[] k[]k shiftin k[] a sin k[] k[] sinh k[] k[] sininv k[] k[] slicearray S[] S[]iip slicearray a[] a[]iip slicearray k[] k[]iip sorta k[] k[] sortd k[] k[] sqrt k[] k[] string2array k[] S tab2array k[] iOOP tan k[] k[] tanh k[] k[] taninv k[] k[] taninv k[] k[]k[] unwrap k[] k[] vbap a[] akOOo vbapg k[] kOOo vbapgmove k[] iiim vbapmove a[] aiiim window k[] k[]Op

Date	2019-04-25 23:41
From	Eduardo Moguillansky
Subject	Re: [Csnd-dev] Array corrections
	Given the requirement that there can't be allocation at perf-time, the semantics for arrays should be: * an array has an allocated size and a perceived size * the allocated size is given at init time and can't be modified after that * the output of any array operation should either result in an array of the expected size or in an error. This needs to be enfor * for element wise operations, like arithmetic between an array and a scalar, the output array should be the same size / shape as the input array, or result in a performance error if this can't be achieved If there is no agreement in the semantics, no code can be correct. There are two issues here: 1) array growth. With the implementation in the array-init-change branch, arrays can be reallocated at perf-time if they need to grow beyond allocated size. According to Victor this needs to be prevented in order to ensure realtime safety. That means that tabensure, as written there, can't be called at perf-time. 2) correct perceived size. At perf-time it must be ensured that an array has the correct perceived size, as long as this size is <= allocated size. This operation does not imply any memory allocation. With my proposed implementation all these constraints are satisfied. Any other solution is either semantically flawed (the currect situation, where no update of perceived size is done, size depends on allocated size, even on previous size in previous instances of the instrument, etc) or does not meet the required conditions (realtime safety) On 26.04.19 00:13, Steven Yi wrote: The code change for tabensure (the one in the array-init-change branch) seems good to me. If the problem is that it should not be called at perf-time, then existing opcodes have problems and need to be rewritten. I'd rather we not have tabensure_perf and just document "only use at init-time". On Thu, Apr 25, 2019 at 5:33 PM Victor Lazzarini <Victor.Lazzarini@mu.ie> wrote: Sounds like the correct thing to me. As with other dynamic memory uses by opcodes, we only allocate or reallocate at init-time (e.g. we don't change delay line max sizes at perf time). Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 25 Apr 2019, at 22:21, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: If this is a requirement, then the array semantics need to be redefined and made clear. There can't be any operation at perf time which can grow an array. Even further, all perf time array operation need to operate with initialized arrays, since that would mean a perf time allocation. Following this semantics, we could go back to two tabensure functions in the form: // this should be called at init time by any opcode returning an array // even if the opcode only works at perf-time. static inline void tabensure_init(CSOUND csound, ARRAYDAT p, int size) { size_t ss = p->arrayMemberSizesize; if (p->data == NULL) { CS_VARIABLE var = p->arrayType->createVariable(csound, NULL); p->arrayMemberSize = var->memBlockSize; p->data = (MYFLT)csound->Calloc(csound, ss); p->allocated = ss; } else if (ss > p->allocated) { p->data = (MYFLT)csound->ReAlloc(csound, p->data, ss); p->allocated = ss; } p->sizes[0] = size; p->dimensions = 1; } // this should be called at perf time static inline void tabensure_perf(CSOUND csound, ARRAYDAT p, int size) { if(p->data == NULL) { return csound->PerformanceError(...); } size_t ss = p->arrayMemberSize*size; if(ss > p->allocated) { return csound->PerformanceError(...); } p->sizes[0] = size; } With this scheme, all allocation and resizing would happen at init time and any attempt to resize at perf time would result in an error. The user then can preallocate arrays as big as needed: kMyArray[] init i_largestsize possible Would that be an acceptable solution? On 25.04.19 21:50, Victor Lazzarini wrote: I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe operation where memory allocation can only happen at init-time on a separate thread. If this is a solution to a problem, then it’s not the right solution. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 On 25 Apr 2019, at 19:38, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. OSClisten kk[] iSS OSCraw S[]k i abs k[] k[] bformdec1 a[] ia[] bformenc1 a[] akk bpf k[] k[]M bpfcos k[] k[]M c2r k[] k[] cbrt k[] k[] ceil k[] k[] ceps k[] k[]k cepsinv k[] k[] cmp k[] kSk[]Sk cmp k[] k[]Sk cmp k[] k[]Sk[] cmplxprod k[] k[]k[] cos k[] k[] cosh k[] k[] cosinv k[] k[] dct k[] k[] dctinv k[] k[] deinterleave k[]k[] k[] directory S[] SN diskin a[] SPooooooo diskin a[] iPooooooo diskin2 a[] SPooooooo diskin2 a[] iPooooooo exp k[] k[] fft k[] k[] fftinv k[] k[] fillarray S[] W fillarray k[] S fillarray k[] m fillarray k[] z floor k[] k[] fluidInfo S[] i fmax k[] k[]k fmax k[] k[]k[] fmin k[] k[]k fmin k[] k[]k[] fmod k[] k[]k fmod k[] k[]k[] frac k[] k[] ftom k[] k[]o genarray k[] kkp genarray_i k[] iip getcol k[] k[]k getrow k[] k[]k getrowlin k[] k[]kOOP getrowlin k[] kiiooop hypot k[] k[]k[] in a[] (null) in a[] (null) init .[] m inletv a[] S int k[] k[] interleave k[] k[]k[] limit k[] k[]kk limit1 k[] k[] linlin k[] k[]kkOP linlin k[] kk[]k[]OP log k[] k[] log k[] k[]i log10 k[] k[] log2 k[] k[] loscilx a[] xkioojjoo mags k[] k[] maparray k[] k[]S mfb k[] k[]kki monitor a[] (null) mtof k[] k[] passign k[] po phs k[] k[] pol2rect k[] k[] pol2rect k[] k[]k[] pow k[] k[]k pow k[] k[]k[] powoftwo k[] k[] pows k[] k[] pvsceps k[] fo pvstrace fk[] fko r2c k[] k[] rect2pol k[] k[] rfft k[] k[] rifft k[] k[] round k[] k[] setcol k[] k[]k setrow k[] k[]k shiftin k[] a sin k[] k[] sinh k[] k[] sininv k[] k[] slicearray S[] S[]iip slicearray a[] a[]iip slicearray k[] k[]iip sorta k[] k[] sortd k[] k[] sqrt k[] k[] string2array k[] S tab2array k[] iOOP tan k[] k[] tanh k[] k[] taninv k[] k[] taninv k[] k[]k[] unwrap k[] k[] vbap a[] akOOo vbapg k[] kOOo vbapgmove k[] iiim vbapmove a[] aiiim window k[] k[]Op

Date	2019-04-25 23:57
From	Steven Yi
Subject	Re: [Csnd-dev] Array corrections
	I was thinking through this and realized even if we say array sizes can only change at init-time, we may still need to set sizes at runtime. It dawned on me that the situation occurs if you reuse an array var: karr[] fillarray 1,2,3 ...1. do something... karr = fillarray 5,6,7,8 ... 2. do something ... so that in code section 1, karr should have a size of 3 at runtime, while code section 2, karr would have a size of 4. So it seems back to two tabensures, one at init and one for perf. On Thu, Apr 25, 2019 at 6:41 PM Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Given the requirement that there can't be allocation at perf-time, the semantics for arrays should be: * an array has an allocated size and a perceived size * the allocated size is given at init time and can't be modified after that * the output of any array operation should either result in an array of the expected size or in an error. This needs to be enfor * for element wise operations, like arithmetic between an array and a scalar, the output array should be the same size / shape as the input array, or result in a performance error if this can't be achieved If there is no agreement in the semantics, no code can be correct. There are two issues here: 1) array growth. With the implementation in the array-init-change branch, arrays can be reallocated at perf-time if they need to grow beyond allocated size. According to Victor this needs to be prevented in order to ensure realtime safety. That means that tabensure, as written there, can't be called at perf-time. 2) correct perceived size. At perf-time it must be ensured that an array has the correct perceived size, as long as this size is <= allocated size. This operation does not imply any memory allocation. With my proposed implementation all these constraints are satisfied. Any other solution is either semantically flawed (the currect situation, where no update of perceived size is done, size depends on allocated size, even on previous size in previous instances of the instrument, etc) or does not meet the required conditions (realtime safety) On 26.04.19 00:13, Steven Yi wrote: The code change for tabensure (the one in the array-init-change branch) seems good to me. If the problem is that it should not be called at perf-time, then existing opcodes have problems and need to be rewritten. I'd rather we not have tabensure_perf and just document "only use at init-time". On Thu, Apr 25, 2019 at 5:33 PM Victor Lazzarini <Victor.Lazzarini@mu.ie> wrote: Sounds like the correct thing to me. As with other dynamic memory uses by opcodes, we only allocate or reallocate at init-time (e.g. we don't change delay line max sizes at perf time). Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 25 Apr 2019, at 22:21, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: If this is a requirement, then the array semantics need to be redefined and made clear. There can't be any operation at perf time which can grow an array. Even further, all perf time array operation need to operate with initialized arrays, since that would mean a perf time allocation. Following this semantics, we could go back to two tabensure functions in the form: // this should be called at init time by any opcode returning an array // even if the opcode only works at perf-time. static inline void tabensure_init(CSOUND csound, ARRAYDAT p, int size) { size_t ss = p->arrayMemberSizesize; if (p->data == NULL) { CS_VARIABLE var = p->arrayType->createVariable(csound, NULL); p->arrayMemberSize = var->memBlockSize; p->data = (MYFLT)csound->Calloc(csound, ss); p->allocated = ss; } else if (ss > p->allocated) { p->data = (MYFLT)csound->ReAlloc(csound, p->data, ss); p->allocated = ss; } p->sizes[0] = size; p->dimensions = 1; } // this should be called at perf time static inline void tabensure_perf(CSOUND csound, ARRAYDAT p, int size) { if(p->data == NULL) { return csound->PerformanceError(...); } size_t ss = p->arrayMemberSize*size; if(ss > p->allocated) { return csound->PerformanceError(...); } p->sizes[0] = size; } With this scheme, all allocation and resizing would happen at init time and any attempt to resize at perf time would result in an error. The user then can preallocate arrays as big as needed: kMyArray[] init i_largestsize possible Would that be an acceptable solution? On 25.04.19 21:50, Victor Lazzarini wrote: I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe operation where memory allocation can only happen at init-time on a separate thread. If this is a solution to a problem, then it’s not the right solution. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 On 25 Apr 2019, at 19:38, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. OSClisten kk[] iSS OSCraw S[]k i abs k[] k[] bformdec1 a[] ia[] bformenc1 a[] akk bpf k[] k[]M bpfcos k[] k[]M c2r k[] k[] cbrt k[] k[] ceil k[] k[] ceps k[] k[]k cepsinv k[] k[] cmp k[] kSk[]Sk cmp k[] k[]Sk cmp k[] k[]Sk[] cmplxprod k[] k[]k[] cos k[] k[] cosh k[] k[] cosinv k[] k[] dct k[] k[] dctinv k[] k[] deinterleave k[]k[] k[] directory S[] SN diskin a[] SPooooooo diskin a[] iPooooooo diskin2 a[] SPooooooo diskin2 a[] iPooooooo exp k[] k[] fft k[] k[] fftinv k[] k[] fillarray S[] W fillarray k[] S fillarray k[] m fillarray k[] z floor k[] k[] fluidInfo S[] i fmax k[] k[]k fmax k[] k[]k[] fmin k[] k[]k fmin k[] k[]k[] fmod k[] k[]k fmod k[] k[]k[] frac k[] k[] ftom k[] k[]o genarray k[] kkp genarray_i k[] iip getcol k[] k[]k getrow k[] k[]k getrowlin k[] k[]kOOP getrowlin k[] kiiooop hypot k[] k[]k[] in a[] (null) in a[] (null) init .[] m inletv a[] S int k[] k[] interleave k[] k[]k[] limit k[] k[]kk limit1 k[] k[] linlin k[] k[]kkOP linlin k[] kk[]k[]OP log k[] k[] log k[] k[]i log10 k[] k[] log2 k[] k[] loscilx a[] xkioojjoo mags k[] k[] maparray k[] k[]S mfb k[] k[]kki monitor a[] (null) mtof k[] k[] passign k[] po phs k[] k[] pol2rect k[] k[] pol2rect k[] k[]k[] pow k[] k[]k pow k[] k[]k[] powoftwo k[] k[] pows k[] k[] pvsceps k[] fo pvstrace fk[] fko r2c k[] k[] rect2pol k[] k[] rfft k[] k[] rifft k[] k[] round k[] k[] setcol k[] k[]k setrow k[] k[]k shiftin k[] a sin k[] k[] sinh k[] k[] sininv k[] k[] slicearray S[] S[]iip slicearray a[] a[]iip slicearray k[] k[]iip sorta k[] k[] sortd k[] k[] sqrt k[] k[] string2array k[] S tab2array k[] iOOP tan k[] k[] tanh k[] k[] taninv k[] k[] taninv k[] k[]k[] unwrap k[] k[] vbap a[] akOOo vbapg k[] kOOo vbapgmove k[] iiim vbapmove a[] aiiim window k[] k[]Op

Date	2019-04-26 15:42
From	Mauro Giubileo
Subject	Re: [Csnd-dev] Array corrections
	I don't know the Bela platform, but I'm curious... Exactly, why on the Bela platform you should not do a memory allocation at perf-time? What is different compared to a memory allocation made on a standard PC? --- Mauro Il 2019-04-25 21:50 Victor Lazzarini ha scritto: I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe operation where memory allocation can only happen at init-time on a separate thread. If this is a solution to a problem, then it's not the right solution. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 On 25 Apr 2019, at 19:38, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. OSClisten kk[] iSS OSCraw S[]k i abs k[] k[] bformdec1 a[] ia[] bformenc1 a[] akk bpf k[] k[]M bpfcos k[] k[]M c2r k[] k[] cbrt k[] k[] ceil k[] k[] ceps k[] k[]k cepsinv k[] k[] cmp k[] kSk[]Sk cmp k[] k[]Sk cmp k[] k[]Sk[] cmplxprod k[] k[]k[] cos k[] k[] cosh k[] k[] cosinv k[] k[] dct k[] k[] dctinv k[] k[] deinterleave k[]k[] k[] directory S[] SN diskin a[] SPooooooo diskin a[] iPooooooo diskin2 a[] SPooooooo diskin2 a[] iPooooooo exp k[] k[] fft k[] k[] fftinv k[] k[] fillarray S[] W fillarray k[] S fillarray k[] m fillarray k[] z floor k[] k[] fluidInfo S[] i fmax k[] k[]k fmax k[] k[]k[] fmin k[] k[]k fmin k[] k[]k[] fmod k[] k[]k fmod k[] k[]k[] frac k[] k[] ftom k[] k[]o genarray k[] kkp genarray_i k[] iip getcol k[] k[]k getrow k[] k[]k getrowlin k[] k[]kOOP getrowlin k[] kiiooop hypot k[] k[]k[] in a[] (null) in a[] (null) init .[] m inletv a[] S int k[] k[] interleave k[] k[]k[] limit k[] k[]kk limit1 k[] k[] linlin k[] k[]kkOP linlin k[] kk[]k[]OP log k[] k[] log k[] k[]i log10 k[] k[] log2 k[] k[] loscilx a[] xkioojjoo mags k[] k[] maparray k[] k[]S mfb k[] k[]kki monitor a[] (null) mtof k[] k[] passign k[] po phs k[] k[] pol2rect k[] k[] pol2rect k[] k[]k[] pow k[] k[]k pow k[] k[]k[] powoftwo k[] k[] pows k[] k[] pvsceps k[] fo pvstrace fk[] fko r2c k[] k[] rect2pol k[] k[] rfft k[] k[] rifft k[] k[] round k[] k[] setcol k[] k[]k setrow k[] k[]k shiftin k[] a sin k[] k[] sinh k[] k[] sininv k[] k[] slicearray S[] S[]iip slicearray a[] a[]iip slicearray k[] k[]iip sorta k[] k[] sortd k[] k[] sqrt k[] k[] string2array k[] S tab2array k[] iOOP tan k[] k[] tanh k[] k[] taninv k[] k[] taninv k[] k[]k[] unwrap k[] k[] vbap a[] akOOo vbapg k[] kOOo vbapgmove k[] iiim vbapmove a[] aiiim window k[] k[]Op

Date	2019-04-26 16:17
From	Victor Lazzarini
Subject	Re: [Csnd-dev] Array corrections
	In general, for realtime safe operation, all blocking operations and resource allocation should not be done in the audio processing thread. In the Bela platform, in particular, this causes context switches, which in its hard-realtime OS will cause interruptions in the processing leading to clicks etc. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 26 Apr 2019, at 15:43, Mauro Giubileo <mgiubileo@computeraltafed.it> wrote: I don't know the Bela platform, but I'm curious... Exactly, why on the Bela platform you should not do a memory allocation at perf-time? What is different compared to a memory allocation made on a standard PC? --- Mauro Il 2019-04-25 21:50 Victor Lazzarini ha scritto: I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe operation where memory allocation can only happen at init-time on a separate thread. If this is a solution to a problem, then it's not the right solution. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 On 25 Apr 2019, at 19:38, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. OSClisten kk[] iSS OSCraw S[]k i abs k[] k[] bformdec1 a[] ia[] bformenc1 a[] akk bpf k[] k[]M bpfcos k[] k[]M c2r k[] k[] cbrt k[] k[] ceil k[] k[] ceps k[] k[]k cepsinv k[] k[] cmp k[] kSk[]Sk cmp k[] k[]Sk cmp k[] k[]Sk[] cmplxprod k[] k[]k[] cos k[] k[] cosh k[] k[] cosinv k[] k[] dct k[] k[] dctinv k[] k[] deinterleave k[]k[] k[] directory S[] SN diskin a[] SPooooooo diskin a[] iPooooooo diskin2 a[] SPooooooo diskin2 a[] iPooooooo exp k[] k[] fft k[] k[] fftinv k[] k[] fillarray S[] W fillarray k[] S fillarray k[] m fillarray k[] z floor k[] k[] fluidInfo S[] i fmax k[] k[]k fmax k[] k[]k[] fmin k[] k[]k fmin k[] k[]k[] fmod k[] k[]k fmod k[] k[]k[] frac k[] k[] ftom k[] k[]o genarray k[] kkp genarray_i k[] iip getcol k[] k[]k getrow k[] k[]k getrowlin k[] k[]kOOP getrowlin k[] kiiooop hypot k[] k[]k[] in a[] (null) in a[] (null) init .[] m inletv a[] S int k[] k[] interleave k[] k[]k[] limit k[] k[]kk limit1 k[] k[] linlin k[] k[]kkOP linlin k[] kk[]k[]OP log k[] k[] log k[] k[]i log10 k[] k[] log2 k[] k[] loscilx a[] xkioojjoo mags k[] k[] maparray k[] k[]S mfb k[] k[]kki monitor a[] (null) mtof k[] k[] passign k[] po phs k[] k[] pol2rect k[] k[] pol2rect k[] k[]k[] pow k[] k[]k pow k[] k[]k[] powoftwo k[] k[] pows k[] k[] pvsceps k[] fo pvstrace fk[] fko r2c k[] k[] rect2pol k[] k[] rfft k[] k[] rifft k[] k[] round k[] k[] setcol k[] k[]k setrow k[] k[]k shiftin k[] a sin k[] k[] sinh k[] k[] sininv k[] k[] slicearray S[] S[]iip slicearray a[] a[]iip slicearray k[] k[]iip sorta k[] k[] sortd k[] k[] sqrt k[] k[] string2array k[] S tab2array k[] iOOP tan k[] k[] tanh k[] k[] taninv k[] k[] taninv k[] k[]k[] unwrap k[] k[] vbap a[] akOOo vbapg k[] kOOo vbapgmove k[] iiim vbapmove a[] aiiim window k[] k[]Op

Date	2019-04-26 19:12
From	Mauro Giubileo
Subject	Re: [Csnd-dev] Array corrections
	Thanks for the explanation. But why memory allocations have to be made on a separated thread? It's a Csound thing or it is related to the Bela platform? --- Mauro Il 2019-04-26 17:17 Victor Lazzarini ha scritto: In general, for realtime safe operation, all blocking operations and resource allocation should not be done in the audio processing thread. In the Bela platform, in particular, this causes context switches, which in its hard-realtime OS will cause interruptions in the processing leading to clicks etc. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 26 Apr 2019, at 15:43, Mauro Giubileo <mgiubileo@computeraltafed.it> wrote: I don't know the Bela platform, but I'm curious... Exactly, why on the Bela platform you should not do a memory allocation at perf-time? What is different compared to a memory allocation made on a standard PC? --- Mauro Il 2019-04-25 21:50 Victor Lazzarini ha scritto: I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe operation where memory allocation can only happen at init-time on a separate thread. If this is a solution to a problem, then it's not the right solution. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 On 25 Apr 2019, at 19:38, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. OSClisten kk[] iSS OSCraw S[]k i abs k[] k[] bformdec1 a[] ia[] bformenc1 a[] akk bpf k[] k[]M bpfcos k[] k[]M c2r k[] k[] cbrt k[] k[] ceil k[] k[] ceps k[] k[]k cepsinv k[] k[] cmp k[] kSk[]Sk cmp k[] k[]Sk cmp k[] k[]Sk[] cmplxprod k[] k[]k[] cos k[] k[] cosh k[] k[] cosinv k[] k[] dct k[] k[] dctinv k[] k[] deinterleave k[]k[] k[] directory S[] SN diskin a[] SPooooooo diskin a[] iPooooooo diskin2 a[] SPooooooo diskin2 a[] iPooooooo exp k[] k[] fft k[] k[] fftinv k[] k[] fillarray S[] W fillarray k[] S fillarray k[] m fillarray k[] z floor k[] k[] fluidInfo S[] i fmax k[] k[]k fmax k[] k[]k[] fmin k[] k[]k fmin k[] k[]k[] fmod k[] k[]k fmod k[] k[]k[] frac k[] k[] ftom k[] k[]o genarray k[] kkp genarray_i k[] iip getcol k[] k[]k getrow k[] k[]k getrowlin k[] k[]kOOP getrowlin k[] kiiooop hypot k[] k[]k[] in a[] (null) in a[] (null) init .[] m inletv a[] S int k[] k[] interleave k[] k[]k[] limit k[] k[]kk limit1 k[] k[] linlin k[] k[]kkOP linlin k[] kk[]k[]OP log k[] k[] log k[] k[]i log10 k[] k[] log2 k[] k[] loscilx a[] xkioojjoo mags k[] k[] maparray k[] k[]S mfb k[] k[]kki monitor a[] (null) mtof k[] k[] passign k[] po phs k[] k[] pol2rect k[] k[] pol2rect k[] k[]k[] pow k[] k[]k pow k[] k[]k[] powoftwo k[] k[] pows k[] k[] pvsceps k[] fo pvstrace fk[] fko r2c k[] k[] rect2pol k[] k[] rfft k[] k[] rifft k[] k[] round k[] k[] setcol k[] k[]k setrow k[] k[]k shiftin k[] a sin k[] k[] sinh k[] k[] sininv k[] k[] slicearray S[] S[]iip slicearray a[] a[]iip slicearray k[] k[]iip sorta k[] k[] sortd k[] k[] sqrt k[] k[] string2array k[] S tab2array k[] iOOP tan k[] k[] tanh k[] k[] taninv k[] k[] taninv k[] k[]k[] unwrap k[] k[] vbap a[] akOOo vbapg k[] kOOo vbapgmove k[] iiim vbapmove a[] aiiim window k[] k[]Op

Date	2019-04-26 19:15
From	Victor Lazzarini
Subject	Re: [Csnd-dev] Array corrections
	It's a realtime safe thing. Any system that is hard realtime low-latency needs that. Csound offers it in its --realtime mode. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 26 Apr 2019, at 19:12, Mauro Giubileo <mgiubileo@computeraltafed.it> wrote: Thanks for the explanation. But why memory allocations have to be made on a separated thread? It's a Csound thing or it is related to the Bela platform? --- Mauro Il 2019-04-26 17:17 Victor Lazzarini ha scritto: In general, for realtime safe operation, all blocking operations and resource allocation should not be done in the audio processing thread. In the Bela platform, in particular, this causes context switches, which in its hard-realtime OS will cause interruptions in the processing leading to clicks etc. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 26 Apr 2019, at 15:43, Mauro Giubileo <mgiubileo@computeraltafed.it> wrote: I don't know the Bela platform, but I'm curious... Exactly, why on the Bela platform you should not do a memory allocation at perf-time? What is different compared to a memory allocation made on a standard PC? --- Mauro Il 2019-04-25 21:50 Victor Lazzarini ha scritto: I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe operation where memory allocation can only happen at init-time on a separate thread. If this is a solution to a problem, then it's not the right solution. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 On 25 Apr 2019, at 19:38, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. OSClisten kk[] iSS OSCraw S[]k i abs k[] k[] bformdec1 a[] ia[] bformenc1 a[] akk bpf k[] k[]M bpfcos k[] k[]M c2r k[] k[] cbrt k[] k[] ceil k[] k[] ceps k[] k[]k cepsinv k[] k[] cmp k[] kSk[]Sk cmp k[] k[]Sk cmp k[] k[]Sk[] cmplxprod k[] k[]k[] cos k[] k[] cosh k[] k[] cosinv k[] k[] dct k[] k[] dctinv k[] k[] deinterleave k[]k[] k[] directory S[] SN diskin a[] SPooooooo diskin a[] iPooooooo diskin2 a[] SPooooooo diskin2 a[] iPooooooo exp k[] k[] fft k[] k[] fftinv k[] k[] fillarray S[] W fillarray k[] S fillarray k[] m fillarray k[] z floor k[] k[] fluidInfo S[] i fmax k[] k[]k fmax k[] k[]k[] fmin k[] k[]k fmin k[] k[]k[] fmod k[] k[]k fmod k[] k[]k[] frac k[] k[] ftom k[] k[]o genarray k[] kkp genarray_i k[] iip getcol k[] k[]k getrow k[] k[]k getrowlin k[] k[]kOOP getrowlin k[] kiiooop hypot k[] k[]k[] in a[] (null) in a[] (null) init .[] m inletv a[] S int k[] k[] interleave k[] k[]k[] limit k[] k[]kk limit1 k[] k[] linlin k[] k[]kkOP linlin k[] kk[]k[]OP log k[] k[] log k[] k[]i log10 k[] k[] log2 k[] k[] loscilx a[] xkioojjoo mags k[] k[] maparray k[] k[]S mfb k[] k[]kki monitor a[] (null) mtof k[] k[] passign k[] po phs k[] k[] pol2rect k[] k[] pol2rect k[] k[]k[] pow k[] k[]k pow k[] k[]k[] powoftwo k[] k[] pows k[] k[] pvsceps k[] fo pvstrace fk[] fko r2c k[] k[] rect2pol k[] k[] rfft k[] k[] rifft k[] k[] round k[] k[] setcol k[] k[]k setrow k[] k[]k shiftin k[] a sin k[] k[] sinh k[] k[] sininv k[] k[] slicearray S[] S[]iip slicearray a[] a[]iip slicearray k[] k[]iip sorta k[] k[] sortd k[] k[] sqrt k[] k[] string2array k[] S tab2array k[] iOOP tan k[] k[] tanh k[] k[] taninv k[] k[] taninv k[] k[]k[] unwrap k[] k[] vbap a[] akOOo vbapg k[] kOOo vbapgmove k[] iiim vbapmove a[] aiiim window k[] k[]Op

Date	2019-04-26 19:53
From	Mauro Giubileo
Subject	Re: [Csnd-dev] Array corrections
	Ok, so the separated thread is to make the audio processing run without interruptions from init-tasks of other instruments instances (I suppose) ? --- Mauro Il 2019-04-26 20:15 Victor Lazzarini ha scritto: It's a realtime safe thing. Any system that is hard realtime low-latency needs that. Csound offers it in its --realtime mode. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 26 Apr 2019, at 19:12, Mauro Giubileo <mgiubileo@computeraltafed.it> wrote: Thanks for the explanation. But why memory allocations have to be made on a separated thread? It's a Csound thing or it is related to the Bela platform? --- Mauro Il 2019-04-26 17:17 Victor Lazzarini ha scritto: In general, for realtime safe operation, all blocking operations and resource allocation should not be done in the audio processing thread. In the Bela platform, in particular, this causes context switches, which in its hard-realtime OS will cause interruptions in the processing leading to clicks etc. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 26 Apr 2019, at 15:43, Mauro Giubileo <mgiubileo@computeraltafed.it> wrote: I don't know the Bela platform, but I'm curious... Exactly, why on the Bela platform you should not do a memory allocation at perf-time? What is different compared to a memory allocation made on a standard PC? --- Mauro Il 2019-04-25 21:50 Victor Lazzarini ha scritto: I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe operation where memory allocation can only happen at init-time on a separate thread. If this is a solution to a problem, then it's not the right solution. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 On 25 Apr 2019, at 19:38, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. OSClisten kk[] iSS OSCraw S[]k i abs k[] k[] bformdec1 a[] ia[] bformenc1 a[] akk bpf k[] k[]M bpfcos k[] k[]M c2r k[] k[] cbrt k[] k[] ceil k[] k[] ceps k[] k[]k cepsinv k[] k[] cmp k[] kSk[]Sk cmp k[] k[]Sk cmp k[] k[]Sk[] cmplxprod k[] k[]k[] cos k[] k[] cosh k[] k[] cosinv k[] k[] dct k[] k[] dctinv k[] k[] deinterleave k[]k[] k[] directory S[] SN diskin a[] SPooooooo diskin a[] iPooooooo diskin2 a[] SPooooooo diskin2 a[] iPooooooo exp k[] k[] fft k[] k[] fftinv k[] k[] fillarray S[] W fillarray k[] S fillarray k[] m fillarray k[] z floor k[] k[] fluidInfo S[] i fmax k[] k[]k fmax k[] k[]k[] fmin k[] k[]k fmin k[] k[]k[] fmod k[] k[]k fmod k[] k[]k[] frac k[] k[] ftom k[] k[]o genarray k[] kkp genarray_i k[] iip getcol k[] k[]k getrow k[] k[]k getrowlin k[] k[]kOOP getrowlin k[] kiiooop hypot k[] k[]k[] in a[] (null) in a[] (null) init .[] m inletv a[] S int k[] k[] interleave k[] k[]k[] limit k[] k[]kk limit1 k[] k[] linlin k[] k[]kkOP linlin k[] kk[]k[]OP log k[] k[] log k[] k[]i log10 k[] k[] log2 k[] k[] loscilx a[] xkioojjoo mags k[] k[] maparray k[] k[]S mfb k[] k[]kki monitor a[] (null) mtof k[] k[] passign k[] po phs k[] k[] pol2rect k[] k[] pol2rect k[] k[]k[] pow k[] k[]k pow k[] k[]k[] powoftwo k[] k[] pows k[] k[] pvsceps k[] fo pvstrace fk[] fko r2c k[] k[] rect2pol k[] k[] rfft k[] k[] rifft k[] k[] round k[] k[] setcol k[] k[]k setrow k[] k[]k shiftin k[] a sin k[] k[] sinh k[] k[] sininv k[] k[] slicearray S[] S[]iip slicearray a[] a[]iip slicearray k[] k[]iip sorta k[] k[] sortd k[] k[] sqrt k[] k[] string2array k[] S tab2array k[] iOOP tan k[] k[] tanh k[] k[] taninv k[] k[] taninv k[] k[]k[] unwrap k[] k[] vbap a[] akOOo vbapg k[] kOOo vbapgmove k[] iiim vbapmove a[] aiiim window k[] k[]Op

Date	2019-04-26 19:58
From	Justin Smith
Subject	Re: [Csnd-dev] Array corrections
	It's about not getting blocked by syscalls. When you allocate memory from the OS there's no way to guarantee it happens within a specific period of time, so an app that is time critical should do memory allocation either before time critical operations start, or in a separate thread that isn't time critical. On Fri, Apr 26, 2019 at 11:53 AM Mauro Giubileo wrote: > > Ok, so the separated thread is to make the audio processing run without interruptions from init-tasks of other instruments instances (I suppose) ? > > --- > Mauro > > > Il 2019-04-26 20:15 Victor Lazzarini ha scritto: > > It's a realtime safe thing. Any system that is hard realtime low-latency needs that. Csound offers it in its --realtime mode. > > Victor Lazzarini > Dean of Arts, Celtic Studies, and Philosophy > Maynooth University > Ireland > > On 26 Apr 2019, at 19:12, Mauro Giubileo wrote: > > Thanks for the explanation. But why memory allocations have to be made on a separated thread? It's a Csound thing or it is related to the Bela platform? > > --- > Mauro > > > Il 2019-04-26 17:17 Victor Lazzarini ha scritto: > > In general, for realtime safe operation, all blocking operations and resource allocation should not be done in the audio processing thread. > > In the Bela platform, in particular, this causes context switches, which in its hard-realtime OS will cause interruptions in the processing leading to clicks etc. > > Victor Lazzarini > Dean of Arts, Celtic Studies, and Philosophy > Maynooth University > Ireland > > On 26 Apr 2019, at 15:43, Mauro Giubileo wrote: > > I don't know the Bela platform, but I'm curious... Exactly, why on the Bela platform you should not do a memory allocation at perf-time? What is different compared to a memory allocation made on a standard PC? > > --- > Mauro > > > Il 2019-04-25 21:50 Victor Lazzarini ha scritto: > > I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls > calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe > operation where memory allocation can only happen at init-time on a separate thread. > > If this is a solution to a problem, then it's not the right solution. > ======================== > Prof. Victor Lazzarini > Dean of Arts, Celtic Studies, and Philosophy, > Maynooth University, > Maynooth, Co Kildare, Ireland > Tel: 00 353 7086936 > Fax: 00 353 1 7086952 > > On 25 Apr 2019, at 19:38, Eduardo Moguillansky wrote: > > Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. > > OSClisten kk[] iSS > OSCraw S[]k i > abs k[] k[] > bformdec1 a[] ia[] > bformenc1 a[] akk > bpf k[] k[]M > bpfcos k[] k[]M > c2r k[] k[] > cbrt k[] k[] > ceil k[] k[] > ceps k[] k[]k > cepsinv k[] k[] > cmp k[] kSk[]Sk > cmp k[] k[]Sk > cmp k[] k[]Sk[] > cmplxprod k[] k[]k[] > cos k[] k[] > cosh k[] k[] > cosinv k[] k[] > dct k[] k[] > dctinv k[] k[] > deinterleave k[]k[] k[] > directory S[] SN > diskin a[] SPooooooo > diskin a[] iPooooooo > diskin2 a[] SPooooooo > diskin2 a[] iPooooooo > exp k[] k[] > fft k[] k[] > fftinv k[] k[] > fillarray S[] W > fillarray k[] S > fillarray k[] m > fillarray k[] z > floor k[] k[] > fluidInfo S[] i > fmax k[] k[]k > fmax k[] k[]k[] > fmin k[] k[]k > fmin k[] k[]k[] > fmod k[] k[]k > fmod k[] k[]k[] > frac k[] k[] > ftom k[] k[]o > genarray k[] kkp > genarray_i k[] iip > getcol k[] k[]k > getrow k[] k[]k > getrowlin k[] k[]kOOP > getrowlin k[] kiiooop > hypot k[] k[]k[] > in a[] (null) > in a[] (null) > init .[] m > inletv a[] S > int k[] k[] > interleave k[] k[]k[] > limit k[] k[]kk > limit1 k[] k[] > linlin k[] k[]kkOP > linlin k[] kk[]k[]OP > log k[] k[] > log k[] k[]i > log10 k[] k[] > log2 k[] k[] > loscilx a[] xkioojjoo > mags k[] k[] > maparray k[] k[]S > mfb k[] k[]kki > monitor a[] (null) > mtof k[] k[] > passign k[] po > phs k[] k[] > pol2rect k[] k[] > pol2rect k[] k[]k[] > pow k[] k[]k > pow k[] k[]k[] > powoftwo k[] k[] > pows k[] k[] > pvsceps k[] fo > pvstrace fk[] fko > r2c k[] k[] > rect2pol k[] k[] > rfft k[] k[] > rifft k[] k[] > round k[] k[] > setcol k[] k[]k > setrow k[] k[]k > shiftin k[] a > sin k[] k[] > sinh k[] k[] > sininv k[] k[] > slicearray S[] S[]iip > slicearray a[] a[]iip > slicearray k[] k[]iip > sorta k[] k[] > sortd k[] k[] > sqrt k[] k[] > string2array k[] S > tab2array k[] iOOP > tan k[] k[] > tanh k[] k[] > taninv k[] k[] > taninv k[] k[]k[] > unwrap k[] k[] > vbap a[] akOOo > vbapg k[] kOOo > vbapgmove k[] iiim > vbapmove a[] aiiim

Date	2019-04-26 20:09
From	Victor Lazzarini
Subject	Re: [Csnd-dev] Array corrections
	That, and in the case of Csound, opening files, and also making blocking calls. All of that goes to separate threads that run in parallel with audio computing in --realtime mode. In normal operation this all happens in the same thread. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland > On 26 Apr 2019, at 19:59, Justin Smith wrote: > > It's about not getting blocked by syscalls. When you allocate memory > from the OS there's no way to guarantee it happens within a specific > period of time, so an app that is time critical should do memory > allocation either before time critical operations start, or in a > separate thread that isn't time critical. > > On Fri, Apr 26, 2019 at 11:53 AM Mauro Giubileo > wrote: >> >> Ok, so the separated thread is to make the audio processing run without interruptions from init-tasks of other instruments instances (I suppose) ? >> >> --- >> Mauro >> >> >> Il 2019-04-26 20:15 Victor Lazzarini ha scritto: >> >> It's a realtime safe thing. Any system that is hard realtime low-latency needs that. Csound offers it in its --realtime mode. >> >> Victor Lazzarini >> Dean of Arts, Celtic Studies, and Philosophy >> Maynooth University >> Ireland >> >> On 26 Apr 2019, at 19:12, Mauro Giubileo wrote: >> >> Thanks for the explanation. But why memory allocations have to be made on a separated thread? It's a Csound thing or it is related to the Bela platform? >> >> --- >> Mauro >> >> >> Il 2019-04-26 17:17 Victor Lazzarini ha scritto: >> >> In general, for realtime safe operation, all blocking operations and resource allocation should not be done in the audio processing thread. >> >> In the Bela platform, in particular, this causes context switches, which in its hard-realtime OS will cause interruptions in the processing leading to clicks etc. >> >> Victor Lazzarini >> Dean of Arts, Celtic Studies, and Philosophy >> Maynooth University >> Ireland >> >> On 26 Apr 2019, at 15:43, Mauro Giubileo wrote: >> >> I don't know the Bela platform, but I'm curious... Exactly, why on the Bela platform you should not do a memory allocation at perf-time? What is different compared to a memory allocation made on a standard PC? >> >> --- >> Mauro >> >> >> Il 2019-04-25 21:50 Victor Lazzarini ha scritto: >> >> I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls >> calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe >> operation where memory allocation can only happen at init-time on a separate thread. >> >> If this is a solution to a problem, then it's not the right solution. >> ======================== >> Prof. Victor Lazzarini >> Dean of Arts, Celtic Studies, and Philosophy, >> Maynooth University, >> Maynooth, Co Kildare, Ireland >> Tel: 00 353 7086936 >> Fax: 00 353 1 7086952 >> >> On 25 Apr 2019, at 19:38, Eduardo Moguillansky wrote: >> >> Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. >> >> OSClisten kk[] iSS >> OSCraw S[]k i >> abs k[] k[] >> bformdec1 a[] ia[] >> bformenc1 a[] akk >> bpf k[] k[]M >> bpfcos k[] k[]M >> c2r k[] k[] >> cbrt k[] k[] >> ceil k[] k[] >> ceps k[] k[]k >> cepsinv k[] k[] >> cmp k[] kSk[]Sk >> cmp k[] k[]Sk >> cmp k[] k[]Sk[] >> cmplxprod k[] k[]k[] >> cos k[] k[] >> cosh k[] k[] >> cosinv k[] k[] >> dct k[] k[] >> dctinv k[] k[] >> deinterleave k[]k[] k[] >> directory S[] SN >> diskin a[] SPooooooo >> diskin a[] iPooooooo >> diskin2 a[] SPooooooo >> diskin2 a[] iPooooooo >> exp k[] k[] >> fft k[] k[] >> fftinv k[] k[] >> fillarray S[] W >> fillarray k[] S >> fillarray k[] m >> fillarray k[] z >> floor k[] k[] >> fluidInfo S[] i >> fmax k[] k[]k >> fmax k[] k[]k[] >> fmin k[] k[]k >> fmin k[] k[]k[] >> fmod k[] k[]k >> fmod k[] k[]k[] >> frac k[] k[] >> ftom k[] k[]o >> genarray k[] kkp >> genarray_i k[] iip >> getcol k[] k[]k >> getrow k[] k[]k >> getrowlin k[] k[]kOOP >> getrowlin k[] kiiooop >> hypot k[] k[]k[] >> in a[] (null) >> in a[] (null) >> init .[] m >> inletv a[] S >> int k[] k[] >> interleave k[] k[]k[] >> limit k[] k[]kk >> limit1 k[] k[] >> linlin k[] k[]kkOP >> linlin k[] kk[]k[]OP >> log k[] k[] >> log k[] k[]i >> log10 k[] k[] >> log2 k[] k[] >> loscilx a[] xkioojjoo >> mags k[] k[] >> maparray k[] k[]S >> mfb k[] k[]kki >> monitor a[] (null) >> mtof k[] k[] >> passign k[] po >> phs k[] k[] >> pol2rect k[] k[] >> pol2rect k[] k[]k[] >> pow k[] k[]k >> pow k[] k[]k[] >> powoftwo k[] k[] >> pows k[] k[] >> pvsceps k[] fo >> pvstrace fk[] fko >> r2c k[] k[] >> rect2pol k[] k[] >> rfft k[] k[] >> rifft k[] k[] >> round k[] k[] >> setcol k[] k[]k >> setrow k[] k[]k >> shiftin k[] a >> sin k[] k[] >> sinh k[] k[] >> sininv k[] k[] >> slicearray S[] S[]iip >> slicearray a[] a[]iip >> slicearray k[] k[]iip >> sorta k[] k[] >> sortd k[] k[] >> sqrt k[] k[] >> string2array k[] S >> tab2array k[] iOOP >> tan k[] k[] >> tanh k[] k[] >> taninv k[] k[] >> taninv k[] k[]k[] >> unwrap k[] k[] >> vbap a[] akOOo >> vbapg k[] kOOo >> vbapgmove k[] iiim >> vbapmove a[] aiiim

Date	2019-04-27 10:25
From	Mauro Giubileo
Subject	Re: [Csnd-dev] Array corrections
	Thank you very much, now everything is clearer to me. --- Mauro Il 2019-04-26 21:09 Victor Lazzarini ha scritto: That, and in the case of Csound, opening files, and also making blocking calls. All of that goes to separate threads that run in parallel with audio computing in --realtime mode. In normal operation this all happens in the same thread. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 26 Apr 2019, at 19:59, Justin Smith <noisesmith@gmail.com> wrote: It's about not getting blocked by syscalls. When you allocate memory from the OS there's no way to guarantee it happens within a specific period of time, so an app that is time critical should do memory allocation either before time critical operations start, or in a separate thread that isn't time critical. On Fri, Apr 26, 2019 at 11:53 AM Mauro Giubileo <mgiubileo@computeraltafed.it> wrote: Ok, so the separated thread is to make the audio processing run without interruptions from init-tasks of other instruments instances (I suppose) ? --- Mauro Il 2019-04-26 20:15 Victor Lazzarini ha scritto: It's a realtime safe thing. Any system that is hard realtime low-latency needs that. Csound offers it in its --realtime mode. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 26 Apr 2019, at 19:12, Mauro Giubileo <mgiubileo@computeraltafed.it> wrote: Thanks for the explanation. But why memory allocations have to be made on a separated thread? It's a Csound thing or it is related to the Bela platform? --- Mauro Il 2019-04-26 17:17 Victor Lazzarini ha scritto: In general, for realtime safe operation, all blocking operations and resource allocation should not be done in the audio processing thread. In the Bela platform, in particular, this causes context switches, which in its hard-realtime OS will cause interruptions in the processing leading to clicks etc. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 26 Apr 2019, at 15:43, Mauro Giubileo <mgiubileo@computeraltafed.it> wrote: I don't know the Bela platform, but I'm curious... Exactly, why on the Bela platform you should not do a memory allocation at perf-time? What is different compared to a memory allocation made on a standard PC? --- Mauro Il 2019-04-25 21:50 Victor Lazzarini ha scritto: I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe operation where memory allocation can only happen at init-time on a separate thread. If this is a solution to a problem, then it's not the right solution. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 On 25 Apr 2019, at 19:38, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. OSClisten kk[] iSS OSCraw S[]k i abs k[] k[] bformdec1 a[] ia[] bformenc1 a[] akk bpf k[] k[]M bpfcos k[] k[]M c2r k[] k[] cbrt k[] k[] ceil k[] k[] ceps k[] k[]k cepsinv k[] k[] cmp k[] kSk[]Sk cmp k[] k[]Sk cmp k[] k[]Sk[] cmplxprod k[] k[]k[] cos k[] k[] cosh k[] k[] cosinv k[] k[] dct k[] k[] dctinv k[] k[] deinterleave k[]k[] k[] directory S[] SN diskin a[] SPooooooo diskin a[] iPooooooo diskin2 a[] SPooooooo diskin2 a[] iPooooooo exp k[] k[] fft k[] k[] fftinv k[] k[] fillarray S[] W fillarray k[] S fillarray k[] m fillarray k[] z floor k[] k[] fluidInfo S[] i fmax k[] k[]k fmax k[] k[]k[] fmin k[] k[]k fmin k[] k[]k[] fmod k[] k[]k fmod k[] k[]k[] frac k[] k[] ftom k[] k[]o genarray k[] kkp genarray_i k[] iip getcol k[] k[]k getrow k[] k[]k getrowlin k[] k[]kOOP getrowlin k[] kiiooop hypot k[] k[]k[] in a[] (null) in a[] (null) init .[] m inletv a[] S int k[] k[] interleave k[] k[]k[] limit k[] k[]kk limit1 k[] k[] linlin k[] k[]kkOP linlin k[] kk[]k[]OP log k[] k[] log k[] k[]i log10 k[] k[] log2 k[] k[] loscilx a[] xkioojjoo mags k[] k[] maparray k[] k[]S mfb k[] k[]kki monitor a[] (null) mtof k[] k[] passign k[] po phs k[] k[] pol2rect k[] k[] pol2rect k[] k[]k[] pow k[] k[]k pow k[] k[]k[] powoftwo k[] k[] pows k[] k[] pvsceps k[] fo pvstrace fk[] fko r2c k[] k[] rect2pol k[] k[] rfft k[] k[] rifft k[] k[] round k[] k[] setcol k[] k[]k setrow k[] k[]k shiftin k[] a sin k[] k[] sinh k[] k[] sininv k[] k[] slicearray S[] S[]iip slicearray a[] a[]iip slicearray k[] k[]iip sorta k[] k[] sortd k[] k[] sqrt k[] k[] string2array k[] S tab2array k[] iOOP tan k[] k[] tanh k[] k[] taninv k[] k[] taninv k[] k[]k[] unwrap k[] k[] vbap a[] akOOo vbapg k[] kOOo vbapgmove k[] iiim vbapmove a[] aiiim window k[] k[]Op

Date	2019-04-27 15:53
From	Mauro Giubileo
Subject	Re: [Csnd-dev] Array corrections
	If for realtime needs we don't want to allow memory allocation syscalls at perf-time, the second fillarray in your example should not be allowed. It could be allowed only if the elements of the second fillarray were less than those of the first. I think that if an audio application needs to let an array grow dinamically at perf-time, one can always preallocate a bigger array during init-time, in anticipation of that growth... --- Mauro Il 2019-04-26 00:57 Steven Yi ha scritto: I was thinking through this and realized even if we say array sizes can only change at init-time, we may still need to set sizes at runtime. It dawned on me that the situation occurs if you reuse an array var: karr[] fillarray 1,2,3 ...1. do something... karr = fillarray 5,6,7,8 ... 2. do something ... so that in code section 1, karr should have a size of 3 at runtime, while code section 2, karr would have a size of 4. So it seems back to two tabensures, one at init and one for perf. On Thu, Apr 25, 2019 at 6:41 PM Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Given the requirement that there can't be allocation at perf-time, the semantics for arrays should be: * an array has an allocated size and a perceived size * the allocated size is given at init time and can't be modified after that * the output of any array operation should either result in an array of the expected size or in an error. This needs to be enfor * for element wise operations, like arithmetic between an array and a scalar, the output array should be the same size / shape as the input array, or result in a performance error if this can't be achieved If there is no agreement in the semantics, no code can be correct. There are two issues here: 1) array growth. With the implementation in the array-init-change branch, arrays can be reallocated at perf-time if they need to grow beyond allocated size. According to Victor this needs to be prevented in order to ensure realtime safety. That means that tabensure, as written there, can't be called at perf-time. 2) correct perceived size. At perf-time it must be ensured that an array has the correct perceived size, as long as this size is <= allocated size. This operation does not imply any memory allocation. With my proposed implementation all these constraints are satisfied. Any other solution is either semantically flawed (the currect situation, where no update of perceived size is done, size depends on allocated size, even on previous size in previous instances of the instrument, etc) or does not meet the required conditions (realtime safety) On 26.04.19 00:13, Steven Yi wrote: The code change for tabensure (the one in the array-init-change branch) seems good to me. If the problem is that it should not be called at perf-time, then existing opcodes have problems and need to be rewritten. I'd rather we not have tabensure_perf and just document "only use at init-time". On Thu, Apr 25, 2019 at 5:33 PM Victor Lazzarini <Victor.Lazzarini@mu.ie> wrote: Sounds like the correct thing to me. As with other dynamic memory uses by opcodes, we only allocate or reallocate at init-time (e.g. we don't change delay line max sizes at perf time). Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 25 Apr 2019, at 22:21, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: If this is a requirement, then the array semantics need to be redefined and made clear. There can't be any operation at perf time which can grow an array. Even further, all perf time array operation need to operate with initialized arrays, since that would mean a perf time allocation. Following this semantics, we could go back to two tabensure functions in the form: // this should be called at init time by any opcode returning an array // even if the opcode only works at perf-time. static inline void tabensure_init(CSOUND csound, ARRAYDAT p, int size) { size_t ss = p->arrayMemberSizesize; if (p->data == NULL) { CS_VARIABLE var = p->arrayType->createVariable(csound, NULL); p->arrayMemberSize = var->memBlockSize; p->data = (MYFLT)csound->Calloc(csound, ss); p->allocated = ss; } else if (ss > p->allocated) { p->data = (MYFLT)csound->ReAlloc(csound, p->data, ss); p->allocated = ss; } p->sizes[0] = size; p->dimensions = 1; } // this should be called at perf time static inline void tabensure_perf(CSOUND csound, ARRAYDAT p, int size) { if(p->data == NULL) { return csound->PerformanceError(...); } size_t ss = p->arrayMemberSize*size; if(ss > p->allocated) { return csound->PerformanceError(...); } p->sizes[0] = size; } With this scheme, all allocation and resizing would happen at init time and any attempt to resize at perf time would result in an error. The user then can preallocate arrays as big as needed: kMyArray[] init i_largestsize possible Would that be an acceptable solution? On 25.04.19 21:50, Victor Lazzarini wrote: I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe operation where memory allocation can only happen at init-time on a separate thread. If this is a solution to a problem, then it's not the right solution. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 On 25 Apr 2019, at 19:38, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. OSClisten kk[] iSS OSCraw S[]k i abs k[] k[] bformdec1 a[] ia[] bformenc1 a[] akk bpf k[] k[]M bpfcos k[] k[]M c2r k[] k[] cbrt k[] k[] ceil k[] k[] ceps k[] k[]k cepsinv k[] k[] cmp k[] kSk[]Sk cmp k[] k[]Sk cmp k[] k[]Sk[] cmplxprod k[] k[]k[] cos k[] k[] cosh k[] k[] cosinv k[] k[] dct k[] k[] dctinv k[] k[] deinterleave k[]k[] k[] directory S[] SN diskin a[] SPooooooo diskin a[] iPooooooo diskin2 a[] SPooooooo diskin2 a[] iPooooooo exp k[] k[] fft k[] k[] fftinv k[] k[] fillarray S[] W fillarray k[] S fillarray k[] m fillarray k[] z floor k[] k[] fluidInfo S[] i fmax k[] k[]k fmax k[] k[]k[] fmin k[] k[]k fmin k[] k[]k[] fmod k[] k[]k fmod k[] k[]k[] frac k[] k[] ftom k[] k[]o genarray k[] kkp genarray_i k[] iip getcol k[] k[]k getrow k[] k[]k getrowlin k[] k[]kOOP getrowlin k[] kiiooop hypot k[] k[]k[] in a[] (null) in a[] (null) init .[] m inletv a[] S int k[] k[] interleave k[] k[]k[] limit k[] k[]kk limit1 k[] k[] linlin k[] k[]kkOP linlin k[] kk[]k[]OP log k[] k[] log k[] k[]i log10 k[] k[] log2 k[] k[] loscilx a[] xkioojjoo mags k[] k[] maparray k[] k[]S mfb k[] k[]kki monitor a[] (null) mtof k[] k[] passign k[] po phs k[] k[] pol2rect k[] k[] pol2rect k[] k[]k[] pow k[] k[]k pow k[] k[]k[] powoftwo k[] k[] pows k[] k[] pvsceps k[] fo pvstrace fk[] fko r2c k[] k[] rect2pol k[] k[] rfft k[] k[] rifft k[] k[] round k[] k[] setcol k[] k[]k setrow k[] k[]k shiftin k[] a sin k[] k[] sinh k[] k[] sininv k[] k[] slicearray S[] S[]iip slicearray a[] a[]iip slicearray k[] k[]iip sorta k[] k[] sortd k[] k[] sqrt k[] k[] string2array k[] S tab2array k[] iOOP tan k[] k[] tanh k[] k[] taninv k[] k[] taninv k[] k[]k[] unwrap k[] k[] vbap a[] akOOo vbapg k[] kOOo vbapgmove k[] iiim vbapmove a[] aiiim window k[] k[]Op

Date	2019-04-27 23:43
From	Steven Yi
Subject	Re: [Csnd-dev] Array corrections
	I'm wondering if we need three functions: 1. array_size_ensure_init : Does size check and resizing (w/mem allocation) but doesn't set size at init time 2. array_size_init : Does size check and resizing, sets size 3. array_size_perf: Does size check and sets size, no resizing and gives error if allocated size too small #1 would be for something like slicearray version that runs at perf time. It'd have to ensure the size, but doesn't actually set the active size, since it doesn't do a calculation until run-time. #2 would be for opcodes that run at init-time. #3 would be for perf-time opcodes that need to set the size. The max size is allocated with #1 or #2 at init-time. Question: Do we need this? I'm thinking it might be a kind of safety for the situation where people mix the order of init- and perf-time array processing opcodes. On Thu, Apr 25, 2019 at 6:57 PM Steven Yi <stevenyi@gmail.com> wrote: I was thinking through this and realized even if we say array sizes can only change at init-time, we may still need to set sizes at runtime. It dawned on me that the situation occurs if you reuse an array var: karr[] fillarray 1,2,3 ...1. do something... karr = fillarray 5,6,7,8 ... 2. do something ... so that in code section 1, karr should have a size of 3 at runtime, while code section 2, karr would have a size of 4. So it seems back to two tabensures, one at init and one for perf. On Thu, Apr 25, 2019 at 6:41 PM Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Given the requirement that there can't be allocation at perf-time, the semantics for arrays should be: * an array has an allocated size and a perceived size * the allocated size is given at init time and can't be modified after that * the output of any array operation should either result in an array of the expected size or in an error. This needs to be enfor * for element wise operations, like arithmetic between an array and a scalar, the output array should be the same size / shape as the input array, or result in a performance error if this can't be achieved If there is no agreement in the semantics, no code can be correct. There are two issues here: 1) array growth. With the implementation in the array-init-change branch, arrays can be reallocated at perf-time if they need to grow beyond allocated size. According to Victor this needs to be prevented in order to ensure realtime safety. That means that tabensure, as written there, can't be called at perf-time. 2) correct perceived size. At perf-time it must be ensured that an array has the correct perceived size, as long as this size is <= allocated size. This operation does not imply any memory allocation. With my proposed implementation all these constraints are satisfied. Any other solution is either semantically flawed (the currect situation, where no update of perceived size is done, size depends on allocated size, even on previous size in previous instances of the instrument, etc) or does not meet the required conditions (realtime safety) On 26.04.19 00:13, Steven Yi wrote: The code change for tabensure (the one in the array-init-change branch) seems good to me. If the problem is that it should not be called at perf-time, then existing opcodes have problems and need to be rewritten. I'd rather we not have tabensure_perf and just document "only use at init-time". On Thu, Apr 25, 2019 at 5:33 PM Victor Lazzarini <Victor.Lazzarini@mu.ie> wrote: Sounds like the correct thing to me. As with other dynamic memory uses by opcodes, we only allocate or reallocate at init-time (e.g. we don't change delay line max sizes at perf time). Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 25 Apr 2019, at 22:21, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: If this is a requirement, then the array semantics need to be redefined and made clear. There can't be any operation at perf time which can grow an array. Even further, all perf time array operation need to operate with initialized arrays, since that would mean a perf time allocation. Following this semantics, we could go back to two tabensure functions in the form: // this should be called at init time by any opcode returning an array // even if the opcode only works at perf-time. static inline void tabensure_init(CSOUND csound, ARRAYDAT p, int size) { size_t ss = p->arrayMemberSizesize; if (p->data == NULL) { CS_VARIABLE var = p->arrayType->createVariable(csound, NULL); p->arrayMemberSize = var->memBlockSize; p->data = (MYFLT)csound->Calloc(csound, ss); p->allocated = ss; } else if (ss > p->allocated) { p->data = (MYFLT)csound->ReAlloc(csound, p->data, ss); p->allocated = ss; } p->sizes[0] = size; p->dimensions = 1; } // this should be called at perf time static inline void tabensure_perf(CSOUND csound, ARRAYDAT p, int size) { if(p->data == NULL) { return csound->PerformanceError(...); } size_t ss = p->arrayMemberSize*size; if(ss > p->allocated) { return csound->PerformanceError(...); } p->sizes[0] = size; } With this scheme, all allocation and resizing would happen at init time and any attempt to resize at perf time would result in an error. The user then can preallocate arrays as big as needed: kMyArray[] init i_largestsize possible Would that be an acceptable solution? On 25.04.19 21:50, Victor Lazzarini wrote: I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe operation where memory allocation can only happen at init-time on a separate thread. If this is a solution to a problem, then it’s not the right solution. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 On 25 Apr 2019, at 19:38, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. OSClisten kk[] iSS OSCraw S[]k i abs k[] k[] bformdec1 a[] ia[] bformenc1 a[] akk bpf k[] k[]M bpfcos k[] k[]M c2r k[] k[] cbrt k[] k[] ceil k[] k[] ceps k[] k[]k cepsinv k[] k[] cmp k[] kSk[]Sk cmp k[] k[]Sk cmp k[] k[]Sk[] cmplxprod k[] k[]k[] cos k[] k[] cosh k[] k[] cosinv k[] k[] dct k[] k[] dctinv k[] k[] deinterleave k[]k[] k[] directory S[] SN diskin a[] SPooooooo diskin a[] iPooooooo diskin2 a[] SPooooooo diskin2 a[] iPooooooo exp k[] k[] fft k[] k[] fftinv k[] k[] fillarray S[] W fillarray k[] S fillarray k[] m fillarray k[] z floor k[] k[] fluidInfo S[] i fmax k[] k[]k fmax k[] k[]k[] fmin k[] k[]k fmin k[] k[]k[] fmod k[] k[]k fmod k[] k[]k[] frac k[] k[] ftom k[] k[]o genarray k[] kkp genarray_i k[] iip getcol k[] k[]k getrow k[] k[]k getrowlin k[] k[]kOOP getrowlin k[] kiiooop hypot k[] k[]k[] in a[] (null) in a[] (null) init .[] m inletv a[] S int k[] k[] interleave k[] k[]k[] limit k[] k[]kk limit1 k[] k[] linlin k[] k[]kkOP linlin k[] kk[]k[]OP log k[] k[] log k[] k[]i log10 k[] k[] log2 k[] k[] loscilx a[] xkioojjoo mags k[] k[] maparray k[] k[]S mfb k[] k[]kki monitor a[] (null) mtof k[] k[] passign k[] po phs k[] k[] pol2rect k[] k[] pol2rect k[] k[]k[] pow k[] k[]k pow k[] k[]k[] powoftwo k[] k[] pows k[] k[] pvsceps k[] fo pvstrace fk[] fko r2c k[] k[] rect2pol k[] k[] rfft k[] k[] rifft k[] k[] round k[] k[] setcol k[] k[]k setrow k[] k[]k shiftin k[] a sin k[] k[] sinh k[] k[] sininv k[] k[] slicearray S[] S[]iip slicearray a[] a[]iip slicearray k[] k[]iip sorta k[] k[] sortd k[] k[] sqrt k[] k[] string2array k[] S tab2array k[] iOOP tan k[] k[] tanh k[] k[] taninv k[] k[] taninv k[] k[]k[] unwrap k[] k[] vbap a[] akOOo vbapg k[] kOOo vbapgmove k[] iiim vbapmove a[] aiiim window k[] k[]Op

Date	2019-04-28 09:36
From	Eduardo Moguillansky
Subject	Re: [Csnd-dev] Array corrections
	What happens to an opcode like kA[] = kB[] * kx, where the size of kB is not known at init time? It needs to possibly allocate an initial size at init time, but size depends on values set later. Should opcodes like this require preallocated arrays? If allocation at perftime is banned, then this seems the case. On a side note,there are many other opcodes doing allocation at perftime, like string opcodes. Should these also require preallocation? On 28.04.19 00:43, Steven Yi wrote: I'm wondering if we need three functions: 1. array_size_ensure_init : Does size check and resizing (w/mem allocation) but doesn't set size at init time 2. array_size_init : Does size check and resizing, sets size 3. array_size_perf: Does size check and sets size, no resizing and gives error if allocated size too small #1 would be for something like slicearray version that runs at perf time. It'd have to ensure the size, but doesn't actually set the active size, since it doesn't do a calculation until run-time. #2 would be for opcodes that run at init-time. #3 would be for perf-time opcodes that need to set the size. The max size is allocated with #1 or #2 at init-time. Question: Do we need this? I'm thinking it might be a kind of safety for the situation where people mix the order of init- and perf-time array processing opcodes. On Thu, Apr 25, 2019 at 6:57 PM Steven Yi <stevenyi@gmail.com> wrote: I was thinking through this and realized even if we say array sizes can only change at init-time, we may still need to set sizes at runtime. It dawned on me that the situation occurs if you reuse an array var: karr[] fillarray 1,2,3 ...1. do something... karr = fillarray 5,6,7,8 ... 2. do something ... so that in code section 1, karr should have a size of 3 at runtime, while code section 2, karr would have a size of 4. So it seems back to two tabensures, one at init and one for perf. On Thu, Apr 25, 2019 at 6:41 PM Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Given the requirement that there can't be allocation at perf-time, the semantics for arrays should be: * an array has an allocated size and a perceived size * the allocated size is given at init time and can't be modified after that * the output of any array operation should either result in an array of the expected size or in an error. This needs to be enfor * for element wise operations, like arithmetic between an array and a scalar, the output array should be the same size / shape as the input array, or result in a performance error if this can't be achieved If there is no agreement in the semantics, no code can be correct. There are two issues here: 1) array growth. With the implementation in the array-init-change branch, arrays can be reallocated at perf-time if they need to grow beyond allocated size. According to Victor this needs to be prevented in order to ensure realtime safety. That means that tabensure, as written there, can't be called at perf-time. 2) correct perceived size. At perf-time it must be ensured that an array has the correct perceived size, as long as this size is <= allocated size. This operation does not imply any memory allocation. With my proposed implementation all these constraints are satisfied. Any other solution is either semantically flawed (the currect situation, where no update of perceived size is done, size depends on allocated size, even on previous size in previous instances of the instrument, etc) or does not meet the required conditions (realtime safety) On 26.04.19 00:13, Steven Yi wrote: The code change for tabensure (the one in the array-init-change branch) seems good to me. If the problem is that it should not be called at perf-time, then existing opcodes have problems and need to be rewritten. I'd rather we not have tabensure_perf and just document "only use at init-time". On Thu, Apr 25, 2019 at 5:33 PM Victor Lazzarini <Victor.Lazzarini@mu.ie> wrote: Sounds like the correct thing to me. As with other dynamic memory uses by opcodes, we only allocate or reallocate at init-time (e.g. we don't change delay line max sizes at perf time). Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 25 Apr 2019, at 22:21, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: If this is a requirement, then the array semantics need to be redefined and made clear. There can't be any operation at perf time which can grow an array. Even further, all perf time array operation need to operate with initialized arrays, since that would mean a perf time allocation. Following this semantics, we could go back to two tabensure functions in the form: // this should be called at init time by any opcode returning an array // even if the opcode only works at perf-time. static inline void tabensure_init(CSOUND csound, ARRAYDAT p, int size) { size_t ss = p->arrayMemberSizesize; if (p->data == NULL) { CS_VARIABLE var = p->arrayType->createVariable(csound, NULL); p->arrayMemberSize = var->memBlockSize; p->data = (MYFLT)csound->Calloc(csound, ss); p->allocated = ss; } else if (ss > p->allocated) { p->data = (MYFLT)csound->ReAlloc(csound, p->data, ss); p->allocated = ss; } p->sizes[0] = size; p->dimensions = 1; } // this should be called at perf time static inline void tabensure_perf(CSOUND csound, ARRAYDAT p, int size) { if(p->data == NULL) { return csound->PerformanceError(...); } size_t ss = p->arrayMemberSize*size; if(ss > p->allocated) { return csound->PerformanceError(...); } p->sizes[0] = size; } With this scheme, all allocation and resizing would happen at init time and any attempt to resize at perf time would result in an error. The user then can preallocate arrays as big as needed: kMyArray[] init i_largestsize possible Would that be an acceptable solution? On 25.04.19 21:50, Victor Lazzarini wrote: I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe operation where memory allocation can only happen at init-time on a separate thread. If this is a solution to a problem, then it’s not the right solution. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 On 25 Apr 2019, at 19:38, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. OSClisten kk[] iSS OSCraw S[]k i abs k[] k[] bformdec1 a[] ia[] bformenc1 a[] akk bpf k[] k[]M bpfcos k[] k[]M c2r k[] k[] cbrt k[] k[] ceil k[] k[] ceps k[] k[]k cepsinv k[] k[] cmp k[] kSk[]Sk cmp k[] k[]Sk cmp k[] k[]Sk[] cmplxprod k[] k[]k[] cos k[] k[] cosh k[] k[] cosinv k[] k[] dct k[] k[] dctinv k[] k[] deinterleave k[]k[] k[] directory S[] SN diskin a[] SPooooooo diskin a[] iPooooooo diskin2 a[] SPooooooo diskin2 a[] iPooooooo exp k[] k[] fft k[] k[] fftinv k[] k[] fillarray S[] W fillarray k[] S fillarray k[] m fillarray k[] z floor k[] k[] fluidInfo S[] i fmax k[] k[]k fmax k[] k[]k[] fmin k[] k[]k fmin k[] k[]k[] fmod k[] k[]k fmod k[] k[]k[] frac k[] k[] ftom k[] k[]o genarray k[] kkp genarray_i k[] iip getcol k[] k[]k getrow k[] k[]k getrowlin k[] k[]kOOP getrowlin k[] kiiooop hypot k[] k[]k[] in a[] (null) in a[] (null) init .[] m inletv a[] S int k[] k[] interleave k[] k[]k[] limit k[] k[]kk limit1 k[] k[] linlin k[] k[]kkOP linlin k[] kk[]k[]OP log k[] k[] log k[] k[]i log10 k[] k[] log2 k[] k[] loscilx a[] xkioojjoo mags k[] k[] maparray k[] k[]S mfb k[] k[]kki monitor a[] (null) mtof k[] k[] passign k[] po phs k[] k[] pol2rect k[] k[] pol2rect k[] k[]k[] pow k[] k[]k pow k[] k[]k[] powoftwo k[] k[] pows k[] k[] pvsceps k[] fo pvstrace fk[] fko r2c k[] k[] rect2pol k[] k[] rfft k[] k[] rifft k[] k[] round k[] k[] setcol k[] k[]k setrow k[] k[]k shiftin k[] a sin k[] k[] sinh k[] k[] sininv k[] k[] slicearray S[] S[]iip slicearray a[] a[]iip slicearray k[] k[]iip sorta k[] k[] sortd k[] k[] sqrt k[] k[] string2array k[] S tab2array k[] iOOP tan k[] k[] tanh k[] k[] taninv k[] k[] taninv k[] k[]k[] unwrap k[] k[] vbap a[] akOOo vbapg k[] kOOo vbapgmove k[] iiim vbapmove a[] aiiim window k[] k[]Op

Date	2019-04-28 10:15
From	Victor Lazzarini
Subject	Re: [Csnd-dev] Array corrections
	In general, if kB does not have allocation at perf-time then an error should result, or something of the kind. String opcodes are a funny case that probably crept in with time, as originally they only worked at i-time. We might need to leave these alone, with a warning that hard-realtime dsp use should avoid their use. It's a different case in that it is not really needed or used in signal processing. Conversely, it's likely that users will want to use k-rate arrays in signal processing. Off the top of my head, it appears that all array operations need to have an i-time pass so that any allocation needed happens at that stage. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 28 Apr 2019, at 09:36, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: What happens to an opcode like kA[] = kB[] * kx, where the size of kB is not known at init time? It needs to possibly allocate an initial size at init time, but size depends on values set later. Should opcodes like this require preallocated arrays? If allocation at perftime is banned, then this seems the case. On a side note,there are many other opcodes doing allocation at perftime, like string opcodes. Should these also require preallocation? On 28.04.19 00:43, Steven Yi wrote: I'm wondering if we need three functions: 1. array_size_ensure_init : Does size check and resizing (w/mem allocation) but doesn't set size at init time 2. array_size_init : Does size check and resizing, sets size 3. array_size_perf: Does size check and sets size, no resizing and gives error if allocated size too small #1 would be for something like slicearray version that runs at perf time. It'd have to ensure the size, but doesn't actually set the active size, since it doesn't do a calculation until run-time. #2 would be for opcodes that run at init-time. #3 would be for perf-time opcodes that need to set the size. The max size is allocated with #1 or #2 at init-time. Question: Do we need this? I'm thinking it might be a kind of safety for the situation where people mix the order of init- and perf-time array processing opcodes. On Thu, Apr 25, 2019 at 6:57 PM Steven Yi <stevenyi@gmail.com> wrote: I was thinking through this and realized even if we say array sizes can only change at init-time, we may still need to set sizes at runtime. It dawned on me that the situation occurs if you reuse an array var: karr[] fillarray 1,2,3 ...1. do something... karr = fillarray 5,6,7,8 ... 2. do something ... so that in code section 1, karr should have a size of 3 at runtime, while code section 2, karr would have a size of 4. So it seems back to two tabensures, one at init and one for perf. On Thu, Apr 25, 2019 at 6:41 PM Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Given the requirement that there can't be allocation at perf-time, the semantics for arrays should be: * an array has an allocated size and a perceived size * the allocated size is given at init time and can't be modified after that * the output of any array operation should either result in an array of the expected size or in an error. This needs to be enfor * for element wise operations, like arithmetic between an array and a scalar, the output array should be the same size / shape as the input array, or result in a performance error if this can't be achieved If there is no agreement in the semantics, no code can be correct. There are two issues here: 1) array growth. With the implementation in the array-init-change branch, arrays can be reallocated at perf-time if they need to grow beyond allocated size. According to Victor this needs to be prevented in order to ensure realtime safety. That means that tabensure, as written there, can't be called at perf-time. 2) correct perceived size. At perf-time it must be ensured that an array has the correct perceived size, as long as this size is <= allocated size. This operation does not imply any memory allocation. With my proposed implementation all these constraints are satisfied. Any other solution is either semantically flawed (the currect situation, where no update of perceived size is done, size depends on allocated size, even on previous size in previous instances of the instrument, etc) or does not meet the required conditions (realtime safety) On 26.04.19 00:13, Steven Yi wrote: The code change for tabensure (the one in the array-init-change branch) seems good to me. If the problem is that it should not be called at perf-time, then existing opcodes have problems and need to be rewritten. I'd rather we not have tabensure_perf and just document "only use at init-time". On Thu, Apr 25, 2019 at 5:33 PM Victor Lazzarini <Victor.Lazzarini@mu.ie> wrote: Sounds like the correct thing to me. As with other dynamic memory uses by opcodes, we only allocate or reallocate at init-time (e.g. we don't change delay line max sizes at perf time). Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 25 Apr 2019, at 22:21, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: If this is a requirement, then the array semantics need to be redefined and made clear. There can't be any operation at perf time which can grow an array. Even further, all perf time array operation need to operate with initialized arrays, since that would mean a perf time allocation. Following this semantics, we could go back to two tabensure functions in the form: // this should be called at init time by any opcode returning an array // even if the opcode only works at perf-time. static inline void tabensure_init(CSOUND csound, ARRAYDAT p, int size) { size_t ss = p->arrayMemberSizesize; if (p->data == NULL) { CS_VARIABLE var = p->arrayType->createVariable(csound, NULL); p->arrayMemberSize = var->memBlockSize; p->data = (MYFLT)csound->Calloc(csound, ss); p->allocated = ss; } else if (ss > p->allocated) { p->data = (MYFLT)csound->ReAlloc(csound, p->data, ss); p->allocated = ss; } p->sizes[0] = size; p->dimensions = 1; } // this should be called at perf time static inline void tabensure_perf(CSOUND csound, ARRAYDAT p, int size) { if(p->data == NULL) { return csound->PerformanceError(...); } size_t ss = p->arrayMemberSize*size; if(ss > p->allocated) { return csound->PerformanceError(...); } p->sizes[0] = size; } With this scheme, all allocation and resizing would happen at init time and any attempt to resize at perf time would result in an error. The user then can preallocate arrays as big as needed: kMyArray[] init i_largestsize possible Would that be an acceptable solution? On 25.04.19 21:50, Victor Lazzarini wrote: I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe operation where memory allocation can only happen at init-time on a separate thread. If this is a solution to a problem, then it’s not the right solution. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 On 25 Apr 2019, at 19:38, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. OSClisten kk[] iSS OSCraw S[]k i abs k[] k[] bformdec1 a[] ia[] bformenc1 a[] akk bpf k[] k[]M bpfcos k[] k[]M c2r k[] k[] cbrt k[] k[] ceil k[] k[] ceps k[] k[]k cepsinv k[] k[] cmp k[] kSk[]Sk cmp k[] k[]Sk cmp k[] k[]Sk[] cmplxprod k[] k[]k[] cos k[] k[] cosh k[] k[] cosinv k[] k[] dct k[] k[] dctinv k[] k[] deinterleave k[]k[] k[] directory S[] SN diskin a[] SPooooooo diskin a[] iPooooooo diskin2 a[] SPooooooo diskin2 a[] iPooooooo exp k[] k[] fft k[] k[] fftinv k[] k[] fillarray S[] W fillarray k[] S fillarray k[] m fillarray k[] z floor k[] k[] fluidInfo S[] i fmax k[] k[]k fmax k[] k[]k[] fmin k[] k[]k fmin k[] k[]k[] fmod k[] k[]k fmod k[] k[]k[] frac k[] k[] ftom k[] k[]o genarray k[] kkp genarray_i k[] iip getcol k[] k[]k getrow k[] k[]k getrowlin k[] k[]kOOP getrowlin k[] kiiooop hypot k[] k[]k[] in a[] (null) in a[] (null) init .[] m inletv a[] S int k[] k[] interleave k[] k[]k[] limit k[] k[]kk limit1 k[] k[] linlin k[] k[]kkOP linlin k[] kk[]k[]OP log k[] k[] log k[] k[]i log10 k[] k[] log2 k[] k[] loscilx a[] xkioojjoo mags k[] k[] maparray k[] k[]S mfb k[] k[]kki monitor a[] (null) mtof k[] k[] passign k[] po phs k[] k[] pol2rect k[] k[] pol2rect k[] k[]k[] pow k[] k[]k pow k[] k[]k[] powoftwo k[] k[] pows k[] k[] pvsceps k[] fo pvstrace fk[] fko r2c k[] k[] rect2pol k[] k[] rfft k[] k[] rifft k[] k[] round k[] k[] setcol k[] k[]k setrow k[] k[]k shiftin k[] a sin k[] k[] sinh k[] k[] sininv k[] k[] slicearray S[] S[]iip slicearray a[] a[]iip slicearray k[] k[]iip sorta k[] k[] sortd k[] k[] sqrt k[] k[] string2array k[] S tab2array k[] iOOP tan k[] k[] tanh k[] k[] taninv k[] k[] taninv k[] k[]k[] unwrap k[] k[] vbap a[] akOOo vbapg k[] kOOo vbapgmove k[] iiim vbapmove a[] aiiim window k[] k[]Op

Date	2019-04-28 11:32
From	Mauro Giubileo
Subject	Re: [Csnd-dev] Array corrections
	Another thing I was thinking... You wrote (by the way, I think there is a little error in the second fillarray. It should be "karr = fillarray(5,6,7,8)" or "karr fillarray 5,6,7,8"): karr[] fillarray 1,2,3 ...1. do something... karr = fillarray 5,6,7,8 ... 2. do something ... so that in code section 1, karr should have a size of 3 at runtime, while code section 2, karr would have a size of 4. When you write "at runtime" you mean "at perf-time"? I'm asking this because, in the above code, the first and the second fillarray run both at i-time, so the perf-time instructions in the code sections 1 and 2 would always see karr as it's declared in the second fillarray (i.e.: 5,6,7,8). --- Mauro Il 2019-04-26 00:57 Steven Yi ha scritto: I was thinking through this and realized even if we say array sizes can only change at init-time, we may still need to set sizes at runtime. It dawned on me that the situation occurs if you reuse an array var: karr[] fillarray 1,2,3 ...1. do something... karr = fillarray 5,6,7,8 ... 2. do something ... so that in code section 1, karr should have a size of 3 at runtime, while code section 2, karr would have a size of 4. So it seems back to two tabensures, one at init and one for perf. On Thu, Apr 25, 2019 at 6:41 PM Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Given the requirement that there can't be allocation at perf-time, the semantics for arrays should be: * an array has an allocated size and a perceived size * the allocated size is given at init time and can't be modified after that * the output of any array operation should either result in an array of the expected size or in an error. This needs to be enfor * for element wise operations, like arithmetic between an array and a scalar, the output array should be the same size / shape as the input array, or result in a performance error if this can't be achieved If there is no agreement in the semantics, no code can be correct. There are two issues here: 1) array growth. With the implementation in the array-init-change branch, arrays can be reallocated at perf-time if they need to grow beyond allocated size. According to Victor this needs to be prevented in order to ensure realtime safety. That means that tabensure, as written there, can't be called at perf-time. 2) correct perceived size. At perf-time it must be ensured that an array has the correct perceived size, as long as this size is <= allocated size. This operation does not imply any memory allocation. With my proposed implementation all these constraints are satisfied. Any other solution is either semantically flawed (the currect situation, where no update of perceived size is done, size depends on allocated size, even on previous size in previous instances of the instrument, etc) or does not meet the required conditions (realtime safety) On 26.04.19 00:13, Steven Yi wrote: The code change for tabensure (the one in the array-init-change branch) seems good to me. If the problem is that it should not be called at perf-time, then existing opcodes have problems and need to be rewritten. I'd rather we not have tabensure_perf and just document "only use at init-time". On Thu, Apr 25, 2019 at 5:33 PM Victor Lazzarini <Victor.Lazzarini@mu.ie> wrote: Sounds like the correct thing to me. As with other dynamic memory uses by opcodes, we only allocate or reallocate at init-time (e.g. we don't change delay line max sizes at perf time). Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 25 Apr 2019, at 22:21, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: If this is a requirement, then the array semantics need to be redefined and made clear. There can't be any operation at perf time which can grow an array. Even further, all perf time array operation need to operate with initialized arrays, since that would mean a perf time allocation. Following this semantics, we could go back to two tabensure functions in the form: // this should be called at init time by any opcode returning an array // even if the opcode only works at perf-time. static inline void tabensure_init(CSOUND csound, ARRAYDAT p, int size) { size_t ss = p->arrayMemberSizesize; if (p->data == NULL) { CS_VARIABLE var = p->arrayType->createVariable(csound, NULL); p->arrayMemberSize = var->memBlockSize; p->data = (MYFLT)csound->Calloc(csound, ss); p->allocated = ss; } else if (ss > p->allocated) { p->data = (MYFLT)csound->ReAlloc(csound, p->data, ss); p->allocated = ss; } p->sizes[0] = size; p->dimensions = 1; } // this should be called at perf time static inline void tabensure_perf(CSOUND csound, ARRAYDAT p, int size) { if(p->data == NULL) { return csound->PerformanceError(...); } size_t ss = p->arrayMemberSize*size; if(ss > p->allocated) { return csound->PerformanceError(...); } p->sizes[0] = size; } With this scheme, all allocation and resizing would happen at init time and any attempt to resize at perf time would result in an error. The user then can preallocate arrays as big as needed: kMyArray[] init i_largestsize possible Would that be an acceptable solution? On 25.04.19 21:50, Victor Lazzarini wrote: I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe operation where memory allocation can only happen at init-time on a separate thread. If this is a solution to a problem, then it's not the right solution. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 On 25 Apr 2019, at 19:38, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. OSClisten kk[] iSS OSCraw S[]k i abs k[] k[] bformdec1 a[] ia[] bformenc1 a[] akk bpf k[] k[]M bpfcos k[] k[]M c2r k[] k[] cbrt k[] k[] ceil k[] k[] ceps k[] k[]k cepsinv k[] k[] cmp k[] kSk[]Sk cmp k[] k[]Sk cmp k[] k[]Sk[] cmplxprod k[] k[]k[] cos k[] k[] cosh k[] k[] cosinv k[] k[] dct k[] k[] dctinv k[] k[] deinterleave k[]k[] k[] directory S[] SN diskin a[] SPooooooo diskin a[] iPooooooo diskin2 a[] SPooooooo diskin2 a[] iPooooooo exp k[] k[] fft k[] k[] fftinv k[] k[] fillarray S[] W fillarray k[] S fillarray k[] m fillarray k[] z floor k[] k[] fluidInfo S[] i fmax k[] k[]k fmax k[] k[]k[] fmin k[] k[]k fmin k[] k[]k[] fmod k[] k[]k fmod k[] k[]k[] frac k[] k[] ftom k[] k[]o genarray k[] kkp genarray_i k[] iip getcol k[] k[]k getrow k[] k[]k getrowlin k[] k[]kOOP getrowlin k[] kiiooop hypot k[] k[]k[] in a[] (null) in a[] (null) init .[] m inletv a[] S int k[] k[] interleave k[] k[]k[] limit k[] k[]kk limit1 k[] k[] linlin k[] k[]kkOP linlin k[] kk[]k[]OP log k[] k[] log k[] k[]i log10 k[] k[] log2 k[] k[] loscilx a[] xkioojjoo mags k[] k[] maparray k[] k[]S mfb k[] k[]kki monitor a[] (null) mtof k[] k[] passign k[] po phs k[] k[] pol2rect k[] k[] pol2rect k[] k[]k[] pow k[] k[]k pow k[] k[]k[] powoftwo k[] k[] pows k[] k[] pvsceps k[] fo pvstrace fk[] fko r2c k[] k[] rect2pol k[] k[] rfft k[] k[] rifft k[] k[] round k[] k[] setcol k[] k[]k setrow k[] k[]k shiftin k[] a sin k[] k[] sinh k[] k[] sininv k[] k[] slicearray S[] S[]iip slicearray a[] a[]iip slicearray k[] k[]iip sorta k[] k[] sortd k[] k[] sqrt k[] k[] string2array k[] S tab2array k[] iOOP tan k[] k[] tanh k[] k[] taninv k[] k[] taninv k[] k[]k[] unwrap k[] k[] vbap a[] akOOo vbapg k[] kOOo vbapgmove k[] iiim vbapmove a[] aiiim window k[] k[]Op

Date	2019-04-28 11:46
From	Eduardo Moguillansky
Subject	Re: [Csnd-dev] Array corrections
	On 28.04.19 11:15, Victor Lazzarini wrote: Off the top of my head, it appears that all array operations need to have an i-time pass so that any allocation needed happens at that stage. In general we can say that for any perftime operation on arrays the size of the output (and of the input) MUST be known at i-time. All array operations MUST have an i-time pass which ensures that 1. allocated size >= output size (possibly allocating / growing the array) 2. perceived size = output size At perftime it must be ensured that: 1. allocated size >= output size, failing otherwise (no allocation at this point) 2. perceived size = output size Some opcodes need to be modified / removed, since the size of the output array depends on k-rate variables, which are not known at i-time. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 28 Apr 2019, at 09:36, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: What happens to an opcode like kA[] = kB[] * kx, where the size of kB is not known at init time? It needs to possibly allocate an initial size at init time, but size depends on values set later. Should opcodes like this require preallocated arrays? If allocation at perftime is banned, then this seems the case. On a side note,there are many other opcodes doing allocation at perftime, like string opcodes. Should these also require preallocation? On 28.04.19 00:43, Steven Yi wrote: I'm wondering if we need three functions: 1. array_size_ensure_init : Does size check and resizing (w/mem allocation) but doesn't set size at init time 2. array_size_init : Does size check and resizing, sets size 3. array_size_perf: Does size check and sets size, no resizing and gives error if allocated size too small #1 would be for something like slicearray version that runs at perf time. It'd have to ensure the size, but doesn't actually set the active size, since it doesn't do a calculation until run-time. #2 would be for opcodes that run at init-time. #3 would be for perf-time opcodes that need to set the size. The max size is allocated with #1 or #2 at init-time. Question: Do we need this? I'm thinking it might be a kind of safety for the situation where people mix the order of init- and perf-time array processing opcodes. On Thu, Apr 25, 2019 at 6:57 PM Steven Yi <stevenyi@gmail.com> wrote: I was thinking through this and realized even if we say array sizes can only change at init-time, we may still need to set sizes at runtime. It dawned on me that the situation occurs if you reuse an array var: karr[] fillarray 1,2,3 ...1. do something... karr = fillarray 5,6,7,8 ... 2. do something ... so that in code section 1, karr should have a size of 3 at runtime, while code section 2, karr would have a size of 4. So it seems back to two tabensures, one at init and one for perf. On Thu, Apr 25, 2019 at 6:41 PM Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Given the requirement that there can't be allocation at perf-time, the semantics for arrays should be: * an array has an allocated size and a perceived size * the allocated size is given at init time and can't be modified after that * the output of any array operation should either result in an array of the expected size or in an error. This needs to be enfor * for element wise operations, like arithmetic between an array and a scalar, the output array should be the same size / shape as the input array, or result in a performance error if this can't be achieved If there is no agreement in the semantics, no code can be correct. There are two issues here: 1) array growth. With the implementation in the array-init-change branch, arrays can be reallocated at perf-time if they need to grow beyond allocated size. According to Victor this needs to be prevented in order to ensure realtime safety. That means that tabensure, as written there, can't be called at perf-time. 2) correct perceived size. At perf-time it must be ensured that an array has the correct perceived size, as long as this size is <= allocated size. This operation does not imply any memory allocation. With my proposed implementation all these constraints are satisfied. Any other solution is either semantically flawed (the currect situation, where no update of perceived size is done, size depends on allocated size, even on previous size in previous instances of the instrument, etc) or does not meet the required conditions (realtime safety) On 26.04.19 00:13, Steven Yi wrote: The code change for tabensure (the one in the array-init-change branch) seems good to me. If the problem is that it should not be called at perf-time, then existing opcodes have problems and need to be rewritten. I'd rather we not have tabensure_perf and just document "only use at init-time". On Thu, Apr 25, 2019 at 5:33 PM Victor Lazzarini <Victor.Lazzarini@mu.ie> wrote: Sounds like the correct thing to me. As with other dynamic memory uses by opcodes, we only allocate or reallocate at init-time (e.g. we don't change delay line max sizes at perf time). Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 25 Apr 2019, at 22:21, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: If this is a requirement, then the array semantics need to be redefined and made clear. There can't be any operation at perf time which can grow an array. Even further, all perf time array operation need to operate with initialized arrays, since that would mean a perf time allocation. Following this semantics, we could go back to two tabensure functions in the form: // this should be called at init time by any opcode returning an array // even if the opcode only works at perf-time. static inline void tabensure_init(CSOUND csound, ARRAYDAT p, int size) { size_t ss = p->arrayMemberSizesize; if (p->data == NULL) { CS_VARIABLE var = p->arrayType->createVariable(csound, NULL); p->arrayMemberSize = var->memBlockSize; p->data = (MYFLT)csound->Calloc(csound, ss); p->allocated = ss; } else if (ss > p->allocated) { p->data = (MYFLT)csound->ReAlloc(csound, p->data, ss); p->allocated = ss; } p->sizes[0] = size; p->dimensions = 1; } // this should be called at perf time static inline void tabensure_perf(CSOUND csound, ARRAYDAT p, int size) { if(p->data == NULL) { return csound->PerformanceError(...); } size_t ss = p->arrayMemberSize*size; if(ss > p->allocated) { return csound->PerformanceError(...); } p->sizes[0] = size; } With this scheme, all allocation and resizing would happen at init time and any attempt to resize at perf time would result in an error. The user then can preallocate arrays as big as needed: kMyArray[] init i_largestsize possible Would that be an acceptable solution? On 25.04.19 21:50, Victor Lazzarini wrote: I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe operation where memory allocation can only happen at init-time on a separate thread. If this is a solution to a problem, then it’s not the right solution. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 On 25 Apr 2019, at 19:38, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. OSClisten kk[] iSS OSCraw S[]k i abs k[] k[] bformdec1 a[] ia[] bformenc1 a[] akk bpf k[] k[]M bpfcos k[] k[]M c2r k[] k[] cbrt k[] k[] ceil k[] k[] ceps k[] k[]k cepsinv k[] k[] cmp k[] kSk[]Sk cmp k[] k[]Sk cmp k[] k[]Sk[] cmplxprod k[] k[]k[] cos k[] k[] cosh k[] k[] cosinv k[] k[] dct k[] k[] dctinv k[] k[] deinterleave k[]k[] k[] directory S[] SN diskin a[] SPooooooo diskin a[] iPooooooo diskin2 a[] SPooooooo diskin2 a[] iPooooooo exp k[] k[] fft k[] k[] fftinv k[] k[] fillarray S[] W fillarray k[] S fillarray k[] m fillarray k[] z floor k[] k[] fluidInfo S[] i fmax k[] k[]k fmax k[] k[]k[] fmin k[] k[]k fmin k[] k[]k[] fmod k[] k[]k fmod k[] k[]k[] frac k[] k[] ftom k[] k[]o genarray k[] kkp genarray_i k[] iip getcol k[] k[]k getrow k[] k[]k getrowlin k[] k[]kOOP getrowlin k[] kiiooop hypot k[] k[]k[] in a[] (null) in a[] (null) init .[] m inletv a[] S int k[] k[] interleave k[] k[]k[] limit k[] k[]kk limit1 k[] k[] linlin k[] k[]kkOP linlin k[] kk[]k[]OP log k[] k[] log k[] k[]i log10 k[] k[] log2 k[] k[] loscilx a[] xkioojjoo mags k[] k[] maparray k[] k[]S mfb k[] k[]kki monitor a[] (null) mtof k[] k[] passign k[] po phs k[] k[] pol2rect k[] k[] pol2rect k[] k[]k[] pow k[] k[]k pow k[] k[]k[] powoftwo k[] k[] pows k[] k[] pvsceps k[] fo pvstrace fk[] fko r2c k[] k[] rect2pol k[] k[] rfft k[] k[] rifft k[] k[] round k[] k[] setcol k[] k[]k setrow k[] k[]k shiftin k[] a sin k[] k[] sinh k[] k[] sininv k[] k[] slicearray S[] S[]iip slicearray a[] a[]iip slicearray k[] k[]iip sorta k[] k[] sortd k[] k[] sqrt k[] k[] string2array k[] S tab2array k[] iOOP tan k[] k[] tanh k[] k[] taninv k[] k[] taninv k[] k[]k[] unwrap k[] k[] vbap a[] akOOo vbapg k[] kOOo vbapgmove k[] iiim vbapmove a[] aiiim window k[] k[]Op

Date	2019-04-28 11:54
From	Victor Lazzarini
Subject	Re: [Csnd-dev] Array corrections
	In cases where a k-rate variable is used to determine the size of an array we will need to do something about it. That's not all the cases, because a k array can be created with 'init' (etc) with a given size, which is a common use. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 28 Apr 2019, at 11:47, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: On 28.04.19 11:15, Victor Lazzarini wrote: Off the top of my head, it appears that all array operations need to have an i-time pass so that any allocation needed happens at that stage. In general we can say that for any perftime operation on arrays the size of the output (and of the input) MUST be known at i-time. All array operations MUST have an i-time pass which ensures that 1. allocated size >= output size (possibly allocating / growing the array) 2. perceived size = output size At perftime it must be ensured that: 1. allocated size >= output size, failing otherwise (no allocation at this point) 2. perceived size = output size Some opcodes need to be modified / removed, since the size of the output array depends on k-rate variables, which are not known at i-time. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 28 Apr 2019, at 09:36, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: What happens to an opcode like kA[] = kB[] * kx, where the size of kB is not known at init time? It needs to possibly allocate an initial size at init time, but size depends on values set later. Should opcodes like this require preallocated arrays? If allocation at perftime is banned, then this seems the case. On a side note,there are many other opcodes doing allocation at perftime, like string opcodes. Should these also require preallocation? On 28.04.19 00:43, Steven Yi wrote: I'm wondering if we need three functions: 1. array_size_ensure_init : Does size check and resizing (w/mem allocation) but doesn't set size at init time 2. array_size_init : Does size check and resizing, sets size 3. array_size_perf: Does size check and sets size, no resizing and gives error if allocated size too small #1 would be for something like slicearray version that runs at perf time. It'd have to ensure the size, but doesn't actually set the active size, since it doesn't do a calculation until run-time. #2 would be for opcodes that run at init-time. #3 would be for perf-time opcodes that need to set the size. The max size is allocated with #1 or #2 at init-time. Question: Do we need this? I'm thinking it might be a kind of safety for the situation where people mix the order of init- and perf-time array processing opcodes. On Thu, Apr 25, 2019 at 6:57 PM Steven Yi <stevenyi@gmail.com> wrote: I was thinking through this and realized even if we say array sizes can only change at init-time, we may still need to set sizes at runtime. It dawned on me that the situation occurs if you reuse an array var: karr[] fillarray 1,2,3 ...1. do something... karr = fillarray 5,6,7,8 ... 2. do something ... so that in code section 1, karr should have a size of 3 at runtime, while code section 2, karr would have a size of 4. So it seems back to two tabensures, one at init and one for perf. On Thu, Apr 25, 2019 at 6:41 PM Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Given the requirement that there can't be allocation at perf-time, the semantics for arrays should be: * an array has an allocated size and a perceived size * the allocated size is given at init time and can't be modified after that * the output of any array operation should either result in an array of the expected size or in an error. This needs to be enfor * for element wise operations, like arithmetic between an array and a scalar, the output array should be the same size / shape as the input array, or result in a performance error if this can't be achieved If there is no agreement in the semantics, no code can be correct. There are two issues here: 1) array growth. With the implementation in the array-init-change branch, arrays can be reallocated at perf-time if they need to grow beyond allocated size. According to Victor this needs to be prevented in order to ensure realtime safety. That means that tabensure, as written there, can't be called at perf-time. 2) correct perceived size. At perf-time it must be ensured that an array has the correct perceived size, as long as this size is <= allocated size. This operation does not imply any memory allocation. With my proposed implementation all these constraints are satisfied. Any other solution is either semantically flawed (the currect situation, where no update of perceived size is done, size depends on allocated size, even on previous size in previous instances of the instrument, etc) or does not meet the required conditions (realtime safety) On 26.04.19 00:13, Steven Yi wrote: The code change for tabensure (the one in the array-init-change branch) seems good to me. If the problem is that it should not be called at perf-time, then existing opcodes have problems and need to be rewritten. I'd rather we not have tabensure_perf and just document "only use at init-time". On Thu, Apr 25, 2019 at 5:33 PM Victor Lazzarini <Victor.Lazzarini@mu.ie> wrote: Sounds like the correct thing to me. As with other dynamic memory uses by opcodes, we only allocate or reallocate at init-time (e.g. we don't change delay line max sizes at perf time). Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 25 Apr 2019, at 22:21, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: If this is a requirement, then the array semantics need to be redefined and made clear. There can't be any operation at perf time which can grow an array. Even further, all perf time array operation need to operate with initialized arrays, since that would mean a perf time allocation. Following this semantics, we could go back to two tabensure functions in the form: // this should be called at init time by any opcode returning an array // even if the opcode only works at perf-time. static inline void tabensure_init(CSOUND csound, ARRAYDAT p, int size) { size_t ss = p->arrayMemberSizesize; if (p->data == NULL) { CS_VARIABLE var = p->arrayType->createVariable(csound, NULL); p->arrayMemberSize = var->memBlockSize; p->data = (MYFLT)csound->Calloc(csound, ss); p->allocated = ss; } else if (ss > p->allocated) { p->data = (MYFLT)csound->ReAlloc(csound, p->data, ss); p->allocated = ss; } p->sizes[0] = size; p->dimensions = 1; } // this should be called at perf time static inline void tabensure_perf(CSOUND csound, ARRAYDAT p, int size) { if(p->data == NULL) { return csound->PerformanceError(...); } size_t ss = p->arrayMemberSize*size; if(ss > p->allocated) { return csound->PerformanceError(...); } p->sizes[0] = size; } With this scheme, all allocation and resizing would happen at init time and any attempt to resize at perf time would result in an error. The user then can preallocate arrays as big as needed: kMyArray[] init i_largestsize possible Would that be an acceptable solution? On 25.04.19 21:50, Victor Lazzarini wrote: I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe operation where memory allocation can only happen at init-time on a separate thread. If this is a solution to a problem, then it’s not the right solution. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 On 25 Apr 2019, at 19:38, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. OSClisten kk[] iSS OSCraw S[]k i abs k[] k[] bformdec1 a[] ia[] bformenc1 a[] akk bpf k[] k[]M bpfcos k[] k[]M c2r k[] k[] cbrt k[] k[] ceil k[] k[] ceps k[] k[]k cepsinv k[] k[] cmp k[] kSk[]Sk cmp k[] k[]Sk cmp k[] k[]Sk[] cmplxprod k[] k[]k[] cos k[] k[] cosh k[] k[] cosinv k[] k[] dct k[] k[] dctinv k[] k[] deinterleave k[]k[] k[] directory S[] SN diskin a[] SPooooooo diskin a[] iPooooooo diskin2 a[] SPooooooo diskin2 a[] iPooooooo exp k[] k[] fft k[] k[] fftinv k[] k[] fillarray S[] W fillarray k[] S fillarray k[] m fillarray k[] z floor k[] k[] fluidInfo S[] i fmax k[] k[]k fmax k[] k[]k[] fmin k[] k[]k fmin k[] k[]k[] fmod k[] k[]k fmod k[] k[]k[] frac k[] k[] ftom k[] k[]o genarray k[] kkp genarray_i k[] iip getcol k[] k[]k getrow k[] k[]k getrowlin k[] k[]kOOP getrowlin k[] kiiooop hypot k[] k[]k[] in a[] (null) in a[] (null) init .[] m inletv a[] S int k[] k[] interleave k[] k[]k[] limit k[] k[]kk limit1 k[] k[] linlin k[] k[]kkOP linlin k[] kk[]k[]OP log k[] k[] log k[] k[]i log10 k[] k[] log2 k[] k[] loscilx a[] xkioojjoo mags k[] k[] maparray k[] k[]S mfb k[] k[]kki monitor a[] (null) mtof k[] k[] passign k[] po phs k[] k[] pol2rect k[] k[] pol2rect k[] k[]k[] pow k[] k[]k pow k[] k[]k[] powoftwo k[] k[] pows k[] k[] pvsceps k[] fo pvstrace fk[] fko r2c k[] k[] rect2pol k[] k[] rfft k[] k[] rifft k[] k[] round k[] k[] setcol k[] k[]k setrow k[] k[]k shiftin k[] a sin k[] k[] sinh k[] k[] sininv k[] k[] slicearray S[] S[]iip slicearray a[] a[]iip slicearray k[] k[]iip sorta k[] k[] sortd k[] k[] sqrt k[] k[] string2array k[] S tab2array k[] iOOP tan k[] k[] tanh k[] k[] taninv k[] k[] taninv k[] k[]k[] unwrap k[] k[] vbap a[] akOOo vbapg k[] kOOo vbapgmove k[] iiim vbapmove a[] aiiim window k[] k[]Op

Date	2019-04-28 12:39
From	Eduardo Moguillansky
Subject	Re: [Csnd-dev] Array corrections
	There are actually very few opcodes which output an array with an unknown size at i-time. * genarray with k-values * trim with k-values * getrowlin with k-values * tab2array with k-values Such opcodes should ensure that the output array has already been initialized and should not touch the size at init time (since they can't determine the size) At perf-time they should work like all other opcodes, failing if an allocation needs to be done. So in the end I agree with Steven that we need three functions. On 28.04.19 12:54, Victor Lazzarini wrote: In cases where a k-rate variable is used to determine the size of an array we will need to do something about it. That's not all the cases, because a k array can be created with 'init' (etc) with a given size, which is a common use. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 28 Apr 2019, at 11:47, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: On 28.04.19 11:15, Victor Lazzarini wrote: Off the top of my head, it appears that all array operations need to have an i-time pass so that any allocation needed happens at that stage. In general we can say that for any perftime operation on arrays the size of the output (and of the input) MUST be known at i-time. All array operations MUST have an i-time pass which ensures that 1. allocated size >= output size (possibly allocating / growing the array) 2. perceived size = output size At perftime it must be ensured that: 1. allocated size >= output size, failing otherwise (no allocation at this point) 2. perceived size = output size Some opcodes need to be modified / removed, since the size of the output array depends on k-rate variables, which are not known at i-time. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 28 Apr 2019, at 09:36, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: What happens to an opcode like kA[] = kB[] * kx, where the size of kB is not known at init time? It needs to possibly allocate an initial size at init time, but size depends on values set later. Should opcodes like this require preallocated arrays? If allocation at perftime is banned, then this seems the case. On a side note,there are many other opcodes doing allocation at perftime, like string opcodes. Should these also require preallocation? On 28.04.19 00:43, Steven Yi wrote: I'm wondering if we need three functions: 1. array_size_ensure_init : Does size check and resizing (w/mem allocation) but doesn't set size at init time 2. array_size_init : Does size check and resizing, sets size 3. array_size_perf: Does size check and sets size, no resizing and gives error if allocated size too small #1 would be for something like slicearray version that runs at perf time. It'd have to ensure the size, but doesn't actually set the active size, since it doesn't do a calculation until run-time. #2 would be for opcodes that run at init-time. #3 would be for perf-time opcodes that need to set the size. The max size is allocated with #1 or #2 at init-time. Question: Do we need this? I'm thinking it might be a kind of safety for the situation where people mix the order of init- and perf-time array processing opcodes. On Thu, Apr 25, 2019 at 6:57 PM Steven Yi <stevenyi@gmail.com> wrote: I was thinking through this and realized even if we say array sizes can only change at init-time, we may still need to set sizes at runtime. It dawned on me that the situation occurs if you reuse an array var: karr[] fillarray 1,2,3 ...1. do something... karr = fillarray 5,6,7,8 ... 2. do something ... so that in code section 1, karr should have a size of 3 at runtime, while code section 2, karr would have a size of 4. So it seems back to two tabensures, one at init and one for perf. On Thu, Apr 25, 2019 at 6:41 PM Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Given the requirement that there can't be allocation at perf-time, the semantics for arrays should be: * an array has an allocated size and a perceived size * the allocated size is given at init time and can't be modified after that * the output of any array operation should either result in an array of the expected size or in an error. This needs to be enfor * for element wise operations, like arithmetic between an array and a scalar, the output array should be the same size / shape as the input array, or result in a performance error if this can't be achieved If there is no agreement in the semantics, no code can be correct. There are two issues here: 1) array growth. With the implementation in the array-init-change branch, arrays can be reallocated at perf-time if they need to grow beyond allocated size. According to Victor this needs to be prevented in order to ensure realtime safety. That means that tabensure, as written there, can't be called at perf-time. 2) correct perceived size. At perf-time it must be ensured that an array has the correct perceived size, as long as this size is <= allocated size. This operation does not imply any memory allocation. With my proposed implementation all these constraints are satisfied. Any other solution is either semantically flawed (the currect situation, where no update of perceived size is done, size depends on allocated size, even on previous size in previous instances of the instrument, etc) or does not meet the required conditions (realtime safety) On 26.04.19 00:13, Steven Yi wrote: The code change for tabensure (the one in the array-init-change branch) seems good to me. If the problem is that it should not be called at perf-time, then existing opcodes have problems and need to be rewritten. I'd rather we not have tabensure_perf and just document "only use at init-time". On Thu, Apr 25, 2019 at 5:33 PM Victor Lazzarini <Victor.Lazzarini@mu.ie> wrote: Sounds like the correct thing to me. As with other dynamic memory uses by opcodes, we only allocate or reallocate at init-time (e.g. we don't change delay line max sizes at perf time). Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland On 25 Apr 2019, at 22:21, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: If this is a requirement, then the array semantics need to be redefined and made clear. There can't be any operation at perf time which can grow an array. Even further, all perf time array operation need to operate with initialized arrays, since that would mean a perf time allocation. Following this semantics, we could go back to two tabensure functions in the form: // this should be called at init time by any opcode returning an array // even if the opcode only works at perf-time. static inline void tabensure_init(CSOUND csound, ARRAYDAT p, int size) { size_t ss = p->arrayMemberSizesize; if (p->data == NULL) { CS_VARIABLE var = p->arrayType->createVariable(csound, NULL); p->arrayMemberSize = var->memBlockSize; p->data = (MYFLT)csound->Calloc(csound, ss); p->allocated = ss; } else if (ss > p->allocated) { p->data = (MYFLT)csound->ReAlloc(csound, p->data, ss); p->allocated = ss; } p->sizes[0] = size; p->dimensions = 1; } // this should be called at perf time static inline void tabensure_perf(CSOUND csound, ARRAYDAT p, int size) { if(p->data == NULL) { return csound->PerformanceError(...); } size_t ss = p->arrayMemberSize*size; if(ss > p->allocated) { return csound->PerformanceError(...); } p->sizes[0] = size; } With this scheme, all allocation and resizing would happen at init time and any attempt to resize at perf time would result in an error. The user then can preallocate arrays as big as needed: kMyArray[] init i_largestsize possible Would that be an acceptable solution? On 25.04.19 21:50, Victor Lazzarini wrote: I have been largely keeping out of this discussion, but this email is sounding alarm bells here. The tabensure() function calls calloc() and that absolutely should be kept out of perf-time. It will break platforms (such as Bela) that require realtime safe operation where memory allocation can only happen at init-time on a separate thread. If this is a solution to a problem, then it’s not the right solution. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 On 25 Apr 2019, at 19:38, Eduardo Moguillansky <eduardo.moguillansky@gmail.com> wrote: Now that the array semantics seem clear (at least for 1D arrays) there is a big rewrite ahead comprising all opcodes which have arrays as output, which need to comply with this behaviour by calling tabensure at perf-time. The complete list of opcode which need to be revised is below (also all opcodes dealing with array arithmetic, which are not listed here). There are probably some of those which already do the right thing but most need to have the call to tabensure added. OSClisten kk[] iSS OSCraw S[]k i abs k[] k[] bformdec1 a[] ia[] bformenc1 a[] akk bpf k[] k[]M bpfcos k[] k[]M c2r k[] k[] cbrt k[] k[] ceil k[] k[] ceps k[] k[]k cepsinv k[] k[] cmp k[] kSk[]Sk cmp k[] k[]Sk cmp k[] k[]Sk[] cmplxprod k[] k[]k[] cos k[] k[] cosh k[] k[] cosinv k[] k[] dct k[] k[] dctinv k[] k[] deinterleave k[]k[] k[] directory S[] SN diskin a[] SPooooooo diskin a[] iPooooooo diskin2 a[] SPooooooo diskin2 a[] iPooooooo exp k[] k[] fft k[] k[] fftinv k[] k[] fillarray S[] W fillarray k[] S fillarray k[] m fillarray k[] z floor k[] k[] fluidInfo S[] i fmax k[] k[]k fmax k[] k[]k[] fmin k[] k[]k fmin k[] k[]k[] fmod k[] k[]k fmod k[] k[]k[] frac k[] k[] ftom k[] k[]o genarray k[] kkp genarray_i k[] iip getcol k[] k[]k getrow k[] k[]k getrowlin k[] k[]kOOP getrowlin k[] kiiooop hypot k[] k[]k[] in a[] (null) in a[] (null) init .[] m inletv a[] S int k[] k[] interleave k[] k[]k[] limit k[] k[]kk limit1 k[] k[] linlin k[] k[]kkOP linlin k[] kk[]k[]OP log k[] k[] log k[] k[]i log10 k[] k[] log2 k[] k[] loscilx a[] xkioojjoo mags k[] k[] maparray k[] k[]S mfb k[] k[]kki monitor a[] (null) mtof k[] k[] passign k[] po phs k[] k[] pol2rect k[] k[] pol2rect k[] k[]k[] pow k[] k[]k pow k[] k[]k[] powoftwo k[] k[] pows k[] k[] pvsceps k[] fo pvstrace fk[] fko r2c k[] k[] rect2pol k[] k[] rfft k[] k[] rifft k[] k[] round k[] k[] setcol k[] k[]k setrow k[] k[]k shiftin k[] a sin k[] k[] sinh k[] k[] sininv k[] k[] slicearray S[] S[]iip slicearray a[] a[]iip slicearray k[] k[]iip sorta k[] k[] sortd k[] k[] sqrt k[] k[] string2array k[] S tab2array k[] iOOP tan k[] k[] tanh k[] k[] taninv k[] k[] taninv k[] k[]k[] unwrap k[] k[] vbap a[] akOOo vbapg k[] kOOo vbapgmove k[] iiim vbapmove a[] aiiim window k[] k[]Op

Date	2019-04-28 16:50
From	john
Subject	Re: [Csnd-dev] Array corrections
	It seems to me simpler if an array size is set on i-time by init or fillarray and cannot be changed; attempts to make larger are an error and making smaller might be OK as we can adjust the perceived size, or it is an error. That might require more calls to init but it is clear and explainable. On Sat, 27 Apr 2019, Steven Yi wrote: > I'm wondering if we need three functions: > > 1. array_size_ensure_init : Does size check and resizing (w/mem allocation) > but doesn't set size at init time > 2. array_size_init : Does size check and resizing, sets size > 3. array_size_perf: Does size check and sets size, no resizing and gives error > if allocated size too small > > #1 would be for something like slicearray version that runs at perf time. It'd > have to ensure the size, but doesn't actually set the active size, since it > doesn't do a calculation until run-time. > > #2 would be for opcodes that run at init-time. > > #3 would be for perf-time opcodes that need to set the size. The max size is > allocated with #1 or #2 at init-time. > > Question: Do we need this? I'm thinking it might be a kind of safety for the > situation where people mix the order of init- and perf-time array processing > opcodes. > > > > On Thu, Apr 25, 2019 at 6:57 PM Steven Yi wrote: > I was thinking through this and realized even if we say array sizes can > only change at init-time, we may still need to set sizes at runtime. It > dawned on me that the situation occurs if you reuse an array var: > > karr[] fillarray 1,2,3 > ...1. do something... > karr = fillarray 5,6,7,8 > ... 2. do something ... > > so that in code section 1, karr should have a size of 3 at runtime, > while code section 2, karr would have a size of 4. > > So it seems back to two tabensures, one at init and one for perf. > > > > > > On Thu, Apr 25, 2019 at 6:41 PM Eduardo Moguillansky > wrote: > Given the requirement that there can't be allocation at > perf-time, the semantics for arrays should be: > > * an array has an allocated size and a perceived size > * the allocated size is given at init time and can't be modified > after that > * the output of any array operation should either result in an > array of the expected size or in an error. This needs to be enfor > * for element wise operations, like arithmetic between an array > and a scalar, the output array should be the same size / shape as > the input array, or result in a performance error if this can't be > achieved > > If there is no agreement in the semantics, no code can be correct. > > There are two issues here: > > 1) array growth. With the implementation in the array-init-change > branch, arrays can be reallocated at perf-time if they need to > grow beyond allocated size. According to Victor this needs to be > prevented in order to ensure realtime safety. That means that > tabensure, as written there, can't be called at perf-time. > 2) correct perceived size. At perf-time it must be ensured that an > array has the correct perceived size, as long as this size is <= > allocated size. This operation does not imply any memory > allocation. > > With my proposed implementation all these constraints are > satisfied. Any other solution is either semantically flawed (the > currect situation, where no update of perceived size is done, size > depends on allocated size, even on previous size in previous > instances of the instrument, etc) or does not meet the required > conditions (realtime safety) > > > On 26.04.19 00:13, Steven Yi wrote: > The code change for tabensure (the one in the > array-init-change branch) seems good to me. If the > problem is that it should not be called at perf-time, > then existing opcodes have problems and need to be > rewritten. I'd rather we not have tabensure_perf and > just document "only use at init-time". > > On Thu, Apr 25, 2019 at 5:33 PM Victor Lazzarini > wrote: > Sounds like the correct thing to me. As with > other dynamic memory uses by opcodes, we only > allocate or reallocate at init-time (e.g. we > don't change delay line max sizes at perf time). > Victor Lazzarini Dean of Arts, Celtic Studies, > and Philosophy > Maynooth University > Ireland > > On 25 Apr 2019, at 22:21, Eduardo Moguillansky > wrote: > > If this is a requirement, then the array > semantics need to be redefined and made > clear. There can't be any operation at > perf time which can grow an array. Even > further, all perf time array operation > need to operate with initialized arrays, > since that would mean a perf time > allocation. Following this semantics, we > could go back to two tabensure functions > in the form: > > > // this should be called at init time by any opcode returning an array > // even if the opcode only works at perf-time. > static inline void tabensure_init(CSOUND csound, ARRAYDAT p, int size) { > size_t ss = p->arrayMemberSizesize; > if (p->data == NULL) { > CS_VARIABLE var = p->arrayType->createVariable(csound, NULL); > p->arrayMemberSize = var->memBlockSize; > p->data = (MYFLT)csound->Calloc(csound, ss); > p->allocated = ss; > } > else if (ss > p->allocated) { > p->data = (MYFLT)csound->ReAlloc(csound, p->data, ss); > p->allocated = ss; > } > p->sizes[0] = size; > p->dimensions = 1; > } > > // this should be called at perf time > static inline void tabensure_perf(CSOUND csound, ARRAYDAT p, int size) { > if(p->data == NULL) { > return csound->PerformanceError(...); > } > size_t ss = p->arrayMemberSize*size; > if(ss > p->allocated) { > return csound->PerformanceError(...); > } > p->sizes[0] = size; > } > > With this scheme, all allocation and resizing would happen at init time and > any attempt to resize at perf time would result in an error. The user then > can preallocate arrays as big as needed: > > kMyArray[] init i_largestsize possible > > Would that be an acceptable solution? > > > On 25.04.19 21:50, Victor Lazzarini wrote: > > I have been largely keeping out of this discussion, but this email is sounding > alarm bells here. The tabensure() function calls > calloc() and that absolutely should be kept out of perf-time. It will break pl > atforms (such as Bela) that require realtime safe > operation where memory allocation can only happen at init-time on a separate t > hread. > > If this is a solution to a problem, then it’s not the right solution. > ======================== > Prof. Victor Lazzarini > Dean of Arts, Celtic Studies, and Philosophy, > Maynooth University, > Maynooth, Co Kildare, Ireland > Tel: 00 353 7086936 > Fax: 00 353 1 7086952 > > On 25 Apr 2019, at 19:38, Eduardo Moguillansky > wrote: > > Now that the array semantics seem clear (at least for 1D arrays) there is a bi > g rewrite ahead comprising all opcodes which have arrays as output, which need > to comply with this behaviour by calling tabensure at perf-time. The complete > list of opcode which need to be revised is below (also all opcodes dealing wi > th array arithmetic, which are not listed here). There are probably some of th > ose which already do the right thing but most need to have the call to tabensu > re added. > > OSClisten kk[] iSS > OSCraw S[]k i > abs k[] k[] > bformdec1 a[] ia[] > bformenc1 a[] akk > bpf k[] k[]M > bpfcos k[] k[]M > c2r k[] k[] > cbrt k[] k[] > ceil k[] k[] > ceps k[] k[]k > cepsinv k[] k[] > cmp k[] kSk[]Sk > cmp k[] k[]Sk > cmp k[] k[]Sk[] > cmplxprod k[] k[]k[] > cos k[] k[] > cosh k[] k[] > cosinv k[] k[] > dct k[] k[] > dctinv k[] k[] > deinterleave k[]k[] k[] > directory S[] SN > diskin a[] SPooooooo > diskin a[] iPooooooo > diskin2 a[] SPooooooo > diskin2 a[] iPooooooo > exp k[] k[] > fft k[] k[] > fftinv k[] k[] > fillarray S[] W > fillarray k[] S > fillarray k[] m > fillarray k[] z > floor k[] k[] > fluidInfo S[] i > fmax k[] k[]k > fmax k[] k[]k[] > fmin k[] k[]k > fmin k[] k[]k[] > fmod k[] k[]k > fmod k[] k[]k[] > frac k[] k[] > ftom k[] k[]o > genarray k[] kkp > genarray_i k[] iip > getcol k[] k[]k > getrow k[] k[]k > getrowlin k[] k[]kOOP > getrowlin k[] kiiooop > hypot k[] k[]k[] > in a[] (null) > in a[] (null) > init .[] m > inletv a[] S > int k[] k[] > interleave k[] k[]k[] > limit k[] k[]kk > limit1 k[] k[] > linlin k[] k[]kkOP > linlin k[] kk[]k[]OP > log k[] k[] > log k[] k[]i > log10 k[] k[] > log2 k[] k[] > loscilx a[] xkioojjoo > mags k[] k[] > maparray k[] k[]S > mfb k[] k[]kki > monitor a[] (null) > mtof k[] k[] > passign k[] po > phs k[] k[] > pol2rect k[] k[] > pol2rect k[] k[]k[] > pow k[] k[]k > pow k[] k[]k[] > powoftwo k[] k[] > pows k[] k[] > pvsceps k[] fo > pvstrace fk[] fko > r2c k[] k[] > rect2pol k[] k[] > rfft k[] k[] > rifft k[] k[] > round k[] k[] > setcol k[] k[]k > setrow k[] k[]k > shiftin k[] a > sin k[] k[] > sinh k[] k[] > sininv k[] k[] > slicearray S[] S[]iip > slicearray a[] a[]iip > slicearray k[] k[]iip > sorta k[] k[] > sortd k[] k[] > sqrt k[] k[] > string2array k[] S > tab2array k[] iOOP > tan k[] k[] > tanh k[] k[] > taninv k[] k[] > taninv k[] k[]k[] > unwrap k[] k[] > vbap a[] akOOo > vbapg k[] kOOo > vbapgmove k[] iiim > vbapmove a[] aiiim > window k[] k[]Op > > > > >