[Cs-dev] opcode request - changing table size
Date | 2011-11-27 20:46 |
From | peiman khosravi |
Subject | [Cs-dev] opcode request - changing table size |
Hello, Would it be possible to have an opcode that let's one dynamically change table sizes? P ------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-novd2d _______________________________________________ Csound-devel mailing list Csound-devel@lists.sourceforge.net |
Date | 2011-11-27 21:03 |
From | jpff@cs.bath.ac.uk |
Subject | Re: [Cs-dev] opcode request - changing table size |
Yes; reasonably easy. Will try tomorrow (too much Lirac tonight) > Hello, > > Would it be possible to have an opcode that let's one dynamically > change table sizes? > > > P > ------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-novd2d _______________________________________________ Csound-devel mailing list Csound-devel@lists.sourceforge.net |
Date | 2011-11-27 21:27 |
From | Victor Lazzarini |
Subject | Re: [Cs-dev] opcode request - changing table size |
what do you mean? On 27 Nov 2011, at 20:46, peiman khosravi wrote: > Hello, > > Would it be possible to have an opcode that let's one dynamically > change table sizes? > > > P > > ------------------------------------------------------------------------------ > All the data continuously generated in your IT infrastructure > contains a definitive record of customers, application performance, > security threats, fraudulent activity, and more. Splunk takes this > data and makes sense of it. IT sense. And common sense. > http://p.sf.net/sfu/splunk-novd2d > _______________________________________________ > Csound-devel mailing list > Csound-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/csound-devel Dr Victor Lazzarini Senior Lecturer Dept. of Music NUI Maynooth Ireland tel.: +353 1 708 3545 Victor dot Lazzarini AT nuim dot ie ------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-novd2d _______________________________________________ Csound-devel mailing list Csound-devel@lists.sourceforge.net |
Date | 2011-11-27 22:06 |
From | Rory Walsh |
Subject | Re: [Cs-dev] opcode request - changing table size |
Attachments | None None |
Let's say you're doing some live writing to a table but you don't know how long the recording will be? It would be nice if every few seconds you could dynamically allocate more space for the table. At present one must allocate loads of space just in case. Know what I mean? On Sunday, 27 November 2011, Victor Lazzarini <Victor.Lazzarini@nuim.ie> wrote: > what do you mean? > > On 27 Nov 2011, at 20:46, peiman khosravi wrote: > >> Hello, >> >> Would it be possible to have an opcode that let's one dynamically >> change table sizes? >> >> >> P >> >> ------------------------------------------------------------------------------ >> All the data continuously generated in your IT infrastructure >> contains a definitive record of customers, application performance, >> security threats, fraudulent activity, and more. Splunk takes this >> data and makes sense of it. IT sense. And common sense. >> http://p.sf.net/sfu/splunk-novd2d >> _______________________________________________ >> Csound-devel mailing list >> Csound-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/csound-devel > > Dr Victor Lazzarini > Senior Lecturer > Dept. of Music > NUI Maynooth Ireland > tel.: +353 1 708 3545 > Victor dot Lazzarini AT nuim dot ie > > > > > ------------------------------------------------------------------------------ > All the data continuously generated in your IT infrastructure > contains a definitive record of customers, application performance, > security threats, fraudulent activity, and more. Splunk takes this > data and makes sense of it. IT sense. And common sense. > http://p.sf.net/sfu/splunk-novd2d > _______________________________________________ > Csound-devel mailing list > Csound-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/csound-devel > |
Date | 2011-11-27 22:18 |
From | peiman khosravi |
Subject | Re: [Cs-dev] opcode request - changing table size |
Yes! Thanks John :-) Best, Peiman On 27 November 2011 22:06, Rory Walsh |
Date | 2011-11-27 22:36 |
From | Victor Lazzarini |
Subject | Re: [Cs-dev] opcode request - changing table size |
Attachments | None None |
Ok, but even if such opcode is provided, it's never going to be as efficient for RT use than allocating a lot of space to start with. Reallocating involves copying a lot of memory, which is always costly. Victor On 27 Nov 2011, at 22:06, Rory Walsh wrote: Let's say you're doing some live writing to a table but you don't know how long the recording will be? It would be nice if every few seconds you could dynamically allocate more space for the table. At present one must allocate loads of space just in case. Know what I mean? Dr Victor Lazzarini Senior Lecturer Dept. of Music NUI Maynooth Ireland tel.: +353 1 708 3545 Victor dot Lazzarini AT nuim dot ie |
Date | 2011-11-27 22:38 |
From | peiman khosravi |
Subject | Re: [Cs-dev] opcode request - changing table size |
How is the inefficiency manifested? Is it only the point at which table size is changed or will if affect the whole performance? Thanks P On 27 November 2011 22:36, Victor Lazzarini |
Date | 2011-11-27 22:44 |
From | Rory Walsh |
Subject | Re: [Cs-dev] opcode request - changing table size |
I take it using vectors instead of fixed size arrays would be a big job? Copying all the data in a table to a new table with a different size will be pretty slow indeed, especially with big tables. On 27 November 2011 22:38, peiman khosravi |
Date | 2011-11-27 22:49 |
From | Victor Lazzarini |
Subject | Re: [Cs-dev] opcode request - changing table size |
The inefficiency is twofold: having to allocate memory in the middle of performance and having to do big copy of contents straight after that. In fact, it's probably possible to do this already (no need for a new opcode), using tablecopy: 1) Allocate a table 2) Before running out allocate another table, bigger and use tablecopy to copy the contents of the first table 3) switch to writing to this new table and back to 2 if you need more space. Victor On 27 Nov 2011, at 22:38, peiman khosravi wrote: > How is the inefficiency manifested? Is it only the point at which > table size is changed or will if affect the whole performance? > > Thanks > > P > > On 27 November 2011 22:36, Victor Lazzarini |
Date | 2011-11-27 22:51 |
From | Victor Lazzarini |
Subject | Re: [Cs-dev] opcode request - changing table size |
Do you mean linked lists? That's not efficient at all. On 27 Nov 2011, at 22:44, Rory Walsh wrote: > I take it using vectors instead of fixed size arrays would be a big > job? Copying all the data in a table to a new table with a different > size will be pretty slow indeed, especially with big tables. > > On 27 November 2011 22:38, peiman khosravi |
Date | 2011-11-27 22:52 |
From | Rory Walsh |
Subject | Re: [Cs-dev] opcode request - changing table size |
I was thinking of stl::vectors? So they're worse?! Ha. Oh well. On 27 November 2011 22:51, Victor Lazzarini |
Date | 2011-11-27 22:52 |
From | Rory Walsh |
Subject | Re: [Cs-dev] opcode request - changing table size |
Or the C equivalent, which I guess is a linked list.. On 27 November 2011 22:52, Rory Walsh |
Date | 2011-11-27 22:53 |
From | peiman khosravi |
Subject | Re: [Cs-dev] opcode request - changing table size |
OK I see. If doing it without a dedicated opcode is just as (non-)efficient then I guess there is no need for an opcode. Thanks Peiman On 27 November 2011 22:51, Victor Lazzarini |
Date | 2011-11-27 22:58 |
From | Victor Lazzarini |
Subject | Re: [Cs-dev] opcode request - changing table size |
That's exactly the same as the case I described. C++ vectors can be resized dynamically, but this will mean allocating and copying the old data to the new location, so it's not different from a simple case of dynamic table allocation. Victor On 27 Nov 2011, at 22:52, Rory Walsh wrote: > I was thinking of stl::vectors? So they're worse?! Ha. Oh well. > > On 27 November 2011 22:51, Victor Lazzarini |
Date | 2011-11-27 22:59 |
From | John Lato |
Subject | Re: [Cs-dev] opcode request - changing table size |
The only real way to do this sort of operation efficiently is to use a rope: a tree (or linked list) of buffers. When the table is lengthened, a new buffer is malloc'ed, then the newly allocated memory is appended to the end of the tree. No copying is necessary, although the malloc will still be costly. John On Sun, Nov 27, 2011 at 10:58 PM, Victor Lazzarini |
Date | 2011-11-27 23:01 |
From | Victor Lazzarini |
Subject | Re: [Cs-dev] opcode request - changing table size |
The only way I see this can be efficient is if a big memory pool is allocated at the start and the size of the table can grow contiguously inside the pool. But I don't then see the difference of just creating a very big table. On 27 Nov 2011, at 22:53, peiman khosravi wrote: > OK I see. If doing it without a dedicated opcode is just as > (non-)efficient then I guess there is no need for an opcode. > > Thanks > > Peiman > > On 27 November 2011 22:51, Victor Lazzarini |
Date | 2011-11-27 23:04 |
From | Victor Lazzarini |
Subject | Re: [Cs-dev] opcode request - changing table size |
yes, I forgot that case. Victor On 27 Nov 2011, at 22:59, John Lato wrote: > The only real way to do this sort of operation efficiently is to use a > rope: a tree (or linked list) of buffers. When the table is > lengthened, a new buffer is malloc'ed, then the newly allocated memory > is appended to the end of the tree. No copying is necessary, although > the malloc will still be costly. > > John > > On Sun, Nov 27, 2011 at 10:58 PM, Victor Lazzarini > |
Date | 2011-11-27 23:21 |
From | Rory Walsh |
Subject | Re: [Cs-dev] opcode request - changing table size |
Attachments | None None |
Isnt that how stl vectors work? On Sunday, 27 November 2011, Victor Lazzarini <Victor.Lazzarini@nuim.ie> wrote: > yes, I forgot that case. > > Victor > On 27 Nov 2011, at 22:59, John Lato wrote: > >> The only real way to do this sort of operation efficiently is to use a >> rope: a tree (or linked list) of buffers. When the table is >> lengthened, a new buffer is malloc'ed, then the newly allocated memory >> is appended to the end of the tree. No copying is necessary, although >> the malloc will still be costly. >> >> John >> >> On Sun, Nov 27, 2011 at 10:58 PM, Victor Lazzarini >> <Victor.Lazzarini@nuim.ie> wrote: >>> That's exactly the same as the case I described. C++ vectors can be resized dynamically, but this will mean allocating and copying the old data to the new location, so it's not different from a simple case of dynamic table allocation. >>> >>> Victor >>> On 27 Nov 2011, at 22:52, Rory Walsh wrote: >>> >>>> I was thinking of stl::vectors? So they're worse?! Ha. Oh well. >>>> >>>> On 27 November 2011 22:51, Victor Lazzarini <Victor.Lazzarini@nuim.ie> wrote: >>>>> Do you mean linked lists? That's not efficient at all. >>>>> >>>>> On 27 Nov 2011, at 22:44, Rory Walsh wrote: >>>>> >>>>>> I take it using vectors instead of fixed size arrays would be a big >>>>>> job? Copying all the data in a table to a new table with a different >>>>>> size will be pretty slow indeed, especially with big tables. >>>>>> >>>>>> On 27 November 2011 22:38, peiman khosravi <peimankhosravi@gmail.com> wrote: >>>>>>> How is the inefficiency manifested? Is it only the point at which >>>>>>> table size is changed or will if affect the whole performance? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> P >>>>>>> >>>>>>> On 27 November 2011 22:36, Victor Lazzarini <Victor.Lazzarini@nuim.ie> wrote: >>>>>>>> Ok, but even if such opcode is provided, it's never going to be as efficient >>>>>>>> for RT use than allocating a lot of space to start with. >>>>>>>> Reallocating involves copying a lot of memory, which is always costly. >>>>>>>> Victor >>>>>>>> On 27 Nov 2011, at 22:06, Rory Walsh wrote: >>>>>>>> >>>>>>>> Let's say you're doing some live writing to a table but you don't know how >>>>>>>> long the recording will be? It would be nice if every few seconds you could >>>>>>>> dynamically allocate more space for the table. At present one must allocate >>>>>>>> loads of space just in case. Know what I mean? >>>>>>>> >>>>>>>> On Sunday, 27 November 2011, Victor Lazzarini <Victor.Lazzarini@nuim.ie> >>>>>>>> wrote: >>>>>>>>> what do you mean? >>>>>>>>> >>>>>>>>> On 27 Nov 2011, at 20:46, peiman khosravi wrote: >>>>>>>>> >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>> Would it be possible to have an opcode that let's one dynamically >>>>>>>>>> change table sizes? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> P >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>>> All the data continuously generated in your IT infrastructure >>>>>>>>>> contains a definitive record of customers, application performance, >>>>>>>>>> security threats, fraudulent activity, and more. Splunk takes this >>>>>>>>>> data and makes sense of it. IT sense. And common sense. >>>>>>>>>> http://p.sf.net/sfu/splunk-novd2d > |
Date | 2011-11-27 23:30 |
From | Victor Lazzarini |
Subject | Re: [Cs-dev] opcode request - changing table size |
Attachments | None None |
No. Vectors allocate new space and then copy the existing data there (unless they are shrinking, but I assume the case in question to be growing) Victor On 27 Nov 2011, at 23:21, Rory Walsh wrote: Isnt that how stl vectors work? Dr Victor Lazzarini Senior Lecturer Dept. of Music NUI Maynooth Ireland tel.: +353 1 708 3545 Victor dot Lazzarini AT nuim dot ie |
Date | 2011-11-28 10:06 |
From | Richard Dobson |
Subject | Re: [Cs-dev] opcode request - changing table size |
There is also the perennial issue of attempting dynamic memory allocation during performance, in the real-time rendering thread. This is universally regarded as a bad idea (as is making just about any system call during real-time performance) if for no other reason than it stalls the rendering thread leading to audio breakup. So if such an implicitly large alloc or realloc is requested during performance, it will need to be done in a background thread, not in the "perf" thread. For mobile devices (which are in any case seriously RAM limited compared to normal machines) it may well not work at all. Richard Dobson On 27/11/2011 22:38, peiman khosravi wrote: > How is the inefficiency manifested? Is it only the point at which > table size is changed or will if affect the whole performance? > > Thanks > > P > ------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-novd2d _______________________________________________ Csound-devel mailing list Csound-devel@lists.sourceforge.net |
Date | 2011-11-28 10:42 |
From | peiman khosravi |
Subject | Re: [Cs-dev] opcode request - changing table size |
I see, that makes sense. I think Victor's suggestion of allocating a maximum size to begin with and appending from it to an existing table still sounds like a cleaner alternative than doing it manually. It may not be any more efficient but it does make life easier somewhat. What do you think Rory? P On 28 November 2011 10:06, Richard Dobson |
Date | 2011-11-28 10:56 |
From | Rory Walsh |
Subject | Re: [Cs-dev] opcode request - changing table size |
This is one for the hardcore programmers, but is John's suggestion a possible solution? To use a rope of buffers? Malloc can be costly, but would it not still be cheaper than copying vast amounts of table data each time one resizes? On 28 November 2011 10:42, peiman khosravi |
Date | 2011-11-28 12:11 |
From | John Lato |
Subject | Re: [Cs-dev] opcode request - changing table size |
As to whether or not it's a solution depends on what you're trying to solve. It would be possible to introduce a new family of opcodes that uses a rope to record an arbitrary length of audio data, and have opcodes to access that data or copy the final amount into a table. I don't think it's reasonable to replace the current table implementation with a more complex data structure though. Everything that touches tables would need to be changed, and there are other tradeoffs too. Reading is less efficient for one. Besides, as Richard points out, malloc'ing shouldn't be done from the audio thread in any case. Maintaining a rope for live writing does seem like a sensible approach, but I'm not certain of the interface. Especially as I think it would only be readable with certain opcodes. You could also stream the data to a file. John On Mon, Nov 28, 2011 at 10:56 AM, Rory Walsh |
Date | 2011-11-28 13:07 |
From | Rory Walsh |
Subject | Re: [Cs-dev] opcode request - changing table size |
I was also thinking it would have to be a new family of opcodes. In which case perhaps it might well be more trouble than it's worth. Streaming to disk is something I've done in the past and it works fine. I think we've reached an impasse. Case closed? On 28 November 2011 12:11, John Lato |
Date | 2011-11-28 13:30 |
From | Andres Cabrera |
Subject | Re: [Cs-dev] opcode request - changing table size |
Hi, It might be a good idea to implement something like what SC does, which is have a preallocated pool of memory and when an opcode requests RTAlloc, give it a part of this pool. The good thing about this is that most opcodes wouldn't have to be changed, since most of them already allocate memory through an API call. Cheers, Andrés On Mon, Nov 28, 2011 at 10:06 AM, Richard Dobson |