| and further...... (too much info??? :-)
>> If you want to make full use of multiple processors using pure python
>> code you'll have to use multiple processes.
>>
>
> This is absolutely true, but in practice it is a little more
> complicated. If you can factor your algorithm so that each process
> can be run completely separately, with no interprocess communication,
> then just using multiple processes will work fine. But once you need
> your processes to communicate in any way using sockets, there is
> another critical requirement to make the multi-process solution work
> in practice. You will absolutely need to use non-blocking sockets and
> the best way to do this in python is using Twisted.
>
> Now a bit of self-promotion....As a part of the IPython project we
> have been implementing an architecture that allows you to do parallel
> and distribted computing interactively in python/Ipython. This
> infrastructure solves many of the basic problems you will run into we
> regularly use it to take advantage of multi-core and multi-CPU
> hardware. For details, see:
>
> http://ipython.scipy.org/moin/Parallel_Computing
>
> Cheers,
>
> Brian
On 03/11/2006, at 10:25 AM, David Worrall wrote:
> from the Pythonmac-SIG guys re whether or not the python threading
> module can make use of multiple processors:
>
>> It can, but not always optimally. Python has a global interpreter
>> lock (GIL) that is held whenever a thread is executing python byte
>> code. This means only one thread at a time can be executing byte
>> code, however multiple threads can be in C code at the same time.
>>
>> Therefore it is possible to use multiple CPUs, but only if you have
>> C code that does some calculation without holding the GIL. Note
>> that several GUI API's from Apple, such as CoreImage make automatic
>> use of multiple processors, and even if you don't use multiple
>> processors yourself the other processor won't be completely idle,
>> that can and will be used for background tasks and other programs.
>>
>> If you want to make full use of multiple processors using pure
>> python code you'll have to use multiple processes.
>
>> Ronald
>
>
> On 30/10/2006, at 8:19 PM, Steven Yi wrote:
>
>> Hi Michael,
>>
>> I can understand to make things as simple as possible. I don't know
>> yet how much this might affect performance but I have the feeling at
>> the moment that it would not be negligible. But also, I don't
>> know if
>> certain things will be possible; there were messages here about
>> threading in Python and that it can't take advantage of multiple
>> processors, which if the threaded handling was done in C, it would be
>> able to do so (I don't know much about Python's threading model so
>> please correct me if I'm wrong). So, this would complicate things,
>> but it also might be the only way to implement this in a fashion that
>> all languages can really take advantage of it.
>>
>> Basically, I'm thinking that Csound should be as performant when used
>> by the API in a non-C/C++ language as it is in C or C++, and I think
>> that one of the ways that happens to slow down now is in the
>> messaging.
>>
>> However, I *do* want to verify this all first with tests before doing
>> anything. If messaging performance is reasonably performant using a
>> threaded solution in a host language (specifically python I'm curious
>> about here) or host program or in the end doesn't gain enough to make
>> a difference, then this whole thread (pun intended! =) is a moot
>> question.
>>
>> steven
>>
>>
>> On 10/30/06, Michael Gogins wrote:
>>> I think it should stay the same as it is now -- hosts should set
>>> up async logging if they want. It's not hard to do.
>>>
>>> One of the things I am constantly worrying about with Csound is
>>> introducing unnecessary complexity. We need to balance introducing
>>> new features with their cost in terms of maintenance and external
>>> dependencies and so on, with their benefits.
>>>
>>> I think Csound is simpler in some ways now than it was, and more,
>>> much more complex than it used to be in other ways. I would like
>>> to simplify certain things. I bet Istvan has simplified some things.
>>>
>>> Regards,
>>> Mike
>>>
>>> -----Original Message-----
>>>> From: Steven Yi
>>>> Sent: Oct 29, 2006 6:08 PM
>>>> To: Developer discussions
>>>> Subject: Re: [Cs-dev] Possible Optimization
>>>>
>>>> Hi Matt,
>>>>
>>>> What you said was basically what I was thinking, so we're
>>>> somewhat on
>>>> the same page. As far as I understand, right now within Csound the
>>>> handling is synchronous, so when an opcode or the engine calls
>>>> csound->Message, by default if no message callback is set by a
>>>> host,
>>>> the default message callback is used and that calls vfprintf to
>>>> output
>>>> to stderr directly.
>>>>
>>>> In MacCsound or Csound5GUI, the callback given to Csound is then
>>>> used
>>>> for Message, but the asynchronous part is then handled in the host.
>>>> The opcodes call Message, the host callback adds the message to its
>>>> buffer, and the lower priority thread in the host is what does
>>>> the end
>>>> printing.
>>>>
>>>> I was suggesting a flag because i thought that if Csound was
>>>> changed
>>>> to do the low priority thread thing inside, it might be less than
>>>> optimal if that's going on and you already have the same thing in
>>>> the
>>>> host.
>>>>
>>>> So, if the flag is off, the asynchronous handling via thread and
>>>> buffer is implemented by a host. If flag is on, it gets a little
>>>> complicated, but there would have to be an extra message callback
>>>> stored somewhere. All of the opcodes and internal code call
>>>> Message,
>>>> but what would have to happen would be that Message would need to
>>>> be a
>>>> function that adds to the buffer, then the low priority thread
>>>> would
>>>> have to grab from the buffer and call the second stored Message
>>>> callback (either the default vfprintf one or a host provided one).
>>>> The low priority thread would also have to be started very early in
>>>> the csound compilation process to get all messages (I guess
>>>> whenever
>>>> setMessageCallback is normally set, at that same step in the
>>>> compilation process).
>>>>
>>>> So, take for instance you're in python. The host sets a callback
>>>> to do
>>>> something like append to a text field. With the flag, csound would
>>>> store messages in a buffer internally and the low priority thread
>>>> would call the host callback. If that callback is slow (which it
>>>> might be with python or other scripting language with all the
>>>> method
>>>> overheads involved), the it'll only slowdown the low priority
>>>> thread,
>>>> not the csound main loop.
>>>>
>>>> If you don't use the flag and you're in a scripting language,
>>>> then you
>>>> can set a callback and do the whole threading thing in the
>>>> scripting
>>>> language, but that callback, even if it's just to add to a linked
>>>> list, isn't going to be as fast as in C, and perhaps neither will
>>>> the
>>>> thread in the scripting language be processing as quickly as in
>>>> C. So
>>>> it tightens up everything to do as much in C as possible I think to
>>>> let the main audio loop go as quickly as possible.
>>>>
>>>> But I was thinking that the situation like csound5gui and
>>>> MacCsound,
>>>> you're already handling it in an optimal way, and just adding it
>>>> into
>>>> csound would only slow it down, which is why I was suggesting a
>>>> flag.
>>>>
>>>> Does that sound alright or is there something I'm missing that this
>>>> wouldn't work out?
>>>> steven
>>>>
>>>>
>>>> On 10/29/06, matt ingalls wrote:
>>>>> i probably don't really understand, but
>>>>> i would assume the asynchronous handling be in the
>>>>> default message routine, which would then no be used
>>>>> if a host app registers a message callback?
>>>>> [ so no need for a commandline flag ]
>>>>>
>>>>> On Oct 29, 2006, at 2:44 AM, Steven Yi wrote:
>>>>>
>>>>>> Hi Matt,
>>>>>>
>>>>>> Thanks for the reply! I'm beginning to think that it's pretty
>>>>>> important to have asynchronous handling for the Message callback,
>>>>>> especially when rendering in realtime. I think I'll experiment
>>>>>> sometime with the commandline to see if implementing a threaded
>>>>>> system
>>>>>> there will help out for blue and also in realtime CSD's. If it
>>>>>> does
>>>>>> indeed improve performance, would it be an issue to you or
>>>>>> ayone else
>>>>>> here if it was put into the core and disabled by default but
>>>>>> enable-able (I've just coined a new word, I know :P ) with a
>>>>>> flag? It
>>>>>> seems like it would be tedious to implement for API users in
>>>>>> non-C/C++
>>>>>> languages.
>>>>>>
>>>>>> BTW: Just as an experiment in Java, I found that the way to
>>>>>> implement
>>>>>> a MessageCallback is to use the CsoundCallbackWrapper as in
>>>>>> below:
>>>>>>
>>>>>> Csound csound = new Csound();
>>>>>> CsoundCallbackWrapper cw = new CsoundCallbackWrapper
>>>>>> (csound) {
>>>>>> public void MessageCallback(int attr, String msg) {
>>>>>> System.out.println(">>" + msg);
>>>>>> }
>>>>>> };
>>>>>> cw.SetMessageCallback();
>>>>>>
>>>>>> csound.Compile("/work/blue/sineTest.csd", "-Wdo", "dac");
>>>>>> csound.Perform();
>>>>>>
>>>>>> Learning something every day! =)
>>>>>>
>>>>>> steven
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 10/29/06, matt ingalls wrote:
>>>>>>>
>>>>>>>
>>>>>>> On Oct 28, 2006, at 4:12 PM, Steven Yi wrote:
>>>>>>>
>>>>>>>
>>>>>>> Do any hosts currently implement setting their own
>>>>>>>
>>>>>>> message callbacks to work in an asynchronous manner?
>>>>>>>
>>>>>>> yes - i have a circular buffer the csoundMessage callback
>>>>>>> writes to,
>>>>>>> and then a lower priority thread handles printing the buffer.
>>>>>>>
>>>>>>>
>>>>>>> ----------------------------------------------------------------
>>>>>>> -
>>>>>>> ----
>>>>>>> ----
>>>>>>> Using Tomcat but need to do more? Need to support web services,
>>>>>>> security?
>>>>>>> Get stuff done quickly with pre-integrated technology to make
>>>>>>> your
>>>>>>> job
>>>>>>> easier
>>>>>>> Download IBM WebSphere Application Server v.1.0.1 based on
>>>>>>> Apache
>>>>>>> Geronimo
>>>>>>> http://sel.as-us.falkag.net/sel?
>>>>>>> cmd=lnk&kid=120709&bid=263057&dat=121642
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Csound-devel mailing list
>>>>>>> Csound-devel@lists.sourceforge.net
>>>>>>> https://lists.sourceforge.net/lists/listinfo/csound-devel
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> -----------------------------------------------------------------
>>>>>> -
>>>>>> ----
>>>>>> ---
>>>>>> Using Tomcat but need to do more? Need to support web services,
>>>>>> security?
>>>>>> Get stuff done quickly with pre-integrated technology to make
>>>>>> your
>>>>>> job easier
>>>>>> Download IBM WebSphere Application Server v.1.0.1 based on Apache
>>>>>> Geronimo
>>>>>> http://sel.as-us.falkag.net/sel?
>>>>>> cmd=lnk&kid=120709&bid=263057&dat=121642
>>>>>> _______________________________________________
>>>>>> Csound-devel mailing list
>>>>>> Csound-devel@lists.sourceforge.net
>>>>>> https://lists.sourceforge.net/lists/listinfo/csound-devel
>>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------
>>>>> -
>>>>> ------
>>>>> Using Tomcat but need to do more? Need to support web services,
>>>>> security?
>>>>> Get stuff done quickly with pre-integrated technology to make
>>>>> your job easier
>>>>> Download IBM WebSphere Application Server v.1.0.1 based on
>>>>> Apache Geronimo
>>>>> http://sel.as-us.falkag.net/sel?
>>>>> cmd=lnk&kid=120709&bid=263057&dat=121642
>>>>> _______________________________________________
>>>>> Csound-devel mailing list
>>>>> Csound-devel@lists.sourceforge.net
>>>>> https://lists.sourceforge.net/lists/listinfo/csound-devel
>>>>>
>>>>
>>>> -------------------------------------------------------------------
>>>> -
>>>> -----
>>>> Using Tomcat but need to do more? Need to support web services,
>>>> security?
>>>> Get stuff done quickly with pre-integrated technology to make
>>>> your job easier
>>>> Download IBM WebSphere Application Server v.1.0.1 based on Apache
>>>> Geronimo
>>>> http://sel.as-us.falkag.net/sel?
>>>> cmd=lnk&kid=120709&bid=263057&dat=121642
>>>> _______________________________________________
>>>> Csound-devel mailing list
>>>> Csound-devel@lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/csound-devel
>>>
>>>
>>>
>>>
>>> --------------------------------------------------------------------
>>> -
>>> ----
>>> Using Tomcat but need to do more? Need to support web services,
>>> security?
>>> Get stuff done quickly with pre-integrated technology to make your
>>> job easier
>>> Download IBM WebSphere Application Server v.1.0.1 based on Apache
>>> Geronimo
>>> http://sel.as-us.falkag.net/sel?
>>> cmd=lnk&kid=120709&bid=263057&dat=121642
>>> _______________________________________________
>>> Csound-devel mailing list
>>> Csound-devel@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/csound-devel
>>>
>>
>> ---------------------------------------------------------------------
>> -
>> ---
>> Using Tomcat but need to do more? Need to support web services,
>> security?
>> Get stuff done quickly with pre-integrated technology to make your
>> job easier
>> Download IBM WebSphere Application Server v.1.0.1 based on Apache
>> Geronimo
>> http://sel.as-us.falkag.net/sel?
>> cmd=lnk&kid=120709&bid=263057&dat=121642
>> _______________________________________________
>> Csound-devel mailing list
>> Csound-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/csound-devel
>>
>
> _______________________________________
> experimental polymedia: www.avatar.com.au
> Sonic Communications Research Group,
> University of Canberra: www.canberra.edu.au
>
>
>
>
>
> ----------------------------------------------------------------------
> ---
> Using Tomcat but need to do more? Need to support web services,
> security?
> Get stuff done quickly with pre-integrated technology to make your
> job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache
> Geronimo
> http://sel.as-us.falkag.net/sel?
> cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Csound-devel mailing list
> Csound-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/csound-devel
>
_______________________________________
experimental polymedia: www.avatar.com.au
Sonic Communications Research Group,
University of Canberra: www.canberra.edu.au
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Csound-devel mailing list
Csound-devel@lists.sourceforge.net |