[Csnd-dev] python issue..
Date | 2017-02-11 05:25 |
From | Rory Walsh |
Subject | [Csnd-dev] python issue.. |
Hi guys. Oeyvind and I are trying to debug with running lots of Cabbage plugins that call Python. At some point it will crash the host. I managed to grab a backtrace. I hope it's an easy fix. BTW, we've been testing on Windows only. We've no idea if the issue is across all platforms. We might be able to test OSX later today. If we do we'll get back to you.
#0 0x0000000058e37100 in python27!PyNode_Compile () from C:\WINDOWS\SYSTEM32\python27.dll #1 0x0000000058e37afb in python27!PyNode_Compile () from C:\WINDOWS\SYSTEM32\python27.dll #2 0x0000000058e37e79 in python27!PyNode_Compile () from C:\WINDOWS\SYSTEM32\python27.dll #3 0x0000000058e30ef7 in python27!PyNode_Compile () from C:\WINDOWS\SYSTEM32\python27.dll #4 0x0000000058e2f9a3 in python27!PyAST_Compile () from C:\WINDOWS\SYSTEM32\python27.dll #5 0x0000000058e6671a in python27!PyRun_FileExFlags () from C:\WINDOWS\SYSTEM32\python27.dll #6 0x0000000058e665af in python27!PyRun_StringFlags () from C:\WINDOWS\SYSTEM32\python27.dll #7 0x000000006c4c17eb in eval_string_in_given_context (string=0x5d4e4a0 "add_one(48.000000)", private=<optimized out>) at C:/Users/rory/sourcecode/csound64/csound/Opcodes/py/pythonopcodes.c:116 #8 0x000000006c4c54fe in pycall1_krate (csound=<optimized out>, p=0x96f6c58) at C:/Users/rory/sourcecode/csound64/csound/Opcodes/py/pycall.auto.c:167 #9 0x0000000006793598 in kperf_nodebug (csound=0x95d9820) at C:/Users/rory/sourcecode/csound64/csound/Top/csound.c:1591 #10 0x00000000067947ca in csoundPerformKsmps (csound=0x95d9820) at C:/Users/rory/sourcecode/csound64/csound/Top/csound.c:1985 #11 0x000000006caf85dd in Csound::PerformKsmps() () from C:\Users\rory\Documents\vsts\analyzer1_py_repeat.dll |
Date | 2017-02-11 06:48 |
From | Victor Lazzarini |
Subject | Re: [Csnd-dev] python issue.. |
It does not look easy. Are they sharing memory or the same interpreter? It could be a reentrancy problem in the python lib. We need to know about linux and OSX. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland > On 11 Feb 2017, at 05:26, Rory Walsh |
Date | 2017-02-11 07:08 |
From | Rory Walsh |
Subject | Re: [Csnd-dev] python issue.. |
Just tested on Linux with Bitwig studio and I get the same result, although it dies with much more decorum than Reaper on Windows. I've no idea how the Python opcodes work, but I imagine it has to be some kind of race condition that's causing the problem. Oeyvind wrote the .csd we are using to purposely try to bring down hosts. It makes randomly timed calls to pycall1. We think at some point some several instances make a call at the same time. I tried to prepare a command line version but I can't seem to recreate the issue. I'm happy to try again with some direction. On 11 February 2017 at 06:48, Victor Lazzarini <Victor.Lazzarini@nuim.ie> wrote: It does not look easy. Are they sharing memory or the same interpreter? It could |
Date | 2017-02-11 07:18 |
From | Victor Lazzarini |
Subject | Re: [Csnd-dev] python issue.. |
It must be to do with reentrancy, maybe the Python library is not designed to have all of these things happening at the same time. The backtrace seems to point to deep inside the Python lib. Can you give more details? Is there any memory sharing? Are you running several Csound instances? We need to look again at the Python opcodes code to see if there are no global statics. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 > On 11 Feb 2017, at 07:08, Rory Walsh |
Date | 2017-02-11 07:31 |
From | Rory Walsh |
Subject | Re: [Csnd-dev] python issue.. |
The crash occurs when we load several Python based Cabbage VSTs in a host. Each VST will create and run an instance of Csound. This is the instrument we are running to test: print sr pyinit pyruni "def add_one(x): return x+1.0" instr 1 seed 0 irnd rnd31 1, 1 print irnd ktrig metro 10+irnd k1 init 0 if ktrig == 1 then k1 pycall1 "add_one", k1 endif k2 = int(k1/10) printk2 k2+irnd endin It's not the started and stopping of the plugins that causes the crash. They all start with the host and eventually crash it. We appreciate it's not an easy one. If you need us to try anything let us know. On 11 February 2017 at 07:18, Victor Lazzarini <Victor.Lazzarini@nuim.ie> wrote: It must be to do with reentrancy, maybe the Python library is not designed to have all of these things |
Date | 2017-02-11 07:43 |
From | Victor Lazzarini |
Subject | Re: [Csnd-dev] python issue.. |
Each Csound will start a new interpreter, so I wonder if it is reentrancy on the Python side that causes it. I see nothing suspicious in the python opcodes sources.
Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy
Maynooth University
Ireland
|
Date | 2017-02-11 07:45 |
From | Oeyvind Brandtsegg |
Subject | Re: [Csnd-dev] python issue.. |
I tried to test if the are sharing the same interpreter, and it seems they are. At least they are sharing the same namespace. |
Date | 2017-02-11 07:46 |
From | Victor Lazzarini |
Subject | Re: [Csnd-dev] python issue.. |
Provided that of course you are not running into memory problems, which I would doubt.
Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy
Maynooth University
Ireland
|
Date | 2017-02-11 07:47 |
From | Victor Lazzarini |
Subject | Re: [Csnd-dev] python issue.. |
How many plugins at the same time?
Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy
Maynooth University
Ireland
|
Date | 2017-02-11 07:52 |
From | Rory Walsh |
Subject | Re: [Csnd-dev] python issue.. |
In my test I seem to have to at least 3 to get the host to crash. On 11 February 2017 at 07:47, Victor Lazzarini <Victor.Lazzarini@nuim.ie> wrote:
|
Date | 2017-02-11 08:00 |
From | Victor Lazzarini |
Subject | Re: [Csnd-dev] python issue.. |
Pyinit should start different interpreters so I don't know why. Need to look. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland > On 11 Feb 2017, at 07:56, Oeyvind Brandtsegg |
Date | 2017-02-11 08:01 |
From | Oeyvind Brandtsegg |
Subject | Re: [Csnd-dev] python issue.. |
For me, it will crash easily with two instances of the plugin. If I load them up one by one, it will take some seconds before it crashes, but if I save a song with two instances and reopen it it will crash immediately. I suspect this is because the two instances then access pycall simultaneously. 2017-02-10 23:52 GMT-08:00 Rory Walsh |
Date | 2017-02-11 08:04 |
From | Oeyvind Brandtsegg |
Subject | Re: [Csnd-dev] python issue.. |
If I run my example on the CLI, I get two separate Python namespaces. When running as Cabbage plugins, I get one namespace (instance 2 can call a function defined in instance 1) 2017-02-11 0:00 GMT-08:00 Victor Lazzarini |
Date | 2017-02-11 08:05 |
From | Victor Lazzarini |
Subject | Re: [Csnd-dev] python issue.. |
I did not read your email completely. Yes, I would expect that. But each Csound instance will have its own interpreter. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland > On 11 Feb 2017, at 07:56, Oeyvind Brandtsegg |
Date | 2017-02-11 08:06 |
From | Oeyvind Brandtsegg |
Subject | Re: [Csnd-dev] python issue.. |
... so no problems on CLI. Here's my namespace testing csd, for convenience. |
Date | 2017-02-11 08:09 |
From | Oeyvind Brandtsegg |
Subject | Re: [Csnd-dev] python issue.. |
It is a little unclear to me what you mean by 'I would expect that'. Did you mean you would expect the two instances to share the same namespace? (I guess not) 2017-02-11 0:05 GMT-08:00 Victor Lazzarini |
Date | 2017-02-11 09:26 |
From | Victor Lazzarini |
Subject | Re: [Csnd-dev] python issue.. |
But here there is only one Csound instance, or are there more. Not sure how you could run two Csound instances with the CLI csound command. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland > On 11 Feb 2017, at 08:16, Oeyvind Brandtsegg |
Date | 2017-02-11 09:29 |
From | Victor Lazzarini |
Subject | Re: [Csnd-dev] python issue.. |
Forget it, I misinterpreted what you said. If it is the case that two separate Csound instances are using the same interpreter, it is very possible that they could clash. The question is how come this is happening. Pyinit should use a clean interpreter, should it not? Need to read the code again. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland > On 11 Feb 2017, at 08:19, Oeyvind Brandtsegg |
Date | 2017-02-11 09:42 |
From | Victor Lazzarini |
Subject | Re: [Csnd-dev] python issue.. |
It is probably the case that there is one interpreter per process, which will make the Python opcodes non-reentrant. So this means that unless we rewrite the code to use a different interpreter per instance, you will not be able to run two or more Csound objects using Python opcodes in the same process. I am looking at Py_Initialize(). The docs say it's a non-op the second time it is called, so that seems to be what is happening. Pyinit attempts to initialize and given that there is an interpreter running, it just uses it. We need a different approach. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland > On 11 Feb 2017, at 09:30, Victor Lazzarini |
Date | 2017-02-11 10:03 |
From | Victor Lazzarini |
Subject | Re: [Csnd-dev] python issue.. |
The other thing is that maybe it's not really a case of not being reentrant, but of thread safety. So if you put locks around perfKsmps() you should be able to fix it. Not ideal, but it should do the job. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland > On 11 Feb 2017, at 09:42, Victor Lazzarini |
Date | 2017-02-11 10:26 |
From | Rory Walsh |
Subject | Re: [Csnd-dev] python issue.. |
Do you mean use a csoundMutexLock? I tend to try to avoid locks in my audio thread but I'll try it out see if it works. On 11 February 2017 at 10:03, Victor Lazzarini <Victor.Lazzarini@nuim.ie> wrote: The other thing is that maybe it's not really |
Date | 2017-02-11 11:02 |
From | Victor Lazzarini |
Subject | Re: [Csnd-dev] python issue.. |
Or a spinlock. You don’t need to use csound locks, anything would do. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 > On 11 Feb 2017, at 10:26, Rory Walsh |
Date | 2017-02-11 11:50 |
From | Rory Walsh |
Subject | Re: [Csnd-dev] python issue.. |
I tried protecting calls to PerformKsmps with csoundSpinLock(), but I still get the same crash as before. Am I using those function correctly: static int lock = 0; I just followed the code from the API docs. On 11 February 2017 at 11:02, Victor Lazzarini <Victor.Lazzarini@nuim.ie> wrote: Or a spinlock. You don’t need to use csound locks, anything would do. |
Date | 2017-02-11 12:17 |
From | Victor Lazzarini |
Subject | Re: [Csnd-dev] python issue.. |
yes, it looks OK. I don’t really know. The opcodes did not seem to have been designed for this type of use. ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 > On 11 Feb 2017, at 11:50, Rory Walsh |
Date | 2017-02-11 12:32 |
From | Rory Walsh |
Subject | Re: [Csnd-dev] python issue.. |
When has that ever stopped Oeyvind ;) On 11 Feb 2017 12:17 pm, "Victor Lazzarini" <Victor.Lazzarini@nuim.ie> wrote: yes, it looks OK. I don’t really know. The opcodes did not seem to have been designed for this type of use. |
Date | 2017-02-11 13:14 |
From | Michael Gogins |
Subject | Re: [Csnd-dev] python issue.. |
This sounds plausible. Regards, Mike On Feb 11, 2017 4:42 AM, "Victor Lazzarini" <Victor.Lazzarini@nuim.ie> wrote: It is probably the case that there is one interpreter per process, which will make the |
Date | 2017-02-11 17:25 |
From | Oeyvind Brandtsegg |
Subject | Re: [Csnd-dev] python issue.. |
It has worked fine up until recently. So it is something that has changed. I understand it would be different if this was a new use case, but it concerns Cabbage plugins that I've been using for several years and now they don't work. The change assumedly happened late last year, but I haven't been able to track down the exact time. 2017-02-11 4:32 GMT-08:00 Rory Walsh |
Date | 2017-02-11 17:42 |
From | Victor Lazzarini |
Subject | Re: [Csnd-dev] python issue.. |
Maybe the change was in Cabbage? We have not touched the Python opcodes for a long time. What I can say by looking at the code is that there is only one interpreter and if there are two threads accessing it, it may or may not be a problem. best regards ======================== Prof. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 > On 11 Feb 2017, at 17:25, Oeyvind Brandtsegg |