strange Python issue on OSX...
Date | 2016-09-19 13:58 |
From | Rory Walsh |
Subject | strange Python issue on OSX... |
Oeyvind and I are currently having some issues running python enabled Csound plugins in Live. Whenever we try to use any of the Python opcodes we get this error:
Csound mailing list
Csound@listserv.heanet.ie
https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
Send bugs reports to
https://github.com/csound/csound/issues
Discussions of bugs and features can be posted here
import _locale SystemError: dynamic module not initialized properly We've no idea why it only happens in Live. Reaper and other hosts have no issues. I'm hoping that someone on this list might have come across this error before. |
Date | 2016-09-19 14:13 |
From | Steven Yi |
Subject | Re: strange Python issue on OSX... |
Is the system using a self-compiled version of Csound? I put the error into Google and it looks like it has happened to others who use homebrew and that it might result from clashing versions of libraries. On Mon, Sep 19, 2016 at 8:58 AM, Rory Walsh |
Date | 2016-09-19 14:16 |
From | Victor Lazzarini |
Subject | Re: strange Python issue on OSX... |
That looks like a Python library conflict. Maybe live loads the Apple-supplied Python and the opcodes are linked to the python.org library or vice-versa. ======================== Dr Victor Lazzarini Dean of Arts, Celtic Studies and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 > On 19 Sep 2016, at 13:58, Rory Walsh |
Date | 2016-09-19 15:19 |
From | Michael Gogins |
Subject | Re: strange Python issue on OSX... |
I have had issues with the re module at my old job, our python was different from python.org. I forget what I did but I found the solution by googling. Regards, On Sep 19, 2016 9:16 AM, "Victor Lazzarini" <Victor.Lazzarini@nuim.ie> wrote: That looks like a Python library conflict. Maybe live loads the Apple-supplied Python and the opcodes are |
Date | 2016-09-19 15:59 |
From | Rory Walsh |
Subject | Re: strange Python issue on OSX... |
Thanks guys, you're thinking along the same lines as I am. But attempts to switch versions of python have failed. I'll keep digging. I'm not using a homebrew of Csound, but I've tried a homebrew python. But it doesn't seem to a make a difference. On 19 September 2016 at 15:19, Michael Gogins <michael.gogins@gmail.com> wrote:
|
Date | 2016-09-19 16:10 |
From | Steven Yi |
Subject | Re: strange Python issue on OSX... |
Although not a fantastic solution, one thing that could be done is to build a special version of the python opcode library that does not link to python, but instead assumes the symbols will be supplied dynamically. This is done with the csnd6 python interface library and shows up in the interfaces/CMakeLists.txt file as: if(APPLE) set_target_properties(${SWIG_MODULE_csnd6_REAL_NAME} PROPERTIES LINK_FLAGS "-undefined dynamic_lookup") endif() You might try to remove linking to python and using the above flag in the python opcode's cmake file. The result should, if I understand correctly, build a python opcode library that would work if Csound is embedded within another program and then loads the python opcodes. On the other hand, the python opcodes alone when run from just Csound, would not load in this case. Anyways, as mentioned, not a great option, but wanted to mention it as a possibility. On Mon, Sep 19, 2016 at 10:59 AM, Rory Walsh |
Date | 2016-09-19 16:27 |
From | Rory Walsh |
Subject | Re: strange Python issue on OSX... |
Thanks Steven, that's a good idea. But could we force Csound to use this special build of the opcode over the one supplied with the OSX package? Would using the --lib flag or whatever it is, force Csound to use the custom opcode lib? On 19 September 2016 at 16:10, Steven Yi <stevenyi@gmail.com> wrote: Although not a fantastic solution, one thing that could be done is to |
Date | 2016-09-19 16:34 |
From | Steven Yi |
Subject | Re: strange Python issue on OSX... |
No I don't think so. This path would only really work if you're in control of what opcode libs are packaged with the target system. If you're trying to do this with a general install of Csound, I think this path wouldn't be appropriate. On Mon, Sep 19, 2016 at 11:27 AM, Rory Walsh |
Date | 2016-09-19 17:03 |
From | Rory Walsh |
Subject | Re: strange Python issue on OSX... |
That's a pity. I don't really want to have to start building my own Csound framework on OSX. But I may not have a choice. On 19 September 2016 at 16:34, Steven Yi <stevenyi@gmail.com> wrote: No I don't think so. This path would only really work if you're in |
Date | 2016-09-19 18:40 |
From | Victor Lazzarini |
Subject | Re: strange Python issue on OSX... |
you don't need to build the whole Csound framework, just the opcode library. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland
|
Date | 2016-09-20 11:14 |
From | Rory Walsh |
Subject | Re: strange Python issue on OSX... |
I don't suppose you could provide the command line to build just these opcodes? I just tried somthing simple like: gcc -O2 -dynamiclib -o py.dylib pythonopcodes.c -DUSE_DOUBLE -I/Library/Frameworks/CsoundLib64.framework/Headers -I/usr/include/Python2.7 This leads to a lot of ld: symbol(s) not found for architecture x86_64 After some reading I see some people recommend using g++ to get rid of these errors. Doing so does get rids of these errors but leads to a whole berth of new ones. Trying to build Csound itself using cmake fails the same symbol(s) not found errors. I'd be happy o build the entire Csound package and then modify the cmake commands for the python opcodes. I often find this easier than trying to build individual opcodes. On 19 September 2016 at 18:40, Victor Lazzarini <Victor.Lazzarini@nuim.ie> wrote:
|
Date | 2016-09-20 11:18 |
From | Victor Lazzarini |
Subject | Re: strange Python issue on OSX... |
Try Steven’s suggestion -undefined dynamic_lookup ======================== Dr Victor Lazzarini Dean of Arts, Celtic Studies and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 > On 20 Sep 2016, at 11:14, Rory Walsh |
Date | 2016-09-20 13:19 |
From | Rory Walsh |
Subject | Re: strange Python issue on OSX... |
That's hardly going to sort the compiler problems I reported? I will give it a shot after lunch. On 20 Sep 2016 11:18 am, "Victor Lazzarini" <Victor.Lazzarini@nuim.ie> wrote: Try Steven’s suggestion |
Date | 2016-09-20 13:56 |
From | Victor Lazzarini |
Subject | Re: strange Python issue on OSX... |
It should because the symbols will not be linked directly. It’s a linker error (ld) not a compiler one. ======================== Dr Victor Lazzarini Dean of Arts, Celtic Studies and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 > On 20 Sep 2016, at 13:19, Rory Walsh |
Date | 2016-09-20 14:36 |
From | Rory Walsh |
Subject | Re: strange Python issue on OSX... |
Thanks Vcitor, that did indeed get rid of the linker errors, and everything seems to have compiled correctly. So I dropped the newly created libpy.lib into the Csound framework instead of the one that was there, but Csound doesn't see it. I get init time errors about the use of the any python opcodes. If you are at your PC, perhaps you wouldn't mind running off a quick Csound build with '-undefined dynamic_lookup' set for the Python opcodes? We'd very much appreciate it. The 'beers owed to Victor' count would really shoot up!! On 20 September 2016 at 13:56, Victor Lazzarini <Victor.Lazzarini@nuim.ie> wrote: It should because the symbols will not be linked directly. It’s a linker error (ld) not a compiler one. |
Date | 2016-09-20 15:01 |
From | Victor Lazzarini |
Subject | Re: strange Python issue on OSX... |
It’s not that it doesn’t see it. It can’t load them if the loading program is not linked to Python. Make sure it does: - host -> compiled & linked to libpython - loads the plugin — all the symbols are resolved. Csound is not normally linked to Python, but we presume whatever other host you were using is. You could even try running your code from a python program, that should work. Regards ======================== Dr Victor Lazzarini Dean of Arts, Celtic Studies and Philosophy, Maynooth University, Maynooth, Co Kildare, Ireland Tel: 00 353 7086936 Fax: 00 353 1 7086952 > On 20 Sep 2016, at 14:36, Rory Walsh |
Date | 2016-09-20 15:17 |
From | Rory Walsh |
Subject | Re: strange Python issue on OSX... |
Thanks Victor. It's starting to make a little more sense now. So with the newly created dylib I am able to launch a Csound/Python plugin in Live that no longer causes a crash. I'm getting some Python errors, but no longer any crashes. Thanks again. The question now is how best to provide this opcode lib for Cabbage users who want to use Python in their Csound VST plugins... On 20 September 2016 at 15:01, Victor Lazzarini <Victor.Lazzarini@nuim.ie> wrote: It’s not that it doesn’t see it. It can’t load them if the loading program is not linked to Python. Make sure it does: |
Date | 2016-09-20 16:18 |
From | Steven Yi |
Subject | Re: strange Python issue on OSX... |
Are you packaging Csound with Cabbage? Also to note, the dynamic lookup is an OSX option. I don't think there is a comparable linking option available on Windows, but I am unsure. On Tue, Sep 20, 2016 at 10:17 AM, Rory Walsh |
Date | 2016-09-20 17:26 |
From | Oeyvind Brandtsegg |
Subject | Re: strange Python issue on OSX... |
Just to chime in, yes, the problem only occurs on OSX, and only under Ableton Live (afaics). Other hosts (Reaper, Bitwig, Logic) on OSX does not trigger the same error. On Windows it does not trigger an error with any hosts. Even though it is a pesky situation occuring in (some would say) and obscure situation, a general solution would be preferred over a customisation of the libraries shipped with Cabbage. For many musicians, running Ableton Live on OSX is not so obscure, rather the contrary. I don't do so (use either Live or OSX) myself, but would be very interested in having my instruments available for those who do. best Oeyvind 2016-09-20 8:18 GMT-07:00 Steven Yi |
Date | 2016-09-20 17:49 |
From | Steven Yi |
Subject | Re: strange Python issue on OSX... |
One thing to understand is that this is not a simple problem. The problem of conflicting dynamically-linked libraries just isn't something one can completely solve as it is an open-ended situation. The reason one gets this here, I assume, is that something being loaded in Live is using Python and it is a different version of Python than what is linked to by Csound. If two different parts load different versions of the same library, you get conflicts. (Same thing happens if you try to mix executables and plugins using QT4 and QT5, for example.) The path being explored here adjusts to the situation by asking for dynamic symbol lookup, which is making the assumption that by the time the python opcode library is loaded, something else would have loaded a version of python already and we can then reuse those symbols. That means when we use this version of the library, it fails if trying to run Csound outside of something that has loaded python (i.e., the csound commandline executable). One possibility, is to have two versions of the python opcode library, and modify Csound's opcode library loading system. Say for example, we had: libpyopcodes.dylib (not linked to python) libpyoopcodes_2.dylib (linked to python) and had a rule where Csound tries to load libraries sorted by name, but where there are multiple versions, skip rest on sucees, the above might work. In the situation where Csound is run from the commandline, trying to load the first would fail, but the latter might pass. In the situation where Csound is loaded from another app, such as the python executable or in Live on OSX, the former would pass and the latter would be skipped. This is, in the end, a bit ugly, but perhaps no less so than the existing scenario. On Tue, Sep 20, 2016 at 12:26 PM, Oeyvind Brandtsegg |
Date | 2016-09-20 17:56 |
From | Rory Walsh |
Subject | Re: strange Python issue on OSX... |
Sounds like a fairly decent solution to me Steven but is it doable considering that Live will allow Csound to load the first library, but then crashes. It's not that it fails to load the plugin lib. It just loads it, tries to run the instrument and then craps out. A far less beautiful solution would be a menu command in Cabbage that lets you switch between versions of libpy.dylib. On 20 September 2016 at 17:49, Steven Yi <stevenyi@gmail.com> wrote: One thing to understand is that this is not a simple problem. The |
Date | 2016-09-20 17:59 |
From | Oeyvind Brandtsegg |
Subject | Re: strange Python issue on OSX... |
I do understand it is not a simple problem. Thanks so much for the effort to try to solve it. I know it is truly gnarly. The solution you propose seems like it could work. Perhaps if there was a better way of choosing which library to load than sorting them by name? Is there a way of asking if Python has been loaded (perhaps that is actually what you already do when trying to load the lib and it fails or succeeds) 2016-09-20 9:49 GMT-07:00 Steven Yi |
Date | 2016-09-20 18:03 |
From | Oeyvind Brandtsegg |
Subject | Re: strange Python issue on OSX... |
I am just poking in the dark here, so apologies if my suggestions does not make sense, but would it be possible to have some kind of verification method to run when the lib is loaded? Something simple that would just return a value if Python can run, and another if it doesn't (or crashed while trying). The we could load the lib, run the verification, and then if it did load but crashes (on the verification method) we could load the other one. 2016-09-20 9:56 GMT-07:00 Rory Walsh |
Date | 2016-09-20 18:46 |
From | Victor Lazzarini |
Subject | Re: strange Python issue on OSX... |
Did you try the library you built? It is important for us to know it works before attempting anything else. We might be barking at the wrong tree. Check it first. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland
|
Date | 2016-09-20 18:52 |
From | Steven Yi |
Subject | Re: strange Python issue on OSX... |
I think Rory's point makes this much trickier. If the library loads alright, then crashes when evaluating python code, then I'm not sure what to do. (There may be other paths to investigate too involving weak linking of symbols and dynamic loading.) On Tue, Sep 20, 2016 at 1:03 PM, Oeyvind Brandtsegg |
Date | 2016-09-20 20:02 |
From | Victor Lazzarini |
Subject | Re: strange Python issue on OSX... |
but let's first see if a library with runtime linkage gets loaded in Live. If it doesn't that's a different issue. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland > On 20 Sep 2016, at 18:52, Steven Yi |
Date | 2016-09-20 21:00 |
From | Rory Walsh |
Subject | Re: strange Python issue on OSX... |
Tell me what to do to figure this out ;) As it stands, my own python plugin opcodes work fine with Steven's linker flag. There was an issue with a path which was causing a python exception, but after fixing that everything works fine. On 20 September 2016 at 20:02, Victor Lazzarini <Victor.Lazzarini@nuim.ie> wrote: but let's first see if a library with runtime linkage gets loaded in Live. If it doesn't that's a different issue. |
Date | 2016-09-21 06:29 |
From | Victor Lazzarini |
Subject | Re: strange Python issue on OSX... |
then it is exactly as we thought, Live must be linked to a different Python lib. Victor Lazzarini Dean of Arts, Celtic Studies, and Philosophy Maynooth University Ireland
|