Csound Csound-dev Csound-tekno Search About

Re: [Cs-dev] Multi-threading spinlocks

Date2008-08-11 09:18
Fromvictor
SubjectRe: [Cs-dev] Multi-threading spinlocks
The problem was that the #include ; line was
causing the next  statement to be hidden (and causing
a compiler warning), so the test passed when it should
have failed.

Victor
----- Original Message ----- 
From: "Michael Gogins" 
To: "Michael Gogins" ; "Developer discussions" 

Sent: Monday, August 11, 2008 12:09 AM
Subject: Re: [Cs-dev] Multi-threading spinlocks


> Your test does not work for me. It must have to do with int32_t. Change 
> your test to check for (int *) 0, 0 instead of 0,0 in the sync call. But 
> this will be no good if we build on 64 bit systems.
>
> Hope this helps,
> Mike
>
> -----Original Message-----
>>From: Michael Gogins 
>>Sent: Aug 10, 2008 7:05 PM
>>To: Developer discussions 
>>Subject: Re: [Cs-dev] Multi-threading spinlocks
>>
>>I am quite sure this is not because of anything I did. I only added two 
>>lines to SConstruct, they are identical in structure to existing lines, 
>>and they worked for me. Either you need to do the whole cleanup.sh thing, 
>>or your custom.py changed or something.
>>
>>Hope this helps,
>>Mike
>>
>>-----Original Message-----
>>>From: victor 
>>>Sent: Aug 10, 2008 4:01 PM
>>>To: Developer discussions 
>>>Subject: Re: [Cs-dev] Multi-threading spinlocks
>>>
>>>Now configure does not pass the stdio.h test.
>>>
>>>Checking for C header file stdio.h... no
>>> *** Failed to compile a simple test program. The compiler is
>>> *** possibly not set up correctly, or is used with invalid flags.
>>> *** Check config.log to find out more about the error.
>>>
>>>
>>>config.log
>>>file C:\msys\1.0\home\Victor\csound5\SConstruct,line 621:
>>> Configure(confdir = .sconf_temp)
>>>scons: Configure: Checking for C header file stdio.h...
>>>scons: Configure: ".sconf_temp\conftest_0.c" is up to date.
>>>scons: Configure: The original builder output was:
>>>  |.sconf_temp\conftest_0.c <-
>>>  |  |
>>>  |  |#include 
>>>  |  |
>>>  |  |
>>>  |
>>>gcc -o
>>>.sconf_temp\conftest_0.o -c -Wall -DNOGETTEXT -g -fomit-frame-pointer -D_WIN32
>>> -DWIN32 -DPIPES -DOS_IS_WIN32 -mthreads -DSDFT -DBETA -I. -IH -IC:\msys\1.0\local\include
>>> -IC:\msys\1.0\include -IC:\Python25\include "-IC:\Program
>>>Files\java\jdk1.6.0_06\include" "-IC:\Program
>>>Files\java\jdk1.6.0_06\include\win32" -IC:\msys\1.0\local\java\include -IC:\msys\1.0\local\java\include\win32
>>> -IC:\usr\local\include -IC:\usr\include .sconf_temp\conftest_0.c
>>>gcc: .sconf_temp\conftest_0.c: No such file or directory
>>>gcc: no input files
>>>scons: Configure: no
>>>
>>>
>>>So??
>>>
>>>Victor
>>>---- Original Message ----- 
>>>From: "Michael Gogins" 
>>>To: "victor" 
>>>Sent: Sunday, August 10, 2008 8:27 PM
>>>Subject: Re: [Cs-dev] Multi-threading spinlocks
>>>
>>>
>>>> That's not a complete cleanup. You need to run cleanup.sh.
>>>>
>>>> You might be able to get by by deleting the configuration stuff
>>>> (.sconf_temp dir and config.log).
>>>>
>>>> If this works, it does the #ifdef that you want.
>>>>
>>>> Regards,
>>>> Mike
>>>>
>>>> -----Original Message-----
>>>>>From: victor 
>>>>>Sent: Aug 10, 2008 3:02 PM
>>>>>To: Michael Gogins 
>>>>>Subject: Re: [Cs-dev] Multi-threading spinlocks
>>>>>
>>>>>scons -c & scons
>>>>>
>>>>>same thing. Can this code be protected by ifdefs? It would be nice
>>>>>to be able to build csound regardless of  compiler version.
>>>>>
>>>>>Victor
>>>>>----- Original Message ----- 
>>>>>From: "Michael Gogins" 
>>>>>To: "victor" 
>>>>>Sent: Sunday, August 10, 2008 7:58 PM
>>>>>Subject: Re: [Cs-dev] Multi-threading spinlocks
>>>>>
>>>>>
>>>>>> This should have failed the configure test. Try a complete blow away 
>>>>>> and
>>>>>> retest?...
>>>>>>
>>>>>> What version of GCC are you using?
>>>>>>
>>>>>> Regards,
>>>>>> Mike
>>>>>>
>>>>>> -----Original Message-----
>>>>>>>From: victor 
>>>>>>>Sent: Aug 10, 2008 2:38 PM
>>>>>>>To: Michael Gogins , Developer discussions
>>>>>>>
>>>>>>>Subject: Re: [Cs-dev] Multi-threading spinlocks
>>>>>>>
>>>>>>>Fails to build here, on windows/mingw
>>>>>>>
>>>>>>>OOps\aops.o(.text+0x3531): In function `in':
>>>>>>>C:/msys/1.0/home/Victor/csound5/OOps/aops.c:952: undefined reference 
>>>>>>>to
>>>>>>>`__sync_
>>>>>>>lock_test_and_set'
>>>>>>>OOps\aops.o(.text+0x3572):C:/msys/1.0/home/Victor/csound5/OOps/aops.c:954:
>>>>>>>undef
>>>>>>>ined reference to `__sync_lock_release'
>>>>>>>OOps\aops.o(.text+0x35a1): In function `ins':
>>>>>>>C:/msys/1.0/home/Victor/csound5/OOps/aops.c:963: undefined reference 
>>>>>>>to
>>>>>>>`__sync_
>>>>>>>lock_test_and_set'
>>>>>>>OOps\aops.o(.text+0x3647):C:/msys/1.0/home/Victor/csound5/OOps/aops.c:971:
>>>>>>>undef
>>>>>>>ined reference to `__sync_lock_release'
>>>>>>>OOps\aops.o(.text+0x36b1): In function `inq':
>>>>>>>C:/msys/1.0/home/Victor/csound5/OOps/aops.c:986: undefined reference 
>>>>>>>to
>>>>>>>`__sync_
>>>>>>>lock_test_and_set'
>>>>>>>OOps\aops.o(.text+0x377e):C:/msys/1.0/home/Victor/csound5/OOps/aops.c:993:
>>>>>>>undef
>>>>>>>ined reference to `__sync_lock_release'
>>>>>>>OOps\aops.o(.text+0x397c): In function `ino':
>>>>>>>C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1035: undefined reference 
>>>>>>>to
>>>>>>>`__sync
>>>>>>>_lock_test_and_set'
>>>>>>>OOps\aops.o(.text+0x3ad1):C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1046:
>>>>>>>unde
>>>>>>>fined reference to `__sync_lock_release'
>>>>>>>OOps\aops.o(.text+0x3b19): In function `inn':
>>>>>>>C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1058: undefined reference 
>>>>>>>to
>>>>>>>`__sync
>>>>>>>_lock_test_and_set'
>>>>>>>OOps\aops.o(.text+0x3b83):C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1063:
>>>>>>>unde
>>>>>>>fined reference to `__sync_lock_release'
>>>>>>>OOps\aops.o(.text+0x3c49): In function `inch_opcode':
>>>>>>>C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1085: undefined reference 
>>>>>>>to
>>>>>>>`__sync
>>>>>>>_lock_test_and_set'
>>>>>>>OOps\aops.o(.text+0x3cbf):C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1092:
>>>>>>>unde
>>>>>>>fined reference to `__sync_lock_release'
>>>>>>>OOps\aops.o(.text+0x3d23): In function `inall_opcode':
>>>>>>>C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1102: undefined reference 
>>>>>>>to
>>>>>>>`__sync
>>>>>>>_lock_test_and_set'
>>>>>>>OOps\aops.o(.text+0x3e0a):C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1112:
>>>>>>>unde
>>>>>>>fined reference to `__sync_lock_release'
>>>>>>>OOps\aops.o(.text+0x3e47): In function `out':
>>>>>>>C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1121: undefined reference 
>>>>>>>to
>>>>>>>`__sync
>>>>>>>_lock_test_and_set'
>>>>>>>OOps\aops.o(.text+0x3ef8):C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1130:
>>>>>>>unde
>>>>>>>fined reference to `__sync_lock_release'
>>>>>>>OOps\aops.o(.text+0x3f50): In function `outs':
>>>>>>>C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1141: undefined reference 
>>>>>>>to
>>>>>>>`__sync
>>>>>>>_lock_test_and_set'
>>>>>>>OOps\aops.o(.text+0x409e):C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1157:
>>>>>>>unde
>>>>>>>fined reference to `__sync_lock_release'
>>>>>>>OOps\aops.o(.text+0x40d2): In function `outq':
>>>>>>>C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1166: undefined reference 
>>>>>>>to
>>>>>>>`__sync
>>>>>>>_lock_test_and_set'
>>>>>>>OOps\aops.o(.text+0x42f9):C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1191:
>>>>>>>unde
>>>>>>>fined reference to `__sync_lock_release'
>>>>>>>OOps\aops.o(.text+0x4346): In function `outs1':
>>>>>>>C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1202: undefined reference 
>>>>>>>to
>>>>>>>`__sync
>>>>>>>_lock_test_and_set'
>>>>>>>OOps\aops.o(.text+0x443e):C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1217:
>>>>>>>unde
>>>>>>>fined reference to `__sync_lock_release'
>>>>>>>OOps\aops.o(.text+0x448b): In function `outs2':
>>>>>>>C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1227: undefined reference 
>>>>>>>to
>>>>>>>`__sync
>>>>>>>_lock_test_and_set'
>>>>>>>OOps\aops.o(.text+0x4583):C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1242:
>>>>>>>unde
>>>>>>>fined reference to `__sync_lock_release'
>>>>>>>OOps\aops.o(.text+0x45d0): In function `outs12':
>>>>>>>C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1252: undefined reference 
>>>>>>>to
>>>>>>>`__sync
>>>>>>>_lock_test_and_set'
>>>>>>>OOps\aops.o(.text+0x4701):C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1268:
>>>>>>>unde
>>>>>>>fined reference to `__sync_lock_release'
>>>>>>>OOps\aops.o(.text+0x474e): In function `outq1':
>>>>>>>C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1279: undefined reference 
>>>>>>>to
>>>>>>>`__sync
>>>>>>>_lock_test_and_set'
>>>>>>>OOps\aops.o(.text+0x486d):C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1296:
>>>>>>>unde
>>>>>>>fined reference to `__sync_lock_release'
>>>>>>>OOps\aops.o(.text+0x48ba): In function `outq2':
>>>>>>>C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1306: undefined reference 
>>>>>>>to
>>>>>>>`__sync
>>>>>>>_lock_test_and_set'
>>>>>>>OOps\aops.o(.text+0x49d9):C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1323:
>>>>>>>unde
>>>>>>>fined reference to `__sync_lock_release'
>>>>>>>OOps\aops.o(.text+0x4a26): In function `outq3':
>>>>>>>C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1333: undefined reference 
>>>>>>>to
>>>>>>>`__sync
>>>>>>>_lock_test_and_set'
>>>>>>>OOps\aops.o(.text+0x4b45):C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1350:
>>>>>>>unde
>>>>>>>fined reference to `__sync_lock_release'
>>>>>>>OOps\aops.o(.text+0x4b92): In function `outq4':
>>>>>>>C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1360: undefined reference 
>>>>>>>to
>>>>>>>`__sync
>>>>>>>_lock_test_and_set'
>>>>>>>OOps\aops.o(.text+0x4cb1):C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1377:
>>>>>>>unde
>>>>>>>fined reference to `__sync_lock_release'
>>>>>>>OOps\aops.o(.text+0x4d35): In function `outh':
>>>>>>>C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1392: undefined reference 
>>>>>>>to
>>>>>>>`__sync
>>>>>>>_lock_test_and_set'
>>>>>>>OOps\aops.o(.text+0x4fca):C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1416:
>>>>>>>unde
>>>>>>>fined reference to `__sync_lock_release'
>>>>>>>OOps\aops.o(.text+0x5064): In function `outo':
>>>>>>>C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1433: undefined reference 
>>>>>>>to
>>>>>>>`__sync
>>>>>>>_lock_test_and_set'
>>>>>>>OOps\aops.o(.text+0x53a1):C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1461:
>>>>>>>unde
>>>>>>>fined reference to `__sync_lock_release'
>>>>>>>OOps\aops.o(.text+0x53e5): In function `outn':
>>>>>>>C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1470: undefined reference 
>>>>>>>to
>>>>>>>`__sync
>>>>>>>_lock_test_and_set'
>>>>>>>OOps\aops.o(.text+0x557f):C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1491:
>>>>>>>unde
>>>>>>>fined reference to `__sync_lock_release'
>>>>>>>OOps\aops.o(.text+0x5673): In function `outch':
>>>>>>>C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1522: undefined reference 
>>>>>>>to
>>>>>>>`__sync
>>>>>>>_lock_test_and_set'
>>>>>>>OOps\aops.o(.text+0x580a):C:/msys/1.0/home/Victor/csound5/OOps/aops.c:1545:
>>>>>>>unde
>>>>>>>fined reference to `__sync_lock_release'
>>>>>>>collect2: ld returned 1 exit status
>>>>>>>scons: *** [csound32.dll.5.1] Error 1
>>>>>>>scons: building terminated because of errors.
>>>>>>>
>>>>>>>----- Original Message ----- 
>>>>>>>From: "Michael Gogins" 
>>>>>>>To: "Michael Gogins" ; "Developer discussions"
>>>>>>>
>>>>>>>Sent: Sunday, August 10, 2008 7:31 PM
>>>>>>>Subject: Re: [Cs-dev] Multi-threading spinlocks
>>>>>>>
>>>>>>>
>>>>>>>>I should have added...
>>>>>>>>
>>>>>>>> There may be some bug fixes required for odd numbers of threads. 
>>>>>>>> But,
>>>>>>>> by
>>>>>>>> the next release, multi-threading should be usable with many Csound
>>>>>>>> orchestras, and will show a real performance gain even on machines
>>>>>>>> with
>>>>>>>> only 2 cores.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Mike
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>>>From: Michael Gogins 
>>>>>>>>>Sent: Aug 10, 2008 2:03 PM
>>>>>>>>>To: Csound Developers 
>>>>>>>>>Subject: [Cs-dev] Multi-threading spinlocks
>>>>>>>>>
>>>>>>>>>I have committed to Csound CVS implementations of spinlocks in
>>>>>>>>>csound.h,
>>>>>>>>>based on compiler intrinsics, for MSVC and GCC. The spinlock macros
>>>>>>>>>compile to no-ops if the compiler does not have the required
>>>>>>>>>intrinsics.
>>>>>>>>>Note that GCC 4.1 and later has these intrinsics, but I think GCC 
>>>>>>>>>3.4
>>>>>>>>>does
>>>>>>>>>not have them.
>>>>>>>>>
>>>>>>>>>(Note: a compiler intrinsic is a function for which code is emitted
>>>>>>>>>directly by the compiler, instead of being linked in from a 
>>>>>>>>>library.
>>>>>>>>>Current compilers use intrinsics for atomic operations and 
>>>>>>>>>sometimes
>>>>>>>>>for
>>>>>>>>>elementary math functions, bit twiddling, memory operations, and 
>>>>>>>>>other
>>>>>>>>>things.)
>>>>>>>>>
>>>>>>>>>I have used these spinlocks to protect the spin and spout buffers 
>>>>>>>>>in
>>>>>>>>>all
>>>>>>>>>in and out opcodes. I have not done anything for other shared 
>>>>>>>>>global
>>>>>>>>>data,
>>>>>>>>>although the channel, mixer, and table opcodes would be obvious 
>>>>>>>>>next
>>>>>>>>>steps.
>>>>>>>>>
>>>>>>>>>Steven Yi raised the possibility of using private spout buffers for
>>>>>>>>>out
>>>>>>>>>opcode instances, to be reduced after multiple opcodes have been
>>>>>>>>>mapped
>>>>>>>>>to
>>>>>>>>>different threads. However, I decided to go ahead and commit 
>>>>>>>>>spinlocks
>>>>>>>>>for
>>>>>>>>>the following reasons:
>>>>>>>>>
>>>>>>>>>(1) Spinlocks have proved useful and scalable in scenarios where 
>>>>>>>>>the
>>>>>>>>>functions that are protected typically execute for only a short 
>>>>>>>>>time.
>>>>>>>>>That
>>>>>>>>>is exactly the case here. Typically, the out opcodes add a few to 
>>>>>>>>>few
>>>>>>>>>hundred samples, and are done. This is usually a rather small part 
>>>>>>>>>of
>>>>>>>>>the
>>>>>>>>>total overhead for an instrument instance.
>>>>>>>>>
>>>>>>>>>(2) Because of this small amount of time required for out opcodes
>>>>>>>>>compared
>>>>>>>>>to other computation in the thread, the likelihood of actually 
>>>>>>>>>having
>>>>>>>>>to
>>>>>>>>>wait in a spinlock is also rather small. This is verified by the
>>>>>>>>>almost
>>>>>>>>>complete lack of clicks or pops in the multi-thread testing that we
>>>>>>>>>have
>>>>>>>>>done to date, without any protection for spout.
>>>>>>>>>
>>>>>>>>>(3) By contrast, private spout buffers would involve adding spout 
>>>>>>>>>once
>>>>>>>>>for
>>>>>>>>>each thread. I do not know the exact overhead of the atomic compare
>>>>>>>>>and
>>>>>>>>>swap intrinsic used in the spinlock, but I think it is probably
>>>>>>>>>comparable
>>>>>>>>>to the cost of adding a few sample frames. More information on this
>>>>>>>>>question would be appreciated! But if I am right, then as soon as
>>>>>>>>>ksmps
>>>>>>>>>is
>>>>>>>>>much greater than 1, spinlocks are more efficient than private 
>>>>>>>>>spouts.
>>>>>>>>>
>>>>>>>>>(4) My tests show no appreciable performance impact from spinlocks
>>>>>>>>>with
>>>>>>>>>1
>>>>>>>>>or 2 threads.
>>>>>>>>>
>>>>>>>>>(5) As the number of cores increases, the value of spinlocks 
>>>>>>>>>increases
>>>>>>>>>even faster, because they involve no operating system locks at all.
>>>>>>>>>The
>>>>>>>>>only OS overhead is memory scheduling, which would also occur with 
>>>>>>>>>OS
>>>>>>>>>locks.
>>>>>>>>>
>>>>>>>>>If anyone has compile-time, run-time, or engineering problems with 
>>>>>>>>>the
>>>>>>>>>spinlocks, please let us know.
>>>>>>>>>
>>>>>>>>>If anyone has an alternative implementation of thread-safety for 
>>>>>>>>>the
>>>>>>>>>out
>>>>>>>>>and in opcodes, please benchmark it against the spinlock
>>>>>>>>>implementation
>>>>>>>>>before committing it.
>>>>>>>>>
>>>>>>>>>Regards,
>>>>>>>>>Mike
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>-------------------------------------------------------------------------
>>>>>>>>>This SF.Net email is sponsored by the Moblin Your Move Developer's
>>>>>>>>>challenge
>>>>>>>>>Build the coolest Linux based applications with Moblin SDK & win 
>>>>>>>>>great
>>>>>>>>>prizes
>>>>>>>>>Grand prize is a trip for two to an Open Source event anywhere in 
>>>>>>>>>the
>>>>>>>>>world
>>>>>>>>>http://moblin-contest.org/redirect.php?banner_id=100&url=/
>>>>>>>>>_______________________________________________
>>>>>>>>>Csound-devel mailing list
>>>>>>>>>Csound-devel@lists.sourceforge.net
>>>>>>>>>https://lists.sourceforge.net/lists/listinfo/csound-devel
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -------------------------------------------------------------------------
>>>>>>>> This SF.Net email is sponsored by the Moblin Your Move Developer's
>>>>>>>> challenge
>>>>>>>> Build the coolest Linux based applications with Moblin SDK & win 
>>>>>>>> great
>>>>>>>> prizes
>>>>>>>> Grand prize is a trip for two to an Open Source event anywhere in 
>>>>>>>> the
>>>>>>>> world
>>>>>>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>>>>>>>> _______________________________________________
>>>>>>>> Csound-devel mailing list
>>>>>>>> Csound-devel@lists.sourceforge.net
>>>>>>>> https://lists.sourceforge.net/lists/listinfo/csound-devel
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>-------------------------------------------------------------------------
>>>This SF.Net email is sponsored by the Moblin Your Move Developer's 
>>>challenge
>>>Build the coolest Linux based applications with Moblin SDK & win great 
>>>prizes
>>>Grand prize is a trip for two to an Open Source event anywhere in the 
>>>world
>>>http://moblin-contest.org/redirect.php?banner_id=100&url=/
>>>_______________________________________________
>>>Csound-devel mailing list
>>>Csound-devel@lists.sourceforge.net
>>>https://lists.sourceforge.net/lists/listinfo/csound-devel
>>
>>
>>
>>
>>-------------------------------------------------------------------------
>>This SF.Net email is sponsored by the Moblin Your Move Developer's 
>>challenge
>>Build the coolest Linux based applications with Moblin SDK & win great 
>>prizes
>>Grand prize is a trip for two to an Open Source event anywhere in the 
>>world
>>http://moblin-contest.org/redirect.php?banner_id=100&url=/
>>_______________________________________________
>>Csound-devel mailing list
>>Csound-devel@lists.sourceforge.net
>>https://lists.sourceforge.net/lists/listinfo/csound-devel
>
>
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's 
> challenge
> Build the coolest Linux based applications with Moblin SDK & win great 
> prizes
> Grand prize is a trip for two to an Open Source event anywhere in the 
> world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Csound-devel mailing list
> Csound-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/csound-devel 


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Csound-devel mailing list
Csound-devel@lists.sourceforge.net