Csound Csound-dev Csound-tekno Search About

[Csnd] New music made with ctcsound and deep learning techniques

Date2022-04-22 23:45
FromPrent Rodgers
Subject[Csnd] New music made with ctcsound and deep learning techniques
I've been very happy with the ctcsound python library. It was a bit challenging to get started, but I was able to get it working with the assistance of Michael Gogins and Victor Lazzarini. I was interested in using csound from python because of the large library of deep learning techniques applied to music, all of which use pythong. I was terribly disappointed with the music that has been produced so far using machine learning and deep learning methods. I've been combing through academic articles on the subject over the past year, searching for something that worked, and all I have heard lacked any kind of musicality. It's kind of like the dog that can stand on his hind legs and sing. At first you are amazed that he can do the trick, until you realize that he's not a very good singer. That's my impression, and I'd love to be proven wrong. 

I set out to try my hand at the task. My first effort took an existing model created by Google Magenta called coconet. The original model was made with Tensorflow. I was never able to get it to work on my system, mostly because it relies on a version of Tensorflow that has long since been replaced by newer versions, and I could not get to run. Another researcher (Kevin Donoghue at https://github.com/kevindonoghue/coconet-pytorch) was able to port it to Torch, and I was able to get that version working. 

The basic idea is that you start with a very large corpus of actual Bach chorales. You mask some notes and then train a deep neural network model to replace the missing notes. As it gets closer to the original, over many tens of thousands of iterations, it gets very good an generating the missing notes. In fact, it can generate a respectable chorale out of a few random notes. This is the dog that can sing. They are still just MIDI files of Bach chorales, that don't sound all that exciting. It's amazing that it can be done at all. Bach was a master after all. But it's just a MIDI file.

What I did at that point is take some existing Bach chorales, and subtract one of the voices, and have the model make replacements. I then subtracted a different voice and had the model replace that one. I did this 16 times, until I had a chorale that was something like the original, but clearly different. There were many wrong notes, and awkward cadences. But it was working. 

I then applied by algorithmic techniques to evaluate the artificial chorales, one time step at a time. I stretched the challenging sections so that they were much longer and louder, anywhere from twice as long to nine times as long. I masked notes to create octave shifts, arpeggios, and doubled up the voices. Finally, I fed the results into some Csound code that used my Bosendorfer sample library, and a tuning that matched what Bach would have used when he wrote the chorales. 

I think they sound pretty good. Maybe even musical. I published the results on github here: https://github.com/prentrodgers/coconet-pytorch-csound
I point to some musical samples created using the techniques here: http://ripnread.com/sample-page/code/fantasia-on-artificial-chorales/

Csound mailing list
Csound@listserv.heanet.ie
https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
Send bugs reports to
        https://github.com/csound/csound/issues
Discussions of bugs and features can be posted here

Date2022-04-23 02:17
FromMichael Gogins
SubjectRe: [Csnd] New music made with ctcsound and deep learning techniques
Thanks, I'll check it out. 

On Fri, Apr 22, 2022, 18:45 Prent Rodgers <prent.rodgers@gmail.com> wrote:
I've been very happy with the ctcsound python library. It was a bit challenging to get started, but I was able to get it working with the assistance of Michael Gogins and Victor Lazzarini. I was interested in using csound from python because of the large library of deep learning techniques applied to music, all of which use pythong. I was terribly disappointed with the music that has been produced so far using machine learning and deep learning methods. I've been combing through academic articles on the subject over the past year, searching for something that worked, and all I have heard lacked any kind of musicality. It's kind of like the dog that can stand on his hind legs and sing. At first you are amazed that he can do the trick, until you realize that he's not a very good singer. That's my impression, and I'd love to be proven wrong.

I set out to try my hand at the task. My first effort took an existing model created by Google Magenta called coconet. The original model was made with Tensorflow. I was never able to get it to work on my system, mostly because it relies on a version of Tensorflow that has long since been replaced by newer versions, and I could not get to run. Another researcher (Kevin Donoghue at https://github.com/kevindonoghue/coconet-pytorch) was able to port it to Torch, and I was able to get that version working.

The basic idea is that you start with a very large corpus of actual Bach chorales. You mask some notes and then train a deep neural network model to replace the missing notes. As it gets closer to the original, over many tens of thousands of iterations, it gets very good an generating the missing notes. In fact, it can generate a respectable chorale out of a few random notes. This is the dog that can sing. They are still just MIDI files of Bach chorales, that don't sound all that exciting. It's amazing that it can be done at all. Bach was a master after all. But it's just a MIDI file.

What I did at that point is take some existing Bach chorales, and subtract one of the voices, and have the model make replacements. I then subtracted a different voice and had the model replace that one. I did this 16 times, until I had a chorale that was something like the original, but clearly different. There were many wrong notes, and awkward cadences. But it was working.

I then applied by algorithmic techniques to evaluate the artificial chorales, one time step at a time. I stretched the challenging sections so that they were much longer and louder, anywhere from twice as long to nine times as long. I masked notes to create octave shifts, arpeggios, and doubled up the voices. Finally, I fed the results into some Csound code that used my Bosendorfer sample library, and a tuning that matched what Bach would have used when he wrote the chorales.

I think they sound pretty good. Maybe even musical. I published the results on github here: https://github.com/prentrodgers/coconet-pytorch-csound
I point to some musical samples created using the techniques here: http://ripnread.com/sample-page/code/fantasia-on-artificial-chorales/

Csound mailing list
Csound@listserv.heanet.ie
https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND
Send bugs reports to
        https://github.com/csound/csound/issues
Discussions of bugs and features can be posted here
Csound mailing list Csound@listserv.heanet.ie https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND Send bugs reports to https://github.com/csound/csound/issues Discussions of bugs and features can be posted here