To begin with, programs exist that solve this problem both in commercial
software (MetaSynth on the Macintosh) and in freeware or academic computer
music software. My package Silence version 2.1 has a program GraphMus that
does this. My newer version of Silence also does this, but is still in beta
version. Other authors have created variations on this theme.

The idea simply applying an inverse FFT to the pixel data, or some
transformation thereof, works, but is too simple and does not produce the
most musical output. It usually sounds better to use additive synthesis
(each raster in the picture becomes one oscillator, and the brightness and
color control the amplitude and perhaps the phase of the oscillator). Partly
this works better because when when we look at a picture we intuitively map
it to pitch, which is the log of the frequencies used by the FFT. Partly
this works better because people don't seem to intuitively grasp what
complex phase relationships between partials do to the sound (maybe some DSP
engineers or acousticians or very experienced computer musicians can grasp
them - I understand the math, but I can't grasp them well enough to hear
them). You get ringing, smearing, unexpected clicks, and the like.

The additive synthesis approach can be implemented easily enough from
scratch, but it is even easier to implement using Csound's adsyn opcode, in
which case the programmer need only translate the bitmap to adsyn data
tracks.

-----Original Message-----
From: Geoff Cluett 
To: Robert Tucker ; csound@maths.ex.ac.uk

Date: Friday, November 20, 1998 11:54 AM
Subject: RE: Visual waveform creator/editor?


>Hi to everyone,
> this is my first post - I haven't actually used CSound at all yet,
although
>I'm really looking forward to finding the time, as the posts I've been
>reading sound like it is one hell of a program.  My main interest is in
>using my PC to do multitrack recording in my small but very effective home
>studio.  I got to here through trying to find source code for reading .wav
>files, as I am trying to help a friend of mine write an audio application
as
>part of his two year Masters degree in Music Technology.  (I have also
>pointed him and the University to these sites).  Part of what he wants do
do
>is to allow the user to split a wave file into frequency components and
>process these using DirectX plugins.  I have just purchased a rather
>excellent book "Developers Guide to Sound" by Tim Kientzle, which is
>published by Addison-Wesley.  This contains the basics on sound, including
>human perception and digital storage.  It delves into compression, basic
>audio processing and sound synthesis.  It also contains the full source
code
>to everything he covers - one of those subjects being the Fourier
Transform,
>which as you probably know turns an audio wave from Time and Amplitude into
>Frequency, Time and Amplitude.  So, I had the idea that to display this
data
>would require a 3D display or a 2D bitmap showing greyscales as the
>Amplitude (as a textured bump map would).  This then got me thinking that
>you could easily use this as a 'painting' board and when the inverse
Fourier
>Transform is applied to that 'painted' on data, a new sound is created.  I
>don't know how you would then convert this into a score file, but I expect
>everyone else does.