Oxide and audio handling

Oxide initially evolved to do image processing, the need to add an audio portion to the API arises quite quickly once the basic image handling portions were in place. Oxide should allow both a simplistic view of audio where input video fragments contribute to a common audio stream, and seperate audio data.

Oxide were initially used to create music videos, which needed a fixed pcm audio track, the portion providing the fixed pcm audio track should be possible to expand into a minimal audio engine, ideally an engine capable of cooprating with audacity.

Video clips, (first candidate source is mlt), should provide an audio stream that is mixed into the current audio fragment, the current audio fragment. If one wants to have audio from one scene continue into another scene, decoupling of audio/video is needed, this action will mute the audio contributed from the video clip,. whilst creating an audio fragment using the same/a derivate of the original movie source op, that does not provide actual image data,. this would allow to stretch the clip in the audio domain. (lip-sync should probably be maintained anyways).

sequence of events

When processing a single frame, the following events currently occur:

1. current fragment is cleared
2. audio track is copied to fragment (at the moment a "toy api" is used that copies from a static pcm buffer loaded from file, this is enough for editing music videos, later more intelligent handling is wanted here
3. gggl graph is processed
1. mlt_load might modify the current fragment buffer by mixing in audio from the audio track in it's file
2. ff_save might use oxide_audio_get_fragment to get the final audio fragment and write it to file
4. boemiite/bauxite display code fetches the fragment and writes it to the soundcard for interactive playback

it might be possible for realtime playback,. to call the audio specific portions, the general audio will be used,. but dropping of associated audiofragments will occur when video fragments are dropped,. this might be good enough for realtime playback,.. decoupling of audio from video might be all that is needed for this to work.

xml serialization

by adding audio as a valid child of sequence,. most audio handling needs should be served, the offset of audio should be implied by the position in the sequence, thus an audio element immediatly preceding a clip, will start playing when the clip starts playing, unless the offset of the audiofragment specifically overrides position, the position of the audiofragment could be relative to the current clip order,. or absolute in relation to the sequence,. absolute in relation to the sequence just means placing the audio clip at the beginning of the sequence.

        oxide
            sequence
                audio (pcmfile='' offset=0 in=0 out=8000)
                clip
                    sequence
                        audio (mlt_extract='mov1.avi')
                        clip
                            mlt_load (mov1.avi)
                        audio (mlt_extract='mov2.avi')
                        clip
                            mlt_load (mov2.avi)
                        ..
                clip
                    sequence
                        clip
                        clip
                        ..
                clip
                    sequence
                        clip
                        clip
                        ..
                    
        

unlinking could be the action of making an audio clip relative to the sequence instead of relative to a clip