Offline pitchshifting

FMOD Engine
#1

Hi! Im working on implementing rubberbands pitchshift in fmod. I got a basic realtime plugin effect working (see link). The realtime shifting is not as good as the offline, so I what Im trying to figure out is the best way of doing that. Seems like these are my options:

  1. create a instrument plugin instead of effect.
  2. use programmer sound
    3.create a codec that implement pitchshift once as the file is opened.

The sound needs to be able to follow FMOD studios timeline and loop settings. My questions are:

If I use (1) or( 2), how do I get the current playback position so that I can seek to that posiition in my sound.
If I use option (3), can I have options for setting different pitches.

I looked through the examples that comes with the FMOD distro, but didnt find a solution for this.

Thanks,
Martin.

#2

I’m not sure I follow what you mean by wanting to do this offline, do you mean modifying the source files before they are processed in FMOD Studio? Usually that kind of thing would be performed on the files before import into Studio using your DAW.

To process audio in realtime in response to automated parameters you would create a custom DSP (as you have done). At runtime FMOD handles decoding the audio files, looping and resampling so that your effect plugin need only deal with a continuous stream of audio.

#3

Thanks for the reply!

Yes I want to process the file before the “realtime” loop, so to say. Yes, its possible to do things in the DAW, but I want that process to react to game elements, ie the user can change the stretch ratio. If I do this in the DAW, the stretch ratio would be constant.

So for best quality stretching rubberband needs to access the whole audio file, which obviosly is not possible in the realtime flow where you only get a fraction of the audio at a time.

I’m looking at doing this using fmod_codec. The stretch ratio is passed as user data and when the file is opened and read ,I stretch the file with that amount. The stretch ratio wont be possible to change in realtime, but it can change for each new open and read. So each time you press play you can have a new ratio.

Im looking at the raw codec example, but what I cant figure out is how to process the audio in the read callback. The example supplied doesnt do any processing. When I try changing values in the buffer, it either crash or does nothing. This is what I have right now:

int ibs = 1024;
size_t channels = 2;
float* fbuf;
float** ibuf;
int frame = 0;
int percent = 0;
size_t countIn = 0, countOut = 0;
const float** ptrs = new const float* [2];
float* bufL = new  float[8192 * 4];
float* bufR = new  float[8192 * 4];

float** tmp = new float* [2];

unsigned int length = 0;
FMOD_RESULT F_CALLBACK rawread(FMOD_CODEC_STATE *codec, void *buffer, unsigned int size, unsigned int *read)
{
    RubberBand::RubberBandStretcher* m_stretcher = (RubberBand::RubberBandStretcher *) codec->plugindata;

    channels = 2;

    ptrs[0] = bufL;
    ptrs[1] = bufR;
    
    length = 1024;
    codec->fileread(codec->filehandle, buffer, size, &length, 0);

    length = 1024;// std::min(1024, (int)*read);
    // De-iterleave
    
    if (length > 0)
    {
        for (int i = 0; i < length - 1; i++) {
            bufL[i] = ((float*)buffer)[0 + i * channels];
            bufR[i] = ((float*)buffer)[1 + i * channels];
        }

        m_stretcher->process(ptrs, length, 0);

        int avail = m_stretcher->available();
        if (avail > 0) {
            m_stretcher->retrieve(tmp, avail);

            // Re-interleave
            for (int i = 0; i < avail - 1; i++) {
                ((float*)buffer)[0 + i * channels] = tmp[0][i];
                ((float*)buffer)[1 + i * channels] = tmp[1][i];
            }

        }
        
    }

    read = &length;
    return FMOD_OK;// 
}
#4

Codec plugins are intended for supporting new file types, the raw codec example we provide makes assumptions about the file being loaded by the user. Typically in the open callback you’d read some of the file to determine whether it’s the format you can decode, then go ahead with your decoder.

If you are looking to process an entire file at a particular pitch, instead of making a plugin I’d recommend you use FMOD to decompress the file then access the memory to make your modifications. So first use System::createSound with the FMOD_CREATESAMPLE flag on any file you like (this will decompress to memory). Then use Sound::lock access to the entire audio data, make any modifications you like, then Sound::unlock to finish. Now you can play the Sound.

EDIT 1: Be mindful that the decompressed audio may not be PCM float, it could be 16bit integer which may require converting if your pitch shifter expects float. You would need to convert it back to reuse the existing sound, or create a new FMOD::Sound to play the float data.

EDIT 2: Additionally to have these files play back within an Event, you would need to use programmer instruments.

1 Like