Internet streaming workaround with an OPENUSER sound?

FMOD Engine
1

Hi, I’m just wondering if there’s some workaround for streaming audio over the internet on FMOD’s HTML5 implementation by managing the stream in JavaScript, and passing the raw PCM data directly to an FMOD sound?

My idea was to calculate the buffer size outside of FMOD and manually stop playback if there wasn’t a big enough buffer. I get that this may be a long shot, but I think it’d be really cool to send live radio through FMOD in the browser.

I’ve managed to get streamed audio to work without errors by using Sound.lock() and Sound.unlock(), but when I play the sound, it gets stuck in FMOD_OPENSTATE_SETPOSITION.

I’ve also tried using pcmreadcallback and got the same result.

Here’s an extract of some of the code I’m experimenting with. Pointer.deref() is just a wrapper for the whole outval.val pattern.

Is there some fundamental barrier to this that I’m missing?

        const sound = new Pointer<any>();
        const info = FMOD.CREATESOUNDEXINFO();
        info.defaultfrequency = sampleRate;
        info.decodebuffersize = 44100;
        info.numchannels = 2;
        info.length = info.defaultfrequency * info.numchannels * 2 * 5; 
        info.format = FMOD.SOUND_FORMAT_PCM16;
        const mode = FMOD.OPENUSER | FMOD.LOOP_NORMAL;

        const ptr1 = new Pointer<any>();
        const ptr2 = new Pointer<any>();
        const len1 = new Pointer<any>();
        const len2 = new Pointer<any>();
        FMOD.Result = FMOD.Core.createSound('', mode, info, sound);
        FMOD.Result = sound.deref().lock(0, info.length, ptr1, ptr2, len1, len2);
        let offset = 0;
        while (offset < len1.deref()) {
            const { done, value } = await reader.read();
            if (done) break;
            const { channelData, samplesDecoded, errors } = await decoder.decode(value);
            console.log(samplesDecoded, offset, len1.deref());

            const [ left, right ] = channelData;
            for (let i = offset; i < (samplesDecoded >> 2); i++) {
                let leftIndex = ptr1.deref() + i << 2;
                let rightIndex = leftIndex + 2;
                FMOD.setValue(leftIndex, left[i], 'i16');    // left channel
                FMOD.setValue(rightIndex, right[i], 'i16');    // right channel
            }
            offset += samplesDecoded;
            // break;
        }
        sound.deref().unlock(ptr1.deref(), ptr2.deref(), len1.deref(), len2.deref());
        const openstate = new Pointer<any>();
        const percentbuffered = new Pointer<any>();
        const starving = new Pointer<any>();
        const diskbusy = new Pointer<any>();
        sound.deref().getOpenState(openstate, percentbuffered, starving, diskbusy);
        console.log(openstate, percentbuffered, starving, diskbusy);
        FMOD.Result = FMOD.Core.playSound(sound.deref(), null, null, {});
        setInterval(() => {
            sound.deref().getOpenState(openstate, percentbuffered, starving, diskbusy);
            console.log(openstate, percentbuffered, starving, diskbusy);
        }, 2000);
2

Hi, apologies for the delayed response.

A few questions:

  • What version of FMOD are you using?
  • Is the FMOD system being updated at any point outside of the snippet you’ve posted? If not, this may be the issue.
  • Does add FMOD_NONBLOCKING to the sound mode make any difference?
  • Failing the above, could I get you to swap to the logging version of the FMOD libs (suffixed with “L”), use Debug_Initialize ahead of system creation and initialization to set your logging level to FMOD_DEBUG_LOG, and post a log where the issue occurs?
3

Hi, no worries at all.

I got it to work!

Thanks for the help! FMOD is being updated in a separate loop. FMOD_NONBLOCKING is not supported I believe. I switched to fmodstudioL.js, and the logs helped quite a bit.

I still have literally no idea what the issue was. Maybe there was some sort of logical mistake in my code. Either way, it’s now definitely possible to stream audio over the internet into FMOD sounds, even on the Emscripten port.

I switched to using pcmreadcallback instead of Sound.lock(), so that may have had something to do with it.

The trick was to move all asynchronous stuff away from FMOD, because when pcmreadcallback is called, it needs bytes right away. The way I’m doing it now is that pcmreadcallback reads from a buffer that’s filled by an AudioWorkletNode that does all the decoding and interleaving of the samples.

Here’s what I’ve got so far.

I get that this use case is quite niche, but do you think that HTML5 internet streaming is something that FMOD might officially support any time soon?

    async fetch() {
        await gesture.promise;
        const context = new AudioContext();
        const source = context.createMediaElementSource(this.element);
        await context.audioWorklet.addModule('/pcmProcessor.js');
        const node = new AudioWorkletNode(context, 'pcm-processor');
        node.port.onmessage = async (event: MessageEvent<ArrayBuffer>) => {
            const { full } = this.buffer.write(event.data);
            if (full && !this.buffer.loop) {
                // TODO
            }
        };
        source.connect(node);
        this.element.play();
    }

/* ... */
    async load() {
        await this.buffer.ready;

        const sound = new Pointer<any>();
        const info = FMOD.CREATESOUNDEXINFO();
        const { sampleRate, numChannels, bytesPerSample } = this.soundInfo;

        info.length = this.length * numChannels * sampleRate * bytesPerSample;
        info.numchannels = numChannels;
        info.defaultfrequency = sampleRate;
        info.decodebuffersize = sampleRate;
        info.format = FMOD.SOUND_FORMAT_PCM16;

        info.pcmsetposcallback = (
            sound: any,
            subsound: any,
            position: any,
            postype: any
        ) => {
            console.log(this.url, 'seeking', position);
            const { sampleRate } = this.soundInfo;
            this.seek(position / sampleRate);
            return FMOD.OK;
        };

        info.pcmreadcallback = (sound: any, data: any, datalen: number) => {
            const { view, wrappedView, wrap, underflow } = this.buffer.read(
                Math.min(datalen, this.buffer.capacity)
            );

            if (underflow) {
                this.stop();
                this.buffer.ready.then(() => this.restart());
                return FMOD.OK;
            }
            FMOD.HEAPU8.set(view, data);
            if (wrap) {
                FMOD.HEAPU8.set(wrappedView, data + view.length);
            }
            return FMOD.OK;
        };
        FMOD.Result = FMOD.Core.createStream('', FMOD.OPENUSER | FMOD.ACCURATETIME | FMOD.LOOP_NORMAL, info, sound);
        this.handle = sound.deref();
        return true;
    };
4

Happy to hear that it’s all working now.

Unfortunately I don’t have any updates to offer on the status HTML5 net streaming, but I’ve noted your interest on our internal feature tracker.