audioSource from callback - irrKlang Forum

Ambiera Forum

Discussions, Help and Support.

	Ambiera Forum > irrKlang
	audioSource from callback

dominikp Registered User
Quote
2021-04-05 08:29:21

Hey folks,
I am currently trying to create a small video viewer using Libav and Irrklang. However, I am struggling to find a nice solution to create a SoundSource for a given video file.

I came up with two approaches that are both rather clunky:
- Decoding the entire audio to PCM-data and calling addSoundSourceFromPCMData
- Following the .mp3 extension example. I implemented a custom IAudioStream so I can just use play2D(<Path to video>)

In both cases Video and Audio decoding must be done independently from one another, which makes syncing quite challenging.

Is there any way I can mimic an SDL-like functionality where I can create a SoundSource from a custom callback (which is similar to the readFrames method in IAudioStream)?

Written in code I would like to something like this:


auto decode_audio = [&](void* target, ik_s32 frameCountToRead) -> ik_32 {
    // Do decoding and syncing using Libav shared context between video and audio
};

...
 
SAudioStreamFormat format; // This should match the actual format returned by decode_audio

irrklang::ISoundSource* videoSound = engine->addSoundSourceFromCallback(decode_audio, "name", format); 
engine->play2D(videoSound);

The engine would periodically call decode_audio to get new PCM-data to play. I think something like this, basically creating an "anonymous" IAudioStream implementation, would be very handy. Does anyone know a hack to achieve something like this?

Sorry for the weird mix of question and feature request ðŸ˜Š. I hope someone can help me and everyone reading this: Have a nice Easter Monday.

erik Registered User
Quote
2021-04-06 07:35:33

I think irrKlang wasn't made for something like this, synching will really be not that easy.

dominikp Registered User
Quote
2021-04-06 16:28:33

Thanks for your answer.
I completely agree that audio-video syncing is not in the scope of irrklang.

However, that was not the point of my question. I think the initial post is a little bloated and the core of the question is hard to see. I will try to strip my question down a little:

Is there a nice way to play continuously, programmatically generated PCM data?
Concretely if I have a function 'generate_sound' that will create new PCM-samples whenever it is called, can I somehow convert this method to a ISoundSource that can be played by the ISoundEngine of Irrklang.

I hope this is more concise. The Video-Audio syncing is only one possible application of that idea. Others would be streaming or generating procedural sounds.

erik Registered User
Quote
2021-04-07 07:10:10

You could probably abuse the IAudioStream interface for that:

Use soundengine::registerAudioStreamLoader() with your own implemented IAudioStreamLoader which creates an implementation of IAudioStream, where you return your audio in the method readFrames(), but I'm not sure how well this would work for your use case.

dominikp Registered User
Quote
2021-04-09 17:02:19

Thank you again for your answer.

I took your idea and implemented procedural sound generation. You can find my attempt here:
https://github.com/DominikPenk/i...

I decided that you will have to register your callback with an associated name with a ProcedrualAudioStreamLoader class. Then when you call play2D with the provided name the stream loader can find the previously registered callback.
Unfortunately, IrrKlang tries to open the file provided to play2D before calling any *Loader classes. So, I had to generate some temporary, empty files, while registering a callback.

All in all, the implementation is quite messy but functional.

Edit:
I had some time to improve the code a little bit. You can Impement a custom irrklang::IFileFactory (and a irrklang::IFileReaderFactory) that specifically handles your "Callback-Files" and circumvent the need for actual files on your harddrive.
I also added a wrapper around ISoundEngine that has an additional function 'addSoundSourceFromCallback' that registers the callback and returns a SoundSource you can pass on to the play2D/play3D functions.

Create reply:

Possible Codes

Feature	Code
Link	[url] www.example.com [/url]
Bold	[b]bold text[/b]
Image	[img]http://www.example.com/image.jpg[/img]
Quote	[quote]quoted text[/quote]
Code	[code]source code[/code]

Emoticons