Sound recording support for Orx

2»

Comments

  • edited November 2010
    I had some problems with the downloadable version of OpenAL soft.
    So I checked it out via git. This newest version support now capturning via WinMM. So no need for portaudio.
    Which is good, because I had some problems with the portaudio implementation. I was not able to compile it.

    Anyway, so I added the "dev" branch of OpenAL soft to the SVN and compiled a static library from it. With this library it is possible to capture audio through WinMM without the library from creative.

    I linked this static library to the orx project. Now everything works fine..playback(I didn't break it^^) and capturing. The only weird thing is that when including the headers of the dev branch(in extern/OpenAL-dev/include) I get several "error LNK2019: unresolved external symbol "__imp__alcCloseDevice" in function "@orxSoundSystem_OpenAL_Init@0";none".
    However if I use the old header files(extern/OpenAL-1.12.854/include) together with my new library, everything works like it should. That's totally weird and I don't get why :/


    I like the possiblity to use both the creative lib an the opensource version in combination. But I think this is a personal preference. I'm thinking of linking soft OpenAL via DLL for my project. But well I will leave the SVN version with the static linking.
  • edited November 2010
    tdomhan wrote:
    I had some problems with the downloadable version of OpenAL soft.
    So I checked it out via git. This newest version support now capturning via WinMM. So no need for portaudio.
    Which is good, because I had some problems with the portaudio implementation. I was not able to compile it.

    Excellent news! :)
    Anyway, so I added the "dev" branch of OpenAL soft to the SVN and compiled a static library from it. With this library it is possible to capture audio through WinMM without the library from creative.

    I linked this static library to the orx project. Now everything works fine..playback(I didn't break it^^) and capturing. The only weird thing is that when including the headers of the dev branch(in extern/OpenAL-dev/include) I get several "error LNK2019: unresolved external symbol "__imp__alcCloseDevice" in function "@orxSoundSystem_OpenAL_Init@0";none".
    However if I use the old header files(extern/OpenAL-1.12.854/include) together with my new library, everything works like it should. That's totally weird and I don't get why :/

    Mmh, I'll check the source but it's possible they don't compile the library statically often and they might have forgotten a declspec somewhere. I remember I had to fix something like this but not sure if it was for OpenAL or libsndfile. It shouldn't be too hard to patch anyway.
    I like the possiblity to use both the creative lib an the opensource version in combination. But I think this is a personal preference. I'm thinking of linking soft OpenAL via DLL for my project. But well I will leave the SVN version with the static linking.

    Great. Why don't you use the Creative dll in your project, in this case? They should be compatible as they share the same API.
  • edited November 2010
    alright I implemented basic sound capturing in the openAL plugin. this was the easiest part of the whole journey^^

    there are still some things that need to be done, but the basic stuff is working.

    what are the sender and the recipient of orxEVENT_SEND?
    to what should I set them?
  • edited November 2010
    Great!

    Well, in your case you have no sender nor recipient, so you can just set both to orxNULL. There are orxEVENT_* macros to help initialize & sent events.
  • edited November 2010
    ok I already did so.

    Furthermore I was not completely sure about creating the clock:
    Which orxCLOCK_TYPE is appropriate? I chose orxCLOCK_TYPE_CORE.
  • edited November 2010
    Mmh, there should only be one core clock in orx in order for people to retrieve it easily.

    You shouldn't actually need to create a clock if you use the global timer (which registers on the core clock too). I think that's how I did streaming in the same plugin.
  • edited November 2010
    ok I used addGlobalTimer now.
    I implemented the capturing in the OpenAL Plugin. Additionally I implemented recording to a file.

    directly playing back recorded data is bit more tricky. we would have to create a interface to append buffers to the playback queue.
    But I don't even know if such a feature is necessary/needed.
  • edited November 2010
    tdomhan wrote:
    ok I used addGlobalTimer now.
    I implemented the capturing in the OpenAL Plugin. Additionally I implemented recording to a file.

    Great! I'll look at this tonight or tomorrow.
    directly playing back recorded data is bit more tricky. we would have to create a interface to append buffers to the playback queue.
    But I don't even know if such a feature is necessary/needed.

    I wouldn't bother with that right now as none of us can't see any use with the current network-less state of orx. (And people use vent/mumble anyway ;)).
  • edited December 2010
    I was wondering, should I write a small tutorial about the sound recording or is it too trivial?
  • edited December 2010
    Experience taught me that there are no trivial tutorials! ^^

    So any tutorials is welcome, including sound recording. =)
  • edited December 2010
    Excellent wiki tutorial! Nice stuff, tdomhan. Getting documentation out for beginners and advanced users alike for all kinds of uses of orx, is very important for increasing the uptake.
  • edited January 2011
    I've just merged your code into the trunk, it works great. Congrats again!

    I've fixed a couple of issues with other platforms and haven't tested on linux/mac yet, but I'm sure it'll work fine.

    I took the liberty to simplify a bit the API while still granting the same degree of freedom you gave the user. I hope you don't mind.

    You can now decide on a per packet basis if you want to save them to a file or not. You can also modify them prior to file saving (I postponed the event handler registration to ensure the user has the time to register his first).
    Custom buffers can now be provided for modified packets, the user is responsible of the life/death of such buffer.

    There's a small example on a simple sound processing in the bounce demo on the svn. While shift is pressed, packets will be saved and their frequency will be doubled. No packet will be saved if shift isn't pressed.

    I'll update your tutorial in the coming days. Thanks again for your contribution! =)
  • edited January 2011
    ok, that's great news.

    I haven't looked into the changes you made to the API, but I will do so later. but I suppose they are sane ;)

    you're welcome. btw, I'm much more thankful for all the effort you put into this project! kudos to you! :D
  • edited January 2011
    tdomhan wrote:
    ok, that's great news.

    I haven't looked into the changes you made to the API, but I will do so later. but I suppose they are sane ;)

    I hope so! ^^

    I've just updated your tutorial so you can judge by yourself.
    you're welcome. btw, I'm much more thankful for all the effort you put into this project! kudos to you! :D

    Ahah, thanks! I hope orx still fits your need then. :)
  • edited January 2011
    yeah orx totally fits my needs so far.

    regarding the capturing:
    this wont work for me, as I definitly need the fixed block size.
    but I guess this is a feature not many can make use of.
  • edited January 2011
    I see, but you can still have a fixed block size on the client side.

    Simply accumulate the info in your own buffer. If you need to write the data to file, just give the buffer back when it's full through the payload.

    In the same way, the polling frequency can be achieved using a orx timer to collect your locally stored data.

    It's actually the same code you wrote except it's not internal anymore. If you think it's too much of a hassle I can put the code back.
    However using a struct to initialize the capture is too error prone as I found when trying your branch first (the boolean where not set in your example resulting to a crash when starting the capture with my vs (boolean had the 0xCDCDCDCD value)). I guess I can add two extra parameters to the function if you'd rather have the feature internally.
  • edited January 2011
    quess I got very uncommon requirements regarding the sound capturing.(and also the playback)

    leave it the way it is and I will figure something out.

    Maybe I will completely switch to portaudio after all, in order to keep everything in sync.
  • edited January 2011
    Out of curiosity, what are your requirements for sound playback? Is it something that could be added to the current plugin?

    I was thinking I could simply write a tutorial on how to have fixed size block for recording. Though it might not be of any use to you as it'd be based on your initial method, it could help any newcomers with similar requirements than yours.
  • edited January 2011
    ah yeah I sure can. well beside the fixed block size there are the following requirements:
    • well most importantly: I would like the playback and the capturing to stay in sync.
    • the least latency possible(ok this one is very generic^^)
    • information about how big the latency actually is, meaning at what time can I expect the current sample to be played back and accordingly on what exact time were the samples captured exactly
    • synchronization between the visual stuff and the audio -> relying on the same clock
    ok my project is really time critical and I guess this are very specifc requirements that normal games won't have.

    I completely switched to portaudio now. all the above is doable with this library. things just got a bit more complicated then "orxSound_Play(...)"^^

    I guess the current API is sufficient for most games.
  • edited January 2011
    tdomhan wrote:
    ah yeah I sure can. well beside the fixed block size there are the following requirements:

    I've been thinking about the fixed block size and was considering adding it back before release 1.3. Not that it will be of any use for you right now, but maybe someone in the future will have the same requirement! ^^

    I've been toying a bit with various sound transformations and I'll probably spend some time toying with things based on Sylvain Marchand's work (he was one of my teachers during my last year of master, 10 years ago): http://dept-info.labri.u-bordeaux.fr/~sm/Publications/
    • well most importantly: I would like the playback and the capturing to stay in sync.none

    Can you elaborate a bit more on this point? But I'm already sensing it's not an easy things to do with OpenAL. =)
    [*]information about how big the latency actually is, meaning at what time can I expect the current sample to be played back and accordingly on what exact time were the samples captured exactlynone

    For the capturing, your system of timestamps should have worked just fine, however I don't think you can accurately predict when the buffers sent by OpenAL will reach the sound card. If I remember correctly, that was also part of a big rant from Jonathan Blow when discussing a port of Braid for linux on his blog.
    [*]synchronization between the visual stuff and the audio -> relying on the same clock
    none

    That isn't a big deal as visuals will react immediately (that is within the same frame, which should be more than enough for human perception, unless you're running at 10 FPS ;)).
    ok my project is really time critical and I guess this are very specifc requirements that normal games won't have.

    I completely switched to portaudio now. all the above is doable with this library. things just got a bit more complicated then "orxSound_Play(...)"^^

    I guess the current API is sufficient for most games.

    I'm glad you found a working solution with PortAudio. Too bad orx didn't fit your requirements, maybe with a future version? ;)

    However I'm always willing to improve orx use while trying to keep things as simple as possible for most of the "usual" needs. That being said, writing a PortAudio based plugin and/or extending features through the use of events, which are less intruding than direct API changes, is something that could totally happen and I'd be happy to work in this direction. :)
  • edited January 2011
    I will write how the whole thing works using portaudio, maybe it will then become a bit clearer what I ment then:
    iarwain wrote:
    tdomhan wrote:
    ah yeah I sure can. well beside the fixed block size there are the following requirements:

    I've been thinking about the fixed block size and was considering adding it back before release 1.3. Not that it will be of any use for you right now, but maybe someone in the future will have the same requirement! ^^

    I've been toying a bit with various sound transformations and I'll probably spend some time toying with things based on Sylvain Marchand's work (he was one of my teachers during my last year of master, 10 years ago): http://dept-info.labri.u-bordeaux.fr/~sm/Publications/
    • well most importantly: I would like the playback and the capturing to stay in sync.none

    Can you elaborate a bit more on this point? But I'm already sensing it's not an easy things to do with OpenAL. =)
    So in portaudio it works like this: You first register a callback that will be called each time the audio hardware needs new data. the buffers will be created by portaudio. the callback will be called from some fancy high priority/low level audio thread. if I understood it correctly it will be triggered/called from some audio hardware interrupt. the result is that you may not allocate memory and do any disk IO. the upside is that it will even be called if the game itself lags/halts etc.

    The important thing is that there is only one callback for the input and the output. this means both use the same clock for the timestamps. furthermore there will be as much samples available from as an input as there are for the output, e.g. if you said you will always want 4096 samples then for each time the callback is called you have to fill the output buffer with 4096 samples and you can fetch 4096 samples from the input buffer.
    [*]information about how big the latency actually is, meaning at what time can I expect the current sample to be played back and accordingly on what exact time were the samples captured exactlynone

    For the capturing, your system of timestamps should have worked just fine, however I don't think you can accurately predict when the buffers sent by OpenAL will reach the sound card. If I remember correctly, that was also part of a big rant from Jonathan Blow when discussing a port of Braid for linux on his blog.
    yep I didn't find a way to determine the latency in OpenAL. portaudio tries to estimate the latency. but depending on the host API that is being used this estimate might not be very accurate.
    [*]synchronization between the visual stuff and the audio -> relying on the same clock
    none

    That isn't a big deal as visuals will react immediately (that is within the same frame, which should be more than enough for human perception, unless you're running at 10 FPS ;)).
    hehe, yeah that is actually the problem, it is too fast!! ;)
    for example in my game I have a background sound. now let's say I would like to have some audio effects as soons as a specific part of the background sound is currently played. because of all the buffering the background music will not be played back as soon as the output buffer is filled but probably several hundret milli seconds later. so I have to delay the visuals by exactly the time. otherwise you would first see the specific effect and then after XY milli seconds you would hear the sound.

    but in order to do that you need to rely on the same clock andsome information about the output latency.

    similar stuff is true for the input.
    ok my project is really time critical and I guess this are very specifc requirements that normal games won't have.

    I completely switched to portaudio now. all the above is doable with this library. things just got a bit more complicated then "orxSound_Play(...)"^^

    I guess the current API is sufficient for most games.

    I'm glad you found a working solution with PortAudio. Too bad orx didn't fit your requirements, maybe with a future version? ;)

    However I'm always willing to improve orx use while trying to keep things as simple as possible for most of the "usual" needs. That being said, writing a PortAudio based plugin and/or extending features through the use of events, which are less intruding than direct API changes, is something that could totally happen and I'd be happy to work in this direction. :)
  • edited January 2011
    tdomhan wrote:
    I will write how the whole thing works using portaudio, maybe it will then become a bit clearer what I ment then:[...]

    So in portaudio it works like this: You first register a callback that will be called each time the audio hardware needs new data. the buffers will be created by portaudio. the callback will be called from some fancy high priority/low level audio thread. if I understood it correctly it will be triggered/called from some audio hardware interrupt. the result is that you may not allocate memory and do any disk IO. the upside is that it will even be called if the game itself lags/halts etc.

    I think it might be the same hardware-interrupt way of working for non-software versions of OpenAL.
    The important thing is that there is only one callback for the input and the output. this means both use the same clock for the timestamps. furthermore there will be as much samples available from as an input as there are for the output, e.g. if you said you will always want 4096 samples then for each time the callback is called you have to fill the output buffer with 4096 samples and you can fetch 4096 samples from the input buffer.

    Thanks for the details! That's very helpful. Currently sound input/output is on the same clock in orx, but not on the same ticks. It's easy to change so I'll do it tonight and make sure that if a recording callback is called it'll always be synchronized with any input buffering.
    hehe, yeah that is actually the problem, it is too fast!! ;)
    for example in my game I have a background sound. now let's say I would like to have some audio effects as soons as a specific part of the background sound is currently played. because of all the buffering the background music will not be played back as soon as the output buffer is filled but probably several hundret milli seconds later. so I have to delay the visuals by exactly the time. otherwise you would first see the specific effect and then after XY milli seconds you would hear the sound.

    Ok, now I understand better your issue. :) I guess that by modifying the numbers of buffers and their size we should be able to obtain a similar result. I could expose those in the config system when one needs other values than the "safe" default ones.

    If you'd like to give it a try, I'd be happy to make such changes. Even if it's no use anymore for you, it'd be so as to see if orx could achieve similar results for projects that are very demanding on sound synchronization. :)
  • edited February 2011
    yes, I guess it wouldn't matter for my project as the deadline is in a couple of weeks.

    however I would be willing to test the implementation inside the orx engine and compare the performance. :D
  • edited February 2011
    Sounds good! I'll make the changes probably just after the 1.3 that is coming very soon.

    I already made the one so that recording and playback streaming are synchronized. :)
Sign In or Register to comment.