New ways of buffering audio

I'm currently using SoundManager as a new playback option with my Python-based cross-platform HTTP music browsing/streaming server (it's still in a proof-of-concept state, but will be open-sourced if it turns out fine). Desktop players such as WinAmp behave nicely with the backend, buffering ahead only an adjustable amount of data, but the way SM loads tracks fully, even if the listener pauses a track after a few seconds, causes extra burden on the server, especially when someone listens to a track for a while, switches to the next on the playlist, and so on.

Would it be possible to add an option to make SM behave the same way, buffering only n seconds or bytes ahead of the playback position? If not, even an option to pause loading along with playback would help preserve precious bandwidth.

And while we're at it, why not throw in a possibility to setPosition beyond the point of bytesLoaded, making the player load and play data from that position onwards, loading the track piece by piece as requested by the user (HTTP Range requests, anyone)? I understand that VBR would cause some difficulties, but that's why it'd be handy to set the position in bytes, which should be accurate enough for seeking in most cases.

If those buffering issues were solved, the only essential thing missing would be Vorbis audio support (which Adobe is already developing, according to rumors). There has been a lot of discussion on that on the forums of some other Flash-based players, and some mad hacker even wrote a Vorbis decoder of his own!
4 people like
this idea
+1
Reply
  • I think this is a really important feature!

    Does anybody know whether the current buffering scheme is due to the inherent design of the Flash libraries / APIs? Would it actually be possible to implement on-demand seeking and buffering?

    Is it perhaps only possible when streaming from some kind of Shoutcast or Flash Media server?
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly indifferent, undecided, unconcerned happy, confident, thankful, excited

  • Many Flash video players seem to be able to seek and buffer on-demand (from the nearest video keyframe onwards, I assume), so it should be possible with audio as well, although I'm not familiar enough with Flash to say if it's got the necessary features under the hood.

    In order to allow fast seeking, a video file must begin with a header describing keyframe positions in the file, but MP3 files don't have such information. What complicates things further with MP3 is a thing called "bit reservoir" (makes frames somewhat dependent on each other), which means that starting playback from an arbitrary MP3 frame is likely to cause decoding errors, but modern decoders are seemingly able to correct those. With newer formats such as Vorbis and AAC things are probably much easier.

    Any comments from SM developers?
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly indifferent, undecided, unconcerned happy, confident, thankful, excited

  • Your ideas are.. sound (zing! :D), but the issue with Flash 8/9 is that with the provided sound API, there is no real way to do bandwidth management or true "streaming" without using some sort of special set-up on the backend eg. Flash + NetStream object requesting sound, via RTMP to Adobe's Flash Media Server (or a free/open-source alternative), to the best of my knowledge. If you watch videos on YouTube, I think the dynamic seek and buffering/loading of video may be achieved with FMS (or something developed in-house, perhaps.)

    Additionally, the Flash sound object does not support pausing or resuming of loading, as far as I know. Once a request is started, it will continue until finished. This ties into RAM use, which I'll get into re: Shoutcast/radio streams.

    One simple way to perhaps optimize the bandwidth use would be to restrict the server's output, throttling it to 16 KB/sec (or whatever the bitrate of the MP3 is, plus a little bit.)

    There have been some developments in this area with Flash 10, and a few people have been working on classes that allow "efficient" streaming from Shoutcast servers, vs. the current method of loading sound which eats more and more RAM over time as Flash holds it all there while the connection is open, and thus requires the sound to be stopped and reloaded every so often so memory will be released. I think SoundCloud may be doing this to support quick seeking within their sounds, which are typically long DJ sets of 30+ minutes.

    I do think it might be possible to support seeking within a sound by closing a HTTP request and opening a new one at the new position eg. /mp3/?position=12345, where the server returns a new header and the start of the new data (?), but that would mean server work to be able to support that sort of feature - the client would be rather "dumb" in this case.

    Flash 10 also allows for alternate formats to be introduced as you can load, generate and manipulate sound data on the fly (ie., read OGG and decode to raw RIFF/WAVE etc., feed to sound buffer), but it is non-trivial and the native Flash sound API which SM2 is based on (including onload() events, etc.) may be lost unless the developers choose to re-implement those methods.

    I have been looking at "as3mp3stream", a class hosted on Google Code, and have a prototype in progress which does allow for streaming from Shoutcast servers (and support for song change events, etc.) It does have some limitations (eg., special cross-domain permissions are required to allow the stream to work), but gives an idea of the sort of dynamic options available.

    http://getsatisfaction.com/schillmani...

    Good questions and points, I would like to see more of these sorts of features become standard with sound APIs.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. kidding, amused, unsure, silly indifferent, undecided, unconcerned sad, anxious, confused, frustrated happy, confident, thankful, excited

  • @Scott

    What you said is pretty much what I expected (mod_throttle, etc), and I also wondered about using Shoutcast or some kind of a "seek to" HTTP request.

    The latter would require a bit of intelligence on the client side, that is it would need to read and store the mp3 header in order to know where to seek to, although in my case I only want to support CBR files which should make that a whole lot easier.

    I'll follow these threads to see what happens. Thanks for the reply!
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly indifferent, undecided, unconcerned happy, confident, thankful, excited

  • Thanks a lot for the info. As I feared, Flash 8/9 obviously doesn't have enough "oomph" under the hood, but it's good to hear that 10 allows more freedom. And that idea behind as3mp3stream sounds absolutely brilliant! It'd not only take care of those pesky Shoutcast issues but would also end bandwidth problems like mine with its chunked loading, which, AFAIK, is exactly what desktop players do.

    As I hinted earlier, the backend I'm developing currently relies on Range headers from the client when it comes to seeking and assumes that the client takes care of locating the next playable MP3/whatever frame within the data stream. WinAmp, for example, does a somewhat magical job estimating seek/playback position within a streamed MP3, even if it's VBR, although on the other hand it doesn't even show the seek bar when streaming OGG Vorbis (that input module of theirs apparently isn't equally advanced yet). Too bad that Flash isn't that "smart" yet.

    It'd feel pretty wasteful to develop an additional method of seeking (queries, as suggested) for my server, when audio streaming can be performed satisfyingly with regular HTTP and a lightweight/plain-dumb backend. The advantage of avoiding backend-specific query strings would be that my web front-end would not only suit my own backend, but probably any server script that can serve M3U (which JS can easily download and parse) files containing the actual track URLs.

    Limiting bandwidth artificially isn't really an option for me, because track URLs must be downloadable as well. What I've done to reduce bandwidth and RAM usage so far is to unload every track which hasn't loaded fully whenever that particular player instance is stopped and another one started, and do periodic playlist cleanups to unload tracks which haven't been played for a while.

    Some have suggested a Java-based approach instead, but that'd feel too "20th century" IMO. :-D

    Oh well, I guess I'll have to put my browser front-end on hold for a while and wait for a more suitable playback engine. I'll definitely keep on following these discussions, because SM is by far the most flexible playback option I've come across, and I'd really like to stick to it. Keep up the great work!
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. kidding, amused, unsure, silly indifferent, undecided, unconcerned sad, anxious, confused, frustrated happy, confident, thankful, excited

  • This reply was removed.
    see the change log