Re: RFC: GNOME 2.0 Multimedia strategy

[Erik Walthinsen <omega temple-baptist com>]

On Sun, 20 May 2001, Jim Gettys wrote: (leaving context for fwd to gst-dev)

> I believe we want a (relatively) simple audio server, that exposes most
> things to clients (and can be audited for network intrusions).
>
> The issue is we don't have a good one.  We have at least 4 not so good
> ones.  And it needs to be satisfactory to not just Gnome, but KDE and others.
> (thank you very much: I want to be able to have both Gnome and KDE apps
> playing audio on my desktop at once).
>
> These include:
> 	esd
> 	NAS
> 	AF
> 	A nameless one Keith Packard wrote.
>
> And there are others I can name, if I scratch my head for a while.  Each
> is deficient in some way or another; some more deficient than others.
>
> Such a server needs to be coordinated with the X server using the X
> syncronization extension.
>
> To try to solve all the world's problems (and world hunger), for multimedia
> in one place stikes me as a violation of modularity.  No matter how hard
> you try, you'll not get it right.  I can tell you (first hand - I am coauthor
> of AF, one of the inadaquate servers we have available), that just doing
> the audio server by itself is enough to keep in one's brain at once, without
> worrying about all of multimedia streaming requirements.
>
> So I believe strongly we need to keep separate the signal processing and
> internal needs  (including synchronization) of an application mostly separate
> from the problem of mixing audio streams from different sources
> (applications).  If you do this right (allow explicit control of time,
> as AF does), multiple applications can synchronize quite nicely on the
> rare occasions that they need to.
>
> So lets see if we can get two threads of discussion going here:
> 	1) an audio server, to deal with the multiple client and hardware abstraction
> 	problem,
> 	2) the general signal processing/synchronization needs of applications.
> The two should only be loosely coupled (making sure 1) provides what 2)
> will need to get its job done, and acknowledging that there will, over
> the years, be many more instances of 2) than of 1).
>
> So I claim 1) needs to be much more than a Gnome discussion to succeed.
Definitely.  On the linux-audio-dev list right now is (among other
semi-related topics) a large discussion on a Rewire equivalent for Linux.
>From what I gather, Rewire (for Windoze) allows all the various audio
applications to use each other as sources as sinks, basically virtual
sound cards.  That way a drum machine can write to a virtual sound card,
and that virtual sound card can then be used as a loopback input to
another program, say a sequencer or mixer.

It seems to me that we might want to make this part of the requirement for
a good sound server, with the understanding that it should be simple, yet
still be able to route any application to any other.

> A year or more a ago, I tried to poke people into working on this problem,
> but didn't get critical mass to make progress.  Keith Packard and I set
> up a "soundserver xfree86 org" mailing list to try to get it going, but
> did not really advertize it much: neither of us has had time to do more
> than comment on what little traffic there has been (both of us have built
> audio servers).  What is really needed is a good architect with enough
> time on their hands (which leaves Keith and I out, being too busy already).
> Maybe we should try again to get things moving in this area.  But it really
> needs at least one full time really good person to make good progress,
> I suspect.  There is alot of code available to help the problem (all of
> the above list's code is available), but the design has to be done and
> a real artifcat built.
>
> That being said, gstreamer, from what I can gather, though I haven't studied
> it carefully, looks like a reasonable library to help the apps do what
> they need and is a strong candidate for Gnome's need for 2).  But I don't
> think it, itself should have to deal with the hardware abstraction and
> network issues on top of everything else.  The problems are enough different
> that trying to mix both into one is likely to fail, and won't withstand
> the test of time; I can guarantee we'll learn lots more about signal
> processing over the next decade.
So, what I proposed to the GStreamer list, and will outline quickly here
(and expand on soon) is that the 'sound server' actually be set up like X,
with the core of the system being a wire protocol, and transports for said
protocol (which is complicated a bit by the fact that there's control and
data flow with different properties, but XShm is the same thing...).  The
software piece would be an xlib-like library that only deals with the
protocol/transport-specific bits of the soundserver API.  A program can
then be written to do the esddsp trick, gnome-libs can have a simple
interface built into it, and the sound server can be written with anything
you want.

One wishlist item is that one be able to gnome_play_sound("bleep.mp3").
The protocol should be able to handle some variance of data type, not just
raw audio (IMO).  This would make GStreamer an ideal basis for the sound
server itself (NOT the clients, unless they need that complexity, which
should be provided via some other interface to the same sound server
process, probably CORBA).

I gotta go, I'll explain more when I get back.

TTYL,
    Omega

      Erik Walthinsen <omega temple-baptist com> - System Administrator
        __
       /  \                GStreamer - The only way to stream!
      |    | M E G A        ***** http://gstreamer.net/ *****
      _\  /_