Re: Polypaudio for Gnome 2.10, the next steps

[Mike Hearn <mike navi cx>]

Hi Lennart,

Thanks! This is the sort of email I was looking for. I'll shut up about
this idea now I've got some feedback on it. By the way
you're doing a great job: this sort of quality argument, responsiveness to
criticism and dedication is a text book case of how to get free software
accepted by people. I for one am well impressed!

OK I have read your points and am mostly convinced now. It seems
Polypaudio has a lot of things going for it. Let me respond to a few
things though. Firstly in another email you said:

> Is the fact that they are emplyed by SuSE a point for or against them?
> ;-)

I didn't mean "they are employed by SuSE so their work must be good", I
was just responding to somebody who implied dmix was unmaintained or
wasn't being worked on. The last commit seems to have been 3 weeks ago so
it clearly is being worked on, I thought I should make it clear. I
certainly wasn't comparing your work to theirs quality wise! I can't judge
that anyway, I don't know enough.

> Polypaudio is very good in these two issues. Since movie players
> usually query the latency repeatedly in very short periods of time and
> getting latency info usually involves a roundtrip, polypaudio
> interpolates the latency information. This works quite well, lip sync
> video playback with mplayer or xine using their polypaudio drivers
> works astonishingly well. In addition to that polypaudio has some very
> nifty techniques for allowing large playback buffers (which is
> required when doing network transfers with large and non-deterministic
> latency) but allows immediate reaction on user events (e.g. the user
> presses "pause").

OK. I don't know much about this. It seems that this is a function of the
code which forwards audio between hosts though, and my proposal
deliberately left this vague. I suggested a gstreamer pipeline but that
was just an example, it could have been Polypaudio listening to the
sockets or anything. 

The key point was grabbing the sound on a per-process basis from the
lowest level possible rather than rewriting apps (or writing lots of
drivers). It's kind of hard to hack the kernel to do this, so alsalib is
the next level up where you can reroute audio. Currently you've written
lots of drivers for different programs which is great but there are still
ones you missed (eg Wine) or cannot change (because they only support OSS
and have no driver framework) and therefore would need to use an
LD_PRELOAD type wrapper. So why not have only one such wrapper program
(aoss) and then redirect audio into Polypaudio at the alsalib level? There
could be similar OSS level redirectors for other operating systems or they
could deal with it at the kernel level.

Well you addressed this point further down so read on for more questions :)

> Besides, Polypaudio already exists and works, while your ALSA pulgin
> doesn't exist yet.

Yes excellent point. You are right. Still the relevant plugin is just a
switch: send data here, or send it there. It should not be hard to
write or debug.
 
> BTW: I investigated to write an ALSA plugin for passing audio data to
> polypaudio. Unfortunately the internal ALSA library API isn't flexible
> enough for allowing such things without major changes (i.e. while the
> external ALSA API exports multiple audio file descriptors for the
> application to select() on, you don't have the ability to export more
> than one fd from a single plugin. In addition to that there's no way to
> change during runtime what to select() on: for readability, writability
> or both).

I don't really understand this. Is it really impossible to have a plugin
which either simply forwards data from the master to slave (ie is a 
no-op) or writes it to a socket then just puts null data into the slave?
That sounds like the simplest sort of plugin possible, it'd be pretty
incredible if you could not do even that :(

> I personally think the ALSA API is ugly, much to complicated and way
> over-engineered. However, I'll keep an eye on that. I am willing to
> provide a polypaudio plugin for ALSA. One of the goals of polypaudio is
> compatibility with as many APIs and applications as possible.

OK that would be great! Especially if it was toggleable via an environment
variable (the alsalib api provides support for this). So then in the case
of no network audio needed and a card which supports HW mixing all apps
could bypass a third party server process and write direct to the sound
card (well, possibly via a media framework ...) and if you wanted these
features you could add the Polypaudio plugin to the ALSA pipeline and it
would magically happen for all ALSA supporting apps.

That sort of thing would then sort of resemble what I was wanting, except
would it be necessary to make it Polypaudio specific? Or could you do what
I suggested and just write the audio maybe with timing data via a simple
protocol to a domain socket in /tmp?  Then it would be usable by any
program.

> Another thing that I miss in your design: sample caching. That's really
> something you want to have (in a networked scenario) when your desktop
> generates those "ding" sounds whenever you press a button.

Yes, but this seems to be impossible with all widely deployed sound APIs
as they all work in terms of writing samples to some device not
uploading samples then playing them. GStreamer is only used currently by
GNOME apps, and Polypaudio isn't used directly by any apps at all. So I
think you'd still have to adjust software to add this support.

> There is a video framework: X11.
>
> And there is such a filing system: gnome-vfs. It even includes its own
> daemon the last time I looked. On my debian system I can find it in
> /usr/lib/gnome-vfs2/gnome-vfs-daemon

I was very sloppy with phrasing. By "video driver framework" I meant like
the DRI, and by filing system I meant like ReiserFS. And anyway I've never
been convinced gnome-vfs was a good idea: from a usability POV it means
even though my desktop is full of apps which look the same (all gtk2) some
of them have magic abilities in their file selectors that others don't
apparently at random. It seems a FUSE type approach is more consistent
because it extends the existing OS level VFS API rather than invent a new
one not much uses. Which is the same approach I'd love to see taken here!

> There are at least five important areas where polypaudio is strong in,
> and where it supersedes most other implementations I know:
> 
> - Networked sound. I do believe that this a very important issue. It's
>   the same as with X11, many people don't use the network transparency,
>   but some people do (including me), and are very happy with it and
>   would not like to loose it. That X11 is networked is one of the great
>   strengths i see in it. That's one of the points, where Linux/Unix is
>   better than Windows since years. When I first started to work with
>   Linux (coming from DOS/Windows) that was one of the points where i
>   said "Wow, that's cool", where I was really impressed and started to
>   believe in Linux's superiority. ;-)
> 
>   It seems only natural to me to extend network transparency to audio.

I agree, this seems sensible.

> - Concurrent playback. You can achieve this with ALSA dmix. However,
>   the last time I checked, it was incompatible like hell and failed to
>   work in more places than it did work in other places. I don't like the
>   implementaion of dmix, it seems that the latency management (low
>   latencies and latency measurement) doesn't work very well yet, if at
>   all. These are fixable things, but it seems as if nobody is actively
>   working on fixing them. Plus, ALSA is (currently) Linux specific. (and
>   it seems that at least the FreeBSD people dislike ALSA)

Well, for me dmix 95% works right now with only a few lines added to a
config file. It works with nearly everything and I don't have to
reconfigure every program to use a specific driver. So from an ease of use
perspective it's quite good. That's because most of my stuff already uses
ALSA and support is being improved for other apps all the time, so it
makes sense to grab the audio and do *something* with it at the alsalib
layer. I don't really care if it's dmix or some other server process as
long as it:

a) Works well
b) Works with all my apps

I agree about it being buggy. I do not know how fast dmix is being fixed,
if at all. CVS activity is slow but it is happening.

> - Portability and compatibility. Polypaudio is (probably) much more
>   portable than ALSA. "probably" because it has been ported to FreeBSD
>   (and to some extent MacOSX) only, as of now. The patch for freebsd was
>   only about 20 lines long, and I don't expect much larger patches for
>   other operating systems. Polypaudio is compatible with ESOUND, which
>   is especially important in Gnome. It is easy to write other protocol
>   implementations for polypaudio (like NAS), which would make it even
>   more compatible.

Yes this is a good thing. Though I don't think it should be the primary
factor in any decision: ALSA isn't magic, FreeBSD etc could implement
software mixing and audio redirection too in their OSS implementations.

> - Polypaudio is more than a simple sound server. You can
>   load modules into Polypaudio for teaching it new tricks. Think of
>   polypaudio as an extensible application server for sound on the
>   desktop. Something like the "Apache for Audio". 

These are all very cool ideas! I love the zeroconf one.

> - Polypaudio is actively maintained and has a maintainer who's really
>   interested in getting polypaudio into gnome. ;-)

Yep this is cool, though I share Colins view that GNOME already has an
audio API in place, so Polypaudio would make sense as a desktop
level component rather than developer platform component. 

I'm not sure it should be a distro-level decision though. Way too much
stuff is already pushed down to the distro level and it causes lots of
painful fragmentation and bizarre quirks you have to work around in 3rd
party software. So I'd like to see GNOME and KDE make a unified decision
on this, whatever that is.

thanks -mike