[hotssh/wip/libssh] Add porting discussion

[Colin Walters <walters src gnome org>]

commit 482f468fcb606509eac8d6d909aada1b6b44851f
Author: Colin Walters <walters verbum org>
Date:   Tue Nov 5 17:17:41 2013 -0500

    Add porting discussion

 libgssh/libssh-porting.txt |  112 ++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 112 insertions(+), 0 deletions(-)
---
diff --git a/libgssh/libssh-porting.txt b/libgssh/libssh-porting.txt
new file mode 100644
index 0000000..7b6816b
--- /dev/null
+++ b/libgssh/libssh-porting.txt
@@ -0,0 +1,112 @@
+<sh4rm4> glib integration ?
+<walters> sh4rm4: using libssh with the GLib mainloop
+<sh4rm4> ah
+<sh4rm4> which kind of complications could arise there ?
+<sh4rm4> walters, ?
+<walters> sh4rm4: well, i'm not sure how to do it at the moment
+<sh4rm4> what makes a glib main loop so special ?
+<walters> does your application use libssh asynchronously?
+<walters> or let me rephrase - do you have any applications which need to watch for libssh events *and* 
other events, such as an X11 socket?
+<walters> in the same process
+<sh4rm4> no, it's in blocking mode
+<walters> right
+<gladiac> walters: ssh_get_status()
+<gladiac> well, it is likely we need something better
+<ar1s> nope, ssh_get_status isn't for this
+<ar1s> ssh_get_fd will get the fd from the session. Put it into a handler in your mainloop
+<ar1s> when something happens on the socket, call ...
+<walters> i'm already doing the socket connection outside of libssh, using 
https://developer.gnome.org/gio/unstable/GSocketClient.html then taking the fd out
+<ar1s> ... hmm I'd say ssh_handle_packets but that function is not public
+<walters> ar1s: but how do i know what directions to select for (POLLIN/POLLOUT)?  libssh2 has 
http://www.libssh2.org/libssh2_session_block_directions.html
+<ar1s> create an ssh_poll_ctx and add the session into it. When glib gives you an event, call 
ssh_poll_ctx_dopoll
+<ar1s> walters: that's a weak spot of libssh, I tried to make it integrable into other main loops but never 
actually did it
+<walters> i think what i really want is to add callbacks to libssh that allow my app to take over the actual 
reading/writing to the socket
+<walters> and for it to have an API like _get_block_directions()
+<ar1s> ok that's maybe what the ssh_get_status we discussed with gladiac last week was from
+<ar1s> walters: I think that's not too hard
+<gladiac> ar1s: yeah, I think so too, just a bad name for the function :)
+<ar1s> in fact we would need two getters that get the value in the ssh_pollfd struct itself
+<ar1s> walters: do you have an example of integration glib/any other network lib like libssh that is 
considered best practice ?
+<walters> libdbus was explicitly designed to only be integrated into foreign mainloops
+<walters> it has a synchronous API but you really don't want to use it that way
+<walters> since the whole idea is your app is doing something else *besides* just sending dbus messages
+<walters> of course nowadays GLib comes with its own implementation of the DBus wire protocol so we are 
making less and less use of libdbus over time
+<ar1s> if the code is still relevant I'd have a look
+<walters> https://git.gnome.org/browse/hotssh/tree/libgssh is my wrapper library for libssh2 that is used 
for my app
+<walters> the core concept is basically here: 
https://git.gnome.org/browse/hotssh/tree/libgssh/gssh-connection.c#n155  and here 
https://git.gnome.org/browse/hotssh/tree/libgssh/gssh-connection.c#n316
+<walters> we call some libssh2 API, and if anything returns EAGAIN, we then ask it which directions to use 
for poll()  (making the optimization that we only recreate the GSource which is effectively poll request + fd 
if it's changed)
+<ar1s> how does glib handle some recursive callback stuff ?
+<walters> i'm not sure which callbacks you're referring to
+<ar1s> a problem I had designing something else based on libssh: 2 sessions in the same mainloop. session 1 
receives some data, user callback reads that data and does a synchronous call on session2
+<ar1s> I had to be very careful to lock session1's socket handlers to avoir reentrancy problems
+<walters> yeah, don't do synchronous calls
+<walters> glib has some nice API for that
+<walters> for example my favorite function: 
https://developer.gnome.org/gio/unstable/GOutputStream.html#g-output-stream-splice-async
+<walters> "take this input stream (fd), and write it to the output continually in a loop, then call me back 
when done"
+<walters> and if you want to interrupt it, you can via the GCancellable
+<walters> that's how you can implement a "Cancel" for a copy operation in a UI for example
+<ar1s> oh yes that's awesome
+<ar1s> I wanted to code this in libssh for ages, that would make many tasks much easier
+<ar1s> you mean the "cancel" button that never works ?
+<walters> well, it certainly can work
+<walters> probably a good example how how this plumbing lets me implement my ssh client app is here: 
https://git.gnome.org/browse/hotssh/tree/src/hotssh-tab.c#n155   - I do an asynchronous read, when it 
returns, feed the data to the terminal widget, then start another async read
+<walters> while this going on of course gtk+ can be talking to X, etc.
+<ar1s> watching, there's a lot of good reading here
+<ar1s> except I really hate glib's C-with-objects-but-its-not-c++-we-swear
+<walters> i understand that perspective for sure, there's certainly a lot of boilerplate.  But in full 
disclosure I've been working on the platform for over 10 years, and have been co-maintaining GLib for the 
last 2, so I'm fairly invested in it =)
+<ar1s> I see, so you're just another victim :)
+<walters> hehe =)
+<ar1s> btw the last link you gave me is interesting, very clean way of implementing async IO
+<ar1s> how would you do if you needed to implement flow control on the vte_terminal_feed
+<walters> yeah it's the general pattern we use now in glib
+<ar1s> (let's say the term has a buffer and you don't want to make it too big)
+<walters> ah, that API simply doesn't block
+<walters> it just draws
+<ar1s> you can have APIs that don't block but instead buffer, the results can be worse
+<ar1s> is there a mechanism to tell you that the output buffer is full and you have a callback to tell you 
when the threshold is safe again ?
+<walters> like say i'm trying to take data from a ssh stream and copy it to another socket?
+<walters> or a local Unix pipe?
+<ar1s> yes
+<walters> in that case, I'll have a GOutputStream (for a pipe I can create one via 
g_unix_output_stream_new()), and then I just call that g_output_stream_splice() function above
+<ar1s> the unix pipe would block, when some asynchronous libs would simply grow the output buffer of the 
socket until gigabytes
+<ar1s> ok, that's plumbing I gess
+<ar1s> guess
+<walters> what happens under the hood is that both of those stream types are pollable
+<walters> so the fundamental GMainContext -> poll() operation will read/write to them when they're ready in 
a nonblocking fashion
+<walters> the implementation is here https://git.gnome.org/browse/glib/tree/gio/goutputstream.c#n1737
+<walters> it's a little complex because we have to account for streams which *aren't* pollable (i.e. don't 
have an underlying fd - e.g. GMemoryOutputStream which is just a memory buffer)
+<walters> but the async splice() is really just "do an async read, get some data, do an async write"
+<walters> repeat
+<ar1s> ok
+<ar1s> I'll watch your implementation and see what APIs we have to provide
+<ar1s> it would be cool if we could just have a contrib/ directory containing bindings to popular IO libs 
like glib, qt or libevent
+<walters> fwiw by default on GNU/Linux userspace targets Qt actually uses the GLib loop  (although it has 
its own)
+<walters> Qt obviously doesn't pull in GLib to Windows targets just for the mainloop though
+<ar1s> are you saying that most apps using QT as backend use glib for IO ?
+<ar1s> oh, glib is a dependancy of Qt on linux ?
+<walters> it's configurable, but last I looked, yes
+<walters> the reason is they want to be able to call out to the GTK+ file chooser and print dialog even in 
Qt apps when those apps are run under GNOME
+<walters> anyways =)  I'll look at ssh_session_get_status()
+<ar1s> ssh_session_get_status in master will give you useful info for output (it says if the output buffer 
is empty or not) but not for reading unfortunately
+<ar1s> but that's not really relevant because libssh consumes every input it receives, so you can always 
poll for POLL_IN
+<walters> ah, duh!  That makes sense, I hadn't thought of htat
+<walters> *that
+<ar1s> yes, ssh has an internal windowing mechanism so it's possible to stop the peer to feed us data until 
we can process it
+<ar1s> (and I think libssh2 doesn't do this properly)
+<sh4rm4> anyway, it would be nice if libssh itself stayed glib-free
+<walters> hm, i'm just reading the code and i'm uncertain about the logic for how libssh chooses whether or 
not to pass POLLOUT to poll()
+<walters> sh4rm4: i'm not suggesting revisiting those old discussions, I just want libssh to export a robust 
mainloop integration, I'm OK maintaining my own nice glib-based wrapper library
+<walters> ar1s: am I reading this right that basically it's ssh_socket_nonblocking_flush() that's saying 
"ok, we have a nonempty output buffer, set POLLOUT"?
+<walters> and then in nonblocking mode we'll invoke poll() ourself with a 0 timeout, get the timeout, but as 
a side effect we'll leave the state in the socket status that we wanted POLLOUT, so my mainloop's poll() can 
then invoke it again
+<ar1s> walters: yes. libssh tries do write without passing by the mainloop if the write_wontblock flag is set
+<ar1s> when the write_wontblock flag is not set, POLLOUT is added to the events to monitor
+<ar1s> I don't think that will cause a problem
+<ar1s> if you use these functions: api.libssh.org/master/libssh_8h_source.html (line 540)
+<ar1s> 541 sorry
+<ar1s> http://api.libssh.org/master/group__libssh__session.html#ga3f1b3c59662464eec3649d3d72a40543
+<ar1s> we should probably reword that doc but the purpose is to tell to libssh "I did poll the socket myself 
and you can write/read"
+<walters> right, OK this is basically fine.  I mean we could avoid the extra poll() call I guess if we added 
API to tell libssh that we'll be invoking poll() externally, but it doesn't really matter
+<ar1s> this api will spare you the poll
+<walters> ok, this gives me a good basis to resume porting my app and library, I'll report back how it goes 
=)  this discussion was very helpful!
+<ar1s> but the best would be to call directly the socket callbacks, but imho these callbacks are not in the 
public API
+<ar1s> you're welcome