[gnome-remote-desktop] session-rdp: Fix invalid colour depth, when codecs or GFX is advertised

[Jonas Ådahl <jadahl src gnome org>]

commit c6fec5aa04c3d7532b53467a69501385eaf83225
Author: Pascal Nowack <Pascal Nowack gmx de>
Date:   Wed Aug 31 11:22:26 2022 +0200

    session-rdp: Fix invalid colour depth, when codecs or GFX is advertised
    
    In RDP, all codecs and the graphics pipeline require the peer to
    support the colour depth 32 bits per pixel.
    While advertising the support for codecs and the graphics pipeline, the
    client can still prefer a specific colour depth, although it is
    nonsense doing it.
    This preferred colour depth setting exists for ancient servers (Win XP
    era), but mstsc in recent MS Windows versions still allows a user to
    set this setting.
    The supported colour depths and support for the graphics pipeline are
    submitted by the client in the Client Core Data, which is read before
    the Capabilities() callback is called.
    The colour depth used in the session is written by the server side in
    the Bitmap Capability Set, which is sent after the Capabilities()
    callback is called.
    
    When reading the Client Core Data of a client, FreeRDP overrides the
    ColorDepth setting, when the preferred colour depth of the client is
    lower than the one on the server side.
    While FreeRDP clients always automatically set the preferred colour
    depth to 32, when any codec or the graphics pipeline is advertised,
    mstsc does not.
    This results into mstsc dropping the connection, if the user overrides
    the preferred colour depth setting.
    
    To fix this issue, check in the Capabilities() callback, whether any
    codec or the graphics pipeline was advertised.
    In such case, set the colour depth back to 32. This will result into
    writing the correct colour depth value in the Bitmap Capability Set and
    mstsc not dropping the connection, when the user selected a different
    preferred colour depth in mstsc.
    MS Windows RDS does the same, when the client supports the graphics
    pipeline.
    
    Fixes: https://gitlab.gnome.org/GNOME/gnome-remote-desktop/-/issues/118

 src/grd-session-rdp.c | 7 +++++++
 1 file changed, 7 insertions(+)
---
diff --git a/src/grd-session-rdp.c b/src/grd-session-rdp.c
index 6b2cb5d3..891c95e3 100644
--- a/src/grd-session-rdp.c
+++ b/src/grd-session-rdp.c
@@ -1749,6 +1749,13 @@ rdp_peer_capabilities (freerdp_peer *peer)
       session_rdp->monitor_config->monitor_count = 1;
     }
 
+  if (rdp_settings->SupportGraphicsPipeline || rdp_settings->RemoteFxCodec ||
+      rdp_settings->NSCodec)
+    {
+      g_debug ("[RDP] Fixing invalid colour depth set by client");
+      rdp_settings->ColorDepth = 32;
+    }
+
   switch (rdp_settings->ColorDepth)
     {
     case 32: