[vte/wip/egmont/bidi: 15/16] unsorted

[Egmont Koblinger <egmontkob src gnome org>]

commit f8c7daf40905a90ec3cdbeb739f3e447cf326e5b
Author: Egmont Koblinger <egmont gmail com>
Date:   Fri Aug 17 18:43:19 2018 +0200

    unsorted

 BIDI-STATUS  |  63 +++++++++++++++
 doc/bidi.txt | 257 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 320 insertions(+)
---
diff --git a/BIDI-STATUS b/BIDI-STATUS
new file mode 100644
index 00000000..4f9b665a
--- /dev/null
+++ b/BIDI-STATUS
@@ -0,0 +1,63 @@
+Done:
+- Implicit (level 1) and Explicit (a small subset only) modes (SM / RM 8).
+- Overall LTR or RTL direction (SCP 1 / 2 or SPD 0 / 3).
+- Possibility to autodetect paragraph direction (DECSET / DECRST 2501).
+- Possibility to make box drawing characters mirrorable (DECSET / DECRST 2500).
+- I-Beam cursor shows the character's resolved directionality when the
+  paragraph has a foreign directionality character.
+- Mouse highlighting, text copying (logical in normal modes, visual in
+  rectangle mode).
+- Mouse reporting.
+- Regex match and explicit hyperlink underlining on hover.
+- VTE_DEBUG=bidi highlights characters with resolved RTL directionality.
+- Arabic shaping using Unicode presentation forms.
+- Test file.
+- Configure flag.
+- Keyboard arrow swapping.
+
+Bugs:
+- The way the modes apply to paragraphs, and what happens when a paragraph
+  is split or two paragraphs are joined is just a first hack, needs to be
+  reviewed, adjusted, fixed properly.
+- SPD should also update all the onscreen lines – really?
+- Or should we drop SPD?
+
+Missing from first release:
+- The entire screen is always invalidated. Have some more fine grained
+  solution (see also vte #26).
+- Probably we'd need to make Erase in Display (Below or All) sequences turn
+  the previous line to hard wrapped, requiring to keep one more line in the
+  ring's writable area.
+- Code cleanup and review, of course.
+
+Planned future improvements:
+- Real shaping (harfbuzz?).
+- Right-align RTL glyphs.
+- Implicit mode level 2 (handling BiDi control characters).
+- Mirror the glyphs that don't have mirrored counterpart.
+- Play with other possibilities for placing the cursor (especially when it's
+  at the end of the logical line).
+- Play with better placement of the preedit box.
+- Remember some lines that are no longer user-accessible as they scroll out,
+  to properly BiDi the still remaining part of that paragraph.
+- Possibility for default RTL directionality?
+- API (for what exactly)?
+
+Not planned at all:
+- Operating on the presentation component (DCSM reset).
+- BiDi in explicit mode by transferring the embedding levels (SDS, SRS...).
+- Escape sequences that modify the emulation behavior (SIMD...).
+
+Useful aliases:
+  alias ltr='echo -ne "\e[1 k"'
+  alias rtl='echo -ne "\e[2 k"'
+  alias implicit='echo -ne "\e[8h"'
+  alias explicit='echo -ne "\e[8l"'
+  alias bidi='echo -ne "\e[8h"'    # same as implicit
+  alias nobidi='echo -ne "\e[8l"'  # same as explicit
+  alias box-mirror='echo -ne "\e[?2500h"'
+  alias box-normal='echo -ne "\e[?2500l"'
+  alias auto='echo -ne "\e[?2501h"'
+  alias noauto='echo -ne "\e[?2501l"'
+  alias kbdswap='echo -ne "\e[?1243h"'
+  alias nokbdswap='echo -ne "\e[?1243l"'
diff --git a/doc/bidi.txt b/doc/bidi.txt
new file mode 100644
index 00000000..a71d35b7
--- /dev/null
+++ b/doc/bidi.txt
@@ -0,0 +1,257 @@
+                         ╔════════════════════════════╗
+═════════════════════════╣ BiDi test – for 80 columns ╠═════════════════════════
+                         ╚════════════════════════════╝
+
+Test cases are structured like this:
+
+Title (bold)
+Optional LTR explanation of the logical order (normal)
+Reference rendering using similarly looking LTR glyphs (green)
+Actual rendering, needs to match the reference rendering (cyan)
+
+All words, except for the explanation of the logical order wherever it’s
+deliberately reversed, should show up in human readable order.
+
+Some test cases are marked as “should look broken”. This means that the
+overall look is not what humans are looking for; still, the actual rendering
+needs to match the reference rendering.
+
+At Arabic shaping tests, runs of connected letters are denoted.
+
+At box tests the reference rendering is upside down, so you should get nice
+squares everywhere.
+
+                              ┌──────────────────┐
+──────────────────────────────┤ 1 – Implicit LTR ├──────────────────────────────
+                              └──────────────────┘
+LTR + RTL
+Hello1 ⸤Hello23⸣ Hello456 Shlm1 ⸤Shlm23⸣ Shlm456
+Hello1 ⸤Hello23⸣ Hello456 456oi7w ⸢23oi7w⸥ 1oi7w
+Hello1 ⸤Hello23⸣ Hello456 שָׁלוֹם1 ⸤שָׁלוֹם23⸣ שָׁלוֹם456
+
+RTL + LTR
+Shlm1 ⸤Shlm23⸣ Shlm456 Hello1 ⸤Hello23⸣ Hello456
+456oi7w ⸢23oi7w⸥ 1oi7w Hello1 ⸤Hello23⸣ Hello456
+שָׁלוֹם1 ⸤שָׁלוֹם23⸣ שָׁלוֹם456 Hello1 ⸤Hello23⸣ Hello456
+
+Paragraph wrapping at foreign word
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor w
+oi7 incididunt ut labore et dolore magna aliqua.
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor שָׁלוֹם incididunt ut 
labore et dolore magna aliqua.
+
+Double wide
+a＜z  n＞x
+a＜z  א＜ת
+
+Arabic shaping
+──╶╶╶─╶ ─── ─────╶ ─ ───╶──╶
+الطرفية – استعمل سطر الأوامر
+
+Box
+┏━┓   a╔═╗z  n┌─┐x   ╭─╮
+┗━┛   a╚═╝z  א┘─└ת   ╰─╯
+
+Box in mirrored mode
+┏━┓   a╔═╗z  n┌─┐x   ╭─╮[?2500h
+┗━┛   a╚═╝z  א└─┘ת   ╰─╯[?2500l
+
+                              ┌──────────────────┐
+──────────────────────────────┤ 2 – Implicit RTL ├──────────────────────────────
+                              └──────────────────┘
+LTR + RTL
+Hello1 ⸤Hello23⸣ Hello456 Shlm1 ⸤Shlm23⸣ Shlm456
+                                456oi7w ⸢23oi7w⸥ 1oi7w Hello1 ⸤Hello23⸣ Hello456
+[2 kHello1 ⸤Hello23⸣ Hello456 שָׁלוֹם1 ⸤שָׁלוֹם23⸣ שָׁלוֹם456[1 k
+
+RTL + LTR
+Shlm1 ⸤Shlm23⸣ Shlm456 Hello1 ⸤Hello23⸣ Hello456
+                                Hello1 ⸤Hello23⸣ Hello456 456oi7w ⸢23oi7w⸥ 1oi7w
+[2 kשָׁלוֹם1 ⸤שָׁלוֹם23⸣ שָׁלוֹם456 Hello1 ⸤Hello23⸣ Hello456[1 k
+
+Paragraph with wrong direction (should look broken)
+w Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
+                                .incididunt ut labore et dolore magna aliqua oi7
+[2 kLorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor שָׁלוֹם incididunt 
ut labore et dolore magna aliqua.[1 k
+
+Double wide
+                                                                      a＜z  n＞x
+[2 kא＜ת  a＜z[1 k
+
+Arabic shaping
+                                                    ──╶╶╶─╶ ─── ─────╶ ─ ───╶──╶
+[2 kالطرفية – استعمل سطر الأوامر[1 k
+
+Old Hungarian Rovásírás
+(font at https://github.com/OldHungarian/old-hungarian-font)
+                                                         ΛᛩHTΛᛩMↄH ↄ4TH4Λↄ¤ᛝ +ΛↃ
+[2 k𐲛𐳖𐳇 𐲏𐳪𐳙𐳍𐳀𐳢𐳐𐳀𐳙 𐲢𐳛𐳮𐳁𐳤𐳑𐳢𐳁𐳤[1 k
+
+Box
+                                                        ╭─╮   a┌─┐z  n╔═╗x   ┏━┓[2 k
+┛━┗   א╝═╚ת  a└─┘z   ╯─╰[1 k
+
+Box in mirrored mode
+                                                        ╭─╮   a┌─┐z  n╔═╗x   ┏━┓[2 k[?2500h
+┗━┛   א╚═╝ת  a└─┘z   ╰─╯[?2500l[1 k
+
+                      ┌──────────────────────────────────┐
+──────────────────────┤ 3 – Implicit auto (LTR fallback) ├──────────────────────
+                      └──────────────────────────────────┘
+LTR + RTL
+Hello1 ⸤Hello23⸣ Hello456 Shlm1 ⸤Shlm23⸣ Shlm456
+Hello1 ⸤Hello23⸣ Hello456 456oi7w ⸢23oi7w⸥ 1oi7w
+[?2501hHello1 ⸤Hello23⸣ Hello456 שָׁלוֹם1 ⸤שָׁלוֹם23⸣ שָׁלוֹם456[?2501l
+
+RTL + LTR
+Shlm1 ⸤Shlm23⸣ Shlm456 Hello1 ⸤Hello23⸣ Hello456
+                                Hello1 ⸤Hello23⸣ Hello456 456oi7w ⸢23oi7w⸥ 1oi7w
+[?2501hשָׁלוֹם1 ⸤שָׁלוֹם23⸣ שָׁלוֹם456 Hello1 ⸤Hello23⸣ Hello456[?2501l
+
+Paragraph wrapping at foreign word
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor w
+oi7 incididunt ut labore et dolore magna aliqua.
+[?2501hLorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor שָׁלוֹם 
incididunt ut labore et dolore magna aliqua.[?2501l
+
+Arabic shaping
+                                                    ──╶╶╶─╶ ─── ─────╶ ─ ───╶──╶
+[?2501hالطرفية – استعمل سطر الأوامر[?2501l
+
+Box
+┏━┓   a╔═╗z  n┌─┐x   ╭─╮[?2501h
+┗━┛   a╚═╝z  א┘─└ת   ╰─╯[?2501l
+
+Box in mirrored mode
+┏━┓   a╔═╗z  n┌─┐x   ╭─╮[?2501h[?2500h
+┗━┛   a╚═╝z  א└─┘ת   ╰─╯[?2500l[?2501l
+
+                      ┌──────────────────────────────────┐
+──────────────────────┤ 4 – Implicit auto (RTL fallback) ├──────────────────────
+                      └──────────────────────────────────┘
+LTR + RTL
+Hello1 ⸤Hello23⸣ Hello456 Shlm1 ⸤Shlm23⸣ Shlm456
+Hello1 ⸤Hello23⸣ Hello456 456oi7w ⸢23oi7w⸥ 1oi7w
+[2 k[?2501hHello1 ⸤Hello23⸣ Hello456 שָׁלוֹם1 ⸤שָׁלוֹם23⸣ שָׁלוֹם456[?2501l[1 k
+
+RTL + LTR
+Shlm1 ⸤Shlm23⸣ Shlm456 Hello1 ⸤Hello23⸣ Hello456
+                                Hello1 ⸤Hello23⸣ Hello456 456oi7w ⸢23oi7w⸥ 1oi7w
+[2 k[?2501hשָׁלוֹם1 ⸤שָׁלוֹם23⸣ שָׁלוֹם456 Hello1 ⸤Hello23⸣ Hello456[?2501l[1 k
+
+Paragraph wrapping at foreign word
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor w
+oi7 incididunt ut labore et dolore magna aliqua.
+[2 k[?2501hLorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor שָׁלוֹם 
incididunt ut labore et dolore magna aliqua.[?2501l[1 k
+
+Arabic shaping
+                                                    ──╶╶╶─╶ ─── ─────╶ ─ ───╶──╶
+[2 k[?2501hالطرفية – استعمل سطر الأوامر[?2501l[1 k
+
+Box
+                                                        ╭─╮   a┌─┐z  n╔═╗x   ┏━┓[2 k[?2501h
+┛━┗   א╝═╚ת  a└─┘z   ╯─╰[?2501l[1 k
+
+Box in mirrored mode
+                                                        ╭─╮   a┌─┐z  n╔═╗x   ┏━┓[2 k[?2501h[?2500h
+┗━┛   א╚═╝ת  a└─┘z   ╰─╯[?2500l[?2501l[1 k
+
+                              ┌──────────────────┐
+──────────────────────────────┤ 5 – Explicit LTR ├──────────────────────────────
+                              └──────────────────┘
+LTR + fake RTL
+⸤Hello⸣ ⸤mlhS⸣
+⸤Hello⸣ ⸤oi7w⸣
+⸤Hello⸣ ⸤םוֹלשָׁ⸣
+
+Fake RTL + RTL
+⸤mlhS⸣ ⸤Hello⸣
+⸤oi7w⸣ ⸤Hello⸣
+⸤םוֹלשָׁ⸣ ⸤Hello⸣
+
+Arabic shaping
+──╶╶╶─╶ ─── ─────╶ ─ ───╶──╶
+رماوألا رطس لمعتسا – ةيفرطلا
+
+Box
+┏━┓   a╔═╗z  n┌─┐x   ╭─╮
+┗━┛   a╚═╝z  ת└─┘א   ╰─╯
+
+Box in mirrored mode
+┏━┓   a╔═╗z  n┌─┐x   ╭─╮[?2500h
+┗━┛   a╚═╝z  ת└─┘א   ╰─╯[?2500l
+
+                              ┌──────────────────┐
+──────────────────────────────┤ 6 – Explicit RTL ├──────────────────────────────
+                              └──────────────────┘
+Fake LTR + RTL
+⸤olleH⸣ ⸤Shlm⸣
+                                                                  ⸢oi7w⸥ ⸢Hello⸥
+[2 k⸤olleH⸣ ⸤שָׁלוֹם⸣[1 k
+
+RTL + fake LTR
+⸤Shlm⸣ ⸤olleH⸣
+                                                                  ⸢Hello⸥ ⸢oi7w⸥
+[2 k⸤שָׁלוֹם⸣ ⸤olleH⸣[1 k
+
+Fullwidth characters with underlines
+                        ［Ｌｏｒｅｍ  ｉｐｓｕｍ  ｄｏｌｏｒ  ｓｉｔ  ａｍｅｔ）
+[2 k（ｔｅｍａ  ｔｉｓ  ｒｏｌｏｄ  ｍｕｓｐｉ  ｍｅｒｏＬ］[1 k
+
+Arabic shaping
+                                                    ──╶╶╶─╶ ─── ─────╶ ─ ───╶──╶
+[2 kالطرفية – استعمل سطر الأوامر[1 k
+
+Box
+                                                        ╭─╮   a┌─┐z  n╔═╗x   ┏━┓[2 k
+┛━┗   א╝═╚ת  z┘─└a   ╯─╰[1 k
+
+Box in mirrored mode
+                                                        ╭─╮   a┌─┐z  n╔═╗x   ┏━┓[2 k[?2500h
+┗━┛   א╚═╝ת  z└─┘a   ╰─╯[?2500l[1 k
+
+                               ┌────────────────┐
+───────────────────────────────┤ 7 – Misc tests ├───────────────────────────────
+                               └────────────────┘
+Numbers inside RTL, across linebreak
+jumps Shalom 123 456 789 Shalom!
+The quick brown fox jumps over the lazy dog The quick brown fox jumps 4 123 oi7w
+oi7w 789 56!
+The quick brown fox jumps over the lazy dog The quick brown fox jumps שָׁלוֹם 123 456 789 שָׁלוֹם!
+
+Mirroring across linebreak
+jumps Shalom <[<[<[<[ Shalom!
+The quick brown fox jumps over the lazy dog The quick brown fox jumps >]>]> oi7w
+oi7w ]>]!
+The quick brown fox jumps over the lazy dog The quick brown fox jumps שָׁלוֹם <[<[<[<[ שָׁלוֹם!
+
+Attributes
+Hello1 Shlm23 Shlm456 Hello7890
+Hello1 456oi7w 23oi7w Hello7890
+Hello1 שָׁלוֹם23 שָׁלוֹם456 Hello7890
+
+Link autodetection, overall LTR
+https!//he.wikipedia.org/wiki/Front_page
+https!//he.wikipedia.org/wiki/'wxá’£_tiny
+https://he.wikipedia.org/wiki/עמוד_ראשי
+
+Link autodetection, overall LTR (should look broken)
+https!//he.wikipedia.org/wiki/Front_page Shalom
+https!//he.wikipedia.org/wiki/oi7w 'wxá’£_tiny
+https://he.wikipedia.org/wiki/עמוד_ראשי שָׁלוֹם
+
+Link autodetection, overall RTL
+Shalom1 http!//example.com/test Shalom23
+                                            23oi7w http!//example.com/test 1oi7w
+[2 kשָׁלוֹם1 http://example.com/test שָׁלוֹם23[1 k
+
+Link autodetection, overall RTL (should look broken)
+Shalom1 http!//example.com/test/ Shalom23
+                                           23oi7w /http!//example.com/test 1oi7w
+[2 kשָׁלוֹם1 http://example.com/test/ שָׁלוֹם23[1 k
+
+Explicit hyperlinks
+Hello1 Shlm23 Shlm456 Hello7890
+Hello1 456oi7w 23oi7w Hello7890
+Hello1 456oi7w 23oi7w Hello7890
+]8;;http://example.com/hello123\Hello1 שָׁלוֹם23]8;;\ 
]8;;http://example.com/hello4567890\שָׁלוֹם456 Hello7890]8;;\
+
+────────────────────────────────────────────────────────────────────────────────