[mosh-users] Alternate character set support (line grpahics) with terminfo

[Vincent Lefevre]

On 2014-03-20 14:21:47 +0100, Keith Winstein wrote:
> Unfortunately, terminfo has not been updated to support the difference
> between byte-based and UTF-8-based terminals. An ANSI terminal whose state
> machine operates on bytes is fundamentally different from an ANSI terminal
> whose state machine operates on Unicode (ISO 10646) scalar values after
> decoding UTF-8, but there is no way to indicate this difference in
> terminfo. Instead, we indicate this with the locale-related environment
> variables, which causes its own set of problems.

Well, not just terminfo. There are old programs written without
knowing anything about the character set. What I mean is that
terminfo could have been updated to take into account UTF-8-based
terminals, but this would have broken these old programs.

> Some terminal authorities (including the authors of PuTTY, the Linux
> Unicode/UTF-8 FAQ, and Mosh) take the position that an ISO 10646 terminal
> (i.e. a Unicode one) should NOT support ISO 2022 escape sequences -- the
> "alternate character set" you're talking about -- because all of those
> coded characters can now be accessed directly without locking shifts.
> Markus Kuhn explains why in the FAQ. We think applications are given fair
> warning of this by the C locale system -- if you run nl_langinfo( CODESET )
> in the implementation-defined locale, you will get "UTF-8". That means (in
> our view) that ISO 2022 is not allowed.
> 
> Otherwise (including the authors of xterm and VTE and Terminal.app) took
> the opposite position -- that the terminal should decode UTF-8, interpret
> the result as ISO 10646 code positions, but should also respect ISO 2022
> locking shifts. These authors believe that a CODESET of UTF-8 also
> implicitly includes ISO 2022 shift sequences.

This is also the case with the Linux virtual terminals, rxvt and
GNOME Terminal (I've just tried). Actually I haven't found a terminal
that behaves in another way (AFAIK, mosh is the first one).

> Mosh has chosen to go with the Linux Unicode FAQ and PuTTY on this one --
> we think locking shifts are bad and that ISO 10646 is better than ISO 2022.

Thanks for the explanations. I think that it would have been better to
make the behavior configurable. Or alternatively, use a terminal name
for TERM that doesn't claim to support ACS. For instance, "mosh". The
corresponding terminfo data could be installed at the same time as
mosh software (in particular the server). IMHO, this would be much
cleaner than the current status.

> If you have an application that speaks ISO 2022, you can certainly run an
> ISO 2022 interpreter in front of Mosh yourself. For example, luit or screen.

Indeed "mosh localhost screen" works. But not with luit, probably
because:

$ luit -list
[...]
  UTF-8 (non-ISO-2022 encoding)
[...]

and there is no UTF-8 version with ISO 2022 support.

> > I *never* get stuck in hieroglyphs with any terminal. So, this is
> > pointless.
> 
> You will if you run something like: echo -e "\033(0"
> 
> (Or if you abort a program when it was in the middle of writing in the ACS.)

No, I don't, thanks to my superior shell. :)

ypig:~> echo -e "\033(0"; echo mq                     <14:52:08

└─
ypig:~>                                               <14:52:10

See, the second prompt is fine.

This is not specific to the ACS. For instance, you'll have problems
with "tput setab 0; tput setaf 0" if the shell doesn't clean up the
terminal status. And here, mosh alone doesn't avoid the problem!

Now, it's true that a program that uses Unicode will behave in a
better way than with ACS under some conditions, e.g. in case the
output is piped while the ACS is still active. But these are very
specific cases, and it's also better to be able to support old
programs that haven't been rewritten yet.

> In theory I think we are willing to look at a pull request that makes this
> an option, and this functionality could be a server-only change. But I will
> be up-front that I am pretty skeptical we need to put this into Mosh, given
> that this has been our behavior for two years, the number of complaints
> we've gotten has been very low, the benefit of avoiding locking shifts is
> clear, and affected users can fix the problem today by running luit or
> screen.

FYI, I've tried mosh since yesterday. This means that I've quickly
found the problem.

Running luit doesn't solve the problem. And if mosh supports mouse
scrollback in the future, that would be a reason not to use screen
(in which one can't scrollback with the mouse).

> I hope this explains our thinking.

Yes, thanks.

-- 
Vincent Lefèvre <vincent at vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)