[mosh-devel] Concerns about mosh's security at the Broad Institute

[Keith Winstein]

Hello Hayden and Simon,

Happy to help if we can. There may be others on the list who are more
familiar with some of these kinds of concerns than me -- hope they
will weigh in. Why don't you let us bat around the response for a bit
before we commit to it. My instincts are that if you really want to
move the needle, perhaps we should get on a conference call and try to
hear them out and understand where they're coming from a bit better.
I'm happy to participate if it would be helpful.

Here would be my initial responses. Our basic perspective is that Mosh
has been out for three and a half years, has been very stable, and has
millions of users, including at major Silicon Valley companies like
Google, Facebook, and Apple. All told, the number of serious security
holes that have been reported in Mosh to date is: zero. This speaks
well of our conservative design and development process, and it
amounts to a better track record than OpenSSL, OpenSSH, or the
underlying SSH and TLS protocols. We have some FAQs on security at
https://mosh.mit.edu that you might find helpful.

On Fri, Aug 7, 2015 at 4:33 PM, Hayden Metsky <hayden at mit.edu> wrote:
> * "Mosh requires opening UDP ports on the Broad perimeter. That makes
>   the Broad network available as a participant in a DDoS performed
>   against external entities, specifically ICMP PORT UNREACHABLE class of
>   attack ("Smack" is one productized version). Even unwitting
>   participation by the Broad in a DDoS could have materially damaging
>   effect upon the Broad as a whole and the outcome of such an event
>   would inevitably involve closing the ports anyway."

Our initial response: We are not sure that we understand the proposed
risk and would need more details to produce a well-informed reply.
Mosh does not require ICMP; your institution can block ICMP and Mosh
will continue to function.

We did not find many references to "Smack." According to
http://www.isi.edu/~mirkovic/bench/attacks.html, Smack attempts to
crash endpoints by sending "random ICMP unreachable packets from
random IP's" in order to bring about "End-point networking crash."

This class of attack is not related to UDP specifically; hosts will
respond with an ICMP port-unreachable message whether the incoming IP
datagram is UDP or TCP. Closing UDP ports will not prevent this
attack. To the extent Broad is worried about phony ICMP unreachable
messages, it will need to mitigate this attack whether the institution
allows UDP or not (perhaps by blocking these ICMP messages, which will
not affect Mosh).

Other large research institutions, such as MIT and Stanford, do not
block UDP and have not suffered this outcome. Many institutions block
all incoming connections (TCP and UDP) by default, but allow them on
an opt-in basis to whitelisted hosts. Stanford does this, and we
understand MIT is moving in this direction -- something similar at
your institution might satisfy all sides.

Mosh does not require opening 1,000 UDP ports; if the institution is
more comfortable with fewer, it could open 10 or 100. The number of
ports is an upper limit on the number of concurrent mosh-servers on
any particular server machine. Mosh uses separate port numbers as a
security feature; different users are isolated from one another by the
kernel. There is no code in Mosh that runs as root or with elevated
privileges. This has allowed Mosh to avoid a class of
privilege-escalation or remote-root security vulnerabilities that have
afflicted various implementations of SSH and HTTPS servers.

> * "Mosh is based on the experimental MIT State Synchronization Protocol
>   (SSP).

Our initial response: SSP is not experimental. It has been widely used
for more than three years. There have been zero vulnerabilities
reported to date. Its security track record is, so far, better than
SSH and TLS.

>   SSP is not know to any commercially available firewalls or
>   perimeter devices, so the use of mosh would not be manageable.

Our initial response: This is becoming the best-practice in secure
protocols. Earlier protocols (like SSH and TLS) are built on top of
unencrypted TCP. More recent protocols (like Mosh's SSP and Google's
QUIC) use fully-encrypted payloads after the UDP header. With SSP and
QUIC, security is end-to-end and there are fewer opportunities for
modification by in-network middleboxes. This increases security and
defeats a class of vulnerabilities that have afflicted SSH and TLS.
Even today, these protocols are vulnerable to phony RST segments,
since they do not authenticate the TCP headers.

However, a Mosh session can always be uniquely tracked by the IP and
port of the server. There is no need to open up UDP to every server at
Broad; the institution could white-list a small set of opt-in servers.
(MIT and Stanford follow a similar approach.)

> Also the first UDP mosh packet is from client to server.

Our initial response: We do not understand the described concern and
may need more information to be able to give a well-informed reply.
Almost all protocols initiate a connection by sending the first packet
from client to server. This also describes all TCP-based protocols
(SSH, TLS, HTTP) -- these also begin with the client sending the first
packet to the server, and the server then replying to the client.

> * "Both Google and Mozilla have rejected mosh."

Our initial response: We do not have direct information about what
these companies think about Mosh today, but we have no reason to think
this information is accurate. We understand that Google performed a
security review in 2012, when Mosh was first released, and concluded
that Mosh was acceptable for their employees to use "at their own
risk" (the category that describes most third-party software in use at
Google). We understand that when Mosh was first released in 2012,
Mozilla took a "wait-and-see" approach on opening up their firewall to
external incoming connections. We have no information about their
current view or practice 3.5 years later. Mosh is now in widespread
use across Silicon Valley, including at Google, Facebook, and Apple.

> * "Mosh sends the session key as an environment variable. User-
>   controlled environment variables are an injection path - their use
>   for sensitive purposes opens security vulnerabilities."

Our initial response: We do not understand the reference to an
"injection path." On the client, the mosh startup script uses an
environment variable to convey the session key to the mosh-client
program. It is cleared from the environment and is not inherited by
user programs in general. Environment variables are confidential and
are not visible to other users on all of the client platforms that
Mosh is distributed for, including OS X and Linux. The mosh-client
audits the key for conformance with the requirements for a key before
proceeding.

To be sure: this is not the first time we have heard concerns about
the use of an environment variable to convey keying material.
Generally these concerns are about the possibility that a future
operating-system vulnerability might accidentally expose environment
variables to other users on the same client machine. We recognize that
OS vendors do make mistakes, and the developers are sensitive to this
concern and, frankly, are sympathetic to it. (OpenBSD did have such an
exposure briefly in 2012 and patched it.)

It may make your institution more comfortable to hear that Mosh plans
to shift to another confidential means of conveying the keying
material in the future. But to be clear, even a hypothetical OS
vulnerability along these lines would result in an exposure to other
users on the same client machine -- not a server-related issue.

> * "Mosh requires server-side modification to /etc/sshd_config to enable
>   'SendEnv LANG LC_*'; this is turned off by default to protect against
>   environment variable injection."

Our response: This is incorrect. Mosh has not required this since Mosh
1.2 (released in April 2012).

> * "The mosh client can, at startup, invoke any program on the server.
>   The program does not have to be the mosh-server. So it is an
>   unrestricted and unmonitored remote execution environment like rsh."

Our initial response: We do not understand the concern and may need to
learn more to prepare a well-informed reply. The mosh startup script
uses "ssh" to log in to the server and run the "mosh-server" process.
It does not use any elevated privileges. It is no more capable of
invoking "any program" on the server than ssh, which is the underlying
tool that is used.

===

Why don't you let my co-developers weigh in on this before we commit
to a response back to your institution. Hope this will be helpful to
you.

Best regards,
Keith