[mosh-devel] Concerns about mosh's security at the Broad Institute

[john hood]

On 8/7/15 7:33 PM, Hayden Metsky wrote:
> Hi,
> 
> A friend (Simon Ye, CC'd) and I are PhD students at MIT who are
> currently working at the Broad Institute (a genomics center) using
> their computing infrastructure. The standard workflow is to use ssh to
> connect to a login server and then do work on that. This login server
> is accessible from anywhere (including outside the VPN) and password
> is the sole method of authentication.

When you say "login server", what exactly do you mean?  Is this a
bastion host that you can ssh to, and then connect to internal hosts
from?  (That's what I'll assume here.)

> We would like the Broad Institute's IT staff to unblock UDP ports
> 60000-61000 so that we can use mosh to connect to the login servers.
> (We've both used mosh elsewhere and loved it.) Unfortunately we've
> encountered resistance from the security team, who have cited a number
> of security concerns. We've responded to many of these concerns but
> have made little progress. Of course, the security team's view is
> going to be biased in a conservative direction.
> 
> Since networking and computer security are well outside our area of
> expertise, we think it would be helpful to pass the concerns along to
> mosh's developers. We would love to hear your thoughts; obviously we
> would welcome rebuttals, but we would also be happy to hear if you
> think the concerns have validity.
> 
> Below I am going to quote six concerns verbatim from a document
> written by the security team. Under some of these concerns I'll
> include a short note from myself giving our thoughts.
> 
> * "Mosh requires opening UDP ports on the Broad perimeter. That makes
>   the Broad network available as a participant in a DDoS performed
>   against external entities, specifically ICMP PORT UNREACHABLE class of
>   attack ("Smack" is one productized version). Even unwitting
>   participation by the Broad in a DDoS could have materially damaging
>   effect upon the Broad as a whole and the outcome of such an event
>   would inevitably involve closing the ports anyway."
>    - This is the team's primary concern. Our initial thought is that it
>      is difficult to imagine a single machine (server or router) having
>      a meaningful impact in a DDoS amplification, but perhaps this is
>      mistaken. Or maybe it is possible to use mosh with restrictions on
>      outgoing traffic that would avoid the Broad from participating in
>      this kind of attack? Even though this is not specific to mosh, we
>      would very much appreciate your thoughts.

Is the concern here that servers running Mosh could be used as DDoS
reflectors, or that opening UDP ports would allow Broad-internal hosts
to be useful for DDoS bots wanting to generate UDP?

For the first issue, the problem here is that if Mosh isn't running on a
specific port in this range, the host OS will respond with an ICMP
UNREACHABLE.  This is a valid concern for Mosh in its current state, for
organizations that are unable/unwilling to disable ICMP replies on
internal hosts, or filter them at the firewall.  Mosh's need for a range
of ports doesn't help matters.

However, Mosh itself doesn't rely on this at all.  The Mosh client pays
no attention to ICMP, and the server will only reply to packets that
successfully pass decryption and authentication.  So your security team
is welcome to filter Mosh-related ICMP with abandon.

If this login server is a bastion host as described above, your security
team should have no real issue with turning off ICMP PORT UNREACHABLE
replies on Mosh's port range.

For the second concern, since the Mosh server never initiates UDP flows,
a firewall is free to drop outgoing UDP packets that don't hash to an
existing, client-initiated flow.

There's discussion on Github on how Mosh might multiplex sessions on a
single UDP port, but nobody's coughed up any code yet.

> * "Mosh is based on the experimental MIT State Synchronization Protocol
>   (SSP). SSP is not know to any commercially available firewalls or
>   perimeter devices, so the use of mosh would not be manageable. Also
>   the first UDP mosh packet is from client to server. That underscores
>   the fact that there is no way for a firewall to have any control of
>   state."

As far as SSP being new...you have to start somewhere.

Mosh is a good example of a UDP protocol with very little identifying
info in the packets; the packets are composed of a 64-bit sequence
number, two 16-bit timestamps, and encrypted data.  It will be difficult
for middleboxes to identify, unfortunately.  We can't change that
without breaking compatibility.

Client-initiated flows?  Mosh is intended for use on servers that are
reachable on the public Internet.  Clients must initiate UDP flows for
its roaming capability to work.

> * "Both Google and Mozilla have rejected mosh."
>    - I don't know about Mozilla, but my understanding is that this is
>      incorrect with regard to Google. Even if Google only allows mosh
>      access from within a VPN and we're asking the Broad to allow access
>      from outside (ssh is allowed from outside), it is difficult to see
>      how this is an argument against mosh.

Different organizations have different security models.  Is the Broad's
the same as Google's or Mozilla's?

> * "Mosh sends the session key as an environment variable. User-
>   controlled environment variables are an injection path - their use
>   for sensitive purposes opens security vulnerabilities."

There are differing opinions on the Mosh team, but I think this is a
significant issue (for another reason-- I had an OS install that exposed
environment variables); it's not yet resolved.

> * "Mosh requires server-side modification to /etc/sshd_config to enable
>   'SendEnv LANG LC_*'; this is turned off by default to protect against
>   environment variable injection."
>    - There's a lot that's misleading or incorrect about this. First, I
>      think they mean 'AcceptEnv' rather than 'SendEnv'. On many OS's,
>      it is my understanding that this is actually turned *on* by default.
>      When I run 'ssh broad locale', I receive back my laptop's locale
>      (even after changing it), indicating that the server is accepting
>      my LANG/LC_* env variables. So I suspect this is in fact turned on
>      (I cannot read the file). Regardless, we're able to install mosh
>      and use it internally among nodes, which we believe suggests that
>      no changes need to be made (besides unblocking ports).

This is quite incorrect, though the correct details may not make your
security team any happier.  Mosh currently passes LC_* variables
out-of-band via the mosh-server command line and sets them itself for
its own use and the user login session.  It doesn't rely on SSH's
SendEnv or AcceptEnv.  Mosh could implement a whitelist to restrict the
variables it will set to LANG and LC_*, though.

> * "The mosh client can, at startup, invoke any program on the server.
>   The program does not have to be the mosh-server. So it is an
>   unrestricted and unmonitored remote execution environment like rsh."
>    - We're a bit confused on this point. The same (we believe) is true
>      of ssh, and since mosh authenticates via ssh we don't quite see how
>      this might be held against mosh given that ssh is already used.

Correct.  Since the mosh wrapper invokes mosh-server via an SSH session,
it's exactly as manageable as any other SSH usage.

regards,

  --jh