[mosh-devel] Mosh on Kuberbnetes

[Keith Winstein]

Hi Thomas,

Hmm, let me try to see if I can say it better. For an outgoing UDP datagram
(from a containerized mosh-server to a mosh-client), the mosh-server will
be sending the datagram from some private IP address (the container's IP
address, e.g. 10.0.0.5) and a source port that is probably going to be the
same across all containers (like 60001), since each container will probably
only be running one mosh-server.

The proxy will want to rewrite the source IP:port so that the source IP
address is the proxy's routable Internet address and the port number is
whatever unique port it assigned and sent to the client in the original
MOSH CONNECT message that the client saw.

-Keith

On Sat, Jul 7, 2018 at 8:32 PM, Thomas Buckley-Houston <tom at tombh.co.uk>
wrote:

> Thanks so much for this idea, I really think it's the one, simple and
> scalable.
>
> I haven't tried but I'm pretty sure the mosh-server's "MOSH CONNECT"
> can be wrapped in plain BASH. I'm already in control of the SSH
> connection as I'm using my own `ForceCommand` script.
>
> Also I can still use this method with extra Round-Robin balanced IP
> addresses giving me multiple sets of 65,000 ports.
>
> The only thing I don't understand is why the outgoing UDP datagram has
> to rewrite the container's source IP. Isn't the original MOSH CONNECT
> IP:port the canonical reference?
>
> On 1 July 2018 at 12:54, Keith Winstein <keithw at cs.stanford.edu> wrote:
> > How about a semi-smart (but mostly Mosh-oblivious) server-side proxy/NAT
> > that works like this:
> >
> > - The proxy service has one public IP address and like 65,000 available
> UDP
> > ports.
> > - The proxy service can itself be redundant with failover...
> > - When a user wants to open a new Mosh connection, they Mosh to a single
> > hostname (which resolves to the IP address of the proxy service).
> > - Your code allocates the necessary container, etc., and also allocates a
> > unique UDP port on the proxy.
> > - Your code runs the new mosh-server process in the target container.
> > - The proxy intercepts the mosh-server's "MOSH CONNECT <port> <key>"
> > message, replacing the port number with the unique public-facing UDP port
> > (and remembering the container's IP address and the original port
> number).
> > - When the proxy receives an incoming UDP datagram destined to a
> particular
> > UDP port, it forwards it to the appropriate container at its IP address
> and
> > at the original port number. It *preserves* the source IP and port of the
> > datagram when forwarding.
> > - When the container wants to send an outgoing UDP datagram, it sends it
> > normally (to whatever IP:port is associated with the client), except the
> > containers are not directly connected to the Internet; they use the
> > proxy/NAT as their next-hop router.
> > - For the outgoing UDP datagram, the proxy/NAT rewrites the container's
> > source IP:port to its own IP and the public port number.
> >
> > I think this will allow you to serve like 65,000 separate mosh
> connections
> > from a single public IP address...
> >
> > The added latency in forwarding a datagram is probably <1 ms, and you
> don't
> > really have to change anything about Mosh itself or its internals.
> >
> > Unfortunately there are no unencrypted identifying marks to a Mosh
> > connection, except the incrementing sequence numbers (which start at 0
> for
> > every connection).
> >
> > -Keith
> >
> > On Fri, Jun 29, 2018 at 12:17 AM, Thomas Buckley-Houston <
> tom at tombh.co.uk>
> > wrote:
> >>
> >> Hey Keith, John, everyone,
> >>
> >> Yeah the more this is looking like a quite a big hurdle. Especially
> >> your point Keith about roaming IPs (which I'd forgotten), it's a
> >> central feature of Mosh I don't want to lose.
> >>
> >> So the only 2 options seems to be exposing multiple IPs for Round
> >> Robin (or other smart DNS routing) or writing a new Mosh proxy that
> >> already has knowledge of the existing keys. Both seem like quite a
> >> challenge. Round Robin DNS seems more approachable and I can imagine
> >> integrating it with the Google Cloud DNS API I'm already using, but I
> >> just wonder how expensive Google (or anyone for that matter) will make
> >> thousands of static IP addresses? Apart from me having to learn Mosh
> >> internals, one difficulty that strikes me about a Mosh proxy is that
> >> it might introduce a non-trivial delay to each datagram arriving?
> >> Though surely only ever in the order of a handful of milliseconds I
> >> suppose.
> >>
> >> Are there not any other identifying marks to a datagram, I don't know
> >> much about low level networking, but maybe something like a MAC
> >> address for example?
> >>
> >> Thanks,
> >> Tom
> >>
> >> On 27 June 2018 at 04:50, Keith Winstein <keithw at cs.stanford.edu>
> wrote:
> >> > Hi Thomas,
> >> >
> >> > Glad you could provoke a very interesting discussion! But I'm still
> >> > confused
> >> > -- how is "sticky IP-based routing" going to work after the client
> roams
> >> > to
> >> > a new IP address (or to a new UDP source port)? When your system seems
> >> > an
> >> > incoming UDP datagram from a previously unseen source IP:port, how
> does
> >> > it
> >> > know which mosh-server (on which server machine) to send it to?
> >> >
> >> > With off-the-shelf Mosh, you basically need a load-balancing strategy
> >> > that
> >> > allows a destination IP:port to uniquely identify a particular
> >> > mosh-server.
> >> > You can do this with multiple DNS A/AAAA records (where the client
> picks
> >> > the
> >> > winning one -- maybe you permute the list), or with a smart DNS server
> >> > that
> >> > serves *one* A or AAAA record to the client at the time of resolution
> >> > (like
> >> > a CDN would use).
> >> >
> >> > Instead of using the mosh wrapper script, you could have your users
> use
> >> > some
> >> > other scheme to figure out the IP:port of the server, but the point is
> >> > that
> >> > once you launch the mosh-client, it's going to keep sending datagrams
> to
> >> > the
> >> > IP:port of the mosh-server, and those datagrams need to get to the
> same
> >> > mosh-server process even if the client roams to a different
> >> > publicly-visible
> >> > IP address or port.
> >> >
> >> > You could imagine writing a very smart mosh proxy that has the keys to
> >> > all
> >> > the sessions and can figure out (for an incoming datagram coming from
> an
> >> > unknown source IP:port) which session it actually belongs to, and then
> >> > makes
> >> > a sticky mapping and routes it to the proper mosh-server. But I don't
> >> > think
> >> > anybody has actually done this yet and of course there's a challenge
> in
> >> > making this reliable/replicated.
> >> >
> >> > -Keith
> >> >
> >> > On Mon, Jun 25, 2018 at 3:10 AM, Thomas Buckley-Houston
> >> > <tom at tombh.co.uk>
> >> > wrote:
> >> >>
> >> >> Thanks so much for the clarification.
> >> >>
> >> >> > UDP is connectionless
> >> >>
> >> >> That's the key here. So I have no choice but to use sticky IP-based
> >> >> routing. Round-robin DNS isn't an option I don't think, because I
> hope
> >> >> one day to be able to scale to thousands of servers.
> >> >>
> >> >> And thanks so much for the heads up about my DNSSEC records. I've
> sent
> >> >> a request for them to be deleted. I'd added them and some SSHFP
> >> >> records to explore automatically passing the StrictHostKey warning.
> >> >> But it's not entirely straight forward. Even with correct DNS records
> >> >> the SSH user still has to have VerifyHostKeyDNS enabled, which as I
> >> >> understand most people don't. And then on top of that my DNS provider
> >> >> (DNSSimple) automatically rotate the keys every 3 months, which means
> >> >> I have to manually send a request to my registrars by email to update
> >> >> the DNSSEC records. Is it all worth it do you think?
> >> >>
> >> >> On 24 June 2018 at 13:36, Anders Kaseorg <andersk at mit.edu> wrote:
> >> >> > You may have a misunderstanding about how a Mosh session is set up.
> >> >> > The
> >> >> > mosh script launches a mosh-server on the remote system via SSH;
> >> >> > mosh-server picks a port number and a random encryption key, and
> >> >> > writes
> >> >> > them to stdout, where they go back over SSH to the mosh script;
> then
> >> >> > the
> >> >> > mosh script launches mosh-client passing the IP address, port
> number,
> >> >> > and
> >> >> > encryption key.  The newly launched mosh-client and mosh-server
> >> >> > processes
> >> >> > exchange UDP packets encrypted with the shared key; communication
> is
> >> >> > successful if the packets can be decrypted.
> >> >> >
> >> >> > There’s no separate “key checking” step to be disabled.  And it
> >> >> > doesn’t
> >> >> > make sense to “refuse more than 1 connection on the same port”,
> both
> >> >> > because UDP is connectionless, and because a new mosh-server is
> >> >> > launched
> >> >> > on a new port for each Mosh session (it is not a daemon like sshd).
> >> >> >
> >> >> > The easiest way to put Mosh servers behind a load balancer is with
> >> >> > round-robin DNS where a single hostname resolves to many addresses,
> >> >> > or
> >> >> > to
> >> >> > different addresses for different clients and/or at different
> times.
> >> >> > We’ve already gone out of our way to make the mosh script resolve
> the
> >> >> > hostname only once and use the same address for the SSH connection
> >> >> > and
> >> >> > the
> >> >> > UDP packets, because that’s needed for MIT’s athena.dialup.mit.edu
> >> >> > pool.
> >> >> >
> >> >> > If that’s not an option and you really need all connections to go
> >> >> > through
> >> >> > a single load balancer address, you could try wrapping mosh-server
> in
> >> >> > a
> >> >> > script that passes different disjoint port ranges (-p) on different
> >> >> > backends, and forwarding those ranges to the corresponding backends
> >> >> > from
> >> >> > the load balancer.
> >> >> >
> >> >> > Unrelatedly, brow.sh doesn’t resolve with DNSSEC-enabled resolvers
> >> >> > like
> >> >> > 1.1.1.1 or 8.8.8.8, seemingly due to some problem with the DS
> records
> >> >> > set
> >> >> > with the registrar: https://dnssec-debugger.
> verisignlabs.com/brow.sh.
> >> >> >
> >> >> > Anders
> >> >>
> >> >> _______________________________________________
> >> >> mosh-devel mailing list
> >> >> mosh-devel at mit.edu
> >> >> http://mailman.mit.edu/mailman/listinfo/mosh-devel
> >> >
> >> >
> >>
> >> _______________________________________________
> >> mosh-devel mailing list
> >> mosh-devel at mit.edu
> >> http://mailman.mit.edu/mailman/listinfo/mosh-devel
> >
> >
>
> _______________________________________________
> mosh-devel mailing list
> mosh-devel at mit.edu
> http://mailman.mit.edu/mailman/listinfo/mosh-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/mosh-devel/attachments/20180707/506d2670/attachment-0001.html