[mosh-devel] Mosh on Kuberbnetes

[Keith Winstein]

Hi Thomas,

Glad you could provoke a very interesting discussion! But I'm still
confused -- how is "sticky IP-based routing" going to work after the client
roams to a new IP address (or to a new UDP source port)? When your system
seems an incoming UDP datagram from a previously unseen source IP:port, how
does it know which mosh-server (on which server machine) to send it to?

With off-the-shelf Mosh, you basically need a load-balancing strategy that
allows a destination IP:port to uniquely identify a particular mosh-server.
You can do this with multiple DNS A/AAAA records (where the client picks
the winning one -- maybe you permute the list), or with a smart DNS server
that serves *one* A or AAAA record to the client at the time of resolution
(like a CDN would use).

Instead of using the mosh wrapper script, you could have your users use
some other scheme to figure out the IP:port of the server, but the point is
that once you launch the mosh-client, it's going to keep sending datagrams
to the IP:port of the mosh-server, and those datagrams need to get to the
same mosh-server process even if the client roams to a different
publicly-visible IP address or port.

You could imagine writing a very smart mosh proxy that has the keys to all
the sessions and can figure out (for an incoming datagram coming from an
unknown source IP:port) which session it actually belongs to, and then
makes a sticky mapping and routes it to the proper mosh-server. But I don't
think anybody has actually done this yet and of course there's a challenge
in making this reliable/replicated.

-Keith

On Mon, Jun 25, 2018 at 3:10 AM, Thomas Buckley-Houston <tom at tombh.co.uk>
wrote:

> Thanks so much for the clarification.
>
> > UDP is connectionless
>
> That's the key here. So I have no choice but to use sticky IP-based
> routing. Round-robin DNS isn't an option I don't think, because I hope
> one day to be able to scale to thousands of servers.
>
> And thanks so much for the heads up about my DNSSEC records. I've sent
> a request for them to be deleted. I'd added them and some SSHFP
> records to explore automatically passing the StrictHostKey warning.
> But it's not entirely straight forward. Even with correct DNS records
> the SSH user still has to have VerifyHostKeyDNS enabled, which as I
> understand most people don't. And then on top of that my DNS provider
> (DNSSimple) automatically rotate the keys every 3 months, which means
> I have to manually send a request to my registrars by email to update
> the DNSSEC records. Is it all worth it do you think?
>
> On 24 June 2018 at 13:36, Anders Kaseorg <andersk at mit.edu> wrote:
> > You may have a misunderstanding about how a Mosh session is set up.  The
> > mosh script launches a mosh-server on the remote system via SSH;
> > mosh-server picks a port number and a random encryption key, and writes
> > them to stdout, where they go back over SSH to the mosh script; then the
> > mosh script launches mosh-client passing the IP address, port number, and
> > encryption key.  The newly launched mosh-client and mosh-server processes
> > exchange UDP packets encrypted with the shared key; communication is
> > successful if the packets can be decrypted.
> >
> > There’s no separate “key checking” step to be disabled.  And it doesn’t
> > make sense to “refuse more than 1 connection on the same port”, both
> > because UDP is connectionless, and because a new mosh-server is launched
> > on a new port for each Mosh session (it is not a daemon like sshd).
> >
> > The easiest way to put Mosh servers behind a load balancer is with
> > round-robin DNS where a single hostname resolves to many addresses, or to
> > different addresses for different clients and/or at different times.
> > We’ve already gone out of our way to make the mosh script resolve the
> > hostname only once and use the same address for the SSH connection and
> the
> > UDP packets, because that’s needed for MIT’s athena.dialup.mit.edu pool.
> >
> > If that’s not an option and you really need all connections to go through
> > a single load balancer address, you could try wrapping mosh-server in a
> > script that passes different disjoint port ranges (-p) on different
> > backends, and forwarding those ranges to the corresponding backends from
> > the load balancer.
> >
> > Unrelatedly, brow.sh doesn’t resolve with DNSSEC-enabled resolvers like
> > 1.1.1.1 or 8.8.8.8, seemingly due to some problem with the DS records set
> > with the registrar: https://dnssec-debugger.verisignlabs.com/brow.sh.
> >
> > Anders
>
> _______________________________________________
> mosh-devel mailing list
> mosh-devel at mit.edu
> http://mailman.mit.edu/mailman/listinfo/mosh-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/mosh-devel/attachments/20180626/07638280/attachment.html