[mosh-devel] Mosh on Kuberbnetes

[Thomas Buckley-Houston]

Hey Keith, John, everyone,

Yeah the more this is looking like a quite a big hurdle. Especially
your point Keith about roaming IPs (which I'd forgotten), it's a
central feature of Mosh I don't want to lose.

So the only 2 options seems to be exposing multiple IPs for Round
Robin (or other smart DNS routing) or writing a new Mosh proxy that
already has knowledge of the existing keys. Both seem like quite a
challenge. Round Robin DNS seems more approachable and I can imagine
integrating it with the Google Cloud DNS API I'm already using, but I
just wonder how expensive Google (or anyone for that matter) will make
thousands of static IP addresses? Apart from me having to learn Mosh
internals, one difficulty that strikes me about a Mosh proxy is that
it might introduce a non-trivial delay to each datagram arriving?
Though surely only ever in the order of a handful of milliseconds I
suppose.

Are there not any other identifying marks to a datagram, I don't know
much about low level networking, but maybe something like a MAC
address for example?

Thanks,
Tom

On 27 June 2018 at 04:50, Keith Winstein <keithw at cs.stanford.edu> wrote:
> Hi Thomas,
>
> Glad you could provoke a very interesting discussion! But I'm still confused
> -- how is "sticky IP-based routing" going to work after the client roams to
> a new IP address (or to a new UDP source port)? When your system seems an
> incoming UDP datagram from a previously unseen source IP:port, how does it
> know which mosh-server (on which server machine) to send it to?
>
> With off-the-shelf Mosh, you basically need a load-balancing strategy that
> allows a destination IP:port to uniquely identify a particular mosh-server.
> You can do this with multiple DNS A/AAAA records (where the client picks the
> winning one -- maybe you permute the list), or with a smart DNS server that
> serves *one* A or AAAA record to the client at the time of resolution (like
> a CDN would use).
>
> Instead of using the mosh wrapper script, you could have your users use some
> other scheme to figure out the IP:port of the server, but the point is that
> once you launch the mosh-client, it's going to keep sending datagrams to the
> IP:port of the mosh-server, and those datagrams need to get to the same
> mosh-server process even if the client roams to a different publicly-visible
> IP address or port.
>
> You could imagine writing a very smart mosh proxy that has the keys to all
> the sessions and can figure out (for an incoming datagram coming from an
> unknown source IP:port) which session it actually belongs to, and then makes
> a sticky mapping and routes it to the proper mosh-server. But I don't think
> anybody has actually done this yet and of course there's a challenge in
> making this reliable/replicated.
>
> -Keith
>
> On Mon, Jun 25, 2018 at 3:10 AM, Thomas Buckley-Houston <tom at tombh.co.uk>
> wrote:
>>
>> Thanks so much for the clarification.
>>
>> > UDP is connectionless
>>
>> That's the key here. So I have no choice but to use sticky IP-based
>> routing. Round-robin DNS isn't an option I don't think, because I hope
>> one day to be able to scale to thousands of servers.
>>
>> And thanks so much for the heads up about my DNSSEC records. I've sent
>> a request for them to be deleted. I'd added them and some SSHFP
>> records to explore automatically passing the StrictHostKey warning.
>> But it's not entirely straight forward. Even with correct DNS records
>> the SSH user still has to have VerifyHostKeyDNS enabled, which as I
>> understand most people don't. And then on top of that my DNS provider
>> (DNSSimple) automatically rotate the keys every 3 months, which means
>> I have to manually send a request to my registrars by email to update
>> the DNSSEC records. Is it all worth it do you think?
>>
>> On 24 June 2018 at 13:36, Anders Kaseorg <andersk at mit.edu> wrote:
>> > You may have a misunderstanding about how a Mosh session is set up.  The
>> > mosh script launches a mosh-server on the remote system via SSH;
>> > mosh-server picks a port number and a random encryption key, and writes
>> > them to stdout, where they go back over SSH to the mosh script; then the
>> > mosh script launches mosh-client passing the IP address, port number,
>> > and
>> > encryption key.  The newly launched mosh-client and mosh-server
>> > processes
>> > exchange UDP packets encrypted with the shared key; communication is
>> > successful if the packets can be decrypted.
>> >
>> > There’s no separate “key checking” step to be disabled.  And it doesn’t
>> > make sense to “refuse more than 1 connection on the same port”, both
>> > because UDP is connectionless, and because a new mosh-server is launched
>> > on a new port for each Mosh session (it is not a daemon like sshd).
>> >
>> > The easiest way to put Mosh servers behind a load balancer is with
>> > round-robin DNS where a single hostname resolves to many addresses, or
>> > to
>> > different addresses for different clients and/or at different times.
>> > We’ve already gone out of our way to make the mosh script resolve the
>> > hostname only once and use the same address for the SSH connection and
>> > the
>> > UDP packets, because that’s needed for MIT’s athena.dialup.mit.edu pool.
>> >
>> > If that’s not an option and you really need all connections to go
>> > through
>> > a single load balancer address, you could try wrapping mosh-server in a
>> > script that passes different disjoint port ranges (-p) on different
>> > backends, and forwarding those ranges to the corresponding backends from
>> > the load balancer.
>> >
>> > Unrelatedly, brow.sh doesn’t resolve with DNSSEC-enabled resolvers like
>> > 1.1.1.1 or 8.8.8.8, seemingly due to some problem with the DS records
>> > set
>> > with the registrar: https://dnssec-debugger.verisignlabs.com/brow.sh.
>> >
>> > Anders
>>
>> _______________________________________________
>> mosh-devel mailing list
>> mosh-devel at mit.edu
>> http://mailman.mit.edu/mailman/listinfo/mosh-devel
>
>