<div dir="ltr">Hi Thomas,<div><br></div><div>Glad you could provoke a very interesting discussion! But I'm still confused -- how is "sticky IP-based routing" going to work after the client roams to a new IP address (or to a new UDP source port)? When your system seems an incoming UDP datagram from a previously unseen source IP:port, how does it know which mosh-server (on which server machine) to send it to?</div><div><br></div><div>With off-the-shelf Mosh, you basically need a load-balancing strategy that allows a destination IP:port to uniquely identify a particular mosh-server. You can do this with multiple DNS A/AAAA records (where the client picks the winning one -- maybe you permute the list), or with a smart DNS server that serves *one* A or AAAA record to the client at the time of resolution (like a CDN would use).</div><div><br></div><div>Instead of using the mosh wrapper script, you could have your users use some other scheme to figure out the IP:port of the server, but the point is that once you launch the mosh-client, it's going to keep sending datagrams to the IP:port of the mosh-server, and those datagrams need to get to the same mosh-server process even if the client roams to a different publicly-visible IP address or port.</div><div><br></div><div>You could imagine writing a very smart mosh proxy that has the keys to all the sessions and can figure out (for an incoming datagram coming from an unknown source IP:port) which session it actually belongs to, and then makes a sticky mapping and routes it to the proper mosh-server. But I don't think anybody has actually done this yet and of course there's a challenge in making this reliable/replicated.</div><div><br></div><div>-Keith</div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jun 25, 2018 at 3:10 AM, Thomas Buckley-Houston <span dir="ltr"><<a href="mailto:tom@tombh.co.uk" target="_blank">tom@tombh.co.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thanks so much for the clarification.<br>
<br>
> UDP is connectionless<br>
<br>
That's the key here. So I have no choice but to use sticky IP-based<br>
routing. Round-robin DNS isn't an option I don't think, because I hope<br>
one day to be able to scale to thousands of servers.<br>
<br>
And thanks so much for the heads up about my DNSSEC records. I've sent<br>
a request for them to be deleted. I'd added them and some SSHFP<br>
records to explore automatically passing the StrictHostKey warning.<br>
But it's not entirely straight forward. Even with correct DNS records<br>
the SSH user still has to have VerifyHostKeyDNS enabled, which as I<br>
understand most people don't. And then on top of that my DNS provider<br>
(DNSSimple) automatically rotate the keys every 3 months, which means<br>
I have to manually send a request to my registrars by email to update<br>
the DNSSEC records. Is it all worth it do you think?<br>
<div class="m_-4240057182193738083HOEnZb"><div class="m_-4240057182193738083h5"><br>
On 24 June 2018 at 13:36, Anders Kaseorg <<a href="mailto:andersk@mit.edu" target="_blank">andersk@mit.edu</a>> wrote:<br>
> You may have a misunderstanding about how a Mosh session is set up. The<br>
> mosh script launches a mosh-server on the remote system via SSH;<br>
> mosh-server picks a port number and a random encryption key, and writes<br>
> them to stdout, where they go back over SSH to the mosh script; then the<br>
> mosh script launches mosh-client passing the IP address, port number, and<br>
> encryption key. The newly launched mosh-client and mosh-server processes<br>
> exchange UDP packets encrypted with the shared key; communication is<br>
> successful if the packets can be decrypted.<br>
><br>
> There’s no separate “key checking” step to be disabled. And it doesn’t<br>
> make sense to “refuse more than 1 connection on the same port”, both<br>
> because UDP is connectionless, and because a new mosh-server is launched<br>
> on a new port for each Mosh session (it is not a daemon like sshd).<br>
><br>
> The easiest way to put Mosh servers behind a load balancer is with<br>
> round-robin DNS where a single hostname resolves to many addresses, or to<br>
> different addresses for different clients and/or at different times.<br>
> We’ve already gone out of our way to make the mosh script resolve the<br>
> hostname only once and use the same address for the SSH connection and the<br>
> UDP packets, because that’s needed for MIT’s <a href="http://athena.dialup.mit.edu" rel="noreferrer" target="_blank">athena.dialup.mit.edu</a> pool.<br>
><br>
> If that’s not an option and you really need all connections to go through<br>
> a single load balancer address, you could try wrapping mosh-server in a<br>
> script that passes different disjoint port ranges (-p) on different<br>
> backends, and forwarding those ranges to the corresponding backends from<br>
> the load balancer.<br>
><br>
> Unrelatedly, brow.sh doesn’t resolve with DNSSEC-enabled resolvers like<br>
> 1.1.1.1 or 8.8.8.8, seemingly due to some problem with the DS records set<br>
> with the registrar: <a href="https://dnssec-debugger.verisignlabs.com/brow.sh" rel="noreferrer" target="_blank">https://dnssec-debugger.verisi<wbr>gnlabs.com/brow.sh</a>.<br>
><br>
> Anders<br>
<br>
______________________________<wbr>_________________<br>
mosh-devel mailing list<br>
<a href="mailto:mosh-devel@mit.edu" target="_blank">mosh-devel@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/mosh-devel" rel="noreferrer" target="_blank">http://mailman.mit.edu/mailman<wbr>/listinfo/mosh-devel</a><br>
</div></div></blockquote></div><br></div></div>