<div dir="ltr">Hello Thomas,<div><br></div><div><div class="gmail_extra"><div class="gmail_quote">On Thu, Jul 12, 2018 at 8:58 PM, Thomas Buckley-Houston <span dir="ltr"><<a href="mailto:tom@tombh.co.uk" target="_blank">tom@tombh.co.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Forgive me for not fully learning about UDP, I've Googled a little<br>
bit, but I'm sure I still have a lot of gaps in my knowledge. So<br>
because UDP is a connectionless protocol, every datagram has to<br>
contain the source IP in order for the receiver to send responses?<br></blockquote><div><br></div><div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">IP is a connectionless protocol, and every IP datagram contains a source and destination IP address in the IP header.</span></div></div><div><br></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Both UDP and TCP put their data in IP datagrams -- so every TCP-in-IP datagram and every UDP-in-IP datagram contains a source and destination IP address.</span></div><div><br></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">TCP and Mosh both add connections on top of IP. (UDP is basically just a thin layer on top of IP that adds a port number so that individual programs can use it.)</span></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
It's not enough for either end to assume an IP based on the *initial*<br>
datagram's IP? So based on that assumption, the best way for a</blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
datagram to get its IP:port is to query the OS.</blockquote><div><br></div><div>Not quite sure what you mean here. When you write a program to receive a UDP datagram, that program can learn the source IP and port of the datagram by calling recvfrom() or recvmsg().</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Which in our scenario<br>
is a privately networked container and so will be unreachable from the<br>
outside.<br></blockquote><div><br></div><div>For the datagrams flowing from the mosh-server to the mosh-client, yeah, the mosh-server is going to send using a private source address. It will be the job of the proxy server to change that source address (and port number) into a publicly reachable one.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
>From a brief research I couldn't find any docs on this, I'm hoping its<br>
possible with Nginx. But maybe it can only be done with something like<br>
iptables?<br></blockquote><div><br></div><div>I think you're going to have to write your own program to do this (using the logic I described on June 30) -- I don't think iptables is going to be able to do exactly what I described out of the box. But it doesn't have to know anything about the SSP protocol and it doesn't have to decrypt anything. It does have to rewrite the "private" IP addresses and port numbers into public ones as I described.</div><div> </div><div>Cheers,</div><div>Keith</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On a separate note I re-launched Texttop as Browsh this week to a big<br>
response: <a href="https://www.brow.sh" rel="noreferrer" target="_blank">https://www.brow.sh</a> The SSH servers (`ssh brow.sh`) held up<br>
really well, they've had thousands of sessions already without issue.<br>
So would be great to get Mosh working as well.<br>
<br>
So just another friendly reminder - I'm having to tell people to<br>
compile Mosh themselves to get true colour support.<br>
<div class="HOEnZb"><div class="h5"><br>
On 8 July 2018 at 11:56, Keith Winstein <<a href="mailto:keithw@cs.stanford.edu">keithw@cs.stanford.edu</a>> wrote:<br>
> Hi Thomas,<br>
><br>
> Hmm, let me try to see if I can say it better. For an outgoing UDP datagram<br>
> (from a containerized mosh-server to a mosh-client), the mosh-server will be<br>
> sending the datagram from some private IP address (the container's IP<br>
> address, e.g. 10.0.0.5) and a source port that is probably going to be the<br>
> same across all containers (like 60001), since each container will probably<br>
> only be running one mosh-server.<br>
><br>
> The proxy will want to rewrite the source IP:port so that the source IP<br>
> address is the proxy's routable Internet address and the port number is<br>
> whatever unique port it assigned and sent to the client in the original MOSH<br>
> CONNECT message that the client saw.<br>
><br>
> -Keith<br>
><br>
> On Sat, Jul 7, 2018 at 8:32 PM, Thomas Buckley-Houston <<a href="mailto:tom@tombh.co.uk">tom@tombh.co.uk</a>><br>
> wrote:<br>
>><br>
>> Thanks so much for this idea, I really think it's the one, simple and<br>
>> scalable.<br>
>><br>
>> I haven't tried but I'm pretty sure the mosh-server's "MOSH CONNECT"<br>
>> can be wrapped in plain BASH. I'm already in control of the SSH<br>
>> connection as I'm using my own `ForceCommand` script.<br>
>><br>
>> Also I can still use this method with extra Round-Robin balanced IP<br>
>> addresses giving me multiple sets of 65,000 ports.<br>
>><br>
>> The only thing I don't understand is why the outgoing UDP datagram has<br>
>> to rewrite the container's source IP. Isn't the original MOSH CONNECT<br>
>> IP:port the canonical reference?<br>
>><br>
>> On 1 July 2018 at 12:54, Keith Winstein <<a href="mailto:keithw@cs.stanford.edu">keithw@cs.stanford.edu</a>> wrote:<br>
>> > How about a semi-smart (but mostly Mosh-oblivious) server-side proxy/NAT<br>
>> > that works like this:<br>
>> ><br>
>> > - The proxy service has one public IP address and like 65,000 available<br>
>> > UDP<br>
>> > ports.<br>
>> > - The proxy service can itself be redundant with failover...<br>
>> > - When a user wants to open a new Mosh connection, they Mosh to a single<br>
>> > hostname (which resolves to the IP address of the proxy service).<br>
>> > - Your code allocates the necessary container, etc., and also allocates<br>
>> > a<br>
>> > unique UDP port on the proxy.<br>
>> > - Your code runs the new mosh-server process in the target container.<br>
>> > - The proxy intercepts the mosh-server's "MOSH CONNECT <port> <key>"<br>
>> > message, replacing the port number with the unique public-facing UDP<br>
>> > port<br>
>> > (and remembering the container's IP address and the original port<br>
>> > number).<br>
>> > - When the proxy receives an incoming UDP datagram destined to a<br>
>> > particular<br>
>> > UDP port, it forwards it to the appropriate container at its IP address<br>
>> > and<br>
>> > at the original port number. It *preserves* the source IP and port of<br>
>> > the<br>
>> > datagram when forwarding.<br>
>> > - When the container wants to send an outgoing UDP datagram, it sends it<br>
>> > normally (to whatever IP:port is associated with the client), except the<br>
>> > containers are not directly connected to the Internet; they use the<br>
>> > proxy/NAT as their next-hop router.<br>
>> > - For the outgoing UDP datagram, the proxy/NAT rewrites the container's<br>
>> > source IP:port to its own IP and the public port number.<br>
>> ><br>
>> > I think this will allow you to serve like 65,000 separate mosh<br>
>> > connections<br>
>> > from a single public IP address...<br>
>> ><br>
>> > The added latency in forwarding a datagram is probably <1 ms, and you<br>
>> > don't<br>
>> > really have to change anything about Mosh itself or its internals.<br>
>> ><br>
>> > Unfortunately there are no unencrypted identifying marks to a Mosh<br>
>> > connection, except the incrementing sequence numbers (which start at 0<br>
>> > for<br>
>> > every connection).<br>
>> ><br>
>> > -Keith<br>
>> ><br>
>> > On Fri, Jun 29, 2018 at 12:17 AM, Thomas Buckley-Houston<br>
>> > <<a href="mailto:tom@tombh.co.uk">tom@tombh.co.uk</a>><br>
>> > wrote:<br>
>> >><br>
>> >> Hey Keith, John, everyone,<br>
>> >><br>
>> >> Yeah the more this is looking like a quite a big hurdle. Especially<br>
>> >> your point Keith about roaming IPs (which I'd forgotten), it's a<br>
>> >> central feature of Mosh I don't want to lose.<br>
>> >><br>
>> >> So the only 2 options seems to be exposing multiple IPs for Round<br>
>> >> Robin (or other smart DNS routing) or writing a new Mosh proxy that<br>
>> >> already has knowledge of the existing keys. Both seem like quite a<br>
>> >> challenge. Round Robin DNS seems more approachable and I can imagine<br>
>> >> integrating it with the Google Cloud DNS API I'm already using, but I<br>
>> >> just wonder how expensive Google (or anyone for that matter) will make<br>
>> >> thousands of static IP addresses? Apart from me having to learn Mosh<br>
>> >> internals, one difficulty that strikes me about a Mosh proxy is that<br>
>> >> it might introduce a non-trivial delay to each datagram arriving?<br>
>> >> Though surely only ever in the order of a handful of milliseconds I<br>
>> >> suppose.<br>
>> >><br>
>> >> Are there not any other identifying marks to a datagram, I don't know<br>
>> >> much about low level networking, but maybe something like a MAC<br>
>> >> address for example?<br>
>> >><br>
>> >> Thanks,<br>
>> >> Tom<br>
>> >><br>
>> >> On 27 June 2018 at 04:50, Keith Winstein <<a href="mailto:keithw@cs.stanford.edu">keithw@cs.stanford.edu</a>><br>
>> >> wrote:<br>
>> >> > Hi Thomas,<br>
>> >> ><br>
>> >> > Glad you could provoke a very interesting discussion! But I'm still<br>
>> >> > confused<br>
>> >> > -- how is "sticky IP-based routing" going to work after the client<br>
>> >> > roams<br>
>> >> > to<br>
>> >> > a new IP address (or to a new UDP source port)? When your system<br>
>> >> > seems<br>
>> >> > an<br>
>> >> > incoming UDP datagram from a previously unseen source IP:port, how<br>
>> >> > does<br>
>> >> > it<br>
>> >> > know which mosh-server (on which server machine) to send it to?<br>
>> >> ><br>
>> >> > With off-the-shelf Mosh, you basically need a load-balancing strategy<br>
>> >> > that<br>
>> >> > allows a destination IP:port to uniquely identify a particular<br>
>> >> > mosh-server.<br>
>> >> > You can do this with multiple DNS A/AAAA records (where the client<br>
>> >> > picks<br>
>> >> > the<br>
>> >> > winning one -- maybe you permute the list), or with a smart DNS<br>
>> >> > server<br>
>> >> > that<br>
>> >> > serves *one* A or AAAA record to the client at the time of resolution<br>
>> >> > (like<br>
>> >> > a CDN would use).<br>
>> >> ><br>
>> >> > Instead of using the mosh wrapper script, you could have your users<br>
>> >> > use<br>
>> >> > some<br>
>> >> > other scheme to figure out the IP:port of the server, but the point<br>
>> >> > is<br>
>> >> > that<br>
>> >> > once you launch the mosh-client, it's going to keep sending datagrams<br>
>> >> > to<br>
>> >> > the<br>
>> >> > IP:port of the mosh-server, and those datagrams need to get to the<br>
>> >> > same<br>
>> >> > mosh-server process even if the client roams to a different<br>
>> >> > publicly-visible<br>
>> >> > IP address or port.<br>
>> >> ><br>
>> >> > You could imagine writing a very smart mosh proxy that has the keys<br>
>> >> > to<br>
>> >> > all<br>
>> >> > the sessions and can figure out (for an incoming datagram coming from<br>
>> >> > an<br>
>> >> > unknown source IP:port) which session it actually belongs to, and<br>
>> >> > then<br>
>> >> > makes<br>
>> >> > a sticky mapping and routes it to the proper mosh-server. But I don't<br>
>> >> > think<br>
>> >> > anybody has actually done this yet and of course there's a challenge<br>
>> >> > in<br>
>> >> > making this reliable/replicated.<br>
>> >> ><br>
>> >> > -Keith<br>
>> >> ><br>
>> >> > On Mon, Jun 25, 2018 at 3:10 AM, Thomas Buckley-Houston<br>
>> >> > <<a href="mailto:tom@tombh.co.uk">tom@tombh.co.uk</a>><br>
>> >> > wrote:<br>
>> >> >><br>
>> >> >> Thanks so much for the clarification.<br>
>> >> >><br>
>> >> >> > UDP is connectionless<br>
>> >> >><br>
>> >> >> That's the key here. So I have no choice but to use sticky IP-based<br>
>> >> >> routing. Round-robin DNS isn't an option I don't think, because I<br>
>> >> >> hope<br>
>> >> >> one day to be able to scale to thousands of servers.<br>
>> >> >><br>
>> >> >> And thanks so much for the heads up about my DNSSEC records. I've<br>
>> >> >> sent<br>
>> >> >> a request for them to be deleted. I'd added them and some SSHFP<br>
>> >> >> records to explore automatically passing the StrictHostKey warning.<br>
>> >> >> But it's not entirely straight forward. Even with correct DNS<br>
>> >> >> records<br>
>> >> >> the SSH user still has to have VerifyHostKeyDNS enabled, which as I<br>
>> >> >> understand most people don't. And then on top of that my DNS<br>
>> >> >> provider<br>
>> >> >> (DNSSimple) automatically rotate the keys every 3 months, which<br>
>> >> >> means<br>
>> >> >> I have to manually send a request to my registrars by email to<br>
>> >> >> update<br>
>> >> >> the DNSSEC records. Is it all worth it do you think?<br>
>> >> >><br>
>> >> >> On 24 June 2018 at 13:36, Anders Kaseorg <<a href="mailto:andersk@mit.edu">andersk@mit.edu</a>> wrote:<br>
>> >> >> > You may have a misunderstanding about how a Mosh session is set<br>
>> >> >> > up.<br>
>> >> >> > The<br>
>> >> >> > mosh script launches a mosh-server on the remote system via SSH;<br>
>> >> >> > mosh-server picks a port number and a random encryption key, and<br>
>> >> >> > writes<br>
>> >> >> > them to stdout, where they go back over SSH to the mosh script;<br>
>> >> >> > then<br>
>> >> >> > the<br>
>> >> >> > mosh script launches mosh-client passing the IP address, port<br>
>> >> >> > number,<br>
>> >> >> > and<br>
>> >> >> > encryption key. The newly launched mosh-client and mosh-server<br>
>> >> >> > processes<br>
>> >> >> > exchange UDP packets encrypted with the shared key; communication<br>
>> >> >> > is<br>
>> >> >> > successful if the packets can be decrypted.<br>
>> >> >> ><br>
>> >> >> > There’s no separate “key checking” step to be disabled. And it<br>
>> >> >> > doesn’t<br>
>> >> >> > make sense to “refuse more than 1 connection on the same port”,<br>
>> >> >> > both<br>
>> >> >> > because UDP is connectionless, and because a new mosh-server is<br>
>> >> >> > launched<br>
>> >> >> > on a new port for each Mosh session (it is not a daemon like<br>
>> >> >> > sshd).<br>
>> >> >> ><br>
>> >> >> > The easiest way to put Mosh servers behind a load balancer is with<br>
>> >> >> > round-robin DNS where a single hostname resolves to many<br>
>> >> >> > addresses,<br>
>> >> >> > or<br>
>> >> >> > to<br>
>> >> >> > different addresses for different clients and/or at different<br>
>> >> >> > times.<br>
>> >> >> > We’ve already gone out of our way to make the mosh script resolve<br>
>> >> >> > the<br>
>> >> >> > hostname only once and use the same address for the SSH connection<br>
>> >> >> > and<br>
>> >> >> > the<br>
>> >> >> > UDP packets, because that’s needed for MIT’s <a href="http://athena.dialup.mit.edu" rel="noreferrer" target="_blank">athena.dialup.mit.edu</a><br>
>> >> >> > pool.<br>
>> >> >> ><br>
>> >> >> > If that’s not an option and you really need all connections to go<br>
>> >> >> > through<br>
>> >> >> > a single load balancer address, you could try wrapping mosh-server<br>
>> >> >> > in<br>
>> >> >> > a<br>
>> >> >> > script that passes different disjoint port ranges (-p) on<br>
>> >> >> > different<br>
>> >> >> > backends, and forwarding those ranges to the corresponding<br>
>> >> >> > backends<br>
>> >> >> > from<br>
>> >> >> > the load balancer.<br>
>> >> >> ><br>
>> >> >> > Unrelatedly, brow.sh doesn’t resolve with DNSSEC-enabled resolvers<br>
>> >> >> > like<br>
>> >> >> > 1.1.1.1 or 8.8.8.8, seemingly due to some problem with the DS<br>
>> >> >> > records<br>
>> >> >> > set<br>
>> >> >> > with the registrar:<br>
>> >> >> > <a href="https://dnssec-debugger.verisignlabs.com/brow.sh" rel="noreferrer" target="_blank">https://dnssec-debugger.<wbr>verisignlabs.com/brow.sh</a>.<br>
>> >> >> ><br>
>> >> >> > Anders<br>
>> >> >><br>
>> >> >> ______________________________<wbr>_________________<br>
>> >> >> mosh-devel mailing list<br>
>> >> >> <a href="mailto:mosh-devel@mit.edu">mosh-devel@mit.edu</a><br>
>> >> >> <a href="http://mailman.mit.edu/mailman/listinfo/mosh-devel" rel="noreferrer" target="_blank">http://mailman.mit.edu/<wbr>mailman/listinfo/mosh-devel</a><br>
>> >> ><br>
>> >> ><br>
>> >><br>
>> >> ______________________________<wbr>_________________<br>
>> >> mosh-devel mailing list<br>
>> >> <a href="mailto:mosh-devel@mit.edu">mosh-devel@mit.edu</a><br>
>> >> <a href="http://mailman.mit.edu/mailman/listinfo/mosh-devel" rel="noreferrer" target="_blank">http://mailman.mit.edu/<wbr>mailman/listinfo/mosh-devel</a><br>
>> ><br>
>> ><br>
>><br>
>> ______________________________<wbr>_________________<br>
>> mosh-devel mailing list<br>
>> <a href="mailto:mosh-devel@mit.edu">mosh-devel@mit.edu</a><br>
>> <a href="http://mailman.mit.edu/mailman/listinfo/mosh-devel" rel="noreferrer" target="_blank">http://mailman.mit.edu/<wbr>mailman/listinfo/mosh-devel</a><br>
><br>
><br>
<br>
______________________________<wbr>_________________<br>
mosh-devel mailing list<br>
<a href="mailto:mosh-devel@mit.edu">mosh-devel@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/mosh-devel" rel="noreferrer" target="_blank">http://mailman.mit.edu/<wbr>mailman/listinfo/mosh-devel</a><br>
</div></div></blockquote></div><br></div></div></div>