[mosh-devel] Impressed

Thu Apr 12 16:14:32 EDT 2012

On Thu, Apr 12, 2012 at 10:40 AM, Hari Balakrishnan <hari at csail.mit.edu> wrote:
> Hi Dave,
>
> The idea of setting the appropriate ToS/DiffServ code point in mosh-generated data makes sense.  Some questions:
>
> 1) What specific DiffServ Code Point would you recommend?  Does openwrt support EF?

Everything supports EF, but that is pretty much intended for voip. It
does have one advantage in that it jumps the default pfifo_fast
*software* queue , and big disadvantage for wifi *hardware* queues in
that packets in the VO queue cannot be aggregated.

So the 'logical' thing would have been to use the IMM bit, however, no
deployed linux based access point that I know of does the right thing
with that. The best alternative
seems to be CS4 or CS5.

I'm actually not going to foist this decisionmaking on you, the
version of netperf now in svn is capable of exercising diffserv,
socket priority, and tcp algorithms, via remote control, on both sides
of a connection.

svn co http://www.netperf.org/svn/netperf2/trunk/

build that, and you can exercise these two very different queues under
saturating loads and observe what happens.

The syntax is rather undocumented at present, but some is up here.

https://www.bufferbloat.net/projects/cerowrt/wiki/Using_the_Networking_Test_Tools

While I have some data on it, it's all with sfq and sfqred in the mix,
which shows the
hardware queues dominating the delays in wifi, by a lot, and the
queues actually allocating bandwidth as per the specification for
802.11e (yea!)

Openwrt has cut their default txqueuelens from 1000 to 30, as of about
september of last year, so current builds are also dominated by wifi
delays and retries.

>
> 2) Would the setting of the codepoint be treated consistently across different network elements? E.g., if we set a particular value but some other network treated it very differently, would it make things worse (I suppose it'll be fine as long as no one treats it as worse than best-effort).

By and large diffserv codepoints do not survive transit across the
internet over ipv4. Surprisingly, they do tend to survive over ipv6.

BK (background, also known as CS0) sometimes survives, as does IMM.

So, within an organisation you will see this stuff be respected by
most gear and mostly following the relevant, if sometimes conflicting,
rfcs, but across the core, no.

I have no good statistical data on any of that, however, I have tons
of captures that appear to show against the test sites I run against.
More data would be good. :)

> 3) What are CS4 and CS5?

Diffserv codepoints. See the rfc... Several others may apply, but as
deployed in Linux wifi, only the CS0-CS7 bits are examined (which are
also present in the other codepoints in an almost-but-not-quite-sane
manner)

I have a backlog of somewhat relevant code and tests and an
increasingly obsolete proposal that needs revision relative to
diffserv, but it mostly applies to 802.11e, and I have not tried it
very much with vlan technologies (802.11q). See

http://www.bufferbloat.net/projects/bloat/wiki/Diffserv_RFC

https://github.com/dtaht/Diffserv

But I note I'm not proud of their current state and some of my
thinking has evolved based on later data.

> And something in your email confuses me: isn't the primary issue in networks with deep buffers caused actually by the bulk transfers that occupy the queues?  So while mosh could in principle be careful not to fill up buffers, realistically, it might not have a huge impact in avoiding filling up queues because most of that fill up comes from non-mosh traffic anyway? Am I missing something?

I can imagine scenarios where mosh-like traffic would need to be sane
about and participate in congestion avoidance, or be affected by it.
Not many! My comment regarding ECN marking it is that makes it mildly
more survivable e2e in some cases.

I'm curious, if you have to retry sending state, I imagine you coalese
the information into larger packets.

(again, I've only been using the stuff, and have read both papers, but
not the code, soooo....)

>Is the issue that ssh by itself fills up buffers badly during interactive sessions even when other traffic is minimal?

ssh can do that yes.

What are the effects of tail -f syslog in your design? I've read up on
supdup, wasn't clear to me how you handled high rates.

>
> Keith and I have been working on other solutions/approaches to bufferbloat (which itself is a symptom of larger problems), but that's independent of the mosh software for the most part.

As clean subsets go, wow. I was fiddling with HIP as a similar
solution (openhip does nat traversal well now) but doing this at a
core application level appears to *just work*, and go beyond to handle
the increasing common nat timeout and wifi mobility problems.

Totally made my month.

>
> Hari
>
> On Apr 12, 2012, at 1:23 PM, Dave Taht wrote:
>
>> On Wed, Apr 11, 2012 at 4:33 PM, Keith Winstein <keithw at mit.edu> wrote:
>>> Thanks Dave! I'm a big lurker on the bufferbloat list and appreciate your
>>> kind words.
>>
>> You've taken a huge subset of the problem with bufferbloat (interactive ssh
>> management sessions suck) and taken care of it innovative, easy to deploy way,
>> that I'd never thought of.
>>
>> Sysadmins around the world will rejoice. In my case I frequently
>> (purposely) blow up the lab network and losing my state on a dozen or
>> more ssh sessions is really annoying.
>>
>> For the last two days, that hasn't happened. It's very relaxing.
>>
>> "Mosh: better than valium. "
>>
>> You can quote me on that. :)
>>
>> My major kvetch, oh lurker, is that had I known about your project
>> before I'd absolutely
>> have take the time to toss it into cerowrt/openwrt! As it is, there seem to be
>> a few large dependencies that would need to be ported, and packaged,
>> and I'm too low on time before this release to do so myself.
>>
>> A couple other notes: The current linux wifi stack does do categorization
>> into the 802.11e hardware queues, based on diffserv priority markings.
>>
>> While these generally do not survive  transit across the internet,
>> they do seem to help locally on a congested network.
>>
>> openssh tries to use the old tos-style imm bit, but due to the installed base
>> not respecting that, I suspect you'd get better results on congested
>> wifi if you tossed mosh's packets into CS4 or CS5 categories, which
>> will utilize the underused VI queue.
>>
>> (cerowrt tosses IMM into the VI queue, I really should push that
>> patch up to mainline)
>>
>> Somewhat similarly, most of the bloat related solutions are working
>> best with ECN, and
>> it would be interesting to try turning that on, as packets marked with
>> that bit have a better
>> chance of survival through an AQM, and possibly getting some more
>> signaling from the path would help, on occasion.
>>
>> I'm painfully aware that ECN appears hard to deploy,
>> but it seems to me you could attempt a robust solution in your protocol here.
>>
>>> We plan to support IPv6, and there are some user branches on github that get
>>> us working IPv6 for IPv6-only hosts. The problem is that for dual-stack
>>> hosts, we think it's pretty likely that the user may start up over IPv6 but
>>> later want to roam to a cellular network (e.g. LTE) that supports IPv4 only.
>>> We want this to keep working as well as it does today.
>>>
>>> I haven't found other applications that have successfully implemented
>>> cross-protocol roaming -- I think it raises some new issues. (At the IETF
>>> last month we had some discussion about the various ways this could be done
>>> but nobody seemed confident in a pre-existing approach that would work.)
>>>
>>> Right now we look up the server's A records, the resolver picks one, and we
>>> use that as the IPv4 address (for both the initial SSH connection that
>>> remotely executes the mosh-server, and then for the mosh-client to connect
>>> to the mosh-server).
>>>
>>> If we support IPv6 and v4, then we're going to potentially get back four
>>> AAAA records and seven A records. How do we know which pairs correspond to
>>> the same host? Right now the client's roaming is totally stateless and it
>>> has no idea that it has even roamed (it just keeps sending heartbeats to the
>>> same target IPv4 address).
>>>
>>> So to do this properly, we'll probably have to provide the client with a
>>> list of possible IPv6 and IPv4 addresses (every AAAA and A record we got
>>> back), and have it start out sending packets to all of them over each
>>> protocol. Whichever one it gets a (cryptographically authentic) reply from
>>> it can lock onto, but if it ever goes more than a few seconds without a
>>> reply from the server, it will have to fall back to the mass-send approach
>>> again.
>>>
>>> This is like Happy Eyeballs on steroids.
>>> (http://tools.ietf.org/html/draft-ietf-v6ops-happy-eyeballs-07)
>>
>> It's good to know there are branches that do ipv6, and I agree that
>> super-happy-eyeballs would be required, and a PITA.
>>
>>> The datagrams all represent idempotent operations so I don't think there's
>>> any theoretical obstacle here, but it means adding logic and state to the
>>> client for something that right now is very dumb.
>>>
>>> So... we're working on it! :-)
>>>
>>> Best regards,
>>> Keith
>>>
>>>
>>> On Wed, 11 Apr 2012, Dave Taht wrote:
>>>
>>>> I've been fiddling with mosh in the lab here and I'm really impressed.
>>>>
>>>> I need to read up on your underlying protocol a bit, but right now
>>>> I'm no longer as motivated to finish fixing tcp.
>>>>
>>>> In what state is the ipv6 support?
>>>>
>>>> --
>>>> Dave Täht
>>>> http://www.bufferbloat.net
>>>>
>>>> _______________________________________________
>>>> mosh-devel mailing list
>>>> mosh-devel at mit.edu
>>>> http://mailman.mit.edu/mailman/listinfo/mosh-devel
>>
>>
>>
>> --
>> Dave Täht
>> SKYPE: davetaht
>> US Tel: 1-239-829-5608
>> http://www.bufferbloat.net
>>
>> _______________________________________________
>> mosh-devel mailing list
>> mosh-devel at mit.edu
>> http://mailman.mit.edu/mailman/listinfo/mosh-devel
>

-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net