[Rooftops] I assume

[Lenny Foner]

Every time I think about this sort of thing, I worry about latency.
An RF mesh tends to have a lot of hops.  Even if you start transmitting
on the next hop as soon as you can, you still have to get enough of a
packet preamble to know which of your (virtual?) beams to send it on.
That means that each hop adds some amount of latency that's hard to
get around, even if it's only a few bits of header (and not, say,
all the bits of a large packet).

Mitigating factors of course include:
(a) Assume users aren't using ssh.  (I do, but I'm a fossil.)
    [And assume that Skype traffic will instead use the real trunks?
    Ick.  So there's a case millions use where real latency will
    really matter.  Simple web browsing may be one of them, too, alas.]
(b) Any increment in bits/second on the links shortens the amount of
    time you have to wait for that preamble, so speak as fast as you
    can.  You'd want that anyway, of course. :)
(c) Any increment in hop length helps.  Of course, this has a direct
    bearing on (b).  Even with beamforming, at some point you'll hit
    annoying limits (or obstacles :).
(d) Assign flow tags that can be abbreviated to a very few bits, and
    preface packets with them.  Won't help you for a coast-to-coast
    UDP query (think DNS), but those are rare.  Will help you for
    virtual circuits, which describe most of the cases where latency
    gets noticeable (ssh, voice, etc).  This is analogous to Van
    Jacobson compression back when 9600 bps modems were all the rage;
    the idea would be to use Huffman coding (for example) to identify
    streams, with some hacks for flushing old state so you only have
    to identify a few (dozens? hundreds?) of streams per node at any
    given time.  I'll bet a careful reading of the literature would be
    rewarding; I'd be shocked if there isn't a ton of research on this
    sort of thing.  This must be Cisco's bread & butter.  Zigbee might
    also be rewarding to study for whatever it does; have you?

Doing (d), in particular, assumes you can get bits as they come in
from the RF system---not after they've all be received and a packet
checksum has been computed.  Does typical 802.whatever HW allow this
under any circumstances?  Possibly, given Airsnort et al.  And what
are the implications of noise in the channel?  [Maybe you start
forwarding as soon as you think you know where the packet's going,
with an oops-never-mind-abort if some really short & simple flow
checksum comes along later and you realize you had noise, and another
never-mind-I-really-blew-it if the packet checksum fails at the end.
And if noise is rare, maybe you just don't bother letting go of the
channel early & just let the receiver drop it if a checksum's wrong.
But probably that means one error early on ripples through a -lot- of
hops before everyone abandons it, so it could cost you bandwidth.]

I also wonder if it could ever pay off to have that preamble be
a different RF modulation---using more bandwidth or more power or
whatever to get it out faster, in the assumption that even though
you're degrading the rest of the channel slightly, the decrease in
latency is worth it.  May not be worth the added RF complexity,
though.  [But it might:  20 bits of header means 20 bits of delay
per hop, plus one bit of delay per hop for the payload, assuming
optimality.  So either making that header shorter, or speaking it
faster, might have a big multiplier on end-to-end latency.  Absolutely
don't use uncompressed TCP!  40 bytes/hop, minimum.  Bleah.]