Got (Buffer) Bloat? 121
mtaht writes, "After a very intense month of development, the Bufferbloat project has announced the debloat-testing git kernel tree, featuring (via suggestion of Van Jacobson) a wireless network latency smashing algorithm, (called eBDP), the SFB and CHOKe packet schedulers, and a slew of driver level fixes that reduce network latency across the Linux kernel by over 2 orders of magnitude. Got Bloat?"
what it is (Score:1, Insightful)
Re:what it is (Score:5, Informative)
It's about the downside of memory becoming cheap causing latency problems with congestion control mechinisms that rely on the endpoints being able to inform the sender when it's sending too fast.
Jim Getty's research blog entry [wordpress.com] explains the problem in detail.
Re:what it is (Score:3, Informative)
oblig. car analogy, by Eric Raymond no less:
https://lists.bufferbloat.net/pipermail/bloat/2011-February/000050.html [bufferbloat.net]
== Packets on the Highway ==
To fix bufferbloat, you first have to understand it. Start by
imagining cars traveling down an imaginary road. They're trying to get
from one end to the other as fast as possible, so they travel nearly
bumper to bumper at the road's highest safe speed.
[snipped]
Re:what it is (Score:0)
...but what is it?
What bufferbloat is (Score:5, Informative)
My understanding may not be correct but:
Bufferbloat (I first came across the term bufferbloat in this blog post by Jim Gettys [wordpress.com]) is the nickname that has been given to the high latency that can occur in modern network connections due to large buffers in the network. An example could be the way that a network game on one computer starts to stutter if another computer starts to use a protocol like bittorrent to transfer files on the same network connection.
The large buffers seem to have arisen from a desire to maximise download throughput regardless of network condition. This can give rise to the situation where small urgent packets are delayed because big packets (which perhaps should not have been sent) are queued up in front of them. The system sending the big packets is not told to stop sending them so quickly because its packets are being delivered...
The linked article sounds like people have modified the Linux kernel source to allow people who know how to compile their own kernels to test ideas people have had for reducing the bufferbloat effect on their hardware and to report back their results.
Does this help explain things a bit?
Re:What bufferbloat is (Score:5, Interesting)
high latency that can occur in modern network connections due to large buffers in the network
Nobody ever explained this to me but I was using ping to measure latency on a network where I was actually most interested in ssh. Ping times went something like 10ms, 50ms, 90ms, 130ms... up to about 500ms, then started again at 10ms, 50ms and so on. Maybe some of my pings shared a buffer with a large, periodic data transfer and when that transfer filled a buffer somewhere my latency dropped.
I am pretty sure the people actually operating the WAN in question had no idea what was going on either.
Re:What bufferbloat is (Score:5, Informative)
I should have also linked to a definition of bufferbloat by Jim Gettys [wordpress.com]. For the curious here's a page of links to bufferbloat resources [bufferbloat.net] and a 5 minute animation that shows the impact of large buffers on network communication (.avi) [bufferbloat.net].
Re:What bufferbloat is (Score:2)
Re:What bufferbloat is (Score:2)
Better link to the animation (Score:1)
Sorry Dave. A Coral cache version of the animation [nyud.net].
Comment removed (Score:1)
Re:What bufferbloat is (Score:5, Informative)
The code changes to the Linux kernel also reduce the size and ill effects of buffers inside the kernel and drivers.
Re:What bufferbloat is (Score:1)
My home internet connections have previously suffered from enormous buffers in the DSLAM - setting off a big download could cause ping times to increase to about 2 seconds, rendering other interactive use of the connection impossible.
Still, either this has been fixed now or more modern versions of TCP are more sophisticated, because it doesn't seem to happen any more - at least not to the same degree.
Re:What bufferbloat is (Score:4, Informative)
I think what you were seeing was more due to ATM overhead than the DSLAM trying to be cute with throttling. Because ADSL encapsulates everything in ATM even small IP / Ethernet frames get broken up into lots of ATM cells which can add upwards of 20% overhead. So an ADSL line trained at 8Mb/s will never provide 8Mb/s of usable throughput to the end user. Some ISPs actually advertise targeted throughput instead of train rate and set the train rate a certain percentage above the target throughput to compensate. Others just advertise train rates and have disclaimers in the fine print.
I've had my hands inside most Gen 1, 1.5 and 2nd Gen DSLAMs and never seen any with automatic throttling like you described.
(Gen 1 being units that just function at the ATM layer requiring an external system to bridge to Ethernet or IP. Gen 1.5's being upgraded Gen 1s with crude bridging, and Gen 2 being units that were designed to terminate connections directly from the ground up.)
Re:What bufferbloat is (Score:1)
To be fair, it was probably the pppoe terminator that throttled your connection. It's the only device in the network that has the correct position to be able to throttle efficiently on a per-customer basis, for both the upstream and the downstream direction, as well as the information required to do this "dynamic" throttling you talk about.
The issue technical support has which leads to bufferbloat is equally simple "but your add said 10 Mbit download speed and I'm getting 9.8 according to speedtest.net". The amount of customers that scream bloody murder and "demand their rights" is ridiculous.
You should check one of the discussions on the so-called "net neutrality" to see how flexible slashdotters are on the subject of isp's delivering slightly less bandwidth than advertised (which will always be the case without huge buffers) to see the problem. Or, God forbid, an isp that demands netflix pays for a customer line instead of a peering line when it is in fact a customer, and not a peer. This results in a *slightly* longer path to netflix, and thus apparently quite a few heathens had to be rounded up and thoroughly burned, despite the fact that this obviously did not involve giving preferential treatment to anyone. Quite the reverse, in fact.
I don't see this getting fixed any time soon.
Re:what it is (Score:2)
Re:what it is (Score:5, Informative)
Re:what it is (Score:5, Insightful)
What routers should do is keep track of how long packets have been in the router (in milliseconds or even microseconds) and use that with QoS stuff (and maybe some heuristics) to figure out which packets to send first, or to drop.
For example, "bulk/throughput" packets might be kept around for hundreds of milliseconds, but while latency sensitive packets get priority they are dropped if they cannot be sent within tens of milliseconds (then the sender will faster realize that it should slow down).
Re:what it is (Score:3, Insightful)
That's a much more complex solution than "don't buffer so much damn stuff for no good reason."
Re:what it is (Score:2)
But you do need big buffers to be able to do fast single tcp transfers! You need at least rtt * bandwidth in buffer in any place that has a faster uplink than downlink, like distribution switches for instance. And that's several megabytes, per port, in the today's gigabit ethernet world. Otherwise you're going to get bad to horrible throughput for high latency transfers.
Now, big buffers also need a decent buffer management (just trivial RED is orders of magnitudes better than "lets just fill the buffer up and then drop everything"), but going to small buffers isn't helping.
This is one of the hardest thing to get in a distribution switch, decently sized buffers. Most switches have horribly small buffers, or no documentation at all on sizes. Usually you have to go up to the chassis based ones to get something not horribly small. And if you want intelligent queue management so you can have both throughput and low interactive latency, well, I've heard Juniper makes one of those. Unfortunately at quite a bit higher price point than the cheap procurve/netgear stuff.
Re:what it is (Score:2)
Re:what it is (Score:2)
The problem with switches is that most switches have not merely small buffers, which would be ok, but microscopic ones. E.g. Cisco 3560G loses traffic on a gigabit port when faced with 50Mbps of bursty traffic in total coming from two ports. 10ms of buffer at 1Gbps is ~1MB, and most switches have nothing near that per port.
Re:what it is (Score:2)
This is an important point - and one most people are confused about. I'd like to add a nuance: my understanding is TX buffers are OK IF the router is very smarting about its queuing algorithm within the buffer, so that it drops packets early for any given sender so that the senders don't mistake a large buffer for a large pipe and overspeed the transmission. I believe Gettys &c just released (this article) is such a router/buffer queuing algorithm - smart enough to be effectively utilized with large buffered routers.
Who pointed it out? (Score:2)
Have you done the research to see just who you're disagreeing with about this?
And why they engineered TCP the way they did?
I won't pretend that I've walked through the experiments to try to verify their conclusions. I'm not even sure I know enough to interpret to interpret But...the people shouting the warnings aren't your average Chicken Littles.
Re:Who pointed it out? (Score:1)
Have you considered they still could be wrong?
Personally I've never seen a buffer in software designed they way they described. I've never heard of hardware acting that way, but as you said they certainly know more than I.
I stopped reading when they said 'it waits for the buffer to fill up until sending', which is true on a per packet level for a lot of things at a end point, in transit everything I've ever dealt with will forward packets without waiting UNTIL the output link becomes too congested to do so, THEN buffering starts happening. So the issue doesn't come into play until you are near saturation. When you hit that point, you're going to want buffering or the latency will be way worse when you start having to wait for retransmission.
Re:Who pointed it out? (Score:2)
Re:Who pointed it out? (Score:2)
Of course they could be wrong!
My only point here was: are you looking at their credentials?
As I understand it, you just re-stated the problem. Which is why they engineered TCP to work the way it does in the first place
Re:what it is (Score:2)
Your alternative isn't as simple as you'd like when you have many self interested clients playing zero sum competition over the router's bandwidth.
Re:what it is (Score:1)
Re:what it is (Score:2)
Keeping big buffers but managing them better (Score:3)
The first problem is that a ton of transit systems on the Internet (like indeed a ton of systems everywhere) are effectively running the default behaviors in this respect, with no special tuning. That means FIFO with whatever queue size is available.
The second is that even if all the ISP operators decided to fix this, "QoS stuff" has the potential to run afoul of Network Neutrality. The current thinking is that they shouldn't be discriminating between "bulk/throughput" packets and others. Some would suggest discrimination between traffic types is okay, so long as you don't discriminate between traffic sources (ie prioritize all VoIP, but don't let Comcast give Comcast VoIP preference over Vonage VoIP) but implementation would be tricky - all too easy to implement a "vendor neutral" policy that coincidentally doesn't seem to identify Vonage's traffic quite right.
The simplest and most neutral solution to all this is simply to decrease the buffer size in those big default FIFO queues. Even the bulk/throughput packets won't really suffer from that - TCP is specifically designed to have packets dropped in a timely manner, rather than held in a queue for a long time. One of the problem behaviors that Jim identifies is that if your real RTT to say, a server in California, is 40 msec, but there's 4 seconds worth of delay in the buffers. TCP will send a window full of data (let's say 64K) then wait for a reply for 40, 80, maybe 120 msec. Not getting it, he sends the whole window again. And again. And again. Finally an ACK squeezes through, and the process begins again. Instead of shrinking his transmission size, he does the opposite - he sends big multiples of every packet, making the congestion worse.
Re:Keeping big buffers but managing them better (Score:2)
I'm proposing they use an AQM algorithm that isn't that stupid/random but rather based on the QoS AND _age_of_packet_. The latter I believe is important.
One can determine the QoS by fields in the packet header and/or guessing.
Guessing isn't necessarily that difficult or error prone - latency sensitive stuff uses mostly small packets (because bigger packets = higher latency). And high throughput stuff uses mostly big "max size" packets.
With my proposal if say a 1Mbps ADSL user gets a quick burst of multiple HTTP downloads (a single page often involves multiple concurrent HTTP downloads) the router can queue them up in its buffers, but this doesn't have to interfere with the user's latency sensitive game connection, nor does it have to significantly lower HTTP throughput.
Whereas if you have small buffers, and the small buffers overflowed by the bursts, you get packet drops which reduce throughput and can still also interfere with latency sensitive packets (because they get dropped it the buffers happen to be full).
Many games (e.g. WoW) use TCP (require nonlossy comms) and a missing packet means an effective delay in the magnitude of RTT + timeouts since they have to detect that the packet got lost and then resend it. If your RTT
For example if you have a 1Mbps connection and your game ping is 200 milliseconds (server is far away), if your 256 byte latency sensitive packet is just delayed for 12 milliseconds (the time it takes to send a 1500 byte bulk http packet down a 1Mbps link - because it was in the process of being sent, it has to be sent before the low latency packet) it doesn't matter that much. But if the packet is dropped just because the buffers are full you're going to have to wait a few hundred milliseconds to get its replacement.
Most devices need to put the packet in a queue first before they can do fancy decisions on it. If the queue is full, they drop the packet. With my proposal the queues can be big so the latency sensitive packets don't have to be dropped just because of bursts.
Re:Keeping big buffers but managing them better (Score:2)
Please feel free to write some code. Writing a new qdisc for Linux and BSD is not very hard.
Writing a good qdisc, insanely so.
That said, I tend to feel that time-stamping more packets and doing more guessing may make sense, as does concepts in TCP vegas.
But: Before starting, read these:
http://pollere.net/Pdfdocs/bcit_6.2001.pdf [pollere.net] Kathleen Nichols - who proved to Van Jacobson that RED was wrong -
And Van Jacobson: http://pollere.net/Pdfdocs/QrantJul06.pdf [pollere.net]
Re:Keeping big buffers but managing them better (Score:1)
Re:Keeping big buffers but managing them better (Score:2)
You may be right about some of your points, but (if I understand your point) you're wrong about the QoS and net neutral stuff. FCC has never indicated that net neutrality regulations will impinge on "reasonable network management" practices. If folks need to route certain kinds of packets or manage certain kinds of buffers in specific ways to get performant networks, that's just great as far they're concerned (or anyone else with a legislative/regulatory angle I've ever read about or talked with).
I corresponded with some FCC staff about this bufferbloat issue specifically, and they are interested in it and definitely talking with the tel/cable cos about how to fix this. Net Neutrality is supposed to prevent these companies from taking unfair advantage of their position as network providers, not about making them worse network providers.
Re:Keeping big buffers but managing them better (Score:2)
Indeed, when I referenced "net neutrality", I wasn't referring to the specific implementation by the FCC, but rather the concept itself. The actual language does include an exception for "reasonable network management", but many network neutrality proponents were (I think somewhat rightfully) concerned at the size and flexibility of such a loophole.
Several providers saw their networks being loaded up with bit-torrent, and believed that limiting that specific protocol would constitute "reasonable network management". Naturally, many readers of slashdot disagreed. Also as I mentioned in the previous comment, I believe a triple-play provider could plausibly concoct a "reasonable network management" policy that would favor their own traffic without referencing themselves specifically. Say that Comcast used a different UDP port range for their RTP/VoIP bearer than Vonage did. Or say that to avoid allowing users to overwhelm priority queues, they refused to mark VoIP based on ports, but rather required it be destined for "verified legitimate voice gateways" (ie those administered by or registered formally with the provider, thus preventing Skype direct calls and the like from being priority-queued).
I am of course speculating wildly, but my point is that these unintended consequences are conceivable and that proponents of absolute network neutrality should be aware of the trade-offs with QoS (and vice versa). Also that while "Net Neutrality is supposed to prevent these companies from taking unfair advantage of their position as network providers, not about making them worse network providers", history is littered with laws and corporate policies that were "supposed to" achieve all sorts of laudable goals, but instead lead to unexpected exploitation by self-interested parties. I have no perfect solution in mind; I'm simply trying to call attention to the trade-offs involved and warn of possible unintended consequences.
Re:Keeping big buffers but managing them better (Score:1)
Re:Keeping big buffers but managing them better (Score:1)
Re:Keeping big buffers but managing them better (Score:2)
I understand that QoS policies can be written neutral, my concern is that they won't be. Dismissing the actions of ISPs and carriers as "a market problem" doesn't really constitute useful policy. Similarly, ignoring all network hosts who aren't FLOSS apps written after 2011 is also not helpful.
The issues being examined in the bufferbloat discussions are not about whether there is a technically possible solution somewhere in the world (indeed, both ECN and AQM have been around for a while). The issues relate to providing a robust Internet which can be/is administered by a large and diverse group of self-interested organizations in such a way that it productively serves a vastly diverse population of clients. Ignoring the ISPs, carriers, and non-FLOSS users isn't solving the problem, it's ignoring the problem.
Re:what it is (Score:2)
Re:what it is (Score:2)
Re:what it is (Score:2)
2) where you see a problem others may see a genuine business opportunity (I claim my proposal will give a better user experience than smaller buffers and/or stuff like RED).
Re:what it is (Score:2)
Re:what it is (Score:2)
Particularly, with carriers throwing bandwidth at the core, this should be an interesting project for DD-WRT, since gateway routers are those getting the most impact of all. With the increase of access capacities, the next hop will be also impacted and so on.
Those using pings to see the latency, don't seem to take into account also that when a long TCP flow gets impacted by a packet loss, throttles down, so the latency AND losses impact also the available capacity to TCP clients.
Now, many Active Queue Management (AQM) mechanisms, such as Random Early Detection (RED), have been proposed. And most of them work well with TCP, but Linux recently moved to CUBIC, is there enough information about the impact of those over CUBIC?
Re:what it is (Score:4, Informative)
re:"Interesting problem for dd-wrt"
Agreed.
We are throwing efforts at both the mainline kernel and openwrt.
Openwrt is foundational for dd-wrt and several other (commercial) distributions of Linux on the router. I have a large set of debloated routers already, I'm just awaiting further work on the eBDP algorithm to make better....
http://www.bufferbloat.net/projects/bloat/wiki/Experiment_-_Bloated_LAGN_vs_debloated_WNDR5700 [bufferbloat.net]
re: "using pings"
httpping is a much saner approach than ping, in many cases. Get it from:
http://www.vanheusden.com/httping/ [vanheusden.com]
re: RED & AQM
SFB and CHOKEe are in the debloat-testing kernel, as is eBDP.
RED 93 isn't going to work. nRED may. Experimentation and scripts highly desired. See the bloat and bloat-devel mailing lists for discussions.
Also:
http://www.bufferbloat.net/projects/bloat/wiki/Dogfood_Principle [bufferbloat.net]
Also:
I've seen some VERY interesting behavior with tcp vegas over bloated connections.
http://www.bufferbloat.net/projects/bloat/wiki/Experiment_-_TCP_cubic_vs_TCP_vegas [bufferbloat.net]
Re:what it is (Score:2)
1. Does said router properly respect QoS when deciding what data gets "rushed"?
2. Does said client have to pay a premium when sending out packets with elevated QoS?
Re:what it is (Score:2)
It's a myth that routers have the ability to tell a client to slow down, at least in the majority of environments (particularly ones with Ethernet segments, but other network types too).
Ethernet Flow Control has very limited utility here. You'll see it kick in in a rare few congestion cases - like if a switch backplane becomes overloaded - but it is used in a very limited number of situations and definitely will not be used end-to-end on a network for a router or host to tell another host (client) to do slow down.
Instead, when buffers fill, packets will be dropped. Yes, congestion control and FLOW CONTROL are handled using *packet drops* as the signalling method. It's assumed that TCP will use one of number of drop detection methods and congestion control algorithms to recover, and that does work better than one might think, but it's still ridiculously inefficient (especially with some types of drop patterns). I'll say it again, because I've been working on this pretty much daily for 10+ years and it still amazes me - you have to DROP PACKETS for any kind of end-to-end flow control to kick in.
In Ethernet land, there are some technologies created by folks like Cisco that are collectively being called "lossless Ethernet" that will definitely help here, and provide end-to-end flow control. But I'll pretty much guarantee you don't have hardware that supports it at the moment.
Re:what it is (Score:3)
And you know, in my experience that's where one of the real problems - and one of the most commonly undiagnosed problems - exists. In nearly 99% of networks I've looked at where buffer overflows were occurring and drops were happening, network admins were not only unaware of the severity of packet drops and didn't understand the impact this was having on their *critical* workloads, they had no idea how to even look for it.
Re:what it is (Score:1)
Re:what it is (Score:4, Informative)
Oh come on, you only have to follow four links to get to the definition [wordpress.com]. What are you, lazy?
Seriously, this is a true failure of web design. You click from the summary, then you go to the wiki, then you go to the faq, and the faq doesn't even tell you, it references a blog post.
Re:what it is (Score:3)
Re:what it is (Score:2)
Re:what it is (Score:2)
Solution: Use a proper protocol (aka ISO) (Score:1)
Instead of using TCP/IP (bastardized version of ISO), people should start using real OSI implementations such as the ISO protocol, with 20 byte addresses and QoS level settings for each of the 7 OSI layers.
Once upon a time it was an issue of cost of h/w logic, IP was the cheaper alternative, today the difference is nil and the benefits of ISO are orders of magnitude better than IP.
bufferbloat, IP address exhaustion, etc are just a few of the reason why we should drop IP altogether.
Re:Solution: Use a proper protocol (aka ISO) (Score:3)
The difference is that you can write an smtp server by reading in strings line by line and treating them as commands, then watch the logs and kludge it until it seems to interoperate well enough. With the OSI way of doing things you have to wear a blue tie for a start then you have to print out all the interface definition documents and spread them out on your desk and write the software to the interface.
You correctly point out that IP is cheaper, but that means all the people who work with it will be cheaper too and the product which is slightly cheaper will always win.
Re:Solution: Use a proper protocol (aka ISO) (Score:3, Funny)
The difference is that you can write an smtp server by reading in strings line by line and treating them as commands, then watch the logs and kludge it until it seems to interoperate well enough. With the OSI way of doing things you have to wear a blue tie for a start then you have to print out all the interface definition documents and spread them out on your desk and write the software to the interface.
man.. I want your desk if you can spread out all the iso interface definition documents on it and be able to read them
Re:Solution: Use a proper protocol (aka ISO) (Score:2)
I'm not sure if this has anything to do with it (or we were just victims of slick salespeople):
Bach in the early days of networking, when I was at Boeing, we (engineering) were starting to write some client-server stuff. Every time our IT folks approached us with ISO/OSI networking products as recommendations, there always seemed to be licensing fees attached. Per seat, per process, per user, per CPU, per whatever. While the software gurus were negotiating licenses and contracts, we just said "Screw it. Give me TCP/IP, a socket and get out of my way."
There may have been some unencumbered implementations out there. But we never saw them. If it wasn't carried in by a vendor in an expensive suit, it got no respect from Boeing Computer Services*.
*Same thing happened to Linux for a while. Everyone in the computing department wanted managing $100 million product portfolios on their resume. Not burning a bunch of free Slackware distro CDs.
Re:Solution: Use a proper protocol (aka ISO) (Score:1)
Erm, that doesn't seem to be the problem here IMHO & as per RTFA the problem is a maladaptive response to packet loss by throwing cheap memory into dumb buffers that effectively break the whole packet loss concept. Packet loss is not the enemy of throughput it is 'big idea' behind maintaing it. But sitting in a bloated buffer is the enemy of throughput, seeing the Internet as as series of sealed pipes is the enemy of throughput, missing the point completely and connecting huge dumb buffers into you OS your router your exchange, *everything* is the enemy of throughput.
Re:Solution: Use a proper protocol (aka ISO) (Score:2)
Why is this only rated a 1?
This may be the best summary of the problem that I've seen yet.
That is a battle which was lost 20 years ago (Score:4, Interesting)
A lot of our problems today would not be here if.
OSI stack instead of TCP/IP.
DCE & DFS instead of passwd/whatever + the bastard abomination which is NFS.
Meh. People are lazy and cheap. Free with the network effect always wins. The Lowest Common Denominator. It's going to take another 15 years before we are near where we were 15 years ago. But this time it will be in Java!
Re:That is a battle which was lost 20 years ago (Score:3)
OSI stack instead of TCP/IP
Can you please elaborate?
Re:That is a battle which was lost 20 years ago (Score:2)
Call me an idiot, but I thought TCP/IP was part of the OSI stack.
I'd also like to hear an explanation.
Re:That is a battle which was lost 20 years ago (Score:2)
Well, no. In my (limited) experience, you'd use CLNP [wikipedia.org] instead of IP if you were using OSI. And instead of IP addresses you would have NSAP [wikipedia.org] addresses. It's a whole different world actually.
Re:That is a battle which was lost 20 years ago (Score:2)
Did you ever use those things? I've never used the OSI stack (though I have had the misfortune of looking at some of the specs), but DCE and DFS had terrible perfomance 15 years ago, and were a bear to set up. Having never worked with the originals (Kerberos and Andrew File System), I don't know if this was a problem added in the "standardization" or if it came with the territory.
Re:That is a battle which was lost 20 years ago (Score:1)
Having never worked with the originals (Kerberos and Andrew File System), I don't know if this was a problem added in the "standardization" or if it came with the territory.
I can't speak about historical implementations, but the current (and I assume most modern implementations elsewhere) implementations of Kerberos used by Microsoft and the FreeBSD project can be configured for a system with a 5 line config file that could be generated from the output of a hostname -f call if the client is otherwise configured properly (Has its domain name set properly). It does require a proper DNS setup which can be obnoxious if you try to configure it by hand, but there again, its an implementation issue. Microsoft's implementation (part of ActiveDirectory) pretty much just works out of the box and requires pretty much no effort to use.
Of course, you still have to sync user ids :) Which is also simple with nss_ldap and ActiveDirectory+Service for Unix, but thats pretty trivial and could be done with a generic script that would work for almost everyone in the world out of the box.
There may be decent UNIX implementations of Kerberos and LDAP out there as well, but Microsoft has at least one implementation that doesn't suck to actually implement for no apparent reason.
I have never dealt with AFS, but making everything on my network use kerberos was pretty easy, but until its the standard way of authenticating I wouldn't expect anyone other than MS to put real effort into making it easy for an idiot ... or just not a pain in the ass.
Re:Solution: Use a proper protocol (aka ISO) (Score:0)
No amount of standardized QoS is going to help when you can't trust the endpoints not to cheat. IP has an 8-bit field you can use for QoS too, but you can't tell realtime from bulk traffic to assign QoS classes.
Re:Solution: Use a proper protocol (aka ISO) (Score:2)
Sure it will. You simply cap the customer at the bandwidth he/she is paying for and do QoS within the customer's allocation. Then you don't oversell the bandwidth. :-) Yeah, right. That'll happen.
Alternatively (and more realistically), you design things such that high priority packets must be sent isochronously (that is, every nth time slot) plus or minus a little jitter. A video codec will have no problem delivering frames at a constant data rate. Random bulk communications will not be able to do so, and those sockets will, after some threshold for maximum failures is exceeded, be automatically de-prioritized down to the level of bulk traffic.
You further penalize the endpoint by cutting it down to 56k modem speeds for a month if you see it making too many short-lived (<5s) connections to multiple IPs and declaring them isoch. By definition, any such connections are an abuse of the standard.
That should cover about 95% of the likely abuses.
Re:Solution: Use a proper protocol (aka ISO) (Score:4, Interesting)
You further penalize the endpoint by cutting it down to 56k modem speeds for a month if you see it making too many short-lived (<5s) connections to multiple IPs and declaring them isoch. By definition, any such connections are an abuse of the standard.
But isn't that exactly how channel surfing would work on IP-based television?
Re:Solution: Use a proper protocol (aka ISO) (Score:2)
No. That should generally look like a single, continuous stream from the head end at your cable company. Your machine sends back-channel messages to the head end to indicate that a channel change should occur, and it sends down a different stream. There's really no good reason to disconnect and reconnect when changing channels; doing so would constitute a rather serious abuse of the network due to all the overhead of setting up and tearing down an isochronous transport stream.
Now if by "changing channels" you mean rapidly switching between separate streams from content sources on the Internet, then yes, but on the other hand, most streaming videos are almost invariably not being transmitted live, and thus have no need to be isochronous in the first place. (The only reason to send data isochronously is if you cannot buffer forwards in time arbitrarily far to compensate for network congestion. Otherwise, you should be sending data in bulk. YouTube, for example, should be sending all of its data as bulk data.) As such, again, using isoch connections in that way would constitute abuse.
In short, you might be able to come up with a few edge case exceptions, but you'd have to try pretty hard. In general, isoch connections should be used for live streams from special events, video chat, VoIP, and a single isoch connection to the head end for anything that resembles IPTV. Chances are, anything that falls very far at all outside those bounds would constitute abuse of the standard.
Which streams does this head end carry? (Score:2)
[Changing channels] should generally look like a single, continuous stream from the head end at your cable company. [...] Now if by "changing channels" you mean rapidly switching between separate streams from content sources on the Internet, then yes, but on the other hand, most streaming videos are almost invariably not being transmitted live
In other words, streams MUST NOT be live unless the viewer subscribes to a cable television service and the streams have been approved by the operator of such service.
In general, isoch connections should be used for live streams from special events, video chat, VoIP, and a single isoch connection to the head end for anything that resembles IPTV.
This raises the question: should each IPTV user have access to multiple head ends? News and sporting events appear to qualify as "special events" for this purpose, but someone might want to watch news and sports offered by a provider without a specific carriage agreement with the head end owned by your particular ISP. For example, a small or remote provider might offer a live stream. Otherwise, it's cable TV carriage disputes all over again.
Re:Which streams does this head end carry? (Score:1)
Re:Solution: Use a proper protocol (aka ISO) (Score:1)
No, it was an issue of the ISO specs being bloated and incomprehensible. The human cost had much more to do with their failure than the hardware cost.
Two orders of magnitude! ? (Score:0)
Does that mean the network becomes a hundred times faster? Oh well, at least I can dream.
Re:Two orders of magnitude! ? (Score:4, Interesting)
It is about reducing latency. So it helps prevent problems with VoIP failing when there is a lot of other data flowing over the same connection(s).
Some buffers are really large and it turns out some introduce latency of several seconds (!).
Re:Two orders of magnitude! ? (Score:2)
Re:Two orders of magnitude! ? (Score:2)
Re:Two orders of magnitude! ? (Score:1)
Re:Two orders of magnitude! ? (Score:5, Interesting)
Most network throughput is at least 80-90% efficient already, so it won't get much faster. It will make it more responsive though, which is good if you're browsing the web, playing an online game or something else interactive.
I assume this is under load though, because on ping there's not much to be saved. On local sites I have 8-12 ms ping, on slashdot I have 140-150 ms. Since the theoretical round trip in a straight line at light speed is some 110 ms, there's not even room for a 50 ms drop. A lot of weirdness can happen under load though if stuff gets buffered up various places.
Re:Two orders of magnitude! ? (Score:5, Informative)
Re:Two orders of magnitude! ? (Score:2)
Re:Two orders of magnitude! ? (Score:2)
Most network throughput is at least 80-90% efficient already, so it won't get much faster. It will make it more responsive though, which is good if you're browsing the web, playing an online game or something else interactive.
I assume this is under load though, because on ping there's not much to be saved. On local sites I have 8-12 ms ping, on slashdot I have 140-150 ms. Since the theoretical round trip in a straight line at light speed is some 110 ms, there's not even room for a 50 ms drop. A lot of weirdness can happen under load though if stuff gets buffered up various places.
Doesn't this also mean that under high bittorrent pressure, pings will not time out and and ssh will remain responsive.
Latency again (Score:4, Insightful)
I've seen it time and time again, people just generally don't care about latency, or even deny it exists in many cases (buffer bloat is certainly one cause of latency).
Everything from changing channel on your TV remote, to a mobile phone number entry, to the frame delays you get from LCD monitors, to the soundcard delay, to the GUI widgets you click on;......... it's all over the place, and it can wreck the experience, or reduce it somewhat according to how big the delay is. Just because latency is harder to measure, that doesn't mean it isn't very important, especially when it builds up with lots of other 'tiny' delays to make one big delay.
Re:Latency again (Score:2)
Re:Latency again (Score:2)
That said, I wish we could settle on an Internet-wide QOS implementation and get both. Some packets have a legitimate need to cut in line. It would be workable if ISPs advertized both 'total' bandwidth and a smaller amount of 'turbo' bandwidth, or whatever stupid name they want to use for it, which is the fraction of your bandwidth that is not over-subscribed. By setting the QOS bits you could prioritize part of your traffic.
Re:Latency again (Score:4, Insightful)
Re:Latency again (Score:4, Informative)
"Why does the web site load so slowly?" is the classic question - caused in many cases by the "eagleholic" having 4 live eagle nest video streams running in one window while trying to post observations and screencaps to the web site in another.
Believe me - there is ample reason to deal with the problem as most of today's home networks are used for more than just one thing at a time. Mom is watching video, sis is uploading pictures of her party to Facebook, son is playing online games and dad is trying to listen to streaming audio - and NOTHING is working correctly despite the fact that this is a trivial load for even a T1 (1.45Mbps) let alone today's high-speed cable (30Mbps down and 5Mbps up). We used to run 30+ modems and web sites and email and all manner of stuff over bonded 56K ISDN lines for pity sake - and we got better latency than the links today.
What's the problem? The latency for the "twitch" game packets has gone from 10ms to 4000ms or more - and the isochronos audio stream is jerky because it's bandwidth starved and the upload takes forever because the ACKs from FB can't get through the incoming video dump from YouTube (with its fast start window pushed from default 3 to 11 or 12) and by the time the video is half over, the link to YouTube has dropped because it took 30 seconds or more for the buffer to drain after the first push and the link had timed out.
That's the problem - you need low latency for some things at the same time you need high throughput for others - and it is possible and can be done - and IS done if things are tuned correctly. But correctly tuning the use of buffers is an art today, not a science - and the ever-changing (by 3-4 orders of magnitude) needs of today's end-point routers has pushed the limits of what AQM (automated queue management) algorithms are currently available, even if they're turned on (which in most cases they're not it seems)
Re:Latency again (Score:2)
Re:Latency again (Score:2)
"I want everything louder than everything else" (Meat Loaf) epitomizes the net today - we have Google screwing with the fast start window and Microsoft pretty much ignoring it and setting it as large as possible in some cases (they do other things right though it seems)
The buffer bloat problem is one born of history and ignorance:
History - it used to be that we could not put enough buffer RAM into the device because it was too expensive - so we designed our algorithms to use all that was available "because there's never enough."
Ignorance - we now have a generation of network "engineers" who have grown up not having to deal with really congested networks (until very recently) and simply don't bother to turn on things like RED (Random Early Detection) in their router products or ECN (Explicit Congestion Notification) on their servers and links - or ensure that ECN is actually passed through and not zero'd. Now they don't even recognize the problem and we have to teach them (and get them to un-learn their bad habits - like "any packet loss is bad")
5 months ago my prototype of our old company's first generation embedded Linux router software, running on an old '486 chassiz, finally died. When I replaced it with a recent D-Link, my connection from my home office to the world went from quite acceptable to almost useless whenever I was up or downloading anything larger than a couple of hundred K. The buffer on the router fills up and the latency goes from 10-20ms to 4000ms (4 seconds) - and my streaming radio stops - and my xload monitors and other remote monitors on servers stop - and my nagios system checks go off thinking that the remote systems are down.
This is unacceptable - and it is a LOCAL problem. In systems where the ISP's equipment is to blame, there's little I could do but rate-limit my connection to something under the threshold of pain.
Even turning on the router's "QOS" setting (switch, not knob - no control over parameters) that should give me "better gaming results" does not eliminate the problem or do much at all.
Bufferbloat is real - but the good news is that it can be fixed if we start asking the right questions of our suppliers and get them to admit there is a problem. The Bufferbloat community is in the process of putting together test facilities to help you and the ISPs and manufacturers get definitive information on the problem; things like a mixed-mode lantency/throughput test that measures 2 or 3 different stream types at the same time instead of just raw "bits through the pipe from source to destination"
I expect that even if ISO had won the war (and I was there in the trenches at the time the war was being fought) we'd have come to this point at some time - but IMHO that would have been some time in the next century as the ISO "standards" regime (and cost) was a huge damper on development and deployment. We would not have had the digital revolution at all if it was ISO anchored.
Re:Latency again (Score:2)
A poorly designed router can sabotage the performance TCP, causing overall slowness to your connections. Particularly, those 10Mbps you're paying for and want them to properly work.
Re:Latency again (Score:2)
(Reasonable) packet loss or ECN - pick one - and then tell your up and downstream neighbors why you picked it (hopefully ECN will find its way into near 100% deployment ASAP) and why they should respect it and follow on.
Then - when the Bufferbloat gurus get the testing systems working, test and report so we can do our jobs and let the world know good/bad setups and such.
Lead, Follow, or get the hell out of the way
Every packet is sacred (Score:5, Funny)
Every packet is great.
If a packet is wasted,
TCP gets quite irate.
Let the heathen drop theirs
When their RAM is spent.
TCP shall make them pay for
Each packet that can't be sent.
Every packet is wanted.
To this we are sworn.
From real-time data from CERN
To the filthiest of porn.
Every packet is sacred.
Every packet is great.
If a packet is wasted,
TCP gets quite irate.
.
Re:Every packet is sacred (Score:2)
Re:Every packet is sacred (Score:2)
Re:Every packet is sacred (Score:2)
Best. Comment. Ever. That's going on my door at work - thanks.
Most delays are due to the ethernet packet buffers (Score:1)
Most delays are due to users connecting to their ADSL modem via Ethernet and not traffic managing properly.
On a congested link this can cause large delays as Ethernet normally has a 1000 packet buffer in the Linux kernel and the ADSL modem has a similar buffer. You only need a couple of heavy connections which want to go faster than the ADSL will support and those buffers start to fill up real fast. You can easily end up with latencies measured in seconds if you have a lot of connections running (say bittorrent).
There are several solutions to this but the best in my experience is to change the queuing discipline to SFQ and rate limit using HTB. This has been in the kernel for years and works extremely well. You need to limit the traffic upstream and downstream to slightly less (5% less) than the ADSL link speed. This ensures that the modem never queues traffic. Uplink you can use all sorts of fancy queuing but downlink all you can really do is policing of traffic unless you install the IMQ patch to the kernel.
I've a script which I got from somewhere a while ago, don't remember where though. I've put it at http://ams1.x31.com/~andy/ppp0-ratelimit.sh if anyone wants to look at it. It expects to work on ppp0 but can be adapted as required.
I've played a lot more recently with Linux kernel disciplines and it has produced surprising performance on congested links. One link is running mail, remote access and Internet access over a 1mbit symmetric link for about 60 users. in the morning it hits 95% link capacity at the start of work and stays there until everyone goes home but ssh sessions are fully interactive without noticeable lag all this time. Yes web browsing is a little slow but it is the same for everyone and one user can't flood the link and upset everyone else.
Linux QOS is the future, pity about the documentation
Buffer bloat is (not) an illusion... (Score:2)
After reading this guys buffer bloat rant I largly agree with him with some exceptions:
1. What does multiple TCP sessions have to do with circumvention of congestion avoidance? TCP congestion avoidance needs to work with lots and lots of TCP sessions at once not just one or two. HTTP 1.1 sessions need NOT be short lived. I don't see why a large number of TCP sessions can't all be subject to congestion avoidance...responding individually to the conditions they see? How does this work to effectivly bypass congestion avoidance? I've seen this talking point in a few places but noone has ever explained WHY this is so. I can see an argument based on a static suboptimal initial congestion window but HTTP 1.1 supports pipelining...
2. Two connections per server is not sufficient for browsers. TCP is a stream protocol with head of line blocking... High latency links will never use the available bandwidth properly unless they either use lots of sessions or start with massive windows which is not good for congestion. There is also a problem of ordering dependancies of resource requests within the web content. Without lots of concurrent fetches the user gets to wait longer for page loads. The presentation sounded to me like someone either not understanding necessary details of TCP and higher layer considerations or trying to have their cake and eat it too.
Lastly we don't need to replace HTTP - we need to replace TCP... HTTP over SCTP would be a much more significant improvement than any reasonable change to HTTP. No matter what you do to HTTP you still have to live with the underlying transports limitations!
Re:Buffer bloat is (not) an illusion... (Score:2)
Sort of in answer to both of your questions the bufferbloat.net servers are configured as follows:
http://www.bufferbloat.net/projects/bloat/wiki/Dogfood_Principle [bufferbloat.net]
trying at every point to make sure http 1.1 actually got used.
We survived today's slashdotting. Handily.
That said, your points are well made. SPDY is part of the chromium browser and looks to have some potential.
In my case, I like the idea of smarter - and eventually sctp-enabled - proxies, especially on wireless hops. See thread at:
https://lists.bufferbloat.net/pipermail/bloat/2011-February/000068.html [bufferbloat.net]
Re:Buffer bloat is (not) an illusion... (Score:3)
Re: 1. I've always thought that the congestion window to the same end-point should be shared: but that's not the way TCP implementations work, and wishing they worked that way won't make the problem go away. And, as I've shown, bufferbloat is not a TCP phenomena in any case.
Re: 2. HTTP is a lousy protocol in and of itself, and having to do it on top of TCP makes it yet harder. It is the fact that HTTP is so ugly that makes so much else difficult. And I disagree with your claim that high latency links won't use the bandwidth; in fact, lots of sessions is just making things harder. You can read our HTTP/1.1 Sigcomm performance paper.
I'll be writing more on this topic this week.
And we need to replace HTTP, and something other than TCP would be highly desirable. Personally, I'm much more fond of CCNx than any IP based transport.
Corrected Git URL (Score:4, Informative)
The link in the /. story to debloat-testing should go here: git://git.infradead.org/debloat-testing.git [git].
git:gitinfradeadorgdebloat-testinggit is not a valid URL.
Re:Do they have a solution (Score:2)