Multi-Threaded SSH/SCP

neo writes "Chris Rapier has presented a paper describing how to dramatically increase the speed of SCP networks. It appears that because SCP relies on a single thread in SSH, the crypto can sometimes be the bottleneck instead of the wire speed. Their new implementation (HPN-SSH) takes advantage of multi-threaded capable systems dramatically increasing the speed of securely copying files. They are currently looking for potential users with very high bandwidth to test the upper limits of the system."

A likely story

[sqldr (838964)]

Hi, I've invented a new way of downloading pron^H^H^H^H^H^Hcopying files across a network. If you have uber bandwidth, please contact me urgently!

Re:A likely story

[slyn (1111419)]

Makes you wonder how many innovations can either be directly attributed to or partially attributed to the distribution of porn (not (necessarily) that this is).

VHS v Betamax comes to mind.

Re:A likely story

[shenanigans (742403)]

I think this story is interesting because it shows a general trend: increased focus on multi-threading. We will see much more of this in the future as multi-core and multi-processor systems become more common. This trend is driven not by porn though, but by that other big driving force behind the computer industry, gaming.

Re:A likely story

[rapier1 (855437)]

Actually, this is one of the main reasons why we did this. If you look at where processor architectures are going they aren't necessarily increasing the pure computational power of each core as much as they are using more and more cores. If you are in a situation where you have a single threaded process that is CPU bound - like SSH can be - you'll find that throughput rates (assuming that you aren't network bound) will remain flat. In order to make full use of the available network capacity we'll have to start doing things like this. There is still a lot of work to be done but we're pleased by our progress so far.

Re:

[dodobh (65811)]

There's also message passing and event driven programming, which can be a much simpler model if done right. Multi threading tends to shared state, and that's bad for programmers.

Re:A likely story

[jovetoo (629494)]

It is rare that you can completely separate every context of every step of your processing. There is always some data that needs to be shared between the threads and they become bottlenecks. The faster you serve your requests, the worse the contention (waiting for a resource) and thus the inefficiency.

It depends on the task at hand and on your architecture. A file or web server is less likely to encounter contention than for example an IRC server. The first requires some authentication and resource resolving through configuration data but the actual data can be send without interference from other requests. An IRC server requires constant lookups in the user database for routing information and this is likely to take longer than actually sending the messages (even without multi-threading). In these cases, you really have to think your locking scheme through or you will lose more time waiting for a lock than doing actual work - defeating much of the purpose of going MT.

When it comes to architecture, multi threading is an option in your architecture, not an architecture in itself. There is no problem doing a multi-threaded event-driven architecture or a MT message passing architecture -- these are actually very effective. For some interesting reading about this, I would suggest you check out the SEDA white paper [harvard.edu] for a pretty in depth list of options and their goals.

Why is it bad for programmers? Because locking is hard to do in itself and if your locking scheme is subobtimal it often requires a lot of work to change it afterwards.

Re:

[dodobh (65811)]

This one perhaps? : Threads.pdf [pacbell.net]

Re:A likely story

[mikael (484)]

People are always willing to pay more to be entertained that to be educated.

Which explains why football players and movie stars will get paid more than the innovators that carried out the research to develop the broadcast technology that helped to make those stars famous.

Re:

[empaler (130732)]

AND they get pregnant.

Re:FUNNY?! That's not funny, try for TRUE

[by Anonymous Coward]

I won't lie to you -- being a parent is no laughing matter. It is a ton of work. It can be amazingly stressful and expensive. I've been through periods that I look back on now and wonder how the hell I managed to pull through without going completely insane. But if you ask me, the rewards outweigh the difficulties ten to one.

When your child first looks up at your face and you see actual recognition in her eyes... when you see all the blocks fall into place as she figures out how to do something for the first time... look, I know it sounds really sappy and smarmy, but seriously (srsly) it is absolutely indescribable. This thing started out as a bit of genetic code from two people, and now it is actually self-aware and sentient. How cool is that? What geek can't be astonished at these emergent properties, derived from a program more complicated than you can possibly imagine -- a program that has spontaneously evolved over time?

And you get to see her mental map evolve. You watch branches get added to her decision tree. You observe as she learns how to acquire information, process it, and decide how to act upon it. And all the while, you mold her view of the world based on your interactions with her. I don't know about you, but I find that not only fascinating, but incredibly rewarding.

Before my daughter was born, I was terrified too, and somebody had said these things to me, I would've said, "Yeah, okay, I'm sure it's great and all, but I'm sure you're exaggerating somewhat." That's because there is something that happens to you when it's your kid. There's some very ancient, very basic code that gets turned on in your brain that says "this life is your responsibility, and you must do everything you can to ensure its safety, survival, and growth". I can't explain it because I honestly believe it's something buried deep beneath the conscious mind.

Whatever the case, if you honestly don't want the baby, for it's sake, put it up for adoption. Don't make it live a life with a father who doesn't care for it. I'm being absolutely serious here. Find a loving couple who are unable to have kids of their own.

(Posting AC because this is way offtopic, and because there are a lot of single, selfish, bitter child-haters out there with mod points to burn... but I had to say something.)

Re:FUNNY?! That's not funny, try for TRUE

[empaler (130732)]

Thank you for your very well-written reply, but I wasn't actually being all that serious. (No, the pregnancy was unplanned, and I am actually opposed to the idea of becoming a parent at my current place in life; however, after talking this over with the only other person to have any say in this (the mother), I've decided to go with it).

Don't mistake my badly crafted joke for being completely ignorant of what's ahead of me; before the final decision came, I had consulted with friends who are also parents (carefully not discussing this with any of my single, singlemindedly free-roaming friends), and I am in no way in doubt that I will make this child a net benefit for the human race. There are simply too many rotten parents, spoilt children, miserable families and bad genes in the world for me to actually fail in that respect.
Plus, living in Denmark*, the baby will have pretty good odds for a good life, my involvement notwithstanding.

I am going to have a lot of fun making tech projects for my little one when that time comes, including audio books with his/her favourite bed time stories, video diaries of how the child evolves, and of course, teaching how to solder before the age of 5. How I survived until 15 without that knowledge eludes me to this day.

*: Studies have shown that there is a tie for Country With Best Quality of Life; Denmark and Iceland. I've been to Iceland, and it smelled like rotten eggs. Denmark takes the lead.

Re:

[empaler (130732)]

No worries, she'll be right.

Re:A likely story

[Mister Whirly (964219)]

Because slasdotters generally can either only afford to pay for the porn site, or pay for the girl, but not both.

Oblig applicable Dilbert

[bigmouth_strikes (224629)]

Wally: "My proposed work plan for the year is to stress-test our product under severe network conditions. I will accomplish this by downloading large image files from the busiest servers on the net."

(PHB rejects suggestion)
(later)

Wally: "I was this close to making it my job to download naughty pictures."
Dilbert : "It's just as well; I would have had to kill you."

( http://books.google.com/books?id=dCeVfKrZ-3MC&pg=PA77&source=gbs_selected_pages&cad=0_1&sig=xD5tmMhG1RcspLch8gCIJu8ro2U#PPA79,M1 [google.com] )

this is just what i've been looking for

[by Anonymous Coward]

not that scp as-is isn'--stalled--

Must be why rsync over ssh is much faster

[MichaelSmith (789609)]

I get a lot of use out of ssh for moving files around and rsync is definitely the best way to do heavy lifting in this area. Improving scp would be good to. I can't wait to hear what Theo thinks about this. I don't see him as a fan of adding complexity to improve performance.

Big scp copies through my wifi router used to cause kernel panics under netbsd current of about a year ago. I never had that problem running rsync inside ssh.

Re:

[rapier1 (855437)]

As a note, the changes are actually to SSH itself and not just SCP. So any application that uses SSH as a transport mechanism can see a performance boost. This isn't to say *every* user will. This is mainly geared towards high bandwidth delay product networks (greater than 1MB) or GigE LANs.

Re:Must be why rsync over ssh is much faster

[MichaelSmith (789609)]

There is definitely something funny(strange) about the way scp does bulk copies. It stops and starts. Other applications happily stream through encrypted ssh connections.

And in my experience rsync is faster.

Re:Must be why rsync over ssh is much faster

[drinkypoo (153816)]

If you just want to copy some files from system to system in an encrypted fashion, then the BEST option by far is to use tar, and pipe it through ssh like so:

tar cvfpz - * | ssh user@host '( cd /destination ; tar xvfpz - )

This example will compress and encrypt your data before sending it; on the other end, the file is streamed to tar. This example requires GNU rar or a close facsimile.

Now, if you want to UPDATE a directory, use rsync:

rsync -av -e ssh * user@host:/destination/

Because rsync will do partial checksums and send parts even of BINARY files if the whole file has not changed, and doesn't re-send unchanged files, rsync makes sense when updating a directory. But it provides no speedup benefit over using tar, and in fact the directory scans it does before the sync mean that it may actually be slower.

Use scp only for copying single files, because you're right, scp chokes between each file.

Re:

[ComputerSlicer23 (516509)]

tar cfpz - . | ssh user@host '( cd /destination ; tar xfpvz - )'

I'd use a "." instead of *, it avoids shell line length problems, and will also copy hidden files... as someone who as learned this the hard way. Also in my experience, on anything faster then 10MB, don't bother with compression (it's really a CPU to network speed ratio, on transfers I did regularly that was the rule of thumb with P4 2.2Ghz Xeons). Also, I removed the "v" from the source tar, as it duplicates every file name twice and can

Re:

[timeOday (582209)]

tar cpzf - * | ssh user@host " cd /destination ; tar xpzf - "

You don't need a quoted pair of commands, just use tar's -C option

ssh [e-mail] tar -C /remote/path -cpzf - remotefile1 remotefile2 | tar -C /local/path -xvzp -

Re:Must be why rsync over ssh is much faster

[AnyoneEB (574727)]

Unless your servers are running rsh, rsync is probably going to get routed through ssh, in which case it gets encrypted just like scp. ref [everythinglinux.org]:

Secure Shell - The security concious of you out there would like this, and you should all be using it. The stream from rsync is passed through the ssh protocol to encrypt your session instead of rsh, which is also an option (and required if you don't use ssh - enable it in your /etc/inet.d and restart your inet daemon if you disabled it for security).

Can't we all just get along?

[piojo (995934)]

Okay, it's very simple.

Encrypted and tunneled over SSH, rsync is spawned by a login shell at the other side:
rsync /some/path myhost.com:my/directory

Not encrypted, rsyncs daemon must be running at other end:
rsync /some/path rsync://myhost.com/my/directory OR
rsync /some/path myhost.com::my/directory

Alternative solution for a trusted LAN

[Diomidis Spinellis (661697)]

If you want to speed up transfers and you're working on a LAN you trust (i.e. you don't worry about the integrity and confidentiality of the data passing through it), you can dramatically increase throughput using socketpipe [spinellis.gr]. Although the initial socketpipe communication setup is performed through client-server intermediaries such as ssh(1), the communication channel that socketpipe establishes is a direct socket connection between the local and the remote commands. This eliminates not only the encryption/description overhead, but also the copying between your processes and ssh or rsh.

Re:Alternative solution for a trusted LAN

[upside (574799)]

You can also use a cheaper cipher. From the ssh manpage:

-c blowfish|3des|des
                          Selects the cipher to use for encrypting the session. 3des is
                          used by default. It is believed to be secure. 3des (triple-des)
                          is an encrypt-decrypt-encrypt triple with three different keys.
                          blowfish is a fast block cipher, it appears very secure and is
                          much faster than 3des. des is only supported in the ssh client
                          for interoperability with legacy protocol 1 implementations that
                          do not support the 3des cipher. Its use is strongly discouraged
                          due to cryptographic weaknesses.

Re:Alternative solution for a trusted LAN

[KiloByte (825081)]

Actually, it appears that (at least on Debian) AES is already the default. Selecting 3des gives tremendous slowdown; blowfish is somewhat slower than AES.

Copying 100MB of data over 100mbit ethernet to a P2 350Mhz box (the slowest I got) gives:
* 3des 1.9MB/s
* AES 4.8MB/s
* blowfish 4.4MB/s

Re:

[Neil Watson (60859)]

Actually, it depends upon the SSH protocol. Both Debian and Cygwin have this to say:

-c cipher_spec
Selects the cipher specification for encrypting the session.

Protocol version 1 allows specification of a single cipher. The

Re:

[HAKdragon (193605)]

I may be speaking out of ignorance, but doesn't that defeat the point of SSH?

Re:

[forkazoo (138186)]

I may be speaking out of ignorance, but doesn't that defeat the point of SSH?

SSH is one of those uberutilities that has a surprising amount of usefulness once you dig a bit. Sure, secure telnet functionality is great, and I use it a lot. But, I still use ssh on my own LAN where I don't really care about security. I use sshfs because it is easier and more convenient for me than bothering with Samba. SCP/SFTP to avoid bothering with ftp. I use it for forwarding ports between various machines, and I use i

Re:

[MoogMan (442253)]

Or use netcat/nc (installed by default on most Linux distros). Server cats it's output directly to a file (or to tar -x). Client grabs it's input from a file.

Server: nc -l 1234 | tar -x
Client: tar -c file_list_here | nc localhost 1234

Re:

[Diomidis Spinellis (661697)]

Nc is useful, but it still involves the overhead of copying the data through it (once at the client and once at the server). Nowadays, in most settings this overhead can be ignored. But, given the fact that a well-behaved application will work with a socket exactly as well as with a pipe or a file descriptor, I thought it would be very elegant to be able to connect two instances of (say) tar through a socket. Hence the implementation of socketpipe. Socketpipe sets up the plumbing and then just waits fo

How will this affect application deployment

[Zombie Ryushu (803103)]

How will this affect the operation of applications wich use SSH to standardie applications accross Linux like urpmi's --parallel parameter?

This is one of the most useful aspects of Mandriva, but as the number of nodes I have to manage increases, I find RPMS being SCPed to other nodes taking longer and longer. I think this is because even though with Kerberos Authentication is much faster, urpmi is waiting until one node finishes copying the files to start copying to the next node in the Domain.

Thoughts?

Sweet!

[miffo.swe (547642)]

I really hope this will make it into OpenSSH after some security auditing. The performance gains was pretty impressive. It will make ssh much more fun for rsync, backups and other times when i transfer large files. I also wonder if one cant get similar performance gains with normal ssh and for example forwarded X-windows. That would be very interesting indeed.

Re:Sweet!

[Per Wigren (5315)]

Use NX [nomachine.com] instead of plain old remote DISPLAY or ssh's X11 forwarding or even VNC! It's silly fast! You get a perfectly usable desktop even on slow, high latency connections. The free edition is free as in GPL.

To *have* such problems...

[pla (258480)]

the crypto can sometimes be the bottleneck instead of the wire speed.

Between two devices on my gigabit home LAN, the CPU barely even registers while SCP'ing a large file (and that with every CPU-expensive protocol option turned on, including compression). What sort of connection do these guys have, that the CPU overhead of en/decryption throttles the transfer???


Coming next week: SSH compromised via a thread injection attack, thanks to a "feature" that only benefits those of us running our own undersea fiber.

Re:To *have* such problems...

[dm(Hannu) (649585)]

They claim that the first bottleneck is actually flow control of buffers, which prevents utilizing full network bandwidth in normal gigabit connections. The threads will help only after this first bottleneck has been cleared. They have patches to fix both problems. The slashdot summary was therefore a bit inaccurate, and reading TFA certainly helps.

Re:To *have* such problems...

[egede (266062)]

The limitations of transfer rates for scp is often the round trip time that consumes time for confirmation of received packages. This is a serious issue for transfers from the Europe to the US West Coast (around 200 ms) or to Australia (around 400 ms). Having several parallel TCP streams can solve this problem and has been in use for many years for transfer of data in High Energy Physics. An example of such a solution is GridFTP http://www.globus.org/toolkit/docs/4.0/data/gridftp/ [globus.org].

Re:

[Andy Dodd (701)]

Older implementations of TCP only allow for a 64 KB window size. (Older meaning "really old" - nearly any implementation in the last decade implements extensions that allow for much larger transmit/receive windows.)

Many apps set fixed window sizes (incl. apparently standard SSH - the webpage implies 64K.)

Linux can "autotune" window sizes, but most OSes don't, hence the need for an app to be able to specify a larger window.

Even with larger window sizes, TCP congestion control starts breaking on networks wit

Re:To *have* such problems...

[totally bogus dude (1040246)]

Have you measured your actual throughput on the file transfer? It tends to take a crapload of tuning to get anywhere near saturating gigabit, even if you're not using encrypted transfers.

I wrote the bit below which I'll keep because it might be interesting to someone, but dm(Hannu) already mentioned the claw flaw in the logic behind the PP and article summary: if the CPU is the bottleneck, how could adding more threads possibly help?

Just for a laugh I used scp to copy a 512 MB file from my file server to itself, an Athlon 3700+ running at 2.2ghz. I got about 18 megabytes / second out of it. I took a snapshot of top's output right at the end (97% complete) and the CPU usage was as follows:

ssh: 48.6%
sshd: 44.9%
scp: 3.7%
scp: 1.3%
pdflush: 0.7%

So this system was pretty much pegged by this copy operation, and it achieved less than a fifth the capacity of a gigabit network link. Obviously the system is capable of transferring data much faster than this; the source was a RAID-5 set of 5 new 500 GB drives, and the destination was a stripe across two old 40 GB drives. I'd also repeated the experiment a few times (and this was the fastest transfer I got) so it's likely the source file was cached, too.

I do agree that there's probably more interesting and useful things to optimise (and make easy to optimise) than scp's speed, but I know for sure that scp'ing iso images to our ESX servers in a crapload slower than using Veeam's copy utility or the upload facility in the new version of Infrastructure Client (at least I think it's new, never noticed it before).

Re:To *have* such problems...

[mikael_j (106439)]

A possible problem source here is that you're also doing disk I/O, when transferring data on my home network I've noticed that rsyncing things for redundancy purposes I end up with a lot more CPU usage (even when reading from a RAID5 via a hardware controller) than if I just pump random data from one machine to another. I reommend you try just transferring random data and piping it directly to /dev/null on the receiving machine to see if there's any difference in CPU usage.

/Mikael

Linux kernel version 2.6.17 to 2.6.24.1

[arkarumba (763047)]

Preferably, we would like to test it any very high bandwidth systems running Linux kernels version 2.6.17 to 2.6.24.1.

Pretty much totaly incorrect summary

[Eunuchswear (210685)]

Almost all the improvements they talk about come from optimising TCP buffer usage. The summary of the fixes:

HPN-13 A la Carte

• Dynamic Windows and None Cipher
  This is a basis of the HPN-SSH patch set. It provides dynamic window in SSH and the ability to switch to a NONE cipher post authentication. Based on the HPN12 v20 patch.
• Threaded CTR cipher mode
  This patch adds threading to the CTR block mode for AES and other supported ciphers. This may allow SSH to make use of multiple cores/cpus during transfers and significantly increase throughput. This patch should be considered experimental at this time.
  
• Peak Throughput
  This patch modifes the progress bar to display the 1 second throughput average. On completion of the transfer it will display the peak throughput through the life of the connection.
• Server Logging
  This patch adds additional logging to the SSHD server including encryption used...

So the main part of the patch set is "It provides dynamic window in SSH and the ability to switch to a NONE cipher post authentication" and the only part that has to do with threading is marked "This patch should be considered experimental at this time".

By the way, does anybody else think "the ability to switch to a NONE cipher post authentication" is pretty dodgy?

Re:

[Arimus (198136)]

By the way, does anybody else think "the ability to switch to a NONE cipher post authentication" is pretty dodgy?

Not really, for some of the stuff I do via SSH: eg logging into my webhost to untar a patch and apply it the only part of the transaction I want to be secure is my initial password/key-exchange post authentication I really don't give a stuff who sees me type

cd ~/www
tar xvfz ~/patch.tar.gz

or any of the other commands I type in. However it should be down to the admin of the system in the first

Re:

[tbuskey (135499)]

By the way, does anybody else think "the ability to switch to a NONE cipher post authentication" is pretty dodgy?

I'd like it when I tunnel a new SSH or scp through another SSH tunnel. We call it a sleeve. I've had to sleeve within a sleeve's sleeve before to get through multiple SSH gateways and firewalls to an inner system. You can tell ssh to use XOR but I'm not sure you can in scp.

Of course, if speed is paramount, you can use netcat inside the sleeve(s) to copy files. No encryption of the netcat

Re:Pretty much totaly incorrect summary

[rapier1 (855437)]

As a note - while the NONE cipher switch turns off data encryption we do *not* disable the message authentication cipher (MAC). So each data packet is still signed and authenticated. If it detects any in transit modification of the packet the connection is immediately dropped.

Hardware acceleration

[AceJohnny (253840)]

I've been wondering, does there exist hardware accelerators usable by OpenSSL or GnuTLS? I work in embedded systems, and our chip includes a crypto and hash processor. I'm surprised nothing equivalent exists on modern PCs, or have I just not been looking in the right places?

News just in!

[j.a.mcguire (551738)]

Crowds were shocked to discover that multi-threaded software runs faster on a multi-core system, it was hard to contain the excitement over the discovery which could revolutionise software development!

Some comments from one of the authors

[rapier1 (855437)]

First off, thank you for taking the time to read down this far. There have been some very interesting and useful comments so far. Second, I need to point out that both Ben Bennett of PSC and Michael Stevens of CMU were instrumental in getting this patch written. Without them there would be no HPN-SSH patch. I also highly suggest that interested people go to the http://www.psc.edu/networking/projects/hpn-ssh [psc.edu] and read about what we've done. There is a lot of good material in the papers and presentations section as well as the FAQ.

A couple notes about the multi-threading: The main goal was to allow SSH to make use of multiple processing cores. The stock OpenSSH is, by design, limited to using one core. As such a user can encounter situations where they have more network capacity and more compute capacity but will be unable to exploit them. The goal of this patch was to allow users to make full use of the resources available too them. The upshot of this is that its best suited for high performance network and compute environments (The HPN in HPN-SSH stands for High Performance Networking). This doesn't mean it won't be useful to home users - only that they might not see the dramatic performance gains someone in a higher capacity environment might see. Its really going to depend on the specifics of their environment.
Based on our research we decided the most effective way to do this would be to make the AES-CTR mode cipher multi-threaded. The CTR mode is well suited to threading because there is no inter block dependency and, even better, the resulting cypher stream is indistinguishable from a single threaded CTR mode cypher stream. As a result, we retain full compatibility with other implementations of SSH - you don't need to have HPN-SSH on both sides of the connection. Of course, you won't see the same improvements unless you do.
We still see this as somewhat experimental because we've not yet implemented a way to allow users to choose between a single threaded AES-CTR and multi-threaded AES-CTR mode. As such users on single core machines - if using AES-CTR may see a decrease in performance. We suggest those users just make use of the AES-CBC mode instead (which is the default anyway). Also, you need to be able to support posix threads.
Future work will involve pipelining the MAC routine and that should provide us with another 30% or so improvement in throughput.

Also, its important to keep in mind that these improvements are *not* just for SCP but for SSH as a whole. People using HPN-SSH as a transport mechanism for rsync, tunnels, pipes, and so forth may also see considerable performance improvements. Additionally, the windowing patches don't necessarily require HPN-SSH to be installed on both ends of the connection. As long as the patch is installed on the receiving side (the data sink) you may (assuming you were previously window limited) see a performance gain.

We welcome any comments, suggests, ideas, or problem reports you might have regarding the HPN-SSH patch. Go the website mentioned above and use the email address there to get in touch with us. This is a work in progress and we are doing what we can to enable line rate easy to use fully encrypted communications. We've a lot more to do but I hope what we've done so far is of use and value to the community.

Re:

[raynet (51803)]

The lack of ASCII transfer mode is a good thing. A LATIN15 transfer mode might be handy. Less time wasted and less confusion and complaints amongst my clients.

Re:

[Mr Z (6791)]

BDP is the bandwidth-delay product. BDP is one of the main things these patches address. Loopback has very, very little delay. You could, I suppose, add artificial delay over loopback, but now you're diverging further from the actual deployment scenario.

The other thing is that when sender and receiver are the same host, you don't engage the full network stack (no ethernet queuing, for example, no dropped packets, etc. etc.), so you don't find out all the curve balls that TCP/IP will throw you.

And yet a