Multi-Threaded SSH/SCP

neo writes "Chris Rapier has presented a paper describing how to dramatically increase the speed of SCP networks. It appears that because SCP relies on a single thread in SSH, the crypto can sometimes be the bottleneck instead of the wire speed. Their new implementation (HPN-SSH) takes advantage of multi-threaded capable systems dramatically increasing the speed of securely copying files. They are currently looking for potential users with very high bandwidth to test the upper limits of the system."

Alternative solution for a trusted LAN

[Diomidis Spinellis]

If you want to speed up transfers and you're working on a LAN you trust (i.e. you don't worry about the integrity and confidentiality of the data passing through it), you can dramatically increase throughput using socketpipe [spinellis.gr]. Although the initial socketpipe communication setup is performed through client-server intermediaries such as ssh(1), the communication channel that socketpipe establishes is a direct socket connection between the local and the remote commands. This eliminates not only the encryption/description overhead, but also the copying between your processes and ssh or rsh.

Re:Must be why rsync over ssh is much faster

[MichaelSmith]

There is definitely something funny(strange) about the way scp does bulk copies. It stops and starts. Other applications happily stream through encrypted ssh connections.

And in my experience rsync is faster.

To *have* such problems...

[pla]

the crypto can sometimes be the bottleneck instead of the wire speed.

Between two devices on my gigabit home LAN, the CPU barely even registers while SCP'ing a large file (and that with every CPU-expensive protocol option turned on, including compression). What sort of connection do these guys have, that the CPU overhead of en/decryption throttles the transfer???


Coming next week: SSH compromised via a thread injection attack, thanks to a "feature" that only benefits those of us running our own undersea fiber.

Re:To *have* such problems...

[totally bogus dude]

Have you measured your actual throughput on the file transfer? It tends to take a crapload of tuning to get anywhere near saturating gigabit, even if you're not using encrypted transfers.

I wrote the bit below which I'll keep because it might be interesting to someone, but dm(Hannu) already mentioned the claw flaw in the logic behind the PP and article summary: if the CPU is the bottleneck, how could adding more threads possibly help?

Just for a laugh I used scp to copy a 512 MB file from my file server to itself, an Athlon 3700+ running at 2.2ghz. I got about 18 megabytes / second out of it. I took a snapshot of top's output right at the end (97% complete) and the CPU usage was as follows:

ssh: 48.6%
sshd: 44.9%
scp: 3.7%
scp: 1.3%
pdflush: 0.7%

So this system was pretty much pegged by this copy operation, and it achieved less than a fifth the capacity of a gigabit network link. Obviously the system is capable of transferring data much faster than this; the source was a RAID-5 set of 5 new 500 GB drives, and the destination was a stripe across two old 40 GB drives. I'd also repeated the experiment a few times (and this was the fastest transfer I got) so it's likely the source file was cached, too.

I do agree that there's probably more interesting and useful things to optimise (and make easy to optimise) than scp's speed, but I know for sure that scp'ing iso images to our ESX servers in a crapload slower than using Veeam's copy utility or the upload facility in the new version of Infrastructure Client (at least I think it's new, never noticed it before).

Re:To *have* such problems...

[mikael_j]

A possible problem source here is that you're also doing disk I/O, when transferring data on my home network I've noticed that rsyncing things for redundancy purposes I end up with a lot more CPU usage (even when reading from a RAID5 via a hardware controller) than if I just pump random data from one machine to another. I reommend you try just transferring random data and piping it directly to /dev/null on the receiving machine to see if there's any difference in CPU usage.

/Mikael

Hardware acceleration

[AceJohnny]

I've been wondering, does there exist hardware accelerators usable by OpenSSL or GnuTLS? I work in embedded systems, and our chip includes a crypto and hash processor. I'm surprised nothing equivalent exists on modern PCs, or have I just not been looking in the right places?

Re:To *have* such problems...

[totally bogus dude]

True enough, but my main point was that getting to actual gigabit speeds in the first place is actually pretty difficult. Plus, I couldn't find an easy way to copy only X amount of "random" data via scp which was the point of the article. Regardless, copying data is rarely if ever a useful thing to do with scp, anyway.

Re:Hardware acceleration

[htd2]

There are a number of vendors who supply PCI/e/x/etc crypto accelerators. However these are mostly targeted at servers where the overhead of serving up numerous encrypted streams of data is much higher than on a client PC.

The Sun T1 and T2 processors in the T2000 and T5000 also have onchip crypto units 1 on the T2000 and 8 on the T5000 which accelerate OpenSSL traffic by offloading DES, AES, MD5 etc.

Old news?

[Damocles the Elder]

Digging around for the best way to apply the patch without screwing up my portage updates, I came across a request for this to be merged into the portage back in 2005, and is apparently usable with the HPN useflag.

Not that I'm that surprised to see this is old news, since they're apparently on major revision 13...

Re:A likely story

[Anonymous Coward]

...what's wrong with looking at porn with a real girl?

Re:Alternative solution for a trusted LAN

[HAKdragon]

I may be speaking out of ignorance, but doesn't that defeat the point of SSH?

Can't we all just get along?

[piojo]

Okay, it's very simple.

Encrypted and tunneled over SSH, rsync is spawned by a login shell at the other side:
rsync /some/path myhost.com:my/directory

Not encrypted, rsyncs daemon must be running at other end:
rsync /some/path rsync://myhost.com/my/directory OR
rsync /some/path myhost.com::my/directory

Re:Must be why rsync over ssh is much faster

[drinkypoo]

If you just want to copy some files from system to system in an encrypted fashion, then the BEST option by far is to use tar, and pipe it through ssh like so:

tar cvfpz - * | ssh user@host '( cd /destination ; tar xvfpz - )

This example will compress and encrypt your data before sending it; on the other end, the file is streamed to tar. This example requires GNU rar or a close facsimile.

Now, if you want to UPDATE a directory, use rsync:

rsync -av -e ssh * user@host:/destination/

Because rsync will do partial checksums and send parts even of BINARY files if the whole file has not changed, and doesn't re-send unchanged files, rsync makes sense when updating a directory. But it provides no speedup benefit over using tar, and in fact the directory scans it does before the sync mean that it may actually be slower.

Use scp only for copying single files, because you're right, scp chokes between each file.

Re:A likely story

[dodobh]

There's also message passing and event driven programming, which can be a much simpler model if done right. Multi threading tends to shared state, and that's bad for programmers.

Re:Alternative solution for a trusted LAN

[forkazoo]

I may be speaking out of ignorance, but doesn't that defeat the point of SSH?

SSH is one of those uberutilities that has a surprising amount of usefulness once you dig a bit. Sure, secure telnet functionality is great, and I use it a lot. But, I still use ssh on my own LAN where I don't really care about security. I use sshfs because it is easier and more convenient for me than bothering with Samba. SCP/SFTP to avoid bothering with ftp. I use it for forwarding ports between various machines, and I use it for forwarding X sessions. There are surely ways to accomplish all the same stuff that I use ssh (and the closely related stuff like scp) for without using ssh, but I'd just rather not bother about it.

Defeating the point of ssh is like defeating the point of a morning star. No matter how dull you make the point, the other 99 of them will still clobber you in the face.

Re:FUNNY?! That's not funny, try for TRUE

[Anonymous Coward]

Sure, it's amazing during the formative years... but later on down the line, when that nurturing environment that made your child into a healthy, independent, and intelligent adult lets him come to his own conclusions that he should vehemently support that which you equally vehemently oppose. When the loving adoration for daddy turns into a sort of pitiful contempt because of some religious, philosophical, or political variable.

Can't see how anyone could feel like anything but a failure in that spot.

Re:FUNNY?! That's not funny, try for TRUE

[empaler]

Thank you for your very well-written reply, but I wasn't actually being all that serious. (No, the pregnancy was unplanned, and I am actually opposed to the idea of becoming a parent at my current place in life; however, after talking this over with the only other person to have any say in this (the mother), I've decided to go with it).

Don't mistake my badly crafted joke for being completely ignorant of what's ahead of me; before the final decision came, I had consulted with friends who are also parents (carefully not discussing this with any of my single, singlemindedly free-roaming friends), and I am in no way in doubt that I will make this child a net benefit for the human race. There are simply too many rotten parents, spoilt children, miserable families and bad genes in the world for me to actually fail in that respect.
Plus, living in Denmark*, the baby will have pretty good odds for a good life, my involvement notwithstanding.

I am going to have a lot of fun making tech projects for my little one when that time comes, including audio books with his/her favourite bed time stories, video diaries of how the child evolves, and of course, teaching how to solder before the age of 5. How I survived until 15 without that knowledge eludes me to this day.

*: Studies have shown that there is a tie for Country With Best Quality of Life; Denmark and Iceland. I've been to Iceland, and it smelled like rotten eggs. Denmark takes the lead.