Multi-Threaded SSH/SCP

neo writes "Chris Rapier has presented a paper describing how to dramatically increase the speed of SCP networks. It appears that because SCP relies on a single thread in SSH, the crypto can sometimes be the bottleneck instead of the wire speed. Their new implementation (HPN-SSH) takes advantage of multi-threaded capable systems dramatically increasing the speed of securely copying files. They are currently looking for potential users with very high bandwidth to test the upper limits of the system."

Must be why rsync over ssh is much faster

[MichaelSmith]

I get a lot of use out of ssh for moving files around and rsync is definitely the best way to do heavy lifting in this area. Improving scp would be good to. I can't wait to hear what Theo thinks about this. I don't see him as a fan of adding complexity to improve performance.

Big scp copies through my wifi router used to cause kernel panics under netbsd current of about a year ago. I never had that problem running rsync inside ssh.

Re:A likely story

[slyn]

Makes you wonder how many innovations can either be directly attributed to or partially attributed to the distribution of porn (not (necessarily) that this is).

VHS v Betamax comes to mind.

Sweet!

[miffo.swe]

I really hope this will make it into OpenSSH after some security auditing. The performance gains was pretty impressive. It will make ssh much more fun for rsync, backups and other times when i transfer large files. I also wonder if one cant get similar performance gains with normal ssh and for example forwarded X-windows. That would be very interesting indeed.

Re:A likely story

[harry666t]

Get a /real/ girl.

Re:Must be why rsync over ssh is much faster

[cheater512]

Rsync doesnt encrypt. SSH/SCP does.

Rsync is only really useful as a synchronizing method between a source and a out of date copy.
Then its real benefits get shown.

Re:A likely story

[shenanigans]

I think this story is interesting because it shows a general trend: increased focus on multi-threading. We will see much more of this in the future as multi-core and multi-processor systems become more common. This trend is driven not by porn though, but by that other big driving force behind the computer industry, gaming.

Re:A likely story

[mikael]

People are always willing to pay more to be entertained that to be educated.

Which explains why football players and movie stars will get paid more than the innovators that carried out the research to develop the broadcast technology that helped to make those stars famous.

Re:sftp and ASCII mode transfers

[raynet]

The lack of ASCII transfer mode is a good thing. A LATIN15 transfer mode might be handy. Less time wasted and less confusion and complaints amongst my clients.

Re:A likely story

[Anonymous Coward]

People are always willing to pay more to be entertained that to be educated.

Which explains why football players and movie stars will get paid more than the innovators that carried out the research to develop the broadcast technology that helped to make those stars famous.

Another big issue is that there are fewer stars than researchers. As such, it is more difficult to replace any given star than to replace any given researcher. The star does a greater portion of the work than the researcher (even if all the stars do a smaller portion than all the researchers).

Replace a star NFL quarterback with his backup, big drop in performance of the team. Replace a top researcher with a new hire, a big drop in performance of her immediate team but the overall group of teams is impacted only slightly.

Multithreading waaay to go...

[Crass Spektakel]

Today you can buy a machine with eight cores, 8gb memory and 1tb harddrive for less than 2000. And most software will only use one core and a maximum of 2gb memory.

WE NEED MULTITHREADING NOW BIG AND EVERYWHERE.

Multithreading is maybe the biggest change in software development. In contrast to advanced command sets like MMX, SSE and so on it is not about some peep hole optimization, about replacing a bunch of x86_32 commands with some SSE commands, it is about changing the whole approach, finding new algorithms and redevelop much if not all software we are used to work with.

And we are lightyears away from using appropiate software on on todays systems.

See compressors: Most can't multithread at all. Those who can have more issues than the SCO buisness plan, eg reduced efficency, scale very bad, can't accept input from stdio or can't output on stdout. Basically compression should be a good starting point but even here todays solutions are incredible far behind the hardware.

Also consider network communication, graphical interfaces, games, printing, non-enterprise realtime presentation and so on and so forth.

The revolution is NOT here. People aren't even talking about it.

Re:Pretty much totaly incorrect summary

[tbuskey]

By the way, does anybody else think "the ability to switch to a NONE cipher post authentication" is pretty dodgy?

I'd like it when I tunnel a new SSH or scp through another SSH tunnel. We call it a sleeve. I've had to sleeve within a sleeve's sleeve before to get through multiple SSH gateways and firewalls to an inner system. You can tell ssh to use XOR but I'm not sure you can in scp.

Of course, if speed is paramount, you can use netcat inside the sleeve(s) to copy files. No encryption of the netcat, but it's inside an encrypted sleeve so the stream is encrypted.

Re:FUNNY?! That's not funny, try for TRUE

[Anonymous Coward]

I won't lie to you -- being a parent is no laughing matter. It is a ton of work. It can be amazingly stressful and expensive. I've been through periods that I look back on now and wonder how the hell I managed to pull through without going completely insane. But if you ask me, the rewards outweigh the difficulties ten to one.

When your child first looks up at your face and you see actual recognition in her eyes... when you see all the blocks fall into place as she figures out how to do something for the first time... look, I know it sounds really sappy and smarmy, but seriously (srsly) it is absolutely indescribable. This thing started out as a bit of genetic code from two people, and now it is actually self-aware and sentient. How cool is that? What geek can't be astonished at these emergent properties, derived from a program more complicated than you can possibly imagine -- a program that has spontaneously evolved over time?

And you get to see her mental map evolve. You watch branches get added to her decision tree. You observe as she learns how to acquire information, process it, and decide how to act upon it. And all the while, you mold her view of the world based on your interactions with her. I don't know about you, but I find that not only fascinating, but incredibly rewarding.

Before my daughter was born, I was terrified too, and somebody had said these things to me, I would've said, "Yeah, okay, I'm sure it's great and all, but I'm sure you're exaggerating somewhat." That's because there is something that happens to you when it's your kid. There's some very ancient, very basic code that gets turned on in your brain that says "this life is your responsibility, and you must do everything you can to ensure its safety, survival, and growth". I can't explain it because I honestly believe it's something buried deep beneath the conscious mind.

Whatever the case, if you honestly don't want the baby, for it's sake, put it up for adoption. Don't make it live a life with a father who doesn't care for it. I'm being absolutely serious here. Find a loving couple who are unable to have kids of their own.

(Posting AC because this is way offtopic, and because there are a lot of single, selfish, bitter child-haters out there with mod points to burn... but I had to say something.)

Re:A likely story

[rapier1]

Actually, this is one of the main reasons why we did this. If you look at where processor architectures are going they aren't necessarily increasing the pure computational power of each core as much as they are using more and more cores. If you are in a situation where you have a single threaded process that is CPU bound - like SSH can be - you'll find that throughput rates (assuming that you aren't network bound) will remain flat. In order to make full use of the available network capacity we'll have to start doing things like this. There is still a lot of work to be done but we're pleased by our progress so far.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Re:FUNNY?! That's not funny, try for TRUE

[sgbett]

There's also other geek parents out there that know exactly where you are coming from. well said.

Re:A likely story

[jovetoo]

It is rare that you can completely separate every context of every step of your processing. There is always some data that needs to be shared between the threads and they become bottlenecks. The faster you serve your requests, the worse the contention (waiting for a resource) and thus the inefficiency.

It depends on the task at hand and on your architecture. A file or web server is less likely to encounter contention than for example an IRC server. The first requires some authentication and resource resolving through configuration data but the actual data can be send without interference from other requests. An IRC server requires constant lookups in the user database for routing information and this is likely to take longer than actually sending the messages (even without multi-threading). In these cases, you really have to think your locking scheme through or you will lose more time waiting for a lock than doing actual work - defeating much of the purpose of going MT.

When it comes to architecture, multi threading is an option in your architecture, not an architecture in itself. There is no problem doing a multi-threaded event-driven architecture or a MT message passing architecture -- these are actually very effective. For some interesting reading about this, I would suggest you check out the SEDA white paper [harvard.edu] for a pretty in depth list of options and their goals.

Why is it bad for programmers? Because locking is hard to do in itself and if your locking scheme is subobtimal it often requires a lot of work to change it afterwards.

Re:FUNNY?! That's not funny, try for TRUE

[empaler]

Yeah. Turns out 0.8% risk of pregnancy isn't as small as I thought it was.