What is the actual situation in technology concerning parallel computing ?
Parallel computing was once hailed as the road to unlimited processing power - anyone remember ‘transputers’? - these were going to be modular computers that you could just plug together like Lego and the more you had, the more processing power you’d be able to use.
Trouble is, with real-time computing, parallel processing runs up against the problem of information interchange - in order to do its job, each of the processors needs to intimately know what all the others are doing and the communication overhead starts to outweigh any benefit gained from extra actual processing ability.
However, for operations such as graphics rendering or statistical analysis, that can be broken down into a number of self-contained operations, parallel processing works really well and is in common use - this can either take the form of dedicated farms of machines, or distributed computing, like the SETI@home project.
Right now, there’s a question as to exactly what problems have efficient parallel algorithms to solve them. It’s possible that every problem that can be solved reasonably quickly has an even better parallel solution, but it’s not known for sure.
For the real geeks out there, the question is whether P = NC.
it works for some things, not for others. and sometimes to diffrent degrees.
haveing 2 people makeing a cake is faster than makeing one… but use 256 to make a cake and the organization overwelms the speedup and you make a mess.
or as my teacher says “you can’t always make a baby in one month by haveing sex with nine women”
Mangetout’s post is really great. I just want to add my two bits.
At the low end-your own PC-there hasn’t been much going on in parallel CPUs. The reasons for this are complex. I like to blame Intel and MS. Neither “gets” parallel processing. People have known for 20 years how to design multiprocessor-friendly CPUs and Intel is still stuck in the dark ages in CPU design. They are just now starting to try to commercialize multithreaded CPUs. That’s being behind the times and then some. (They are also doing it wrong.) MS OSes have just flat out been primitive, behind-the-times crap. There’s no reason they couldn’t have done true multiprogramming on 8088s, but never did. They just don’t “think parallel” in their designs. If you can’t handle multiprogramming well, forget doing multi-CPUs entirely.
There has also been this maddening “upgrade mania” in the PC world. MB makers can just barely keep up making single processor boards. 2 and 4 processor boards don’t stay “modern” long enough to have long enough economic lifespans so they can come down in price to really pay off. If you designed a 64 processor Intel CPU MB, you would never make your investment back. By the time it was stable enough for market release, it would be using out of date CPUs and such.
However, if you pry off the lid of your PC and really check into things, there really are several processors in it. A couple of nice processors for handling buses, a very powerful processor on the graphics card, etc. I’ve even seen 8088 processors on pre-winmodem modems.
But the mentality is still “the user is only running one task at a time”. As opposed to doing a “ps” on a Unix box and seeing 30+ processes running.
Second issue: The right way to use multiple processors. As Mangetout points out, even if the task could be parallelized (many can’t), you usually run into i/o limits. The right way to use 100 processors is to run 100 different tasks on them. Somehow people refuse to think this way. They insist that the first task get split into 100 parts, run in parallel, and when it’s finished the 2nd task gets split into 100 parts, etc. This is just plain stupid.
Third issue: The “parallel computing is for number crunching” myth. The overwhelming majority of computing is for for simple data processing. Moving data from point a to point b. Comparing 2 strings. Searching for a record. Etc. But everyone thinks multiprocessors = supercomputers = number crunchers. (2 bad steps of logic.) Forget number crunching. That’s a small niche market. Ordinary business data processing needs far more computing power. But that’s not “sexy”. (No, I don’t get it either.) So hardly anyone is building large scale multiprocessors for the majority of people who need them. That’s explains a lot of stagnation right there.
So, answering the OP on state of technology: it’s been there for decades. The issue is poor state of human thought.
Good point; I remember seeing a Z80 processor in something recently (I think it was a network card, but I can’t remember).
Could you give us a bit more to go on here? Are you interested strictly in symmetric shared-memory multiprocessing? Distributed processing? MIMD? SIMD?
If you’ll accept distributed architectures, then consider Virgina Tech’s 1100 node G5 cluster (http://www.infoworld.com/article/03/09/22/HNvatech_1.html). I imagine the granularity of the problem division is rather large to make up for communication delays, so you have to pick algorithms that are nicely separable.
I haven’t been paying all that much attention to what’s going on in the world of shared-memory multiprocessing lately, but a lot of the commercial ventures that were offering them seem to no longer be around. Real shared memory computers need a lot of specialized hardware so they have had a tough time keeping up with Moore’s Law – by the time you’ve built an architecture for connecting a bunch of processors @X Mhz in parallel, Intel or IBM has come out with a chip that’s almost X times as fast without the overhead of parallelism. Not to mention that the very fast clock speeds require very fast bus speeds which makes sharing memory increasingly problematic.
-
-
- Parallel computing works great, if the software is designed to operate with parallel computers and the problem can be properly split: ideally you would want not one big problem, but a lot of little problems that can each be solved separately. Such as, suppose you had a list of one million numbers, and you needed to factorize all those numbers. You see that for however many processors you have, you have one processor that “passes out numbers” to the others when requested and stores the answers, and each of all the others can take one number, take however long it needs to for factorizing it, and then return the result and get another number to work on–and not require results from, or interfere with any of the other processors. In reality, most computing tasks are not like this; they are a bunch of interlinked problems that need to be solved in a specific order, like a spreadsheet. So until software is written in such a way as to identify separate problems, you aren’t gong to see much (if any) improvement in desktop PC’s with multiple processors. Apple makes a big deal out of some of their computers having dual CPU’s, but these computers do not run twice as fast as their single-CPU counterparts, and the things that Apple computers do faster than PC’s don’t have much to do with a couple of the Apple computers having dual CPU’s.
-
- So basically, for at least the next ten years or so, it will be hype.
- And by the by: Apple did not invent desktop computers with multiple CPU’s. A long time ago there used to be a few companies making PC motherboards that supported multiple 486 and 586 processors. I remember a few duals, and one quad. I also remember that you had to run Solaris on these machines because Windows did not support multiple CPU’s, and Linux wasn’t around yet–and even after all that, they weren’t real good for much, because they needed special software (that you basically had to write yourself) to show any benefit of multiple CPU’s.
~
One of the new-old ideas in multiprocessing is the Beowulf Cluster, which is simply harnessing multiple low-to-midrange PCs into a single entity that has the advantages of multiple CPUs, disk drives, network links, and, maybe, terminals. It aims to Put A Supercomputer In Every Garage and a Teraflops In Every Quake System and it has attracted quite a bit of geeky attention in the homebrew *nix world, to the point of creating a massive running joke on Slashdot.
(The idea of jamming multiple physical drives into a single logical drive is called RAID (Redundant Arrays of Inexpensive Disks) and it’s been around on high-end PCs and low-end workstations and the like for a while. Basically, each disk write happens to multiple physical disks at once. That way, if a write fails on one disk (distressingly common, if it’s a business-related thing (cf. Murphy)), it probably succeeded on a different one. A head crash moves from the realm of Oh Shit I’m Dead to Damn, I’m Out $200.)
Anyway, Beowulf Clusters don’t solve any of the problems associated with multiprocessing (to wit: Communications/synchronisation and algorithm parallelization) but they do provide a new way of looking at PCs: Single-celled beings that may one day aspire to become slime mold. 
I’m one of several administrators working with a common multi-terabyte Oracle database running on several different computers simultaneously. In total there’s over 60 individual processors spread across several separate computers.
One problem, indeed, is the coordination of all the processes between the different computers. The application, it turns out, spends a lot of time just flopping about the cluster trying to be a clustered application rather than actually performing the meat-and-potatoes of processing the actual data.
It’s a problem to the point that, later this year, we’re looking at de-clustering the database and moving to a larger, more monolithic architecture.
When a process can be broken down well and the data interchange between elements can be small in comparison to the processing within the elements, then parallel processing might be a good option. A good example would be finite element analysis.
There are a whole bunches of levels of parallelism, which have been mostly mixed in these responses. (Though less in ftg’s response.) Most people have discussed parallelizing a big computation. That’s still done in a lot of places, but isn’t quite as exciting as processor speeds have exceeded those of vector supercomputers.
All modern processors have multiple arithmetic units, and pipelines. So, the processor you are running is executing several instructions at the same time, and is likely doing several computations in parallel, assuming the code you are running needs it. This is partially where all those transistors go (the other place is the cache.) In addition, my old PDP 11/20 had to do print spooling itself. Today even a cheapo printer has a processor and enough memory to handle print jobs in parallel with the CPU in your processor. Your home PC is a machine with an impressive number of jobs running in parallel.
Then we have workstations with multiple CPUs. I’ve had a few - in one place I worked my second CPU was reserved for distributed simulation jobs run by other people. Instead of one CPU swapping between jobs, you’ve got two.
Finally, the big trend in processor design is to have one chip with multiple CPU cores, each of which is running an independent thread. This is a good way of using up transistors without having to spend a lot of design time.
Bottom line, parallelism is so common you don’t even notice it anymore.
“can’t always”? Perhaps you might ask your teacher when this would ever work?
I say we need to do some research in this matter. I’ll volunteer to make the attempts, if nine suitable subjects can be found.