Parallel and Distributed Computing

AP CSP Big Idea 4.3: Parallel and Distributed Computing

By J. Moore

Introduction

Imagine a huge task, like analyzing billions of data points or rendering a complex animation. These tasks can take a very long time on a single computer. That's where parallel and distributed computing come in. They involve using multiple computers or processors to work together, significantly speeding up the process.

Key Concepts

●      Sequential Computing: Traditional computing where operations are performed one after another.

●      Parallel Computing: A method where a problem is broken down into smaller, independent parts that are executed simultaneously by multiple processors within the same computer.

●      Distributed Computing: A method where a problem is broken down into smaller, independent parts that are executed simultaneously by multiple computers connected over a network.

●      Okay, imagine you have a big job to do, like cleaning your whole house.

• Sequential Computing: This is like cleaning one room at a time. You finish the living room, then you move to the kitchen, then the bathroom, and so on. One thing happens after the other.

• Parallel Computing: This is like having your family help you clean. While you're vacuuming the living room, your brother can be dusting the kitchen, and your sister can be cleaning the bathroom all at the same time in the same house.

• Distributed Computing: This is like having your friends in different houses help you with a giant project. One friend might work on the introduction, another on the middle part, and another on the conclusion at the same time, and then you all put it together.

In short:

• Sequential: One thing at a time.

• Parallel: Many things at once, in the same place.

• Distributed: Many things at once, in different places.

Key Concepts

●      Speedup: The improvement in execution time achieved by using parallel or distributed computing compared to sequential computing.

●      Scalability: The ability of a system to handle larger workloads by adding more processors or computers.

●      Overhead: The extra time and resources required to manage and coordinate parallel or distributed computing, such as communication between processors/computers.

Let's go back to cleaning the house:

• Speedup: Imagine it takes you 3 hours to clean the house by yourself (sequential). If your family helps (parallel), and it only takes 1 hour, the speedup is that it's 3 times faster! It's how much quicker you get the job done by working together.

• Scalability: Let's say your house gets bigger! Scalability means that if you need to clean a bigger house, you can just add more helpers (more family members or even friends in different houses - distributed). The system can handle more work by adding more workers.

• Overhead: When everyone is helping, you need to tell them what to do, make sure they don't clean the same thing twice, and maybe even provide cleaning supplies. This extra effort of organizing and coordinating everyone is the overhead. It's the extra work needed to make the teamwork happen.

In short:

• Speedup: How much faster you finish the job by working together.

• Scalability: How easily you can handle a bigger job by adding more workers.

• Overhead: The extra effort needed to organize the teamwork.

●      Parallel Computing: A modern computer with a multi-core processor rendering different parts of a video game scene simultaneously.

Examples

Imagine playing a video game on a computer with a super-powered brain that has many "mini-brains" inside (that's a multi-core processor).

When you see a scene in the game with lots of things happening at once – like characters moving, explosions, and the background – a parallel computer (your computer with the multi-core processor) can have each of those "mini-brains" work on drawing a different part of the scene at the exact same time.

Instead of drawing the whole picture bit by bit, it's like having multiple artists each drawing a different piece of the same picture all at once, making the whole picture appear much faster and smoother.

So, parallel computing in a game means the computer's many "mini-brains" work together to quickly draw all the different parts of what you see on the screen at the same time.

●      Distributed Computing: A search engine like Google using many servers to process search queries from millions of users at the same time.

Examples

●      Weather forecasting: Complex simulations are run on supercomputers using parallel processing to predict weather patterns.

Examples

Imagine trying to predict if it will rain tomorrow. To do that accurately, you need to look at a lot of things happening in the air right now all over the place – the temperature, the wind, how much moisture is in the clouds, and so on.

Without parallel processing (like a regular computer): It would be like one person trying to measure the temperature of every single tiny bit of air across the whole country, one after the other. It would take forever!

With parallel processing (like a supercomputer): It's like having thousands of super-smart helpers all over the country taking measurements at the exact same time. Each helper focuses on a small area, gathers their information, and sends it back to the main weather center.

The supercomputer then takes all that information gathered at the same time and uses it to quickly run its complex calculations to predict the weather.

So, for weather forecasting, parallel processing lets supercomputers look at many different pieces of the puzzle happening simultaneously across a huge area, allowing them to make predictions much faster and more accurately than if they had to do everything one step at a time.

Open Ended

1.     Explain the difference between parallel and distributed computing, and provide an example of each.

Open Ended

1.     Discuss the factors that can limit the speedup achieved by parallel or distributed computing. *use internet to look up specific factors

Multiple Choice

Which of the following is an advantage of parallel computing?

• Increased complexity

• Reduced memory usage

• Faster execution time

• Lower cost

Multiple Choice

In which computing method do multiple computers work together over a network?

• Sequential computing

• Parallel computing

• Distributed computing

• Quantum computing