Magical World of Parallel Computing

This is part-2 of the "EnchantingProgram" post, read this post first: http://magical-parallel-computing.blogspot.in/2017/04/a-simple-python-program-using.html So, let's see the actual reason of the speedup in the C and C++ programs. Lo and Behold, it is the effect of Branch Prediction ! Surprised? Well, at least my comments in the programs should have given you some direction!!! The if condition leads to a branch in the control flow. We know that branch predictions lead to pipeline flushes and create a delay in the piped execution scheme. Modern microprocessors utilize complex run-time systems for throughput, execution speed and memory efficiency. One example of such a technique is dynamic branch prediction. A long time ago, microprocessors used only a basic technique called static branch prediction, with two general rules: A forward branch is presumed to be not taken A backward branch is presumed to be taken Now, static branch p...

Parallel is always better than Serial... right? NO Well, in the most general case, parallel computing is better than serial computing in terms of speed and throughput. Sometimes, we have to make other considerations too. As a comparison , consider computer networking where serial transmissions are straight-forward and faster than their parallel SCSI  counterparts! Some processes are inherently not parallelizable, due to presence of data dependency. (Two bank account withdrawals from different locations which may lead to negative account balance if done simultaneously! Anyway, such a pair of withdrawals with critical section management using semaphores/mutexes conceptually and momentarily reduces to serial execution...) On a lighter note, the process of Sneezing and keeping your eyes open is not parallelizable for example! Before jumping into the concepts and principles of parallelizing a given task, let us go through some interesting set of ( controversial ...

A magical world where clever algorithms meet elegant multitasking models! Suppose you were given the following open-ended problem: Given an array of n bits , perform a negation operation on each of the bits. The most obvious brute force approach would be to go from index 0  to index n-1  and invert the  i th  bit on the way. Well, this works in a clean way and is the simplest formulation which delivers correct results. Nice! Now suppose you have an array of about a million digits; our algorithm directs us to go through each of the elements serially, one by one, giving each bit a feeling of self-importance! (The CPU dedicates few clock cycles exclusively on accessing, processing and finally writing out the result). Can we make this faster? Can we use the fact that an operation on the  i th  bit is independent of  the operation on the ( i+1) th  bit (or any other bit in general)? Of course we can! Imagine a switchbo...

The magical world of parallel computing awaits you! We all know Serial computing using our favourite mainstream programming language( C, C++, Java, Python ) on a single faithful CPU with a module of RAM of a standard PC, one machine instruction after another. (Don't fret even if you don't! We'll discuss concepts gradually transitioning from serial processing to concurrency to parallelism and beyond!) Now, suppose we have a 2.0 GHz single core CPU, so it is capable of about 2 billion instructions per second. Compare that to Human calculation time, about 2 seconds per instruction !!! (assume simple addition or relocation instruction) What if we want more than that? Just visit the closest computer hardware store and buy another PC with: n processors with  m  cores, x.yz GHz Superscalar architecture,  Hyper threading, etc r GB  GDDR5 RAM and s GB SSD  Integrated Graphics and Discrete Graphics card  with abcd GPGPU cores (Wauw, that...