Term 2 2020 MAIN TOPICS IN THE LAST LECTURE (MAINLY IN BEN-ARI - - PowerPoint PPT Presentation

1

Different Concurrency Paradigms and Constructs for Distributed Systems Dr Vladimir Z. Tosic Term 2 2020

MAIN TOPICS IN THE LAST LECTURE… (MAINLY IN BEN-ARI CHAPTER 10)

2

• Some “Big

ig Ideas” about concurrency in distributed systems (and wider)

• Ben-Ari’s distributed

ributed system stem model (remember these assumptions!)

• Ric

Ricart-Ag Agra rawala wala algo lgorit ithm hm for dist istrib ributed uted mutu tual al exclusion n – DM DME (distributed critical sections)

• Token-pa

passi ssing algo lgorit ithms hms for distributed mutual exclusion – another Ricart-Agrawala algorithm

MAIN TOPICS IN THIS LECTURE… (NOT IN THE BEN-ARI TEXTBOOK! )

3

• Ricart-Agrawala algorithm demo in DA

in DAJ

• Revision of message-passing using CSP channels
• The actor
• r model for message-passing concurrency
• Brief overview of some other dist

istrib ribute uted d messag age- passing ing and dist istrib ribute uted d shared d memory

• ry paradigm

igms

• Notes on some other concepts for programming

asynchron chronou

• us

s dist istrib ribute uted d syste stems ms

REVISION AND DAJ DEMO OF RICART-AGRAWALA ALGORITHMS

4

From Chapter 10 and Appendix D.3 in Ben-Ari’s Textbook and DAJ docu cumentat entation ion

RICART-AGRAWALA ALGORITHM FOR DME – MAIN IDEAS (REVISION)

5

• Using tick

icket et numbers ers, similarly to Lamport’s bakery algorithm

• Nodes choose ticket numbers and compare them
• In a distributed system these numbers cannot be compared

directly, so they have to be sent t in in messag sages es

• Node with low

lowest t number er can enter CS (critical section)

• Other nodes have to wait until CS is free again
• Vir

irtua ual l queue does not exist as a data structure, but it is the effect of messages – nodes ordered as if in a queue

RICART-AGRAWALA ALGORITHM – COMPLETE (1/3)

6

RICART-AGRAWALA ALGORITHM – COMPLETE (2/3)

7

RICART-AGRAWALA ALGORITHM – COMPLETE (3/3)

8

RICART-AGRAWALA ALGORITHM DEMO IN BEN-ARI’S DAJ TOOL

9

Read about DAJ: https://github.com/motib/daj (URL), Appendix D.3 in textbook Task for you: Experiment with DAJ

TOKEN-PASSING ALGORITHMS FOR DME – MAIN IDEAS (REVISION)

10

• Token is permission to enter CS – only 1 node has token at any 1 time

so mutua tual l exclus lusion ion holds

• Effic

ficiencies iencies: token not passed unless necessary (contin tinge gency ncy), only 1 message needed to transfer token between nodes, node with token can enter CS multiple times without token transfer

• Ch

Chall llenges: ensuring freedom from deadlock and starvation

• In Ricart-Agrawala token passing algorithm, token includes array

grante nted with ticket number of each node the las last tim ime it had permission to CS – enables determining what outstanding request st messages have not yet been satisfied

REVISION OF SYNCHRONOUS VS. ASYNCHRONOUS COMMUNICATION

From Chapter 8 in Ben-Ari’s Text xtboo book

11

SOME IMPACTS OF THE DISTRIBUTED COMPUTING CONTEXT (REVISION)

12

• In compl

plex, x, decent ntrali ralised system stems (e.g. on the Internet)

• message passing has advantages over shared memory
• asynchronous communication has advantages over

synchronous communication

• immutable data has advantages over mutable data
• …
• Flex

lexibil ibilit ity is needed because in distributed systems change is frequent and often unpredictable

• caused by technology or by business aspects

SYNCHRONOUS VS. ASYNCHRONOUS COMMUNICATION (REVISION)

13

• Synchron

hronou

• us – communication cannot happen until both

parties are ready at the same tim ime, e.g. telephone call

• Asynchr

chron

• nou
• us – parties need not be ready at the same time,

e.g. email; requires storing data elements

• Shared memory – asynchronous
• Message passing – synchronous or asynchronous

SYNCHRONOUS COMMUNICATION USING MESSAGES (REVISION)

14

• Exchange of a message is an atomic

mic action tion requiring participation of both the sender and the recei eiver er

• If sender is ready, but receiver is not – sender

r is is b bloc locked

• If sender is not ready, but receiver is – receiv

iver er is is b bloc locked

• Exchange of a message synchro

chroni nises the execution sequences of these 2 processes

• Can be 1-wa

way y or 2-way y communication

ASYNCHRONOUS COMMUNICATION USING MESSAGES (REVISION)

15

• No

No tempora poral l depende dence nce (other then ordering) between the execution sequences of these 2 processes, but must st be 1-wa way

• Sender can send a message and contin

tinue wit without ut bloc locking ing

• The message is stored in a buff

ffer er – a (finite size > 0) queue queue of data elements in the communication channel

• Issues: writing into a full buffer, reading from an empty buffer
• Receiver can later check the buffer for new messages

16

• In distributed/decentralised systems (e.g. Internet), flexibility
• ffered by asynchron

chronou

• us communication is often advantageous
• E.g. due to considerable communication delays, failures of

communication links or distributed computers, …

• But this can lead to more

e dif ifficult icult program rammin ming

THE NEED FOR ASYNCHRONOUS COMMUNICATION

17

• Modern programming languages have various advance

ced d constr structs ucts for asynchronous communication (and other concurrency)

• Simple additions: callback function, promise/future

mise/future, async/await /await, …

• Bigger paradigm shifts: CS

CSP channe nels ls, acto tor r model, (tuple) spaces, …

• Not only purpose-built concurrent languages, but also extensions of

popular general programming languages

• We will give a brief overview of some paradigms and constructs, but

details remain for your independent research

LANGUAGE SUPPORT FOR ASYNCHRONOUS COMMUNICATION

SOME MESSAGE PASSING CONCURRENCY PARADIGMS

18

From Chapter 8 in Ben-Ari’s Text xtboo book

ADDRESSING IN MESSAGE PASSING

19

• Symmetric

metric addressing essing: sender and receiver know each

• ther’s address
• Channels

ls: instead of addresses, 2 or more processes use shared named d channels

• Asymmetric

etric addressin essing: a client knows address of a server, but the server need not know address the client

• No addressin

ssing: receiver is determined based on matching

• n the message structure

RENDEZVOUS – OVERVIEW

20

• Type of synchronous (blocking) communication
• Can be appropriate for some servers (but asynchronous is often better)
• Supported in Ada and some other programming languages
• Ca

Call lling ing proce cess ss must know identity of accept pting ing process ess and entry ry point (in the accepting process)

• Accepting proc. need not know identity of calling proc.
• Optional parameters and return values
• Limitations: tigh

ight t coupli ling, active participation of both processes

RENDEZVOUS – EXAMPLE

21

RENDEZVOUS SEMANTICS – 2 TIME DIAGRAMS

22

a) b)

REMOTE PROCEDURE CALL (RPC)

23

• Another type of synchronous (blocking) communication
• Supported (differently) in many prog. languages and distributed system

frameworks: Java RMI, CORBA, some Web service frameworks, …

• Client calls server in the way identical to local procedure call
• Different ways to locate servers, e.g. registry
• New process is created (locally or remotely) to handle this call, but

this is transparent to the client

• Easier programming, somewhat less tight coupling than rendezvous
• More widely used (but asynchronous is still better)

REMOTE PROCEDURE CALL (RPC) – COMPONENTS

24

CSP (COMMUNICATING SEQUENTIAL PROCESSES) CHANNELS (REVISION)

25

• Connecting senders and receivers
• Typed: type of messages must be declared and matched
• Can be synchro

chrono nous us or asynchron chronou

• us
• Originally synchronous, then extended to asynchronous
• Sending value of variable x on channel ch :

𝑑ℎ ⇐ 𝑦 (pseudocode), 𝑑ℎ ! 𝑦 (Promela)

• Receiving value from channel ch and storing it in variable y :

𝑑ℎ ⇒ 𝑧 (pseudocode), 𝑑ℎ ? 𝑧 (Promela)

• Effic

ficient ient and easy to use communication mechanism

CSP CHANNELS – LIMITATIONS

26

• Channels lack flexibility – programs using channels must be

configured (at compile-time or initialisation-time)

• Not a dynamic, run-time change of configuration
• E.g. difficult to program (without workarounds) a realistic,

flexible server using channels because it could offer only specific channels for access

• Deadloc

lock or starvation arvation possible, if not programmed carefully

THE ACTOR MODEL

27

From online resources

???

28

This is a cropped version of the CC BY 2.0 image from: https://www.flickr.com/photos/35332562@N04/3363633021

THE ACTOR MODEL – INTRODUCTORY VIDEOS

29

• Task for you: Watch these 2 videos (in the given order)
• Finematics, “Actor Model Explained” [video, 4:32],

YouTube, 17 Apr. 2018, at: https://www.youtube.com/watch?v=ELwEdb_pD0k

• See also: https://finematics.com/actor-model-explained/
• Mark Lewis, “The Actor Model” [video, 4:29], YouTube, 13
• Feb. 2016, at: https://www.youtube.com/watch?v=un-

pSOlTaY0

THE ACTOR MODEL – SOME KEY CHARACTERISTICS (1/2)

30

• 1 actor is not useful by itself, you need a system

tem of actors

• rs
• In a (pure) system of actors, everything is an actor
• Actors can be compo

posed sed into higher-level actors

• Actor can be viewed as an object (!) with a li

lightwe weigh ight t thre read ad that is run only when needed

• Asynchron

chronou

• us

s messag sage e passing sing concurr urren ency cy – each actor has a mail ilbox (FIFO queue) and an address ess

THE ACTOR MODEL – SOME KEY CHARACTERISTICS (2/2)

31

• Actor is iso

isolat lated (“share nothing” concurrency)

• Actor has internal state

ate but messages are im immuta tabl ble (data only created or copied)

• Easy to dist

istrib ribute ute: working remotely in the same way as locally

• Built-in fault

lt toler leran ance ce (“let it crash” then deal with this)

THE ACTOR MODEL – SOME BENEFITS (1/2)

32

• No

No loc locks and no bloc locking ing of senders ers (only of receivers)

• Prese

served rved encapsu psula latio tion because message passing does not transfer execution

• Eli

limina ination ion of races because modifying internal state of an actor

• nly possible via immutable messages processed sequentially
• Co

Corresp esponding nding to modern ern memory

• ry hierarch

rchies because state of actors is local and not shared, while changes and data are propagated via messages

THE ACTOR MODEL – SOME BENEFITS (2/2)

33

• Task

k deleg legation ion, scali ling and dist istrib ribution ution are natural with agents

• Suit

itable le for r decentrali ntralised d computi puting (e.g. Internet) due to built-in support for fault-tolerance (e.g. supervisors)

• Can work well when a problem can be decomp

mposed

• sed in a set of

ind independent ent tasks ks or when tasks sks are li linked throu

• ugh

gh a clea lear wo workflo kflow w

• Co

Concepts pts easier ier to understa rstand nd than in some other paradigms

THE ACTOR MODEL – SOME LIMITATIONS

34

• De

Deadloc lock or starvation arvation possible

• Following some design patterns can help avoid this in many situations
• Should use formal

mal me methods

• ds for analys

ysis/v s/ver erific ficati ation

• n of actor programs

(several methods have been developed)

• Mail

ilboxes s can overf rflow low

• Most implementations of actors do not guarant

ntee e timed message sage

• rder

r nor tim ime li limit it for actin ting upon a m messag sage

• Too high

igh overhe head ad for some problems

• …

THE ACTOR MODEL – SOME IMPLEMENTATIONS

35

• Somewhat different implementations of the conceptual

actor model

• Programming languages: Erlang, Eli

Elixir, Dart, …

• Frameworks: AKKA

AKKA for Scala and Java (and Kotlin), …

• Optional task for you: Read the AKKA “Getting Started

Guide” (hyperlink) for Java and run the given examples

• Task for you: Consider studying an actor language or

framework for your Assignment 2

CSP VS. THE ACTOR MODEL – SIMILARITIES AND DIFFERENCES

36

• Task for you: Discuss sim

imil ilarit ities ies between the CSP model and the actor model

• Task for you: Discuss differe

erences nces between the CSP model and the actor model

• Task for you: Whic

ich h one would ld you use in a:

• (non-distributed) multithreaded system?
• parallel computing system?
• distributed system?
• Justify your answers!

SOME SHARED MEMORY CONCURRENCY PARADIGMS

37

From online sources and Chapter 9 in Ben-Ari’s Text xtbo book

• k

38

• Sh

Shared ed memory

• ry concurren

urrency cy contr trol

• l mechan

anism

• Implemented in software (not HW), simplifies concurrent programming
• Growin

ing g in in popul ular arity ty: Haskell, Clojure, Java Multiverse STM, …

• Motivated by database

ase tra ransacti nsactions and their atomicity, consistency and isolation (but not always durability)

• Begin/commit/rollback
• Can be implemented lock-free or with encounter-time locking or

with commit-time locking

• FYI: Dr. Liam O’Connor has expertise and experience with STM

SOFTWARE TRANSACTIONAL MEMORY (STM)

(TUPLE) SPACES

39

• Shared,

ed, vir irtua ual, l, associativ ciative memory

• ry for distributed systems
• Loose coupli

ling of processes, data a persistency istency

• Typed notes (tuples) posted on a space (similar to bulletin

board) and then read or removed

• Matching of notes based on content/parameters
• Different implementations in Linda, C-Linda, JavaSpaces, …
• Lim

imit itatio tions ns: for many (but not all) application domains message passing is usually more efficient

MASTER-WORKER (MASTER-SLAVE, MANAGER-WORKERS) PATTERN

40

• Not only for (tuple) spaces, but easy to implement there
• Master

ster process assigns tasks ks to wo workers ers that perform required computations and send results to their master for merging

• Divide and conquer to achieve scalability
• It is possible to specify granula

larit rity of tasks to suit relative performances of processors and communication systems

• Somewhat similar patterns: farmer-worker, MapRedu

duce ce, …

• FYI: See Chapter 9 of Ben-Ari’s textbook and online sources

SOME OTHER CONSTRUCTS FOR ASYNCHRONOUS SYSTEMS

41

From various sources, including personal experience

CALLBACK FUNCTION

42

• Function

ction passed as a pa paramet meter to another function and invoked later at the end of this other function

• Usually asynchronous: the original caller continues executing
• Many problems, often leading to “callback hell”:
• Difficulty nesting callbacks (resulting in “Christmas tree” blocks)
• Complicated error handling
• Different prog. language support: JavaScript, C++, Lua, Python, …

PROMISE AND FUTURE

43

• Plac

laceholder lder for an object whose value lue wil will l be determ ermined ined only ly lat later after some other code executes

• Future

ure – read-only container for a result that does not yet exist; this result is set (resolved) by some other code asynchronously

• Pr

Promis mise – write-once (single-assignment) container setting value of a future; can be sent to other code but its value is set (fulfilled) later

• Prog. language implementation: Scala, JavaScript, C++ 11, Java 8, …
• Cleaner asynchronous programming using chained calls
• However, propagation and chaining of errors can be tricky

COROUTINE

44

• Generalised subroutine that may suspen

end d and resume ume it its execut ution ion by expli licit itly ly yieldin ielding control to another coroutine wit without ut bloc locking ing

• For cooperat

rative, e, non-pre preempt emptive multitaski tasking

• Go(lang) goroutine

utine similar, but for preemptive multitasking

• Much more

re li lightwe weigh ight t than a threa read (cf. actor)

• Many corou
• utine

tines execut ute on 1 t threa read (concurrency, not parallelism)

• Can be used for programming asynchronous systems
• Programming languages: assembly, Ruby, Kotli

lin, Julia, Haskell, …

• Recent extensions of other languages: C++, Python, …

ASYNC/AWAIT

45

• “Syntactic sugar” for writing asynchronou

chronous s funct ctions similarly to synchronous functions

• Function denoted as async

nc can contain awa wait it expressions and will bind result to a promise that is the function’s return value

• awa

wait it in front of an expression returning a promise pauses execution until this promise is fulfilled

• awa

wait it can be used only in async functions

• Implementation: coroutines, event loop, …
• Included into: Python 3.5, C#

C# 5. 5.0, JavaScript, Rust, Dart, …

• Improved code understandability and, possibly, error handling

STREAM AND REACTIVE STREAM

46

• St

Stre ream am – a sequence of data and on each of its elements a series of

• perations is performed, usually in a pipeline, possibly in parallel
• Related to dataflow programming, reactive programming, …
• As

Asynchronou chronous publication-subscription (producer-consumer) model

• No

Non-blocking locking back-pressure pressure – subscriber can asynchronously notify its publisher that the data rate has to be reduced

• Re

Reactive tive St Strea eams ms – a standard for asynchronous stream processing with non-blocking back pressure, implemented in: Java 9 (prog. lang.), RxJava (API), AKKA AKKA (toolkit & runtime), Spring (framework), MongoDB (database), …

FUNCTIONAL PROGRAMMING FOR ASYNCHRONOUS DISTRIBUTED SYSTEMS

47

• Many modern distributed systems written in functional programming

languages, e.g. Erlang, Elixir, Scala, Clojure, Haskell, OCaml, …

• or functional extensions of OO languages: Java 8, Kotlin, C++ 11, …
• Inter-service calls are non

non-blocking locking and use resources (e.g. CPU) well

• Immutable

mutable data passed around without issues in synchronising data modifications

• Code using only immutable data can be safely copied/moved around
• Co

Composi sing proce cessing ssing pipelines to abstract over asynchronicity

• Monadic separat

ation ion of handli ling regular lar cases es and fail ilures, …

CONCLUSION

48

From various sources, including personal experience

A CONCURRENT PROGRAMMING TOOLBOX

49

Image by Per Erik Strandberg sv:User:PER900 0 / CC BY-SA 2.5

BIG IDEA 5: THERE ARE SEVERAL (NEW AND OLD) CONCURRENCY PARADIGMS

50

• “If your only tool is a hammer, then every problem looks like a nail”
• P.S. Most problems are NOT nails
• Modern concurrent computing is much more than threads and

locks (semaphores, monitors)

• Dif

iffere erent nt concurr urrency ncy paradigm igms are used for different problem types or in different contexts

• Comeback

ack of some old computer science ideas - changed context led to new use cases for old ideas

• E.g., the actor model from 1973 became very popular in 2010s due to

cloud computing

NEXT TIME… (PREVIEW HIGHLIGHTS)

51

From Chapter 12 in Ben-Ari’s Textbook

MAIN TOPICS IN THE NEXT LECTURE… (BEN-ARI TEXTBOOK CHAPTER 12 )

52

• Fault

lt toler leran ance ce and inc inconsisten istent t inf inform rmation ation in distributed systems – the problem of consen ensus sus

• By

Byzant ntine General rals algorithm hm explanation

• Byzantine Generals algorithm examples and demo in

in DA DAJ

• King

ing algo lgorit ithm hm explanation and examples