Logistics

• Secret code:
• What is the highest number of threads you’ve gotten to?
• How much of the test interface have you implenmented?
• Any questions?

Scalability Pattern: Concurrency

• Do more with the available resources
• Do more things at the same time
• Resource idle time is your enemy
  1. CPU
  2. Network
  3. Disk
  4. Database

Scenarios when concurrency is an option

• Users can ask for a detailed report about their twitter traffic
  1. It can be a one shot, or regenerated at midnight
  2. It can be displayed on the screen or emailed
  3. Inline is no good because of computational cost and error handling
• User enters a new value in a spreadsheet
  1. The cell can update immediately
  2. The recalculation can be done concurrently (asynchronously)
• Note: AJAX uses a very common form of concurrency
• Note: Count the users’ own computer as one of the processors!

Scaling pattern: concurrency on a single computer

• Difference between syncrhonous and asynchronous
• Difference between concurrent and parallel

Processes (“forking”)

• Use more memory (new VM for each process)
• for the data + the program + everything
• “Copy on write”
• If parent dies before children, they can become “zombie” processes
• Context switching very expensive
• Communication expensive (IPC or file system)
• Slower to create and destroy
• Less hard to program and debug (not easy!)

Threads

• Use less memory (Shared memory space)
• All threads die when oricess dies
• Context switching cheap
• Communication cheap (via queues and shared memory)
• Fast to create and destroy
• Harder to program and debug

Ruby-related issues

• Ruby and Pyton “GIL”
  1. Global interpretter lock
  2. Essentially they become single threaded
  3. Except for asynchrony provded by OS via IO operations
  4. “It’s complicated”
• Ruby libraries that use processes
  1. Resque - background processing framework
  2. Unicordn - http server
• Ruby Libraries that use Threads
  1. Sidekiq - background processing framework
  2. Puma - http server
  3. Thin - http server

Thread-safe

• A property of software, or a routine or a class
• Does it behave ‘well’ when running in a thread (sharing memory)
• Deadlock (“deadly embrace”)
  1. Example with two people and two tools
• Race Condition
  1. When the results vary due to
  2. How to avoid: using semaphors, queues, and other techniques

:slide_title

• Let’s take a look at Multithreaded Ruby
• Let’s take a look at Introduction to Concurrency Models in RUby

Scaling Pattern: Asycnhronous processing

• Real examples
  1. Account registraton confirmation emails (actually all emails)
  2. Daily (periodic or episodic) notification emails
  3. Automatic backups or archiving
  4. Image Resizing
  5. Spam checking
• Synchronous: Caller waits for response
• Asynchronous: Request returns immediately, but result comes later
• Web server mainly responds to http requests!
• Background processing happens even if no requests!

How do processes work in a modern Operating System?

• Scheduler part of the OS
• Processes can be fairly heavy weight
• Let’s estimate how many background processs you would need
• How would you handle processes that:
  1. Were taking too long?
  2. Had crashed?
  3. Needed to be restarted?
  4. Or what if the server crashes as a whole?
  5. It’s a mess. Need abstractions!

Case 1: Schedule based

• “CRON jobs”: Most elementary. Require “privileges”
• Alternative: Native “scheduler” application (e.g. Heroku Simple Job Scheduler)

Case 2: Request based

• Request handled ‘asynchronously’ of course
• By local server, in a separate process
• By a remote server
• Request queueing
  1. What happens when requests come faster than we can handle them
  2. How do we add more performance in handling request
• SOA API
  1. Make sure the API returns right away and requestor has a way to check status

Thank you. Questions?  (random Image from picsum.photos)