Tuning for Scale (Tue Mar 15, lect 15) | previous | next | slides |

Logistics

• Magic Word
• Adding to Project readme.md: new work in the release but also who was primarily responsible

Tuning for Scalability

• We start with the simplest possible set up. If you have stand alone servers (which we don’t) you would run the database and web server on the same box.
• In our case we are deploying to a cloud server, where you aren’t getting a full box but a virtual slice, from e.g. Heroku. In that case our base configuration is a single “web worker” to run Sinatra and a single “database” to run our database, which would be Postgress.
• With that base, measure performance. If you have a real load, you can measure. In our case we have to create an artificial load send traffic to the server. We want to see how many simultanuous sessions we support.

Stage One Scalability Tuning

• As you do scalability testing, keep an eye out for errors that are happening. A server will appear to respond super fast to a request if all it is doing is returning an error code!
• Analyze and think about whether the servers are poweful enough. Is it your old laptop running a database server or is it a fairly new computer with no other load. If you are using a cloud service, like BlueMix, Heroku or Digital Ocean, what kind of virtual capacity or limits does it have? You might need to simply up the capacity.
• Examine your database access and queries. Can you determine that pages that don’t have a database call are really fast and the ones that do are slower? Can you see what pages slow down the most?

If queries are the problem

• Make sure you clearly understand the separation between the web and the database server.
• Consider whether you should put the database on it’s own server
• Consider whether any tables need indexes
• Consider whether you are going back to the database more than a few times for each page displayed.
  • Consider whether you are hitting the database once for each record displayed (so called N+1 problem).
  • If you are, look at making your queries ‘greedy’ meaning that they bring back more in a single call to the server
• Consider whether you are issuing the exact same query over and over again.
• Consider whether an intermediate query result that is expensive is being requested again. If so, caching those results is a strategy.
• Look at the metrics on your database servers, that is simple

If the app/web server seems to be the problem

• Consider what app server you are using
  • For example: webrick, unicorn, puma, thin, passenger
  • See App Server Arena
• Consider whether you are fully/over utilizing resources
  • Make sure that the resources (memory, cpu) are being used but not pinned to maximum.
  • You don’t want to hit your caps on resources, otherwise the app will start thrashing.
• Consider adding more “web workers” or concurrent threads.
  • You have to be careful that your architecture allows that. Is your design ready to run concurrently.
  • Is there any way that one request can corrupt another one running at the same time?

If you still have scaling issues

• Consider adding another discrete server
  • Once you see one box does not cut it any longer add another server, you will need a load balancer for this.
  • Adding another server, increases your redundancy. Depending on how valuable this app is and how badly it needs to stay up 2 app servers is a good idea, as well as 2 database servers, 1 primary and 1 follower on stand by. AWS RDS makes that last part easy.
• Consider caching services
  • Are there intermediate results that can be cached to avoid hitting the database at all?
• Consider partitioning the databases
  • One replica for update transactions and multiple replicas for read transactions
  • Partitioning vertically, by moving certain columns of tables to a separate db servers.
  • Partitioning horizontally (sharding) by moving certain sets of rows to separate db servers. (For example, profile records for half the users on one server and the other half on another server.)
• Consider breaking into services
  • Are there separable major bits of functionality that you can carve off into fully independent services, with their own app and db servers

Load Testing

• Testing a product to see how it performs under load
• Obviously to do load testing, your app needs to be running on a server, separately from your own computer.
• This could be any server, even your neighbor’s laptop.
• For our purposes, it needs to be on one of the cloud services like Digital Ocean or Heroku
  1. We are all using Heroku
  2. if you are adventurous you could deploy it elsewhere too.
  3. But then we won’t really be able to give you too much help
• Cloud deployment implies that there is a well known ‘fixed’ domain name (e.g. wild-mushroom-2312.herokuapp.com)

App functionality

• Each team has deployed nanoTwitter and has supplied us with the domain name
• Each implementation of nanoTwitter follows the same spec (NanoTwitter Functionality) including the nT Test Interface
• In particular, implement the url routes exactly as specified.
• This allows a unified test to be run against all of the apps
• To allow us to compare apples to apples we have Seed Data
• Adapt your code to use this Seed Data

Load Testing Tools

• There are numerous Load Testing tools. Here are two lists: open source Load Testing Tools and Performance Testing Tools
• We are using loader.io
• You need to understand the different types of tests

How to deal with authentication

• The nT Test Interface / specifies an authentication bypass argument
• All your urls will need to check for the special argument in the url: ?user_id=x
• If you see that argument, then you bypass the normal login sequence and instead set x as your currently logged in user by storing it in your session variable.

Reporting

• Keep a careful lab notebook!
• Make note of configuration changes that you try
• Collect data and try to interpret it

When your app is failing to scale

• Think about what happens when your app response time is longer than the time between new connections. Requests will start to pile up.
• If it takes too long for a request to reach the front of the line, it will time out.
• If your app is too overloaded, it may shut down entirely. Make sure to consult your Heroku logs after a test to identify this.
• When an app hasn’t been accessed for a while, Heroku will put it in a sleep state. If your app is asleep, make sure to access it manually to wake it up before beginning a test. Waking up may take several seconds, which would influence your results.

Using Loader.io

• Our goal is to see how well each of your servers survive the onslaught of traffic
• Take a look at Loader.io
• Use the “maintain client load” option (at least to start.)

Test 1: Simple display of home page

• Visit your root page (which will display a set number of tweets)
• Use Loader.io to hit that page using maintain client load
• Collect and record data

Test 2: Display of a certain user’s home page

• Visit /user/1 to display User 1’s home page
• Visit /user/100 to display another home page
• Collect and record data

Test 3: Do some analysis

• Form some hypotheses about what is going on with your performance
• Make a report correlating load to performance

Thank you. Questions?  (random Image from picsum.photos)