Peterbe.com

This isn't rocket science in the world of Web Development but I think it's so darn useful that if you've been unlucky to miss this, it's worth mentioning one more time.

Suppose you're on a site with lots of links. Like https://www.peterbe.com/plog.
And you want to get a list of all URLs that contain word "react" in the URL pathname.
And you want to get this into your clipboard.

Here's what you do:

1. Open your browsers Web Console. In Firefox it's Alt+Cmd+K on OSX (or F12). In Chrome it's Alt+Cmd+J.

2. Type in the magic: copy([...document.querySelectorAll('a')].map(a => a.href).filter(u => u.match(/react/i)))

3. Hit Enter, go to a text editor and paste like regular.

It should look something like this:

[
  "https://www.peterbe.com/plog/10-reasons-i-love-create-react-app",
  "https://www.peterbe.com/plog/how-to-deploy-a-create-react-app",
  "https://www.peterbe.com/plog/4-different-kinds-of-react-component-styles",
  "https://www.peterbe.com/plog/onchange-in-reactjs",
  "https://www.peterbe.com/plog/tips-on-learning-react",
  "https://www.peterbe.com/plog/visual-speed-comparison-of-angularjs-and-reactjs",
  "https://www.peterbe.com/plog/600-billion-challenge-reactions",
  "https://www.peterbe.com/plog/active-reactor-watches"
]

The example is just that. An example. The cool thing about this is:

• The Web Console built-in command copy().
• That [...document.querySelectorAll('a')] turns the NodeList object into a regular array.
• That the .map(a => a.href) is super simple and it turns each Node into its href attribute value.
• That you could have used a more advanced CSS selector like document.querySelectorAll(a[target="_blank"]) for example.

The limit is your imagination. You can also do things like copy([...document.querySelectorAll('a')].filter(a => a.textContent.match(/react/i)).map(a => a.href)) and you filter by the links' text.

tl;dr; I have a lot of code that does response = requests.get(...) in various Python projects. This is nice and simple but the problem is that networks are unreliable. So it's a good idea to wrap these network calls with retries. Here's one such implementation.

The First Hack

import time
import requests

# DON'T ACTUALLY DO THIS. 
# THERE ARE BETTER WAYS. HANG ON!

def get(url):
    try:
        return requests.get(url)
    except Exception:
        # sleep for a bit in case that helps
        time.sleep(1)
        # try again
        return get(url)

This, above, is a terrible solution. It might fail for sooo many reasons. For example SSL errors due to missing Python libraries. Or the URL might have a typo in it, like get('http:/www.example.com').

Also, perhaps it did work but the response is a 500 error from the server and you know that if you just tried again, the problem would go away.

# ALSO A TERRIBLE SOLUTION

while True:
    response = get('http://www.example.com')
    if response.status_code != 500:
        break
    else:
        # Hope it won't 500 a little later
        time.sleep(1)

What we need is a solution that does this right. Both for 500 errors and for various network errors.

The Solution

Here's what I propose:

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry


def requests_retry_session(
    retries=3,
    backoff_factor=0.3,
    status_forcelist=(500, 502, 504),
    session=None,
):
    session = session or requests.Session()
    retry = Retry(
        total=retries,
        read=retries,
        connect=retries,
        backoff_factor=backoff_factor,
        status_forcelist=status_forcelist,
    )
    adapter = HTTPAdapter(max_retries=retry)
    session.mount('http://', adapter)
    session.mount('https://', adapter)
    return session

Usage example...

response = requests_retry_session().get('https://www.peterbe.com/')
print(response.status_code)

s = requests.Session()
s.auth = ('user', 'pass')
s.headers.update({'x-test': 'true'})

response = requests_retry_session(session=s).get(
    'https://www.peterbe.com'
)

It's an opinionated solution but by its existence it demonstrates how it works so you can copy and modify it.

Testing The Solution

Suppose you try to connect to a URL that will definitely never work, like this:

t0 = time.time()
try:
    response = requests_retry_session().get(
        'http://localhost:9999',
    )
except Exception as x:
    print('It failed :(', x.__class__.__name__)
else:
    print('It eventually worked', response.status_code)
finally:
    t1 = time.time()
    print('Took', t1 - t0, 'seconds')

There is no server running in :9999 here on localhost. So the outcome of this is...

It failed :( ConnectionError
Took 1.8215010166168213 seconds

Where...

1.8 = 0 + 0.6 + 1.2

The algorithm for that backoff is documented here and it says:

A backoff factor to apply between attempts after the second try (most errors are resolved immediately by a second try without a delay). urllib3 will sleep for: {backoff factor} * (2 ^ ({number of total retries} - 1)) seconds. If the backoff_factor is 0.1, then sleep() will sleep for [0.0s, 0.2s, 0.4s, ...] between retries. It will never be longer than Retry.BACKOFF_MAX. By default, backoff is disabled (set to 0).

It does 3 retry attempts, after the first failure, with a backoff sleep escalation of: 0.6s, 1.2s.
So if the server never responds at all, after a total of ~1.8 seconds it will raise an error:

In this example, the simulation is matching the expectations (1.82 seconds) because my laptop's DNS lookup is near instant for localhost. If it had to do a DNS lookup, it'd potentially be slightly more on the first failure.

Works In Conjunction With timeout

Timeout configuration is not something you set up in the session. It's done on a per-request basis. httpbin makes this easy to test. With a sleep delay of 10 seconds it will never work (with a timeout of 5 seconds) but it does use the timeout this time. Same code as above but with a 5 second timeout:

t0 = time.time()
try:
    response = requests_retry_session().get(
        'http://httpbin.org/delay/10',
        timeout=5
    )
except Exception as x:
    print('It failed :(', x.__class__.__name__)
else:
    print('It eventually worked', response.status_code)
finally:
    t1 = time.time()
    print('Took', t1 - t0, 'seconds')

And the output of this is:

It failed :( ConnectionError
Took 21.829053163528442 seconds

That makes sense. Same backoff algorithm as before but now with 5 seconds for each attempt:

21.8 = 5 + 0 + 5 + 0.6 + 5 + 1.2 + 5

Works For 500ish Errors Too

This time, let's run into a 500 error:

t0 = time.time()
try:
    response = requests_retry_session().get(
        'http://httpbin.org/status/500',
    )
except Exception as x:
    print('It failed :(', x.__class__.__name__)
else:
    print('It eventually worked', response.status_code)
finally:
    t1 = time.time()
    print('Took', t1 - t0, 'seconds')

The output becomes:

It failed :( RetryError
Took 2.353440046310425 seconds

Here, the reason the total time is 2.35 seconds and not the expected 1.8 is because there's a delay between my laptop and httpbin.org. I tested with a local Flask server to do the same thing and then it took a total of 1.8 seconds.

Discussion

Yes, this suggested implementation is very opinionated. But when you've understood how it works, understood your choices and have the documentation at hand you can easily implement your own solution.

Personally, I'm trying to replace all my requests.get(...) with requests_retry_session().get(...) and when I'm making this change I make sure I set a timeout on the .get() too.

The choice to consider a 500, 502 and 504 errors "retry'able" is actually very arbitrary. It totally depends on what kind of service you're reaching for. Some services only return 500'ish errors if something really is broken and is likely to stay like that for a long time. But this day and age, with load balancers protecting a cluster of web heads, a lot of 500 errors are just temporary. Obivously, if you're trying to do something very specific like requests_retry_session().post(...) with very specific parameters you probably don't want to retry on 5xx errors.

tl;dr; I'm not a TC39 member and I barely understand half of what those heros are working on but there is one feature they're working on that really stands out, in my view, for React coders; Public Class Fields.

The Problem?

Very common pattern in React code is that have a component that has methods that are tied to DOM events (e.g. onClick) and often these methods need acess to this. The component's this. So you can reach things like this.state or this.setState().

You might have this in your code:

class App extends Component {
  state = {counter: 0}

  constructor() {
    super()

    // Like homework or situps; something you have to do :(
    this.incrementCounter = this.incrementCounter.bind(this) 
  }

  incrementCounter() {
    this.setState(ps => {
      return {counter: ps.counter + 1}
    })
  }

  render() {
    return (
      <div>
        <p>
          <button onClick={this.incrementCounter}>Increment</button>
        </p>
        <h1>{this.state.counter}</h1>
      </div>
    )
  }
}

Demo

If you don't bind the class method to this in the constructor, this will be undefined inside the class instance field incrementCounter. Buu!

Suppose you don't like having the word incrementCounter written in 4 placecs, you might opt for this shorthand notation where you create a new unnamed function inside the render function:

class App extends Component {
  state = {counter: 0}

  render() {
    return (
      <div>
        <p>
          <button onClick={() => {
            this.setState(ps => {
              return {counter: ps.counter + 1}
            })
          }}>Increment</button>
        </p>
        <h1>{this.state.counter}</h1>
      </div>
    )
  }
}

Demo

Sooo much shorter and kinda nice that the code can be so close in proximity to the actual onClick event definition.

But this notation has a horrible side-effect. It creates a new function on every render. If instead of a regular DOM jsx object <button> it might be a sub-component like <CoolButton/> then that sub-component would be forced to re-render every time (unless you write your own shouldComponentUpdate.

Also, this notation works when the code is small and light but it might get messy quickly if you need that functionality on other element's onClick. Or it might become mess with really deep indentation.

The Solution?

Public Class Fields.

That new feature is currently in the "Draft" stage at TC39. Aka. stage 1.

However, I discovered that you can use stage-2 in Babel to use this particular feature.

Note! I don't understand why you only have to put on your stage-2-brave socks for this feature when it's part of a definition that is stage 1.

Anyway, what it means is that you can define your field (aka method) like this instead:

class App extends Component {
  state = {counter: 0}

  incrementCounter = () => {
    this.setState(ps => {
      return {counter: ps.counter + 1}
    })
  }

  render() {
    return (
      <div>
        <p>
          <button onClick={this.incrementCounter}>Increment</button>
        </p>
        <h1>{this.state.counter}</h1>
      </div>
    )
  }
}

Demo

Now it's only mentioned by name incrementCounter twice. And no need for that manual binding in a constructor.
And since it's automatically bound, the function isn't recreated and those can easily make sub-components be pure.

So, let's always write our React methods this way from now on.

Oh, and in case you wonder. Inheritence works the same with these public class fields as the regular class instance fields.

The title is a bit of an understatement because I think it's pretty good. It's not perfect and it's not guaranteed to scale, but it works pretty well. Especially on search term typos.

This, my own blog, now has a search engine built with Elasticsearch using the Python library elasticsearch-dsl. The algorithm (if you can call it that) is my own afternoon hack invention. Before I explain how it works try out a couple of searches:

Try a couple of searches:

(each search appends &debug-search for extended output)

• corn - finds Cornwall, cron, Crontabber, crontab, corp etc.
• crown - finds crown, Crowne, crowded, crowds, crowd etc.
• react - finds create-react-app, React app, etc.
• jugg - finds Jung, juggling, judging, judged etc.
• pythn - finds Python, python2.4, python2.5 etc.

Also, by default it uses Elasticsearch's match_phrase so when you search for a multi-word thing, it requires a match on each term. E.g. date format which finds Date formatting, date formats etc.

But if you search for something where the whole phrase can't match, it splits up the search an uses a match operator instead (minus any stop words).

Typo-focussed

This solution is very much focussed on typos. One thing I really dislike in non-Google search engines is when you make a search and nothing is found and it says "Did you mean ...?". Quite likely I did, but why do I have to click it? Can't it just be clicked for me?

Also, if there's ambiguity and possibly some results based on what you typed and multiple potential "Did you mean...?". Why not just blend them alltogether like Google does? Here is my attempt to solve that. Come with me...

Figuring Out ALL Search Terms

So if you type "Firefix" (not "Firefox", also scroll to the bottom to see the debug table) then maybe, that's an actual word that might be in the database. Then by using the Elasticsearch's Suggesters it figures out alternative spellings based on frequency distributions within the indexed content. This lookup is actually really fast. So now it figures out three alternative ways to spell this term:

• firefox (score 0.9, 1 character different)
• firefli (score 0.7, 2 characters different)
• firfox (score 0.7, 2 characters different)

And, very arbitrarily I pick a score for the default term that the user typed in. Let's pick 1.1. Doesn't matter gravely and it's up for future tuning. The initial goal is to not bury this spelling alternative too far back.

Here's how to run the suggester for every defined doc type and generate a list of other search terms tuples (minimum score >=0.6).

search_terms = [(1.1, q)]
_search_terms = set([q])
doc_type_keys = (
    (BlogItemDoc, ('title', 'text')),
    (BlogCommentDoc, ('comment',)),
)
for doc_type, keys in doc_type_keys:
    suggester = doc_type.search()
    for key in keys:
        suggester = suggester.suggest('sugg', q, term={'field': key})
    suggestions = suggester.execute_suggest()
    for each in suggestions.sugg:
        if each.options:
            for option in each.options:
                if option.score >= 0.6:
                    better = q.replace(each['text'], option['text'])
                    if better not in _search_terms:
                        search_terms.append((
                            option['score'],
                            better,
                        ))
                        _search_terms.add(better)

Eventually we get a list (once sorted) that looks like this:

search_terms = [(1.1 'firefix'), (0.9, 'firefox'), (0.7, 'firefli'), (0.7, 'firfox')]

The only reason the code sorts this by the score is in case there are crazy-many search terms. Then we might want to chop off some and only use the 5 highest scoring spelling alternatives.

Building The Boosted OR-query

In this scenario, we're searching amongst blog posts. The title is likely to be a better match than the body. If the title mentions it we probably want to favor that over those where it's only mentioned in the body.

So to build up the OR-query we'll boost the title more than the body ("text" in this example) and we'll build it up using all possible search terms and boost them based on their score. Here's the complete query.

strategy = 'match_phrase'
if original_q:
    strategy = 'match'
search_term_boosts = {}
for i, (score, word) in enumerate(search_terms):
    # meaning the first search_term should be boosted most
    j = len(search_terms) - i
    boost = 1 * j * score
    boost_title = 2 * boost
    search_term_boosts[word] = (boost_title, boost)
    match = Q(strategy, title={
        'query': word,
        'boost': boost_title,
    }) | Q(strategy, text={
        'query': word,
        'boost': boost,
    })
    if matcher is None:
        matcher = match
    else:
        matcher |= match

search_query = search_query.query(matcher)

The core is that it does Q('match_phrase' title='firefix', boost=2X) | Q('match_phrase', text='firefix', boost=X).

Here's another arbitrary number. The number 2. It means that the "title" is 2 times more important than the "text".

And that's it! Now every match is scored based on how suggester's score and whether it be matched on the "title" or the "text" (or both). Elasticsearch takes care of everything else. The default is to sort by the _score as ultimately dictated by Lucene.

Match Phrase or Match

In this implementation it tries to match using a match phrase query which basically tries to find matches where every word in the query matches.

The cheap solution here is to basically keep whole search function as is, but if absolutely nothing is found with a match_phrase, and there were multiple words, then just recurse over one more time and do it with a match query instead.

This could probably be improved and do the match_phrase first with higher boost and do the match too but with a lower boost. All in one big query.

Want A Copy?

Note, this copy is quite a mess! It's a personal side-project which is an excuse for experimentation and goofing around.

The full search function is here.

Please don't judge me for the scrappiness of the code but please share your thoughts on this being a decent application of Elasticsearch for smallish datasets like a blog.

tl;dr; Redis is twice as fast as memcached as a Django cache backend when installed using AWS ElastiCache. Only tested for reads.

Django has a wonderful caching framework. I think I say "wonderful" because it's so simple. Not because it has a hundred different bells or whistles. Each cache gets a name (e.g. "mymemcache" or "redis append only"). The only configuration you generally have to worry about is 1) what backed and 2) what location.

For example, to set up a memcached backend:

# this in settings.py
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'KEY_PREFIX': 'myapp',
        'LOCATION': config('MEMCACHED_LOCATION', '127.0.0.1:11211'),
    },
}

With that in play you can now do:

>>> from django.core.cache import caches
>>> caches['default'].set('key', 'value', 60)  # 60 seconds
>>> caches['default'].get('key')
'value'

Django comes without built-in backend called django.core.cache.backends.locmem.LocMemCache which is basically a simply Python object in memory with no persistency between Python processes. This one is of course super fast because it involves no further network (local or remote) beyond the process itself. But it's not really useful because if you care about performance (which you probably are if you're here because of the blog post title) because it can't be reused amongst processes.

Anyway, the most common backends to use are:

• Memcached
• Redis

These are semi-persistent and built for extremely fast key lookups. They can both be reached over TCP or via a socket.

What I wanted to see, is which one is fastest.

The Experiment

First of all, in this blog post I'm only measuring the read times of the various cache backends.

Here's the Django view function that is the experiment:

from django.conf import settings
from django.core.cache import caches

def run(request, cache_name):
    if cache_name == 'random':
        cache_name = random.choice(settings.CACHE_NAMES)
    cache = caches[cache_name]
    t0 = time.time()
    data = cache.get('benchmarking', [])
    t1 = time.time()
    if random.random() < settings.WRITE_CHANCE:
        data.append(t1 - t0)
        cache.set('benchmarking', data, 60)
    if data:
        avg = 1000 * sum(data) / len(data)
    else:
        avg = 'notyet'
    # print(cache_name, '#', len(data), 'avg:', avg, ' size:', len(str(data)))
    return http.HttpResponse('{}\n'.format(avg))

It records the time to make a cache.get read and depending settings.WRITE_CHANCE it also does a write (but doesn't record that).
What it records is a list of floats. The content of that piece of data stored in the cache looks something like this:

1. [0.0007331371307373047]
2. [0.0007331371307373047, 0.0002570152282714844]
3. [0.0007331371307373047, 0.0002570152282714844, 0.0002200603485107422]

So the data grows from being really small to something really large. If you run this 1,000 times with settings.WRITE_CACHE of 1.0 the last time it has to fetch a list of 999 floats out of the cache backend.

You can either test it with 1 specific backend in mind and see how fast Django can do, say, 10,000 of these. Here's one such example:

$ wrk -t10 -c400 -d10s http://127.0.0.1:8000/default
Running 10s test @ http://127.0.0.1:8000/default
  10 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    76.28ms  155.26ms   1.41s    92.70%
    Req/Sec   349.92    193.36     1.51k    79.30%
  34107 requests in 10.10s, 2.56MB read
  Socket errors: connect 0, read 0, write 0, timeout 59
Requests/sec:   3378.26
Transfer/sec:    259.78KB

$ wrk -t10 -c400 -d10s http://127.0.0.1:8000/memcached
Running 10s test @ http://127.0.0.1:8000/memcached
  10 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    96.87ms  183.16ms   1.81s    95.10%
    Req/Sec   213.42     82.47     0.91k    76.08%
  21315 requests in 10.09s, 1.57MB read
  Socket errors: connect 0, read 0, write 0, timeout 32
Requests/sec:   2111.68
Transfer/sec:    159.27KB

$ wrk -t10 -c400 -d10s http://127.0.0.1:8000/redis
Running 10s test @ http://127.0.0.1:8000/redis
  10 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    84.93ms  148.62ms   1.66s    92.20%
    Req/Sec   262.96    138.72     1.10k    81.20%
  25271 requests in 10.09s, 1.87MB read
  Socket errors: connect 0, read 0, write 0, timeout 15
Requests/sec:   2503.55
Transfer/sec:    189.96KB

But an immediate disadvantage with this is that the "total final rate" (i.e. requests/sec) is likely to include so many other factors. However, you can see that LocMemcache got 3378.26 req/s, MemcachedCache got 2111.68 req/s and RedisCache got 2503.55 req/s.

The code for the experiment is available here: https://github.com/peterbe/django-fastest-cache

The Infra Setup

I created an AWS m3.xlarge EC2 Ubuntu node and two nodes in AWS ElastiCache. One 2-node memcached cluster based on cache.m3.xlarge and one 2-node 1-replica Redis cluster also based on cache.m3.xlarge.

The Django server was run with uWSGI like this:

uwsgi --http :8000 --wsgi-file fastestcache/wsgi.py  --master --processes 6 --threads 10

The Results

Instead of hitting one backend repeatedly and reporting the "requests per second" I hit the "random" endpoint for 30 seconds and let it randomly select a cache backend each time and once that's done, I'll read each cache and look at the final massive list of timings it took to make all the reads. I run it like this:

wrk -t10 -c400 -d30s http://127.0.0.1:8000/random && curl http://127.0.0.1:8000/summary
...wrk output redacted...

                         TIMES        AVERAGE         MEDIAN         STDDEV
memcached                 5738        7.523ms        4.828ms        8.195ms
default                   3362        0.305ms        0.187ms        1.204ms
redis                     4958        3.502ms        1.707ms        5.591ms

Best Averages (shorter better)
###############################################################################
█████████████████████████████████████████████████████████████  7.523  memcached
██                                                             0.305  default
████████████████████████████                                   3.502  redis

Things to note:

• Redis is twice as fast as memcached.
• Pure Python LocMemcache is 10 times faster than Redis.
• The table reports average and median. The ASCII bar chart shows only the averages.
• All three backends report huge standard deviations. The median is very different from the average.
• The average is probably the more interesting number since it more reflects the ups and downs of reality.
• If you compare the medians, Redis is 3 times faster than memcached.
• It's luck that Redis got fewer datapoints than memcached (4958 vs 5738) but it's as expected that the LocMemcache backend only gets 3362 because the uWSGI server that is used is spread across multiple processes.

Other Things To Test

Perhaps pylibmc is faster than python-memcached.

TIMES        AVERAGE         MEDIAN         STDDEV
pylibmc                   2893        8.803ms        6.080ms        7.844ms
default                   3456        0.315ms        0.181ms        1.656ms
redis                     4754        3.697ms        1.786ms        5.784ms

Best Averages (shorter better)
###############################################################################
██████████████████████████████████████████████████████████████   8.803  pylibmc
██                                                               0.315  default
██████████████████████████                                       3.697  redis

Using pylibmc didn't make things much faster. What if we we pit memcached against pylibmc?:

TIMES        AVERAGE         MEDIAN         STDDEV
pylibmc                   3005        8.653ms        5.734ms        8.339ms
memcached                 2868        8.465ms        5.367ms        9.065ms

Best Averages (shorter better)
###############################################################################
█████████████████████████████████████████████████████████████  8.653  pylibmc
███████████████████████████████████████████████████████████    8.465  memcached

What about that fancy hiredis Redis Python driver that's supposedly faster?

TIMES        AVERAGE         MEDIAN         STDDEV
redis                     4074        5.628ms        2.262ms        8.300ms
hiredis                   4057        5.566ms        2.296ms        8.471ms

Best Averages (shorter better)
###############################################################################
███████████████████████████████████████████████████████████████  5.628  redis
██████████████████████████████████████████████████████████████   5.566  hiredis

These last two results are both surprising and suspicious. Perhaps the whole setup is wrong. Why wouldn't the C-based libraries be faster? Is it so incredibly dwarfed by the network I/O in the time between my EC2 node and the ElastiCache nodes?

In Conclusion

I personally like Redis. It's not as stable as memcached. On a personal server I've run for years the Redis server sometimes just dies due to corrupt memory and I've come to accept that. I don't think I've ever seen memcache do that.

But there are other benefits with Redis as a cache backend. With the django-redis library you have really easy access to the raw Redis connection and you can do much more advanced data structures. You can also cache certain things indefinitely. Redis also supports storing much larger strings than memcached (1MB for memcached and 512MB for Redis).

The conclusion is that Redis is faster than memcached by a factor of 2. Considering the other feature benefits you can get out of having a Redis server available, it's probably a good choice for your next Django project.

Bonus Feature

In big setups you most likely have a whole slur of web heads that are servers that do nothing but handle web requests. And these are configured to talk to databases and caches over the near network. However, many of us have cheap servers on DigitalOcean or Linode where we run web servers, relational databases and cache servers all on the same machine. (I do. This blog is one of those where there is Nginx, Redis, memcached and PostgreSQL on a 4GB DigitalOcean SSD Ubuntu).

So here's one last test where I installed a local Redis and a local memcached on the EC2 node itself:

$ cat .env | grep 127.0.0.1
MEMCACHED_LOCATION="127.0.0.1:11211"
REDIS_LOCATION="redis://127.0.0.1:6379/0"

Here are the results:

TIMES        AVERAGE         MEDIAN         STDDEV
memcached                 7366        3.456ms        1.380ms        5.678ms
default                   3716        0.263ms        0.189ms        1.002ms
redis                     5582        2.334ms        0.639ms        4.965ms

Best Averages (shorter better)
###############################################################################
█████████████████████████████████████████████████████████████  3.456  memcached
████                                                           0.263  default
█████████████████████████████████████████                      2.334  redis

The conclusion of that last benchmark is that Redis is still faster and it's roughly 1.8x faster to run these backends on the web head than to use ElastiCache. Perhaps that just goes to show how amazingly fast the AWS inter-datacenter fiber network is!

tl;dr; You can download files from S3 with requests.get() (whole or in stream) or use the boto3 library. Although slight differences in speed, the network I/O dictates more than the relative implementation of how you do it.

I'm working on an application that needs to download relatively large objects from S3. Some files are gzipped and size hovers around 1MB to 20MB (compressed).

So what's the fastest way to download them? In chunks, all in one go or with the boto3 library? I should warn, if the object we're downloading is not publically exposed I actually don't even know how to download other than using the boto3 library. In this experiment I'm only concerned with publicly available objects.

The Functions

f1()

The simplest first. Note that in a real application you would do something more with the r.content and not just return its size. And in fact you might want to get the text out instead since that's encoded.

def f1(url):
    r = requests.get(url)
    return len(r.content)

f2()

If you stream it you can minimize memory bloat in your application since you can re-use the chunks of memory if you're able to do something with the buffered content. In this case, the buffer is just piled on in memory, 512 bytes at a time.

def f2(url):
    r = requests.get(url, stream=True)
    buffer = io.BytesIO()
    for chunk in r.iter_content(chunk_size=512):
        if chunk:
            buffer.write(chunk)
    return len(buffer.getvalue())

I did put a counter into that for-loop to see how many times it writes and if you multiple that with 512 or 1024 respectively it does add up.

f3()

Same as f2() but with twice as large chunks/

def f3(url):  # same as f2 but bigger chunk size
    r = requests.get(url, stream=True)
    buffer = io.BytesIO()
    for chunk in r.iter_content(chunk_size=1024):
        if chunk:
            buffer.write(chunk)
    return len(buffer.getvalue())

f4()

I'm actually quite new to boto3 (the cool thing was to use boto before) and from some StackOverflow-surfing I found this solution to support downloading of gzipped or non-gzipped objects into a buffer:

def f4(url):
    _, bucket_name, key = urlparse(url).path.split('/', 2)
    obj = s3.Object(
        bucket_name=bucket_name,
        key=key
    )
    buffer = io.BytesIO(obj.get()["Body"].read())
    try:
        got_text = GzipFile(None, 'rb', fileobj=buffer).read()
    except OSError:
        buffer.seek(0)
        got_text = buffer.read()
    return len(got_text)

Note how it doesn't try to find out if the buffer is gzipped but instead relying on assuming it is plus a raised exception.
This feels clunky, around the "gunzipping", but it's probably quite representative of a final solution.

Complete experiment code here

The Results

At first I ran this on my laptop here on my decent home broadband whilst having lunch. The results were very similar to what I later found on EC2 but 7-10 times slower here. So let's focus on the results from within an EC2 node in us-west-1c.

The raw numbers are as follows (showing median values):

Function	18MB file	Std Dev	1MB file	Std Dev
f1	1.053s	0.492s	0.395s	0.104s
f2	1.742s	0.314s	0.398s	0.064s
f3	1.393s	0.727s	0.388s	0.08s
f4	1.135s	0.09s	0.264s	0.079s

I ran each function 20 times. It's interesting, but not totally surprising that the function that was fastest for the large file wasn't necessarily the fastest for the smaller file.

The winners are f1() and f4() both with one gold and one silver each. Makes sense because it's often faster to do big things, over the network, all at once.

Or, are there winners at all?

With a tiny margin, f1() and f4() are slightly faster but they are not as convenient because they're not streams. In f2() and f3() you have the ability to do something constructive with the stream. As a matter of fact, in my application I want to download the S3 object and parse it line by line so I can use response.iter_lines() which makes this super convenient.

But most importantly, I think we can conclude that it doesn't matter much how you do it. Network I/O is still king.

Lastly, that boto3 solution has the advantage that with credentials set right it can download objects from a private S3 bucket.

Bonus Thought!

This experiment was conducted on a m3.xlarge in us-west-1c. That 18MB file is a compressed file that, when unpacked, is 81MB. This little Python code basically managed to download 81MB in about 1 second. Yay!! The future is here and it's awesome.

tl;dr; More experiments with different ways to lazy load CSS with and without web fonts. Keeping it simple is almost best but not good enough.

Background

Last week I blogged about Web Performance Optimization of a Single-Page-App and web fonts in which I experimented with tricks to delay the loading of some massive CSS file. The app is one of those newfangled single-page-app where JavaScript (React in this case) does ALL the DOM rendering, which is practical but suffers the first-load experience.

Again, let's iterate that single-page-apps built with fully fledged JavaScript frameworks isn't the enemy. The hope is that your visitors will load more than just one page. If users load it once, any further click within the app might involve some "blocking" AJAX but generally the app feels great since a click doesn't require a new page load.

Again, let's iterate what the ideal is. Ideally the page should load and be visually useful by the server sending all the DOM you almost need and once loaded, some JavaScript can turn the app "alive" so that further clicks just update the DOM instead of reloading. But that's not trivial. Things like react-router (especially version 4) makes it a lot easier but you now need a NodeJS server to render the pages server-side and you can't just easily take your "build" directory and put into a CDN like Firebase or CloudFront.

When I first started these experiements I was actually looking at what I can do with the JavaScript side to make first loads faster. Turns out fixing your CSS render blocking load and assessing use of web fonts has a bigger "bang for the buck".

Go Straight to the Results

Visual comparison of 6 different solutions

Same visual but as a comparative video

Stare at that for a second or two. Note that some versions start rendering something earlier than others. Some lose some of their rendered content and goes blank. Some change look significantly more than once before the final rendered result. As expected they almost all take the same total time till the final thing is fully rendered with DOM and all CSS loaded.

What Each Experimental Version Does

Original

A regular webapp straight out of the ./build directory built with Create React App. Blocking CSS, blocking JS and no attempts to show any minimal DOM stuff first.

Optimized Version 1

Inline CSS with basic stuff like background color and font color. CSS links are lazy loaded using tricks from loadCSS.

Optimized Version 2

Same as Optimized version 1 but both the JavaScript that lazy loads the CSS and the React JavaScript has a defer on it to prevent render blockage.

Optimized Version 3

Not much of an optimization. More of a test. Takes the original version but moves the <link rel="stylesheet" ... tags to just above the </body> tag. So below the <script src="..."> tags. Note how, around the 15 second mark, this renders the DOM (by the JavaScript code) before the CSS has loaded and it gets ugly. Note also how much longer this one takes because it has to context switch and go back to CSS loading and re-rendering after the DOM is built by the JavaScript.

Optimized Version 4

All custom web font loading removed. Bye bye "Lato" font-family. Yes you could argue that it looks worse but I still believe performance makes users more happy ultimately than prettiness (not true for all sites, e.g. a fashion web app).
Actually this version is copied from the Optimized version 1 but with the web fonts removed. The final winner in my opinion.

Optimized Version 5

Out of curiousity, let's remove the web fonts one more time but do it based on the Original. So no web fonts but CSS still render blocking.

So, Which Is Best?

I would say Optimized version 4. Remove the web font and do the loadCSS trick of loading the CSS after some really basic rendering but do that before loading the JavaScript. This is what I did to Podcast Time except slightly improved the initial server HTML so that it says "Loading..." immediately and I made the header nav look exactly like the one you get when everything is fully loaded.

Also, Chrome is quite a popular browser after all and the first browser that supports <link rel="preload" ...>. That means that when Chrome (and Opera 43 and Android Browser) parses the initial HTML and sees <link rel="preload" href="foo.css"> it can immediately start downloading that file and probably start parsing it too (in parallel). The visual comparison I did here on WebPagetest used Chrome (on 3G Emerging Markets) so that gives the optimized versions above a slight advantage only for Chrome.

What's Next?

To really make your fancy single-page-app fly like greased lightning you should probably do two things:

1. Server-side render the HTML but perhaps do it in such a way that if your app depends on a lot of database data, skip that so you at least send a decent amount of HTML so when React/Angular/YouNameIt starts executing it doesn't have to do much DOM rendering and can just go straight to a loading message whilst it starts fetching the actual AJAX data that needs to be displayed. Or if it's a news article or blog post, include that in the server rendered HTML but let the client-side load other things like comments or a sidebar with related links.

2. Tools like critical that figures out the critical above-the-fold CSS don't work great with single-page-apps where the DOM is generated by loaded JavaScript but you can probably hack around that by writing some scripts that generate a DOM as a HTML string by first loading the app in PhantomJS in various ways. That way you can inline all the CSS needed even after the JavaScript has built the DOM. And once that's loaded, then you can lazy load in the remaining CSS (e.g. bootstrap.min.css) to have all styles ready when the user starts clicking around on stuff.

Most important is the sites' functionality and value but hopefully these experiments have shown that you can get pretty good mileage from your standard build of a single-page-app without having to do too much tooling.

tl;dr; Don't worry so much about your bloated JavaScript and how to async it. Worry about your CSS and your Web fonts. Here I demonstrate a web performance optimization story with Webpagetest.org results demonstrating the improvements.

The app playground

I have a little app called Podcast Time. It's a single-page-app that renders all the functionality in React with the help of MobX and I use Bootstrap 4 and Bootswatch to make it "pretty". And that Bootswatch theme I chose depends on the Lato font by Google Fonts.

I blogged about it last month. Check it out if you're curious what it actually does. As with almost all my side-projects, it ultimately becomes a playground to see how I can performance optimize it.

The first thing I did was add a Service Worker so that consecutive visits all draws from a local cache. But I fear most visitors (all 52 of them (I kid, I don't know)) come just once and don't benefit from the Service Worker. The compiled and minified JavaScript weighs 124 KB gzipped (440 KB uncompressed) and the CSS weighs 20 KB gzipped (151 KB uncompressed). Not only that but the bootstrap.min.css contained an external import to Google Fonts's CSS that loads the external font files. And lastly, the JavaScript is intense. It's almost 0.5 MB of heavy framework machinery. Of that JavaScript about 45 KB which is "my code". By the way, as ES6 code it's actually 66 KB that my fingers have typed. So that's an interesting factoid about the BabelJS conversion to ES5 code.

Anyway, to make site ultra fast the bestest thing would be to rewrite all the JavaScript in vanilla JavaScript and implement, with browser quirks (read: polyfills), HTML5 pushState etc. Also, I guess I should go through the bootstrap.css file and extract exactly only the selectors I'm actually using. But I'm not going to do that. That would be counter productive and I wouldn't have a chance to add the kind of features I have and can add. So that fat JavaScript and that fat CSS needs to be downloaded.

So what can I do to make that slightly less painful? ...when Service Workers isn't an option.

1. The baseline

Here's the test results on Webpagetest.org

Click on the first Filmstrip View and notice that the rendering both starts and finishes between the 3.0 and 3.5 second mark. The test is done using Chrome on what they call "Mobile 3G Fast".

2. Have some placeholder content whilst it's downloading

I poked around at the CSS to get some really basics out of it. The background color, the font color, the basic grid. I then copied that into the index.html as an inline stylesheet. You can see it here.

The second thing I did was to replace all <link rel="stylesheet" href="..."> meta tags with this:

<link rel="preload" href="FOO.css" as="style" onload="this.rel='stylesheet'">
<noscript><link rel="stylesheet" href="FOO.css"></noscript>

Also, near the bottom of the index.html I took the loadCSS.js and cssrelpreload.js scripts from Filament Group's loadCSS solution.

And lastly I copied the header text and the sub-title from the DOM, once rendered, plus added a little "Loading..." paragraph and put that into the DOM element that React mounts into once fully loaded.

Now the cool effect is that the render blocking CSS is gone. CSS loads AFTER some DOM rendering has happened. Yes, there's going to be an added delay when the browser has to re-render everything all over again, but at least now it appears that the site is working whilst you're waiting for the site to fully load. Much more user-friendly if you ask me.

Here's the test results on Webpagetest.org

Click on the first Filmstrip View and notice that some rendering started at the 0.9 second mark (instead of 3 seconds in the previous version). Then, after the browser notices that it's not the right font so it blanks the DOM, waits for the web font to load and around the time the web font is loaded, the JavaScript is loaded and ready so it starts rendering the final version at around the 2.8 second mark.

Note that the total time from start to finish is still the same. 3.1 second vs. 3.5 second. It's just that in this second version the user gets some indication that the site is alive and loading. I think the pulsating "Loading..." message puts the user at ease and although they're unhappy it still hasn't loaded I think it buys me a smidge of patience.

3. Hang on! Are web fonts really that important?

That FOIT is still pretty annoying though. When the browser realizes that the font is wrong, it makes the DOM blank until the right font has been loaded. My poor user has to stare at a blank screen for 1.3 seconds instad of seeing the "Loading..." message. Not only that but instead of just saying "Loading..." this could be an opportunity to display something more "constructive" that the user can actually start consuming with their eyes and brains. Then, by the time the full monty has been downloaded and ready the user will not have been waiting in vain.

So how important are those web fonts really? It's a nerdy app where search, content and aggregates matter more than the easthetic impression. Sure, a sexy looking site (no matter what the context/environment) does yield a better impression but there's tradeoffs to be made. If I can load it faster, my hunch is that that'll be more impressive than a pretty font. Secs sells.

I know there are possible good hacks to alleviate these FOIT problems. For example, in "How We Load Web Fonts Progressively" they first set the CSS to point to a system font-family, then, with a JavaScript polyfill they can know when the external web font has been downloaded and then they change the CSS. Cute and cool but it means yet another piece of JavaScript that has to be loaded. It also means "FOUT" (Flash Of Unstyled Text) which can disturb the view quite a lot if the changed font changes layout shifts.

Anyway, I decided I'll prefer fast loading over pretty font so I removed all font family hacks and references to the external font. Bye bye!

Here's the test results on Webpagetest.org

Click on the first Filmstrip View and notice that the rendering starts already at 1.2 seconds and stay with the "Loading..." message all the way till around 3.0 seconds when both the CSS and the full JavaScript has been loaded.

Side note; Lato vs. Helvetica Neue vs. Helvetica

You decide, is the Lato font that much prettier?

Lato

Helvetica Neue

Helvetica

In Conclusion

I started this exploration trying to decide if there are better ways I can load my JavaScript files. I did some experiments with some server-side loading and async and defer. I scrutinized my ES6 code to see if there are things I can cut out. Perhaps I should switch to Preact instead of React but then I'd have to own the whole es-lint, babel and webpack config hell that create-react-app has eliminated and I'd get worried about not being able to benefit from awesome libraries like MobX and various routing libraries. However, when I was doing various measurements I noticed that all the tricks I tried, for a better loading experience, dwarfed in comparison to the render blocking CSS and the web font loading.

Service Workers is to web apps, what smartphone native apps are to mobile web pages in terms of loading performance. But Service Workers only work if you have a lot of repeat users.

If you want a fast loading experience and still have the convenience of building your app in a powerful JavaScript framework; focus on your CSS and your web font loading.

I had this piece of code:

new_all_ids = set(
    x for x in all_ids if x not in to_process_ids
)

The all_ids is a list object of 1.1 million IDs. Some repeated. And to_process_ids is a list sample of 1,000 randomly selected IDs from that list but all unique. The objective of the code was to first extract 1,000 IDs, do something when them, then remove those 1,000 from the original list and once that's done, update all_ids back with those from to_process_ids removed.

Only problem was that I noticed that this operation took 30+ seconds! How can it take 30 seconds to do a little bit of list comprehension? The explanation is the lack of index lookups.

What the code actually does is this:

new_all_ids = set()
for x in all_ids:
    not_found = False
    for y in to_process_ids:
        if y != x:
            not_found = True
    if not_found:
         new_all_ids.add(x)

Can you see it? It's doing a nested loop. Or, O(n^2) in computer lingo. That's 1.1 million * 1,000 iterations of that conditional.

What sets do, on the other hand, is that they convert the list of keys in the set to a hash table. Kinda similar to how dict works except it doesn't point to a value. That means that asking a set if it contains a certain key is a O(1) operation.

You might have spotted the solution already.
If you instead do:

to_process_ids_set = set(to_process_ids)
new_all_ids = set(
    x for x in all_ids if x not in to_process_ids_set
)

Now, in my particular example, instead of taking 30+ seconds this implementation only takes 0.026 seconds. 1,000 times faster.

You might also have noticed that there's a much more convenient way to do this very same thing, namely set operations! The desired new_all_ids is going to become a set anyway. And if we can first convert it to a set, then do the operation on it we can avoid looping over repeats as we do the checking.

Final solution:

new_all_ids = set(all_ids) - set(to_process_ids)

You can try it yourself with this little benchmark:

"""
Demonstrate three ways how to reduce a non-unique list of integers
by 1,000 randomly selected unique ones.
Demonstrates the huge difference between lists and set lookups.
"""
import time
import random


# Range numbers chosen quite arbitrarily to get a benchmark that lasts
# a couple of seconds with a realistic proportion of unique and 
# repeated integers.
items = 200000
all_ids = [random.randint(1, items * 2) for _ in range(items)]

print('About', 100. * len(set(all_ids)) / len(all_ids), '% unique')

to_process_ids = random.sample(set(all_ids), 1000)


def f1(to_process_ids):
    return set(x for x in all_ids if x not in to_process_ids)


def f2(to_process_ids):
    to_process_ids_set = set(to_process_ids)
    return set(x for x in all_ids if x not in to_process_ids_set)


def f3(to_process_ids):
    return set(all_ids) - set(to_process_ids)


functions = [f1, f2, f3]
results = {f.__name__: [] for f in functions}
for i in range(10):
    random.shuffle(functions)
    for f in functions:
        t0 = time.time()
        f(to_process_ids)
        t1 = time.time()
        results[f.__name__].append(t1 - t0)

for function in sorted(results):
    print(function, sum(results[function])/ len(results[function]))

When I run that with my Python 3.5.1 I get:

About 78.616 % unique
f1 3.9494598865509034
f2 0.041156983375549315
f3 0.02245485782623291

Seems to match expectations.

tl;dr; In other words, you can now have multiple servers with crontabber, all talking to a central PostgreSQL for its state, and not have to worry about jobs being started more than exactly once. This will be super useful if your crontabber apps are such that they kick of stored procedures that would freak out if run more than once with the same parameters.

crontabber is an advanced Python program to run cron-like applications in a predictable way. Every app is a Python class with a run method. Example here. Until version 0.18 you had to do locking outside but now the locking has been "internalized". Meaning, if you open two terminals and run python crontabber.py --admin.conf=myconfig.ini in both you don't have to worry about it starting the same apps in parallel.

General, business logic locking

Every app has a state. It's stored in PostgreSQL. It looks like this:

# \d crontabber
              Table "public.crontabber"
    Column    |           Type           | Modifiers
--------------+--------------------------+-----------
 app_name     | text                     | not null
 next_run     | timestamp with time zone |
 first_run    | timestamp with time zone |
 last_run     | timestamp with time zone |
 last_success | timestamp with time zone |
 error_count  | integer                  | default 0
 depends_on   | text[]                   |
 last_error   | json                     |
 ongoing      | timestamp with time zone |
Indexes:
    "crontabber_unique_app_name_idx" UNIQUE, btree (app_name)

The last column, ongoing used to be just for the "curiosity". For example, in Socorro we used that to display a flashing message about which jobs are ongoing right now.

As of version 0.18, this ongoing column is actually used to NOT run apps again. Basically, when started, crontabber figures out which app to run next (assuming it's time to run it) and now the first thing it does is look up if it's ongoing already, and if it is the whole crontabber application exits with an error code of 3.

Sub-second locking

What might happen is that two separate servers which almost perfectly synchronoized clocks might have cron run crontabber at the "exact" same time. Or rather, only a few milliseconds apart. But the database is central so what might happen is that two distinct PostgreSQL connection tries to send a... UPDATE crontabber SET ongoing=now() WHERE app_name='some-app-name' at the very same time.

So how is this solved? The answer is row-level locking. The magic sauce is here. You make a select, by app_name with a suffix of FOR UPDATE WAIT. Imagine two distinct PostgreSQL connections sending this:

BEGIN;
SELECT ongoing FROM crontabber WHERE app_name = 'my-app-name'
FOR UPDATE NOWAIT;

-- do some other stuff in Python

UPDATE crontabber SET ongoing = now() WHERE app_name = 'my-app-name';
COMMIT;

One of them will succeed the other will raise an error. Now all you need to do is catch that raised error, check that it's a row-level locking error and not some other general error. Instead of worrying about the raised error you just accept it and exit the program early.

This screenshot of a test.sql script demonstrates this:

Two terminals lined up and I start one and quickly switch and start the other one

Another way to demonstrate this is to use psycopg2 in a little script:

import threading
import psycopg2


def updater():
    connection = psycopg2.connect('dbname=crontabber_exampleapp')
    cursor = connection.cursor()
    cursor.execute("""
    SELECT ongoing FROM crontabber WHERE app_name = 'bar'
    FOR UPDATE NOWAIT
    """)
    cursor.execute("""
    UPDATE crontabber SET ongoing = now() WHERE app_name = 'bar'
    """)
    print("JOB CAN START!")
    connection.commit()


# Use threads to simulate starting two connections virtually 
# simultaneously.
threads = [
    threading.Thread(target=updater),
    threading.Thread(target=updater),
]
for thread in threads:
    thread.start()
for thread in threads:
    thread.join()

The output of this is:

▶ python /tmp/test.py
JOB CAN START!
Exception in thread Thread-1:
Traceback (most recent call last):
...
OperationalError: could not obtain lock on row in relation "crontabber"

With threads, you never know exactly which one will work and which one will not. In this case it was Thread-1 that sent its SQL a couple of nanoseconds too late.

In conclusion...

As of version 0.18 of crontabber, all locking is now dealt with inside crontabber. You still kick off crontabber from cron or crontab but if your cron does kick it off whilst it's still in the midst of running a job, it will simply exit with an error code of 2 or 3.

In other words, you can now have multiple servers with crontabber, all talking to a central PostgreSQL for its state, and not have to worry about jobs being started more than exactly once. This will be super useful if your crontabber apps are such that they kick of stored procedures that would freak out if run more than once with the same parameters.