Django under the Hood - Debugging

Speaker: Aymeric Augustin: DDTB maintainer for a while, in Django: Python3, templating stuff with Jinja2, Time zones,and much more.

Links: App and Code, guide on DRF profiling

Response Times

  • Fast is up to 0.1s
  • Normal is up to a second
  • Anything faster catches the user's flow of thought

Measuring page load time

  • Chrome Developer Tools, emulate bad network speed
  • Google Analytics, sampling 1% of users
  • Application performance monitoring solutions

Timeline

  • Redirects
  • App Cache for some apps
  • DNS lookup
  • TCP handshaking
  • Start request
  • Receive response
  • Process response
    • domLoading
    • No event
    • Parse HTML, build DOM
    • Download and run sync js
    • domInteractive
    • DOMCOntentLoades event
    • Download CSS and images
    • Parse CSS, build CSSOM
    • domComplete
    • Page is fully "loaded"
    • Load event
  • onLoad

Consequences

  • DNS cache helps a lot
  • Backend varies a lot, but 10%-20% is expected
  • HTTP 1 is fairly bad at this - HTTP2 can do much more
  • We cache, we minify, we inline, we use other connections for assets
  • Only proper way to cache JS and CSS: put hash in filename, set validity to infinite.
  • Use HSTS, to remove a redirect (moves redirect to browser)
  • Critical Path: HTML ➡️ CSS ➡️ Sync JS on CSS
    • CSS on top, JS on bottom
  • Browsers optimize heavily
    • Parse HTML incrementally
    • Paint while waiting for Sync JS, after CSS
    • Paint while waiting for fonts
    • Preload scanner

Order or optimizations - Frontend

  1. Optimize HTML load time
  2. Optimize CSS load time
  3. Avoid sync JS, including inline, or put it at the bottom

Sync scripts

  • Download & inject script via another script (this does not block)
  • Use <script async src=''></script>
    • For non-critical: in the body
    • For critical: in the head

Order of optimizations - Backend

  1. Look at SQL queries
  2. select_related
    • 1/N to 1 only
    • more elegant, but not always faster, fetching some instance repeatedly (more data)
  3. prefetch_related
    • use a Prefetch object for ordering
    • Makes several queries, resulting in transactional consistency issues
  4. Look at DB interaction
  5. use only, defer, and especially values_list, values
  6. use aggregate and annotate
  7. use iterator() (not a common use case, all data is still fetched)

Q&A

  • Async CSS?
    • Kind of, inline CSS later on, it might help a bit.
  • Content hash vs e-tag?
    • Didn't manage to make it work in all the browsers due to gzipping etc. Good idea, but not practical.
  • Future version where ORM would error out if queries were not prefetched?
    • Tricky to read minds … Could be as doable as it is annoying.
  • Does iterate() cache and re-use?
    • No, we want garbage collection here.
  • What would make debugging easier with annotate() and aggregate() in combination with values_list()?
    • Perhaps better documentation is a good way to go here.