Django under the Hood - Testing

Speaker: Ana Balica, works for Potato (London).

Archeology

  • 1.0
    • Ticket 2333: add test framework, ./manage.py test
    • The TestRunner does setup, points to method executing tests.py and doctests, teardown
    • TestCase provides Client, bypassing WSGI
  • 1.1
    • We get PUT, HEAD, DELETE, OPTIONS
    • TestCase gets transactions for rollbacks on test data
    • TransactionTestCase flushes the database instead
  • 1.2
    • DjangoTestSuiteRunner returned number of failed tests as exit codes which does not work for more than 255 failed tests, and also disregards common system codes.
    • Support for multiple databases: by redirecting all replica requests to the main database in tests
  • 1.3
    • RequestFactory does the request and Client inherits from it (even this should be moved to composition instead)
    • doctests: Neither high-quality tests nor high-quality documentation, discouraged
    • @skipIfDBFeature, @skipUnlessDBFeature
  • 1.4
    • SimpleTestCase (no DB hits, no setup, no teardown)
    • LiveServerTestCase (based on transaction, runs server)
    • New Tutorial for testing
    • New assertions for JSON etc
    • Python3
    • unittest2
  • 1.5
    • TransactionTestCase flushes after instead of before TestCase, not leaving a dirty state
  • 1.6
    • PATCH
    • Test discovery
    • Full paths instead of pseudo paths
    • doctest removal
  • 1.7
    • StaticLiveServerTestCase
  • 1.8
    • TRACE (complete verbs)
    • TestCase now encapsulates fixtures in their own additional transaction
  • 1.9
    • Parallel testing with multiprocessing
    • Also in Django's own tests
    • Find number of workers, build a database per test
    • Partition tests, and workers grab tests
    • For older versions, there is nose with multiprocessing (better than none)
  • 1.10
    • tag as decorator to group or exclude tests

Test Bed

  • ./manage.py test
  • Instantiate TestRunner, and run tests
  • setup_test_environment
    • Locmem email
    • Instrumented test render method, sending signals prior to rendering
    • Deactivate translations
  • build_suite: aggregation of all relevant tests
  • setup_databases
  • run_suite
  • teardown_databases
  • teardown_test_environment (reverse of setup_test_environment)
  • Return amount of failures and successes

Classes

  • SimpleTestCase has the Client, but no DB
  • TransactionTestCase: flushes per test, is slow
  • TestCase: faster, transaction per test
  • LiveServerTestCase: launches server
  • StaticLiveServerTestCase: launches server, serving also static files

Client

  • RequestFactory: constructs requests and encodes data
  • ClientHandler: inherits from HTTPBaseHandler, returns response, emulates browser behaviour, loads middleware, disables CSRF
  • Client: stateful, counts responses, and knows about contest, handles redirects

Quality

  • Introduce FactoryBoy
    • Replace fixtures with random and realistic values
    • Those are generated by Faker
  • Or use Hypothesis
    • using randomized-ish generated values
    • provide a description of the parameters to be tested (think functional)
    • Property based testing might look complicated
  • Quality is not expressed by Coverage
    • Low coverage is bad
    • High coverage does not imply high quality of tests
  • Mutation testing (mutpy)
    • Run tests on code and mutated code
    • If tests fail on mutant, it is killed and raises our score
    • If tests don't fail on the mutant
    • Mutates logic, decorator, constant, conditionals, break points
    • Mutation score: low is worse

Django testing tutorial

  • Yay, it's in the tutorial.
  • Yay, it's in startproject

More is better

  • More is not better: slow tests suck
  • Use MD5PasswordHasher
  • Use an in-memory sqlite3 if doable
  • Write more SimpleTestCases
  • Use setUpTestData
  • Don't use mocks everywhere
  • Optimize setUp()
  • Don't save unless necessary (in-memory model, or build() or stub() in FactoryBoy)
  • Isolate unit tests from the rest, due to their speed, and tag them

Write better code by writing better tests

  • Isolate business logic in your code, so that you can isolate it in your tests

Q&A

  • Do you have an opinion on if the Django TestRunner is necessary?
    • It could be stripped, but it's too tied into internals to be able to remove.
  • What are you missing from Django testing tools?
    • Pluggable components, Channels will bring improvements, too.
  • Hypothesis might hurt with ORM performance - isn't that bad?
    • Use hypothesis testing for unit tests, and if you really need them, just run them on a CI against the DB.
  • How can Django encourage devs to pull logic out of code?
    • Be more explicit about not bloating views etc, improve docs.
  • How do you go about finding your problems in an existing test suite?
    • Run modules separately. (Side note by me: for pytest there is pytest-profiling)
  • Migrations might take a looong time in setup/teardown …
    • Ana doesn't have that problem due to the DataStore. Maybe concatenate/squash? There is also keepdb
  • Multiple DBs are hard, Django only flushes the default one.
    • Ana recommended a more pluggable system anyways, it's not easily doable.
  • Testing with files?
    • Minimum possible example. Don't muck about with the storage backend.
  • What can Django do to improve their own tests?
    • 10 minutes parallelized is okay. Maybe try decorators for local vs remote environments? FactoryBoy/Hypothesis might be cool, too.