When I attended the London Django Meetup in May one of my fellow attendees asked how best to test files uploaded to Amazon S3 via a form. I used the question as the basis of a talk I gave at a later meet up in July. This post is the written form of that talk I gave.
When you test, patch network requests
When I write tests I try and patch all external network requests in order to:
- Isolate the behaviour of my code during testing.
- Speed up the test by avoiding network operations.
- Avoid leaving leftover files in storage buckets in the cloud.
Get the boilerplate code out of the way
The full source code for this tutorial can be found on GitHub.
To start, I installed various requirements I'll need both to run and to test this project:
$ cat requirements.txt
Django==1.6.5
argparse==1.2.1
boto==2.30.0
django-boto==0.3.1
django-storages==1.1.8
mock==1.0.1
wsgiref==0.1.2
$ pip install -r requirements.txt
I then created a Dajngo project called meetup-testing and created a candidate application within it. Here is the layout of the files and folders:
$ tree
.
├── base
│ ├── __init__.py
│ ├── local_settings.py
│ ├── settings.py
│ ├── urls.py
│ └── wsgi.py
├── candidate
│ ├── admin.py
│ ├── fixtures
│ │ └── gradient.jpg
│ ├── forms.py
│ ├── __init__.py
│ ├── models.py
│ ├── tests.py
│ └── views.py
├── manage.py
├── README.md
├── requirements.txt
└── templates
├── candidate.html
└── thanks.html
I use boto to perform the I/O between my server and Amazon S3. It requires an access key id, secret access key and bucket name. I've kept these out of settings.py and instead placed them in local_settings.py which is excluded from the git repo. I also took SECRET_KEY out of the boilerplate settings.py Django generates and placed it in local_settings.py as well.
$ cat base/local_settings.py
AWS_S3_ACCESS_KEY_ID = 'Get this from Amazon'
AWS_S3_SECRET_ACCESS_KEY = 'This as well'
AWS_STORAGE_BUCKET_NAME = 'your-s3-bucket-name'
SECRET_KEY = 'A long string with many different types of characters'
In settings.py I made the following additions to the boilerplate file:
import os
BASE_DIR = os.path.dirname(os.path.dirname(__file__))
TEMPLATE_DIRS = os.path.join(BASE_DIR, 'templates')
INSTALLED_APPS = (
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'storages',
'candidate',
)
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
And, at the very bottom of settings.py I added the following in:
try:
from local_settings import *
except ImportError:
pass
assert len(SECRET_KEY) > 20, 'Please set SECRET_KEY in local_settings.py'
Writing the application code
I wrote a model called Candidate, created a form for it and used those in a view method:
candidate/models.py:
from django.db import models
class Candidate(models.Model):
first_name = models.CharField(max_length=30)
photo = models.FileField(upload_to='candidate-photos')
candidate/forms.py:
from django.forms import ModelForm
from candidate.models import Candidate
class CandidateForm(ModelForm):
class Meta:
model = Candidate
candidate/views.py:
from django.core.urlresolvers import reverse
from django.http import HttpResponseRedirect
from django.shortcuts import render
from candidate.forms import CandidateForm
def create(request):
if request.method == 'POST':
form = CandidateForm(request.POST, request.FILES)
if form.is_valid():
form.save()
return HttpResponseRedirect(reverse('created'))
else:
form = CandidateForm()
return render(request, 'candidate.html', {
'form': form,
})
def created(request):
return render(request, 'thanks.html')
Normally I create a urls.py for each application in a Django project but since this was a 2-view project I just used the main base/urls.py file to configure the two endpoints:
base/urls.py:
from django.conf.urls import patterns, url
urlpatterns = patterns('',
url(r'^$', 'candidate.views.create', name='create'),
url(r'^thanks/', 'candidate.views.created', name='created'),
)
Testing uploads
With the application built I wanted to write a test to make sure the form will accept a file, a first name and save them.
I created a small JPEG file and saved it to candidate/fixtures/gradient.jpg. This will be the example image I upload when I'm testing the form.
The default file storage class in settings.py is S3BotoStorage. This class will upload any files to whichever cloud provider configured. I don't want to upload to the cloud when I'm running my tests so I've patched S3BotoStorage with FileSystemStorage.
But by using FileSystemStorage I'll end up with any files uploaded during testing being stored in my Django project's folder. To avoid this littering I'll create a temporary folder, save the files there and then remove that folder when testing is complete.
Here is candidate/test.py:
from os.path import abspath, join, dirname
from shutil import rmtree
from tempfile import mkdtemp
from django.core.files.storage import FileSystemStorage
from django.core.urlresolvers import reverse
from django.test import TestCase
from django.test.client import Client
from django.test.utils import override_settings
import mock
class ViewTest(TestCase):
def setUp(self):
self.client = Client()
self.media_folder = mkdtemp()
def tearDown(self):
rmtree(self.media_folder)
@mock.patch('storages.backends.s3boto.S3BotoStorage', FileSystemStorage)
def test_post_photo(self):
photo_path = join(abspath(dirname(__file__)), 'fixtures/gradient.jpg')
with open(photo_path) as photo:
with override_settings(MEDIA_ROOT=self.media_folder):
resp = self.client.post(reverse('create'), {
'first_name': 'Test',
'photo': photo
})
redirect = 'Location: http://testserver%s' % reverse('created')
self.assertTrue(redirect in str(resp))
Now I can run the test command and see that my test passes:
$ python manage.py test -v1
Creating test database for alias 'default'...
.
----------------------------------------------------------------------
Ran 1 test in 0.026s
OK
Destroying test database for alias 'default'...