🔖 Day22 - Django with AWS S3

2018 - 06 - 26
🔖 Day22 - Django with AWS S3
Static is always a topic in Django projects that it requires careful settings before the projects can work smoothly. In Django, static refers to a collections of static files or a static folder which stores the common settings of your web applications such as CSS, Javascripts. Usually we will have the static folder put right inside the project folder, at the same level of manage.py. By calling collectstatic, Django gathers the static files from this folder before running the project such that it knows where to lookup when you use the tag {% load static %}.
0. [Advantages] The advantages of using AWS S3 include but not limit to:
  • Cloud storages which means you can save more spaces for your project
  • Secure online storages with AWS IAM which we will talk about later
  • Unlimited spaces for uploads
1. [Register and IAM] First of all, register an AWS account to use this amazing service. Note that this is not the same as your amazon account for shopping.
  • Go to https://aws.amazon.com/free/ and create an account.
    • Once you are registered and logged into the console, open the Services tab on the left top corner, and find a service named IAM under 'Security, Identity & Compliance'. IAM is short for 'Identity and Access Management' where you can create groups or users to use your AWS services. Instead of using your AWS root account, the benefit of using IAM group+user is that you can control their permission to use your AWS services. Furthermore, it also helps to protect your files that every user needs credential key ID + secret key to access (i.e. programmatic access). Read their documents here: https://aws.amazon.com/documentation/iam/ for more detail.
  • So in the IAM page, click 'Users' tab and 'Add User'. Give it a username such as 'django-user' or anything you prefer. Check the 'Programmatic Access' checkbox and proceed.
    • Now, we need to create a 'Group' for that 'django-user' in order to give it rights to access our coming AWS S3.
  • Click 'Create Group' and assign a name e.g. 'django-group'. Search for the policy 'AmazonS3FullAccess' and check the box. Click 'Create Group' and 'Next: Review'.
In this page, you will found the Access key ID and Secret Access for this group of user(s). You may cap-screen for the detail but if you forgot to do so, no worries because you can find them again from the Users tab.
2. [Set up S3] After creating IAM, we are now going to register the de facto AWS S3. Click 'Create Bucket', assign a name for your S3 bucket(the cloud storage), e.g. 'django-s3'. Choose the region you preferred then click 'Next' to finish.
3. [Install packages] There are 2 packages you will need for assigning the storages option.
  • Open terminal and type: 
    $pipenv install boto3 django-storages
  • Add storages to INSTALLED_APP in settings.py: 
    INSTALLED_APP = [
	...,
	'storages'
]
4. [Django Settings] Here comes the main part. In this session, we need to let Django knows our 'static' location is now moved to AWS. To do so, we need to connect AWS via the 'group&user' created earlier.
  • In settings.py, head to your 'static' settings and replace it into these: (Following keys are not real, only resemble) 
    #AWS S3 connection
AWS_ACCESS_KEY_ID = 'HASBDJASBDBBNBDSLFHA'
AWS_SECRET_ACCESS_KEY = 'bhasd7ASVDLASJDB9ADNASDBLSDBJASDB+SDNads/ASDHJ'
AWS_STORAGE_BUCKET_NAME = 'django-s3'
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
AWS_S3_OBJECT_PARAMETERS = {'CacheControl': 'max-age=86400'}
    • Now this part tells how Django should connect to AWS S3. Note that those private data such as ID, keys and name should be enclosed in .env file by using Python-Decouple as mentioned in previous posts.
  • Next we have to tell it where and in what manner should it store the 'static', 'private media' and 'public media' files. 
    #AWS S3 Static
AWS_STATIC_LOCATION = 'static'
STATIC_URL = 'http://%s/%s/' % (AWS_S3_CUSTOM_DOMAIN, AWS_STATIC_LOCATION)
STATICFILES_DIR = [os.path.join(BASE_DIR, 'static')]
STATICFILES_STORAGE = '<YOUR_PROJECT_NAME>.storage_backends.StaticStorage'

#AWS S3 Private Media Upload
AWS_MEDIA_LOCATION = 'media'
PRIVATE_FILE_STORAGE = '<YOUR_PROJECT_NAME>.storage_backends.MediaStorage'

#AWS S3 Public Media Upload
AWS_PUBLIC_LOCATION = 'public'
DEFAULT_FILE_STORAGE = '<YOUR_PROJECT_NAME>.storage_backends.PublicStorage'
    • Here we tells django we are going to use 3 different folders inside the AWS S3 bucket, 'static', 'media' and 'public'. Such that static files such as CSS and JS will be kept in 'static', files uploaded by you/web admin will be stored in 'media', and visitor may upload files into 'public' folder.
  • Then we have to create our python file storage_backends.py so that the snippet above can make use of. create a new file named as storage_backends.py and save it next to settings.py. Make the content looks like this: 
    from django.conf import settings
from storages.backends.s3boto3 import S3Boto3Storage

class StaticStorage(S3Boto3Storage):
	location = settings.AWS_STATIC_LOCATION

class MediaStorage(S3Boto3Storage):
	location = settings.AWS_MEDIA_LOCATION
	default_acl = 'private' #To turn access control list into private use only. Will use it in models.py
	file_overwrite = False #Not to replace files even they have same name
	custom_domain = False

class PublicStorage(S3Boto3Storage):
	location = settings.AWS_PUBLIC_LOCATION
	file_overwrite = False
5. [FileField or ImageField in models] By this far, the settings are almost ready. The things that you still have to amend is models.py where you need to tell which field you are going to make it 'private upload only'.
  • Open models.py and look for the fields that you wish to restrict as admin upload only: 
    from <YOUR_PROJECT_NAME>.storage_backends import MediaStorage <--
from django.db import models

class Profile(models.Model):
	name = models.CharField(max_length=50)
	birthday = models.DateField()
	picture = models.ImageField(storage=MediaStorage()) <--
    • So that the Django model understand this ImageField is set as MediaStorage which has default_acl set as private.
    6. [Collectstatic] Before started running the web application, you have to call collectstatic locally in order to have your staticfiles gathered/uploaded to your AWS S3 bucket 'django-s3'. Note, always change back to local database when you want to run Django via local manage.py: 
    """
import dj_database_url
DATABASES = {
    'default': dj_database_url.config(
        default=config('DATABASE_URL')
    )
}
"""

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'database_name',
        'USER': 'postgres',
        'DBPASSWORD': config('DBPASSWORD'),
        'HOST': '127.0.0.1',
        'PORT': '5432'
    }
}
    Make sure local database is running, open terminal and collectstatic: 
    $pipenv run python manage.py collectstatic
    This takes much longer than it usually does as it uploads to AWS. Once finished, take a look at the AWS S3 online, you will see a folder named 'static' created. All your static files are now stored in there.
    7. [Deploy to Heroku] Quite a number of things are set. Now tell Heroku what we upgraded:
    • You have installed 2 new packages, so you need to include them into your requirements.txt: 
      boto3==1.5.29
django-storages==1.6.5
    • Also some AWS keys and ID entered in .env should be entered manually on Heroku dashboard:
    • Change back to dj-database-url for using database on Heroku: 
      import dj_database_url
DATABASES = {
    'default': dj_database_url.config(
        default=config('DATABASE_URL')
    )
}

"""
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'database_name',
        'USER': 'postgres',
        'DBPASSWORD': config('DBPASSWORD'),
        'HOST': '127.0.0.1',
        'PORT': '5432'
    }
}
"""
    Git add, commit and push the changes to Heroku. Now everything should be running smoothly with AWS S3! If you used Whitenoise as static files adaptor previously, you may remove it and all related settings! As you are now using the S3! p.s. As you no longer use the static files in your project on Heroku platform but AWS S3, you may just simply disable collectstatic function (I don't recommend) by typing: 
    $heroku config:set DISABLE_COLLECTSTATIC=1
** DISABLE_COLLECTSTATIC=0 to enable it **
    So that Heroku simply skips the procedures to collectstatic during deployment. Since you actually collectstatic from local command already, it really doesn't matter.

Comments

There is no comment yet

New Comment

Please Login to comment

Sections

Tweets

Pinterest

About Me

Kavie. A registered optometrist from HK. An ICT graduate of University of Tasmania. Have a passion in webdev and machine learning.
This site is built in Python & Django, JS ES6 & jQuery & Bootstrap.

Contact

Social Networks
Email
devjuniorkm@gmail.com