The instructions to set up an AWS instance here is adapted from EECS 485’s tutorial.

Create an account

Create an AWS account at the AWS Registration. You should be eligible for their free tier, which means that you will be able to run an instance for free for the duration of the course.

Despite that, you will need to enter a credit card number on the account, even if you only use free tier resources. This is how Amazon charges, in case you request more resources than provided in the free tier. Do not launch any additional instances other than the one we specify in order to avoid additional charges.

Optionally, you may redeem extra benefits as a student, including $100 in AWS credits.

Start instance

Navigate to the AWS Management Console. Select the “Services” dropdown menu, then “EC2”. An EC2 “instance” is a virtual machine running on Amazon AWS hardware.

Click launch an instance. It may take a few minutes to initialize before it’s running.

Select the “Ubuntu Server 22.04.01 [or 20.04] LTS” Amazon Machine Image (AMI).

Select the “t2.micro” instance type. You should see “free tier eligible”. When you create an instance, AWS automatically creates user “ubuntu” for you on the instance.

Create a key pair for user “ubuntu”. The private key will be automatically downloaded for you once the key pair is created. You’ll use this private key later to ssh to the instance. Click on Create new key pair:

Then enter “eecs441” as the key pair name and click Create key pair:

Allow SSH, HTTP, and HTTPS and click Edit on the top right:

Verify if three security group rules have been added. After this, Click on Add security group rule.

Edit the “Security group rule 4 (TCP, 0)” section, and add “Custom TCP” for port 8000 to allow Django traffic in and out of your instance. [Thanks to Z. Liang and R. Nagavardhan ‘23.]

Click “Launch Instance”.

Instance status

Navigate to the AWS Management Console. Select the “Services” dropdown menu, then “EC2”. Click “Instances”. Select one of the instances and view its status and Public DNS.

In the remainder of this spec, and this term, we will refer to your “IPv4 Public IP” as YOUR_SERVER_IP (in the image it’s 3.141.23.144).

ssh to instance

On your development host (laptop):

If AWS gave you a eec441.cer instead of eecs441.pem, just use eecs441.cer everywhere you see eecs441.pem in this spec.

MacOS on Terminal:

#         👇👇👇👇👇👇👇👇👇👇
laptop$ cd YOUR*LABS*FOLDER
laptop$ mv ~/Downloads/eecs441.pem eecs441.pem
laptop$ chmod 400 eecs441.pem
laptop$ ssh -i eecs441.pem ubuntu@YOUR_SERVER_IP
#                                 👆👆👆👆👆👆👆👆👆

Windows on PowerShell [thanks to Jad Beydoun (F21) for use of icacls]:

#            👇👇👇👇👇👇👇👇👇👇
PS laptop> cd YOUR*LABS*FOLDER
PS laptop> mv ~\Downloads\eecs441.pem eecs441.pem
PS laptop> icacls eecs441.pem /grant "$($env:username):(r)" /inheritance:r
PS laptop> ssh -i eecs441.pem ubuntu@YOUR_SERVER_IP
#                                    👆👆👆👆👆👆👆👆👆

  laptop$ sudo vi /etc/wsl.conf

  [automount]
  options = "metadata"

  #          👇👇👇👇👇👇👇👇👇
  laptop$ cd YOUR*LABS*FOLDER
  #                       👇👇👇👇👇👇👇👇👇👇👇👇👇
  laptop$ mv /mnt/c/Users/YOUR_WINDOWS_USERNAME/Downloads/eecs441.pem eecs441.pem
  laptop$ chmod 400 eecs441.pem
  laptop$ ssh -i eecs441.pem ubuntu@YOUR_SERVER_IP
  #                                 👆👆👆👆👆👆👆👆👆

In both cases, what the above does:

1. change working directory to YOUR*LABS*FOLDER,
2. move the private ssh key you created and downloaded earlier into YOUR*LABS*FOLDER,
3. set its permissions to read-only, and
4. ssh to your AWS instance as user “ubuntu” using the downloaded private key. (Make sure your instance is running. See Instance status.)

Stop instance

Check your Instance status.

Right click on your instance Instance State > Stop.

You should now see that your instance is stopped.

server$ sudo apt install cloud-utils awscli

server$ ec2metadata
server$ aws configure
server$ aws ec2 help

server$ aws iam list-access-keys

server$ aws ec2 authorize-security-group-ingress --group-id YOUR_GROUP_ID --ip-permissions IpProtocol=tcp,FromPort=8000,ToPort=8000,Ipv6Ranges=[{CidrIpv6=::/0}]

Google Cloud Platform has a free-tier, with free credits that are easy to qualify for.

Login to Google account

You’ll need a personal Google account to use the Google Cloud Platform. Do not use your umich email address. The following steps are adapted from Google Cloud’s Quickstart using a Linux VM, though they have been adapted to the course. For example, we choose an E2 instance that is eligible for the free-tier.

Create project

Go to the Google Cloud Platform Project Selector page and create a project. Click AGREE AND CONTINUE if you agreed to Google’s Terms of Service.

Create a project by clicking on the Create Project button.

Give your project a name unique to you and click Create.

Add billing

Add a billing method to your project.

The side menu may have additional items pre-pinned, however the two items we need Billing and Compute Engine are easily identifiable. Please consult the teaching staff if you couldn’t find any menu items referenced in this spec.

When you fill out your billing information, select “individual” as account type. Make sure you see something like this:

Add a credit or debit card. If your back end qualifies for free-tier (it should), this card will not be charged. Select START MY FREE TRIAL. Return to the project console.

Enable Compute Engine API

Visit GCP’s Compute Engine API site and select ENABLE.

Create VM instance

Return to the console. Hover over Compute Engine on the left navigation bar and select VM Instances.

Select CREATE INSTANCE.

Review the free tier options at the Google Cloud Free Program page by scrolling to the section titled “Free Tier usage limits”. Look under the “Compute Engine” section and check regions eligible for free tier. Free tier usage is governed by time used. Currently, an E2 in Oregon, Iowa, or South Carolina is eligible for free tier if used for the number of hours in a month.

Give the instance a name and carefully select the regions that are available as free tier with an e2-micro configuration.

The monthly estimate does not factor in free tier. If the steps are followed, your account should not be billed.

Scroll down until you see the **Boot disk**** section. Click “CHANGE” under Boot disk and configure it to Ubuntu 22.04.01 (or 20.04) LTS. Be sure to select STANDARD PERSISTENT DISK. Any other Boot Disk type option will cost you.

After you’ve chosen “Ubuntu 22.04.01 [or 20.04] LTS” and “Standard persistent disk”, click the blue SELECT button.

Back in the “Machine configuration” page, scroll down further, pass the “Boot disk” section, to get to the “Firewall” section. In the “Firewall” section, allow both HTTP and HTTPS traffic. You should see two boxes like this:

Press “CREATE” to create the instance. Wait for the instance to initialize. When the loading animations are done, write down the external IP address shown on the screen. In the remainder of this spec, and this term, we will refer to your “external IP” as YOUR_SERVER_IP (in the image it’s 34.138.61.201). You’ll never need the internal IP (and GCP doesn’t provide any public DNS for your instance).

Next select the triple dots on your E2. Select “Network Details”.

Select “Firewall”. We have to change one more firewall setting to allow us to test the web server we’ll be setting up later.

Create a firewall rule.

Give your rule a name. Scroll down to “Targets”. Enter “http-server” into the “Target tags” box. Enter 0.0.0.0/0 into the source IPv4 ranges box. Enter port 8000 into the tcp box. Press “CREATE”.

ssh to instance

The back-end specs in this course will assume that you have a user with username “ubuntu” created on your instance. The specs further assume that you’re doing all your back-end work, building your server, under user “ubuntu”. You’re free to build under a different username, e.g., your Google account name, however you will then have to map between the instructions in the specs and your setup. More importantly, we will NOT be able to help you should you need help debugging your back-end server. To build your back end as user ubuntu, please do the following:

  laptop$ sudo vi /etc/wsl.conf

  [automount]
  options = "metadata"

  #                       👇👇👇👇👇👇👇👇👇👇👇👇👇
  laptop$ ls /mnt/c/Users/YOUR_WINDOWS_USERNAME/

First generate a public/private key pair for user “ubuntu” in a safe place you can easily remember, for example YOUR*LABS*FOLDER.:

#         👇👇👇👇👇👇👇👇👇👇
laptop$ cd YOUR*LABS*FOLDER
laptop$ ssh-keygen -C ubuntu

when ssh-keygen prompts you for the file in which to save the key, enter “eecs441.pem”. It will then prompt you for a passphrase. Leave it empty. Hit return or enter, twice. Your identification (private key) would have been saved in eecs441.pem and your public key in eecs441.pem.pub. You can view the content of your public key for posting to Google below by:

laptop$ cat eecs441.pem.pub

Go to GCP Metadata page, open the SSH KEYS tab and click on EDIT.

On the edit page, click + ADD ITEM, copy and paste the content of your “eecs441.pem.pub” to the empty box that + ADD ITEM brought up, and hit the blue SAVE button.

Your SSH KEYS tab should now list “ubuntu” under Username with its corresponding public key:

To ssh to your GCP instance as user “ubuntu” using your private key, eecs441.pem, you must first set its permissions to read-only. In the following, YOUR_SERVER_IP always refers to the external IP address you’ve noted down earlier.

laptop$ chmod 400 eecs441.pem
laptop$ ssh -i eecs441.pem ubuntu@YOUR_SERVER_IP
#                                 👆👆👆👆👆👆👆👆👆

PS laptop> icacls eecs441.pem /grant "$($env:username):(r)" /inheritance:r
PS laptop> ssh -i eecs441.pem ubuntu@YOUR_SERVER_IP
#                                    👆👆👆👆👆👆👆👆👆 

Stop instance

Head to your E2 dashboard. Select “Compute Engine”.

When you are completely done with your E2, delete it to ensure you are not charged. A day or two later, ensure that there are no charges for your E2 at all.

Install Go

• ssh to your server and download the latest version of Go: Check Go’s Downloads page for the current latest version. As of the time of writing, the latest version is 1.23.0.

  server$ cd /tmp
  server$ wget https://go.dev/dl/go1.23.0.linux-amd64.tar.gz
  

  The version of Go distributed with the Ubuntu package manager apt was not the latest version and was incompatible with our solution, which requires Go version 1.18 or later.

• Go to Go’s download and install page.

• Skip the first section on Go download, you’ve already downloaded Go.

• Click on the Linux tab in the second (Go install) section of the instructions.

• Follow the instructions.

To update to a later version of Go, follow the instructions in Managing Go installations—which would have you manually delete the existing Go folder (usually /usr/local/go/), so don’t put any custom files in there.

chatterd module

First create and change into a directory where you can keep your Go module files:

server$ mkdir ~/441/chatterd
server$ cd ~/441/chatterd

Create a Go module called chatterd to house the Chatter back end:

server$ go mod init chatterd
# output:
go: creating new go.mod: module chatterd

Create a file called main.go:

server$ vi main.go

Put the following lines in your main.go:

package main

import(
    "log"
    "chatterd/router"
    "chatterd/views"
)

func main() {
    views.New()
    go func() { log.Fatal(router.Redirect().ListenAndServe()) }()
    log.Fatal(router.New().ListenAndServe())
}

We imported two Go packages chatterd/router and chatterd/chatter and instantiated a chatter and two routers. One router serves as the HTTPS web server by calling the ListenAndServe() function of router. The other router redirects HTTP to HTTPS and runs on a separate goroutine.

chatterd/router package

We set up url routing in the router package. Create the directory router and add the file router.go to this directory:

server$ cd ~/441/chatterd
server$ mkdir router
server$ vi router/router.go

Put the following in router.go:

package router

import (
    "chatterd/views"
    "net/http"

    "github.com/savsgio/atreugo/v11"
)

type Route struct {
    HTTPMethod string
    URLPath    string
    URLHandler atreugo.View
}

var routes = []Route {
    {"GET", "/getchatts/", views.GetChatts},
    {"POST", "/postchatt/", views.PostChatt},
}

func New() *atreugo.Atreugo {
    router := atreugo.New(atreugo.Config{
        Addr:               ":443",
        TLSEnable:          true,
        CertKey:            "/etc/ssl/private/selfsigned.key",
        CertFile:           "/etc/ssl/certs/selfsigned.cert",
        //Prefork:            true,
        Reuseport:          true,
    })
    router.RedirectTrailingSlash(true)

    for _, route := range routes {
        router.Path(route.HTTPMethod, route.URLPath, route.URLHandler)
    }

    return router
}

func Redirect() *atreugo.Atreugo {
    return atreugo.New(atreugo.Config{
        Addr:      ":80",
        Reuseport: true,
        NotFoundView: func(c *atreugo.RequestCtx) error {
            views.LogHTTP(http.StatusPermanentRedirect, c)
            return c.RedirectResponse("https://"+string(c.Host())+string(c.RequestURI()), http.StatusPermanentRedirect)
        },
    })
}

The code above sets up the router to serve Chatter’s two APIs, getchatts and postchatt. It routes HTTP GET requests with path /getchatts/ to the GetChatts() function of the views package and HTTP POST requests with path /postchatt/ to the PostChatt() function of the views package.

It also created two atreugo instantiation functions: New() creates a router to serve HTTPS requests following the two paths above, while Redirect() creates a router that permanently redirects all HTTP requests to the HTTPS server.

We have disabled Prefork above. If enabled, 8 servers will be preforked by default. To control the number of preforked servers, set the shell environment variable GOMAXPROCS prior to starting the server, for example:

server# export GOMAXPROCS=3

This will prefork 3 instances of the HTTPS server and 3 instances of the HTTP redirect server.

chatterd/views package

We implement the URL path API handlers in the views package. First create the package directory:

server$ mkdir views

Create and edit a file called views.go:

server$ vi views/views.go

with the following imports:

package views

import (
    "context"
    "encoding/json"
    "log"
    "net/http"
    "time"

    "github.com/savsgio/atreugo/v11"
    "github.com/jackc/pgx/v4/pgxpool"
)

and struct definitions:

type Chatt struct {
    Username  string    `json:"username"`
    Message   string    `json:"message"`
    Id        string    `json:"id"`
    Timestamp time.Time `json:"timestamp"`
}

var emptyJSON = struct {
    Empty *string `json:"-"`
}{}

We set up the backend with a log function and a database connection pool:

func LogHTTP(sc int, c *atreugo.RequestCtx) {
    log.Println("[ATR] |", sc, `|`, c.RemoteAddr().String(), `|`, string(c.Method()), string(c.RequestURI()))
}

var ctx = context.Background()
var chatterDB *pgxpool.Pool

func New() {
    var err error
    const (
        psqlUser   = "chatter"
        psqlPasswd = "chattchatt"
        psqlDB     = "chatterdb"
    )

    chatterDB, err = pgxpool.Connect(ctx, "host=localhost user="+psqlUser+" password="+psqlPasswd+" dbname="+psqlDB)
    if err != nil {
        panic(err)
    }
}

The New() function instantiates a new views by allocating a pool of open connections to our PostgreSQL chatterdb database. Maintaining a pool of open connections avoids the cost of opening and closing a connection on every database operation. The psqlPasswd here must match the one you used when setting up Postgres earlier.

The URL handler GetChatts() uses the pool to query the database for stored chatts and returns them to the client in the expected JSON format. Similarly, PostChatt() receives a posted chatt in the expected JSON format, unmarshalls the Chatt struct, and inserts it into the database through the pool. The UUID and time stamp of each chatt are automatically generated at insertion time.

func GetChatts(c *atreugo.RequestCtx) error {
    var chattArr [][]any
    var chatt Chatt

    rows, err := chatterDB.Query(ctx, `SELECT username, message, id, time FROM chatts ORDER BY time DESC`)
    if err != nil {
        panic(err)
    }
    defer rows.Close()

    for rows.Next() {
        rows.Scan(&chatt.Username, &chatt.Message, &chatt.Id, &chatt.Timestamp)
        chattArr = append(chattArr, []any{chatt.Username, chatt.Message, chatt.Id, chatt.Timestamp})
    }

    LogHTTP(http.StatusOK, c)
    return c.JSONResponse(map[string][][]any{"chatts": chattArr}, http.StatusOK)
}

func PostChatt(c *atreugo.RequestCtx) error {
    var chatt Chatt

    if err := json.Unmarshal(c.Request.Body(), &chatt); err != nil {
        LogHTTP(http.StatusUnprocessableEntity, c)
        return c.JSONResponse([]byte(err.Error()), http.StatusUnprocessableEntity)
    }

    _, err := chatterDB.Exec(ctx, `INSERT INTO chatts (username, message, id) VALUES ($1, $2, gen_random_uuid())`, chatt.Username, chatt.Message)
    if err != nil {
        LogHTTP(http.StatusInternalServerError, c)
        return c.JSONResponse([]byte(err.Error()), http.StatusInternalServerError)
    }

    LogHTTP(http.StatusOK, c)
    return c.JSONResponse(emptyJSON, http.StatusOK)
}

Build and run

To build your server:

server$ go get
server$ go build

Go is a compiled language, like C/C++ and unlike Python, which is an interpreted language. This means you must run go build each and every time you made changes to your code, for the changes to show up in your executable.

To run your server:

server$ sudo ./chatterd

If you had Prefork enabled and wanted to prefork 3 servers of HTTPS and HTTP redirect each, do instead:

server$ sudo GOMAXPROCS=3 ./chatterd

You can test your implementation following the instructions in the Testing Chatter APIs section.

References

• Working with Go
  • Golang tutorial series gentle and clear, though a bit out of date in parts now.
• Running multiple HTTP servers in Go
• Making a RESTful JSON API in Go
• Golang Json Marshal Example
• HTTP Status Code
• Go: Format a time or date
• Dipping Your Feet Into Golang Servers with Fiber
• atreugo

The following is based on DigitalOcean’s How To Set Up Django with Postgres, Nginx, and Gunicorn on Ubuntu 20.04 and How to Create a Self-Signed SSL Certificate for Nginx in Ubuntu 18.04, though the instructions have been customized to the specifics of our Chatter project, especially when it comes to directory and variable naming, for the sake of narrative consistency across all our labs.

Install Python

We will first use the apt package manager to install the following:

• Python 3.8 or later (up to 3.12.3 tested)
• Nginx (latest version)

Then we’ll install the following using the Python package manager pip:

• Django 4.1 or later (up to 5.1 tested)
• Gunicorn (latest version)

server$ sudo apt update
server$ sudo apt install python3-pip python3-dev python3-venv nginx
server$ sudo ln -s /usr/bin/python3 /usr/bin/python

Confirm that you’ve installed python version 3.8 or later:

server$ python --version
# output:
Python 3.8 # or later

Python virtual environment

Next install Python within a virtual environment for easier management.

• Create and change into a directory where you can keep your project files:

  server$ mkdir ~/441/chatterd
  server$ cd ~/441/chatterd
  

• Within the project directory, create a Python virtual environment:

  server$ python -m venv env
  

  This will create a directory called env within your chatterd directory. Inside, it will install a local version of Python and a local version of pip. We can use this to install and configure an isolated Python environment for our project.

• Activate your virtual environment:

    server$ source env/bin/activate
  

  Your prompt should change to indicate that you are now operating within a Python virtual environment. It will look something like this: (env) ubuntu@YOUR_SERVER_IP:~/441/chatterd$.

  Henceforth we will represent the prompt as just (env):chatterd$ to indicate being inside the python virtual environment.

• Within your activated virtual environment, install django, gunicorn, and the psycopg PostgreSQL adaptor using the local instance of pip:

  (env):chatterd$ pip install django gunicorn "psycopg[binary]"
  

  You should now have all of the software packages needed to start a Django project.

  You can check whether Django is installed and which version is installed with:

  server$ python -m django --version
  # output:
  4.1 # or later
  

Django web framework

• Create a Django project:

  (env):chatterd$ django-admin startproject routing ~/441/chatterd
  

  At this point your project directory (~/441/chatterd) should have the following content:

  • ~/441/chatterd/env/: the virtual environment directory we created earlier.
  • ~/441/chatterd/manage.py: the Django project management script.
  • ~/441/chatterd/routing/: the Django project package. This should contain the __init__.py, asgi.py, settings.py, urls.py, and wsgi.py files.

• Edit the project settings:

  (env):chatterd$ vi routing/settings.py
  

  At the top of the file add:

  import os
  

  Next locate the ALLOWED_HOSTS directive. This defines a list of addresses or domain names clients may use to connect to the Django server instance. Any incoming requests with a Host header that is not in this list will raise an exception. Django requires that you set this to prevent a certain class of security vulnerability.

  In the square brackets, list the IP addresses or domain names that are associated with your Django server. Each item mut be listed in single quotes, with entries separated by a comma. In the snippet below, a few commented out examples are provide as examples. Those with a period prepended to the beginning of an entry serves an entire domain and its subdomains.

  Note: Be sure to include ‘localhost’ as one of the options, for testing. You can also add ‘127.0.0.1’, the IP address indicating localhost. Some online examples may use it instead of ‘localhost’.

  . . .
  # The simplest case: just add the domain name(s) and IP addresses of your Django server
  # ALLOWED_HOSTS = [ 'example.com', '203.0.113.5']
  # To respond to 'example.com' and any subdomains, start the domain with a dot
  # ALLOWED_HOSTS = ['.example.com', '203.0.113.5']
  # ALLOWED_HOSTS = ['your_server_domain_or_IP', 'second_domain_or_IP', . . ., 'localhost']
  #                👇👇👇👇👇👇👇👇👇
  ALLOWED_HOSTS = ['YOUR_SERVER_IP', 'localhost', '127.0.0.1']
  

  YOUR_SERVER_IP will be your external IP address. Don’t list your internal IP address nor your DNS.

  It is a common bug not to replace YOUR_SERVER_IP with your external IP address in the above file.

  Next find the DATABASES configuration. The default configuration in the file is for a SQLite database. We want to use a PostgreSQL database for our project, so we need to change the settings to our PostgreSQL database information. We need to give the database name, the database username, the database user’s password, and then specify that the database is located on the local computer. The PASSWORD here must match the one you used when setting up Postgres earlier. You can leave the PORT setting as an empty string:

  DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'chatterdb',
        'USER': 'chatter',
        'PASSWORD': 'chattchatt',
        'HOST': 'localhost',
        'PORT': '',
    }
  }
  

  Scroll to the bottom of the file and add a setting indicating where the static files should be placed. The Nginx web server is optimized to serve static files fast, calling your python code only when needed to serve dynamic content. Here we’re telling Django to put static files for Nginx in a directory called static in the base project directory (~/441/chatterd/static/):

  STATIC_URL = '/static/'
  STATIC_ROOT = BASE_DIR / 'static'   # added line
  

  Save and close the settings.py file when you’re done.

• We can now migrate Django’s administrative database schema to our PostgreSQL database using the management script (and the exptected output):

  (env):chatterd$ ./manage.py makemigrations
  # output:
  No changes detected
  (env):chatterd$ ./manage.py migrate
  # output:
  Operations to perform:
    Apply all migrations: admin, auth, contenttypes, sessions
  Running migrations:
    Applying contenttypes.0001_initial... OK
    ...
  

• Create an administrative user for the project:

  (env):chatterd$ ./manage.py createsuperuser
  

  You can use your uniqname@umich.edu, and choose and confirm a password.

• Collect all static content into the static directory we configured:

  (env):chatterd$ ./manage.py collectstatic
  # output:
  130 static files copied to '/home/ubuntu/chatterd/static'.
  

Testing Django

If you’re on AWS, first allow access to port 8000 which we’ll be using for testing (this step is not needed for GCP):

(env):chatterd$ sudo ufw allow 8000

To run and test your development server, on either platform:

(env):chatterd$ ./manage.py runserver 0.0.0.0:8000

In your web browser on your laptop, visit http://YOUR_SERVER_IP:8000/.

You should see the default Django index page:

You can also test from the server host using curl, but you’ll see only the HTML source of the graphical page:

server$ curl http://localhost:8000/
# output:
<!doctype html>

<html lang="en-us" dir="ltr">
    <head>
        <meta charset="utf-8">
        <title>The install worked successfully! Congratulations!</title>
...        

When you are satisfied that Django is working, hit Ctl-C in server’s terminal window to shut down the development server.

Testing Django with Gunicorn

• The last thing we want to do before leaving our virtual environment is test Gunicorn to make sure that it can serve the application. We can do this by entering our project directory and using gunicorn to load the project’s WSGI (Web Server Gateway Interface) module:

  (env):chatterd$ gunicorn --bind 0.0.0.0:8000 routing.wsgi
  

  This will start Gunicorn on the same interface that the Django development server was running on. You should see output similar to this:

  [2022-07-21 21:43:14 -0400] [32224] [INFO] Starting gunicorn 20.1.0
  [2022-07-21 21:43:14 -0400] [32224] [INFO] Listening at: http://0.0.0.0:8000 (32224)
  [2022-07-21 21:43:14 -0400] [32224] [INFO] Using worker: sync
  [2022-07-21 21:43:14 -0400] [32227] [INFO] Booting worker with pid: 32227
  

• When you are done testing, hit Ctl-C in the server’s terminal window to stop Gunicorn.

• We’re done configuring Django. Exit the virtual environment:

  (env):chatterd$ deactivate
  

Setup Gunicorn

We use the Unix tool systemd to run Gunicorn whenever a connection attempt is made (by Nginx) to a socket we associate with Gunicorn. The socket will be created by systemd at Gunicorn service start time. We set up Gunicorn service configuration to have it started by systemd automatically on system boot.

• Start by creating and opening a systemd socket configuration file for Gunicorn with sudo privileges:

  server$ sudo vi /etc/systemd/system/gunicorn.socket
  

• Inside, we create a [Unit] section to describe the socket, a [Socket] section to specify the socket location, and an [Install] section to make sure the socket is created at the right time:

  [Unit]
  Description=gunicorn socket
  
  [Socket]
  ListenStream=/run/gunicorn.sock
  
  [Install]
  WantedBy=sockets.target
  

  Save and close the file when you are finished.

• Next, create and open a systemd service file for Gunicorn, with sudo privileges. The service file name must match the socket file name, except for the extension:

  server$ sudo vi /etc/systemd/system/gunicorn.service
  

• Enter the following into your gunicorn.service file:

  [Unit]
  Description=gunicorn daemon
  Requires=gunicorn.socket
  After=network.target
  
  [Service]
  User=ubuntu
  Group=www-data
  WorkingDirectory=/home/ubuntu/441/chatterd
  ExecStart=/home/ubuntu/441/chatterd/env/bin/gunicorn \
          --access-logfile - \
          --workers 3 \
          --bind unix:/run/gunicorn.sock \
          routing.wsgi:application
  
  [Install]
  WantedBy=multi-user.target
  

  With that, our Gunicorn service configuration is done. Save and close the file.

• We can now start and enable the Gunicorn socket. This will create the socket file at /run/gunicorn.sock now and at boot. When a connection is made to that socket, systemd will automatically start the gunicorn.service to handle it:

  server$ sudo systemctl start gunicorn.socket
  server$ sudo systemctl enable gunicorn.service
  server$ sudo systemctl start gunicorn
  

  If you subsequently made changes to the /etc/systemd/system/gunicorn.service file, you’d need to reload the daemon to reinitialize the service configuration and restart the Gunicorn process:

  server$ sudo systemctl daemon-reload
  server$ sudo systemctl restart gunicorn
  

  We can confirm that the operation was successful by checking for the socket file.

  server$ file /run/gunicorn.sock
  # output:
  /run/gunicorn.sock: socket
  

  and Gunicorn’s status:

  server$ systemctl status gunicorn
  # output:
  ● gunicorn.service
      Loaded: loaded (/etc/systemd/system/gunicorn.service; disabled; vendor preset: enabled)
      **Active: active (running)** since Wed 2022-05-27 16:47:07 UTC; 18s ago
      . . .     👆👆👆👆👆👆👆👆👆👆
  

  Confirm that Gunicorn’s status reported on the third line is Active: active (running).

TIP: sudo systemctl status gunicorn is your BEST FRIEND in debugging Django. If you get an HTTP error code 500 Internal Server Error or if you just don’t know whether your HTTP request has made it to the server, first thing you do is run sudo systemctl status gunicorn on your server and study its output. It also shows error messages from your python code, including any debug printouts from your code. The command systemctl status gunicorn is by far the most useful go-to tool to diagnose Django back-end server problem.

Nginx to Gunicorn

Now that Gunicorn is set up, we configure Nginx to pass traffic to it.

• Start by creating and opening a web site configuration file we’ll call chatterd in Nginx’s sites-available directory:

  server$ sudo vi /etc/nginx/sites-available/chatterd
  

• Inside, open up a new server block. We specify that the server will listen on the default HTTPS port 443, for both IPv4 and IPv6. We use the private key and certificate you created earlier to serve HTTPS traffic:

  server {
    listen 443 ssl;
    listen [::]:443 ssl;         # add support for IPv6
  
    ssl_certificate     /etc/ssl/certs/selfsigned.cert;
    ssl_certificate_key /etc/ssl/private/selfsigned.key;
    ssl_protocols TLSv1.2;
    ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
  
    location = /favicon.ico { access_log off; log_not_found off; }
    location /static/ {
      root /home/ubuntu/441/chatterd;
    }
  
    location / {
      include proxy_params;
      proxy_pass http://unix:/run/gunicorn.sock;
    }
  }
  

• So that visitors to your web site who accidentally entered http instead of https wouldn’t be confronted with an error message, we automatically redirect them to https permanently. In your web site configuration file above, create a second server block after the one above. Remember to replace YOUR_SERVER_IP with your external IP:

  # . . .
  server {
    listen 80;
    listen [::]:80;  # IPv6
  
    #          👇👇👇👇👇👇👇👇👇
    server_name YOUR_SERVER_IP;
  
    return 308 https://$server_name$request_uri; # permanent redirect
  }
  

• Save and close the file. Now, we can enable the file by linking it to the sites-enabled directory:

  server$ sudo ln -s /etc/nginx/sites-available/chatterd /etc/nginx/sites-enabled
  

• Test your Nginx configuration for syntax errors:

  server$ sudo nginx -t
  # output:
  nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
  nginx: configuration file /etc/nginx/nginx.conf test is successful
  

• If no errors are reported, go ahead and restart Nginx:

  server$ sudo systemctl restart nginx
  

  Be sure to do this every time you make changes to the chatterd server configuration file.

  To check that your nginx is running:

  server$ sudo systemctl status nginx
  

• Finally, we need to open up our firewall to normal traffic on port 80:

  server$ sudo ufw allow 'Nginx Full'
  

• To test, you can use curl:

  laptop$ curl --insecure https://YOUR_SERVER_IP/
  

  or, from your Chrome browser on your laptop, browse to https://YOUR_SERVER_IP/. It will warn you that “Your connection is not private”. Click the Advanced button. Then bravely click Proceed to YOUR_SERVER_IP (unsafe). You should see the Nginx rocket.

  On macOS, Safari won’t allow you to visit the site, you have to use Chrome.

• You can verify that your redirect for HTTP to HTTPS functions correctly by accessing http://YOUR_SERVER_IP/ (note use of http not https). Again, you must do this from a Chrome browser or curl --insecure.

  Troubleshooting

  If this last step does not show your application, consult DigitalOcean’s tutorial and search for “Troubleshooting Nginx and Gunicorn” or consult the teaching staff.

Summary: configuration files

In summary, here are the configuration files for the three packages we rely on for a Python-based back-end stack:

Nginx: web server that listens on port 80

file: /etc/nginx/sites-enabled/chatterd, if modified run:

server$ sudo nginx -t
server$ sudo systemctl restart nginx

Gunicorn: serves Django project

file: /etc/systemd/system/gunicorn.service, if modified run:

server$ sudo systemctl daemon-reload
server$ sudo systemctl restart gunicorn

Django: framework to route HTTP requests to your python code

directory: ~/441/chatterd/routing/, in particular urls.py (see below). If modified run:

server$ sudo systemctl restart gunicorn

Congratulations! Your server is all set up! Now to implement Django’s views, which comprises our Chatter back-end code.

Chatter back end

Start by creating the model-view-controller (MVC) framework expected by Django for all python projects:

server$ cd ~/441/chatterd
server$ source env/bin/activate
(env):chatterd$ ./manage.py startapp app
(env):chatterd$ deactivate

This will create a directory ~/441/chatterd/app with the necessary python files in it. In your ~/441/chatterd project directory, you should now have two directories that were created by Django:

• ~/441/chatterd/routing/ created with startproject earlier. It contains the Django web framework for your app. We will be modifying settings.py and urls.py in this directory.
• ~/441/chatterd/app/ that we just created with startapp. It contains your app’s domain/business logic and views and controllers. We will be modifying views.py in this directory.

These are distinct directories and both must be retained (don’t delete or merge them!).

urls.py

We set up URL path API routing in the routing/urls.py file. Open and edit the file to add the following import line below the two existing ones:

from app import views

Next add to the contents of urlpatterns array:

    path('getchatts/', views.getchatts, name='getchatts'),
    path('postchatt/', views.postchatt, name='postchatt'),

The code above sets up Django’s router to serve Chatter’s two APIs, getchatts and postchatt. It routes HTTP GET requests with path getchatts/ to the getchatts() function of the views module and HTTP POST requests with path postchatt/ to the postchatt() function of the views module.

The Chatter APIs, getchatts and postchatt will be implemented in ~/441/chatterd/app/views.py.

views.py

Add to ~/441/chatterd/app/views.py:

from django.http import JsonResponse, HttpResponse
from django.db import connection
from django.views.decorators.csrf import csrf_exempt
import json

def getchatts(request):
    if request.method != 'GET':
        return HttpResponse(status=404)

    with connection.cursor() as cursor:
        cursor.execute('SELECT username, message, id, time FROM chatts ORDER BY time DESC;')
        rows = cursor.fetchall()

    response = {}
    response['chatts'] = rows
    return JsonResponse(response)

In getchatts(), we use the database cursor to retrieve chatts. The cursor in psycopg is a context manager, which releases any resources used when its with block is closed. Once you have retrieved all the rows from the database, you need to insert it into the response dictionary to be returned to the front end.

For postchatt(), by default, Django wants to see CSRF (cross-site request forgery) cookies for posting. Since we’re not implementing csrf, we ask for exemption. In views.py add:

@csrf_exempt
def postchatt(request):
    if request.method != 'POST':
        return HttpResponse(status=404)

    json_data = json.loads(request.body)
    username = json_data['username']
    message = json_data['message']

    with connection.cursor() as cursor:
        cursor.execute('INSERT INTO chatts (username, message, id) VALUES '
                       '(%s, %s, gen_random_uuid());', (username, message))

    return JsonResponse({})

For more Python-PostgreSQL interaction, see Passing parameters to SQL queries.

As before, everytime you make changes to either app/views.py or routing/urls.py, you need to restart Gunicorn:

server$ sudo systemctl restart gunicorn

Leave your nginx and gunicorn running until you have received your lab grade.

References

• Nginx+Gunicorn+Django+PostgreSQL setup
  • Django NameError: name ‘os’ is not defined
• AWS Services You Should Know When Deploying Your Django App
• Django introduction
• To read more about how nginx, gunicorn, and django work together:
  • Django Runserver is not your Production Server
  • Gunicorn and Nginx in a Nutshell
  • What is Gunicorn, and What does it do?

Install Rust

Note that GCP gives you 2 GB more disk space than AWS, which allows Rust to build dependencies without complaining of running out of space.

ssh to your server and install Rust:

server$ sudo apt install gcc          # cargo depends on gcc's linker
server$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
server$ rustup update

If you see:

rustup: Command not found.

try to logout from your server and ssh back to it again. If the problem persisted and you need help updating your PATH shell environment variable, please see a teaching staff.

The command rustup update is also how you can subsequently update your installation of Rust to a new version.

chatterd package

First create and change into a directory where you can keep your Rust chatterd package files:

server$ cd ~/441
server$ cargo new chatterd
# output:
     Created binary (application) `chatterd` package   

This will create the ~/441/chatterd/ directory for you. Change to this directory and edit the file Cargo.toml to list all the 3rd-party libraries (crates in Rust-speak) we will be using.

server$ cd chatterd
server$ vi Cargo.toml

In Cargo.toml, add the following below the [dependencies] tag:

axum = "0.7.5"
axum-server = { version = "0.7.1", features = ["tls-rustls"] }
bb8 = "0.8.3"
bb8-postgres = "0.8.1"
chrono = { version = "0.4.38", features = ["serde"] }
postgres = { version = "0.19.7", features = ["with-chrono-0_4", "with-uuid-1"] }
serde = { version = "1.0.197", features = ["derive"] }
serde_json = "1.0.117"
tokio = { version = "1.37.0", features = ["full"] }
tokio-postgres = "0.7.10"
tracing = "0.1.40"
tracing-subscriber = { version = "0.3.18", features = ["env-filter"] }
uuid = { version = "1.10.0", features = ["v4", "macro-diagnostics", "serde"] }

In ~/441/chatterd/src/ a file main.rs has also been created for you. Edit the file:

server$ vi src/main.rs

and replace the existing lines in main.rs with the following:

#![allow(non_snake_case)]
use axum::{
    extract::Host,
    handler::HandlerWithoutStateExt,
    http::{StatusCode, Uri},
    response::Redirect,
    routing::{get, post},
    BoxError,
    Router,
};
use axum_server::{
  tls_rustls::RustlsConfig,
  Server,
};
use bb8::Pool;
use bb8_postgres::PostgresConnectionManager;
use std::{net::SocketAddr};
use tokio_postgres::NoTls;
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};

pub mod handlers;

#[tokio::main]
async fn main() {
    tracing_subscriber::registry()
        .with(tracing_subscriber::EnvFilter::new("chatterd=trace"))
        .with(tracing_subscriber::fmt::layer())
        .init();

    // run http to https redirect
    tokio::spawn(http_redirect());

    // setup connection pool for PostgreSQL
    let pgmanager = PostgresConnectionManager::new_from_stringlike(
        "host=localhost user=chatter password=chattchatt dbname=chatterdb",
        NoTls,
    )
    .unwrap();
    let pgpool = Pool::builder().build(pgmanager).await.unwrap();

    let router = Router::new()
        .route("/getchatts/", get(handlers::getchatts))
        .route("/postchatt/", post(handlers::postchatt))
        .with_state(pgpool); // must always be last line in Router set up

    // port number the HTTPS server will bind to:
    let addr = SocketAddr::from(([0, 0, 0, 0], 443));

    tracing::debug!("https server listening on {}", addr);

    // certificate and private key used with HTTPS
    let certkey = RustlsConfig::from_pem_file(
        "/etc/ssl/certs/selfsigned.cert",
        "/etc/ssl/private/selfsigned.key",
    )
    .await
    .unwrap();

    // run the HTTPS server
    axum_server::bind_rustls(addr, certkey)
        .serve(router.into_make_service_with_connect_info::<SocketAddr>())
        .await
        .unwrap();
}

After listing all our imports, we export a module handlers, which we will define later. In main(), we enable logging (tracing) and spawn an asynchronous function to redirect all HTTP requests to our HTTPS server. Next we set up a pool of open connections to our PostgreSQL chatterdb database. Maintaining a pool of open connections avoids the cost of opening and closing a connection on every database operation. The password used in creating pgmanager must match the one you used when setting up Postgres earlier. The connection pool is passed to both URL path API handlers as Router State.

The code above also sets up the axum server to route Chatter’s two APIs, getchatts and postchatt. It routes HTTP GET requests with path /getchatts/ to the getchatts() function of the handlers module and HTTP POST requests with path /postchatt/ to the postchatt() function of the handlers module.

It then starts the axum_server bound to the default HTTPS port, with the given certificate and private key, and the provided URL path API routing.

Earlier, we spawned an asynchronous function to redirect all HTTP requests to this HTTPS server. Here’s the code to perform the redirection (from axum’s example-tls-rustls). Add it to the end of main.rs:

async fn http_redirect() {
    fn make_https(host: String, uri: Uri) -> Result<Uri, BoxError> {
        let mut parts = uri.into_parts();
        parts.scheme = Some(axum::http::uri::Scheme::HTTPS);
        if parts.path_and_query.is_none() {
            parts.path_and_query = Some("/".parse().unwrap());
        }
        parts.authority = Some(host.parse()?);
        Ok(Uri::from_parts(parts)?)
    }

    let redirect = move |Host(host): Host, uri: Uri| async move {
        match make_https(host, uri) {
            Ok(uri) => Ok(Redirect::permanent(&uri.to_string())),
            Err(error) => {
                tracing::warn!(%error, "failed to convert URI to HTTPS");
                Err(StatusCode::BAD_REQUEST)
            }
        }
    };

    let addr = SocketAddr::from(([0, 0, 0, 0], 80));
    tracing::debug!("http redirect listening on {}", addr);

    Server::bind(addr)
        .serve(redirect.into_make_service())
        .await
        .unwrap();
}

handlers module

We implement the URL path API handlers in the module handlers.rs:

server$ vi src/handlers.rs

Add the following contents:

#![allow(non_snake_case)]
use axum::{
    extract::{ConnectInfo, Json, State},
    http::{Method, StatusCode, Uri},
};
use bb8::Pool;
use bb8_postgres::PostgresConnectionManager;
use chrono::{DateTime, Local};
use serde::{Deserialize, Serialize};
use serde_json::{json, Value};
use std::{net::SocketAddr, str};
use tokio_postgres::NoTls;
use uuid::Uuid;

type PGPool = Pool<PostgresConnectionManager<NoTls>>;

#[derive(Debug, Serialize, Deserialize)]
pub struct Chatt {
    username: String,
    message: String,
    id: Option<String>,    
    timestamp: Option<DateTime<Local>>,    
}

pub async fn getchatts(
    State(pgpool): State<PGPool>,
    ConnectInfo(clientIP): ConnectInfo<SocketAddr>,
    method: Method,
    uri: Uri,
) -> Json<Value> {
    let chatterDB = pgpool
        .get()
        .await
        .unwrap();

    let mut chattArr: Vec<Vec<Option<String>>> = Vec::new();

    for row in chatterDB
        .query(
            "SELECT username, message, id, time FROM chatts ORDER BY time DESC",
            &[],
        )
        .await
        .unwrap()
    {
        chattArr.push(vec![
            row.get(0),
            row.get(1),
            Some(row.get::<usize, Uuid>(2).to_string()),
            Some(row.get::<usize, DateTime<Local>>(3).to_string()),
        ]);
    }

    tracing::debug!(
        " {:?} | {:?} | {:?} {:?}",
        StatusCode::OK,
        clientIP,
        method,
        uri.path()
    );
    Json(json!({ "chatts": chattArr }))
}

pub async fn postchatt(
    State(pgpool): State<PGPool>,
    ConnectInfo(clientIP): ConnectInfo<SocketAddr>,
    method: Method,
    uri: Uri,
    Json(chatt): Json<Chatt>,
) -> (StatusCode, Json<Value>) {
    let chatterDB = pgpool
        .get()
        .await
        .unwrap();

    let dbStatus = chatterDB
        .execute(
            "INSERT INTO chatts (username, message, id) VALUES ($1, $2, gen_random_uuid())",
            &[&chatt.username, &chatt.message],
        )
        .await
        .map_or_else(|err| (StatusCode::INTERNAL_SERVER_ERROR, Json(json!(err.to_string()))), |_| (StatusCode::OK, Json(json!({}))));

    tracing::debug!(
        " {:?} | {:?} | {:?} {:?}",
        dbStatus.0,
        clientIP,
        method,
        uri.path()
    );
    dbStatus
}

The handler getchatts() uses the connection pool to query the database for stored chatts and returns them to the client in the expected JSON format. Similarly, postchatt() receives a posted chatt in the expected JSON format, has it deserialized into the Chatt struct, and inserts it into the database through the connection pool. The UUID and time stamp of each chatt are automatically generated at insertion time.

Build and run

To build your server:

server$ cargo build --release
server$ ln -s target/release/chatterd chatterd

The first time around, it will take some time to download and build all the 3rd-party crates. Be patient.

  error: linking with cc failed: exit status: 1
  note: collect2: fatal error: ld terminated with signal 9 [Killed]

To run your server:

server$ sudo ./chatterd

You can test your implementation following the instructions in the Testing Chatter APIs section.

References

• The Rust Programming Language the standard and best intro to Rust.
• axum
• axum_server
• axum_server::tls_rustls
• axum examples
• http::StatusCode see the list of Associated Constants on the left menu.
• axum::extract
• axum::Extension
• Serde JSON
• Postgres with Rust
• postgres::types::FromSql
• chrono::DateTime
• http::Uri