Editing handlers.go

Add the following libraries to the import block at the top of the file:

import (
    //...
    "errors"
    "io"
    "os"
    "path"
    "sync"
    //...
)

Now add two new properties to the Chatt struct in handlers.go:

type Chatt struct {
    // . . .
    ImageUrl  *string    `json:"imageUrl"`
    VideoUrl  *string    `json:"videoUrl"`
}

Next add the postimages() function along with the saveFormFile() and copyZeroAlloc() helper functions. We also specify our designated media directory to store image/video files.

const MEDIA_ROOT = "/home/ubuntu/441/chatterd/media/"

func postimages(c echo.Context) error {
	var err error
	var chatt Chatt

	chatt.Username = string(c.FormValue("username"))
	chatt.Message = string(c.FormValue("message"))
	chatt.ImageUrl, err = saveFormFile(c, "image", chatt.Username, ".jpeg")
	if err == nil {
		chatt.VideoUrl, err = saveFormFile(c, "video", chatt.Username, ".mp4")
	}
	if err != nil {
		return logClientErr(c, http.StatusUnprocessableEntity, err)
	}

	_, err = chatterDB.Exec(background, `INSERT INTO chatts (username, message, id, imageurl, videourl) VALUES ($1, $2, gen_random_uuid(), $3, $4)`, chatt.Username, chatt.Message, chatt.ImageUrl, chatt.VideoUrl)
	if err != nil {
		return logClientErr(c, http.StatusBadRequest, err)
	}

	logOk(c)
	return c.JSON(http.StatusOK, "")
}

func saveFormFile(c echo.Context, media string, username string, ext string) (*string, error) {
	content, err := c.FormFile(media)
	if err != nil { return nil, nil } // not an error, media not posted

	securename := path.Base(path.Clean(username))
	if securename[0] == '.' || securename == "/" {
		return nil, errors.New("invalid username")
	}
	filename := securename + "-" + strconv.FormatInt(time.Now().Unix(), 10) + ext

	src, err := content.Open()
	if err != nil { return nil, err }
	defer src.Close()

	dst, err := os.Create(MEDIA_ROOT + filename)
	if err != nil { return nil, err }
	defer dst.Close()

	if _, err = copyZeroAlloc(dst, src); err != nil {
		return nil, err
	}

    mediaUrl := "https://" + string(c.Request().Host) + "/media/" + filename
	return &mediaUrl, nil
}

// adapted from "github.com/valyala/fasthttp"
var copyBufPool = sync.Pool{ New: func() any {return make([]byte, 16384) } }
func copyZeroAlloc(w io.Writer, r io.Reader) (int64, error) {
	if directRead, ok := r.(io.WriterTo); ok {
		return directRead.WriteTo(w)
	}
	if directWrite, ok := w.(io.ReaderFrom); ok {
		return directWrite.ReadFrom(r)
	}
	// else copy in chunks of 16 KB.
	vbuf := copyBufPool.Get()
	buf := vbuf.([]byte)
	n, err := io.CopyBuffer(w, r, buf)
	copyBufPool.Put(vbuf)
	return n, err
}

Make a copy of your getchatts() in handlers.go and name the copy getimages(). In getimages(), replace the SELECT statement with the following: `SELECT username, message, id, time, imageurl, videourl FROM chatts ORDER BY time DESC`. This statement will retrieve all data, including our new image and video URLs from the PostgreSQL database.

Still in getimages(), replace the rows.Scan() call in the for rows.Next() {} block with:

        rows.Scan(&chatt.Username, &chatt.Message, &chatt.Id, &chatt.Timestamp, &chatt.ImageUrl, &chatt.VideoUrl)

and if the returned err is not nil:

        chattArr = append(chattArr, []any{chatt.Username, chatt.Message, chatt.Id, chatt.Timestamp, chatt.ImageUrl, chatt.VideoUrl})

In addition to the original three columns, we added reading the imageurl and videourl columns and included them in the chatt data returned to the front end.

Save and exit handlers.go.

Routing for new URLs

For the newly added getimages() and postimages() functions, add the following new routes to the routes array in main.go:

var routes = []Route {
    // . . .
    {"GET", "/getimages/", getimages},
    {"POST", "/postimages/", postimages},
}

To serve image/video static files, in the main() function add the following before launching the server:

	server.Static("/media", MEDIA_ROOT)
	server.Use(middleware.BodyLimit("10M"))

This tells Echo to redirect URL path /media to its static-file server and to serve media files from our designated media directory, whose value is stored in MEDIA_ROOT, and to limit each upload to 10 MB.

Save and exit main.go.

Go is a compiled language, like C/C++ and unlike Python, which is an interpreted language. This means you must run go build each and every time you made changes to your code, for the changes to show up in your executable.

Rebuild chatterd:

server$ go get
server$ go build

and restart it:

server$ sudo systemctl restart chatterd

References

• Uploading Files in Golang with Multipart Request
• Build a web application in Go: Serving static files
• Ways to handle SQL NULL values in Go

To support uploading of large media files, first install the multer package as dependency:

server$ npm install multer
server$ npm install -D @types/multer

This should add to your package.json, in the dependencies block:

    "multer": "^1.4.5-lts.1",

and in the devDependencies block:

    "@types/multer": "^1.4.12",

Editing handlers.ts

Add the postimages() function:

export async function postimages(req: Request, res: Response) {
    let chatt: Chatt
    try {
        chatt = req.body
    } catch (error) {
        res.status(422).json(error)
        return
    }

    let imageurl: string | null
    let videourl: string | null
    try {
        // https://github.com/expressjs/multer#usage
        // https://www.typescriptlang.org/docs/handbook/advanced-types.html
        // https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Optional_chaining#syntax
        const files = req.files as { [fieldname: string]: Express.Multer.File[] }
        const imagefilename = files['image']?.[0]['filename']
        const videofilename = files.video?.[0].filename  // alternate syntax, syntactic sugar?
        imageurl = imagefilename ? `https://${req.hostname}/media/${imagefilename}` : null
        videourl = videofilename ? `https://${req.hostname}/media/${videofilename}` : null
    } catch (error) {
        res.status(422).json(`${error}`)
        return
    }

    try {
        await chatterDB`INSERT INTO chatts (username, message, id, imageurl, videourl) VALUES (${chatt.username}, ${chatt.message}, ${randomUUID()}, ${imageurl}, ${videourl})`
        res.json({})
    } catch (error) {
        res.status(400).json(`${error as PostgresError}`)
    }
}

We also add postuser() function to return the username of a posted chatt. We will use this in the next section to construct filenames for uploaded files:

export function postuser(req: Request): string | null {
    let chatt: Chatt
    try {
        chatt = req.body
        return chatt.username
    } catch (error) {
        return null
    }
}

Finally, make a copy of your getchatts() in handlers.ts and name the copy getimages(). In getimages(), replace the declaration of chatts with:

        const chatts = await chatterDB`SELECT username, message, id, time, imageurl, videourl FROM chatts ORDER BY time DESC`.values()

This will retrieve all data, including our new image and video URLs from the PostgreSQL database.

Save and exit handlers.ts.

Routing for new URLs

To serve image/video static files, add the following lines of import at the top of main.ts:

import { basename } from 'path'
import multer from 'multer'

We also specify our designated media directory to store image/video files, and maximum allowed media file size. Add the following lines to your main.ts, before the try block, right after the declaration of chatterDB, for example:

const MEDIA_ROOT = '/home/ubuntu/441/chatterd/media'
const MEDIA_MXSZ = 10485760 // 10 MB, default is infinity

Next, we set up the multer middleware. Inside your try block, before the instantiation of the Express app, add:

  const storage = multer.diskStorage({
    destination: (req, file, cb) => {
      cb(null, MEDIA_ROOT)
    },
    filename: (req, file, cb) => {
      const username = handlers.postuser(req)
      let securename: string
      if (username) {
        securename = basename(username)
      } else {
        securename = req.ip
      }
      const time = new Date(Date.now()).toISOString()
      const filename = `${securename}-${time}.${basename(file.mimetype)}`
      cb(null, filename)
    }
  })

  const upload = multer({storage: storage, limits: {fileSize: MEDIA_MXSZ}})

We have set up multer to store uploaded files to disk. The destination directory for uploaded files is in MEDIA_ROOT, which we defined earlier to be /home/ubuntu/441/media. Each uploaded file will be given a file name of the form securename-uploadTime.mimeType, where securename is chatt.username, with any path-looking part strips off, or the client’s IP address. We further limit the upload file size to MEDIA_MXSZ, which we have previously defined to be 10 MB.

For the newly added getimages() and postimages() functions, add the following to the Express app instantiation in main.ts:

      .get('/getimages/', handlers.getimages)
      .post('/postimages', upload.fields([
        {name: 'image', maxCount: 1},
        {name: 'video', maxCount: 1}
      ]), handlers.postimages)
      .use('/media', express.static(MEDIA_ROOT))

In addition to routing the two new APIs, the .use() line tells Express to redirect URL path /media to serve media files from our designated media directory.

Save and exit main.ts. Rebuild and restart chatterd:

server$ npx tsc
server$ sudo systemctl restart chatterd

References

• Multer Usage <– Handling File Uploads with Multer in a Class-based Node.js and Express Application –>
  • Multer: Easily upload files with Node.js and Express
  • Multer: storage

To support uploading of large media files, first install the following packages in your python’s virtual environment:

server$ source venv/bin/activate
(env):chatter$ pip install -U python-multipart werkzeug

Editing handlers.py

Add the following libraries to the import block at the top of the file:

import os
from werkzeug.utils import secure_filename

Next add the postimages() function along with the saveFormFile() helper function. We also specify our designated media directory to store image/video files, and maximum allowed media file size.

MEDIA_ROOT = '/home/ubuntu/441/chatterd/media/'
MEDIA_MXSZ = 10485760 # 10 MB

async def saveFormFile(fields, media, url, username, ext):
    try:
        file = fields[media]
        if file.size > MEDIA_MXSZ:
             # but the whole file will still be received, just not saved
            raise BufferError
    except KeyError:
        return None # not an error, media not sent
    except Exception:
        raise

    try:
        if not (filename := secure_filename(username)):
            raise NameError

        filename = f'{filename}-{str(time.time())}{ext}'
        filepath = os.path.join(MEDIA_ROOT, filename)

        # open(): https://docs.python.org/3/library/functions.html#open
        with open(filepath, 'wb') as f:
            # write(): https://docs.python.org/3/tutorial/inputoutput.html#tut-files
            # form.UploadFile.read(): https://www.starlette.io/requests/#request-files
            f.write(await file.read(MEDIA_MXSZ))
            f.close()
        
        # url to string: https://stackoverflow.com/a/57514621/
        return f'{url}{filename}'
    except BaseException:
        raise

async def postimages(request):
    try:
        url = str(request.url_for('media', path='/'))
        # loading form-encoded data
        async with request.form() as fields:
            username = fields['username']
            message = fields['message']
            imageurl = await saveFormFile(fields, 'image', url, username, '.jpeg')
            videourl = await saveFormFile(fields, 'video', url, username, '.mp4')
    except BaseException as err:
        print(f'{err=}')
        return JSONResponse('Unprocessable entity', status_code=422)

    try:
        async with main.server.pool.connection() as connection:
            async with connection.cursor() as cursor:
                await cursor.execute('INSERT INTO chatts (username, message, id, imageurl, videourl) VALUES '
                                 '(%s, %s, gen_random_uuid(), %s, %s);', (username, message, imageurl, videourl))
        return JSONResponse({})
    except StringDataRightTruncation as err:
        print(f'Message too long: {str(err)}')
        return JSONResponse(f'Message too long', status_code = 400)
    except Exception as err:
        print(f'{err=}')
        return JSONResponse(f'{type(err).__name__}: {str(err)}', status_code = 500)

Make a copy of your getchatts() in handlers.py and name the copy getimages(). In getimages(), replace the SELECT statement with the following: 'SELECT username, message, id, time, imageurl, videourl FROM chatts ORDER BY time DESC;'. This statement will retrieve all data, including our new image and video URLs from the PostgreSQL database.

Save and exit handlers.py.

Routing for new URLs

To serve image/video static files, add the StaticFiles import and add the Mount module to the existing import from starlette.routing such that the lines near the top of the file reads:

from starlette.routing import Route, Mount
from starlette.staticfiles import StaticFiles

For the newly added getimages() and postimages() functions, add the following new routes to the routes array in main.py:

    Route('/getimages/', handlers.getimages, methods=['GET']),
    Route('/postimages/', handlers.postimages, methods=['POST']),
    # static files: https://www.starlette.io/staticfiles
    Mount('/media/', app=StaticFiles(directory=handlers.MEDIA_ROOT, follow_symlink=True), name='media'),

In addition to routing the two new APIs, the next line line tells Starlette to redirect URL path /media to serve media files from our designated media directory, whose value is stored in MEDIA_ROOT in handlers.py above.

Save and exit main.py and restart chatterd:

server$ sudo systemctl restart chatterd

References

• Loading form-encoded data in python
• Exception hierarchy

Adding dependencies

In your package directory, ~/441/chatterd/, open Cargo.toml and replace the axum line in the dependencies block with:

axum = { version="0.8.0-alpha.1", features = ["multipart"] }

and add the following dependencies, replacing the tower-http line with the one here that imports additional features:

futures = "0.3.30"
tokio-util = { version="0.7.12", features = ["io"] }
tower-http = { version = "0.6.1", features = ["fs", "limit", "trace"] }

Save and exit Cargo.toml.

Editing handlers.rs

Add the following crates/modules in the use axum::{} block at the top of the file:

use axum::{
    body::Bytes, 
    // you can choose to "merge" the following with the existing
    // `extract` module or add it as a separate line. Both work.
    extract::{Host, Multipart},
    // ...
};

and add the following as additional imported crates/modules:

use chrono::SecondsFormat;
use futures::{Stream, TryStreamExt};
use std::{
    io, 
    io::{Error, ErrorKind}, 
    path::{Path, PathBuf},
    str,};
use tokio::{fs::File, io::{copy, BufWriter}};
use tokio_util::io::StreamReader;
use tower_http::BoxError;

As with the axum::extract module, you can “merge” the chrono and std modules to their respective existing imports, or leave them as separate lines.

Next add the postimages() function along with the saveFormFile() and stream_to_file() helper functions. We also specify our designated media directory to store image/video files.

pub const MEDIA_ROOT: &str = "/home/ubuntu/441/chatterd/media";

pub async fn postimages(
    State(pgpool): State<PGPool>,
    ConnectInfo(clientIP): ConnectInfo<SocketAddr>,
    Host(host): Host,
    mut form: Multipart,
) -> Result<Json<Value>, (StatusCode, String)> {
    let mut username = String::new();
    let mut message = String::new();
    let mut imageurl: Option<String> = None;
    let mut videourl: Option<String> = None;

    let chatterDB = pgpool
        .get()
        .await
        .map_err(|err| logServerErr(clientIP, err.to_string()))?;

    while let Some(field) = form.next_field().await.unwrap_or(None) {
        let ext = field
            .content_type()
            .map(|mimeType| mimeType[6..].to_string());

        match &*field.name().unwrap_or_default() {
            "username" => username.push_str(
                str::from_utf8(&field.bytes().await.map_or(Bytes::from("unknown"), |s| s)).unwrap(),
            ),
            "message" => message.push_str(
                str::from_utf8(&field.bytes().await.map_or(Bytes::from(""), |s| s)).unwrap(),
            ),
            "image" => {
                imageurl = Some(
                    saveFormFile(field, &username, &host, &ext.unwrap_or("jpeg".to_string()))
                        .await
                        .map_err(|err| {
                            logClientErr(clientIP, StatusCode::UNPROCESSABLE_ENTITY, err.to_string())
                        })?,
                )
            }
            "video" => {
                videourl = Some(
                    saveFormFile(field, &username, &host, &ext.unwrap_or("mp4".to_string()))
                        .await
                        .map_err(|err| {
                            logClientErr(clientIP, StatusCode::UNPROCESSABLE_ENTITY, err.to_string())
                        })?,
                )
            }
            &_ => unreachable!(),
        }
    }

    chatterDB
        .execute(
            "INSERT INTO chatts (username, message, id, imageurl, videourl) VALUES ($1, $2, gen_random_uuid(), $3, $4)",
            &[&username, &message, &imageurl, &videourl],
        )
        .await
        .map_err(|err| logClientErr(clientIP, StatusCode::NOT_ACCEPTABLE, err.to_string()))?;

    logOk(clientIP);
    Ok(Json(json!({})))
}

async fn saveFormFile<S, E>(stream: S, username: &str, host: &str, ext: &str) -> io::Result<String>
where
    S: Stream<Item = Result<Bytes, E>>,
    E: Into<BoxError>,
{
    let postTime = Utc::now().to_rfc3339_opts(SecondsFormat::Secs, true);
    let securename = Path::new(username)
        .file_stem()
        .and_then(|stem| { stem.to_str() })
        .ok_or_else(|| Error::from(ErrorKind::InvalidData))?;
    let filename = &*format!("{securename}-{postTime}.{ext}");

    stream_to_file(filename, stream)
        .await
        .map(|filename| format!("https://{host}/media/{filename}"))
}

// Save a `Stream` to a file
// from https://github.com/tokio-rs/axum/blob/main/examples/stream-to-file/src/main.rs
async fn stream_to_file<S, E>(filename: &str, stream: S) -> io::Result<&str>
where
    S: Stream<Item = Result<Bytes, E>>,
    E: Into<BoxError>,
{
    let body_with_io_error = stream.map_err(|err| Error::new(ErrorKind::Other, err));
    let body_reader = StreamReader::new(body_with_io_error);
    futures::pin_mut!(body_reader);

    let mut filepath = PathBuf::from(MEDIA_ROOT);
    filepath.push(filename);
    let mut file = BufWriter::new(
        File::create(
            filepath
                .to_str()
                .ok_or_else(|| Error::from(ErrorKind::PermissionDenied))?,
        )
        .await?,
    );

    // Copy the body into the file.
    copy(&mut body_reader, &mut file).await?;

    Ok(filename)
}

Make a copy of your getchatts() in handlers.rs and name the copy getimages(). In getimages(), replace the SELECT statement with the following: "SELECT username, message, id, time, imageurl, videourl FROM chatts ORDER BY time DESC". This statement will retrieve all data, including our new image and video URLs from the PostgreSQL database.

Still in getimages(), replace the chattArr.push(/*...*/); line with:

            chattArr.push(vec![
                row.get("username"),
                row.get("message"),
                Some(row.get::<&str, Uuid>("id").to_string()),                
                Some(row.get::<&str, DateTime<Local>>("time").to_string()),
                row.get("imageurl"),
                row.get("videourl")
            ]);   

In addition to the original three columns, which we now access by column name instead of by index, we added reading the imageurl and videourl columns and included them in the chatt data returned to the front end.

Save and exit handlers.rs.

Editing main.rs

Add the following crates/modules at the top of your src/main.rs:

use axum::{
    // you can choose to "merge" the following with the existing
    // `extract` module or add it as a separate line. Both work.
    // If your merge list has more than one elements, put both inside a pair of parantheses.
    extract::DefaultBodyLimit,
    handler::HandlerWithoutStateExt,
    http::StatusCode,
    // ...
};

then add the following crate:

use tower_http::{
    limit::RequestBodyLimitLayer,
    services::ServeDir,
};

For the newly added getimages() and postimages() functions, add the following to the Router instantiation of the router variable in main.rs, right after the call to Router::new():

let router = Router::new() // look for this line and add the following right below the line
    .route("/postimages/", post(handlers::postimages))
    .layer(DefaultBodyLimit::disable())  // first disable default 2 MB limit
    .layer(RequestBodyLimitLayer::new(10485760)) // then add 10 MB limit       
    .route("/getimages/", get(handlers::getimages))
    .nest_service("/media",
        ServeDir::new(handlers::MEDIA_ROOT)
            .not_found_service(handle_not_found.into_service()))

For postimages(), we limit the size of each uploaded media file to 10 MB. To set this limit, we must first disable the default limit of 2MB/upload.

In addition to routing the two new APIs, the .nest_service line tells axum to redirect URL path /media to serve media files from our designated media directory, whose value is stored in handlers::MEDIA_ROOT. If the requested file is not found at the MEDIA_ROOT path, we call handle_not_found() to inform the user:

async fn handle_not_found() -> (StatusCode, &'static str) {
    (StatusCode::NOT_FOUND, "Not found")
}

Save and exit main.rs. Rebuild and restart chatterd:

server$ cargo build --release
server$ sudo systemctl restart chatterd

References

• Axum multipart/form-data example
• axum::extract::Multipart
• Axum static file path example
• axum_extra::routing::SpaRouter