It’s Not a Web Application Without a Database

The all important database ORM tooling… a light shining in the dark!

Part of a Series: Designing a Full-Featured WebApp with Rust
Part 1: Piecing Together a Rust Web Application
Part 2: My Next Step in Rust Web Application Dev
Part 3: It’s Not a Web Application Without a Database
Part 4: Better Logging for the Web Application
Part 5: Rust Web App Session Management with AWS
Part 6: OAuth Requests, APIs, Diesel, and Sessions
Part 7: Scraping off the Dust: Redeploy of my Rust web app
Part 8: Giving My App Secrets to the AWS SecretManager

Rack of hard drives perhaps attached to a database server supporting a web application
I think I just skipped a bit!?

What did we even use computers for before the Internet!? How much is your smartphone or laptop worth if you have zero Internet connectivity? What good is a web application without a database? Which is more important, the language or the architecture? The actors or the writing? These… are all questions… but one wouldn’t have a job without the other. Still, a web application needs a strong database layer but with abstractions to hide the raw SQL inside the code. So, we need to pick a database solution as carefully as we choose the programming language. True, that is, if we aren’t talking about the solution for our father’s VHS rental store and its single register (been there before!)

Don’t Spare the Database Layer

As a developer, the idea of Object-relational mapping (ORM) is about decoupling the low-level semantics of data storage and retrieval from the application code. An ORM abstracts database functionality inside your code just like a templating system abstracts HTML or JavaScript in your code. Everyone’s favorite, Stack Overflow, has a really good posted definition of ORMs along with the pros and cons of using one in a project. While you’re there, also check out Stack Overflow’s latest questions tagged [Rust] – maybe you can help someone out!

The ORM tools handle a big responsibility in app development. The Diesel crate has taken on this task for my sample app here at the beginning. With Rust and Diesel both being so new, I need to cut these tools some slack, but I also desperately want them to be fairly mature for the benefit of the language.




Sample Web Application Goals

Let me explain my concept for this sample app first – it likely won’t ever get finished, but that’s ok, it gives me something to practice with. A friend (check out his Kindle book, Seeing in the Dark: A Guide to Night Photography) and I came up with this idea over 10 years ago when we were scouring the local 200 mi radius for great places to shoot photographs. My 500px profile is filled with them – nearly all landscapes, I don’t do portraits or models.

Our idea came to be known as PinpointShooting.com (which isn’t a website yet, duh). Photographers could sign up for free and, using a Google Maps interface, indicate the best places to get some terrific shots. Some might be obvious, but there are some hidden places out there to spice up your photography.

Woman laying on stomach to take photograph, maybe my web application could give her some tips
She seems to be having fun!!

Handy tip: a photograph is often more interesting because it was taken from an angle or a hidden spot that no one else has picked. That way, your picture doesn’t look so much like a “touristy postcard.” That’s a reason photography books tell you to kneel or lay down or climb on things to take a picture. In fact, anything to change the normal viewing angle. What’s more boring than seeing a photo of a flower from 5′-6′ above? That’s the angle we see ALL the time, just walking around! Plus, that’s why photographers often refer to it as “making a shot” rather than “taking a shot.” There’s art involved above pressing the shutter release!

Back to the Database Chat

Anyway, enough about that – I need to store some data!! I’m sure this will expand and evolve, but this is what I’m thinking for tables so far:

  • Users – to store the free user profiles, passwords, email address
  • Locations – to store the address and/or lat/long of the good spot to shoot from that a user has shared
  • Photos – by users to store the pictures (or links to pictures) taken from a location in the database
  • Ratings – by users of the locations they have been to and if the agree
  • Likes – by users of the photos uploaded or linked to
  • Visits – to the location linking together a User/Location when they share a good spot, or a User/Photo when they upload/link one, or a User/Rating when they leave one

I start the DB design with some Enum types inside Postgresql. Most of these tables are going to have a status field with just a few possible choices. The enums turn out to be something that Diesel doesn’t seem to handle well (yet). In fact, I found a crate because of that fact. Wait… I take that back… I found 3 crates. Diesel-derive-enum has, by far, the most downloads, so I download that one. However, I only fooled with it a short time before I gave up. This is just a silly practice app, so I will switch to charfields. Well, they don’t work right away for me either, and it is getting late. I’ll just use some varchars which I already have a ton of.

A database was not the first thing to migrate, here are some birds headed south
Databases were not the first things to migrate!

Diesel Database Migration Tool

Diesel has a database migration tool to allow your project to record (and store in the repository) the entire path of changes the database undergoes during the product lifetime. It seems that tool name may have come from Rails – or there are only so many synonyms to choose from. I have used (and beat my head against) Sqitch and I’m just becoming aware of Skeema. Database schema management turns out to be a somewhat complex and fragile flow when you work with over 100 other developers in a fast-moving codebase. Because of the field issues I had (and because this project is ONLY me and is just starting), I reset my database to start over several times. In the end, I have a successful compile of the above DB structure abstracted into my code.

Diesel, btw, has a wonderfully impressive website to introduce you to the available functionality. I use the DBIX::Class module (often referred to as DBIC) with Perl, and it is as complicated as DB queries can become! But those are mostly the exception; DBIC generally abstracts the SQL nicely into some OO code to get me the data I need from the database. Most importantly, the Diesel Getting Started Guide is a great introduction to what using this crate is like. Plus, in addition to the normal crate API-style documentation, there are several other guides – some diving deep into the mechanics and benefits of Diesel.

I need to really get into some code to see how Diesel will be, but I have high hopes from what I’ve seen so far. A systems programmer (which I’m not) probably has far less need for the tools and add-ons that I’ve been playing with these past three posts. However, for big-scale web application development, most of these are key requirements!

More to come…

I hope all of these tools continue to develop with the pace of Rust itself, and I hope to get good enough with Rust that I can help out with some of the modules in the future! Once again, here’s the repository link to my code (at whatever state it exists when you read this post). I expect to have more to share about the PinpointShooting web application soon.

Edit: Meanwhile, go check out this fantastic post that looks at using Rust on both the front-end AND back-end of a web application! Web Assembly is hot right now, so this is a great read!

My Next Step in Rust Web Application Dev

Part II: I am impressed with Rocket and the add-ons and pile things into my sample web application

Part of a Series: Designing a Full-Featured WebApp with Rust
Part 1: Piecing Together a Rust Web Application
Part 2: My Next Step in Rust Web Application Dev
Part 3: It’s Not a Web Application Without a Database
Part 4: Better Logging for the Web Application
Part 5: Rust Web App Session Management with AWS
Part 6: OAuth Requests, APIs, Diesel, and Sessions
Part 7: Scraping off the Dust: Redeploy of my Rust web app
Part 8: Giving My App Secrets to the AWS SecretManager

Bridge Day 2017, New River Gorge, WV – a framework is important!

My previous post on working towards a shell of a Rust web application covered global, static application settings and logging. There is still, probably, a change to come to allow for structured logging. However, this time lets start with the actual framework I’m using. First I download Iron, but it doesn’t feel right and I notice it hasn’t been worked on for two years. That’s an immediate clue to keep looking, so I move on to another I have heard mentioned: Rocket.

Check out the podcast archives!

I’ve been listening to the archives of the mostly-discontinued podcast New Rustacean by Chris Krycho and, even though they are a couple of years old now (which is pretty old considering the pace of Rust), I still get an idea of where Rust started and how far it has come. Two episodes I just listened to – a 2016 episode about Diesel and a 2017 episode about Rocket – even helped me with this web application example I’m working on. It’s impressive these two Crates have continued getting updates and support after a couple of years. I’ll talk about Diesel in the next post, but for now let’s look at Rocket, plus some add-ons for templating, static web serving, web security, and interactive communication.

Rocket – Rust Framework

The Rocket crate received some high praise from Krycho in his “Crates You Should Know” episode and it seems the maintainers have continued with what impressed him. Rocket has a fantastic website and documentation guide making it a breeze to get started. The built-in route handling reminds me of perl Catalyst (how different could it be?). This feels very familiar. Here is my (simplified at this early point) implementation of getting Rocket started in my web application. The global static CONFIG I talked about last time continues to help here:

pub fn start_webservice() {
    // start rocket webservice
    let bind_address = &CONFIG.webservice.bind_address;
    let bind_port = &CONFIG.webservice.bind_port;
    let version = include_str!("version.txt");

    rocket::ignite()
        .attach(Template::fairing())
        .attach(SpaceHelmet::default())
        .mount("/", routes![routes::index])
        .mount("/img", StaticFiles::from("src/view/static/img"))
        .mount("/css", StaticFiles::from("src/view/static/css"))
        .mount("/js", StaticFiles::from("src/view/static/js"))
        .launch();

    warn!(
        "Listening on {}:{} as version {}",
        bind_address, bind_port, version
    );
}

Just as important as Rocket, in my view, are the capabilities added with the rocket_contrib crate. You can see some of them referenced in the chain of functions getting Rocket configured and running in my code above. So far, I already pull in the handlebars_templates, serve, helmet and JSON features.




rocket_contrib::handlebars_templates

I am pretty familiar with Template::Toolkit for use with Perl, but I’m only getting started with handlebars. Also, there is another rocket_contrib::templates option called tera templates. I mostly picked one of the two at random. I’m not yet to the point of working in HTML templates for this sample app, so this choice might change in the future. Who knows – maybe we need the option of a Template::Toolkit add-on as well!?

rocket_contrib::serve

Since Rocket builds-in the webserver funcionality (and I’m not sure I like this – I think I can front the web application with Nginx or some other service, so I need to look into that), I need it to serve local, static files (like CSS, Javascript, and images) as well. Rocket_contrib::serve handles this job for me with just some mount calls in the Rocket chain (see above). Again, I’m just getting set up, but this seems a good first step to get me going.

rocket_contrib::helmet

Security needs to extend past the coding language, of course, and rocket_contrib::helmet gives you a start at just a couple of HTTP concerns. The web is an absolutely crazy place – having read The Tangled Web by Michal Zalewski, I’m even more acutely aware of the convoluted and contentious history of web servers, web browsers, and the web languages we’ve ended up with, and I’m surprised things work at all! If helmet stops even one or two classes of those dangers, in it goes.

rocket_contrib::JSON

JSON is convenient to let the client browser (with Javascript) communicate back and forth with the server via (ahem) AJAX calls. I’ve included rocket_contrib::json just for this eventuality. Maybe there a more Rustic device or maybe this is still just “one way to do it”. I’ll learn more when I get to that point.

Ok, next up, we talk about the all-important ORM piece. I’m really impressed with Diesel so far, especially with the guide and documentation… but we’ll get to that with the next post.

Piecing Together a Rust Web Application

Part I: I work on a global static Config and on logging

Part of a Series: Designing a Full-Featured WebApp with Rust
Part 1: Piecing Together a Rust Web Application
Part 2: My Next Step in Rust Web Application Dev
Part 3: It’s Not a Web Application Without a Database
Part 4: Better Logging for the Web Application
Part 5: Rust Web App Session Management with AWS
Part 6: OAuth Requests, APIs, Diesel, and Sessions
Part 7: Scraping off the Dust: Redeploy of my Rust web app
Part 8: Giving My App Secrets to the AWS SecretManager

Hrm… how to find the right piece on Crates.io ???

For over a decade, I’ve worked on web apps with Perl, the last several years with Catalyst/Moose/DBIC and a slew of internal abstractions. There are a bunch of features I expect to need in any web app: config files (at different platform levels); structured logging; database ORM; templating; cookie, authentication, and session controls; en/decryption for access secrets; etc. I spent most of the overnight hours playing with piecing together a Rust web application, though I still have much more to do. After struggling for hours for what amounts to 279 lines of Rust code, I decided it was well worth it. I’ll try to explore some of my problems and what I worked out. This might take more than one post, so I don’t put you to sleep.

Global (Static) Application Config

This probably isn’t idiomatic Rust and frowned upon for even more reasons. I’m sure I’ll adapt to Rustic thinking as I learn, but for now, I’m liking this. I like to have a Config struct that is setup at init time and is immutable (music to the Rust compiler’s ears, I’m sure). I battled this with const and lazy_static and numerous other things. Eventually, I settled on a Crate that the author seems to have stumbled into writing/publishing (unless I’m missing some context): OnceCell. Where I was having trouble getting lazy_static to work, once_cell::sync::OnceCell seemed to work for me rather quickly.

Coupled with that, I like the notion of having Config settings initialized by YAML or JSON or TOML files and also able to be overridden in some way – usually environment variables. This path (and an earlier post) took me to the aptly named Config Crate. It does just what I need for pulling settings into a config from various places. I ended up adding the dotenv Crate as well, because something else used it in an example. I’m not sure I’ll keep ALL of these options forever, but it’s in the mix for now. There are, obviously, many ways to allow overrides vs protect the settings on disk, and many ways to decide which platform you are running on besides an ENV variable – and I’m flexible.

EDIT: I just realized that putting <Mutex> on my type means I have to lock() it to read it – and that prevents other functions (and other threads) from reading it. Since I’m ok with CONFIG being immutable, I really don’t need the Mutex, so I dropped it.

My settings.rs module looks like this for now:

use config::{Config, Environment, File};
use dotenv::dotenv;
use once_cell::sync::Lazy;
use serde_derive::Deserialize;
use std::env;

#[derive(Debug, Deserialize)]
pub struct Server {
    pub run_level: String,
}

#[derive(Debug, Deserialize)]
pub struct WebService {
    pub bind_address: String,
    pub bind_port: u16,
}

#[derive(Debug, Deserialize)]
pub struct Settings {
    pub server: Server,
    pub webservice: WebService,
    pub database_url: String,
}

pub static CONFIG: Lazy<Settings> = Lazy::new(|| {
    dotenv().ok();
    let mut config = Config::default();
    let env = env::var("PPS_RUN_MODE").unwrap_or("development".into());

    config
        .merge(File::with_name("conf/default"))
        .unwrap()
        .merge(File::with_name(&format!("conf/{}", env)).required(false))
        .unwrap()
        .merge(File::with_name("conf/local").required(false))
        .unwrap()
        .merge(Environment::with_prefix("PPS"))
        .unwrap();
    match config.try_into() {
        Ok(c) => c,
        Err(e) => panic!("error parsing config files: {}", e),
    }
});

And config files like these examples:

conf/default.toml:

[server]
run_level = "default"

conf/development.toml:

[server]
run_level = "development"

[webservice]
bind_address = "0.0.0.0"
bind_port = 3000

conf/staging.toml:

[server]
run_level = "staging"

[webservice]
bind_address = "0.0.0.0"
bind_port = 3000

and, conf/production.toml:

[server]
run_level = "production"

[webservice]
bind_address = "0.0.0.0"
bind_port = 80



Application Logging

Piecing together a Rust web application includes another area of big concern – logging! Logging can easily become a tremendous burden of bandwidth and storage space, but a single log record might explain a production incident and lead you to a quick fix! Structured logging is great for logging platforms because storing and especially searching can be greatly improved when the message is static and data fields attached to the log record fill in the variable gaps.

To get things going, I started with the Crates log and simple_logger, but I probably will move to slog for the structured logging. The very first line in my main() is to call setup_logging() so if anything breaks on app initialization, we should get a log for it. With CONFIG a global static, this simple function looks like this for now, but soon I will work out specifying the logging level in the settings so it can be verbose for devs, but tamer on production:

pub fn setup_logging() {
    simple_logger::init_with_level(log::Level::Info)
        .expect("Trouble starting simple_logger");
    let run_level = &CONFIG.server.run_level;
    warn!("Running as run_level {}", run_level);
}

We still have the web framework, the database, encryption and more to come. Next up, more single-word Crates: Iron, Rocket and Diesel. I’d love to hear what Rust developers think of this so far – suggestions are welcome. Here’s the repository on Github – you can skip ahead and see what other messes I’ve made.