Start Coding Now

Empty Image Fields in Django

2024-10-29T00:00:00-07:00

tl;dr

Django FileFields and ImageFields are stored as strings in your database. Don't use null=True with them, instead use blank=True and filter by .filter(image__exact="").

Discovering a Bug

I ran into a bug where I was trying to filter a QuerySet by image__isnull=False to determine if the image field was set. It ended up returning all objects where image was not yet set, so I did some digging.

How `FileField` and `ImageField` Look in the Database

The FileField and ImageField are both implemented in your database table as a varchar field. Note the type column from the psql display table below:

                               Table "public.myapp_mymodel"
    Column    |          Type          | Collation | Nullable |             Default
--------------+------------------------+-----------+----------+----------------------------------
 id           | bigint                 |           | not null | generated by default as identity
 image        | character varying(100) |           | not null |

This stores the relative path of the file, including your upload_to and filename.

The problem is that these strings have two values that might mean "empty": None and "".

How to Define a Model with Not Required File Fields

I had defined my model like this:

class MyModel(models.Model):
    image = models.ImageField(
        upload_to="images/",
        null=True,
        blank=True,
    )

But I should have removed the null=True option:

class MyModel(models.Model):
    image = models.ImageField(
        upload_to="images/",
        blank=True,
    )

How to Filter for Empty File Fields

Now, I still cannot filter by image__isnull, because the image field should never be set to NULL in the database and None in Python.

But I can filter by .filter(image__exact=""), which is the empty string.

Or exclude those values with .exclude(image__exact="").

Opening Django Model Files in PyMuPDF

2024-10-28T00:00:00-07:00

I have been working on a file upload feature for a Django project. We use django-storages to store files in AWS S3, but we store them on the filesystem for local development.

PyMuPDF is a Python library for reading and writing PDF files. It's very fast and I've been delighted to use it.

I ran into a small issue getting my Django file uploads to work with PyMuPDF. This post will outline my notes on how I worked through it.

Opening a file from the local filesystem

Let's say we have a model SlideDeck with a FileField that stores a PDF file:

class SlideDeck(models.Model):
    pdf = models.FileField(
        upload_to="slide_decks",
        max_length="500",
    )

And a model SlideImage that stores an image for each slide in the PDF:

class SlideImage(models.Model):
    slide_deck = models.ForeignKey(
        SlideDeck,
        on_delete=models.CASCADE,
    )
    img = models.ImageField(
        upload_to="slide_decks/images",
        max_length="500",
    )

And we have a helper function to create a list of SlideImage objects from a SlideDeck object:

def _create_empty_slides(slide_deck: SlideDeck) -> list[SlideImage]:
    pdf_document = pymupdf.open(slide_deck.file.path)
    slide_img_objs = SlideImage.objects.bulk_create(
        [
            SlideImage(slide_deck=slide_deck)
            for i in range(len(pdf_document))
        ]
    )
    pdf_document.close()
    return slide_img_objs

The above works with the Django default FileSystemStorage backend. But if you're using a remote storage backend like the AWS S3 backend from django-storages, the file will not be available on the filesystem.

We are using AWS S3 and this was the error I ran into when trying to open a file with open(model_obj.file.path):

NotImplementedError: This backend doesn't support absolute paths.

Should you open with `file.name` or `file.path`?

Assuming we have a model FieldFile object:

file = model_obj.file

file.name returns the relative path to the file.

file.path returns the absolute path on the local filesystem.

We could then rewrite the above _create_empty_slides() function like this:

def _create_empty_slides(slide_deck: SlideDeck) -> list[SlideImage]:
    pdf_document = pymupdf.open(slide_deck.file.name)
    slide_img_objs = SlideImage.objects.bulk_create(
        [
            SlideImage(slide_deck=slide_deck)
            for i in range(len(pdf_document))
        ]
    )
    pdf_document.close()
    return slide_img_objs

And that would still work on the filesystem, but it's not portable to a remote storage backend. More on that later.

Opening an `InMemoryUploadedFile`

There's a GitHub discussion about uploaded files in PyMuPDF. The author had to specify to seek back to the beginning of the file to open it:

import pymupdf


def get_pages_of_uploaded_pdf(request: HttpRequest) -> int:
    # Get the file
    uploaded_pdf = request.FILES.get("user_uploaded_pdf")

    # Set read location
    uploaded_pdf.seek(0)

    # Open the file
    pdf_document = pymupdf.open(
        stream=uploaded_pdf.read(), filetype="pdf"
    )
    num_pages = len(pdf_document)
    pdf_document.close()
    return num_pages

Opening a file from remote storage

The PyMuPDF docs have a section on opening remote files. They outline getting the remote file content as a bytes object, and then opening the file with PyMuPDF:

import pymupdf
import requests


r = requests.get("https://mupdf.com/docs/mupdf_explored.pdf")
data = r.content
doc = pymupdf.open(stream=data)

I started to implement this in our project, but didn't want to have to check for what backend we're using every time we want to open a file.

So instead, we can leverage the default_storage class from Django to open the file:

import pymupdf
from django.core.files.storage import default_storage


def get_pages_from_pdf(slide_deck: SlideDeck) -> int:
    # Get the file bytes data
    with default_storage.open(slide_deck.file.name, "rb") as f:
        content: bytes = f.read()

    # Open the file with pymupdf
    pdf_document = pymupdf.open(stream=content)
    num_pages = len(pdf_document)
    pdf_document.close()
    return num_pages

This reads the entire file into memory as a bytes object in the content variable. If you're low on memory, this might create problems.

Opening a file with `pymupdf` as a context manager

We can rewrite the above function to use a context manager, automatically closing the file when we're done:

import pymupdf
from django.core.files.storage import default_storage


def get_pages_from_pdf(slide_deck: SlideDeck) -> int:
    # Get the file bytes data
    with default_storage.open(slide_deck.file.name, "rb") as f:
        content: bytes = f.read()

    # Open the file with pymupdf
    with pymupdf.open(stream=content) as pdf_document:
        num_pages = len(pdf_document)
    return num_pages

Custom Django Python Migration

2024-10-03T00:00:00-07:00

Sometimes I find myself wanting to add some model objects along with a feature. You can make a custom migration to handle that!

Overview of what we will do

Create an empty migration
Create a forwards operation
Create a backwards operation
Tie those operations to the migration
Apply, un-apply, and re-apply the migration

Whole file contents

For a tl;dr, make an empty migration and populate it like this:

# Generated by Django 5.1.1 on 2024-10-03 12:59

from django.db import migrations


def forwards(apps, schema_editor):
    MyModel = apps.get_model('my_app', 'MyModel')
    MyModel.objects.create(
        field_name1="name", field_name2="describe"
    )


def backwards(apps, schema_editor):
    MyModel = apps.get_model('my_app', 'MyModel')
    MyModel.objects.get(
        field_name1="name", field_name2="describe"
    ).delete()


class Migration(migrations.Migration):

    dependencies = [
        ("app_name", "0001_initial"),  # could be more here
    ]

    operations = [
        migrations.RunPython(forwards, backwards),
    ]

Create an empty migration

python manage.py makemigrations  --empty

I tend to add a descriptive name as well:

python manage.py makemigrations  \
    --name  \
    --empty

The empty file should look something like this:

# Generated by Django 5.1.1 on 2024-10-03 12:59

from django.db import migrations


class Migration(migrations.Migration):

    dependencies = [
        ("app_name", "0001_initial"),  # could be more here
    ]

    operations = []

Create the change you want to see

Next, I'm going to add a top-level function that runs the code I seek to run:

def forwards(apps, schema_editor):
    MyModel = apps.get_model('my_app', 'MyModel')
    MyModel.objects.create(
        field_name1="name", field_name2="describe"
    )

Note the part where we set the model with apps.get_model. This ensures the migration uses the correct version of the model at the time the migration is applied even if the model changes later.

Undo the change you made for migrating back in time

Then, we'll write the equivalent of an "undo" operation in another top-level function:

def backwards(apps, schema_editor):
    MyModel = apps.get_model('my_app', 'MyModel')
    MyModel.objects.get(
        field_name1="name", field_name2="describe"
    ).delete()

Tie these operations to the migration

Now we need to tell Django that we're looking to run Python code for this migration:

class Migration(migrations.Migration):

    dependencies = [...]

    operations = [
        migrations.RunPython(forwards, backwards),
    ]

Apply, un-apply, and re-apply the migration

Assuming, like our example, that the migration we've been working on is 0002:

# Apply the migration
python manage.py migrate  0002

# Check for the expected change

# Un-apply the migration
python manage.py migrate  0001

# Check for reversal of change

# Re-apply the migration
python manage.py migrate  0002

DjangoCon US 2024 - Short Summary

2024-09-25T00:00:00-07:00

I apologize in advance for (1) the brevity, and (2) that not every talk is listed here. I didn't get a chance to watch them all yet. I do not mean any disrespect to any speaker who spoke, but is not listed here.

Hopefully this can serve as a taste of the conference. I had a great time at DjangoCon US 2024, and I hope to see you next year!

Day 1

Learning, Teaching

Sheena O'Connell - Power to the People who Teach the People (Sheena's slides)

"Make It Stick" by Peter C. Brown, Henry I. Roediger III, and Mark A. McDaniel.
Mastery-based Learning
Self-efficacy

SPA vs Progressive Enhancement

Chris May - Choosing Wisely: SPA vs. HTMX for Your Next Web Project

"Progressive enhancement" means your site still works without JavaScript.
The longer someone is on a page, the more likely a SPA would be useful. - 20+ operations on the same subset of data? - UI for chat or conversations?
Katie Sylor-Miller's "Component Islands" explained in "Islands Architecture" by Jason Miller.
The browser can handle a lot of things without JavaScript.
Streamed HTML components
Article series from Alex Russell on JavaScript excess.
Packages: django-unicorn, htmx, alpinejs, django-template-partials, django-components, datastar, unpoly

Error Culture

Ryan Cheley - Error Culture

Reactive, firefighting Culture
Provide useful and relevant context along with your alerts: actionable, important items sent to the correct people.

Lightning Talks

Calvin Hendry-Parker: scaf
Paul Everitt: PEP750
Tim Allen: pyromania
Jay Miller: learn about Python
Santos: bandit, CodeQL
Sangeeta Jadoonanan: Django Stickers, Django Brew
Megan Voss: Carpalx, Tarmak (transitional colemak layout)
Norma Miller: White Coat Captioning

Django Forms

Josh Thomas - An Opinionated Guide to Modern Django Forms

Forms as_field_group
django-template-partials and django-cotton

SQL Queries

Chris Adams - A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related(), and prefetch_related()

django-debug-toolbar
select_related() for foreign keys
prefetch_related() for many-to-many relationships and reverse foreign keys
Make sure to inspect the 95th and 99th percentile queries on sites with high traffic because highly active users may be encountering problems that your other users don't feel.

Passkeys

Ryan Hiebert - Passkeys: Your password-free future

Passkeys are in development for django-allauth
Passkeys are inherently multi-factor (you own the device, you have the fingerprint/retina)
webauthn.guide, webauthn.io, passkeys.dev, Passkeys: A Shattered Dream

Running Django in Production

Vince Salvino - One Thousand and One Django Sites

Error detection might auto-trigger a self-healing patch or rollback
MaxMind for fraud detection
codered.cloud for deploying Django and Wagtail apps

Security

Adarsh Divakaran - Pygoat - Learn django security the hard way

Day 2

Self-Improvement

Mario Munoz - How To Be A Developer and Other Lies We Tell Ourselves

The way to be a better developer is to be a better human.

htmx + Alpine.js

Karen Tracey - Django + Alpine.js + htmx Ups & Downs

htmx for interacting with the server
Alpine.js for interacting with the user
These tools increase the shared knowledge between the frontend and backend
django-formset, django-template-partials, django-components, and django-cotton

Maps

Paolo Melchiorre - Maps with Django

SQLite uses SpatiaLite
PostgreSQL uses PostGIS
Django uses GeoDjango

Lightning Talks

Richard Terry: nanodjango
Adam Fast: what it takes to bring DjangoCon to your city
Jib Adegunloye: Brute forcing a new MongoDB integration for Django
Lilian: Djangonaut space mentorship program
Reddy Tintaya: TDD
Lacey Henschel: Django Commons supporting community package maintenance
Daniele Procida: Twelve rules for job applications and interviews
Rachell Calhoun: Django Girls
Sheena O'Connell: DjangoCon Africa 2023

End-to-End Testing

Avindra Fernando - Lessons from E2E Testing Web Applications

Cypress and Playwright
Testing Playground for selecting elements in your tests

Atomic Transactions

Eva Nanyonga - Only reliable Data: Protecting Database Integrity

Database transactions

Plugins

Simon Willison - How to design and implement extensible software with plugins

The beauty of plugins is that they allow you to extend the functionality of an app without changing the app itself
pluggy and djp

Migrations

Tim Bell - Deploying Django migrations at Kraken scale (Tim's slides)

Locks
Break up migrations into smaller chunks
"Async Migrations" from Yakko Majuri at Posthog

Large Scale Project Considerations

Benjamin "Zags" Zagorsky - Fighting Homelessness with Django (Zags' slides)

Used Vue with one Vue app per Django template that needed it
Sticky sessions on load balancer

Day 3

Django Fellows

Natalia Bidart - The Fellowship of the Pony

Who is a Django Fellow and what do they do?

Release Notes

Sarah Boyce - Hidden gems of Django 5.x

Not all changes are listed in the release notes

Django History

Frank Wiles - A Brief History of Django

How Django came to be (starting out known as "The CMS")

Lightning Talks

Natalia Bidart: https://inclusivenaming.org/
Brent O'Connor: https://github.com/epicserve/django-base-site
Levi Mann: Debugging SQL
Josh O'Brien: Tips for improving your CV
Sam Eure: Custom querysets
Marcelo Elizeche Lando: https://airelib.re/
Abigail (Afi) G.: Updates from the Ghanaian community
Jon Gould: Light work now makes your next job search easier
Alvaro Justen (Turicas): Storing data, vcard info in QR codes
Thibaud Colas: DSF updates

WebRTC

Ken Whitesell - WebRTC with Django, Channels, HTMX, and coturn

STUN = "What's my IP?" for WebRTC; like a ping
TURN = distribution hub for WebRTC data
SDP = data defining the content of the connection
https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Perfect_negotiation
chrome://webrtc-internals

PostgreSQL

Elizabeth Christensen - A Guided Tour Through Postgres Internals

Tons of tips on how to use PostgreSQL and the psql commmand

Django User Model

Will Vincent - Django User Model: Past, Present, and Future

Did you know Django has a wiki?
Django's current auth provides friction for new users trying to learn Django
Suggesting a profile model, ship with a login template, add Profile model to docs
Simon Willison also suggested maybe a profile JSONField on the User model

Bootstrapping

Carlton Gibson - API Maybe: Bootstrapping a Web Application circa 2024

Django is the perfect framework when money is tight

Understanding the Template Method Pattern in Python

2024-08-24T00:00:00-07:00

Introduction

I found myself wanting to write an extensible class for handling a few fancy new APIs out there. There are a ton that do the same thing, and the field is constantly evolving, so it's useful to be able to easily switch from one to the next.

I chose the Template Method design pattern. This is useful when you have a series of steps that need to be executed in a specific order, but the implementation of some steps may vary.

What is the Template Method Pattern?

The Template Method pattern defines the skeleton of an algorithm in a base class but lets subclasses override specific steps of the algorithm without changing its structure. It's called "template method" because it defines a template for an algorithm's steps.

A Simple Example: Coffee and Tea

Let's illustrate this pattern with a simple example of brewing beverages. We'll create a base class for brewing beverages and then implement specific classes for coffee and tea.

from abc import ABC
from abc import abstractmethod


class Beverage(ABC):
    def prepare(self):
        self.boil_water()
        self.brew()
        self.pour_in_cup()
        self.add_condiments()

    def boil_water(self):
        print("Boiling water")

    def pour_in_cup(self):
        print("Pouring into cup")

    @abstractmethod
    def brew(self):
        pass

    @abstractmethod
    def add_condiments(self):
        pass


class Coffee(Beverage):
    def brew(self):
        print("Dripping coffee through filter")

    def add_condiments(self):
        print("Adding sugar and milk")


class Tea(Beverage):
    def brew(self):
        print("Steeping the tea")

    def add_condiments(self):
        print("Adding lemon")

# Usage
coffee = Coffee()
tea = Tea()

print("Making coffee:")
coffee.prepare()

print("\nMaking tea:")
tea.prepare()

In this example, Beverage is our abstract base class that defines the template method prepare(). This method outlines the steps for preparing a beverage, but leaves the implementation of brew() and add_condiments() to the subclasses.

Benefits of the Template Method Pattern

Reduce duplication by implementing the common parts of the algorithm in the base class.
Subclasses can easily override certain steps without changing the overall algorithm structure.
The base class handles the algorithm's structure, while the subclasses focus on the specific details of each step.

As long as you're clear on the data structure you want your methods to return, you can swap in many different subclasses.

Real-World Applications

The Template Method pattern is widely used in frameworks and libraries. For instance:

In web frameworks, the base controller class might define a template method for handling HTTP requests, with subclasses implementing specific behaviors for different routes.
In testing frameworks, the setup-run-teardown process is often implemented as a template method, allowing test classes to override specific parts of this process.
When calling external services, your template transforms the API response into the data structure your application is expecting.

Example Template Class

You might want to call a nutrition facts API to get info about food, but not feel locked in to a single API provider. You could write a NutritionFacts base class which outlines the get_nutrition_facts method and returns a Django NutritionInfo model object, outlining the fat, carbohydrates, and protein in a particular meal.

Then, when you're fed up (get it?) with the Spoonacular API, you can implement the Edamam API with .

Conclusion

The Template Method pattern allows for the creation of flexible and reusable code structures, making it easier to implement variations of an algorithm while keeping the overall process consistent. By understanding and applying this pattern, you can write more maintainable and extensible code in your Python projects.

Remember, while design patterns like the Template Method are useful, they should be applied judiciously. If you're only making one class, it might not be helpful to make that inherit from a base class.

Best of luck in your design endeavors.

Custom Error Messages on Model Deletion in the Django Admin

2024-08-21T00:00:00-07:00

I have been working on a project where we might want to delete models even just for testing purposes, but we don't want to accidentally delete models.

Protect Model Deletion

Let's say you have a Django model:

from django.db import models


class User(models.Model):
    name = models.CharField(max_length=255)
    email = models.EmailField()


class Rental(models.Model):
    name = models.CharField(max_length=255)
    current_renter = models.ForeignKey(
        User, on_delete=models.CASCADE, null=True, blank=True
    )

At some point, I might sell and delete my Rental object, but I don't want to do that if I have someone renting it at the moment.

Override Model `delete()` Method

We can put in our own custom logic to prevent this deletion if there's an active renter:

from django.core.exceptions import ValidationError  # new


class Rental(models.Model):
    # ...
    def delete(self, *args, **kwargs):
        # Check if someone is renting at the moment
        if self.user_set.exists():
            raise ValidationError(
                f"Cannot delete rental {self.name} while someone is in it."
            )

        # If no renter, proceed with deletion
        super().delete(*args, **kwargs)

Message the User by Overriding the `delete_model()` Method

In our Model's Admin class, we can override the delete_model() method to give the user feedback as to why we stopped their deletion so they aren't left totally confused:

from django.contrib import admin
from django.contrib import messages


@admin.register(Rental)
class RentalAdmin(admin.ModelAdmin):
    list_display = ("name", "current_renter")
    list_filter = ("current_renter",)
    search_fields = ("name", "current_renter__name", "current_renter__email")

    def delete_model(self, request, obj):
        try:
            return super().delete_model(request, obj)
        except ValidationError as e:
            self.message_user(request, str(e.message), messages.ERROR)
            return

This will pass along our error message through the Django Messages framework.

The ModelAdmin.delete_model() method normally calls the object's deletion. This is what we just overrode, and we can use the ValidationError we've raise to communicate across methods.

Neat.

Customize the Deletion HTTP Response by Overriding the `response_delete()` Method

You might notice that if you test this code, you'll get two messages: the one you wrote that says the deletion failed, and another one that said it succeeded.

Weird!

That's because Django's method version is cuing up a success message.

We can override this method an add a check related to our deletion logic to divert the response if it makes sense:

from django.http import HttpResponseRedirect  # new


@admin.register(Rental)
class RentalAdmin(admin.ModelAdmin):
    # ...
    def response_delete(self, request, obj_display, obj_id):
        if Rental.objects.filter(pk=obj_id).exists():
            # Return to the admin's Rental detail page
            return HttpResponseRedirect(
                reverse("admin:rentals_rental_change", args=[obj_id])
            )

        # Otherwise proceed as usual
        return super().response_delete(request, obj_display, obj_id)

This will stop Django from adding that confusing "successful deletion" message.

Voila.

Bulk Deletions Follow a Different Course of Action

To my knowledge, the bulk deletion option in the Django Admin uses different methods to handle the bulk deletion behavior.

If you're familiar with how to override this behavior, I'd be curious to hear!

You can disable bulk deletions by overriding the method ModelAdmin.get_actions():

admin.register(Rental)
class RentalAdmin(admin.ModelAdmin):
    # ...
    def get_actions(self, request):
        actions = super().get_actions(request)
        if "delete_selected" in actions:
            del actions["delete_selected"]
        return actions

Custom Dictionary Representation of Python Dataclass

2024-07-10T00:00:00-07:00

I've really enjoyed the shortcuts provided by Python's dataclasses.

Recently, I found myself working with an API that required camelCase keys, but I wanted to keep my Python keys in snake_case as I find it comforting.

Creating a dataclass

I created the following dataclass for working with cell locations in Google Sheets:

from dataclasses import dataclasses

@dataclass
class Coordinate:
    sheet_id: int
    row_index: int
    column_index: int

To instantiate an object:

coord = Coordinate(0, 1, 3)  # cell D2

Turning the object into a dictionary

The dataclasses.asdict() function transforms your dataclass object. It reuses your declared key names. For example:

from dataclasses import asdict

expected = {
    "sheet_id": 0,
    "row_index": 1,
    "column_index": 3
}
actual = asdict(coord)
assert(expected == actual)  # True

Making a custom dict factory

asdict() accepts an optional parameter dict_factory which is a function that tells asdict how to parse the dataclass object.

# help us transform the keys strings
def snake_case_to_camel_case(value: str) -> str:
    # split character and remove underscores
    data_list: list[str] = data.split("_")

    # capitalize the first letter of all words except the first
    for i in range(1, len(data_list)):
        data_list[i] = data_list[i][0].upper() + data_list[i][1:]
    return "".join(data_list)

# our method to plug into `asdict()`
def camel_case_dict_factory(data):
    return {snake_case_to_camel_case(field[0]): field[1] for field in data}

To use this new factory:

expected = {
    "sheetId": 0,
    "rowIndex": 1,
    "columnIndex": 3
}
actual = asdict(
    coord, dict_factory=camel_case_dict_factory
)
assert(expected == actual)  # True

For reference, here's the standard implementation of asdict().

Where Does Validation Occur?

2024-03-04T00:00:00-07:00

We can be more confident that our application will not break during runtime if we validate our data is of an expected size/shape/range.

If we have a user account, for example, we don't want to save "larryjohnson" as an email address because when we try to email that address, the application will fail. We might want to restrict names to a reasonable character length, numbers to be only positive, or start dates to be earlier than end dates.

Where Does Validation Occur?

There are many places where data is validated in a frontend + DRF backend situation:

The user validates what they write when they enter it, assuming they are not malicious
The browser validates a field or form using HTML5's constraint validation for common use cases, assuming the correct usage of the type attribute on an input field or other validation-related attributes such as required and pattern
The browser can also validate a field or form in custom situations using JavaScript
The serializer validates each field on the serializer with their respective validate_(self, value) methods
The serializer validates the serializer as a whole with the validate(self, attrs) method
The database validates the data does not violate any database constraints

Simple Validation of Data Types in the Serializer

When using a serializer, DRF maps JSON values back into native Python values. This is the simplest form of validation within our API.

As an example, if the supplied string cannot be coerced into a datetime, DRF will raise a serializers.ValidationError on the field defining the datetime.

Custom Data Validation in the Serializer

TestDriven.io has a good overview of custom data validation in DRF.

As mentioned above, there are two access points for validating your serializer data:

Field
Object

When checking serializer validity, first each field is checked, and lastly the object as a whole is checked.

Here's an example serializer with custom validation for the name and rating fields as well as a custom object-level validator:

from rest_framework import serializers
from .models import Product, User


class ProductSerializer(serializers.ModelSerializer):
    class Meta:
        model = Product
        fields = ["name", "creator_id", "description", "rating"]

    def validate_name(self, value):
        if value.length > 140:
            raise serializers.ValidationError(
                "We play by old school Twitter rules here"
            )

    def validate_rating(self, value):
        if value < 1 or value > 5:
            raise serializers.ValidationError(
                "Rating must be between 1 and 5"
            )
        return value

    def validate(self, data):
        user = User.objects.get(id=data["creator_id"])
        if Product.objects.filter(
            creator=user,
            name__iexact=data["name"],
        ).exists():
            raise serializers.ValidationError(
                f"A product with this name already exists for creator {user}"
            )

Writing Reusable Validators

If you have a validator to reuse in multiple serializers, you can write it separately and then set it on the field under the validators argument.

from rest_framework import serializers
from .models import Product


def is_rating(value):
    if value < 1:
        raise serializers.ValidationError(
            "Rating cannot be lower than 1."
        )
    if value > 5:
        raise serializers.ValidationError(
            "Rating cannot be higher than 5."
        )


class ProductSerializer(serializers.ModelSerializer):
    rating = serializers.IntegerField(validators=[is_rating])

    class Meta:
        model = Product
        fields = ["name", "creator_id", "description", "rating"]

How to Kill the Django Development Server Without Ctrl+C

2023-11-05T00:00:00-07:00

Let's say you're developing a Django project and you've lost track of the terminal window that ran your python manage.py runserver command. If you can't press Ctrl+C in the terminal, how do you stop the server?

Finding the `runserver` Process

First, we need to find the ID of the manage.py runserver process.

On Linux, you can list all processes with the following command:

ps aux

To search the results, you can pipe the output of that command into the grep command:

ps aux | grep manage.py

If you want to search multiple words, enclose them in single quotes ' or double quotes ":

ps aux | grep 'manage.py runserver'

This should give you rows of table data such as:

lance    12839  0.0  0.1  57068 49840 pts/25   S+   07:11   0:00 python manage.py runserver
lance    12840 23.0  0.2 435156 67752 pts/25   Sl+  07:11   4:40 /home/lance/django-workout/.venv/bin/python manage.py runserver
lance    23752  0.0  0.0   4040  1980 pts/30   S+   07:31   0:00 grep --color=auto manage.py runserver

This can be confusing because there are a few numbers here and I'm not sure what each number represents.

To prepend a row of headers, we can run two commands back-to-back:

ps aux | head --lines=1 && ps aux | grep 'manage.py runserver'

The head command is used for printing the beginning of a file, or, if no file is specified, it reads standard input in the console.

We're looking for the PID or the "Process ID":

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
lance    12839  0.0  0.1  57068 49840 pts/25   S+   07:11   0:00 python manage.py runserver
lance    12840 23.0  0.2 435156 67752 pts/25   Sl+  07:11   4:40 /home/lance/django-workout/.venv/bin/python manage.py runserver
lance    23752  0.0  0.0   4040  1980 pts/30   S+   07:31   0:00 grep --color=auto manage.py runserver

There are three to pick from. Which do we choose?

We can kill either of the first two, PID 12839 or PID 12840. They are related. Notice that they have the same start time.

If we read the command from the third process, we can see it's a grep command. That's the one we just ran to search for results. That process is already gone by the time we're reading the table.

Killing a Process

Now that we know the process ID, we can tell the operating system to stop it:

kill 12839

Zero results suggests the command was successful, but we can double check.

Verify the Process was Terminated

We can confirm this worked by checking our process list again:

ps aux | head --lines=1 && ps aux | grep 'manage.py runserver'

You could also load up http://127.0.0.1:8000 in your browser and see if your Django project is still being served there.

Summary

I had this happen to me when I started a server from the VSCode integrated terminal right when VSCode decided to restart.

Introducing djarter – An opinionated Django starter project

2023-11-02T00:00:00-07:00

I have spent six years resisting the urge to make a starter template because (a) it locks you in to a particular development style, and (b) it prevents the repetition that promotes learning.

But in the spirit of deploying high quality work, this repo contains a few best practices and useful tools for Django projects.

These features would take a while to get set up from scratch, and might not be worth it for every project. But cloning this starter template can save you roughly a full day of work if you're planning to extend the built-in user authentication.

Most importantly, though, no Bootstrap styling was used in the making of this template.

Check out djarter on GitHub.

Logging in Google Apps Script

2023-08-25T00:00:00-07:00

I have some Google Apps Script code that archives emails. I can see if the code ran without errors in the execution log, but I would like to also see how many emails were archived on a run to know if it changed the state of my inbox.

We can use the execution log to do some simple logging.

function archiveInbox() {
    var threads = GmailApp.search('label:inbox from:(jobalerts-noreply@linkedin.com) older_than:4d');
    for (var i = 0; i < threads.length; i++) {
        console.log(
            'Moving thread to archive: "%s"',
            threads[i].getFirstMessageSubject()
        );
        threads[i].moveToArchive();
    }
}

In the snippet above, the console.log statement gets pushed directly to the execution log. This will leave a note whenever a thread gets archived with the subject of the thread.

To view the logs, check the execution log from the Executions sidebar menu.

Auto-Archive Emails in Gmail

2023-07-28T00:00:00-07:00

Goal

I'm looking for a job in software engineering and have signed up for job alerts from LinkedIn.

I still, however, have a personal training business to run, and these emails are muddying my inbox, making it easier to miss important emails from my clients.

Given these job alerts are time-sensitive, I am okay with archiving emails that have been in my inbox for three days.

A good alternative would be to check these emails right away, but hey, sometimes I get busy!

Starting Point

I'm starting with the following snippet from Amit Agarwal.

function cleanInbox() {

    var delayDays = 2;

    var maxDate = new Date();
    maxDate.setDate(maxDate.getDate()-delayDays);

    var threads = GmailApp.getInboxThreads();

    for (var i = 0; i < threads.length; i++) {
        if (threads[i].getLastMessageDate() function archiveInbox() {
            var threads = GmailApp.search('label:inbox is:read older_than:2d');
            for (var i = 0; i < threads.length; i++) {
                threads[i].moveToArchive();
            }
    }

Google Apps Script Project Setup

Head to https://script.google.com
Make sure you're logged in with the account associated with the Gmail inbox you want to clean
Click on "+ New Project"
Click on the "+" button next to Services and add "Gmail" to give the script permission to use Gmail, an advanced service.

Your script will be written inside the code.gs file that automatically opens when creating a new project.

Search and Archive Script

Open up code.gs under Files and enter your script. Here's how I adjusted Amit's code to serve my purposes:

function archiveInbox() {
    var threads = GmailApp.search('label:inbox from:(jobalerts-noreply@linkedin.com) older_than:3d');
    for (var i = 0; i < threads.length; i++) {
        threads[i].moveToArchive();
    }
}

Read the documentation to learn more about the GmailApp. For example, you may want to skip archiving and move the thread straight to the trash.

A Note on Feature Design

When searching for the features I wanted, it was helpful to take a step back from the Gmail Service documentation and think about how I would accomplish this task using the standard Gmail graphical user interface.

For my situation, I would use search to filter emails by sender and date. I figured out who the sender was and used the date filtering similar to Amit's example.

Test Your Script

Once you think your code is correct, click on the "Save" button and then the "Run" button.

An Execution log should open at the bottom. If everything worked, you should see "Execution started" and "Execution completed".

You can add console.log("message") statements to tell you what's happening when the code is run.

Enable Time Trigger

As it stands, the script is actually done.

The issue is that it's not saving any time to have to open up Google Apps Script every time you want to archive a few emails.

To run this script automatically, click on "Triggers" with the time clock icon in the leftmost sidebar.

Then click "+ Add Trigger" and choose your settings. Mine were like this:

Choose which function to run on: archiveInbox
Choose which deployment should run: Head
Select event source: Time-driven
Select type of time based trigger: Day timer
Select time of day: Midnight to 1am
Failure notification settings: Notify me daily

If you changed the function name or you have multiple, they should show up in this dialog. You can customize the time to fit your needs.

Troubleshooting

When first running, Apps Script will tell you "Authorization required". You must review and allow permissions using Google Single Sign-On.

If you forgot to add the Gmail service, the execution log will tell you that "Execution started", but it will hit an error of "Script function not found".

If you haven't placed your script within a function, you will not be able to run it.

If the code appears to run correctly, but you do not notice the desired outcome in your Gmail inbox, double check that the project was created under the correct account.

Conclusion

I'm seeing the power of Google Apps Script as someone using many Google services. Perhaps we can start utilizing Google Sheets and Docs more?

3 Git Configuration Tips I Use Every Day

2023-06-15T00:00:00-07:00

Do you use Git literally every day? Me too.

I've been using the basics of Git for years, but only recently did I learn some new powerful tricks.

If you want to move past add, commit, and push, keep reading.

Praise for Boost Your Git DX by Adam Johnson
Make a dotfiles Repository
Use Shell Aliases for Git Commands
Automatically Create Upstream Branches
Summary

Praise for Boost Your Git DX by Adam Johnson

All of these tips have come from Adam Johnson's book, Boost Your Git DX. I'm only sharing (with permission) three tips from the book. He's written almost 300 more pages than what I outline here.

This book transformed my day-to-day operations. As a self-taught engineer, I have found myself starved for mentorship, and Boost Your Git DX has been my most influential mentor in 2023.

If you're looking to maximize the usefulness of Git past basic version control, you will love this book.

Make a `dotfiles` Repository

The dotfiles repository serves as a place to save your configuration files, allowing you to git clone it down to new computers or accounts. This is a huge plus for your developer experience if you work on multiple machines.

I can now use my optimized shell on Windows Subsystem for Linux for every version of Ubuntu I have installed, my home Ubuntu server, my powerhouse desktop computer, and my more portable laptop. You could even clone it into your web servers!

Check if you have files to save:

ls ~
ls ~/.config

If so, set up a dotfiles repository:

mkdir ~/dotfiles
cd ~/dotfiles
git init

Set up a config directory to store config files and move in your Git config:

mkdir config
mv ~/.config/git config/git

Removing the prepended . from .config means the folder won't be hidden by default.

Then, make a script to create a symlink for the config files back to their expected location inside the ~/.config directory:

touch setup.sh
chmod +x setup.sh
$EDITOR setup.sh

Inside setup.sh, add the following:

#!/bin/sh

# Enable shell script strictness
set -eu

# Show each command when running script
set -x

# Ensure config directory exists
mkdir -p ~/.config

# Link Git config if it doesn’t exist
[ ! -e ~/.config/git ] && ln -s "$PWD/config/git" ~/.config/git

If the target ~/.config/git file already exists, it will not be overwritten.

If it doesn't exist, the script will create a symbolic link or "symlink" with ln -s.

Run the script to link your config files from your dotfiles directory into your home directory:

./setup.sh

Check if it's working, commit the files to your repo, and push them up to your preferred Git host.

When you want to pull your config down to a new server

git clone 
./setup.sh

If you've made updates to your config from one machine, add + commit + push them to your dotfiles repo, then pull them down to your other machines.

Use Shell Aliases for Git Commands

Though I believe the dotfiles repo has saved me the most headache when working on multiple machines, these shell aliases are the change that I use the absolute most.

Aliases allow you to type abbreviations for longer commands. Some examples:

g for git
ga for git add
gc for git commit

To enable these, you can type them directly into a single shell session, or you can add a line to your shell's startup file (Bash: ~/.bashrc, Zsh: ~/.zshrc) to persist them across all interactive login shell sessions.

# ex: alias =
alias g=git
alias ga=git add
alias gc=git commit
alias gp=git push

Make sure to restart your shell to load the new configuration if you put it in your .bashrc or .zshrc file.

source ~/.zshrc
ga .
gc -m "added new config"
gp --set-upstream origin main

Automatically Create Upstream Branches

Once a project is more established, I prefer to work on new features in their own respective Git branches.

The problem is that when I push my changes to GitHub, the branch does not yet exist on the remote, so I have to type this wordy gp -u origin new-feature-1.

But what I really want to type is gp.

To automatically create an upstream branch and link it to your local branch, you can use the push.autoSetupRemote option.

git config --global push.autoSetupRemote true

This will add a new setting to your global Git configuration file:

[push]
    autoSetupRemote = true

This Git feature was added in Git v2.37, so you may check your git --version to see if you need to upgrade.

Summary

There you have it! Three Git tips I use all the time since discovering them in Adam Johnson's book, Boost Your Git DX.

May your contribution streak be long and light green 🙏.

Save a Django Query to the Database

2022-01-30T00:00:00-07:00

I found a StackOverflow question on saving a Django query to the database, but I was left super unclear on what to do. There’s all this talk about pickling and serialization that I’m sure makes a ton of sense to someone smarter than me.

The current answer is old, however, and in 2022, I found another solution: turning a dict into JSON.

Plus, I found a way to make it work with my htmx live search, allowing me to view the results of my query on the front end of my site (read: pretty) to double-check that things look okay before saving.

I’ll show you what I did below.

I believe pickling still works, so at the end I will show some more thoughts there.

models.py - example database structure

class Transaction(models.Model):
    id = models.CharField(max_length=24, primary_key=True)
    date = models.DateField(null=False)
    amount = models.IntegerField(null=False)
    info = models.CharField()
    account = models.ForiegnKey(Account, on_delete=models.SET_NULL, null=True)
    category = models.ForeignKey(Category, on_delete=models.SET_NULL, null=True, blank=False, default=None)

class Account(models.Model):
    name = models.CharField()
    email = models.EmailField()

class Category(models.Model):
    name = models.CharField(unique=True)

class Rule(models.Model):
    category = models.ForeignKey(Category, on_delete=models.SET_NULL, blank=False, null=True, default=None)
    criteria = models.JSONField(default=dict)  # this will hold our query

My models store financial transactions, the category the transaction fits into (e.g., salaried income, 1099 income, office expenses, labor expenses, etc…), and a rule to save a query to automatically categorize future transactions without having to remember the query every year when doing taxes.

I know, for example, that all my transactions with my consulting clients should be marked as 1099 income. So I want to create a rule for clients that will grab each monthly transaction and mark it as 1099 income.

Making the query the old-fashioned way

>>> from transactions.models import Category, Rule, Transaction
>>>
>>> client1_transactions = Transaction.objects.filter(account__name="Client One")
<QuerySet [<Transaction: Transaction object (1111111)>, <Transaction: Transaction object (1111112)>, <Transaction: Transaction object (1111113)...]>
>>> client1_transactions.count()
12

Twelve transactions, one for each month. Beautiful.

But how do we save this to the database?

Save query to database in JSONField

We now have Django 4.0 and a bunch of support for JSONField.

I’ve been able to grab the filtering values out of a form POST request, then add them in view logic.

urls.py

from transactions import views
app_name = "transactions"
urlpatterns = [
    path("categorize", views.categorize, name="categorize"),
    path("", views.list, name="list"),
]

transactions/list.html

<form action="{% url 'transactions:categorize' %}" method="POST">
  {% csrf_token %}
  <label for="info">Info field contains...label>
  <input id="info" type="text" name="info" />
  <label for="account">Account name contains...label>
  <input id="account" type="text" name="account" />
  <label for="category">New category should be...label>
  <input id="category" type="text" name="category" />
  <button type="submit">Make a Rulebutton>
form>

views.py

def categorize(request):
    # get POST data from our form
    info = request.POST.get("info", "")
    account = request.POST.get("account", "")
    category = request.POST.get("category", "")

    # set up query
    query = {}
    if info:
        query["info__icontains"] = info
    if account:
        query["account__name__icontains"] = account

    # update the database
    category_obj, _ = Category.objects.get_or_create(name=category)
    transactions = Transaction.objects.filter(**query).order_by("-date")
    Rule.objects.get_or_create(category=category_obj, criteria=query)
    transactions.update(category=category_obj)

    # render the template
    return render(
        request,
        "transactions/list.html",
        {
            "transactions": transactions.select_related("account"),
        },
    )

How to do it with htmx and live search

I am using htmx to get live search to ensure that my query is correct. This shows the filtered queryset to me as I type, then I can click “Create Rule” to save this query to the database.

transactions/list.html

<form
  hx-post="{% url 'transactions:perform_categorization' %}"
  hx-target="#tables"
  hx-indicator=".htmx-indicator"
>
  <input
    id="info"
    type="text"
    name="info"
    hx-post="{% url 'transactions:categorize' %}"
    hx-trigger="keyup changed delay:250ms"
    hx-indicator=".htmx-indicator"
    hx-target="#tables"
    placeholder="info"
  />
  <input
    id="account"
    type="text"
    name="account"
    hx-post="{% url 'transactions:categorize' %}"
    hx-trigger="keyup changed delay:250ms"
    hx-indicator=".htmx-indicator"
    hx-target="#tables"
    placeholder="account"
  />
  <span class="htmx-indicator">...span>
  <input
    id="category"
    type="text"
    name="category"
    placeholder="new category..."
  />
  <button type="submit">Make a Rulebutton>
form>

<div id="tables" class="stack">{% include "transactions/tables.html" %}div>

<script
  src="https://unpkg.com/htmx.org@1.6.1"
  integrity="sha384-tvG/2mnCFmGQzYC1Oh3qxQ7CkQ9kMzYjWZSNtrRZygHPDDqottzEJsqS4oUVodhW"
  crossorigin="anonymous"
>script>

<script>
  document.body.addEventListener("htmx:configRequest", (event) => {
    event.detail.headers["X-CSRFToken"] = "{{ csrf_token }}";
  });
script>

views.py

I’ve got one view tied to {% url 'transactions:categorize' %} that will display the list of transactions. The htmx live active search works there.

Then I have another view tied to {% url 'transactions:perform_categorization' %} which actually saves the query to the database by creating a new Rule. The code is the same as above.

How to do it with pickle?

So, I actually lied before. I have a little experience with pickle and I do like it, but I am not sure on how to save it to the database. My guess is that you’d then save the pickled string to a BinaryField.

Perhaps something like this:

>>> # imports
>>> import pickle  # standard library
>>> from transactions.models import Category, Rule, Transaction  # my own stuff
>>>
>>> # create the query
>>> qs_to_save = Transaction.objects.filter(account__name="Client 1")
>>> qs_to_save.count()
12
>>>
>>> # create the pickle
>>> saved_pickle = pickle.dumps(qs_to_save.query)
>>> type(saved_pickle)
<class 'bytes'>
>>>
>>> # save to database
>>> # make sure `criteria = models.BinaryField()` above in models.py
>>> # I'm unsure about this
>>> test_category, _ = Category.objects.get_or_create(name="Test Category")
>>> test_rule = Rule.objects.create(category=test_category, criteria=saved_pickle)
>>>
>>> # remake queryset at a later date
>>> new_qs = Transaction.objects.all()
>>> new_qs.query = pickle.loads(test_rule.criteria)
>>> new_qs.count()
12

Hope this helps!

Get With the Program: The Basics for Freelancing as a Coder

2021-12-19T00:00:00-07:00

Written by Ed Carter

If the pandemic has confirmed one thing it’s that the tech sector isn’t slowing down and its willful stewards, the computer programmers (or ‘coders’), continue to enjoy high demand. If you’re an aspiring coder, it’s now more viable than ever to go it alone and enjoy freedom and profit as a ‘codepreneur’ - here’s how you can get started.

The Coding Landscape

Before embarking on a solo career of any kind, it’s always important to first understand the scope and pitfalls of the business landscape. In the case of coding, that means finding the right target market and business types in requirement of your services. Some industries, such as finance, work in close coalition with tech and are already well saturated with programmers - finding a niche in these sectors (although lucrative) can be difficult. Finding the right market (or just a good client) means finding somewhere you’re valued and where you can be aptly compensated.

Fortunately, regardless of where you choose to work, research shows that computer and mathematical jobs continue to see increasing growth and at high salaries. Especially with a burgeoning ‘startup’ economy (with heavy reliance on tech), part-time or contract coders are an increasingly necessary expense, as companies seek to build websites, apps, and proprietary algorithms. It’s important to remember this when you first begin to set rates and working hours - unlike many freelance professionals, you may have more freedom to call the shots.

Finding Your First Client

Unless you are already well connected, the first step for a codepreneur is always to develop a marketing strategy. You’ll need to think carefully about your brand - what is your business ethos, what do you do well and where would you like to work? Answering these three questions will allow you to more easily set pricing, identify potential clients, and engage in a productive dialogue with businesses. If you haven’t already, it may be necessary to develop a portfolio of your work. If you’re starting from scratch, this inevitably will mean having to build websites or write code in exchange for a testimonial or creating speculative work either for yourself or for ghost clients.

Once you have the assets to market yourself, the next step is to develop leads. You can begin by reaching out directly to individuals via social media or you could take a more passive, paid approach and build PPC campaigns/marketing campaigns. Either Way, it’s important that you have completed work that you can reference and preferably some credentials/qualifications, which allude to your ability.

Conducting Business

Sometimes all you need is that first client. Once you’ve got your foot through the door and a job well done, your business can multiply itself via recommendation. It’s of the utmost importance, therefore, to keep your clients happy. This means clear, regular communication but, more crucially, good quality work at reasonable rates. If you can prove yourself easy to work with, convenient at short notice, and not overly expensive, you might also become irreplaceable.

Remember, before entering into any contract, ensure you’ve got your head around the basic freelance admin. That means understanding how to file your own taxes, how to manage invoices, and maintain spreadsheets. If you don’t know your way around these essential processes, not only will it appear unprofessional but it can slow down negotiations.

Life is good in the tech sector and, if you can find the right clients, for a talented codepreneur. If you can do the simple things well and keep yourself on task, there’s no reason you can’t have a successful, lucrative career in the years to come.

Start Coding Now is a blog and information resource for Coders and programming professionals. Learn from or spread the wealth of knowledge, at: www.startcodingnow.com

Image by Pexels

Get Started with Python

2021-11-01T00:00:00-07:00

If you’re looking to build your Python skills, there are a bunch of resources out there. Being self-taught, I’ve gone through quite a few of them!

Here’s a list of the most memorable stuff I’ve come across, as well as an explanation of a few basic concepts that most people tend to find confusing.

Books

A book or course is the best way to begin. Tutorials are great at some point, but when you need an overview of the entire language, look for something more comprehensive.

Learn Python the Hard Way by Zed Shaw

Of all the books and such that I read, Learn Python the Hard Way was my favorite intro to Python because it has a little bit of general programming sprinkled in.

Automate the Boring Stuff with Python by Al Sweigart

Automate the Boring Stuff with Python was the first Python book I went through and it’s also good (and free to read online!).

Video Courses

You can try your luck with YouTube which usually has something great, but Real Python traditionally has the best stuff on Python. They have a ton of great, free articles, but they also have a membership that will give you access to some video courses if you prefer to learn that way. Here are two that tackle some of the most important fundamentals of Python:

And then once you can figure out pip, packages, and virtual environments, your most powerful tools are just learning all the magic that’s out there.

Useful Packages

Python Standard Library

The Python Standard Library has a bunch of stuff available to help you manipulate strings, use different data structures, and more. Doug Hellman wrote a bunch of short articles on these packages called Python Module of the Week.

Third-Party Packages

You can get similar benefits out of learning powerful third party packages (those unassociated with Python proper). Here are some examples that you would just install with `pip install package-name`:

django - a framework for building web applications
requests - helps you make web requests
black - a formatter to give you a consistent, “pythonic” code style
borb - for working with PDFs
pillow - for working with images
ffmpeg - for working with video

Projects

Lastly, make sure you get your hands dirty! You can’t truly understand how to structure a project until you build one!

Learn Python the Hard Way and Automate the Boring Stuff with Python have a bunch of projects that get you going.

If you’re feeling ambitious, try building your own project or tinkering around with someone else’s to get going!

Cloning a Single Production Database Table from Heroku to Local Database with PostgreSQL in Docker Compose

2021-09-22T00:00:00-07:00

I have a few Django apps in a Django project. I used factory_boy to mock a bunch of the data, but I have a few hundred rows of accurate data in my production database. It made sense to work on a copy of that data as it contains no sensitive user information.

My project is hosted on Heroku with one of their free tier PostgreSQL databases. My local environment is a Docker Compose stack with images for Python, PostgreSQL, and Redis.

The trickiest part here, in my opinion, is finding out just how to get the backup to the PostgreSQL docker container.

Table of Contents
Backup the Production Database
Clone to Local Database
Conclusion

Backup the Production Database

There are two easy ways to do this on Heroku

Use the Heroku CLI
Use the Heroku web interface

The Heroku docs detail how to make a backup.

Use the Heroku CLI to Backup Database

So first, if you aren’t already using it, you’ll need to install the Heroku CLI and run heroku login.

Next, you’ll need to know your app’s name in Heroku. Mine is mastering-fitness, but you can get a list of all your Heroku apps right there in the terminal.

heroku apps

Before we make a backup, let’s get acquainted with the pg:backups command.

heroku pg:backups --help

Read through that briefly as it shows you your options.

To make a backup manually, I will run the following command. You should put your app name in place of mine.

heroku pg:backups:capture --app mastering-fitness

Next, download the file locally.

heroku pg:backups:download --app mastering-fitness

Note that this file will download to your active directory in the terminal. I am working out of VSCode so it went right into my root project folder. But we need to give it to our PostgreSQL Docker container…

Let’s talk about making a backup with the Heroku web interface next. If you’ve made your backup, feel free to skip to Cloning to Local Database.

Use the Heroku Web Interface to Backup Database

Find and click on your app.

Find and click on your app

Find and click on your Postgres database.

Find and click on your Postgres database

Click on the Durability tab.

Click on the Durability tab

Click on Create a Manual Backup.

Once it’s finished, you should see it right there on the page.

Download the backup and then you’re ready to clone it to your local table.

Cloning to Local Database

I ran into trouble when first trying this because my local database already had some data. I got a bunch of errors when trying to import the production data, but I didn’t want to mess with ALL the data, just a single table in the database.

Destroying Your Old Local Data

If you’re okay with destroying everything, you can use the --clean flag as Matt Segal noted in his article How to pull production data into your local Postgres database.

I have plenty of usable data in most of my apps, but one of them is a list of fitness exercises that I wanted to just nuke and replace. I used the Django admin to remove the five or so testing records I had locally. I am not sure if the --clean flag will delete that single table if you specify a table, so (1) be careful, and (2) if you know the truth, leave a comment below. I’d love to know.

The pg_restore Command

The basic command to import the backup into your local Postgres database is…

pg_restore --dbname DB_NAME --host localhost --port 5432 --username USERNAME --no-owner FILE_TO_IMPORT

But we need to run this in our Postgres Docker container. We might try something like this…

docker-compose exec db pg_restore --dbname postgres --host localhost --port 5432 --username postgres --no-owner latest.dump

…where

“db” is the name of our Docker Compose service for Postgres
“postgres” is the name we gave our database AND the user
“latest.dump” is the file we downloaded from Heroku

But the command won’t be able to find the file because it does not exist there. We need to move it to the Postgres volume where we’re persisting our database’s data.

Help Docker db Service Find Our Postgres Backup File

Here’s the docker-compose.yml file

version: "3.7"

services:
    web:
        build: ./app
        command: python manage.py runserver 0.0.0.0:8000
        volumes:
            - ./app/:/usr/src/app/
        ports:
            - 8000:8000
        env_file:
            - ./.env.dev
        depends_on:
            - db
    db:
        image: postgres:12-alpine
        volumes:
            - postgres_data:/var/lib/postgresql/data/
        environment:
            - POSTGRES_USER=postgres
            - POSTGRES_PASSWORD=secretsecretivegotasecret
            - POSTGRES_DB=postgres
    redis:
        restart: always
        image: redis:6.2.3
        expose:
            - "6379"

volumes:
    postgres_data:

We’ve got a volume called “postgres_data” that maps to the /var/lib/postgresql/data/ directory inside our Postgres container. But where is that on our local filesystem?

Let’s examine our Docker Volumes.

docker volume ls

This will show you the full name of that “postgres_data” volume, prepended with the project name. Mine is “lance-fitness-store_postgres_data”. Let’s inspect that volume.

docker volume inspect lance-fitness-store_postgres_data

In there we see a value for “Mountpoint”. This is what we’ve been looking for! We need to copy our database backup file “local.dump” over to that folder if we want the “db” service to have access to it. To do it in the terminal…

cp latest.dump /var/lib/docker/volumes/lance-fitness-store_postgres_data_dev/_data/latest.dump

Now, we need to point to the file in the volume’s folder in the container. Remember the file location we found in docker-compose.yml?

db:
    image: postgres:12-alpine
    volumes:
        - postgres_data:/var/lib/postgresql/data/

So inside the container my backup will be located at: /var/lib/postgresql/data/latest.dump

Importing Only One Table

We’re almost there!

We just need to add a --table flag to our pg_restore command.

You’ll have to know the name of your Django table. The Django style is appname_modelname, but you can override it if you’d like. I have an Exercise model in an app named exercises. My table is exercises_exercise.

So my final command becomes…

docker-compose exec db pg_restore --dbname store_dev --host localhost --port 5432 --username lance_dev --data-only --table exercises_exercise --no-owner /var/lib/postgresql/data/latest.dump

Conclusion

Hopefully that went without any hiccups! But just in case it did, I tried to outline the goal and thought process of each step here.

Best of luck!

Header database icon: database by Anton Outkine from the Noun Project

Getting Access to the Wide Internet in Your Hyper-V Virtual Machine

2021-04-26T00:00:00-07:00

I’m using Hyper-V on my workstation to run a Ubuntu Linux virtual machine.

I recently moved and changed my office setup. Previously I had a Cat 6 cable into a 2.5G Realtek Ethernet Adapter running into my router, plus a SFP+ DAC cable running into a 10G switch to connect to my local network server.

With the move, though, I wanted to simplify the setup. I’m no longer using a Cat 6 cable and 2.5G NIC to get Internet access. Now I’m only using the small 10 gigabit switch, the Mikrotik CRS305-1G-4S+IN, and that cable supplies my internal and external network needs.

I highly recommend this setup even if it’s just for a local connection. It’s really cool to saturate RAID’ed drive transfer speeds when moving my videos around. If you’re curious, here’s a longer post detailing how I did it all: 10 Gigabit Networking Between Home Server and Computer.

The problem is that removing that 2.5G network cable broke my VM’s internet access.

In this post, I’ll detail what happened and how to fix it.

Losing Internet Access from Hyper-V Virtual Machine

How does Hyper-V manage networks? What are “virtual switches”?

Hyper-V creates a virtual switch to connect your VM to your physical switch.

If you open up Hyper-V and right click on the server (the name of the Hyper-V host, a.k.a. the name of my workstation), you can click on Virtual Switch Manager… to open up some helpful settings.

My workstation has four virtual switches:

The first listed there is the Realtek 2.5G network card I created
The second listed is the default that Hyper-V created
The third listed is one tied to my WiFi network card that I created
The fourth listed is one tied to my 10G network card that I created

There are quite a few settings when making these virtual switches, but they are described well.

How to Create a New Virtual Switch in Hyper-V

When you click on New virtual network switch, the dialog asks what type it should be.

External switches give you access to the outside network
Internal switches connect your VMs on the same server and your host OS
Private switches connect only your VMs on the same server

If you want to get wide internet access, you need a new External switch.

The next screen allows you to pick the “Connection Type”. Since my 10G network adapter is what I want to use, I selected that.

Then, I also created one for my WiFi adapter in case my wired connection is ever down.

Add the New Virtual Switch to the Hyper-V Virtual Machine

Now that I have made the virtual switch, I need to tell the virtual machine that it’s there.

In the Hyper-V manager, right-click on your VM and select Settings….

Now, if you previously had a virtual switch there, it’s likely you still have the network adapter listed there. If you click on it, you can change the virtual switch.

After hitting Apply, you’ll likely need to re-authenticate into your VM. But let’s go in and see if we have access to the network now.

This little network icon in the top right of the screen gives me hope! Let’s open the Network settings now.

Beautiful! And if we test internet access in the web browser, it works.

Success!

Extra Credit: Adding the WiFi Switch to Our Hyper-V Virtual Machine

As a backup and as further practice, let’s add the new WiFi switch as well.

Back in our virtual machine’s settings, we want to Add Hardware. Select Network Adapter and click Add.

Then select the correct switch and hit Apply.

WiFi switch selected

And then let’s check our network settings in ubuntu again.

Awesome! If it looks like the network card is there, but you’re getting a “cable unplugged” message, just make sure your WiFi is turned on in your host machine.

Conclusion

That’s it! I’m excited to be up-and-running once again with my virtual machine.

Good luck!

Create Your Own YouTube Analytics Dashboard SUPER FAST with Google Apps Script

2021-04-20T00:00:00-07:00

In my day job as a personal trainer, I find myself doing a lot of repetitive tasks. When writing a training program, there’s SO MUCH clicking to format a Google Sheet well enough to make a professional product.

So I started playing around with macros and found it easy to augment a macro with Google Apps Script.

When recording a macro, the code is saved in the “Script Editor”.

Tools > Script editor will open the script editor.

The Google Apps Script Editor.

And I come to find out that this Script Editor can do a lot for us!

In this post, we’re going to make your own personal YouTube dashboard.

Table of Contents
Create a New Google Sheet
Find a YouTube Video ID
Write the Script
- Line By Line Code Analysis
Add the YouTube Service to the Script
Run the Script
Next Ideas

Create a New Google Sheet

We’ll need a place to store your YouTube data. Create a new sheet and give it three headings in the first columns:

Video ID
Title
Views

The first three cells have the headers “Video ID”, “Title”, and “Views”.

Find a YouTube Video ID

First, we’re going to need a video to analyze.

I went and grabbed one of my top performers: Elbow Pain During Push Ups. Let’s examine the link:

https://www.youtube.com/watch?v=2TPwtBl6_KQ

That value after the ?v= is the ID of the video.

https://www.youtube.com/watch?v=**2TPwtBl6_KQ**

So we’re going to copy that for our Google Sheet and paste it into cell A2.

The video ID has been pasted in cell A2.

Write the Script

The inspiration for this post came from another post by Wesley Chun on the Google Developers blog. It’s a few years old now, but the code mostly works. The user interface has changed a bit, though, so I’m going to walk your through it.

Open up the Script Editor from your Google Sheet: Tools > Script Editor.

And type in the following code:

function getVideoInfo() {
  let sheet = SpreadsheetApp.getActiveSheet();
  let vid = sheet.getRange("A2").getValue();
  let data = YouTube.Videos.list('snippet, statistics', {id: vid});
  let item = data.items[0];
  let info = [item.snippet.title, item.statistics.viewCount];
  sheet.getRange("B2:C2").setValues([info]);
}

Wesley does a great job walking through the code in the post, but I’ll do so here for completeness.

Line By Line Code Analysis

function getVideoInfo() {

Here we’re declaring the function we’re going to run.

let sheet = SpreadsheetApp.getActiveSheet();

This tells the script what sheet we’re working on and is tied to our current sheet

let vid = sheet.getRange("A2").getValue();

This selects the video ID we just pasted in. If we need it, we can use the vid variable.

let data = YouTube.Videos.list('snippet, statistics', {id: vid});

Here we’re using the YouTube Data API to grab info about our video.
We use the YouTube.Videos.list method.
- 'snippet, statistics' tells the API what we want it to give us.
- The snippet property contains the channelId, title, description, tags, and categoryId properties.
- The statistics property contains the viewCount, likeCount, dislikeCount, favoriteCount, and commentCount properties.
And we pass in the video ID from the vid variable we just set.

let item = data.items[0];

We’ve only asked for one video, so our response from the YouTube Data API has only given us one video in our array (at index 0).

let info = [item.snippet.title, item.statistics.viewCount];

Grab the title and view count from the JSON response data.

sheet.getRange("B2:C2").setValues([info]);

Select our B2:C2 range.
Add the two values from the info variable we just set.

Add the YouTube Service to the Script

This is where things start to look a little different from Wesley’s original article. Here’s how we do it in April 2021.

Click + in Services.

The plus button for adding a new service.

Scroll down and select the YouTube Data API.

The Add a Service dialog box. The YouTube Data API v3 contains a link to the documentation link and can be identified with “YouTube”.

Then click “Add” and you should see it in the left menu.

The YouTube service is showing.

Run the Script

Then we can “Run” the script.

The run button is towards the top of the window.

A dialog box will open asking for you to authenticate your Google Account.

You may also get a warning that executing the code from this sheet is not safe. Chrome likes to warn you by hiding the ability to go on and showing a blue “Back to Safety” button. I was able to bypass this by clicking “Advanced” and proceeding anyways.

Once you’ve done that, you might have to click “Run” one more time.

When it runs, the Execution Log should open at the bottom of the window.

Execution log shows execution started and completed.

And if all went well, you’ll see what I’ve pasted above.

Check the Google Sheet…

The video title has been placed in cell B2. The video view count has been placed in C2.

And there’s our data!

Next Ideas

Track your top ten videos and find their views.
Track your content marketing videos to see which perform best.
Test different methods of engagement to see which videos get the most comments, likes, and dislikes.
Send these stats to your Gmail.
Store a new file for every month with the highest performing videos of the month.
Grab data from a playlist you’ve created (perhaps a tutorial series?)
Retrieve the IDs for your most popular videos and get all the data you want about them.
Create visual tables and graphs inside your Google Sheet to better visualize your data.
Share your results with a friend to hold yourself accountable.

Any other ideas? How are you using Google Apps Script? Share in a comment below!

Using sorl-thumbnail with Redis on Heroku

2020-09-23T00:00:00-07:00

I recently added sorl-thumbnail to a project for creating smaller image files. I chose to configure with a Redis Key-Value Store, as I’ve heard Redis is super hip, but ran into trouble during deployment.

In this post, I’ll document how I set up Redis locally with Docker, then in production on Heroku.

Local Docker Setup

The logical place to start is with the sorl-thumbnail documentation.

Install the package

python3 -m pip install sorl-thumbnail

Add package to settings.py

# settings.py
INSTALLED_APPS += ['sorl.thumbnail']

Migrate database

python manage.py migrate

This should create a model called KVStore.

Add redis server

If we navigate to the documentation’s requirements page, we’ll see we need some sort of key-value store and image processing library.

As the title suggests, we’re going to use Redis. Let’s add that in our docker-compose.yml file.

# docker-compose.yml
version: "3.7"

services:
  web:
    build: ./django
    command: python /code/manage.py runserver 0.0.0.0:8000
    env_file:
      - ./.env.dev
    volumes:
      - ./django:/code
    ports:
      - 8000:8000
      - 443:443
    depends_on:
      - db
      - redis

  db:
    image: postgres
    volumes:
      - postgres_data:/var/lib/postgresql
    environment:
      - POSTGRES_HOST_AUTH_METHOD=trust
      - POSTGRES_USER=username
      - POSTGRES_PASS=supercomplexpasswordthatishardtocrack
      - POSTGRES_DBNAME=pg
    ports:
      - "5432:5432"

  redis:
    restart: always
    image: redis:latest
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/var/lib/redis

volumes:
  postgres_data:
  redis_data:

networks:
  default:

I’m assuming here that your Django web and db services and volumes are already configured. Use this tutorial to get started with Docker, Django, and Postgres.

First, we’re adding a redis service (redis:) and volume (redis_data). Then, we’re telling the web service that we need redis for things to work properly.

Add Redis client

python3 -m pip install redis

Here’s a link to this redis-py project on Github.

Add a Python image library

If you’ve gotten to this page, you probably already have Pillow installed, but you can use a few different image libraries.

Connect Redis server and client

Lastly, sorl-thumbnail has some settings to configure to link it up to your Redis key-value store.

# settings.py
THUMBNAIL_KVSTORE = 'sorl.thumbnail.kvstores.redis_kvstore.KVStore'

But if we try to docker-compose up with just this settings, we receive an error from Django:

**Connection error.** Error 99 connecting to localhost:6379. Cannot assign requested address.

The sorl-thumbnail package tries to use some sensible default values for connecting to the Redis server:

THUMBNAIL_REDIS_DB = 0
THUMBNAIL_REDIS_PASSWORD = ''
THUMBNAIL_REDIS_HOST = 'localhost'
THUMBNAIL_REDIS_PORT = 6379

The problem is that we called our Redis host redis, so sorl-thumbnail cannot connect.

I like to be explicit about variables at the expense of verbosity. Here’s all my sorl-thumbnail settings:

# settings.py

# sorl-thumbnail
THUMBNAIL_KVSTORE = 'sorl.thumbnail.kvstores.redis_kvstore.KVStore'
THUMBNAIL_REDIS_DB = 0  # the default
THUMBNAIL_REDIS_HOST = 'redis'
THUMBNAIL_REDIS_PORT = 6379  # the default

Then you should be all set locally!

Heroku Production Setup

Production is a little different, but the principles still apply:

Redis server
Redis client
Connect sorl-thumbnail to server

Sorl-thumbnail uses the redis-py package (the Redis client) to connect to the Redis server. At the time of writing, sorl-thumbnail’s reference documentation only showed our above settings as configurable.

But I went and looked at the code and found a nice little secret! We’ll use it later.

Provision heroku-redis add-on

The Heroku documentation shows you how to create a new heroku-redis add-on from the command line or Heroku dashboard.

heroku addons:create heroku-redis:hobby-dev -a your-app-name

The hobby-dev tier is free.

This will add a REDIS_URL configuration or environment variable to your Heroku app. It should look like “redis://…”.

But we didn’t use this variable to connect our project to Redis in our local environment. How will we use REDIS_URL?

That’s the secret I just mentioned at the end of the previous section!

Configure Django settings to use `REDIS_URL`

Here’s a link to the sorl-thumbnail code used to connect to Redis. It first searches for a THUMBNAIL_REDIS_URL environment variable, which contains the Redis username, password, host, and port.

We want to use the REDIS_URL variable set by heroku-redis because that URL might change in the future without us knowing, but will always be stored as REDIS_URL.

We need to replace our previous Django settings for sorl-thumbnail and Redis connections.

# sorl-thumbnail
THUMBNAIL_KVSTORE = 'sorl.thumbnail.kvstores.redis_kvstore.KVStore'
THUMBNAIL_REDIS_URL = os.environ.get('REDIS_URL')

Conclusion

And that should be all! Hopefully everything works for you on your first try and you feel like a genius.

I’ve opened an issue for including the THUMBNAIL_REDIS_URL in the Reference documentation.

Overall lesson today is to keep your eye out for strange anomalies like this. Sometimes there is a way to do what you need to do, but someone hasn’t gotten around to documenting it yet.

Maybe that someone could be you!

Authenticating Django PostgreSQL User in Multiple Docker Compose Environments

2020-09-01T00:00:00-07:00

I’ve been building a Django project template using Docker Compose, PostgreSQL, and Nginx. Docker is a definite weak point for me, so I used an article from Michael Herman to set it up: Dockerizing Django with Postgres, Gunicorn, and Nginx.

I made some additions to that tutorial – custom user model, re-organized config files – but I kept running into the same issue…

Django Can’t Connect to Postgres

When switching between development and production environments, Django could not connect to my postgres database:

docker-compose up -d --build
docker-compose exec web python manage.py migrate

django.db.utils.OperationalError: FATAL: password authentication failed for user ""

I had set up the same username for both the development and the production environment postgres services, but the passwords were different. For some reason, Docker Compose wasn’t re-configuring postgres with the new information when I switched containers.

Temporary Fix

I was able to avoid the problem with some help from the testdriven.io tutorial mentioned above:

docker-compose down -v
docker-compose -f docker-compose.prod.yml down -v

The -v flag brings down all volumes, i.e. my postgres database. Then, when bringing the containers back up, Docker Compose would say, “I don’t have any volume called postgres_data. I better make one!”

This would totally work, but I would lose any data that I’d put in postgres. In the long term, this will prove to be a headache.

More Permanent Fix

I decided to give each postgres database service it’s own name in my docker-compose.yml and docker-compose.prod.yml files. Instead of postgres_data, I used postgres_data_dev and postgres_data_prod.

But wait!

In this process, I received the following warning:

docker-compose up

WARNING: Service "db" is using volume "/var/lib/postgresql/data" from the previous container.

I had taken down the containers, but since I hadn’t removed them, changing docker-compose.yml tried to use the same part of my filesystem.

To fix, I removed the volumes:

docker-compose down -v

I also heard something about orphan containers. The warning in the command line will help you solve that.

Featured Image Photo Credit: Jeff Kubina, unmodified (CC BY-SA 2.0)

Caution About Faking Django Migrations

2020-07-22T00:00:00-07:00

I ran into a problem recently when trying to reconfigure my Django project’s database.

I had a DateTimeField on my model that I was updating to have null=True and blank=True.

There were no rows in this table. I was able to make the migrations, but then ran into this error when running python manage.py migrate.

TypeError: expected string or bytes-like object

What is This TypeError?

Honestly, after my research, I’m not totally sure. I believe Django was trying to put a NULL value in that DateTime table column, but since the migration hadn’t been run, NULL values were still not allowed.

I found something that made my error message go away… but also introduced a bug.

Django’s Migrations Table

Django’s migrations actually create a table in your database where database updates are tracked. Here’s a look at that table from one of my newer projects:

postgres=# \d django_migrations
                                     Table "public.django_migrations"
 Column  |           Type           | Collation | Nullable |                    Default
---------+--------------------------+-----------+----------+-----------------------------------------------
 id      | integer                  |           | not null | nextval('django_migrations_id_seq'::regclass)
 app     | character varying(255)   |           | not null |
 name    | character varying(255)   |           | not null |
 applied | timestamp with time zone |           | not null |
Indexes:
    "django_migrations_pkey" PRIMARY KEY, btree (id)

At least in my simplified view of Django migrations, there are two things that need to happen:

Django needs to know the migration happened so it doesn’t try to repeat actions
The database actually needs to be updated with SQL commands

This django_migrations table tells Django what has been done.

Faking Migrations

django-admin and manage.py have --fake flag you can add to a migrate command. This argument tells Django that a migration has happened, but DOES NOT RUN IT. That is, it creates a row in the django_migrations table, but does not run the SQL to change the database structure. Here’s a link to the docs.

And here’s another link to an article on Django Migrations from Real Python.

So I ran the command:

python manage.py migrate app_name --fake

And I saw no errors! Yay!

But I had a sneaking suspicion I should do more digging…

Checking on the Table

Since this was the first time I’ve run the migrate --fake command, and the docs clearly state, “without actually running the SQL to change your database schema,” I figured there was a good chance my DateTime field would still not accept null values.

Run psql to see the command line interface:

python manage.py dbshell

or, if you’re using Docker Compose on a Linux VM like me…

sudo docker-compose exec db_service_name psql -U postgres_username

When in psql, your prompt should change to something like postgres=#. Let’s inspect the table.

Get a list of all tables:

postgres=# \d

Then inspect the appropriate table. Mine was a Mail models inside an app named “mail”, so my table was named mail_mail.

postgres=# \d mail_mail
                                        Table "public.mail_mail"
    Column    |           Type           | Collation | Nullable |                Default
--------------+--------------------------+-----------+----------+---------------------------------------
 id           | integer                  |           | not null | nextval('mail_mail_id_seq'::regclass)
 to_email     | character varying(254)   |           | not null |
 subject      | character varying(78)    |           | not null |
 body_text    | text                     |           | not null |
 body_html    | text                     |           | not null |
 created      | timestamp with time zone |           | not null |
 sent         | timestamp with time zone |           | not null |
 content_id   | integer                  |           | not null |
 recipient_id | integer                  |           | not null |

The DateTime field I was trying to allow NULL values in is called “sent”, describing when a mail was sent. The NULL value represents that it has not been sent, otherwise a DateTime value represents when it was sent.

If you notice up there, the “Nullable” column in our psql output tells us that “sent” cannot be null.

If we don’t change this, we’ll run into trouble.

Revert the Table

So again, I’m confused as to why this would happen considering I didn’t have any data in the table. But no data means we can start from scratch.

First, unapply your migrations:

python manage.py migrate mail zero

Then reapply migrations:

python manage.py migrate mail

If you have trouble, unapply back to “zero” and delete the migrations files in your app_name/migrations/ directory. These files usually look like “0001_initial.py”. Then you can makemigrations again, then re-try migrate.

Check Database to Confirm Success

Alright, let’s go back to psql and see if our migration was actually applied this time.

postgres=# \d mail_mail
                                        Table "public.mail_mail"
    Column    |           Type           | Collation | Nullable |                Default
--------------+--------------------------+-----------+----------+---------------------------------------
 id           | integer                  |           | not null | nextval('mail_mail_id_seq'::regclass)
 to_email     | character varying(254)   |           | not null |
 subject      | character varying(78)    |           | not null |
 body_text    | text                     |           | not null |
 body_html    | text                     |           | not null |
 created      | timestamp with time zone |           | not null |
 content_id   | integer                  |           | not null |
 recipient_id | integer                  |           | not null |
 sent         | timestamp with time zone |           |          |

Hey! The “sent” column is updated (as evidenced by the absence of “not null”).

Conclusion

Hopefully you’ve taken in the word of caution about using --fake in your Django migrations. And hey, maybe you even learned a few ways to use psql in your project!

How to Use psql in Docker Compose

2020-07-15T00:00:00-07:00

As a Django newbie, I’ve found it easy to consider the database stuff “magic” and just leave it alone.

I’m starting to realize, however reluctantly, that it’s super important to know how databases work!

After seeing a software engineer YouTuber talk through a few of his database tables with some simple commands, I decided that would be a helpful skill.

So I ran psql and then… obviously it didn’t work because that would be too easy.

In this post, I’ll talk about how to use psql to view your PostgreSQL database from your Django project.

The Docker Problem

Docker has been great to me, but it’s also been a huge headache. There’s just so much going on behind the scenes; it makes reality difficult to comprehend. Layer that on top of learning Django and you’ve got a clear path to overwhelm.

For context, I can attribute my knowledge of Docker and Docker Compose to Will Vincent from his book “Django for Professionals”. That book really helped getting going, but deviating from that path is definitely challenging.

How to Run the Django Server Locally

Here’s a quick review of my local setup.

Dockerfile

# pull base image
FROM python:3.8

# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# set work directory
WORKDIR /code

# install dependenceies
COPY Pipfile Pipfile.lock /code/
RUN pip install pipenv && pipenv install --system

# copy project
COPY . /code/

docker-compose.yml

version: "3.7"

services:
  web:
    build: .
    command: python /code/manage.py runserver 0.0.0.0:8000
    env_file:
      - .env
    volumes:
      - .:/code
    ports:
      - 8000:8000
    depends_on:
      - db
  db:
    image: postgres:12
    volumes:
      - postgres_data:/var/lib/postgresql/data
    environment:
      - "POSTGRES_HOST_AUTH_METHOD=trust"
    env_file:
      - .env
    ports:
      - 5432:5432

volumes:
  postgres_data:

Then to run the local Django server and local PostgreSQL database:

sudo docker-compose up -d --build

Accessing the Python Container

When running management commands with Docker Compose, we have to specify which service we want to run the command.

I have two services: web and db.

When I need to use Django’s manage.py, I have to prepend this command:

sudo docker-compose exec web...

So, for example, when I want to migrate my users app migrations, it looks like this:

sudo docker-compose exec web python manage.py migrate users

Python Container or Database Container?

So there are two ways (that I know) to access the psql command.

python manage.py dbshell
psql

python manage.py dbshell

As specified in the Django docs:

dbshell

Runs the command-line client for the database engine specified in your ENGINE setting, with the connection parameters specified in your USER, PASSWORD, etc., settings.

For PostgreSQL, this runs the psql command-line client.
For MySQL, this runs the mysql command-line client.
For SQLite, this runs the sqlite3 command-line client.
For Oracle, this runs the sqlplus command-line client.

This sounds like the way to go, so let’s run it.

sudo docker-compose exec web python manage.py dbshell

And then we get an error:

CommandError: You appear not to have the 'psql' program installed or on your path.

WHAT!? How is that possible?

Our PostgreSQL service “db” surely has psql installed, but that’s a different container. Instead of trying to install psql inside our “web” container, let’s try running psql directly from the “db” service.

psql

To run, we’ll have to specify the “db” service:

sudo docker-compose exec db psql

Annnnnnd we get another error:

psql: error: could not connect to server: FATAL:  role "root" does not exist

This error is telling us that psql is, in fact, available. Confirm by checking the version.

sudo docker-compose exec db psql --version

Now the error told us that “role ‘root’ does not exist”. The default command with no arguments is trying to log in as root. But if we check out the .env file, we can see we specified different values for the PostgreSQL database name, username, and password:

POSTGRES_DB=postgres
POSTGRES_USER=postgres
POSTGRES_PASSWORD=supersecretpassword
POSTGRES_HOST=db
POSTGRES_PORT=5432

Well hey then, we don’t NEED to connect from “root”. What if we specify the username with the -U flag?

sudo docker-compose exec db psql -U postgres

Then we get a new prompt:

postgres=#

This is the psql prompt.

If you’re still getting an error:

psql: error: FATAL:  database "postgres" does not exist

It’s because psql is trying to login to the database “postgres” as the user “postgres”, but your database and user are different. You can specify them both with the -d and -U flags:

sudo docker-compose exec db psql -d postgres_db -U postgres_username

Next Steps

Hey congrats! Now you can use psql!

Or should I say, “Now you have to learn psql commands!”

Here’s a few to get your started:

List all databases: \l
View all tables in the current database: \d
Describe a table: \d table_name
List all available users: \du
Quit psql: \q

Giving Django Project a “Two Scoops” Configuration

2020-07-01T00:00:00-07:00

Years back I bought a physical copy of Two Scoops of Django 1.11 after seeing it highly recommended in a book called Hello Web App by Tracy Osborn. Little did I know it was way over my head at the time.

With the release of Two Scoops of Django 3.X, and with a lot of recent practice, the book is now only slightly over my head, so I’ve been trying to implement more of the practices recommended in the book.

I’ve also been continuing to use Docker and Docker Compose after learning the basics in Django for Professionals by Will Vincent.

Fixing Problems?

Early on in Two Scoops, the authors discuss how the standard project layout from django-admin startproject leaves a lot to be desired.

I definitely have NOT experienced that pain because I don’t have the skills to make any complex projects. I do, however, see the value in getting comfortable with a new layout. It teaches you how some of the module inheritance in Django works. Python import statements are one of the most challenging things for me, personally, so this is a good way to train your coding skills.

Let’s talk about two different project layouts.

Standard layout

Here’s the base layout you get from running django-admin startproject :

django-admin startproject basic_startproject .
tree.
.
├── basic_startproject/
│   ├── asgi.py
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
└── manage.py

Two Scoops layout

Here’s the directory layout recommended in Two Scoops:


├── 
└──

Here’s what’s stored in the directory


├── 
├── asgi.py
├── urls.py
└── wsgi.py

The folder holds onto your multiple settings files. This is another notable difference. See the Two Scoops book for more info on how to set that up; I found it helpful. Their guidance left me with the following project structure:

.
├── config/
│   ├── asgi.py
│   ├── settings/
│   │   ├── base.py
│   │   ├── __init__.py
│   │   ├── local.py
│   │   └── production.py
│   ├── urls.py
│   └── wsgi.py
├── /
│   └── /
│       └── __init__.py
├── docker-compose.yml
├── Dockerfile
├── docs/
├── manage.py
├── Pipfile
└── Pipfile.lock

Note that I’m using pipenv, Docker, and docker-compose as well.

Moving files breaks things

Now it’s easy enough to move files around, but if you move things around, import statements break.

Where to put docker-compose.yml

I decided to place the Dockerfile and docker-compose.yml alongside the manage.py file. This meant I did not need to change these files from a familiar setup:

# Dockerfile
# pull base image
FROM python:3.8

# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# set work directory
WORKDIR /code

# install dependenceies
COPY Pipfile Pipfile.lock /code/
RUN pip install pipenv && pipenv install --system

# copy project
COPY . /code/

# docker-compose.yml
version: "3.7"

services:
  web:
    build: .
    command: python /code/manage.py runserver 0.0.0.0:8000
    env_file:
      - .env
    volumes:
      - .:/code
    ports:
      - 8000:8000
    depends_on:
      - db
  db:
    image: postgres:12
    volumes:
      - postgres_data:/var/lib/postgresql/data
    environment:
      - "POSTGRES_HOST_AUTH_METHOD=trust"

volumes:
  postgres_data:

Finding the project app

The first major problem was that I ran into a ModuleNotFoundError when starting the development server. The module with the name of my Django project could not be found.

I found a nice Stack Overflow question addressing this. manage.py needs to know where to look for the apps. Since we moved things around, we’ll have to tell the system where to find that Django project folder.

In my base settings file, base.py:

# config/settings/base.py
import sys
from pathlib import Path

# Project root directory
BASE_DIR = Path(__file__).resolve().parent.parent.parent
sys.path.append(str(BASE_DIR / ''))  # replace

Finding `ROOT_URLCONF`

Running the development server again left me with this error: ModuleNotFoundError: No module named '.urls'

Now is a good time to compare our directories again. With the standard django-admin startproject command, our settings.py file is in this relative path: //settings.py. The standard ROOT_URLCONF value is: '.urls'. This works because the project urls.py file is in the same directory as the settings.py file.

With our new structure, we’ve separated these files and put them in a new folder named config/, so we need to update this setting.

.
├── config/
│   ├── settings/
│   │   ├── base.py
│   │   ├── __init__.py
│   │   ├── local.py
│   │   └── production.py
│   └── urls.py
└── /

# config/settings/base.py
ROOT_URLCONF = 'config.urls'

Finding `WSGI_APPLICATION`

Same goes for finding the WSGI application.

# config/settings/base.py
WSGI_APPLICATION = 'config.wsgi.application'

Success!

It seems simple, but it took me an entire morning to figure this out! Hopefully it speeds up your progress.

Be sure to check out Two Scoops of Django 3.X for more best practices in Django development.

Fancy New Windows Terminal

2020-06-07T00:00:00-07:00

I have a buddy who recommended switching to z shell as my terminal. He showed me some pictures and it looks nice and pretty, so I started researching.

Most results suggested installing Cygwin and then using the Cygwin terminal to install z shell. I went and did all that, it seems cool, but when I tried that apt-cyg install zsh, I received an error that said the certificate had expired. It wouldn’t install! Let’s try to find another solution.

Enter Windows Subsystem for Linux

I’ve been wary about trying this. It seems pretty new still, but the install directly from the Windows store was SO EASY. I have high hopes.

Just enable Windows Subsystem for Linux, find your Linux distro and install. I went with Ubuntu because I’m familiar with it.

I found this article from howtogeek.com to be really helpful: “How to Use Zsh (or Another Shell) in Windows 10”.

Install oh-my-zsh!

As a total newb, I don’t even know what kind of upgrades I could expect to make to my shell. Here’s an article on becoming a command line power user. You’ll walk through the new shell, theme, and adding customization.

Install PowerLevel10k

Here’s a Z shell theme recommended to me. I was on board as soon as I saw the pretty colors, but I’ll probably stay forever due to the Dragon Ball Z nostalgia.

Check out the manual there as there’s plenty of things you can do. I would at least install the fonts they recommend and see how you like them.

Annoying Windows Alert Sound

I found my Windows was making that terrible du-da-duuunn sound every time I double tabbed in the new shell. I figured I probably don’t want the terminal to play sounds, so I just opened the “volume mixer” in Windows and muted the console window.

Wrapping Up and Next Steps

Turns out there’s a lot of customization available. Definitely look into different plugins and try using the ones you think you’ll be using a lot. And I encourage you to get creative with your aliases as well.

Lack of Volume Persistence in kartoza/postgis Docker Container

2020-06-06T00:00:00-07:00

I have been working on a Django project that needs maps integration. I looked into using GeoDjango, but needed to upgrade my current Postgresql database to add PostGIS.

This was totally outside what I’ve done before. Here’s how I got it to work.

Starting Point

My knowledge of Docker comes entirely from Will Vincent’s Django for Professionals book. In there, he builds from the official postgres Docker image within a docker-compose.yml file.

version: "3.7"

services:
  web:
    build: .
    command: python /code/manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/code
    ports:
      - 8000:8000
    depends_on:
      - db
  db:
    image: postgres:11
    volumes:
      - postgres_data:/var/lib/postgresql/data/
    environment:
      - "POSTGRES_HOST_AUTH_METHOD=trust"

volumes:
  postgres_data:

Learning to Use GeoDjango

Luckily, I found a great tutorial over at Real Python to introduce myself to GeoDjango: “Make a Location-Based Web App With Django and GeoDjango” by Ahmed Bouchefra. If you’re totally new to GeoDjango, I highly recommend this article.

Ahmed even uses Docker in the article, so super win if you’re coming from Vincent’s book like me. PostGIS is an extension of PostgreSQL, so it’s kind of like a standard postgres database you might use for your Django project, but adds support for geographic objects.

In the article, we use a new PostgreSQL image to build our database: the kartoza/postgis image. If you check out the tags, you’ll see some have TWO sets of numbers. These are versions for PostgreSQL and PostGIS. Ex: kartoza/postgis:11.5-2.5 uses PostgreSQL 11.5 and PostGIS 2.5.

We spin up this database with the following command in the terminal:

$ docker run --name=postgis -d -e POSTGRES_USER=user001 -e POSTGRES_PASS=123456789 -e POSTGRES_DBNAME=gis -p 5432:5432 kartoza/postgis:9.6-2.4

What’s all this mean?

docker run means let’s run a Docker command
--name is the container’s name
-d means detach this container from the terminal so I don’t have to open another one
-e are environment variables
-p publishes a container’s port to the host
and the last kartoza/postgis:9.6-2.4 is the name of the image from Docker Hub you want to use to build the container

Once that’s running, the rest of the stuff eventually works and it’s great! All is well in the world.

The tutorial, however, doesn’t show me how to use it with my docker-compose.yml file. So let’s dive into that.

Using docker-compose.yml

To get this part going, I started with the PostgreSQL image we made in Django for Professionals. The file we had before becomes…

version: "3.7"

services:
  web:
    build: .
    command: python /code/manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/code
    ports:
      - 8000:8000
    depends_on:
      - db
  db:
    # NOTE the new image
    image: kartoza/postgis:11.5-2.5
    volumes:
      - postgres_data:/var/lib/postgresql/data/
    environment:
      - "POSTGRES_HOST_AUTH_METHOD=trust"
      # NOTE the new environment variables
      - POSTGRES_USER=user001
      - POSTGRES_PASS=123456789
      - POSTGRES_DBNAME=gis

volumes:
  postgres_data:

Bring it all up with docker-compose up -d --build and things should work! Everything appears to be fine until you bring the containers down and back up again.

docker-compose down

docker-compose up -d --build

Could not connect to host in Firefox

Our docker logs show us an error message Django is trying to tell us:

django.db.utils.OperationalError: could not connect to server: Connection refused

Is the server running on host "db" (192.168.96.2) and accepting TCP/IP connections on port 5432?

I had previously enabled python manage.py runsslserver so that I could have an SSL connection to work with some public APIs. When I ran the Django server with runsslserver, I get the beautiful Django DEBUG screen:

Programming error at /: django.site does not exist

Note that the exception is in the “django/db/backends/utils.py” file. It’s a database problem. The database is gone! We just need to migrate it.

python manage.py migrate

docker-compose exec web python manage.py migrate (for our docker-compose Django service named “web”.

The terminal shows a ton of green “OK” notifications as it applies your database migrations. Check the site now and the error should be gone.

But now the main point of the article. Let’s bring the containers down and then up again.

docker-compose down

docker-compose up -d --build

And that error is still there!

Retain the Local kartoza/postgis Database

I started digging into why the database wouldn’t stick around. It worked when I used the official postgres image, but not with this new one.

If you look at the Docker Hub page for the postgres image, they say:

The default [database file location] is /var/lib/postgresql/data.

That’s already where we store it! But what does the kartoza/postgis image say?

Docker volumes can be used to persist your data.

mkdir -p ~/postgres_data

docker run -d -v $HOME/postgres_data:/var/lib/postgresql kartoza/postgis

Ahhhhh, different location! Let’s try that out. Updated docker-compose.yml file:

version: "3.7"

services:
  web:
    build: .
    command: python /code/manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/code
    ports:
      - 8000:8000
    depends_on:
      - db
  db:
    image: kartoza/postgis:11.5-2.5
    volumes:
      # NOTE the new storage location
      - postgres_data:/var/lib/postgresql
    environment:
      - "POSTGRES_HOST_AUTH_METHOD=trust"
      # NOTE the new environment variables
      # NOTE use your correct credentials
      - POSTGRES_USER=user001
      - POSTGRES_PASS=123456789
      - POSTGRES_DBNAME=gis

volumes:
  postgres_data:

Migrate the database, bring the containers down, then back up.

If you’re lucky, it worked!

Conclusion

This whole scenario is a great example of how touchy coding can be. You need to specify the right folder. If you can’t quite appreciate that, start tinkering around with Linux and file permissions. HUGE PAIN. But valuable experience, too.

Oops! Removing Secrets from Django Project in Docker

2020-04-22T00:00:00-07:00

So you’ve been frantic as you try to fix a bug. You get lost in the work and totally forget where you are. All of a sudden… SUCCESS! It works!

Amazing.

But oh no, you just put your secrets into a tracked file. What to do, what to do…

Project Overview

So I’ve been saying “you”, but I really mean “me”. Here’s a short summary of my project, then we’ll get into how I fixed it.

Web app coded in Python
Using the Django web framework
Using Docker with a Dockerfile and docker-compose.yml for containerization
Deployed to Heroku
Had a problem serving static files on Heroku and not locally, so I implemented an Amazon S3 bucket for remote file storage, which requires secrets!

Up until this point, I’ve tracked my docker-compose.yml file so that if I want to work on this project on another computer, it will be easy to clone and get going. But I’m hosting this code on GitHub and may share it some day, so I don’t want secrets available.

Fixing Django’s `SECRET_KEY`

My docker-compose.yml file has my Django SECRET_KEY inside it. When prepping to deploy to Heroku, all I did was generate a new key and place that in my Heroku app as a config var.

This means that the public version of my site has a hidden SECRET_KEY.

Beautiful.

But I didn’t want to try to generate a whole new set of credentials for my AWS S3 bucket because this is really my first time using it. I have been debugging this static files problem for over two weeks now and just want it to work. So instead of messing around with more credentials, let’s find another solution.

Serving Django’s Static Files from AWS S3 Bucket

To solve my static files problems, I went looking for a mentor. Matthew Freire of JustDjango (site, YouTube) suggested that Heroku won’t let me serve my files from the same server as my Django site and that I should use an S3 bucket.

I found a tutorial from testdriven.io that was fairly recent and uses Docker, and that was extremely helpful.

I needed to commit and push my changes to Heroku to see if this change would work. When it did, then I realized I needed to remove those keys. Oops!

Fixing AWS Keys

After realizing my keys were in my settings.py file, I set up environment variables and put the keys in my docker-compose.yml file, which is where I’ve stored all my environment variables.

But then I realized I’m tracking docker-compose.yml, too! So I went looking for a new solution.

I found a concise article from David Walsh on using ``docker-compose.override.yml`. <https://davidwalsh.name/docker-compose-override>`__ When building your Docker image with docker-compose.yml, it will take whatever’s in your docker-compose.override.yml file and use it to replace old things and add new things.

Don’t forget to add docker-compose.override.yml to .gitignore.

Then check to make sure it works. Mine does!

What Changed in the Previous Commit?

For my particular scenario, my previous commit was the one that had the secrets in it. So I want to amend that one.

But let’s make sure we’re not missing anything.

git reflog

Outputs hashes of previous commits.

git diff

This will walk you through the differences between the two commits. I used the command git diff a08db22 04d80e7, which says, “What changed from commit a08db22 to 04d80e7?” Keep hitting enter to scroll.

I first see that my Pipfile and Pipfile.lock have changed. This is good because I wanted to install boto3 and django-storages to get my S3 bucket working.

Next change I see is in settings.py, where we’ve added 'storages', to my INSTALLED_APPS. All good here.

Then, further down, we see the problem. My AWS key variables are pasted in there! Make a note that we need to make sure that we want to amend that change in settings.py.

There are just a few other changes, but nothing that needs to be hidden.

Amend Previous Commit

So I had four files to change:

docker-compose.yml
docker-compose.override.yml
.gitignore
settings.py

But since we ignored the new docker-compose.override.yml, then only three files should show up as changed when we ask git to check.

git status

Perfect, our three files show up. Now let’s check what is different since the last commit.

git diff

Nice! Environment variables! Repeatedly hit ENTER to make sure everything is good.

git add -A

Stage these modified files.

git commit --amend

Amend the previous commit. A screen will pop up with the message you gave the commit. Adjust as needed, save, and exit.

Success!

One last note: maybe push your changes to your remote repository :)

git push origin master

All set!

10 Gigabit Networking Between Home Server and Computer

2020-01-13T00:00:00-07:00

In my other life, I film a lot of videos on fitness.

After a recent lecture a few friends and I did for charity, I found out… I’m out of storage space!

I built a server for my home to not only store the videos I film, but also as a media server to store and watch some of the bluray and DVD content I’ve purchased over the years.

Problem is I have about 5TB of my own storage. And I have a few computers lying around that can help with the job of transcoding my ripped bluray and DVD media.

But if I do that on multiple devices, I’ll still have to transfer them back to the server.

So I decided to set up 10 gigabit ethernet between my main computer and my server.

Bits vs Bytes Networking often describes speed in bits. In our daily computing, we usually use bytes. One byte is comprised of eight bits.

Ex 1) 1 gigabit networking means 125 megabytes per second Ex 2) 10 gigabit networking means 1.25 gigabytes per second

Inventory - what you’ll need to get this going

I will outline what I bought exactly at the end of this section, but here’s an overview so you can pick and choose what you need.

At least two computers
Two networking interfaces
A 10 gigabit switch
The appropriate cables (one for each computer)

Network card

My server is in a huge tower (Fractal Design Define XL R2) with a huge motherboard (Supermicro X9DRH-7F). There is a variant of this motherboard with onboard 10 gigabit ethernet, but the cost was double at the time of purchase, so I opted for this cheaper one.

To get 10 gigabit networking, I will need a 10 gigabit network adapter. I can use one of my six PCIe 3.0 x8 slots, with a max transfer rate of about 8GBps, is plenty fast enough to take a 10 gigabit network adapter card. There could even be two or four 10GbE interfaces on that card and it would still work, since 10 Gb = 1.25 GB and 40 Gb = 5 GB.

Now there are two major types of connectors used for 10 gigabit ethernet: the standard-looking RJ45, and the intimidating SFP+.

Connector type

There are four major methods for home use

Direct Attach Copper (DAC) cable – cheap but only works for short distances
LC connector for multi-mode fiber optic cable – better for longer distances across the home
LC connector for single-mode fiber optics – good for REALLY long distances, but doesn’t seem to make sense for home use
RJ45 – only really makes sense if you already have at least Cat 6a ethernet cables

I found this video from Craft Computing on YouTube very helpful here. More on this when we talk about the 10 gigabit switch in a few sections.

My server is sitting right underneath my work desk, so I didn’t need long cables. I was going to pick an RJ45 connector until I discovered…

Thunderbolt 3 to 10 Gigabit Adapter

It would have been easy to just buy two of those network cards above and call it day, but my main computer is actually a Lenovo X1 Extreme Gen 1 laptop. I can’t just plug in a traditional PCIe network card.

This laptop is very capable (read: expensive) and comes with Thunderbolt 3, which is EVEN FASTER than 10 GbE, running 40 Gbps through the proper cables. But I need an adapter to get the 10 GbE into the TB 3 port on my laptop.

QNAP makes just such an adapter – one for SFP+ connections, and one for RJ45 connections. The SFP+ adapter was cheaper, so I chose that.

10 Gigabit Switch

I had intended to just plug the two computers together and call it a day, but I decided a much more elegant solution would be to plug each machine into a switch and let the switch facilitate the connections.

The aforementioned YouTube video from Craft Computing detailed a super affordable product: the Mikrotik CRS305-1G-4S+IN. It comes with a power supply, is nearly silent when running, and has one gigabit port to hook up to the main router (if desired) and four 10 gigabit ports to use as a switch.

Summary Purchase List

This is assuming your have a laptop with Thunderbolt 3 and a tower server with open PCIe lanes.

Mikrotik CRS305-1G-4S+IN - 10 gigabit switch
Mellanox MNPA19-XTR - 10 gigabit SFP+ network card for server (2-port and 4-port variants)
QNAP QNA-T310G1S - 10 gigabit to Thunderbolt 3 network adapter for laptop
(2x) DAC cable

Getting It Working - NOT quite plug and play

One thing I need to make perfectly clear: I am not a networking guru. I just like to tinker around and learn new stuff. So when that Craft Computing video said this solution was “plug and play”, it left me a little confused when I plugged everything in and it wasn’t working.

I just needed to realize how my computers were connecting.

Samba Shares

When I first got my server running, I set up a Samba SMB share to my main storage folder so that I had a new place to move my old files.

That worked pretty quickly. I was happy with that.

So I plugged in all my 10 gigabit hardware and tried transferring files…

I was able to transfer files, but only at a dismal 40-50 MBps. That was not even saturating the write speed on my HDD, which I expected to be at MINIMUM 80 MBps.

The problem was that after setting up my 10 gigabit connections, I kept using that plain ol’ 1 gigabit connection I had already been using. Notice what is highlighted on the left side of the following two screenshots.

Both of those network locations appear the same. When I click one of them in the sidebar on my Windows 10 laptop, it shows the same root folders. Navigating through shows the same folders as well.

The difference is that when connecting, I typed in a different IP address.

Setting Up Windows Network Share

First, we need to find addresses. The Mikrotik CRS305-1G-4S+IN comes configured with an IP of 192.168.88.1. I decided to move my Mellanox 10 GbE server card and my QNAP QNA-T310G1S 10 GbE laptop adapter to the 192.168.88.xxx subnet.

Configure IP on Windows Adapter

Right click your networking icon in your Windows task tray and select “Open Network & Internet settings”.

Select “Change Adapter Options”.

Right-click the 10 gigabit network adapter and select “Properties”.

Make sure “Internet Protocol Version 4 (TCP/IPv4)” is enabled and double click it.

Configure it manually to use the same sub-network as your switch. If you bought the Mikrotik one and didn’t change it, the subnet is 192.168.88.xxx. You just have to fill in the last number to give this network adapter it’s own location.

Since I’m not connecting to the internet, I just left the default gateway and DNS server addresses blank because I don’t need them. My Internet traffic is routed through my Gigabit ethernet or my WiFI adapter.

Accept the changes and move to your server.

Configure IP on Ubuntu

Go to network settings.

Click the settings gear icon for your 10 gigabit network card.

In IPv4, add an IP address on the same sub-network you selected earlier. If you went with the default for the Mikrotik switch, then the first three numbers have to be 192.168.88.

Set up a Samba Share on Linux

Now we need to have a folder that we want to share. In my case, I have four 10 TB HDDs that I have mirrored and pooled together. So two serve as a RAID 1 redundant mirror for the other two, which are pooled together to be displayed on my file browser as one folder. It’s pretty neat and ensures I’ll always have my stuff even if a drive fails.

I have mounted this pooled virtual drive in my /mnt folder. The easiest and most visual way to share this folder on my local network is to navigate to it in my Ubuntu Desktop file browser. The default file browser is “Nautilus”.

Right-click the folder and select “Local Network Share”.

Configure your sharing settings. I kept the name of the folder and allowed more access to it so that people on my local network dont need my help getting logged into the folder.

Then hit “Create Share”.

If you run into an error that says Nautilus needs to add some permissions to your folder in order to share it, try to add them automatically (if that is indeed what you want to do).

If you’re hit with another error that says Could not change the permissions of folder, then we need superuser privileges to get things straightened out.

To confirm this is actually the problem, right-click on your desired share folder and select “Properties” then “Permissions”. Here’s a screenshot from another folder of mine that I want to share.

This is showing us that the folder is owned by the user “root”. Since I am logged in as “lance”, I cannot change root’s folder. We’ll need to go to the terminal and tell it “I am root”.

Hit Ctrl + Alt + T to open up the Linux Terminal.

We need superuser privileges to change the permissions on this folder. And we need to know the path of the folder. Mine is /mnt/Intel660P. First we’ll change the owner of the directory and all of its contents.

$ sudo chown --recursive username:username /path/to/folder

sudo gives us superuser privileges (telling the terminal “I am root”)

chown means “change the owner”

--recursive means “do this on all the sub-directories and contained files”

username:username represents the user and the group that owns this folder. Each user has a group that is the same as their username, and that’s what I want to set my folder to.

/path/to/folder designates the file or folder we want to change

Putting it all together, here is what I wrote to change the owner of my folder

$ sudo chown --recursive lance:lance /mnt/Intel660P

And we can check to see if it worked by refreshing our Nautilus file viewing and checking the Properties > Permissions again.

Nice! Now, I need others to be able to access the folder, so we’ll need to change the permissions of this folder and the enclosed files.

Easiest way on Ubuntu desktop is to change the “access” for “group” and for “others”.

Then click on “Change Permissions for Enclosed Files…” and change to the settings you want.

Check out the chmod command if you want to do it in the terminal.

You can get a lot fancier with permissions if you set up samba shares. Click here for the basics of setting up samba shares and click here to get a more advanced samba configuration.

Create New Network Share on Windows

NOW! This step was my big problem. I already had a folder on my Linux server marked as a share. And I already had a link to that share set up on my Windows computer. This gave my Windows computer access to the storage on my Linux server.

The problem was that this link was made BEFORE installing the 10 gigabit network cards. So it was using plain ol’ gigabit (which is honestly fine for my home use, but doesn’t seem as cool so I “needed” to upgrade).

Let’s create a new network share on Windows that now uses this new connection.

On your Windows computer, hit WIN + E to open up Windows Explorer.

Right-click “This PC” in the left sidebar and select “Add a Network Location”. Note: you can also select “Map network drive…” if you want it to be stored as a letter.

This will open the Add Network Location Wizard.

Hit Next to continue.

Select “Choose a custom network location” and select Next.

Now is time to tell Windows where to look. You need your server’s 10 gigabit IPv4 address for this step. This was the last IP address we set just a few steps ago. We will write…

\\IPA.DDR.ESS.XXX\SHARED_FOLDER

Mine looks like this…

Success!

Check out those transfer speeds! This is from a local Samsung NVMe drive, through a Direct Attach Copper cord, into a 10 gigabit switch, out another DAC cord, and writing onto one of two spinning hard disks in my server.

As with most technology-based projects, this was TREMENDOUSLY FRUSTRATING and then TREMENDOUSLY REWARDING. Hope this helps one of you out there.

What are you using your 10 gigabit network for? Anybody doing any photo or video editing through it? Leave a comment below if you want an excuse for a humble brag.

Fixing Bad Blocks Finder burnin “ERROR - unable to open device: /dev/sda [Device or resource is busy]”

2019-12-20T00:00:00-07:00

My situation

I had four equivalent drives.

I had 2 x RAID 1 mirrors created with mdadm (ref)

I had both of these RAID 1 mirrors pooled together using mergerfs (ref)

The problem

I tried to run a burnin with bad blocks finder.

sudo bbf -c  burnin /dev/sda

I received an error.

ERROR - unable to open device: /dev/sda [Device or resource is busy]

Unmounting the drives

I went in and unmounted my mergerfs pool and both of my mdadm arrays from their respective spots in my /mnt folder.

$ sudo umount /mnt/storage
$ sudo umount /mnt/mirror1
$ sudo umount /mnt/mirror2

I tried the burnin again here and was still receiving the ERROR.

Deactivating the mdadm array

I then stopped the mdadm arrays, zeroed their superblocks, commented out persistent references to the array in /etc/fstab file, commented out the array definition in my /etc/mdadm/mdadm.conf file, and updated initramfs so that the early boot process doesn’t try to find these arrays. (ref)

Happily ever after

Now my first 10TB drive is doing a burnin… only 28 more hours to go ?

Hopefully this helps someone!

Special thanks to trapexit for not only his help figuring out the issue, but for developing these tools in the first place. Check out his website or his Github.

Here’s a link to the original issue a opened on Github.

Start Learning Web Development

2019-11-05T00:00:00-07:00

Skip to tl;dr

There is a LOT of good information out there. Tutorials, videos, paid courses, free courses… so much good stuff.

When I decided I wanted to learn to code, I just started doing whatever looked interesting to me. It’s easy to fall into the trap of trying to pick the “optimal” syllabus for your personal learning journey. I wanted to start coding now.

I actually started years ago – overnight in an airport in Bergen, Norway – doing the Django polls tutorial. This is a good one, but my goodness it was a hard time.

Yes, we can attribute that partially to pulling an all-nighter. I definitely get dumber without sleep. But the real problem was that I just didn’t understand much of anything about web development. Plus, my Python was meh at best.

The Django polls tutorial is good, but probably not where you want to start.

First, learn Python

I found that whole experience kind of frustrating. I made it through the whole app, but I didn’t know any programming and didn’t really understand how this web development framework even works. I didn’t understand any of the anatomy – the structure – of the code, so it was all too mysterious.

And when you don’t know how it works, you can’t learn.

The best thing I did when learning to code was treating myself like more of a newbie than I originally felt.

I bought Zedd Shaw’s Learn Python 3 the Hard Way and started toiling away. I think it took me about a week to make it through nearly everything. I attempted to go slowly, making sure I researched anything that was unclear. I even filled up a few dozen pages of a notebook, taking actual handwritten notes.

Second, learn Django

Since I knew coding was not going to be a full-time job for me, I was wary about learning a JavaScript framework. They seem to change faster than I learn! I wanted to find something old and boring. Something nearly set in stone. Something I could rely upon.

That’s why I picked Django.

Django is proven. It’s not terribly flashy or hyped up, but it gets the job done while handling a lot of best practices for you. I like code that doesn’t change because I didn’t even know how ANY frameworks work, much less the specifics on Django.

It’s easy to pick up a random tutorial on someone’s blog or on YouTube, but when you’re at this early stage, you cannot afford to be random.

If you jump around from tutorial to tutorial, you miss the basics. You want to build skills like a pyramid: foundation first, then layer by layer bring the rest together. What we end up doing in this Age of (Free) Information is building our skills more like a puzzle. We piece together sections as we see them, and then eventually try to tie the whole thing together.

If you lay the foundation first, you learn the bits that teach you the most. It gives you a broad view which you can then deepen as you need to learn new skills.

The most comprehensive piece I’ve found on learning Django is Django for Beginners by Will Vincent.

If you don’t know HTML or CSS, you may want to do any old book on those, too.

Third, Go Further

Once you’ve laid your Django foundation, you have to start building depth.

The recommended next step is either:

Try to build your own thing that fits the structure of a project you’ve already built
Go through another book that’s more advanced

If you feel comfortable, you can try to build your own stuff. I did that because I didn’t have a book in mind. And it worked for a little while, but I was unsuccessful. I just didn’t have the mental model of web development ingrained in my brain. You might have more success than me.

What saved me was a follow-up to Django for Beginners: Django for Professionals.

This book wasn’t published when I was ready for it, but it came out only a few short weeks later. When I snatched it up, I felt really hopeful about my future as a coder.

So hopeful, in fact, that I actually developed an app I could use with some of my fitness mentoring clients.

Django for Professionals teaches you Docker right away, gets you a ton more practice with Django in general, and brings you through the whole process of deploying an app professionally (i.e. keeping private things private).

Fourth…

After that, start building your own project! Now that you have your foundation set, you can deepen your skills.

Come up with a project idea that is super duper simple. It’s okay if someone has already done it before. You can steal their structure and try to recreate it.

Write out your plans on a napkin or Post-It note. Start building and search for help when you need it.

tl;dr

There’s no reason to re-create the wheel when we already have steps that work. Use these affordable resources to level up your programming skills right away.

Start now. Put in the work and you will be rewarded.

Have you tried out any of these resources? Are there other ones you found helpful? Leave a comment below or submit your own version of this article.

Django Projects - 99 ideas for beginners

2019-10-01T00:00:00-07:00

Learning Django, I found it really helpful to go through some beginner tutorials.

But eventually, you have to fly on your own. You have to learn how to plan out a project and how to research what you don’t know.

We believe in learn by doing. And that there’s no better time to start than now.

It doesn’t matter if it already exists. Your not building software to change the world, you’re building software to learn web development.

Here are some ideas to get you started.

99 Beginner Django Projects

Track sleep
Track customized habit performance
Track customized habits of users
Send daily reminders in email
Send daily reminders in SMS
Store a list of current projects and log entries
Freelance time tracking on current projects
Freelance project tracking with view ability for project owners
Freelance project tracking with comment ability for project owners
Send daily inspirational quote to your email
Send daily inspirational quote in SMS
Make a blog like WordPress
Make an e-commerce store like Gumroad
Make someone else a blog
Make someone else an e-commerce store
Collect your thoughts on books
Make an e-commerce store with Amazon Affiliate links
Make an e-commerce store that automatically pulls Amazon product info
Make a list with reviews of your favorite notebooks
Make a list with reviews of your favorite kitchen utensils
Catalog your favorite meal recipes
Pin a list of DIY projects
Pin a list of DIY projects that lists needed items from Amazon product API
Remake an app from a tutorial you’ve done
Pin a list of your favorite single origin coffees and put them on a map
Pin a list of camping gear you recommend for overnight car camping
Pin a list of camping gear you recommend for backpacking
Pin a list of climbing gear you recommend for beginners
Create a forum like Discourse
Make your own personal Wiki
Make a portfolio of projects
Compress and crop uploaded photos
Install Wagtail and see if you can get it to work
Test out django-filter on one of your old projects
Track your hours of programming
Track your hours of working out
Track your medical condition
Store research about your medical condition
Log stories from your favorite non-fiction books and write your own reflections
Invite others to log their stories
Log your favorite songs and the memories they make
Invite others to log their favorite songs
List out business ideas, why they would work, and why they wouldn’t work
List out the building blocks or DIY projects for your home
Make a Django source code highlighter for your site
Make a form builder for non-coders
Make a list of coding interview questions and practice
Make a list of certification questions and practice
Generate reports on your progress
Create timelines and anticipated milestones for entrepreneurial projects
Log your workouts
Track your favorite workouts
Track your favorite exercises
Create your own RSS reader
RSS track your favorite blogs AND YouTube videos
Email yourself price alerts for a desired online product
Grab local air quality data daily
Grab local weather forecast daily
Grab weather forecast by user inputted location
Grab air quality by user inputted location
Grab other surrounding day weather forecast
Grab other surrounding day air quality forecast
Grab water contaminant data daily
Grab water contaminant data for user inputted location
Auto-generate a workout
Generate a body part workout based on selected body part
Generate a body part workout based on selected exercise
Generate a random conditioning workout
Generate a random conditioning workout based on selected duration
Log your runs
Track average mile speed during runs
Make a journal online
Make a simple payment platform to collect freelance payments using Stripe
Generate invoices for freelance work
Generate invoices and allow clients to pay right there
Track weather in desired vacation destinations
Track weather in national parks
Track weather in state parks
Track weather in coldest areas of the world
Log podcast episodes and memorable quotes
Log podcast episodes and show all quotes for a given show’s collection of episodes
Log songs you’re trying to learn on the guitar
Log the hours you practice on the guitar by song
Pin guitar scale exercises
Pin music theory notes
Pin your favorite guitars
Implement guitar search by color, brand, price, and style
Track desired travel destinations
Share travel destinations with viewers
Share travel destinations with travel buddies or significant other
List average weather for potential travel destinations based on user inputted date
Make a photo sharing app
Add sorting to photo sharing app based on DSLR metadata
Add ability to download various photo sizes and qualities
Link photos to your previous travel destinations to create a picture journal
Make a flashcard app like Quizlet
Make a polling app with user submitted answer
Add upvoting to polling app
Make a simple app you can submit as your own tutorial

There you go!

Did you finish any of these projects? Or have your own project to share? Leave a comment below.

Embedding Videos Into Django Project

2019-09-24T00:00:00-07:00

Have you ever tried to embed a video into your Django website so find that it wouldn’t work?

Well, I have.

Let’s go over the solution I found to this problem.

The problem

I’ve been making an app for some other people that I mentor. I decided it would be easier on me and more beneficial to my mentees if I answered their questions in video instead of writing; speeds things along and I can increase my output while conveying my non-verbal communication.

I expected to just hold a Vimeo URL in the database and plug it into a set embed code in my template.

Long story short: that didn’t work. So I went searching for a new answer.

Answer: django-embed-video

I found the solution in a lovely plugin package, django-embed-video.

Install the package

pip install django-embed-video

Change `settings.py`

INSTALLED_APPS = (
    ...
    'embed_video',
)

Add the field to your model

# app_name/models.py
from django.db import models
from embed_video.fields import EmbedVideoField

class ModelName(models.Model):
    video = EmbedVideoField(blank=True) # an empty field is fine for me

Load the tags at the top of your template

{# for me, this was app_name/model_name_detail.html #}
{% load embed_video_tags %}

Call the video in the template

{# still app_name/model_name_detail.html #}
{# the string at the end is the size of the video #}
{# options: tiny, small, medium, large, huge #}
{% video model_name.video 'medium' %}

Other features

You can also add your new model field to the Django admin and customize the video backends.

Plus, you can embed audio from SoundCloud if video isn’t your thing.

My favorite part about it is that it will auto-detect which backend to use based on the link. Super simple in case I end up switching from Vimeo to YouTube.

Alternative Solutions?

That’s it! If you’ve ever solved this problem, tell us a story in the comments below. I’d love to hear what other solutions are out there.

Likewise, if you’ve ever written your own custom backend, share the tale!

Some Shortcuts in Atom Text Editor

2019-09-23T00:00:00-07:00

One of the things that blew me away when I started to get back into coding in this new day and age was the text editors.

I knew about Notepad and Notepad++, but working with Atom really sped up the code writing process.

Here are a few shortcuts that I couldn’t live without.

Delete line (Ctrl + Shift + K)
Duplicate line (Ctrl + Shift + D)
Select whole term (Ctrl + D)
Select next copy of this term (Ctrl + D)
Close window (Ctrl + W)
Comment out line or group of lines (Ctrl + /)
Toggle sidebar visibility (Ctrl + \)
Snippets! (Type something, then hit TAB)

You can type “html” then hit TAB and it will give you a bunch of boilerplate.

Duplicating lines is good when you have different, but related functions.

I’ve even been writing a few articles in Atom, working locally and easily making lists with (“ol” + TAB), (“ul” + TAB), and (“li” + TAB).

Which ones did I miss that you can’t live without?

Learning Django Past the Documentation

2019-09-20T00:00:00-07:00

After you’ve read documentation from a bunch of different projects, you’ll come to realize the Django Documentation is fantastic. It’s concise and contains lots of examples and suggested use cases, which I have found invaluable as a n00b.

But sometimes it still leaves you with wtf feels.

When you get to a point where something you think you need to know is far outside your scope, what do you do?

Realize you probably don’t need it

Most of the books, well-written documentation, and tutorials you’ve read and watched cover the parts of Django that are most useful or best practice. If you haven’t heard about something, it’s because most people aren’t using it. And there’s usually a reason for that. You may want to ask a professional’s opinion.

Google search

There may be a rando somewhere who truly DOES think this is useful to know, and they may have written something about it somewhere. There are two main types of search results in my head:

Private sites, such as this one
Stack Overflow questions

A third, less likely scenario is that your google search actually points you to a “hidden-to-you” part of the documentation that actually answers your question.

Take your search results and start tinkering around. There will likely be some differences, but hopefully not TOO many since you chose an “old” (read: stable) framework like Django. Even lessons or questions that are five or more years old can still be helpful.

Use pen and paper

Don’t let your propensity for computers bias you.

Writing a blog on the computer is nice because you can type faster than you can handwrite. Your brain moves faster than your muscles can, so it helps you get in that state of flow.

That’s no help when you don’t know what you need to do, though.

When my mind is racing and unfocused, that’s when I reach for a pen and paper. The goal is to just start writing anything and whittle it down into something that makes sense.

Especially write down questions you don’t have answers to. That will help you focus your research.

And if you DO figure something out, write that down, too.

Django source code

This feels like the “long way” to do things, but when you consider that the “short way” doesn’t actually get you anywhere, it’s appeal grows.

Head over to the Django GitHub repository. Find the files you’re importing and start reading through the classes you’re using. They probably inherit from many other classes, so you’ll have to read those, too.

Getting into the guts of the code can elucidate new pieces you didn’t know about.

If you’ve ever worked out, it’s kind of like that: you can better understand what your exercises do when you know a little bit of anatomy.

Summary

The docs don’t contain everything you need. There is always some implicit information flying under the radar that really simplifies a topic. When trying to learn a new-to-you piece of Django…

Figure out if you REALLY need to know this thing now
Search it on Google and read through other people’s problems
Write out your questions with pen and paper
Start reading the Django source code

Making Your Own Email Templates in Django

2019-09-19T00:00:00-07:00

I’m looking to make my app notify the user when their object is updated by an admin.

Problem is that I had no idea how to use my own email templates.

But I got it working! Here’s how you can put it into your Django project.

I’m using Django 2.2.5.

Email settings in `settings.py`

There are two useful ways that I know of to send emails:

Logging directly to the console window
Sending via SMTP

The first is much quicker, but doesn’t actually send emails. It’s good for testing.

The second is what you need if you need to actually send emails to a user’s inbox.

Send to console

To log emails to the console, you just have to set EMAIL_BACKEND in settings.py.

EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'

Here’s the documentation for reference.

Send via SMTP

There are a few settings you’ll need to take care of if you want to send live SMTP emails:

EMAIL_BACKEND
EMAIL_HOST
EMAIL_PORT
EMAIL_HOST_USER
EMAIL_HOST_PASSWORD
EMAIL_USE_TLS or EMAIL_USE_SSL

If you’re ready to give this a try, I would recommend making an account on SendGrid. Even when your free trial expires, you’ll have 100 free emails to send every day. Not enough to run a big project, but plenty of leeway to learn how to use it.

Once you’re signed up, you want to find the “Send Your First Email” instructions to set up SMTP.

Then choose SMTP relay, not Web API.

Click Choose under SMTP Relay

Then you need to create a SendGrid API key. This will be your EMAIL_HOST_PASSWORD in your settings.py file.

After naming and creating your key, step two will autopopulate. These are what you’ll use in your settings.py file. Here’s a link to the SendGrid documentation on exactly how to set it up.

Make sure all the settings outlined at the beginning of this section are set, then you can try to actually send an email.

from django.core.mail import send_mail()

The send_mail() function has been in Django for a while now. It takes a few parameters and sends an email. Very simple.

You must supply:

subject: string
message: string
from_email: string
recipient_list: list of strings

Worth noting: message is the plain text body content of the email. If you want to send an HTML email, you want to set html_message. We’ll do both to get some practice with templates.

Here’s a link to the Django documentation for reference.

Sending a plain email

Let’s practice. I’m assuming you have your EMAIL_BACKEND all set up. If you can’t get SMTP working, make it log to console to get the templates working first. It’s much more encouraging than sitting there trying to troubleshoot why SMTP isn’t sending.

# settings.py
EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'

So for my app, I want to send an email when my UpdateView form is submitted. This will notify the user related to that model object that they should come check it out. So I will override the form_valid() method to send an email when the UpdateView form is submitted.

# app_name/views.py
from django.core.mail import send_mail
from django.views.generic import UpdateView

from .models import ModelName

class ModelNameUpdateView(UpdateView):
  model = ModelName
  fields = ['field_1', 'field_2',]
  template_name = 'app_name/model_name_update.html'

  def form_valid(self, form):
    if request.method.POST:
      from_email = 'me@email.com'
      recipient_list = ['yourpersonalemail@email.com', ]  # put your real email here
      subject = 'Subject text'
      message = 'This is the body of the email.'
      send_mail(subject, message, from_email, recipient_list)
    return super(ModelNameUpdateView, self).form_valid(form)

Update your model with your UpdateView and check your console or inbox (depending on your EMAIL_BACKEND) to see if it worked!

from django.template.loader import render_to_string()

Now that emails are working, we can discuss the templates. Email templates work just like your traditional Django templates that load pages: we put the content in its own file, then we point to that file.

Create templates

We’re going to make three templates:

app_name/email/model_name_update_email_subject.txt
app_name/email/model_name_update_email_message.txt
app_name/email/model_name_update_email_message.html

Make those three files in their respective folders, then move your strings from views.py to each respective template.

app_name/email/model_name_update_email_subject.txt

Subject text

app_name/email/model_name_update_email_message.txt

This is the body of the email.

app_name/email/model_name_update_email_message.html

This is the html body of the email.

Call render_to_string()

Now we move back to views.py where we want to point to these template files.

# app_name/views.py
from django.template.loader import render_to_string  # new
from django.core.mail import send_mail
from django.views.generic import UpdateView

from .models import ModelName

class ModelNameUpdateView(UpdateView):
  model = ModelName
  fields = ['field_1', 'field_2',]
  template_name = 'app_name/model_name_update.html'

  def form_valid(self, form):
    if request.method.POST:
      from_email = 'me@email.com'
      recipient_list = ['yourpersonalemail@email.com', ]  # put your real email here
      subject = render_to_string(
        template_name='app_name/email/model_name_update_email_subject.txt'
      )
      message = render_to_string(
        template_name='app_name/email/model_name_update_email_message.txt'
      )
      html_message = render_to_string(
        template_name='app_name/email/model_name_update_email_message.html'
      )
      send_mail(subject, message, from_email, recipient_list)
    return super(ModelNameUpdateView, self).form_valid(form)

render_to_string() is kind of like the render() function you may have used before, but returns the template as a string instead of an HttpResponse object.

Give that a shot and see if it works… hint: it probably won’t.

Python strip() function

I got a variety of errors here.

When using the console backend, I just didn’t see any email at all.
When using the SMTP backend, I received a BadHeaderError that says “Header values can’t contain newlines”.

You might be thinking “I don’t want any newlines in there!”

Well, we have a solution: strip the whitespace off the ends with the python strip() function. Update your views.py file…

# app_name/views.py
subject = render_to_string(
  template_name='app_name/email/model_name_update_email_subject.txt'
).strip() # new

And try to trigger your email again.

And it should be successful!

Django 3.0 Alpha Released

2019-09-18T00:00:00-07:00

Big news! The alpha version of Django 3.0 has been released.

This is not suitable for production!

BUT, this is suitable for sandboxing. Feel free to install it and see if it breaks anything. And if you notice something, make sure to submit a bug!

Or you can try to get up and running with ASGI. How exciting!

Here’s a link to the official release statement.

And here’s a link to the Django 3.0 release notes.

Limit Access to Only Superusers in Django

2019-09-18T00:00:00-07:00

So in my other life as a strength and conditioning coach, I have a few online mentees. They basically email me questions and I answer them.

I’m using Django to make an interface for them to submit questions and see all the other questions that other members have asked. Plus adding some searching so the old stuff doesn’t just die.

I am using the generic UpdateView to answer the questions. This view is marking the question “published” and storing my answer in the database.

My issue is that I don’t want the users to be able to see this because then they could be adding answers to questions and people are paying for ME to answer the question.

This entails two major parts:

Adding a link to the UpdateView in the DetailView
Blocking permissions to access the form with UserPassesTestMixin

Let’s walk through it.

Add link to UpdateView

Order doesn’t particularly matter here. You’ll have to do each step. I like to start with the URL, so first, we need to add the URL to our UpdateView to our urls.py file.

# app_name/urls.py
from .views import AppNameUpdateView # make sure to import your view

urlpatterns = [
    # other URL patterns
    path('/update/', AppNameUpdateView.as_view(), name='model_name_update'),
]

Once we have that, let’s go ahead and make this UpdateView in our views.py.

# app_name/views.py
from .models import ModelName

class AppNameUpdateView(UpdateView):
    model = ModelName
    fields = ['field_1', 'field_2',']
    template_name = 'app_name/model_name_update.html'

Now we just need to get the template running.

{# app_name/model_name_update.html #}
Update

  {% csrf_token %}
  {{ form.as_p}}

Check to make sure it works before moving on.

Once that skeleton is working, let’s link to this page on the DetailView page. Remember we only want admins to be able to get to this UpdateView.

{# app_name/templates/app_name/model_name_update.html #}

{% if user.is_superuser %}

  Update

{% endif %}

Awesome, now we have a working link to the UpdateView so we don’t have to hard type it directly into our browser.

Adding permissions

Now what if a user finds or guesses the link? They’ll still be able to see the UpdateView because we’ve only authenticated the vision of the link to the page, not the view and its logic.

Fortunately, Django makes this super easy. You just have to know about the UserPassesTestMixin. Here’s a link to the documentation.

UserPassesTestMixin is designed to work with class-based views. All we need to do is

define the test as a function within the view by overriding test_func()
define the login_url attribute to tell Django where to redirect an unauthorized user

# app_name/views.py
from .models import ModelName

class AppNameUpdateView(UpdateView):
    model = ModelName
    fields = ['field_1', 'field_2',']
    template_name = 'app_name/model_name_update.html'
    login_url = reverse_lazy('login') # new

    def test_func(self): # new
        return self.request.user.is_superuser

And there we go!

Django Has Changed Over Time

2019-09-17T00:00:00-07:00

Repetition is king for learning. Working with models, views, and templates over and over gets repetitive, but that’s a good thing.

Have you ever seen a tutorial online and thought, “That code doesn’t look very familiar.” No repetition, no knowledge.

Usually these differences are not WAY off. The similarities give you sense for what you have to do. But after you give it a shot, you get an error.

One of the biggest reasons to start your web development journey by learning Django is because it is OLD. Old things are set in their ways. They won’t change so much on you as you slog along.

In fact, this was one of the biggest reasons I decided to pick Django as my learning language. You can be sure that you won’t lose much on your investment because what you learn will still be relevant in the future.

But still, things have changed in Django. And this makes it difficult to learn on the internet. There’s a lot of wasted time. And time is more valuable than money.

To help save you time, I want to outline some of the major differences I’ve run into while learning Django.

Note

When reading a tutorial, attempt to find out the version of Django the author is using. You can view changes in the Django release notes.

urlpatterns

Before version 2.0, Django used a url() function to add a URL to your urlpatterns list. Here’s a link to the 1.11 django.conf.urls documentation.

Django 2.0 added path() and re_path() functions for putting URL items in your urlpatterns list in urls.py. Here’s a link to the 2.0 django.urls documentation.

They function similarly, but url() will likely be deprecated in the future.

Import statements

Import statements have been the single biggest challenge for me. With all of this DRY, reusable code, there is this complex web of python files. It’s kind of like navigating through a maze: you can see the path ahead of you, but you need a memory sense of everything around you if you’re going to make it out of the maze.

The design is fantastic, but it’s difficult to pick up.

If you get an ImportError, it’s possible the file you’re looking for has just been moved. Make sure to read the error because it might tell you what’s going on. Then, as always, Google is your friend. You may even want to search directly on the Django Documentation.

An example to note that tripped me up recently: mapping urls.

You may have noticed in the previous section that the import path is different for those URL functions.

Previously your import statement at the top of your urls.py file would be:

from django.conf.urls import url

In Django 2.0, it looks like:

from django.urls import path

Another thing to note: the include() function for including other urls.py urlpatterns has followed this move, located at django.urls.include() in Django 2.0.

Python print statements

Python 3 had a major update in print statements. Here’s the differences listed directly from the Python 3 Release Notes.

Old: print "The answer is", 2*2
New: print("The answer is", 2*2)

Old: print x,           # Trailing comma suppresses newline
New: print(x, end=" ")  # Appends a space instead of a newline

Old: print              # Prints a newline
New: print()            # You must call the function!

Old: print >>sys.stderr, "fatal error"
New: print("fatal error", file=sys.stderr)

Old: print (x, y)       # prints repr((x, y))
New: print((x, y))      # Not the same as print(x, y)!

Help us all!

Got another example? Leave a comment below because others are having the same problem!

Why I Wanted to Learn to Code

2019-09-14T00:00:00-07:00

Every answer is at your fingertips, but time is scarce. What will you choose to do?

I encourage everyone I meet to take the mindset of the “lifelong learner”. If you aren’t a little embarrassed by who you were ten years ago, then you’re probably not improving. And the best way to help the world is by first helping yourself.

But why coding?

Backstory

My first love was actually ice hockey. I grew up in Michigan, USA, where hockey is a big deal. I’ve been on ice since I was three years old.

Hockey taught me work ethic. I couldn’t be the best if I didn’t work hard. But it also taught me what it’s like to feel superior, to win.

And I had to learn how to lose. The hard way.

Ego fragility paved the path of lifelong learning. The only reason you would seek to change is if you’d like to see something different about yourself.

Practice

I got a heavier dose of repeated practice just before high school when I picked up my first guitar. How cool would it be to make those sounds that all my favorite bands make.

There’s that ego again.

Guitar is much different than sports. You don’t have the same physical inputs. You have to listen to what your ears are telling you to know if you’re doing it well or not.

Monotonous. Repetition. Repetition. Repetition. Over and over.

You can practice for an hour and not get much better.

But if you practice for an hour every day for six months…

Fitness

Once in high school, I started working out. And my oh my was that fun.

If you like to be rewarded for your effort, start working out. The strength you gain in the first few months, known colloquially as “newbie gains,” is perhaps the single best investment anyone can make.

You see, your body is capable of magical things. But your brain slows you down, fearing injury. There are stories of women lifting cars off of their children. Now that is motivation.

What brakes are you riding with?

Career

I ended up turning fitness into a career, a job title I still carry to this day. I’ve learned a lot in the field, coaching for 10+ years and even acquiring a masters degree in Anatomy & Cell Biology.

If you’ve never dedicated time to help someone out, do it. Talk about rewarding. It’s especially important if you’re the type of person who isn’t quite sure that you’re worth anything. We’ve all been there. When you teach someone something new, that doubt disappears (temporarily, so remember to practice that skill).

What can you teach?

Coding

I started to ask myself, “How can I make a greater impact?” Though the impact is great, coaching people to be stronger and fitter doesn’t scale well. I can help 30-50 people, but any more than that and there’s no way I’ll be able to sustain it with how I operate. I think in our Information Age, a personal touch goes a long way. How can I free up resources to allow for interpersonality?

That’s why I look to tech.

I always liked math in grade school. And I picked up computers pretty quickly, at least relative to the people I knew. Maybe coding could be the answer?

I tried it in college and failed. I didn’t know how to teach it to myself. And I did not have a very involved teacher. I fell behind and never caught up.

But in the last couple years, I decided to pick it back up again. I wanted to tinker with my personal website. I already had a little sandbox to play in. Low risk, high reward; might as well try it out.

So I taught myself HTML and CSS. I learned how WordPress worked. Then I learned how use PHP to change up a WordPress website. I learned how the web works and started a few websites for some fitness friends.

Then that started to get boring. I can help others spread their messages, but surely there’s a more powerful way.

That’s when I started diving into Python and the Django web framework. Maybe I can start making websites that aren’t WordPress websites?

I considered a few coding bootcamps and even going back to college. But it didn’t make sense to me to totally abandon a career that is still working for me. I know some people are there, but that’s not what drives me.

What drives me is getting better. Learning new things. Developing myself.

If you want to improve, one of the easiest things to learn is basic coding.

And don’t wait for the time to be right. The time is never right.

Start coding now.

Start Coding Now

Empty Image Fields in Django

tl;dr

Discovering a Bug

How FileField and ImageField Look in the Database

How to Define a Model with Not Required File Fields

How to Filter for Empty File Fields

Opening Django Model Files in PyMuPDF

Opening a file from the local filesystem

Should you open with file.name or file.path?

Opening an InMemoryUploadedFile

Opening a file from remote storage

Opening a file with pymupdf as a context manager

Custom Django Python Migration

Overview of what we will do

Whole file contents

Create an empty migration

Create the change you want to see

Undo the change you made for migrating back in time

Tie these operations to the migration

Apply, un-apply, and re-apply the migration

DjangoCon US 2024 - Short Summary

Day 1

Learning, Teaching

SPA vs Progressive Enhancement

Error Culture

Lightning Talks

Django Forms

SQL Queries

Passkeys

Running Django in Production

Security

Day 2

Self-Improvement

htmx + Alpine.js

Maps

Lightning Talks

End-to-End Testing

Atomic Transactions

Plugins

Migrations

Large Scale Project Considerations

Day 3

Django Fellows

Release Notes

Django History

Lightning Talks

WebRTC

PostgreSQL

Django User Model

Bootstrapping

Understanding the Template Method Pattern in Python

Introduction

What is the Template Method Pattern?

A Simple Example: Coffee and Tea

Benefits of the Template Method Pattern

Real-World Applications

Example Template Class

Conclusion

Custom Error Messages on Model Deletion in the Django Admin

Protect Model Deletion

Override Model delete() Method

Message the User by Overriding the delete_model() Method

Customize the Deletion HTTP Response by Overriding the response_delete() Method

Bulk Deletions Follow a Different Course of Action

Custom Dictionary Representation of Python Dataclass

Creating a dataclass

Turning the object into a dictionary

Making a custom dict factory

Where Does Validation Occur?

Where Does Validation Occur?

Simple Validation of Data Types in the Serializer

Custom Data Validation in the Serializer

Writing Reusable Validators

How to Kill the Django Development Server Without Ctrl+C

Finding the runserver Process

Killing a Process

Verify the Process was Terminated

Summary

Introducing djarter – An opinionated Django starter project

How `FileField` and `ImageField` Look in the Database

Should you open with `file.name` or `file.path`?

Opening an `InMemoryUploadedFile`

Opening a file with `pymupdf` as a context manager

Override Model `delete()` Method

Message the User by Overriding the `delete_model()` Method

Customize the Deletion HTTP Response by Overriding the `response_delete()` Method

Finding the `runserver` Process

Make a `dotfiles` Repository