Albertyw.com

Upgrading MariaDB Database Versions

June 2, 2024

This script updates the database version for MariaDB docker containers. Given the container name, it will stop the container and launch a new MariaDB container with the updated version of MariaDB but with the original data and network.

Notes:

The script requires the name of the docker container as an argument
The new docker container has some typical recommended configurations set like --cap-add=sys_nice to optimize cpu scheduling.
The script will prompt for the root password in order to execute upgrade scripts

#!/bin/bash

set -exuo pipefail
IFS=$'\n\t'

BASEDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
cd "$BASEDIR/.." || exit 1

NAME="$1"
VERSION="11.4.2"
VOLUME="$(docker container inspect "$NAME" | jq -r .[0].Mounts[0].Name)"
NETWORK="$(docker container inspect "$NAME" | jq -r '.[0].NetworkSettings.Networks | keys[0]')"

echo "$VOLUME"

docker pull "mariadb:$VERSION"
docker stop "$NAME" || true
docker rm "$NAME" || true
docker run \
    --detach \
    --restart=always \
    --network="$NETWORK" \
    --name="$NAME" \
    --volume "$VOLUME:/var/lib/mysql" \
    --cap-add=sys_nice \
    "mariadb:$VERSION" \
    --character-set-server=utf8mb4 \
    --collation-server=utf8mb4_unicode_ci \
    --binlog_expire_logs_seconds=3600
docker exec -it "$NAME" mariadb-upgrade -u root -p --force

Permalink

Concurrent Python Example

January 1, 2024

Reference example of using concurrent.futures in Python >3.2. Standard parallel programming warnings still apply (race conditions, sharing state, inter-process communication):

import concurrent.futures
import time


def calculate(limit: int) -> int:
    """ An example function that takes a long time to run """
    x = 1
    for i in range(1, limit):
        x = (x + i) % 10
    return x


def iterate(data: list[int]) -> list[int]:
    """ A base case without concurrency """
    return [calculate(i) for i in data]


def concurrent_submit(data: list[int]) -> list[int]:
    """ Concurrency using executor.submit """
    result: list[int] = []
    with concurrent.futures.ProcessPoolExecutor() as executor:
        futures: list[concurrent.futures.Future[int]] = []
        for i in data:
            future = executor.submit(calculate, i)
            futures.append(future)
        for future in futures:
            result.append(future.result())
    return result


def concurrent_map(data: list[int]) -> list[int]:
    """ Concurrency using executor.map """
    with concurrent.futures.ProcessPoolExecutor() as executor:
        # Use list to check for exceptions
        data = list(executor.map(calculate, data))
    return data


def main() -> None:
    """ Execute each of the implementations and time it """
    data = [10**7 + x for x in range(20)]
    print(data)

    """
    # Non-parallel implementation
    start = time.time()
    output = iterate(data)
    duration = time.time() - start
    print('iterate: %f' % duration)
    print(output)
    """

    # Parallel with executor.submit()
    start = time.time()
    output = concurrent_submit(data)
    duration = time.time() - start
    print('concurrent_submit: %f' % duration)
    print(output)

    # Parallel with executor.map()
    start = time.time()
    output = concurrent_map(data)
    duration = time.time() - start
    print('concurrent_map: %f' % duration)
    print(output)


if __name__ == '__main__':
    main()

Permalink

Updating UUIDField on MariaDB to Django 5

December 27, 2023

Updating from Django 4 to Django 5 comes with an incompatible change to how Django UUIDFields are stored in MariaDB databases - in Django 4, Django would store UUIDFields as char(32) types but in Django 5, Django would store UUIDFields as uuid types. However, the Django 5 upgrade notes are incomplete. Following the directions on a nontrivial Django project, it’s still likely that you’ll get database errors like django.db.utils.OperationalError: (4078, "Cannot cast 'int' as 'uuid' in assignment of test_database.table_column.id") and errors from fitting 36-character UUID strings into 32-character database fields.

In order to fix this problem, I found that it’s better to convert char(32) MariaDB fields into uuid fields first before migrating to Django 5. That way, there are fewer changes during the update. In order to do so,

While still on Django 4, replace django.models.UUIDField with a uuid field using a uuid database type with:

# app/models.py

from django.db import models


class RealUUIDField(models.UUIDField):
    def db_type(self, connection):
        return "uuid"


class Model(models.Model):
    id = RealUUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    # Formerly:
    # id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)

Explicitly declare older migrations on the uuid field as a char(36) field. Note that this needs to be char(36) because Django will attempt to store UUIDs with dashes ('06cb5e67-467f-4675-91f3-ca466bcee805' instead of '06cb5e67467f467591f3ca466bcee805'). In migration files, replace models.UUIDField with:

# app/migrations/0001_migration.py
class Char36UUIDField(models.UUIDField):
    def db_type(self, connection):
        return "char(36)"

class Migration(migrations.Migration):
    operations = [
        migrations.CreateModel(
            name='Model',
            fields=[
                ('id', Char36UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)),
                # Formerly:
                # ('id', Char36UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)),
            ],
        ),
    ]

Run ./manage.py makemigrations to generate a migration that will convert the database column from a char(32) to a uuid type.
Install the django==5.0 pip package. Fix any other incompatibilities.
Replace RealUUIDField back to models.UUIDField.

Permalink

Replacing Setup.py

December 7, 2023

Setup.py is more or less deprecated and should be replaced by alternatives like pyproject.toml or poetry. This requires installing a few dependencies:

pip install build twine

setup.py command	alternative command
`setup.py sdist`	`python -m build`
`setup.py bdist_wheel`	`python -m build`
`setup.py test`	`python -m unittest`
`setup.py install`	`pip install .`
`setup.py develop`	`pip install -e .`
`setup.py upload`	`twine upload`
`setup.py check`	`twine check`

Permalink

Fixing Mariadb --Column-Statistics Errors

June 5, 2023

Recent versions of mysqldump include a config to include a COLUMN_STATISTICS table. While StackExchange answers recommand passing a --column-statistics=0 flag to mysqldump, this isn’t always possible, e.g. when some code like Django is running mysqldump for you.

Instead of patching your mysqldump calls, if you recently upgraded from using mysql to mariadb, you should check if you’re still using the Oracle mysql version of mysqldump. Switching to the mariadb version of mysqldump may solve your problems. On ubuntu/debian, you can do so with

apt update
apt remove mysql-client
apt install mariadb-client

Permalink