Notes
Atom FeedSwitching From Python requirements.txt to pyproject.toml
In python, a lot of projects are switching over from the old
requirements.txt format of declaring dependencies to pyproject.toml and
others. pyproject.toml can condense multiple requirements-*.txt files as
well as other package metadata into a single file, simplifying several package
maintenance processes.
The steps to switch are:
Create a bare bones
pyproject.tomlfile in the root of your package:[project] name = "<NAME>" version = "<VERSION>" dependencies = [ "dependency1", "dependency2", ]Copy a name and version from your
setup.pyif you’re still using that. Copy dependencies from yourrequirements.txt. Dependencies inpyproject.tomlfollow the same format asrequirements.txtand support dependency versioning with==,>=, and<limits.If you’ve split your
requirements-test.txtor other types of dependencies, you can include them in yourpyproject.tomlas optional dependencies:[project] ... [project.optional-dependencies] test = [ "dependency3", "dependency4", ]Delete your
requirements.txt. Hopefully you’re using version control.Switch your build commands from
pip install -r requirements.txttopip install -e .. If you want to install optionaltestdependencies too, you can do that withpip install -e .[test].
Generating Cloudflare Origin Certificate for Multiple Domains
Cloudflare recommends having an Origin Certificate installed on the server that hosts your website (your Origin) so that requests between Cloudflare and your Origin are encrypted and Cloudflare can authenticate your server’s data.
This gets problematic if you host multiple domains on your server. It’s not possible to do so through the UI, but Cloudflare actually supports multi-domain Origin Certificates through its API. To generate and install a multi-domain certificate, use this script:
"""
This script generates an origin certificate from Cloudlare using their API.
It requires the 'requests' library to make HTTP requests.
"""
from cryptography import x509
from cryptography.x509.oid import NameOID
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import rsa
import requests
# Origin SSL Certificate Update API Token
# Generate at https://dash.cloudflare.com/profile/api-tokens
# Create a Token -> Create Custom Token
# Settings: Permissions Zone + SSL and Certificates + Edit
AUTH_TOKEN = "<TOKEN>"
INSTRUCTIONS = """
sudo cp certificates/server.key /etc/nginx/ssl/server.key
sudo cp certificates/server.pem /etc/nginx/ssl/server.pem
sudo chmod 640 /etc/nginx/ssl/server.key
# Restart nginx
# /etc/init.d/nginx restart
"""
def get_domains() -> list[str]:
return [
"example.com",
"example2.com",
"example3.com",
]
def generate_key() -> rsa.RSAPrivateKey:
# Generate our key
key = rsa.generate_private_key(
public_exponent=65537,
key_size=2048,
)
# Write our key to disk for safe keeping
with open("certificates/server.key", "wb") as f:
private_bytes = key.private_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PrivateFormat.TraditionalOpenSSL,
encryption_algorithm=serialization.NoEncryption(),
)
print("Private Key for signing")
print(private_bytes.decode("utf-8"))
f.write(private_bytes)
return key
def generate_csr(key: rsa.RSAPrivateKey) -> str:
# Generate a CSR
csr = x509.CertificateSigningRequestBuilder().subject_name(x509.Name([
# Provide various details about who we are.
x509.NameAttribute(NameOID.COUNTRY_NAME, "<COUNTRY>"),
x509.NameAttribute(NameOID.STATE_OR_PROVINCE_NAME, "<STATE>"),
x509.NameAttribute(NameOID.LOCALITY_NAME, "<CITY>"),
x509.NameAttribute(NameOID.ORGANIZATION_NAME, "<ORGANIZATION>"),
x509.NameAttribute(NameOID.COMMON_NAME, "<COMMON_NAME>"),
])).sign(key, hashes.SHA256())
csr_bytes = csr.public_bytes(serialization.Encoding.PEM)
with open("certificates/csr.pem", "wb") as f:
f.write(csr_bytes)
csr_string = csr_bytes.decode("utf-8")
print("Certificate Signing Request:")
print(csr_string)
return csr_string
def request_origin_certificate(csr: str, domains: list[str]) -> None:
# Request the origin certificate from Cloudflare
url = "https://api.cloudflare.com/client/v4/certificates"
headers = {
"Content-Type": "application/json",
"Authorization": "Bearer %s" % AUTH_TOKEN,
}
domains += ["*.%s" % d for d in domains]
print("Domains:")
print(domains)
data = {
"csr": csr,
"hostnames": domains,
"request_type": "origin-rsa",
"requested_validity": 5475, # Valid for 15 years
}
response = requests.post(url, json=data, headers=headers)
if response.status_code == 200:
print("Origin certificate requested successfully.")
data = response.json()
certificate_pem = data["result"]["certificate"] # type: ignore
with open("certificates/server.pem", "w") as f:
f.write(certificate_pem.strip())
print("Origin certificate:")
print(certificate_pem)
else:
print("Failed to request origin certificate.")
print("Status Code:", response.status_code)
print(response.text)
def main() -> None:
domains = get_domains()
key = generate_key()
csr_string = generate_csr(key)
request_origin_certificate(csr_string, domains)
print(INSTRUCTIONS)
if __name__ == "__main__":
main()
Introductory Computer Science and Software Engineering Topics
A list of practical topics to learn as an introduction to computer science and beginner software engineering:
- Algorithms and Data Structures (textbook) - the basics of computer science; emphasis on the data structures such as arrays, maps, and graphs
- Programming languages to learn well:
- Python (or some scripting language)
- Typescript (for a well-implemented but accessible strongly-typed language)
- Go (for a simple language with good support for parallelization)
- SQL (widely applicable for talking to databases)
- Programming languages to learn:
- C (the fount of current languages)
- Java (the most common language for backend engineering)
- Javascript (the Lingua Franca for frontend engineering)
- HTML (because you’re sooner or later going to need to build a website)
- Bash (so you can improve your tools)
- How to learn a language: write small programs for it in Leetcode, Rosalind, Advent of Code, or similar to learn the syntax. Write small projects in it to learn the language ecosystem.
- Topics to go deeper on:
- Unix/Linux basics - Environment variables, everything is a file
- 12-factor app - Some rules for designing backend systems
- HTTP/JSON - How to send messages between systems like they’re websites
- gRPC/Proto - How to send messages between systems (grown up)
- Designing Data-Intensive Applications - A dive into architecture around storage, indexing, and processing of data
- Code architecture - Learn how to break software down into modules with abstractions
- Tooling:
- IDE and/or Text editor (try Neovim) - Know and/or configure the capabilities of your text editor or terminal environment
- Terminal - Learn CLI commands as you need them but be familiar with things like piping data from one command to another
- Git - Version control everything. Be familiar with rebase.
- (Optional) Set up and configure your dotfiles - Once you start version controlling your configuration, you’ll more actively customize your working environment
- Projects (because building things is more fun than reading)
- Build a static website
- Build a web scraper
- Build a CLI
- Build a program that reads/writes data to a database
- Build a program that reads, processes, and writes data to a file
PID Controller
A PID controller is a common systems control function based on a feedback loop. PID stands for:
- Proportional: A multiplier on the error between the measured and set/target value. This effect increases the farther the output is from the target (error).
- Integral: A multiplier on the accumulated error over time. This effect increases the more time (and error) the output is from the target.
- Derivative: A multiplier on the rate of change of error. This is meant to dampen oscillations.
Below is an example chart that shows a PID controller in a system where the target is “10”, the starting value is “0”, and with a variety of PID tuple configurations.
PermalinkAvailability Percentages
Availability percentages measured in 9s of uptime, qualified by (rounded) expected downtime
| Availability % | Downtime per year | Downtime per month | Downtime per week | Downtime per day |
|---|---|---|---|---|
| 90% | 37 days | 4400 minutes (73 hours) | 1000 minutes (17 hours) | 140 minutes |
| 99% | 3.7 days | 440 minutes (7.3 hours) | 100 minutes (1.7 hours) | 14 minutes |
| 99.5% | 1.8 days | 220 minutes (3.7 hours) | 50 minutes | 7.2 minutes |
| 99.9% | 8.7 hours | 43.8 minutes | 10 minutes | 1.4 minutes |
| 99.99% | 53 minutes | 4.4 minutes | 1.0 minutes | 8.6 seconds |
Source: Wikipedia; High availability
Permalink