It might be pretty out-of-vogue to host your own servers in 2024, but I still do it and I quite like it. There's a lot of great benefits to building software on basic infrastructure and I always find myself appreciating the flexibility that simple VPS hosting brings. That said, you do need to be careful. Life as a developer on the Internet requires a pretty decent familiarity with security best-practices.

Being self-hosted, there's also a lot of simple design decisions that you need to make when running a system on simple VPSs, and there's really not a lot of good info out there to help with this.

So, here's a simple guide to some of the things I do when setting up my servers.

This post isn't sponsored by Linode. It's just what I know best.

1. Follow Linode/Akamai's Great Guide for Securing Your Server.

Always set up ssh key forwarding and use it. That said, I usually end up leaving Password Authentication Enabled, since I've had problems remoting in from a known machine (i.e. my laptop isn't available). In these cases I use a very strong password (Apple's Keychain app will make up to 31 character random passwords) and synced with iCloud so it's on my phone.

2. Set Up a Cloud Firewall

These days, I prefer using the cloud firewalls provided by the platform (here's a guide for setting this up on Linode). I still do set up simple on-device firewalls on the server itself, but I depend more on the cloud firewall since it's easier to configure.

3. Configure Basic Maintenance Cronjobs

These days, I usually set up a few basic cron jobs to monitor things and keep things running. Here's a list of my typical cron jobs. This list will vary a bit depending on what kind of server you're setting up.

• Postgres nightly/weekly vacuum.
• Disk Usage & Avg CPU Usage monitoring using Nine9s Measures
• Nightly DB Backup & Archive (if needed, see 4)
• Nightly Log Trimming (keep 30 days)
• Automatic Deploys (see 5)

As an aside here, I use Pushover in a lot of my shell scripts. I send a quick API call to the service which forwards Push Notifications to my phone. That way I get notified of things like build failures which normally fail silently.

4. Set Up Automated Backups

If your platform supports it, just use the automated backups for your servers. It's so easy and well worth a few extra bucks per month. If you're not in the position to use those (e.g. Linode when wont't back up encrypted partitions), then make sure you have a cron job that backs up your database. I use simple pgdump | gzip commands in those cases.

If you do the backups yourself, be sure to test restoring from your backups as well!

5. Apps go in Containers

I use Docker, but any container system works. I've been told this setup isn't "how Docker is intended to be used", but that's fine with me. It works. I deploy my apps to my servers using a build script that usually runs automatically. It uses the production git repo for my docker compose files and runs a simple docker compose up with the configuration for production. I even have it set up to boot a backup copy of the current app while it deploys the new one so I get zero-downtime deployments.

It's probably overkill to do this, but it works well and avoids the possibility of corrupting code files on the host machine. The automatic deploy (which I frequently do nightly) is awesome because it means I never have to think about which changes have made it to production. If it's in main it's in prod.

I also run my databases (which is almost always Postgres) in a container as well, only because it makes minor upgrades easier.

6. Run Webservers on the Host

Above, I told you that I use Docker. However these days I always run the webserver (nginx in my case, though Caddy is also very nice) directly on the host. Nginx then proxy-passes to my my containers using the loopback address. That way nothing is exposed (Docker likes to punch holes in firewalls). Running nginx on the host let's certbot work as expected.

7. Set Up Break-Glass Measures

The single most popular post on this blog is about my (in)famous 8GB Empty File break-glass measure. I don't always do this anymore, but it has saved me a bunch of times. To me, SSH password auth is a kind of break-glass fallback too.

What Else?

That's all I can think of at the moment. It's a lot, but it's also not too bad. You don't need to be scared of hosting things yourself, but you do need to be careful. It's a wild world out there on the internet.

As a final note, if you're looking to get a handle on web architecture stuff, I recommend this blog post for a quick rundown on that. Practical Web Architecture is difficult to learn in my experience, and it takes a lot of pondering, trial, and error to do it right. Most resources are intended for enterprise-scale deployments and not smaller projects. In general, don't over-complicate things if you don't need to. One of the best parts about hosting your own apps this way is that you have the flexibility to change things around as you grow (at the cost of some work and likely some downtime).

I've been doing this for a while now, but I'm sure there's something I've missed. This guide is certainly not extensive, but it should at least help you avoid some major headaches. I can say that from experience.

My band, The Fourth Section has been playing more frequent shows lately and along side all of that we've been recording select songs and publishing them on YouTube.

We're well on the way to another album (though the timing is very uncertain). That said though, it feels good to be able to share these live versions of some of our new music. We've even put together a playlist of these edited versions which will continue to grow as we write and perform new material.

I've been playing in a band for nearly a decade now and while my love of writing and playing music has been a crucial release valve and creative outlet for me, one of the best parts of playing in a band is playing live. It's hard work, but it's so fulfilling.

[For nearly all of human history] in order to hear a piece of music you had to be within earshot of someone playing it.
- John Green (vlogbrothers)

Live music is something special. Unlike recorded music—which really hasn't been around all that long in human history—live music has been with us since the beginning. Regardless of form, there's something primal about being in a room where music is being played. It's a communal experience where a group of people sit and appreciate a thing together. Live music is where art meets mind.

And it's not just bands. I'm always struck by just how captivating a solo artist with a guitar can be in a live performance. I love a good acoustic song, but there's something indescribable about being in the room that just can't be replicated by the best recording. A good acoustic song recording can make me feel things, but a good live performance can enthrall me; it can make time stop.

Live music is one of many reminders that we are still very human. Even in this modern and increasingly disconnected world, we still need the same things that drove early humans in times long gone to pound a drum and sing together.

There's tremendous beauty in that, I think.

Yesterday I went hiking at Lake Cuyamaca and took some pictures. Let's look at them as we discuss something I've been wanting to write for a long time.

The Past

When this blog began in 2012 I was in the very early days of learning to code. In fact I built this site back then exactly for the purpose of teaching myself how websites (and the internet) actually worked. Back then I was in school to become an Aerospace Engineer (a subject I never really liked) and I found myself both excelling at and being interested in web development.

For the next almost-decade I hopped between jobs, built several apps and services, joined a startup, founded more than one company, and worked as a contractor. Then, as go most stories in our present era: 2020 happened.

It was a very strange time. I was unemployed for a while with nothing but my side projects to occupy my time. I built a bunch of new things—and even wrote a book about how to build things. I was perhaps the most productive developer I've ever been. To this day there are times when I wax nostalgic for those days (just without the 2020-current-events part).

But when things finally settled down I found myself working full-time, finally, for my own company: SkyRocket Software, doing software contracting. I'm decent enough at the work that I suspect I'll be fine professionally for some time, but during all the 2020/21 craziness something changed. Not just with the world, but with me.

The Present

My thirties came and with that my interests began to shift dramatically. I still love coding and I still do it for a living, but it's not something I obsess over anymore. It's not something I want to do much outside of work (though this isn't universally true). I still run my own servers and all that, but I've stopped dreaming up new projects and instead I've begun to consider shutting most of my existing ones down. As time has gone on I've even found that I care less and less about tech in general. I still exist within this world, but I don't feel anymore that I live in it. As well I feel the industry has drifted away from the things I'm personally interested in—or I'm just getting cranky and old. ¯\_(ツ)_/¯

Either way something has changed.

These days I'm far more interested in reading, writing, and music than coding. I read more books now than I have at any point in my life and I write a lot as well (yes by hand). In part this change hasn't been reflected much in this blog because I haven't wanted to write this very post explaining the shift. This is that attempt.

I've spent a fair bit of time over these past four or five years now thinking about this experience I've had, and I've kept coming back to one particular method of describing what I mean when I say something has changed.

In short, when I think about myself and what I call myself that definition has changed. I used to say I was a software developer who did a bunch of other stuff too. These days, I don't much feel like a developer. I still am of course, but I don't see myself through that lens anymore. My self-conception has changed.

The Future

This blog has always been a general-purpose place where I could write about anything that interests me, but it's always had a technical bent. That is very likely to change in the months and years to come. I have a lot of things on my mind (and a few new projects I'm working on) and while some of those are tech-related, most are not—as such the subject matter on this blog is going to reflect that shift.¹

I hope everyone reading this will stick around and see what's coming. I'm very excited about the future and I'm excited to have a space here where I'll be more willing to write about the things I'm doing these days. It should be fun.

🚀

Recently, John Green announced a now-released podcast, The Universe hosted by himself and Dr. Katie Mack, which attempts to detail the entire history, present, and future of our universe as we currently understand it. While I haven't listened to the podcast beyond the trailer, I am confident it will be quite good. But that isn't really what I wanted to talk about here.

Instead I wanted to talk about how the trailer for that podcast reminded me of my own journey with the topic of Astronomy and Astrophysics.

As a teenager, I spent a fair bit of time in my school library doing one of three things:

1. Playing games on the library computers on whatever sites were as-of-yet undiscovered (read unblocked) by the IT admin.¹
2. Reading fantasy (see Ursula Le Guin's Earthsea Series).
3. Trying to read whatever I could to answer what I thought was a pretty simple question.

What is Time?

Why I was trying to answer this question is lot to my memory, but if I recall correctly the Doctor Who episode Blink came out around that time with its famous "wibbly wobbly" description of time. And Doctor Who was my jam at the time.

Einstein famously quipped: "time is what clocks measure". Which is delightfully unhelpful to the philosophically inclined.

To answer this question I read whatever I could find on the subject including Steven Hawking's A Brief History of Time, The Universe in a Nutshell, and others. Most of what I read went over my head as you'd expect for a teenager reading books on Theoretical Physics, but books like ABHoT lead me eventually to Astronomy. Perhaps if I was going to undertand Time, I needed to understand how it began.

Introduction to Stars, Galaxies, & the Universe

I don't remember exactly how, but eventually I stumbled on the medium of Podcasts and specifically a podcast by Professor Richard Pogge at Ohio State University called Astronomy 162. The podcast is effectively just the lectures for that class which Pogge describes as astronomy for non-science majors. In other words, it's an overview without all the math. A perfect find!

I listened to that podcast (and it's predecessor Astronomy 161) over and over again in order to extract the maximum amount of information from them. I can recall sitting in the backyard of my parent's house for hours devouring these episodes. In particular, the episode that fascinated me the most (and which is the subject of the first episode of The Universe) is titled, The First Three Minutes.

I'd recommend that episode today for anyone who is curious about astronomy. The whole series of lectures is very approachable and understandable to anyone, and but The First Three Minutes is what truly gripped me. As many of my friends know, I love things like that.

An Aside

I remember discovering other podcasts by similarly forward-thinking professors and those podcasts both indulged my curiosity and augmented my own college education later on.

The Internet serving to make more college-level content available to the general public has been one of the coolest things to see happen in recent decades. And it was through podcasting that I also discovered the excellent show Writing Excuses which taught me so much about the craft of writing and continues to be a show I go back to when I need help.

Bringing it Back

The release of The Universe by John Green and Dr. Katie Mack has put me in a bit of a reminiscent mood. I'm thrilled to see a new show attempt to capture the next generation of curious minds and I hope those who listen to it find what I did in Professor Pogge's podcast nearly two decades ago. That podcast was ultimately the reason I wanted to study Astrophysics in college—something I later abandoned to study Aerospace Engineering which turned out to be decidedly less interesting to me—and to this day I am still very interested in Astronomy and Physics.

In a way, perhaps I can credit Doctor Who, Professor Pogge, Steven Hawking, and being a 6th period Library Assistant for the path I eventually took and for kindling the ongoing desire for learning that I keep to this today.

I'm not sure, but it's been fun to think about. What I can be sure of is that podcasts are incredible.

I've been around long enough now to remember when we used different words to refer to our favorite apps and websites. Long before the words Platform, Service, or Ecosystem infected our discourse there were Tools. And I think we need fewer of the former and more of the latter.

The word User is also partly a holdout of this now bygone age. A User is anyone (even a dog) that uses the website or app. By traditional parlance, a User is the person being served by the software. On an Ecosystem, Service, or Platform though the User is likely not the Customer, they're often the Product.

A hammer is a tool and it is used for a purpose by a user—it's right there in the name. The user uses the tool to accomplish a goal in their lives and the tool exists to make the user's life easier in doing the thing they want to do. We used to think of software that way, but alas that mode of thought has long faded from popular conception. These days a User is more likely to be a person used by the software rather than someone who uses it to accomplish a goal. The software's goal is paramount, not the human's.

No doubt many of us out there still believe the old adage I've recited as:

A computer is a tool. I don't wait for a hammer, it waits for me. Hammers exist to make life easier not harder. Computers and software should be no different.

These days it is more likely for software to feel like an entertainment product, a movie theatre. The User then isn't a person trying to accomplish a goal, but is a captive audience who paid for one experience but gets a bunch of extra content thrown at them for the privilege of getting what they paid for. Though, thinking about it more, perhaps the better comparison is that of an airport. A user's every movement is tracked and monitored both for security and sales, everything is overpriced and cheaply-made, yet at the end of the day you still get where you need to go (assuming your bags make it and the plane has no loose bolts).

Perhaps this transformation is part of the increased scope and scale of modern software or perhaps it has to do with the professionalization and financialization of the industry. Maybe it's something else. I don't know.

Apps like NetNewsWire or Mastodon (just to name two examples) remind me of what we've lost. They're not trying to get me to adopt some platform. They're tools to help me get work done, like good software should.

My last post seemed to garner a fair amount of interest and so I figured it would be good to cover some of the unspecified particulars that I left out of the previous post.

On Writing Cursive Left-Handed

I mentioned in the previous post that I write using a fountain pen (and occasionally with a dip pen). Given that, it may come as a surprise to some, but I am left handed. And while yes, writing with a dip pen does occasionally leave me with blackened palms from ink-smear, I've found this to be increasingly rare. It was certainly the case early on, but it's actually rare these days. Most modern inks dry incredibly quickly (so long as the right amount of ink lands on the page). That said I do end up with lots of ink on my fingers whenever I change the cartridges on my fountain pen.

I've also found that the amount of ink that smears on my hand went down now that I write in cursive. That may seem strange as cursive uses a lot more ink than print, but I think it has to do with the extreme bias I tend to write with. As I'm left-handed, I also tilt the page the wrong way. This helps with ink smear because I'm almost always writing on already dry text.

I was taught to write in the Zaner-Bloser cursive style as a kid and that's what I picked up again last year. As with all forms of handwriting, I've simplified my take on the style to be what I now consider to be a mix of Zaner-Bloser and the New American style which incorporate several print letters for capitals (ridding me of the annoying Zaner-Bloser F, G, H, and its uninspired A). None of these tweaks were born of me trying to adopt a formal style, instead each came from me tweaking my own style which happened to converge and resemble those above.

As part of this quest of mine, I spent a fair amount of time researching the history of cursive and its various styles. I learned a lot about how different techniques evolved and how the style I most recognize as "fancy, old-style cursive" (i.e. the Palmer Method) lost the battle mid-century to the style that I was taught as a kid. I still think the Palmer Method is cooler looking, but I have no interest in breaking my brain again to learn it.

On the Gear

As this is partly a techie blog, I'd be remis if I didn't talk about the gear I use, though it's not particularly exciting.

I use a wide variety of notebooks (some. are. linked. here), though most of the time I just write on cheap legal pads.

As for the pens I use, my daily driver is a Schneider Base Uni. I have two of them (one stocked with black ink and the other with blue which I rarely use).

For my dip pen, I really like this ink. It's nothing special but it's cheap and works well.

Now What?

I continue to write a fair amount each day, and like I mentioned in my other post, that's largely because I find it fun to do. I've ended up performing a sort of near-nightly ritual where I write about something, anything just to feel the pen in my hand and hear the scratch of nib on paper. Sometimes it's (bad) poetry other times it's just stream-of-conscious rambling. Most of what comes out is uninspired or repetitive, but some aren't! Some are (dare I say it) even good!

Writing for me has become a new hobby, something I do to unwind and de-stress. That's not something I expected, but it is what happened.

I'm a software developer; I make my living on a computer. In this age there just isn't much reason for me to bother improving my handwriting.

I've thought that for years. While like most people my age, I learned to write in cursive in school (and to write in general) I'd essentially stopped using such an all-important skill in my daily life, save for the odd sticky-note here and reminder scribble there.

How It Began

I never wrote by hand. Why would I? You can't ⌘+f handwritten text, you can't change-control it, you can't even back it up easily. Why would you bother writing anything significant by hand at all?

An old programming teacher of mine once quipped:

People say that stuff on a computer isn't real, it's just digital. You can't touch it or save it. If the computer dies, it's gone. But I say they've got it backwards. Files on a computer are so much more real. I can prevent them from getting old, I can back them up for free, and I can send them to you without losing my original. On a computer, it's more real than it is on paper.

While I'm not sure I ever truly agreed with his point there, I did take that advice to heart for most of the past 12 years. In college, most of my work was digital already, and once school ended, I stopped writing by hand for all but a few rare tasks. I've always been a fan of writing in principle and many years ago I got a few fountain pens as a gift. Mostly they sat in a drawer, but occasionally I'd take them out and pine over how cool they were, then go about my day. I do keep a journal by hand, but writing in it has been an oft-forgot afterthought for most of my adult life.

Until last year.

How it Went

I've been working on a bit of a project now for over two years and part of that project has been writing a ton of notes for myself. I used to keep them in Notes.app on my Mac (since it syncs with all my devices), but after a while I got sort of disillusioned on doing that. As mentioned above, I live on my computer and it was starting to really get to me just how much time I spend at my desk, staring at a screen. So I took out a fresh notebook I'd been given years ago, and I started to take notes there instead.

I was hooked.

Within a few short weeks I had dozens of pages of notes, sketches, and musings in this little book (that was poorly made and now falling apart), so I dusted off the old fountain pens, bought some better notebooks on Amazon, and kept at it. I'd been taken in by writing. Writing by hand had so many downsides, but there were incredible upsides too. I no longer got distracted by notifications while writing. I loved flipping through my notes to review them. I rarely lost track of notes because I knew where they were in relation to other notes because human brains know where stuff is in space (mind-blowing, I know). I even got to talking to some friends about this new hobby of mine.

And that's when I went into overdrive. You see, a friend of mine gifted me a dip pen as a birthday present and I was supercharged.

One thing I've learned, that I find difficult to describe, is just how expressive it feels to write with pen on paper, even when you have nothing to write about. I was genuinely taken in by the sound of scritches of metal on paper. Over that first half of the year I wrote, by hand, over one hundred pages of text! That's likely more than I'd written in the past decade (and likely even further back) combined!

One thing still bothered me though: my terrible handwriting.

It's always been bad. It was bad when I was 5, 12, 20, and it's still bad. I knew it, everyone knew it. I never bothered to improve it either because:

1. I knew how to read it (most of the time).
2. Writing is something I rarely did, so why get better?
3. I'm lazy.

However, writing poorly didn't just mean my scribbles were sometimes difficult for me to read (and impossible for others to do so), it also meant something more important: my hand hurt when I wrote. My grip was too tight and my hand was unpracticed and sloppy. That meant I couldn't write for very long, even when I wanted to. What's more I've always romanticized writing (see the fountain pens above). I wanted to write better, but I didn't want it enough and I never really needed to. Now though, I was writing constantly and my bad penmanship was getting in my way.

It was time. I needed to revisit cursive.

How It's Going

I hadn't written in cursive since second grade. I stopped doing it the moment it was no longer required and I never looked back — until last October.

I can confidently say that I've written more cursive in the past four months than I have in my whole life before, combined. Easily. Over two hundred pages.

And it's great.

I'm by no means good at it, but I'm improving. I started with just trying to remember the letters. I have to imagine this is how learning to type on a DVORAK keyboard feels: your brain just breaks for a while. I even used those cursive worksheets they give to kids (with custom text of course).

After that I graduated to journaling in cursive and later writing in general in cursive. After four months, I can say that almost everything I write these days is in cursive by default. In fact I really only print when I need to scribble down a quick sticky-note or reminder (exactly as I did before all this).

My handwriting is by no means good, but I think my cursive is better than my print (especially if you take into account how long I've been practicing the latter vs. the former).

I've fallen in love with writing by hand. It's something I've always wanted to do well, but it took until now for me to set aside the time and effort to do it. I only regret waiting this long.

In what seems to be a new annual tradition, I'd like to discuss my favorite books from 2023. While I didn't achieve the same level of reading as 2022, last year was still one of the most productive reading years I've ever had—18 in total! You can see the totals of the books on my reading progress page, which at time of writing will look off since it doesn't count audiobooks in the counts (something I need to fix).

As a side note, I am curious if I can somehow automate the posting of recommended/recent books on that page. An interesting idea 🤔

In no particular order here are the books I recommend from last year:

The Book of Hidden Things, Francesco Dimitri

Francesco is quickly becoming my favorite author. I read Never the Wind last year and fell in love with his writing, and The Book of Hidden Things is more of the same excellent storytelling. Seriously, check him out.

Also he has a new book coming out this year, which I have already pre-ordered.

The Ocean at the End of the Lane, Neil Gaiman

This one was a really fun read. Neil Gaiman is a master and it shows. Really enjoyed this one, but I don't have much to say about it. A lovely story.

The Internet Newspaper, Adam Gnade

The cover hooked me immediately when I picked this one up on a whim from a local bookstore. The story takes place in San Diego (the author's and my hometown). If you like San Diego, 90's/00's music, and early 20-somethings mischief you'll like this. Also I love the size of this thing. Pocket novels are a lost medium.

The Secrets of Alchemy, Lawrence Principe

The book is an overview of the history of the practices of alchemy (mostly attempts to create a philosopher's stone and turn lead to gold), though it touches on the other uses for alchemy including metallurgy, medicine, and pre-modern chemistry. However the book really hits it out of the park when Principe does the homework and attempts to replicate several of the ancient alchemical recipes! Instantly I was hooked.

I read a lot of ancient history and one things I always try to keep in mind is that while ancient worldviews are very different from our own, and scientific thinking as we know it today didn't exist yet, that doesn't mean that people weren't experimenting, verifying, and sharing their ideas. For years people dismissed old alchemical recipes because we know their goal (chemically turn lead to gold) to be impossible, but that doesn't mean their experiments were useless, as Principe discovers. Some were very nuanced and precise. 10/10, fascinating read. Highly recommended.

It's available to rent for free on the Internet Archive, so please check it out.

Beren and Luthien, J.R.R. Tolkein

I read a lot of Tolkien this year and this one was my favorite. It's a little happier than the excellent Children of Hurin and features some of my favorite scenes from The Silmarillion. Basically if you enjoyed that chapter with the same name in The Silmarillion, you'll love this.

My favorite part: the song battle between a demi-god and an elf lord. 10/10. No notes.

As I am want to do, I spent an evening a while back putting together a Python version of John Conway's Game of Life: a "zero-player game" in which the board state progresses automatically according to a set of rules.

You can find the code for the game here. It requires just the Python standard library to run, so its easy to get started assuming you have Python >=3.10 installed. (It might run on lower than that but I didn't test it.)

The initial board state for the game is randomized so each run-through is unique. Some end immediately, so you may have to run the script several times to see something cool happen.

Because it's written in Python, the game is single threaded and quite lacking in raw performance, but because it's written in Python the game was also incredibly easy to put together—taking about three hours in total including debugging, though I did spend three more hours optimizing the render performance for no reason.

As with most of my recent script projects, I don't really have a point or reason for doing it other than to have some fun building toy projects. Like I've said before:

Scripts are programming candy whereas app development is the real meat and potatoes. In a script you can take shortcuts, be a bit messy, and forgo worrying about the complexities of large software.
- Take A Break, Script Something

In that same time I've also thrown together a few little scripts to visualize the chaos of the Collatz Conjecture and simulate the dynamics of 2-D gas expansion. Are these important? Nope, not at all. Were they fun? Yes, yes they were.

For all of the good things about Mastodon and the Fediverse more broadly, the technology has so far struggled to spread far outside of tech circles and into the broader public. Mastodon especially is pretty big these days, but its mostly filled with a lot of computer nerds like myself; there really isn't a very large music scene like there is on other social media sites.

I wanted to help fix that, so I threw together a simple Mastodon bot (a toot bot?) that regularly shares information about local music shows in San Diego.

With the San Diego Reader as its data source, the bot posts every hour about shows happening in the next day or two. Hopefully it will help my fellow San Diegans keep tabs on the music scene, and help to inspire more musicians and local music fans onto the platform.

So if you're at all interested in the San Diego music scene, be sure to follow sdmusiceventsbot on Mastodon! And if you're a local artist, make sure you add your event to the Reader to get yourself noticed by the bot! Be warned, San Diego has a lot of shows (more than the bot could ever reasonably post without looking like spam), so not all shows will get posted. The choices are random, so it's all up to the luck of the dice.