What I've been working on

The thing that really clicks with me about programming is the puzzle-like nature of pretty much any project. Having a challenge in front of me and then "cracking it" is a feeling that is kinda hard to beat. I previously wrote <link here> about a small, fun project I played with and how finishing it made me feel like "<quote here>".

I've recently heard this notion encapsulated by the term "meaningful obstacle", and I reckon this is exactly why I do what I do both for a living (coding) and for fun (tinkering with stuff, code included).

Ever since I started playing around with self-hosting my own stuff, I found myself not only starting a series of small and medium personal projects, but motivated to finish the projects I start. And it's been a real cool learning experience too.

With that in mind, I thought I'd share a handful of "meaningful obstacles" (at various levels of doneness and not in any particular order of importance) that occupied my time recently, and some of the things I've learned along the way.

Linux 

Linux is the backbone of the internet. And so, when I've decided that "knowing my tools" was a priority, then what  followed was the need to always be as close as possible to where we deploy code to.

Over the last couple of months, I revived (a second time) an old MacBook that, back in 2019 I installed Ubuntu Linux on during my first mission to revive it. By old I mean one of those white plastic case ones that would be basically HORRENDOUS to use these days. It's honestly very usable with the new setup, albeit a bit slow for certain things.

But the performance is more than enough for the (new) purpose I've given this machine: hosting a few websites (including this blog and testing sites for clients). It also hosts a minimal, yet powerful, backend where I author the blog, and other services for personal and professional use.

My dev environment

The other side of the same coin. Being intimately familiar with the tools I use as a dev — my IDE, debuggers, the shell, the file system, etc — and completely in control of their behavior is also paramount to me.

Whatever drives this obviously stems from the same place where the puzzle cracker in me lives. But, the more I think about it, the more I think there is to it: it's about being intentional with what I do when I code.

Sure, there are effiency gains. One can move quite fast when their environment is perfectly set up, LSP is offering great auto-completion suggestions, and documentation is at your fingertips.

But there's also an incredible amount of time spent on setting all these things up. I must've dedicated at least a  week or two worth of late nights just configuring neovim (btw) for my linking, and I've barely scratched the surface. Then there was zsh (the shell I use), alacritty (terminal emulator), git and Lazygit (dont sleep on Lazygit!), tmux (multiplexer). And that was only on my Apple laptop. There's a lot more going on in Ubuntu. 

It's almost like I'm arguing against having your custom enviroment. But I'm not. It's all about being very much in control on the occasions I'm building something. Yes, mastering my tools takes time, but it's so worth it even before I fully get there. And I even made a repo out of it, the famous .dotfiles, where I have version-controlled config files that make my environment reproducible if I ever need to set up a new computer.

SQL

This one is more a product of my current job situation (I work on enterprise data migration these days. Sounds fun, I know) than anything else. But it's among the most important things I dove into recently.

SQL underpins the vast majority of all applications we all use, particularly on the web or our electronic devices. Anything on a computer or phone is very likely to be running at least an instance of SQLite — which I also have come to appreciate for it's simplicity over more powerful flavors of SQL — if not relying on external data sources which are, very likely, relational databases.

Writing lots and lots of SQL against enormous databases with hundreds of tables, complex relationships accross millions of records, and different architectures made me a much more well-rounded developer. I haven't seriously worked on a front end project in about 6 months but, through some sort of learning-more-SQL-butterfly-effect, I now get how websites work much better.

And, hey, don't take it from me. Take it from this guy who thinks SQL skills will "pay dividends until the end of time." 

Deploying a spider

The project itself is still ~top-secret~, but I've invested a lot of time recently building a complete solutiont to scrape data from a certain website.

And putting all the pieces together was a lot of fun: from thinking about how to structure the code to connecting a bunch of different services together. Now I have a deployed Scrapy spider project, the Scrapyd daemon to basically work as an engine to trigger scrapes, and Scrapyd Web to give me an interface to schedule and monitor all scraping jobs.

And this process taught me BIG lessons about containerization (all hail docker compose), the inner workings of ASP websites, keeping good version control habits, and my ability to develop something of this scale one my own.

Leaning towards simplicity

This last sentence about working solo on a big project is tightly connected to the idea of simplicity. I have forever been guilty of over-engineering projects (or pretty much anything in my life) from the moment of inception, and then never following through because... well, grandiose ideas tend to be overwhelming and scary.

But three things have been really helpful in the last couple of months building this big spider:

Breaking things down into achievable chunks

"Oh, Agile, you mean...". Whatever you wanna call it. There's no methodology behind this point. And what even is the point of getting all theoretical about SLDC when you're working on your own....

But having small targets for every couple of days really kept my motivation high and the code base moving forward.

Aiming for what "just works" instead of the "perfect solution"

It was a step in the right direction battling the over-engineering-induced stage-fright. But I came to terms with the fact that I'd rather have an imperfect project running than feel like a failure for not making it happen.

Keeping a daily(-ish) project log

This may have been the glue that held it all together. Every day I did work on the project I made an effort to 1- keep a tidy commit history and write decent commit messages, and, more importantly, 2- write a few paragraphs about what I've done, what blocks/challenges I had, and what are potential next steps.

It was a game changer for me. And the reason I think it helped is that articulating on the regular what's on my mind, what's happening with the code base, and/or what needs doing made my vision of the project super clear. It also reduced the friction of context-switching when I came back to the project each night.

I used pjs (a cool TUI, terminal user interface, I randomly learned about on YouTube) as a way to dump my thoughts somewhere I could easily refer back to. I also kept a few hand-written notes and checklists that I created couple of days or every week to track the chunks of work I wanted to get done in a reasonable time frame.