Rabbit holes
I disappear from time to time. No, not in the physical sense. Rabbit holes are everywhere. I follow them and emerge a couple of days (or weeks) later, hopefully wiser.
For illustration, I wandered through Docker land for a whole 6 months. That particular rabbit hole went deep. The weird and wonderful container landscape is still maturing and changing fast. Below the surface of a seemingly simple virtualisation technology lies a myriad of tools, concepts and philosophies. We have built out a single, auto-scaling CI grid that runs in containers, using other containers to build new container images, spinning up containers from those newly built images and testing them before publishing them. It’s turtles all the way down.
Other than the standard unit and integration testing, we have to add static code analysis, application vulnerability analysis and image vulnerability analysis before an image even makes it to a private registry. Then there are production considerations - orchestration with Rancher and Kubernetes (amongst other flavours), vulnerability analysis of production images and network security scans, scaling across geographic locations with redundancy, logging infrastructure, monitoring and reporting. Inside environments you can have containers that do nothing but provide services for other containers - whether that be providing data volumes by being a data container or providing signals for routing and scheduling through labels. Yes, it’s possible to build a container that is less than 1kb for those use cases. It’s mind blowing. “The Docker way” is an enabler for micro-services and has changed the way I architect applications. It’s difficult to explain until you’ve chosen the red pill.
You take the blue pill - the story ends, you wake up in your bed and believe whatever you want to believe. You take the red pill - you stay in Wonderland, and I show you how deep the rabbit hole goes. Remember: all I’m offering is the truth. Nothing more.
Morpheus, The Matrix
Sometimes following rabbit holes lead to procrastination. I haven’t posted to this blog in a long time because I wanted to:
- change my blog from Octopress to pure Jekyll
- convert all my old posts from HTML to Markdown
- migrate my blog to run on Docker in AWS
- recreate my automated deployment workflow
- set up SSL using Let’s Encrypt
It took me a while to get around to it. I also refused to post until I could do my writing on a mobile device:
Soon after, I became frustrated with the limitations of Jekyll. I found the excuse I was looking for to try out Gatsby.js. After all, by now I have stopped treating JavaScript like a toy language and delved deep into it. I got my hands dirty with React and had GraphQL on my todo list. Now, on top of linting my JavaScript code my workflow incorporates a linter for Markdown, spelling and prose. There is no reason not to treat your blog like you would any other production application.
In preparation for future posts I wrote a Gatsby plugin called gatsby-remark-draw to render static diagrams using SvgBob. I proceeded down the rabbit hole and added Graphviz and Mermaid.js support as well. Combine it with React hot loader and DrawIt and you can see your ASCII drawings come to life as you hit save. It’s a beautiful thing.
Faster and faster
Technology is moving at an incredible pace. This is evident in the plethora of new languages, frameworks and paradigms we are coming up with. Not only are we working with new tools but we need to adapt to entire new ways of thinking and reasoning about solutions. The pace of innovation is accelerating and there is no sign of it dissipating.
We’ve seen an entire transformation in web development over the last decade. JavaScript flourished as browsers matured aided by the increase of computing power in the hands of developers and consumers of software. Transpiling is now commonplace where we have SASS, LESS, CoffeeScript, TypeScript, and others. Tools like Babel allows us to use new JavaScript features by transpiling to older versions of the language. We can tie everything together with webpack to hide some of the inherit complexities in polyglot programming and large scale application development. We are challenging conventional wisdom with Atomic CSS, CSS in JavaScript and Component Styles.
And then, the uber frameworks. Angular.js and React have completely changed the way we build user interfaces and handle data and user interaction. Node.js brought JavaScript to the server side and the desktop. It was a humbling experience back in the day when I first ran a proper command line utility on my machine that was pure JavaScript.
The thread is worth reading.
The “big data” revolution has spawned an industry by itself where open source solutions became ubiquitous. Data collection, analysis and prediction tools became accessible to the masses. General use databases are now commodity. PostgreSQL, MySql, MongoDB, Riak, Redis and Cassandra (to mention a few) gained widespread adoption.
I have been around for long enough to witness the server revolution first-hand:
These are only some examples. Don’t get me started on languages.
The list goes on and on. Novelty only lasts so long before it becomes generalised. There are so… …many… …shiny… …things. It’s exhausting and exhilarating at the same time.
I am not surprised that developers feel overwhelmed. Building a modern web application requires knowledge of a thousand different frameworks, tools and languages. Take a closer look at the dependencies of the create-react-app application to see what I mean. Staying on top of it all is hard for anyone.
Nobody knows everything
Let us deal with a common theme in technology community - imposter syndrome. Scott’s post from 2011 still resonates with me. In particular, Scott quotes Dave Ward:
I think the more you know, the more you realize just how much you don’t know. So paradoxically, the deeper down the rabbit hole you go, the more you might tend to fixate on the growing collection of unlearned peripheral concepts that you become conscious of along the way.
That can manifest itself as feelings of fraudulence when people are calling you a “guru” or “expert” while you’re internally overwhelmed by the ever-expanding volumes of things you’re learning that you don’t know.
However, I think it’s important to tamp those insecurities down and continue on with confidence enough to continue learning. After all, you’ve got the advantage of having this long list of things you know you don’t know, whereas most people haven’t even taken the time to uncover that treasure map yet. What’s more, no one else has it all figured out either. We’re all just fumbling around in the adjacent possible, grasping at whatever good ideas and understanding we can manage to wrap our heads around.
The first part is an adaption from a quote widely attributed to Albert Einstein but Dave adds beautiful context to it. That statement made a profound impact on my life, my career, and my self-image.
The more I learn, the more I realize how much I don’t know.
Albert Einstein
I cannot stress this point enough. Nobody knows everything. People who boast about their knowledge and abilities are unaware of what they don’t know. It’s healthy to keep your skills up to date and increase your knowledge, but comparing yourself to others will only lead to increased anxiety. Attempting to know more than everyone else is an exercise in futility.
Here is what is more important. Have confidence in what you do know while keeping to the spirit of “strong opinions, weakly held”. Make a decent attempt at knowing what you don’t know.
In a world where there are too many things and too little time, focus on your ability to digest and process information. Let go of what is not useful any more.
In this industry, if you are not moving forward you are moving backwards. Become good at learning and unlearning.
See how to do that in the follow up post - The Magpie.
Photo by Glenn Carstens-Peters on Unsplash