It's Still Difficult
There is more tooling and abstraction than ever, and shipping code is still difficult.
What I'm Writing
GitOps Superpowers: Now your code changes can show up on your cloud bill! Reflections on the past few years using Git to drive systems.
A long thread on cloud native data science and engineering: I'll be continuing to update this for awhile I think.
In the Buffer
K8s Storage Provider Benchmarks: This epic series continues to tickle my heartstrings. The cloud native storage projects are trying to solve a critical problem, but the end user usually has to pick a solution by the numbers.
Multi-Task Robotic Reinforcement Learning at Scale: It feels almost akin to applying the lessons of large language models to behavior (and they do explain that they're seeking the same economies of scale). By training on diverse tasks in the same job, the models can learn to do multiple skills well - even outperforming on some of the tasks where data is less plentiful. And then, by relabeling existing data, the models can learn to accomplish novel goals (see image).
Game Theory as an Engine for Large-scale Data Analysis: Components as players in a game, with discrete utility functions, allowing for distributed and scalable PCA. The potential applications to other problem domains are tantalizing.
Construction Costs Around The World: On the physical infrastructure side of the house, some compelling analysis by Brian Potter on the hyper-local economics of construction. Why is it so expensive to build in the United States?
I'm not complaining, but...
As contemporary developers, what a time it is to be alive.
We have modern operating systems, which lets us abstract away from system calls.
We have package repositories with hundreds of thousands of tools to choose from.
We have higher level languages that massively increase our productivity.
Our machines are more powerful than ever, and they tend not to use cards, tapes, or HDDs anymore.
There are now so many ways for me to create, from interactive notebooks like Jupyter, completely new takes on developer experience like Replit, No Code platforms like Webflow, and modern IDEs like VSCode with huge plugin ecosystems.
There's even ML tools like DeepCode, CodeGuru, and Tabnine to help me write code faster and better.
Standards, technical specifications, norms, best practices, and reference implementations are all readily available to me from such trusted organizations as the IEEE, IETF, CNCF, ECMA, and ACM among others.
Despite all this progress, it can still be hard to ship in 2021.
Rough Edges
We can't talk about all the rough edges, but there are several that I want to call out here as "still tough":
AAA and SSO: Arguably, this one is the worst. In 2021 it's still a PITA to deal with authentication, authorization, audit, and SSO. The easiest advice I ever give is to solve this problem in exactly one place throughout the organization, for each language/framework you use. I badly want a one-liner that just adds baseline AAA/SSO to my app, and arguably that's the sort of thing enterprise should be good at solving centrally. Instead this actually can become harder to accomplish in an enterprise for all kinds of least-common-denominator reasons.
Encryption: Even just setting up TLS is still nontrivial in 2021 for some architectures. Can I encrypt my Kubernetes secrets? Do I need SOPS? How often should I rotate my keys? What's the right algorithm for this?
Money: Stripe might be the next megacorp just on the basis of making an API for money.
~~a11y~~: I looked into it and it seems like there are good a11y resources now. There's just a perception that it's still technically difficult. IBM Equal Access Toolkit looks neat.
DNS: It's always DNS. DNS changes seemingly scare everyone. They can take a long time to propagate, they can be confusing, and it always feels like undifferentiated heavy lifting.
So what?
I should note is that developer platforms and PaaS products like Heroku, Vercel, Replit, and Netlify hide many of these problems away from you very nicely. And Stripe is a great API for payments. They solve hard problems and make it easy for developers to move on to focus on the business. What I hope and wonder, is whether these problems will ever be abstracted away, the same way syscalls are abstracted away by Linux.