Over our combined many years of working with Kubernetes, me and Márk have seen some weird things. Many resulting in us scratching our head and going "WTF?". Regardless of if you're brand new to the cloud native world or you're a Kubernetes veteran I hope you'll find something interesting in this post - or at the very least a shared frustration! 😁
With KubeCon just 1 week away I thought it would be good to share some of my tips and suggestions on how to make the most of the week without burning out. I've been to a few KubeCon's now, both EU and US, and I’ve learned a lot about how to make the most of the week and what not to do in order to protect your health!
For months now (maybe years, who knows 🤷) I've had the following error sprinkled throughout my Kubelet logs across multiple of my worker nodes:
StopPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to destroy network for sandbox "055b221e44a28ce8d9120f771d5e1ef201f2457ce49c58999a0147104cca2708": failed to get network "cbr0" cached result: decoding version from network config: unexpected end of JSON input
As there has never been any associated impact with this that I could see I just ignored it... for a long time!
Well, fast forward to last Friday night where a sudden hyper-focus hit and I decided I wanted to know what was going on and how to stop it - here's how it went.
I've been giving talks at meetups and conferences for a few years now. I started off after the encouragement of my friends giving their own talk and looking so cool doing it! It's taken a while but I think I'm at a stage now where I'm not only good at it (at least I hope so 😅) but I feel confident and comfortable while doing it. I want everyone to have that same confidence and I want to hear ALL OF YOU giving talks too! You have stories to tell, lessons to share and experience to pass on. So here is my learnings on how I approach giving a talk in front of a crowd of techies, mainly focussed on technical talks but most of this should apply to most public speaking.
I run several bots that post to social media and the number seems to keep growing. I've had a few people in the past ask me how I run them so I thought I'd finally get around to writing down how I manage it by leveraging Node-RED.
I have a habit of automating as much of my life as I can. As part of that I have some small automations that handle posting to social media automatically when I publish a new blog post. (More on this in a future post). In the past this was Twitter but that is dead so these days I post to my Mastodon and Bluesky accounts automatically whenever a new blog post is available on my website. Maybe that's how you ended up here today! One platform I had been avoiding for a long time was LinkedIn. I had the impression that they didn't have a free, personal API available for users to use to post status updates. Well, turns out I was mistaken and it's actually not too difficult to setup, there's just a LOT of outdated information out there. So lets fix this...
Last week I announced the launch of a new project I've been working on - Cloud Native Now - a new monthly newsletter that will provide a roundup of all the happenings in the cloud native world. This newsletter is my attempt at keeping myself, and others, up-to-date on all the latest news, tools and events happening in the cloud native world. A new issue will be published each month on the last Friday of that month and contain a roundup of articles, announcements, tools, tutorials, events and CFPs relating to cloud native technologies and the community.
As part of a side project I'm currently working on I needed to spin up a new Kubernetes cluster that I could manage via GitOps. I decided to take this opportunity to take a look at OpenTofu and see how it handles as it's been several years now since I last used Terraform. My plan was to use OpenTofu to scaffold a fairly basic Civo Kubernetes cluster and then use Flux to handle installing workloads into the cluster. It took me a little trial-and-error so I thought I'd write up my final setup to help others avoid the issues and to help my future self when I come to do this again in a couple years!
Recently, a friend of mine asked me what resources I'd recommend to start learning about Kubernetes. He was a victim of the layoffs that seem to be so prevalent right now and has experience as a classic SysOps / SysAdmin engineer but no expose to Kubernetes yet and wanted to learn to help improve his job-hunting prospects.
I wasn't sure what to recommend at first, it's been a long time since I was learning Kubernetes for the first time and wasn't sure what was still useful and relevant but what follows is what I ended up sharing with him, and now with all of you.
Last weekend I finally went to my first ever EMF Camp after wanting to go for 4+ years now. I had a thoroughly good time while I was there and wanted to share my experience to encourage others to go in the future.
I'm quite a fan of Renovate for automating my dependancy updates. I've been using it quite extensively - at Giant Swarm, with my personal infrastructure and with k8s.social - to keep things up-to-date in the various Kubernetes clusters I manage. This has been working great for container images and Helm charts, with both being managed via GitOps and Renovate automating the version updates via PRs, but I have now found myself wanting to automate the updating of the Kubernetes version for my Civo cluster. Renovate doesn't have built in support for this but we can make it work with the use of custom datasources!
I’ve recently been seeing some stability issues with Redis that I have running for the k8s.social Mastodon instances. After looking into it I realised that I had it configured in a master/replica architecture but I wasn’t actually making any use of the replicas as Mastodon was configured to do everything via the master. There’s two things wrong with this - firstly I’m wasting resources by having the replicas running but more importantly I created a single point of failure. When the Redis master went down, so did Mastodon.
This post is based on a webinar i've previously given where I go through some of my favourite tips for working with Kubernetes clusters all day long. The goal of all of these techniques is to make my life easier and (hopefully) less error prone. I start off with the first 5 tips being applicable to anyone working with Kubernetes and can be picked up right away. From there I move on to a couple that would benefit from having some old-skool Linux sys-admin experience. Finally I finish of with some more advanced techniques that require some previous programming experience.
Generally, and by default, operators of the cluster are assigned to the cluster-admin ClusterRole. This gives the user access and permission to do all operations on all resources in the cluster. There's very good reason for this, an admin generally needs to be able to setup and manage the cluster, including the ability to define and assign roles. But what if we need to block an action performed by cluster admins? We can't do it with RBAC, it only allows for adding of permissions, not taking them away.
I was recently asked by a collegue at work if there are any resource I recommend with getting familiar with Go. It turned into quite a list so I thought i'd share it with everyone (and hopefully keep updating it) in the hopes that others will find it useful.
Docker has recently announced that Docker Desktop will soon require a subscription and, based on the size of your company, may require a paid subscription. (It remains free for personal use). There has been quite a bit of reaction to this news...
I’ve recently accepted a new job offer and wanted to share a little about my experience job hunting during a global pandemic. Even after more than a year it was clear that some companies still hadn’t adjusted or taken the time to stop and look at the approach they were taking and the effect it had on the interview candidate. Hopefully some of this will be of use to someone looking to hire during similar situations. Most of what’s to come is aimed at those doing the hiring rather than the hunting but I think it could be of benefit for both sides to be aware of what might come up.
I've been using Scaleway's Kapsule managed Kubernetes offering for my personal projects for a while now (this blog is running on it) so I was pretty excited when they announced a new managed Kubernetes offering dubbed Kosmos. What makes Kosmos really interesting is it's sold as a multi-cloud Kubernetes offering.
Ever since the first batch of Android phones were released in the UK I have been a fan. My first, the HTC Dream (also known as the T-Mobile G1), is still my all time favourite phone. The slide out physical keyboard was fantastic. I’ve had a whole range of different Android-based phones both high-end and low-end, some with quirks, some fairly standard, most rooted and tweaked to their limits.
But, in recent years, my enthusiasm for both Android and Google in general has waned. I have been slowly moving away from various Google services over the past couple years. Google search replace by DuckDuckGo, Gmail replaced with ProtonMail, Drive replaced with a self-hosted Nextcloud, etc. while also getting frustrated with the way most Android devices get abandoned soon after launch.
When the new iPhone 12 was released I decided to give it a try (the blue does look very pretty). Rather than just settling for the new iPhone 12 I decided to also pick up an iPad Air for the family (what I’m currently writing this post on).
It’s been at least a couple weeks now since I made the switch so I thought I’d collect some of my thoughts on how I’m finding the change so that maybe it’d be of use to others.