March 1, 2020 · project kubernetes cloud

Evaluating Kubernetes IaaS offerings for personal projects

It's been almost a year since I first started using Kubernetes. After trying out Kubernetes on AWS Elastic Kubernetes Service, AWS provisioned with kops, Google Kubernetes Engine, and DigitalOcean Kubernetes Service, I have a couple of comments about each of them. All of these services are provisioned using Terraform. And just remember, opinions are like assholes, everyone's got one.

Let's get started

tl;dr use EKS for production, kops if you want to tweak all the knobs, and DOKS for your blog.

EKS

AWS Elastic Kubernetes Service (they finally gave up on insisting people call it Elastic Container Service for Kubernetes) is a great hands off way to provision a Kubernetes cluster on AWS. You pay $0.10 USD per hour per cluster for the AWS managed master nodes, and then any additional AWS resources created. It is fully supported with Terraform's AWS provider, and super quick to setup - only two Terraform resources needed to create the masters and node groups. You also get the benefit of being on AWS and use all the other services they offer.

kops

Kubernetes Operations (kops) describes itself as "the easiest way to get a production grade Kubernetes cluster up and running", and it's true. It is the easiest (the most popular and officially supported by Kubernetes) way to deploy a self-managed production grade Kubernetes. While it doesn't have direct Terraform support, kops resources are declaratively managed in YAML manifests similar to Kubernetes, and you can use kops toolbox template on the kops manifests with outputs from Terraform. Not a bad compromise.

Senior YAML Engineer, amirite?

Since this isn't managed service, you do have full access to the master nodes (they're just EC2 instances). Downside is you have more knobs to turn, and more things to break. It is also the most expensive of the four options, a highly available setup with masters running on m5.large instances will cost you $0.183 USD per hour (that's reversed instance pricing too). A small cluster could easily cost hundreds of dollars a month. That said, it's got great documentation, a very active community, and of the two production clusters I've managed at two separate companies, I've never had any major issues with it. The provisioning and rolling upgrade process is very robust.

DOKS

DigitalOcean Kubernetes Service is a managed service similar to EKS, but with a much lower barrier to entry. You don't have to worry about VPCs, ASGs, deploying your own cluster autoscalers, etc. Like their other products, it is targeted to developers who don't want to do ops. Their cheapest node type runs shared CPU droplets which is by far the cheapest of all of the options here. But once you switch to their dedicated CPU droplets, it gets quite expensive, comparable to GCP instances. These are just based on the numbers on the page though, I haven't done any benchmarking of the different instances' performance. The easy of use does come at a cost of functionality. Unlike AWS or GCP, DigitalOcean does not have NAT gateways and more advanced networking configuration (like subnets, custom routing tables, private networking, IP based firewalling on the load balancer). The master node is also public only, which may be an issue for those who are more security conscious. Being the newer (and smallest) kid on the block, there are a few issues that are definitely deal breakers for production. Bottom line is it's good enough for a hobby project.

GKE

Google Kubernetes Engine is the home of Kubernetes. I don't have much to say about it, other than GCP as a whole isn't well supported by Terraform, and my crude calculations show that GKE should be cheaper than AWS. I originally ran my own services on GKE, a private master, two node GKE cluster, with a iptables-based NAT instance, SSH bastion and VPN, and a load balancer cost me around $80 USD.

But do you need Kubernetes?

Is this your Wordpress site running in Kubernetes?

You could definitely get away with docker-compose for a lot of personal projects. Docker Swarm was something I experimented before settling with Kubernetes. The smaller ecosystem of Docker Swarm was the biggest downside, when you google a bug or problem you run into with Docker Swarm you'll see many posts with the sentiment of "we started with Docker Swarm, but then we switched to Kubernetes". Also with Mirantis' recent acquisition of Docker EE, Docker Swarm's future is up the air. But with low-cost and low-barrier options like DOKS, there's not many reasons not to use Kubernetes.