Using Prometheus-operator with Minikube

In this article I will describe how to install Prometheus-operator into Minikube for evaluation and testing purposes. Minikube is a tiny flavor of Kubernetes and some of Prometheus exporters need special tuning to run under Minikube.

My Minikube version:

$ minikube version
minikube version: v1.2.0

First, let’s install Prometheus-operator using Helm:

$ helm fetch --untar stable/prometheus-operator
$ cd prometheus-operator

Now we need to adapt values.yaml for Minikube environment. First, let’s disable https scraping for kubelet:

kubelet:
  enabled: true
  namespace: kube-system
  serviceMonitor:
    interval: ""
    https: false

This will make Prometheus alert saying “Kubelet target is down” go away. Easy fix.

The next issue is that Prometheus cannot scrape Etcd cluster’s metrics. Issue actually consists of two independent issues. First, due to security constraints Etcd cluster (although running in pod) is not available from pods network. It is only available using node (Minikube VM) IP address. So first, we must obtain Minikube VM IP:

$ minikube ip
192.168.99.100

Then we must explicitly define this endpoint in values.yaml:

kubeEtcd:
  enabled: true
  endpoints:
  - 192.168.99.100

Second issue is that Etcd listens on HTTPS with client SSL authentication. That means that we must provide Prometheus with set of certificates to be able to talk to /metrics endpoint of Etcd. Those certificates are stored on Minikube VM under /var/lib/minikube/certs/etcd directory. We must download those certificates and store them as Kubernetes secret.

$ minikube ssh -- sudo cat /var/lib/minikube/certs/etcd/ca.crt > ca.crt
$ minikube ssh -- sudo cat /var/lib/minikube/certs/etcd/healthcheck-client.key > healthcheck-client.key
$ minikube ssh -- sudo cat /var/lib/minikube/certs/etcd/healthcheck-client.crt > healthcheck-client.crt
$ kubectl create secret generic etcd-certs --from-file healthcheck-client.crt --from-file healthcheck-client.key --from-file ca.crt

We then must specify created secret in our values.yaml :

prometheusSpec:
  secrets:
  - etcd-certs

And finally specify those certificates underkubeEtcd key:

kubeEtcd:
  enabled: true
  endpoints:
  - 192.168.99.100
  service:
    port: 2379
    targetPort: 2379
    selector:
      component: etcd
  serviceMonitor:
    scheme: https
    insecureSkipVerify: false
    serverName: localhost
    caFile: /etc/prometheus/secrets/etcd-certs/ca.crt
    certFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.crt
    keyFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.key

One more highly desired thing is to expose Prometheus dashboard using NodePort service:

prometheus:
  service:
    type: NodePort

And the last step is obviously to install Prometheus-operator with all our fixes:

helm install . --name prometheus -f values.yaml

After that we can go to Prometheus dashboard. To find out where it is located we may use minikube service list command. In my case it was http://192.168.99.100:30090. So we go to Prometheus -> Targets and here is the picture we may see:

It shows us that all the targets are up. Another way to verify is to use PromQL query:

It also shows that all scrape jobs are running without issues.

Thanks for reading!