Change Service Network in running OpenShift 4 cluster

Use case: we want to change serviceNetwork in running OpenShift 4.5 cluster from 172.25.0.0/16 to 172.30.0.0/16 without recreating it from scratch. ServiceNetwork as it name stands is used by Services, not Pods. It is important to add that this operation is certainly disruptive and causes some amount of downtime for services depending on the size of your cluster.

There is no Red Hat’s documentation on how to that. Officially changing network configuration of running cluster is not possible, see https://access.redhat.com/solutions/4873171.

However, changing serviceNetwork in OpenShift 4 can be done in 4 steps:

1. Create new TLS certificate for kube-apiserver with its new clusterIP address (for service default/kubernetes). Remember that this service always uses the first IP address from serviceNetwork. In our case its IP will change from 172.25.0.1 to 172.30.0.1. We will create TLS certificate with both old and new addresses to ensure kube-apiserver will continue to be available in case something goes really wrong.
2. Change serviceNetwork using network operator.
3. Recreate ALL services to force their IP address change.
4. Update MachineConfigs with new DNS IP address.

Create new TLS certificate for kube-apiserver

Export the certificate for internal OpenShift CA.

oc -n openshift-kube-apiserver-operator get secret service-network-serving-signer -o=jsonpath='{.data.tls\.crt}' | base64 -d > ca.crt
oc -n openshift-kube-apiserver-operator get secret service-network-serving-signer -o=jsonpath='{.data.tls\.key}' | base64 -d > ca.key

Export the certificate for kube-apiserver.

oc -n openshift-kube-apiserver get secret service-network-serving-certkey -o=jsonpath='{.data.tls\.crt}' | base64 -d > tls.crt
oc -n openshift-kube-apiserver get secret service-network-serving-certkey -o=jsonpath='{.data.tls\.key}' | base64 -d > tls.key

Inspect the apiserver certificate, make note of its X509v3 Subject Alternative Name field. You will need that for the next step.

openssl x509 -text -noout -in tls.crt

Create options file for openssl. Make sure that alt_names section corresponds to those in your apiserver certificate. Add new IP address to alt_names and req_distinguished_namesection.

cat > ssl.conf <<EOF
[ req ]
default_bits       = 4096
distinguished_name = req_distinguished_name
req_extensions     = req_ext
prompt             = no
[ req_distinguished_name ]
commonName = 172.30.0.1
[ req_ext ]
subjectAltName = @alt_names
[alt_names]
DNS.1  = kubernetes
DNS.2  = kubernetes.default
DNS.3  = kubernetes.default.svc
DNS.4  = kubernetes.default.svc.cluster.local
DNS.5  = openshift
DNS.6  = openshift.default
DNS.7  = openshift.default.svc
DNS.8  = openshift.default.svc.cluster.local
DNS.9  = 172.25.0.1
DNS.10 = 172.30.0.1
IP.1   = 172.25.0.1
IP.2   = 172.30.0.1
EOF

Create new TLS certificate for apiserver.

openssl genrsa -out new.key 4096
openssl req -new -sha256 -out new.csr -key new.key -config ssl.conf
openssl x509 -req -sha256 -days 3650 -in new.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out new.crt -extensions req_ext -extfile ssl.conf

It can be seen that new certificate has the correct issuer and SANs.

$ openssl x509 -text -noout -in new.crt | grep Issuer
        Issuer: OU=openshift, CN=kube-apiserver-service-network-signer
$ openssl x509 -text -noout -in new.crt | grep -A1 "Subject Alternative Name"
            X509v3 Subject Alternative Name:
                DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:openshift, DNS:openshift.default, DNS:openshift.default.svc, DNS:openshift.default.svc.cluster.local, DNS:172.25.0.1, DNS:172.30.0.1, IP Address:172.25.0.1, IP Address:172.30.0.1

Insert new certificates into apiserver secret.

oc -n openshift-kube-apiserver patch secret service-network-serving-certkey --type='merge' --patch "{ \"data\": { \"tls.crt\": \"$(base64 -w0 new.crt)\", \"tls.key\": \"$(base64 -w0 new.key)\" } }"

Verify that apiserver is now working with new certificate (in this example my API endpoint is api.crc.testing:6443).

$ echo | openssl s_client -connect api.crc.testing:6443 2>/dev/null | grep subject
subject=/CN=172.30.0.1

Change serviceNetwork using network operator

First we need to tell network operator that we are changing the network settings, otherwise it will decline any modifications to serviceNetwork.

oc annotate network.operator cluster 'networkoperator.openshift.io/network-migration'=""

Change serviceNetwork CIDR.

oc patch network.config cluster --type='merge' --patch '{ "spec": { "serviceNetwork": ["172.30.0.0/16"] } }'

Network operator will restart kube-apiserver with new settings, there will be a brief downtime of kube API.

Recreate all services

Here comes the fun part. We need to recreate all services to force them to get new IP addresses.

Back up all services:

oc get svc -A -o yaml --export | sed '/clusterIP/d' > all_svc.yaml

Delete all services.

oc delete -f all_svc.yaml

Wait until OpenShift operators recreate system services and then create all other application related.

oc create -f all_svc.yaml

Don’t bother all those “services *** already exists“ errors — it’s normal in this case.

Watch OpenShift operators reaching normal state.

oc get co

Update MachineConfigs

Node MachineConfigs need to be updated in order to update kubelet clusterDNS setting (clusterDNS is always 10th IP address in serviceNetwork).

oc get machineconfig -o yaml | sed 's/172.25.0.10/172.30.0.10/g' | oc apply -f-

Another way to achieve the same is via KubeletConfig CR.

Watch nodes are being rebooted to apply new config.