ETCD cluster won't start and fails with "request sent was ignored by remote peer due to cluster ID mismatch"
Scenario
Starting your etcd cluster throws an error like:
{"level":"warn","ts":"2022-03-09T01:06:57.767Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"99fbb86961c11d8f","remote-peer-cluster-id":"d0d0a4fb77ca1d5f","local-member-id":"5892b00848500dd8","local-member-cluster-id":"464487e7a78e118c","error":"cluster ID mismatch"}
Common Causes
- Kubernetes PVC/PV contain older configuration or cluster information and you’re trying to start with
initialClusterState: new. - A pod in the cluster failed and the rescheduled one tries to add itself as a member with
ETCD_INITIAL_CLUSTER_STATEset tonew. This will not work.
Solution
- This is a common problem with Bitnami’s Helm Chart. Be sure you follow the proper instructions when adding new replicas or upgrading to a newer version of the Helm chart as
helm installandhelm upgradebehave differently. Otherwise, you need to wipe your PVC/PV to successfully start a new etcd cluster. MORE INFO - Right after deployment of the cluster with
ETCD_INITIAL_CLUSTER_STATE: "new", change it toETCD_INITIAL_CLUSTER_STATE: "existing"and runhelm upgrade.