A plain-language breakdown of what ZooKeeper and etcd are, why systems like HBase and Kubernetes need them, how leader election actually works, and whether a web/backend developer needs them — written from the perspective of an app developer, not a distributed-systems builder.

1. Why coordination systems exist at all

When you go from one machine to many machines, new problems appear that don't exist on a single box:

• Who is the leader/coordinator?
• Which nodes are currently alive?
• How do all nodes agree on shared config?
• How do two processes on different machines take turns on a shared resource?

A single file or database can't answer these reliably during network failures. That gap is what coordination services (ZooKeeper, etcd, Consul) fill.

   SINGLE NODE (e.g. a cron + SQLite app)      DISTRIBUTED SYSTEM
   -------------------------                   --------------------------
   cron IS the coordinator.                    Something must decide leader,
   No coordination needed.                     membership, locks, config.

2. What a coordination service is

Both ZooKeeper and etcd are the same category of tool:

• A consistent, replicated key-value store
• Kept in agreement by a consensus algorithm (ZAB for ZooKeeper, Raft for etcd)
• Fault-tolerant via quorum (a majority must agree)
• Support auto-expiring entries (ephemeral nodes / leases) that vanish when a client dies
• Support watches — clients subscribe to a key and get notified on change

Used for: leader election, distributed locks, config distribution, membership/service discovery.

Key mechanisms:

• Ephemeral node / lease — tied to a client session/TTL; auto-deleted if the client disconnects. Basis of "who's alive" and locks.
• Watch — instant notification on change. Basis of config propagation and failover.
• Quorum consensus — a write commits only when a majority agrees, so the cluster stays consistent through failures.

3. Coordinator vs. Manager (an important distinction)

ZooKeeper/etcd are NOT node managers. They are the source of truth (a shared brain). They store state and notify on change — they never start/stop your processes.

Analogy: ZooKeeper is the air traffic control tower (knows every plane's position, tells the truth, doesn't fly planes). The managers (Kubernetes, YARN NodeManager, Nomad) are the pilots/ground crew (actually move the planes, ask the tower for truth). The manager usually sits on top of a coordination service.

4. Who uses ZooKeeper

Systems built on it: Kafka (older versions), HBase, Hadoop HA (HDFS/YARN), SolrCloud, Pulsar, Storm, ClickHouse Keeper, NiFi, Druid, Mesos.

Trend: usage is declining — systems are designing it out:

   THEN                    ->  NOW
   Kafka + ZooKeeper       ->  Kafka + KRaft (Raft built-in, ZK removed)
   Coordination clusters   ->  etcd (Kubernetes standard)
   ClickHouse + ZooKeeper  ->  ClickHouse Keeper (ZK-compatible, no JVM)
   Service discovery        ->  Consul / etcd / built-in mesh

5. Why HBase needs ZooKeeper

HBase is a distributed system by design — even on one machine it's architecturally a cluster (HMaster(s) + RegionServers + regions). What ZooKeeper does for HBase:

1. HMaster election — decide the active master; fail over to standby.
2. RegionServer liveness — each RS registers an ephemeral node; crash → node vanishes → HMaster reassigns its regions.
3. Locating hbase:meta — clients ask ZooKeeper where the meta table lives.
4. Region assignment/state tracking.
5. Cluster-wide config (splits, replication, table state).

   client -> ZooKeeper: "Where is hbase:meta?"  -> "RegionServer X"
          -> meta table: "Which RS has my row?" -> "RegionServer Y"
          -> RegionServer Y: GET the row

Local standalone HBase bundles a ZooKeeper in the same JVM (HBASE_MANAGES_ZK=true by default). ZooKeeper listens on port 2181.

Honest take: using HBase as a single-node local DB means paying for lots of distributed machinery you don't benefit from. HBase shines at billions of rows / TB–PB scale with heavy random read/write across a cluster. For modest single-node needs, PostgreSQL / SQLite / RocksDB are far simpler.

6. HBase across multiple nodes — how ZooKeeper works

Set HBASE_MANAGES_ZK=false and run ZooKeeper as its own ensemble that all HMasters and RegionServers connect to.

Why odd numbers (3 or 5): need a majority (quorum). 3 → tolerate 1 failure; 5 → tolerate 2. Even numbers waste a node.

Failure handling:

• RegionServer dies → ephemeral node vanishes → HMaster (watching) reassigns its regions.
• Active HMaster dies → /hbase/master vanishes → standby grabs it → becomes active (automatic failover).
• One ZK node dies → 2 of 3 alive → quorum intact → HBase unaffected. (Lose majority → HBase halts to stay consistent.)

Operational gotchas: open the client/quorum/election ports between nodes; use private IPs/internal DNS (public IPs change); run NTP/chrony (drift breaks ZK); spread ZK nodes across different availability zones; give ZK its own disk (transaction-log latency matters).

7. How ZooKeeper decides the master (leader election)

ZooKeeper does not pick the "best" node. It provides a lock: only one client can create the ephemeral znode /hbase/master. Whoever succeeds is the master.

   HMaster-A -+  both try: create /hbase/master (ephemeral)
   HMaster-B -+
        A succeeds -> ACTIVE master
        B gets "NODE EXISTS" -> STANDBY, sets a WATCH on /hbase/master

Two properties make it work: ephemeral (the znode is tied to A's session; if A dies, ZK auto-deletes it) and watch (B is notified the instant it disappears). A refinement uses sequential ephemeral nodes (lowest number wins, each watches the one below) to avoid a "herd effect" stampede.

8. etcd in Kubernetes

etcd = Kubernetes' brain. A distributed key-value store (Raft consensus) that holds all cluster state — the single source of truth.

   kubectl / app -> kube-apiserver -> etcd
   (only the API server talks to etcd; scheduler, controllers, kubelets
    all go THROUGH the API server, never directly to etcd)

What's stored under /registry/: pods, deployments, services, configmaps, secrets, nodes, namespaces — every Kubernetes object. The reconciliation loop = compare desired vs actual (both in etcd), fix the diff. Watches make it feel live/self-healing.

HA/quorum (same concept as ZK): 3 members → tolerate 1 failure; 5 → tolerate 2. Lose quorum → cluster goes read-only. As an app developer you never touch etcd directly — you use kubectl/the API server.

9. etcd vs. ZooKeeper — same category, different flavor

   ZooKeeper: create ephemeral znode /leader -> others watch -> dies -> re-elect
   etcd:      create key with a LEASE /leader -> others watch -> lease expires -> re-elect

• Consensus — ZooKeeper: ZAB  |  etcd: Raft
• Auto-expire unit — ZooKeeper: ephemeral node (session)  |  etcd: lease (TTL)
• Data model — ZooKeeper: tree of znodes  |  etcd: flat key-value (prefixes)
• API — ZooKeeper: custom binary  |  etcd: gRPC / HTTP + JSON
• Language — ZooKeeper: Java (JVM)  |  etcd: Go
• Ecosystem — ZooKeeper: Hadoop, Kafka(old), HBase, Solr  |  etcd: Kubernetes, cloud-native/CNCF

Mental model: ZooKeeper : Hadoop/Java world ≈ etcd : Kubernetes/Go world (like Postgres vs MySQL — same category, different engine). Why etcd is winning: Raft is simpler than ZAB, gRPC/HTTP friendlier than ZK's binary protocol, Go fits cloud-native, and Kubernetes chose it.

10. Do you (a web/backend API developer) need these?

Almost certainly no — directly. A standard backend is stateless API servers + load balancer + database, and it already has coordination tools:

   "Which servers are up?"      -> load balancer health checks
   "Share state between them"   -> your database / Redis
   "Only one runs this job"     -> DB lock, Redis lock, or a cron/worker box
   "Config for all servers"     -> env vars, config service, AWS SSM
   "Route requests"             -> load balancer / API gateway

The one problem that tempts devs toward ZK/etcd — "run a scheduled job on only one of N servers" — does NOT need them. Use a database lock, a Redis lock (SET key NX EX 60), or a dedicated scheduler (EventBridge + Lambda, ECS scheduled task).

Distributing a secret (e.g. Rails master.key) to multiple servers — do you need etcd?

No. That's secret distribution, not cluster coordination. Use AWS Secrets Manager / SSM Parameter Store (each host fetches at boot via its IAM role), deploy-tool secrets (Kamal, Capistrano, CI secrets), or ansible-vault. Don't commit it to git, bake it into an image, stand up etcd/ZK for it, or scp it around by hand.

When would you use etcd directly?

Only when you are building a distributed system yourself that needs strongly-consistent live coordination across many nodes — leader election among your own replicas, dynamic service discovery, strongly-correct distributed locks, or live config that many nodes must see change instantly — and Redis/your DB can't cleanly do the job.

   Need to share state across servers?
     |-- Business data (users, orders)?           -> DATABASE
     |-- Cache / simple lock / rate limit?         -> REDIS
     |-- Static secret / config file?              -> Secrets Manager / SSM / env vars
     +-- LIVE, strongly-consistent coordination
         across many nodes you operate?            -> etcd (only here)

TL;DR

• ZooKeeper and etcd are the same category of tool: quorum-based distributed coordination services for leader election, locks, config, and membership.
• etcd is the modern, cloud-native successor (Raft, Go, powers Kubernetes); ZooKeeper is the battle-tested elder (ZAB, Java, powers HBase/Hadoop/old Kafka).
• Leader election in both = "one client grabs an auto-expiring key, others watch, re-elect when it disappears." The service guarantees exactly one winner.
• HBase needs ZooKeeper to elect its master, track region servers, and let clients bootstrap to hbase:meta — even standalone (it bundles ZK on port 2181).
• As a web/backend developer you rarely touch either directly. You consume systems that embed them. Solve your real problems with your database, Redis, load balancer, and managed AWS services.