one elasticsearch shard will not initialize . 07/29/2014

Working with Elasticsearch, you end up obsessing over cluster health. A cluster is either red, yellow, or green. Apparently yellow is okay, but it means that replica shards are missing. Elasticsearch should automatically take care of this dilemma for you to get a cluster status to green. In my experience, it usually does. Today it did not. A cluster I'm maintaining was yellow all day and I wasted a lot of time scouring logs.

> curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

1
2
3
4
5
6
7
8
9
10
11
12

{
  "cluster_name" : "whatever-cluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 12,
  "number_of_data_nodes" : 12,
  "active_primary_shards" : 30,
  "active_shards" : 89,
  "relocating_shards" : 0,
  "initializing_shards" : 1,
  "unassigned_shards" : 0
}

If this happens to you, do not scour the logs. Use cat recovery.

> curl -XGET 'localhost:9200/_cat/recovery?v'

Look for a row that looks like this:

172.31.9.96 datapoints 2 41390776 replica init n/a some-host n/a n/a 0 0.0% 0 0.0%

In my case, this shard was attempting to initialize with "n/a" for source_host.

Restart Elasticsearch on the target_host ("some-host" in the example) and Elasticsearch should be able to take care of the rest.