Stop a shard stacking its replicas on one host

[bjmeetsfo]

Independent of the other open work — main base, meta/partitioning.rs only.

The problem

A shard could put both its replicas on the same physical host while another host
sat idle. It reported the full replica count, and one host failure took all of
them.

Four datanodes, two per host, all in one rack — an ordinary small deployment:

servers:  host-a:1001  host-a:1002  host-b:1001  host-b:1002
replica_count: 2
placement:  ["host-a:1001", "host-a:1002"]      <-- both on host-a

Why. push_replica records used_hosts but does not enforce it — only
the separation ladder above checked it. When every server shares a location, the
ladder rejects every candidate after the first (shared == depth, never
< separation), so every replica after the first came from the unconditional
fallback fill, which never looked at used_hosts at all.

The one-rack case is not exotic. It is what a cluster looks like before it grows
into multiple availability units, and it is exactly when the ladder can separate
nothing and the fallback does all the work.

The change

The fallback fills in two passes instead of one: a host already holding a
replica of this shard is taken only when nothing else is left.

That keeps the fallback's purpose — reach replica_count — while not spending
redundancy it did not have to spend.

Tests

• a shard does not stack its replicas on one host — the case above. Fails
  without the change.
• a host is reused only when there is no other — two datanodes on one host,
  replica_count: 2, still returns two replicas.

The second is the one that matters for review: the point is to spread, not to
refuse, and it would be easy to "fix" this into a shard that comes back short.
It passes with and without the change, on purpose — it guards the fix rather
than proving it.

Scope

Placement order otherwise is untouched: the load-based candidate ordering, the
separation ladder, the owner handling and the primary selection all behave
exactly as before. Only the final fill changed.

Verification

• cargo check --all-targets — 0 errors
• cargo test --bin metaserver -- --test-threads=1
• cargo test --lib -- --test-threads=1

The data_node::…jitter_backoff… and engine::…recovery_validates_… failures
reproduce on an unmodified tree at this base and are untouched here.

---

Correction to the verification note above

I described two failures as reproducing on an unmodified tree. Re-checked on
a quiet machine with disk headroom, run in isolation against unmodified main:

test	unmodified main
data_node::…jitter_backoff…	fails 3/3 — genuinely pre-existing, deterministic
engine::…recovery_validates_all_timestamped_kv_page_families	passes 3/3

Only the first is pre-existing. My original evidence for the second came from a
run taken immediately after the disk hit 100%, so it was environmental — that
test is sensitive to disk pressure and load, not broken on main.

The conclusion this change is not responsible for either failure is unchanged.
The evidence offered for half of it was wrong, and the record should say so.