fix(ironic): Fix anti affinity rules for default and update replica count

[nicholaskuechler]

The ironic-api deployment included anti-affinity rules (under 'default' section), and uses 4 replicas, but we only have 3 noddes. This combination with preferredDuringSchedulingIgnoredDuringExecution prevents our ability to deploy the ironic-api as it’s just stuck forever scheduling due to never meeting these combined conditions.

[cardoe]

The reasoning doesn't reconcile with the Kubernetes docs.

[cardoe]

To quote the Kubernetes docs:

preferredDuringSchedulingIgnoredDuringExecution: The scheduler tries to find a node that meets the rule. If a matching node is not available, the scheduler still schedules the Pod.

You've said this prevents us from scheduling workloads but the Kubernetes docs say otherwise.

[cardoe]

https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity

[nicholaskuechler]

I've also read the docs but what they say doesn't match our reality. Our reality is that we literally can't complete an ironic-api deployment without manual intervention. The pods are stuck in pending for at least 30 minutes after which time I intervene. I've been feeding the logs/output/etc in to LLM over the past few weeks but it hasn't found anything conclusive either.

[skrobul]

Which environment did you see that problem in?

❯ kubectl --context uc_dfw3_prod -o wide get pods | grep ironic-api
ironic-api-9ffd89855-4tngk                                        1/1     Running     0             5d6h    10.64.52.40    infra3   <none>           <none>
ironic-api-9ffd89855-f98nd                                        1/1     Running     0             5d6h    10.64.50.216   infra1   <none>           <none>
ironic-api-9ffd89855-n9s6r                                        1/1     Running     0             5d6h    10.64.51.243   infra2   <none>           <none>
ironic-api-9ffd89855-tpbj4                                        1/1     Running     0             5d6h    10.64.50.142   infra1   <none>           <none>

or in other words, how to reproduce this?