Hello, @dimitri , thank you so much for pg_auto_failover! For us, this is the best cluster solution for PostgreSQL.
Quite a long time ago, more than a year ago, I discovered a bug when changing the monitor of the cluster's datanode related to NodeID.
Description of the situation:
We have two monitors monitorA and monitorB, each with its own data cluster:
- monitorA: datanodeA_1 (NodeID=1), datanodeA_2 (NodeID=2)
- monitorB: datanodeB_1 (NodeID=1), datanodeB_2 (NodeID=2)
To save resources or for some other reason, we decide that one monitor - monitorA is enough for us, and we attempt to replace the monitorB monitor online from the datanodeB_1 and datanodeB_2 cluster using the following functionality https://pg-auto-failover.readthedocs.io/en/main/operations.html#replacing-the-monitor-online
Since our clusters are independent and initially each has its own monitor, the datanodes may have the same NodeID as indicated in the configuration example above. We are successfully disconnecting datanodeB_1 and datanodeB_2 from monitorB.
Next, we try to connect datanodeB_1 and datanodeB_2 datanodes to the monitor, but we get an error that tells us that we cannot add NodeID datanodes to the monitor with which the monitor already has datanodes - the monitor already has NodeID 1 and 2 datanodes (these are datanodeA_1 and datanodeA_2).
It seems that this behavior is caused by the fact that the datanodeB_1 and datanodeB_2 datanodes, when trying to connect them to the monitor Monitor, pass it their NodeID from the state file as desiredNodeId. Since the desired NodeIDs have been transferred to the monitor, the IDs of the nodes to be added are not assigned from the sequence, but those transmitted by the datanodes are taken, and if they match the NodeID datanodes of another cluster on the monitor, an error occurs.
The monitor probably runs the monitor_register_node(...) function, and the datanode runs keeper_register_again(Keeper *keeper),
the KeeperConfig *config = &(keeper->config) structure contains the NodeID from the state file. (But I could be wrong)
At one time, I tried to change the code of the stored procedure that was responsible for adding the datanode to the monitor, passing the desired_node_id always null, as a result, it helped - the ID for the nodes added (switched) to the monitor was taken from the sequence and everything was ok - the nodes were added. But it required changing the code.
I recently found a second way - after disabling datanodeB_1 and datanodeB_2, you can find the necessary bytes responsible for the NodeID in their state files, change them to those IDs with which there are no datanodes on monitorA, and then add datanodeB_1 and datanodeB_2 to the monitorA monitor. You also need to advance the value of the sequence on monitorA (if the nodeIDs following the current value of the sequence were selected).
If possible, then you need to make sure that when you enable monitor, the datanodes do not transfer their nodeIDs as desired nodeIDs, otherwise the process of replacing the monitor may not be possible (if there is a match NodeID). Either omit the desired NodeID in the monitor_register_node function or the pgautofailover.register_node database function. Then the NodeID will always be selected from the sequence.
We have configurations where one monitor manages several clusters at once, and similar situations are possible when switching new databases from other monitors to such a monitor.
Thank you again for developing pg_auto_failover!
Hello, @dimitri , thank you so much for pg_auto_failover! For us, this is the best cluster solution for PostgreSQL.
Quite a long time ago, more than a year ago, I discovered a bug when changing the monitor of the cluster's datanode related to NodeID.
Description of the situation:
We have two monitors monitorA and monitorB, each with its own data cluster:
To save resources or for some other reason, we decide that one monitor - monitorA is enough for us, and we attempt to replace the monitorB monitor online from the datanodeB_1 and datanodeB_2 cluster using the following functionality https://pg-auto-failover.readthedocs.io/en/main/operations.html#replacing-the-monitor-online
Since our clusters are independent and initially each has its own monitor, the datanodes may have the same NodeID as indicated in the configuration example above. We are successfully disconnecting datanodeB_1 and datanodeB_2 from monitorB.
Next, we try to connect datanodeB_1 and datanodeB_2 datanodes to the monitor, but we get an error that tells us that we cannot add NodeID datanodes to the monitor with which the monitor already has datanodes - the monitor already has NodeID 1 and 2 datanodes (these are datanodeA_1 and datanodeA_2).
It seems that this behavior is caused by the fact that the datanodeB_1 and datanodeB_2 datanodes, when trying to connect them to the monitor Monitor, pass it their NodeID from the state file as desiredNodeId. Since the desired NodeIDs have been transferred to the monitor, the IDs of the nodes to be added are not assigned from the sequence, but those transmitted by the datanodes are taken, and if they match the NodeID datanodes of another cluster on the monitor, an error occurs.
The monitor probably runs the
monitor_register_node(...)function, and the datanode runskeeper_register_again(Keeper *keeper),the
KeeperConfig *config = &(keeper->config)structure contains the NodeID from the state file. (But I could be wrong)At one time, I tried to change the code of the stored procedure that was responsible for adding the datanode to the monitor, passing the desired_node_id always null, as a result, it helped - the ID for the nodes added (switched) to the monitor was taken from the sequence and everything was ok - the nodes were added. But it required changing the code.
I recently found a second way - after disabling datanodeB_1 and datanodeB_2, you can find the necessary bytes responsible for the NodeID in their state files, change them to those IDs with which there are no datanodes on monitorA, and then add datanodeB_1 and datanodeB_2 to the monitorA monitor. You also need to advance the value of the sequence on monitorA (if the nodeIDs following the current value of the sequence were selected).
If possible, then you need to make sure that when you enable monitor, the datanodes do not transfer their nodeIDs as desired nodeIDs, otherwise the process of replacing the monitor may not be possible (if there is a match NodeID). Either omit the desired NodeID in the
monitor_register_nodefunction or thepgautofailover.register_nodedatabase function. Then the NodeID will always be selected from the sequence.We have configurations where one monitor manages several clusters at once, and similar situations are possible when switching new databases from other monitors to such a monitor.
Thank you again for developing pg_auto_failover!