Topology information is updated on topology poll. So if one of your nodes reports that this port is linked to some other device, manually set peer will be removed.
You can try running topology poll manually on these nodes to see which one of them does that.
You can try running topology poll manually on these nodes to see which one of them does that.