I have had issues when using just three loss perimeters I am not sure how you have it set up, but you might consider calculating which are convergent and which are divergent and update the gradient in the direction you wish to go, where you have multiple agree experts or multiple disagreeing experts and you choose to "side" with either the multiple experts or the divergent.