Colour tracking thrown off due to white balance

When colour tracking an object the white balance is off because an object starts in front of the camera. How do I make my camera so it does not white balance right at the beginning so my colouring is not off. I need to have a grey/yellow object start in front of the camera but later on track other colors like orange and green so I can not have my white balance thrown off.

I have these set like this:

sensor.set_auto_gain(False)
sensor.set_auto_whitebal(False)

Hi, make the amount of time skip frames runs shorter. If you make the auto functions run for only a short amount of time they barely have an effect. Try 100ms.