Groups whose vanishing class sizes are not divisible by a given prime