The conjugate point condition for smooth control sets