We show by explicit computation that the recently discovered duality invariance of D=4 linearized gravity is lost, already at first self-interacting, cubic approximation of general relativity. In contrast, the cubic Yang-Mills extension of Maxwell does admit a simple deformed duality.