If you shot at it or not, if you missed it twice you deserve the extra penalty
But in reality, I usually try to call it and point as soon as it happens so other people see that I saw it. Sometimes if a person does a wild ass plan it's hard to notice and it's definitely easier to call it when working a major when you are way more familiar with the stage.
We shot a Halloween themed match and all the targets on the memory stage had tombstones next to them. After working the whole weekend we could call by name which target got skipped.