We studied the role of attention and task demands for implicit change detection. Subjects engaged in an object sorting task performed in a virtual reality environment, where we changed the properties of an object while the subject was manipulating it. The task assures that subjects are looking at the changed object immediately before and after the change. Our results demonstrate that in this situation subjects’ ability to notice changes to the object strongly depends on momentary task demands. Surprisingly, frequent noticing is not guaranteed by task relevance of the changed object attribute per se, but the changed object attribute needs to be task relevant at exactly the right times. Also, the simplicity of the used objects indicates that change blindness occurs in situations where the visual short term memory load is minimal, suggesting a potential dissociation between short term memory limitations and change blindness. Finally, we found that changes may even go unnoticed if subjects are visually tracking the object at the moment of change. Our experiments suggest a highly purposive and task specific nature of human vision, where information extracted from the fixation point is used for certain computations only “just in time” when needed to solve the current goal.