An important aspect of mathematics teachers’ diagnostic competences is the ability to judge the difficulty of a mathematical task. The process of judging task difficulty includes the perception and interpretation of task characteristics that are potentially challenging for students. Such judgement processes are often quick and difficult to assess. Most previous studies described these processes on the basis of teachers’ verbal reports. A more recent approach to tap into cognitive processes is eye tracking. However, there is no firm knowledge yet whether eye tracking allows for a reliable assessment of teachers’ judgements of mathematical task difficulty. The present study aims at filling this gap. We asked