Like you, I saw many more hazards than the ones a learner was expected to see. I also spotted the "real" hazards sooner than a learner would, and as a result would press the button too soon. I had to "train myself" to do the test like a learner.
Once you learn that the motorway scenario is about vehicles joining or changing lanes unexpectedly, the country scenario, tractors, horses round bends, water on road, town scenario, cars pulling out, pedestrians running across the road, vehicles overtaking as they approach you, you can disregard the "subtle" hazards an experienced driver spots.
Our PCV instructor told us how the four instructors all took the test when it was first introduced (tests are done from the bus depot, not from a testing centre) and all four failed the hazard perception test!