Yes, men don't generally look like women or vice versa. So, men will look at a woman's facial features for features that express youth, fertility, and general health. Then if asked to rate a man on looks will tend to try to take those features in how they would assess a man's looks, where women might look to features that show strength, confidence, or power (whatever the actual list would be is less important, other than to note the list is different).
Clooney is a fair example; just going off of the wiki photo, since that was the top search result, I would probably rate him as somewhere around 7-8, better than average looks. However, clearly when women look at him they are rating him as exceptionally good looking and so, from a female perspective must be getting rated somewhere between 8-10...
The other factor to consider is that when men rate women, it usually falls in a bell curve distribution where most get rated around average. Women rate men typically in an inverse bell curve, as in moat are rated 1-3 or 8-10 with very few landing in the average range.