I have a hard time looking away. I tend to maintain eye contact of EVERYONE I am speaking to. I have to conciously remember to look away occasionally.
One time, this woman stopped by when I was still with my ex, to discuss this extra-curricular program that she was doing with little kids. She was strangely mezmerizing and attractive, so I maintained eye contact. After a couple minutes of this, she could barely function! Later, my ex mentioned the situation in a jealous tone of voice. I know that I wasn't scaring this woman because to this day when we run into each other, we talk like we are old friends.
Since then I have read that one should look away when talking or having job interviews, etc., because it can be intimidating otherwise.