Right, getting this out of the way. The man is not human.

Look, he's swirly!
The mazoku or monster race are Astral, which is to say non-corporeal beings. They're able to take on human form, although there's usually something a bit off about them. Xel's most obvious marker is his unnaturaly smooth hair. Also noteworthy are his eyes, which are almost always closed because they have no shine, and his ever-present gloves and turtleneck. If they're hiding anything, it's still a secret.
Mazoku can eat ordinary food,

but their real food pyramid runs off negative emotions like, fear, hatred, and pain

and, in Xellos's case at least, probably annoyance.

(But honestly, she kind of deserved that.)