"Because I've met some other Avatars who haven't. There was an older man and his adoptive daughter. Both served the Hunt, mind and body, both spent their entire lives as themselves."

Granted, said older man and daughter, Trevor Herbert and Julia Montauk, aren't exactly the best examples of 'stable people who works for the Hunt.' But Elias is keeping that to himself!