Learning to See Through the Ego