The Tavern

I've never seen him twice. Isn't he supposed to come back? What am I doing "wrong"?

You're not doing anything wrong, it's just rather uncommon to see him more than once due to Dungeon being a lot shorter (as well as the low odds of seeing him in the first place).

If you want a grudge match with him, he has a chance of spawning in Tomb, so you can start routing that in (do not do tomb casually it is a slog for most characters)

I think he ran out of robes to wear.