One of the things which breaks that mindset for me, is thinking about and trying to really appreciate the 'multiple constructibility' of mathematics. In other words, that our ordinary arithmetic of naturals, and countless other nonsense-to-us-but-internally-consistent arithmetics besides, and our set theories, ZFC and ZF~C and all the others, all of them... they all have unfathomably many equivalent, alternative constructions, in all kinds of symbolic and computational languages. And insofar as they can all construct each other, these equivalencies are infinitely nested.
That is to say, Von Neumann numbers are no more or less real than Peano numbers, are no more or less real than Peano numbers constructed *out of* Godel-numbering of Von Neumann numbers. All three of these systems of positive integers behave in the same way and have an equivalency between them, and the equivalence is shared by uncountably many others. But out of all those possibilities, we happen to have a theory of 'numbers', rather than, say, a theory of 'schmumbers', which behave exactly like numbers except for that they differ in some totally-indescribable way.
Somehow, this all gives me a feeling of "arbitrariness" about the foundational mathematics that we humans happen to have based our everything on. If we encountered extraterrestrial intelligence it's possible that all their Pure Maths would be formulated in a completely bizarre, unfamiliar way to ours. And so, pure mathematicians are really just like all the other hyper specific niche researchers. It's *all* stamp collecting.