I may be more of a peculiar case because I actually come from a physics background initially before I started working more towards theoretical physics/mathematical physics/math after going back and formally studying math. Nevertheless, I don’t think that makes my opinion completely obsolete so here goes.
Firstly, it may *seem* like some pure math students are able to quickly pick up necessary physics knowledge, but the reality is that they don’t, it’s fairly specific subfield knowledge and the minimum they need to know to function. It’s not as daunting of a task as obtaining a physicists’ knowledge of physics in a fraction of the time, just a lot of bits and pieces.
Secondly, a significant part of mathematical physics is just mathematics and the physics emphasis required isn’t problematically deep in many cases. This does depend on the subfield of mathematical physics and the type of research you are doing, but it is a general principle.
Essentially, if you’re a mathematician, it’s often not that incredibly large of a jump to bridge over to some field in mathematical physics, since you’ll find yourself at home with the nature of the subject.