There is a great talk by Freeman Dyson, called Missed Opportunities, where he argues that mathematicians could have anticipated many of the most important discoveries in modern physics. For example, the Galilei group can be understood as a limit case of the more general Poincare group for $c\rightarrow \infty$. Why should nature make use of such a peculiar special case? With this reasoning it would've been possible to anticipate special relativity solely on mathematical ground.
Equally, the Poincare group can be understood as limit of the more general de Sitter group in the limit $\Lambda \rightarrow 0$. Thus again, it could've been anticipated on mathematical grounds that the expansion of the universe accelerates.
Another such missed opportunity, not mentioned by Dyson, is the curious behaviour of spin $1/2$ particles. The Poincare group is not simply connected and therefore in mathematical terms a rather incomplete object. (This is comparable to how $SO(3)$ is as a Lie group is only “half of the three sphere”, whereas its simply connected double cover $SU(2)$ is the complete three sphere). In contrast, the double cover of the Poincare group is simply connected and exhibits additional representations: the spinor representations. Thus, instead of going the complicate route via Dirac's derivation of his famous equation, someone could have revealed the existence of spinors and their strange behaviour solely through abstract mathematics.
Therefore it seems worthwhile to look for similar opportunities that we are currently missing.
See also Reflections on the Evolution of Physical Theories by Henri Bacry:
“I will give in the present article examples of missed opportunities of a special kind. Very often, a physicist has the intuition of some principle but, for obscure reasons, does not state it explicitly. Similarly, a physicist has the idea of a thought experiment but does not explore it completely. ”