First, the answer. As a hobbyist, I relearn math all the time. It's almost the name of the game. As a high school (engineering) teacher, some of my engagement with mathematical ideas is expressly to find new ways of communicating them, which requires relearning.
Second, the metaphor. This reminds me of an idea I came across previously in the comments. What you know in your head is your toolbox — what you carry around with you to solve problems you might encounter. What you've learned but no longer know (perhaps codified in your texts/notes) is your workshop. There's some heavy machinery that's not really feasible to carry around, but there's also tools that don't fit in your toolbox. You know how to use them, but you have to go and get them. Maybe even find them if your workshop is as disorganized as most.
Third, the advice. Don't feel shame over needing to revisit an idea or subject — it is not only a natural part of learning but a necessary one. A similar pridefulness hindered me greatly in college. Had I the humility to accept the various types of difficulty that accompany continued study perhaps things would look differently; at the very least my GPA certainly would. To that end, measure yourself by your effort not strictly the outcome. This is something I continue to struggle with, but when I'm able to do so it makes many more things–including relearning**–**worthwhile and enjoyable.
