I'd say it much depends on your natural talent. I won't sugarcoat, but from my experience if you struggle at maths at 16 you'll definitely struggle way more at the university. Hell, I was one of the best students in my highschool, top 2 in my class and I barely passed calculus. The math related courses really are fast paced and if you know that you're a slow learner, I'd say pass.
If you really do want to learn high level maths and be considered a good mathematician, it'll require a ton of work and you'll have 0 free time. There are probably easier ways to become a good mathematician than going to college/university/whatever though, but I don't really have any experience in that regard. Wish you luck though.