I'll relay the advice I recently received from one of my professors when I was deciding between grad schools, since I had the same dilemma. I am (or was?) interested in going into algebraic topology, but considering a school with no one doing algebraic topology proper, although with several people doing similar flavored research I'd want to work with. I ended up committing to the school based on his advice.
He told me that I should stay flexible and keep my options open instead of specializing too early, since there'll be professors doing interesting research in algebra nowhere near AT that I might want to work with (and similarly for topology). He said that advisor fit is important, and if you specialize in an area and end up not liking your advisor, you're not going to have the best time in graduate school.
I decided he was very right after visiting and talking to some professors there and realized there were fields I'd never considered that actually sounded incredibly cool. There's honestly too much math out there (that's both interesting to me and something I could be good at) to decide so early in my career that I absolutely prefer one field over everything else.
