If you only have studied Calculus previously, the concepts introduced in Category Theory might feel weird and it will be hard to see the motivation behind them (this includes even the very first definitions). Also, you will have a hard time finding examples to work with, since they usually come from various areas of math. I would suggest studying some Linear Algebra (vector spaces and linear transformations stuff), Abstract Algebra or Topology beforehand, but it might still be a bit hard to grasp on the beginning.
Category Theory is better taken later on because it generalizes some properties one may have encountered during their studies which show up in seemingly very distant parts of math. So I guess it's not the best subject to rush towards. It is, however, certainly still possible to learn it, I just think it will be a lot harder to grasp the material without good examples and without the required mathematical maturity.