There is the whole theory of representation theory, where you study ways to represent a group as actions on a vector space. The vector are abstract, what you care about is what kind of representations are possible. Many huge research program in math involves representations, such as the one that solve Fermat's Last Theorem.
On the opposite side, you can distinguish between various geometrical object by the group. In physics and geometry people distinguish between vector, covector, various form of tensor, spinor, and so on, which are all vectors in the abstract sense of vector. They are distinguished by the group that act on them. Vector is acted on by the orthogonal group (or analog of that in Lorentzian spacetime), or the general linear group. Covector is the acted on by the opposite group. Spinor get acted on by the spin group.