It's the other way round: They are the building blocks because they are simple.

In mathematics (or science in general, really), we tend to express things in other, known things that are simple. This way we can build up quite complex knowledge in a Lego like fashion. Another of these ubiquitous concepts in math is linear algebra. Vectors and matrices (or more accurately linear maps) are very easy to understand and use, yet make up the basis of a lot of the advanced math like, e.g., functional analysis, which can be described as linear algebra with functions instead of vectors and matrices.