EL here of \~2yrs. From my perspective, the benefits of working in office are probably more important the younger you are in your career. I work 100% remote though. I think there's one thing a lot of people don't realize. For some people, even if they wanted to go in the office, they don't get any benefit from it. Me for example, I would go in if a lot of other people did or if my teammates did, but they don't. Essentially if I go into the office it's the same as being remote.
To answer your question directly though, I don't really think it's limiting, but it depends a lot on what specific company you work for. I started 100% remote, get great reviews, and have networked effectively.