The local aspects of Riemannian geometry largely boil down to the Levi-Civita connection --- which is indeed quite special to the Riemannian setting --- but of course the global aspects have a lot to do with the calculus of variations and elliptic PDE theory.
Anyway, your question is probably best answered via the theory of G-structures. A G-structure on a manifold M is a principal G-subbundle of the general frame bundle FM, where G is a (usually closed) subgroup of GL(n,R). Specifying an O(n)-structure on M is equivalent to specifying a Riemannian metric on M. A connection on an O(n)-structure is then equivalent to a metric-compatible connection on TM.
To each G-structure B in FM, there is a G-equivariant function T\_B called the "intrinsic torsion" (an older name is "structure function") whose domain is B and whose codomain is a particular R-vector space equipped with a G-action that I won't explain here. The function T\_B is zero if and only if B admits a torsion-free G-connection. In this language, the existence part of the fundamental lemma of Riemannian geometry states that every O(n)-structure has T\_B = 0.