Question

[Riemannian Geometry] Why did we give "Exponential map" this name?

Answer 1

One way of seeing the analogy, think of R^+ as a Riemann geometry such that its metric is invariant under *multiplicative* translation.

Another way, is to go through Lie group.

On a matrix Lie group, the exponent map of a matrix A is exp(tA)=1+tA+t^2 A^2 /2!+... which is the usual Taylor's series, and the derivative is Aexp(tA). The parallel transport of A at the identity (t=0) to some point along this curve at time t is Aexp(tA), where parallel transport is defined using the fact that every tangent space can be identified using left-invariant action (or right-invariant, doesn't matter). And exp(A) is the value when t=1.

Hence, we have this notion of an exponential curve: the exponential curve is one in which all velocity vector are parallel transport of each other along the curve. This notion makes sense whenever you have a connection. And on a Riemannian geometry, there is a Levi-Civita connection. Then for a tangent vector v, you can plug in the value when t=1.

Answer 2

Lee's book on smooth manifolds asserts that the reason is this: in GL(n,R) considered as a Lie group, the exponential map exp A turns out to be precisely the matrix exponential e^(A). The exponential map in the sense of a Lie group is not quite the same notion as the exponential map on a general Riemannian manifold - in Riemannian terms we would call it exp_0(A) - but the terminology carried over.

I don't know if this is historically accurate, but it seems at least plausible as an explanation.