I remember thinking the exact same thing when I learned integration, here's what made it eventually click for me (though this is not a formal proof by any means)

Let the area under the graph of y = f(x) from 0 to x be A(x). If you extend x by a tiny amount h, that's like adding one of those tiny rectangles to the sum - in this case that rectangle will be h \* f(x). When h is small, you expect this to be close to the actual difference of A(x) and A(x + h). In other words:

A(x + h) \~= A(x) + hf(x)

Then by rearranging:

f(x) \~= (A(x + h) - A(x))/h

When h tends to 0, this gives the exact defintion of the derivative, showing that the function of the area under a graph of a function differentiates to give the function itself. A stronger argument would show that the approximation tends to an equation as h tends to 0.