I don't know why you're not telling us what these data sets actually represent. That would be pretty helpful.
However, it sounds like each one intentionally contains exactly one large number and a bunch of smaller ones. So you probably need to produce two numbers -- one for the large score, and another for some kind of average of the small ones.
Then if you *really* need to combine those into a single number, you can average the two numbers (possibly a weighted average).