why is 17x^(2)/15 correct, but not 17/15x^(2)?

First, you are writing 1/8 without the x in your answer, but your work obviously shows you know there is an x there.  So, I am ignoring that.

When we write mixed numbers like 1 2/15, we understand that there is a plus sign between them.  Thus, 1+2/15 = 17/15.

1 2/15 x^2 would confuse the math, since we know the 2/15 is being multiplied by x^2 and BEDMAS/PEDMAS rules all.

This is one of the reasons we rarely use mixed numbers in math, sticking to improper fractions.

With respect to the + -1x/8, I would expect my students to simplify this to - x/8.
Why did the -1/8x become -1/8? To be more clear on the computer although it probably is less confusing on paper, you can write (17/15)x\^2 and (1/8)x instead.

In any case (1/8)x is clearly not the same as (1/8) so that is already a mistake.
>where 17x^(2)/15 came from/why its true versus 17/15x^2

Using BODMAS or PEMDAS,  17x^(2)/15 and 17/15x^(2) are the same thing.

However you're going to find later in algebra that another convention called "multiplication by juxtaposition" is often used instead.   And using that convention, 17/15x^2 means 17/(15x^(2)).    Basically the convention says that when you use juxtaposition (e.g.  3x instead of 3*x)) to represent multiplication, then *that* multiplication is done before any division.

My point is not that you're wrong or right.   I just wanted you to be aware that unfortunately expressions like 17/15x^(2) and 3/8x can be ambiguous so it's often better to either write 17x^(2)/15 or 3x/8 if that's what you mean.   Those expressions are unambiguous because they are the same whether or not you're using multiplication by juxtaposition.