Simply replace the x in the function's formula with whatever you see in the parentheses to evaluate g at that point. We get:
g(5n) = (5n)^2 - 5n
Recall that (ab)^2 = a^2b^2, so:
g(5n) = 25 * n^2 - 5n
The tricky part here is that parentheses like (5n)^2 are often omitted (which is obviously a mistake) and are often interpreted like 5n^2, which just means 5 times n^2, or 25n, which means 5^2 * n (again, those are just common mistakes). However, the correct way is to always use them when substituting a "bigger" term in place of a smaller one, for example when using abc instead of d in the formula d^3, the correct substitution is (abc)^3 = a^3b^3c^3, and none other.