What is the domain of #h(x)=sqrt(( x- (3x^2)))#?

Right from the start, you know that the domain of the function must only include values of #x# that will make the expression under the square root positive.

In other words, you need to exclude from the function's domain any value of #x# will result in

#x - 3x^2 < 0#

The expression under the square root can be factored to give

#x - 3x^2 = x * (1 - 3x)#

Make this expression equal to zero to find the values of #x# that make it negative.

#x * (1 - 3x) = 0 implies {(x = 0), (x = 1/3) :}#

So, in order for this expression to be positive, you need to have
#x>0# and #(1-3x) > 0#, or #x<0# and #(1-3x)<0#.

Now, for #x<0#, you have

#{(x<0), (1 - 3x > 0) :} implies x * (1-3x) < 0#

Likewise, for #x > 1/3#, you have

#{(x > 0), (1 - 3x > 0) :} implies x * (1-3x) < 0#

This means that the only values of #x# that will make that expression positive can be found in the interval #x in (0, 1/3)#.

Any other value of #x# will cause the expression under the square root to be negative. The domain of the function will thus be #x in (0, 1/3)#.

graph{sqrt(x-3x^2) [-0.466, 0.866, -0.289, 0.377]}