We can call the side of the square base B and the height H. The total surface area of the open box is 4HB+B^2=975, so H=(975-B^2)/4B or B^2+4HB+4H^2-4H^2=975, (B+2H)^2=975+4H^2 and B=sqrt(975+4H^2)-2H. The volume V=H(sqrt(975+4H^2)-2H)^2.
Let X=sqrt(975+4H^2), X^2=975+4H^2 and 2XdX/dH=8H, so dX/dH=4H/X. V=H(X-2H)^2. If we differentiate V with respect to H we get:
dV/dH=(X-2H)^2+2H(X-2H)(dX/dH-2)=
(X-2H)^2+4H(X-2H)(2H/X-1)=
(X-2H)^2-4H(X-2H)^2/X=(X-2H)^2(1-4H/X)/X.
When dV/dH=0 V is at a maximum or minimum: so X=2H or 4H. (X=2H is not valid, because 4H^2=975+4H^2, resulting in 0=975, which is never true.)
So 16H^2=975+4H^2 and 12H^2=975 so H=sqrt(975/12)=sqrt(325/4)=5sqrt(13)/2=9.014m approx.
From this value of H,
B=sqrt(975+4H^2)-2H=sqrt(975+325)-5sqrt(13)=10sqrt(13)-5sqrt(13)=5sqrt(13)=18.03m approx. V=5sqrt(13)/2*325=2929.51 sq m.
To prove this is a maximum, we need to take H=9 (close to 9.014) making B=sqrt(975+324)-18=18.04m. V=2929.50 sq m. Now take H=9.1, so B=17.94 and V=2929.41 sq m. So maxV=2929.51 sq m, height=9.014m and base side is 18.028m. The height is half the base side length and the maximum volume is B^(3/2)/2=325^(3/2)/2.