cosx-cos^3x=cosx(1-cos^2x)=cosxsin^2x. Square root is sinxsqrt(cosx).
Integral becomes S(sinxsqrt(cosx/(1-cos^3x)dx), where S denotes integral.
Let y=cosx, dy/dx=-sinx, so dy=-sinxdx.
-S(sqrt(y/(1-y^3)dy) is the new integral=-S(y^(1/2)/(1-y^3)^(1/2)dy). Let z=y^(3/2), so dz=(3/2)y^(1/2)dy and y^3=z^2.
Integral becomes -(2/3)S(dz/sqrt(1-z^2)). Let z=cos(p), dz=-sin(p)dp.
Integral becomes (2/3)S(sin(p)dp/sin(p))=
(2/3)p=(2/3)cos^-1(z)=
(2/3)cos^-1(y^(3/2))=(2/3)cos^-1((cos(x))^(3/2))+k, where k is constant of integration.