I think this is a double integral:
∫∫f(x,y)dxdy=∫∫kx^2dxdy=∫(kx^2y[0,x])dx=∫kx^3dx[0,1]=kx^4/4[0,1]=k/4.
The explanation for the integration is that, treating x as a constant wrt y, we get kx^2y, and then we apply the limits for y which gives us the result x. When combined with kx^2, we thus get kx^3, which is then integrated wrt x to give us kx^4/4. Then apply the interval to get k/4. Taken over the whole range the PDF = 1 (meaning the sum of all possibilities is certainty), so since the double integral = 1, k/4=1 and k=4.