Let O(h,k) be a point in an x-y Cartesian frame of reference. O is the origin of another Cartesian frame of reference parallel to the x-y plane. Now let P'(x',y') be a point in the second frame of reference. O'(0,0) is the origin in this plane and is equivalent to O(h,k) in the first frame of reference. So we can write x=x'+h and y=y'+k. That’s translation.
Let O'P'=r and the angle between O'P' and the x' axis be θ. We can write x'=rcosθ and y'=rsinθ. Now we rotate P' anti-clockwise ɸ degrees about O'.
The new position of P' after rotation is given by polar coordinates r and θ+ɸ.
So the new values for x' and y' are:
x''=rcos(θ+ɸ)=rcosθcosɸ-rsinθsinɸ=x'cosɸ-y'sinɸ,
y''=rcos(θ+ɸ)=rsinθcosɸ+rcosθsinɸ=y'cosɸ+x'sinɸ.
But x'=x-h and y'=y-k so the point (x'',y'') becomes ((x-h)cosɸ-(y-k)sinɸ,(y-k)cosɸ+(x-h)sinɸ). In the first frame of reference the point becomes (h+(x-h)cosɸ-(y-k)sinɸ,k+(y-k)cosɸ+(x-h)sinɸ).
If we are given a point A(a₁,a₂) before translation and rotation, we can relate it to the corresponding point A'(a₁',a₂') on the image with respect to the first frame of reference.
a₁'-h=(a₁-h)cosɸ-(a₂-k)sinɸ,
a₂'-k=(a₂-k)cosɸ+(a₁-h)sinɸ.
From these we get (a₁'-h)²+(a₂'-k)²=(a₁-h)²+(a₂-k)²=r² and we get a linear relationship between h and k of the form h=Ak+B, where A and B are constants.
We are trying to find h, k and ɸ, but we only have two equations. So we need another point on the original figure and the corresponding point on the rotated image to give us another pair of equations.
So, for a point B we can similarly write:
b₁'-h=(b₁-h)cosɸ-(b₂-k)sinɸ,
b₂'-k=(b₂-k)cosɸ+(b₁-h)sinɸ.
From these we get (b₁'-h)²+(b₂'-k)²=(b₁-h)²+(b₂-k)² from which we get another linear relationship between h and k (h=Ck+D, where C and D are constants) enabling us to find h and k, because we can write Ak+B=Ck+D, from which we get k.
Because of size limits on this website, there is insufficient space here to show the actual solutions for h and k.
Knowing h and k we can move to the second frame of reference and we can calculate for one point (let’s use A) the angle θ that point makes with the x' axis before rotation: tanθ=(a₂-k)/(a₁-h), so θ=arctan((a₂-k)/(a₁-h)). We can write tan(θ+ɸ)=(a₂'-k)/(a₁'-h).
So θ+ɸ=arctan((a₂'-k)/(a₁'-h)). Therefore ɸ=arctan((a₂'-k)/(a₁'-h))-arctan((a₂-k)/(a₁-h)).
The picture shows an object (a square) and one of its vertices rotated through angle ɸ. The coords are with respect to the x-y frame of reference (the calibrated axes).