pi < theta < 3pi/2 is the 3rd quadrant.
In this quadrant, both the adjacent side and opposite side of the triangle formed by the angle are negative.
cos(theta) = -5/13 implies that the adjacent side is -5 and the hypotenuse is 13.
Thus, the opposite side is -sqrt(13^2 - (-5)^2) = -sqrt(144) = -12
This means sin(theta) = opposite side / hypotenuse = -12 / 13
sin(2theta)
= 2sin(theta)cos(theta)
= 2(-12/13)(-5/13)
= 120 / 169
cos(2theta)
= 2cos^2 (theta) - 1
= 2(-5/13)^2 - 1
= 2(25/169) - 1
= (50 / 169) - 1
= -119 / 169