this equation expands toThe vertices share no common coordinate so the hyperbola cannot be orientated in the usual x-y form. The hyperbola must therefore be skew (tilted axis) and the line joining the vertices is sloping. This line is given by the equation y=((7+1)/(5-3))x+c where c is to be found. y=4x+c, and -1=12+c, plugging in (3,-1). c=-13. Therefore the axis of the hyperbola is y=4x-13.
The centre or origin of the hyperbola lies midway between them at ((1/2)(5+3),(1/2)(7-1))=(4,3) and the equation of the conjugate axis is given by y=-x/4+k where k is found by plugging in the midpoint: 3=-4/4+k, so k=4 and y=4-x/4 is the equation of the conjugate axis.
Let's define the axes of the hyperbola as X and Y instead of x and y. We can place the centre of the hyperbola at (0,0) in the X-Y plane so that in this plane the equation of the hyperbola is Y^2/A^2-X^2/B^2=1. The vertices are found by finding Y when X=0, so (0,A) and (0,-A) are the vertices. The asymptotes are given by (Y/A+X/B)(Y/A-X/B)=0. So Y=AX/B and Y=-AX/B are their equations.
The distance between the vertices is 2A so we can work this out from the x-y coordinates of the vertices using Pythagoras' Theorem: 2A=sqrt((7+1)^2+(5-3)^2)=sqrt(68). A=sqrt(68)/2, A^2=68/4=17.
Since the hyperbola is rectangular, B=A, and the equation of the hyperbola is Y^2-X^2=17. The asymptotes then become Y=X and Y=-X.
Now we have to transform X-Y to x-y, so that the equation can be expressed in the x-y plane. We use geometry and trigonometry for this.
Let O'(4,3) be the location of the origin of the X-Y plane. Let P(x,y) be any point. In the X-Y plane this is P'(X,Y). Join O'P'=O'P. This line can be represented in both reference frames. O'P^2=X^2+Y^2=(x-4)^2+(y-3)^2 by Pythagoras.
The gradient of the Y axis with respect to the x-y frame is 4, and we can represent this as tan(w) so w=tan^-1(4) or tan(w)=4; sin(w)=4/sqrt(17); cos(w)=1/sqrt(17). The angle z that O'P makes with the line y=3 is given by tan(z)=(y-3)/(x-4). sin(z)=(y-3)/O'P; cos(z)=(x-4)/O'P. Let v=90-w. sin(v+z)=Y/O'P; cos(v+z)=X/O'P. v+z=90-w+z=90-(w-z), so sin(v+z)=cos(w-z)=cos(w)cos(z)+sin(w)sin(z)=(x-4)/(O'Psqrt(17))+4(y-3)/(O'Psqrt(17))=(x+4y-16)/(O'Psqrt(17)).
cos(v+z)=sin(w-z)=sin(w)cos(z)-cos(w)sin(z)=4(x-4)/(O'Psqrt(17))-(y-3)/(O'Psqrt(17))=(4x-y-13)/(O'Psqrt(17)).
Y=O'P*sin(v+z)=(x+4y-16)/sqrt(17); X=O'P*cos(v+z)=(4x-y-13)/sqrt(17).
So P maps to the x-y plane using these transformations.
Y^2-X^2=17 becomes (x+4y-16)^2-(4x-y-13)^2=289, the equation of the hyperbola.
This expands to x^2+8x(y-4)+16y^2-128y+256-(16x^2-8x(y+13)+y^2+26y+169)=289;
-15x^2+16xy+15y^2+72x-154y-202=0.
The asymptotes are x+4y-16+4x-y-13=0; 5x+3y-29=0 and x+4y-16-4x+y+13=0; 5y-3x-3=0.