Let's assume that 3x^2-5xy=12y^2 is the intended second equation.
From the first equation, 3x^2=26+y^2, therefore, by substitution:
26+y^2-5xy=12y^2 and 26-5xy=11y^2, so x=(26-11y^2)/5y and x^2=(26-11y^2)^2/25y^2.
So 3(26-11y^2)^2/25y^2-y^2=26.
3(676-572y^2+121y^4)-25y^4=650y^2.
2028-1716y^2+363y^4-25y^4=650y^2 and 338y^4-2366y^2+2028=0, y^4-7y^2+6=0=(y^2-6)(y^2-1).
So y^2=6 or 1 and y=±√6 or ±1, so x^2=(26+y^2)/3=32/3 or 9, and x=±√(32/3) or ±3.
To decide which roots to take we look at 3x^2-5xy-12y^2=0. Consider 3x^2-12y^2: for (√(32/3),√6) 32-72=-40, and 5xy=-40, so xy=-8 and the roots are (√(32/3),-√6) or (-√(32/3), √6); or for (3,1) 27-12=15. So 5xy=15 and xy=3, therefore the solutions are (1,3) or (-1,-3). [Note that √(32/3)*√6=√64=8 and √32/3=4√6/3.]
ALTERNATIVE METHOD
3x^2-5xy-y^2=(3x+4y)(x-3y) so x=-4y/3 or 3y.
Substitute these in the first equation:
x=-4y/3:
3x^2-12y^2=26: 3(16y^2)/9-y^2=26; 13(y^2)/3=26; y^2=6, y=±√6, x=T4√6/3.
x=3y:
3x^2-y^2=26: 27y^2-y^2=26; 26y^2=26, y=±1, x=±3.
The solutions are (1,3), (-1,-3), (4√6/3,-√6), (-4√6/3,√6) by either method.