2x2y2-5xy-12 is a quadratic. To see this more clearly let z=xy then:
2z2-5z-12 is the quadratic and there are various ways to solve it. Here's one:
Identify the factors of 2: 1×2; factors of 12: 1×12, 2×6, 3×4.
So we have (z-a)(2z+b) or (z+a)(2z-b) and we have to find a and b. ab=12 and a and b have opposite signs
Expanding this: 2z2±z(2a-b)-ab=2z2-5z-12, so |2a-b|=|-5|=5, and we have all possible factors of 12, so we need to find which pair of factors out of the three pairs satisfies 2a-b=-5.
It's 3×4, because 2×4-3=8-3=5, so a=4 and b=3 (as you already found out!). Therefore (z-4)(2z+3) or (z+4)(2z-3) is the factorisation. Only the first of these gives -5z as the middle term.
Now we can replace z with xy: (xy-4)(2xy+3). (From this xy=z=4 or -3/2.)
The other ways are to use the quadratic formula: z=(5±√(25+96))/4=(5±√121)/4=(5±11)/4=4 or -3/2. This gives the factors z-4 and z+3/2 or 2z+3 (because z-4=0 and z+3/2=0 is the same as z=4 and z=-3/2 or 2z=-3).
Completing the square is effectively the same as using the quadratic formula.
Your method relies on the way you split -5xy into -8xy+3xy. This is a trial-and-error way, which may or may not work, because the quadratic may have had irrational factors. The quadratic formula or completing the square will solve the quadratic even if it has irrational or complex factors and it doesn't rely on trial and error. The method I used initially is based on Rational Zeroes theory. Basically it will determine fairly quickly whether there is a rational factorisation through logic, not trial and error.