OK, start from the first equation, a^2  b^2 = ab ± 1. Since a and b are both threedigit numbers, ab is going to be five to seven digits, meaning that that ±1 is almost irrelevant. So to a very good approximation, a^2  ab + b^2 = 0. Defining x == a/b, this means that x^2  x  1 ~= 0, or x ~= (1+sqrt(5))/2 ~= 1.61803 (incidentally, the Golden Ratio, phi). In other words, a ~= 1.61803 * b, or ab ~= 0.61803*b is a perfect square, or b is a threedigit perfect square times 1.61803 . That only leaves us nine numbers to check, 100 to 324 (any higher square would leave a and/or b with four digits).
Just computing the b values corresponding to those perfect squares, I notice very quickly that phi*144 is very, very close to an integer. So let's start there. phi*144 = 232.99632, so call it 233. And 233*phi = 377.001919, also really close to an integer. So it looks really likely that our numbers will be 233 and 377.
377^2  233^2 = 142129  54289 = 87840, and 233*377 = 87841. Looks like we've got it.
3
