You’re on the right track, or should I say you’re on a right track. Your next step would be to divide the top row by 108 to get:
0 0 1 1/4
which yields the result z = 1/4 (which is correct).
The way you seem to be going about it looks like it will have bigger and bigger numbers. It might eventually get the right answers, but the chances of error grow.
The classic way (which is also error prone) is to set up the augmented matrix:
2 6 8 5
-2 9 -12 -1
4 6 -4 3
Now employ a process called “row reduction”.
The first step is to divide the top row by 2:
1 3 4 5/2
-2 9 -12 -1
4 6 -4 3
With a 1 in the first row of the first column, the idea is to put zeros in every other position in the first column.
To put a zero in the second row, first column; multiply the first row by 2 and add it to the second row:
1 3 4 5/2
0 15 -4 4
4 6 -4 3
To put a zero in the third row, first column; multiply the first row by -4 and add it to the third row:
1 3 4 5/2
0 15 -4 4
0 -6 -20 -7
That takes care of the first column. To get to work on the second column, divide the second row by 15:
1 3 4 5/2
0 1 -4/15 4/15
0 -6 -20 -7
Then put zeros in all the other positions in the second column (I’ll spare you the details):
1 0 24/5 17/5
0 1 -4/15 4/15
0 0 108/5 27/5
which takes care of the second column.
Finally, take care of the third column. Start by putting a 1 in the third row, third column by multiplying the third row by 5/108 to get:
1 0 24/5 17/5
0 1 -4/15 4/15
0 0 1 1/4
Add appropriate multiples of the third row to the second and first rows to put zeros in the third column to get:
1 0 0 1/2
0 1 0 1/3
0 0 1 1/4
and you’re done.
I’ve found that the best way to keep from making errors is to not be afraid to completely re-write the whole matrix for each step along the way. If you try to do too many steps in your head involving things like (-24/15)-20 you’re bound to make a mistake.