Cam,
don't abandon it, your almost there!
http://en.wikipedia.org/wiki/Cross_product
Basically the tension in the link, the moment arm, and the resultant moment are all vectors. They have magnitude and direction. The moment is a result of the cross product of the force with the moment arm vector.
The error is in how you built up your A Matrix.