Rotation matrix to quaternion