This turned out to be a rather interesting one. The water-tight
ray/triangle intersection algorithm, while very accurate for
finding if there is an intersection with a line segment, is
not as remarkably accurate for determining if that intersection
is within the interval of the ray.
This is because of the coordinate transformation it does
depending on ray direction: for triangles laying flat on one of
the axis planes near zero, that near-zero coordinate can get
transformed to a much less accurate space for testing. In fact,
generally speaking, beause of the coordinate transform, you can
only rely on the test being as accurate as the least accurate
axis.
The ray-origin offset code was doing offsets based on the
assumption that the error on the major axes are independent, but
as this triangle intersection algorithm shows, you can't actually
depend on that being the case. So rather than handling triangle
intersection as a special case, I've changed the intersection
position error to be a single float, representing the maximum
possible error on any axis. This should be robust for any
geometry type added in the future, and also solves the immediate
issue in a correct way.
BVH traversal still happens in local space, but final actual
surface intersection calculations are done in world space by
transforming the triangle into world space. This is to improve
numerical consistency between intersections.