This turned out to be a rather interesting one. The water-tight
ray/triangle intersection algorithm, while very accurate for
finding if there is an intersection with a line segment, is
not as remarkably accurate for determining if that intersection
is within the interval of the ray.
This is because of the coordinate transformation it does
depending on ray direction: for triangles laying flat on one of
the axis planes near zero, that near-zero coordinate can get
transformed to a much less accurate space for testing. In fact,
generally speaking, beause of the coordinate transform, you can
only rely on the test being as accurate as the least accurate
axis.
The ray-origin offset code was doing offsets based on the
assumption that the error on the major axes are independent, but
as this triangle intersection algorithm shows, you can't actually
depend on that being the case. So rather than handling triangle
intersection as a special case, I've changed the intersection
position error to be a single float, representing the maximum
possible error on any axis. This should be robust for any
geometry type added in the future, and also solves the immediate
issue in a correct way.
For some reason the ulp incrementing is unreliable when starting
at zero. It creates subnormal numbers, and that seems to be an
issue somewhere in the pipeline, ultimately leading to weird
render artifacts. Not entirely sure why.
This fixes it by avoiding subnormal numbers in the final offset
ray origin. Left a note suggesting investigating in more detail
at some point.
