[Gtkradiant] [Bug 1063] New: Cap selection consumes 100% CPU and does nothing

[Michael]

bugzilla-daemon at zerowing.idsoftware.com wrote:

>http://zerowing.idsoftware.com/bugzilla/show_bug.cgi?id=1063
>
>           Summary: Cap selection consumes 100% CPU and does nothing
>           Product: GtkRadiant
>           Version: 1.5
>          Platform: PC
>        OS/Version: Linux
>            Status: NEW
>          Severity: normal
>          Priority: P2
>         Component: editor
>        AssignedTo: gtkradiant at zerowing.idsoftware.com
>        ReportedBy: ondrej at svetlik.info
>
>
>After installing gtkradiant-1.5.0-2005-06-20.i386.rpm I tried to create some
>simple arena in doom3 and q3a mode. I added an end cap, inverted it's matrix and
>tried to cap the selection with inverted endcap. Then I had to kill radiant, it
>was using 100%CPU with no result.
>
>  
>

Not 100% sure if this is related but ....

There's a problem in BuildVertexArray in radiant/patch.cpp.

The vector3_normalise() function it uses can go into an infinite loop if 
the sqrt() is passed a nan.
I've noticed this compiling from source with -march=k8, the sse 
instruction sequence ends with

  8352:       f2 0f 51 c0           sqrtsd %xmm0,%xmm0
  8356:       66 0f 2e c0          ucomisd %xmm0,%xmm0
  835a:       75 d5                   jne    8331 
<_ZN5Patch16BuildVertexArrayEv+0xe81>
  835c:       7a d3                   jp     8331 
<_ZN5Patch16BuildVertexArrayEv+0xe81>

The 2 conditional jumps send you back to the start of the inline 
function [i.e 100%cpu loop]

Quick fix, don't divide by zero / check with isnan(foo) before doing 
sqrt etc.

But since it's doing this irrespective of this code sequence, it's 
obviously not the real bug [e,g compiling with debug doesn't seem to 
emit code that causes the hang but
the code is still creating nan's - although it's difficult to see what 
negative effect this has?]

A better fix needs some understanding of what it's actually trying to do 
- some of the tangents it picks to normalise are degenerate :-

For example, create a new map, do curve cylinder, shift-c and then 
choose 'cylinder' from the popup, in BuildVertexArrays, after the
tangentX and tangentY arrays are calculated and 
tangents_remove_degenerate is called the tangent arrays look like this :-

(gdb) print tangentY
$2 = {{m_elements = {0, -64, 0}}, {m_elements = {64, 0, 0}}, {m_elements 
= {7.66589433e-39,
      0, 0}}, {m_elements = {64, 0, 0}}, {m_elements = {0, 64, 0}}, 
{m_elements = {64, 0, 0}}}
(gdb) print tangentX
$3 = {{m_elements = {0, 64, 0}}, {m_elements = {0, 64, 0}}, {m_elements 
= {0, 64, 0}}, {
    m_elements = {0, 64, 0}}, {m_elements = {0, 64, 0}}, {m_elements = 
{0, 64, 0}}}

The degenerate flags were nFlagsX == 9 and nFlagsY == 2.

The TangentsX for the cap are all the same. Later, in BestTangents00, the
index0 and index1 start with 0, 0, which is correctly detected as 
degenerate, but the
index0 and index1 are set to 2 and 0 [because nflagsX and degen_1a is false]

But TangentX[2] and TangentY[0] is still degenerate - and the subsequent 
processing using those tangents results in a nan when the null vector 
generated from the crossproduct is normalised.
This is then added to m_tess.m_vertices corrupting that too [and, as I 
think the bug reporter and I have seen, causes an infinite loop with 
some of the code generated]

Hope that helps,
-- 
Michael.