We are running a simple filter recursion equation y(n) = b0*x(n) + b1*x(n-1) + b2*x(n-2) - a0*y(n-1) - a1*y(n-2)
We had a case where a certain combination of filter coefficients cause the filter to become unstable resulting in overflow.
Under some cases, we get a filter output of infinity (0x7F800000) ... this is the expected behavior.
However, in other cases, we get a filter output of NaN (0x7FFFFFFF). This is not expected.
We are setting the round mode in FADCR and FMCR to their default value of 0 (Round toward nearest representable floating-point number)
So the question... why could either a ADDSP or an MYPSP produce a NaN value of 0x7FFFFFFF. Shouldn't the ALU be forcing the output to a valid floating point number (even if that number is infinity?