Hi
Base on my previous questions Could you let me know what I am doing wrong, I am tray to calculate gain error based on equation 13 in datasheet.
Ex:
Measured (V) | Actual (V) | |
Vmin | 0.00994 | 0.00941 |
Vmax | 0.24449 | 0.24329 |
Gain Error = 1- (0.24449-0.00994)/(0.24329-0.00941) = -0.00286472
= -0.00286472 / (1.19*10^-7) = -24073 => HEX => FFFFFFA1F7
Based on datasheet the result should be between 0-2 (2^24-1)/2^24 not negative.
Even I try just calculating slop (y=mx+b). As you can see the result is worst after calibration.
EX:
m = 1.0055730019019 so I program register with 80F091
Thanks