Vref = 2.5V
VDD = 5V
Sweeping the differential input Vin = VINP-VINN, and taking 1024 samples for each Vin, I get the following plot:
In red, I show the ideal transfer curve, and in blue my samples. In the y axis I just converted the number of counts to volts to make it easier to read.
The measured transfer curve agrees with the ideal transfer curve from Vin = -4.84V to Vin = 4.85V. I took three samples close to the points where it fails, which is hard to see in the transfer curve plot because they are too close. But here are the histograms for the different cases when Vin is close to VDD and -VDD.
When Vin is close to -VDD, it seems that it wants to toggle between +5V and -5V, but it also shows values in between.
This is when it started to fail.
For Vin > -4.84V the ADC works perfect.
For Vin higher than 4.85V it starts to fail again, and note that the output is toggling to -2.5V when it should be saturating (clipping) to 2.5V all the time.
Here it really fails, and -2.5V gets more counts than 2.5V, even though 2.5V is the expected output.
Thanks a lot and please let me know if I need to provide more details about the driving circuit.