am335x realtime linux driver problem

Hi all, I try to make the am335x evm board to be a realtime linux system, using AM335x-PSP 04.06.00.02 and with the osadl patch :patch-3.2-rt10.patch. The problem is that ,when I select the config : Preemption Mode (Complete Preemption (Real-Time)) and make it.  the system make a very big latency in the latency test.  and when I disable ti's cpsw driver and any other net service support,  I can get a much better latency result.   But I have to use the net function in the system, how can I solve this problem? I suffer it for days, and had no way to deal with it.  Can some one tell me how to make the latency fighting? Thanks.