TUSB9261 very slow sync writes

Christophe Lorenz

Other Parts Discussed in Thread: TUSB9261

We need to use synchronous writes to disk using the TUSB9261 - that is writes that bypass os and possibly disk caching to make sure data is written to disk in case of power failure or other bad events.

We understand it has a performance impact, however using the TUSB9261 it is huge.

Cached writes : 165MB/sec

Sync writes : 12MB/sec

Our previous PCB design was using competitors bridges ICs, that also suffered some perf impact, but not so much :

Cached writes : 155MB/sec

Sync writes : 65MB/sec

The tests were done using the same hard drives and software install, just the bridge and hub have changed to TI brand.

Those are real application figures, raw sequential IO perform a bit better, in the range of 45-55MB/sec.

Application runs under linux, FW is 1.05.

Do you have some details on the impact of sync writes handled in by firmware ?

(I actually suspect the competitor IC was not honouring the sync write completely)

over 8 years ago

0 Elias Villegas M. over 8 years ago

TI__Guru* 76130 points

Hello,
Do you have access to a USB protocol analyzer? That will be very helpful.
Regards

0 Christophe Lorenz over 8 years ago in reply to Elias Villegas M.

Intellectual 270 points

No ... :-(

0 Brian Quach over 8 years ago

TI__Expert 4325 points

Default FW for TUSB9261 does not allow OS to disable write cache via Mode Select command. So you must have a custom version of FW. Cached vs non-cached is handled completely by the SATA device. We only send the SET FEATURES command to disable or enable the write cache. When flushing the cache, 9261 does not respond until the sync is completed by the SATA device.

0 Christophe Lorenz over 8 years ago in reply to Brian Quach

Intellectual 270 points

So you mean a custom FW that allows disabling write cache on the sata device would perform better in this case ?

0 Brian Quach over 8 years ago in reply to Christophe Lorenz

TI__Expert 4325 points

I can't speculate on performance. It depends entirely on the SATA device.

0 Brian Quach over 8 years ago in reply to Brian Quach

TI__Expert 4325 points

You can try this FW to determine if there is any difference.

TUSB926x_FW_v1.06_write-cache_changeable.zip

0 Christophe Lorenz over 8 years ago in reply to Brian Quach

Intellectual 270 points

Could you make a version with No polarity swap and another with RX swapped for the SATA lines ?

Thanks.

0 Brian Quach over 8 years ago in reply to Christophe Lorenz

TI__Expert 4325 points

Here's all varieties:

5657.TUSB926x_FW_v1.06_write-cache_changeable.zip

0 Christophe Lorenz over 8 years ago in reply to Brian Quach

Intellectual 270 points

Great ! thanks a lot !
I'll try that tomorrow.

Chris.

0 Christophe Lorenz over 8 years ago in reply to Christophe Lorenz

Intellectual 270 points

We tried the cache control enabled version but there's no noticable performance improvement.

We also enabled UAS which provided 20-30% more speed in all cases, sync or async, but still sync write is extremely slow.

When the host detects the drive, I still see

sd 12:0:0:0: [sdh] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

Which is no different between the cache control enabled or the vanilla firmware.

Shouldn't the DPO or FUA flags be controllable to bypass caching ?

Using the competitor's IC, the cache settings were somewhat different :

sd 126:0:0:0: [sdc] No Caching mode page found
sd 126:0:0:0: [sdc] Assuming drive cache: write through

That just makes me think that we were not actually performing sync writes - hence the higher speed.

Chris.

0 Brian Quach over 8 years ago in reply to Christophe Lorenz

TI__Expert 4325 points

I fixed the FUA reporting:

[596227.000709] usb 1-2: New USB device found, idVendor=0451, idProduct=9261
[596227.000714] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[596227.000718] usb 1-2: Product: TUSB9261 Firmware v1
[596227.000720] usb 1-2: Manufacturer: Texas Instruments
[596227.000723] usb 1-2: SerialNumber: DA7CD570E2F68D111316522843E1DF372
[596227.033491] usb-storage 1-2:1.0: USB Mass Storage device detected
[596227.033866] scsi5 : usb-storage 1-2:1.0
[596228.053500] scsi 5:0:0:0: Direct-Access TS512GMTS800I 6KB PQ: 0 ANSI: 6
[596228.057781] sd 5:0:0:0: Attached scsi generic sg2 type 0
[596228.079530] sd 5:0:0:0: [sdb] 1000215216 512-byte logical blocks: (512 GB/476 GiB)
[596228.094626] sd 5:0:0:0: [sdb] Write Protect is off
[596228.094633] sd 5:0:0:0: [sdb] Mode Sense: 17 00 10 00
[596228.110465] sd 5:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
[596228.164321] sdb: sdb1
[596228.249139] sd 5:0:0:0: [sdb] Attached SCSI disk

Please try the attached FW and let me know the result:

TUSB926x_Firmware_v1.06_FUA_fix.zip

0 Christophe Lorenz over 8 years ago in reply to Christophe Lorenz

Intellectual 270 points

After checking with a native SATA port, I found out that DPO and FUA are not controllable on any of the drives we have.

So the firmware is not in cause, as it just reports what the drive supports as you indicated.

Digging into linux kernel, the behaviour when caching is not controllable is to issue a flush command after every write.

This is clearly the cause of the slow performance.

I'll check with the drive manufacturer engineers why those cache flags are not supported.

They however confirmed that the hardware cache flushing is not an issue even in case of power loss, the drives have enough power reserve and momentum to perform the flush properly.

Currently we disabled the sync on write and it is causing us filesystem reliability issues during the tests.

So the next question is - can you release a firmware that doesn't expose the cache feature page ?

Linux would then treat the drive as "write through cache" effectively suppressing the flush after every write, while enabling us to keep the sync options in the filesystem.

Also you talked about monitoring the serial port output - I already did that for another purpose, but the data that comes out is undocumented and not obvious.

Do you have some more information on the data format ? (Serial decoding was ok but I decoded some binary values with no meaning for me)

Thanks.

Chris.

0 Brian Quach over 8 years ago in reply to Christophe Lorenz

TI__Expert 4325 points

Chris,
DPO and FUA aren't controllable but if the drive supports FUA, you can use use it. I'm surprised you drive doesn't support it. Did you try the new FW? Any change in performance?

Even with cache features page reported, you should always be able to use non-sync writes.

The serial port output is standard ASCII and human readable. But you need an appropriate adapter as it's 3.3V.
For example:

========================================================
|| TUSB926x Firmware v1.05 [Oct 3 2014 16:33:57] ||
|| Device ID: 0x0000 ||
========================================================

Reset Flag(s): [SW] [Watchdog] [Power-Up]

[0000000001] Datapath RAM Usage: 80208 / 81920 bytes.
[0000000001] Supported NCQ Depth: 32
[0000000001] U1/U2 Transistions: OFF
[0000000001] USB PHY Suspend: ON
[0000000001] SATA LPM: OFF
[0000000001] Device is Self-powered.
[0000000001] -> usb_hal_init()
[0000000001] USB Core Ver: 0x120a.
[0000000001] USB SSC is OFF.
[0000000051] -> usb_hal_connect()
[0000000051][0000000051] LTSSM state = (0x5) RX DETECT.
-> ahci_init()
[0000000051] -> ahci_hba_reset()
[0000000057] SATA Gen-2 speed negotiated.
[0000000065]
[0000000065] ==========================[=====================
[0000000065] IDENTIFY DEVICE INFO
[0000000065] ================================================
[0000000065]
[0000000065] Model: TS512GMTS800I
[0000000065] FW Rev: N1126KB
[0000000065] Serial: C441060023
[0000000065] TRIM Support: Yes
[0000000065]
[0000000065] Spec Compliance: ATA-7
[0000000065] Removable Media: No
[0000000065] UDMA Modes = 0x407f
[0000000065] PIO Modes = 0x0003
[0000000065]
[0000000065] LBA48: Yes
[0000000065] Max LBA = 0x00000000 3b9e12b0
[0000000065] Write FUA: Yes
[0000000065] World Wide Name: N/A
[0000000065]
[0000000065] SATA Speed: Gen3
[0000000065] NCQ Support: Yes
[0000000065] Queue Depth = 31
[0000000065]
[0000000065] Logical Sector Size = 512 bytes
[0000000065] Physical Sector Size = 512 bytes
[0000000065] Logical Sector Offset = 0
[0000000066]
[0000000066] ================================================

[0000000069] Connected to 1 AHCI device(s).
[0000000069] Connected at SUPER speed.

0 Christophe Lorenz over 8 years ago in reply to Brian Quach

Intellectual 270 points

To be sure, I reflashed 24 bridges and ran another test with various disks.

The firmware now correctly reports DPO/FUA on drives that support it :

Feb 8 10:43:32 amsterdam kernel: usb 6-1.3.2: new SuperSpeed USB device number 22 using xhci_hcd
Feb 8 10:43:32 amsterdam kernel: usb 6-1.3.2: New USB device found, idVendor=0451, idProduct=9261
Feb 8 10:43:32 amsterdam kernel: usb 6-1.3.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Feb 8 10:43:32 amsterdam kernel: usb 6-1.3.2: Product: Maud bridge V3 UAS NOCACHE
Feb 8 10:43:32 amsterdam kernel: usb 6-1.3.2: Manufacturer: Maud Technology
Feb 8 10:43:32 amsterdam kernel: usb 6-1.3.2: SerialNumber: 534E31363039503030303034534132413031
Feb 8 10:43:32 amsterdam kernel: usb-storage 6-1.3.2:1.0: USB Mass Storage device detected
Feb 8 10:43:32 amsterdam kernel: scsi host17: usb-storage 6-1.3.2:1.0
Feb 8 10:43:32 amsterdam mtp-probe: checking bus 6, device 22: "/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:02.0/0000:04:00.0/usb6/6-1/6-1.3/6-1.3.2"
Feb 8 10:43:32 amsterdam mtp-probe: bus: 6, device: 22 was not an MTP device
Feb 8 10:43:45 amsterdam kernel: scsi 17:0:0:0: Direct-Access WDC WD6001F4PZ-49CWH RAE1 PQ: 0 ANSI: 6
Feb 8 10:43:45 amsterdam kernel: sd 17:0:0:0: Attached scsi generic sg3 type 0
Feb 8 10:43:45 amsterdam kernel: sd 17:0:0:0: [sdd] 1465130646 4096-byte logical blocks: (6.00 TB/5.45 TiB)
Feb 8 10:43:45 amsterdam kernel: sd 17:0:0:0: [sdd] Write Protect is off
Feb 8 10:43:45 amsterdam kernel: sd 17:0:0:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
Feb 8 10:43:45 amsterdam kernel: sd 17:0:0:0: [sdd] 1465130646 4096-byte logical blocks: (6.00 TB/5.45 TiB)
Feb 8 10:43:45 amsterdam kernel: sdd: sdd1
Feb 8 10:43:45 amsterdam kernel: sd 17:0:0:0: [sdd] 1465130646 4096-byte logical blocks: (6.00 TB/5.45 TiB)
Feb 8 10:43:45 amsterdam kernel: sd 17:0:0:0: [sdd] Attached SCSI disk

Note that only a very small set of drives we use support the FUA control, despite being enterprise grade...

But worst, it didn't improve the performance at all, we are still at 15MB/sec on those, just like the others.
We have to dig further in the kernel to find out what's causing this... or find a protocol analyser that can sniff what's happening on the bus.

In the meantime we really would like to test a FW version that doesn't report the cache support page.
At least we could reach the performance and confidence level we had in our previous boards.
Also patching the kernel is a solution, but it's not very user friendly way of adressing the problem...

If a no cache page version still suffer from bad performance, I'm running out of ideas for the cause.

Chris.