Write a file to a disk with O_DIRECT flag

Hello,

l will try to involve someone to help here with IPNC_RDK.

Best Regards,

Margarita

Constantin,

Three questions.

1. Why are you using O_DIRECT here? Are you targeting data integrity or performance?

2. write() with Vsys_allocBuf() when used without O_DIRECT is working fine?

3. Have you written some valid data to the array after allocation, at least one byte?

Thanks  for answer.

1. Yes, we are targeting performance.
2. As I  mentioned  before,  everything is working properly without O_DIRECT. Just it isn’t fast enough. With O_DIRECT it is much better.
3. Sure, we copy a frame to this array.

Can you reproduce this issue? Just open a device/file with O_DIRECT and try to copy the allocated to a SATA HD.

Best Regards,

Constantin 

Constantin,

I don't have a setup with SATA drive to reproduce the issue. But I have resolved similar performance issues before. So, let me know what is your current throughput with and without O_DIRECT?

What is the expected througput?

Also are you using O_SYNC | O_DIRECT? 

Yes, I'm using O_SYNC | O_DIRECT 

The expected throughput is  100 MB  and we have it when using O_SYNC | O_DIRECT and the frame is located in user space.

Regards,

Constantin

Constantin,

What is your current throughput? I feel that 100MB/sec is some thing difficult to achieve, some where close to 80MB/sec should be feasible. 

Hi Renjith,

We don't use file system. So when we have have a frame in user space and use O_DIRECT|O_SYNC speed can be more then 100 MB.

80 MB if use a file system and it depends from length of file and directories tree.

Thanks,

Constantin

Constantin,

I still didn't understand what is the current throughput your getting. 

Also which is the IO scheduler that you are using? Is it CFQ/Deadline or Noop? Have you tried changing the IO scheduler and analyzed the throughput difference?

Hi Renjith, 

The  throughput without O_DIRECT option is about 50 MBytes. 

I don't know which IO scheduler I'm using and I cannot check it today  but I didn't try to change IO scheduler, so it should be default.

I just want to remind of my question: " I'm using O_DIRECT for write to SSD. It is working properly when the array is allocated in user space via malloc(). But when I try to use an array which allocated via Vsys_allocBuf() the write() return -1."

Regards,

Constatin

Constantin,

Sorry, I missed the original question completely. 

According to the man page, "The O_DIRECT flag may impose alignment restrictions on the length and address of userspace buffers and the file offset of I/Os".

So could you please print the addresses of the buffers in both cases and share the logs? Also what is the buffer sizes used? This is just to be doubly sure.

But what I think the real reason could be, when you use O_DIRECT is because the physical pool where Vsys_allocBuf() doesn't belong to kernel's memory range. I'll really have to check the kernel code to see which layer is returning the error code. 

Hi,

I allocate 4718592 (4096*1152) from start of CMEM the address 0x85000000. I'm sure it's alighned

One time more: when buffer allocated in user space everything is working properly.

Regards,

Constantin

Constantin,

constantin ostrovsky said:

when buffer allocated in user space everything is working properly.

When the buffer is allocated from user space, the buffer actually belongs to kernel's memory address pool. Ultimately the physical address belongs to the Linux's address space. Whereas CMEM pool is outside the kernel's (user space + kernel space) pool. 

Need to look deep in the kernel code to understand how the buffers outside of kernel's memory pool is handled.

So, can I use O_DIRECT when a file located in CMEM pool?

If not, can I use DMA for move a video  frame to a buffer allocated in user space

Regards,

Constantin

Constantin,

I am not sure about this as I haven't tried the same. It should be possible to do it, but I think we've to take care of this pool in kernel as well. Block layer code needs to be analyzed more. Using DMA to user space buffer may not be really a good solution for this issue. But there might be some kind of workaround possible. I am not sure about the workaround though.

OK,

So can you please suggest a way to save a stream of raw video frames (speed about 100MBytes)  to a SATA hard drive?

Regards,

Constantin

Constantin,

I have some basic questions.

1. Are you particular about storing raw frames or is it okay for you to store compressed frames as well?

2. What is the current throughput that you are getting now with the SSD you are using? Ideally SSD should give 2-3 times better throughput when compared to HDD. I have not used SSD in DM81xx. 

3. What is the chunk size you are trying this file write with? Have you tried changing the chunk size and observed any performance difference. 

In my opinion, optimizing the SATA driver performance will enable you in achieving 100MB/sec on SSD. On DM816x we've achieved close to 80MB/sec with O_SYNC | O_DIRECT of regular files with 128KB block size. 

Hi Renjith,

1. We have to  store raw images

2. If the data located in user space we can get  100 MBytes/sec with O_DIRECT option. 

3. I tried different chunk sizes from 2^12 to 2^22.

Can you please suggest how to optimize the  SATA driver performance?

Regards,

Constantin

Constantin,

constantin ostrovsky said:

2. If the data located in user space we can get  100 MBytes/sec with O_DIRECT option. 

Since you are already getting 100MB/sec there is no need to optimize SATA driver. Then it looks likes to me that if we can crack the failure issue when CMEM buffers are used, you problem will get resolved. If you can spare one hardware setup, I can try to debug this issue. 

Hi 

I use the DN814x/AM387x evaluation module + SSD and the demo from  /demos /mcfw_api_demos/multich_usercase  of ipnc_rdk.

Thanks,

Constantin

Constantin,

I don't have an SSD with me. I have an DM814x EVM with me. If you can spare one SSD I can try this here.

Hi,

You can use any fast SATA drive.

Thanks,

Constantin