EMAC EOQ flag problem on AM3505

I will forward this to the AM35x experts. Feedback will be posted directly here.

Just because there is no errata does not mean that it works as intended. It just means that no one has documented a problem, not that a problem does not exist. From our observations, the hardware does not work as described in the TRM, and we have to employ a workaround in order to queue EMAC descriptors when this potential race condition occurs. Furthermore, the Linux kernel davinci_emac driver employs the same workaround we do.

Here is the sequence of events:
1. Assemble a linked list of TX descriptors with OWNER = 1
2. Set TX0HDP to the head of the list
3. Create a new TX descriptor, and call it "new_tail" with OWNER = 1
4. Copy the address of the last TX descriptor in the linked list to "tail"
5. Set tail->next = new_tail
6. Read the EOQ flag for tail
7. The EOQ flag for tail is 0
8. Observe that not all buffers were transmitted, the last two buffers were never transmitted by the EMAC
9. Observe that the OWNER flag for tail and new_tail is still 1

This does not happen every time, but it does happen very often, and is very easy to reproduce.

Here is the relevant code from the Linux kernel:
[code=C]
send(...) {
...
tail_bd = txch->active_queue_tail;
tail_bd->next = curr_bd;
txch->active_queue_tail = curr_bd;
tail_bd = EMAC_VIRT_NOCACHE(tail_bd);
tail_bd->h_next = (int)emac_virt_to_phys(curr_bd, priv);
frame_status = tail_bd->mode;
if (frame_status & EMAC_CPPI_EOQ_BIT) {
emac_write(EMAC_TXHDP(ch),
emac_virt_to_phys(curr_bd, priv));
frame_status &= ~(EMAC_CPPI_EOQ_BIT);
tail_bd->mode = frame_status;
++txch->end_of_queue_add;
}
}
tx_irq() {
...
while ((curr_bd) &&
((frame_status & EMAC_CPPI_OWNERSHIP_BIT) == 0) &&
(pkts_processed < budget)) {
emac_write(EMAC_TXCP(ch), emac_virt_to_phys(curr_bd, priv));
txch->active_queue_head = curr_bd->next;
if (frame_status & EMAC_CPPI_EOQ_BIT) {
if (curr_bd->next) { /* misqueued packet */
emac_write(EMAC_TXHDP(ch), curr_bd->h_next);
++txch->mis_queued_packets;
} else {
txch->queue_active = 0; /* end of queue */
}
}

dma_unmap_single(emac_dev, curr_bd->buff_ptr,
curr_bd->off_b_len & EMAC_RX_BD_BUF_SIZE,
DMA_TO_DEVICE);

*tx_complete_ptr = (u32) curr_bd->buf_token;
++tx_complete_ptr;
++tx_complete_cnt;
curr_bd->next = txch->bd_pool_head;
txch->bd_pool_head = curr_bd;
--txch->active_queue_count;
pkts_processed++;
txch->last_hw_bdprocessed = curr_bd;
curr_bd = txch->active_queue_head;
if (curr_bd) {
BD_CACHE_INVALIDATE(curr_bd, EMAC_BD_LENGTH_FOR_CACHE);
frame_status = curr_bd->mode;
}
} /* end of pkt processing loop */
}
[/code]

If the EMAC hardware was working as described, then "mis_queued_packets" should always be 0, but it's not and it is constantly being incremented. Infact, if the hardware were working as described, the IRQ handler would have no need to write anything to TXHDP.

Yes, and all of this is described in the TRM chapter quoted in the initial post on this thread.

So is that an admission that there is a problem and that the EMAC hardware is not behaving as intended?

After updating the tail->next pointer of an active list, we always check the EOQ flag of the tail, but this value is always 0.  However, the newly appended descriptors were never transmitted, and their OWNER bit remains set to 1.  In other words, even though the EOQ bit is still 0, the EMAC has stopped transmission, but has not updated the EOQ bit., and the EOQ bit will not be updated until a TX complete IRQ is received/triggered.  Are you saying that this is the expected behavior?  In other words, the EOQ bit is not valid until an IRQ is triggered by the EMAC?

From the condition described it almost sounds like the queue was modified after the EOQ of the tail descriptor was checked but before the next descriptor pointer could be modified. Perhaps the port is detecting the EOQ just after the host is reading the descriptor. This would be the race condition described in the TRM.

It looks like it should be a misqueued packet condition which is what the race condition is called but the problem description does not match since the ownership bit is set.

The driver being used here have may have to have some type of scheduling event to insure all the packets were sent.

The TRM may imply that the tx complete event is not needed but there is still a race condition described and must be handled. The driver needs a method to look at the list again through some methods such as event, signal or polling to check the status of the tx descriptor queue.

The problem of the ownership bit set is not really a misqueued packet, it looks to be one where the descriptor is being appended to a completed list that the driver thinks is still active. Perhaps the driver needs a critical section around where the tx descriptor being appended so it is not interrupted. And check the EOQ bit after modification and before exiting the critical section to look for the misqueue packet then. This is just really a guess though.

It is not that the driver thinks the completed list is still active, but rather that the EMAC has not updated the EOQ bit faster enough for the driver to see it when the EMAC has completed iterating through the list.  We already have a critical section in place where the driver updates the list, and it already re-reads and checks the EOQ bit after updating the NEXT pointer and before exiting the critical section.  It looks like the EMAC is just updating the EOQ bit some time after it has determined that it has reached the end of the list, and that checking that NEXT == NULL and setting the EOQ bit in the EMAC is not an atomic operation.  In other words, in the time between when the EMAC checks the NEXT pointer and sets the EOQ bit, the driver has already updated the NEXT pointer and read the EOQ bit.

The sequence of events seems to be as follows:

1. EMAC reads the NEXT pointer, and it is NULL
2. Driver updates NEXT pointer
3. Driver reads EOQ bit, and it is 0
4. EMAC sets EOQ bit to 1 and OWNER bit to 0

Is it possible that if the EMAC is running at a slower clock rate, then the driver could update the queue and read the EOQ bit while the EMAC is reading the NEXT pointer.  This would imply that the EMAC does not atomically check the NEXT pointer and set the EOQ bit.

We have a workaround in place that waits for the TX complete interrupt to basically re-queue a packet that was misqueued.  The main question we had is whether or not this is the intended behavior of the EMAC, since the TRM seems to imply that this would be unnecessary, and that the race condition could be sufficiently handled by just re-reading and checking checking the EOQ bit after setting the NEXT pointer while it is in the active queue.  If this is the intended behavior, then it would be nice if the TRM were more explicit regarding this behavior (i.e. it could mention that the EOQ bit is not valid when the OWNER bit is set).