AM6442: Access to DDR4 controller register stalls indefinitely while in LPDDR4 frequency update

Dominic,

Just checking the place inside k3_lpddr4_info_handler where you see an indefinite stall is the driverdt->readreg below i colored red (copy paste is from https://git.ti.com/cgit/ti-u-boot/ti-u-boot/tree/drivers/ram/k3-ddrss/k3-ddrss.c )?  

#define TH_OFFSET_FROM_REG(REG, SHIFT, offset) do {\
	char *i, *pstr = xstr(REG); offset = 0;\
	for (i = &pstr[SHIFT]; *i != '\0'; ++i) {\
		offset = offset * 10 + (*i - '0'); } \
	} while (0)

static u32 k3_lpddr4_read_ddr_type(void)
{
	u32 status = 0U;
	u32 offset = 0U;
	u32 regval = 0U;
	u32 dram_class = 0U;

	TH_OFFSET_FROM_REG(LPDDR4__DRAM_CLASS__REG, CTL_SHIFT, offset);
	status = driverdt->readreg(&pd, LPDDR4_CTL_REGS, offset, &regval);
	if (status > 0U) {
		printf("%s: Failed to read DRAM_CLASS\n", __func__);
		hang();
	}

	dram_class = ((regval & TH_FLD_MASK(LPDDR4__DRAM_CLASS__FLD)) >>
		TH_FLD_SHIFT(LPDDR4__DRAM_CLASS__FLD));
	return dram_class;
}

  Pekka

Hello Pekka,

yes, that's where I'm seeing the stall. That read is performed when the LPDDR4 FSP switch is requested. Surprisingly this appears to work if the code gets there "fast enough" and fails if it is too slow, e.g. due to logging (I tried with a small delay-loop counting 1,000,000 to 0 -> fails, 100,000 to 0 -> works).

Regards,

Dominic

Hello Dominic,
I'm passing the e2e to my colleague on his follow-up comments on your inquiry.
Best,
-Hong

Hi Dominic, can you tell me which call is getting stuck?  There are 2 in the driver below.  I believe it is probably the one in ack_freq_upd_req(), since it is around that point that the PLL to the DDR subsystem is changing and there may be a window of time where a register read from the controller is problematic.  If that is the problematic read, can you modify the code to remove that register read, and just pass the value from the initial read in k3_ddrss_init_freq().  If this is successful, I will have to submit this as a code bug

static void k3_lpddr4_ack_freq_upd_req(void)
{
	u32 dram_class;

	debug("--->>> LPDDR4 Initialization is in progress ... <<<---\n");

	dram_class = k3_lpddr4_read_ddr_type();

	switch (dram_class) {
	case DENALI_CTL_0_DRAM_CLASS_DDR4:
		break;
	case DENALI_CTL_0_DRAM_CLASS_LPDDR4:
		k3_lpddr4_freq_update();
		break;
	default:
		printf("Unrecognized dram_class cannot update frequency!\n");
	}
}

static int k3_ddrss_init_freq(struct k3_ddrss_desc *ddrss)
{
	u32 dram_class;
	int ret;

	dram_class = k3_lpddr4_read_ddr_type();

Hello James,

it's the call from k3_lpddr4_ack_freq_upd_req. Yes, this is due to the DDR controller asking "the system" to switch to a new FSP.

Like I wrote in my original post we have a workaround that gets rid of the read by caching the read, and that solves the immediate issue. What I don't know, since the DDR controller is basically undocumented, is whether this is "by design" and expected, or if it's a mere side effect of an underlying problem.

The thing that I don't understand is why this read works when it happens "soon enough" and why it fails "later". E.g. if the controller is still accessible "earlier", then what happens if I switch the clock "too soon"?

If it can be explained why our workaround is necessary we want to bring it upstream, but it's kind of difficult for me to phrase WHY this change is necessary, when the SK-AM64 apparently works without (probably because it is "fast enough", but we've only got EVMs and our custom hardware, so I can't verify that assumption).

Regards,

Dominic

Dominic, i will need to investigate this with our design team.

Regards,

james

Hi Dominic, sorry for the delay.  Unfortunately, i don't have a good answer for you yet.  But i do have a question.  The frequency update is called several times during the initialization.  Do you know if the stall always occurs on the first call, or some of the subsequent ones?  

I can't see any reason why the R5F is stalling when accessing these registers.

Regards,

James

Hello James,

there are mulitple frequency update steps, but these are performed in a loop inside k3_lpddr4_freq_update(). The read from the controller register happens only once, when am64_lpddr4_info_handler() calls am64_lpddr4_ack_freq_upd_req(), before it enters k3_lpddr4_freq_update().

-> The stall occurs on the first call (there is no second call).

The code remains in am64_lpddr4_freq_update() for a while, waiting for clock-change requests, configuring the clock, and acknowledging the change, waiting for the request to be cleared. Cycling through the FSPs is apparently a state machine that is driven from within the DDR controller.

The actual sequence is this:

• the cadence driver calls the am64_lpddr4_info_handler at a point where the initialization sequence has been started, but the FSP change is not yet requested by the controller
• the k3-ddrss reads the control register to determine the type of memory (DDR4 vs. LPDDR4)
• am64_lpddr4_freq_update waits for the FSP change requests and performs the handshakes and clock changes, without touching any of the DDR controller registers, only CTRLMMR and clocking changes.

I've put in some debug code to measure how "long" it takes until the FSP change request is registered, by simply counting how often I can poll CTRLMMR_DDR4_FSP_CLKCHNG_REQ until bit 7 is set. That number varies greatly, between ~70 times and ~1600 times.

If I try and read the DDR control register as long as the FSP request is not yet set, I can successfully read every once in a while (1-2 out of 10) - I guess that's plausible, since reading the FSP request register and reading from the controller can't happen atomically, so by the time I read from the DDR control register the FSP request might already be set.

If I try and read the DDR control register (0x0f308000) AFTER the FSP change request is set, I have 100% failure rate (tried ~100 times).

It seems that the DDR control register can't be accessed once the FSP change request is set.

That means there's no chance that the clock switch is performed "too soon", since the code actually waits for the FSP change request bit to be set. If the code is "too slow" and the FSP frequency change request bit is already set by the controller, the DDR control register access stalls indefinitely.

Regards,

Dominic

Hi Dominic, i have discussed this with our team, and our theory here is that the PLL frequency change is temporarily stopping the clock to the DDR subsystem, and this is affecting the read from the controller register (the read is being missed by the controller since it has no clock).  Once the init sequence is started in the controller (with the write to the START bit), there is no way to determine the time when the FSP request is generated by the controller and when the PLL to the DDR subsystem is temporarily halted, so our recommendation is to not access any of the DDRSS regs once the init sequence is started until the last FSP is completed.  

The read of the DRAM_CLASS bits would have to come before the init is started with the START bit, which i think is what your workaround is doing.

Let me know if you need more info.  I can alert our software teams of this finding to modify the driver.

Regards,

James