PROCESSOR-SDK-AM335X: ADC FSM BUSY Issue

Gary Wu

Hi,

My customer is doing final prototype test before mass production. And they found that one of 16 prototypes reported an ADC error.

The SDK version is 05.00.00.15. I know that this verion is old. But they have developed several products on this for many years.

Is there any idea about this issue?

I have suggested customer to test latest SDK to check if there is the same issue. But the time is very tight.

The problem is as below:
1- The application is stuck while getting the ADC, the staus of the process is "D (Disk sleep)"
2- The stack of the process is as below:
[<c0599c9c>] am335x_tsc_se_set_once+0x138/0x1a0
[<c0781f3c>] tiadc_read_raw+0xe4/0x1b4
[<c077d174>] iio_channel_read+0xa8/0xb8
[<c077d220>] iio_read_channel_raw+0x4c/0x6c
[<bf09c070>] wfet_adc_read+0x70/0xb8 [wfos_adc]
[<bf0a4358>] wfet_battery_voltage_get_by_chan+0xc0/0xf8 [wfos_pm]
[<bf0a4650>] wfet_battery_read+0x54/0xdc [wfos_pm]
[<c0274684>] __vfs_read+0x38/0x12c
[<c0274810>] vfs_read+0x98/0x11c
[<c0274cf0>] SyS_read+0x4c/0xa0
[<c0108180>] arm_elf_read_implies_exec+0x50/0x50
[<ffffffff>] 0xffffffff

They have found the finally position of the stack, it is in ti_am335x_tscadc.c and the function is am335x_tscadc_need_adc

static void am335x_tscadc_need_adc(struct ti_tscadc_dev *tscadc)
{
DEFINE_WAIT(wait);
u32 reg;

regmap_read(tscadc->regmap, REG_ADCFSM, &reg);
if (reg & SEQ_STATUS) {
tscadc->adc_waiting = true;
prepare_to_wait(&tscadc->reg_se_wait, &wait,
TASK_UNINTERRUPTIBLE);
spin_unlock_irq(&tscadc->reg_lock);

schedule();

spin_lock_irq(&tscadc->reg_lock);
finish_wait(&tscadc->reg_se_wait, &wait);

/*
* Sequencer should either be idle or
* busy applying the charge step.
*/
regmap_read(tscadc->regmap, REG_ADCFSM, &reg);
WARN_ON((reg & SEQ_STATUS) && !(reg & CHARGE_STEP));
tscadc->adc_waiting = false;
}
tscadc->adc_in_use = true;
}

when the REG_ADCFSM is busy, then the process enter in sleep mode and wait for wake-up, but nobody wake it,

so the process stuck all the time.

over 3 years ago

0 Nick Saulnier over 3 years ago

TI__Guru** 100740 points

Hello Gary,

It looks like the am335x_tscadc_need_adc function in the ADC driver is the same in SDK 8.2 as it is in SDK 5.0. Please note that I will not be able to fully support questions about SDK 5.0, since that release is pretty old.

To confirm: This behavior is only observed on one of the boards? Is the behavior repeatable? Are there any differences between the problem board and the other boards?

Regards,

Nick

0 Gary Wu over 3 years ago in reply to Nick Saulnier

TI__Expert 3313 points

Hi Nick,

Thanks for your reply.

The issue can be reproduced in other boards. Not only one deticated board.

0 Gary Wu over 3 years ago in reply to Nick Saulnier

TI__Expert 3313 points

Hi Nick,

Compared to SDK5.0, did the newer SDK fix this ADC bug ? I didn't find this item in SDK6/7/8 release note.

0 Nick Saulnier over 3 years ago in reply to Gary Wu

TI__Guru** 100740 points

Hello Gary,

What steps are taken to reproduce the issue?

It has been years since I did a deep dive on ADC programming, but this thread may be useful for your customers: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/688440/am3354-adc-fsm-busy-issue

Regards,

Nick

0 Gary Wu over 3 years ago in reply to Nick Saulnier

TI__Expert 3313 points

Hello Nick,

Yes, this thread is useful. But customer don't know well about ADC driver. And the MP schedule is very tight for customer.

They want to know how to do this workaround as below shown:

"polling for STEP_ID == IDLE or waiting two adc_clk cycles before writing to STEPENABLE”

Could you help which file and which line ?

Thanks.

0 Gary Wu over 3 years ago in reply to Nick Saulnier

TI__Expert 3313 points

Hello Nick,

Can they call "schedule" funtion instead of "schedule_timeout" function?

Does it make sense to fix the issue?

0 Nick Saulnier over 3 years ago in reply to Gary Wu

TI__Guru** 100740 points

Hello Gary,

I did not have time to dig into the driver today. I can take a look tomorrow.

Regards,

Nick

0 Nick Saulnier over 3 years ago in reply to Nick Saulnier

TI__Guru** 100740 points

In the meantime, please attach the steps needed to replicate the customer's results. I also expect to see a terminal log for the crash.

Regards,

Nick

0 Gary Wu over 3 years ago in reply to Nick Saulnier

TI__Expert 3313 points

Hello Nick,

It's not easy to reproduce it on your side.

Cutomer are setting 48 unit prototypes running at the same time for several days. For now they found 3 of them can see this issue. For now, customer is still finding out how to reproduct it regularly.

And they dumped ADC registers when the issue happens.

address：		value：
0x44E0D000		0x47300001
0x44E0D010		0x0000000C
0x44E0D024		0x0000000E
0x44E0D028		0x00000000
0x44E0D02C		0x00000000
0x44E0D030		0x00000000
0x44E0D034		0x00000000
0x44E0D038		0x00000000
0x44E0D03C		0x00000000
0x44E0D040		0x00000007
0x44E0D044		0x00000030
0x44E0D048		0x00000000
0x44E0D04C		0x00000007
0x44E0D050		0x00000000
0x44E0D054		0x00000002
0x44E0D058		0x00000000
0x44E0D05C		0x00000000
0x44E0D060		0x00000001
0x44E0D064		0x05843010
0x44E0D068		0x00000098
0x44E0D06C		0x058C3010
0x44E0D070		0x00000098
0x44E0D074		0x05943010
0x44E0D078		0x00000098
0x44E0D07C		0x059C3010
0x44E0D080		0x00000098
0x44E0D084		0x05A43010
0x44E0D088		0x00000098
0x44E0D08C		0x05AC3010
0x44E0D090		0x00000098
0x44E0D094		0x05B43010
0x44E0D098		0x00000098
0x44E0D09C		0x05BC3010
0x44E0D0A0		0x00000098
0x44E0D0A4		0x00000000
0x44E0D0A8		0x00000000
0x44E0D0AC		0x00000000
0x44E0D0B0		0x00000000
0x44E0D0B4		0x00000000
0x44E0D0B8		0x00000000
0x44E0D0BC		0x00000000
0x44E0D0C0		0x00000000
0x44E0D0C4		0x00000000
0x44E0D0C8		0x00000000
0x44E0D0CC		0x00000000
0x44E0D0D0		0x00000000
0x44E0D0D4		0x00000000
0x44E0D0D8		0x00000000
0x44E0D0DC		0x00000000
0x44E0D0E0		0x00000000
0x44E0D0E4		0x00000001
0x44E0D0E8		0x00000000
0x44E0D0EC		0x00000000
0x44E0D0F0		0x00000000
0x44E0D0F4		0x00000013
0x44E0D0F8		0x00000000
0x44E0D100		0x0000090C
0x44E0D200		0x00040909

The ADC is used to be as general ADC to monitor battary voltage of external battary.

0 Nick Saulnier over 3 years ago in reply to Gary Wu

TI__Guru** 100740 points

I am not going to replicate on my side, since SDK 5.0 is too old for me to support. However, I need to have an idea of how they are using the driver before I spend a bunch of time doing a deep dive on the driver and the registers.

so:

* When are they seeing the behavior? i.e., is this happening during Linux boot and ADC initialization? Is this happening during runtime?

* How are they using the ADC? Information like one-shot mode vs continuous mode, how many reads are done in what timeframe, etc are important.

* What commands are they using?

* What is the structure of their userspace code? (assuming they are writing a userspace program)

* What is their devicetree entry?

If the code is customer sensitive, feel free to send to me offline.

Regards,

Nick

0 Gary Wu over 3 years ago in reply to Nick Saulnier

TI__Expert 3313 points

Hello Nick,

* When are they seeing the behavior? i.e., is this happening during Linux boot and ADC initialization? Is this happening during runtime?

Gary: It happened after Linux boot. They have added log store to check whether ADC initialization or runtime.

* How are they using the ADC? Information like one-shot mode vs continuous mode, how many reads are done in what timeframe, etc are important.

Gary: one-shot mode

Read 4 times per 1 second

* What commands are they using?

Gary: you can refer below log for commands they used

* What is the structure of their userspace code? (assuming they are writing a userspace program)

Yes, open dev and read

* What is their devicetree entry?

&tscadc{

status = "okay";

adc{

ti,adc-channels = <0 1 2 3 4 5 6 7>;

};

Processors

Processors forum

PROCESSOR-SDK-AM335X: ADC FSM BUSY Issue