• State Resolved
  • Replies 9 replies
  • Subscribers 98 subscribers
  • Views 299 views
  • Users 0 members are here
Support feedback
Options
Options
Related

AM6412: OTP Programming with keywriter sometimes fails

Lee
Prodigy 100 points
Part Number: AM6412
Other Parts Discussed in Thread: SK-AM64B, TCA6424

Tool/software:

When loading and executing 'keywriter.bin' (renamed from 'tiboot3.bin') built from the MCU SDK 10.1.0.32 to program the OTP on an AM6412, it usually succeeds the first time but sometimes fails. After failing, if re-attempted, it may succeed, heating the board seems to increase the probability of success.
We captured and decoded the TISCI traces from UART1 and captured the keywriter output from UART0.
When it fails:
UART1:
Configuring trace data version to: 0x03007
0x00C20101:   BasePort: Unknown Action: 0x03 MSG:0x020101
0x00C20024:   BasePort: Unknown Action: 0x03 MSG:0x020024
0x61800101: Power Management:    MSG_RECEIVED(TI-SCI message received): Message ID: 0x00000101
0x61C0008A: Power Management: MSG_PARAM_DEV_CLK_ID(TI-SCI message content: dev/clk-ids): Device ID: 138 Clock ID: 0
0x00C20104:   BasePort: Unknown Action: 0x03 MSG:0x020104
0x00C20024:   BasePort: Unknown Action: 0x03 MSG:0x020024
0x61800104: Power Management:    MSG_RECEIVED(TI-SCI message received): Message ID: 0x00000104
0x61C0008A: Power Management: MSG_PARAM_DEV_CLK_ID(TI-SCI message content: dev/clk-ids): Device ID: 138 Clock ID: 0
0x00C20100:   BasePort: Unknown Action: 0x03 MSG:0x020100
0x00C20024:   BasePort: Unknown Action: 0x03 MSG:0x020024
0x61800100: Power Management:    MSG_RECEIVED(TI-SCI message received): Message ID: 0x00000100
0x61C0008A: Power Management: MSG_PARAM_DEV_CLK_ID(TI-SCI message content: dev/clk-ids): Device ID: 138 Clock ID: 0
0x62000000: Power Management: MSG_PARAM_VAL(TI-SCI message content: value): Target Value: 0x00000000
0x00C2010D:   BasePort: Unknown Action: 0x03 MSG:0x02010D
0x00C20024:   BasePort: Unknown Action: 0x03 MSG:0x020024
0x6180010D: Power Management:    MSG_RECEIVED(TI-SCI message received): Message ID: 0x0000010D
0x61C0008A: Power Management: MSG_PARAM_DEV_CLK_ID(TI-SCI message content: dev/clk-ids): Device ID: 138 Clock ID: 0
0x612D7C83: Power Management: CLOCK_SET_RATE(Clock Frequency has been changed: significand * 2^Exponent Hz): Clock ID: 131 Clock Frequency{significand}: 95 Clock Frequency{Exponent}: 22
0x00C2010C:   BasePort: Unknown Action: 0x03 MSG:0x02010C
0x00C20024:   BasePort: Unknown Action: 0x03 MSG:0x020024
0x6180010C: Power Management:    MSG_RECEIVED(TI-SCI message received): Message ID: 0x0000010C
0x61C0008A: Power Management: MSG_PARAM_DEV_CLK_ID(TI-SCI message content: dev/clk-ids): Device ID: 138 Clock ID: 0
0x612D7C83: Power Management: CLOCK_SET_RATE(Clock Frequency has been changed: significand * 2^Exponent Hz): Clock ID: 131 Clock Frequency{significand}: 95 Clock Frequency{Exponent}: 22
0x612D7C83: Power Management: CLOCK_SET_RATE(Clock Frequency has been changed: significand * 2^Exponent Hz): Clock ID: 131 Clock Frequency{significand}: 95 Clock Frequency{Exponent}: 22
0x00C2010E:   BasePort: Unknown Action: 0x03 MSG:0x02010E
0x00C20024:   BasePort: Unknown Action: 0x03 MSG:0x020024
0x6180010E: Power Management:    MSG_RECEIVED(TI-SCI message received): Message ID: 0x0000010E
0x61C0003D: Power Management: MSG_PARAM_DEV_CLK_ID(TI-SCI message content: dev/clk-ids): Device ID: 61 Clock ID: 0
UART0:
<nothing>
When it succeeds:
UART1:
Configuring trace data version to: 0x03007
0x00C20101:   BasePort: Unknown Action: 0x03 MSG:0x020101
0x00C20024:   BasePort: Unknown Action: 0x03 MSG:0x020024
0x61800101: Power Management:    MSG_RECEIVED(TI-SCI message received): Message ID: 0x00000101
0x61C0008A: Power Management: MSG_PARAM_DEV_CLK_ID(TI-SCI message content: dev/clk-ids): Device ID: 138 Clock ID: 0
0x00C20104:   BasePort: Unknown Action: 0x03 MSG:0x020104
0x00C20024:   BasePort: Unknown Action: 0x03 MSG:0x020024
0x61800104: Power Management:    MSG_RECEIVED(TI-SCI message received): Message ID: 0x00000104
0x61C0008A: Power Management: MSG_PARAM_DEV_CLK_ID(TI-SCI message content: dev/clk-ids): Device ID: 138 Clock ID: 0
0x00C20100:   BasePort: Unknown Action: 0x03 MSG:0x020100
0x00C20024:   BasePort: Unknown Action: 0x03 MSG:0x020024
0x61800100: Power Management:    MSG_RECEIVED(TI-SCI message received): Message ID: 0x00000100
0x61C0008A: Power Management: MSG_PARAM_DEV_CLK_ID(TI-SCI message content: dev/clk-ids): Device ID: 138 Clock ID: 0
0x62000000: Power Management: MSG_PARAM_VAL(TI-SCI message content: value): Target Value: 0x00000000
0x00C2010D:   BasePort: Unknown Action: 0x03 MSG:0x02010D
0x00C20024:   BasePort: Unknown Action: 0x03 MSG:0x020024
0x6180010D: Power Management:    MSG_RECEIVED(TI-SCI message received): Message ID: 0x0000010D
0x61C0008A: Power Management: MSG_PARAM_DEV_CLK_ID(TI-SCI message content: dev/clk-ids): Device ID: 138 Clock ID: 0
0x612D7C83: Power Management: CLOCK_SET_RATE(Clock Frequency has been changed: significand * 2^Exponent Hz): Clock ID: 131 Clock Frequency{significand}: 95 Clock Frequency{Exponent}: 22
0x00C2010C:   BasePort: Unknown Action: 0x03 MSG:0x02010C
0x00C20024:   BasePort: Unknown Action: 0x03 MSG:0x020024
0x6180010C: Power Management:    MSG_RECEIVED(TI-SCI message received): Message ID: 0x0000010C
0x61C0008A: Power Management: MSG_PARAM_DEV_CLK_ID(TI-SCI message content: dev/clk-ids): Device ID: 138 Clock ID: 0
0x612D7C83: Power Management: CLOCK_SET_RATE(Clock Frequency has been changed: significand * 2^Exponent Hz): Clock ID: 131 Clock Frequency{significand}: 95 Clock Frequency{Exponent}: 22
0x612D7C83: Power Management: CLOCK_SET_RATE(Clock Frequency has been changed: significand * 2^Exponent Hz): Clock ID: 131 Clock Frequency{significand}: 95 Clock Frequency{Exponent}: 22
0x00C2010E:   BasePort: Unknown Action: 0x03 MSG:0x02010E
0x00C20024:   BasePort: Unknown Action: 0x03 MSG:0x020024
0x6180010E: Power Management:    MSG_RECEIVED(TI-SCI message received): Message ID: 0x0000010E
0x61C0003D: Power Management: MSG_PARAM_DEV_CLK_ID(TI-SCI message content: dev/clk-ids): Device ID: 61 Clock ID: 0
0x00C20002:   BasePort: Unknown Action: 0x03 MSG:0x020002
0x00C20024:   BasePort: Unknown Action: 0x03 MSG:0x020024
0x04003007:   BasePort: TRACE_DATA_VERSION(OSAL/Baseport trace data version): Trace version major: 0x03 Trace version minor: 0x007
0x04400A08:   BasePort:   SYSFW_VERSION(System Firmware version): version: 10 subversion: 0 patch: 8
0x00409031:   BasePort: TISCI_MSG_RECEIVED(TISCI Message interrupt handled): Queue ID: 0 Message ID: 9031
0x00800023:   BasePort: TISCI_MSG_SENDER_HOST_ID(Message from secure host received): Queue ID: 0 Host ID: 35
Note: the output is the same as the failing attempt, except for six additional lines at the end.
UART0:
Combined boot mode
Starting Keywriting
Enabled VPP
DMSC Firmware Version 10.0.8-v10.00.08_am64x_keywrite
DMSC Firmware revision 0xa
DMSC ABI revision 4.0
keys Certificate found: 0x70011b80
Keywriter Debug Response:0x0
Success Programming Keys
Can you please help us understand what could cause the failures? The 'Unknown Action' (0x3) doesn't seem to be listed in https://software-dl.ti.com/tisci/esd/latest/4_trace/trace.html#trace-debug-data-format is it defined elsewhere? Let us know if there's any further information we can share.
Thanks,
Lee
  • 0
    TI__Guru 69181 points

    Hello,

    Lee said:
    UART0:
    <nothing>

    Can you please connect to the R5FSS0-0 core using the CCS and see where it is stuck?

    Thanks!

  • 0 in reply to Prashant Shivhare
    Prodigy 100 points

    Hi Prashant,

    Building the keywriter.bin is somewhat procedural for us, we did just enough to create and include our keys, certs and enable Vpp. We don't normally use CCS, nor have we developed any application firmware for the R5F.

    How do we 'connect' to the R5F core? Is there some additional h/w (e.g. a JTAG debug probe) necessary? Is there a specific document you can point us to that would help us get this up and running?

    Thanks,

    Lee

  • 0 in reply to Lee
    Prodigy 100 points
    Hi Prashant,
    We tried our keywriter.bin file on one of our SK-AM64B evaluation boards. Note we did modify the SK-AM64B board to allow USB DFU to work. While the Vpp enable control is different to our board, we're not aware of any other h/w dependencies. We were expecting it to get further. Interestingly, it didn't, it failed in the same way.
    So tried the same experiment again on the SK-AM64B, only this time without making any modifications to the reference keywriter source, except for the following (see '//....') in 'sbl_keywriter/am64x-evm/r5fss0-0_nortos/board.c':
     
        /* set VPP core */
        if (status == SystemP_SUCCESS)
        {
            status = TCA6424_config(&TCA6424_IOexp_config, EFUSE_VPP_PIN, TCA6424_MODE_OUTPUT);
        }
        if (status == SystemP_SUCCESS)
        {
            // don't enable Vpp (so we can retry w/o the OTP being committed)
            //status = TCA6424_setOutput(&TCA6424_IOexp_config, EFUSE_VPP_PIN, TCA6424_OUT_STATE_HIGH);
            status = TCA6424_setOutput(&TCA6424_IOexp_config, EFUSE_VPP_PIN, TCA6424_OUT_STATE_LOW);
        }
    Again, this failed in the same way. I also tried heating the board, that did not affect the outcome either.
    FYI, we are using:
        mcu_plus_sdk_am64x_10_01_00_32-linux-x64
        otp_keywriter_am64x_v10.00.08_am64x_keywriter-linux
        sysconfig-1.21.2_3837
        CCS12.8.1.00005_linux-x64
    I will endeavor to send you the keywriter.bin (tiboot3.bin) binary, with just the one change shown above.
    Can you try on one of your eval boards and confirm you see the same failing outcome?
    Thanks,
    Lee
  • 0 in reply to Lee
    TI__Guru 69181 points

    Hello,

    The keywriter package does not provide the support for the SK-AM64B board. If you are still running the provided keywriter example on this board then the failure is expected for the reasons described in the following response:

    https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1473209/processor-sdk-am64x-flashing-otp-keywriter-per-sd-card/5669823

    Regards,

    Prashant

  • 0 in reply to Prashant Shivhare
    Prodigy 100 points

    Hi Prashant,

    First, note our board does have an eMMC (on MMC0) and we don't use an I2C IO expander for Vpp enable. We are confident that we're controlling Vpp enable using a GPIO.

    None-the-less, we'd still like to see the OTP programming steps run on the SK-AM64B eval board.

    We implemented the changes suggested in https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1473209/processor-sdk-am64x-flashing-otp-keywriter-per-sd-card/5669823 by removing the references to the EMMC from the example.syscfg file:

    ~/.../sbl_keywriter/am64x-evm/r5fss0-0_nortos/ti-arm-clang$ diff ../example.syscfg.orig ../example.syscfg
    44,48c44,48
    < bootloader1.appImageOffset      = "0x800000";
    < bootloader1.appImageBaseAddress = "0";
    < bootloader1.bootMedia           = "EMMC";
    < bootloader1.$name               = "CONFIG_BOOTLOADER_EMMC0";
    < bootloader1.EMMCAppImageOffset  = "0x800000";
    ---
    > //bootloader1.appImageOffset      = "0x800000";
    > //bootloader1.appImageBaseAddress = "0";
    > //bootloader1.bootMedia           = "EMMC";
    > //bootloader1.$name               = "CONFIG_BOOTLOADER_EMMC0";
    > //bootloader1.EMMCAppImageOffset  = "0x800000";
    55,58c55,58
    < const mmcsd             = scripting.addModule("/drivers/mmcsd/mmcsd", {}, false);
    < const mmcsd1            = mmcsd.addInstance({}, false);
    < mmcsd1.$name            = "CONFIG_MMCSD0";
    < bootloader1.MMCSDDriver = mmcsd1;
    ---
    > //const mmcsd             = scripting.addModule("/drivers/mmcsd/mmcsd", {}, false);
    > //const mmcsd1            = mmcsd.addInstance({}, false);
    > //mmcsd1.$name            = "CONFIG_MMCSD0";
    > //bootloader1.MMCSDDriver = mmcsd1;
    219c219
    < mmcsd1.MMC0.$suggestSolution                = "MMC0";
    ---
    > //mmcsd1.MMC0.$suggestSolution                = "MMC0";

    Please confirm this is effectively the same change as suggested using the GUI.

    We also implemented the changes suggested in https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1198808/faq-how-to-run-otp-keywriter-on-sk-am64b, noting:

    • the path mentioned in the helper script contained a ‘tifs’ sub-dir, we removed it (as it doesn’t exist) in the build

    • we set ‘Vpp enable' LOW as we don’t want it to actually write the OTP, just attempt to do so (and see it fail)

    Everything compiles.

    No difference to the original outcome. Specifically the UART1 (J12) trace output stops at:
    0x61C0003D: Power Management: MSG_PARAM_DEV_CLK_ID(TI-SCI message content: dev/clk-ids): Device ID: 61 Clock ID: 0

    and no output on UART0 (J11).

    If the trace output stops at '0x61C0003D', does any code from 'sbl_keywriter/am64x-evm/r5fss0-0_nortos/ti-arm-clang/', i.e. main.c or board.c, even run?

    Also (an earlier question), how do we 'connect' to the R5F core? Is there some additional h/w (e.g. a JTAG debug probe) necessary? Is there a specific document you can point us to that would help us get this up and running?

    Thanks,

    Lee

  • 0 in reply to Lee
    Prodigy 100 points

    Hi Prashant,

    Update: We do now see the expected OTP programming steps occur on the SK-AM64B eval board.

    We'll now focus our attention on our board and the differences between the keywriter firmware.

    Thanks,

    Lee 

  • 0 in reply to Lee
    TI__Guru 69181 points

    Hello,

    Lee said:
    Also (an earlier question), how do we 'connect' to the R5F core? Is there some additional h/w (e.g. a JTAG debug probe) necessary? Is there a specific document you can point us to that would help us get this up and running?

    Please see if the following guide helps:

    https://software-dl.ti.com/mcu-plus-sdk/esd/AM64X/11_01_00_17/exports/docs/api_guide_am64x/CCS_LAUNCH_PAGE.html

    https://software-dl.ti.com/mcu-plus-sdk/esd/AM64X/11_01_00_17/exports/docs/api_guide_am64x/CCS_SETUP_PAGE.html

    Regards,

    Prashant

  • 0 in reply to Prashant Shivhare
    Prodigy 100 points

    Hi Prashant,

    Thanks for your continued support.

    If we remove (comment-out) the eMMC related parameters in the example.syscfg, our boards that previously failed to program OTP now succeed. The exact change is included in an earlier post. This gets us over our immediate OTP programming issue.
    However, our board does use an eMMC, indeed we boot and run our application code from it. Should we be concerned that the original keywriter.bin sometimes 'hangs' when initializing the eMMC? In such cases, applying heat and re-attempting succeeds. Can you think of anything we can try to re-assure ourselves that the observed/marginal behavior only occurs when using keywriter.bin and not when we're booting/running our normal application code?

    Thanks,

    Lee
  • +1 in reply to Lee
    TI__Guru 69181 points

    Hello,

    Lee said:
    Can you think of anything we can try to re-assure ourselves that the observed/marginal behavior only occurs when using keywriter.bin and not when we're booting/running our normal application code?

    Without knowing the eMMC failure signature, I can't say for sure if the issue won't occur in the normal application code. The keywriter doesn't have any use of eMMC so it should definitely not include the eMMC. As for the application, you may test it continuously. If the issue occurs, please feel free to create a separate thread.

    Thanks!