This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM3352: I2C issue

Part Number: AM3352
Other Parts Discussed in Thread: PCA9555, TPS65910

When the system starts up, it will have the following information:

U-Boot SPL 2018.01-g1272a9523a (Jul 31 2019 - 16:00:59)
Timed out in wait_for_bb: status=1000
Timed out in wait_for_bb: status=1000
Trying to boot from MMC1


U-Boot 2018.01-g1272a9523a (Jul 31 2019 - 16:00:59 +0000)

CPU  : AM335X-GP rev 2.1
Model: EPC-R3220 A101-1
DRAM:  1 GiB
NAND:  0 MiB
MMC:   OMAP SD/MMC: 0, OMAP SD/MMC: 1
SF: Detected w25q32bv with page size 256 Bytes, erase size 4 KiB, total 4 MiB
*** Warning - bad CRC, using default environment

Net:   SF: Detected w25q32bv with page size 256 Bytes, erase size 4 KiB, total 4 MiB
cpsw
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0 is current device
SD/MMC found on device 0
** Unable to read file boot.scr **
** Unable to read file uEnv.txt **
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
switch to partitions #0, OK
mmc0 is current device
SD/MMC found on device 0
4096512 bytes read in 388 ms (10.1 MiB/s)
40033 bytes read in 36 ms (1.1 MiB/s)
## Flattened Device Tree blob at 88000000
   Booting using the fdt blob at 0x88000000
   Loading Device Tree to 8fff3000, end 8ffffc60 ... OK

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.14.79-gbe5d9a5075 (oe-user@oe-host) (gcc version 7.2.1 20171011 (Linaro GCC 7.2-2017.11)) #1 PREEMPT Wed Jul 31 08:42:48 UT
[    0.000000] CPU: ARMv7 Processor [413fc082] revision 2 (ARMv7), cr=10c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: EPC-R3220 A101-1
[    0.000000] Memory policy: Data cache writeback
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 48 MiB at 0xbd000000
[    0.000000] CPU: All CPU(s) started in SVC mode.
[    0.000000] AM335X ES2.1 (neon)
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 260416
[    0.000000] Kernel command line: console=ttyO0,115200n8 root=PARTUUID=000254d6-02 rw rootfstype=ext4 rootwait
[    0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Memory: 976284K/1048576K available (8192K kernel code, 326K rwdata, 2676K rodata, 1024K init, 277K bss, 23140K reserved, 49152K cma-reserve highmem)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
[    0.000000]     vmalloc : 0xf0800000 - 0xff800000   ( 240 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xf0000000   ( 768 MB)
[    0.000000]     pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
[    0.000000]     modules : 0xbf000000 - 0xbfe00000   (  14 MB)
[    0.000000]       .text : 0xc0008000 - 0xc0900000   (9184 kB)
[    0.000000]       .init : 0xc0c00000 - 0xc0d00000   (1024 kB)
[    0.000000]       .data : 0xc0d00000 - 0xc0d51b40   ( 327 kB)
[    0.000000]        .bss : 0xc0d51b40 - 0xc0d972ac   ( 278 kB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000]  Tasks RCU enabled.
[    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[    0.000000] IRQ: Found an INTC at 0xfa200000 (revision 5.0) with 128 interrupts
[    0.000000] OMAP clockevent source: timer2 at 24000000 Hz
[    0.000020] sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 89478484971ns
[    0.000047] clocksource: timer1: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635851949 ns
[    0.000061] OMAP clocksource: timer1 at 24000000 Hz
[    0.000292] timer_probe: no matching timers found
[    0.000545] Console: colour dummy device 80x30
[    0.000600] Calibrating delay loop... 597.60 BogoMIPS (lpj=2988032)
[    0.118755] pid_max: default: 32768 minimum: 301
[    0.119007] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
[    0.119031] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
[    0.119993] CPU: Testing write buffer coherency: ok
[    0.120073] CPU0: Spectre v2: using BPIALL workaround
[    0.120902] Setting up static identity map for 0x80100000 - 0x80100060
[    0.121099] Hierarchical SRCU implementation.
[    0.121550] EFI services will not be available.
[    0.123485] devtmpfs: initialized
[    0.134897] random: get_random_u32 called from bucket_table_alloc+0x8c/0x1ac with crng_init=0
[    0.135469] VFP support v0.3: implementor 41 architecture 3 part 30 variant c rev 3
[    0.135846] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.135873] futex hash table entries: 256 (order: -1, 3072 bytes)
[    0.140301] pinctrl core: initialized pinctrl subsystem
[    0.141179] DMI not present or invalid.
[    0.141688] NET: Registered protocol family 16
[    0.144422] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.163336] omap_hwmod: debugss: _wait_target_disable failed
[    0.217005] cpuidle: using governor ladder
[    0.217050] cpuidle: using governor menu
[    0.222937] OMAP GPIO hardware version 0.1
[    0.236585] No ATAGs?
[    0.236608] hw-breakpoint: debug architecture 0x4 unsupported.
[    0.254738] edma 49000000.edma: TI EDMA DMA engine driver
[    0.256092] reg-fixed-voltage fixedregulator@2: could not find pctldev for node /ocp/l4_wkup@44c00000/scm@210000/pinmux@800/pinmux_wl12xx_gpio, deferri
[    0.259859] usbcore: registered new interface driver usbfs
[    0.259938] usbcore: registered new interface driver hub
[    0.260079] usbcore: registered new device driver usb
[    0.260416] omap_i2c 44e0b000.i2c: could not find pctldev for node /ocp/l4_wkup@44c00000/scm@210000/pinmux@800/pinmux_i2c0_pins, deferring probe
[    0.260496] omap_i2c 4802a000.i2c: could not find pctldev for node /ocp/l4_wkup@44c00000/scm@210000/pinmux@800/i2c1_pins, deferring probe
[    0.260656] media: Linux media interface: v0.10
[    0.260718] Linux video capture interface: v2.00
[    0.260852] pps_core: LinuxPPS API ver. 1 registered
[    0.260864] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.260893] PTP clock support registered
[    0.260938] EDAC MC: Ver: 3.0.0
[    0.261491] dmi: Firmware registration failed.
[    0.262014] omap-mailbox 480c8000.mailbox: omap mailbox rev 0x400
[    0.262439] Advanced Linux Sound Architecture Driver Initialized.
[    0.263116] Bluetooth: Core ver 2.22
[    0.263188] NET: Registered protocol family 31
[    0.263201] Bluetooth: HCI device and connection manager initialized
[    0.263222] Bluetooth: HCI socket layer initialized
[    0.263237] Bluetooth: L2CAP socket layer initialized
[    0.263291] Bluetooth: SCO socket layer initialized
[    0.264035] clocksource: Switched to clocksource timer1
[    0.274929] NET: Registered protocol family 2
[    0.275849] TCP established hash table entries: 8192 (order: 3, 32768 bytes)
[    0.275970] TCP bind hash table entries: 8192 (order: 3, 32768 bytes)
[    0.276080] TCP: Hash tables configured (established 8192 bind 8192)
[    0.276230] UDP hash table entries: 512 (order: 1, 8192 bytes)
[    0.276268] UDP-Lite hash table entries: 512 (order: 1, 8192 bytes)
[    0.276492] NET: Registered protocol family 1
[    0.277044] RPC: Registered named UNIX socket transport module.
[    0.277063] RPC: Registered udp transport module.
[    0.277073] RPC: Registered tcp transport module.
[    0.277082] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.278139] hw perfevents: no interrupt-affinity property for /pmu, guessing.
[    0.278278] hw perfevents: enabled with armv7_cortex_a8 PMU driver, 5 counters available
[    0.280112] workingset: timestamp_bits=14 max_order=18 bucket_order=4
[    0.285990] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.287021] NFS: Registering the id_resolver key type
[    0.287074] Key type id_resolver registered
[    0.287085] Key type id_legacy registered
[    0.287143] ntfs: driver 2.1.32 [Flags: R/O].
[    0.293896] bounce: pool size: 64 pages
[    0.294015] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)
[    0.294152] io scheduler noop registered
[    0.294163] io scheduler deadline registered
[    0.294462] io scheduler cfq registered (default)
[    0.294478] io scheduler mq-deadline registered
[    0.294489] io scheduler kyber registered
[    0.296100] pinctrl-single 44e10800.pinmux: 142 pins at pa f9e10800 size 568
[    0.298771] pwm-backlight backlight: backlight supply power not found, using dummy regulator
[    0.363493] Serial: 8250/16550 driver, 10 ports, IRQ sharing disabled
[    0.367385] omap_uart 44e09000.serial: no wakeirq for uart0
[    0.367646] 44e09000.serial: ttyO0 at MMIO 0x44e09000 (irq = 30, base_baud = 3000000) is a OMAP UART0
[    1.093650] console [ttyO0] enabled
[    1.098041] omap_uart 481a6000.serial: no wakeirq for uart3
[    1.104189] 481a6000.serial: ttyO3 at MMIO 0x481a6000 (irq = 31, base_baud = 3000000) is a OMAP UART3
[    1.114499] omap_uart 481a8000.serial: no wakeirq for uart4
[    1.120600] 481a8000.serial: ttyO4 at MMIO 0x481a8000 (irq = 32, base_baud = 3000000) is a OMAP UART4
[    1.131832] omap_rng 48310000.rng: Random Number Generator ver. 20
[    1.140181] ch7055_init
[    1.157105] brd: module loaded
[    1.168754] loop: module loaded
[    1.175460] m25p80 spi0.0: found w25q32, expected m25p80
[    1.181069] m25p80 spi0.0: w25q32 (4096 Kbytes)
[    1.185972] 1 ofpart partitions found on MTD device spi0.0
[    1.191740] Creating 1 MTD partitions on "spi0.0":
[    1.196814] 0x000000000000-0x000000400000 : "BootLoader"
[    1.205957] libphy: Fixed MDIO Bus: probed
[    1.284134] davinci_mdio 4a101000.mdio: davinci mdio revision 1.6, bus freq 1000000
[    1.292202] davinci_mdio 4a101000.mdio: detected phy mask ffffff7c
[    1.301888] libphy: 4a101000.mdio: probed
[    1.306251] davinci_mdio 4a101000.mdio: phy[0]: device 4a101000.mdio:00, driver RTL8211F Gigabit Ethernet
[    1.316383] davinci_mdio 4a101000.mdio: phy[1]: device 4a101000.mdio:01, driver RTL8211F Gigabit Ethernet
[    1.326585] davinci_mdio 4a101000.mdio: phy[7]: device 4a101000.mdio:07, driver RTL8211F Gigabit Ethernet
[    1.337870] cpsw 4a100000.ethernet: Detected MACID = c4:00:ad:38:3e:a9
[    1.344995] cpsw 4a100000.ethernet: initialized cpsw ale version 1.4
[    1.351681] cpsw 4a100000.ethernet: ALE Table size 1024
[    1.357259] cpsw 4a100000.ethernet: cpts: overflow check period 500 (jiffies)
[    1.365808] cpsw 4a100000.ethernet: cpsw: Detected MACID = c4:00:ad:38:3e:aa
[    1.374444] PPP generic driver version 2.4.2
[    1.379410] usbcore: registered new interface driver usbserial
[    1.385656] usbcore: registered new interface driver option
[    1.391569] usbserial: USB Serial support registered for GSM modem (1-port)
[    1.399288] i2c /dev entries driver
[    1.403507] IR NEC protocol handler initialized
[    1.408468] IR RC5(x/sz) protocol handler initialized
[    1.413786] IR RC6 protocol handler initialized
[    1.418573] IR JVC protocol handler initialized
[    1.423337] IR Sony protocol handler initialized
[    1.428210] IR SANYO protocol handler initialized
[    1.433155] IR Sharp protocol handler initialized
[    1.438117] IR MCE Keyboard/mouse protocol handler initialized
[    1.444269] IR XMP protocol handler initialized
[    1.451103] cpuidle: enable-method property 'ti,am3352' found operations
[    1.458684] sdhci: Secure Digital Host Controller Interface driver
[    1.465217] sdhci: Copyright(c) Pierre Ossman
[    1.470588] omap_hsmmc 48060000.mmc: Got CD GPIO
[    1.478391] sdhci-pltfm: SDHCI platform and OF driver helper
[    1.485338] ledtrig-cpu: registered to indicate activity on CPUs
[    1.495898] NET: Registered protocol family 10
[    1.502273] Segment Routing with IPv6
[    1.506402] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    1.513508] NET: Registered protocol family 17
[    1.518493] Bluetooth: RFCOMM TTY layer initialized
[    1.523636] Bluetooth: RFCOMM socket layer initialized
[    1.529136] Bluetooth: RFCOMM ver 1.11
[    1.533086] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[    1.538769] Bluetooth: BNEP filters: protocol multicast
[    1.544294] Bluetooth: BNEP socket layer initialized
[    1.549832] Key type dns_resolver registered
[    1.554657] omap_voltage_late_init: Voltage driver support not added
[    3.604069] omap_i2c 44e0b000.i2c: controller timed out
[    3.609714] tps65910 0-002d: No interrupt support, no core IRQ
[    5.684062] omap_i2c 44e0b000.i2c: controller timed out
[    6.374035] random: fast init done
[    7.764055] omap_i2c 44e0b000.i2c: controller timed out
[    9.844053] omap_i2c 44e0b000.i2c: controller timed out
[    9.849554] tps65910 0-002d: Error in configuring external control EN1
[    9.856441] tps65910 0-002d: Failed to initialise ext control config
[    9.863245] vrtc: supplied by vbat
[   11.924060] omap_i2c 44e0b000.i2c: controller timed out
[   14.004056] omap_i2c 44e0b000.i2c: controller timed out
[   14.034055] vrtc: failed to enable
[   14.037645] tps65910 0-002d: failed to register tps65910-pmic regulator
[   14.044670] tps65910-pmic: probe of tps65910-pmic failed with error -110
[   16.084069] omap_i2c 44e0b000.i2c: controller timed out
[   16.114060] rtc-s35390a 0-0030: error resetting chip
[   16.119971] rtc-s35390a: probe of 0-0030 failed with error -5
[   18.244062] omap_i2c 44e0b000.i2c: controller timed out
[   18.249562] adv-wdt-i2c 0-0029: adv_wdt_i2c_read: i2c transfer failed
[   18.256354] adv-wdt-i2c 0-0029: Advantech Watchdog Timer enabled. timeout=60s (nowayout=0), Ver.0
[   18.266085] pca953x 0-0022: 0-0022 supply vcc not found, using dummy regulator
[   20.324059] omap_i2c 44e0b000.i2c: controller timed out
[   20.354058] pca953x 0-0022: failed reading register
[   20.359359] pca953x: probe of 0-0022 failed with error -110
[   20.365513] omap_i2c 44e0b000.i2c: bus 0 rev0.11 at 50 kHz
[   20.372698] omap_i2c 4802a000.i2c: bus 1 rev0.11 at 50 kHz
[   20.379012] pwm-backlight backlight: backlight supply power not found, using dummy regulator
[   20.389745] omap_hsmmc 48060000.mmc: Got CD GPIO
[   20.399268] hctosys: unable to open rtc device (rtc0)
[   20.405366] lis3_reg: disabling
[   20.408684] vwl1271: disabling
[   20.411893] ALSA device list:
[   20.415127]   No soundcards found.
[   20.419634] Waiting for root device PARTUUID=000254d6-02...
[  216.114092] random: crng init done

This phenomenon is probabilistic. I keep restarting the board, more than 500 times appeared.

Because PMIC is on this I2C bus, the system can not start.

The SDK version we used is: 05.03.00.07

Do you have any suggestions for reference?

  • I don't believe your boot is failing because of I2C. It seems to be failing because it can't find the root partition on the SD Card or eMMC you are booting from on device 0.

    [   20.419634] Waiting for root device PARTUUID=000254d6-02...

    This should be the same device you booted from, but the second partition. This is all sent to the kernel from U-Boot. So, you may want to stop U-Boot at a prompt and use the mmc commands to explore what U-Boot is seeing.

  • Does the "waiting for root device" message coincide with the I2C bus busy and timeout messages precisely?  In other words, do you always see both issues or neither issue?  What is the state of the I2C bus?  Is either line stuck low in the failed boot case?

    For example in this thread there was a hardware issue with the board manufacturing that caused the issue.

  • Hi Brad:

    Thank you for your reply.

    Every time the system fails to start, I can see both issues. At this point, we found that i2c_SDA was pulled down.

  • Have you determined which device is pulling it low?  What devices are on the bus?

  • Hi Brad

    PCA9555/RTC/MSP430/TPM/tps65910

    We now remove them in turn for testing. But because it takes too long to test once, so the test results will take a long time to come out.

    Do you have any suggestions on this issue?

  • Implement a I2C software reset on boot.

    Toggle the CLK line, read the DATA line, until the DATA line is permanent HIGH.

    You should find some information on the internet if you search for I2C reset.

  • Hi Wolfgang:

    Thank you for your reply.

    The SDK version we use is 05.03.00.07. I have seen that i2c reset already exists in omap_i2c_wait_for_bb_valid and omap_i2c_wait_for_bb.

    omap_i2c_wait_for_bb_valid and omap_i2c_wait_for_bb will be called by omap_i2c_xfer. So I was wondering why it didn't work.

  • This comes down to  i2c_generic_scl_recovery().

    IMHO, this code has 2 errors:

    1) They are doing a set of SCL pulses, and end.

    The right thing to do would be using a special counter and counting only pulses where SDATA is HIGH after the pulse, and resetting the counter to zero if not.

    So the function would stop if N consecutive SDATA=HIGH will be encountered.

    2) They use N = 9. This will only work for simple I2C devices. If you have I2C devices which uses 4 bytes as address/data sequence, you will have to wait for N = 4 x 9 clocks.

  • zhengguang,

    I recommend that you complete your investigation to understand which specific device is holding the line low.  It would be useful to understand which device is having an issue and why.

    In addition, Wolfgang has made some good suggestions.  My understanding was that you should be able to keep toggling the SCL line and repeatedly checking the SDA line.  Once SDA goes high, then you should be able to issue a normal command.  The "start" condition of the command is supposed to force all I2C state machines back to a known good condition (ready for slave address).

    Best regards,
    Brad

  • Hi Brad:

    Thank you for your reply.

    We have removed TPM from i2c bus. Then it has been tested for three days and is still normal. We'll keep testing for longer.

  • Does the TPM have a reset pin that can be controlled from AM335x GPIO? Perhaps you could simply reset the TPM on every power-up to put it in a good state.

  • Brad wrote: "My understanding was that you should be able to keep toggling the SCL line and repeatedly checking the SDA line.  Once SDA goes high, then you should be able to issue a normal command."

    SDA == HIGH is not sufficient. SDA can go high in the middle of a transfer. If you want realy be sure that the I2C bus is clear, you need to have SDA == HIGH for the maximum transfer lenght of all devices on the bus.

    That means;

    Device 1 has a command/data sequence of 2 bytes = 18 clocks

    Device 2 has 4 bytes = 36 clocks

    etc...

    so you want to wait for SDA == HIGH for 36 consecutive clocks.

  • SDA high is sufficient  if all the slaves fully comply with the I2C spec. In that case an “unexpected start condition” will force their state machine back to waiting for a slave address.

    Also, I don’t agree with 36 clocks as a magic number. Most slaves support consecutive transfers so they could go on sending indefinitely (well, likely to some address boundary).

    It is critical to look for SDA to be released and not just blindly sending clocks.

  • Thinking about that some more, it shouldn’t go indefinitely since any data transfers should be NACK’d. So there probably are cases where that works. Perhaps we agree there is more than one way to recover an I2C bus!

  • Yes, the NAK will stop the data transfer. You only have to send enough clocks to terminate the longest data transfer.

    As this is only a recovery procedure, I do not mind if there are < 100 clocks in this situation.

    And yes, I have seen this behaviour with I2C devices in the past.

  • Hi Wolfgang:

    Thank you for your reply.

    I2C timeout time is 1s.

    Waiting for SDA == HIGH for 36 consecutive clocks. 36 clocks are 360ms.  I'm afraid there's a risk of overtime. 

    I'll try to import the solution and do a long time test.

  • Hi Brad:

    Thank you for your reply.

    TPM reset pin is not connected to CPU. 

    So far, after removing TPM, we have tested 4,000 times and it's normal.

    But TPM is needed for customers.

  • The slowest I2C clock pulse is 10us.

  • zhengguang yang said:
    So far, after removing TPM, we have tested 4,000 times and it's normal.

    That gives strong evidence not only that the TPM is the issue, but that it's the only problematic device.  So that is good to have it narrowed down.  Next I suggest looking at a board with the TPM present to try and understand why it is having the issue.  I suggest reaching out to the vendor.  Perhaps they have some ideas of why it would behave that way.  I would look carefully at power sequencing and reset timing to be sure you meet all requirements of the TPM.

    Are you able to modify one PCB to connect an AM335x GPIO pin to the TPM reset?  It would be another good test to understand if giving a reset at power-up to the TPM clears the condition.

    You should also try the I2C bus recovery method to see if you can clear the condition.  Given that it's a power-up failure and not a run-time issue, it might not be clearable short of resetting it.