PMU with Custom Module

John Demme

Other Parts Discussed in Thread: 4460, 4430

Hello All,

I'm trying to write a custom module on top of the Android 4.1 kernel to use the OMAP4460's performance counters. Before you ask, no, I cannot use perf or oprofile; they do not yield the information I need. This is rather odd data I'm attempting to collect.

Regardless, I can't get the counters to, well, count. I'm using the v7_pmu.S code from:

http://code.google.com/p/mycodespot/wiki/DirectPMUCodeGCC

Along with my module code, which I've put here:

http://gist.github.com/3289014#file_pmu_sync_sample_main.c

And build it against the Android 3.0.31 kernel ("3.0.31-g6fb96c9-dirty"). I'm then running it on a rooted Samsung Galaxy Nexus. Examples of the dmesg output can be found on the above page.

I'm having two problems:

1) After configuring and enabling the counters, they remain at 0 -- they don't count. The cycle counter, however, seems to work OK. I've looked at a lot of code for using these counters -- from oprofile, perf and the directPMU link above -- and as far as I can tell, I'm doing the same thing. Yet no luck. What am I missing? Is there another bit somewhere that needs to be flipped? Is Linux enabling some oddball security mode (with which I'm not familiar) that has to be disabled?

2) If I sleep for a second before reading the counters, they come up as gibberish. It's like some other process (or part of the kernel) is using them. In addition to disabling all PMU stuff in my kernel's config, I've grepped the kernel code for all writes to the PMU registers and commented them out. I am now running a custom kernel, thought it made no difference. (I only found PMU register writes in some SMC security code and the perf_event code.) So what could be interfering? Are these registers being context swapped? Is there some other code I haven't found that uses them or the PMU?

I've been banging my head against the wall on this for longer than I'd care to admit, so I'd appreciate some help.

Thanks in advance,

John

over 13 years ago

0 Magdalena Iovescu over 13 years ago

TI__Intellectual 2505 points

Hello John,

Have you seen this thread about a similar issue: http://e2e.ti.com/support/omap/f/849/p/184885/667248.aspx#667248

Regards,

Magdalena

0 John Demme over 13 years ago in reply to Magdalena Iovescu

Prodigy 90 points

Yes I have. Unfortunately, none of the pointers from that response helped.

0 Gina Glaser over 13 years ago in reply to John Demme

TI__Expert 8055 points

John,

Something people have run into in the past is that, in order to get to the counters, you need to clock them. If you are connecting a debugger, then these clocks I believe are automatically enabled. For example:

        writereg(OMAP4430_CM_EMU_CLKSTCTRL, 0x2);
        writereg(OMAP4430_CM_L3INSTR_L3_3_CLKCTRL, 0x1);
        writereg(OMAP4430_CM_L3INSTR_L3_INSTR_CLKCTRL, 0x1);
        writereg(OMAP4430_PRM_PRM_PROFILING_CLKCTRL, 0x1);

Regards,
Gina

0 Jon Hunter over 13 years ago in reply to Gina Glaser

TI__Expert 3245 points

Gina Glaser said:

Something people have run into in the past is that, in order to get to the counters, you need to clock them. If you are connecting a debugger, then these clocks I believe are automatically enabled. For example:

        writereg(OMAP4430_CM_EMU_CLKSTCTRL, 0x2);
        writereg(OMAP4430_CM_L3INSTR_L3_3_CLKCTRL, 0x1);
        writereg(OMAP4430_CM_L3INSTR_L3_INSTR_CLKCTRL, 0x1);
        writereg(OMAP4430_PRM_PRM_PROFILING_CLKCTRL, 0x1);

If you are using an OMAP4460 then only the EMU power domain needs to be enabled to get the PMU counters working. To enable the EMU power domain you just need to turn on the clocks to the EMU power domain by writing 0x2 to OMAP4430_CM_EMU_CLKSTCTRL as Gina mentions above. The other 3 register writes are only necessary for OMAP4430 that does not have dedicated PMU interrupts and PMU events are routed using the cross trigger interface (CTI).

I have tested PMU with the latest linux kernel with OMAP4460 plus a patch series I am keeping here:

git@gitorious.org:linux-omap-dev/linux-omap-dev.git -b dev-pmu

Cheers
Jon

0 John Demme over 13 years ago in reply to Jon Hunter

Prodigy 90 points

Thanks for the help. I think this has gotten me one step closer, but I'm still missing something. I've added this code:

__raw_writel(2, OMAP4430_CM_EMU_CLKSTCTRL);
__raw_writel(1, OMAP4430_CM_L3INSTR_L3_3_CLKCTRL);
__raw_writel(1, OMAP4430_CM_L3INSTR_L3_INSTR_CLKCTRL);
__raw_writel(1, OMAP4430_CM_PRM_PROFILING_CLKCTRL);

Which seems to have helped a bit. At boot, the register contents were:

[ 135.988250] State: OMAP4430_PM_EMU_PWRSTCTRL 0x30000
[ 135.988494] State: OMAP4430_PM_EMU_PWRSTST 0x0
[ 135.988647] State: OMAP4430_RM_EMU_DEBUGSS_CONTEXT 0x101
[ 135.988891] State: OMAP4430_CM_EMU_CLKSTCTRL 0x3
[ 135.989044] State: OMAP4430_CM_EMU_DYNAMICDEP 0x4000040
[ 135.989288] State: OMAP4430_CM_EMU_DEBUGSS_CLKCTRL 0x9060001
[ 135.989471] State: OMAP4430_CM_L3INSTR_L3_3_CLKCTRL 0x1
[ 135.989685] State: OMAP4430_CM_L3INSTR_L3_INSTR_CLKCTRL 0x1
[ 135.989868] State: OMAP4430_CM_PRM_PROFILING_CLKCTRL 0x20001

So the module was definitely disabled. After writing, I get the state to:

[ 135.990264] State: OMAP4430_PM_EMU_PWRSTCTRL 0x30000
[ 135.990417] State: OMAP4430_PM_EMU_PWRSTST 0x37
[ 135.990661] State: OMAP4430_RM_EMU_DEBUGSS_CONTEXT 0x101
[ 135.990814] State: OMAP4430_CM_EMU_CLKSTCTRL 0x302
[ 135.991058] State: OMAP4430_CM_EMU_DYNAMICDEP 0x4000040
[ 135.991210] State: OMAP4430_CM_EMU_DEBUGSS_CLKCTRL 0x9040001
[ 135.991455] State: OMAP4430_CM_L3INSTR_L3_3_CLKCTRL 0x1
[ 135.991607] State: OMAP4430_CM_L3INSTR_L3_INSTR_CLKCTRL 0x1
[ 135.991851] State: OMAP4430_CM_PRM_PROFILING_CLKCTRL 0x1

So it appears to be enabled. However, I still have the same problem: the counters still don't increment.

So, two more questions: (1) If the above registers say that the modules are enabled (as they appear to) are the modules definitely enabled, or could other disabled modules (i.e. disabled clock domains feeding these modules) still cause the module to be disabled? (2) Are there any other common problems?

As for your kernel branch Jon, thanks for the pointer. I've actually been trying to clone your repo there for about a week (gitorious disconnect problems) and finally succeeded today. I will look further at the patches in that branch to determine what I'm missing. I should probably also use the Linux clock domain management stuff rather than just writing these values, yes? I'm not familiar with that subsystem so I haven't tried to use it yet.

Thanks again,

John

0 Jon Hunter over 13 years ago in reply to John Demme

TI__Expert 3245 points

John Demme said:

Thanks for the help. I think this has gotten me one step closer, but I'm still missing something. I've added this code:

__raw_writel(2, OMAP4430_CM_EMU_CLKSTCTRL);
__raw_writel(1, OMAP4430_CM_L3INSTR_L3_3_CLKCTRL);
__raw_writel(1, OMAP4430_CM_L3INSTR_L3_INSTR_CLKCTRL);
__raw_writel(1, OMAP4430_CM_PRM_PROFILING_CLKCTRL);

Which seems to have helped a bit. At boot, the register contents were:

[ 135.988250] State: OMAP4430_PM_EMU_PWRSTCTRL 0x30000
[ 135.988494] State: OMAP4430_PM_EMU_PWRSTST 0x0
[ 135.988647] State: OMAP4430_RM_EMU_DEBUGSS_CONTEXT 0x101
[ 135.988891] State: OMAP4430_CM_EMU_CLKSTCTRL 0x3
[ 135.989044] State: OMAP4430_CM_EMU_DYNAMICDEP 0x4000040
[ 135.989288] State: OMAP4430_CM_EMU_DEBUGSS_CLKCTRL 0x9060001
[ 135.989471] State: OMAP4430_CM_L3INSTR_L3_3_CLKCTRL 0x1
[ 135.989685] State: OMAP4430_CM_L3INSTR_L3_INSTR_CLKCTRL 0x1
[ 135.989868] State: OMAP4430_CM_PRM_PROFILING_CLKCTRL 0x20001

So the module was definitely disabled. After writing, I get the state to:

[ 135.990264] State: OMAP4430_PM_EMU_PWRSTCTRL 0x30000
[ 135.990417] State: OMAP4430_PM_EMU_PWRSTST 0x37
[ 135.990661] State: OMAP4430_RM_EMU_DEBUGSS_CONTEXT 0x101
[ 135.990814] State: OMAP4430_CM_EMU_CLKSTCTRL 0x302
[ 135.991058] State: OMAP4430_CM_EMU_DYNAMICDEP 0x4000040
[ 135.991210] State: OMAP4430_CM_EMU_DEBUGSS_CLKCTRL 0x9040001
[ 135.991455] State: OMAP4430_CM_L3INSTR_L3_3_CLKCTRL 0x1
[ 135.991607] State: OMAP4430_CM_L3INSTR_L3_INSTR_CLKCTRL 0x1
[ 135.991851] State: OMAP4430_CM_PRM_PROFILING_CLKCTRL 0x1

So it appears to be enabled. However, I still have the same problem: the counters still don't increment.

The above registers look good.

So, two more questions: (1) If the above registers say that the modules are enabled (as they appear to) are the modules definitely enabled, or could other disabled modules (i.e. disabled clock domains feeding these modules) still cause the module to be disabled? (2) Are there any other common problems?

(1). Yes the MPU power domain. If you are using the Android kernel then most likely the MPU power domain is transitioning in and out of low power. This will definitely cause the PMU to lose its logic state. Right now I am using setting a QoS constraint when PMU is active to prevent the MPU sub-system from hitting low power states. See:

https://gitorious.org/linux-omap-dev/linux-omap-dev/commit/12cb83b749337a5c2edcdc92e29f22493f7ce94a

For a test you could also try disabling CPU-idle in the kernel configuration too.

(2). Not that I can think of. Just MPU and EMU power domains.

As for your kernel branch Jon, thanks for the pointer. I've actually been trying to clone your repo there for about a week (gitorious disconnect problems) and finally succeeded today. I will look further at the patches in that branch to determine what I'm missing. I should probably also use the Linux clock domain management stuff rather than just writing these values, yes? I'm not familiar with that subsystem so I haven't tried to use it yet.

Sorry about that I have been having some problems with gitorious myself. The gitorious folks were pushing some fixes to fix some problems, but I may switch over to github if it continues to be a problem.

Cheers
Jon

0 John Demme over 13 years ago in reply to Jon Hunter

Prodigy 90 points

Jon Hunter said:

(1). Yes the MPU power domain. If you are using the Android kernel then most likely the MPU power domain is transitioning in and out of low power. This will definitely cause the PMU to lose its logic state. Right now I am using setting a QoS constraint when PMU is active to prevent the MPU sub-system from hitting low power states. See:

https://gitorious.org/linux-omap-dev/linux-omap-dev/commit/12cb83b749337a5c2edcdc92e29f22493f7ce94a

For a test you could also try disabling CPU-idle in the kernel configuration too.

OK. This has gotten me one step closer. If I recompile without CPU-idle support, then I no longer ever read gibberish from the counters. So that solves original problem number 2. Thanks!

Unfortunately, the counters still aren't counting. (I've also switched to turning on the clock/power with calls to the clock subsystem rather than register writes to no avail.) There seems to be one more possible problem: non-invasive debug authentication. According to table c7-3 in the ARM archiecture manual, if non-invasive debug isn't enabled, the event counters won't be enabled. Further, whether or not non-invasive debug is enabled is a combination of DBGEN, NIDEN and secure mode signals. I haven't seen mention of these signals in the OMAP documentation. Is it possible they are set wrong? Does anyone know how they are controlled?

Jon Hunter said:

Sorry about that I have been having some problems with gitorious myself. The gitorious folks were pushing some fixes to fix some problems, but I may switch over to github if it continues to be a problem.

Personally, I love Github. I've never seen it beat, so vote++.

Thanks again for the help,

John

0 John Demme over 13 years ago in reply to John Demme

Prodigy 90 points

John Demme said:

Unfortunately, the counters still aren't counting. (I've also switched to turning on the clock/power with calls to the clock subsystem rather than register writes to no avail.) There seems to be one more possible problem: non-invasive debug authentication. According to table c7-3 in the ARM archiecture manual, if non-invasive debug isn't enabled, the event counters won't be enabled. Further, whether or not non-invasive debug is enabled is a combination of DBGEN, NIDEN and secure mode signals. I haven't seen mention of these signals in the OMAP documentation. Is it possible they are set wrong? Does anyone know how they are controlled?

Sorry, I've posted too quickly. Turns out DBGAUTHSTATUS lets me read these values. In fact bit 2 is "non-secure non-invasive debug enabled" and bit 6 is the secure version. (Table C10.8.7 of the ARM manual). When I read the register on my processor, it's 0xAA. So non-invasive debug is not enabled. This must be why my counters aren't counting. Any clues how to push DBGEN or NIDEN high? (Are they signals exposed to the PCB, software controlled or something else?)

Thanks,

John

0 Mihai Pricopi over 13 years ago in reply to John Demme

Prodigy 10 points

Hi John,

I am facing the same problem here with a OMAP4460. I am not very experienced with the linux kernel. Can I ask where did you call these functions;

I tried setting OMAP4430_CM_EMU_CLKSTCTRL inside omap_pm_init() in pm44xx.c but the value was not set. Do I miss something ?

0 Jon Hunter over 13 years ago in reply to John Demme

TI__Expert 3245 points

John Demme said:

Sorry, I've posted too quickly. Turns out DBGAUTHSTATUS lets me read these values. In fact bit 2 is "non-secure non-invasive debug enabled" and bit 6 is the secure version. (Table C10.8.7 of the ARM manual). When I read the register on my processor, it's 0xAA. So non-invasive debug is not enabled. This must be why my counters aren't counting. Any clues how to push DBGEN or NIDEN high? (Are they signals exposed to the PCB, software controlled or something else?)

The signals DBGEN and NIDEN are definitely internal signals, so they are nothing that you would be exposed externally and connect to the PCB.

In my testing, I am using PERF and so the kernel PMU driver is configuring the counters. So may be there is something different in the programming of the PMU between your kernel module and the kernel itself. I am currently out of the office but I can probably take a closer look at this later in the week.

Cheers
Jon

0 John Demme over 13 years ago in reply to Mihai Pricopi

Prodigy 90 points

Mihai Pricopi said:

Hi John,

I am facing the same problem here with a OMAP4460. I am not very experienced with the linux kernel. Can I ask where did you call these functions;

__raw_writel(2, OMAP4430_CM_EMU_CLKSTCTRL);
__raw_writel(1, OMAP4430_CM_L3INSTR_L3_3_CLKCTRL);
__raw_writel(1, OMAP4430_CM_L3INSTR_L3_INSTR_CLKCTRL);
__raw_writel(1, OMAP4430_CM_PRM_PROFILING_CLKCTRL);

I tried setting OMAP4430_CM_EMU_CLKSTCTRL inside omap_pm_init() in pm44xx.c but the value was not set. Do I miss something ?

Hi Mihai,

I'm writing a custom kernel module, so I'm calling them in my own code. If you want to put them in existing kernel code, I'm not sure where an appropriate place would be. However, you probably have to be careful about putting them in initialization code -- Linux could easily override your values shortly after you set them. There's some pretty extensive clock/power domain management code which determines which domains have to be enabled. I should also add that I haven't made them work yet, so you shouldn't necessarily believe what I'm saying.

~John

0 Jon Hunter over 13 years ago in reply to John Demme

TI__Expert 3245 points

John Demme said:

Sorry, I've posted too quickly. Turns out DBGAUTHSTATUS lets me read these values. In fact bit 2 is "non-secure non-invasive debug enabled" and bit 6 is the secure version. (Table C10.8.7 of the ARM manual). When I read the register on my processor, it's 0xAA. So non-invasive debug is not enabled. This must be why my counters aren't counting. Any clues how to push DBGEN or NIDEN high? (Are they signals exposed to the PCB, software controlled or something else?)

Hi John,

I did some quick testing today and I found that if I keep the EMU power domain in the off state (by setting CM_EMU_CLKSTCTRL = 0x3), then I also see that DBGAUTHSTATUS = 0xAA. However, once I power up the EMU power domain (by setting CM_EMU_CLKSTCTRL = 0x2), then I see that DBGAUTHSTATUS = 0xAF. So you are definitely on the right lines here. The problem is I am not sure why powering up the EMU power domain is not working for you.

When you power up the EMU power domain, are you waiting for the EMU power domain to come out of idle? The IDLEST field in the CM_EMU_DEBUGSS_CLKCTRL register will indicate when the power domain is out of idle. The PM_EMU_PWRSTST register should also show the power domain is ON-ACTIVE too.

Cheers
Jon

0 Jon Hunter over 13 years ago in reply to Jon Hunter

TI__Expert 3245 points

By the way, I am reading the DBGAUTHSTATUS just as the counters are being enabled in the kernel. Cheers Jon

0 Jon Hunter over 13 years ago in reply to John Demme

TI__Expert 3245 points

John Demme said:

Personally, I love Github. I've never seen it beat, so vote++.

I have now created a github account and mirrored my pmu branch here:

https://github.com/jonhunter/linux/tree/dev-pmu

Cheers
Jon

0 John Demme over 13 years ago in reply to Jon Hunter

Prodigy 90 points

Jon Hunter said:

I did some quick testing today and I found that if I keep the EMU power domain in the off state (by setting CM_EMU_CLKSTCTRL = 0x3), then I also see that DBGAUTHSTATUS = 0xAA. However, once I power up the EMU power domain (by setting CM_EMU_CLKSTCTRL = 0x2), then I see that DBGAUTHSTATUS = 0xAF. So you are definitely on the right lines here. The problem is I am not sure why powering up the EMU power domain is not working for you.

When you power up the EMU power domain, are you waiting for the EMU power domain to come out of idle? The IDLEST field in the CM_EMU_DEBUGSS_CLKCTRL register will indicate when the power domain is out of idle. The PM_EMU_PWRSTST register should also show the power domain is ON-ACTIVE too.

Hmm. Interesting. I am indeed waiting for the IDLEST field to go to 0 and indeed it does. It seems to happy very quickly. However, I just noticed that the next field, CM_EMU_DEBUGSS_CLKCTRL.STBYST remains as 1, indicating the module is in standby. If I wait on STBYST as well, it freezes, indicating that the module never exits standby. So, what can cause it to remain in standby after I ungate the clock with CM_EMU_CLKSTCTRL = 0x2?

As for PM_EMU_PWRSTST, I'm finding the documentation a bit confusing.

LASTPOWERSTATEENTERED: " Last low power state entered. Set to 0x3 upon write of the same only. This register is intended for debug purpose only."

Does this mean I have to set it to 0x3 and see if it stays there or something odd? If I only read from it, it stays at 0x0. This indicates to me that the module isn't powered. However, if I set it to 0x3 then it stays there.

Any further suggestions? Thanks a lot for your help

~John

0 Jon Hunter over 13 years ago in reply to John Demme

TI__Expert 3245 points

John Demme said:

Hmm. Interesting. I am indeed waiting for the IDLEST field to go to 0 and indeed it does. It seems to happy very quickly. However, I just noticed that the next field, CM_EMU_DEBUGSS_CLKCTRL.STBYST remains as 1, indicating the module is in standby. If I wait on STBYST as well, it freezes, indicating that the module never exits standby. So, what can cause it to remain in standby after I ungate the clock with CM_EMU_CLKSTCTRL = 0x2?

As for PM_EMU_PWRSTST, I'm finding the documentation a bit confusing.

Don't worry about standby just check that the module is coming out of idle so that it is accessible.

John Demme said:

LASTPOWERSTATEENTERED: " Last low power state entered. Set to 0x3 upon write of the same only. This register is intended for debug purpose only."

Does this mean I have to set it to 0x3 and see if it stays there or something odd? If I only read from it, it stays at 0x0. This indicates to me that the module isn't powered. However, if I set it to 0x3 then it stays there.

I have not looked at the behaviour of the LASTPOWERSTATEENTERED field, however, the POWERSTATEST field in the same register should reflect the current status correctly.

Cheers

Jon

0 John Demme over 13 years ago in reply to Jon Hunter

Prodigy 90 points

Jon Hunter said:

I have not looked at the behaviour of the LASTPOWERSTATEENTERED field, however, the POWERSTATEST field in the same register should reflect the current status correctly.

I forgot to mention that: POWERSTATEST is 0x3 and LOGICSTATEST is 1. Also, EMU_BANK_STATEST goes to 0x3. Before turning on the EMU clock domain, the entire PM_EMU_PWRSTST register reads as 0x0.

So it certainly seems like it's powering up, yet DBGAUTHSTATUS stays at 0xAA, leading me to believe that NIDEN is being de-asserted somewhere.

~John

0 John Demme over 13 years ago in reply to John Demme

Prodigy 90 points

Hi All,

Quick update: I got my hands on a pandaboard running an OMAP4430. My code works fine on that processor running Linaro Android (which uses a 3.2 -based kernel). So I still don't know what the problem is with my Galaxy Nexus (which was running Android AOSP w/ 3.0 -based kernel on an OMAP4460), but I'm just going to move to the pandaboard to fix my problem.

Thanks for all your help!

~John

0 Jon Hunter over 13 years ago in reply to John Demme

TI__Expert 3245 points

Hi John,

Thanks for the update and sorry for the delay. This has been on my to-do list but I have not had time this week.

Is your latest source on github? I would like to give it a try on my omap4430/60 boards. I am wondering if your Galaxy Nexus has a security enabled device. The register STD_FUSE_PROD_ID_0 should tell us. I would expect the lower 8-bits of this register to return 0xF0 for a non-secure device (which I have).

Cheers
Jon

0 Xueyang Wang over 13 years ago in reply to John Demme

Prodigy 200 points

Hi All,

Did you get any update on this?

I am struggling with the same problem. I applied Jon's patches to the Linux kernel and tested it on a pandaboard with omap4460. The counters run well. I also tested with John's kernel module and it's working, too.

However, the same patches are not working for the Linaro android kernel 3.2 on pandaboard 4460. I cannot get the counters to count.

Thanks,

Xueyang Wang

0 Yongbing Huang over 12 years ago in reply to Xueyang Wang

Prodigy 20 points

Hi Xueyang,

I also meet the same problem on pandaboard 4460 with linaro kernel , and already struggle on it for several days. Have you already solve it?

Thanks.

Yongbing Huang

0 Jon Hunter over 12 years ago in reply to Yongbing Huang

TI__Expert 3245 points

Hi Yongbing,

What kernel version are you using?

As of kernel release v3.7, PMU is now supported for OMAP4460. OMAP4430 is still not supported but I have patches to enable PMU on 4430 for v3.7.

If you are on an older kernel release, there are several patches that we would need to back-port.

Cheers
Jon

0 Yongbing Huang over 12 years ago in reply to Jon Hunter

Prodigy 20 points

Hi Jon,

I use linaro 12.04 with LInux kernel 3.2. And I finally make it work by setting OMAP4430_CM_EM_CLKSTCTRL just as you mentioned earlier.

Thanks very much and I will try your kernel tree later.

Best regards,

Yongbing

0 Benedikt Dietrich over 12 years ago in reply to Yongbing Huang

Hi,

did anybody read out the PMU successfully on the Galaxy Nexus?

Best regards,

Benedikt

0 Guru Prasad3 over 11 years ago in reply to Jon Hunter

Prodigy 10 points

Hi Jon Hunter,

I've been trying to get the PMU counters to work on the Galaxy Nexus.

I have followed most of the advice up until this point. Upon reading STD_FUSE_PROD_ID_0 register, I receive the value 0x08200CC. This value along with your definition of a 'non-secure device' leads me to believe that the Galaxy Nexus is a 'secure device'.

Is there any way to enable PMU registers given this configuration?

Processors

Processors forum

PMU with Custom Module