Hello All,
I'm trying to write a custom module on top of the Android 4.1 kernel to use the OMAP4460's performance counters. Before you ask, no, I cannot use perf or oprofile; they do not yield the information I need. This is rather odd data I'm attempting to collect.
Regardless, I can't get the counters to, well, count. I'm using the v7_pmu.S code from:
http://code.google.com/p/mycodespot/wiki/DirectPMUCodeGCC
Along with my module code, which I've put here:
http://gist.github.com/3289014#file_pmu_sync_sample_main.c
And build it against the Android 3.0.31 kernel ("3.0.31-g6fb96c9-dirty"). I'm then running it on a rooted Samsung Galaxy Nexus. Examples of the dmesg output can be found on the above page.
I'm having two problems:
1) After configuring and enabling the counters, they remain at 0 -- they don't count. The cycle counter, however, seems to work OK. I've looked at a lot of code for using these counters -- from oprofile, perf and the directPMU link above -- and as far as I can tell, I'm doing the same thing. Yet no luck. What am I missing? Is there another bit somewhere that needs to be flipped? Is Linux enabling some oddball security mode (with which I'm not familiar) that has to be disabled?
2) If I sleep for a second before reading the counters, they come up as gibberish. It's like some other process (or part of the kernel) is using them. In addition to disabling all PMU stuff in my kernel's config, I've grepped the kernel code for all writes to the PMU registers and commented them out. I am now running a custom kernel, thought it made no difference. (I only found PMU register writes in some SMC security code and the perf_event code.) So what could be interfering? Are these registers being context swapped? Is there some other code I haven't found that uses them or the PMU?
I've been banging my head against the wall on this for longer than I'd care to admit, so I'd appreciate some help.
Thanks in advance,
John
Hello John,
Have you seen this thread about a similar issue: http://e2e.ti.com/support/omap/f/849/p/184885/667248.aspx#667248
Regards,
Magdalena
Please click the Verify Answer button on this post if it answers your question
_____________________________________________
Be sure to read the OMAP4 and OMAP5 Forum Guidelines and FAQ
Yes I have. Unfortunately, none of the pointers from that response helped.
John,
Something people have run into in the past is that, in order to get to the counters, you need to clock them. If you are connecting a debugger, then these clocks I believe are automatically enabled. For example: writereg(OMAP4430_CM_EMU_CLKSTCTRL, 0x2); writereg(OMAP4430_CM_L3INSTR_L3_3_CLKCTRL, 0x1); writereg(OMAP4430_CM_L3INSTR_L3_INSTR_CLKCTRL, 0x1); writereg(OMAP4430_PRM_PRM_PROFILING_CLKCTRL, 0x1);
Regards,Gina
_______________________________________________________
Gina Glaser Something people have run into in the past is that, in order to get to the counters, you need to clock them. If you are connecting a debugger, then these clocks I believe are automatically enabled. For example: writereg(OMAP4430_CM_EMU_CLKSTCTRL, 0x2); writereg(OMAP4430_CM_L3INSTR_L3_3_CLKCTRL, 0x1); writereg(OMAP4430_CM_L3INSTR_L3_INSTR_CLKCTRL, 0x1); writereg(OMAP4430_PRM_PRM_PROFILING_CLKCTRL, 0x1);
Something people have run into in the past is that, in order to get to the counters, you need to clock them. If you are connecting a debugger, then these clocks I believe are automatically enabled. For example:
writereg(OMAP4430_CM_EMU_CLKSTCTRL, 0x2); writereg(OMAP4430_CM_L3INSTR_L3_3_CLKCTRL, 0x1); writereg(OMAP4430_CM_L3INSTR_L3_INSTR_CLKCTRL, 0x1); writereg(OMAP4430_PRM_PRM_PROFILING_CLKCTRL, 0x1);
If you are using an OMAP4460 then only the EMU power domain needs to be enabled to get the PMU counters working. To enable the EMU power domain you just need to turn on the clocks to the EMU power domain by writing 0x2 to OMAP4430_CM_EMU_CLKSTCTRL as Gina mentions above. The other 3 register writes are only necessary for OMAP4430 that does not have dedicated PMU interrupts and PMU events are routed using the cross trigger interface (CTI).
I have tested PMU with the latest linux kernel with OMAP4460 plus a patch series I am keeping here:
git@gitorious.org:linux-omap-dev/linux-omap-dev.git -b dev-pmu
CheersJon
Thanks for the help. I think this has gotten me one step closer, but I'm still missing something. I've added this code:
__raw_writel(2, OMAP4430_CM_EMU_CLKSTCTRL); __raw_writel(1, OMAP4430_CM_L3INSTR_L3_3_CLKCTRL); __raw_writel(1, OMAP4430_CM_L3INSTR_L3_INSTR_CLKCTRL); __raw_writel(1, OMAP4430_CM_PRM_PROFILING_CLKCTRL);
Which seems to have helped a bit. At boot, the register contents were:
[ 135.988250] State: OMAP4430_PM_EMU_PWRSTCTRL 0x30000[ 135.988494] State: OMAP4430_PM_EMU_PWRSTST 0x0[ 135.988647] State: OMAP4430_RM_EMU_DEBUGSS_CONTEXT 0x101[ 135.988891] State: OMAP4430_CM_EMU_CLKSTCTRL 0x3[ 135.989044] State: OMAP4430_CM_EMU_DYNAMICDEP 0x4000040[ 135.989288] State: OMAP4430_CM_EMU_DEBUGSS_CLKCTRL 0x9060001[ 135.989471] State: OMAP4430_CM_L3INSTR_L3_3_CLKCTRL 0x1[ 135.989685] State: OMAP4430_CM_L3INSTR_L3_INSTR_CLKCTRL 0x1[ 135.989868] State: OMAP4430_CM_PRM_PROFILING_CLKCTRL 0x20001
So the module was definitely disabled. After writing, I get the state to:
[ 135.990264] State: OMAP4430_PM_EMU_PWRSTCTRL 0x30000[ 135.990417] State: OMAP4430_PM_EMU_PWRSTST 0x37[ 135.990661] State: OMAP4430_RM_EMU_DEBUGSS_CONTEXT 0x101[ 135.990814] State: OMAP4430_CM_EMU_CLKSTCTRL 0x302[ 135.991058] State: OMAP4430_CM_EMU_DYNAMICDEP 0x4000040[ 135.991210] State: OMAP4430_CM_EMU_DEBUGSS_CLKCTRL 0x9040001[ 135.991455] State: OMAP4430_CM_L3INSTR_L3_3_CLKCTRL 0x1[ 135.991607] State: OMAP4430_CM_L3INSTR_L3_INSTR_CLKCTRL 0x1[ 135.991851] State: OMAP4430_CM_PRM_PROFILING_CLKCTRL 0x1
So it appears to be enabled. However, I still have the same problem: the counters still don't increment.
So, two more questions: (1) If the above registers say that the modules are enabled (as they appear to) are the modules definitely enabled, or could other disabled modules (i.e. disabled clock domains feeding these modules) still cause the module to be disabled? (2) Are there any other common problems?
As for your kernel branch Jon, thanks for the pointer. I've actually been trying to clone your repo there for about a week (gitorious disconnect problems) and finally succeeded today. I will look further at the patches in that branch to determine what I'm missing. I should probably also use the Linux clock domain management stuff rather than just writing these values, yes? I'm not familiar with that subsystem so I haven't tried to use it yet.
Thanks again,
John Demme Thanks for the help. I think this has gotten me one step closer, but I'm still missing something. I've added this code: __raw_writel(2, OMAP4430_CM_EMU_CLKSTCTRL); __raw_writel(1, OMAP4430_CM_L3INSTR_L3_3_CLKCTRL); __raw_writel(1, OMAP4430_CM_L3INSTR_L3_INSTR_CLKCTRL); __raw_writel(1, OMAP4430_CM_PRM_PROFILING_CLKCTRL); Which seems to have helped a bit. At boot, the register contents were: [ 135.988250] State: OMAP4430_PM_EMU_PWRSTCTRL 0x30000[ 135.988494] State: OMAP4430_PM_EMU_PWRSTST 0x0[ 135.988647] State: OMAP4430_RM_EMU_DEBUGSS_CONTEXT 0x101[ 135.988891] State: OMAP4430_CM_EMU_CLKSTCTRL 0x3[ 135.989044] State: OMAP4430_CM_EMU_DYNAMICDEP 0x4000040[ 135.989288] State: OMAP4430_CM_EMU_DEBUGSS_CLKCTRL 0x9060001[ 135.989471] State: OMAP4430_CM_L3INSTR_L3_3_CLKCTRL 0x1[ 135.989685] State: OMAP4430_CM_L3INSTR_L3_INSTR_CLKCTRL 0x1[ 135.989868] State: OMAP4430_CM_PRM_PROFILING_CLKCTRL 0x20001 So the module was definitely disabled. After writing, I get the state to: [ 135.990264] State: OMAP4430_PM_EMU_PWRSTCTRL 0x30000[ 135.990417] State: OMAP4430_PM_EMU_PWRSTST 0x37[ 135.990661] State: OMAP4430_RM_EMU_DEBUGSS_CONTEXT 0x101[ 135.990814] State: OMAP4430_CM_EMU_CLKSTCTRL 0x302[ 135.991058] State: OMAP4430_CM_EMU_DYNAMICDEP 0x4000040[ 135.991210] State: OMAP4430_CM_EMU_DEBUGSS_CLKCTRL 0x9040001[ 135.991455] State: OMAP4430_CM_L3INSTR_L3_3_CLKCTRL 0x1[ 135.991607] State: OMAP4430_CM_L3INSTR_L3_INSTR_CLKCTRL 0x1[ 135.991851] State: OMAP4430_CM_PRM_PROFILING_CLKCTRL 0x1 So it appears to be enabled. However, I still have the same problem: the counters still don't increment.
The above registers look good.
(1). Yes the MPU power domain. If you are using the Android kernel then most likely the MPU power domain is transitioning in and out of low power. This will definitely cause the PMU to lose its logic state. Right now I am using setting a QoS constraint when PMU is active to prevent the MPU sub-system from hitting low power states. See:
https://gitorious.org/linux-omap-dev/linux-omap-dev/commit/12cb83b749337a5c2edcdc92e29f22493f7ce94a
For a test you could also try disabling CPU-idle in the kernel configuration too.
(2). Not that I can think of. Just MPU and EMU power domains.
Sorry about that I have been having some problems with gitorious myself. The gitorious folks were pushing some fixes to fix some problems, but I may switch over to github if it continues to be a problem.
Jon Hunter (1). Yes the MPU power domain. If you are using the Android kernel then most likely the MPU power domain is transitioning in and out of low power. This will definitely cause the PMU to lose its logic state. Right now I am using setting a QoS constraint when PMU is active to prevent the MPU sub-system from hitting low power states. See: https://gitorious.org/linux-omap-dev/linux-omap-dev/commit/12cb83b749337a5c2edcdc92e29f22493f7ce94a For a test you could also try disabling CPU-idle in the kernel configuration too.
OK. This has gotten me one step closer. If I recompile without CPU-idle support, then I no longer ever read gibberish from the counters. So that solves original problem number 2. Thanks!
Unfortunately, the counters still aren't counting. (I've also switched to turning on the clock/power with calls to the clock subsystem rather than register writes to no avail.) There seems to be one more possible problem: non-invasive debug authentication. According to table c7-3 in the ARM archiecture manual, if non-invasive debug isn't enabled, the event counters won't be enabled. Further, whether or not non-invasive debug is enabled is a combination of DBGEN, NIDEN and secure mode signals. I haven't seen mention of these signals in the OMAP documentation. Is it possible they are set wrong? Does anyone know how they are controlled?
Jon Hunter Sorry about that I have been having some problems with gitorious myself. The gitorious folks were pushing some fixes to fix some problems, but I may switch over to github if it continues to be a problem.
Personally, I love Github. I've never seen it beat, so vote++.
Thanks again for the help,
John Demme Unfortunately, the counters still aren't counting. (I've also switched to turning on the clock/power with calls to the clock subsystem rather than register writes to no avail.) There seems to be one more possible problem: non-invasive debug authentication. According to table c7-3 in the ARM archiecture manual, if non-invasive debug isn't enabled, the event counters won't be enabled. Further, whether or not non-invasive debug is enabled is a combination of DBGEN, NIDEN and secure mode signals. I haven't seen mention of these signals in the OMAP documentation. Is it possible they are set wrong? Does anyone know how they are controlled?
Sorry, I've posted too quickly. Turns out DBGAUTHSTATUS lets me read these values. In fact bit 2 is "non-secure non-invasive debug enabled" and bit 6 is the secure version. (Table C10.8.7 of the ARM manual). When I read the register on my processor, it's 0xAA. So non-invasive debug is not enabled. This must be why my counters aren't counting. Any clues how to push DBGEN or NIDEN high? (Are they signals exposed to the PCB, software controlled or something else?)
Thanks,
Hi John,
I am facing the same problem here with a OMAP4460. I am not very experienced with the linux kernel. Can I ask where did you call these functions;
__raw_writel(2, OMAP4430_CM_EMU_CLKSTCTRL);__raw_writel(1, OMAP4430_CM_L3INSTR_L3_3_CLKCTRL);__raw_writel(1, OMAP4430_CM_L3INSTR_L3_INSTR_CLKCTRL);__raw_writel(1, OMAP4430_CM_PRM_PROFILING_CLKCTRL);
I tried setting OMAP4430_CM_EMU_CLKSTCTRL inside omap_pm_init() in pm44xx.c but the value was not set. Do I miss something ?
John Demme Sorry, I've posted too quickly. Turns out DBGAUTHSTATUS lets me read these values. In fact bit 2 is "non-secure non-invasive debug enabled" and bit 6 is the secure version. (Table C10.8.7 of the ARM manual). When I read the register on my processor, it's 0xAA. So non-invasive debug is not enabled. This must be why my counters aren't counting. Any clues how to push DBGEN or NIDEN high? (Are they signals exposed to the PCB, software controlled or something else?)
The signals DBGEN and NIDEN are definitely internal signals, so they are nothing that you would be exposed externally and connect to the PCB.
In my testing, I am using PERF and so the kernel PMU driver is configuring the counters. So may be there is something different in the programming of the PMU between your kernel module and the kernel itself. I am currently out of the office but I can probably take a closer look at this later in the week.
Mihai Pricopi Hi John, I am facing the same problem here with a OMAP4460. I am not very experienced with the linux kernel. Can I ask where did you call these functions; __raw_writel(2, OMAP4430_CM_EMU_CLKSTCTRL);__raw_writel(1, OMAP4430_CM_L3INSTR_L3_3_CLKCTRL);__raw_writel(1, OMAP4430_CM_L3INSTR_L3_INSTR_CLKCTRL);__raw_writel(1, OMAP4430_CM_PRM_PROFILING_CLKCTRL); I tried setting OMAP4430_CM_EMU_CLKSTCTRL inside omap_pm_init() in pm44xx.c but the value was not set. Do I miss something ?
Hi Mihai,
I'm writing a custom kernel module, so I'm calling them in my own code. If you want to put them in existing kernel code, I'm not sure where an appropriate place would be. However, you probably have to be careful about putting them in initialization code -- Linux could easily override your values shortly after you set them. There's some pretty extensive clock/power domain management code which determines which domains have to be enabled. I should also add that I haven't made them work yet, so you shouldn't necessarily believe what I'm saying.
~John
I did some quick testing today and I found that if I keep the EMU power domain in the off state (by setting CM_EMU_CLKSTCTRL = 0x3), then I also see that DBGAUTHSTATUS = 0xAA. However, once I power up the EMU power domain (by setting CM_EMU_CLKSTCTRL = 0x2), then I see that DBGAUTHSTATUS = 0xAF. So you are definitely on the right lines here. The problem is I am not sure why powering up the EMU power domain is not working for you.
When you power up the EMU power domain, are you waiting for the EMU power domain to come out of idle? The IDLEST field in the CM_EMU_DEBUGSS_CLKCTRL register will indicate when the power domain is out of idle. The PM_EMU_PWRSTST register should also show the power domain is ON-ACTIVE too.
By the way, I am reading the DBGAUTHSTATUS just as the counters are being enabled in the kernel. Cheers Jon
John Demme Personally, I love Github. I've never seen it beat, so vote++.
I have now created a github account and mirrored my pmu branch here:
https://github.com/jonhunter/linux/tree/dev-pmu
Jon Hunter I did some quick testing today and I found that if I keep the EMU power domain in the off state (by setting CM_EMU_CLKSTCTRL = 0x3), then I also see that DBGAUTHSTATUS = 0xAA. However, once I power up the EMU power domain (by setting CM_EMU_CLKSTCTRL = 0x2), then I see that DBGAUTHSTATUS = 0xAF. So you are definitely on the right lines here. The problem is I am not sure why powering up the EMU power domain is not working for you. When you power up the EMU power domain, are you waiting for the EMU power domain to come out of idle? The IDLEST field in the CM_EMU_DEBUGSS_CLKCTRL register will indicate when the power domain is out of idle. The PM_EMU_PWRSTST register should also show the power domain is ON-ACTIVE too.
Hmm. Interesting. I am indeed waiting for the IDLEST field to go to 0 and indeed it does. It seems to happy very quickly. However, I just noticed that the next field, CM_EMU_DEBUGSS_CLKCTRL.STBYST remains as 1, indicating the module is in standby. If I wait on STBYST as well, it freezes, indicating that the module never exits standby. So, what can cause it to remain in standby after I ungate the clock with CM_EMU_CLKSTCTRL = 0x2?
As for PM_EMU_PWRSTST, I'm finding the documentation a bit confusing.
LASTPOWERSTATEENTERED: " Last low power state entered. Set to 0x3 upon write of the same only. This register is intended for debug purpose only."
Does this mean I have to set it to 0x3 and see if it stays there or something odd? If I only read from it, it stays at 0x0. This indicates to me that the module isn't powered. However, if I set it to 0x3 then it stays there.
Any further suggestions? Thanks a lot for your help