Hi !
I am seeing kernel soft lock up messages when processor is trying to enter into the cpuidle state sometimes. The crash is very random and reproducible.
Here is the kernel log with our diagnostic code.
Jan 30 07:07:00 [53960.444224] Took more than 10m second!! endtime: 55506628834640 starttime: 55506535701000 difftime: 93133
Jan 30 07:07:00 [53960.444276] ****Entered state was 2
Jan 30 07:07:00 [53960.554325] Took more than 10m second!! endtime: 55506738963600 starttime: 55506630135760 difftime: 108827
Jan 30 07:07:00 [53960.554359] ****Entered state was 2
Jan 30 07:07:00 [53960.591769] Took more than 10m second!! endtime: 55506776458480 starttime: 55506739783360 difftime: 36675
Jan 30 07:07:00 [53960.591800] ****Entered state was 2
Jan 30 07:07:00 [53961.586285] Took more than 10m second!! endtime: 55507770955960 starttime: 55506776606760 difftime: 994349
Jan 30 07:07:00 [53961.586310] ****Entered state was 2
Jan 30 07:07:00 [53962.629275] Took more than 10m second!! endtime: 55508813939480 starttime: 55507771526760 difftime: 1042412
Jan 30 07:07:00 [53962.629306] ****Entered state was 2
Jan 30 07:07:00 [53964.033383] Took more than 10m second!! endtime: 55510218051480 starttime: 55508814486160 difftime: 1403565
Jan 30 07:07:00 [53964.033412] ****Entered state was 2
Jan 30 07:07:00 [53964.135364] Took more than 10m second!! endtime: 55510319971000 starttime: 55510218571960 difftime: 101399
Jan 30 07:07:00 [53964.135393] ****Entered state was 2
Jan 30 07:07:00 [53964.189344] Took more than 10m second!! endtime: 55510374012000 starttime: 55510327028280 difftime: 46983
Jan 30 07:07:00 [53964.189374] ****Entered state was 2
Jan 30 07:07:00 [53964.264176] Took more than 10m second!! endtime: 55510448835520 starttime: 55510374531320 difftime: 74304
Jan 30 07:07:00 [53964.264204] ****Entered state was 2
Jan 30 07:07:00 [53964.298661] Took more than 10m second!! endtime: 55510483331960 starttime: 55510449378120 difftime: 33953
Jan 30 07:07:00 [53964.298689] ****Entered state was 2
Jan 30 07:07:00 [53965.329430] Took more than 10m second!! endtime: 55511514091560 starttime: 55510484269680 difftime: 1029821
Jan 30 07:07:00 [53965.329463] ****Entered state was 2
Jan 30 07:07:00 [53966.373439] Took more than 10m second!! endtime: 55512558115440 starttime: 55511514365800 difftime: 1043749
Jan 30 07:07:00 [53966.373466] ****Entered state was 2
Jan 30 07:07:00 [53967.419082] Took more than 10m second!! endtime: 55513603763480 starttime: 55512558355080 difftime: 1045408
Jan 30 07:07:00 [53967.419108] ****Entered state was 2
Jan 30 07:07:00 [53968.448609] Took more than 10m second!! endtime: 55514633290960 starttime: 55513603994160 difftime: 1029296
Jan 30 07:07:00 [53968.448636] ****Entered state was 2
Jan 30 07:07:00 [53969.493756] Took more than 10m second!! endtime: 55515678435440 starttime: 55514633521440 difftime: 1044914
Jan 30 07:07:00 [53969.493782] ****Entered state was 2
Jan 30 07:07:00 [53971.131687] Took more than 10m second!! endtime: 55517316371400 starttime: 55515678666560 difftime: 1637704
Jan 30 07:07:00 [53971.131714] ****Entered state was 2
Jan 30 07:07:00 [53971.233832] Took more than 10m second!! endtime: 55517418443880 starttime: 55517316598760 difftime: 101845
Jan 30 07:07:00 [53971.233858] ****Entered state was 2
Jan 30 07:07:00 [53971.286936] Took more than 10m second!! endtime: 55517471619800 starttime: 55517425164840 difftime: 46454
Jan 30 07:07:00 [53971.286963] ****Entered state was 2
Jan 30 07:07:00 [53971.361722] Took more than 10m second!! endtime: 55517546387440 starttime: 55517471847400 difftime: 74540
Jan 30 07:07:00 [53971.361748] ****Entered state was 2
Jan 30 07:07:00 [53971.397006] Took more than 10m second!! endtime: 55517581691520 starttime: 55517546649880 difftime: 35041
Jan 30 07:07:00 [53971.397032] ****Entered state was 2
Jan 30 07:07:00 [53972.427626] Took more than 10m second!! endtime: 55518612307400 starttime: 55517581915600 difftime: 1030391
Jan 30 07:07:00 [53972.427653] ****Entered state was 2
Jan 30 07:07:00 [53973.470970] Took more than 10m second!! endtime: 55519655651920 starttime: 55518612537600 difftime: 1043114
Jan 30 07:07:00 [53973.470997] ****Entered state was 2
Jan 30 07:07:00 [53974.516937] Took more than 10m second!! endtime: 55520701619960 starttime: 55519655883120 difftime: 1045736
Jan 30 07:07:00 [53974.516963] ****Entered state was 2
Jan 30 07:07:00 [53975.546064] Took more than 10m second!! endtime: 55521730747920 starttime: 55520701847200 difftime: 1028900
Jan 30 07:07:00 [53975.546090] ****Entered state was 2
Jan 30 07:07:00 [53976.591137] Took more than 10m second!! endtime: 55522775819920 starttime: 55521730974040 difftime: 1044845
Jan 30 07:07:00 [53976.591164] ****Entered state was 2
Jan 30 07:07:00 [53978.231145] Took more than 10m second!! endtime: 55524415827360 starttime: 55522776050560 difftime: 1639776
Jan 30 07:07:00 [53978.231171] ****Entered state was 2
Jan 30 07:07:00 [53978.333251] Took more than 10m second!! endtime: 55524517867360 starttime: 55524416055440 difftime: 101811
Jan 30 07:07:00 [53978.333277] ****Entered state was 2
Jan 30 07:07:00 [53978.385118] Took more than 10m second!! endtime: 55524569803400 starttime: 55524524589320 difftime: 45214
Jan 30 07:07:00 [53978.385145] ****Entered state was 2
Jan 30 07:07:00 [53978.459888] Took more than 10m second!! endtime: 55524644554920 starttime: 55524570029160 difftime: 74525
Jan 30 07:07:00 [53978.459914] ****Entered state was 2
Jan 30 07:07:00 [53978.500104] Took more than 10m second!! endtime: 55524684787400 starttime: 55524644799520 difftime: 39987
Jan 30 07:07:00 [53978.500130] ****Entered state was 2
Jan 30 07:07:00 [53979.524372] Took more than 10m second!! endtime: 55525709050920 starttime: 55524685015240 difftime: 1024035
Jan 30 07:07:00 [53979.524399] ****Entered state was 2
Jan 30 07:07:00 [53980.569304] Took more than 10m second!! endtime: 55526753987960 starttime: 55525709303080 difftime: 1044684
Jan 30 07:07:00 [53980.569331] ****Entered state was 2
Jan 30 07:07:00 [53981.614416] Took more than 10m second!! endtime: 55527799099400 starttime: 55526754216480 difftime: 1044882
Jan 30 07:07:00 [53981.614442] ****Entered state was 2
Jan 30 07:07:00 [53982.644673] Took more than 10m second!! endtime: 55528829355400 starttime: 55527799325480 difftime: 1030029
Jan 30 07:07:00 [53982.644699] ****Entered state was 2
Jan 30 07:07:00 [53983.689511] Took more than 10m second!! endtime: 55529874194960 starttime: 55528829583360 difftime: 1044611
Jan 30 07:07:00 [53983.689539] ****Entered state was 2
Jan 30 07:07:00 [53985.329415] Took more than 10m second!! endtime: 55531514099400 starttime: 55529874424440 difftime: 1639674
Jan 30 07:07:00 [53985.329442] ****Entered state was 2
Jan 30 07:07:00 [53985.431523] Took more than 10m second!! endtime: 55531616139400 starttime: 55531514326040 difftime: 101813
Jan 30 07:07:00 [53985.431549] ****Entered state was 2
Jan 30 07:07:00 [53985.483607] Took more than 10m second!! endtime: 55531668292160 starttime: 55531622860520 difftime: 45431
Jan 30 07:07:00 [53985.483635] ****Entered state was 2
Jan 30 07:07:00 [53985.558375] Took more than 10m second!! endtime: 55531743019400 starttime: 55531668520280 difftime: 74499
Jan 30 07:07:00 [53985.558402] ****Entered state was 2
Jan 30 07:07:00 [53985.598543] Took more than 10m second!! endtime: 55531783227920 starttime: 55531743289080 difftime: 39938
Jan 30 07:07:00 [53985.598570] ****Entered state was 2
Jan 30 07:07:00 [53986.623632] Took more than 10m second!! endtime: 55532808315440 starttime: 55531783455040 difftime: 1024860
Jan 30 07:07:00 [53986.623658] ****Entered state was 2
Jan 30 07:07:00 [53987.667641] Took more than 10m second!! endtime: 55533852323920 starttime: 55532808543040 difftime: 1043780
Jan 30 07:07:00 [53987.667668] ****Entered state was 2
Jan 30 07:07:00 [53988.712818] Took more than 10m second!! endtime: 55534897499880 starttime: 55533852551920 difftime: 1044947
Jan 30 07:07:00 [53988.712844] ****Entered state was 2
Jan 30 07:07:00 [53989.742824] Took more than 10m second!! endtime: 55535927507920 starttime: 55534897747640 difftime: 1029760
Jan 30 07:07:00 [53989.742850] ****Entered state was 2
Jan 30 07:07:00 [53990.787881] Took more than 10m second!! endtime: 55536972563000 starttime: 55535927734440 difftime: 1044828
Jan 30 07:07:00 [53990.787907] ****Entered state was 2
Jan 30 07:07:00 [53992.425993] Took more than 10m second!! endtime: 55538610675440 starttime: 55536972791560 difftime: 1637883
Jan 30 07:07:00 [53992.426020] ****Entered state was 2
Jan 30 07:07:00 [53992.528154] Took more than 10m second!! endtime: 55538712771400 starttime: 55538610905640 difftime: 101865
Jan 30 07:07:00 [53992.528181] ****Entered state was 2
Jan 30 07:07:00 [53992.581013] Took more than 10m second!! endtime: 55538765699880 starttime: 55538719491320 difftime: 46208
Jan 30 07:07:00 [53992.581040] ****Entered state was 2
Jan 30 07:07:00 [53992.655769] Took more than 10m second!! endtime: 55538840434960 starttime: 55538765924640 difftime: 74510
Jan 30 07:07:00 [53992.655795] ****Entered state was 2
Jan 30 07:07:00 [53992.705966] Took more than 10m second!! endtime: 55538890650960 starttime: 55538840678880 difftime: 49972
Jan 30 07:07:00 [53992.705993] ****Entered state was 2
Jan 30 07:07:00 [53993.751192] Took more than 10m second!! endtime: 55539935874960 starttime: 55538890877920 difftime: 1044997
Jan 30 07:07:00 [53993.751218] ****Entered state was 2
Jan 30 07:07:00 [53994.784431] Took more than 10m second!! endtime: 55540969114960 starttime: 55539936101720 difftime: 1033013
Jan 30 07:07:00 [53994.784458] ****Entered state was 2
Jan 30 07:07:00 [53995.826152] Took more than 10m second!! endtime: 55542010835840 starttime: 55540969341600 difftime: 1041494
Jan 30 07:07:00 [53995.826178] ****Entered state was 2
Jan 30 07:07:00 [53996.871305] Took more than 10m second!! endtime: 55543055987480 starttime: 55542011062840 difftime: 1044924
Jan 30 07:07:00 [53996.871331] ****Entered state was 2
Jan 30 07:07:00 [53997.903289] Took more than 10m second!! endtime: 55544087971000 starttime: 55543056217120 difftime: 1031753
Jan 30 07:07:00 [53997.903315] ****Entered state was 2
Jan 30 07:07:00 [53999.523384] Took more than 10m second!! endtime: 55545708067480 starttime: 55544088218240 difftime: 1619849
Jan 30 07:07:00 [53999.523410] ****Entered state was 2
Jan 30 07:07:00 [53999.625570] Took more than 10m second!! endtime: 55545810187440 starttime: 55545708296160 difftime: 101891
Jan 30 07:07:00 [53999.625597] ****Entered state was 2
Jan 30 07:07:00 [53999.679517] Took more than 10m second!! endtime: 55545864202960 starttime: 55545816910040 difftime: 47292
Jan 30 07:07:00 [53999.679543] ****Entered state was 2
Jan 30 07:07:00 [53999.754274] Took more than 10m second!! endtime: 55545938939400 starttime: 55545864427120 difftime: 74512
Jan 30 07:07:00 [53999.754300] ****Entered state was 2
Jan 30 07:07:00 [53999.804367] Took more than 10m second!! endtime: 55545989051040 starttime: 55545939184560 difftime: 49866
Jan 30 07:07:00 [53999.804393] ****Entered state was 2
Jan 30 07:07:00 [54000.849385] Took more than 10m second!! endtime: 55547034066960 starttime: 55545989278560 difftime: 1044788
Jan 30 07:07:00 [54000.849411] ****Entered state was 2
Jan 30 07:07:00 [54001.880520] Took more than 10m second!! endtime: 55548065203480 starttime: 55547034296240 difftime: 1030907
Jan 30 07:07:00 [54001.880547] ****Entered state was 2
Jan 30 07:07:00 [54002.925104] Took more than 10m second!! endtime: 55549109787960 starttime: 55548065431600 difftime: 1044356
Jan 30 07:07:00 [54002.925130] ****Entered state was 2
Jan 30 07:07:00 [54003.969652] Took more than 10m second!! endtime: 55550154331440 starttime: 55549110015600 difftime: 1044315
Jan 30 07:07:00 [54003.969678] ****Entered state was 2
Jan 30 07:07:00 [54005.001808] Took more than 10m second!! endtime: 55551186491040 starttime: 55550154562040 difftime: 1031929
Jan 30 07:07:00 [54005.001834] ****Entered state was 2
Jan 30 07:07:00 [54006.622850] Took more than 10m second!! endtime: 55552807531960 starttime: 55551186719120 difftime: 1620812
Jan 30 07:07:00 [54006.622876] ****Entered state was 2
Jan 30 07:07:00 [54006.724980] Took more than 10m second!! endtime: 55552909595920 starttime: 55552807761080 difftime: 101834
Jan 30 07:07:00 [54006.725006] ****Entered state was 2
Jan 30 07:07:00 [54006.777877] Took more than 10m second!! endtime: 55552962563240 starttime: 55552916317600 difftime: 46245
Jan 30 07:07:00 [54006.777904] ****Entered state was 2
Jan 30 07:07:00 [54006.852595] Took more than 10m second!! endtime: 55553037259800 starttime: 55552962789000 difftime: 74470
Jan 30 07:07:00 [54006.852621] ****Entered state was 2
Jan 30 07:07:00 [54006.901799] Took more than 10m second!! endtime: 55553086482960 starttime: 55553037506320 difftime: 48976
Jan 30 07:07:00 [54006.901825] ****Entered state was 2
Jan 30 07:07:00 [54007.947908] Took more than 10m second!! endtime: 55554132586880 starttime: 55553086709800 difftime: 1045877
Jan 30 07:07:00 [54007.947934] ****Entered state was 2
Jan 30 07:07:00 [54008.976943] Took more than 10m second!! endtime: 55555161627080 starttime: 55554132818880 difftime: 1028808
Jan 30 07:07:00 [54008.976969] ****Entered state was 2
Jan 30 07:07:00 [54010.021968] Took more than 10m second!! endtime: 55556206651960 starttime: 55555161853960 difftime: 1044798
Jan 30 07:07:00 [54010.021994] ****Entered state was 2
Jan 30 07:07:00 [54011.066985] Took more than 10m second!! endtime: 55557251667920 starttime: 55556206877640 difftime: 1044790
Jan 30 07:07:00 [54011.067011] ****Entered state was 2
Jan 30 07:07:00 [54012.097168] Took more than 10m second!! endtime: 55558281851400 starttime: 55557251894440 difftime: 1029956
Jan 30 07:07:00 [54012.097195] ****Entered state was 2
Jan 30 07:07:00 [54013.720295] Took more than 10m second!! endtime: 55559904979440 starttime: 55558282079440 difftime: 1622900
Jan 30 07:07:00 [54013.720322] ****Entered state was 2
Jan 30 07:07:00 [54013.822317] Took more than 10m second!! endtime: 55560006931520 starttime: 55559905207040 difftime: 101724
Jan 30 07:07:00 [54013.822343] ****Entered state was 2
Jan 30 07:07:00 [54013.875134] Took more than 10m second!! endtime: 55560059819760 starttime: 55560013652560 difftime: 46167
Jan 30 07:07:00 [54013.875162] ****Entered state was 2
Jan 30 07:07:00 [54013.949906] Took more than 10m second!! endtime: 55560134571800 starttime: 55560060046840 difftime: 74524
Jan 30 07:07:00 [54013.949933] ****Entered state was 2
Jan 30 07:07:00 [54014.000182] Took more than 10m second!! endtime: 55560184867000 starttime: 55560134816720 difftime: 50050
Jan 30 07:07:00 [54014.000208] ****Entered state was 2
Jan 30 07:07:00 [54015.045273] Took more than 10m second!! endtime: 55561229954880 starttime: 55560185092800 difftime: 1044862
Jan 30 07:07:00 [54015.045300] ****Entered state was 2
Jan 30 07:07:00 [54016.076320] Took more than 10m second!! endtime: 55562261003480 starttime: 55561230185440 difftime: 1030818
Jan 30 07:07:00 [54016.076347] ****Entered state was 2
Jan 30 07:07:00 [54017.120434] Took more than 10m second!! endtime: 55563305114920 starttime: 55562261232040 difftime: 1043882
Jan 30 07:07:00 [54017.120460] ****Entered state was 2
Jan 30 07:07:00 [54018.165335] Took more than 10m second!! endtime: 55564350019000 starttime: 55563305345000 difftime: 1044674
Jan 30 07:07:00 [54018.165361] ****Entered state was 2
Jan 30 07:07:00 [54019.197423] Took more than 10m second!! endtime: 55565382107480 starttime: 55564350246600 difftime: 1031860
Jan 30 07:07:00 [54019.197449] ****Entered state was 2
Jan 30 07:07:00 [54020.833976] Took more than 10m second!! endtime: 55567018659440 starttime: 55565382333080 difftime: 1636326
Jan 30 07:07:00 [54020.834002] ****Entered state was 2
Jan 30 07:07:00 [54020.935971] Took more than 10m second!! endtime: 55567120587400 starttime: 55567018886680 difftime: 101700
Jan 30 07:07:00 [54020.935997] ****Entered state was 2
Jan 30 07:07:00 [54020.989623] Took more than 10m second!! endtime: 55567174307440 starttime: 55567127310720 difftime: 46996
Jan 30 07:07:00 [54020.989649] ****Entered state was 2
Jan 30 07:07:00 [54021.064384] Took more than 10m second!! endtime: 55567249051880 starttime: 55567174553200 difftime: 74498
Jan 30 07:07:00 [54021.064411] ****Entered state was 2
Jan 30 07:07:00 [54021.104583] Took more than 10m second!! endtime: 55567289267440 starttime: 55567249296360 difftime: 39971
Jan 30 07:07:00 [54021.104609] ****Entered state was 2
Jan 30 07:07:00 [54022.129682] Took more than 10m second!! endtime: 55568314362960 starttime: 55567289493840 difftime: 1024869
Jan 30 07:07:00 [54022.129708] ****Entered state was 2
Jan 30 07:07:00 [54023.172761] Took more than 10m second!! endtime: 55569357443920 starttime: 55568314592080 difftime: 1042851
Jan 30 07:07:00 [54023.172787] ****Entered state was 2
Jan 30 07:07:00 [54024.218815] Took more than 10m second!! endtime: 55570403498960 starttime: 55569357671560 difftime: 1045827
Jan 30 07:07:00 [54024.218842] ****Entered state was 2
Jan 30 07:07:00 [54025.250968] Took more than 10m second!! endtime: 55571435651480 starttime: 55570403726320 difftime: 1031925
Jan 30 07:07:00 [54025.250994] ****Entered state was 2
Jan 30 07:07:00 [54026.292832] Took more than 10m second!! endtime: 55572477515480 starttime: 55571435881000 difftime: 1041634
Jan 30 07:07:00 [54026.292858] ****Entered state was 2
Jan 30 07:07:00 [54027.931001] Took more than 10m second!! endtime: 55574115682960 starttime: 55572477742080 difftime: 1637940
Jan 30 07:07:00 [54027.931027] ****Entered state was 2
Jan 30 07:07:00 [54028.032995] Took more than 10m second!! endtime: 55574217611800 starttime: 55574115911000 difftime: 101700
Jan 30 07:07:00 [54028.033022] ****Entered state was 2
Jan 30 07:07:00 [54028.086974] Took more than 10m second!! endtime: 55574271659400 starttime: 55574224343640 difftime: 47315
Jan 30 07:07:00 [54028.087001] ****Entered state was 2
Jan 30 07:07:00 [54028.161762] Took more than 10m second!! endtime: 55574346427920 starttime: 55574271885120 difftime: 74542
Jan 30 07:07:00 [54028.161788] ****Entered state was 2
Jan 30 07:07:00 [54028.199113] Took more than 10m second!! endtime: 55574383795480 starttime: 55574346671840 difftime: 37123
Jan 30 07:07:00 [54028.199140] ****Entered state was 2
Jan 30 07:07:00 [54029.228080] Took more than 10m second!! endtime: 55575412763960 starttime: 55574384043680 difftime: 1028720
Jan 30 07:07:00 [54029.228107] ****Entered state was 2
Jan 30 07:07:00 [54030.271111] Took more than 10m second!! endtime: 55576455795480 starttime: 55575412991120 difftime: 1042804
Jan 30 07:07:00 [54030.271138] ****Entered state was 2
Jan 30 07:07:00 [54031.316696] Took more than 10m second!! endtime: 55577501379440 starttime: 55576456023040 difftime: 1045356
Jan 30 07:07:00 [54031.316722] ****Entered state was 2
Jan 30 07:07:00 [54032.348120] Took more than 10m second!! endtime: 55578532803880 starttime: 55577501607560 difftime: 1031196
Jan 30 07:07:00 [54032.348146] ****Entered state was 2
Jan 30 07:07:00 [54033.391059] Took more than 10m second!! endtime: 55579575739880 starttime: 55578533029920 difftime: 1042709
Jan 30 07:07:00 [54033.391085] ****Entered state was 2
Jan 30 07:07:00 [54035.029352] Took more than 10m second!! endtime: 55581214035400 starttime: 55579575969440 difftime: 1638065
Jan 30 07:07:00 [54035.029379] ****Entered state was 2
Jan 30 07:07:00 [54035.131345] Took more than 10m second!! endtime: 55581315963440 starttime: 55581214263440 difftime: 101700
Jan 30 07:07:00 [54035.131371] ****Entered state was 2
Jan 30 07:07:00 [54035.185325] Took more than 10m second!! endtime: 55581370011240 starttime: 55581322684280 difftime: 47326
Jan 30 07:07:00 [54035.185352] ****Entered state was 2
Jan 30 07:07:00 [54035.260255] Took more than 10m second!! endtime: 55581444922920 starttime: 55581370235800 difftime: 74687
Jan 30 07:07:00 [54035.260281] ****Entered state was 2
Jan 30 07:07:00 [54035.299342] Took more than 10m second!! endtime: 55581484027480 starttime: 55581445165440 difftime: 38862
Jan 30 07:07:00 [54035.299369] ****Entered state was 2
Jan 30 07:07:00 [54036.327382] Took more than 10m second!! endtime: 55582512059880 starttime: 55581484253520 difftime: 1027806
Jan 30 07:07:00 [54036.327409] ****Entered state was 2
Jan 30 07:07:00 [54037.369650] Took more than 10m second!! endtime: 55583554331920 starttime: 55582512293920 difftime: 1042038
Jan 30 07:07:00 [54037.369676] ****Entered state was 2
Jan 30 07:07:00 [54038.414498] Took more than 10m second!! endtime: 55584599179440 starttime: 55583554580200 difftime: 1044599
Jan 30 07:07:00 [54038.414524] ****Entered state was 2
Jan 30 07:07:00 [54039.445657] Took more than 10m second!! endtime: 55585630339440 starttime: 55584599408800 difftime: 1030930
Jan 30 07:07:00 [54039.445684] ****Entered state was 2
Jan 30 07:07:00 [54040.489763] Took more than 10m second!! endtime: 55586674443920 starttime: 55585630567400 difftime: 1043876
Jan 30 07:07:00 [54040.489789] ****Entered state was 2
Jan 30 07:07:00 [54042.126896] Took more than 10m second!! endtime: 55588311579960 starttime: 55586674673840 difftime: 1636906
Jan 30 07:07:00 [54042.126923] ****Entered state was 2
Jan 30 07:07:00 [54042.230201] Took more than 10m second!! endtime: 55588414818920 starttime: 55588311807720 difftime: 103011
Jan 30 07:07:00 [54042.230227] ****Entered state was 2
Jan 30 07:07:00 [54042.282743] Took more than 10m second!! endtime: 55588467427440 starttime: 55588421540760 difftime: 45886
Jan 30 07:07:00 [54042.282770] ****Entered state was 2
Jan 30 07:07:00 [54042.357552] Took more than 10m second!! endtime: 55588542219480 starttime: 55588467654000 difftime: 74565
Jan 30 07:07:00 [54042.357578] ****Entered state was 2
Jan 30 07:07:00 [54042.397798] Took more than 10m second!! endtime: 55588582483520 starttime: 55588542463680 difftime: 40019
Jan 30 07:07:00 [54042.397824] ****Entered state was 2
Jan 30 07:07:00 [54043.421897] Took more than 10m second!! endtime: 55589606578960 starttime: 55588582709320 difftime: 1023869
Jan 30 07:07:00 [54043.421923] ****Entered state was 2
Jan 30 07:07:00 [54044.466784] Took more than 10m second!! endtime: 55590651467440 starttime: 55589606808520 difftime: 1044658
Jan 30 07:07:00 [54044.466810] ****Entered state was 2
Jan 30 07:07:00 [54045.512970] Took more than 10m second!! endtime: 55591697650960 starttime: 55590651693600 difftime: 1045957
Jan 30 07:07:00 [54045.512996] ****Entered state was 2
Jan 30 07:07:00 [54046.541986] Took more than 10m second!! endtime: 55592726667400 starttime: 55591697880560 difftime: 1028786
Jan 30 07:07:00 [54046.542012] ****Entered state was 2
Jan 30 07:07:00 [54047.587113] Took more than 10m second!! endtime: 55593771795440 starttime: 55592726915080 difftime: 1044880
Jan 30 07:07:00 [54047.587139] ****Entered state was 2
Jan 30 07:07:00 [54049.225392] Took more than 10m second!! endtime: 55595410075920 starttime: 55593772024320 difftime: 1638051
Jan 30 07:07:00 [54049.225419] ****Entered state was 2
Jan 30 07:07:00 [54049.327963] Took more than 10m second!! endtime: 55595512579880 starttime: 55595410304960 difftime: 102274
Jan 30 07:07:00 [54049.327989] ****Entered state was 2
Jan 30 07:07:00 [54049.381142] Took more than 10m second!! endtime: 55595565827240 starttime: 55595519300560 difftime: 46526
Jan 30 07:07:01 [54049.381169] ****Entered state was 2
Jan 30 07:07:01 [54049.455920] Took more than 10m second!! endtime: 55595640587400 starttime: 55595566053440 difftime: 74533
Jan 30 07:07:01 [54049.455946] ****Entered state was 2
Jan 30 07:07:01 [54049.496296] Took more than 10m second!! endtime: 55595680979880 starttime: 55595640830520 difftime: 40149
Jan 30 07:07:01 [54049.496322] ****Entered state was 2
Jan 30 07:07:01 [54050.522219] Took more than 10m second!! endtime: 55596706899920 starttime: 55595681205880 difftime: 1025694
Jan 30 07:07:01 [54050.522246] ****Entered state was 2
Jan 30 07:07:01 [54051.565280] Took more than 10m second!! endtime: 55597749963480 starttime: 55596707130600 difftime: 1042832
Jan 30 07:07:01 [54051.565306] ****Entered state was 2
Jan 30 07:07:01 [54052.597600] Took more than 10m second!! endtime: 55598782283000 starttime: 55597750189000 difftime: 1032094
Jan 30 07:07:01 [54052.597626] ****Entered state was 2
Jan 30 07:07:01 [54053.640457] Took more than 10m second!! endtime: 55599825139960 starttime: 55598782511120 difftime: 1042628
Jan 30 07:07:01 [54053.640483] ****Entered state was 2
Jan 30 07:07:01 [54054.685584] Took more than 10m second!! endtime: 55600870267360 starttime: 55599825367600 difftime: 1044899
Jan 30 07:07:01 [54054.685611] ****Entered state was 2
Jan 30 07:07:01 [54056.323681] Took more than 10m second!! endtime: 55602508363440 starttime: 55600870495560 difftime: 1637867
Jan 30 07:07:01 [54056.323708] ****Entered state was 2
Jan 30 07:07:01 [54056.425772] Took more than 10m second!! endtime: 55602610387440 starttime: 55602508611640 difftime: 101775
Jan 30 07:07:01 [54056.425798] ****Entered state was 2
Jan 30 07:07:01 [54056.479642] Took more than 10m second!! endtime: 55602664324240 starttime: 55602617107600 difftime: 47216
Jan 30 07:07:01 [54056.479668] ****Entered state was 2
Jan 30 07:07:01 [54056.554297] Took more than 10m second!! endtime: 55602738963400 starttime: 55602664552040 difftime: 74411
Jan 30 07:07:01 [54056.554323] ****Entered state was 2
Jan 30 07:07:01 [54056.604479] Took more than 10m second!! endtime: 55602789163960 starttime: 55602739207920 difftime: 49956
Jan 30 07:07:01 [54056.604505] ****Entered state was 2
Jan 30 07:07:01 [54057.649778] Took more than 10m second!! endtime: 55603834459880 starttime: 55602789390680 difftime: 1045069
Jan 30 07:07:01 [54057.649804] ****Entered state was 2
Jan 30 07:07:01 [54058.680747] Took more than 10m second!! endtime: 55604865427480 starttime: 55603834688000 difftime: 1030739
Jan 30 07:07:01 [54058.680773] ****Entered state was 2
Jan 30 07:07:01 [54059.723784] Took more than 10m second!! endtime: 55605908466960 starttime: 55604865657520 difftime: 1042809
Jan 30 07:07:01 [54059.723810] ****Entered state was 2
Jan 30 07:07:01 [54060.769856] Took more than 10m second!! endtime: 55606954538960 starttime: 55605908694680 difftime: 1045844
Jan 30 07:07:01 [54060.769882] ****Entered state was 2
Jan 30 07:07:01 [54061.800824] Took more than 10m second!! endtime: 55607985507000 starttime: 55606954766800 difftime: 1030740
Jan 30 07:07:01 [54061.800850] ****Entered state was 2
Jan 30 07:07:01 [54063.421064] Took more than 10m second!! endtime: 55609605747480 starttime: 55607985735320 difftime: 1620012
Jan 30 07:07:01 [54063.421090] ****Entered state was 2
Jan 30 07:07:01 ABB user.emerg kernel: [54063.472305] BUG: soft lockup - CPU#0 stuck for 98s! [swapper:0]
Jan 30 07:07:01 ABB user.emerg kernel: [54063.478575] INFO: Duration: 98
Jan 30 07:07:01 ABB user.emerg kernel: [54063.481805] INFO: now is: 50350
Jan 30 07:07:01 ABB user.emerg kernel: [54063.485125] INFO: touch_ts is: 50252
Jan 30 07:07:01 ABB user.emerg kernel: [54063.488903] INFO: get_softlockup_thresh is: 20
Jan 30 07:07:01 ABB user.emerg kernel: [54063.493596] INFO: delay (now - touch_ts): 98
Jan 30 07:07:01 ABB user.emerg kernel: [54063.498121] 6.30 6.47 6.52 28/115 10767
Jan 30 07:07:01 [54063.502172] Modules linked in:
Jan 30 07:07:01 [54063.502203]
Jan 30 07:07:01 [54063.502243] CPU: 0 PID: 0 Comm: swapper Not tainted 3.12.52-rt70 #6
Jan 30 07:07:01 [54063.502281] task: c062d0a8 ti: c0622000 task.ti: c0622000
Jan 30 07:07:01 [54063.502344] PC is at cpuidle_enter_state+0x88/0x158
Jan 30 07:07:01 [54063.502381] LR is at ktime_get+0xcc/0x100
Jan 30 07:07:01 [54063.502418] pc : [<c0324be8>] lr : [<c006aea0>] psr: 20050013
Jan 30 07:07:01 [54063.502418] sp : c0623f50 ip : 00000006 fp : c07d92a8
Jan 30 07:07:01 [54063.502453] r10: 00000002 r9 : 00003293 r8 : 9ecf6958
Jan 30 07:07:01 [54063.502481] r7 : 00003293 r6 : a1cc1598 r5 : c0630ae0 r4 : c0622000
Jan 30 07:07:01 [54063.502509] r3 : 00000000 r2 : 02fcac40 r1 : 00003293 r0 : c0623f60
Jan 30 07:07:01 [54063.502542] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
Jan 30 07:07:01 [54063.502573] Control: 10c5387d Table: 8ba40019 DAC: 00000015
Jan 30 07:07:01 [54063.502607] CPU: 0 PID: 0 Comm: swapper Not tainted 3.12.52-rt70 #6
Jan 30 07:07:01 [54063.502706] [<c00128b4>] (unwind_backtrace+0x0/0xdc) from [<c0010c04>] (show_stack+0x10/0x14)
Jan 30 07:07:01 [54063.502769] [<c0010c04>] (show_stack+0x10/0x14) from [<c0093bb8>] (watchdog_timer_fn+0x1d4/0x228)
Jan 30 07:07:01 [54063.502832] [<c0093bb8>] (watchdog_timer_fn+0x1d4/0x228) from [<c0054514>] (__run_hrtimer+0x84/0x12c)
Jan 30 07:07:01 [54063.502885] [<c0054514>] (__run_hrtimer+0x84/0x12c) from [<c0054bb8>] (hrtimer_interrupt+0x114/0x274)
Jan 30 07:07:01 [54063.502948] [<c0054bb8>] (hrtimer_interrupt+0x114/0x274) from [<c0021b5c>] (omap2_gp_timer_interrupt+0x20/0x30)
Jan 30 07:07:01 [54063.503011] [<c0021b5c>] (omap2_gp_timer_interrupt+0x20/0x30) from [<c0065b34>] (handle_irq_event_percpu+0x50/0x1b0)
Jan 30 07:07:01 [54063.503064] [<c0065b34>] (handle_irq_event_percpu+0x50/0x1b0) from [<c0065cd0>] (handle_irq_event+0x3c/0x5c)
Jan 30 07:07:01 [54063.503119] [<c0065cd0>] (handle_irq_event+0x3c/0x5c) from [<c006811c>] (handle_level_irq+0xdc/0xf4)
Jan 30 07:07:01 [54063.503170] [<c006811c>] (handle_level_irq+0xdc/0xf4) from [<c00653ec>] (generic_handle_irq+0x20/0x30)
Jan 30 07:07:01 [54063.503222] [<c00653ec>] (generic_handle_irq+0x20/0x30) from [<c000e5c8>] (handle_IRQ+0x60/0x80)
Jan 30 07:07:01 [54063.503272] [<c000e5c8>] (handle_IRQ+0x60/0x80) from [<c00085d8>] (omap3_intc_handle_irq+0x64/0x9c)
Jan 30 07:07:01 [54063.503325] [<c00085d8>] (omap3_intc_handle_irq+0x64/0x9c) from [<c0011640>] (__irq_svc+0x40/0x88)
Jan 30 07:07:01 [54063.503354] Exception stack(0xc0623f08 to 0xc0623f50)
Jan 30 07:07:01 [54063.503394] 3f00: c0623f60 00003293 02fcac40 00000000 c0622000 c0630ae0
Jan 30 07:07:01 [54063.503436] 3f20: a1cc1598 00003293 9ecf6958 00003293 00000002 c07d92a8 00000006 c0623f50
Jan 30 07:07:01 [54063.503470] 3f40: c006aea0 c0324be8 20050013 ffffffff
Jan 30 07:07:01 [54063.503525] [<c0011640>] (__irq_svc+0x40/0x88) from [<c0324be8>] (cpuidle_enter_state+0x88/0x158)
Jan 30 07:07:01 [54063.503579] [<c0324be8>] (cpuidle_enter_state+0x88/0x158) from [<c0324da0>] (cpuidle_idle_call+0xe8/0x144)
Jan 30 07:07:01 [54063.503628] [<c0324da0>] (cpuidle_idle_call+0xe8/0x144) from [<c000e6f8>] (arch_cpu_idle+0x8/0x44)
Jan 30 07:07:01 [54063.503676] [<c000e6f8>] (arch_cpu_idle+0x8/0x44) from [<c0065360>] (cpu_startup_entry+0x6c/0xc8)
Jan 30 07:07:01 [54063.503746] [<c0065360>] (cpu_startup_entry+0x6c/0xc8) from [<c05e8a04>] (start_kernel+0x2ec/0x358)
Jan 30 07:07:01 [54063.505728] Took more than 10m second!! endtime: 55609656087960 starttime: 55609605974360 difftime: 50113
Jan 30 07:07:01 [54063.505763] ****Entered state was 2
Jan 30 08:01:04 [57307.705423] hrtimer: interrupt took 226080 ns
Jan 30 08:05:28 [57469.046505] Took more than 10m second!! endtime: 59015231116000 starttime: 59015212535080 difftime: 18580
Jan 30 08:05:28 [57469.046559] ****Entered state was 2
Jan 30 08:05:28 [57469.107424] Took more than 10m second!! endtime: 59015292085880 starttime: 59015231943680 difftime: 60142
Jan 30 08:05:28 [57469.107456] ****Entered state was 2
Jan 30 08:05:28 [57469.254827] Took more than 10m second!! endtime: 59015439530640 starttime: 59015293187480 difftime: 146343
Jan 30 08:05:28 [57469.254857] ****Entered state was 2
Jan 30 08:05:28 [57470.092006] Took more than 10m second!! endtime: 59016276674920 starttime: 59015439663680 difftime: 837011
Jan 30 08:05:28 [57470.092032] ****Entered state was 2
We currently have 3 cpuidle states C0,C1 and C2. C1 and C2 are same. Here are the power state settings. (Also we have a LPDDR RAM on our board).
The board is running on 300 MHz and kernel tick is 100Hz. We are running a initramfs.gz image and our persistent memory storage is eMMC.
struct cpuidle_state am33xx_ddr2_states[] = {
{
.exit_latency = 72,
.target_residency = 150,
.power_usage = 625,
.flags = CPUIDLE_FLAG_TIME_VALID,
.enter = am33xx_enter_idle,
.name = "C0",
.desc = "WFI",
},
{
.exit_latency = 235, /* Empirically Measured at 300 MHz */
.target_residency = 500, /* Optimized to be 30 % of time */
.power_usage = 562,
.flags = CPUIDLE_FLAG_TIME_VALID | AM33XX_FLAG_MPU_PLL,
.enter = am33xx_enter_idle,
.name = "C1",
.desc = "Bypass MPU PLL",
},
{
.exit_latency = 235, /* Empirically Measured at 300 MHz */
.target_residency = 500, /* Optimized to be 30 % of time */
.power_usage = 562,
.flags = CPUIDLE_FLAG_TIME_VALID | AM33XX_FLAG_MPU_PLL,
.enter = am33xx_enter_idle,
.name = "C2", /* Added for Compatibility. same as state 1*/
.desc = "C1 - DUP",
},
The prints before the kernel crash gives the time information of processor in that state. (/drivers/cpuidle/cpuidle.c). We have configured the tick to be 100Hz but most of the time when there is a kernel bug message the time spent in the state 2 is more than a second!
int cpuidle_enter_state(struct cpuidle_device *dev, struct cpuidle_driver *drv,
int index)
{
int entered_state;
struct cpuidle_state *target_state = &drv->states[index];
ktime_t time_start, time_end;
s64 diff;
time_start = ktime_get();
entered_state = target_state->enter(dev, drv, index);
time_end = ktime_get();
local_irq_enable();
diff = ktime_to_us(ktime_sub(time_end, time_start));
if (diff > INT_MAX)
diff = INT_MAX;
if (diff > 10500){
printk("************ Took more than 10m second!! endtime: %lld starttime: %lld difftime: %lld********\n",time_end.tv64, time_start.tv64, diff);
printk("****Entered state was %d\n",entered_state );
}
....
I was wondering if anyone have similar problems running realtime and power management.
Thanks,
Harsha
