unknown issue causing "APEX CPU time limit exceeded"

  • 2
  • Problem
  • Updated 10 months ago
  • In Progress
Photo of mB Pat Vachon

mB Pat Vachon, Champion

  • 43,056 Points 20k badge 2x thumb

Posted 2 years ago

  • 2
Photo of Barry Schnell

Barry Schnell, Champion

  • 18,266 Points 10k badge 2x thumb
Pat -

If you're not familiar with Salesforce Governors, I'd recommend first reading up on them starting with https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/apex_gov_limits.htm.

The intermittent nature of what you are seeing can be explained by the not to obvious fact that SFDC states that they have the "right to kill" your process if you exceed a governor.  The key word is "right" - note that it doesn't say they "will."  In short, if an execution context exceeds one of the governors, sometimes SFDC will kill your request and sometimes it won't.  There's no docs as to when they will or when they won't unfortunately.  That said, if the load on your instance is low enough at the time of the request SFDC will likely let things run a little longer than the governor allows for.  When load is higher, they will likely kill the request as soon as you exceed the limit.  In short, this is more than likely why sometimes things work and sometimes they don't.

Another possibility are due to other factors that come in to play regarding CPU usage keeping in mind that the same exact request will always vary a little bit with regards to the amount of CPU time required.  It is possible that your current config is right on the border line of the max CPU and so sometimes things run a bit faster and the request succeeds while other times, things run a bit slower and the request fails.

As for why the page is taking a long time, I'd recommend using the developer console and setting up DEBUG tracing to try to determine where the request is spending the bulk of its time to start to isolate the root cause.
(Edited)
Photo of mB Pat Vachon

mB Pat Vachon, Champion

  • 43,056 Points 20k badge 2x thumb
Still uber weird. Setup a debug log with everything turned on. A SOQL query that triggers the issue takes about 30 seconds before returning 0 rows. Copying that query into dev console has no issues and returns 0 rows immediately.

Debug log
Photo of Shmuel Kamensky

Shmuel Kamensky, Champion

  • 4,644 Points 4k badge 2x thumb
Maybe it's the amount of metadata being requested? What happens if you decrease the number of fields being queried? Are there any relationship and/or polymorphic fields being queried?
Photo of mB Pat Vachon

mB Pat Vachon, Champion

  • 43,056 Points 20k badge 2x thumb
Nope. Removing the id parameter let's the page load fine.
Photo of Shmuel Kamensky

Shmuel Kamensky, Champion

  • 4,644 Points 4k badge 2x thumb
Does the metadata still load if the query produces no results? I see some of your queries have noquery behavior if the conditions aren't satisfied.
Photo of mB Pat Vachon

mB Pat Vachon, Champion

  • 43,056 Points 20k badge 2x thumb
Yup. All the models load without issue.
Photo of Shmuel Kamensky

Shmuel Kamensky, Champion

  • 4,644 Points 4k badge 2x thumb
That is weird. How are you doing on API Call Usage?
Photo of mB Pat Vachon

mB Pat Vachon, Champion

  • 43,056 Points 20k badge 2x thumb
HUH?
Photo of Shmuel Kamensky

Shmuel Kamensky, Champion

  • 4,644 Points 4k badge 2x thumb
It's far out, but skuid makes API calls to Salesforce. It's worth checking to see if your API request limit has been reached. You can check it in the System Overview section. 
Photo of mB Pat Vachon

mB Pat Vachon, Champion

  • 43,056 Points 20k badge 2x thumb
Photo of Shmuel Kamensky

Shmuel Kamensky, Champion

  • 4,644 Points 4k badge 2x thumb
Can you post the debug log that showed the 30 second query somewhere?
Photo of mB Pat Vachon

mB Pat Vachon, Champion

  • 43,056 Points 20k badge 2x thumb
Strange. I had posted something earlier with that.

Here it is.

https://docs.google.com/spreadsheets/...
Photo of Shmuel Kamensky

Shmuel Kamensky, Champion

  • 4,644 Points 4k badge 2x thumb
I see. You do query lots of relationships going three levels deep in one of them. I think CreatedBy and LastModifedBy are polymorphic relationships since they could be groups or users. But like you said that shouldn't be the issue since it's working when you remove the where condition.

 Some possible contributors to the issue:
Photo of Karen Waldschmitt

Karen Waldschmitt, Official Rep

  • 8,540 Points 5k badge 2x thumb
Pat~

Wanted to check in and see if you're still having this issue. 

Thanks!
Karen
Photo of mB Pat Vachon

mB Pat Vachon, Champion

  • 43,056 Points 20k badge 2x thumb
Yes. It is still an issue. Very very odd behaviour.
Photo of Rick Nendza

Rick Nendza

  • 110 Points 100 badge 2x thumb
na43 has been having issues since a couple of days prior to new years (they only admitted it as of today on trust). We encountered the same thing. They probably start cracking down on CPU limits when their instance starts barfing. Considering cpu time is cumulative of apex code, workflows, managed packages etc.. this can be one of the biggest pita's to track down as tools / logging leave a lot to be desired on this platform.
Photo of Barry Schnell

Barry Schnell, Champion

  • 18,226 Points 10k badge 2x thumb
Hey Pat -

If NA43 is/was having issues, then all bets are off of course.  According to trust, NA43 is green - take that for what it's worth though especially given what Rick had to share about apparent issues late last year that weren't acknowledged until today.

Assuming you still have issues on this next time you try, let us know the following:

1) How many records are in the object involved in the query?
2) Are you using client side model or server side model?
3) If you remove the Line_Number__c part of the where clause what happens?
4) Do you have a sandbox that you can test this in?  If yes, how many records in that object and what are result of the page?
5) Was this page working previously and all of a sudden started experiencing this issue without any configuration changes?  Or is this a new page?  Or did the page change recently and then the issue start?
(Edited)
Photo of mB Pat Vachon

mB Pat Vachon, Champion

  • 43,056 Points 20k badge 2x thumb
Yup. The page has since started to perform significantly better. Not 100%,but high nineties. Super annoying since I spent something like 5 hours on the issue. ERG! Wasn't a complete loss as I found a few small issues that needed addressing and brushed up on my debug log skills.
Photo of Rick Nendza

Rick Nendza

  • 110 Points 100 badge 2x thumb
That's great!. If you click on the orange section you'll see the history which sort of coincides. https://status.salesforce.com/status/.... (you may have to page back a day) I had an office of 20 lawyers that was sent home due to this. So there is friendship in misery my friend. :) Silver lining....Perhaps it will generate another project for me tuning the 70k lines of code that the client has. :) Beware that most likely you will not encounter this issue in a sandbox, probably not even a full sandbox if your lucky enough to have one ($$$)... since they are not as harsh on killing processes that violate the 10k milli cpu limit since sandbox hardware is not as powerful. The only true way is to prepare ahead of time is via sifting through the logs (grrr) and make sure the processes are well under the limit.
(Edited)
Photo of mB Pat Vachon

mB Pat Vachon, Champion

  • 43,056 Points 20k badge 2x thumb
That's the oddest part of it. The "offending" query ran in dev. console in less than a second but was timing out in the debug logs.
Photo of Rick Nendza

Rick Nendza

  • 110 Points 100 badge 2x thumb
Yep. It's sort of a rough one.. Somewhat non-related.. Note. SOQL does NOT count towards these limits. It's more any looping to build the soql / and processing of soql results in a loop. and any line of managed packaged code / workflow / process / flow..etc that runs in the same transaction.. https://developer.salesforce.com/foru...
Photo of mB Pat Vachon

mB Pat Vachon, Champion

  • 43,056 Points 20k badge 2x thumb
Hmmm .... so that suggests that it wasn't the queries but something else to do with the page? Stranger still, because I had stripped the page down to just 4 models without components, js, css. The page was being used in via preview, so no page overrides. No custom vf, apex, flows or process builders on the objects. One workflow, but it wouldn't run unless the specified change happened. It was as if the object being queried didn't like being queried via Skuid. That to me suggests the version of the API, if different from dev. console, was the issue.
Photo of Rick Nendza

Rick Nendza

  • 110 Points 100 badge 2x thumb
I'm not really at all familiar with SKUID to be honest.. more of an SFDC guy. I'm assuming it's some managed package installed on the SFDC instance that takes this xml and builds queries / results behind the scenes.
Photo of mB Pat Vachon

mB Pat Vachon, Champion

  • 43,056 Points 20k badge 2x thumb
Yup. Spot on. The queries being built were asking for a record each and approx. 25-50 fields each. So really not something for SF to trip up on.
Photo of Rick Nendza

Rick Nendza

  • 110 Points 100 badge 2x thumb
I encountered the cpu limit on various queries that returned only 1 row as well. For us. it wasn't so much the nbr of rows returned but rather the looping that was building the soql as we were doing describe calls and then evaluating each field deciding whether or not to put it in the SOQL string.
Photo of Shmuel Kamensky

Shmuel Kamensky, Champion

  • 4,644 Points 4k badge 2x thumb
Rick, depending on how the SOQL is implemented, it's possible that it could eat away at your CPU limits. E.g. if you did a Database.QueryLocator 
Photo of Rick Nendza

Rick Nendza

  • 110 Points 100 badge 2x thumb
We did not. It's a simple 1 row result / real time query not used in Batch Apex but rather a controller backing a VF Page.
Photo of Shmuel Kamensky

Shmuel Kamensky, Champion

  • 4,644 Points 4k badge 2x thumb
I think Database.query can also give a CPU timeout exception. Is that how skuid runs queries? Or do they have some other system for dynamic query strings?
Photo of Karen Waldschmitt

Karen Waldschmitt, Official Rep

  • 8,540 Points 5k badge 2x thumb
Hi community~

We tried to contact Salesforce about getting more details about what happened with NA43. Here's what we heard back: 

If you need more information outside of what is posted to TRUST, you'll have to log a case to make a formal request.

At this time, Salesforce only sends proactive alerts for system incidents to our premier customers / partners.  
All other customers should proactively use TRUST to check for incidents when issues are noticed, or use the RSS function on instances to setup a RSS feed for status updates when they are posted.

Sorry couldn't help shed more light on what happened but hopefully you can do one of the suggestions above to learn more about things as they affect you ...
Karen
Photo of Rick Nendza

Rick Nendza

  • 110 Points 100 badge 2x thumb
Thank you so much for trying! Typical of SFDC.. Unfortunately my clients do not pay the bucho $$$ for premier support. Any case we submit with any substance ultimately gets thrown back at us telling us to pay the man. lol. Anyway, thanks so much for trying! Maybe others with premier support can share the wealth. :)
Photo of Jack Sanford

Jack Sanford, Champion

  • 8,978 Points 5k badge 2x thumb
I'm having this issue across the board with a large page that is not even querying any models upon page load since Salesforce updated to Summer '18 on NA38. Anyone else having this issue?
Photo of Stephen Sells

Stephen Sells

  • 17,326 Points 10k badge 2x thumb
Are you able to reduce the size of the page via page includes or anything?