Options around managing large datasets and Apex Heap Size

  • 3
  • Question
  • Updated 3 years ago
  • Answered
We're doing some data visualisations in our Skuid pages using several Javascript libraries. They work beautifully, but the only challenge we're facing is the Apex heap size limit. Being data visualisations, they want to access chunky datasets from our Salesforce objects. The performance in the browser is fine, but eventually we inevitably run into the heap size limit when we exceed 3MB.

So I did some experimenting and wanted to verify it here.

I separated one visualisation into a separate Skuid page and included that in a new tab in our main page using a page include. I checked the debug logs and we seem to now end up with two heap sizes to work with: one for our main page (which is housed in a Visualforce page) and one for the page include (which runs a Skuid Visualforce page called Include). That suggests that if we do a page include for each one of our visualisations, then we may get around the heap size limit.

  1. Is this the best approach for what I'm trying to achieve?
  2. When I access the new tab in our main page, the included Skuid page takes a few seconds, even when its model's Max # of Records is set to a low number like 10. Is that to be expected for includes?
Many thanks.

Photo of Glenn Elliott

Glenn Elliott, Champion

  • 7,738 Points 5k badge 2x thumb

Posted 5 years ago

  • 3
Photo of Ben Hubbard

Ben Hubbard, Employee

  • 12,490 Points 10k badge 2x thumb
Hi Glenn,

1. For now I think your method is a good way to get more heap.  

2. Do you know if you just pull up the included page separately if it takes about the same time to load?

For the future, we're thinking through adding an API method to models called "getMoreData()" or something like that.  Basically, it would use SOQL's OFFSET and LIMIT functionality to get the next set of data for the model.  This could be repeated as many times as you would like to get as much data from salesforce as you would like.  It may take a few round trips to get all your data, but once you had it in your browser you could do whatever you wanted with it.  We're thinking you could just specify a "chunk size" and and maybe "max round trips" and Skuid would handle the rest to fill up your model.

Does that sound like something you guys could use?
(Edited)
Photo of Glenn Elliott

Glenn Elliott, Champion

  • 7,738 Points 5k badge 2x thumb
1. OK, cool.
2. No, it's very quick. The page on its own takes about 1 sec (loading 3000 records), when included it takes about 3-4 seconds with the "Loading..." label showing.

I like the getMoreData() idea. We could certainly make use of that.
Photo of Ben Hubbard

Ben Hubbard, Employee

  • 12,490 Points 10k badge 2x thumb
Update for this post.  A while back we added an API method to skuid models called loadNextOffsetPage().  Documentation on this function is here... http://help.skuidify.com/m/11720/l/205447-skuid-model-model 
Photo of Moshe Karmel

Moshe Karmel, Champion

  • 8,646 Points 5k badge 2x thumb
If anybody wants an awesome little function to load data by offset:
function load(model){    
    model.loadNextOffsetPage(function(){
        load(model);
    });
}
load(skuid.model.getModel('MyModel'));
Just put this in a pageload function, and enjoy.
Photo of Rob Hatch

Rob Hatch, Official Rep

  • 44,006 Points 20k badge 2x thumb
Thanks for sharing Moshe! Happy Friday.