Options around managing large datasets and Apex Heap Size

edited April 11, 2017 in Questions
We're doing some data visualisations in our Skuid pages using several Javascript libraries. They work beautifully, but the only challenge we're facing is the Apex heap size limit. Being data visualisations, they want to access chunky datasets from our Salesforce objects. The performance in the browser is fine, but eventually we inevitably run into the heap size limit when we exceed 3MB.

So I did some experimenting and wanted to verify it here.

I separated one visualisation into a separate Skuid page and included that in a new tab in our main page using a page include. I checked the debug logs and we seem to now end up with two heap sizes to work with: one for our main page (which is housed in a Visualforce page) and one for the page include (which runs a Skuid Visualforce page called Include). That suggests that if we do a page include for each one of our visualisations, then we may get around the heap size limit.
  1. Is this the best approach for what I'm trying to achieve?
  2. When I access the new tab in our main page, the included Skuid page takes a few seconds, even when its model's Max # of Records is set to a low number like 10. Is that to be expected for includes?
Many thanks.

Comments

  • edited April 11, 2017
    Hi Glenn,

    1. For now I think your method is a good way to get more heap.  

    2. Do you know if you just pull up the included page separately if it takes about the same time to load?

    For the future, we're thinking through adding an API method to models called "getMoreData()" or something like that.  Basically, it would use SOQL's OFFSET and LIMIT functionality to get the next set of data for the model.  This could be repeated as many times as you would like to get as much data from salesforce as you would like.  It may take a few round trips to get all your data, but once you had it in your browser you could do whatever you wanted with it.  We're thinking you could just specify a "chunk size" and and maybe "max round trips" and Skuid would handle the rest to fill up your model.

    Does that sound like something you guys could use?
  • edited December 1, 2014
    1. OK, cool.
    2. No, it's very quick. The page on its own takes about 1 sec (loading 3000 records), when included it takes about 3-4 seconds with the "Loading..." label showing.

    I like the getMoreData() idea. We could certainly make use of that.
  • edited March 1, 2017
    Update for this post.  A while back we added an API method to skuid models called loadNextOffsetPage().  Documentation on this function is here... http://help.skuidify.com/m/11720/l/205447-skuid-model-model 

  • edited September 1, 2016
    If anybody wants an awesome little function to load data by offset:
    function load(model){    
        model.loadNextOffsetPage(function(){
            load(model);
        });
    }
    load(skuid.model.getModel('MyModel'));
    Just put this in a pageload function, and enjoy.


  • Rob HatchRob Hatch 💎💎💎
    edited September 1, 2016
    Thanks for sharing Moshe! Happy Friday. 
  • I am trying to query a model multiple times by setting different conditions in snippet in a for loop and tried to get data batch wise. But get more data is not working as expected. Any thought?
  • Rich SlackRich Slack ✭✭
    edited July 9, 2020

    Disclaimer- you may have more chunky data than I do. But one way I have avoided page heap size is by breaking the page itself into multiple page includes. IE I may have a couple objects on the beefy page but then I separated the Charts into their own page include and just fed the parameters into the page include. When I did this I could include multiple page include charts in the same page without issue. (reminder-object names within the page include have to be unique so there isn't conflicts). But it allowed me to bring all my data into one page without an issue. You have to keep the page include below the HEAP limit. So in your case maybe one chart per include in some cases. But collectively I haven't seen an impact on the main page as long as the page include didn't bust the limits. I am not an expert coder like many of you, but that was my experience.

  • As well use a lot of page includes for those situations, but another thing that can maybe help, is the models you query at page load

    So example instead of querying let's say 10 models at page load, I query let's say 2-3 and in those i create an model action, when requeried to query the other model(s). But this slows down a bit loading time

    As well if you using tabs on that page, the model in tabs you don't need right away can be queried only at the moment you go on that tab and not on page load

    Or if you have TFG component installed, I use this often i put a collapsible wrapper on my page, I hide wrapper header, and in actions i query model(s) on 'Before first open'

    There's many other little tricks like this if you need, let me know

    Hope it helps

Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!