Hey Everyone, Hopefully I can clear some of this up.
1. Skuid only uses OFFSET for pagination some of the time. In the vast majority of cases, it uses Standard Set Controllers. It uses Standard Set Controllers for all custom objects and for the small list of standard objects (shown below) that support Standard Set Controllers.
(‘Account’,‘Asset’,‘Campaign’,‘Case’,‘Contact’,‘Contract’,‘Idea’,‘Lead’,‘Opportunity’,‘Quote’,‘Pricebook2’,‘Product2’,‘Solution’)
For all other standard objects, (Task is a common one) Skuid has to use OFFSET because Standard Set Controllers are not supported for that object. OFFSET is very limited because the max OFFSET that can be used is 2000, that means the largest amount of data you can retrieve via this method is 2000 + your query limit.
2. Standard Set Controllers / OFFSET were chosen over queryMore() because they are available to be called straight from APEX, and do not require an API call. I believe missing data would be a problem in all of these approaches. The best way to mitigate this would be to put a good “Order by” on your model. I think “Id” is usually a good choice.
3. In my experience, modern browsers are great at handling large amounts of data in-memory without many problems. 100s of thousands or even millions of records should be possible. You will start running into performance issues when you start trying to iterate over these models to do summaries, or try to use something like MODEL_LOOKUP ui-only formula fields. It’s the DOM that gets really slow, quickly. Skuid already doesn’t add records to the DOM until you actually paginate to that page. I don’t think clearing records out of models because of pagination would be really worthwhile in most cases.
4. Actually, since skuid runs in “read-only” mode, the max row count is 1,000,000. But try processing anything near that amount of data in apex (outside of batch apex) and you’ll run into heap size errors. The export has to come from the client, as I’ll explain in #5. So the limit here is going to be how much data you can get into your model, and possibly heap size issues in Apex.
5. I haven’t looked at the export code in a while, but I believe you can export template fields, ui-only fields, formatted reference fields (with display templates), etc. This is all processed on the client. If you just need a straight up DB dump of your data, there are other tools that may be better suited for that job.
As for the issue with Skuid not alerting users to truncated data, I think this is a bug. You actually get this message when you create a table on the Task object. Once you try to “load more” past 2000 + query limit, you will see a message at the bottom of your table indicating that the pagination limit has been reached. This should be happening for objects that use the standard set controller, but doesn’t seem to be doing that.
I think the reason that for your particular table, you’re getting 11,000 rows instead of 10,000 is because of the differences in how default ordering works in SOQL vs Standard set controllers. I would guess if you told your model to order by “Id”, it would stop at 10,000 records.
