Custom Metadata - A better way to manage Skuid page/pageassignment/etc. migrations?

  • 1
  • Idea
  • Updated 1 year ago
  • Not Planned
As discussed in the community in the past, managing migrations of Skuid pages/pageassignments/etc. can be difficult.  Additionally, tracking pages in a VCS is challenging and often times, what I've heard is that a lot of users building Skuid based pages/apps have a dedicated sandbox org that always contains the "truth" (aka. latest version).

With the introduction of Custom Metadata in Summer '15 and enhancements coming in Winter '16 (native ui support, enhanced permission management, etc.), my thinking is that this might be a great solution to overcoming some of these challenges.

I'm curious if leveraging Custom Metadata would be a fit for pages, page assignments, etc., if this has been explored/evaluated yet and if so, what the future might hold?

Thanks!
Photo of Barry Schnell

Barry Schnell, Champion

  • 18,076 Points 10k badge 2x thumb

Posted 3 years ago

  • 1
Photo of Ben Hubbard

Ben Hubbard, Employee

  • 12,490 Points 10k badge 2x thumb
We would love to move to Custom Metadata for Pages, Page Assignments, Model Services, Model Service Adapters and other things. It's probably something we'll be exploring in the next Major Release after Banzai. The biggest issue right now is their lack of support for any long text. So we won't be able to move Pages until something like that is supported.
Photo of Zach McElrath

Zach McElrath, Employee

  • 49,056 Points 20k badge 2x thumb
There are 3 major reasons why we cannot use Custom Metadata types yet for Skuid, much as we would like to:

  1. No Long Textarea (LOB) fields --- Skuid Page XML metadata is stored in a combination of several Long Textarea fields on the Page object. Because Custom Metadata types do not offer Long Textarea fields, we really can't use them right now as a viable alternative for storing Skuid Page metadata.
  2. No Relationship / Lookup fields --- the Skuid Page Version and Page Assignment objects have lookups / master-detail relationship fields to Page records. And with Banzai, the Page object has a Lookup field to Page, to allow for Master Page functionality. Custom Metadata Types do not have a concept of Relationship fields. 
  3. No Triggers on Custom Metadata types --- the Page and Page Version objects have trigger logic that is essential for their functionality. Custom Metadata types do not support triggers.
  4. No DML support on Custom Metadata Types --- currently we use DML to create/modify/delete Skuid objects / custom settings. Custom Metadata Types can only be edited with the Metadata API. This presents substantial migration challenges for our existing codebase, including creation / updates of records from our Post-Install Apex script, modification of pages from the Skuid UI / Composer... it would take a lot of complex migration work.

In short, we are functionally unable to migrate any of the core Skuid objects to Custom Metadata types in the foreseeable future. Some of the new configuration objects we have added with Banzai could be migrated to Custom Metadata types, but we do not have immediate plans to do this.

I have heard that Relationship fields are up soon on the Custom Metadata Types roadmap, and after that LOB fields, but no timelines have been disclosed yet.
Photo of Barry Schnell

Barry Schnell, Champion

  • 18,076 Points 10k badge 2x thumb
Thanks for the insight Ben/Zach, makes sense why moving there now isn't possible but glad to hear that it's a potential future direction once MDT matures.  Having this in place would make things so much easier in terms of deployment management, version control, etc.

I've read the roadmap includes Relationship/Lookup Fields and APEX DML support, hopefully LOB & Trigger support will be included as well.  Though not ideal, I could see ways to overcome the lack of trigger support but lacking LOB would be make things difficult, or at the very least, extremely messy.
Photo of mB Pat Vachon

mB Pat Vachon, Champion

  • 42,714 Points 20k badge 2x thumb
So I've been setting up Source Control in BitBucket. One thing that would make this setup much better would be if Skuid were to use Custom Metadata.

I've looked at the list of the 4 issues provided and it seems all but "No Triggers on Custom Metadata types" is now available. Is there any update on this? A workaround maybe?

I know that static resources can be used to commit changes to source control, but the issue with this is that there is still a very manual process to ensure there are no overwrites. Can't very well pack everything into one static resource, so steps must be taken to pack changed pages into their own static resource.
Photo of mB Pat Vachon

mB Pat Vachon, Champion

  • 42,714 Points 20k badge 2x thumb
Any tips and tricks anyone would recommend on this?

My current strategy is clone the Skuid Page into new Record/Named version and update Visualforce. Allows for one page pack everytime.
Photo of Zach McElrath

Zach McElrath, Employee

  • 49,056 Points 20k badge 2x thumb
Pat, any reason why you're not using Skuid Grunt or Skuid CLI (https://github.com/skuid/skuid) to do source control on Skuid Pages? This is our recommended approach. When doing active development on Skuid Pages, I would definitely use one of these two tools instead of Page Packs.
Photo of Barry Schnell

Barry Schnell, Champion

  • 18,076 Points 10k badge 2x thumb
Glad you enjoyed Matt :) 

Technically, there's nothing wrong with lots of static resources per se.  However, the more "stuff" your managing via non-native processes the more risk you introduce in to your overall solution.  Additionally, org refreshes with dev tools, etc. become slower as do deployments (more files to sync) thereby decreasing developer efficiency.  Lastly, there are governors on static resources (no file larger than 5MB & 250MB total).  Technically it can work, it just introduces variables and in-turn risk that would otherwise be unnecessary.

I do understand and appreciate your guys desire to find declarative solutions to manage Skuid Pages with a VCS and there are some solutions that will work.  The main challenge is that there isn't an available tool today that does exactly what you guys want/need and therefore a custom solution is required to facilitate it.  While this can be done, that solution needs to be maintained & supported which takes away from focusing on running your business and adding value to that business.  Personally, I just think that the increased "risk" and "overhead" associated with any of those processes isn't worth the "reward."  That said, happy to help continue to find possible solutions to the problem - i just wouldn't recommend any of them :)
(Edited)
Photo of Matt Sones

Matt Sones, Champion

  • 31,478 Points 20k badge 2x thumb
The idea of just notifying the VCS (or some kind of system) to run 'skuid-pull' for the changed page seems ideal to me.  Are there servics that could do that kind of thing?

I'm completely out of my depth here. I've picked up skuid-grunt and use the CLI for skuid and git with bitbucket (turns out the commands are much faster than using SourceTree).

We've been pounding away at trying to automate some kind of deployment system that would work well for both salesforce metadata and skuid data.

Let me try to come up with a fancy picture to demonstrate what our system looks like, and then maybe you folks can offer some suggestions?
Photo of Barry Schnell

Barry Schnell, Champion

  • 18,076 Points 10k badge 2x thumb
That would be very helpful Matt.  The more details on what you guys currently have process & tools wise the more meaningful our discussions around them can be.

Regarding just notifying the VCS to run the CLI commands, if it were me and I was absolutely forced to find a declarative solution, I would target the absolute minimum to get the job done.  This would mean, as you suggest, finding a simple way to initiate a "skuid-pull/push/etc." when a page is changed.  In my other post, I mentioned creating a GUI wrapper on top of the CLI tools or a browser extension - either of these would probably be where I'd start.  This avoids having to do anything on the SFDC side (e.g. triggers, events, etc.) and instead, just requires the developer to remember to push the button in the GUI when they are done developing their page.  The GUI could use SFDC apis and SOQL to query for information, pages, etc. and present the user with a drop down that allows them to "pick" which page they want to "commit to VCS". The browser extension could just take the current page information and initiate the CLI commands although this gets a little complicated due to security limitations from within a browser.
Photo of Bill McCullough

Bill McCullough, Champion

  • 12,436 Points 10k badge 2x thumb
Barry,

Nice start...where's the Github/Gitlab/Bitbucket repo???  :-)

Unfortunately, I don't have experience using my proposed design.  I was 'threading' off a Pat's design and adding some options to the mix.  I have used Skuid Grunt, but primarily for page backup and check in to Git.  I have had the same thoughts as Pat in that there 'ought to be a better way'.  I just haven't pursued them in earnest.

I think BlueCanvas has the nicest out of box experience for declarative and Apex developers.  It keeps track of everything with limited fuss.

Thanks,

Bill
Photo of Barry Schnell

Barry Schnell, Champion

  • 18,076 Points 10k badge 2x thumb
Thanks Bill, all makes sense and agreed - there "should" be a better way :)
Photo of mB Pat Vachon

mB Pat Vachon, Champion

  • 42,714 Points 20k badge 2x thumb
How about mirroring (auto saving) each Skuid page, page assignment and custom theme as a Static Resource? That way none of this malarkey about command line needs to be done.

Seems too simple to me, but can this be done?

More and more tools allow for simplified SDLC processes. We're using Gearset and BitBucket, so we can do everything from the cloud. ie. no local repository or command line.
Photo of Barry Schnell

Barry Schnell, Champion

  • 18,076 Points 10k badge 2x thumb
Hey Pat -

I'm with Zach in that cli tools are the best technical solution available currently.  Hopefully Custom Metadata based Skuid is in the future :)

That said, you asked for a way to avoid using cli.  So while it wouldn't be recommended, here's a couple of thoughts.  I haven't completely flushed these out in my head but I think they could be made to work:

1) Build a GUI based tool on top of the CLI tools.  This avoids your users having to use the CLI but functionally still uses it.  You could write the tool in any language you choose (Java, c#, etc.) and make it portable for different operating systems.  This would accomplish your goal of "clicks" for the users that are maintaining pages.  They would just need to remember go "click" in the GUI after they are done making changes.

2) Using a trigger/flow/process builder or some other SFDC feature, when a record changes, use an HTTP callout to commit that change to your VCS.  If I'm remembering correctly, bitbucket supports a REST API for various actions that can be taken.  You can leverage that such that when someone "saves a page", it invokes a back-end process to commit that change to your VCS.  You could get really fancy here and use something like Mulesoft or Jitterbit (or some other ESB type product) that have orchestration capabilities to do more than just "commit" something.

3) Similar to #2, build a "sync" tool that takes data from Pages object and sync to your VCS.  You would build a queu'ing mechanism that would put a message in the queue indicating that someone saved a page, which org the save was in, what the record id of the save was, etc.  Then, your sync tool would poll for messages, grab the correct version of the page and sync to your VCS.

4) You could write a browser extension that the user could interact with.  The extension would "save" the skuid page and make callouts to the VCS, etc.

Again, just thinking out loud but hopefully this gives you some direction to potentially find a solution to avoid using CLI.  One thing to keep in mind with all the above is that some of them are better than others with regards to making the "save & commit to VCS" atomic - meaning that you are guaranteed to get your commit whenever someone saves.

Tthought of 1 more...

5) You could submit a feature request to Skuid to add extensibility to the Page Builder so that you can put your own "Save" button on it which would save the page and do the callout :)
(Edited)
Photo of Matt Sones

Matt Sones, Champion

  • 31,478 Points 20k badge 2x thumb
I suppose that if all you're doing with VCS is record keeping, then the trigger for creating a static resource would be fine. But you're not going to be able to make changes to the resource and send it back to skuid that way (you could send them back to the resource in salesforce, but it wouldn't impact the skuid page at all).
Photo of mB Pat Vachon

mB Pat Vachon, Champion

  • 42,714 Points 20k badge 2x thumb
It would be easy enough to add code for the Skuid page builder to check for an update on open.

I'd simply like a simplified approach for the strictly declarative devs. Which, again, is most Skuid devs.
Photo of Matt Sones

Matt Sones, Champion

  • 31,478 Points 20k badge 2x thumb
Sounds exciting... but you want to actually change the page without opening the builder, right? It's almost like you're asking skuid to use your generated static resource as the page xml instead of the Page Layout text fields?

Also, Gearset look awesome!
Photo of Ralph Loeffler

Ralph Loeffler

  • 60 Points
I Am Jesus and I want a raise
Photo of mB Pat Vachon

mB Pat Vachon, Champion

  • 42,714 Points 20k badge 2x thumb
Hmmmm .... that would pretty slick, but I'm not sure it would be enough as there are other settings stored in other fields for the page. 

Gearset is awesome for sure. Great fit for Skuid devs as it enables a declarative solution for source control solely  based in the cloud. No CLI or local repos. Compare and deploy between Salesforce org, Local Repo and Source Control.

Photo of Matt Sones

Matt Sones, Champion

  • 31,478 Points 20k badge 2x thumb
Ok, here's approximately what we're doing at the moment... 
It's a manually intensive process, which is mostly working, but we run into hiccups primarily with merging SFDC metadata between orgs after a git merge.

We were attempting to use grunt to create an automated deployment process--basically looping through all our customers and running all the commands that we run on the command line in the right sequence--but it's fraught with opportunities for failure.

Suggestions? 

Photo of Matt Sones

Matt Sones, Champion

  • 31,478 Points 20k badge 2x thumb
We don't necessarily need a clicks-only solution (although it would make it easier for the over Dev on my team). I'm mostly interested in finding a way to better streamline or automate the deployment process.
Photo of Barry Schnell

Barry Schnell, Champion

  • 18,076 Points 10k badge 2x thumb
Hey Matt -

Appreciate the pictures, helpful to better understand, thank you!  The one thing I learned from working with SFDC is that managing "source" is extremely challenging and ripe for failure.  You do need to be meticulous in the way that you manage things and unfortunately, the tools available today just aren't what they need to be.  The introduction of skuid-grunt was very helpful and moved the needle forward but challenges still are present with SFDC metadata and its limitations as well as skuid resources.  In short, as much as SFDC wants everyone to believe that it's an "enterprise development" platform, in my opinion it's not.  The tooling just isn't there - although it's come a long way in the last year (e.g. VS Code extension, SFDC DX, etc.) and hopefully will continue to get better.

Questions:
1) Can you provide some specifics on what challenges you're having with merging metadata?  One thing I've noticed in the past (although it's been quite a while) is that if you don't keep your sandboxes in sync then merges become much more difficult especially around "SFDC Upgrade" time when new features/metadata gets introduced to the orgs.

2) For the automation you do/did have in place, looping through customers, what challenges or opportunities for failure were you facing?
Photo of mB Pat Vachon

mB Pat Vachon, Champion

  • 42,714 Points 20k badge 2x thumb
So many ways to do this! I already know what Skuid likely does with their own org since they've got engineers up the wazoo.

Assuming no failed tests, our process is as follows.
1. Start new feature.
 a) Create Page Pack with needed pages named as per Jira issue key.
 b) Create new sandbox named as per Jira issue key.
 c) Create branch from Jira off master named as per Jira issue key. (BitBucket integration in Jira)
2. Develop
 a) On dev sandbox
3. Commit to Branch
 a) Update page pack
 a) Compare and Deploy from Sandbox to Branch using Gearset
4. Pull Request
 a) Create Pull Request from Jira. (BitBucket integration in Jira)
5. Code Review and Merge
 a) In BitBucket
6. Continuous Integration
 a) Auto Deployed to partial sandbox for apex tests.
7. Deploy to UAT
 a) Deploy to Full Sandbox using Gearset.
8. Deploy to Production

Workflow in Jira


Diagram of the process
Photo of Barry Schnell

Barry Schnell, Champion

  • 18,076 Points 10k badge 2x thumb
I'll leave to you to decide if everything is achieved when deploying from local system w/ Skuid CLI since you know your requirements best :)  Without knowing how GearSet works, I can't be sure but it's possible you can leverage whatever the process is that GearSet uses and invoke a CLI command from that process.

Regarding not liking to go deeper into the weeds "unless it's easy" - I'd challenge you on that :)  I recall a time a while back when you dug in to javascript and Skuid API despite no prior knowledge of JS and, unlike using a CLI tool, involved actual coding.  Just like everything else, there's what you can do out of the box declaratively but to really harness the "power", sometimes you need to jump in to the deep end.  :)
Photo of mB Pat Vachon

mB Pat Vachon, Champion

  • 42,714 Points 20k badge 2x thumb
I'm not too worried about me. It's everyone else I typically work with.
Photo of Matt Sones

Matt Sones, Champion

  • 31,478 Points 20k badge 2x thumb
Even the other folks you work with can handle the command line. It's really just memorizing (or keeping a list) of 1/2 a dozen commands. The CLI may feel intimidating at first (it certainly did to me, I avoided it for a long time), but in the end I've found it to be much easier to work with than any of the graphic tools.

For you team to learn "when I finish making changes to my page, i go to the little black box and type `skuid-pull`" is not really any harder than learning to build a page pack, right?
Photo of mB Pat Vachon

mB Pat Vachon, Champion

  • 42,714 Points 20k badge 2x thumb
I'm not too worried about the ability to use Skuid CLI in a bubble. It's more about getting users who've never used VCS of any kind to know how to do git on a local machine to begin with. This is/was especially significant when the project timeline allowed for only 2 weeks to get Agile Development, SDLC and VCS up and running.

My desire is to get VCS integration done declaratively. Like Skuid. ;)
Photo of Matt Sones

Matt Sones, Champion

  • 31,478 Points 20k badge 2x thumb
Yep!