Custom Metadata - A better way to manage Skuid page/pageassignment/etc. migrations?

Glad you enjoyed Matt :) 

Technically, there’s nothing wrong with lots of static resources per se.  However, the more “stuff” your managing via non-native processes the more risk you introduce in to your overall solution.  Additionally, org refreshes with dev tools, etc. become slower as do deployments (more files to sync) thereby decreasing developer efficiency.  Lastly, there are governors on static resources (no file larger than 5MB & 250MB total).  Technically it can work, it just introduces variables and in-turn risk that would otherwise be unnecessary.

I do understand and appreciate your guys desire to find declarative solutions to manage Skuid Pages with a VCS and there are some solutions that will work.  The main challenge is that there isn’t an available tool today that does exactly what you guys want/need and therefore a custom solution is required to facilitate it.  While this can be done, that solution needs to be maintained & supported which takes away from focusing on running your business and adding value to that business.  Personally, I just think that the increased “risk” and “overhead” associated with any of those processes isn’t worth the “reward.”  That said, happy to help continue to find possible solutions to the problem - i just wouldn’t recommend any of them :slight_smile:

The idea of just notifying the VCS (or some kind of system) to run ‘skuid-pull’ for the changed page seems ideal to me.  Are there servics that could do that kind of thing?

I’m completely out of my depth here. I’ve picked up skuid-grunt and use the CLI for skuid and git with bitbucket (turns out the commands are much faster than using SourceTree).

We’ve been pounding away at trying to automate some kind of deployment system that would work well for both salesforce metadata and skuid data.

Let me try to come up with a fancy picture to demonstrate what our system looks like, and then maybe you folks can offer some suggestions?

That would be very helpful Matt.  The more details on what you guys currently have process & tools wise the more meaningful our discussions around them can be.

Regarding just notifying the VCS to run the CLI commands, if it were me and I was absolutely forced to find a declarative solution, I would target the absolute minimum to get the job done.  This would mean, as you suggest, finding a simple way to initiate a “skuid-pull/push/etc.” when a page is changed.  In my other post, I mentioned creating a GUI wrapper on top of the CLI tools or a browser extension - either of these would probably be where I’d start.  This avoids having to do anything on the SFDC side (e.g. triggers, events, etc.) and instead, just requires the developer to remember to push the button in the GUI when they are done developing their page.  The GUI could use SFDC apis and SOQL to query for information, pages, etc. and present the user with a drop down that allows them to “pick” which page they want to “commit to VCS”. The browser extension could just take the current page information and initiate the CLI commands although this gets a little complicated due to security limitations from within a browser.

Ok, here’s approximately what we’re doing at the moment…
It’s a manually intensive process, which is mostly working, but we run into hiccups primarily with merging SFDC metadata between orgs after a git merge.

We were attempting to use grunt to create an automated deployment process–basically looping through all our customers and running all the commands that we run on the command line in the right sequence–but it’s fraught with opportunities for failure.

Suggestions?

We don’t necessarily need a clicks-only solution (although it would make it easier for the over Dev on my team). I’m mostly interested in finding a way to better streamline or automate the deployment process.

Barry,

Nice start…where’s the Github/Gitlab/Bitbucket repo???  :slight_smile:

Unfortunately, I don’t have experience using my proposed design.  I was ‘threading’ off a Pat’s design and adding some options to the mix.  I have used Skuid Grunt, but primarily for page backup and check in to Git.  I have had the same thoughts as Pat in that there ‘ought to be a better way’.  I just haven’t pursued them in earnest.

I think BlueCanvas has the nicest out of box experience for declarative and Apex developers.  It keeps track of everything with limited fuss.

Thanks,

Bill

So many ways to do this! I already know what Skuid likely does with their own org since they’ve got engineers up the wazoo.

Assuming no failed tests, our process is as follows.

  1. Start new feature.
    a) Create Page Pack with needed pages named as per Jira issue key.
    b) Create new sandbox named as per Jira issue key.
    c) Create branch from Jira off master named as per Jira issue key. (BitBucket integration in Jira)
  2. Develop
    a) On dev sandbox
  3. Commit to Branch
    a) Update page pack
    a) Compare and Deploy from Sandbox to Branch using Gearset
  4. Pull Request
    a) Create Pull Request from Jira. (BitBucket integration in Jira)
  5. Code Review and Merge
    a) In BitBucket
  6. Continuous Integration
    a) Auto Deployed to partial sandbox for apex tests.
  7. Deploy to UAT
    a) Deploy to Full Sandbox using Gearset.
  8. Deploy to Production

Workflow in Jira

Diagram of the process

Hey Matt -

Appreciate the pictures, helpful to better understand, thank you!  The one thing I learned from working with SFDC is that managing “source” is extremely challenging and ripe for failure.  You do need to be meticulous in the way that you manage things and unfortunately, the tools available today just aren’t what they need to be.  The introduction of skuid-grunt was very helpful and moved the needle forward but challenges still are present with SFDC metadata and its limitations as well as skuid resources.  In short, as much as SFDC wants everyone to believe that it’s an “enterprise development” platform, in my opinion it’s not.  The tooling just isn’t there - although it’s come a long way in the last year (e.g. VS Code extension, SFDC DX, etc.) and hopefully will continue to get better.

Questions:
1) Can you provide some specifics on what challenges you’re having with merging metadata?  One thing I’ve noticed in the past (although it’s been quite a while) is that if you don’t keep your sandboxes in sync then merges become much more difficult especially around “SFDC Upgrade” time when new features/metadata gets introduced to the orgs.

2) For the automation you do/did have in place, looping through customers, what challenges or opportunities for failure were you facing?


Thanks Bill, all makes sense and agreed - there “should” be a better way :slight_smile:

Hey Pat -

Thanks for this, very helpful.  A few thoughts:

1) Your approach seems risky unless you are very careful about which pages you are modifying in each feature.  For example, what do you do when “Feature A” and “Feature B” are worked on simultaneously and both need to modify “Page ABC”?  Using page packs would make this very difficult to merge properly.

2) You mentioned each page pack has its own “module” name.  Did you mean Skuid “Module”?  If so, how do you reconcile these upon merge to CI/UAT/etc?  In other words, if Feature A gets a module name of “Feature A” then all pages will be in that Module.  What if this feature needs to modify a page that already exists in a different module?  Or what if this feature is creating a new master page or shared page of some type that another feature at some point needs?  Do the pages remain in Feature A Module? 

3) If you were to use the CLI (I understand you don’t want to so just humor me :)), do you think there would be other challenges you’d face if you just swapped in CLI to your process instead of page packs?

4) With your current process, what challenges are you facing specifically?

Never responded to this.

We’ve settled on working on Skuid pages one at a time.

I see challenges in using CLI since most of the developers i work with aren’t coding developers. They never use CLI. They are declarative developers like myself. The Gearset UX is way way easier to learn and execute on.

Page packs are required in order to get the pages into other orgs via source control. Do Skuid Grunt or Skuid CLI tools alleviate this? If so how?

Hey Pat - Yes, Skuid CLI (or Grunt) provide a mechanism to handle individual pages and pull/push to any org.  After “pulling” the pages from the org, you use your VCS of choice to commit them to source control.  See the readme at https://github.com/skuid/skuid for more details on how to use.

I guess what I’m missing is an understanding of being able to view the Skuid page XML as metadata so we can commit, review and merge changes to the page. Such as adding an action to a button. Right now, the only thing I see options for is to replace the page in it’s entirety.

When you “pull” (using Skuid CLI), it will grab the latest XML for the specified resources from the org and place a copy of those on your file system.  As you mention, this will “replace” the copy that you currently have on your file system.  You can then view the XML with any editor (e.g. VS Code, Eclipse, notepad++, vi, etc.) When you commit those changes (typically via a Pull Request if using GIT), you’ll need to merge them.  The “merge” is the same process that would be needed for any type of source controlled file (e.g. sfdc metadata files, js files, etc.).

Yup. Still not clear for me. Is this process going to allow the XML of the page to be updated in the merge? Will the dif show the that on line 68 the URL parameter was changed from “Id” to “id”? If so, what’s the process to subsequently deploy the page from source control to a Salesforce org? As a page pack it goes in with a change set or a deployment in Gearset. Final step would be to unpack the page.

Yes, the diff would show each character/line that changed from what’s in your VCS to what’s being proposed to being committed.  

Deployment to an org would come via an automated release process that uses the CLI.

I’m not familiar with Gearset so not familiar with how it works.  Assuming Gearset uses the local file system to manage pushes/pulls to your org, using the CLI should be no different than managing SFDC metadata or the page packs.  If GearSet doesn’t use the local file system (or doesn’t at least provide the option to use the local file system), then CLI might not be an option.

 If you’re managing your page packs such that each pack contains 1 and only 1 page, then using the CLI won’t buy you much other than alleviating having to manually create packs for each page. 

My best advice here is to try it and see if it yields the results you’re after :slight_smile:

Ok. Seems like everything desired is achieved when deploying from a local system w/ Skuid CLI?

I already know that local file system and CLI is a barrier of entry for most declarative developers. We’re declarative because we don’t like to go deeper into the weeds unless it’s easy. :wink:

So everything except easy.

I’ll leave to you to decide if everything is achieved when deploying from local system w/ Skuid CLI since you know your requirements best :)  Without knowing how GearSet works, I can’t be sure but it’s possible you can leverage whatever the process is that GearSet uses and invoke a CLI command from that process.

Regarding not liking to go deeper into the weeds “unless it’s easy” - I’d challenge you on that :)  I recall a time a while back when you dug in to javascript and Skuid API despite no prior knowledge of JS and, unlike using a CLI tool, involved actual coding.  Just like everything else, there’s what you can do out of the box declaratively but to really harness the “power”, sometimes you need to jump in to the deep end.  :slight_smile:

I’m not too worried about me. It’s everyone else I typically work with.

Even the other folks you work with can handle the command line. It’s really just memorizing (or keeping a list) of 1/2 a dozen commands. The CLI may feel intimidating at first (it certainly did to me, I avoided it for a long time), but in the end I’ve found it to be much easier to work with than any of the graphic tools.

For you team to learn “when I finish making changes to my page, i go to the little black box and type ‘skuid-pull’” is not really any harder than learning to build a page pack, right?