Update 9pm 5/23: Problem with Salesforce Summer 16 patch 5 released on 5/17.

  • 5
  • Announcement
  • Updated 2 years ago
  • (Edited)
Salesforce released a patch to the Summer 16 release last night - and installed it automatically in all sandbox instances that are on Summer 16.  The patch breaks customer's ability to use custom components and custom themes.  

The patch was supposed to include a "fix" to this Known Issue: https://success.salesforce.com/issues_view?id=a1p300000001ACTAA2 but we believe they have made the issue worse instead. 

Some Key notes: 

1. This issue is not related to Rockaway.  It impacts earlier releases including Banzai.  


2. This issue only affects selected sandbox instances (that are on Summer 16).  We have no reports of this impacting any production. 

We will keep you up to date as we know more about this situation.
Photo of Rob Hatch

Rob Hatch, Official Rep

  • 44,006 Points 20k badge 2x thumb

Posted 2 years ago

  • 5
Photo of Henry Goddard

Henry Goddard

  • 776 Points 500 badge 2x thumb
Ah great news! Have been trying to get to the bottom of this all morning. Seems that it's similar to this update https://community.skuid.com/skuid/topics/alert-upcoming-salesforce-critical-update-will-block-usage-...
Photo of Henry Goddard

Henry Goddard

  • 776 Points 500 badge 2x thumb
Also it would seem the fonts on our custom theme aren't loading.
Photo of Rob Hatch

Rob Hatch, Official Rep

  • 44,006 Points 20k badge 2x thumb
Here is an update on the progress made today. We are running a two prong strategy to get this issue addressed. 

1. Since this was an issue introduced by Salesforce, we are asking them to reverse the changes made last night, or address them more favorably. There is a case currently open  on the matter. If you could escalate this case through your support channels it will give Salesforce true visibility to the issue. The case # is 13869100 

2. But we are not just waiting for Salesforce to act here. We have made substantial progress addressing areas within skuid that are impacted by this change. There are still a number of areas where work is in progress. We hope to have a patch update available for Banzai and Rockaway orgs tomorrow. 

Either way we will be communicating with you regularly until this issue is corrected.  

Thanks again for your partnership. 
Photo of Zach McElrath

Zach McElrath, Employee

  • 49,004 Points 20k badge 2x thumb
Update on the root cause of the issue and approaches to consider for temporary workarounds:

The change made by Salesforce made it impossible to use relative URLs to access StaticResource files residing in the local namespace from managed-package namespace subdomains of the .visual.force.com domain. However, such StaticResources can still be accessed via absolute URLs.

Example:
- You have a StaticResource storing some JavaScript defined locally in your org, called "MyCommonJS".
- You click "Preview" on a Skuid Page you have which requests this JS file via a JS Resource load of type "Static Resource"

- In orgs with this Salesforce patch applied, when previewing this Skuid Page, your JS file will be loaded into the browser normally. Also, you can enter the following relative URL into your browser: /resource/MyCommonJS, and you will be redirected to the contents of your Static Resource file.
- In orgs WITHOUT this patch, when you preview the Skuid Page, your JS file no longer gets loaded into the browser. If you enter the relative URL "/resource/MyCommonJS" into the browser, no file is returned. 

However, absolute URLs in the local Visualforce namespace subdomain can be used to access this Static Resource. So for instance if your Sandbox is on cs4, you could enter the following into your browser and you would be served the content of the Static Resource as expected:

https://c.cs4.visual.force.com/resource/MyCommonJS

Therefore, absolute URLs can be used as a workaround. If Static Resource files are failing to load in your Skuid Pages, one approach to getting around this issue while we work on getting out our own patch is to replace JS/CSS Resource loads of type "Static Resource" in your Skuid Pages / Component Packs with "External" resource loads, which allows you to enter absolute URLs. Obviously this is not a viable long-term solution because absolute URLs cannot be migrated between different environments because they will break, e.g. if you try the above URL in your Production NA12 org it will fail to load any resource, whereas a relative URL (normally) would. However, we present this as an option in order to provide a short-term workaround.

(NOTE: this will NOT work for previewing / composing pages within the Lightning Composer. Use of a Visualforce absolute URL will not work within the Lightning runtime.)

Workaround for JS/CSS Resource loads in Skuid Pages

Approach: Switch the type of all "Static Resource" loads to "External".

Example: your Resource Name is "MyCommonJS" and you have no Resource Namespace (e.g. it is a local resource). Resource URL should be set to your local Visualforce subdomain + /resource/MyCommonJS, e.g. "https://mydomain--test--c.cs9.visual.force.com/resource/MyCommonJS"; or "https://c.cs9.visual.force.com/resource/MyCommonJS";

Workaround for Component Pack manifest files

Approach: To fix JS/CSS resource loads requested by Component Pack manifest files, you could replace all "local" referenceType JS/CSS dependency loads with "external" (see this tutorial under  "Resource Reference Definition Properties" for more details: http://help.skuid.com/m/11720/l/288289-component-packs-and-custom-components)

Example: in one of your Manifest files you currently have something like:

{ "resource": "AcmeComponents", "path": "runtime.js" }

you could replace this with an "external" resource load containing an appropriate absolute url, e.g.

{ "referenceType" : "external", "path": "https://c.cs9.visual.force.com/resource/AcmeComponents/runtime.js" }
Photo of Henry Goddard

Henry Goddard

  • 776 Points 500 badge 2x thumb

Hi Zach, thanks for addressing this.

With regards to your second fix, I'm unfortunately not familiar with a manifest file that makes reference to a 'resource'. The progressbarcomponent you've give as an example has a runtime JSON like this:

{
   "id": "progressindicator",
   "components": [
      {
         "id": "progress_indicator"
      }
   ],
   "js": [
      {
         "path": "/progressIndicatorRuntime.js"      }
   ],
   "css": [
      {
         "path": "/progressIndicator.css"      }
   ]
}

Are you suggesting to have it load absolutely it should look like this?

{
   "id": "progressindicator",
   "components": [
      {
         "id": "progress_indicator"
      }
   ],
   "js": [
      {
         "referenceType" : "external",
         "path": "https://c.cs9.visual.force.com/resource/progressindicatorpack/progressIndicatorRuntime.js";      }
   ],
   "css": [
      {
         "referenceType" : "external",
         "path": "https://c.cs9.visual.force.com/resource/progressindicatorpack/progressIndicator.css";   }
   ]
}

EDIT: Pretty JSON Format
(Edited)
Photo of Zach McElrath

Zach McElrath, Employee

  • 49,004 Points 20k badge 2x thumb
Henry, what you have done is correct.

 "referenceType" : "resource" is the assumed default for a js/css manifest file resource request, and if no "resource" property is specified the "path" will be assumed to be relative to the Component Pack's Static Resource, so yes for the Progress Indicator Component Pack you have made the correct changes.
Photo of Henry Goddard

Henry Goddard

  • 776 Points 500 badge 2x thumb
Awesome, thanks for the help.
Photo of TJ

TJ

  • 952 Points 500 badge 2x thumb

Waiting for the latest update on the Skuid patch availability. Please keep us posted.

Photo of Rob Hatch

Rob Hatch, Official Rep

  • 44,006 Points 20k badge 2x thumb
Update 5:30 pm ET  5/19.  Fix Package Available !!! 

Skuid has released a set of packages to addresses the issue introduced with the Salesforce Summer 15 patch 5 update.  These are: 
- Banzai - Update 9 Patch 1
- Rockaway - Point Release 3

This release is available at https://www.skuid.com/skuidreleases



Below are some more details about the issue,  our resolution,  and answers to some common questions we've gotten over the last 2 days. 


Issue Summary

On Tuesday May 17 Salesforce implemented patch 5.0 to all Summer 16 Sandbox instances.  

Here is their release note documentation:   https://success.salesforce.com/issues_releases_view?release=202005000

The release notes indicate that a fix was implemented to this issue: “ Static resources not loading when the critical update for "Serve Static Resources from the Visualforce Domain" is enabled"  (https://success.salesforce.com/issues_view?id=a1p300000001ACTAA2)   We have been following this issue for a long time because it had affected our customers.  However the implementation did not address the defect and introduced additional issues.

Technical Details:

The release removed the ability to use relative URLs to access StaticResource files residing in the local namespace from managed-package namespace subdomains of the .visual.force.com domain.   

This means that any static resources that you as a customer have uploaded  to support your Skuid implementation cannot be accessed from within our managed-package namespace.  

- You might directly load a resource called  “TestJSResource" in your org (which means it’s in the local namespace),

- A relative link to that resource (defined in skuid as   /resource/TestJSResource)  will get appended to the skuid subdomain.  It will look like:  

https://mydomain--skuid.cs4.visual.force.com/resource/TestJSResource

This link will return a 404 error.  An absolute link that does not include the skuid subdomain will continue to work:

https://mydomain--c.cs4.visual.force.com/resource/TestJSResource

Impact:  The following items will not work in your Summer 16 sandbox until this is fixed.
- Javascript or CSS files loaded from static resources
- Custom components
- Custom themes and font files
- Images being sourced from static resources

Extent:   This patch update has only been added to Summer 16 sandboxes.  It affects no production instances yet.   You can see the instances where this patch has been applied on the release notes page (first link above)


This issue is not related to our recent Rockaway release.  It impacts all versions of Skuid.

Are other vendors impacted by this?  We assume the answer is yes,  but we have no specific information about other vendors.  Skuid is unique in the level of customization that is allowed on top of a managed package.  Most packages do not allow users to add their own static resources that are used by the package.  

What have we done about the issue.  

1. Since this was an issue introduced by the Salesforce release, we are continuing to follow protocol to get it addressed leveraging our partner relationship.  There is a case opened which is being escalated by our customers who are impacted by the issue.  Thank you for your assistance in this escalation.  The case number is 13869100. We will continue to work with Salesforce on this case. 

2. Skuid also has taken action directly.  We have released a set of packages as of 5:30 pm on 5/19/2016.  These are available for Banzai and for Rockaway.  You can get these packages at  https://www.skuid.com/skuidreleases

Why did Skuid not know about this:  Early in a release cycle, Salesforce provides preview sandboxes that we use to ensure Skuid compatibility.  Now that the Summer 16 release is well along and has been more widely distributed we no longer had that luxury.  Salesforce pushed the patch into all Summer 16 sandbox orgs.  Release timelines do not allow for preview staging for every patch.  Sandboxes are what this is designed for.   We do however recognize that the activities occurring in sandboxes are business critical (like training and development), which is why we are acting with great urgency to address this.

We certainly have a strong relationship with Salesforce.  At every level of the company we have gotten assurances that they will respect Skuid and not break our integrations.  This is the first time in our history where a change made by Salesforce has had this sort of widespread customer impact.  We will certainly be debriefing this issue with connections in Management at Salesforce, and exploring ways to better prevent it in the future.
(Edited)
Photo of Henry Goddard

Henry Goddard

  • 776 Points 500 badge 2x thumb
Thanks for the quick turnaround guys. I can confirm that we're back to normal on our sandboxes.
Photo of Matt Sones

Matt Sones, Champion

  • 31,478 Points 20k badge 2x thumb
Photo of Rob Hatch

Rob Hatch, Official Rep

  • 44,006 Points 20k badge 2x thumb
Yes Matt.  We believe it is related. 
Photo of Rob Hatch

Rob Hatch, Official Rep

  • 44,006 Points 20k badge 2x thumb
Update End of Day 5/20/2016

Today many customers implemented the patch that we distributed last night into their Summer 16 sandboxes.  Most issues with resources not loading were resolved.   However,  today we have continued to uncover situations where the changes made by Salesforce impact the customizations created by our customers.  We really appreciate customers who have quickly notified us about these remaining issues.   The following items have been reported (among others) as failing

  • Custom CSS included in Theme files 
  • Custom Fonts included in Themes
  • Custom components included in Page Includes
  • Formula fields accessing static resources 





We continue to analyze these issues and explore how they can be fixed.   We hope to release a second patch early next week. 

We also continue to work with Salesforce support on this issue and have been assured that it is widely escalated and has affected a significant number of ISV partners.  You are not alone... 

I’m sorry today’s update is not as encouraging as yesterday’s.  But we continue to work the situation. 

Have a great weekend. 
(Edited)
Photo of Rob Hatch

Rob Hatch, Official Rep

  • 44,006 Points 20k badge 2x thumb
Another update at the “very end" of the day.  Salesforce will be reversing the issues.

We just received word from Salesforce that they will be pushing out a new e-release to all sandboxes this weekend that will reverse the issues created by Summer 16 release patch 5.  The specific timing has not been shared beyond the roll out occurring sometime this weekend. 

Please note that there will be one side effect - The first user that logs into a Skuid page accessing resources (custom components, themes, java script, etc) in the local namespace will receive a 404 error. After that page is refreshed, normal operation can be expected.

As mentioned previously, the impact of this release has been widespread felt across many customers and ISVs.  Salesforce has indicated that while this will address the issues, it is a short term fix and their R&D team will continue to prioritize work on a longer term solution.

Thanks for your continued patience as we work with Salesforce to get this resolved.  We sincerely value your partnership.
(Edited)
Photo of Rob Hatch

Rob Hatch, Official Rep

  • 44,006 Points 20k badge 2x thumb
Update Monday morning 5/23 at 10:00 am. 
Over the weekend, Salesforce completed a rollback of the Summer 16 patch 5 update.  We have confirmed the rollback in our Summer 16 sandboxes.  

However,  if you did implement the Patch we provided on Thursday there will be several items that are still not correct.  We started sourcing static resources with absolute URL’s rather than relative ones.  We feel this is the better way and will continue do do so where we can.  But we continue to see problems with: 

-  Web Fonts

-  Custom CSS included as part of Themes

-  The file picker that allows you to navigate through the ZIP file in a static resource and choose resources inside it. 

We plan on releasing another patch early this week to address these issues. 



Thanks again for your patience. 
Photo of Rob Hatch

Rob Hatch, Official Rep

  • 44,006 Points 20k badge 2x thumb
A quick update at the end of our day.. 

This evening Skuid released a set of patches that revert changes made last week to correct an issue introduced by Salesforce Summer 16 Update 5.

Salesforce has fixed the issue, so the changes to Skuid are no longer needed.

You can get those patches at skuid.com/skuidreleases   Release notes there outline the specifics of the changes



Have a great evening.