S3 Upload "Error uploading to S3. Make sure bucket name is correct and CORs is configured for the bucket"

duane · July 19, 2022, 1:30pm

I’m working on configuring S3 for file storage and have everything working except file upload.

Does someone else have S3 working for file upload that might point me in the right direction?

I receive the error: “Error uploading to S3. Make sure bucket name is correct and CORs is configured for the bucket”

I’m starting off using Skuid’s Access Key authentication method (Because I can’t figure out how to get the AWS: Assume Role with ARN working. Skuid requires “skuid-assumable-role/” in the ARN. I can’t get Amazon to accept the / and I can’t get Skuid to accept it without the /).

The IAM Access Keys have the AmazonS3FullAccess policy. I’m trying to get it working with the least restrictive security and then will start to lock it down after it is working.

The Access Key security model works for everything, including upload, using an app like S3 Browser, but when I use the upload component in Skuid, I get the error above. I’ve researched configure CORs on the bucket, but I can’t find any examples of what settings Skuid is looking for.

Any guidance for me?

Elena_ONeil · July 19, 2022, 9:28pm

Hi @duane,

This article has a lot of helpful instructions for how to set up an S3 data source connection: Simple Storage Service (S3) — Skuid v13.0.6 Documentation

I also grabbed a few screenshots of the model and file upload setup for an S3 data source if that’s helpful to compare to your page.

Model and condition setup:

File upload setup:

Cody_Taylor · July 19, 2022, 9:40pm

I think it’s also worth checking the Network tab in your browser console to see what sort of responses AWS is sending back. It’s been a minute since I’ve tried any AWS data sources, but the error code you see on the network request could shed more light on the issue.

Here’s a link to AWS’s error codes for S3.

Alternatively, you could try the assume role authentication method again. I need to update the docs to include more detailed information on this, but a summary is:

The IAM role’s path must be skuid-assumable-role
However (and this is where the docs need to be updated) this can only be set through the AWS API or AWS CLI.

“Many of the following examples include paths in the resource part of the ARN. Paths cannot be created or manipulated in the AWS Management Console. To use paths, you must work with the resource by using the AWS API, the AWS CLI, or the Tools for Windows PowerShell.” — AWS docs

If you’re willing to give that auth method another shot, try creating the role using the AWS CLI with the necessary trust relationship policy (which the docs should be up-to-date on—link to the proper section here)

duane · July 19, 2022, 10:11pm

@Elena_ONeil - Thank you. I am using the documentation and my setup matched your screenshots. At least I’m on the right path.

@Cody_Taylor - Thank you. It looks like I will need to seek a higher power since I can’t build the role using the AWS Console. I’m using the low-code environment of Skuid as it matches my skillset until we get into requirements like this. Thanks for pointing me in the right direction to find some help.

duane · July 21, 2022, 1:18am

The CLI does not allow a “/“ in the name either. Do I need to make the Skuid role without the / and then make a second role that assumes the Skuid role?

~Duane

Cody_Taylor · July 21, 2022, 4:58pm

Hey Duane! Sorry, I (and the docs) should have been more clear—the forward slash / goes on the path instead of the name.

I’ll be adjusting these instructions a bit for the official docs, but the process should look something like this:

Create the role through the CLI with a command like this:

aws iam create-role --role-name Skuid-S3-Access --path skuid-assumable-role/ --assume-role-policy-document {}

(The path flag is where we’re setting that required assumable role string. And note the empty trust relationship policy, only here since it’s required for creation—you’ll update that part of the role after creating the Skuid authentication provider)

Create the Skuid authentication provider with the ARN of your newly created role, steps for this in the docs.
After creation, you’ll see the trust policy you need. Copy this policy and update the IAM role to use it through either the console or the CLI.

duane · July 21, 2022, 8:06pm

Thank you, @Cody_Taylor. With your guidance, I have made it further. I apologize for not having the technical chops for this.

I now have S3 listing buckets and objects using both the Assume Role and Access Key authentication providers.

But I get the same upload error using both methods.
s3-error

This is what my policy looks like. The action names are a little different than the documentation and I can’t find an action close to “listObjects” as shown in the documentation:
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Sid”: “VisualEditor0”,
“Effect”: “Allow”,
“Action”: [
“s3:PutObject”,
“s3:GetObject”,
“s3:ListBucket”,
“s3:DeleteObject”
],
“Resource”: [
“arn:aws:s3:::frf-skuid/",
“arn:aws:s3:::frf-skuid”
]
},
{
“Sid”: “VisualEditor1”,
“Effect”: “Allow”,
“Action”: “s3:ListAllMyBuckets”,
“Resource”: "”
}
]
}

Do I need to do anything to set up CORs?

FYI: the path required a “/” in front as well (i.e. --path /skuid-assumable-role/)

duane · July 24, 2022, 3:27pm

My journey to get S3 to work as a data source

In case the results from my struggles getting S3 to work can help others, here is what I have found so far to get S3 to work.

As a prerequisite, you must install and configure the Amazon Command Line Interface (CLI) first. If you do a web search, you can find documentation and youtube videos to help you.

Login to Skuid
Go to Data Sources->Authentication Providers
Click Create
Choose AWS: Assume Role with ARN
Copy/Paste the Trust Relationship Policy to a text file (e.g., assume-policy-document.json) and save
** NOTE ** – The Skuid docs Amazon Web Services — Skuid v13.0.6 Documentation indicate you must first create role in AWS and then create the authentication provider but to create the role in AWS you must apply a trust policy. Skuid exposes the trust policy without having to first create & save the auth provider so there is no reason to have to create the role first.
Execute the following command in CLI (assumes you have configured it)
aws iam create-role --role-name Skuid-S3-Access --path /skuid-assumable-role/ --assume-role-policy-document file://role-trust-policy.json
** NOTE ** – Path, not name as the Skuid docs Amazon Web Services — Skuid v13.0.6 Documentation mention, must be used to ensure skuid-assumable-role is contained in the ARN. The AWS docs create-role — AWS CLI 1.25.36 Command Reference (amazon.com) state the path must begin & end with a “/”. Additionally, the CLI must be used – AWS console does not provide a way to specify path.
Create a text file role-policy.json

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action": [
            "s3:ListBucket",
            "s3:GetObject",
            "s3:PutObject",
            "s3:DeleteObject"
        ],
        "Resource": [
         "arn:aws:s3::: <bucket-name>",
         "arn:aws:s3::: <bucket-name>/*"
        ]
      },
      {
         "Effect":"Allow",
         "Action": [
            "s3:ListAllMyBuckets"
        ],
        "Resource": "*"
      }      
   ]
}

Execute the following command in CLI
aws iam put-role-policy --role-name Skuid-S3-Access --policy-name Skuid-S3-Access-Policy --policy-document file://role-policy.json
** NOTE ** The Skuid docs Simple Storage Service (S3) — Skuid v13.0.6 Documentation mention listBuckets & listObjects permissions but these are not valid actions
Create a text file cors-configuration.json

{
    "CORSRules": [
        {
            "AllowedHeaders": [
                "*"
            ],
            "AllowedMethods": [
                "PUT",
                "POST",
                "DELETE",
                "GET",
                "HEAD"
            ],
            "AllowedOrigins": [
                "https://<siteroot>.skuidsite.com"
            ],
            "ExposeHeaders": [
                "Access-Control-Allow-Origin"
            ]
        }
    ]
}

Execute the following command in CLI
aws s3api put-bucket-cors --bucket <bucket-name> --cors-configuration file://cors-config.json
** NOTE ** The docs mention nothing about CORS. Additionally, certain file types (e.g., txt) work without specifying “AllowHeaders” while other file types (e.g., xlsx) do not work unless specified. From reviewing the calls made by Skuid, the only difference between a txt file and an xlsx file is the inclusion of the header “Access-Control-Request-Headers” on the pre-flight check but specifying only that in AllowHeaders does not seem to resolve the CORS issue. Specific documentation on what a proper CORS configuration needs to be to ensure all files types work successfully should be added to the Skuid docs and from all supported browsers
Add the auth provider to Skuid
Add the data source to Skuid
** NOTE ** - The docs mention nothing about this but it is unclear if “Proxy” should be enabled or disabled. I have it working with Proxy disabled but the proper setting should be identified in the Skuid docs and/or information on when to enable and when to disable.
Build the page, etc. – An example page that supports listing all buckets and listing bucket objects along with a working file upload, view, download, delete and create folder would be a valuable addition to the Skuid docs.

Cody_Taylor · July 25, 2022, 9:07pm

Duane, your debugging steps and documentation improvements here are immensely valuable.

I had a ticket for fixing the assume-role flow in docs, but your work investigating the CORS issue (while it shouldn’t be your work to do) has been very illuminating for another area where we need more information. Thank you.

I know it’s frustrating to do something that requires this much hoop jumping, especially when the instructions aren’t clear or complete. I apologize for that. I’ll have updates for the S3 topic soon, and I’ll post here when they’re published.

It’s a side note at this point, but we do have docs on the proxy if it’d be helpful for your implementation: Authentication Providers and Data Sources — Skuid v13.0.6 Documentation

Thank you for your continued work on this. Alongside this specific improvement story, we’ll be reviewing ways to keep this conundrum from happening in the future.

duane · July 25, 2022, 9:34pm

Thank you @Cody_Taylor! I looked at the proxy docs. It might be helpful for each data source that if you know the service accepts a proxy, say it is recommended. If you know the data source does not accept a a proxy, inform us not to set the proxy. If you don’t know, provide a link to the docs which sounds like try proxy first, if it works, sweet life, if not, try direct connect.

Cody_Taylor · July 26, 2022, 7:02pm

That’s a good idea. I’ll make a ticket to address this for existing docs, as well as making it part of the process for future data source docs.

Cody_Taylor · August 26, 2022, 9:50pm

Hey Duane. We’ve got a lot of updates published to our S3 docs (and some adjustments to other AWS docs) based on your feedback. Since the effort was inspired by your work in this thread, I wanted to post an update now that the docs (hopefully) account for all setup issues.

If you spot anything else, don’t hesitate to send me an email at documentation@skuid.com.
And again, appreciate your work here.

https://docs.skuid.com/nlx/v2/en/data/aws/s3.html