← Data Sources and Connectivity

Starts this lesson and continues through 24 more to the end of the course.

Creating data sources through the API

A data source is coordinates plus credentials

A data source in Quick Sight is not data. It's a small named object holding three things: where the source is (DataSourceParameters), how to authenticate (Credentials), and how to get there (VpcConnectionProperties, SslProperties). Datasets — the things SPICE actually fills — are built on data sources. Get the separation straight and half the permission model of Q3 becomes obvious.

The call

From CreateDataSource (verified 2026-08-16):

POST /accounts/{AwsAccountId}/data-sources HTTP/1.1

Three fields are required: DataSourceId (unique per account per Region), Name (display, 1–128 chars), and Type (the 38-value enum from lesson 1). Everything else is optional — which is how you get the classic broken state: a data source that exists but can't connect, because nothing forced you to supply parameters or credentials at creation time.

The optional fields, with their documented constraints:

Response: Arn, DataSourceId, RequestId, and CreationStatus — one of CREATION_IN_PROGRESS | CREATION_SUCCESSFUL | CREATION_FAILED | UPDATE_IN_PROGRESS | UPDATE_SUCCESSFUL | UPDATE_FAILED | DELETED.

⚠️ CreateDataSource returning 200 does not mean the connection works. You get CREATION_IN_PROGRESS back; the connection test happens asynchronously. Poll DescribeDataSource until the status resolves, and if it's CREATION_FAILED, read ErrorInfo (lesson 5). Pipelines that fire-and-forget this call discover the failure a day later, as a refresh error on some downstream dataset.

The parameters catalog

DataSourceParameters has one member object per source type. Representative shapes, verbatim from the request syntax (verified 2026-08-16):

"RedshiftParameters":  { "ClusterId", "Database", "Host", "Port",
                         "IAMParameters": { "AutoCreateDatabaseUser", "DatabaseGroups",
                                            "DatabaseUser", "RoleArn" },
                         "IdentityCenterConfiguration": { "EnableIdentityPropagation" } }
"AthenaParameters":    { "WorkGroup", "RoleArn", "ConsumerAccountRoleArn",
                         "IdentityCenterConfiguration": { "EnableIdentityPropagation" } }
"S3Parameters":        { "ManifestFileLocation": { "Bucket", "Key" }, "RoleArn" }
"RdsParameters":       { "InstanceId", "Database" }
"MySqlParameters":     { "Database", "Host", "Port" }

Read those field names closely — they encode the authorization models of lesson 3:

• RdsParameters takes an InstanceId, not a host — Quick Sight can find RDS instances natively.
• RedshiftParameters accepts ClusterId or Host/Port, plus IAMParameters for connecting via IAM instead of a database password.
• AthenaParameters.RoleArn and S3Parameters.RoleArn let a specific data source assume a specific role, instead of riding the account-wide service role. ConsumerAccountRoleArn exists for cross-account Athena. (Field names verified on the API page; the full semantics live on the AthenaParameters and S3Parameters type pages — cite those before you architect around them.)

For S3, the ManifestFileLocation points at a JSON manifest. From Supported formats for Amazon S3 manifest files (verified 2026-08-16): a Quick Sight-format manifest must have a .json extension; a Redshift-format manifest (also accepted, with its mandatory flag honored) can have any extension. All files in one manifest must share format, column count, and column types. URIs lists exact files; URIPrefixes pulls whole folders recursively. For JSON files, set globalUploadSettings.format but not delimiter/textqualifier/containsHeader.

Credentials — three shapes, one winner

The Credentials object offers, per the request syntax: a CredentialPair (username/password, with optional AlternateDataSourceParameters), a CopySourceArn (borrow credentials from an existing data source), or a SecretArn.

Prefer the SecretArn. From Using AWS Secrets Manager secrets instead of database credentials (verified 2026-08-16):

aws quicksight create-data-source \
  --aws-account-id 111122223333 \
  --data-source-id sales-mysql \
  --name "Sales MySQL" \
  --type MYSQL \
  --data-source-parameters '{"MySQLParameters":{"Database":"sales","Host":"db.example.internal","Port":3306}}' \
  --credentials '{"SecretArn":"arn:aws:secretsmanager:us-east-1:111122223333:secret:sales-db"}' \
  --region us-east-1

The rules that matter, all from that page:

• The secret must be JSON with username and password keys; both are required, all other keys are ignored.
• An admin must first grant Quick Sight access to the specific secrets (Manage Quick → Security & permissions). That creates the IAM role aws-quicksight-secretsmanager-role-v0.
• Supported for "data source types that support credential pair authentication. Jira and ServiceNow are not currently supported."
• Customer-managed KMS key on the secret? The role needs kms:Decrypt. AWS-managed key? No extra setup.
• At create/update time, secretsmanager:GetSecretValue is authorized against the API caller's IAM policy, not the service role. The service role is what's assumed later, when viewers open dashboards. Two different identities need access, at two different times — this asymmetry is behind most "works when I create it, fails when they view it" mysteries.

⚠️ The UI eats secrets. Verbatim: "Secrets are automatically removed from a data source when the data source is altered in the UI." Someone edits the host or port in the console, the SecretArn silently drops off, and the next connection fails auth. The restore is update-data-source through the API. If your data sources are API-managed, say so in the console name (sales-mysql [IaC — do not edit in console]) — it's crude and it works.

Errors worth memorizing

From the API page (verified 2026-08-16): AccessDeniedException 401, InvalidParameterValueException 400, CustomerManagedKeyUnavailableException 400, ResourceExistsException 409, LimitExceededException 409, ConflictException 409, ResourceNotFoundException 404, ThrottlingException 429, InternalFailureException 500.

Same pattern you learned on CreateIngestion in Q2: LimitExceededException is a 409, not a 429 — don't blind-retry it. And CustomerManagedKeyUnavailableException is this API's tell that your account's registered CMK is the problem, not your parameters.

Check yourself

1. create-data-source returned 200. What do you actually know, and what's your next call?
2. Why is CopySourceArn a maintenance hazard compared to SecretArn?
3. A colleague edits an API-created MySQL data source's port in the console. What silently broke?
4. Which two SaaS sources can't use Secrets Manager credentials?
5. Your manifest sales_manifest.txt in Quick Sight format is rejected. Why?
6. Who needs secretsmanager:GetSecretValue — the creator, the service role, or both, and when?

Teaching this section

• Open with the async trap. Ask who has a pipeline that checks CreationStatus after creating a data source. The honest answer is usually nobody, and it frames the lesson.
• Demo: create a data source with deliberately wrong credentials — show the 200, then the CREATION_FAILED in describe-data-source. Cheap, vivid, reusable in lesson 5.
• Good discussion: "Console-editable or IaC-only — pick a policy for data sources and defend it." The secret-eating behavior is the forcing argument.
• Watch for: learners conflating data source permissions (the 64 ResourcePermission entries) with database permissions. Different layer; Q3 and Q5 territory.
• Timing: 30 min presented, 15 for the failed-create demo.