← SPICE Internals and Data at Scale

▶ Play from here · 151 min

Starts this lesson and continues through 19 more to the end of the course.

The quotas that stop you

Why this lesson is the one to memorise

When you tell AWS Support "we can't get our data in", the first thing a support engineer does is work out which of about six ceilings you've hit. If you can name it yourself, the case skips straight to the useful part. If you can't, you spend a round trip or three establishing it.

More importantly: some of these quotas are adjustable and some are not. That single fact decides whether the correct response is "open a limit-increase case" or "go redesign the dataset". Getting it wrong costs weeks, and it is the single most common way teams waste time on this service.

There are two authoritative sources and you should treat them as such:

• Data source quotas — the SPICE and direct-query shape limits
• Amazon Quick Sight endpoints and quotas in the AWS General Reference — the Service Quotas table, which is the only place that states adjustable: yes/no

Blog posts are not a source for a quota. Neither is this page, six months from now.

Per-dataset SPICE quotas

Verbatim from the data source quotas page (verified 2026-08-09):

Two details in that list earn their own paragraphs.

The row limit and the size limit are an OR, not an AND. AWS says so directly:

"In rare cases, if you're ingesting large rows into SPICE, you might reach the quota for gigabytes per dataset before you reach the quota on rows. The size is based on the SPICE capacity the data occupies after ingestion into SPICE."

So a 900-million-row dataset with wide text columns can fail on 2 TB while sitting at 45% of the row limit. Whichever ceiling you touch first is the one that stops you. And note "the SPICE capacity the data occupies after ingestion" — it's logical size from lesson 1, not source size.

The field-length quota has two values and which one applies depends on how the dataset was built. 2,047 characters normally; 65,534 "if you use the new data preparation experience to create your SPICE dataset". That's a thirty-two-fold difference hanging on which UI created the dataset. If you are truncating long text — JSON blobs, log lines, free-text notes — this is the first thing to check.

⚠️ Row-level security does not buy you a smaller footprint. "All quotas apply to SPICE datasets with row-level security, as well." An RLS dataset is stored whole, and filtered at query time. If ten tenants share one 500 GB dataset, that is 500 GB, not ten smaller slices.

The two ceilings people confuse

This is the distinction the whole module is built around.

If a single dataset needs more than 2 billion rows or 2 TB, no support case will help you. The answer is architectural, and lesson 5 covers the options.

The quota that surprises everyone: ingestion calls

From the Service Quotas table in the AWS General Reference (verified 2026-08-09):

The description is precise about the window: "The maximum number of calls to the createIngestion API function in a floating 24-hour window. The time period is measured starting 24 hours before the current date and time."

Floating, not calendar. There is no midnight reset to wait for. If you burn 32 calls between 09:00 and 10:00, you get one back at 09:00 tomorrow, then another, and so on.

The CreateIngestion API page states the same limit in different words:

"You can manually refresh datasets in an Enterprise edition account 32 times in a 24-hour period. You can manually refresh datasets in a Standard edition account 8 times in a 24-hour period." — CreateIngestion

⚠️ A live documentation ambiguity — and it matters. The API page says "manually refresh". The Service Quotas table says "calls to the createIngestion API function". Neither page states whether a scheduled refresh consumes the quota.

Do not guess, and do not accept a blog post's answer. The API gives you RequestSource with values MANUAL and SCHEDULED on every ingestion — lesson 3 shows you how to list your own ingestions and settle it empirically for your account. If it turns out to matter for a design you're committing to, that is a legitimate, well-formed question to put to AWS Support, and one they can answer in a sentence.

What this quota means in practice

You are designing a near-real-time dashboard. You want a 15-minute refresh. That is 96 ingestions a day, and the quota is 32 — three times over.

Your options, in the order you should consider them:

1. Scheduled incremental refresh. The console supports 15-minute and 30-minute incremental schedules on Enterprise (lesson 4). Whether these consume the CreateIngestion quota is exactly the ambiguity above — measure it.
2. Direct query. If the requirement is genuinely "current to the minute", SPICE is the wrong engine. Lesson 5.
3. Split by dataset. The quota is on calls, per Region. Four datasets refreshed hourly is 96 calls; one dataset refreshed hourly is 24. Splitting a dataset multiplies your consumption — it does not divide it. This is the trap in the option, not the escape from it.
4. Reduce the cadence and be honest with stakeholders about the freshness they're actually getting.

Because it is not adjustable, "we'll ask AWS to raise it" is not on the list.

Direct query quotas

Different engine, different ceilings (Data source quotas):

⚠️ The 2-minute timeout does not always stop the query. This is one of the most operationally important sentences in the entire user guide:

"not all database drivers react to the 2-minute timeout, for example Amazon Redshift. In these cases, the query runs for as long as it takes for the response to return, which can result in long-running queries on your database."

Read that again in terms of what your DBA sees. A user opens a dashboard, the visual times out after two minutes and shows an error — and the query keeps running on Redshift, consuming a WLM slot, until it finishes on its own. Twenty users refreshing a slow dashboard can pin a cluster with queries that nobody is waiting for any more.

AWS's remedy is to cancel from the database side: for Redshift, see Canceling a query and workload management.

Design rule: if you run direct query against Redshift, put Quick Sight in its own WLM queue with a statement timeout. Do not rely on the 2-minute client-side timeout to protect the cluster, because for Redshift it doesn't.

Quotas from the Service Quotas table worth knowing

All values below are per Region, verbatim from the AWS General Reference (verified 2026-08-09). Every one of them is marked not adjustable.

The fields per dataset row carries a note worth quoting, because it describes a workaround rather than a hard stop:

"File imports and query result sets can contain more than 2,000 columns. However, you must edit the dataset settings and manually exclude fields until there are less than 2,000 selected or included."

And the email one is a genuine silent failure: "If you try to send reports to larger groups, the report fails." Not truncates — fails.

Quick Suite era quotas

The same table now carries quotas for the newer Quick Suite capabilities. A few, for orientation (covered properly in Q9):

Note the pattern: the Quick Automate count quotas are adjustable; the runtime quotas are not. That's a useful heuristic across the whole table — AWS will sell you more objects far more readily than it will change engine behaviour.

How to check a quota for real

Never from memory. Never from here.

# Is there a Service Quotas entry, and what is my applied value?
aws service-quotas list-service-quotas \
  --service-code quicksight \
  --region us-east-1 \
  --query 'Quotas[].{Name:QuotaName,Value:Value,Adjustable:Adjustable}' \
  --output table

# Applied value (accounts for any increase already granted)
aws service-quotas list-aws-default-service-quotas \
  --service-code quicksight --region us-east-1 --output table

⚠️ The service code is quicksight, not quick. The rename did not touch API or Service Quotas namespaces. The console URL is still console.aws.amazon.com/servicequotas/home/services/quicksight/quotas.

Note also that not every documented limit appears in Service Quotas. The per-dataset row and size quotas (2 billion / 2 TB) are documented in the user guide but are not entries in the quota table — which is consistent with them being non-adjustable. If it's not in the table, you cannot request an increase for it.

Check yourself

1. An Enterprise dataset fails at 800 million rows. Which quota did you hit, and is a support case the right move?
2. You need a 15-minute SPICE refresh. Why is "split into four datasets" the wrong instinct?
3. A dashboard visual errors after two minutes against Redshift. What is still happening on the cluster, and what protects you?
4. Long JSON strings are truncating in a SPICE dataset. What's the first thing to check?
5. Ten tenants share one RLS dataset. How much SPICE capacity does the tenth tenant's data consume?

Answers

1. Not the row quota — 800 million is under 2 billion. You hit the 2 TB size quota, because the two are an OR and wide rows reach the size ceiling first. Neither is adjustable, so a support case won't help; redesign the dataset.
2. Because the quota counts CreateIngestion calls per Region, not per dataset. Four datasets each refreshed hourly is 96 calls against a 32-call quota. Splitting multiplies consumption.
3. The query keeps running on Redshift — the Redshift driver doesn't react to the client-side 2-minute timeout. Only a database-side control protects you: a dedicated WLM queue with a statement timeout, or manual cancellation.
4. Which data preparation experience created the dataset. The field-length quota is 2,047 Unicode characters normally, but 65,534 with the new data preparation experience.
5. The same as everyone else's — the whole dataset is stored once and filtered at query time. All quotas apply to RLS datasets as well; RLS is not a storage optimisation.

Teaching this section

• Start with the adjustable/not-adjustable column, not the numbers. The numbers are lookup-able. The reflex "is this a purchase order or a redesign?" is the actual skill, and it's what separates someone who resolves this in a day from someone who resolves it in a fortnight.
• Demo: run the list-service-quotas command live and let them see that 2 billion rows is not in the table. The absence is the lesson.
• Good discussion: "We need 15-minute freshness on 400 million rows." Let the room design it and watch them walk into the 32-call quota unaided. It lands far harder than being told.
• Watch for: treating the row limit and size limit as a single limit. Ask explicitly: "which one do you hit first, and how would you know?"
• Timing: 25 min presented, 15 discussion. This section always overruns; let it.