← SPICE Internals and Data at Scale

▶ Play from here · 159 min

Starts this lesson and continues through 20 more to the end of the course.

What SPICE is, and how it is sized

The thing people get wrong first

SPICE size has almost nothing to do with the size of your source data.

A 40 GB Parquet file in S3 might occupy 12 GB in SPICE. Or 300 GB. The difference isn't compression luck — it's your column types, and specifically how many string columns you kept. If you've ever watched an ingestion blow through a capacity estimate that "should have been fine", this is why.

So before anything else: SPICE has a published sizing formula, it is not a rule of thumb, and you should run it on paper before you build a large dataset.

What SPICE actually is

"SPICE (Super-fast, Parallel, In-memory Calculation Engine) is the robust in-memory engine that Amazon Quick Sight uses. It's engineered to rapidly perform advanced calculations and serve data. In Enterprise edition, data stored in SPICE is encrypted at rest." — Importing data into SPICE (verified 2026-08-09)

When you build a dataset you choose SPICE or direct query — except for uploaded files, which are always SPICE. The documented reasons to import are:

• analytical queries process faster
• you don't wait on a direct query to run
• data in SPICE is reused without re-querying the source, so on a per-query-priced source such as Amazon Athena you pay at dataset creation and at each refresh, not at every visual render

That third point is the one that pays for the whole feature. A dashboard on Athena in direct query mode bills a scan for every filter change by every viewer. The same dashboard on SPICE bills once per refresh.

Capacity is per Region, and it is shared

This trips up almost every multi-Region account.

"SPICE capacity is allocated separately per AWS Region. For each AWS account, SPICE capacity is shared by all the people using Quick in a single AWS Region. The other AWS Regions have no SPICE capacity unless you choose to purchase some." — Configure SPICE memory capacity (verified 2026-08-09)

Three consequences worth writing down:

1. Capacity you bought in us-east-1 does nothing for a dataset in eu-west-1.
2. Every author in the account draws from the same pool. One person's exploratory 400 GB dataset can fail everyone else's overnight refresh.
3. Purchasing and releasing capacity applies only to the Region currently selected in the Quick console — which is not the same selector as the AWS Management Console's Region selector.

⚠️ The Region-selector trap. The docs are explicit that these are two independent selectors:

"The AWS Management Console and the Amazon Quick console each have an AWS Region selector that works independently from the other. Changing the selected AWS Region in the AWS console doesn't change the AWS Region in Amazon Quick."

And if you open the product via http://quicksight.aws.amazon.com, AWS's own guidance is to stop:

"Amazon Quick automatically selects your account and AWS Region. You can't view your AWS account from Amazon Quick. We recommend using a different method to open Amazon Quick when you want to work with SPICE capacity."

The Region is visible in the URL — https://us-east-1.quicksight.aws.amazon.com/sn/admin. Read it before you spend money.

The sizing formula

The number that counts against your capacity is the dataset's logical size, and AWS is precise about when it is computed:

"The computation of a dataset's logical size occurs after all the data type transformations and calculated columns are defined during data preparation. … Any changes you make in an analysis have no effect on the logical size of the data in SPICE. Only changes that are saved in the dataset apply to SPICE capacity."

Three storage classes: decimals, dates, strings.

Logical dataset size in bytes =
  (Number of Numeric cells * (12 bytes per cell))
+ (Number of Date cells    * (12 bytes per cell))
+ SUM ((24 bytes + UTF-8 encoded length) per Text cell)

Numerics and dates are 8 bytes plus 4 bytes of auxiliary data. Strings carry a flat 24-byte overhead per cell on top of their UTF-8 length, because of the indexing SPICE builds to keep queries fast.

Geospatial fields follow their underlying physical type:

"Geospatial data types use metadata to interpret the physical data type. Latitude and longitude are numeric. All other geospatial categories are strings."

Work an example

100 million rows. Fifteen columns: 8 numeric, 2 dates, 5 strings averaging 20 UTF-8 bytes.

Now notice what dominates. The five string columns are 65% of the footprint while being a third of the schema. And of those 22 GB, 12 GB is pure per-cell overhead — the 24-byte tax — not your actual text.

That single observation drives most real SPICE cost work:

• A high-cardinality string you don't chart on costs you ~24 bytes per row minimum, forever.
• Casting a string that holds a number into a decimal takes it from 24+len to 12.
• Dropping one 20-byte string column from a billion-row dataset saves ~44 GB.

⚠️ The formula is for one dataset only. AWS states this explicitly, and it is not a hedge:

"The formula above should only be used to estimate the size of a single dataset in SPICE. The SPICE capacity usage is the total size of all datasets in an account in a specific region. Quick Sight does not recommend that you use this formula to estimate the total SPICE capacity that your Quick Sight account is using."

Read for the account total from the SPICE capacity page in the admin console, never by summing your own estimates.

Bundled versus purchased capacity

The admin SPICE capacity page splits your total into two, and the distinction decides what you can give back:

The usage view adds a third figure — releasable unused capacity — which is "the purchased capacity that isn't in use, and so can be released to reduce costs."

"You can only release SPICE capacity that isn't currently used by a dataset. Datasets in SPICE stay there until someone remove them from SPICE. To change that, you can either delete the datasets or change them so they aren't stored in SPICE."

So the sequence to actually reduce spend is: remove or convert datasets first, then release. Not the other way round.

I could not verify the exact bundled GB per Author or per Reader from the pages fetched for this lesson — the user guide describes the category without publishing the per-user figure. Check Amazon Quick pricing for the current number rather than trusting any blog post, including this one.

Auto capacity purchasing — read this before you enable it

This is the highest-consequence toggle on the page, and its defaults are not what you'd assume.

Behaviour, verbatim from the docs:

• It only goes up. "Auto capacity purchasing doesn't support auto-decrement. If users want to reduce their SPICE usage, capacity release must be done manually."
• With it off, over-capacity refreshes just die. "When auto purchase capacity is turned off, ingestions or refreshes that exceed the account's SPICE capacity automatically fail."
• Turning it off converts what you're using into what you've bought. "If you turn auto capacity purchasing off after it's been in use, your account's current capacity becomes your account's purchased capacity. If your account has no remaining capacity when you turn auto purchase off, the next ingestion or refresh will fail."
• Turning it on while already over capacity trues you up. "If your account already exceeds its SPICE capacity when you turn auto capacity purchasing on, Amazon Quick automatically matches your account's capacity to your current usage. After Amazon Quick matches your account's capacity, the auto-purchase logic starts."

The opinionated take: turn auto-purchase on for any account where a failed overnight refresh is worse than an unplanned invoice line — which is most production accounts — and pair it with a monthly review that releases unused purchased capacity. Auto-purchase without that review is a ratchet that only ever tightens.

Hiding capacity from authors

Enterprise accounts using custom permissions can hide the account-wide SPICE usage labels from authors: Manage users → Manage permissions → Restrict access to → Datasets → Viewing account SPICE capacity, then assign the permission with the UpdateUser API.

Useful in a multi-tenant or agency account. Also a good way to guarantee nobody notices capacity pressure until a refresh fails — so if you turn it on, own the monitoring centrally.

Check yourself

1. Your source table is 50 GB of Parquet. What does that tell you about its SPICE size?
2. You have 200 GB of purchased capacity in us-east-1 and a new team spinning up in eu-west-1. How much capacity do they have?
3. A dataset has one billion rows and a 30-character UTF-8 customer_notes column nobody charts. Roughly what does that column cost, and what's the fix?
4. Auto-purchase has been on for a year. You turn it off to control cost. What happens to your capacity figure, and what's the immediate risk?
5. Why does AWS tell you not to use the sizing formula to estimate account capacity?

Answers

1. Essentially nothing. Logical size is computed from cell counts and data types after transformations and calculated columns, not from the source file size. Run the formula on the schema instead.
2. None. Capacity is allocated per Region and other Regions have no capacity unless purchased. They need their own purchase in eu-west-1.
3. Roughly (24 + 30) × 1e9 ≈ 54 GB. The fix is to exclude the field in the dataset — not in the analysis, because only changes saved in the dataset affect SPICE size.
4. Your current usage becomes your purchased capacity. The risk is that you now have no headroom: if the account has no remaining capacity at that moment, the next ingestion or refresh fails.
5. Because account usage is the total of all datasets in that Region, and AWS explicitly does not recommend the single-dataset formula for that purpose. Read the admin SPICE capacity page.

Teaching this section

• Open with the string tax. Put the 34 GB worked example on the board and ask which column type to attack first. People reliably guess "the numerics, there are eight of them." The reveal that five string columns are 65% of the bill is the moment the formula stops being trivia.
• Demo: the two independent Region selectors. Change the Region in the AWS console, show that the Quick console did not follow, then read the Region out of the URL. Two minutes, prevents a real incident.
• Good discussion: "Auto-purchase on or off for our account?" There is no universally right answer — it's a trade between a failed refresh and an unbudgeted invoice. Make them argue it.
• Watch for: learners estimating account capacity by summing per-dataset formula results. Correct it immediately; AWS says not to, and the drift compounds.
• Timing: 25 min presented, 10 discussion.