AWS Training
Modules Listen Certification
What SPICE is, and how it is sized Q2 — SPICE Internals and Data at Scale
0:00 0:00

← SPICE Internals and Data at Scale

Q2 Lab — break an ingestion, then diagnose it from the API

Target: a real AWS account with Amazon Quick Sight (Enterprise edition preferred — several steps are Enterprise-only and are marked).

Cost: small but not zero. You will hold a SPICE dataset for the duration and run Athena or Redshift queries. Everything is torn down in Part 6. If you are on Standard edition, the quota numbers differ (25 M rows / 25 GB, 8 ingestion calls) — substitute them and the reasoning is identical.

Write your answers down. The point of this lab is the evidence trail, not the clicking.

Setup

export ACCOUNT=<your 12-digit account id>
export REGION=<the region your Quick Sight account lives in>
export DATASET=<a SPICE dataset id you can safely disrupt>

aws quicksight list-data-sets --aws-account-id $ACCOUNT --region $REGION \
  --query 'DataSetSummaries[].{Id:DataSetId,Name:Name,Mode:ImportMode}' --output table

Q1. Which Region did you use, and how did you confirm it is the Region your Quick Sight account actually lives in — not just your CLI default?

Q2. How many of the datasets listed are SPICE versus DIRECT_QUERY? Was that split deliberate, or accidental?

Part 1 — Size a dataset on paper before you look

Pick your largest SPICE dataset. Do not look at its reported size yet.

aws quicksight describe-data-set --aws-account-id $ACCOUNT --data-set-id $DATASET \
  --region $REGION --query 'DataSet.OutputColumns[].{Name:Name,Type:Type}' --output table

Q3. Classify every column as numeric, date, or text. For text columns, estimate the average UTF-8 byte length. Show your working.

Q4. Apply the formula. What is your estimated logical size?

(numeric cells × 12) + (date cells × 12) + Σ(24 + avg UTF-8 length) per text cell

Q5. Now check the real figure — Quick Sight console → Manage Quick → SPICE capacity, or describe-data-set → ConsumedSpiceCapacityInBytes. How far off were you, and in which direction?

Q6. What percentage of your estimate is the 24-byte per-text-cell overhead alone? Which single column would you drop first, and how many GB would that save?

Part 2 — Read the ingestion history properly

aws quicksight list-ingestions --aws-account-id $ACCOUNT --data-set-id $DATASET \
  --region $REGION --max-results 100 \
  --query 'Ingestions[].{When:CreatedTime,Status:IngestionStatus,Src:RequestSource,
      Type:RequestType,Secs:IngestionTimeInSeconds,Bytes:IngestionSizeInBytes,
      In:RowInfo.RowsIngested,Dropped:RowInfo.RowsDropped,Err:ErrorInfo.Type}' --output table

Q7. Any RowsDropped > 0 on a COMPLETED ingestion? If yes — how long has that been happening, and would anyone have noticed?

Q8. Any RequestType: EDIT rows? Match them to a time of day. Who was editing, and did they know they were triggering a reload?

Q9. Plot IngestionTimeInSeconds over the last 20 runs. Is it flat, or is it creeping up?

Q10. Split the history by RequestSource:

aws quicksight list-ingestions --aws-account-id $ACCOUNT --data-set-id $DATASET \
  --region $REGION --max-results 100 \
  --query 'Ingestions[].RequestSource' --output text | tr '\t' '\n' | sort | uniq -c

How many SCHEDULED versus MANUAL in the last 24 hours?

Part 3 — The deliberate failure ⚠️

This part intentionally breaks something. Use a dataset nobody depends on.

3a. Break the schema contract

  1. In your source database, add a column to the table (or rename one) that the dataset reads.
  2. Trigger a refresh:
aws quicksight create-ingestion --aws-account-id $ACCOUNT --data-set-id $DATASET \
  --ingestion-id "lab-schema-$(date +%Y%m%dT%H%M%S)" \
  --ingestion-type FULL_REFRESH --region $REGION
  1. Poll until it settles:
aws quicksight describe-ingestion --aws-account-id $ACCOUNT --data-set-id $DATASET \
  --ingestion-id <the id you used> --region $REGION

Q11. What is the exact ErrorInfo.Type? Write it down verbatim.

Q12. What did the console show you for the same failure? Which of the two would you rather have in a support case, and why?

Q13. Fix it — open the dataset, save it, refresh again. Confirm success from the API, not the UI. Why can Quick Sight not do this itself?

3b. Exhaust the ingestion quota (Enterprise: 32 · Standard: 8)

⚠️ Do this on a small dataset and understand that you are consuming your account's ingestion budget for the next 24 hours, in this Region, for everyone.

for i in $(seq 1 40); do
  aws quicksight create-ingestion --aws-account-id $ACCOUNT --data-set-id $DATASET \
    --ingestion-id "lab-quota-$i-$(date +%s)" --ingestion-type FULL_REFRESH \
    --region $REGION 2>&1 | tail -1
  sleep 2
done

Q14. At which iteration did it start failing? What exception name, and what HTTP status?

Q15. Did the count you achieved equal 32 exactly, or 32 minus the scheduled refreshes that ran in the same window? This is the empirical answer to the open documentation question from lesson 2 — record it, with the date, because it is the single most valuable finding in this lab.

Q16. Write the retry logic you would ship. Which HTTP statuses do you retry, and what do you do with a 409 LimitExceededException?

Part 4 — Incremental refresh and the window trap (Enterprise)

Configure an incremental refresh on a SQL-backed dataset with a 7-day look-back window on a date column.

  1. Note a row in the source whose date is older than the window — say 60 days ago.
  2. Change a value in that row at source.
  3. Run an incremental refresh.

Q17. Did the change appear in the dashboard? Why not?

Q18. What is your organisation's correction horizon — how far back does the source system amend data? Who did you have to ask, and did they know?

Q19. Given that answer, what window size and what full-refresh cadence do you actually need?

Q20. Now delete the incremental refresh configuration. What ingestion appears in the history immediately afterwards? What would that have cost you on a 2-billion-row dataset?

Part 5 — Capacity and the Region trap

Q21. Record: total capacity, bundled, purchased, used, and releasable unused — for each Region where you have Quick Sight assets.

Q22. Change the Region in the AWS Management Console. Does the Quick Sight console follow? Read the Region out of the Quick Sight URL and record it.

Q23. Is auto capacity purchasing on or off for your account? Was that a decision anybody made, or an artefact of how the account was created (console ⇒ on, API ⇒ off)?

Q24. If you turned auto-purchase off right now, what would your purchased capacity become, and what is your headroom immediately afterwards?

Part 6 — Write the support case

Using only the evidence you gathered above, write a technical support case body for the failure in Part 3a. It must contain, in order:

  1. Account ID and Region
  2. Dataset ID and the failed Ingestion ARN
  3. ErrorInfo.Type, verbatim
  4. RequestId
  5. UTC timestamps
  6. RowInfo
  7. What you ruled out, with the commands
  8. One question with a factual answer

Q25. Choose a severity and justify it in one sentence against AWS's published definitions.

Q26. Rewrite your question twice more — once badly (unanswerable) and once as a design question suitable for a TAM rather than a case. What's different about each?

Teardown

# Release only capacity that is genuinely unused, and only after removing datasets.
# Delete any lab dataset you created:
aws quicksight delete-data-set --aws-account-id $ACCOUNT --data-set-id <lab dataset> --region $REGION

Restore the source schema change from Part 3a, and re-save any dataset you disturbed.

⚠️ Your ingestion budget for this Region is now largely consumed for 24 hours. Tell your team.

Done when you can

Facilitator notes