What an AWS DuckDB walkthrough can teach you about choosing a data-processing engine
By Upstep · 10 October 2026
1. The question behind a slow pipeline
Imagine joining a data team and inheriting a daily sales pipeline. It reads files, joins a product table, calculates revenue, and writes a summary. The dashboard arrives on time, but the job seems expensive for such a small transformation.
Your first instinct might be to tune Spark. That is reasonable. A more useful starting question is: what resources does this workload actually need?
Early-career engineers often learn tools before learning how to choose between them. Production work asks for both. You need to explain why a design meets its deadline, produces correct results, and remains manageable when something breaks.
A recent AWS walkthrough provides a useful opportunity to practise that judgment.
2. What changed
On 6 October 2026, AWS published “Cost-effective ETL with DuckDB and Amazon S3 Tables on AWS Glue.” Its official index describes running DuckDB with Glue 6.0 on a single worker, reading Parquet from S3 and writing Iceberg tables to S3 Tables. It also describes a comparison with an equivalent Spark job. [1]
ETL means extract, transform, and load: retrieve data, reshape it, and save it for another system.
This is a newly published implementation pattern, not evidence that DuckDB or single-machine analytics was invented this week. The useful development is a concrete example connecting an embedded engine with managed cloud execution and table storage.
My interpretation: it is a reason to evaluate the engine inside a pipeline, while considering the surrounding platform separately. It is not a reason to replace every Spark job.
3. Understand the layers before choosing the engine
DuckDB is an analytical SQL engine that can run inside your application’s process. A Python program can call it directly. Its columnar, parallel execution is designed for analytical work such as filtering and aggregation. [2]
Consider a report that needs only order date, region, and amount from a file containing fifty columns. DuckDB can read the required Parquet columns and push suitable filters into the file scan. File statistics can sometimes let it skip irrelevant sections. [3]
Parquet is a column-oriented file format. Iceberg is a table format: it organizes data files and metadata so engines can work with a table rather than an arbitrary folder. DuckDB’s Iceberg extension documents catalog connections and supported reads and writes. A catalog helps resolve table names and metadata. [4]
In the AWS pattern, these components have separate jobs: Glue provides the execution environment, DuckDB performs transformations, and S3 Tables holds the Iceberg output.
Changing the engine does not remove the need for permissions, scheduling, monitoring, or recovery.
4. How teams have approached this problem
One established approach is to run transformations in Spark. In cluster mode, a driver coordinates work and executor processes perform computations. This lets an application use resources across multiple machines. [5]
That architecture brings useful capacity, but it also involves coordination. A small job may spend a meaningful part of its elapsed time starting resources or preparing work. Whether that matters must be measured.
Teams also use warehouse SQL, Python scripts, and other single-machine engines. DuckDB does not introduce the idea of small-scale ETL.
The practical comparison is therefore with the team’s actual alternative. A well-tuned existing Spark pipeline has different migration economics from a new job with no established implementation.
5. What the approach can improve
For a suitable workload, an embedded engine may make the transformation easier to develop and inspect. You can explore the SQL locally, examine its output, and then evaluate it in a managed environment.
However, local success is only the first step. Cloud storage access, dependencies, authentication, and table writes introduce additional behavior.
My assessment: the strongest potential benefit is proportionality. The execution design can match the work rather than inherit a larger architecture by default.
For example, a narrow daily aggregation deserves its own evaluation even if the organization also processes enormous historical datasets. Those workloads do not necessarily need identical execution paths.
Measure improvements across three dimensions: correct output, complete job duration, and operational cost. Faster SQL alone does not establish a better pipeline.
6. Limits: one machine still has boundaries
Avoid the rule “DuckDB only works when the dataset fits in RAM.” DuckDB can process data in chunks and spill intermediate results to disk when necessary. Spilling means temporarily moving working data out of memory. It adds disk activity and can affect performance. [6]
Input size is also an incomplete guide. A small compressed dataset can produce large intermediate results. Joining tables with repeated keys may multiply rows, while grouping by many distinct values can require substantial working memory.
DuckDB’s documentation acknowledges that out-of-memory failures remain possible for some operations and query combinations. [7]
I would be cautious when a workload needs more throughput than the available machine can deliver, repeatedly creates huge intermediate results, or depends on established Spark-specific behavior.
Operational consistency matters too. Adding another engine creates another dependency and troubleshooting path. A modest runtime saving may not justify that burden.
AWS documents responsibility for additional Python dependencies and their compatibility. Pin and test your environment; do not assume a package that works on your laptop will behave identically in Glue. [8]
7. An enterprise scenario: daily sales reporting
Consider a hypothetical retailer receiving order files from several countries. Each morning, finance needs yesterday’s paid sales by country and product category.
The pipeline must validate order identifiers, join a product reference table, aggregate amounts, and publish a result before the reporting deadline. Missing product codes must be visible rather than silently disappearing through an inner join.
I would first benchmark one representative day, then a peak trading day and a historical rerun. That rerun is a backfill: processing earlier dates again after a correction.
The engine decision should include the backfill. A design that handles Tuesday comfortably may struggle when asked to rebuild six months.
The team also needs idempotency: rerunning the same input should not double the published sales. That requirement remains whether DuckDB or Spark performs the calculation.
8. Build a small, meaningful comparison
Start locally so the experiment is accessible. You need basic SQL and Python, plus an understanding of joins and grouped totals.
Generate two reproducible datasets: orders and products. Give orders an identifier, date, product identifier, payment status, region, and amount in integer cents. Use a fixed random seed so another person can recreate the inputs.
Create three dataset sizes, starting around 100 MB and increasing within your machine’s capacity. Include a separate test with disproportionately popular products.
Implement the same transformation in DuckDB and local Spark:
1. Select paid orders for one reporting period.
2. Join product categories, explicitly handling unmatched products.
3. Calculate revenue and order counts by date, region, and category.
4. Write the result to Parquet.
Match data types, null handling, and date rules. Compare results by their grouping keys before comparing speed.
Record end-to-end duration, transformation duration, peak process memory, and temporary disk usage where measurable. Run each case several times and report the median and range. Separate first-run behavior from subsequent runs; cached data can change the result.
A local Spark comparison does not measure distributed Spark’s scaling capabilities or Glue billing. Label that boundary clearly.
For an optional cloud extension, reproduce the validated transformation in Glue and add Iceberg output using the supported catalog setup. Record actual billed resources and storage requests. No benchmark results are claimed here: the measurements are your project’s output.
9. What the project demonstrates
This project can show more than familiarity with two libraries.
It demonstrates SQL correctness, controlled experimentation, resource measurement, and awareness of workload growth. Testing unmatched keys and reruns shows that you understand failures beyond syntax errors.
The strongest portfolio artifact is a short decision record. State what you tested, what you observed, what you chose, and what would make you reconsider.
“DuckDB met our deadline on these workloads” is a defensible conclusion. “Spark is unnecessary” is a much broader claim that this experiment cannot establish.
10. Your next steps
Choose one understandable transformation and establish correct output first. Then identify where time goes: startup, reading, computation, or writing.
Inspect the query plan—the engine’s proposed execution steps—before changing infrastructure. DuckDB’s performance guidance recommends investigating memory, storage, data layout, and execution plans when workloads are slow. [9]
Finally, write a recommendation with explicit conditions: expected volume, completion deadline, rerun behavior, and maintenance responsibility.
That is the career skill this exercise develops: connecting a technical choice to evidence and explaining its boundaries.
11. Sources
Primary sources; documentation checked 10 October 2026.
1. [AWS: Cost-effective ETL with DuckDB and Amazon S3 Tables on AWS Glue](https://aws.amazon.com/blogs/big-data/cost-effective-etl-with-duckdb-and-amazon-s3-tables-on-aws-glue/), 6 October 2026. Publication date and scope confirmed through the [official AWS S3 Tables index](https://aws.amazon.com/blogs/big-data/category/storage/amazon-s3-tables/). The full article was inaccessible during preparation; its detailed benchmark numbers are deliberately omitted.
2. [DuckDB: Project overview](https://duckdb.org/).
3. [DuckDB: Reading and Writing Parquet Files](https://duckdb.org/docs/current/data/parquet/overview).
4. [DuckDB: Iceberg Extension](https://duckdb.org/docs/current/core_extensions/iceberg/overview).
5. [Apache Spark: Cluster Mode Overview](https://spark.apache.org/docs/latest/cluster-overview.html).
6. [DuckDB: Memory Management](https://duckdb.org/2024/07/09/memory-management), 9 July 2024.
7. [DuckDB: Out of Memory Errors](https://duckdb.org/docs/lts/guides/troubleshooting/oom_errors).
8. [AWS Glue: Using Python Libraries](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python-libraries.html).
9. [DuckDB: My Workload Is Slow](https://duckdb.org/docs/current/guides/performance/my_workload_is_slow).