1. The problem behind a successful pipeline
Imagine joining a data team and delivering your first ingestion pipeline. The job finishes successfully. Rows appear in the warehouse. Your manager asks whether yesterday’s cancelled orders are reflected in the revenue dashboard.
You check the row count. It looks reasonable.
But a count cannot tell you whether an order changed from “paid” to “cancelled,” whether a deleted record disappeared, or whether yesterday’s backlog has finished processing. You have shown that data moved. You have not yet shown that the reporting data is trustworthy.
That gap is where production data engineering begins. A recent Databricks release makes a useful starting point for learning it: integrated change data capture for MySQL.
2. What changed
Databricks’ October 15, 2026 release notes announce general availability of its integrated MySQL CDC pipeline. It combines extraction and application in one pipeline update, removing the requirement for a separate ingestion gateway. [1]
CDC means change data capture: identifying changes in a source database so another system can incorporate them.
The announcement is about a simpler deployment architecture. It does not mean CDC was invented this week, or that every pipeline becomes real-time.
There is also a documentation inconsistency worth acknowledging: the October 9 source-setup page still labels the MySQL connector Public Preview. The newer release notes explicitly announce integrated-pipeline GA. Confirm availability for your workspace rather than assuming every page has been updated together. [1][3]
3. How CDC works
Consider an orders table. Order 104 starts with a status of “pending,” becomes “paid,” and is later cancelled. Copying only newly created orders misses the later changes.
MySQL’s binary log, commonly called the binlog, records database changes. With row-based logging, events describe changes to individual rows. A replication consumer can use those events to maintain another copy. This is different from repeatedly downloading an entire table. [5]
An initial snapshot establishes the starting dataset. Subsequent changes keep it current.
In the integrated Databricks pipeline, extraction writes source data into a staging volume, a managed file-storage location. Application then incorporates those changes into destination tables. A primary key, such as order_id, identifies the record being changed. [2]
These tables sit at the ingestion boundary of your analytics system. Further transformations still need to define revenue, handle currencies, and apply business rules. Replicating a cancelled order does not automatically teach a dashboard what cancellation means.
4. How teams handled this previously
The standard Databricks architecture separates a continuously running ingestion gateway from the ingestion pipeline that applies captured data. The integrated approach combines those responsibilities. [1][2]
A simpler home-built alternative might query rows whose updated_at timestamp is newer than the last extraction. That can be adequate when its assumptions are explicit.
However, imagine an application that forgets to update that timestamp. Or imagine a hard delete: the row is removed completely, leaving nothing for the next timestamp query to select. A design must account for those cases.
A full-table refresh avoids some incremental bookkeeping, but repeatedly moving an entire large table can be expensive. The practical choice depends on data volume, acceptable delay, deletion behavior, and recovery requirements.
5. What improves and what still needs judgment
My interpretation is that the main improvement is a smaller operational surface: fewer separately configured components and fewer connections between them to troubleshoot.
That can help a team deliver a manageable ingestion service. It does not prove that the new design costs less for every workload.
Databricks documents a runtime policy called smart closure. An update can finish after catching up, or stop at a runtime limit and leave remaining changes for another update. The event log distinguishes those outcomes. [4]
This distinction matters when you monitor the system. A completed execution and a current dataset are different conditions.
I would therefore track three separate questions: Did the run succeed? How far behind is the destination? Does the destination match the source for a controlled comparison? Each answers something the others cannot.
6. Limitations and poor-fit situations
The integrated MySQL path requires classic compute. Tables without primary keys can have duplicate rows collapse, and expired source binlogs can require a full refresh. These are material design constraints. [2]
The source setup requires row-based binary logging with full row images, replication privileges, and network access. Supported read replicas can reduce primary-database load, but introduce another source of delay; Aurora read replicas are excluded by the setup documentation. [3]
Continuous integrated execution has separate Beta documentation. Do not interpret the GA announcement as making every execution mode generally available. [6]
For a system requiring seconds-level reaction, I would first measure whether this approach meets the requirement. An hourly analytics refresh and an operational fraud decision have different needs.
I would also question adopting this platform solely to copy a tiny daily table. Evaluate the total operating cost and existing infrastructure. Finally, do not treat a current-state replica as an audit trail: retaining every historical business state is a separate requirement.
7. A realistic enterprise example
Consider a hypothetical retailer whose checkout application uses MySQL. Analysts need an hourly view of paid, cancelled, and refunded orders.
The team agrees on a freshness target and a precise revenue definition. For example, cancelled orders must be excluded, and refunds must be represented consistently with the finance model.
The ingestion layer maintains the source records. A transformation layer applies those business rules. The dashboard displays both the metric and the latest confirmed refresh time.
Suppose ingestion stops during a promotion. When it resumes, the team checks whether the backlog is shrinking before declaring recovery. It also reconciles a fixed set of orders whose changes are known.
This example is an architectural proposal, not a claim that a named retailer has deployed the feature.
8. Build a small CDC acceptance lab
The following project is proposed and has not been executed for this article. Use synthetic data, a supported MySQL deployment, and a compatible Databricks workspace. Check permissions, connectivity, and cost before provisioning.
Create an orders table with order_id as its primary key, plus status, amount, currency, and updated_at. Use a fixed-precision decimal for amounts and keep the first experiment to one currency.
Load 1,000 predictable records. Configure a current-state destination using the official setup example, then capture a baseline after ingestion finishes. [2]
Apply a controlled sequence of changes:
Action | Expected result after catch-up |
|---|---|
Insert 20 new orders | Each new key exists once |
Change 10 existing statuses | Those keys have their new statuses |
Delete 5 different original orders | Those keys are absent |
Change one order twice | Its final state matches the source |
The expected final row count is 1,015, but that is only the first check.
Export the source and destination after writes have stopped and ingestion has caught up. Compare key sets and business-column values. Also compare counts by status and monetary totals. A matching total can hide offsetting errors, so retain the row-level comparison.
Next, pause scheduled ingestion while keeping source log retention comfortably longer than the experiment. Make another recorded batch of changes. Resume ingestion and measure how long it takes to converge.
Repeat an update without changing the source. Confirm that the final business records remain unchanged. This tests a useful outcome without claiming that you have proven all internal delivery guarantees.
Save the change script, expected results, observed differences, run timestamps, and cleanup instructions. If you cannot access the managed connector, build a local change-event simulation but label it clearly as concept practice, not connector validation.
9. Skills this project demonstrates
The project demonstrates SQL, data modeling, reconciliation, failure testing, and operational communication.
More importantly, it gives you evidence for an interview. Instead of saying “I built a CDC pipeline,” you can explain how you checked deletions, distinguished stale data from failed execution, and measured recovery.
A short incident-style write-up is valuable: describe the interruption, expected behavior, observations, and remaining uncertainty. Do not invent performance improvements; report what your experiment actually measured.
10. Practical next steps
Start by writing your acceptance criteria before configuring the connector. Define what “correct” and “fresh enough” mean for the use case.
Then run the smallest baseline, add controlled changes, and test interruption recovery. Investigate mismatches before increasing scale.
For a production decision, review source-log retention, schema-change behavior, refresh procedures, and ownership with the database and analytics teams. Measure costs over a representative workload.
The professional skill to develop is knowing what evidence is sufficient to trust the data—and being precise about what your test has not established.
11. Sources
Primary sources checked on 18 October 2026. Product statements above are attributed to documentation; recommendations and the retailer scenario are the author’s interpretation.
Databricks October 2026 release notes — October 15 integrated MySQL CDC GA announcement.
Create an integrated CDC pipeline for MySQL — architecture, setup, compute, and limitations.
Configure MySQL for ingestion — source configuration and replica restrictions.
Smart closure for integrated CDC pipelines — stopping conditions and backlog continuation.
MySQL 8.4: Binary logging formats — row-based logging.
Continuous integrated CDC — separate Beta execution mode.
