Data Integrity in Distributed Systems and the Debezium Call

Many Places, One Truth

A distributed system, at its simplest, is parts in different places acting as if they were one network. You run search on MeiliSearch, but the real data sits in PostgreSQL. Your marketing tool runs on its own MySQL. Your website is in one database, your accounting in another. They all work together for the same product.

The trouble starts right there: all of these parts have to see the same truth. Data that changes in one has to change in the other, or a user updates something in one place and sees the old version in another. This piece is about the pain of keeping that consistent, and the fix I eventually settled on.

The Hard Ways to Stay Consistent

The first methods that come to mind for syncing databases usually dig you into a deeper hole.

Polling. Let a worker run on a schedule and go update the data. Sounds reasonable, but as the database grows these queries do full table scans and lock the CPU. It is never truly real time; the gap between the two only widens. And you cannot catch deleted rows, because a select cannot see the absence of a row. So you end up writing a second control layer just for that.

Spaghetti code and coupling. Burying the sync logic inside application code. Say you have a UserService, and when a user updates their profile you also need to update ElasticSearch. You add an ElasticSearch call inside the method. Now UserService does not just do user work; if ElasticSearch is slow the “Update” button is slow, if that service errors the button breaks. It did its own job fine, but the dependency you glued on took it hostage.

The dual write curse. The sneakiest enemy of distributed systems. You say “I write the user to the database, then right after I push an event to RabbitMQ.” You are writing to two separate places with no guarantee both succeed. If the queue is down at that moment, the data stays in the database but the event never goes out, and when the queue comes back it has no idea about that record. Missing data, duplicate data, silent inconsistency.

Race conditions. Thought to be rare, but in chained event sequences and async setups the order gets tangled. When more than two places update, one write lands on top of something that has not been created yet, and you get an error.

The Fix: Stop Asking “What Changed?”

Underneath all of these is one wrong reflex: constantly asking the database “what changed in you?”

But the database already notes every operation somewhere. PostgreSQL calls it the WAL (Write Ahead Log), MySQL calls it the Binlog. Even if the database crashes, these logs bring it back. The fix is right there: without touching application code or loading the database with extra queries, listen to those logs directly. We call this Change Data Capture, CDC for short.

So who reads the logs? Parsing a raw log file is an engineering job on its own. This is where Debezium comes in. Debezium introduces itself to the database as a replica; the moment the engine writes “this row was added, that row was deleted” to the log, Debezium catches it, turns it into standard JSON, and drops it onto Apache Kafka. (A line I like, by the way: if you solve a problem with Kafka, congratulations, now you have two problems.)

What happens to the pain once you move to this architecture?

• Dual write ends. No more “write to the DB and push to the queue.” You only write to the DB, and Debezium turns what was written into an event, reliably. This is the basis of the outbox pattern.
• Polling ends. No SELECT *, no extra load on the database. Reading the log is nearly free for it.
• Hard delete is solved. When you delete a row, the log gets “this ID was deleted”; Debezium catches that too and tells you “delete this from ElasticSearch as well.”
• Closest to real time. Instead of a cron job’s five-minute delay, you catch changes in milliseconds.

How the Flow Works

A simple flow to picture it:

• The application just sends INSERT INTO users... to PostgreSQL and is done.
• PostgreSQL writes the data to disk and records it in the WAL.
• Debezium sees that change in the WAL instantly.
• Kafka receives it from Debezium as a message.
• The consumers (ElasticSearch, cache, marketing tool) listen to that message and take the data.

So that complex orchestra starts playing in sync, taking its cues from a single conductor: the database logs.

Here is where I have to be honest: there is no free lunch. Debezium is a great tool, but it brings Kafka to manage, schema changes to handle, and an operational cost. Writing a cron job is easier than running Kafka. But if data consistency in a scaling system is keeping you up at night, that cost is small next to sleeping well.

There Is No Perfect Code, Only Manageable Chaos

Building a distributed system is not like clicking Lego together. In Lego the pieces are fixed; in software they keep moving, networks drop, servers sulk. What we call seniority is really the sum of the nights spent next to databases blowing up in production. The clearest thing I learned in there: there is no perfect code, only manageable chaos.

Debezium and CDC are the approach that makes that chaos manageable, pulling the sync work out of spaghetti code and tying it to a single source of truth. I wrote elsewhere about how complexity quietly eats a product from the inside; the point here is the same: protecting simplicity with a method.

In my architecture calls, the priority was never just saving the day, it was sleeping well at night. Because good engineering is not only that the code runs, but that the people keeping it alive are at peace too. Healthy, consistent, commit-heavy days.

The Turkish version of this piece is here.