Consistency is not a localized property

Lobsters Hottest News

Summary

一篇关于分布式系统一致性的技术博客,指出一致性并非单个组件的局部属性,即使 Apache Kafka 花费数年实现 exactly-once 语义,仍需要应用端到端设计和专人对整体属性负责。

<p><a href="https://lobste.rs/s/rhy5zp/consistency_is_not_localized_property">Comments</a></p>
Original Article
View Cached Full Text

Cached at: 10/11/26, 01:15 AM

# Consistency is not a localized property Source: [https://n-the-loop.com/blog/consistency-is-not-a-localized-property/](https://n-the-loop.com/blog/consistency-is-not-a-localized-property/) Oct 6, 2026 It is a common pattern for developers to assume that consistency is something they can achieve inside a single component\. Apache Kafka is the cautionary tale\. Making writes exactly\-once took a team of distributed systems engineers years and several new components and a complete rewrite of older components \([Gustafson et al\. 2016](https://n-the-loop.com/blog/consistency-is-not-a-localized-property/#citeproc_bib_item_1)\)\.[1](https://n-the-loop.com/blog/consistency-is-not-a-localized-property/#fn:1) And after all of that, the people who built it are blunt about what you get: > Exactly\-once processing is an end\-to\-end guarantee and the application has to be designed to not violate the property as well\.\([Narkhede and Wang 2017](https://n-the-loop.com/blog/consistency-is-not-a-localized-property/#citeproc_bib_item_2)\) If the component cannot promise the property, the promise has to live somewhere else\. Two practical consequences: - Someone has to**understand**the system end to end\. Since no component can confirm it, something outside the components has to keep it all consistent\. - Someone has to**own**it\. Component owners will each correctly say their part works\. A global property needs a person who is liable for the whole thing, with some guarantee that it makes sense\. The lesson: do not hand global, valuable properties to machines and assume they are held\. Such guarantees are grounded in understanding, not code\. ## References Gustafson, Jason, Flavio Junqueira, Apurva Mehta, Sriram Subramanian, and Guozhang Wang\. 2016\. “KIP\-98: Exactly Once Delivery and Transactional Messaging\.” Apache Software Foundation\. 2016\.[https://cwiki\.apache\.org/confluence/display/KAFKA/KIP\-98\+\-\+Exactly\+Once\+Delivery\+and\+Transactional\+Messaging](https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging)\. Narkhede, Neha, and Guozhang Wang\. 2017\. “Exactly\-Once Semantics Are Possible: Here’s How Kafka Does It\.” Confluent\. 2017\.[https://www\.confluent\.io/blog/exactly\-once\-semantics\-are\-possible\-heres\-how\-apache\-kafka\-does\-it/](https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/)\. Olshan, Justine, and Calvin Liu\. 2022\. “KIP\-890: Transactions Server\-Side Defense\.” Apache Software Foundation\. 2022\.[https://cwiki\.apache\.org/confluence/display/KAFKA/KIP\-890:\+Transactions\+Server\-Side\+Defense](https://cwiki.apache.org/confluence/display/KAFKA/KIP-890:+Transactions+Server-Side+Defense)\. [← All posts](https://n-the-loop.com/blog/)

Similar Articles

Data liberation: Apache Kafka's native cluster mirroring

Hacker News Top

Apache Kafka introduces native cluster mirroring to embed cross-cluster replication directly into the broker, simplifying data movement between clusters with byte-for-byte transfer and exact offset preservation.

Retries don't fix eventual consistency

Lobsters Hottest

The article argues that retrying messages in eventually consistent distributed systems treats normal state as failure, and instead suggests storing incoming data and processing when all prerequisites arrive, eliminating the need for retries and dead-letter queues.

@vintcessun: The barrier to developing multi-agent systems is too high; those who haven't studied Agent theory dare not touch it. As a result, project implementation is difficult, and teams can only rely on a few experts. This paper directly takes mature architectural patterns from distributed systems (publish-subscribe, message queues, etc.) and defines a minimal set of Agent concepts mapped onto them. Even students with no DS experience can use it...

X AI KOLs Timeline

This paper proposes directly mapping mature architectural patterns from distributed systems (such as publish-subscribe and message queues) to multi-agent systems to lower the development barrier. It was validated in a course: even students with no distributed systems experience could get started with gRPC and RabbitMQ, achieving an average score above 80%.

@siddontang: A few days ago, someone asked me: What Multi-Agent System is TiDB using internally now? I answered very directly: Actually, we are deliberately avoiding Multi-Agent. If one Agent can handle it, let one Agent do it. If it really needs to be split, at most put a Coordinator above…

X AI KOLs Timeline

The TiDB team deliberately avoids using Multi-Agent systems internally, based on lessons learned from distributed systems and referencing Anthropic's related research, emphasizing that optimizing a single Agent might be more effective.