Designing Modern Data Systems

<p>Data systems outlive applications frameworks and infrastructure.<br>They encode decisions that shape what a system can become-and what it can never safely change.</p><p><em>Designing Modern Data Systems</em> is a deep decision-driven guide to building data systems that are reliable scalable and adaptable over time. Rather than focusing on tools or trends this book teaches how to reason about architecture itself: how guarantees are chosen where authority lives how failures manifest and how systems evolve under real-world pressure.</p><p>Written for experienced engineers and architects the book treats data systems as long-lived sociotechnical systems-not just databases or pipelines. It focuses on clarity of responsibility explicit trade-offs and preserving meaning as data moves changes and ages.</p><p>This book takes a structured journey through modern data system design:</p><ul><li>How to define <strong>data systems</strong> as distinct from applications and infrastructure</li><li>How <strong>non-functional requirements</strong> like reliability availability latency and cost shape architecture long before technology choices</li><li>How to design <strong>data models storage engines and indexing strategies</strong> that survive product evolution</li><li>How to reason about <strong>replication partitioning coordination and distributed transactions</strong> without accidental complexity</li><li>How batch and stream processing fit into a unified view of data over time</li><li>How <strong>logs history and derived data</strong> enable recovery reprocessing and safe change</li><li>How to operate systems in production with observability backpressure and failure isolation</li><li>How to design data systems that support <strong>machine learning and large language model platforms</strong> including feature pipelines and embeddings</li><li>How to migrate evolve and decommission systems without outages or loss of trust</li></ul><p>Throughout the book ideas are grounded in a single evolving reference system allowing readers to see how architectural decisions accumulate and interact as requirements change.</p><p>What Makes This Book Different</p><ul><li><strong>Decision-focused not tool-driven</strong><br>The book avoids product comparisons and instead teaches how to evaluate any technology within clear architectural constraints.</li><li><strong>Explicit trade-offs not recipes</strong><br>Every design choice is examined in terms of what it enables what it forbids and what it costs.</li><li><strong>Modern without being trendy</strong><br>AI and LLM systems are addressed where they introduce real architectural pressure-without hype or speculation.</li><li><strong>Written for longevity</strong><br>The principles in this book are designed to remain relevant as tools platforms and organizational structures change.</li></ul><p>This book is written for:</p><ul><li>Software engineers designing backend and platform systems</li><li>Data engineers responsible for storage processing and pipelines</li><li>Staff principal and senior engineers shaping architectural direction</li><li>Architects and technical leaders responsible for long-term system evolution</li><li>Practitioners preparing for system design interviews who want judgment not templates</li></ul><p>This is not:</p><ul><li>A beginner's introduction to databases</li><li>A step-by-step tutorial for specific tools</li><li>A catalog of technologies or patterns</li></ul><p>Instead it is a book about <strong>how to think clearly about data systems</strong> and how to design them so they remain understandable trustworthy and changeable over time.</p><p>If you are responsible for making architectural decisions-and living with their consequences-<em>Designing Modern Data Systems</em> is written for you.</p>