Appearance
System Design Basics 🏗️ (Data Engineering Perspective)
System design in data engineering is about designing end-to-end data systems that are scalable, reliable, and efficient.
🧠 It is not about memorizing architecture diagrams — it is about understanding how data flows under constraints.
🎯 Why System Design Matters
In real interviews, you will be asked:
- Design a real-time analytics system
- Build a scalable ETL pipeline
- Handle billions of events per day
- Design a recommendation data pipeline
These are system design problems.
🧭 What You Are Designing
A complete data system includes:
- Data Sources
- Ingestion Layer
- Processing Layer
- Storage Layer
- Serving Layer
Each layer must work together under scale.
⚙️ High-Level Data System Architecture
Data Sources ↓ Ingestion Layer (Kafka / APIs / CDC) ↓ Processing Layer (Spark / Flink) ↓ Storage Layer (Data Lake / Warehouse) ↓ Serving Layer (BI / APIs / ML)
🧱 Core Design Principles
1. Scalability
System must handle increasing data volume.
Approaches:
- Horizontal scaling
- Partitioning data
- Distributed processing
2. Fault Tolerance
System should not break when failures happen.
Techniques:
- Retry mechanisms
- Data replication
- Checkpointing (streaming systems)
3. Data Consistency
Ensure data correctness across systems.
Models:
- Strong consistency
- Eventual consistency
4. Latency vs Throughput
| Metric | Meaning |
|---|---|
| Latency | Speed of single request |
| Throughput | Volume processed over time |
Tradeoff is always required.
5. Idempotency
A system should safely reprocess data:
Same input → same output (no duplicates)
Critical for pipelines.
🔄 Batch vs Real-Time Systems
Batch Systems
- Process data periodically
- High throughput
- Higher latency
Used in:
- Reporting
- Analytics dashboards
Real-Time Systems
- Process data instantly
- Low latency
- Complex design
Used in:
- Fraud detection
- Live dashboards
⚡ Common Architecture Patterns
1. Lambda Architecture
Combines batch + streaming:
- Batch layer → accurate processing
- Speed layer → real-time processing
- Serving layer → merged results
✔ Accurate
❌ Complex
2. Kappa Architecture
Stream-only system:
- Everything is a stream
- Batch is replay of events
✔ Simple
✔ Unified logic
❌ Requires strong streaming infra
🧠 Data Flow Thinking
In system design, always think:
- Where does data come from?
- How is it ingested?
- How is it processed?
- Where is it stored?
- Who consumes it?
🚨 Common System Design Problems
- Late arriving data
- Data duplication
- Schema evolution
- Backpressure in streaming systems
- Bottlenecks in processing layer
🔗 How Everything Connects
- Data Modeling → defines structure
- Storage → holds data
- Processing → transforms data
- Pipelines → moves data
- Warehouses → enables analytics
- System Design → connects everything
🎯 Goal of System Design Basics
You should be able to:
- Think in system components
- Identify bottlenecks
- Design scalable pipelines
- Choose correct architecture pattern
- Explain tradeoffs clearly
“System design is not about drawing boxes — it is about understanding how data behaves at scale.”