MindMap Gallery The most powerful Kafka3.x mind map from entry to mastery
The most powerful Kafka3.x mind map from entry to mastery, the total structure of kafka, including entry, external system integration, production tuning, source code analysis, etc.
Edited at 2023-11-06 21:19:48This is a mind map about bacteria, and its main contents include: overview, morphology, types, structure, reproduction, distribution, application, and expansion. The summary is comprehensive and meticulous, suitable as review materials.
This is a mind map about plant asexual reproduction, and its main contents include: concept, spore reproduction, vegetative reproduction, tissue culture, and buds. The summary is comprehensive and meticulous, suitable as review materials.
This is a mind map about the reproductive development of animals, and its main contents include: insects, frogs, birds, sexual reproduction, and asexual reproduction. The summary is comprehensive and meticulous, suitable as review materials.
This is a mind map about bacteria, and its main contents include: overview, morphology, types, structure, reproduction, distribution, application, and expansion. The summary is comprehensive and meticulous, suitable as review materials.
This is a mind map about plant asexual reproduction, and its main contents include: concept, spore reproduction, vegetative reproduction, tissue culture, and buds. The summary is comprehensive and meticulous, suitable as review materials.
This is a mind map about the reproductive development of animals, and its main contents include: insects, frogs, birds, sexual reproduction, and asexual reproduction. The summary is comprehensive and meticulous, suitable as review materials.
Kafka3.x from entry to proficiency
Chapter 1 Getting Started
Chapter 1 Kafka Overview
1.1 Definition
1.2 Message queue
1.2.1 Application scenarios of traditional message queues
1.2.2 Two modes of message queue
1.3 Kafka infrastructure
Chapter 2 Quick Start with Kafka
2.1 Installation and deployment
2.1.1 Cluster planning
2.1.2 Cluster deployment
2.1.3 Cluster startup and shutdown scripts
2.2 Kafka command line operation
2.2.1 Theme command line operation
2.2.2 Producer command line operation
2.2.3 Consumer command line operations
Chapter 3 Kafka Producer
3.1 Producer message sending process
3.1.1 Sending principle
3.1.2 List of important parameters of the producer
3.2 Asynchronous sending API
3.2.1 Ordinary asynchronous sending
3.2.2 Asynchronous sending with callback function
3.3 Synchronous sending API
3.4 Producer partition
3.4.1 Benefits of Partitioning
3.4.2 Partitioning strategy for producers to send messages
3.4.3 Custom partitioner
3.5 Production experience - how producers can improve throughput
3.6 Production experience - data reliability
3.7 Production experience - data deduplication
3.7.1 Data transfer semantics
3.7.2 Idempotence
3.7.3 Producer transactions
3.8 Production experience - data in order
3.9 Production experience - data out of order
Chapter 4 Kafka Broker
4.1 Kafka Broker workflow
4.1.1 Kafka information stored by Zookeeper
4.1.2 Kafka Broker overall workflow
4.1.3 Broker important parameters
4.2 Production experience - node service and retirement
4.2.1 Serving new nodes
4.2.2 Retire old nodes
4.3 Kafka replica
4.3.1 Basic information about the copy
4.3.2 Leader election process
4.3.3 Leader and Follower fault handling details
4.3.4 Partition copy allocation
4.3.5 Production experience - manually adjust partition copy storage
4.3.6 Production experience - Leader Partition load balancing
4.3.7 Production experience - increasing the copy factor
4.4 File storage
4.4.1 File storage mechanism
4.4.2 File cleaning strategy
4.5 Efficiently read and write data
Chapter 5 Kafka Consumer
5.1 Kafka consumption method
5.2 Kafka consumer workflow
5.2.1 Overall consumer workflow
5.2.2 Principle of consumer group
5.2.3 Important parameters of consumers
5.3 Consumer API
5.3.1 Independent consumer case (subscription topic)
5.3.2 Independent consumer case (subscription partition)
5.3.3 Consumer group case
5.4 Production experience - partition allocation and rebalancing
5.4.1 Range and rebalancing
5.4.2 RoundRobin and rebalancing
5.4.3 Sticky and rebalancing
5.5 offset displacement
5.5.1 Default maintenance position of offset
5.5.2 Automatically submit offset
5.5.3 Submit offset manually
5.5.4 Specify Offset consumption
5.5.5 Specified time consumption
5.5.6 Missing consumption and repeated consumption
5.6 Production Experience – Consumer Affairs
5.7 Production experience - data backlog (how consumers can improve throughput)
Chapter 6 Kafka-Eagle Monitoring
6.1 MySQL environment preparation
6.2 Kafka environment preparation
6.3 Kafka-Eagle installation
6.4 Kafka-Eagle page operation
Chapter 7 Kafka-Kraft Pattern
7.1 Kafka-Kraft architecture
7.2 Kafka-Kraft cluster deployment
7.3 Kafka-Kraft cluster start and stop script
Part 2 External System Integration
Chapter 1 Integrating Flume
1.1 Flume Producer
1.2 Flume Consumer
Chapter 2 Integrating Flink
2.1 Flink producer
2.2 Flink consumer
Chapter 3 Integrating SpringBoot
3.1 SpringBoot producer
3.2 SpringBoot consumer
Chapter 4 Integrating Spark
4.1 Spark producer
4.2 Spark consumer
Part 3 Production Tuning Manual
Chapter 1 Kafka Hardware Configuration Selection
1.1 Scenario description
1.2 Selection of number of servers
1.3 Disk selection
1.4 Memory selection
1.5 CPU selection
1.6 Network selection
Chapter 2 Kafka Producer
2.1 Kafka producer core parameter configuration
2.2 How producers improve throughput
2.3 Data reliability
2.4 Data deduplication
2.5 Data orderliness
2.6 Data out of order
Chapter 3 Kafka Broker
3.1 Broker core parameter configuration
3.2 Serving new nodes/retiring old nodes
3.3 Add partition
3.4 Increase the copy factor
3.5 Manually adjust partition copy storage
3.6 Leader Partition load balancing
3.7 Automatically create themes
Chapter 4 Kafka Consumer
4.1 Kafka consumer core parameter configuration
4.2 Consumer rebalancing
4.3 Specify Offset consumption
4.4 Consumption at designated time
4.5 Consumer Affairs
4.6 How consumers can improve throughput
Chapter 5 Kafka in general
5.1 How to improve throughput
5.2 Data is accurate once
5.3 Set the number of partitions appropriately
5.4 A single log is larger than 1m
5.5 The server is down
5.6 Cluster stress test
Part 4 Source Code Analysis
Chapter 1 Source code environment preparation
1.1 Source code download address
1.2 Install JDK&Scala
1.3 Load source code
1.4 Install gradle
Chapter 2 Producer Source Code
2.1 Initialization
2.1.1 Program entry
2.1.2 Initialization of producer main thread
2.1.3 Producer sender thread initialization
2.2 Send data to buffer
2.2.1 Overall sending process
2.2.2 Partition selection
2.2.3 Sending message size verification
2.2.4 Memory pool
2.3 The sender thread sends data
Chapter 3 Consumer Source Code
3.1 Initialization
3.1.1 Program entry
3.1.2 Consumer initialization
3.2 Consumers subscribe to topics
3.3 Consumers pull and process data
3.3.1 Consumer/consumer group initialization
3.3.2 Pull data
3.3.3 Interceptor processing data
3.4 Consumer Offset submission
3.4.1 Submit Offset manually and synchronously
3.4.2 Manual asynchronous submission of Offset
Chapter 4 Server Source Code
4.1 Program entry
floating theme
Getting Started with Kafka3.x
Chapter 1 Kafka Overview
1.1 Definition
1.2 Message queue
1.2.1 Application scenarios of traditional message queues
1.2.2 Two modes of message queue
1.3 Kafka infrastructure
Chapter 2 Quick Start with Kafka
2.1 Installation and deployment
2.1.1 Cluster planning
2.1.2 Cluster deployment
2.1.3 Cluster startup and shutdown scripts
2.2 Kafka command line operation
2.2.1 Theme command line operation
2.2.2 Producer command line operation
2.2.3 Consumer command line operations
Chapter 3 Kafka Producer
3.1 Producer message sending process
3.1.1 Sending principle
3.1.2 List of important parameters of the producer
3.2 Asynchronous sending API
3.2.1 Ordinary asynchronous sending
3.2.2 Asynchronous sending with callback function
3.3 Synchronous sending API
3.4 Producer partition
3.4.1 Benefits of Partitioning
3.4.2 Partitioning strategy for producers to send messages
3.4.3 Custom partitioner
3.5 Production experience - how producers can improve throughput
3.6 Production experience - data reliability
3.7 Production experience - data deduplication
3.7.1 Data transfer semantics
3.7.2 Idempotence
3.7.3 Producer transactions
3.8 Production experience - data in order
3.9 Production experience - data out of order
Chapter 4 Kafka Broker
4.1 Kafka Broker workflow
4.1.1 Kafka information stored by Zookeeper
4.1.2 Kafka Broker overall workflow
4.1.3 Broker important parameters
4.2 Production experience - node service and retirement
4.2.1 Serving new nodes
4.2.2 Retire old nodes
4.3 Kafka replica
4.3.1 Basic information about the copy
4.3.2 Leader election process
4.3.3 Leader and Follower fault handling details
4.3.4 Partition copy allocation
4.3.5 Production experience - manually adjust partition copy storage
4.3.6 Production experience - Leader Partition load balancing
4.3.7 Production experience - increasing the copy factor
4.4 File storage
4.4.1 File storage mechanism
4.4.2 File cleaning strategy
4.5 Efficiently read and write data
Chapter 5 Kafka Consumer
5.1 Kafka consumption method
5.2 Kafka consumer workflow
5.2.1 Overall consumer workflow
5.2.2 Principle of consumer group
5.2.3 Important parameters of consumers
5.3 Consumer API
5.3.1 Independent consumer case (subscription topic)
5.3.2 Independent consumer case (subscription partition)
5.3.3 Consumer group case
5.4 Production experience - partition allocation and rebalancing
5.4.1 Range and rebalancing
5.4.2 RoundRobin and rebalancing
5.4.3 Sticky and rebalancing
5.5 offset displacement
5.5.1 Default maintenance position of offset
5.5.2 Automatically submit offset
5.5.3 Submit offset manually
5.5.4 Specify Offset consumption
5.5.5 Specified time consumption
5.5.6 Missing consumption and repeated consumption
5.6 Production Experience – Consumer Affairs
5.7 Production experience - data backlog (how consumers can improve throughput)
Chapter 6 Kafka-Eagle Monitoring
6.1 MySQL environment preparation
6.2 Kafka environment preparation
6.3 Kafka-Eagle installation
6.4 Kafka-Eagle page operation
Chapter 7 Kafka-Kraft Pattern
7.1 Kafka-Kraft architecture
7.2 Kafka-Kraft cluster deployment
7.3 Kafka-Kraft cluster start and stop script
Kafka3.x external system integration
Chapter 1 Integrating Flume
1.1 Flume Producer
1.2 Flume Consumer
Chapter 2 Integrating Flink
2.1 Flink producer
2.2 Flink consumer
Chapter 3 Integrating SpringBoot
3.1 SpringBoot producer
3.2 SpringBoot consumer
Chapter 4 Integrating Spark
4.1 Spark producer
4.2 Spark consumer
Kafka3.x Production Tuning Manual
Chapter 1 Kafka Hardware Configuration Selection
1.1 Scenario description
1.2 Selection of number of servers
1.3 Disk selection
1.4 Memory selection
1.5 CPU selection
1.6 Network selection
Chapter 2 Kafka Producer
2.1 Kafka producer core parameter configuration
2.2 How producers improve throughput
2.3 Data reliability
2.4 Data deduplication
2.5 Data orderliness
2.6 Data out of order
Chapter 3 Kafka Broker
3.1 Broker core parameter configuration
3.2 Serving new nodes/retiring old nodes
3.3 Add partition
3.4 Increase the copy factor
3.5 Manually adjust partition copy storage
3.6 Leader Partition load balancing
3.7 Automatically create themes
Chapter 4 Kafka Consumer
4.1 Kafka consumer core parameter configuration
4.2 Consumer rebalancing
4.3 Specify Offset consumption
4.4 Consumption at designated time
4.5 Consumer Affairs
4.6 How consumers can improve throughput
Chapter 5 Kafka in general
5.1 How to improve throughput
5.2 Data is accurate once
5.3 Set the number of partitions appropriately
5.4 A single log is larger than 1m
5.5 The server is down
5.6 Cluster stress test
Kafka3.x source code analysis
Chapter 1 Source code environment preparation
1.1 Source code download address
1.2 Install JDK&Scala
1.3 Load source code
1.4 Install gradle
Chapter 2 Producer Source Code
2.1 Initialization
2.1.1 Program entry
2.1.2 Initialization of producer main thread
2.1.3 Producer sender thread initialization
2.2 Send data to buffer
2.2.1 Overall sending process
2.2.2 Partition selection
2.2.3 Sending message size verification
2.2.4 Memory pool
2.3 The sender thread sends data
Chapter 3 Consumer Source Code
3.1 Initialization
3.1.1 Program entry
3.1.2 Consumer initialization
3.2 Consumers subscribe to topics
3.3 Consumers pull and process data
3.3.1 Consumer/consumer group initialization
3.3.2 Pull data
3.3.3 Interceptor processing data
3.4 Consumer Offset submission
3.4.1 Submit Offset manually and synchronously
3.4.2 Manual asynchronous submission of Offset
Chapter 4 Server Source Code
4.1 Program entry