1.1 Definition

1.2 Message queue

1.3 Kafka infrastructure

2.1 Installation and deployment

2.2 Kafka command line operation

3.1 Producer message sending process

3.2 Asynchronous sending API

3.3 Synchronous sending API

3.4 Producer partition

3.5 Production experience - how producers can improve throughput

3.6 Production experience - data reliability

3.7 Production experience - data deduplication

3.8 Production experience - data in order

3.9 Production experience - data out of order

4.1 Kafka Broker workflow

4.2 Production experience - node service and retirement

4.3 Kafka replica

4.4 File storage

4.5 Efficiently read and write data

5.1 Kafka consumption method

5.2 Kafka consumer workflow

5.3 Consumer API

5.4 Production experience - partition allocation and rebalancing

5.5 offset displacement

5.6 Production Experience – Consumer Affairs

5.7 Production experience - data backlog (how consumers can improve throughput)

6.1 MySQL environment preparation

6.2 Kafka environment preparation

6.3 Kafka-Eagle installation

6.4 Kafka-Eagle page operation

7.1 Kafka-Kraft architecture

7.2 Kafka-Kraft cluster deployment

7.3 Kafka-Kraft cluster start and stop script

1.1 Flume Producer

1.2 Flume Consumer

2.1 Flink producer

2.2 Flink consumer

3.1 SpringBoot producer

3.2 SpringBoot consumer

4.1 Spark producer

4.2 Spark consumer

1.1 Scenario description

1.2 Selection of number of servers

1.3 Disk selection

1.4 Memory selection

1.5 CPU selection

1.6 Network selection

2.1 Kafka producer core parameter configuration

2.2 How producers improve throughput

2.3 Data reliability

2.4 Data deduplication

2.5 Data orderliness

2.6 Data out of order

3.1 Broker core parameter configuration

3.2 Serving new nodes/retiring old nodes

3.3 Add partition

3.4 Increase the copy factor

3.5 Manually adjust partition copy storage

3.6 Leader Partition load balancing

3.7 Automatically create themes

4.1 Kafka consumer core parameter configuration

4.2 Consumer rebalancing

4.3 Specify Offset consumption

4.4 Consumption at designated time

4.5 Consumer Affairs

4.6 How consumers can improve throughput

5.1 How to improve throughput

5.2 Data is accurate once

5.3 Set the number of partitions appropriately

5.4 A single log is larger than 1m

5.5 The server is down

5.6 Cluster stress test

1.1 Source code download address

1.2 Install JDK&Scala

1.3 Load source code

1.4 Install gradle

2.1 Initialization

2.2 Send data to buffer

2.3 The sender thread sends data

3.1 Initialization

3.2 Consumers subscribe to topics

3.3 Consumers pull and process data

3.4 Consumer Offset submission

4.1 Program entry

1.2.1 Application scenarios of traditional message queues

1.2.2 Two modes of message queue

2.1.1 Cluster planning

2.1.2 Cluster deployment

2.1.3 Cluster startup and shutdown scripts

2.2.1 Theme command line operation

2.2.2 Producer command line operation

2.2.3 Consumer command line operations

3.1.1 Sending principle

3.1.2 List of important parameters of the producer

3.2.1 Ordinary asynchronous sending

3.2.2 Asynchronous sending with callback function

3.4.1 Benefits of Partitioning

3.4.2 Partitioning strategy for producers to send messages

3.4.3 Custom partitioner

3.7.1 Data transfer semantics

3.7.2 Idempotence

3.7.3 Producer transactions

4.1.1 Kafka information stored by Zookeeper

4.1.2 Kafka Broker overall workflow

4.1.3 Broker important parameters

4.2.1 Serving new nodes

4.2.2 Retire old nodes

4.3.1 Basic information about the copy

4.3.2 Leader election process

4.3.3 Leader and Follower fault handling details

4.3.4 Partition copy allocation

4.3.5 Production experience - manually adjust partition copy storage

4.3.6 Production experience - Leader Partition load balancing

4.3.7 Production experience - increasing the copy factor

4.4.1 File storage mechanism

4.4.2 File cleaning strategy

5.2.1 Overall consumer workflow

5.2.2 Principle of consumer group

5.2.3 Important parameters of consumers

5.3.1 Independent consumer case (subscription topic)

5.3.2 Independent consumer case (subscription partition)

5.3.3 Consumer group case

5.4.1 Range and rebalancing

5.4.2 RoundRobin and rebalancing

5.4.3 Sticky and rebalancing

5.5.1 Default maintenance position of offset

5.5.2 Automatically submit offset

5.5.3 Submit offset manually

5.5.4 Specify Offset consumption

5.5.5 Specified time consumption

5.5.6 Missing consumption and repeated consumption

2.1.1 Program entry

2.1.2 Initialization of producer main thread

2.1.3 Producer sender thread initialization

2.2.1 Overall sending process

2.2.2 Partition selection

2.2.3 Sending message size verification

2.2.4 Memory pool

3.1.1 Program entry

3.1.2 Consumer initialization

3.3.1 Consumer/consumer group initialization

3.3.2 Pull data

3.3.3 Interceptor processing data

3.4.1 Submit Offset manually and synchronously

3.4.2 Manual asynchronous submission of Offset