MindMap Gallery Java architect p5-p7 advancement path with annual salary of 500,000-700,000
Java architect p5-p7 advancement path with an annual salary of 500,000-700,000, practical data structure and algorithm, microservice solution data structure model, JVM virtual machine practical performance tuning. Concurrent programming practice, microservice framework source code interpretation, collection framework source code interpretation, distributed architecture solutions. Internet distributed transaction architecture design and actual implementation, distributed message middleware principles, MySQL actual performance optimization, and Netty in-depth source code interpretation.
Edited at 2023-11-21 16:53:38This is a mind map about bacteria, and its main contents include: overview, morphology, types, structure, reproduction, distribution, application, and expansion. The summary is comprehensive and meticulous, suitable as review materials.
This is a mind map about plant asexual reproduction, and its main contents include: concept, spore reproduction, vegetative reproduction, tissue culture, and buds. The summary is comprehensive and meticulous, suitable as review materials.
This is a mind map about the reproductive development of animals, and its main contents include: insects, frogs, birds, sexual reproduction, and asexual reproduction. The summary is comprehensive and meticulous, suitable as review materials.
This is a mind map about bacteria, and its main contents include: overview, morphology, types, structure, reproduction, distribution, application, and expansion. The summary is comprehensive and meticulous, suitable as review materials.
This is a mind map about plant asexual reproduction, and its main contents include: concept, spore reproduction, vegetative reproduction, tissue culture, and buds. The summary is comprehensive and meticulous, suitable as review materials.
This is a mind map about the reproductive development of animals, and its main contents include: insects, frogs, birds, sexual reproduction, and asexual reproduction. The summary is comprehensive and meticulous, suitable as review materials.
The path to advancement as a JAVA architect
Data structure and algorithm practice
Mathematics knowledge review
index
logarithm
series
Modulo arithmetic
the complexity
time complexity
space complexity
data structure
stack model
First in, last out principle
Pop/Push
stack application
queue model
first in first out principle
sequential queue
chained queue
deque
priority queue
tree model
Binary tree
AVL tree
clue binary tree
red black tree
spreading tree
Hash model
hash function
Linear detection method
square detection method
double hash
Hash function method
direct addressing method
digital analytics
Square-Medium Method
folding method
division leaving remainder method
random number method
How to handle conflicts
Linear probing and then hashing
squared probing and hashing
heap model
Left-hand pile
skew pile
Algorithm practice
Sorting Algorithm
insertion sort
Hill sort
Heap sort
merge sort
Quick sort
Bubble Sort
Practical algorithm
greedy algorithm
Huffman coding
Approximate bin packing problem
divide and conquer algorithm
Divide and conquer algorithm running time
Recent issues
Select question
dynamic programming
Use a table instead of recursion
The order of matrix multiplication
optimal binary search tree
The shortest path for all pairs of points
randomization algorithm
random number generator
jump table
aptitude test
Microservice solution data structure model
Registration center cluster algorithm
Nacos cluster election Raft algorithm
Epidemic spread algorithm P2P-Gossip protocol
Eureka message broadcast consistency protocol
Distributed Transaction Consistency Protocol
Strong consistency protocol
Weak consistency protocol
eventual consistency protocol
Zookeeper consensus protocol algorithm
ZAB Atomic Broadcast Protocol
Paxos consensus protocol
Distributed cache elimination algorithm
LRU (less recently used) cache eviction algorithm
LFU (least frequently used algorithm) cache eviction algorithm
ARC (Adaptive Cache Replacement Algorithm) cache elimination algorithm
FIFO (first in, first out) cache eviction algorithm
MRU (most recently used) cache eviction algorithm
Distributed cache breakdown protocol
bloom filter
cuckoo filter
Load balancing algorithm
Polling mechanism algorithm
random algorithm
fixed ip algorithm
weight algorithm
Microservice current limiting algorithm
Counter (fixed window) algorithm
sliding window algorithm
leaky bucket algorithm
Token Bucket Algorithm
Distributed computing task scheduling algorithm (xxl-job)
rotation method
weighting method
Hashing
least join method
minimum missingness method
fastest response method
genetic algorithm
Ant colony algorithm solution
Distributed data synchronization and consistency protocol
MySQL
semisynchronous replication
Asynchronous replication
Global synchronization
cache
canal
otter
(openresty/tengine) Actual high concurrency
Native Nginx core module
Nginx module types
Standard HTTP module
Optional HTTP module
Third-party module (lua)
Nginx event-driven model and features
Nginx core configuration
Virtual host configuration
upstream
location
Static directory configuration
https protocol configuration
Nginx load balancing algorithm configuration
Polling mechanism algorithm
random algorithm
fixed ip algorithm
weight algorithm
openresty/tengine
Openresty combines with Lua to realize billion-level product details page
Tengine is a web server project developed by Alibaba
Practical performance tuning of JVM virtual machine
Programming language development history
Assembly language/machine language
C/C language
java/php/easy language/ython
JVM class loaderClassLoader
Analysis of class loader implementation principle from HotSpot virtual machine (C) source code
What is a class loader
Read our class file into memory
Classification of class loaders
Bootstrap class loader (written in C)
$JAVA_HOME/jre/lib
Extension class loader (written in Java)
JAVA_HOME/jre/lib/ext
(Application) AppClassLoader class loader (written in Java)
Class and jar packages under classpath
Custom class loader (written in java)
Custom class file directory
Benefits of the Parental Delegation Mechanism
The purpose is to prevent conflicts between the classes defined by developers and the source code classes defined by jdk, and to ensure the uniqueness of the class in memory.
Class loader parent delegation principle and how to break it
spi mechanism
Startup class/extension/application class loader in-depth source code analysis
Java SPIService Provider Interface principle
Tomcat/Jetty class loader implementation principle
How to customize a class loader to load bytecode and handwritten hot deployment plug-in
Local Disk
Network acquisition
database
Dynamic proxy generation
war/jar
Analysis of JVM memory structure principles
JVM runtime zone
program counter
Record the line number of the current thread executing code to facilitate the next CPU context switch to resume execution.
heap
new objects are stored in the heap (shared by threads)
Troubleshooting heap memory leaks in production environment
memory leak
What is a heap memory leak
Heap memory application space, GC cannot release the memory
Scenarios where memory leaks occur
threadlocal memory leak
hashMap custom key object memory leak
Troubleshooting heap memory leaks in production environment
Jps/Jmap
Jvisualvm/jconsole
memory overflow
What is heap memory overflow
When applying for space in the heap memory, there is not enough space.
Heap memory overflow
The amount of data loaded in the memory is too large, such as fetching too much data from the database at one time
There is a reference to the object in the collection class, which is not cleared after use, making the JVM unable to recycle it.
There is an infinite loop in the code or the loop generates too many duplicate object entities.
The startup parameter memory value sets the heap memory too small.
Memory overflow solution
Increase heap memory
Heap memory structure details
Heap structure division
New Generation/Old Generation
new generation
The newly created object is stored in the new generation
old age
Frequently used objects are stored in the old generation
JDK version
jdk1.7
eden (Garden of Eden) from (s0) to (s1) permanent generation (method area)
jdk1.8
eden (Garden of Eden) from (s0) to (s1) permanent generation (metaspace)
characteristic:
The default ratio between the new generation and the old generation is 1:2
-XX:NewRatio
Default Eden with s0/s1 8:1:1
-XX:SurvivorRatio
common words
YoungGen (new generation)
oldGen (old generation)
s0(from)
s1(to)
PermGen (permanent generation)
Metaspace
Classification of GC
partial collection
New generation collection (Minor GC/Young GC)
Old generation collection (Major GC/Old GC)
CMS GC will collect the old generation separately.
Heap and method area collection
(Full GC): Collect garbage collection of the entire java heap and method area (meta space)
Common interview questions
The difference between full gc and Minor GC/Young GC
Full gc will trigger new generation and old generation collection
Minor GC triggers new generation gc collection
Minor GC collection times are very frequent
full gc triggers recycling when there is insufficient memory in the old generation or method area (metaspace)
The GC recovery efficiency of full GC is very low
Try not to trigger full gc recycling
Where are large objects stored?
If the eden area cannot be saved, it will be directly stored in the old generation.
Analysis of GC logs
memory escape
Space guarantee
Analysis of internal structure of thread stack and stack frame
local variable table
operand stack
dynamic connection
method export
Local method stack (JNI calling C code) technology
String constant pool
Constant pool classification
static constant pool
Run constant pool
String constant pool
Constant pool to different JDK version storage areas
The string constant pool in JDK1.6 and before is stored in the method area (permanent generation)
In JDK1.7, the string constant pool begins to be stored in the heap
The string constant pool in JDK1.8 is stored in the heap
Common String interview questions
Why did JDK1.7 start to store the string constant pool in the heap?
In JDK7, the string constant pool is placed in the heap space. Because the recycling efficiency of the permanent generation is very low, the garbage collection of the permanent generation will only be executed during the Full GC, and the Full GC will be triggered when there is insufficient space in the old generation and the permanent generation is insufficient.
This results in a low recycling efficiency of the string constant pool. In our development, a large number of strings will be created, and the recycling efficiency is low, resulting in insufficient permanent generation memory. Putting it on the heap can reclaim memory in time.
String s1 = "mayikt";
String s1 = "mayikt"; s1 points to the string constant pool
new String("mayikt");
Create a heap space address in the heap and return it to s2 s2 obtains the heap memory address, and mayikt introduces the string constant pool in the String class.
JVM object memory layout
object layout
Object header
Instance data
Align padding
How many bytes does a new object occupy?
Object access location
handle mode
A piece of memory may be divided into the Java heap as a handle pool. What is stored in the reference in the local variable is the handle address of the object.
direct pointer method
What is stored in reference is directly the object address.
How objects are created
1.new object
2. Create objects using reflection
3. Use clone() to clone the object
4. Use deserialized objects
5. Third-party library Objenesls creates objects
Create object details
1. Check whether the object's class has been loaded, parsed and initialized
1. When the Java virtual machine encounters a bytecode new instruction, it will first check whether the parameters of this instruction can locate the symbolic reference of the class in the constant pool (in metaspace), and check the class represented by this symbolic reference. Whether it has been loaded, parsed and initialized (to determine whether the class has been loaded by the metaspace)
If not, the corresponding class loading process must be executed first (then use the parent delegation mechanism to find the class (full path of the class), if not found, an error will be reported: ClassNotFoundException).
2. After the class loading check passes, the virtual machine will allocate memory for the new object.
Use if the heap memory is regular
pointer collision
Use if the heap memory is irregular
free list
3. Object creation is a very frequent behavior in a virtual machine. Even if you only modify the location pointed by a pointer, it is not thread-safe under concurrent circumstances.
1. Use CAS failure retry to ensure the atomicity of updates;
2. Allocate a piece of TLAB space unique to the current thread in Eden (Eden Campus) for each thread.
4. Set the default value of the attribute. After the memory allocation is completed, the virtual machine must initialize the allocated memory space (but not including the object header) to zero. If TLAB is used, this work can also be allocated in advance to TLAB. By the way. This operation ensures that the instance fields of the object can be used directly in Java code without assigning initial values, so that the program can access the zero values corresponding to the data types of these fields.
5. Set the object header information. The Java virtual machine also needs to make necessary settings for the object, such as which class the object is an instance of, how to find the metadata information of the class, and the hash code of the object (actually the hash code of the object It will be delayed until the Object::hashCode() method is actually called), the GC generation age of the object and other information. This information is stored in the object header (Object Header) of the object. Depending on the current running status of the virtual machine, such as whether bias locking is enabled, etc., the object header will be set in different ways.
6. Execute the init method for initialization
JVM garbage collection algorithm
Garbage collection algorithm
Mark and sweep algorithm
Advantages: high efficiency/no moving memory addresses
Disadvantages: prone to fragmentation
tag sorting algorithm
Advantages: avoids fragmentation problems
Disadvantages: moving memory address, low efficiency
Marked copy algorithm
Advantages: avoids fragmentation problems
Disadvantages: moving memory address, high efficiency
Principle of generational algorithm
new generation
old age
JDK1.0-14 collector
Garbage collector combination
(Parallel in the new generation) ParNew (serial in the old generation) Serial old
(New generation serial) Serial, (Old generation) Serial old
(New generation serial) Serial/CMS (Old generation concurrency)
(JDK9 obsolete)
(Concurrency in the new generation) ParNew/CMS (Concurrency in the old generation)
(Not recommended for JDK9)
(New generation parallel) Parallel/SerialOld (Old generation serial)
Not recommended
(Parallel in the new generation) Parallel Scavenge/Parallel Old (parallel in the old generation)
JDK8 default
G1 (JDK9 default) entire heap collection
JDK8 default
Serial (single GC thread recycling)
Detailed explanation of Serial garbage collector
combination
New Generation (Serial)-Marked Replication Algorithm
Serial old-mark compression algorithm
Application scenarios
Use single-core CPU and small memory
Parallel (multiple GC threads recycling)
Detailed explanation of ParNew garbage collector
Application scenarios
new generation
Suitable for multi-core CPUs
combination
(Old generation) Serial old (mark compression algorithm)
(Old Generation) CMS (Mark and Sweep Algorithm)
Detailed explanation of Parallel garbage collector
Application scenarios
New generation/old generation multiple GC collections at the same time
combination
Parallel
Parallel oldGC
Concurrency (GC and user threads execute simultaneously)
CMS garbage collector principle (key points)
Implementation principle
initial mark
concurrent marking
relabel
Concurrent clear
What are the shortcomings of the CMS collector?
Mark-and-sweep algorithm causes fragmentation problem
Alternative SerialOld is inefficient
Solve the problem of missing labels based on incremental update method
G1 collector principle (key points)
Implementation principle
n multiple different regions
Region memory set (Remember Set)
Detailed explanation of Cardtable
Solve the building marking problem based on original snapshot (satb)
Detailed explanation of ZGC garbage collector
Detailed explanation of Epsilon and Shenandoah garbage collectors
garbage copy algorithm
Principle of three-color marking algorithm of garbage collection mechanism
Multiple labeling/missing labeling problems caused by three-color marking algorithm
floating garbage
Missing bid problem
Object missing label solution
Based on incremental update method (CMS)
Based on original snapshot method satb(G1)
Detailed explanation of Region memory set (Remember Set) and card table (Cardtable)
stop the world
Why does the stop the world problem occur?
How to avoid the stop the world problem
Detailed explanation of JVM tuning tools
Detailed explanation of JDK's own Jstat, Jinfo, Jmap, Jhat and Jstack tuning commands
Detailed explanation of Jvisualvm and Jconsole tuning tools
Detailed explanation of the use of Alibaba tool arthas
GC log analysis tool
How to read GC log details
Use of GCEasy log analysis tool
Use of GCViewer log analysis tool
JVM parameter tuning practice
100 million level traffic project heap memory young generation and old generation garbage collection parameter setting and tuning
Online production environment OMM memory overflow monitoring tool and positioning solution
Optimization practices on how to reduce severe Full GC in online production environments that lead to direct system freezes
High concurrency system, how to avoid frequent GC operations in online production environment
High concurrency system, how to optimize G1/CMS collector
How to set the initial memory size of the JVM heap for the average daily million PV service
Concurrent programming in practice
Operating system basics
Switching process between user mode and kernel mode
Linux process model management
Linux inter-process communication principle
Linux network communication principles
Multithreading basics
Quick Start with Multithreading
What is a process/thread
Process is the smallest unit of resource allocation
Thread is the smallest unit of program execution
Multi-threaded application scenarios
Client (mobile App/) development
Send SMS/Send Email Asynchronously
Change code that takes time to execute to multi-threaded asynchronous execution.
Write log asynchronously
Multi-threaded download
Difference Between Multithreading and Single Threading
Multithreading (parallel execution)
Single thread (synchronous execution)
How to understand the concept of multi-threaded cpu switching
The current CPU switches to execute another thread
Is it true that the more multi-threading is enabled, the better?
Not necessarily. Opening too many threads can easily cause CPU context switching.
The difference between user threads and daemon threads
How to gracefully stop a thread
Analysis of seven states of multithreading
initial state
ready state
Operating status
<font face="宋体"><span style="font-size: 14px;">Death state</span></font><br>
blocking state
timeout wait
waiting state
Five ways to create multithreading
Inherit the Thread class to create a thread
Implement the Runnable interface to create a thread
Create threads using Callable and Future
Use a thread pool such as the Executor framework
Create a thread using the @Async asynchronous annotation
Create a thread using lambda expression
Multi-threaded thread safety
What is thread safety issue
When multiple threads write to the same global variable at the same time, they may be interfered by other<br>threads, which may lead to thread safety issues.
The difference between lock and Synchronized lock
lock Manually acquire and release locks
Synchronized automatically acquires and releases locks
SynchronizedJDK6 starts the automatic lock upgrade process
The underlying implementation of the lock lock is based on the aqs lock and requires manual upgrade of the lock.
Multi-thread deadlock problem
How to troubleshoot deadlock with multi-threading
jconsole.exe diagnoses deadlock
Causes of multi-thread deadlock threads
Nested sync within sync
How to solve thread safety issues (how to ensure thread synchronization)
Synchronized solves thread safety issues
synchronized synchronization code is fast
synchronized modified instance method
synchronized modified static method
Lock lock solves thread safety problem
Communication between multiple threads
wait and notify
wait releases the current lock and blocks the current thread
notify wakes up the blocked thread
The principle of join method
The bottom layer is based on wait encapsulation
synchronized principle
How an object is composed
Object header
Mark Word
Hash code (HashCode), GC generation age, lock status flag, lock held by thread, biased thread ID, biased timestamp
Klass Pointer<br>
Instance data
member properties
Align padding
The size of the object must be an integer multiple of 8 bytes. If it is not an integer multiple of 8 bytes, it will be aligned and padded.
Synchronized lock upgrade process
bias lock
Locking and unlocking do not require additional overhead. They are only suitable for the same thread to access the synchronized code block without additional overhead. If multiple threads compete at the same time, the lock will be revoked.
Lightweight lock
Competing threads will not block, which improves program response speed. If the competing thread for the lock is never obtained, spin will be used to consume CPU resources. It is suitable for situations where synchronized code blocks execute very quickly. Spin (jdk1 .Intelligent rotation after 7)
weight lock
Thread competition does not use spin, does not consume CPU resources, and is suitable for synchronous code execution for a relatively long time.
Analysis on the principle of synchronized lock expansion process
Bias lock(101)
1. The current thread obtains whether it is a biased lock from the markword in the object header. If it is a biased lock, determine whether the thread's id=== the current thread id.
2. If it is equal to the current thread id, the CAS operation will not be performed repeatedly, but will directly enter our synchronization code block.
3. If it is not equal to the current thread id, if it is a lock-free situation and there are no other threads competing with me, directly use CAS to modify the lock status in markword to 101 and also store the current thread id in markword.
4. Other threads start to compete with the bias lock thread. If the number of bias lock cancellations reaches 20, they will directly redirect the bias lock to T2 thread in batches (note: there are no other threads competing with t2). If the number of bias lock cancellations reaches 20, 40 times, batch undoing will begin later.
5. To cancel the biased lock, we need to stop our biased lock thread at a global safe point, modify our markword to a lightweight lock, and wake up the biased lock thread.
6. Note: JDK15 turns off biased lock optimization by default.
Lightweight lock(000)
1. If multiple threads compete for the same lock at the same time, upgrade the lightweight lock and use CAS (modify markword lock status = 00). If successful, replace it with markword and store the HashCode value directly in our stack frame. Currently, The lock record address is stored in markword
2. When we use a lightweight lock to release the lock, the markword value content is restored
Heavyweight (010) (C Monitor)
1. If our thread has retried multiple times and still has not acquired the lock, the current lock will be upgraded to a heavyweight lock.
2.2. Threads that have not acquired the lock will be stored in the C Monitor object EntryList collection. At the same time, the current thread will directly block and release the cpu execution rights. The process cost of waking up and re-entering the competition lock in the later period is very high because cpu context switching needs to occur. Switch from user mode to kernel and change the markword value in our object header to C Monitor. The memory address pointer Java object is associated with C Monitor.
C Monitor monitor lock (heavyweight lock)
Monitor (object heavyweight lock)
recursions (number of recursions/number of reentries)
owner (records the thread ID currently holding the lock)
waitSet (threads waiting for the pool to be in wait state will be added to _WaitSet)
entryList (lock pool: threads waiting for the lock block state will be added to the list)
When the thread in the waiting pool is awakened, will the lock be acquired immediately?
After the thread in the waiting pool is awakened, the waiting pool is transferred to the lock pool and queued to compete for the lock again.
Lock coarsening, elimination and performance optimization
Roughening of locks
Each thread holds the lock for as short a time as possible
Elimination of locks
Lock elimination is a lock optimization method that occurs at the compiler level
volatile keyword principle
keyword properties
Ensure visibility
Disable reordering
No guarantee of atomicity
java memory model
Analysis of CPU multi-core hardware architecture
jmm eight synchronization specifications
Volatile Cache Coherence Protocol
bus lock
MESI protocol
The problem of false sharing
Basic concepts of cache lines
Bank filling plan
Why do you need to add volatile to reordering/memory barrier/double check lock?
The difference between synchronized and volatile
Why Volatile doesn't guarantee atomicity
Classification of concurrent locks
pessimistic lock
optimistic locking
spin lock
Reentrancy lock
fair lock
unfair lock
aqs source code interpretation
LockSupport source code interpretation
AbstractQueuedSynchronizer source code interpretation
ReentrantLock/ReentrantReadWriteLock, ReadWriteLock source code interpretation
Semaphore/CountDownLatch/CyclicBarrie source code interpretation
How to implement the bottom layer of AQS
A.Cas ensures the thread safety of AQS
B. Doubly linked list stores blocked threads
C.LockSupport blocks and wakes up threads
core attributes
State state value
Node status(waitStatus)
-2 The current thread blocks and releases the lock at the same time
-1 wakes up the thread of subsequent nodes<br>
AQS application scenarios
The underlying implementation of lock lock
Fair lock and unfair lock
The default is unfair lock
Contention lock time
Fair lock: When competing for a lock, if the lock is already held by another thread, it will be directly stored at the end of the doubly linked list.
Unfair lock: When competing for a lock, if the lock is already held by another thread, CAS will still be tried again.
Core design principles
lockmethod()
Use CAS to modify the status value in the AQS class from 0 to 1
If the change is successful (the lock is acquired successfully)
If the change fails (failed to obtain the lock)
Stored in AQS doubly linked list
unlockmethod()
Use CAS to modify the status in the AQS class from 0 to 1
If CAS is successful
Need to wake up the thread cached by the next node node of the head node in the AQS doubly linked list
Question: Why does unlock release the lock and only wake up one thread instead of multiple threads?
The bottom layer uses a two-way linked list to store blocked threads. The cost of waking up all threads is very high.
There is no lock upgrade process in the lock, and developers need to expand it themselves.
ConcurrentHashMap1.7 source code underlying Lock
Condition
Note: Condition and lock do not use the same doubly linked list
Lock pool and wait pool
Lock pool: refers to the threads in the lock that failed to compete for the lock and are stored in the AQS doubly linked list.
Waiting pool: refers to the current thread calling the await method, actively releasing the lock (aqs status value = 0), and storing it in the waiting pool doubly linked list
How to wake up the thread in the waiting pool?
Calling this signal() only transfers the node nodes (threads) in the waiting pool to the AQS doubly linked list lock pool to compete for the lock resources again.
After calling this signal() and calling the unlock method, the threads in the wake-up lock pool begin to compete for lock resources.
CountDownLatch(counter)
Through the constructor new CountDownLatch(1)
The underlying implementation based on AQS sets the status value in the AQS class to 1
await method
Store the current thread block in the AQS doubly linked list
countDown
Operate the status value -1 in the AQS class. If the AQS status is changed to 0, wake up the thread stored in the doubly linked list in the AQS class.
Semaphore(Semaphore)
Through the constructor new Semaphore(3)
The underlying implementation based on AQS sets the status value in the AQS class to 3
The underlying operation of acquire is to modify the status value -1 in the AQS class. If the status value in the AQS class is changed to 0, the current thread needs to be blocked and stored in the doubly linked list in the AQS class.
The bottom layer of release operates on the status 1 in the AQS class<br>
Wake up blocked threads in the AQS class at the same time (only one will be woken up)
CyclicBarrier (synchronization barrier)
Through constructor new CyclicBarrier(2)<br>
The bottom layer assigns a value = 2 to the CyclicBarrier count attribute
Call await bottom layer
Operation on CyclicBarrier count-1
count=0
Continue execution and wake up all threads in the waiting pool
count!=0
Stored in waiting pool
Concurrent atomic operations
Atomic class
CAS (optimistic locking) principle
Detailed explanation of Unsafe magic class
Blocking QueueBlockingQueue principle
Blocking queue classification
ArrayBlockingQueue array bounded queue
ConcurrentLinkedQueue linked list bounded queue
PriorityBlockingQueue priority sorting unbounded queue
DelayQueue delay unbounded queue
Framework application
Handwritten thread pool based on BlockingQueue
Handwritten message middleware based on BlockingQueue
Handwritten log framework based on BlockingQueue
Executor thread pool detailed explanation and core source code analysis
Why use thread pool
Reusability
Unified management
Improve response
Four ways to create a thread pool
newCachedThreadPool(); Cacheable thread pool
newFixedThreadPool(); can be fixed in length and limit the maximum number of threads
newScheduledThreadPool(); can be scheduled
newSingleThreadExecutor(); singleton
The real bottom layer is based on ThreadPoolExecutor constructor encapsulation thread pool
Analysis of core principles of thread pool
LinkedBlockingQueue
producer and consumer model
Why Alibaba does not recommend using Executors
The underlying use of LinkedBlockingQueue unbounded queue is prone to memory overflow
What to do if the thread pool queue is full
Deny policy
AbortPolicy discards tasks and throws runtime exceptions
CallerRunsPolicy execution task
DiscardPolicy Ignore, nothing will happen
DiscardOldestPolicy kicks from the queue and enters the queue first.
Implement the RejectedExecutionHandler interface and customize the processor
How to properly configure thread pool parameters
CPU intensive
Optimal number of threads = number of cpu cores or number of cpu cores ±1
Intensive
Optimal number of threads = ((thread waiting time thread CPU time)/thread CPU time) * number of CPUs
How to customize the thread pool based on ThreadPoolExecutor
FutureTask source code interpretation
Implementing FutureTask based on LockSupport
Implementing FutureTask based on Wait/Notify
ForkJoin source code interpretation
Concurrent programming development
work stealing mechanism
Fork Join principle
Threadlocal source code interpretation
What is Threadlocal
Threadlocal application scenarios
1.Spring transaction template class
2. Get httprequest
3.Aop call chain passes parameters
The difference between Threadlocal and Synchronized
How to prevent Threadlocal memory leaks
1. Call the remove method
2. When using the set method, it will be cleared that the previous key was null.
Microservice framework source code interpretation
SpringBoot source code interpretation
How SpringBoot automatic configuration works
SpringBoot core module source code interpretation
SpringBoot core annotation source code interpretation
SpringBoot embedded Servlet container source code interpretation
SpringBoot packaging deployment and operation and maintenance management
How Servlet Containers Overcome Webflux
SpringCloudNetfilix (first generation) core component source code interpretation
Eureka service registration and discovery source code interpretation
Eureka service renewal (heartbeat)
Eureka server-side service elimination
Eureka service self-protection mechanism
What if the service does go down?
Local services use a retry mechanism
Local service implements address failover
Eureka service offline notification
Eureka cluster data synchronization
Fegin declarative service call source code interpretation
Hystrix implements service current limiting, downgrading, and circuit breaker source code interpretation
Detailed explanation of Zuul unified gateway, service routing, filter source code interpretation
Config distributed configuration center source code interpretation
Sleuth distributed link tracking source code interpretation
Ribbon client load balancing detailed explanation and source code analysis
SpringCloudAlibaba (second generation) source code interpretation
Nacos distributed registration center source code interpretation
Service registration and discovery
Nacos service registration and discovery source code interpretation
Service registration principle
eurekaClient
Use jerseyClient to send a request to register
eurekaServer side
Use ConcurrentHashMap to cache interface addresses
key is the service name
value is the cache interface address
Naocs service heartbeat detection and renewal source code interpretation
By default, EurekaClient sends a heartbeat renewal package every 30s to extend the time and tell EurekaServer that I am still alive.
By default, the EurekaServer side searches for expired addresses in the cache address every 60s, stores them in a new collection, and uses a random algorithm to clear them.
Naocs service offline and health check source code interpretation
Nacos cluster Raft election algorithm source code interpretation
Nacos server long polling processing mechanism
Principle of data synchronization between Nacos cluster nodes
Interpretation of AP mode source code in Nacos
Interpretation of CP mode source code in Nacos
Nacos cluster split-brain solution
Distributed configuration center
Implementation principle of Nacos distributed configuration center
How to dynamically refresh configuration files in Nacos
Gateway New generation microservice gateway source code interpretation
Why is Gateway performance better than Zuul performance?
Gateway dynamic routing source code analysis
Specify time rule matching route
cookie matching route
Header matching route
Host matching route
Request method matching route
Request path matching route
Gateway filter source code analysis
Custom GatewayFilter
Gateway integrates Nacos to achieve load balancing
Gateway integrates Sentinel to implement gateway current limiting
Seata distributed transaction framework source code interpretation
Interpretation of the source code of Seata's three core components to solve distributed transactions
Seata reversely generates sql statement based on undo_log table and rolls back source code interpretation
Seata branch transaction global lock design source code interpretation
GlobalTransactionalInterceptor source code interpretation
How TM remotely connects to TC to obtain the global transaction ID source code interpretation
Seata front and rear mirror source code in-depth source code interpretation
What are the differences between Seata and LCN rollback?
Canal distributed data synchronization framework source code interpretation
Canal overall architecture source code interpretation
MySQL master-slave replication principle architecture analysis
How Canal pretends to subscribe to BinLog files from nodes
Design principles of EventParser and EventSink
Canal incremental subscription/consumption design principle
Canal high concurrent data synchronization performance optimization
How to reduce the latency of synchronization between data
How to avoid data synchronization message sequence consistency
How to avoid data synchronization loss problem
Sentinel
service isolation
Current limiting algorithm
token bucket
leaky bucket
Implemented based on semaphore Semaphore
Fixed counting window
sliding technology window
Flow control rules
Threads
Implemented based on semaphore Semaphore
QPS
token bucket
Hystrix
Flow control rules
Semaphore isolation
In Sentinel
QPS
token bucket
Thread pool isolation
Defects consume CPU resources
Current limiting method
Google Guava(RateLimiter)
Alibaba Sentinel
Nginx
Redis lua implements current limiting
It is recommended that current limiting be done on nginx or gateway
Collection framework source code interpretation
Hash (hash function) Map collection framework source code interpretation
basic knowledge
The difference between == and equals and the underlying implementation
Why rewrite equals and also rewrite hashcode
Binary and decimal conversion/^ (exclusive OR operation)/>>> (unsigned right shift)/ & (AND operation)
Low-level implementation
JDK1.7
Array linked list
Head insertion method (concurrent expansion infinite loop problem)
Code writing is simple
JDK1.8
Array linked list red black tree
tail insertion method
High-end code writing
Red-black tree conversion
(Array capacity >= 64 & linked list length greater than 8)
Number of red-black tree nodes <6 conversion linked list
Hash function calculation
(h = key.hashCode()) ^ (h >>> 16)
i = (n - 1) & hash
time complexity
Key does not conflict
The time complexity is O(1)
Key conflict
Linked list storage is O(N)
Red-black tree storage is O(LogN)
hashcode collision problem
The hashcode values are the same but the content values are different.
Features:
The key is null to store the 0 position of the array.
Implemented using a one-way linked list
Unordered hash storage
Performance optimization
How to avoid memory leak problem with HashMap
How HashMap reduces the probability of Hash conflicts
How to reasonably specify the initial value size of the collection in HashMap
HashMap common interview questions
Why should we rewrite the HashCode method when rewriting Equals?
How HashMap avoids memory leak problems
How is the bottom layer of HashMap1.7 implemented?
Where is the HashMapKey stored when it is null?
How is the bottom layer of HashMap1.7 implemented?
Where is the HashMapKey stored when it is null?
How HashMap solves Hash conflict problem
How HashMap implements array expansion problem
Does the bottom layer of HashMap use a single linked list or a double linked list?
Time complexity of HashMap query based on key
What are the differences between HashMap1.7 and 1.8?
How to avoid the infinite loop problem of multi-thread expansion in HashMap1.8
Why does HashMap1.8 need to introduce red-black trees?
Why is the loading factor 0.75 instead of 1
How does the bottom layer of HashMap reduce the probability of Hash conflicts?
How does HashMap store 10,000 keys most efficiently?
What are the benefits of HashMap's high and low bits and modulo operations?
Why not just use the key as a hash value but do an XOR operation with the high 16 bits?
How is the hash function implemented in HashMap?
Is the bottom layer of HashMap stored in order? <br>
How do the underlying layers of LinkedHashMap and TreeMap achieve ordering?
How to use HashMap in high concurrency situations
The principle of underlying implementation of ConcurrentHashMap
ConcurrentHashMap collection framework source code interpretation
basic knowledge
synchronized and lock
How to understand the segmentation lock concept
CAS algorithm and volatile
underlying implementation
JDK1.7
data structure
Array Segments segmentation lock HashEntry linked list implementation
Lock implementation
Lock lock CAS optimistic lock UNSAFE class
Capacity expansion implementation
Supports simultaneous expansion of multiple Segments
Jdk1.8
data structure
Use Node array directly to save data
Array linked list red black tree
Lock implementation
Cancel segment segmentation design
Index does not conflict and uses cas lock
Index conflicts use synchronized
Capacity expansion implementation
Support concurrent expansion
Features: Supports multi-threading, high efficiency, divided into 16 Segments by default
Features:
ConcurrentHashMap does not support null key
List collection source code analysis
Arraylist underlying implementation
data structure
array
time complexity
Subscript query time complexity o(1)
Expansion
The expansion is 1.5 times the original
Not thread safe
Advantages and Disadvantages
The addition and deletion efficiency is low and requires expansion, but the query efficiency is relatively high.
Vector underlying implementation
data structure
array
time complexity
Subscript query time complexity o(1)
Expansion
The expansion is 2 times the original
Thread safety
LinkedList underlying implementation
Based on linked list data structure, subscript query time complexity ()
data structure
linked list
time complexity
Subscript query time complexity log2(n) binary search
Not thread safe
Advantages and Disadvantages
The efficiency of addition, deletion and modification is high, but the query efficiency is relatively low.
Distributed architecture solutions
Internet microservice idempotent architecture design and practice
What is idempotent design? Idempotent production background
Client response timeout
When the business execution time is very long, it is recommended to use mq asynchronous instead.
retry policy
Solve the core reasons for the idempotent problem of service interfaces
database level
Analyze the reasons for idempotence from the architectural level
gateway layer
interface layer
Query in advance whether the business logic has been executed based on the global ID
DB layer
insert type unique annotation constraint
update type optimistic locking mechanism
Does global id really guarantee interface idempotence?
Not necessarily, you need to consider the database level
How to ensure the idempotence of RPC interfaces
global id
Locking mechanism (not recommended due to low efficiency)
Implement idempotent design based on real business scenarios
Internet distributed lock architecture design and practice
What scenarios use distributed locks?
The problem of ensuring idempotence of scheduled task scheduling
Guaranteed flash sales to prevent overselling issues
The essential implementation principle of distributed locks
Retry strategy
Suitable for business execution very fast
Timeout control
Life extension design
How to avoid deadlock problems when continuing life
Performance optimization
Consider the herd effect
High availability
Set blocking timeout
Lock granularity reduced
fairness
Distributed lock compatibility testing and recovery design
Distributed lock implementation solution
Zookeeper implements distributed locks (CP mode)
main idea
Temporary node
path uniqueness
watcher event
Features
Use cp mode
advantage
Congenital solution to split-brain problem (more than half mechanism)
More reliable and stable
shortcoming
Cluster synchronization data efficiency is low
Low performance
Two implementation methods
Implemented based on the same temporary node
Herd effect may occur
Implemented based on temporary sequential nodes
Avoid herding problems
Temporary sequential number node
If the current node is the smallest node, it means that the lock is acquired successfully.
If the current node is not the smallest node, subscribe to the previous node
Curator framework distributed lock
Get lock
Implementation based on temporary sequential numbering nodes
Create a temporary sequence node, and whoever is the smallest will acquire the lock.
block
The current temporary sequence node created by yourself is not the smallest, subscribe to the previous node
release lock
Delete the temporary sequential number node
Interview difficulties
How does the zk master node go down and what are the impacts?
ZK re-election occurs, and the entire ZK environment is temporarily unavailable.
Need to consider zab protocol
Compare zxid first, then compare myid
How to avoid zk client deadlock problem
zkserver is down
zk client sets blocking timeout
After monitoring zk downtime, it is actively woken up.
zk client is down
The innate characteristics of zk avoid deadlock problems and proactively release locks
Other zk clients set blocking timeout
The jvm that acquired the lock never releases the lock.
Control the number of life extensions, actively release the lock after multiple life extensions, and the transaction will be rolled back
Redis implements distributed lock (AP mode)
main idea
Setnx implements distributed lock
Get lock
Multiple jvms setnx at the same time. In the end, only one jvm succeeds
release lock
Delete the key
Features
Use AP mode
advantage
Support high concurrency
Efficiency is okay
Using asynchronous synchronization data method
shortcoming
Congenital split-brain problem (Redis cluster does not have a majority mechanism)
Not very stable
Redis's congenital single thread can ensure setnx thread safety issues
Redisson framework
Get lock
Create hash key using lua script
release lock
Delete the key
Life extension design
By default, the watchdog thread is activated every 10 seconds to prevent key expiration.
Interview difficulties
How to avoid client deadlock problems
Set expired key
Limit the number of life extensions
Rollback transaction
The key has expired, but the business has not been executed yet. What should I do?
Life extension design
Global life extension (not recommended)
Incremental life extension (recommended)
By default, life is renewed in advance every 10 seconds to prevent key expiration.
Renew life multiple times to limit the number of life renewals due to deadlock issues
Rollback transaction
Actively release lock
In redis cluster, what should I do if the master node is down? <br>
Use RedLock red lock algorithm
There is no distinction between master and slave in redis cluster
The client meets more than half of the setnx requirements for multiple redis servers and successfully acquires the lock.
If the client sets the total time taken to obtain the lock > the key expiration time, the lock will be automatically released.
This algorithm actually uses ZK to implement distributed locks.
Internet distributed transaction architecture design and practice
The background of distributed transactions
Single project multiple data sources
jta Atomikos solves distributed transactions
RPC remote calling interface
Distributed Transaction Consistency Protocol
Strong consistency protocol
In the cluster, the copy data of each node must be consistent
Weak consistency protocol
In the cluster, some node replicas are allowed to have inconsistent data
eventual consistency protocol
Short data delays are allowed, but eventually data consistency is required
Distributed transaction design ideas
Base and CAP theory
Base theory
Basically available<br>
soft state
eventual consistency
CAP
Basic theory
Consistency(C)
In the cluster, the copy data of each node must be consistent
Availability(A)
After some nodes in the cluster fail, can the entire cluster still respond to the client's read and write requests?
Partition Tolerance (P)
Partition fault tolerance (P) mainly represents errors caused by network fluctuations, which are inevitable, and these three modes cannot be achieved simultaneously, so there are currently only two modes: CP and AP modes
Mode selection
CP (guaranteed data consistency)
Availability not guaranteed
Ap (guaranteed availability)
However, data consistency of each copy cannot be guaranteed<br>
Comparative analysis
Nacos
Nacos supports CP/AP mixed mode clusters starting from version 1.0. By default, Ap mode
Eureka
Ap mode
Zookeeper
Cp mode
The registration center recommends Ap mode
Flexible and rigid affairs
Rigid transactions satisfy ACID theory
Flexible transactions satisfy BASE theory (basically available, eventually consistent)
2PC/3PC/TCC<br>
Distributed transaction resolution framework
LCN solves distributed transaction problems
Seata solves distributed transaction problems
MQ solves distributed transaction problems
TCC solves distributed transaction problems
Retry callbacks to solve distributed transaction problems<br>
Internet distributed task scheduling architecture design and practice
Disadvantages of traditional scheduled tasks
Consume cpu resources
Non-decoupling affects business logic
Core design ideas of distributed task scheduling
Log traceability
Flexible expansion and contraction
Support parallel scheduling
High availability strategy
Failure handling strategy
Dynamic sharding strategy
Distributed task scheduling framework
XXLJob source code interpretation
ElasticJob source code interpretation
mysql and Redis data consistency protocol
solution
1. Update mysql data and manually clear the Redis cache
Advantages: low latency
Disadvantages: Not decoupled
2. Update mysql data and synchronize data to Redis in the form of mq asynchronous
Advantages: Implement decoupling, retry compensation strategy, improve interface response
Disadvantages: high latency
3. Synchronize data to Redis based on subscribing to MySQLBinLog combined with mq asynchronous form (canal framework implementation)
Advantages: More decoupling, retry compensation strategy, improved interface response
Disadvantages: Latency is getting higher and higher
Core design ideas
The idea of eventual consistency is that temporary data inconsistency is allowed, but the final data must be consistent.
Principle of double-write consistency protocol
Delete the cache first and then update the db (not recommended)
Need to delay double deletion, delete twice
Delete the cache for the first time
The second time to avoid the concurrency of other threads, synchronize the dirty read data to Redis, so delay for a few seconds (the time when the second thread business updates the old data to Redis) to delete the cache
Update the db first, then delete the cache
It needs to be combined with mq and ensure message sequence consistency. Deleting the cache must be successful.
DoubleWrite Consistency Protocol
What is double writing
Update db first, then update cache
What problems will arise
In a concurrent situation, multiple threads write at the same time. Another thread may write dirty read data into the cache.
How to solve dirty reads
Using the mysql transaction row lock mechanism, multiple threads write operations at the same time. In the end, only one thread acquires the row lock. After acquiring the row lock, Redis must be synchronized successfully before the row lock can be released. Only other threads can write
Implemented using distributed locks (not recommended, can be implemented directly based on mysql row locks)
Canal frame design principle
canal solves the principle of data synchronization between mysql and Redis
1.canal disguises itself as a mysql slave node and subscribes to the binlog file of the mysql master node;
2. When the binlog file of our mysql master node changes, the binlog file<br> is sent to the canal server;
3. The canal server converts the binlog file into json format and sends it to the canal client;
4. The canal client synchronizes the data to Redis/ES;
synchronization mode of canal
tcp (low efficiency)
kafka (recommended)
How to improve the efficiency of canal integrating kafka data synchronization
Integrate MQ topic mode
Single topic and single partition (global binlog strict order)
Multi-topic single partition can ensure table-level ordering.
The same table name is placed in the same partition and eventually consumed by the same consumer.
Single topic, multi-topic multi-partition combined with hash mode
How to solve the message sequence consistency problem
Set up multiple partitions and calculate hash based on the fields in the table such as primary key id. The same id will be placed in the same partition and consumed by the same consumer.
Is there any delay in synchronizing data between mysql and redis?
During the data synchronization process, there will be a short delay, which is normal. It is difficult to achieve strong consistency and follow the idea of eventual consistency.
Distributed message middleware principle
MQ basic conceptual model
Synchronous and asynchronous
Traffic peak clipping
Scalability/decoupling
Buffering/recoverability
producers and consumers
MQ common solutions
How MQ avoids message accumulation
Increase consumer rate (cluster)
Consumers get messages in batches
How does MQ avoid repeated consumption by consumers (idempotent problem)
Global ID business scenario ensures uniqueness
How does MQ ensure that messages are not lost?
Message confirmation mechanism
Persistence
message ack
How MQ ensures message sequence consistency
Bind the same consumer and queue
MQ push and pull architecture model
How producers obtain consumption results
Asynchronously returns a global ID, and the front end uses ajax to actively query regularly.
Mainstream MQ framework
rabbitmq
Architectural Thoughts
Rabbitmq management platform center
Virtual Hosts
Store message queues in separate team development paths
Exchange
route distribution message
routing key
RabitMQ queue model
simple mode<br>
work mode
Broadcast mode---fanout
Routing mode --direct
Theme mode--topic
RabitMQ four switch types
Direct exchange (direct switch)
Fanout exchange
Topic exchange
Headers exchange
RabbitMQ common interview questions
How RabbitMQ ensures messages are not lost
producer
Ensure that the producer successfully delivers the message to the MQ server
Ack message confirmation mechanism (Confirms)
Synchronous or asynchronous form
Transaction messages
consumer
Consumption manual receipt mode
Automatic signature (not recommended)
Manual signature (recommended)
MQ server-side message persistence
RabbitMQ dead letter queue
The message is delivered to MQ and stored. The message has expired.
The queue container is full
If the consumer fails to consume multiple messages, it will be transferred to the dead letter queue.
RabbitMQ message automatic retry mechanism
What should I do if I still fail after trying multiple times?
Move to dead letter queue
Record in the log table for scheduled compensation or manual compensation
How to avoid consumer idempotence problems?
Global ID business scenario ensures uniqueness
Kafka
Kafka core architecture design model
Broker (MQ server side)
Topic (topics are classified according to business)
Partiiton (partition storage message)
Producer
Consumer
Consumer Group
Replica (replica mechanism)
Offset (consumption record)
Why kafka can support high concurrency
storage structure level
Messages will be compressed to reduce bandwidth transmission
kafka partition partition storage structure model
.log storage message file
.index stores the index of the message
.timeIndex, time index file
Segmented storage log (segment file)
Use a sparse index to find the physical location of the message (no index is created for each message)
Benefits: Save space
After the message is stored, it will not be deleted immediately after successful consumption. The message is obtained based on the offset (you need to consider configuring the log cleaning strategy)
java application level
producer
Producers deliver messages in batches (buffer pool design)
consumer
Consumers obtain messages in batches (multiple offsets)
One consumer per partition (scalability)
linux kernel level
Use sequential reading and writing
Use zero copy mechanism
sendfile mmap user mode and kernel mode mapping
No need for cpu to copy data
Reduce the number of switching between user mode and kernel mode
Use Page Cache to improve reading and writing
Need to consider the issue of disk brushing
How Kafka ensures reliable messages
Replication
ISR replica reliable mechanism
Replica election in partition
HW high water mark: the maximum offset that consumers can consume
The largest offset value in the LEO queue
Producer delivers message ack
0 means the producer does not wait (messages may be lost)
1 means that the producer is waiting and the leader is flushing the disk (default configuration recommended)
-1 means that the producer needs to wait for all nodes to be synchronized.
Kafka election principle controller principle
Relying on Zookeeper temporary nodes to implement elections
Consumer manually submits offset
How does Kafka find the Message with a specified offset?
1. Search Segment segment files based on offset (binary search)
2. Access the index file (sparse index) and find the corresponding physical storage location
3. According to the physical access location, access the log to find the physical corresponding message.
Find the message index value
Return physical messages directly (time complexity o(1))
The message index value was not found
Search in sequence (time complexity o(N))
Kafka performance optimization
producer
Producer memory buffer size
Retry policies "retries" and "retries.backoff.ms"
This parameter sets the number of retries and the interval.
Confirmation mechanism: acks It is recommended to set it to 1 for balance
consumer
The number of consumer partitions
Consumers obtain messages in batches based on multiple offsets
Consumer enables manual offset submission
Broker (MQ server side)
Log retention policy configuration
Log data file flushing strategy
replica replication configuration
Network and IO thread configuration optimization
MySQL practical performance optimization
MySQL performance optimization
MySQL architecture and execution process principles
How SQL statements are executed
Built-in query cache
Grammar and lexical parsing
semantic processor
Optimizer/execution plan
query execution engine
InnoDb memory structure and disk structure
Buffer design
Buffer Pool function
The memory buffer is full, what to do?
How to configure the size of your Buffer Pool?
MySQL latency issues and data flushing strategies
How to understand the abstract data page unit in MySQL
How do data pages and cache pages on disk correspond to each other?
What is the description information corresponding to the cache page?
How to set up the Buffer Pool appropriately based on machine configuration
How much memory should be set for the buffer pool in a production environment?
Total size = 2 times (chunk size * number of buffer pools)
MySQL underlying communication protocol principle
linux
TCP/IP socket
mysql message
Three-way handshake authentication
Unix socket
windows
named pipe
memory sharing
MySQL underlying module division
Initialization module
Core API
Network interaction module
Client & Server interaction protocol module
user module
access control module
Connection management, connection threads and thread management
Query parsing and forwarding module
Query optimizer module
Logging module
Storage engine interface module
The underlying implementation principle of MySQL index
Index data structure model
hash table
binary search tree
red black tree
Balanced multi-fork search tree
B Trees
What categories does the index have?
Full text index
primary key index
combined index
unique index
The difference between innodb and myisam The difference in index<br>
Basic principles of establishing and using MySQL indexes
The underlying principles of MySQL transactions
The difference between Spring declaration and programming transaction
What problems will occur if the transaction only begins and does not commit/rollback?
mysql multi-version control MVCC principle
MySQL transaction isolation level
repeatable read
read commit
Read uncommitted
serialization
MySQL locking and releasing lock principles
Row lock, table lock, page lock
Pessimistic lock/optimistic lock
Gap Lock
What is gap lock?
Why is gap lock the main reason to prevent phantom reads under RR isolation level?
Primary key index/unique index Will a gap lock be added to the current read?
Whether the gap lock will be added through range query
Will Gap be added to the current read if the search conditions do not exist?
Deadlock analysis principle
Two-Phase Locking
Why does deadlock occur?
The underlying principle of MySQLUndo log
The difference between Undo-log and Redo-log
UndoLog implements the principle of transaction atomicity
RedoLog implements transaction persistence principle
MySQL practical performance optimization
SQL slow query analysis and solutions
How to enable MySQL slow query
Interpretation of the principle of execution plan explain
id: The larger the column, the higher the execution priority. If the id is the same, it will be executed from top to bottom. If the id is NULL, it will be executed last.
select_type: indicates the type of query
table:explain which table is being accessed by a row.
type column
system
const
eq_ref
ref
<p class="MsoNormal"><b><span style="font-family: "Times New Roman"; font-size: 10.5pt;">range </span></b></p>
index
ALL
Extra means additional information
Principles of SQL and Index Optimization
Follow the best left prefix rule to prevent index failure
Try to use covering indexes to avoid table back queries
Sorting follows the best left prefix method to avoid filesort
The minimum type satisfies the range range query level
Pagination optimization where id condition filters offSet or subquery locates id association
Join table query optimizes small tables to drive large table data. It is prohibited to use join for more than three tables.
Like fuzzy follows the best left prefix rule or use composite index fuzzy query
MySQL configuration optimization principles
Storage engine and table structure optimization
Alibaba Development Manual Optimizing MySQL from Perspective
MySQL table and database
When the single table reaches the maximum size, it will be divided into tables and databases.
The difference between horizontal and vertical splits
Sub-table and sub-database strategy
Remainder/range modulo
Split by range
Split by date
Split by month
Shard by enumeration value
Binary modulo range slicing
Consistent hash sharding
Partition as specified by the target field prefix<br>
Segment the modulo range according to the prefix ASCII code and value
Developer customization
common problem
What are the advantages and disadvantages of querying after table and database partitioning?
How to implement paging query after splitting tables and databases
How to implement join table query after splitting tables and databases
Why MyCat is not recommended
Netty in-depth source code interpretation
Conceptual basis of network model
TCP/IP five-layer architecture model
Application layer
transport layer
Network layer
data link layer
physical layer
Socket network programming
TCP protocol
UDP protocol
How to solve the IP address principle by entering a url address
The underlying principles of the HTTP protocol
HTTPS protocol request and response principle
HTTPS and SSL/TLS principles
The difference between http protocol and Socket
IO model principle
Blocking I/O model
Non-blocking I/O model
Multiplexed I/O model
Signal-driven I/O model
Asynchronous I/O model
The difference between NIO and BIO
AIO basic principles
The evolution from BIO to NIO
Stream-oriented and buffer-oriented
Blocking and non-blocking
NIO implementation principle (operating system kernel)
Core concept
kernel buffer
process buffer
linux kernel
select
The time complexity is O(n), and there are certain restrictions on monitoring file descriptors.
poll
The time complexity is O(n), and there is no limit on listening to file descriptors.
epoll
The time complexity is O(1), and there is no limit on listening to file descriptors.
Netty source code interpretation
Netty common usage scenarios
RPC framework
Tomcat server
Online game
NIO architecture principles
buffer
Selector
aisle
Netty high performance design
Asynchronous non-blocking communication
Zero copy/memory pool
MMAP write
Sendfile
Things to think about:
How to reduce the number of cpu copies
Direct memory dma copy principle
How to reduce the number of kernel switches
Efficient Reactor threading model
Single Reactor single thread
Single Reactor multi-threading
Reactor master-slave model
Lock-free serial design concept
Serialization framework support
Netty source code interpretation
Netty thread model and source code analysis
High-performance serialization protocol protobuf and source code analysis
Sticky packet unpacking phenomenon and solution, codec source code analysis
Detailed explanation of direct memory and Netty zero copy
Netty framework practice
Based on Netty handwritten RPC framework (high imitation Dubbo)
Based on Netty handwriting multiplayer online version of backgammon game
Based on Netty handwritten web server (high imitation Tomcat)
Analysis of Linux system kernel principles
Linux kernel preparation work
A brief analysis of the Linux kernel architecture
The difference between Linux architecture and kernel structure
Linux-driven platform mechanism
Linux kernel architecture
Spring System Framework Source Code Interpretation
Spring5 source code interpretation
SpringMVC source code interpretation
SpringBoot source code interpretation
New retail e-commerce project
Architecture design ideas
New retail concept<br>
Technical architecture plan
SpringBoot
SpringCloudAlibaba
Intermediate design
Technical center
Business center
Organization center
cloud computing
SaaS (Software as a Service)
PaaS (Platform Service)
IaaS (Infrastructure Services)
Devops and K8S operation and development integration
Apm
Implement service tracking
Monitoring alarm
Separation of front and back ends
Front-end-----vue is similar to ajax technology and implemented by front-end engineers
Backend ------ Interface form Backend engineer implements java
Build infrastructure services
Deploy Nacos service registration center/configuration center
Build an enterprise-level Maven private server
Implement the call of RPC interface in microservice team
Define API interface specification protocol
Build an enterprise-level code warehouse management platform
po/do/vo/dto/bo select application
The purpose is to ensure the security of data transmitted by RPC
Select app
DO (Data Object): corresponds to the database table structure one-to-one, and transmits data source objects upward through the DAO layer.
DTO (Data Transfer Object): Data transfer object, an object transferred externally by service or manager
BO (Business Object): Business object. Object that encapsulates business logic output by the service layer
AO (Application Object): Application object. Abstract reuse object model between web layer and service layer
VO (View Object): Display layer object, usually an object transmitted by the web to the template rendering engine layer
Member center design
Login interface implementation
Why not use session
When the session is saved on the jvm server, the node cluster synchronization issue needs to be considered.
Token implementation method
Implementation principle
Randomly generate a token (UUID) as the key value of Redis as userid
Return the token to the client, and the client calls the interface each time it passes the token.
Advantages and Disadvantages
advantage
Hide parameter authenticity
shortcoming
Need to go through Redis query
JWT implementation
components
header(header)
Payload(payload)
Signature
Advantages and Disadvantages
advantage
No need to store user data on the server, reducing server-side pressure
Lightweight, json style is relatively simple
cross language
shortcoming
Unable to update validity period
Unable to destroy a jwt
How to implement logout in Jwt
Clear browser cookies (but the server still exists)
It is recommended to set the time slightly shorter
Integrate multithreading and thread pooling
Use multi-threading to process emails, text messages, and coupons sent after login, thereby improving interface response efficiency.
Large projects use mq asynchronous and time-consuming processing to reduce server CPU resources.
SSO single sign-on
web form
Implemented based on cookies
Separation of front and back ends
Implemented based on token or jwt
common problem
How to get the real client’s IP information
By setting the user’s real IP in nginx
How to solve the cross-domain problem of front-end and back-end separation
Use jsonp but does not support post requests (not recommended)
Use SpringMVC @CrossOrigin annotation (recommended)
Solve cross-domain problems based on gateway (recommended)
Access based on different projects based on Nginx (recommended)
Joint login implementation
oauth2 open protocol
1. Generate authorization link address based on appid
2. Obtain authorization code
3. Obtain the accessToken based on the authorization code
4. Obtain user information based on authorization code
Distributed solutions
Disadvantages of traditional collection logs
Use tail to search logs on each server
solution
aop elk kafka implements distributed log collection
Why elk needs to add kafka
Reduce the cost of operation and maintenance on each Logstash installation
Precautions
AOP caches the collected logs in a concurrent queue, and delivers the logs to Kafka in an asynchronous separate thread.
Aggregated payment design
Payment architecture process
Verify signature method
RSA asymmetric encryption
MD5
callback method
Synchronous callback
After the third-party payment is successful, it will jump to the merchant end in the form of payment browser redirection.
Asynchronous callback
Third-party payment sends notifications to merchants using technology similar to HttpClient
Design Patterns
strategy pattern
template method pattern
common problem
How synchronous callbacks and asynchronous callbacks ensure interface security
The synchronous callback jumps in the form of a browser and does not modify the order status.
After the asynchronous callback verifies the signature successfully, pay attention to the idempotent issue to modify the order status.
How to prevent users from paying repeatedly
The same order number is submitted from the form to third-party payment, and the third-party payment will ensure global uniqueness based on the order number.
What should I do if the user's payment is successful but the order status is still unpaid?
This phenomenon is normal. Through the idea of eventual consistency, you can actively call the Alipay interface to query whether the order has been<br>paid.
How to handle the inconsistency between the user's payment amount and the order amount
In the asynchronous callback, it will be queried based on the order whether the user's real order number is consistent with the user's payment amount<br>If they are inconsistent, it is an abnormal order.
What types of payment table amount fields are suitable?
Direct storage of integer types can later be converted into elements
decimal type
How to handle an order that has been unpaid for 30 minutes overtime
Implemented based on MQ delay queue (recommended)
1. After the order is placed successfully, deliver a delayed queue message to MQ
2. When the message expires, it will be transferred to the dead letter queue.
3. The dead letter queue consumer listens to the message and checks whether the order status is paid.
Implemented based on Redis expired key (not recommended)
1. After placing the order successfully, set a 30-minute expiration key to Redis.
2. When the client monitors the expiration of the key, it checks whether the order status is paid.
3. To enable expired key monitoring, you need to pay attention to the prefix and subscribe to a separate Redis library.
common problem
The user's card pays within 30 minutes. How to ensure the consistency of the order status?
1. Set the timeout for redirecting Alipay orders to 30 minutes
2. The delay queue actively calls the Alipay interface to check the payment status based on the order number around 31-35 minutes.
3. If the payment is still not made after calling the Alipay interface, the order will be considered time-consuming.
Product service system architecture design
How to achieve resistance to high concurrency
Front-end layer
Optimization
Dynamic and static separation architecture
Static resource server
dynamic resource server
Static resource compression
Generate .min file
CDN cache
Visit according to the principle of proximity
final effect
Reduce bandwidth transmission
interface layer
Optimization
JVM parameter tuning to reduce the number of GC recycling stw issues
Handle time-consuming operations in the form of multi-threading/MQ asynchronous decoupling
Use Redis cache to reduce the pressure of DB access
Consider splitting tables and databases/index optimization for MySQL massive data
final effect
Operation and maintenance layer
Use docker or K8S to elastically expand/shrink deployment
JavaSE
basic grammar
type of data
Basic data types
Numeric type
Integer (byte, short, int, long)
Floating point number (float,double)
character(char)
non-numeric type
boolean
Reference data type
class
interface
array[]
object-oriented
collection framework
IO stream
reflection mechanism
Multithreading
JDBC
JavaWeb
Base
Servlet
JSP
frame
Spring5
Spring5 and SpringBoot relationship
SpringBean injection method
SpringBean injection method
Parametric constructor injected properties
p namespace injection
Inject null values and special symbols
Inject internal beans
Inject external beans
Inject cascade assignment
Inject collection type properties
Spring factory beans
SpringBean life cycle
Step 1: Use reflection technology to initialize the object and call the no-argument constructor
Step 2: Use reflection to call the set method to assign values to the properties.
The third step of execution: bean’s post-processor pre-method
Step 4: Call the init method in the object
Step 5: Bean’s post-processor post-method
Step 6: Destroy the object and call the destruction method
SpringBean scope
Singleton object
Multiple instance objects
SpringBean automatic wiring
SpringBean external property file
SpringBean annotation form
SpringBean annotation startup method
SpringBean's annotation startup function
SpringBean annotation scanning configuration
Autowired and Qualifier annotations
@Resource usage
SpringBean AOP
AOP basic concepts
The basic role of AOP
Static proxy and dynamic proxy
@AspectJ annotation usage
Use aop to print logs uniformly
SpringBean transaction operations
Classification of transactions
Manual transactions
programming matters
Seven communication behaviors of affairs
PROPAGATION_REQUIRED (default propagation behavior)
If there is a transaction in the current thread, join the current transaction
If a transaction does not exist for the current thread, create a new transaction
PROPAGATION_SUPPORTS
If there is a transaction in the current thread, join the current transaction
If there is no transaction in the current thread, it will be executed in a non-transaction<br>way.
PROPAGATION_MANDATORY
If there is a transaction in the current thread, join the current transaction
Throws an exception if there is a transaction in the current thread
PROPAGATION_REQUIRES_NEW
If a transaction exists in the current thread, the current transaction will be suspended and a new transaction will be created.
PROPAGATION_NOT_SUPPORTED
Always executed in a non-transactional manner
PROPAGATION_NEVER
Always executed in a non-transactional manner, an exception will be thrown if a transaction exists in the current thread
PROPAGATION_NESTED
If there is a transaction in the current thread, a transaction will be nested
SpringMVC
Mybatis
Hibernate
microservices
SpringBoot2.0
Why you need to use SpringBoot framework
Can help developers quickly integrate third-party frameworks (Principle: Maven dependency encapsulation)
Remove xml configuration and completely use annotation (principle: built-in annotation method in Spring system)
No need for external Tomcat and internal implementation server (principle: Java language supports embedded Tomcat server)
The difference between SpringBoot and SpringCloud
SpringCloud dependencies and SpringBoot components
Use SpringMVC to write Http protocol interface
Spring Cloud is a complete microservice solution framework
Introduction to SpringBoot dependency introduction
spring-boot-start-parent,
spring-boot-starter-web
@RestController role
All methods of Controller return JSON format
SpringBoot startup method
@EnableAutoConfiguration
@ComponentScan
@SpringBootApplication
All classes under the current package or sub-packages can be scanned
SpringBoot integrates static resource access
What template engine framework
Render the web, which is beneficial to SEO search
Integrate ftl template engine
Integrate thymeleaf template engine
SpringBoot integrates data sources
JdbcTemplate
mybatis
hibernate
SpringBoot integrated hot deployment
Integrate devtools tools
Class loader implementation
Integrate lombok
SpringBoot integration configuration file
Use @value annotation to read configuration files
Properties convert yml format
@ConfigurationProperties
Configuration file placeholder usage
Integrate different configuration files in multiple environments
Modify port and context path
SpringBoot integrated log framework
logback
log4j
Use aop to print log information uniformly
SpringBoot scheduled tasks
Integrate scheduled tasks @Scheduled annotation
Timed integration tasks combined with Quartz expressions
SpringBoot integrates asynchronous multi-threading
Pay attention to @Async invalidation problem
@Async integrates thread pool
Integrate global catch exceptions
Package and run release
Project tools
docker
basic concept
Why do you need to use docker
Benefits of using docker
The difference between containers and virtual machines
Environment installation
Install docker in Linux environment
Install docker in Win environment
Three major elements
Image file
container
storehouse
Mirror principle
Docker download image principle
Docker image loading principle
bootfs
rootfs
Union fs
Cloud accelerated image configuration
Alibaba Cloud accelerated image
Huawei Cloud Accelerated Image
HKUST Accelerated Mirror
Docker common commands
docker --help (help command)
docker --version (view version)
docker images (view images)
docker search (search image)
docker pull (download image)
latest -----tag latest version of image file
docker container
docker run (start container)
docker start starts container id
docker stop container id
docker rm container id
docker exec -it [CONTAINER ID] bash (enter container)
docker logs --since 30m CONTAINER_ID (view container logs)
Docker Commit (made into an image file based on the current container)
Docker data volume -v
Install commonly used software
Tomcat
docker run -p 8081:8080 -d tomcat:8
Nginx
docker run --name nginx81 -d -p 81:80
MySQL
docker create --name mysql3308 -e MYSQL_ROOT_PASSWORD=root -p 3308:3306 mysql:5.7
DockerFile parsing
DockerFile writing specifications
DockerFile directive
Build springboot project based on Dockerfile
Docker Compose
Compose common commands
docker-compose -h
docker-compose up
docker-compose down
docker-compose logs
docker-compose pull
dokcer-compose config
docker-compose restart
docker-compose start
Compose template file
Deploy SpringBoot MySQL Nginx microservice project
Docker visualization tool usage
Portainer
DockerUI
k8s
meta-native concept
microservices
Restful communication between applications
Can be independently deployed/elastically expanded and reduced
devops
Automated release pipeline, Ci/CD tools
Quickly deploy production environment
Integration of development and operation and maintenance
continuous delivery
Frequent releases, fast delivery, quick feedback, and reduced release risks
Containerization
The best carrier for microservices
Design Patterns
proxy mode