prerequisites

direction

database

other

Cleaning (pretreatment)

Convert

extract

calculate

data analysis

data mining

chart

sheet

Word

Array and matrix operations

Extremely efficient

Charts, visualizations

origin of name

Function

Advanced data structures

Stores a multidimensional array of a single data type

Create multidimensional arrays from existing data

Create a multidimensional array of a specific shape

Create multidimensional array from numerical range

ndim

shape

size

dtype

itemsize

Same as list

reshape(a,b)

repeat(4, axis=1)

np.random.rand(2, 3)

np.random.randint(5, size = (2, 3))

Is a subclass of ndarray

Create matrix

matrix properties

Matrix Operations

Functions that can process ndarray arrays can be used directly.

Arithmetic

comparison operation

logic operation

Refers to the way arithmetic operations are performed between arrays of different shapes

in principle

numpy.save(file, arr, allow_pickle=True, fix_imports=True)

numpy.load(file, mmap_mode=None, allow_pickle=True, fix_imports=True, encoding='ASCII')

np.savetxt(fname, X, fmt='%.18e', delimiter=' ', newline=' ', header='', footer='', comments='# ')

numpy.loadtxt(FILENAME, dtype=int, delimiter=' ')

direct sorting

indirect sorting

numpy.unique

Create canvas

Create canvas

Selected subplot

Set X, Y axis

Add legend (details)

definition

create

Basic operations

Common properties

Common methods

definition

create

Column index operations

Row index operations

Common properties

Common methods

Traverse

df.fillna(value=0)

df['prince'].fillna(df['prince'].mean())

df['city']=df['city'].map(str.strip)

df['city']=df['city'].str.lower()

df['price'].astype(int)

Modify all

Partial modification

Find duplicates: df.duplicated() can return a boolean array indicating whether each row is a duplicate.

Duplicate values ​​that appear after deletion

Remove duplicate values ​​that appear first

Select primary key

df2=df.dropna(axis=0,how="all",inplace=False)

df['city'].replace('sh', 'shanghai')

pd=pd.merge(df,df1,how='inner') #match, merge, intersection, default df_left=pd.merge(df,df1,how='left') df_right=pd.merge(df,df1,how='right') df_outer=pd.merge(df,df1,how='outer') #Union, the effect is the same as the first two combinations

Has been deprecated, it is recommended to use concat

pd.concat(objs,axis=0,join='outer',join_axes=None,ignore_index=False,keys=None,levels=None,names=None,verify_integrity=False,copy=True)

variance

Covariance

df = pd.DataFrame({ 'A': ['a', 'b', 'a', 'c', 'a', 'c', 'b', 'c'], 'B': [2, 8, 1, 4, 3, 2, 5, 9], 'C': [102, 98, 107, 104, 115, 87, 92, 123]})

method

df = pd.DataFrame({' A': list('XYZXYZXYZX'), 'B': [1, 2, 1, 3, 1, 2, 3, 3, 1, 2], 'C': [12, 14, 11, 12, 13, 14, 16, 12, 10, 19]})

method

df['ExtraScore'] = df['Nationality'].apply (lambda x : 5 if x != '汉' else 0)

df['pass_reading'] = df['reading score'].apply (lambda x: 'Pass' if x >= 60 else 'Fail')

alpha is transparency

read csv

Write to Excel

Write to Excel

Write to CSV

import datetime cur = datetime.datetime(2018,12,30, 15,30,59) print cur,type(cur) d = datetime.date(2018,12,30) printd t = datetime.datetime(2018,12,30).now() print t

2018-12-30 15:30:59 <type 'datetime.datetime'> 2018-12-30 2018-12-16 15:35:42.757826

import datetime cur0 = datetime.datetime(2018,12,30, 15,30,59) print cur0 cur1 = cur0 datetime.timedelta(days = 1) print cur1 cur2 = cur0 datetime.timedelta(minutes = 10) print cur2 cur3 = cur0 datetime.timedelta(minutes = 29, seconds = 1) print cur3

2018-12-30 15:30:59 #cur0 2018-12-31 15:30:59 #cur1 2018-12-30 15:40:59 #cur2 2018-12-30 16:00:00 #cur3

from datetime import datetime, timedelta import numpy as np import pandas as pd b = datetime(2018,12,16, 17,30,55) vi = np.random.randn(60) ind = [] for x in range(60): bi = b timedelta(minutes = x) ind.append(bi) ts = pd.Series(vi, index = ind) print ts[:5]

2018-12-16 17:30:55 -1.469098 2018-12-16 17:31:55 -0.583046 2018-12-16 17:32:55 -0.775167 2018-12-16 17:33:55 -0.740570 2018-12-16 17:34:55 -0.287118 dtype: float64

Maps a value to a quantity represented as an integer

Maps a value to a probability expressed as a floating point number

The above process is called normalization, that is, the probability sums to 1

Disadvantages of PMF

CDF

percentile rank

CCDF(a) = P(X > a)= 1- CDF(a)

Probability density function

cumulative distribution function

If a set of values ​​follows a normal distribution after logarithmic transformation, it is said to follow a lognormal distribution. That is, use log(x) to replace x in the normal distribution.

Covariance can be used to measure whether the changing trends of related variables are the same, and it can also be used to measure the overall error of two variables.

Because values ​​and units are difficult to interpret, they are generally less used.

Variance can be viewed as a special case of covariance, when two variables are identical.

If the changing trends of two variables are consistent, that is, if one of them is greater than its own expected value and the other is greater than its own expected value, then the covariance between the two variables is positive;

If the changing trends of two variables are opposite, that is, one variable is greater than its own expected value and the other is less than its own expected value, then the covariance between the two variables is negative;

Scope of application

Substitute the standard fraction for the original value and calculate the product of two standard fractions

is called the Pearson correlation coefficient, where -1<=p<=1, p=1: Indicates that the two variables are completely positively correlated; p=-1: Indicates that the two variables are completely negatively correlated;

Scope of application

First calculate the rank of the value in the sequence: that is, the sorted position of a certain value in the sequence, and then calculate the Pearson correlation coefficient of the rank.

Sample