Wondershare EdrawMind
MindMap Gallery data folder and model folder
- 48
data folder and model folder
This is a mind map about the data folder and model folder. The data folder and model folder are respectively responsible for storing data and model-related files in the project, and are an important part of project management and operation.
Edited at 2024-02-08 15:42:24
- Recommended to you
- Outline
central theme
data folder
Cub
Split
Test.csv
test set
Train.csv
Training set
Val.csv
Validation set
Miniimagenet
Split
Get_unlabel.py
Define image category list
Extract data from three CSV files (train.csv, val.csv and test.csv) related to the Mini-ImageNet dataset, collating the file names and category labels into a dictionary (mini_info) with the category labels sorted in order
Read data from three CSV files in the tieredimagenet/split directory and organize the filenames and category labels into another dictionary (tiered_info)
Categories that appear in the tiered_class_list but not in the mini_class_list are identified and these unlabeled categories are stored in the list unlabel. Then, print the number of unlabeled categories and their contents
Write unlabeled information into a CSV file to form unlabel.csv file
original data set
train.csv
val.csv
test.csv
Generated dataset
unlabel.csv
Additional testing and validation data
Aux_test.csv
Aux_val.csv
Tieredimagenet
Split
Get_spilt.py
Traverse different parts of the data set (train, val, test), generate corresponding CSV files, and write the file path and label into the CSV file for subsequent data processing, model training and other tasks.
Generated dataset
train.csv
val.csv
test.csv
model folder
Dataloader
Cub.py
A PyTorch dataset class CUB for loading the CUB (Caltech-UCSD Birds-200-2011) dataset.
Defines constants such as data set image path, segmentation path and cache path.
Data preprocessing function get_transforms:
Generate transformations required for data transformation based on input parameters, including random cropping, random horizontal flipping, color perturbation, and normalization.
Set different data normalization according to different backbone types to adapt to different model training needs.
Dataset class CUB:
Inherits PyTorch's Dataset class and is used to load CUB datasets.
The following operations are performed in the initialization function __init__: parse the CSV file and obtain data and labels.
Count the number of categories in a dataset.
Call the get_transforms method to obtain the transformations required for data transformation.
The parse_csv method is used to parse CSV files and obtain data and label information.
The __len__ method returns the length of the data set.
The __getitem__ method is used to get the sample at the specified index, which includes: trying to use the jpeg4py library to decode the JPEG image, and if that fails, using the PIL library to open the image.
Apply data transformation (preprocessing) and return images and labels.
Mini_imagenet.py
Create a PyTorch dataset class (MiniImageNet class) for working with the MiniImageNet dataset. This dataset class is used to load image data and provide corresponding data enhancement and preprocessing operations for training neural network models.
Data enhancement function get_transforms: Returns the transformation sequence of two image transformation methods (enhanced and non-enhanced) based on the specified image size, neural network structure (backbone), and data enhancement intensity
MiniImageNet data set class MiniImageNet: includes functions such as data loading, image path analysis, and data enhancement.
__init__ method: Initialize the data set class, accept the name of the data set division and some parameters, and call the get_transforms function to obtain the data conversion method.
parse_csv method: Parses image paths and labels from CSV files for building datasets.
__len__ method: Returns the length of the dataset (number of images).
__getitem__ method: Get the data item according to the index, try to use the jpeg4py library to decode the JPEG format image, if it fails, use the PIL library.
Samplers.py
For meta-learning and classification tasks, the code implements different sampler classes to provide flexible data sampling and loading methods for meta-learning tasks and a flexible data sampling and loading framework for meta-learning tasks. This makes it easier for researchers and developers to implement and test different meta-learning algorithms.
CategoriesSampler class:
During initialization, pass in the label (label), the number of samples in each batch (n_batch), the number of samples in each category (n_cls), and the number of samples sampled in each batch of each category (n_per).
Generate each batch of data through the __iter__ method, randomly select n_per samples for each category, and return a flattened one-dimensional tensor.
CategoriesViewSampler class:
During initialization, pass in the label (label), the number of samples in each batch (n_batch), the number of samples in each category (n_cls), the number of samples in each category for the support set (n_shot), and the number of samples in each category for query. The number of samples in the set (n_query) and the number of views in each task (n_view).
The __iter__ method generates each batch of data. For each category, n_shot n_query samples are randomly selected, of which the first n_shot are used for the support set and the last n_query are used for the query set. Support sets are labeled with additional information indicating the view number.
RandomSampler class:
During initialization, pass in the label (label), the number of samples in each batch (n_batch), and the number of labels per sample in each batch (n_per).
The __iter__ method generates data for each batch, and n_per samples are randomly selected for each batch.
ClassSampler class:
When initializing, pass in the label (label) and the number of samples per batch (n_per) for each category.
The __iter__ method generates data for each batch, and for each category, n_per samples are randomly selected.
InSetSampler class:
Used for ResNet Fine-Tune, the number of samples in each batch (n_batch), the number of labels for each sample in each batch (n_sbatch), and the sample pool (pool) are passed in during initialization.
The __iter__ method generates data for each batch, and each batch randomly selects n_sbatch samples from the sample pool.
Tiered_imagenet_raw.py
Mainly used to load the Tiered ImageNet data set, providing data preprocessing, image loading and label processing functions
Models
_init_.py
If a directory contains a file named __init__.py, Python will treat the directory as a package, allowing you to use the import statement to import modules or subpackages in the directory.
Tells Python to also import the FewShotModel class in the model.models.base module when importing the current package.
Maml.py
The Model-Agnostic Meta-Learning (MAML) framework implemented using PyTorch is used for meta-learning, implements MAML models, and provides different evaluation methods, including permutation evaluation and integrated evaluation. This is a general framework for meta-learning, applicable to a variety of tasks.
Section 1: Importing libraries and modules
Paragraph 2: update_params function
This function accepts a loss value, a dictionary of parameters (params), and updates the parameters using gradient descent. The function returns the updated parameters.
Paragraph 3: inner_train_step function
The inner_train_step function represents the inner training step in meta-learning. It accepts a model, supporting data and some parameters, then performs internal training and returns updated parameters after internal training.
Paragraph 4: MAML classes
Performs a forward pass of MAML, using internally trained parameters. There are other methods for evaluation such as forward_eval, forward_eval_perm and forward_eval_ensemble.
Paragraph 5: Model implementation
Select ResNetMAML as the backbone network according to the given backbone network category and initialize the model parameters. The update of parameters is performed in inner_train_step, using the cross-entropy loss function for internal training. The prediction results are scaled using the temperature parameter.
MAMLUnicorn.py
Implemented a meta-learning algorithm, specifically a variant based on Model-Agnostic Meta-Learning (MAML), for Few-Shot learning tasks
Networks
Convnet.py
Defines a basic convolutional neural network (ConvNet) model
Its main function is to process the input image through convolution and pooling operations, and finally output a feature tensor with z_dim channels, which can be used for various tasks, such as image classification, feature extraction, etc.
Convnet_maml.py
A model based on meta-learning is implemented, specifically the MAML (Model-Agnostic Meta-Learning) algorithm for meta-learning on small sample learning tasks.
Dropblock.py
Defines a PyTorch module called DropBlock that implements a regularization technique called DropBlock
Res12.py
Neural network designed based on ResNet architecture, mainly used for Few-Shot Learning tasks
The entire network adopts the structure of ResNet, but the design takes into account the characteristics of the Few-Shot Learning task, including the introduction of DropBlock and the dynamic adjustment of some parameters. At the same time, the code also contains initialization operations for model parameters.
Res12_maml.py
A neural network based on ResNet architecture, mainly used for model adaptability (Model-Agnostic Meta-Learning, MAML) and sharing of model parameters in meta-learning (Meta-Learning).