Application of a Pretrained Classifier

The project aims to use maschine learning methods to emulate a cloud classification scheme. The classifer can be trained using large amounts of data and later be used to predict cloud types from satelite data. Those two steps can be run separately.

This notebook contains a short explanation how to use a pretrained classifier in order to predict labels from new input data.

Imports

At first we need to point python to the project folder. The path can be assigned as a relative path as shown below, or as an absolute system path. Than the module can be imported via the import cloud_classifier command.

import sys
import ctypytool

import warnings
warnings.filterwarnings("ignore")

Initialization

Our first step is to create a classifier object:

cc = ctypytool.cloud_classifier()

Than we need to point our classifier object to an already existing classifier. The load_project() method will load an existing classifier into our classifier object.

path = "../classifiers/TreeClassifier"
cc.load_project(path)

Applying the Classifier: Prediction of Cloud Type Labels

Using a User-Defined File List

In order to predict labels with the now loaded classifier, we need to specify input files of satelite data. This can be done manually via in input_files option in the set_project_parameters method.

file_1 = "../data/example_data/msevi-medi-20190317_1800.nc"
file_2 = "../data/example_data/msevi-medi-20190318_1100.nc"

cc.set_project_parameters(input_files = [file_1, file_2])

We now run the prediction pipeline (with the run_prediction_pipeline() method) which * applies the classifier to our input data and * stores the predicted labels.

The option create_filelist is set to False to take the user-defined input file list.

cc.run_prediction_pipeline(create_filelist = False)

Classifier loaded!
Masked indices set!
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190317_1800_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190318_1100_predicted.nc

Using an Automatically Generated Input File List

Alternatively to the manual definition, the input file list can be generated automatically.

The easiest way to do so is to put all input files into an input data folder (here it is set to ../data_example_data) and just tell the classifier where to look via the input_source_folder option.

%%bash

ls -l ../data/example_data

lrwxrwxrwx 1 b380352 bb1174 49 Apr  9 10:42 ../data/example_data -> /work/bb1174/data/meteosat-seviri-subregions/medi

cc.set_project_parameters(input_source_folder = "../data/example_data")

In a next step, we can let the classifier predict labels from the input files we have specified. This is again done with the run_prediction_pipeline() method.

If we want the classifier to automatically generate a list of input files and therefore set the option create_filelist to True.

cc.run_prediction_pipeline(create_filelist = True)

Input filelist created!
Classifier loaded!
Masked indices set!
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20201105_0500_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200106_0700_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20191118_1700_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200526_1700_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200921_1300_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20201104_0700_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190207_0000_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190818_0700_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20191212_1500_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200807_1200_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190823_2300_predicted.nc
Removed 201441 vectors for containig 'Nan' values
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200708_0200_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190301_0200_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20201116_0900_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200211_1000_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200213_1800_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190826_1400_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200302_0400_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190820_0400_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20191025_0100_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20191111_0600_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20191118_0300_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190619_0800_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190502_1700_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200521_2300_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200518_0800_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20191205_0700_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200721_1800_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200407_1700_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200321_0000_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190318_1100_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190903_0000_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190310_1300_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190606_0100_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190224_1400_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200516_1700_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190314_2000_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190601_1400_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190305_2200_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190823_2000_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190907_0600_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190523_0200_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200405_0500_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190204_1300_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200724_2300_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200129_0800_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200518_1000_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190420_2300_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20191016_1600_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190106_0700_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200205_0600_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190402_0100_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190103_0300_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200411_1800_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190826_0300_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20191128_0400_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190113_0700_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20191215_1400_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190512_0000_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200308_0700_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190619_1300_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20201107_0000_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190413_1800_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20201122_0800_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190828_0200_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200802_0400_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190217_0100_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190616_0900_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190107_1200_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190815_0900_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190611_0100_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190225_0800_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200131_2300_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200422_0600_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20191102_1800_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200914_2300_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200111_1100_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20191119_0500_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200816_0400_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190403_0600_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190615_1000_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190906_0900_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20191021_1400_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190318_2300_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190410_0800_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200110_2200_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190326_1500_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200206_0300_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200809_1700_predicted.nc
Removed 201441 vectors for containig 'Nan' values
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200426_1100_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190726_1600_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190306_0700_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190818_0400_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190101_1600_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200222_2100_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200103_2000_predicted.nc
Removed 201441 vectors for containig 'Nan' values
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20191129_1200_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20191024_1700_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20191030_1000_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20191016_1200_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20201124_1500_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20191104_1000_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200110_2000_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20191212_1600_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200823_0300_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190301_0800_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200809_1400_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190101_0800_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200209_0400_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190612_0700_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200406_0300_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200101_2300_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20191011_0600_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190905_1000_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20191109_0000_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20191112_2200_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200814_1300_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200830_0800_predicted.nc
Removed 18781 vectors for containig 'Nan' values
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200717_2000_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200803_0500_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20190113_0400_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200730_1100_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200107_2300_predicted.nc
Input vectors created!
Labels saved as nwcsaf_msevi-medi-20200317_0800_predicted.nc

Accessing predicted labels

The predicted labels are stored in the folder of the classifier we are using. They are located in the subfolder labels.

%%bash

ls ../classifiers/TreeClassifier/labels

nwcsaf_msevi-medi-20190317_1800_predicted.nc
nwcsaf_msevi-medi-20190318_1100_predicted.nc