5.1.1 Replace np.NaN with specific values

df_replace_1 = df.copy()
df_replace_1

Missing values from only one column (here ‘Name’) are replaced:

df_replace_1['Name'] = df_replace_1['Name'].fillna(0)
df_replace_1

Missing values from the complete dataset will be replaced:

df_replace_1.fillna(0, inplace=True)
df_replace_1

Does that make sense? Probably not. So let’s look at imputations that follow a certain logic.

5.1.2 Replace np.NaN with MEAN

A popular metric for replacing missing values is the use of mean.

df_replace_2 = df.copy()
df_replace_2

df_replace_2['Age'].mean()

Here all missing values of the column ‘Age’ are replaced by their mean value.

df_replace_2['Age'] = df_replace_2['Age'].fillna(df_replace_2['Age'].mean())
df_replace_2

scikit-learn - SimpleImputer

Always keep in mind that you will need all the steps you take to prepare for model training to make predictions later.

What do I mean by that exactly? If the data set you have available for model training already has missing values, it is quite possible that future data sets for which predictions are to be made will also contain missing values. In order for the prediction model to work, these missing values must be replaced by metrics that were also used in the model training.

df_replace_3 = df.copy()
df_replace_3

imp_age_mean = SimpleImputer(missing_values=np.nan, strategy='mean')

imp_age_mean.fit(df_replace_3[['Age']])
df_replace_3['Age'] = imp_age_mean.transform(df_replace_3[['Age']])
df_replace_3

In the steps shown before, I used the .fit and .transform functions separately. If it’s not about model training, you can also combine these two steps and save yourself another line of code.

df_replace_4 = df.copy()
df_replace_4

imp_age_mean = SimpleImputer(missing_values=np.nan, strategy='mean')

df_replace_4['Age'] = imp_age_mean.fit_transform(df_replace_4[['Age']])
df_replace_4

This way you can see which value is behind imp_age_mean concretely:

 imp_age_mean.statistics_

5.1.3 Replace np.NaN with MEDIAN

Other metrics such as the median can also be used instead of missing values:

df_replace_5 = df.copy()
df_replace_5

imp_age_median = SimpleImputer(missing_values=np.nan, strategy='median')

df_replace_5['Age'] = imp_age_median.fit_transform(df_replace_5[['Age']])
df_replace_5

imp_age_median.statistics_

5.1.4 Replace np.NaN with most_frequent

For some variables, it makes sense to use the most frequently occurring value for NaNs instead of mean or median.

df_replace_6 = df.copy()
df_replace_6

imp_age_mfreq = SimpleImputer(missing_values=np.nan, strategy='most_frequent')

df_replace_6['Age'] = imp_age_mfreq.fit_transform(df_replace_6[['Age']])
df_replace_6

Since there is no value in the variable ‘Age’ that occurs twice or more often, the lowest value is automatically taken. The same would apply if there were two equally frequent values.

5.2.1 Replace np.NaN with most_frequent

The most_frequent function can be used for numeric variables as well as categorical variables.

df_replace_7 = df.copy()
df_replace_7

imp_job_mfreq = SimpleImputer(missing_values=np.nan, strategy='most_frequent')

df_replace_7['Job'] = imp_job_mfreq.fit_transform(df_replace_7[['Job']])
df_replace_7

Here we see that with a frequency of 2, the job ‘student’ is the most common, so this is used for the missing value here.

imp_job_mfreq.statistics_

But what happens if we just don’t have a most frequent value in a categorical column like in our example within the column ‘Name’?

df_replace_8 = df.copy()
df_replace_8

imp_name_mfreq = SimpleImputer(missing_values=np.nan, strategy='most_frequent')

df_replace_8['Name'] = imp_name_mfreq.fit_transform(df_replace_8[['Name']])
df_replace_8

Again, the principle that the lowest value is used applies. In our example, this is the name Anton, since it begins with A and thus comes before all other names in the alphabet.

5.2.2 Replace np.NaN with specific values

df_replace_9 = df.copy()
df_replace_9

However, we also have the option of using certain values:

imp_job_const = SimpleImputer(missing_values=np.nan, 
                              strategy='constant',
                              fill_value='others')

df_replace_9['Job'] = imp_job_const.fit_transform(df_replace_9[['Job']])
df_replace_9

5.3.1 single values

For the following example we take the last version of the last used data set, here df_replace_9:

df_replace_9

rep_job_const = SimpleImputer(missing_values='others', 
                              strategy='constant',
                              fill_value='not_in_scope')

df_replace_9['Job'] = rep_job_const.fit_transform(df_replace_9[['Job']])
df_replace_9

As we can see, ‘others’ became ‘not_in_scope’.

5.3.2 multiple values

Unfortunately, we cannot work with lists for multiple values. But with the use of the pipeline function it works. We use for our following example again the last state of the dataset ‘df_replace_9’:

rep_pipe = Pipeline([('si1',SimpleImputer(missing_values = 'Manager', 
                                          strategy='constant',
                                          fill_value='not_relevant')),
                     ('si2', SimpleImputer(missing_values = 'Teacher', 
                                           strategy='constant', 
                                           fill_value='not_relevant'))])

df_replace_9['Job'] = rep_pipe.fit_transform(df_replace_9[['Job']])
df_replace_9

In my opinion, however, this approach has two disadvantages. Firstly, the values used are not saved (so cannot be reused automatically) and secondly, this is a lot of code to write. With an if-else function you would be faster:

def rep_func(col):

    if col == 'Student':
        return 'useless'
    if col == 'not_relevant':
        return 'useless'
    else:
        return 'useless'

df_replace_9['Job'] = df_replace_9['Job'].apply(rep_func)
df_replace_9

But again, the values used cannot be easily reused. You would have to create your own dictionary.

8.1.1 Train-Test Split

df_practice_simpl_imp = df_practice.copy()
df_practice_simpl_imp

In model training, I first divide the data set into a training part and a test part.

x = df_practice_simpl_imp.drop('Target_Var', axis=1)
y = df_practice_simpl_imp['Target_Var']

trainX, testX, trainY, testY = train_test_split(x, y, test_size = 0.2)

print()
print('trainX')
print()
print(trainX)
print('-------------------------')
print()
print('testX')
print()
print(testX)

8.1.2 Fit&Transform (trainX)

Then I check if there are any Missing Values in trainX.

trainX.isnull().sum()

As we can see, we need to replace missing values in the columns ‘Age’ and ‘Salary’. For this I use the SimpleImputer with the strategy=‘mean’.

# Fit and Transform trainX column 'Age' with strategy='mean'
imp_age_mean1 = SimpleImputer(missing_values=np.nan, strategy='mean')
imp_age_mean1.fit(trainX[['Age']])
trainX['Age'] = imp_age_mean1.transform(trainX[['Age']])

# Fit and Transform trainX column 'Salary' with strategy='mean'
imp_salary_mean1 = SimpleImputer(missing_values=np.nan, strategy='mean')
imp_salary_mean1.fit(trainX[['Salary']])
trainX['Salary'] = imp_salary_mean1.transform(trainX[['Salary']])

print(trainX)
print()
print('Number of missing values:')
print(trainX.isnull().sum())

When I use the SimpleImputer, I save it separately to be able to use it again later.

pk.dump(imp_age_mean1, open('imp_age_mean1.pkl', 'wb'))
pk.dump(imp_salary_mean1, open('imp_salary_mean1.pkl', 'wb'))

8.1.3 Model Training

I won’t do the model training at this point, because that would still require me to either remove the categorical variable ‘name’ or convert it to a numeric one. This would only cause confusion at this point. Let’s assume we have done the model training like this.

dt = DecisionTreeClassifier()
dt.fit(trainX, trainY)

The execution of the prediction function (apart from the still existing categorical variable) would not work like this.

y_pred = df.predict(testX)

Because we also have Missing Values in the testX part.

testX.isnull().sum()

In order to be able to test the created model, we also need to replace them. For this purpose, we saved the metrics of the two SimpleImputers used in the previous step and can use them again here.

8.1.4 Transform (testX)

In the following, I will show the syntax to replace missing values for both columns (‘Age’ and ‘Salary’). I am aware that in this example only the ‘Age’ column contains a missing value. But in practice the data sets are usually larger than 10 observations.

# Transform testX column 'Age'
testX['Age'] = imp_age_mean1.transform(testX[['Age']])

# Transform testX column 'Salary'
testX['Salary'] = imp_salary_mean1.transform(testX[['Salary']])

print(testX)
print()
print('Number of missing values:')
print(testX.isnull().sum())

Now the test of our model training would also work.

8.2.1 Train-Test Split

df_practice_ffill_bfill = df_practice.copy()
df_practice_ffill_bfill

x = df_practice_ffill_bfill.drop('Target_Var', axis=1)
y = df_practice_ffill_bfill['Target_Var']

trainX, testX, trainY, testY = train_test_split(x, y, test_size = 0.2)

print()
print('trainX')
print()
print(trainX)
print('-------------------------')
print()
print('testX')
print()
print(testX)

8.2.2 Use ffill or backfill

Now I will replace the missing values in our example with the ffill method.

# ffill column 'Age'
trainX['Age'] = trainX['Age'].fillna(method='ffill')
# ffill column 'Salary'
trainX['Salary'] = trainX['Salary'].fillna(method='ffill')

print(trainX)
print()
print('Number of missing values:')
print(trainX.isnull().sum())

8.2.3 Fit (trainX)

# Fit trainX column 'Age' with strategy='mean'
imp_age_mean2 = SimpleImputer(missing_values=np.nan, strategy='mean')
imp_age_mean2.fit(trainX[['Age']])

# Fit trainX column 'Salary' with strategy='mean'
imp_salary_mean2 = SimpleImputer(missing_values=np.nan, strategy='mean')
imp_salary_mean2.fit(trainX[['Salary']])

pk.dump(imp_age_mean, open('imp_age_mean2.pkl', 'wb'))
pk.dump(imp_salary_mean, open('imp_salary_mean2.pkl', 'wb'))

8.2.4 Transform (testX)

I’ll leave out the part about model training at this point, since it would only be fictitiously presented anyway.

testX.isnull().sum()

# Transform testX column 'Age'
testX['Age'] = imp_age_mean2.transform(testX[['Age']])

# Transform testX column 'Salary'
testX['Salary'] = imp_salary_mean2.transform(testX[['Salary']])

print(testX)
print()
print('Number of missing values:')
print(testX.isnull().sum())

8.3.1 Train-Test Split

df_practice_knn = df_practice.copy()
df_practice_knn

x = df_practice_knn.drop('Target_Var', axis=1)
y = df_practice_knn['Target_Var']

trainX, testX, trainY, testY = train_test_split(x, y, test_size = 0.2)

print()
print('trainX')
print()
print(trainX)
print('-------------------------')
print()
print('testX')
print()
print(testX)

8.3.2 Fit&Transform (trainX)

# Fit and Transform trainX column 'Age' and 'Salary'
imp_age_salary_knn1 = KNNImputer(n_neighbors=2)
trainX[['Age', 'Salary']] = imp_age_salary_knn1.fit_transform(trainX[['Age', 'Salary']])

print(trainX)
print()
print('Number of missing values:')
print(trainX.isnull().sum())

pk.dump(imp_age_salary_knn1, open('imp_age_salary_knn1.pkl', 'wb'))

8.3.3 Transform (testX)

I’ll leave out again the part about model training at this point, since it would only be fictitiously presented anyway.

testX.isnull().sum()

# Transform testX column 'Age' and 'Salary'
testX[['Age', 'Salary']] = imp_age_salary_knn1.transform(testX[['Age', 'Salary']])

print(testX)
print()
print('Number of missing values:')
print(testX.isnull().sum())