Another little snippet that I sometimes forget 🙂
Suppose you have a dataframe with a column that has data in a string format and you need to transform that into a way that a machine learning algorithm can use. One good way is with one-hot encoding which will take the values in the column and create new columns with 1’s & 0’s representing the original data.
Have a look at this dataframe snippet:
| prod | rev | |
|---|---|---|
| 0 | Solaris 11.3 | 2 |
| 1 | Solaris 11.3 | 1 |
| 2 | Solaris 11.4 | 4 |
| 3 | Solaris 11.4 | 5 |
| 4 | Solaris 11.3 | 1 |
One-hot encoding can be use to transform that:
OH = pd.DataFrame(OH_encoder.fit_transform(mydf[['prod']]))
display(OH.head())
pd.concat([mydf, OH], axis=1)| prod | rev | 0 | 1 | 2 | 3 | |
|---|---|---|---|---|---|---|
| 0 | Solaris 11.3 | 2 | 0.0 | 1.0 | 0.0 | 0.0 |
| 1 | Solaris 11.3 | 1 | 0.0 | 1.0 | 0.0 | 0.0 |
| 2 | Solaris 11.4 | 4 | 0.0 | 0.0 | 1.0 | 0.0 |
| 3 | Solaris 11.4 | 5 | 0.0 | 0.0 | 1.0 | 0.0 |
| 4 | Solaris 11.3 | 1 | 0.0 | 1.0 | 0.0 | 0.0 |
And that’s fine for machine learning. Your models and pipelines will just handle it once it added to the original dataframe. But what if you actually want to poke around in the dataframe and use the data yourself? Then it would be really useful to have the label of the column reflect what the data in it actually is. No problem just get the feature names from the encoder and rename the columns on the one-hot dataframe before concating it to the original frame:
column_name = OH.get_feature_names(['prod'])
OH_cols_train.columns=column_name
df3 = pd.concat([df2, OH_cols_train], axis=1)
| prod | rev | prod_Solaris 11.1 | prod_Solaris 11.3 | prod_Solaris 11.4 | |
|---|---|---|---|---|---|
| 0 | Solaris 11.3 | 2 | 0.0 | 1.0 | 0.0 |
| 1 | Solaris 11.3 | 1 | 0.0 | 1.0 | 0.0 |
| 2 | Solaris 11.4 | 4 | 0.0 | 0.0 | 1.0 |
| 3 | Solaris 11.4 | 5 | 0.0 | 0.0 | 1.0 |