Artist Detection from Artwork
In this blog I will use Convolutional Neural Network to classify images of different artists. For this project I have used dataset from https://www.kaggle.com/ikarus777/best-artworks-of-all-time.
This dataset contains 8764 paintnings of most influential artists of all time. My intuition is most artist has signature stroke, theme, choice of medium, and color preference and their art can be identified from their painting styles prevalent in their artwork which could help retrieve artwork that has not been identified with any artist.
For building this art classifier, I have used keras from Tensorflow library. Following is how I have developed the classifier
Lets import necessary libraries to help with machine learning, deep learning, and visualization
import osimport numpy as np
import pandas as pd import randomfrom sklearn import metricsimport tensorflow as tf
from tensorflow import keras
from tensorflow.keras import models, preprocessing, layers, callbacks, optimizers
import sklearnimport matplotlib.pyplot as plt
import seaborn as sns
Now lets read the artist details
img_dir = "../input/images/images/"label_df = pd.read_csv("../input/artists.csv")
label_df.head()
Lets visualize distribution of image count of artists
df = label_df
df = df.sort_values(by=['name'], ascending=True)figsize=(20, 5)ticksize = 14
titlesize = ticksize + 8
labelsize = ticksize + 5xlabel = "Artist"
ylabel = "Painting Count"title = "Painting Count by Artist"
params = {'figure.figsize' : figsize,
'axes.labelsize' : labelsize,
'axes.titlesize' : titlesize,
'xtick.labelsize': ticksize,
'ytick.labelsize': ticksize}plt.rcParams.update(params)col1 = "name"
col2 = "paintings"sns.barplot(x=col1, y=col2, data=df)
plt.title(title)
plt.xlabel(xlabel)
plt.ylabel(ylabel)
plt.xticks(rotation=90)
plt.plot()
Lets see few paintings of famous artists
Edvard Munch
Years: 1863–1944 , Genre: Symbolism,Expressionism , Nationality: Norwegian , Number of Paintings: 67
Frida Kahlo
Years: 1907–1954 , Genre: Primitivism,Surrealism , Nationality: Mexican , Number of Paintings: 120
Michelangelo
Years: 1475–1564 , Genre: High Renaissance , Nationality: Italian , Number of Paintings: 49
Pablo Picasso
Years: 1881–1973 , Genre: Cubism , Nationality: Spanish , Number of Paintings: 439
Salvador Dali
Years: 1904–1989 , Genre: Surrealism , Nationality: Spanish , Number of Paintings: 139
Vincent van Gogh
Years: 1853–1890 , Genre: Post-Impressionism , Nationality: Dutch , Number of Paintings: 877
Now we can start seting parameters for data preparation and augmentation. I have rescaled all the images and set image size to 150 and set batch size to 128. Alternatively, smaller batch size will yield to more robust models. But lets, go with one I was experimenting with.
rescale = 1.0/255
IMG_SIZE = 150
TARGET_SIZE = (IMG_SIZE, IMG_SIZE)
CLASSES = os.listdir(img_dir)
NUM_CLASSES = len(CLASSES)
BATCH_SIZE = 128train_batch_size = BATCH_SIZE
validation_batch_size = BATCH_SIZE * 5
test_batch_size = BATCH_SIZE * 5
As data set is highly imalanced, we need to calculate class weights for all the classes in the dataset
# Calculate Class Weights
def get_weight(y, NUM_CLASSES):
class_weights = sklearn.utils.class_weight.compute_class_weight('balanced', np.unique(y), y)
return dict(enumerate(class_weights))Now lets use Keras preprocesing unit’s ImageDataGenerator too preprocess image dataset along with performing image augmentation. I have added rotation upto 45, width and height shift of 0.2x and horizontal flip. The added augmentation of the data will help model.
Also, I have set a validation set with 5% of dataset. As I have much less number of images for 51 classes, I have set validation dataset to be much less than usual. Usually, 15–25% of data set should be in validation set. More specifically, training, validation, and testing should be 60–20–20 or such ratio.
train_datagen = preprocessing.image.ImageDataGenerator(
rescale=rescale,
rotation_range=45,
width_shift_range=0.2,
height_shift_range=0.2,
horizontal_flip=True,
validation_split=0.05)train_generator = train_datagen.flow_from_directory(
img_dir,
target_size=TARGET_SIZE,
classes=CLASSES,
class_mode="categorical",
batch_size=train_batch_size,
shuffle=True,
seed=42,
subset='training')validation_generator = train_datagen.flow_from_directory(
img_dir,
classes=CLASSES,
target_size=TARGET_SIZE,
class_mode="categorical",
batch_size=validation_batch_size,
shuffle=True,
seed=42,
subset='validation')class_weights = get_weight(train_generator.classes, NUM_CLASSES)steps_per_epoch = len(train_generator)
validation_steps = len(validation_generator)
Now, lets define the model. I have used InceptionResNetV2 as base model. As the model of identifying artists from artwork is intricate, I have chosen InceptionResNetV2 rather than more simpler models like ResNet50.
I have added more layers to the basemodel. For each added layer, I have added batch normalization. Batch normalization can significantly reduce training time, reduce the problem of covariant shift. Instead of using normalization on just input like other machine learning dataset, I have added Batch normalization after each layer. Also, I have added dropout upto 50% for each additional layer, which has significantly improved generalization and improved model performance on validation dataset.
I have added Dense layer with 256 unit, but later experimented with more units including 512, 1024.
INPUT_SHAPE = (IMG_SIZE, IMG_SIZE, 3)weights = 'imagenet',
dense_units = 256inputs = layers.Input(INPUT_SHAPE)base_model = tf.keras.applications.InceptionResNetV2(
include_top=False,
weights="imagenet",
input_shape=INPUT_SHAPE,
)
x = base_model.output
x = layers.BatchNormalization()(x)
x = layers.Dropout(0.5)(x)
x = layers.GlobalAveragePooling2D()(x)
x = layers.BatchNormalization()(x)
x = layers.Dropout(0.5)(x)x = layers.Dense(dense_units)(x)
x = layers.BatchNormalization()(x)
x = layers.Activation(activation='relu')(x)
x = layers.Dropout(0.5)(x)outputs = layers.Dense(NUM_CLASSES, activation="softmax")(x)
model = keras.Model(inputs=base_model.input, outputs=outputs)# model.summary()
Lets define callbacks. I have added early stopper to prevent model overfitting, also added function to reduce learning rate to help model reach global minima.
OPTIMIZER = optimizers.Adam(learning_rate = 0.0001)EARLY_STOPPING = callbacks.EarlyStopping(
monitor='val_loss',
patience=5,
verbose=1,
restore_best_weights=True)
REDUCE_LR = callbacks.ReduceLROnPlateau(
monitor='val_loss',
factor=0.5,
patience=1,
min_lr=0.000001,
verbose=1)CALLBACKS = [REDUCE_LR, EARLY_STOPPING]
Model is trained for 20 epochs with classweights, for loss categorical crossentropy was used. But Sparse categorical crossentropy can also be used. For accuracy, F-1 score was used.
model.compile(optimizer=OPTIMIZER, loss='categorical_crossentropy', metrics=['accuracy'])
VERBOSE = 1
EPOCHS = 20print("Trainning Model ...\n")
history = model.fit_generator(
train_generator,
steps_per_epoch=steps_per_epoch,
epochs=EPOCHS,
verbose=VERBOSE,
callbacks=CALLBACKS,
validation_data=validation_generator,
validation_steps=validation_steps,
class_weight=class_weights
)
Following is the training result
Trainning Model ...Epoch 1/20
66/66 [==============================] - 321s 5s/step - loss: 4.3694 - acc: 0.0258 - val_loss: 3.8243 - val_acc: 0.0507
Epoch 2/20
66/66 [==============================] - 225s 3s/step - loss: 4.0910 - acc: 0.0362 - val_loss: 3.7736 - val_acc: 0.1184
Epoch 3/20
66/66 [==============================] - 232s 4s/step - loss: 3.8778 - acc: 0.0798 - val_loss: 3.6010 - val_acc: 0.1522
Epoch 4/20
66/66 [==============================] - 234s 4s/step - loss: 3.4394 - acc: 0.1410 - val_loss: 3.3721 - val_acc: 0.2271
Epoch 5/20
66/66 [==============================] - 235s 4s/step - loss: 3.0210 - acc: 0.2036 - val_loss: 3.0342 - val_acc: 0.3019
Epoch 6/20
66/66 [==============================] - 239s 4s/step - loss: 2.6962 - acc: 0.2652 - val_loss: 2.6467 - val_acc: 0.3671
Epoch 7/20
66/66 [==============================] - 239s 4s/step - loss: 2.3670 - acc: 0.3300 - val_loss: 2.5613 - val_acc: 0.4034
Epoch 8/20
66/66 [==============================] - 241s 4s/step - loss: 2.1304 - acc: 0.3734 - val_loss: 2.1946 - val_acc: 0.4155
Epoch 9/20
66/66 [==============================] - 245s 4s/step - loss: 1.8924 - acc: 0.4200 - val_loss: 2.0486 - val_acc: 0.4469
Epoch 10/20
66/66 [==============================] - 249s 4s/step - loss: 1.7094 - acc: 0.4572 - val_loss: 1.9722 - val_acc: 0.4734
Epoch 11/20
66/66 [==============================] - 249s 4s/step - loss: 1.5027 - acc: 0.5031 - val_loss: 1.9029 - val_acc: 0.4903
Epoch 12/20
66/66 [==============================] - 254s 4s/step - loss: 1.3059 - acc: 0.5548 - val_loss: 1.8206 - val_acc: 0.5048
Epoch 13/20
66/66 [==============================] - 255s 4s/step - loss: 1.1500 - acc: 0.5916 - val_loss: 1.8078 - val_acc: 0.5048
Epoch 14/20
66/66 [==============================] - 259s 4s/step - loss: 1.0319 - acc: 0.6202 - val_loss: 1.6887 - val_acc: 0.5580
Epoch 15/20
66/66 [==============================] - 259s 4s/step - loss: 0.8887 - acc: 0.6567 - val_loss: 1.5898 - val_acc: 0.5797
Epoch 16/20
65/66 [============================>.] - ETA: 3s - loss: 0.7832 - acc: 0.6877
Epoch 00016: ReduceLROnPlateau reducing learning rate to 4.999999873689376e-05.
66/66 [==============================] - 277s 4s/step - loss: 0.7824 - acc: 0.6886 - val_loss: 1.6000 - val_acc: 0.5725
Epoch 17/20
66/66 [==============================] - 263s 4s/step - loss: 0.6824 - acc: 0.7161 - val_loss: 1.4691 - val_acc: 0.5821
Epoch 18/20
65/66 [============================>.] - ETA: 3s - loss: 0.5901 - acc: 0.7482
Epoch 00018: ReduceLROnPlateau reducing learning rate to 2.499999936844688e-05.
66/66 [==============================] - 270s 4s/step - loss: 0.5882 - acc: 0.7480 - val_loss: 1.5289 - val_acc: 0.5845
Epoch 19/20
66/66 [==============================] - 272s 4s/step - loss: 0.5300 - acc: 0.7620 - val_loss: 1.4138 - val_acc: 0.6425
Epoch 20/20
65/66 [============================>.] - ETA: 3s - loss: 0.4889 - acc: 0.7793
Epoch 00020: ReduceLROnPlateau reducing learning rate to 1.249999968422344e-05.
66/66 [==============================] - 271s 4s/step - loss: 0.4945 - acc: 0.7791 - val_loss: 1.4553 - val_acc: 0.6159
Lets visualize training
def plot_performance(history=None, figure_directory=None):
xlabel = 'Epoch'
legends = ['Training', 'Validation']# ylim_pad = [0.1, 0.005]
ylim_pad = [0, 0.5]
plt.figure(figsize=(20, 5)) # Plot training & validation Accuracy values y1 = history.history['acc']
y2 = history.history['val_acc'] min_y = min(min(y1), min(y2))-ylim_pad[0]
max_y = max(max(y1), max(y2))+ylim_pad[0]
min_y = 0
max_y = 1
plt.subplot(121) plt.plot(y1)
plt.plot(y2) plt.title('Model Accuracy\n', fontsize=17)
plt.xlabel(xlabel, fontsize=15)
plt.ylabel('Accuracy', fontsize=15)
plt.ylim(min_y, max_y)
plt.legend(legends, loc='upper left')
plt.grid()
# Plot training & validation loss values y1 = history.history['loss']
y2 = history.history['val_loss'] min_y = min(min(y1), min(y2))-ylim_pad[1]
max_y = max(max(y1), max(y2))+ylim_pad[1]# min_y = 0
# max_y = 4 plt.subplot(122) plt.plot(y1)
plt.plot(y2) plt.title('Model Loss\n', fontsize=17)
plt.xlabel(xlabel, fontsize=15)
plt.ylabel('Loss', fontsize=15)
plt.ylim(min_y, max_y)
plt.legend(legends, loc='upper left')
plt.grid()
plt.show()plot_performance(history=history)
Lets validate
validation_generator_test = train_datagen.flow_from_directory(
img_dir,
classes=CLASSES,
target_size=TARGET_SIZE,
class_mode="categorical",
batch_size=validation_batch_size,
shuffle=False,
seed=42,
subset='validation')y_trues = validation_generator_test.labels
y_preds = model.predict(validation_generator_test)
y_preds = y_preds.argmax(axis=1)
Lets generate confusion matrix
matrix = metrics.confusion_matrix(y_trues, y_preds)plt.figure(figsize = (18,10))
sns.heatmap(matrix/np.max(matrix), cmap='Blues')
plt.show()
print(metrics.classification_report(y_trues, y_preds, digits=3))precision recall f1-score support 0 1.000 1.000 1.000 1
1 0.000 0.000 0.000 16
2 0.800 0.667 0.727 12
3 0.667 0.500 0.571 4
4 0.750 1.000 0.857 6
5 0.667 0.750 0.706 8
6 0.727 0.615 0.667 13
7 0.773 0.395 0.523 43
8 0.600 0.600 0.600 5
9 0.750 1.000 0.857 6
10 1.000 1.000 1.000 2
11 0.000 0.000 0.000 4
12 1.000 0.500 0.667 2
13 0.625 0.833 0.714 6
14 0.500 0.800 0.615 5
15 0.500 0.667 0.571 3
16 0.545 0.857 0.667 7
17 1.000 1.000 1.000 3
18 0.545 0.857 0.667 7
19 0.600 0.750 0.667 4
20 1.000 0.750 0.857 4
21 0.500 0.250 0.333 12
22 1.000 0.733 0.846 15
23 0.875 0.500 0.636 14
24 0.923 0.686 0.787 35
25 0.588 0.476 0.526 21
26 1.000 1.000 1.000 3
27 0.400 1.000 0.571 2
28 0.889 0.889 0.889 9
29 0.500 0.500 0.500 8
30 0.400 0.667 0.500 6
31 0.667 1.000 0.800 6
32 0.000 0.000 0.000 1
33 0.571 1.000 0.727 4
34 0.500 0.800 0.615 5
35 1.000 0.889 0.941 9
36 0.000 0.000 0.000 2
37 1.000 1.000 1.000 2
38 0.714 1.000 0.833 5
39 0.833 0.625 0.714 16
40 0.875 0.778 0.824 9
41 0.273 0.750 0.400 4
42 0.667 0.500 0.571 4
43 0.375 0.750 0.500 4
44 0.222 0.667 0.333 3
45 0.600 0.545 0.571 11
46 0.667 0.444 0.533 9
47 0.000 0.000 0.000 3
48 0.414 0.750 0.533 16
49 0.889 0.889 0.889 9
50 0.400 0.667 0.500 6 accuracy 0.623 414
macro avg 0.623 0.672 0.624 414
weighted avg 0.672 0.623 0.623 414
References:
Full notebook can be found here:
Project Link: https://github.com/anjanatiha/Artist-Prediction-from-Artworks
Dataset - https://www.kaggle.com/ikarus777/best-artworks-of-all-time
Thanks for reading