GCN¶
Node Property Prediction
In this section we will build our first graph neural network using PyTorch Geometric. Then we will apply it to the task of node property prediction (node classification).
Specifically, we will use GCN as the foundation for your graph neural network (Kipf et al. (2017)). To do so, we will work with PyG’s built-in GCNConv layer.
Setup¶
[1]:
import os
import torch
import pandas as pd
import torch.nn.functional as F
print(torch.__version__)
# The PyG built-in GCNConv
from torch_geometric.nn import GCNConv
import torch_geometric.transforms as T
from ogb.nodeproppred import PygNodePropPredDataset, Evaluator
1.11.0
Load and Preprocess the Dataset¶
[2]:
if 'IS_GRADESCOPE_ENV' not in os.environ:
dataset_name = 'ogbn-arxiv'
dataset = PygNodePropPredDataset(name=dataset_name,
transform=T.ToSparseTensor())
data = dataset[0]
# Make the adjacency matrix to symmetric
data.adj_t = data.adj_t.to_symmetric()
device = 'cuda' if torch.cuda.is_available() else 'cpu'
# If you use GPU, the device should be cuda
print('Device: {}'.format(device))
data = data.to(device)
split_idx = dataset.get_idx_split()
train_idx = split_idx['train'].to(device)
Device: cuda
GCN Model¶
Now we will implement our GCN model!
Please follow the figure below to implement the forward function.
[3]:
class GCN(torch.nn.Module):
def __init__(self, input_dim, hidden_dim, output_dim, num_layers,
dropout, return_embeds=False):
# TODO: Implement this function that initializes self.convs,
# self.bns, and self.softmax.
super(GCN, self).__init__()
# A list of GCNConv layers
self.convs = None
# A list of 1D batch normalization layers
self.bns = None
# The log softmax layer
self.softmax = None
############# Your code here ############
## Note:
## 1. You should use torch.nn.ModuleList for self.convs and self.bns
## 2. self.convs has num_layers GCNConv layers
## 3. self.bns has num_layers - 1 BatchNorm1d layers
## 4. You should use torch.nn.LogSoftmax for self.softmax
## 5. The parameters you can set for GCNConv include 'in_channels' and
## 'out_channels'. More information please refer to the documentation:
## https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.nn.conv.GCNConv
## 6. The only parameter you need to set for BatchNorm1d is 'num_features'
## More information please refer to the documentation:
## https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm1d.html
## (~10 lines of code)
# 1. use torch.nn.ModuleList for self.convs
# 2. self.convs has num_layers GCNConv layers
# 5. use 'in_channels' and 'out_channels'
self.convs = torch.nn.ModuleList(
[GCNConv(in_channels=input_dim, out_channels=hidden_dim)] +
[GCNConv(in_channels=hidden_dim, out_channels=hidden_dim) for i in range(num_layers-2)] +
[GCNConv(in_channels=hidden_dim, out_channels=output_dim)]
)
# 1. use torch.nn.ModuleList for self.bns
# 3. self.bns has num_layers - 1 BatchNorm1d layers
# 6. The only parameter you need to set for BatchNorm1d is 'num_features'
self.bns = torch.nn.ModuleList(
[torch.nn.BatchNorm1d(num_features=hidden_dim) for i in range(num_layers-1)]
)
# 4. use torch.nn.LogSoftmax for self.softmax
self.softmax = torch.nn.LogSoftmax()
#########################################
# Probability of an element to be zeroed
self.dropout = dropout
# Skip classification layer and return node embeddings
self.return_embeds = return_embeds
def reset_parameters(self):
for conv in self.convs:
conv.reset_parameters()
for bn in self.bns:
bn.reset_parameters()
def forward(self, x, adj_t):
# TODO: Implement this function that takes the feature tensor x,
# edge_index tensor adj_t and returns the output tensor as
# shown in the figure.
out = None
############# Your code here ############
## Note:
## 1. Construct the network as showing in the figure
## 2. torch.nn.functional.relu and torch.nn.functional.dropout are useful
## More information please refer to the documentation:
## https://pytorch.org/docs/stable/nn.functional.html
## 3. Don't forget to set F.dropout training to self.training
## 4. If return_embeds is True, then skip the last softmax layer
## (~7 lines of code)
# 2. use torch.nn.functional.relu and torch.nn.functional.dropout
# 3. set F.dropout training to self.training
# from the first layer to n-1 layer
for conv, bn in zip(self.convs[:-1], self.bns):
x_f = torch.nn.functional.relu(bn(conv(x, adj_t)))
if self.training:
x_f = torch.nn.functional.dropout(x_f, p=self.dropout)
x = x_f
# 4. If return_embeds is True, then skip the last softmax layer
# the last layer
x = self.convs[-1](x, adj_t)
out = x if self.return_embeds else self.softmax(x)
#########################################
return out
[4]:
def train(model, data, train_idx, optimizer, loss_fn):
# TODO: Implement this function that trains the model by
# using the given optimizer and loss_fn.
model.train()
loss = 0
############# Your code here ############
## Note:
## 1. Zero grad the optimizer
## 2. Feed the data into the model
## 3. Slicing the model output and label by train_idx
## 4. Feed the sliced output and label to loss_fn
## (~4 lines of code)
# 1. Zero grad the optimizer
optimizer.zero_grad()
# 2. Feed the data into the model
out = model(data.x, data.adj_t)
# 3. Slicing the model output and label by train_idx
sliced_output, sliced_label = out[train_idx], data.y[train_idx].reshape(-1)
# 4. Feed the sliced output and label to loss_fn
loss = loss_fn(sliced_output, sliced_label)
#########################################
loss.backward()
optimizer.step()
return loss.item()
[5]:
# Test function here
@torch.no_grad()
def test(model, data, split_idx, evaluator):
# TODO: Implement this function that tests the model by
# using the given split_idx and evaluator.
model.eval()
# The output of model on all data
out = None
############# Your code here ############
## (~1 line of code)
## Note:
## 1. No index slicing here
out = model(data.x, data.adj_t)
#########################################
y_pred = out.argmax(dim=-1, keepdim=True)
train_acc = evaluator.eval({
'y_true': data.y[split_idx['train']],
'y_pred': y_pred[split_idx['train']],
})['acc']
valid_acc = evaluator.eval({
'y_true': data.y[split_idx['valid']],
'y_pred': y_pred[split_idx['valid']],
})['acc']
test_acc = evaluator.eval({
'y_true': data.y[split_idx['test']],
'y_pred': y_pred[split_idx['test']],
})['acc']
return train_acc, valid_acc, test_acc
[6]:
# Please do not change the args
if 'IS_GRADESCOPE_ENV' not in os.environ:
args = {
'device': device,
'num_layers': 3,
'hidden_dim': 256,
'dropout': 0.5,
'lr': 0.01,
'epochs': 100,
}
args
[7]:
model = GCN(data.num_features, args['hidden_dim'],
dataset.num_classes, args['num_layers'],
args['dropout']).to(device)
evaluator = Evaluator(name='ogbn-arxiv')
[8]:
import copy
if 'IS_GRADESCOPE_ENV' not in os.environ:
# reset the parameters to initial random value
model.reset_parameters()
optimizer = torch.optim.Adam(model.parameters(), lr=args['lr'])
loss_fn = F.nll_loss
best_model = None
best_valid_acc = 0
for epoch in range(1, 1 + args["epochs"]):
loss = train(model, data, train_idx, optimizer, loss_fn)
result = test(model, data, split_idx, evaluator)
train_acc, valid_acc, test_acc = result
if valid_acc > best_valid_acc:
best_valid_acc = valid_acc
best_model = copy.deepcopy(model)
print(f'Epoch: {epoch:02d}, '
f'Loss: {loss:.4f}, '
f'Train: {100 * train_acc:.2f}%, '
f'Valid: {100 * valid_acc:.2f}% '
f'Test: {100 * test_acc:.2f}%')
C:\Users\user\anaconda3\envs\gnn\lib\site-packages\ipykernel_launcher.py:94: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.
Epoch: 01, Loss: 4.0346, Train: 29.90%, Valid: 32.52% Test: 30.71%
Epoch: 02, Loss: 2.3318, Train: 24.67%, Valid: 22.16% Test: 27.45%
Epoch: 03, Loss: 1.9201, Train: 32.05%, Valid: 27.90% Test: 32.16%
Epoch: 04, Loss: 1.7597, Train: 37.44%, Valid: 34.33% Test: 35.43%
Epoch: 05, Loss: 1.6729, Train: 34.51%, Valid: 30.50% Test: 27.29%
Epoch: 06, Loss: 1.5858, Train: 35.09%, Valid: 32.65% Test: 29.51%
Epoch: 07, Loss: 1.5176, Train: 36.10%, Valid: 34.45% Test: 31.95%
Epoch: 08, Loss: 1.4656, Train: 37.75%, Valid: 36.59% Test: 35.23%
Epoch: 09, Loss: 1.4186, Train: 39.37%, Valid: 39.08% Test: 39.62%
Epoch: 10, Loss: 1.3821, Train: 40.02%, Valid: 37.46% Test: 39.68%
Epoch: 11, Loss: 1.3509, Train: 39.30%, Valid: 33.63% Test: 35.36%
Epoch: 12, Loss: 1.3273, Train: 39.09%, Valid: 31.49% Test: 32.18%
Epoch: 13, Loss: 1.3065, Train: 41.37%, Valid: 34.63% Test: 35.20%
Epoch: 14, Loss: 1.2826, Train: 45.66%, Valid: 41.70% Test: 43.19%
Epoch: 15, Loss: 1.2578, Train: 49.50%, Valid: 48.52% Test: 50.58%
Epoch: 16, Loss: 1.2423, Train: 51.93%, Valid: 52.34% Test: 54.49%
Epoch: 17, Loss: 1.2293, Train: 53.44%, Valid: 54.21% Test: 56.26%
Epoch: 18, Loss: 1.2122, Train: 55.02%, Valid: 56.13% Test: 57.58%
Epoch: 19, Loss: 1.1993, Train: 56.16%, Valid: 57.53% Test: 58.74%
Epoch: 20, Loss: 1.1868, Train: 57.14%, Valid: 58.59% Test: 59.97%
Epoch: 21, Loss: 1.1736, Train: 58.00%, Valid: 59.74% Test: 61.07%
Epoch: 22, Loss: 1.1651, Train: 59.11%, Valid: 60.87% Test: 61.98%
Epoch: 23, Loss: 1.1510, Train: 60.39%, Valid: 62.16% Test: 62.73%
Epoch: 24, Loss: 1.1427, Train: 61.67%, Valid: 63.23% Test: 63.72%
Epoch: 25, Loss: 1.1331, Train: 63.23%, Valid: 64.29% Test: 64.53%
Epoch: 26, Loss: 1.1208, Train: 64.69%, Valid: 64.79% Test: 64.73%
Epoch: 27, Loss: 1.1179, Train: 66.00%, Valid: 65.56% Test: 64.87%
Epoch: 28, Loss: 1.1079, Train: 67.00%, Valid: 66.25% Test: 65.31%
Epoch: 29, Loss: 1.1002, Train: 67.60%, Valid: 66.83% Test: 66.02%
Epoch: 30, Loss: 1.0954, Train: 67.92%, Valid: 67.37% Test: 66.91%
Epoch: 31, Loss: 1.0861, Train: 68.10%, Valid: 67.25% Test: 67.29%
Epoch: 32, Loss: 1.0792, Train: 68.01%, Valid: 67.08% Test: 67.53%
Epoch: 33, Loss: 1.0778, Train: 68.19%, Valid: 67.39% Test: 67.80%
Epoch: 34, Loss: 1.0725, Train: 68.60%, Valid: 68.06% Test: 68.09%
Epoch: 35, Loss: 1.0662, Train: 68.95%, Valid: 68.61% Test: 68.46%
Epoch: 36, Loss: 1.0624, Train: 69.14%, Valid: 68.81% Test: 68.73%
Epoch: 37, Loss: 1.0562, Train: 69.22%, Valid: 68.92% Test: 68.75%
Epoch: 38, Loss: 1.0517, Train: 69.22%, Valid: 68.83% Test: 68.74%
Epoch: 39, Loss: 1.0479, Train: 69.38%, Valid: 68.56% Test: 68.25%
Epoch: 40, Loss: 1.0421, Train: 69.57%, Valid: 68.31% Test: 67.59%
Epoch: 41, Loss: 1.0387, Train: 69.80%, Valid: 68.56% Test: 67.77%
Epoch: 42, Loss: 1.0382, Train: 70.01%, Valid: 69.15% Test: 68.68%
Epoch: 43, Loss: 1.0301, Train: 70.22%, Valid: 69.57% Test: 69.47%
Epoch: 44, Loss: 1.0247, Train: 70.34%, Valid: 69.88% Test: 69.75%
Epoch: 45, Loss: 1.0274, Train: 70.52%, Valid: 70.03% Test: 69.75%
Epoch: 46, Loss: 1.0193, Train: 70.89%, Valid: 70.28% Test: 69.58%
Epoch: 47, Loss: 1.0181, Train: 71.12%, Valid: 70.36% Test: 69.29%
Epoch: 48, Loss: 1.0129, Train: 71.28%, Valid: 70.59% Test: 69.76%
Epoch: 49, Loss: 1.0100, Train: 71.38%, Valid: 70.87% Test: 70.41%
Epoch: 50, Loss: 1.0108, Train: 71.36%, Valid: 70.86% Test: 70.43%
Epoch: 51, Loss: 1.0038, Train: 71.45%, Valid: 70.77% Test: 70.31%
Epoch: 52, Loss: 1.0017, Train: 71.58%, Valid: 70.80% Test: 70.03%
Epoch: 53, Loss: 0.9999, Train: 71.63%, Valid: 70.83% Test: 69.91%
Epoch: 54, Loss: 0.9978, Train: 71.67%, Valid: 70.94% Test: 70.20%
Epoch: 55, Loss: 0.9929, Train: 71.67%, Valid: 70.92% Test: 70.63%
Epoch: 56, Loss: 0.9915, Train: 71.71%, Valid: 71.00% Test: 70.72%
Epoch: 57, Loss: 0.9857, Train: 71.88%, Valid: 71.17% Test: 70.34%
Epoch: 58, Loss: 0.9881, Train: 72.04%, Valid: 70.71% Test: 69.24%
Epoch: 59, Loss: 0.9838, Train: 71.97%, Valid: 70.22% Test: 68.34%
Epoch: 60, Loss: 0.9827, Train: 72.13%, Valid: 70.59% Test: 68.77%
Epoch: 61, Loss: 0.9793, Train: 72.24%, Valid: 71.03% Test: 69.90%
Epoch: 62, Loss: 0.9751, Train: 72.21%, Valid: 71.13% Test: 70.34%
Epoch: 63, Loss: 0.9745, Train: 72.28%, Valid: 70.95% Test: 69.88%
Epoch: 64, Loss: 0.9735, Train: 72.34%, Valid: 70.59% Test: 68.97%
Epoch: 65, Loss: 0.9713, Train: 72.32%, Valid: 70.58% Test: 68.83%
Epoch: 66, Loss: 0.9678, Train: 72.37%, Valid: 70.98% Test: 69.96%
Epoch: 67, Loss: 0.9678, Train: 72.47%, Valid: 71.21% Test: 70.48%
Epoch: 68, Loss: 0.9659, Train: 72.64%, Valid: 71.23% Test: 70.14%
Epoch: 69, Loss: 0.9607, Train: 72.66%, Valid: 70.93% Test: 69.29%
Epoch: 70, Loss: 0.9606, Train: 72.67%, Valid: 70.83% Test: 69.22%
Epoch: 71, Loss: 0.9544, Train: 72.65%, Valid: 71.06% Test: 69.98%
Epoch: 72, Loss: 0.9544, Train: 72.63%, Valid: 71.28% Test: 70.38%
Epoch: 73, Loss: 0.9525, Train: 72.87%, Valid: 71.47% Test: 70.52%
Epoch: 74, Loss: 0.9505, Train: 73.01%, Valid: 71.40% Test: 70.13%
Epoch: 75, Loss: 0.9473, Train: 73.00%, Valid: 71.35% Test: 70.03%
Epoch: 76, Loss: 0.9487, Train: 73.02%, Valid: 71.43% Test: 70.16%
Epoch: 77, Loss: 0.9469, Train: 73.07%, Valid: 71.36% Test: 70.30%
Epoch: 78, Loss: 0.9468, Train: 73.07%, Valid: 71.46% Test: 70.24%
Epoch: 79, Loss: 0.9420, Train: 73.12%, Valid: 71.31% Test: 70.14%
Epoch: 80, Loss: 0.9408, Train: 73.14%, Valid: 71.27% Test: 70.24%
Epoch: 81, Loss: 0.9410, Train: 73.09%, Valid: 71.28% Test: 70.34%
Epoch: 82, Loss: 0.9390, Train: 73.17%, Valid: 71.40% Test: 70.41%
Epoch: 83, Loss: 0.9380, Train: 73.31%, Valid: 71.67% Test: 70.47%
Epoch: 84, Loss: 0.9336, Train: 73.39%, Valid: 71.46% Test: 70.02%
Epoch: 85, Loss: 0.9334, Train: 73.48%, Valid: 71.69% Test: 70.21%
Epoch: 86, Loss: 0.9305, Train: 73.49%, Valid: 71.51% Test: 70.52%
Epoch: 87, Loss: 0.9295, Train: 73.45%, Valid: 71.35% Test: 70.46%
Epoch: 88, Loss: 0.9247, Train: 73.54%, Valid: 71.31% Test: 69.91%
Epoch: 89, Loss: 0.9279, Train: 73.64%, Valid: 71.10% Test: 69.24%
Epoch: 90, Loss: 0.9232, Train: 73.72%, Valid: 71.12% Test: 69.20%
Epoch: 91, Loss: 0.9246, Train: 73.84%, Valid: 71.45% Test: 69.85%
Epoch: 92, Loss: 0.9209, Train: 73.86%, Valid: 71.45% Test: 70.26%
Epoch: 93, Loss: 0.9207, Train: 73.86%, Valid: 71.74% Test: 70.68%
Epoch: 94, Loss: 0.9184, Train: 73.93%, Valid: 71.98% Test: 70.87%
Epoch: 95, Loss: 0.9168, Train: 73.87%, Valid: 71.88% Test: 70.64%
Epoch: 96, Loss: 0.9141, Train: 73.70%, Valid: 71.83% Test: 70.60%
Epoch: 97, Loss: 0.9149, Train: 73.58%, Valid: 71.86% Test: 71.15%
Epoch: 98, Loss: 0.9123, Train: 73.69%, Valid: 71.84% Test: 70.87%
Epoch: 99, Loss: 0.9116, Train: 73.86%, Valid: 71.85% Test: 70.56%
Epoch: 100, Loss: 0.9121, Train: 73.86%, Valid: 71.62% Test: 70.58%
[9]:
best_result = test(best_model, data, split_idx, evaluator)
train_acc, valid_acc, test_acc = best_result
print(f'Best model: '
f'Train: {100 * train_acc:.2f}%, '
f'Valid: {100 * valid_acc:.2f}% '
f'Test: {100 * test_acc:.2f}%')
Best model: Train: 73.93%, Valid: 71.98% Test: 70.87%
C:\Users\user\anaconda3\envs\gnn\lib\site-packages\ipykernel_launcher.py:94: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.
references¶
cs224w colab 2 https://colab.research.google.com/drive/1BRPw3WQjP8ANSFz-4Z1ldtNt9g7zm-bv?usp=sharing