Recently I set out to train a Transformer model, based on Distil-GPT2, to write something like my mothers’ poetry.
After much searching for the most concise way to do this, I think I’ve figured out a reasonable easy-to-understand approach that works for me in Google Colab.
In [2]:
!pip install fastai
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/ Requirement already satisfied: fastai in /usr/local/lib/python3.7/dist-packages (2.7.9) Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from fastai) (2.23.0) Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from fastai) (1.7.3) Requirement already satisfied: fastcore<1.6,>=1.4.5 in /usr/local/lib/python3.7/dist-packages (from fastai) (1.5.16) Requirement already satisfied: spacy<4 in /usr/local/lib/python3.7/dist-packages (from fastai) (3.4.1) Requirement already satisfied: torch<1.14,>=1.7 in /usr/local/lib/python3.7/dist-packages (from fastai) (1.12.1+cu113) Requirement already satisfied: pillow>6.0.0 in /usr/local/lib/python3.7/dist-packages (from fastai) (7.1.2) Requirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from fastai) (3.13) Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from fastai) (3.2.2) Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from fastai) (1.0.2) Requirement already satisfied: torchvision>=0.8.2 in /usr/local/lib/python3.7/dist-packages (from fastai) (0.13.1+cu113) Requirement already satisfied: fastprogress>=0.2.4 in /usr/local/lib/python3.7/dist-packages (from fastai) (1.0.3) Requirement already satisfied: pip in /usr/local/lib/python3.7/dist-packages (from fastai) (21.1.3) Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from fastai) (1.3.5) Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from fastai) (21.3) Requirement already satisfied: fastdownload<2,>=0.0.5 in /usr/local/lib/python3.7/dist-packages (from fastai) (0.0.7) Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.7/dist-packages (from spacy<4->fastai) (3.3.0) Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.7/dist-packages (from spacy<4->fastai) (2.0.8) Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/dist-packages (from spacy<4->fastai) (1.0.7) Requirement already satisfied: thinc<8.2.0,>=8.1.0 in /usr/local/lib/python3.7/dist-packages (from spacy<4->fastai) (8.1.0) Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.7/dist-packages (from spacy<4->fastai) (4.64.0) Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.10.0,>=1.7.4 in /usr/local/lib/python3.7/dist-packages (from spacy<4->fastai) (1.9.1) Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.9 in /usr/local/lib/python3.7/dist-packages (from spacy<4->fastai) (3.0.9) Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from spacy<4->fastai) (57.4.0) Requirement already satisfied: typer<0.5.0,>=0.3.0 in /usr/local/lib/python3.7/dist-packages (from spacy<4->fastai) (0.4.2) Requirement already satisfied: pathy>=0.3.5 in /usr/local/lib/python3.7/dist-packages (from spacy<4->fastai) (0.6.2) Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.7/dist-packages (from spacy<4->fastai) (1.21.6) Requirement already satisfied: wasabi<1.1.0,>=0.9.1 in /usr/local/lib/python3.7/dist-packages (from spacy<4->fastai) (0.10.1) Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from spacy<4->fastai) (1.0.3) Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy<4->fastai) (2.0.6) Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from spacy<4->fastai) (2.11.3) Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.7/dist-packages (from spacy<4->fastai) (2.4.4) Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy<4->fastai) (3.0.6) Requirement already satisfied: typing-extensions<4.2.0,>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from spacy<4->fastai) (4.1.1) Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from catalogue<2.1.0,>=2.0.6->spacy<4->fastai) (3.8.1) Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->fastai) (3.0.9) Requirement already satisfied: smart-open<6.0.0,>=5.2.1 in /usr/local/lib/python3.7/dist-packages (from pathy>=0.3.5->spacy<4->fastai) (5.2.1) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->fastai) (2022.6.15) Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->fastai) (3.0.4) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->fastai) (1.24.3) Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->fastai) (2.10) Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.7/dist-packages (from thinc<8.2.0,>=8.1.0->spacy<4->fastai) (0.7.8) Requirement already satisfied: click<9.0.0,>=7.1.1 in /usr/local/lib/python3.7/dist-packages (from typer<0.5.0,>=0.3.0->spacy<4->fastai) (7.1.2) Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->spacy<4->fastai) (2.0.1) Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->fastai) (0.11.0) Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->fastai) (2.8.2) Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->fastai) (1.4.4) Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.1->matplotlib->fastai) (1.15.0) Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas->fastai) (2022.1) Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->fastai) (1.1.0) Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->fastai) (3.1.0)
In [3]:
from fastai.text.all import *
import fastai;
fastai.__version__
Out[3]:
'2.7.9'
In [4]:
if torch.cuda.is_available:
print('GPU available')
else:
print('Please set GPU via Edit -> Notebook Settings.')
print(torch.cuda.device_count())
GPU available 1
Next, define the path that will contain our poems, named as $NAME.txt, e.g. “a-fairy-tale.txt”. You may need to create this and upload files.
In [6]:
path = Path('/content/poetry')
path.ls()
Out[6]:
(#86) [Path('/content/poetry/anzac-day.txt'),Path('/content/poetry/trauma.txt'),Path('/content/poetry/enigma.txt'),Path('/content/poetry/gar.txt'),Path('/content/poetry/extreme-feeling.txt'),Path('/content/poetry/memories-of-clivedon.txt'),Path('/content/poetry/regret.txt'),Path('/content/poetry/cloughey.txt'),Path('/content/poetry/holiday-memory.txt'),Path('/content/poetry/paddy.txt')...]
Define our DataLoader, which finds all the .txt files in path, and keeps 10% for the validation set.
In [7]:
dls_lm = TextDataLoaders.from_folder(path, is_lm=True, valid_pct=0.1)
Confirm that we have valid data.
In [9]:
dls_lm.show_batch(max_n=5)
| text | text_ | |
|---|---|---|
| 0 | xxbos * * i remember when … xxmaj xxunk from xxunk xxunk . * * \n\n * xxmaj in primary school we were all given a small bottle of xxunk – about xxunk xxunk – at morning xxunk . \n * xxmaj riding trams after climbing the xxunk stairs to the open air top xxunk . \n * xxmaj riding to local beaches along xxmaj belfast xxunk with lunch in our xxunk | * * i remember when … xxmaj xxunk from xxunk xxunk . * * \n\n * xxmaj in primary school we were all given a small bottle of xxunk – about xxunk xxunk – at morning xxunk . \n * xxmaj riding trams after climbing the xxunk stairs to the open air top xxunk . \n * xxmaj riding to local beaches along xxmaj belfast xxunk with lunch in our xxunk bags |
| 1 | xxmaj night time xxunk carry cello in canvas case on my shoulder . \n * xxmaj prom concert on hot , xxunk summer night standing in xxmaj arena in bare feet . xxmaj travelled home with xxunk shoes hidden in xxunk . \n * xxmaj xxunk xxunk . xxmaj xxunk xxunk have left their mark . xxmaj xxunk i climb , xxunk xxunk . xxmaj longing for bed and comfort for feet | night time xxunk carry cello in canvas case on my shoulder . \n * xxmaj prom concert on hot , xxunk summer night standing in xxmaj arena in bare feet . xxmaj travelled home with xxunk shoes hidden in xxunk . \n * xxmaj xxunk xxunk . xxmaj xxunk xxunk have left their mark . xxmaj xxunk i climb , xxunk xxunk . xxmaj longing for bed and comfort for feet . |
| 2 | . xxmaj local xxunk xxunk to plans to build more xxunk and xxunk about where they should be built . xxmaj the xxunk xxunk of xxunk and open space xxunk like a giant xxunk xxunk through every xxunk and seems to be just as xxunk . \n xxmaj the sad fact is that within a very short time the traffic jam will be back . \n\n xxmaj coming back to xxmaj melbourne | xxmaj local xxunk xxunk to plans to build more xxunk and xxunk about where they should be built . xxmaj the xxunk xxunk of xxunk and open space xxunk like a giant xxunk xxunk through every xxunk and seems to be just as xxunk . \n xxmaj the sad fact is that within a very short time the traffic jam will be back . \n\n xxmaj coming back to xxmaj melbourne after |
| 3 | xxunk xxunk train that ran on its track along the side of the road . i wanted to see where xxmaj xxunk had lived and view xxmaj xxunk xxmaj xxunk and the xxunk of the xxmaj xxunk . i was told to be out early as often it was xxunk lost in the mist . i stood on the xxunk point high above the clouds on a fresh sunny morning and gazed | xxunk train that ran on its track along the side of the road . i wanted to see where xxmaj xxunk had lived and view xxmaj xxunk xxmaj xxunk and the xxunk of the xxmaj xxunk . i was told to be out early as often it was xxunk lost in the mist . i stood on the xxunk point high above the clouds on a fresh sunny morning and gazed across |
| 4 | \n xxmaj for the distant point , xxunk by the xxunk \n xxmaj gathering treasures as we go to share and show \n xxmaj xxunk xxunk with xxunk colours \n xxmaj xxunk , xxunk , xxunk and xxunk that xxunk xxunk \n xxmaj hiding in large sandy holes xxunk the tall bracken \n xxmaj we watch the xxunk ’s xxunk xxunk \n xxmaj waiting our chance to run fast for the xxunk | xxmaj for the distant point , xxunk by the xxunk \n xxmaj gathering treasures as we go to share and show \n xxmaj xxunk xxunk with xxunk colours \n xxmaj xxunk , xxunk , xxunk and xxunk that xxunk xxunk \n xxmaj hiding in large sandy holes xxunk the tall bracken \n xxmaj we watch the xxunk ’s xxunk xxunk \n xxmaj waiting our chance to run fast for the xxunk with |
Create a “Learner”, which is a FastAI object that encapsulates data loading, hyperparameters, and the model itself.
In [10]:
learn = language_model_learner(dls_lm, AWD_LSTM, metrics=[accuracy, Perplexity()], path=path, wd=0.1).to_fp16()
100.00% [105070592/105067061 00:09<00:00]
Out[10]:
<fastai.text.learner.LMLearner at 0x7f98e2f27550>
Do one cycle of training to confirm that everything works, with an initial learning rate of 1e-2.
In [11]:
learn.fit_one_cycle(1, 1e-2)
| epoch | train_loss | valid_loss | accuracy | perplexity | time |
|---|---|---|---|---|---|
| 0 | 4.607269 | 4.493447 | 0.229802 | 89.429161 | 00:04 |
Now that we know it works, run the model through 100 fine-tuning iterations. Your results will vary because my data here may have been fine tuned several times while writing this notebook.
In [23]:
learn.fine_tune(100)
| epoch | train_loss | valid_loss | accuracy | perplexity | time |
|---|---|---|---|---|---|
| 0 | 3.053174 | 3.644066 | 0.295351 | 38.247047 | 00:00 |
| epoch | train_loss | valid_loss | accuracy | perplexity | time |
|---|---|---|---|---|---|
| 0 | 3.030195 | 3.643797 | 0.297637 | 38.236744 | 00:00 |
| 1 | 3.020059 | 3.643773 | 0.298018 | 38.235821 | 00:00 |
| 2 | 3.014916 | 3.643056 | 0.299162 | 38.208420 | 00:00 |
| … redacted… | |||||
| 82 | 2.125976 | 3.867096 | 0.291159 | 47.803360 | 00:00 |
| 83 | 2.125021 | 3.867073 | 0.289634 | 47.802277 | 00:00 |
| 84 | 2.114496 | 3.868427 | 0.289253 | 47.867023 | 00:00 |
| 85 | 2.114918 | 3.869371 | 0.289253 | 47.912224 | 00:00 |
| 86 | 2.114381 | 3.870305 | 0.288872 | 47.957001 | 00:00 |
| 87 | 2.105759 | 3.870988 | 0.290396 | 47.989796 | 00:00 |
| 88 | 2.098240 | 3.872298 | 0.290777 | 48.052708 | 00:00 |
| 89 | 2.091323 | 3.872824 | 0.290396 | 48.077965 | 00:00 |
| 90 | 2.088586 | 3.872958 | 0.290396 | 48.084408 | 00:00 |
| 91 | 2.085745 | 3.872825 | 0.290396 | 48.077999 | 00:00 |
| 92 | 2.080667 | 3.873065 | 0.290396 | 48.089554 | 00:00 |
| 93 | 2.076997 | 3.873293 | 0.290396 | 48.100517 | 00:00 |
| 94 | 2.072073 | 3.873533 | 0.290015 | 48.112068 | 00:00 |
| 95 | 2.072201 | 3.873621 | 0.290015 | 48.116310 | 00:00 |
| 96 | 2.072754 | 3.873685 | 0.290396 | 48.119385 | 00:00 |
| 97 | 2.073644 | 3.873664 | 0.290396 | 48.118389 | 00:00 |
| 98 | 2.076322 | 3.873671 | 0.290396 | 48.118710 | 00:00 |
| 99 | 2.074253 | 3.873690 | 0.290396 | 48.119637 | 00:00 |
Here we save the model state.
Later you can run learn = learn.load('100epochs') to load.
Loaded models are frozen. You unfreeze and train like this:
learn.unfreeze()
learn.fit_one_cycle(10, 1e-3)In [28]:
learn.save('100epochs')
Out[28]:
Path('/content/poetry/models/100epochs.pth')
Tou can save just the encoder like this, for example to plug into a classifier:
In [30]:
learn.save_encoder('finetuned')
In [31]:
# let's do some prediction
TEXT = "A Perfect Afternoon.\nIn the summer of 1967 I was"
N_WORDS = 200
N_SENTENCES = 2
preds = [learn.predict(TEXT, N_WORDS, temperature=0.75)
for _ in range(N_SENTENCES)]
In [32]:
print("\n".join(preds))
a Perfect Afternoon . In the summer of xxunk i was glad to rest . How could anyone feel the fresh glow of the early morning . The warmth of the early summer sunlight meant stirring . Once grey we slipped around the road . Moving slowly we saw two girls race against their bikes . We drove the bikes to the downhill path . As they paused , Peter felt at his pace . He saw his children rise and rise . The other scouts came round and saw him rise . Suddenly his face came from its higher ground . It was a quiet day . He gazed at himself as he walked towards it . He wondered what he would start his day . He would be back to contact with his mother . He would say he had been in the same city in the summer as he found the garden so he could get his parents to join them . He wondered why he was not so tall and so he could get his breath back . He stood close and ran slowly . Looking down he a Perfect Afternoon . In the summer of xxunk i was first to ask the girls about the boys too . Soon it was hard work and they just seemed to be climbing the downhill track so that would be great . Soon the boys were on the same track . Soon i was on the top of the hill . My father was out of the bag and he came up to see Kathy ’s father . His father seemed to be out of the bag . It seemed to be a DARK . As the children came round and were there you could see the younger younger sisters . One guy ’s a tall hand . They seemed to be coming close in on a sandy top with a ball and the boys were just coming to join them . It was easy for parents to be close . They talked with a maturity beyond their years . Mum was really looking at her mother ’s side when she was first and the situation came different . She had a lot of fun and did not want to know about anyone , so
We can fix the punctuation in a very basic way with this simple function:
In [36]:
import re
fix_spaces = re.compile(r'\s*([?!.,]+(?:\s+[?!.,]+)*)\s*')
def fix_punctuation(text):
return fix_spaces.sub(lambda x: "{} ".format(x.group(1).replace(" ", "")), text)
print("\n".join(map(fix_punctuation, preds)))
a Perfect Afternoon. In the summer of xxunk i was glad to rest. How could anyone feel the fresh glow of the early morning. The warmth of the early summer sunlight meant stirring. Once grey we slipped around the road. Moving slowly we saw two girls race against their bikes. We drove the bikes to the downhill path. As they paused, Peter felt at his pace. He saw his children rise and rise. The other scouts came round and saw him rise. Suddenly his face came from its higher ground. It was a quiet day. He gazed at himself as he walked towards it. He wondered what he would start his day. He would be back to contact with his mother. He would say he had been in the same city in the summer as he found the garden so he could get his parents to join them. He wondered why he was not so tall and so he could get his breath back. He stood close and ran slowly. Looking down he a Perfect Afternoon. In the summer of xxunk i was first to ask the girls about the boys too. Soon it was hard work and they just seemed to be climbing the downhill track so that would be great. Soon the boys were on the same track. Soon i was on the top of the hill. My father was out of the bag and he came up to see Kathy ’s father. His father seemed to be out of the bag. It seemed to be a DARK. As the children came round and were there you could see the younger younger sisters. One guy ’s a tall hand. They seemed to be coming close in on a sandy top with a ball and the boys were just coming to join them. It was easy for parents to be close. They talked with a maturity beyond their years. Mum was really looking at her mother ’s side when she was first and the situation came different. She had a lot of fun and did not want to know about anyone, so
Now let’s try something more like poetry. Note that I have changed the temperature to 0.9 to induce more randomness and less repetition.
In [42]:
TEXT = "A Beautiful Event.\nWavering candlelights bathes the young child’s face"
N_WORDS = 200
N_SENTENCES = 2
preds = [learn.predict(TEXT, N_WORDS, temperature=0.9)
for _ in range(N_SENTENCES)]
In [43]:
print("\n".join(map(fix_punctuation, preds)))
a Xxunk Event. Xxunk xxunk xxunk the young child ’s face To the young man He has his breath, Create a new world, Now you go into the family Lost in the world Now you stand close by you. ’ Daniel, Daniel, and Daniel looked up at the water on a hot day, and dressed in the dark sky. We slipped around the snow, pausing at the boat stop. “ daniel, Daniel, i ’m here ” Daniel stands beside your feet. He stands tall in the wind, watching from the below. He waking in a cold wind as he waits for his turn. He is late enjoying the warmth of the day and then wind the warmth of the sun away. As the sun briefly rises, Daniel suddenly moves out of the mist and face the large black night sky. Sunlight is moving slowly. Leaving the little sun waiting for the sunlight to rise, Daniel waits outside his bedroom window a Xxunk Event. Xxunk xxunk xxunk the young child ’s face ’s face is the answer. Open the door in an upstairs window You can hear the bell rise in. ‘ let ’s hope you will be caught. ’ ‘ what ’s the hour? ’ i hear you gazing at the door and ask you to come out in the fridge. i wait for you to ask you to come away and find the car. i ’m late for the night and we wander slowly among the slow moving river decker cars. As you are there for two voices, i hear a voice calling out into the bus. He calls and has been riding along the bus track to see him. Luckily he said that he could get everything in time for his friend. Next day he was coming home to join with local man Peter Daniel. As they walked along it we saw him riding along with the young man watching the man ’s face. Daniel was coming to join them. He paused to see them
In [ ]: