Skip to content
Nat Lee

Keyboard shortcuts

Turn these off if they get in the way of a screen reader or voice control. The setting is saved in this browser.

1
Home
2
Work
3
Projects
4
About
5
Experience
/
Search projects
?
Show this panel

AI/ML

Spin-off · Master's thesis

IMDb Preprocessing Study

Text Preprocessing in Neural Text Classifiers

An unofficial, reproducible re-run of a BlackboxNLP 2018 paper's IMDb experiments: four ways to preprocess the text, two neural models, every run traceable. It started as research for my master's thesis in 2019. That version had three bugs; this one fixes them and says where they were.

Role
Creator & Maintainer
Period
2019 – 2026.11
Status
Completed
Categories
Language & sentimentAI & machine learning
Origin
Master's thesis
IMDb Preprocessing Study, screenshot 1

Overview

Does it matter whether you lowercase, lemmatize or group multiword expressions before training a neural text classifier? Camacho-Collados and Pilehvar measured it in On the Role of Text Preprocessing in Neural Network Architectures (BlackboxNLP 2018). This repository re-implements the IMDb part of that study: pick one of four preprocessing variants (vanilla, lowercased, lemmatized, multiword) and one of two models (CNN or CNN+LSTM), and preproc-imdb trains on the 25,000 training reviews and reports the accuracy on the 25,000 test reviews. Every run leaves a folder with its settings, library versions, training history and every prediction. It started in 2019 as a Python 3 port of the authors' sensecnn code for my master's research, and was rewritten in 2026 on Keras 3.15 and TensorFlow 2.21, fixing three bugs in the 2019 port.

Why I built it

My master's research needed to know whether lowercasing, lemmatizing or joining multiword phrases changes what a neural sentiment model learns, so I re-ran a paper's IMDb experiments. In the rerun, changing the random seed moved accuracy more than changing the preprocessing did.

What it's good for

For anyone wondering whether lowercasing or lemmatizing is worth it: pick a variant and a model, get an accuracy traceable to a folder, and compare it with the paper. Your own preprocessing needs four text files, one command and patience: a CNN+LSTM run takes 13 to 19 minutes on CPU.

The problem

  • The 2019 port had three bugs that quietly changed the numbers
  • Every number in the results table had to be traceable to a run

What I did

  • Rebuilt word ids, embedding rows and early stopping, and documented each 2019 bug next to its fix
  • Each run writes a folder with its settings, versions, history and every prediction

Results

  • A results table that puts this repository's accuracy next to the paper's for every variant and model
  • Close to the paper on CNN+LSTM, for example 88.6 against 88.9 on the vanilla text

Features

  • Four preprocessing variants: vanilla, lowercased, lemmatized and multiword
  • The paper's two models: CNN and CNN+LSTM
  • Trains on the 25,000 IMDb training reviews and tests on the other 25,000
  • Every run saves its settings, library versions, training history and predictions
  • Fixed seeds, so the same setup gives the same accuracy again
  • Pinned Keras and TensorFlow versions, with a Docker option

What I learned

  • The 2019 port numbered words 'count + 2', so words with the same count shared an id and the most frequent words became unknown.
  • The 2019 port also stopped early on the test set; the rewrite trains 3 epochs by default, like the authors' code.
  • Fine-tuning hides missing vectors: on vanilla text the lowercased embeddings covered 13,265 of 19,998 words, yet accuracy barely moved.

How it fits together

Simplified on purpose. The real thing has more arrows and at least one config file nobody understands.

  1. IMDb text variants
    • Vanilla, lowercased
    • Lemmatized, multiword
    • UMBC word embeddings
    → trained by
  2. This onepreproc-imdb
    • Frequency-ranked word ids
    • CNN or CNN+LSTM
    • Keras 3.15, TensorFlow 2.21
    → writes
  3. Run folder
    • results.json
    • predictions.tsv
    • model.keras
    → summarized in
  4. Paper comparison
    • Tables 2 and 3
  • IMDb Preprocessing Study, screenshot 2
  • Spin-off · Master's thesis + Big Data
    Language & sentimentAI & machine learning11 stars on GitHub

    Sentiment Bundle Benchmark

    TF-IDF, embeddings, fine-tuned encoders and a local language model, evaluated on fixed splits across six sentiment datasets. The successor to GURA.

    2026Completed
  • Spin-off · Master's thesis + Big Data
    Language & sentimentAI & machine learning33 stars on GitHub

    GURA

    The earlier BiGRU + self-attention sentiment experiment. Preserved for reference; current comparisons live in Sentiment Bundle Benchmark.

    2019Archived
  • Work · Big Data
    Language & sentimentAI & machine learning

    Sentiment Analysis Platform

    Led the sentiment engine behind a company's social-listening products, reading more than three billion crawled posts. Lesson learned: the internet is mostly complaints.

    2021Completed