Zero-Shot Text Classification: Sorting Emails Without Training Data

Hello, tech enthusiasts! 👋

Imagine your team inbox has 400 new emails every week. Some are invoices. Some are complaints. Some are meeting requests. Someone has to read each one and move it to the right folder. That is slow and boring work.

The usual way to automate this is “training” a model. Training means you show the computer thousands of sorted examples, so it learns the pattern. But most small offices do not have thousands of sorted examples. Good news: you can skip that step. Today we will use zero-shot classification.

“Zero-shot” means the model has seen zero examples of your groups. You just give it the group names, like “invoice” or “complaint”, and it guesses which one fits best. It is like asking a smart new colleague to sort mail on their first day. They have never seen your inbox, but they understand what an invoice looks like.

What you need

  • Python 3.9 or newer (https://www.python.org)
  • About 2 GB of free disk space for the model
  • A normal laptop. No graphics card needed, it runs on the CPU.
  • A few emails saved as text (we will use made-up examples)

Step 1: Install the libraries

Open a terminal and install three packages. transformers is the Hugging Face library that runs the model. torch does the math. sentencepiece helps the model split words into small pieces.

pip install transformers torch sentencepiece

Step 2: Load the model

We will use a model called MoritzLaurer/mDeBERTa-v3-base-mnli-xnli. It is multilingual, which means it understands many languages, not only English. It was trained on a task called NLI (Natural Language Inference). NLI means the model checks if one sentence “fits” another sentence. Zero-shot classification uses this skill: it asks “Does this email fit the sentence *This email is about an invoice*?”

from transformers import pipeline

# device=-1 means "use the CPU"
classifier = pipeline(
    "zero-shot-classification",
    model="MoritzLaurer/mDeBERTa-v3-base-mnli-xnli",
    device=-1,
)

The first run downloads the model, so it takes a few minutes. After that it loads from your disk.

Step 3: Try one email

Now give the model an email and your list of candidate labels. Candidate labels are simply the group names you want.

labels = ["invoice", "complaint", "meeting request", "job application", "newsletter"]

email = "Hi, please find attached invoice #1042 for the March order. Payment is due in 30 days."

result = classifier(
    email,
    candidate_labels=labels,
    hypothesis_template="This email is about {}.",
)

print(result["labels"][0], round(result["scores"][0], 2))

The result has two lists. labels is sorted from best to worst. scores holds the confidence for each label. Confidence is a number from 0 to 1. It shows how sure the model is. Here you should see “invoice” with a high score.

The hypothesis_template is the sentence the model tests. A clear sentence often gives better results than just the bare label.

Step 4: Sort many emails and save the results

Let’s process a whole list and save everything to a CSV file. A CSV file is a simple table that opens in Excel.

import csv

emails = [
    "I ordered a chair two weeks ago and it still has not arrived. This is very disappointing.",
    "Can we meet on Tuesday at 10 to review the sales report?",
    "Dear team, I would like to apply for the accountant position posted on your website.",
    "Our May newsletter: new products, tips and a special discount for readers.",
    "Please see the attached bill for consulting services in April.",
    "Quick question about the thing we talked about.",
]

rows = []
for text in emails:
    out = classifier(text, candidate_labels=labels, hypothesis_template="This email is about {}.")
    rows.append({
        "email": text,
        "label": out["labels"][0],
        "confidence": round(out["scores"][0], 3),
    })

with open("sorted_emails.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["email", "label", "confidence"])
    writer.writeheader()
    writer.writerows(rows)

print("Saved", len(rows), "emails")

We use encoding="utf-8" so emails in other languages are saved correctly.

Step 5: Review the unsure ones

The model is good, but not perfect. Look at the last email: “Quick question about the thing we talked about.” Even a human cannot sort that one! The model will give it a low confidence score.

So here is a simple rule: trust high scores, and let a person check low scores.

THRESHOLD = 0.6  # change this after you test with your own emails

for row in rows:
    if row["confidence"] < THRESHOLD:
        print(f"CHECK: {row['label']} ({row['confidence']}) -> {row['email'][:60]}")

Now a person only reads a few emails instead of all of them. That is a big time saver.

Common problems

  • Error about sentencepiece or the tokenizer → Run pip install sentencepiece, then restart Python.
  • It is slow → Long emails take more time. Send only the subject and the first 500 characters: text[:500].
  • Two labels are often mixed up (for example “invoice” and “payment reminder”) → Make the label names more different, or join them into one label.
  • Scores look strange with many labels → The scores are shared across all labels and add up to 1. With 10 labels, a top score of 0.4 can still be a good answer. Test a few emails and pick your threshold from what you see.

Wrap-up

You just built an email sorter with no training data at all. You gave the model some label names, and it did the reading for you. You also kept a human in the loop for the unsure cases, which is the safe way to automate.

Next idea: connect this to your mailbox with Python’s built-in imaplib module, and move each email to a folder with the same name as its label. Happy automating!

fsaeed

, , , ,