license: mit
task_categories:

• text-classification
• feature-extraction
  language:
• en
  tags:
• cybersecurity
• pdf
• malware
• prompt-injection
• evaluation
• benchmark
• embeddings
  size_categories:
• 1K<n<10K
  configs:
• config_name: embeddings
  data_files:
  • split: train
    path: Part_A_Outputs/corpus_embeddings.parquet
• config_name: answers
  data_files:
  • split: train
    path: Part_B_Outputs/part_b_answers.json

Injected PDFs - Model Evaluation

This repository holds the model evaluation stage of a project on detecting harmless-but-real
attack payloads injected into PDF files, together with the artefacts it produced for the
application.

Nothing is trained here. Seven off-the-shelf models are measured against the same 1,100 PDFs,
and the two winners are exported for the app to load.

Everything is in the notebook Final_Project_Evaluation_V1.ipynb. The corpus it reads is the
1,100-file dataset built by the EDA stage,
Cyber-security-final-project/HARMLESS_Synthetic_Injected_PDFs_EDA:
900 injected files against 200 clean ones, 12 attack families, with the payload's location and
type known for every file.

The bar: what does doing nothing score?

This is the number the whole repository is organised around.

The corpus is 81.8% injected, so a "detector" consisting of the single line return "malicious" -
no code, no model, no reading - scores:

A model reporting 85% accuracy has done worse than a constant. So has a model reporting F1 0.88.
Accuracy is never used to rank anything here, and every result below is quoted against this line.
One scoring function, fixed once, judges every model; the evaluation set is frozen at all 1,100
files in their original order, so no model ever sees an easier subset than another.

Part A - Choosing an embedding model

An embedding turns a PDF into a point in space, so "which files look like this one?" becomes
geometry. Three candidates were chosen to differ on the axis this data makes expensive - how much
of a document the model can see - and each was tested on two inputs: the 3,000-character
payload window, and the full document (median ~8,800 tokens).

Metric: family purity@20 - of a file's 20 nearest neighbours, what share carry the same attack
family. Random baseline: 6.8%, computed from the actual class sizes rather than assumed.

Results

Colab T4, SEED = 42, all six combinations completed.

More context is not better - it is worse, and this is Part A's real result. The EDA left an open
question: whole-document embeddings scored badly, but the model that produced that finding could
only see 256 tokens, so nobody knew whether the payload was diluted or simply never read. Nomic
settles it. With a 8,192-token window it reads essentially the whole document - and scores at the
random baseline, after 24 GPU-minutes, twelve times longer than any other run. The same model on a
3,000-character window is the best in the table. On a corpus where the signal occupies 4.5% of the
text, choosing what not to show a model matters more than choosing the model.

Reputation did not transfer. BGE-small is the better-regarded retrieval model on public
benchmarks and has twice MiniLM's context window. It lost to MiniLM on every input and every metric.
Leaderboards are built on prose; PDF internals are not prose.

Where the winner is strong and weak

All twelve families beat the 6.8% baseline, from 2.0x to 7.4x:

The EDA's loud/silent split does not predict this - silent families average 27.3% against the
loud families' 24.4%, and they hold both the best and worst results. "Loud" means a file trips a
keyword flag, and an embedding model does not check a keyword list. What actually predicts the score
is how much readable text the payload adds: llm_prompt_injection tops the table because its
payload is English prose, and ssrf comes last because its payload is a bare IP address.

The decision

Nomic-embed-v1.5 on the payload window, exported as Part_A_Outputs/corpus_embeddings.parquet.
It costs 24x MiniLM's encode time for 5 points of purity, but that cost is paid once - the app
loads the result and only embeds the single PDF a user uploads, about a tenth of a second on either
model. At hundreds of thousands of files the answer would flip to MiniLM.

What is honestly claimed: precision@5 of 35.6% means fewer than 2 of the 5 files shown to a
user are the same kind of attack as the one they uploaded. Far better than chance, and not good.
The app presents these as "the most similar files in the corpus", with the family shown as a label.

Part B - Choosing a detection model

A nearest-neighbour lookup cannot point at a line and say there, that is the injection. So the
detector is a language model, prompted rather than trained: each is shown a PDF extract and answers
in JSON with a verdict, a family from a closed list of twelve, the exact substring that convinced
it, and one sentence of reasoning.

Four models, one per major lab, all quantised identically to 4-bit NF4 so the comparison is not
secretly "whichever models fit the GPU": google/gemma-2-9b-it, Qwen/Qwen2.5-7B-Instruct,
XiaomiMiMo/MiMo-7B-RL, microsoft/Phi-4-mini-instruct. 4,400 generations, greedy decoding,
SEED = 42.

Results

1. Three models beat the constant. The fourth only appears to.

Phi-4-mini scores F1 0.900 - the baseline's number to three decimal places - because it flagged
190 of the 200 clean files. It is not detecting injections, it is very nearly saying yes to
everything. Read alone, its 99.1% recall is the best number in the table. This is exactly why recall
and the false-alarm rate are reported separately and never averaged into "accuracy".

Splitting F1 back into its two halves puts Phi beside the always malicious cross - all recall, no
discrimination - while the other three sit in the top-left corner, catching the attacks and
leaving the clean files alone. Gemma catches 851 of 900 attacks while wrongly flagging 6.

2. No model is riding the positional tell - this was checked, not assumed

For injected files the 3,000-character window is centred on the payload; for clean files it is the
first 3,000 characters. That is a tell, and high recall with a clean sheet on the negatives is what
exploiting it would look like - which is Gemma's profile. So the evidence strings were matched
against the source text:

The near-misses drift or run past the window edge; they are not invented, and they are payloads, not
boilerplate. The models are reading the injection, not its position.

The same check condemns Phi. Of its 190 false alarms, 26% carry no evidence at all and almost
all the rest quote document furniture - Adobe XMP Core 4.0-c321, BINARY 67 bytes. Only 4 of 190
quote anything resembling a payload.

3. No model can name the family reliably, even where its verdict is strong

One family is named by everyone, one by nobody, and in between the models disagree with each other
rather than failing together. A single "family accuracy" figure hides all of it.

4. The missing recall is a format failure, not a detection failure

Roughly one document in seven produced an unreadable answer, and every one is scored as not
injected: 33 of Gemma's 49 missed attacks, and 41 of Qwen's 71, are unreadable answers rather
than wrong ones. A better parser would not recover them - the model either continues the PDF text
it was shown or degenerates into repeated tokens, so there is no verdict inside to rescue. Only
constrained decoding would fix it, and that is a change to the harness.

The decision

Gemma-2-9B, for the verdict and only for the verdict. It wins on F1 (0.969) and on the number
that decides trust (family accuracy 0.630), and no other model is better on both. The margin over
the constant is the honest headline: 0.969 against 0.900 is a 7.7% relative improvement over a
rule that reads nothing at all.

Mimo is the latency fallback. It gives up 0.024 F1 for 2.6x the speed - 77 minutes over the
corpus against 201. If the deployment turns out to be latency-bound rather than accuracy-bound, that
is the trade to make.

At 10.95 s/doc Gemma cannot serve a free Hugging Face Space, which has no GPU. The honest options
are a paid GPU Space, an inference API, or a smaller model for the live demo with these numbers
reported as the offline result.

So the application:

1. Shows Gemma's verdict as the verdict, with its quoted evidence highlighted in the text.
2. Shows the family as a suggestion, never as a verdict, beside Part A's nearest neighbours, so
   a user can judge for themselves where the model is weak.
3. Never hides the baseline - it reports that flagging everything scores F1 0.900.

Limitations

• These are the four models as quantised to 4-bit NF4. A full-precision ranking could differ.
  Nothing here measures them at fp16.
• The Microsoft row is Phi-4-mini standing in for MAI-1, which has no public weights and cannot
  be run locally under the same conditions. No claim about MAI-1 follows from it, and its 95%
  false-alarm rate is a fact about a 3.8B model.
• One model was given one extra token. MiMo's prompt carries a prefilled empty <think> block;
  without it, its reasoning consumed the entire 200-token budget and it emitted no JSON at all, so
  the alternative was not a fairer comparison but no measurement. It changes where MiMo starts
  answering, not what it was shown. The other three are byte-identical to each other.
• This measures detection on injected PDFs, not general PDF malware. The corpus is 1,100
  synthetic files built by injecting harmless payloads into ordinary documents; the EDA established
  that it behaves differently from authored real-world malware.

Repository contents

Final_Project_Evaluation_V1.ipynb        Parts A and B, end to end
App_Creation.ipynb                       how the application wraps these two artefacts
Part_A_Outputs/
  corpus_embeddings.parquet              the winner's vectors - 1,100 x 768 (4.9 MB)
  part_a_results.json                    all six runs, baselines, per-family breakdown, provenance
Part_B_Outputs/
  part_b_answers.json                    every answer from every model - 4,400 rows (3.1 MB)
  part_b_results.json                    the scored head-to-head table
  partB_<model>_shard<0-3>.json          the 16 raw generation shards, as written by the GPU run
images/                                  the 5 figures reproduced above, as the notebook rendered them

corpus_embeddings.parquet

file_id plus e0-e767. Unit-normalised, so a dot product is cosine similarity. Produced by
nomic-ai/nomic-embed-text-v1.5 on the payload_window column with the required
search_document: prefix - all four facts are recorded in part_a_results.json under winner.

part_b_answers.json

One row per model per file, 4 x 1,100:

Verdicts are always re-derived from raw rather than trusted from the stored fields. Because
decoding was greedy the text is final, which means the parser can be corrected without spending GPU
time again - and it was: the first version used the regex \{[^{}]*\}, and since every injected
file carries a marker containing a }, the match truncated exactly when a model quoted its
evidence correctly. It discarded about a third of every shard and biased recall down by ~35
points. The replacement counts braces and tracks string literals.

Loading

import pandas as pd

BASE = "https://huggingface.co/datasets/<HF_REPO_ID>/resolve/main/"

emb = pd.read_parquet(BASE + "Part_A_Outputs/corpus_embeddings.parquet")
answers = pd.read_json(BASE + "Part_B_Outputs/part_b_answers.json")

# nearest neighbours, no index needed at this size
import numpy as np
V = emb.drop(columns="file_id").to_numpy(np.float32)
sims = V @ V.T                      # vectors are unit-normalised
np.fill_diagonal(sims, -np.inf)
top5 = emb["file_id"].to_numpy()[np.argsort(-sims, axis=1)[:, :5]]

Reproducing

SEED = 42 throughout, the same seed as the EDA notebooks. The evaluation set is frozen at all
1,100 files in their original order.

The notebook runs top to bottom and skips the expensive work when its outputs are already
present - delete Part_A_Outputs/ or a shard file to force that piece to re-run. Costs on a
Colab T4: Part A's six embedding runs total 29 minutes (about 4.5 hours on CPU, which is why
they moved to Colab); Part B's four models total 8.6 GPU-hours. Steps 0, 1 and the Part B
scoring run locally in minutes.

Gemma-2 is a gated repo: accept the licence and provide a read token via huggingface-cli login or
HF_TOKEN. No token is hard-coded anywhere in the notebooks.

pandas  numpy  matplotlib  seaborn  scikit-learn  pyarrow
sentence-transformers  transformers>=4.45  accelerate  bitsandbytes  sentencepiece

Safety note

Every payload in the underlying corpus comes from a public, deliberately harmless security test
corpus (EICAR, WICAR, AMTSO, RANSIM) - the standard inert files used to verify that a scanner works.
Nothing here executes a real attack. Some files will still be flagged by antivirus software; that is
the intended behaviour of those test strings.

Note that part_b_answers.json contains model-quoted payload substrings as evidence, and
corpus_embeddings.parquet contains vectors only - no PDF text.

Related repositories

Citation

The real half of the EDA's Part 1 comparison is the CIC-Evasive-PDFMal2022 dataset from the Canadian
Institute for Cybersecurity, University of New Brunswick. Models evaluated: Nomic AI, Google,
Alibaba, Xiaomi and Microsoft, each under its own licence.