license: mit
task_categories:

• text-classification
• feature-extraction
  language:
• en
  tags:
• cybersecurity
• pdf
• malware
• prompt-injection
• eda
• clustering
  size_categories:
• 1K<n<10K
  configs:
• config_name: default
  data_files:
  • split: train
    path: Datasets/synthetic_corpus_part2_clustered.parquet

Injected PDFs - EDA and Evaluation Corpus

This repository holds the exploratory data analysis for a project on detecting harmless-but-real
attack payloads injected into PDF files, together with the dataset that analysis produced.

The project has two halves, both in the notebook Final_project_V7_EDA.ipynb:

Part 2 exports Datasets/synthetic_corpus_part2_clustered.parquet, the dataset the model evaluation
stage runs on.

The data

Two tables are read directly from
Cyber-security-final-project/Generated_Injected_PDFs_HARMLESS:

• combined (11,126 x 34) - one row per PDF, described by 32 structural features. It holds two
  different populations:
  • source = "Real" - 10,025 real PDFs from CIC-Evasive-PDFMal2022;
  • source = "Synthetic" - 1,100 PDFs we generated ourselves by injecting harmless payloads.
• manifest (1,100 x 11) - the ground truth for the synthetic half: which payload went into
  which file, where, and how it was obfuscated.

Target: Class (Malicious / Benign). Grouping variable: source (Real / Synthetic).

The payloads come from standard, deliberately harmless malware test corpora - EICAR (221 files),
WICAR (255), AMTSO (215) and RANSIM (209). They do no damage, but they are the exact strings and
structures the security industry uses to check that a scanner works.

Part 1 - How similar is the synthetic malware to the real thing?

Step 1 - Cleaning the feature table

The raw table looks almost perfect and is not. Five repairs were needed before any analysis:

1. Column names. A typo (Fine name), spaces inside names, and three competing naming
   conventions. All renamed to snake_case.
2. pdf_size was in different units in each half - kilobytes for the real files, bytes for the
   synthetic ones, a factor of roughly 3,000. Rescaled to bytes.
3. 147 cells contained the extraction tool's crash message, written into the CSV as if it were
   data (pdfid.py, bytes[endHeader], (most). Given their own -2 sentinel.
4. 638 cells used a 29(2) notation - a count and a second measurement crammed into one text
   cell. Parsed out, so every measurement column became a proper integer column.
5. metadata_size was not a byte size at all on the synthetic side. Voided to <NA> rather
   than repaired.

The important finding of this step is about missing values. The file declares only 74 NaN out
of ~378,000 cells. In reality 15,016 cells carry a sentinel instead of a measurement, across
every one of the 30 feature columns - and they are not spread evenly:

The extraction tool breaks specifically on malformed malicious PDFs - and malicious PDFs are often
deliberately malformed to defeat parsers. This is informative missingness: the fact that a value is
missing is itself a clue about the class. Tempting to exploit, but a model that learns "extraction
failed, therefore malicious" has learned about our tool, not about malware. The sentinels were
converted to NaN and kept out of the statistics.

Outliers were detected but not removed. The IQR rule flags 4-19% of rows, but in the real half
those rows are the most class-informative in the table, and they point both ways: files with an
extreme js count are 91.3% malicious, files with an extreme obj count only 5.1%, against
a 55.4% base rate. Deleting them would have destroyed the signal, and destroyed it asymmetrically.

Step 2 - Comparing the two corpora

Graph 1 - How is the data composed?

The dataset is dominated by the real data: 10,025 real files (90%) against 1,100 synthetic ones
(10%), so any statistic on the pooled table is really a statistic about the real half. The class
balance also differs sharply - the real half is close to balanced at 55.4/44.6, the synthetic half
is deliberately skewed at 81.8/18.2. Every graph from here on separates the two.

Graph 2 - Do the two sources measure the same thing?

This is the check on the unit fix. Before rescaling the two distributions were two separate humps
roughly 2,900x apart - not because the files differ, but because one half was recorded in
kilobytes and the other in bytes. After the fix they overlap across most of their range.

One real difference survives, and it is a believable one: the real files have a median of 38 KB
against 107 KB for the synthetic files, about 2.8x. We injected payloads into full-length
documents, whereas the CIC corpus contains a large population of very small files. Note the log
scale - size data is almost always log-normal, which is also why the median is the honest summary
here and not the mean.

Graph 3 - Which features separate Malicious from Benign?

In the real data, malicious PDFs are structurally much simpler than benign ones - 47 objects vs
10, 19 streams vs 2, 2 pages vs 1. That is the opposite of the intuitive guess: a weaponised PDF is
a nearly empty shell whose only job is to carry the payload, while a genuine document carries fonts,
images and text.

In the synthetic data the same four boxes are effectively identical - 32.5 vs 35 objects, 12 vs
12 streams, 2 vs 2 pages. Our injection did not change the host document's structure, because we
started from ordinary complete PDFs and added a small payload to them.

Graph 4 - Correlations between features and the target

The real panel contains genuine signal: open_action +0.52, stream -0.39, startxref
-0.38, obj -0.26. The signs tell a coherent story - an instruction that runs something the
moment the file opens is associated with malice, while everything measuring document richness is
associated with benignity.

The synthetic panel is almost blank in the target row. Its strongest correlation is
keyword_embedded_file at 0.167 - weaker than the sixth strongest feature on the real side.

Graph 5 - Does the synthetic data behave like the real data?

In the real panel the two classes form two visibly different clouds - benign files spread up and to
the right (many objects, many streams), malicious files bunch into the bottom-left corner. In the
synthetic panel the two clouds sit directly on top of each other.

The flag table makes the gap explicit: js is present in 71.8% of real malicious files but only
19.8% of synthetic malicious ones, and open_action in 51.4% vs 17.9%. Conversely
launch and keyword_embedded_file are more common in our files (8.9% and 26.4%) than in real
malware (2.3% and 8.1%) - because those are the payload types our generator favoured.

Graph 6 - Every feature ranked by its correlation with the target

This is the compact statement of Part 1's main result. The blue bars form a clear staircase; the
orange bars are nearly flat. More importantly, the two rankings do not merely weaken - they
disagree. open_action is the best real predictor at +0.52 and -0.07 on the synthetic
side: the sign is reversed. A rule taken from one corpus points the wrong way on the other.

Graph 7 - UMAP: the final test of similarity

Given all features at once and no labels at all, UMAP still sorts the two corpora apart.

• The relationship is containment, not overlap. The real corpus spreads across the whole map in
  dozens of thin filaments; the synthetic corpus sits in one compact band covering roughly a fifth
  of the occupied area.
• The synthetic corpus is 9.9% of the data, so if the two were interchangeable a synthetic file's
  neighbours would be about 9.9% synthetic. Instead 77.9% are - an eightfold enrichment - and
  39.1% of synthetic files have no real file at all among their 20 nearest neighbours.

Part 1 conclusion - two corpora, two different notions of "malicious"

The reason is not statistical but procedural. Real malware is authored - built minimally from
scratch to carry a payload. Our files were injected - ordinary complete documents with a payload
added, so the host structure barely moved. The two corpora encode two different threat models.

Why that makes the synthetic corpus the right thing to evaluate on

That difference is usually described as a limitation. For an evaluation it is the entire point.

1. The payloads are largely invisible to the CIC feature vocabulary.

37.3% of our injected files raise no CIC flag at all - more than double the evasion rate of real
malware. A flag-counting scanner of the kind these features describe would pass them straight
through.

2. They are nonetheless genuinely dangerous. Every payload comes from a standard malware test
corpus - harmless by design, but functionally an attack in every respect except the damage.

3. Real scanners do detect them. While transferring the corpus from Google Drive, every
object_action_injection file (80 of them) was blocked by Google's malware scanner - independent
confirmation from a production security system that these files are not inert props.

So the corpus is provably malicious to a production scanner while remaining largely invisible
to the structural feature set the academic benchmark is built on. That gap is what makes it useful.

The variety is graded, which is what makes it a measuring instrument

900 injected files built from 337 distinct attack configurations, varying along five axes at
once: 12 attack families, 9 source frameworks, 550 payload variants, 3 insertion
points, 6 obfuscation strategies - and 1,045 distinct host documents for 1,100 files, so a
model cannot succeed by memorising the container.

Counting CIC flags per family splits the corpus cleanly in two:

The loud families check that a model has basic competence; the silent families are the real test.
Because both are in one corpus, a model's score profile across families is informative in a way a
single average is not.

What this corpus can and cannot establish. It can measure evasion resistance and compare models
fairly, since every model faces the identical 1,100 files. It cannot stand in for a general
accuracy claim on real-world PDFs - the host documents cover about a fifth of the UMAP map.

Part 2 - Preparing the corpus for evaluation

Part 1 ended on a negative result, and that result has a direct consequence: the CIC feature table
is the wrong input for Part 2. If the features cannot see the payload, no analysis of those
features can tell us whether the corpus is varied enough to test a model on. So Part 2 goes back to
the 1,100 actual PDF files and builds its own table from them.

Step 1 - Designing the extraction

A PDF is a binary container of numbered objects, most of them compressed. Three choices had to be
made, and each one can quietly delete the payload - so each was measured rather than guessed, on
a random sample of 60 injected files.

Probe 1 - where is the payload? after_first_endobj puts it a median of 0.2% into the file;
before_trailer and before_final_eof put it at a median of 99.8%. 40 of 60 payloads are
past the halfway mark. Keeping only the start of each file would have silently deleted about two
thirds of the ground truth - and the dataset would still have looked perfectly healthy.

Probes 2 and 3 - what about the compressed streams? Throwing the stream bodies away destroys
22 of 60 payloads, because many injections sit inside streams. Decompressing first keeps
60 of 60. The payloads were never missing; they were compressed.

This is Part 1's lesson in a different form: there, the feature vocabulary had no column for the
payload; here, a naive text extractor cannot see it because it sits behind a decompression step. In
both cases the data looks clean and the information is simply gone.

Step 2-3 - The extractor, and its verification

The design that follows: keep a head and a tail window (never just the head), decompress
streams and keep the text, replace a body with a placeholder only when it is genuinely binary. The
result is a skeleton - the structure and readable content of the PDF with the binary bulk
removed, roughly what a security analyst would look at.

The masking matters: skeleton_masked removes the give-away brand name without removing the attack,
which is what will let us tell "the model recognised a famous test string" apart from "the model
understood the PDF".

Step 4 - Descriptive statistics

The entire class of defect that dominated Part 1 is gone: zero declared missing values and zero
negative values across all 21 numeric features. Part 1 needed five subsections to repair 15,016
sentinel cells written by the CIC tool; here there is no tool between us and the bytes. The data
quality problems in Part 1 were a property of that feature table, not of PDF data in general.

The skew is still extreme (12 of 21 features above 10), so medians remain the honest summary. But
only one feature has a median of zero, against 18 of 30 in Part 1 - the first numerical sign that
we are measuring something different.

Three near-duplicate column pairs were found and one member of each dropped, leaving 18 features:
n_obj ~ n_endobj (r = 1.000, a well-formed PDF closes every object it opens), n_startxref ~
n_eof (0.997, both count revisions), and n_images ~ binary_streams_dropped (0.991, an artefact
of our own extraction). Left alone, all three would have silently given object count, revision count
and image count double weight in every distance calculation downstream.

Step 5 - Is there enough variety?

Graph 1 - Coverage

All 36 cells of the 12 attack types x 3 insertion positions grid are populated, from 15 files to
33. No attack type was only ever tried in one position, so a model cannot score well by being good
at spotting objects appended at the end of a file.

Graph 2 - Composition

Each of the 12 types draws on exactly two frameworks. That rules out the worst problem - where
"detecting ssrf" would really mean "detecting the one framework it came from" - but two is a thin
margin, and this is the corpus's one real limitation.

Graph 3 - Spread within each type

This is where the corpus earns its keep:

• distinct_carriers equals the file count for every single type - no carrier document is ever
  reused across all 900 injected files.
• Carrier size varies by a factor of 21x to 281x within a type.
• pos_iqr is 1.0 for every type: the interquartile range of the payload's relative position
  covers the whole file, in every category.

But the same fact carries a warning. If every file sits in its own unique carrier, the
differences between files are mostly differences between carrier documents, and the payload is a
small addition on top.

Step 6 - Embedding and mapping the corpus

The same vectoriser, run at two scopes, measured by neighbourhood purity - out of each file's 20
nearest neighbours, what share share its injection type? Random baseline: 8.9%.

So the signal is real, and it is local. The payload is a genuine and very distinctive signal
occupying about one twentieth of the text, and averaging over the whole document drowns it.

Two consequences:

• For Part 1's argument, this is confirmation. The CIC features were not unlucky - any
  whole-document summary, hand-built counters or learned embeddings alike, will miss a small local
  change. They were the wrong kind of instrument.
• For the evaluation, this defines the task. Testing a model here is a needle-in-a-haystack
  detection problem, not a document classification problem: find a few hundred suspicious
  characters inside a hundred thousand characters of ordinary PDF.

Step 7 - How many natural groups does the corpus have?

Three algorithms, because each defines "a cluster" differently and they fail differently. All run on
the 384-dimensional payload-window embeddings, not on the 2-D UMAP coordinates - UMAP
deliberately distorts large distances, so clustering its output would describe the picture rather
than the data.

Chosen: KMeans with k = 6, taken from the inertia elbow rather than the silhouette peak. In 384
dimensions the silhouette rewards any tight little blob, so a split that isolates a few outliers and
leaves everything else in one lump can win it.

• Agglomerative (average linkage, cosine) forms one long chain: wherever the tree is cut, one
  cluster holds nearly the whole corpus (65.1% at its own elbow) plus a handful of slivers. Those
  small clusters are outliers being peeled off, not groups.
• DBSCAN never clusters most of the data. The embeddings sit at roughly uniform cosine distance,
  so the k-distance curve has no real knee; at the best eps most files come back labelled noise,
  and a slightly larger eps collapses everything into one cluster.

KMeans at k = 6 is the only setting that gives a balanced split with 0% of files unassigned.

Step 8 - The clustering, and what it actually found

The clustering is well behaved. It is simply not clustering on the payload.

Six clusters of 116-275 files, largest holding 25.0%, nothing unassigned. The profile table shows
what they are made of:

Every row rises steadily left to right. The clusters are sorting PDFs by document size and
structural complexity, not by attack. entropy_file (7.72-7.81) and printable_frac (0.46-0.50)
barely move at all - the byte-level character of the files is the same, only their scale differs.

Agreement with our design is almost zero: ARI +0.025, NMI 0.046. There are two small, explicable
departures: cluster 3 is 98.1% injected and holds three times its share of llm_prompt_injection -
the one attack type whose payload is ordinary written instructions rather than PDF syntax, so the one
a sentence embedding is best placed to notice - while clusters 1 and 4 are where the clean files
gather.

Part 2 conclusion

1. The data quality problems of Part 1 came from the CIC feature table, not from PDF data.
   Reading the files ourselves gave zero missing values and zero sentinels.
2. The corpus is genuinely varied where it matters: all 36 coverage cells filled, no carrier
   document ever reused, carrier sizes spanning 21x-281x within every type, payload positions
   spanning the whole file.
3. The one real limitation is framework breadth - two frameworks per attack type. The corpus
   varies the shape of the attack far more than the style of the payload.
4. The payload is a small, local change taking up about 4.5% of the text: 12.1% neighbourhood
   purity over the whole document against 42.0% in the payload window, on an 8.9% baseline.
5. The corpus varies mostly by carrier, not by injection, and this is a property we want:
   • a model cannot score well by picking up a generator artefact, because the strongest signal
     available is document size, which is unrelated to the label by construction;
   • any model that does recover injection type is demonstrably doing something the
     representation cannot do unaided - the unsupervised floor is measured at ARI 0.025, and
     that is the number later results must be read against;
   • cluster is therefore a control variable, not a finding. If a model's accuracy tracks
     cluster, we will know it is tracking document size rather than doing security reasoning.

Repository contents

Final_project_V7_EDA.ipynb                       Parts 1 and 2, end to end
Datasets/
  synthetic_corpus_part2_clustered.parquet       the Part 2 dataset (1,100 x 44)
  synthetic_corpus_part2_clustered.csv           same, for tools that cannot read parquet
images/                                          the 15 figures reproduced above

Dataset columns

The two marker columns are stored as |-joined strings so that both file formats round-trip.

Loading

from datasets import load_dataset
ds = load_dataset("<your-username>/<your-repo>", split="train")

# or straight to pandas
import pandas as pd
df = pd.read_parquet("Datasets/synthetic_corpus_part2_clustered.parquet")

Reproducing

The notebook runs top to bottom. Part 2's extraction cell needs the 1,100 raw PDFs from the source
dataset repo and takes roughly 12 minutes. SEED = 42 throughout, so the quoted numbers are
reproducible.

pandas  numpy  matplotlib  seaborn  scikit-learn  scipy
umap-learn  sentence-transformers  pymupdf  pyarrow

Safety note

Every payload in this corpus is drawn from a public, deliberately harmless security test corpus
(EICAR, WICAR, AMTSO, RANSIM). These are the standard inert files used to verify that a scanner is
working. Nothing here executes a real attack or causes damage. The corpus exists to evaluate
detection, and some files will still be flagged by antivirus software - that is the intended
behaviour of those test strings.

Citation

The real half of the Part 1 comparison is the CIC-Evasive-PDFMal2022 dataset from the Canadian
Institute for Cybersecurity, University of New Brunswick.