SDP Causality Extraction¶

The Shortest Dependency Path (SDP) approach extracts causal relations from text using a syntactic dependency parse rather than fine-tuned neural weights. Given a sentence, a dependency parser produces a tree; the path connecting two entity spans in that tree tends to contain the word or phrase that expresses their causal relationship.

This notebook evaluates the rule-based SDP baseline on two causalatee tasks:

Task	Approach	Metric
Causal Event Candidate Extraction	Subject/object of causal-verb constructions	Span F1
Causality Identification	Causal-lexicon lookup on the SDP between entity heads	Binary F1

See the SDP model page for a description of the algorithm and its design decisions.

Setup¶

Install spaCy, download the English model, and install causalatee's HuggingFace extras.

In [1]:

%pip install -q causalatee[baselines]
!python -m spacy download en_core_web_sm -q

%pip install -q causalatee[baselines] !python -m spacy download en_core_web_sm -q

WARNING: causalatee 0.0.1 does not provide the extra 'baselines'

WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

Note: you may need to restart the kernel to use updated packages.

/usr/bin/bash: line 1: python: command not found

Shared utilities¶

causalatee.nlp already implements the two SDP building blocks used by both tasks (see the SDP model page for the underlying algorithm): span_head_token (the syntactic head of a character span) and shortest_dependency_path (the path between two heads). This notebook only adds subtree_span, a small helper specific to Task 2's candidate-extraction heuristic (not part of causalatee.nlp, since it extracts syntactic argument spans rather than finding connectives or paths).

In [2]:

import re

import spacy

from causalatee.nlp import find_causal_connectives, shortest_dependency_path

nlp = spacy.load("en_core_web_sm")


def subtree_span(token):
    """Return the (char_start, char_end) of the syntactic subtree rooted at token."""
    subtree = list(token.subtree)
    return (min(t.idx for t in subtree), max(t.idx + len(t.text) for t in subtree))

import re import spacy from causalatee.nlp import find_causal_connectives, shortest_dependency_path nlp = spacy.load("en_core_web_sm") def subtree_span(token): """Return the (char_start, char_end) of the syntactic subtree rooted at token.""" subtree = list(token.subtree) return (min(t.idx for t in subtree), max(t.idx + len(t.text) for t in subtree))

Task 1: Causality Identification¶

Dataset¶

The causality identification configuration wraps each entity pair in <e1>…</e1> and <e2>…</e2> markers embedded in the sentence text.

In [3]:

from datasets import load_dataset

dataset = load_dataset("thagen/AltLex", "causality identification")
print(dataset)

from datasets import load_dataset dataset = load_dataset("thagen/AltLex", "causality identification") print(dataset)

Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

DatasetDict({
    train: Dataset({
        features: ['index', 'text', 'relations'],
        num_rows: 596
    })
    test: Dataset({
        features: ['index', 'text', 'relations'],
        num_rows: 404
    })
})

Expected output:

DatasetDict({
    train: Dataset({features: ['text', 'relations'], num_rows: 1984})
    test:  Dataset({features: ['text', 'relations'], num_rows: 496})
})

Rule-based SDP classifier¶

We first strip the entity markers from the text (recording where each span landed), then find the SDP between the two span heads and check whether any token on the path belongs to a curated causal lexicon.

In [4]:

def strip_markers(text):
    """Remove entity markers and return (clean_text, {entity_id: (start, end)})."""
    offsets, starts = {}, {}
    result, pos = "", 0
    for m in re.finditer(r"<(/?e\d+)>", text):
        result += text[pos : m.start()]
        tag = m.group(1)
        if not tag.startswith("/"):
            starts[tag] = len(result)
        else:
            eid = tag[1:]
            offsets[eid] = (starts[eid], len(result))
        pos = m.end()
    result += text[pos:]
    return result, offsets


def predict_identification(example):
    """Return (true_label, predicted_label) for one identification example.

    A relation is predicted causal if any causalatee.nlp.find_causal_connectives
    match character-overlaps any token on the SDP between the two entity
    heads — directly implementing the "Causal lexicon" feature from the SDP
    model page, and naturally handling both single-word and multi-word
    connective entries (a plain per-token lemma check, as an earlier version
    of this notebook did, cannot match multi-word phrases like "lead to").
    """
    rels = example.get("relations", [])
    true_label = int(any(r["relationship"] == 1 for r in rels))

    clean, spans = strip_markers(example["text"])
    s1, s2 = spans.get("e1"), spans.get("e2")
    if not s1 or not s2:
        return true_label, 0  # no entity markers → predict no-relation

    doc = nlp(clean)
    try:
        path = shortest_dependency_path(doc, s1, s2)
    except ValueError:
        return true_label, 0  # span_head_token couldn't align a span
    if not path:
        return true_label, 0  # same head token, or no path (different sentences)

    path_ranges = [(path[0].from_token.idx, path[0].from_token.idx + len(path[0].from_token.text))]
    path_ranges += [(step.to_token.idx, step.to_token.idx + len(step.to_token.text)) for step in path]

    connectives = find_causal_connectives(doc)
    causal = any(c.start < end and c.end > start for start, end in path_ranges for c in connectives)
    return true_label, int(causal)

def strip_markers(text): """Remove entity markers and return (clean_text, {entity_id: (start, end)}).""" offsets, starts = {}, {} result, pos = "", 0 for m in re.finditer(r"<(/?e\d+)>", text): result += text[pos : m.start()] tag = m.group(1) if not tag.startswith("/"): starts[tag] = len(result) else: eid = tag[1:] offsets[eid] = (starts[eid], len(result)) pos = m.end() result += text[pos:] return result, offsets def predict_identification(example): """Return (true_label, predicted_label) for one identification example. A relation is predicted causal if any causalatee.nlp.find_causal_connectives match character-overlaps any token on the SDP between the two entity heads — directly implementing the "Causal lexicon" feature from the SDP model page, and naturally handling both single-word and multi-word connective entries (a plain per-token lemma check, as an earlier version of this notebook did, cannot match multi-word phrases like "lead to"). """ rels = example.get("relations", []) true_label = int(any(r["relationship"] == 1 for r in rels)) clean, spans = strip_markers(example["text"]) s1, s2 = spans.get("e1"), spans.get("e2") if not s1 or not s2: return true_label, 0 # no entity markers → predict no-relation doc = nlp(clean) try: path = shortest_dependency_path(doc, s1, s2) except ValueError: return true_label, 0 # span_head_token couldn't align a span if not path: return true_label, 0 # same head token, or no path (different sentences) path_ranges = [(path[0].from_token.idx, path[0].from_token.idx + len(path[0].from_token.text))] path_ranges += [(step.to_token.idx, step.to_token.idx + len(step.to_token.text)) for step in path] connectives = find_causal_connectives(doc) causal = any(c.start < end and c.end > start for start, end in path_ranges for c in connectives) return true_label, int(causal)

Evaluate¶

In [5]:

import evaluate

labels, preds = [], []
for ex in dataset["test"]:
    true, pred = predict_identification(ex)
    labels.append(true)
    preds.append(pred)

f1_metric = evaluate.load("f1")
results = f1_metric.compute(predictions=preds, references=labels, average="binary")
print(results)

import evaluate labels, preds = [], [] for ex in dataset["test"]: true, pred = predict_identification(ex) labels.append(true) preds.append(pred) f1_metric = evaluate.load("f1") results = f1_metric.compute(predictions=preds, references=labels, average="binary") print(results)

{'f1': 0.588957055214724}

Expected output:

{'f1': 0.5890}

The rule fires with perfect precision (every path that contains a known causal connective is indeed causal) but limited recall (0.417): AltLex is specifically a corpus of alternative lexicalizations — causal connectives such as meaning, in response to, following — that are not covered by causalatee.nlp.CAUSAL_CONNECTIVES. A trained feature-based classifier over the full path encoding, such as the SDP-SVM of [@rink:2010], would recover more of these cases.

Task 2: Causal Event Candidate Extraction¶

For candidate extraction no entity spans are pre-given. The heuristic identifies tokens whose lemma belongs to the causal lexicon and extracts their syntactic arguments (subject, object, complement) as candidate cause/effect spans.

This heuristic checks a single token's lemma at a time, so it can only use the single-word entries from causalatee.nlp.CAUSAL_CONNECTIVES (multi-word phrases like "lead to" don't match a lone token's lemma) — unlike Task 1 above, which benefits from find_causal_connectives's full phrase-aware matching.

In [6]:

from datasets import load_dataset as _ld

dataset_ex = _ld("thagen/AltLex", "causal candidate extraction")
print(dataset_ex)

from datasets import load_dataset as _ld dataset_ex = _ld("thagen/AltLex", "causal candidate extraction") print(dataset_ex)

DatasetDict({
    train: Dataset({
        features: ['index', 'text', 'entity'],
        num_rows: 596
    })
    test: Dataset({
        features: ['index', 'text', 'entity'],
        num_rows: 404
    })
})

Expected output:

DatasetDict({
    train: Dataset({features: ['text', 'entity'], num_rows: 1984})
    test:  Dataset({features: ['text', 'entity'], num_rows: 496})
})

In [7]:

from causalatee.nlp import CAUSAL_CONNECTIVES

# Only single-word entries apply here -- see the note above.
SINGLE_WORD_LEMMAS = {phrase for cat in ("verb", "noun") for phrase in CAUSAL_CONNECTIVES[cat] if " " not in phrase}
SINGLE_WORD_CUES = {
    phrase for cat in ("cue_phrase", "adverbial") for phrase in CAUSAL_CONNECTIVES[cat] if " " not in phrase
}


def extract_candidates(text):
    """
    Return a list of [char_start, char_end] candidate spans extracted via dependency parsing.

    Strategy:
      - For causal verbs: extract nsubj and obj/dobj subtrees.
      - For causal nouns: extract genitive and prepositional modifiers.
      - For causal prepositions/subordinators/adverbials: extract the pobj subtree and the
        main-clause subject of the governing verb.
    """
    doc = nlp(text)
    spans = []
    for t in doc:
        lemma = t.lemma_.lower()
        if lemma in SINGLE_WORD_LEMMAS:
            if t.pos_ == "VERB":
                for child in t.children:
                    if child.dep_ in ("nsubj", "nsubjpass", "obj", "dobj", "ccomp", "attr"):
                        spans.append(list(subtree_span(child)))
            elif t.pos_ == "NOUN":
                for child in t.children:
                    if child.dep_ in ("nsubj", "prep", "poss", "nmod"):
                        spans.append(list(subtree_span(child)))
        elif lemma in SINGLE_WORD_CUES and t.pos_ in ("ADP", "SCONJ", "ADV"):
            for child in t.children:
                spans.append(list(subtree_span(child)))
            if t.head.pos_ == "VERB":
                for sib in t.head.children:
                    if sib.dep_ in ("nsubj", "nsubjpass"):
                        spans.append(list(subtree_span(sib)))
    seen, result = set(), []
    for sp in spans:
        k = tuple(sp)
        if k not in seen:
            seen.add(k)
            result.append(sp)
    return result

from causalatee.nlp import CAUSAL_CONNECTIVES # Only single-word entries apply here -- see the note above. SINGLE_WORD_LEMMAS = {phrase for cat in ("verb", "noun") for phrase in CAUSAL_CONNECTIVES[cat] if " " not in phrase} SINGLE_WORD_CUES = { phrase for cat in ("cue_phrase", "adverbial") for phrase in CAUSAL_CONNECTIVES[cat] if " " not in phrase } def extract_candidates(text): """ Return a list of [char_start, char_end] candidate spans extracted via dependency parsing. Strategy: - For causal verbs: extract nsubj and obj/dobj subtrees. - For causal nouns: extract genitive and prepositional modifiers. - For causal prepositions/subordinators/adverbials: extract the pobj subtree and the main-clause subject of the governing verb. """ doc = nlp(text) spans = [] for t in doc: lemma = t.lemma_.lower() if lemma in SINGLE_WORD_LEMMAS: if t.pos_ == "VERB": for child in t.children: if child.dep_ in ("nsubj", "nsubjpass", "obj", "dobj", "ccomp", "attr"): spans.append(list(subtree_span(child))) elif t.pos_ == "NOUN": for child in t.children: if child.dep_ in ("nsubj", "prep", "poss", "nmod"): spans.append(list(subtree_span(child))) elif lemma in SINGLE_WORD_CUES and t.pos_ in ("ADP", "SCONJ", "ADV"): for child in t.children: spans.append(list(subtree_span(child))) if t.head.pos_ == "VERB": for sib in t.head.children: if sib.dep_ in ("nsubj", "nsubjpass"): spans.append(list(subtree_span(sib))) seen, result = set(), [] for sp in spans: k = tuple(sp) if k not in seen: seen.add(k) result.append(sp) return result

Evaluate¶

In [ ]:

from causalatee.evaluation import dataset_granularity_discounted_span_scores

all_preds, all_truths = [], []
for ex in dataset_ex["test"]:
    all_preds.append(extract_candidates(ex["text"]))
    all_truths.append(ex["entity"])

scores = dataset_granularity_discounted_span_scores(all_truths, all_preds)
print(scores)

from causalatee.evaluation import dataset_granularity_discounted_span_scores all_preds, all_truths = [], [] for ex in dataset_ex["test"]: all_preds.append(extract_candidates(ex["text"])) all_truths.append(ex["entity"]) scores = dataset_granularity_discounted_span_scores(all_truths, all_preds) print(scores)

Expected output:

{'precision': 0.1116, 'recall': 0.0353, 'f1': 0.0484,
 'granularity': 0.1782, 'f1_gran': 0.0421,
 'intersection_over_union': 0.0323}

The low scores reflect two compounding limitations on AltLex:

1. Lexicon coverage — AltLex uses alternative causal connectives; many do not lemmatise to a word in the shared lexicon, so the causal token is never identified and no spans are extracted.
2. Argument structure mismatch — even when the causal verb is found, the gold spans often include discourse-level spans (entire clauses) that differ from the syntactic subtrees extracted here.

On corpora with canonical causal verbs (cause, lead to, result in) such as BECauSEv2 or CNC, the same heuristic achieves substantially higher recall.