Information Extraction¶

`words`	Extract an ordered sequence of words from a document processed by spaCy, optionally filtering words by part-of-speech tag and frequency.
`ngrams`	Extract an ordered sequence of n-grams (`n` consecutive words) from a spacy-parsed doc, optionally filtering n-grams by the types and parts-of-speech of the constituent words.
`entities`	Extract an ordered sequence of named entities (PERSON, ORG, LOC, etc.) from a `Doc`, optionally filtering by entity types and frequencies.
`noun_chunks`	Extract an ordered sequence of noun chunks from a spacy-parsed doc, optionally filtering by frequency and dropping leading determiners.
`pos_regex_matches`	Extract sequences of consecutive tokens from a spacy-parsed doc whose part-of-speech tags match the specified regex pattern.
`matches`	Extract `Span` s from a `Doc` matching one or more patterns of per-token attr:value pairs, with optional quantity qualifiers.
`subject_verb_object_triples`	Extract an ordered sequence of subject-verb-object (SVO) triples from a spacy-parsed doc.
`acronyms_and_definitions`	Extract a collection of acronyms and their most likely definitions, if available, from a spacy-parsed doc.
`semistructured_statements`	Extract “semi-structured statements” from a spacy-parsed doc, each as a (entity, cue, fragment) triple.
`direct_quotations`	Baseline, not-great attempt at direction quotation extraction (no indirect or mixed quotations) using rules and patterns.

textacy.extract: Functions to extract various elements of interest from documents already parsed by spaCy, such as n-grams, named entities, subject-verb-object triples, and acronyms.

textacy.extract.words(doc: Union[spacy.tokens.doc.Doc, spacy.tokens.span.Span], *, filter_stops: bool = True, filter_punct: bool = True, filter_nums: bool = False, include_pos: Optional[Union[str, Set[str]]] = None, exclude_pos: Optional[Union[str, Set[str]]] = None, min_freq: int = 1) → Iterable[spacy.tokens.token.Token][source]¶

Extract an ordered sequence of words from a document processed by spaCy, optionally filtering words by part-of-speech tag and frequency.

Parameters

doc –
filter_stops – If True, remove stop words from word list.
filter_punct – If True, remove punctuation from word list.
filter_nums – If True, remove number-like words (e.g. 10, “ten”) from word list.
include_pos – Remove words whose part-of-speech tag IS NOT in the specified tags.
exclude_pos – Remove words whose part-of-speech tag IS in the specified tags.
min_freq – Remove words that occur in doc fewer than min_freq times.

Yields

Next token from doc passing specified filters in order of appearance in the document.

Raises

TypeError – if include_pos or exclude_pos is not a str, a set of str, or a falsy value

Note

Filtering by part-of-speech tag uses the universal POS tag set; for details, check spaCy’s docs: https://spacy.io/api/annotation#pos-tagging

textacy.extract.ngrams(doc: Union[spacy.tokens.doc.Doc, spacy.tokens.span.Span], n: int, *, filter_stops: bool = True, filter_punct: bool = True, filter_nums: bool = False, include_pos: Optional[Union[str, Set[str]]] = None, exclude_pos: Optional[Union[str, Set[str]]] = None, min_freq: int = 1) → Iterable[spacy.tokens.span.Span][source]¶

Extract an ordered sequence of n-grams (n consecutive words) from a spacy-parsed doc, optionally filtering n-grams by the types and parts-of-speech of the constituent words.

Parameters

doc –
n – Number of tokens per n-gram; 2 => bigrams, 3 => trigrams, etc.
filter_stops – If True, remove ngrams that start or end with a stop word
filter_punct – If True, remove ngrams that contain any punctuation-only tokens
filter_nums – If True, remove ngrams that contain any numbers or number-like tokens (e.g. 10, ‘ten’)
include_pos – Remove ngrams if any constituent tokens’ part-of-speech tags ARE NOT included in this param
exclude_pos – Remove ngrams if any constituent tokens’ part-of-speech tags ARE included in this param
min_freq – Remove ngrams that occur in doc fewer than min_freq times

Yields

Next ngram from doc passing all specified filters, in order of appearance in the document

Raises

ValueError – if n < 1
TypeError – if include_pos or exclude_pos is not a str, a set of str, or a falsy value

Note

Filtering by part-of-speech tag uses the universal POS tag set; for details, check spaCy’s docs: https://spacy.io/api/annotation#pos-tagging

textacy.extract.entities(doc: spacy.tokens.doc.Doc, *, include_types: Optional[Union[str, Set[str]]] = None, exclude_types: Optional[Union[str, Set[str]]] = None, drop_determiners: bool = True, min_freq: int = 1) → Iterable[spacy.tokens.span.Span][source]¶

Extract an ordered sequence of named entities (PERSON, ORG, LOC, etc.) from a Doc, optionally filtering by entity types and frequencies.

Parameters

doc –
include_types – Remove entities whose type IS NOT in this param; if “NUMERIC”, all numeric entity types (“DATE”, “MONEY”, “ORDINAL”, etc.) are included
exclude_types – Remove entities whose type IS in this param; if “NUMERIC”, all numeric entity types (“DATE”, “MONEY”, “ORDINAL”, etc.) are excluded
drop_determiners –
Remove leading determiners (e.g. “the”) from entities (e.g. “the United States” => “United States”).

Note

Entities from which a leading determiner has been removed are, effectively, new entities, and not saved to the Doc from which they came. This is irritating but unavoidable, since this function is not meant to have side-effects on document state. If you’re only using the text of the returned spans, this is no big deal, but watch out if you’re counting on determiner-less entities associated with the doc downstream.
min_freq – Remove entities that occur in doc fewer than min_freq times

Yields

Next entity from doc passing all specified filters in order of appearance in the document

Raises

TypeError – if include_types or exclude_types is not a str, a set of str, or a falsy value

textacy.extract.noun_chunks(doc: spacy.tokens.doc.Doc, *, drop_determiners: bool = True, min_freq: int = 1) → Iterable[spacy.tokens.span.Span][source]¶

Extract an ordered sequence of noun chunks from a spacy-parsed doc, optionally filtering by frequency and dropping leading determiners.

Parameters

doc –
drop_determiners – Remove leading determiners (e.g. “the”) from phrases (e.g. “the quick brown fox” => “quick brown fox”)
min_freq – Remove chunks that occur in doc fewer than min_freq times

Yields

Next noun chunk from doc in order of appearance in the document

textacy.extract.pos_regex_matches(doc: Union[spacy.tokens.doc.Doc, spacy.tokens.span.Span], pattern: str) → Iterable[spacy.tokens.span.Span][source]¶

Extract sequences of consecutive tokens from a spacy-parsed doc whose part-of-speech tags match the specified regex pattern.

Parameters

doc –
pattern –
Pattern of consecutive POS tags whose corresponding words are to be extracted, inspired by the regex patterns used in NLTK’s nltk.chunk.regexp. Tags are uppercase, from the universal tag set; delimited by < and >, which are basically converted to parentheses with spaces as needed to correctly extract matching word sequences; white space in the input doesn’t matter.

Examples (see constants.POS_REGEX_PATTERNS):
- noun phrase: r’<DET>? (<NOUN>+ <ADP|CONJ>)* <NOUN>+’
- compound nouns: r’<NOUN>+’
- verb phrase: r’<VERB>?<ADV>*<VERB>+’
- prepositional phrase: r’<PREP> <DET>? (<NOUN>+<ADP>)* <NOUN>+’

Yields

Next span of consecutive tokens from doc whose parts-of-speech match pattern, in order of appearance

Warning

DEPRECATED! For similar but more powerful and performant functionality, use textacy.extract.matches() instead.

textacy.extract.matches(doc: spacy.tokens.doc.Doc, patterns: Union[str, List[str], List[Dict[str, str]], List[List[Dict[str, str]]]], *, on_match: Callable = None) → Iterable[spacy.tokens.span.Span][source]¶

Extract Span s from a Doc matching one or more patterns of per-token attr:value pairs, with optional quantity qualifiers.

Parameters

doc –
patterns –
One or multiple patterns to match against doc using a spacy.matcher.Matcher.

If List[dict] or List[List[dict]], each pattern is specified as attr: value pairs per token, with optional quantity qualifiers:
- [{"POS": "NOUN"}] matches singular or plural nouns, like “friend” or “enemies”
- [{"POS": "PREP"}, {"POS": "DET", "OP": "?"}, {"POS": "ADJ", "OP": "?"}, {"POS": "NOUN", "OP": "+"}] matches prepositional phrases, like “in the future” or “from the distant past”
- [{"IS_DIGIT": True}, {"TAG": "NNS"}] matches numbered plural nouns, like “60 seconds” or “2 beers”
- [{"POS": "PROPN", "OP": "+"}, {}] matches proper nouns and whatever word follows them, like “Burton DeWilde yaaasss”
If str or List[str], each pattern is specified as one or more per-token patterns separated by whitespace where attribute, value, and optional quantity qualifiers are delimited by colons. Note that boolean and integer values have special syntax — “bool(val)” and “int(val)”, respectively — and that wildcard tokens still need a colon between the (empty) attribute and value strings.
- "POS:NOUN" matches singular or plural nouns
- "POS:PREP POS:DET:? POS:ADJ:? POS:NOUN:+" matches prepositional phrases
- "IS_DIGIT:bool(True) TAG:NNS" matches numbered plural nouns
- "POS:PROPN:+ :" matches proper nouns and whatever word follows them
Also note that these pattern strings don’t support spaCy v2.1’s “extended” pattern syntax; if you need such complex patterns, it’s probably better to use a List[dict] or List[List[dict]], anyway.
on_match – Callback function to act on matches. Takes the arguments matcher, doc, i and matches.

Yields

Next matching Span in doc, in order of appearance

Raises

TypeError –
ValueError –

Keyterm Extraction¶

`textrank.textrank`	Extract key terms from a document using the TextRank algorithm, or a variation thereof.
`yake.yake`	Extract key terms from a document using the YAKE algorithm.
`scake.scake`	Extract key terms from a document using the sCAKE algorithm.
`sgrank.sgrank`	Extract key terms from a document using the SGRank algorithm.

textacy.ke.textrank.textrank(doc: spacy.tokens.doc.Doc, *, normalize: Optional[Union[str, Callable[[spacy.tokens.token.Token], str]]] = 'lemma', include_pos: Optional[Union[str, Collection[str]]] = 'NOUN', 'PROPN', 'ADJ', window_size: int = 2, edge_weighting: str = 'binary', position_bias: bool = False, topn: Union[int, float] = 10) → List[Tuple[str, float]][source]¶

Extract key terms from a document using the TextRank algorithm, or a variation thereof. For example:

TextRank: window_size=2, edge_weighting="binary", position_bias=False
SingleRank: window_size=10, edge_weighting="count", position_bias=False
PositionRank: window_size=10, edge_weighting="count", position_bias=True

Parameters

doc – spaCy Doc from which to extract keyterms.
normalize – If “lemma”, lemmatize terms; if “lower”, lowercase terms; if None, use the form of terms as they appeared in doc; if a callable, must accept a Token and return a str, e.g. textacy.spacier.utils.get_normalized_text().
include_pos – One or more POS tags with which to filter for good candidate keyterms. If None, include tokens of all POS tags (which also allows keyterm extraction from docs without POS-tagging.)
window_size – Size of sliding window in which term co-occurrences are determined.
edge_weighting ({"count", "binary"}) – : If “count”, the nodes for all co-occurring terms are connected by edges with weight equal to the number of times they co-occurred within a sliding window; if “binary”, all such edges have weight = 1.
position_bias – If True, bias the PageRank algorithm for weighting nodes in the word graph, such that words appearing earlier and more frequently in doc tend to get larger weights.
topn – Number of top-ranked terms to return as key terms. If an integer, represents the absolute number; if a float, value must be in the interval (0.0, 1.0], which is converted to an int by int(round(len(set(candidates)) * topn)).

Returns

Sorted list of top topn key terms and their corresponding TextRank ranking scores.

References

Mihalcea, R., & Tarau, P. (2004, July). TextRank: Bringing order into texts. Association for Computational Linguistics.
Wan, Xiaojun and Jianguo Xiao. 2008. Single document keyphrase extraction using neighborhood knowledge. In Proceedings of the 23rd AAAI Conference on Artificial Intelligence, pages 855–860.
Florescu, C. and Cornelia, C. (2017). PositionRank: An Unsupervised Approach to Keyphrase Extraction from Scholarly Documents. In proceedings of ACL*, pages 1105-1115.

textacy.ke.yake.yake(doc: spacy.tokens.doc.Doc, *, normalize: Optional[str] = 'lemma', ngrams: Union[int, Collection[int]] = 1, 2, 3, include_pos: Optional[Union[str, Collection[str]]] = 'NOUN', 'PROPN', 'ADJ', window_size: int = 2, topn: Union[int, float] = 10) → List[Tuple[str, float]][source]¶

Extract key terms from a document using the YAKE algorithm.

Parameters

doc – spaCy Doc from which to extract keyterms. Must be sentence-segmented; optionally POS-tagged.
normalize –
If “lemma”, lemmatize terms; if “lower”, lowercase terms; if None, use the form of terms as they appeared in doc.

Note

Unlike the other keyterm extraction functions, this one doesn’t accept a callable for normalize.
ngrams – n of which n-grams to consider as keyterm candidates. For example, (1, 2, 3)` includes all unigrams, bigrams, and trigrams, while 2 includes bigrams only.
include_pos – One or more POS tags with which to filter for good candidate keyterms. If None, include tokens of all POS tags (which also allows keyterm extraction from docs without POS-tagging.)
window_size – Number of words to the right and left of a given word to use as context when computing the “relatedness to context” component of its score. Note that the resulting sliding window’s full width is 1 + (2 * window_size).
topn – Number of top-ranked terms to return as key terms. If an integer, represents the absolute number; if a float, value must be in the interval (0.0, 1.0], which is converted to an int by int(round(len(candidates) * topn))

Returns

Sorted list of top topn key terms and their corresponding YAKE scores.

References

Campos, Mangaravite, Pasquali, Jorge, Nunes, and Jatowt. (2018). A Text Feature Based Automatic Keyword Extraction Method for Single Documents. Advances in Information Retrieval. ECIR 2018. Lecture Notes in Computer Science, vol 10772, pp. 684-691.

textacy.ke.scake.scake(doc: spacy.tokens.doc.Doc, *, normalize: Optional[Union[str, Callable[[spacy.tokens.token.Token], str]]] = 'lemma', include_pos: Optional[Union[str, Collection[str]]] = 'NOUN', 'PROPN', 'ADJ', topn: Union[int, float] = 10) → List[Tuple[str, float]][source]¶

Extract key terms from a document using the sCAKE algorithm.

Parameters

doc – spaCy Doc from which to extract keyterms. Must be sentence-segmented; optionally POS-tagged.
normalize – If “lemma”, lemmatize terms; if “lower”, lowercase terms; if None, use the form of terms as they appeared in doc; if a callable, must accept a Token and return a str, e.g. textacy.spacier.utils.get_normalized_text().
include_pos – One or more POS tags with which to filter for good candidate keyterms. If None, include tokens of all POS tags (which also allows keyterm extraction from docs without POS-tagging.)
topn – Number of top-ranked terms to return as key terms. If an integer, represents the absolute number; if a float, value must be in the interval (0.0, 1.0], which is converted to an int by int(round(len(candidates) * topn))

Returns

Sorted list of top topn key terms and their corresponding scores.

References

Duari, Swagata & Bhatnagar, Vasudha. (2018). sCAKE: Semantic Connectivity Aware Keyword Extraction. Information Sciences. 477. https://arxiv.org/abs/1811.10831v1

class textacy.ke.sgrank.Candidate(text, idx, length, count)¶

count¶: Alias for field number 3

idx¶: Alias for field number 1

length¶: Alias for field number 2

text¶: Alias for field number 0

textacy.ke.sgrank.sgrank(doc: spacy.tokens.doc.Doc, *, normalize: Optional[Union[str, Callable[[spacy.tokens.span.Span], str]]] = 'lemma', ngrams: Union[int, Collection[int]] = 1, 2, 3, 4, 5, 6, include_pos: Optional[Union[str, Collection[str]]] = 'NOUN', 'PROPN', 'ADJ', window_size: int = 1500, topn: Union[int, float] = 10, idf: Dict[str, float] = None) → List[Tuple[str, float]][source]¶

Extract key terms from a document using the SGRank algorithm.

Parameters

doc – spaCy Doc from which to extract keyterms.
normalize – If “lemma”, lemmatize terms; if “lower”, lowercase terms; if None, use the form of terms as they appeared in doc; if a callable, must accept a Span and return a str, e.g. textacy.spacier.utils.get_normalized_text()
ngrams – n of which n-grams to include. For example, (1, 2, 3, 4, 5, 6) (default) includes all ngrams from 1 to 6; 2 if only bigrams are wanted
include_pos – One or more POS tags with which to filter for good candidate keyterms. If None, include tokens of all POS tags (which also allows keyterm extraction from docs without POS-tagging.)
window_size – Size of sliding window in which term co-occurrences are determined to occur. Note: Larger values may dramatically increase runtime, owing to the larger number of co-occurrence combinations that must be counted.
topn – Number of top-ranked terms to return as keyterms. If int, represents the absolute number; if float, must be in the open interval (0.0, 1.0), and is converted to an integer by int(round(len(candidates) * topn))
idf – Mapping of normalize(term) to inverse document frequency for re-weighting of unigrams (n-grams with n > 1 have df assumed = 1). Results are typically better with idf information.

Returns

Sorted list of top topn key terms and their corresponding SGRank scores

Raises

ValueError – if topn is a float but not in (0.0, 1.0] or window_size < 2

References

Danesh, Sumner, and Martin. “SGRank: Combining Statistical and Graphical Methods to Improve the State of the Art in Unsupervised Keyphrase Extraction.” Lexical and Computational Semantics (* SEM 2015) (2015): 117.

Keyterm Extraction Utils¶

textacy.ke.utils.normalize_terms(terms: Union[Iterable[spacy.tokens.span.Span], Iterable[spacy.tokens.token.Token]], normalize: Optional[Union[str, Callable[[Union[spacy.tokens.span.Span, spacy.tokens.token.Token]], str]]]) → Iterable[str][source]¶

Transform a sequence of terms from spaCy Token or Span s into strings, normalized by normalize.

Parameters

terms –
normalize – If “lemma”, lemmatize terms; if “lower”, lowercase terms; if None, use the form of terms as they appear in terms; if a callable, must accept a Token or Span and return a str, e.g. textacy.spacier.utils.get_normalized_text().

Yields

str

textacy.ke.utils.aggregate_term_variants(terms: Set[str], *, acro_defs: Optional[Dict[str, str]] = None, fuzzy_dedupe: bool = True) → List[Set[str]][source]¶

Take a set of unique terms and aggregate terms that are symbolic, lexical, and ordering variants of each other, as well as acronyms and fuzzy string matches.

Parameters

terms – Set of unique terms with potential duplicates
acro_defs – If not None, terms that are acronyms will be aggregated with their definitions and terms that are definitions will be aggregated with their acronyms
fuzzy_dedupe – If True, fuzzy string matching will be used to aggregate similar terms of a sufficient length

Returns

Each item is a set of aggregated terms.

Notes

Partly inspired by aggregation of variants discussed in Park, Youngja, Roy J. Byrd, and Branimir K. Boguraev. “Automatic glossary extraction: beyond terminology identification.” Proceedings of the 19th international conference on Computational linguistics-Volume 1. Association for Computational Linguistics, 2002.

textacy.ke.utils.get_longest_subsequence_candidates(doc: spacy.tokens.doc.Doc, match_func: Callable[[spacy.tokens.token.Token], bool]) → Iterable[Tuple[spacy.tokens.token.Token, …]][source]¶

Get candidate keyterms from doc, where candidates are longest consecutive subsequences of tokens for which all match_func(token) is True.

Parameters

doc –
match_func – Function applied sequentially to each Token in doc that returns True for matching (“good”) tokens, False otherwise.

Yields

Next longest consecutive subsequence candidate, as a tuple of constituent tokens.

textacy.ke.utils.get_ngram_candidates(doc: spacy.tokens.doc.Doc, ns: Union[int, Collection[int]], *, include_pos: Optional[Union[str, Collection[str]]] = 'NOUN', 'PROPN', 'ADJ') → Iterable[Tuple[spacy.tokens.token.Token, …]][source]¶

Get candidate keyterms from doc, where candidates are n-length sequences of tokens (for all n in ns) that don’t start/end with a stop word or contain punctuation tokens, and whose constituent tokens are filtered by POS tag.

Parameters

doc –
ns – One or more n values for which to generate n-grams. For example, 2 gets bigrams; (2, 3) gets bigrams and trigrams.
include_pos – One or more POS tags with which to filter ngrams. If None, include tokens of all POS tags.

Yields

Next ngram candidate, as a tuple of constituent Tokens.

Information Extraction¶

Keyterm Extraction¶

Keyterm Extraction Utils¶

Navigation

Related Topics