Skip to content

Preprocessing for Drift Dimension Generation

pos_annotation

annotate_pos(dataset, dataset_name, device='cuda')

Annotate part-of-speech (POS) information for a dataset.

This function detects the language of each document and applies language-specific POS tagging pipelines to generate token-level POS annotations. Annotated datasets can optionally be exported to CSV files.

Parameters:

Name Type Description Default
dataset DataFrame

Input dataset containing at least a "content" column with the documents to process.

 required
dataset_name str

Dataset name used to export the generated files. If None, results are not exported.

 required
device str

Computation device used during POS annotation, such as "cpu" or "cuda". Defaults to "cuda".

 'cuda'

Returns:

Type Description
tuple

Tuple containing:

  • pd.DataFrame: input dataset enriched with document identifiers, processed content, and detected language labels.
  • pd.DataFrame: token-level POS annotations.
Notes

Language detection is performed before POS annotation in order to apply language-specific tagging pipelines.

POS annotations are generated independently for each detected language group.

pos_distributions

get_syntactic_content_distribution(pos_annotations, docs_df, allowed_upos=['NOUN', 'PRON', 'VERB', 'ADV', 'ADJ', 'DET'], **kwargs)

Calculate syntactic content distributions from UPOS-based rules.

This function computes the distribution of syntactic content rules across periods using Universal Part-of-Speech (UPOS) annotations. A rule is defined as a permutation of the UPOS tags in allowed_upos. For each document, a rule is considered present if it appears as an ordered, non-contiguous subsequence in the document's UPOS sequence.

Parameters:

Name Type Description Default
pos_annotations DataFrame

DataFrame containing Stanza POS annotations. It must include at least the following columns:

  • "doc_id": document identifier.
  • "id": token-level identifier within the document.
  • "upos": UPOS tag for the token.
 required
docs_df DataFrame

DataFrame containing document metadata. It must include at least the following columns:

  • "doc_id": document identifier.
  • "period_id": period identifier.
 required
allowed_upos list[str]

UPOS tags used to build syntactic content rules. Defaults to ["NOUN", "PRON", "VERB", "ADV", "ADJ", "DET"].

 ['NOUN', 'PRON', 'VERB', 'ADV', 'ADJ', 'DET']
**kwargs

Additional keyword arguments.

 {}

Returns:

Type Description
DataFrame

DataFrame representing the syntactic content distribution.

The DataFrame contains one row per period, a "period_id" column, and one column per rule. Rule columns are sorted alphabetically, and values represent normalized rule probabilities within each period.
Notes
  • If pos_annotations is empty, the function returns an empty DataFrame with the expected column structure.
  • Rules are generated as all permutations of the UPOS tags in allowed_upos.
  • Probabilities are normalized within each period.

get_syntactic_style_distribution(pos_annotations, docs_df, context_size=4, min_count=4, **kwargs)

Compute syntactic style distributions from UPOS tag sequences.

This function estimates conditional UPOS transition probabilities based on sequential POS contexts. For each context, the probability of a subsequent UPOS tag is computed across temporal periods.

Parameters:

Name Type Description Default
pos_annotations DataFrame

DataFrame containing token-level UPOS annotations with at least the columns "doc_id", "id", and "upos".

 required
docs_df DataFrame

DataFrame containing document metadata with at least the columns "doc_id" and "period_id".

 required
context_size int

Number of preceding UPOS tags used to define the context. Defaults to 4.

 4
min_count int

Minimum number of global occurrences required for a context to be included in the distribution. Defaults to 4.

 4
**kwargs

Additional keyword arguments.

 {}

Returns:

Type Description
DataFrame

Wide-format DataFrame where:

  • each row corresponds to a period_id,
  • each column represents a conditional probability of the form P(next_upos | context),
  • values correspond to the estimated conditional probabilities.
Notes

Contexts with fewer than min_count global occurrences are excluded from the computation.

Missing conditional probabilities are filled with 0.0.

If no valid contexts are found, an empty DataFrame containing only the "period_id" column is returned.

get_lexical_distribution(pos_annotations, docs_df, allowed_upos=['NOUN', 'VERB', 'ADV', 'ADJ'], **kwargs)

Calculate lexical distributions from part-of-speech annotations.

This function computes the distribution of lexical items across periods using POS annotations and document metadata. Frequencies are normalized within each period to obtain probability distributions.

Parameters:

Name Type Description Default
pos_annotations DataFrame

DataFrame containing POS annotations. It must include at least the columns "doc_id", "upos", and either "lemma" or "text".

 required
docs_df DataFrame

DataFrame containing document metadata. It must include at least the columns "doc_id" and "period_id".

 required
allowed_upos list

Universal POS tags included in the lexical distribution. Defaults to ["NOUN", "VERB", "ADV", "ADJ"].

 ['NOUN', 'VERB', 'ADV', 'ADJ']
**kwargs

Additional keyword arguments.

 {}

Returns:

Type Description
DataFrame

DataFrame representing the lexical distributions.

Each row corresponds to a period and each column corresponds to a lexical item. Values represent normalized lexical frequencies within each period.
Notes
  • The POS annotations are merged with the document metadata to associate each annotation with its corresponding period.
  • Only annotations whose UPOS tags are included in allowed_upos are considered.
  • Lexical frequencies are normalized by row sums so that each period distribution sums to 1.
  • The resulting DataFrame is sorted by column names for consistent ordering.

topic_distributions

get_thematic_dimension(docs_df, nr_topics=30, embedding_model='intfloat/multilingual-e5-large', **kwargs)

Compute thematic topic distributions across periods using BERTopic.

This function applies BERTopic to the input documents and calculates the distribution of topics across periods. Topic frequencies are normalized within each period to obtain probability distributions.

Parameters:

Name Type Description Default
docs_df DataFrame

DataFrame containing the input documents. It must include:

  • "content": text content of each document.
  • "period_id": identifier of the associated period.
 required
nr_topics int

Number of topics generated by BERTopic. Defaults to 30.

 30
embedding_model str

Embedding model used by BERTopic. Defaults to "intfloat/multilingual-e5-large".

 'intfloat/multilingual-e5-large'
**kwargs

Additional keyword arguments passed to the BERTopic model.

 {}

Returns:

Type Description
DataFrame

DataFrame representing thematic topic distributions across periods.

The DataFrame contains a "period_id" column and one column per topic. Values represent normalized topic frequencies within each period.

TokenGenerator

A class for tokenizing text inputs using a pretrained transformer model.

Attributes:

pretrained_model : str The identifier of the pretrained transformer model. batch_size : int The number of text samples to process in a single batch. max_length : int The maximum sequence length for tokenization. padding : bool or str The padding strategy ('max_length', 'longest' (or True)). truncation : bool Whether to truncate inputs to max_length. tokenizer : AutoTokenizer The tokenizer instance loaded from the Hugging Face Transformers library. Encoding : namedtuple A named tuple structure for storing tokenized inputs (input_ids, attention_mask).

__init__(pretrained_model, batch_size=8, tokenizer_max_len=512, tokenizer_padding='max_length', tokenizer_truncation=True)

Initialize with tokenizer and configuration settings.

Parameters:

pretrained_model : str The name or path of the pretrained transformer model. batch_size : int, optional (default=8) The batch size for processing text inputs. tokenizer_max_len : int, optional (default=512) The maximum length for tokenized sequences. tokenizer_padding : bool or str, optional (default=max_length) The padding strategy ('max_length', 'longest' (or True)). tokenizer_truncation : bool, optional (default=True) Whether to truncate inputs to max_length.

tokenize_texts(texts, output=list)

Tokenize a list of text inputs in batches.

Parameters:

texts : list of str A list of text strings to tokenize. output : type, optional (default=list) The type of the output (dict or list of namedtuples).

Returns:

dict or list - If output is dict: A dictionary with 'input_ids' and 'attention_mask' as tensor lists. - If output is list: A list of namedtuples with 'input_ids' and 'attention_mask' attributes.

EmbeddingsGenerator

A class for generating text embeddings using a pretrained transformer model.

Attributes:

train_device : str The device to use for computation ('cpu' or 'cuda'). pretrained_model : str The identifier of the pretrained transformer model. batch_size : int The number of input samples to process in a single batch. pooling_mode : str The pooling strategy to apply to the embeddings. Options: - 'cls_pooling': Use the [CLS] token embedding. - 'mean_pooling': Average all token embeddings. - 'no_pooling': Return all token embeddings. device : torch.device The PyTorch device to which the model is loaded. model : AutoModel The transformer model loaded from the Hugging Face Transformers library. encoder : AutoModel The encoder module of the model, used for generating embeddings.

__init__(pretrained_model, train_device='cpu', batch_size=8, pooling_mode='cls_pooling')

Initializes the EmbeddingsGenerator with a pretrained model.

Parameters:

pretrained_model : str The name or path of the pretrained transformer model. train_device : str, optional (default='cpu') The device to use for computation ('cpu' or 'cuda'). batch_size : int, optional (default=8) The batch size for processing inputs. pooling_mode : str, optional (default='cls_pooling') The pooling strategy ('cls_pooling', 'mean_pooling', or 'no_pooling').

generate_embeddings(inputs_tokenized)

Generate embeddings for tokenized inputs.

Parameters:

inputs_tokenized : list A list of tokenized inputs, where each input is an encoding with 'input_ids' and 'attention_mask'.

Returns:

torch.Tensor The generated embeddings as a tensor.