EmbeddingDiagnostic: Soft similarity over normalized textual serializations

This metric aims to evaluate near-miss similarity between generated and gold standard structural elements that fail strict matching. Each element (a triple or a TBox axiom) is rendered as a normalised text serialisation, encoded with a pretrained sentence-transformer, and compared by cosine similarity. The score is reported separately and does not affect the strict Precision, Recall, or F1 of the dimension.

Example:

Gold_Axiom: Animal SubClassOf eats some Plant

Generated_Axiom: Animal SubClassOf consumes some Vegetable

Result:

  1. Strict_match = 0 (different property and class names)
  2. EmbeddingDiagnostic ≈ 0.84 (close in meaning, flagged as a near miss)

What is being measured?

This metric helps diagnose how close a model came to a correct structural answer when it did not produce an exact match. It is a diagnostic for systematic deviations rather than a scoring signal, used to investigate why a generated element was rejected by strict comparison.

Metric information

RDF Resource Description Framework Flyer Icon
Title
Soft similarity over normalized textual serializations
Identifier
Keywords
Ontology Evaluation, Diagnostic, Embedding, Triples, Axioms
Contact Point
Publisher
Version
0.1.0
License
Category
Diagnostic Evaluation
Dimension
Task
Applicable for
Benchmark
CQ2OE Benchmark: Competency Questions for Ontology Engineering. A benchmark for evaluating LLM-assisted ontology generation from competency questions.
Status
Active
Active Proposed Inactive Deprecated
Ontology Engineering Group https://oeg.fi.upm.es