CQ4OE Catalogue

Tasks, dimensions, and atomic metrics for the CQ4OE Benchmark

Tasks

The two evaluation tracks of the benchmark. Each task takes a set of competency questions as input and produces a different artefact.

Short Name	Title	Coverage	Identifier	Keywords
CQ2Term	Term-level generation of classes and properties from competency questions	2 dimensions	https://w3id.org/cq4oe/task/CQ2Term	Term Generation, Task, Competency Questions
CQ2Onto	Full OWL ontology generation from competency questions	6 dimensions	https://w3id.org/cq4oe/task/CQ2Onto	Ontology Generation, Task, Competency Questions

Evaluation Dimensions

Each dimension defines a scoped target against which the atomic metrics are computed.

Short Name	Title	Task	Identifier	Keywords
ClassProperty	Class and property generation	CQ2Term, CQ2Onto	https://w3id.org/cq4oe/dimension/ClassProperty	Term Generation, Ontology Generation, Dimension, Classes, Properties
PropertyCharacteristics	Property characteristics modelling	CQ2Onto	https://w3id.org/cq4oe/dimension/PropertyCharacteristics	Property Modelling, Dimension
Triple	Domain and range triple evaluation	CQ2Onto	https://w3id.org/cq4oe/dimension/Triple	Domain/Range, Dimension, Triples
Axiom	TBox axiom-level structural evaluation	CQ2Onto	https://w3id.org/cq4oe/dimension/Axiom	TBox Axioms, Dimension
HierarchyClosure	Hierarchy closure under reasoning	CQ2Onto	https://w3id.org/cq4oe/dimension/HierarchyClosure	Hierarchy Closure, Dimension
CQCoverage	CQ condition coverage under clouser level	CQ2Term, CQ2Onto	https://w3id.org/cq4oe/dimension/CQCoverage	Competency Questions, Coverage, Dimension

Metrics

The atomic measurements computed by the evaluation pipeline. Each metric is a single, reusable measurement that may be applied across one or more dimensions.

Short Name	Title	Category	Dimension(s)	Identifier	Keywords	Status
HardMatch	Exact lexical string comparison	Ontology Concept Matching	ClassProperty	https://w3id.org/cq4oe/metric/HardMatch	Ontology Concept Matching, Metric, Classes, Properties, Lexical	Active
SequenceMatch	Token-level string similarity comparison	Ontology Concept Matching	ClassProperty	https://w3id.org/cq4oe/metric/SequenceMatch	Ontology Concept Matching, Metric, Classes, Properties, Lexical	Active
LevenshteinDistance	Character-level edit distance approximation	Ontology Concept Matching	ClassProperty	https://w3id.org/cq4oe/metric/LevenshteinDistance	Ontology Concept Matching, Metric, Classes, Properties, Edit Distance	Active
JaroWinklerDistance	String similarity emphasizing common prefixes	Ontology Concept Matching	ClassProperty	https://w3id.org/cq4oe/metric/JaroWinklerDistance	Ontology Concept Matching, Metric, Classes, Properties, Lexical	Active
SemanticCosineSimilarity	Semantic distance over dense vector representations	Ontology Concept Matching	ClassProperty	https://w3id.org/cq4oe/metric/SemanticCosineSimilarity	Ontology Concept Matching, Metric, Classes, Properties, Encoder, Semantic	Active
AggregatedTop3	Mean of the three highest non-hard similarity scores	Ontology Concept Matching	ClassProperty	https://w3id.org/cq4oe/metric/AggregatedTop3	Ontology Concept Matching, Metric, Score Aggregation	Active
Synonyms	Exact match against WordNet synonym relations	Ontology Concept Matching	ClassProperty	https://w3id.org/cq4oe/metric/Synonyms	Ontology Concept Matching, Metric, Classes, Properties, WordNet	Inactive
InformationContent	Semantic-level comparison using WordNet synsets and IC scores	Ontology Concept Matching	ClassProperty	https://w3id.org/cq4oe/metric/InformationContent	Ontology Concept Matching, Metric, Classes, Properties, WordNet	Inactive
Precision	Fraction of generated elements that match the gold standard	Set-Based Evaluation	ClassProperty, PropertyCharacteristics, Triple, Axiom, HierarchyClosure	https://w3id.org/cq4oe/metric/Precision	Ontology Evaluation, Metric, Precision	Active
Recall	Fraction of gold elements matched by the generation	Set-Based Evaluation	ClassProperty, PropertyCharacteristics, Triple, Axiom, HierarchyClosure	https://w3id.org/cq4oe/metric/Recall	Ontology Evaluation, Metric, Recall	Active
F1	Harmonic mean of precision and recall	Set-Based Evaluation	ClassProperty, PropertyCharacteristics, Triple, Axiom, HierarchyClosure	https://w3id.org/cq4oe/metric/F1	Ontology Evaluation, Metric, F1	Active
EmbeddingDiagnostic	Soft similarity over normalized textual serializations	Diagnostic Evaluation	Triple, Axiom	https://w3id.org/cq4oe/metric/EmbeddingDiagnostic	Ontology Evaluation, Diagnostic, Embedding, Triples, Axioms	Active
AtLeastOneCoverage	Fraction of CQs with at least one required element correctly generated	Competency Question Coverage	CQCoverage	https://w3id.org/cq4oe/metric/AtLeastOneCoverage	Ontology Engineering, Metric, Competency Questions, Coverage	Active
MeanCoverage	Average per-CQ coverage rate across all retained CQs	Competency Question Coverage	CQCoverage	https://w3id.org/cq4oe/metric/MeanCoverage	Ontology Engineering, Metric, Competency Questions, Coverage	Active
FullCoverage	Fraction of CQs whose required elements are fully generated	Competency Question Coverage	CQCoverage	https://w3id.org/cq4oe/metric/FullCoverage	Ontology Engineering, Metric, Competency Questions, Completeness, Coverage	Active
ClosureRescueCoverage	CQ coverage credited via reasoner-inferred subsumption closure	Competency Question Coverage	CQCoverage, HierarchyClosure	https://w3id.org/cq4oe/metric/ClosureRescueCoverage	Ontology Engineering, Metric, Competency Questions, Reasoning, HermiT, Closure	Active