The Evaluator class evaluates the quality of an aggregate list with respect to a given set of judgments that determine the relevance of (some, or all) list elements. The set of relevance judgments is stored in a Rels object. The results of the evaluation (i.e. evaluation measures) are written in a CSV file.
Implementation files
The Evaluator class is defined in the src/Evaluator.h header file; its member functions are implemented in src/Evaluator.cpp.
Technical Details
The evaluation process takes place inside a Query object, provided that a file of relevance judgments is provided to FLAGR (see more in the article about Input and Output files). For this reason, a Query object contains a pointer to a single Evaluator object.
The evaluation process is performed by the evaluate member function. The procedure is based on the Rels object that contains the aforementioned relevance judgments. Finally, the four following arrays are created:
precision: in its \(i\)-th position, it stores the value of the Precision measure at the \(i\)-th element of the aggregate list.recall: in its \(i\)-th position, it stores the value of the Recall measure at the \(i\)-th element of the aggregate list.dcg: in its \(i\)-th position, it stores the value of the Discounted Cumulative Gain (DCG) measure at the \(i\)-th element of the aggregate list.ndcg: in its \(i\)-th position, it stores the value of the normalized Discounted Cumulative Gain (nDCG) measure at the \(i\)-th element of the aggregate list.
The following interactive block diagram depicts the functionality of the Evaluator class inside the entire architecture of FLAGR.
