The Influence of Background Data Size on the Performance of a Score-based Likelihood Ratio System: A Case of Forensic Text Comparison

1 Abstract This study investigates the robustness and stability of a likelihood ratio–based (LR- based) forensic text comparison (FTC) sys- tem against the size of background popula- tion data. Focus is centred on a score-based approach for estimating authorship LRs. Each document is represented with a bag- of-words model, and the Cosine distance is used as the score-generating function. A set of population data that differed in the num- ber of scores was synthesised 20 times us- ing the Monte-Carol simulation technique. The FTC system’s performance with differ- ent population sizes was evaluated by a gra- dient metric of the log–LR cost (Cllr). The experimental results revealed two out- comes: 1) that the score-based approach is rather robust against a small population size—in that, with the scores obtained from the 40~60 authors in the database, the sta- bility and the performance of the system become fairly comparable to the system with a maximum number of authors (720); and 2) that poor performance in terms of Cllr, which occurred because of limited background population data, is largely due to poor calibration. The results also indi- cated that the score-based approach is more robust against data scarcity than the fea- ture-based approach; however, this finding obliges further study. 1 Introduction: The Likelihood Ratio Framework and Forensic Text Com- parison The likelihood ratio (LR) conceptual framework has been studied for its effect on various types of forensic evidence; it was mathematically shown that, with some very reasonable assumptions, the LR is the only way of assessing the uncertainty inherited in evidential evaluation (Aitken, 2018; Aitken and Taroni, 2004; Good, 1991). It is be- coming recognised as the logical and legally cor- rect framework for both analysing forensic evi- dence and presenting it in court (Balding, 2005; Evett et al., 1998; Marquis et al., 2011; Morrison, 2009; Neumann et al., 2007). Yet, some argue that the LR is one possible tool for communicating to decision makers (Lund and Iyer, 2017: 1). Alt- hough forensic text comparison (FTC) currently lags behind other forensic sciences, some studies have demonstrated that linguistic text evidence can be properly analysed using the LR framework (Ishihara, 2014, 2017a, 2017b). In the LR framework, instead of assessing the probabilities of two competing hypotheses given the evidence, the probabilities of observing the ev- idence (E) are assessed given the hypotheses: the prosecution hypothesis (Hp) against the defence hypothesis (Hd) (Aitken and Stoney, 1991; Aitken and Taroni, 2004; Robertson et al., 2016). There- fore, the LR can be defined as in Equation (1). LR= p(E|Hp) p(E|Hd) (1) In the case of FTC, the LR is the ratio between the two conditional probabilities of the measured difference (considered the evidence E) between the source-known texts (i.e., from the suspect) and the source-questioned texts (i.e., from the offender): one represents the probability of the evidence if they had been produced by the same author (Hp), and the other represents the probability of observ- ing the same evidence if they had originated from different authors (Hd). …

Content truncated to 500 words