We next evaluated the performance of PRISM on biological data derived from yeast. Without changing any of the parameters from those determined to be optimal for the synthetic dataset, we ran PRISM on regulons obtained from the S. Cerevisiae Protein Database (SCPD). The SCPD regulons have been used for comparisons between motif-finding programs previously (Sinha and Tompa, 2000; Shinozake et al., 2003). Following the procedure outlined in these earlier performance comparisons, we ran PRISM on the 10 regulons in the SCPD database with the most degenerate cis-regulatory elements. Phi scores were obtained for the three most over-represented motifs against the list of binding sites available on SCPD (Table 3). This method provides an objective, biologically relevant basis for comparison between motif-finding programs based on different motif models, since it is based on the degree of overlap between the reported binding sites (output from each program) and the known (biologically determined) transcription factor binding sites.
We compared the Phi scores obtained by PRISM to the previously published results of four other programs: MEME, YMF, AlignACE and SuperPosition (Shinozake et al., 2003). The results are summarized in Table 4. We employed a number of summary metrics that were first used by Sinha and Tompa, 2003. While the dataset is small, PRISM is the best performing of all five programs by nearly all of these summary metrics. PRISM has the highest average Phi score, the highest number of motifs scoring above 0.5 and 0.33, and the highest number of ‘wins’ (as defined by Sinha and Tompa, 2000). By the clear win/loss metric PRISM clearly outperforms SuperPosition, YMF and AlignACE.
The regulons in the Saccharomyces Cerevisiae Protein database (SCPD) are carefully curated and contain no extraneous upstream sequences. In contrast, input sequences for motif-finding programs typically consist of a set of coordinately expressed genes derived from microarray data. Parallel regulation, as well as a rapid serial response of downstream genes are both capable of generating coordinated expression patterns in the absence of coordinate regulation. In addition, errors in clustering may also lead to the inclusion of extraneous upstream sequences.
In order to test the robustness of the PRISM algorithm to increased levels of extraneous upstream sequences in the dataset, we performed the following experiment. For every regulon in our original test set, we added a number of randomly selected upstream regions (corresponding to 0.5, 1.0, 2.0 and 4.0 times the number of sequences in the original regulon). This created 5 datapoints for each regulon, each corresponding to a different level of noise. We ran PRISM on each of these data points and assessed its accuracy, once again, via the Phi score metric. The results are summarized in Figure 9. As can be seen, the PRISM algorithm is exceptionally stable to noise. When four times as many extraneous upstream sequences are present in the dataset as real ones, PRISM’s performance on the top 5 regulons decreases from 0.48 to 0.20, and the rate of decay averaged out across all 5 regulons is extremely gradual. (Adding in the 5 regulons for which PRISM’s initial Phi scores are lower leads to a somewhat artificial improvement in its performance in this experiment, since PRISM will be trivially robust to noise on regulons where its original Phi score was 0). This behavior is in marked contrast to other motif-finding programs (such as Gibbs sampler, whose performance decays by 70% in the presence of 50% noisy genes), and makes PRISM a practical solution for motif-finding from microarray experiments.
