 Method
 Open Access
 Published:
Coverage and error models of proteinprotein interaction data by directed graph analysis
Genome Biology volume 8, Article number: R186 (2007)
Abstract
Using a directed graph model for bait to prey systems and a multinomial error model, we assessed the error statistics in all published largescale datasets for Saccharomyces cerevisiae and characterized them by three traits: the set of tested interactions, artifacts that lead to falsepositive or falsenegative observations, and estimates of the stochastic error rates that affect the data. These traits provide a prerequisite for the estimation of the protein interactome and its modules.
Background
Within the past decade a large amount of data on proteinprotein interactions in cellular systems has been obtained by the highthroughput scaling of technologies, such as the yeast twohybrid (Y2H) system and affinity purificationmass spectrometry (APMS) [1–15]. This opens the possibility for molecular and computational biologists to obtain a comprehensive understanding of cellular systems and their modules [16]. There are many references in the literature, however, to the apparent noisiness and low quality of highthroughput protein interaction data. Evaluation studies have reported discrepancies between the datasets, large error rates, lack of overlap, and contradictions between experiments [17–30]. The interpretation and integration of these large sets of protein interaction data represents a grand challenge for computational biology.
In essence, inference on the existence of an interaction between two proteins is made based on the measured data, and such inference can either be right or wrong. Most publicly available data are stored as positive measured results, and therefore most analyses have employed the most obvious method to infer interactions; a positive observation indicates an interaction, whereas a negative observation or no observation does not. This method, although useful and sometimes unavoidable, does not make use of other indicators for the presence or absence of interactions.
The most useful and yet seldom used indicator is the information about which set of interactions were tested. As mentioned, most studies report positively measured interactions but few report the negative measurements. It is quite often the case that untested protein pairs and negative measurements are not distinguished. A second indicator of the presence of an interaction is reciprocity. Bait to prey systems allow for the testing of an interaction between a pair of proteins in two directions. If bidirectionally tested, we anticipate the result as both positive or both negative. Failure to attain reciprocity indicates some form of error. A third indicator is the type of interaction being assayed; direct physical interactions must be differentiated from indirect interactions, and this difference plays an important role in inference. In the Y2H system, two proteins are modified so that a physical interaction between the two can reconstitute a functioning transcription factor. In APMS, a single protein is chosen and modified, and each pulldown detects proteins that are in some complex with the selected one but may not necessarily directly interact with the chosen protein.
Restricting our attention to bidirectionally tested interactions, we can use a binomial model to identify proteins that either find a disproportionate number of prey relative to the number of baits that find them or vice versa. For the APMS experiments, there is an association between whether a protein exhibits this discrepancy and its relative abundance in the cell. For the Y2H system, analyses conducted separately by Walhout and coworkers [31], Mrowka and colleagues [19], and Aloy and Russell [32] have reported on this type of artifact and have discussed a relationship between it and some bait proteins' propensity to act alone as activators of the reporter gene. Our methods provide a simple test to identify proteins that are probably affected by such systematic errors. Such diagnostics can aid in the interpretation of the data and in the design of future experiments. By restricting attention to proteins that are not seen to be affected by this artifact, we can refine the error modeling and the subsequent biologic analysis.
Results and discussion
Tested interactions and their representations
In the Y2H system, the bait is the protein tagged with the DNA binding domain, and the prey is the hybrid with the activation domain. Only those constructs that result in a functional fusion protein will be tested as bait or as a prey. In APMS, a piece of DNA encoding a tag is inserted into a proteincoding gene, so that yeast cells express the tagged protein. These are the baits. The prey are unmodified proteins expressed under the conditions of the experiment. The set of tested baits, even in experiments intended to be genome wide, can be quite restricted. For example, Gavin and coworkers [10] designed their experiment to employ the 6,466 open reading frames that were at that time annotated with the Saccharomyces cerevisiae genome, but successfully obtained tandem affinity purifications for 1,993 of those. The remaining 4,473 (69%) failed at various stages, because, for example, the tagged protein failed to express or the bands resulting from the gel electrophoresis were not well separated.
It is difficult to give an accurate enumeration of the sets of tested baits and tested prey in an experiment, and often the published data do not contain sufficient detail to allow identification of these sets. As a proxy, we introduce the concepts of viable baits and viable prey; the first is the set of baits that were reported to have interacted with at least one prey, and the latter is similarly defined. These quantities are unambiguously obtained from the reported data and provide reasonable surrogate estimates for what are the tested baits and tested prey. The set of ordered pairs, one being a viable bait and the other a viable prey, are interactions for which we have a level of confidence that were experimentally tested and could, in principle, have been detected. The failure to detect an interaction between a viable bait and a viable prey is informative, whereas the absence of an observed interaction between an untested bait and prey is not. This approach overemphasizes positive interactions; potentially, valid data on tested proteins that have truly no interactions with any other tested protein will be discarded.
Protein interactions have been generally modeled by ordinary graphs [33]. The proteins correspond to the nodes of the graph, and edges between protein pairs indicate an interaction (either physical interaction or complex comembership). For measured data from bait to prey systems, protein pairs are ordered (b,p) to distinguish a bait b from a prey p. There are three types of relationships between protein pairs of an experimental dataset: tested with an observed interaction, tested with no observed interaction, and untested. An adequate representation for this type of datum would be a directed graph with edge attributes. A directed edge (b,p)_{+} signals testing with an observed interaction, whereas a directed edge (b,p)_{} signals testing without an observed interaction. Interactions between proteins that are not adjacent were not tested. In those cases in which all protein pairs were reciprocally tested, we can suppress the (b,p)_{} edges, and a directed graph (digraph) is an adequate representation.
As mentioned above, information on which protein pairs were tested for an interaction is rarely explicitly reported, and so we represent the current data by a directed graph with node attributes. Using viability as a proxy for testing, the nodes with nonzero outdegree are presumed to be the set of viable baits, and similarly the nodes with nonzero indegree are presumed to be the viable prey. Isolated nodes become identified as the set of untested proteins (both as bait and prey). We make use of such a digraph data structure in this report (Figure 1).
Interactome coverage
Given the experimental data, one can partition the proteins into four different sets: viable bait only (VB), viable prey only (VP), viable bait/prey (VBP), and the untested proteins. Figure 2 shows these proportions of the yeast genome as measured by each experiment. For most experiments, relatively large portions of the proteome were untested by the assay (gray area), thereby rendering an incomplete picture of the overall interactome [18, 21, 25, 34].
We considered whether the sets of viable bait and viable prey exhibited a coverage bias in the experimental assays. Applying a conditional hypergeometric test [35] to the terms within the cellular component branch of Gene Ontology (GO), we found that proteins annotated to categories such as nucleus (primarily Y2H), cytoplasm, and protein complex were overrepresented among the viable protein population relative to the yeast genome. This is not surprising because both Y2H and APMS assay two kinds of interactions in protein complexes. The Y2H technology is more successful in generating viable proteins within the nucleus because this is the cellular location where the test is performed, and so native proteins tend to work more successfully.
The conditional hypergeometric tests can also identify portions of the cellular component missed by either Y2H or APMS. For the Y2H technology, terms associated with mitochondrion, ribosome, and integral to membrane were underrepresented by viable proteins. Like the Y2H systems, the viable proteins from APMS assays were also underrepresented with respect to terms associated with mitochondrion and integral to membrane, but instead of ribosome APMS showed underrepresentation in vacuole. These underrepresented categories are limited by the technologies because all datasets were derived before progress had been made to probe membranebound proteins.
Every dataset, whether Y2H or APMS, exhibited underrepresentation for the term cellular component unknown. One possible explanation for this phenomenon can be attributed to the correlation between different technologies. It seems that proteins that are problematic in the Y2H and APMS systems might also be problematic in systems to determine their cellular localization. Ultimately, further experiments are needed to determine why certain GO categories are underrepresented. The hypergeometric analysis on each dataset can be found in the Additional data files.
These findings point to the fact that the subset of the interactome is either nonrandomly sampled or nonrandomly covered by the experiment. Either effect limits the type of inference that can be conducted on the resulting data. For instance, inference on statistics such as the degree distribution or the clustering coefficient of the overall graph is less meaningful as long as the direction and magnitude of the coverage or sampling biases are not well understood [20, 36, 37].
Systematic bias: per protein and experiment wide
The interactions between VBP proteins were tested in both directions, and a surprising yet useful observation is that there is a large number of unreciprocated edges in the data [32]. These unreciprocated interactions can be used to understand better the experimental errors.
Each VBP protein p has n_{ p }unreciprocated edges, and under the assumption of randomness we expect the number of unreciprocated inedges and outedges to be similar. More precisely, under the assumption that the direction of the edge is random, the number of unreciprocated inedges is distributed as the number of heads obtained by tossing a fair coin n_{ p }times. Based on this coin tossing model, we used a per protein binomial error model (see Materials and methods, below) to test the statistical significance for the number of unreciprocated inedges (heads) against the number unreciprocated outedges (tails). Figure 3 shows a partition of the VBP proteins from the data of Krogan and coworkers [11] based on the twosided statistical test derived from the binomial model with a P value threshold of 0.01. Those proteins falling outside the diagonal band are considered to be affected by a systematic bias.
It is interesting to note that the proportion of VBP proteins identified by the binomial error model as potentially affected by bias is quite small for the Y2H experiments and the smaller scale APMS experiments (<3%), whereas the two larger scale APMS experiments showed relatively greater proportions (>14%). It is equally important to note that although these proportions still constitute a minority of VBP proteins, these proteins (within the largescale APMS experiments) participate in a relatively large number of observed interactions, most of which are unreciprocated.
Having identified sets of proteins that are likely to have been affected by this systematic bias, we considered whether these proteins could be associated with biologic properties. To this end, we fit logistic regression models (Additional data files) to predict this effect, and in the APMS system we found evidence that the codon adaptation index (CAI) and protein abundance are associated with the highly unreciprocated indegree of VBP proteins (proteins that were found by an exceptionally high number of baits relative to the number of prey they found themselves when tested as baits). The CAI is a pergene score that is computed from the frequency of the usage of synonymous codons in a gene's sequence, and can serve as a proxy for protein abundance [38].
To visualize the association between such proteins and CAI, we plotted diagrams of the adjacency matrix. If the value of CAI is associated with the tendency of a protein to have a large number of unreciprocated edges, then we should see a pattern in the adjacency matrix when the rows and columns are ordered by ascending CAI values. We do this for the data reported by Gavin and coworkers [10] in Figure 4. We see a dark vertical band in Figure 4b representing a relatively high volume of prey activity. There is no corresponding horizontal band in Figure 4a, which suggests that the relationship of CAI to the APMS system is primarily reflected in a protein's indegree.
Next, we standardized the indegree for each protein by calculating its zscore (see Materials and methods, below) and then plotted the distributions of these zscores by their density estimates. Four experiments appeared to exhibit particularly distinct distributions (ItoFull, ItoCore, Gavin et al. 2006, and Krogan et al. 2006; Figure 5) [1, 10, 11]. The ItoFull [1] dataset shows the largest mean (approximately two to four times the mean of the other Y2H distributions). This is consistent with reports that there were many autoactivating baits in the ItoFull datasets [32]; if a relatively small number of baits autoactivate, resulting in the cell's expression of the reporter gene, then this artificially increases the number of inedges for a large number of prey proteins. Autoactivation would cause a shift in the zscore distribution in the positive direction. This effect is not seen in the ItoCore data.
Although Ito and coworkers [1] tried to eliminate systematic errors by generating the ItoCore subset of interactions, it is noteworthy to recall that they only used reproducibility as a criterion for validation without considering reciprocity. Consequently, almost half of the reciprocated interactions were not recorded in the ItoCore set. Although reproducibility is a necessary condition for validation, it is insufficient because systematic errors are often reproducible.
Among the APMS datasets, the data reported by both Gavin and coworkers [10] and Krogan and colleagues [11] display negative means. A possible interpretation of this effect can be attributed to the abundance of the prey under the conditions of the experimental assay. The APMS system is more sensitive in detecting the complex comembers of a particular bait than in the reverse. For instance, if a lowly expressed protein p is tagged and expressed as a bait and pullsdown proteins p_{1},...,p_{ k }as prey, then the reverse tagging of each protein of p_{1},...,p_{ k }will have a smaller probability of finding p. Even if the lowly abundant protein p is pulled down in the reverse tagging, the mass spectrometry may fail to detect p within the complex mixture [39, 40]. Both of these observations could explain why we observed proteins having an overall slightly higher outdegree than indegree, and therefore an overall slightly negative mean for the zscore distribution.
Finally, we wished to crosscompare the systematic errors between experiments. Only two experiments had sufficient size to give reasonable statistical power. Thus, to compare systematic errors of Gavin and coworkers [10] against those of Krogan and colleagues [11], we generated twoway tables (Tables 1 to 4; also, see Materials and methods, below). Although the concordance is not complete, there is evidence that overlapping sets of proteins are affected. This indicates that both experiment specific and more general factors could be at work, resulting in these unreciprocated edges.
Stochastic error rate analysis
There has been confusion in the literature when analyzing error statistics, because different articles have used different definitions for the same statistic. Proteins pairs can either interact or not, and so the pairs themselves can be partitioned into two distinct sets; the set of interacting pairs, I, and the set of noninteracting pairs, I^{C}. False negative (FN) interactions and true positive (TP) interactions can only occur within the set I, and therefore the false negative probability (P_{FN}) and the true positive probability (P_{TP}) are properties on I. Similarly, the false positive (P_{FP}) and true negative (P_{TN}) probabilities are properties on I^{C}[41]. These standard definitions, along with the values n = I and m = I^{C}, allow us to set up equations for the expectation values of three random variables: the number of reciprocated edges (X_{1}), the number of protein pairs between which no edge exists (X_{2}), and the number of unreciprocated edges (X_{3}).
E[X_{1}] = n (1  P_{FN})^{2} + mP_{FP}^{2} (1)
E[X_{2}] = nP_{FN}^{2} + m(1  P_{FP})^{2} (2)
E[X_{3}] = 2nP_{FN}(1  P_{FN}) + 2mP_{FP}(1  P_{FP}) (3)
We recall that if N is the number of proteins, then n + m = $\left(\begin{array}{c}N\\ 2\end{array}\right)$, which is the number of all pairs of proteins. Any two of these three equations imply the third, and therefore there are three unknowns and two independent equations. By the method of moments[42], we replace the left hand side of Equations1 to 3 with the observed values for the number of reciprocated interactions (x_{1}), for the number of reciprocally noninteracting protein pairs (x_{2}), and for the number of unreciprocated interactions (x_{3}); it follows that knowledge of any one of (P_{FP},P_{FN},n) yields the other two through an application of the quadratic formula (see Materials and methods, below). Otherwise, if none of these three parameters is known from other sources, then Equations 1 to 3 define a family of solutions (a onedimensional set of solutions in a space of three variables; Figure 6).
The variability, or stochastic error, that affects a bait to prey system can thus be characterized by a onedimensional curve in a threedimensional space, {(P_{FP},P_{FN},n)}, which depends on the experiment and can be estimated from the three experimentspecific numbers x_{1}, x_{2}, and x_{3}. If we can identify portions of the data that appear to be affected by systematic bias, such as that described in the preceding section, then we can set these aside and focus the characterization of the experimental errors on the remaining filtered set of data, typically with lower estimates for P_{FP} and P_{FN}.
To gain insight into the prevalence of FP and FN stochastic errors, we calculated estimates of the expected number of FP and FN observations using Equations 1 to 3, and present the results in Tables 5 and 6. Table 5 considers the worstcase scenario for FP errors, setting P_{FN} = 0, and hence assuming that all errors are false positives. We discuss the first row, corresponding to the data of ItoFull [1], as an example. A total of 720 proteins were not rejected in the twosided binomial test, and there are $\left(\begin{array}{c}720\\ 2\end{array}\right)$ = 258,840 protein pairs, excluding homomers. This gives us an upper limit for m. From the solution manifold shown in Figure 6d, we see that an estimate for P_{FP} is approximately 0.0008. From this it follows that the expected number of unreciprocated FP interactions is 414 and of reciprocated FP interactions is 0.17. The actual data contain 435 unreciprocated interactions and 68 reciprocated ones. So, even in the estimated worst case, when all errors are FP observations, reciprocated observations are still most likely due to true interactions.
It is important to contrast the nature of the stochastic error rates because there is confusion in the literature concerning these statistics. From Figure 6, the solution curve gives an estimate for the P_{FP} rate at 0.0008 conditioned on the ItoFull VBP data and conditioned on P_{FN} = 0; a similar estimate for the ItoCore dataset yields P_{FP} at 0.0025. The reason for this is because the number of noninteracting protein pairs in the former is estimated to be approximately 250,000, whereas this number is 8,000 for the latter. Table 5 shows that the number of expected false positively identified unreciprocated interactions for ItoFull is 414 and for the ItoCore is 41. Thus, although the P_{FP} rate of ItoFull is three times smaller than that of ItoCore, the expected number of falsely discovered interactions is an order of magnitude greater. Therefore, a generic interaction contained within ItoCore is much more likely to be true than one from ItoFull. Comparing the P_{FP} rate from ItoFull with the P_{FP} rate from ItoCore is unreasonable when the underlying sets of noninteracting proteins pairs are entirely different. The false discovery rate is more intuitive, and this statistic has often been confused in the literature with the FP rate.
We also considered the worstcase scenario for FN errors. By setting P_{FP} = 0, we calculated the expected number of unreciprocated and reciprocated false negatives in the absence of FP errors. These numbers are presented in Table 6. Because of the size of P_{FN}, we find that a large number of protein pairs between which no edge was reported in either direction may still, in truth, interact.
Ultimately, an observed unreciprocated interaction in the data indicates that either a FP or a FN observation was made. Computational models cannot definitively conclude which of these two occurred, but these models indicate the magnitude and nature of the problem and can be used to compare experiments, because those with relatively higher error rates should be discounted in any downstream analyses.
Conclusion
We have shown that protein interaction datasets can be characterized by three traits: the coverage of the tested interactions, the presence of biases in the assay that systematically affect certain subsets of proteins, and stochastic variability in the measured interactions. In turn, these three characteristics can benefit the design of future protein interaction experiments.
The set of interactions tested is important because datasets usually report positive results, but tend to be ambiguous on the significance of the unreported interactions. Is it because the interaction was tested and not detected, or because it was not tested in the first place? Distinguishing the two cases is important for inference and for integration across datasets. For the currently available datasets from Y2H and APMS, a practical estimate of what is the set of tested interactions is all pairs of tested bait and tested prey. A comprehensive list of tested proteins is usually not reported. We can, however, obtain a useful approximation for the tested baits and prey using the notion of viability. However, this assumption does introduce some bias, especially for experiments with relatively few bait proteins, because proteins that were tested but did not interact with any bait protein will not be counted, falsely raising the proportion of interactions. On the other hand, when complete data are not reported the presumption that interactions were tested, when they were not, introduces bias in the other direction.
There has been substantial interest in crossexperiment analysis, or in integrating data from multiple sources [19, 23, 24, 29, 30]. The possible pitfalls of naïve comparisons between two experimental datasets are depicted in Figure 7. The interactions in the intersection of the rectangles (red) were tested by both; the interactions in the green and purple areas were tested by one experiment but not the other; and the interactions in the light gray areas were tested by neither experiment. Any data analysis that does not keep track of these different coverage characteristics risks being misled. Therefore, coverage must be taken into consideration when integrating and comparing multiple datasets. Additionally, systematic bias due to the experimental assay affects the detection of certain interactions between protein pairs, and these systematic errors should be isolated from the dataset before the estimating the stochastic errors. Ultimately, many more steps are still needed to integrate datasets, and we discuss a few necessary components.
If the assay system were perfect, then all bidirectionally tested protein pairs would either be reciprocally adjacent or not. In practice, unreciprocated edges are observed, and they can be used to understand better the sources of error. Measurement error can be divided into two categories: systematic and stochastic. We have shown that there are proteins with an inordinate imbalance between unreciprocated inedges and outedges, and they behave in a systematically different way when used as a bait than when found as a prey. This is an indication that the interaction data involving these proteins contain either a large number of false positives or of false negatives. Further data are needed to differentiate between these two alternatives. The mode in which they fail is distinct from the unspecific stochastic errors that we model via the FP and FN rates, and hence they should be excluded from these analyses.
It is useful to distinguish between the concepts of stochastic and systematic measurement error. Systematic errors are due to imperfections or biases in the experimental system, and they occur in a correlated or reproducible manner. Stochastic errors occur at random in an irreproducible manner; in principle, they can be averaged out by repeating the experiment often enough. There are many benefits to an analysis that identifies and separates these two types of measurement error. We have identified one type of systematic error in bait to prey systems that appears to be associated with artifacts of the technologies.
The occurrence of unreciprocated edges also points to some of the aspects of the technologies that could be improved. In APMS experiments, this artifact shows a strong association to CAI and protein abundance. Because mass spectrometry techniques are known to be, at times, less sensitive in identifying proteins with low abundance in a complex mixture, refinements of such methodology could potentially yield more accurate measurements.
The methods we have described are useful for future application of Y2H or APMS. Newer experiments can, and should, take into consideration relative protein abundance when assaying protein interactions. Besides this, the GO category analysis for underrepresentation shows certain proteins and protein complexes that do not work as intended under the conditions of the assay system. Knowing which categories are underrepresented allows experimenters to adjust the technologies or create new technologies (such as the Y2H test for membrane bound proteins [43]).
These elementary questions of data preprocessing, quality assessment, and error modeling may appear far removed from the systemslevel modeling of biologic systems. Such a modeling, however, requires the use and integration of multiple different datasets, to increase the breadth and depth of the data compared with those from a single experiment. This can only be done if the error statistics and possible patterns in the errors are sufficiently understood. We believe that the methods and tools developed in this work provide a step in this direction.
Materials and methods
Graph theory
We use a directed graph with node attributes to represent each measured dataset. The proteins correspond to the node set, and directed edges correspond with ordered protein pairs of the form (b,p) showing that a bait b detects a prey p. The node set with nonzero outdegree corresponds with the set of viable baits, and the node set with nonzero indegree corresponds with the viable prey. We remove selfloops because we set aside homomer relationships. The subgraph generated by nodes that are both viable baits and viable prey will have tested all protein pairs bidirectionally.
Protein interaction data
We investigated 12 publicly available datasets for S. cerevisiae, of which seven were assayed by Y2H and five were assayed by APMS. We obtained [1–6, 10] from the IntAct repository [44] and [7–9, 11] from their primary sources. All datasets have two key properties: information on the bait to prey directionality is retained; and the prey population is documented as genome wide. A table with an overview of the datasets can be found in the Additional data files.
Statistical analysis
Binomial error model: detecting bias
The binomial error model assumes that indegrees and outdegrees are equally likely among unreciprocated edges of a bidirectionally tested protein. Thus, we presume that the number of unreciprocated outedges for any bidirectionally tested protein p is distributed as B(n_{ p },$\frac{1}{2}$), where n_{ p }is the total number of unreciprocated edges of p. Under this hypothesis, we can compute the P value for the observed measured directed degree for each protein p. The null hypothesis is rejected at the 0.01 threshold. Proteins for which we reject the null hypothesis are deemed likely to be affected by a systematic bias in the assay.
Multinomial error model
Let N be the number of proteins in an interactome of interest, then the total number of distinct protein pairs, excluding homomers, is $\left(\begin{array}{c}N\\ 2\end{array}\right)$. Denote the set of all unique interacting protein pairs among the N proteins by I and its complement by I^{C}. Recall that $\left(\begin{array}{c}N\\ 2\end{array}\right)$ = n + m, where n = I and m = I^{C}.
Only two of the three Equations 1 to 3 are independent, any two of them imply the third. We parameterize the onedimensional solution manifold by n (0 ≤ n ≤ $\left(\begin{array}{c}N\\ 2\end{array}\right)$). Relevant solutions are those for which 0 ≤ P_{FP},P_{FN} ≤ 1. Consider n given, then we can solve for P_{FN} in terms of P_{FP}:
Where we have defined Δ = (x_{2}  m)  (x_{1}  n). Here, x_{1} is the observed number of reciprocated interactions and x_{2} is the number of reciprocated noninteracting protein pairs. Making a substitution for P_{FN} in Equation 2, the problem reduces to a quadratic equation in one parameter, P_{FP}:
If we let
a = (n + m), b = (Δ  2n), and $\text{c}=n+\frac{{\Delta}^{2}}{4m}\frac{n}{m}{x}_{2}$,
then an application of the quadratic formula gives two solutions for p_{FP}:
Then, substituting an estimate of P_{FP} back into Equation 4 gives a solution for P_{ FN }. A similar argument carries through given any one of the three parameters {n,P_{FP},P_{FN}}. Thus, an estimate of one of the parameters generates estimates of the other two.
Conditional hypergeometric, logistic regression tests, and twoway tables
We grouped the yeast genome into several defined subsets (VB, VP, VBP, and those proteins appearing to be affected by bias), and we wished to determine whether the subsets showed overrepresentation/underrepresentation among biologic categories such as GO, Kyoto Encyclopedia of Genes and Genomes, and Pfam. We used the conditional hypergeometric testing as described by Falcon and Gentleman [35] to probe for such overrepresentation/underrepresentation at a P value threshold of 0.01. A list of such GO categories and Pfam domains can be generated by the R scripts hgGO.R and hgPfam.R, contained with the Additional data files.
For those proteins that are affected by a systematic bias of each experiment, we fitted a logistic regression on these sets against 31 protein properties reported in the Saccharomyces Genome Database [45] and set a P value threshold at 0.01.
Let S_{ i }be the set of proteins identified to be affected by a systematic bias in dataset i, and suppose we wish to compare S_{ i }against S_{ j }; we define two methods of generating S_{ i }and S_{ j }for such a comparison. One method is the application of the binomial test on the VBP subgraph of each dataset i exclusively to determine each S_{ i }. The second method aims to streamline the experimental conditions of i with that of j. First, we compute X = VBP_{ i }∩ VBP_{ j }; then we apply the binomial test on the X_{ i }subgraph as well as the X_{ j }subgraph (because the edgesets will be different). Obtaining such subsets allows us to generate a twoway table, T, to compare S_{ i }against S_{ j }. If the first method is used to generate the subsets S_{ i }and S_{ j }, then we must still restrict to X when computing T. T_{(2,2)} counts S_{ i }∩ S_{ j }; T_{(1,2)} and T_{(2,1)} count S_{ i }\S_{ j } and S_{ j }\S_{ i }, respectively; and T_{(1,1)} counts ${S}_{i}^{c}\cap {S}_{j}^{c}$. We can apply Fisher's exact test to ascertain the independence of these two sets at a designated P value threshold.
Per protein indegree zscore and cross experimental comparisons
Let o_{ p }be the unreciprocated outdegree for a protein p and i_{ p }its unreciprocated indegree. Then denote the number of unreciprocated edges by n_{ p }= i_{ p }+ o_{ p }. Assuming the distribution i_{ p }~ B(n_{ p },$\frac{1}{2}$), we can compute the standardized indegree (zscore) for p:
Estimating the number of stochastic false positive/false negative observations
We used the filtered data after setting aside proteins rejected by the twosided binomial tests to calculate the results presented in Tables 5 and 6. In the first case, we set P_{FN} = 0, and P_{FN} is the maximal value in the solution curve shown in Figure 6. m is estimated as $\left(\begin{array}{c}N\\ 2\end{array}\right)$. The expected number of unreciprocated FP observations is 2P_{FP}(1  P_{FP})m and of reciprocated FP observations is ${P}_{FP}^{2}m$. In the second case, we set P_{FP} = 0 and obtain n from the solution curve. The expected number of unreciprocated FN observations is 2P_{FN}(1  P_{FN})n and of reciprocated FN observations is ${P}_{FN}^{2}n$.
Software implementation and availability
The R/Bioconductor packages used in the statistical analysis in this report are all available as freely distributed and open source software packages with an Artistic license. They are integrated into the R/Bioconductor environment for statistical computing and bioinformatics and run on operating systems Windows, Mac OS X, and Unix.
Additional data files
The following additional data are available with the online version of this paper. Additional data file 1 demonstrates a full endtoend analysis of the protein interaction datasets. Additional data file 2 is the Bioconductor package ppiStats (version 1.3.5 of 22 June 2007) in 'source' format. Additional data file 3 is the ppiStats package in the 'Windows Binary' format.
Abbreviations
 APMS:

affinity purificationmass spectrometry
 CAI:

codon adaptation index
 FN:

false negative
 FP:

false positive
 GO:

Gene Ontology
 TN:

true negative
 TP:

true positive
 VB:

viable bait only
 VBP:

viable bait/prey
 VP:

viable prey only
 Y2H:

yeast twohybrid.
References
 1.
Ito T, Chiba T, Ozawa R, Yoshida M, Hattori M, Sakaki Y: A comprehensive twohybrid analysis to explore the yeast protein interactome. Proc Natl Acad Sci USA. 2001, 98: 45694574. 10.1073/pnas.061034498.
 2.
Cagney G, Uetz P, Fields S: Twohybrid analysis of the Saccharomyces cerevisiae 26S proteasome. Physiol Genomics. 2001, 7: 2734.
 3.
Tong A, Drees B, Nardelli G, Bader G, Brannetti B, Castagnoli L, Evangelista M, Ferracuti S, Nelson B, Paoluzi S, et al: A combined experimental and computational strategy to define protein interaction networks for peptide recognition modules. Science. 2002, 295: 321324. 10.1126/science.1064987.
 4.
Hazbun T, Malmstrom L, Anderson S, Graczyk B, Fox B, Riffle M, Sundin B, Aranda J, McDonald W, CH C, et al: Assigning function to yeast proteins by integration of technologies. Mol Cell. 2003, 12: 13531365. 10.1016/S10972765(03)004763.
 5.
Zhao R, Davey M, Hsu Y, Kaplanek P, Tong A, Parsons A, Krogan N, Cagney G, Mai D, Greenblatt J, et al: Navigating the chaperone network: an integrative map of physical and genetic interactions mediated by the Hsp90 chaperone. Cell. 2005, 120: 715727. 10.1016/j.cell.2004.12.024.
 6.
Uetz P, Giot L, Cagney G, Mansfield T, Judson R, Knight J, Lockshon D, Narayan V, Srinivasan M, Pochart P, et al: A comprehensive analysis of proteinprotein interactions in Saccharomyces cerevisiae. Nature. 2000, 403: 623627. 10.1038/35001009.
 7.
Gavin A, Bosche M, Krause R, Grandi P, Marzioch M, Bauer A, Schultz J, Rick J, Michon A, Cruciat C, et al: Functional organization of the yeast proteome by systematic analysis of protein complexes. Nature. 2002, 415: 141147. 10.1038/415141a.
 8.
Ho Y, Gruhler A, Heilbut A, Bader G, Moore L, Adams S, Millar A, Taylor P, Bennett K, Boutilier K, et al: Systematic identification of protein complexes in Saccharomyces cerevisiae by mass spectrometry. Nature. 2002, 415: 180183. 10.1038/415180a.
 9.
Krogan N, Peng WT, Cagney G, Robinson MD, Haw R, Zhong G, Guo X, Zhang X, Canadien V, Richards DP, Beattie BK, et al: Highdefinition macromolecular composition of yeast RNAprocessing complexes. Mol Cell. 2004, 13: 225239. 10.1016/S10972765(04)000036.
 10.
Gavin A, Aloy P, Grandi P, Krause R, Boesche M, Marzioch M, Rau C, Jensen L, Bastuck S, Dumpelfeld B, et al: Proteome survey reveals modularity of the yeast cell machinery. Nature. 2006, 440: 631636. 10.1038/nature04532.
 11.
Krogan N, Cagney G, Yu H, Zhong G, Guo X, Ignatchenko A, Li J, Pu S, Datta N, Tikuisis A, et al: Global landscape of protein complexes in the yeast Saccharomyces cerevisiae. Nature. 2006, 440: 637643. 10.1038/nature04670.
 12.
Giot L, Bader J, Brouwer C, Chaudhuri A, Kuang B, Li Y, Hao Y, Ooi C, Godwin B, Vitols E, et al: A protein interaction map of Drosophila melanogaster. Science. 2003, 302: 17271736. 10.1126/science.1090289.
 13.
Li S, Armstrong C, Bertin N, Ge H, Milstein S, Boxem M, Vidalain P, Han J, Chesneau A, Hao T, et al: A map of the interactome network of the metazoan C. elegans. Science. 2004, 303: 540543. 10.1126/science.1091403.
 14.
Rual J, Venkatesan K, Hao T, HirozaneKishikawa T, Dricot A, Li N, Berriz G, Gibbons F, Dreze M, AyiviGuedehoussou N, et al: Towards a proteomescale map of the human proteinprotein interaction network. Nature. 2005, 437: 11731178. 10.1038/nature04209.
 15.
Stelzl U, Worm U, Lalowski M, Haenig C, Brembeck F, Goehler H, Stroedicke M, Zenkner M, Schoenherr A, Koeppen S, et al: A human proteinprotein interaction network: a resource for annotating the proteome. Cell. 2005, 122: 957968. 10.1016/j.cell.2005.08.029.
 16.
Hartwell LH, Hopfield J, Leibler S, Murray A: From molecular to modular cell biology. Nature. 1999, 4752. 10.1038/35011540. Suppl
 17.
Walhout A, Boulton S, Vidal M: Yeast twohybrid systems and protein interaction mapping projects for yeast and worm. Yeast. 2000, 17: 8894. 10.1002/10970061(20000630)17:2<88::AIDYEA20>3.0.CO;2Y.
 18.
Schwikowski B, Uetz P, Fields S: A network of proteinprotein interactions in yeast. Nat Biotechnol. 2000, 18: 12571261. 10.1038/82360.
 19.
Mrowka R, Patzak A, Herzel H: Is there a bias in proteome research?. Genome Res. 2001, 11: 19711973. 10.1101/gr.206701.
 20.
Tucker C, Gera J, Uetz P: Towards an understanding of complex protein networks. Trends Cell Biol. 2001, 11: 102106. 10.1016/S09628924(00)019024.
 21.
Hazbun TR, Fields S: Networking proteins in yeast. Proc Natl Acad Sci USA. 2001, 98: 42774278. 10.1073/pnas.091096398.
 22.
Deane C, Salwinski L, Xenarios I, Eisenberg D: Two methods for assessment of the reliability of high throughput observations. Mol Cell Proteomics. 2002, 1: 349356. 10.1074/mcp.M100037MCP200.
 23.
Edwards A, Kus B, Jansen R, Greenbaum D, Greenblat J, Gerstein M: Bridging structural biology and genomics: assessing protein interaction data with known complexes. Trends Genet. 2002, 18: 529536. 10.1016/S01689525(02)027634.
 24.
von Mering C, Krause R, Snel B, Cornell M, Oliver S, Fields S, Bork P: Comparative assessment of largescale data sets of proteinprotein interactions. Nature. 2002, 417: 399403. 10.1038/nature750.
 25.
Thomas A, Cannings R, Monk N, Cannings C: On the structure of proteinprotein interactions networks. Biochem Soc Trans. 2003, 31: 14911496.
 26.
Lappe M, Holm L: Unraveling protein interaction networks with nearoptimal efficiency. Nat Biotechnol. 2004, 22: 98103. 10.1038/nbt921.
 27.
Poyatos J, Hurst L: How biologically relevant are interaction based modules in protein networks. Genome Biol. 2004, 5: R9310.1186/gb2004511r93.
 28.
Vidalain P, Boxem M, Ge H, Li S, Vidal M: Increasing specificity in highthroughput yeast twohybrid experiments. Methods. 2004, 32: 363370. 10.1016/j.ymeth.2003.10.001.
 29.
Goll J, Uetz P: The elusive yeast interactome. Genome Biol. 2006, 7: 223
 30.
Gagneur J, David L, Steinmetz L: Capturing cellular machines by systematic screens of protein complexes. Trends Microbiol. 2006, 14: 336339. 10.1016/j.tim.2006.06.002.
 31.
Walhout A, Vidal M: A genetic strategy to eliminate selfactivator baits prior to highthroughput yeast twohybrid screens. Genome Res. 1999, 9: 11281134. 10.1101/gr.9.11.1128.
 32.
Aloy P, Russell RB: Potential artefacts in proteininteraction networks. FEBS Lett. 2002, 530: 253254. 10.1016/S00145793(02)034270.
 33.
Stanley RP: Enumerative Combinatorics I. 1997, New York, NY: Cambridge University Press
 34.
de Silva E, Thorne T, Ingram P, Agrafioti I, Swire J, Wiuf C, Stumpf MP: The effects of incomplete protein interaction data on structural and evolutionary inferences. BMC Biol. 2006, 4: 3910.1186/17417007439.
 35.
Falcon S, Gentleman R: Using GOstats to test gene lists for GO term association. Bioinformatics. 2006, 2: 257258.
 36.
Han J, Dupuy D, Bertin N, Cusick M, Vidal M: Effect of sampling on topology predictions of proteinprotein interaction networks. Nat Biotechnol. 2005, 23: 839844. 10.1038/nbt1116.
 37.
Stumpf MPH, Wiuf C: Sampling properties of random graphs: the degree distribution. Phys Rev E Stat Nonlin Soft Matter Phys. 2005, 72: 036118
 38.
Sharp PM, Li WH: The codon adaptation index: a measure of directional synonymous codon usage bias, and its potential applications. Nucleic Acid Res. 1987, 15: 12811295. 10.1093/nar/15.3.1281.
 39.
Domon B, Aebersold R: Mass spectrometry and protein analysis. Science. 2006, 312: 212217. 10.1126/science.1124619.
 40.
Aebersold R, Mann M: Mass spectrometrybased proteomics. Nature. 2003, 422: 198207. 10.1038/nature01511.
 41.
Kelsey J, Whittemore A, Evans A, Thompson W: Methods in observational epidemiology. Monographs in Epidemiology and Biostatistics. 1996, New York, NY: Oxford University Press
 42.
Bickel P, Doksum K: Mathematical Statistics: Basic Ideas and Selected Topics. 2001, New Jersey: Prentice Hall
 43.
Miller J, Lo R, BenHur A, Desmarais C, Stagljar I, Noble W, Fields S: Largescale identification of yeast integral membrane protein interactions. Proc Natl Acad Sci USA. 2005, 102: 1212312128. 10.1073/pnas.0505482102.
 44.
Kerrien S, AlamFaruque Y, Aranda B, Bancarz I, Bridge A, Derow C, Dimmer E, Feuermann M, Friedrichsen A, Huntley R, et al: IntAct: open source resource for molecular interaction data. Nucleic Acids Res. 2006, 35: D561D565. 10.1093/nar/gkl958.
 45.
Saccharomyces Genome Database. [http://www.yeastgenome.org]
Acknowledgements
We thank AnneClaude Gavin and Rob Russell for insightful discussions and Jack Greenblatt for making their primary data available. We would also like to thank Li Wang for careful reading of this manuscript. We acknowledge funding through the Human Frontiers Science Program Research Grant RGP0022/2005 to WH and RG.
Author information
Affiliations
Corresponding author
Additional information
Authors' contributions
TC, RG and WH conceived and designed the investigations described in this report. TC, DS and DS performed the computational and statistical analyses. TC, RG and WH wrote paper. All authors read and approved the final version of the manuscript.
Electronic supplementary material
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
Rights and permissions
About this article
Cite this article
Chiang, T., Scholtens, D., Sarkar, D. et al. Coverage and error models of proteinprotein interaction data by directed graph analysis. Genome Biol 8, R186 (2007). https://doi.org/10.1186/gb200789r186
Received:
Revised:
Accepted:
Published:
Keywords
 Additional Data File
 Protein Pair
 Stochastic Error
 Protein Interaction Data
 Bait Protein