Feature-level exploration of a published Affymetrix GeneChip control dataset
© BioMed Central Ltd 2005
Published: 1 September 2006
A comment on Preferred analysis methods for Affymetrix GeneChips revealed by a wholly defined control dataset by SE Choe, M Boutros, AM Michelson, GM Church and MS Halfon. Genome Biology 2005, 6:R16.
In a recent Genome Biology article, Choe et al.  describe a spike-in experiment that they use to compare expression measures for Affymetrix GeneChip technology. In this work, two sets of triplicates were created to represent control (C) and experimental (S) samples. We describe here some properties of the Choe et al.  control dataset one should consider before using it to assess GeneChip expression measures. In  and  we describe a benchmark for such measures based on experiments developed by Affymetrix and GeneLogic. These datasets are described in detail in . A web-based implementation of the benchmark, is available at . The experiment described in  is a worthy contribution to the field as it permits assessments with data that is likely to better emulate the nonspecific binding (NSB) and cross-hybridization seen in typical experiments. However, there are various inconsistencies between the conclusions reached by  and  that we do not believe are due to NSB and cross-hybridization effects. In this Correspondence we describe certain characteristics of the feature-level data produced by  which we believe explain these inconsistencies. These can be divided into characteristics induced by the experimental design and an artifact.
There are three characteristics of the experimental design described by  that one should consider before using it for assessments like those carried out by Affycomp. We enumerate them below and explain how they may lead to unfair assessments. Other considerations are described by Dabney and Storey .
Second, a large percentage of the genes (about 10%) are spiked-in to be differentially expressed and all of these are expected to be upregulated. This design makes this spike-in data very different from that produced by many experiments where at least one of the following assumptions is expected to hold: a small percentage of genes are differentially expressed, and there is a balance between up- and downregulation. Many preprocessing algorithms (for example, loess normalization, variancestabilizing normalization (VSN), rank-invariant) implement normalization routines motivated by one or both of these assumptions; thus we should not expect many of the existing expression measure methodologies to perform well with the Choe et al.  data.
Third, a careful look at Table 1 in  shows that nominal concentrations and fold-change sizes are confounded. This problem will slightly cloud the distinction between ability to detect small fold changes from the ability to detect differential expression when concentration is low. Why this distinction isimportant is shown in . However, Figure ADF5-1 in Additional data file 1 of Choe et al.  demonstrates that this difference in nominal concentrations does not appear to translated into observed intensities. This could, however, be an indication of saturation, which is a common problem when high intensities are observed (see the first point of this argument above). One case of the confounding is seen: genes with nominal fold-changes larger than 1 result in intensities that, on average, are about three times larger than genes with nominal fold-changes of 1.
This problem implies that, unless an ad hoc correction is applied, what Choe et al.  define as false positive might in fact be true positives. Figure 2 shows that this problem persists even after quantile normalization . In Choe et al.  a normalization scheme based on knowledge of which genes have fold-changes of 1 is used to correct this problem. However, preprocessing algorithms are not designed to work with data that has been manipulated in this way, which makes this dataset particularly difficult to use in assessment tools such as Affycomp. Furthermore, Figure 1c,d of this Correspondence shows that the data produced by  is quite different from data from typical experiments which most preprocessing algorithms were developed.
Currently, experiments where the normalization assumptions do not hold seem to be a small minority. However, our experience is that they are becoming more common. For this type of experiment we will need new preprocessing algorithms, and the Choe et al.  data may be useful for the development of these new methods.
Sung E Choe, Michael Boutros, Alan M Michelson, George M Church and Marc S Halfon respond:
Irizarry et al. raise a number of interesting points in their Correspondence that highlight the continued need for carefully designed control microarray experiments. They posit that "the spikein concentrations are unrealistically high" in our experimental design. Although we have estimated that the average per-gene concentration is similar to that in a typical experiment , we do not know individual RNA concentrations and so cannot verify or deny this assertion. Since the majority of probesets in our dataset correspond to non-spiked-in genes, and therefore have a signal range consistent with absent genes, we think it seems reasonable that the spiked-in genes have higher signal than the rest of the chip. Regardless of this, in Additional Data File 5 of , we repeated the receiver-operator characteristics (ROC) analysis using as the "known differentially expressed" probe sets only the subset with low signal levels. The results we obtained for gcrma (robust mutli-array average using sequence information)  were very similar to the conclusions in  and ; in addition, the performance of MAS5  was similar between  and . The inconsistencies between the different studies may therefore be less extreme than they seem. In particular, we think that a large source of the disagreement between  and  is simply the different choice of metric for the ROC curves.
There is no question that our analysis of low-signal-intensity probesets as well as the specific selection of non-differentially expressed genes to use for normalization purposes required prior knowledge of the composition of the dataset. This, of course, is one of the great strengths of a wholly-defined dataset such as that from  - we can choose idealized conditions for assessing the performance of different aspects of the analysis. Unfortunately, as Irizarry et al. correctly point out, it also makes it difficult to use for certain other types of assessment, such as those provided by Affycomp .
A more critical consideration lies in the point raised by Irizarry et al. that our dataset violates two main assumptions of most normalization methods: that a small fraction of genes should be differentially expressed; and that there should be roughly equal numbers of up and down regulated genes. It is important to note that these two assumptions are just that - assumptions - and ones that are extremely difficult to prove or disprove in any given microarray experiment. Thus there is an inherent circularity in the design of analysis algorithms that explicitly rely on these assumptions: they perform well on data assumed to have the properties based on which they are designed to perform well. This is an issue all too often overlooked in the microarray field. The violation of these two core assumptions seen in our dataset may be more common than generally appreciated; certainly we can conceive of many situations in which they are unlikely to hold (for example, when comparing different tissue types, in certain developmental time courses, or in cases of immune challenge). Developing assumption-free normalization methods, and diagnostics to assess the efficacy of the normalization used for a given dataset (see  for an example), should thus be important research priorities.
This discussion underscores the need for more control datasets that specifically address matters of RNA concentration, fractions of differentially expressed genes, direction of changes in gene regulation, and the like. Only then can we truly devise and assess the performance of analysis methods for the large variety of possible scenarios encountered in the course of conducting microarray experiments focused on real biological problems.
Correspondence should be sent to Marc S Halfon: Department of Biochemistry and Center of Excellence in Bioinformatics and the Life Sciences, State University of New York at Buffalo, Buffalo, NY 14214, USA. Email: firstname.lastname@example.org
Additional data files
Additional data file 1 contains MA plots for 100 randomly chosen pairs of arrays from the Gene Expression Omnibus (GEO) is available.
The work of R.A.I. is partially funded by the National Institutes of Health Specialized Centers of Clinically Oriented Research (SCCOR) translational research funds (212- 2492 and 212-2496).
- Choe SE, Boutros M, Michelson AM, Church GM, Halfon MS: Preferred analysis methods for AffymetrixGeneChips revealed by a wholly defined control dataset. Genome Biol. 2005, 6: R16-10.1186/gb-2005-6-2-r16.PubMedPubMed CentralView Article
- Cope L, Irizarry R, Jaffee H, Wu Z, Speed T: A benchmark for Affymetrix GeneChip expression measures. Bioinformatics. 2004, 20: 323-331. 10.1093/bioinformatics/btg410.PubMedView Article
- Irizarry R, Wu Z, Jaffe H: Comparisonof Affymetrix GeneChip expressionmeasures. Bioinformatics. 2006, 22: 789-794. 10.1093/bioinformatics/btk046.PubMedView Article
- Affycomp II: A benchmark for Affymetrix GeneChip expression measures. [http://affycomp.biostat.jhsph.edu]
- Dabney A, Storey J: A reanalysis of a published Affymetrix GeneChip control dataset. Genome Biol. 2006, 7: 401-10.1186/gb-2006-7-3-401.PubMedPubMed CentralView Article
- Saran NG, Pletcher MT, Natale JE, Cheng Y, Reeves RH: Global disruption of the cerebellar transcriptome in a Down syndrome mouse model. Hum Mol Genet. 2003, 12: 2013-2019. 10.1093/hmg/ddg217.PubMedView Article
- One hundred MA plots from GEO. [http://www.biostat.jhsph.edu/~ririzarr/papers/hundredMAs.pdf]
- Bolstad B, Irizarry R, Åstrand M, Speed T: A comparison of normalization methods for high density oligonucleotide array data based on variance and bias. Bioinformatics. 2003, 19: 185-193. 10.1093/bioinformatics/19.2.185.PubMedView Article
- Wu Z, Irizarry R, Gentleman RC, Martinez-Murillo F, Spencer F: A model-based background adjustment for oligonucleotide expression arrays. Journal of the American Statistical Association. 2004, 99: 909-917. 10.1198/016214504000000683.View Article
- Qin LX, Beyer RP, Hudson FNX, Linford NJ, Morris DE, Kerr KF: Evaluation of methods for oligonucleotide array data via quantitative real-time PCR. BMC Bioinformatics. 2006, 7: 23-10.1186/1471-2105-7-23.PubMedPubMed CentralView Article
- GeneChip Expression Analysis: dataanalysis fundamentals. [http://www.affymetrix.com/support/downloads/manuals/data_analysis_fundamentals_manual.pdf]
- Putative null distributionscorresponding to tests of differentialexpression in the Golden Spike dataset are intensity dependent. Technical report 06-01. Buffalo, N.Y.: Department of Biostatistics, State University. [http://sphhp.buffalo.edu/biostat/research/techreports/UB_Biostatistics_TR0601.pdf]