An Introduction to Dimensionality Analysis
What exactly is a "dimension"? By dimension(s), we are referring to the construct (or constructs) being measured by an assessment instrument. For example, one might assume that a science assessment is measuring the construct called "knowledge of science." Another person looking at the same assessment might assert that it really measures multiple constructs, such as, physical science, earth science, biology, chemistry, etc. Thus, one sees, even from this simple example, that dimensionality can be a controversial topic.
So, how do we resolve this controversy? In particular, how might we determine whether the science assessment is unidimensional (measuring one dimension, "knowledge of science") or multidimensional (more than one dimension, "physical science, . . .")? In general, we want to know:
(1) Whether the assessment is unidimensional or multidimensional,
(2) How many dimensions an assessment is measuring (if not unidimensional) and how distinct (how different) the dimensions are from each other,
and, the Holy Grail of dimensionality analysis,
(3) The degree to which each assessment item measures each of the dimensions.
Is the assessment unidimensional?
The most straightforward way that a psychometrician would answer this question would be to simply apply an appropriate hypothesis testing statistic with the null hypothesis (or starting assumption) being unidimensionality. Basically, this hypothesis test looks at how the items covary with each other, knowing how these covariances should look under the assumption of unidimensionality. If the item covariances significantly differ from this pattern, then the null hypothesis is rejected and we can safely assume that the assessment is not unidimensional, i.e., the assessment is multidimensional.
Assuming the null hypothesis of unidimensionality is rejected, let us now discuss the further questions that arise (as we have stated above).
How many dimensions are there and how distinct are they?
Now that we know the assessment is multidimensional, it is possible to statistically investigate the pattern of item covariances to determine how many dimensions are needed to account for the observed pattern. Once the dimensions are determined, the degree of distinctiveness of the dimensions is typically measured using one (or both) of the following two ways: (a) the strength of the correlations between the dimensions (for example, the correlation between knowledge of physical science and knowledge of biology as measured by the appropriate subscores) and (b) the difference between the covariances expected for a unidimensional assessment and the observed covariances. The lower the correlation between two dimensions, the more distinct the dimensions are from each other. The bigger the difference between the unidimensional covariances and the observed covariances, the more distinct the dimensions.
To what degree does each assessment item measure each of the dimensions?
As we mentioned above, this is the Holy Grail of dimensionality analysis; and, like the real Holy Grail, it is frequently not completely attainable. Statistical methods exist that either (a) tell you which dimension is best measured by each item, or (b) tell you the degree that each item measures each dimension (isn't that exactly what you want?).
The problem with method (a) is obvious: it gives you no information about how each item relates to the other dimensions (besides the one it best measures). The problem with method (b) is that in order to get the Holy Grail, it has to make a lot of assumptions—giving only a rough approximation of what is really going on. The best that you can do is perform both types of analyses and simply accept that the truth is probably somewhere in between. Generally, the combined results will provide you with a good understanding of the dimensionality structure of the items.
This tool provides a brief overview of dimensionality analysis relating to assessments. As one can imagine, in practice this analysis is complex and there is rarely one "right" answer.
