A Preliminary Evaluation and recommendations for use
of the Mental Health Indicator in the NHIS
This appendix is based on the May 1999 analyses completed by
Thomas M.Achenbach, Ph.D., of the University of Vermont.
Introduction
The NHIS mental health indicator consists of items from the Child Behavior
Checklist (CBCL) that were identified by Dr. Thomas Achenbach as providing the
best discrimination between demographically similar children referred for mental
health services versus nonreferred. To take account of gender and age differences
between in the discriminative power of particular items, they were selected separately
for each gender at ages 2-3, 4-11, and 12-17. From the original ten items identified
in Dr. Achenbach’s 1995 analyses, the 1997 NHIS elected to include only 4 items, with
the specific items differing somewhat for children of each gender in the age ranges
2-3, 4-11, and 12-17 years. It is essential to note that such a small set of items
cannot be used to evaluate individual children for clinical or other purposes. Even
for use as mental health indicators in large surveys such as the NHIS, very small
sets of items can serve only as approximate indicators of needs for mental health
services. Multiple items tapping each of several specific areas of functioning would
be needed to identify specific disorders, such as Attention Deficit Hyperactivity
Disorder (ADHD), Depression, Conduct Disorder, and Somatization Disorder.
It should also be noted that different cutpoints on the distributions of item scores
may be needed for different purposes. For example, a very low cutpoint may be useful
if the goal is to identify every possible case for which mental health services might
be considered. However, very low cutpoints result in relatively high false positive
rates, i.e., the inclusion of substantial numbers of healthy individuals among those
identified as potentially needing services. Conversely, higher cutpoints may yield
greater overall accuracy in classifying potential cases versus noncases, but at the
cost of missing more cases.
Data Analyses
Since the preliminary data tape could not be released due to confidentiality issues,
Dr. Achenbach specified and reviewed data analyses which were done at NCHS. These
included tabulations of specific responses to each behavioral/emotional problem item;
tabulations of relations between total problem scores and classification of children
as deviant versus nondeviant on the basis of external criteria (e.g., parents ever
being told by health professionals that their child had ADHD, mental retardation,
other developmental delay, autism, down syndrome, or a learning disability; parents
having talked to mental health professionals about their child in the preceding 12
months; or parents needing mental health services for their child but unable to afford
it); and Relative Operating Characteristic (ROC) analyses of cutpoints on the total
problem scores. Because each behavioral/emotional problem item was scored 0 (not true
of the child), 1 (somewhat or sometimes true), or 2 (very true or often true), total
scores across the 4 items for each gender/age group could range from 0 to 8.
Dr. Achenbach examined the results and recommended changes and additions to the
analyses.
Based on the analyses to date, Dr. Achenbach makes the following recommendations:
1. Boys and Girls Ages 2-3. Total scores on the 8 problem items are useful for
quantitative analyses in relation to other variables. However, categorical
mental health indicators should not be derived from specific cutpoints on the
total scores for the 4 behavioral/emotional problem items on the basis of 1997
NHIS data for ages 2-3 for the following reasons:
(A) The total number of children classified as deviant according to external
criteria (e.g., parents being told their child had ADHD; talking to mental
health professionals about their child) was too small to provide a sound
basis for establishing cutpoints (N = 44 boys, 27 girls).
(B) Many disorders relevant to defining criterion groups (e.g., ADHD) are not
identified as early as age 2-3.
(C) The rates of referral for mental health services and other possible
indicators of deviance are much lower at ages 2-3 than later.
2. Boys Ages 4-11. Detailed comments for this group will be used to illustrate
considerations relevant to the selection and application of cutpoints for all
groups.
The number of boys classified as deviant according to external criteria (N = 252
out of the sample of 1545) was sufficient for testing cutpoints on the distribution
of the total scores for the 4 behavioral/emotional problem items. However, it
should be remembered that 4 items cannot be an adequate basis for assessing
individual children for clinical and other purposes.
A much larger pool of items, including multiple items for tapping each one of a
variety of domains, would be necessary for assessment of individual children.
For purposes of analyzing relations between categorical cutpoints on the distribution
of total scores and other variables in large samples such as the NHIS, we need to
consider the different consequences of different cutpoints. If the goal is merely
to maximize the probability of identifying every possible case (i.e., to minimize
false negatives), an optimal strategy would be simply to classify all subjects as
positive. This would mean that all scores on the problem scale (even scores of 0)
would be used to indicate "caseness." Sensitivity would be 100%, but specificity
would be 0. In the data for the age 4-11 boys, the false negative rate would be 0,
but the false positive rate would be 83.7%.
Because a false positive rate of 83.7% is probably too high to be practical, we
need to reduce this rate by moving up the distribution of total problem scores.
If we classify all subjects who obtained scores >1 as cases, we can still achieve
an excellent false negative rate of only 5% (with sensitivity = 82.5%) at the
expense of a false positive rate of 69% (specificity = 64.2%). The overall
accuracy of classification would be 67.2%.
If our goal is to maximize the identification of possible cases without classifying
every subject as a case, then scores >1 achieve this goal for age 4-11 boys.
However, if we wish to maximize the accuracy of classification with respect to both
the detection of cases and noncases, we need to user higher cutpoints to reduce the
percentage of nondeviant boys who are incorrectly classified as cases. Because 83.7%
of the age 4-11 boys were classified as nondeviant according to the external criteria,
reducing the percentage of misclassifications for this group increases the overall
accuracy of classification more than if we reduce the percentage of misclassifications
for the deviant group. By moving the cutpoint to scores >4, we can raise the overall
accuracy of classification to 83.5%, with sensitivity = 26.6%, specificity = 94.6%,
false positives = 51.1%, and false negatives = 11.0%.
As a compromise between the goals of maximizing overall accuracy of classification
and maximizing sensitivity in detecting possible cases, a cutpoint >2 yields 78.8%
overall accuracy, 63.1% sensitivity, 81.8% specificity, 59.6% false positives, and
8.1% false negatives. For most screening purposes, classification of age 4-11 boys
as possible cases if they score >2 would be reasonably efficient, although a cutpoint
>1 would increase detection of possible cases at the expense of increasing false
positives.
Cutpoints of >1 and >2 may seem remarkably low, even on distributions of scores that
can range only from 0 to 8. However, the 4 items used in each gender/age group were
selected by multiple discriminant analyses as the best combination from a pool of 118
items for discriminating between demographically matched children who were referred
versus children who were not referred for mental health services. Furthermore, items
were selected for their discriminative power within each specific gender/age group,
such as age 4-11 boys. A score of even 1 on any of the 4 selected items is thus apt
to be a powerful indicator of certain kinds of needs for professional help.
It is important to note that there is great diversity in needs for mental health
services. Consequently, no single item and no very small set of items can accurately
reflect all the different needs. Furthermore, the external criteria for deviance
used in the NHIS were quite coarse. They not only included talking to a mental health
professional about the child, but also being told at any time in the child’s life that
the child had a disorder such as ADHD, mental retardation, or learning disability.
Even if a parent had been told this by a professional at some time in the child’s
life, this does not mean that the child currently displays deviant levels of
behavioral/emotional problems. The NHIS external criteria may therefore include
children who are not currently deviant or in need of services for behavioral/emotional
problems.
On the other hand, the distribution of problem scores for the children classified as
nondeviant according to the external criteria suggest that small groups of quite
deviant children were missed by the external criteria. The evidence for this is as
follows: Among children who did not meet the external criteria for deviance, the
number of children who obtained a score of 8 was considerably larger than the number
who obtained scores of 5, 6, or 7. This tendency toward bi-modality, which was
evident in the distribution of total problem scores for all 6 gender/age groups,
suggests that exceptionally high scores on the problem items identify children who
are quite deviant even though the NHIS external criteria for deviance failed to
identify them.
No external criteria can be perfect, especially when the mental health questions
are limited to a small portion of the lengthy NHIS interview. However, the inclusion
of possibly quite deviant children in the groups classified a priori as nondeviant
by the external criteria would weaken the association between cutpoints on the problem
scores and classification of children as deviant according to the NHIS external
criteria. It is possible that other NHIS data can be used to test the hypothesis
that the children receiving high problem scores but not identified by the NHIS
external criteria for deviance do need mental health services. If this hypothesis
is supported, exclusion of these children from the group classified a priori as
nondeviant could improve the accuracy with which cutpoints classify NHIS children
as deviant versus nondeviant.
3. Summary for Boys and Girls Ages 4-11 and 12-17. The foregoing comments re:
tradeoffs for lower vs. higher cutpoints and re: the coarseness of the external
criteria pertain to all gender/age groups. The ROC data for both genders at
ages 4-11 and 12-17 are summarized in Table 1.
Table 1. Summary of ROC Analyses *Revised
| Group | Total Na | Deviant N | Cutpoint | Sensitivity |
Specificity | False
Positives | False
Negatives | % Correct |
| Boys 4-11 | 1519 | 243 | >1 | 82.3 | 64.4 | 69.4 |
5.0 | 67.3 |
|
|
|
>2b | 63.0 | 82.1 | 59.9 | 7.9 | 79.0 |
|
|
|
>3 | 42.4 | 90.4 | 54.4 | 10.8 | 82.7 |
|
|
|
>4 | 26.7 | 94.7 | 50.8 | 12.8 | 83.9 |
| Girls 4-11 | 1466 | 138c | >1 | 79.0 | 66.3 | 80.4 |
3.2 | 67.5 |
|
|
|
>2b | 58.0 | 81.6 | 75.3 | 5.1 | 79.4 |
|
|
|
>3 | 37.0 | 90.4 | 71.3 | 6.8 | 85.4 |
| Boys 12-17 | 2441 | 522 | >1 | 85.8 | 63.4 | 61.0 |
5.7 | 68.2 |
|
|
|
>2b | 67.4 | 79.3 | 53.0 | 10.0 | 76.8 |
|
|
|
>3 | 49.0 | 89.0 | 45.3 | 13.5 | 80.4 |
| Girls 12-17 | 2277 | 308 | >1 | 81.5 | 57.5 | 76.9 |
4.8 | 60.8 |
|
|
|
>2b | 58.1 | 75.7 | 72.8 | 8.0 | 73.3 |
|
|
|
>3 | 39.0 | 87.7 | 66.9 | 9.8 | 81.1 |
a Each age/gender group only contains records that contain yes/no responses to
the appropriate mental health indicator questions and to the criterion group
questions.
b A cutpoint of >2 appears to offer a reasonable compromise among the various
measures of performance for all 4 gender/age groups.
c For girls ages 4-11, the N=138 classified as deviant according to external
criteria is somewhat small to provide a basis for cutpoints. However, the
performance of a cutpoint at >2 is similar to that for the other gender/age
groups.
In addition to the ROC data shown in Table 1, ROC analyses were also performed in
which children receiving special education were added to the group that met the
other external criteria for deviance. The addition of special education caused the
biggest increase in cases among 4-11-year-old boys, where the addition of 32 who
were receiving special education increased the number of cases from 252 to 284,
and the percentage classified as deviant from 16.3% to 18.4%. However, the addition
of recipients of special education to the criterion group of cases did not have
large or consistent effects on the ROC results shown in Table 1. Although it may
be worth making finer-grained analyses of specific kinds of special education and
other specific kinds of external criteria for deviance, the results shown in Table 1
are probably fairly robust with respect to broad classifications of children who
provide an appropriate criterion group defined in terms of previously identified
mental health needs.
Summary
On the basis of the data provided to Dr. Achenbach from the 1997 NHIS, he makes
the following recommendations regarding use of categorical cutpoints on scores of
0 to 8 obtained from parents’ 0-1-2 ratings of the 4 behavioral/emotional problem
items:
- For reasons specified earlier, no categorical cutpoints should be used for
ages 2-3.
- For ages 4-17, problem scores >2 on the items specific to each gender/age
group can be used to indicate possible needs for mental health services.
The data shown in Table 1 can be used to select higher or lower cutpoints
according to users’ aims.
- It must be remembered that 4 items rated by parents cannot provide an
adequate basis for assessment of individual children for clinical or other
purposes. Much more comprehensive and individualized assessment is needed
to draw conclusions about the functioning and needs of individual children.
Note: Further evaluation of the Mental Health Indicator is being planned.
