View Full Version : Are EF Tasks Measuring ADD/ADHD?


ADDarkD@y
08-02-13, 06:17 PM
Here's the deal ...

Neuropsych tests DO NOT work to "measure" Executive Dysfunction. The obtained results are for what's referred to in the literature as "cold" EF....

there ARE several EF tests which measure "hot" EF. That refers to "real life, real world" ADHD related impairments. Those are confined to a small number of Rating Scales....


Hi Tavish, I took the time after reading your post to review research on cold and hot EF, concepts which I previously did not know about. Below is a page on EF from the University of Northern Colorado:

http://mast.unco.edu/programs/fsi/summer/2011/projects/winchester/intro.html

Hot and Cold in fact refer to emotion-loaded and non-emotion-loaded cognitive functions centralized in the prefrontal cortex. The difference between them is in their involvement in emotion regulation and sensitivity to arousal. These concepts have nothing to do with the validity of the measured trait in theory or the "real world". Both types of EF are measured by specific tasks (e.g. number/letter sequence, Iowa gambling, etc.). I don't get the impression that number of "rating scales" has anything to do with the assessment of EFs of either type.

mctavish23
08-02-13, 06:29 PM
I BEG TO DIFFER.

One reference source doesn't make for statistical significance.

My reference source is from the ADHD REPORT.

I'm home on medical leave, recovering from a stroke, so my copy of the REPORT is at the

office. Because I've done this for a living since 1977, the last 29+ years at my current

practice, I'm comfortable that my statement IS evidence based.

For the record, I was actually giving you some support.

tc

mctavish23

(Robert)

ADDarkD@y
08-03-13, 05:54 PM
I BEG TO DIFFER....

I'm home on medical leave, recovering from a stroke...

...Because I've done this for a living since 1977, the last 29+ years at my current practice, I'm comfortable that my statement IS evidence based.
...

For the record, I was actually giving you some support.
(Robert)


Tavish, I respect you and your knowledge, but I must disagree where I believe I am correct even if I am no expert myself. This was not one source, in fact I read two that said this. I can probably find 100 if needed. But it doesn't matter because we have already stated our positions and I will leave it at that.

Hope you get better soon.

mctavish23
08-03-13, 07:44 PM
I'll be happy to get the reference source and show you what I'm talking about. Keep in mind I'm NOT refuting what you stated. I'm merely stating an evidence based fact, that I'll be glad to explain.

Just recently, I made several copies of the article in question, which is certainly not the first one I've read to that effect, and gave them to our Clinical Director and the two Internship Supervisors. While things are much better than they used to be where I work, I'm aware that there are still a great many clinicians, both here and around the country (USA), who still use test scores alone for diagnositic purposes. My goal is to try and get everyone here "on the same page."

One thing you'll learn about me, is that I have no problem admitting when I'm wrong; as that's how we all learn. By the same token, please note that if I say something is evidence based, it means I can prove it unequivicably.

Hope that helps.

tc

mctavish23

(Robert)

mctavish23
08-05-13, 11:28 PM
Now Where Were We ? Oh Yes, Responding To A New (Separate) Thread, Designed To Answer A Post Of Mine, From An Entirely Different Thread; By The Same OP.

When I posted on this in the OP's other thread, it was to lend a measure of support for the OP's remarks regarding the reported use of Executive Function (EF) testing in his diagnostic process. (Please Refer To That Response - "So I Saw The Psych Finally," #12,
dated 08/01/13; INATTENTIVE ADD).

That led to THIS thread ... "Are EF Tests Measuring ADD/ADHD?" In the process, I brought up the concept of "Hot" and "Cold" EF; something the OP acknowledged having never heard of until then. In turn, the OP respectfully disagreed with the operational definitions used.

This is really NOT that complicated, as I was specifically referring to TESTING; as applies to the operational definitions. Specifically, how "Hot" EF referred to "real world" examples of impairments in major life activities. While "Cold" EF, was defined as a strictly "laboratory" derived measure of some type of cognitive construct; derived from a standardized psychometric instrument (e.g., neuropsychological tests and test batteries, IQ tests, as well as computerized Continuous Performance Tests (CPT's).

In response, the OP (correctly) pointed out how "Hot" EF refers to Emotionally Motivated Decisions, while "Cold" EF refers to Purely Cognitive EF. I DO NOT disagree with that. The point I'm trying to make is how ADHD has been (incorrectly) evaluated over the years; chiefly through the use of standardized tests, such as the groups listed above.

For the record, Don't You Think I Already Knew Those Definitions? Where we ultimately "disconnected" was in the APPLICATION. The point I was trying to make, and what led to this post, stemmed from the APPLICATION of those operational definitions to TESTING (SOLELY FOR DIAGNOSTIC PURPOSES).

THAT is what I meant when I mentioned "one reference source." However, that WAS INCORRECT AND CONFUSING on my part. What I was referring to is more accurately described as how those definitions are APPLIED DIFFERENTLY in DIFFERENT SETTINGS, and for DIFFERENT PURPOSES (e.g., Experimental (Research) versus Clinical (Applied). Hopefully, this will clarify matters.

I mentioned earlier that my main reference source, a recent article in the ADHD REPORT, was at the office; which it still is. However, my binder with several years worth of ADHD REPORT's, dating from February 2006 - February 2013, is at home. Therefore, the
articles relevant to this particular point, specifically, from August 2011, Number 4; August 2012, Number 4; December 2012, Number 6; and February 2013; Number 1, which only cover the first portion of the binder, which is all I looked at, will serve as the references for this response.

We're talking about whether those constructs can be accurately measured by standardized tests, as well as how the operational definitions MEAN SOMETHING SLIGHTLY DIFFERENT IN THIS CONTEXT; which they do. In an earlier post, I'd mentioned how some operational definitions by the same name (i.e., Working Memory), mean one thing in one context (e.g., Neuropsychology), and something entirely different in others (e.g., Clinical, as well as Tests and Measurements). This is a PERFECT EXAMPLE of what I'm talking about.

REFERENCES :
Here are some of the excerpts which I felt were relevant to my effort to try and "get us on the same page," instead of disagreeing on something we're really not that far apart on :

(ADHD REPORT, Number 4, August 2011)

IS EXECUTIVE FUNCTION DEFICIENT IN ADHD? IT DEPENDS ON YOUR DEFINITIONS AND YOUR MEASURES - Russell Barkley, Ph.D.

(Very) briefly stated, Russ points how "the severity of ADHD symptoms may only be weakly correlated, if at all, with deficits on EF tests." Please refer to the article itself for the individual references listed for each qoute.

He goes on to state that, based on similar test results, some researchers (incorrectely) concluded that ADHD is not a disorder of EF.

He further states, "Underlying this conclusion, however, is a very important premise - that EF tests are the gold standard or most valid indicator of the presence of deficits in EF."

Russ then lists a few of the many evidence based difficulties with the use of tests for the sole purpose of defining EF; in the diagnosis of ADHD.

LACK OF A CONSENSUS DEFINITION OF EF - There are currently 33 Operational Definitions.

He and other researchers have narrowed those down to the most important four as :
1) Inhibition; 2) Working Memory; 3) (Cognitive) Shifting; and 4) Planning (which includes Problem Solving).

THE INSTALLATION OF A "CENTRAL EXECUTIVE" OR HUMUNCULUS IN EF - What he means is that there's a real "disconnect" on who or what is actually "running the show" in directing these "mental mechanisms," so often referred to in defining EF. He even goes so far as to invoke the idea of "The Great and Powerful Oz" in our brain, or the brain having it's very own "Mini-Me" to perform those higher order cognitive functions.

MISSING THE IMPORTANCE OF "THE SELF" IN EF - He believes that the conscious self IS the Central Executive.

(Fear Not, I'm NOT Really Off Task Here).

He goes on to list several other problems, including the OVERSIGHT OF THE IMPORTANCE OF EMOTION AND MOTIVATION IN EF, as well as DIFFICULTIES IN THE ASSESSMENT OF EF BY TESTS.

He further states how standardized (neuropsych) tests of EF were originally developed to assess psychological functions, such as Abstract Thinking in patients with Pre-Frontal Cortex (PFC) damage.

I'm going to sum this (complicated subject matter) up for now, as I'm tired and don't want to overdue it, by stating that "This sole focus on tests of cold cognition as the gold standard for measuring EF, has proven to be a major mistake, I believe".

That's it for tonight, as I have to attend a Stroke Seminar tomorrow. The Bottom Line here is that the OP and I are NOT that far apart here.

tc

mctavish23

(Robert)

Amtram
08-06-13, 11:33 AM
Nicely explained. Experimental testing and evaluation are important tools, but they are not the be-all and end-all, especially for complex conditions. Even clinical testing does not provide all the answers, as we've seen over and over in things as relatively straightforward as medication and other therapeutic trials.

Tests are simply a step in the process of overall evaluation. They have been developed under controlled conditions with a segment of specific populations to determine the most likely correlations/responses/outcomes/etc. Once anything moves from the controlled environment of a well-designed scientific study, numerous variables are introduced that weren't included and that can strongly affect and even reverse the study findings.

Taken out into the real world, some of these "Cold" functions can be overpowered by "Hot" functions, rendering a person's high scores on cognitive EF testing completely inconsequential.

Dizfriz
08-06-13, 01:01 PM
In his book Executive Functions (2012) Barkley strongly argues the idea that the current tests on executive functions are not really measuring what they are proposed to measure.

Don't have time to go into this very deep right now but thought I might mention it.

Dizfriz

Blanched Dubois
08-06-13, 05:47 PM
http://www.youtube.com/watch?v=xKBttQmhDBw

mctavish23
08-06-13, 07:19 PM
Amtram,

Thank you for the feedback. Ever since the stroke, I've wondered if my ability to explain things so they made sense, had deteriorated. I know I went into "overkill" in explaining my rationale and the corresponding methodology, so that I only touched on one of the specific reasons why "cold" or "cool" EF tests were inappropriate for measuring (let alone diagnosing) ADHD.

Blanche,

I still have that album.


tc

Robert

Blanched Dubois
08-06-13, 07:24 PM
{{{{{<3}}}}}

i know, McTavish

janiew
08-06-13, 07:52 PM
Robert, you are past the point where you need to school anyone. Please don't let anyone stress you out - unless you like it. Lol. I always like a good debate.

I think the true measure EF is subjective and not necessarily predictive throughout the lifespan - too many factors play into it. Especially when looking at otherwise successful adults. And there are plenty of us out there. I think that's where we need more study.

Personally, I think I'm on the autism spectrum in addition to whatever. Doesn't matter other than people like me need to develop coping skills for life and communication. It can be a huge learning curve. Coping skills are key because our internal experience is so different than the norm. Not wrong, just different.

Interested to hear what you think.

Peace.

daveddd
08-06-13, 07:56 PM
so hot EF is our executive function failing because our emotions took the wheel??

that makes perfect sense if so

i cant see a test testing those


unless maybe there is a marshmallow involved

Amtram
08-06-13, 08:09 PM
You can pretty much apply this to a whole slew of conditions, including ADHD. The behaviors that are used to inform a diagnosis are not necessarily reflective of how they affect the individual expressing them.

Just for an example, we can look at this young girl and clearly see from her behaviors that she is autistic. All the tests we have at our disposal indicate that.

However, none of the tests can tell us how she is affected from her point of view - and those tests fail massively when they indicate that her autism has made her intellectually impaired:

http://www.youtube.com/watch?v=vNZVV4Ciccg

Having tests to inform diagnoses or even just to better understand a condition is a good thing. It just isn't the be-all and end-all of diagnosis.

You're doing great as usual, Robert!

janiew
08-06-13, 08:16 PM
Oh hell my EF can go out then window when I feel my emotions too much. I try to keep a cap on it to deal with societal expectations and my own goals.

You know?

It's anecdotal of course, but probably could be applied to many of us.

I.E. we learn to deal in response to adverse consequences and positive ones. Or we don't. Some of us figure out how to adapt/cope survive on some level. The extent to which we survive and thrive is hard to quantify compared to the population because society is so is rife with other forms of mental illness and things many of us find hypocritcal or shallow. The statistics as they are now assume lack of educational intervention/modification and lack of therapy to learn how to deal because all we've tried to do so far is medicate. And I am happy for medication, personally.

It might hit a little too close to home for a lot of parents. I will shut up on that and my opinion is based on my experience.

In my opinion, we need more genetic testing to identify, target and treat ADHD and other neurological disorders. Treatment will not always consist of meds in the future but will need to be targeted to the individual.

Later and peace out.

mctavish23
08-06-13, 11:34 PM
janiew,

Thank you for the feedback. I appreciate it.

I've been around a long time, and have seen all kinds of people come and go in here.

There's a real difference between a contrarian / " devil's advocate" position, and a

passive - aggressive / "one upmanship," hidden agenda.

What helps differentiate between those scenarios, are sincere responses acknowledging

the validity of what was presented.

tc

Robert

mctavish23
08-08-13, 08:57 PM
Received the Latest ADHD REPORT Today (08/15/13). It contained yet another summary

finding of a soon to be published study, related to the thread question posed by the OP :

Neuropsychological Tasks Do Not Predict ADHD-Related Emotional Regulation Deficits -

(Briefly stated) "Problems with emotional self-regulation in ADHD could not be explained by neuropsychological deficits as measured by "cold" cognitive tasks, and further research is needed to characterize the factors underlying these deficits."

Surman, C.B., et.al. (2013). Neuropsychological deficits are not predictive of deficient emotional self-regulation in adilts with ADHD. Journal of Attention Disorders. [Epub ahead of print].

tc

mctavish23

(Robert)

daveddd
08-08-13, 09:18 PM
so if i read this right?

does say prove that inattention, to us is secondary

in the same way inattention is secondary to say depression or anxiety

sarahsweets
08-09-13, 06:26 AM
EX stands for extra cool.

Addersaurus
08-09-13, 06:49 AM
I have only skimmed this thread and not read any of the links so please excuse any ignorance on my part.

Surely "cold" testing cannot be reflective of reality, as it removes significant environmental interaction, so to test in an artifical environment absent of significant stressors is to test in an unrealistic environmental context, and obtain unrealistic results.

mctavish23
08-09-13, 04:52 PM
dave,

Nooo... that has nothing to do with it. It has to do with the evidence based

ASSESSMENT of ADHD, and why "real world" / "real life" measures ALWAYS trump the use

of "standardized" tests; including computerized tests, neuropsych tests, IQ tests, etc.

Hope that helps some.

tc

Robert

daveddd
08-09-13, 05:18 PM
Ok

I thought if our EFs were only affected when emotional regulation was involved. It would mean attention was a secondary problem to emotional regulation


My bad, I misunderstood the hot part as affected by emotional difficulty otherwise. Not disordered

daveddd
08-09-13, 07:04 PM
So attention is an executive function


Buts its disordered even when emotional regulation issues aren't present


Wouldn't that make it detectable on "cold" EF tests?

mctavish23
08-10-13, 02:44 PM
No Standardized Pyschometric Instruments Are Valid And Reliable For Use In Diagnosing

ADHD, Because Of The Factors Already Mentioned, As Well As The Following Concerns:

1) Poor Ecological Validity - Taking a test in a quiet room with (or without) an examiner,
is NOT the same as sitting in a classroom, operating a vehicle, doing chores, or being on the job at work (ECOLOGICAL = "Real World" Environment);

2) Small Sample Sizes - These tests were normed on extremely small numbers of people, so that they're lacking in statistical "power;"

3) Different Operational Definitions For the Same Terms.

Bottom Line, Please Don't "Over Think" This.

tc

Robert

Dizfriz
08-10-13, 03:53 PM
Thought I might share some of my notes from Barkley's book Executive Functions (2012)

These are just some of my notes and thoughts and are not intended to be anything conclusive or definitive. The book is complex and dense and so these have to be taken as just some areas that I picked out as significant to me with the caution that I could have missed or misinterpreted some of his writing. (It would be easy to do with this book. I have read it twice and am just beginning to get a grasp on what he is proposing).

From my notes:

Why EF? EF exists to solve problems that arise in group living, especially individuals with whom we are unrelated. Barkley proposes EF as an extended phenotype that covers a broad range. Their purpose is to allow us to get our needs met in the social, cultural environment in which we mostly exist. Working with and within groups in other words.

Definition:

EF is the use of self directed actions so as to choose goals and to select, enact, and sustain actions across time toward those goals. Barkley adds usually in the context of others often relying on social and means for the maximization of one's longer-term welfare as the person defines it to be.

Executive Functions: (there are several formulations of EF in the book, this is the ones I liked best.

EF consists of:

1 Self directed attention to create self-awareness.
2 Self directed inhibition to create self restraint (Executive Inhibition).
3 Self directed sensory-motor actions to create mental representations, and simulations
(ideations).
4 Self directed speech to create verbal thinking (private speech).
5 Self directed emotion and motivation to create conscious appraisal (this is where emotional self regulation comes in).
6 self directed play (nonverbal and verbal recombination to create problem solving, fluency or innovation).

Think of EF as working toward goals in the future and how a deficit would affect the individual in achieving those goals.


Some criticisms of current EF assessment

They do not assess self regulation directly.

They are conducted over incredibly short time intervals and so miss he cross temporal aspects of EF.

They ignore any social purposes for which EF may have evolved.

They fail to grasp the significance of social reciprocity, cooperation, mutualism and culture in their construction


Just something I thought might add a little to the conversation.

Dizfriz

daveddd
08-11-13, 10:03 AM
i over think everything, its what i do, i dont view it as a bad thing

i dont see anything said as proving what i said has nothing to do with hot EF functioning as the primary issue effecting attention, making attention secondary

as opposed to simple inability to pay attention



depression-biased attention to negative events/thought

anxiety- biased attention towards anxiety provoking stimilus/situations

adhd- biased attention towards (or possible away from) emotionally laden events or in short , impulsiveness

daveddd
08-11-13, 11:23 AM
either way, i was just wondering how this hot EF thing could be useful in diagnosing

instead of the attentional questions my doctor and every doctor ive been to asks all the time

it just seems the right questions are not focused on, or even the right knowledge is out there

understanding something, to me , is more important than a varying outward presentation

daveddd
08-11-13, 11:28 AM
i wonder if something could be rigged up with a super chaotic waiting room, and hidden testable questions on the intake paperwork

obviously that wouldnt be a very practical diagnostic method, maybe a study or something

mctavish23
08-11-13, 11:43 PM
There ARE ecologically valid instruments that can help in confirming the diagnosis, by

assessing both hot + cool/cold EF in the "real world;" most notably the Behavior Rating

Inventory of Executive Function (BRIEF).

tc

Robert

daveddd
08-12-13, 10:53 AM
Cool thanks


Third party report? I suck at self awareness and description


Out of curiosity I did some digging and found a couple frustration inducing hot EF studies as well.

Over thinking again

But before I over thought everything. I under thought everything and just did a bunch of cocaine instead. So I'll live with it for now

Thanks for the info

mctavish23
08-15-13, 11:13 AM
The BRIEF has both 3rd party and self-report forms.

tc

Robert

mctavish23
08-15-13, 11:19 AM
As a follow up to post # 15, from (08/06/13), one of the other things I look for is

whether or not the OP returns.

tc

mctavish23

(Robert)

ADDarkD@y
08-19-13, 01:56 PM
I'll be happy to get the reference source and show you what I'm talking about. Keep in mind I'm NOT refuting what you stated. I'm merely stating an evidence based fact, that I'll be glad to explain.
...
One thing you'll learn about me, is that I have no problem admitting when I'm wrong; as that's how we all learn. By the same token, please note that if I say something is evidence based, it means I can prove it unequivicably.


Hi Tavish, I do apologize for the long response. I have been totally side-tracked with other matters and then completely forgot we ever had this debate until I remembered the other day. Lol, that's ADD for you though.

I would like you to produce this evidence. I am the same as you and will happily eat my own words if they are wrong.

mctavish23
08-19-13, 03:43 PM
I've already started. What you see in two posts in particular, COUNTS as "proof."

If you read what I've said carefully, we're not that far apart. Why don't YOU try and

take issue with what I've already posted for starters ?

While you're at it, try Reading Chapters 8 & 9 of Attention Deficit Hyperactivity Disorder,

A Handbook For Diagnosis And Treatement, 2nd Edition, 2005, Guilford Press, by Russell

Barkley, Ph. D. ? That's the best reference I can think of. However, I'll continue to post

more, with the caveat that YOU actually read them. There's no way in hell I'll continue

to do all the work, while you post a couple of operational definitions, which I don't

disagree with.

tc

mctavish23

(Robert)

ADDarkD@y
08-19-13, 04:00 PM
[1]We're talking about whether those constructs can be accurately measured by standardized tests…


Right. Here is a study I found on the subject:

(Burgess et. Al, 1998) The ecological validity of tests of executive function
Available at: http://eprints.ucl.ac.uk/4537/1/4537.pdf

From Abstract: Ninety-two mixed etiology neurological patients and 216 control participants were assessed on a range of neuropsychological tests, including 10 neuropsychological measures of executive function derived from 6 different tests. People who knew the patients well (relatives or careers) completed a questionnaire about the patient’s dysexecutive problems in everyday life, and this paper reports the extent to which the tests predicted the patients’ everyday life problems. All of the tests were significantly predictive of at least some of the behavioral and cognitive deficits reported by patients’ careers. However, factor analysis of the patients’ dysexecutive symptoms suggested a fractionation of the dysexecutive syndrome, with neuropsychological tests loading differentially on 3 underlying cognitive factors (Inhibition, Intentionality, and Executive Memory), supporting the conclusions that different tests measure different cognitive processes, and that there may be limits to the fractionation of the executive system.

From Body:

Trials A: .48
Trials B: .43
Recognition Memory Test Faces: .49
Six Element Test: .46
Fluency–Animals: .40
MWCST: .
Categories: 54
Total Errors: .49
RBMT Profile Score: .41
Recognition Memory Test – Words: .54

So EF studies do measure what they are supposed to moderately well, but not all the subtests are useful. Given this information, it seems at least some of the existing EF tests should be modified to eliminate useless subtests and to increase ecological reliability on those subtests which work.

Please note that I never said the EF tests were infallible. I said they distinguished reliably and validly between neurologically impaired individuals (I should clarify: as measured by rating scales or neuro-imaging) and controls. And they in fact do. And that therefore they would work for ADD clinicals as well, and they in fact DO. Not surprisingly, the results show the same pattern as observed with the DEF patients – moderate correlations between some subtests and symptoms and low or trivial correlations between others. That is expected.

Also, there is one more thing. The sampling method in studies of EF in ADD clinicals is no good. Apparently they are not distinguishing reliably between different sorts of patients, ADD, ADHD, and SCT. As I have pointed out elsewhere these are separate disorders not variants of the same disorder, and even if they were, they require individual analysis. As you can see, if patients of different types exhibit structural differences in EF deficits then the overall analysis of EF in the total sample will be confounded. This is especially true if the deficits are categorical!!! (i.e. completely structured between patient groups). And, as we already know from EF testing in different ADD-type disorders as well as self-reporting, and rating scales, the deficits are in many cases categorical. As you know, this means measures of EF deficits in ADD clinicals are understated.

[2] He further states how standardized (neuropsych) tests of EF were originally developed to assess psychological functions, such as Abstract Thinking in patients with Pre-Frontal Cortex (PFC) damage.

If they were subsequently standardized against normal controls and have high test reliability and validity, then they are actual measures of the thing they purport to assess. I could probably find you a 1000 examples of how something in science was applied first in one field or research area and then later on coopted for another purpose. Pointing out the origin of EF tests in abnormal psychology is not saying much.

[3] I'm going to sum this (complicated subject matter) up for now..

Thank you for clarifying and summarizing. Let’s syllogize my argument.

If (1) EF tests do in fact measure EF well, and if (2) ADD is a product of EF dysfunction, then (3) tests of EF function in ADD will demonstrate a specific profile of dysfunction relative to controls.

Since (1) and (2) are true, and (4) ADD research studies have applied these tests of EF function in ADD subjects, (4) these studies demonstrate a discrete profile of dysfunction that is very significantly correlated with ADD.

That is, what the studies are finding in their empirical results agrees with what must happen logically.

mctavish23
08-19-13, 04:18 PM
Not bad. I appreciate the referefences. However, your data are 15 years old, with what appears to be a sample size of less than one hundred. The oldest primary reference I've posted is 2005; although I recognize there are older references within those articles.

Once again, I don't think we're that far apart. You have to remember that there are NO professional associations or academies, which allow test scores to be used for the sole purpose of diagnosing. Furthermore, Chapters 8 & 9 of the ADHD Handbook, 2nd Edition, in particular Chapter 9, covers EVERY type of test category + numerous individual tests, used in addressing ADHD; NONE of which are diagnostic in nature, for the EXACT same reasons I've already stated.

My ultimate goal is to try and educate, not just the consumer, but also other clinicians, in terms of what constitutes a best standard of practice, evidence based ADHD diagnosis.

tc

mctavish23

(Robert)

mctavish23
08-19-13, 06:20 PM
Here's a follow up to the April 2011 article from The ADHD REPORT previously referenced,

indicating there are 33 operational definitions of "Executive Function" currently available.

Again, it's by Russell Barkley, Ph.D., and is from the April 2012 issue of The ADHD

REPORT, Number 2; entitled ASSESSING ADHD AS AN EXTENDED PHENOTYPE.

The parts I'd like to focus are are contained in the following qoutes "

"A further problem is that tests designed to assess EF, have exceptionally poor ecological validity (Barkley, (2011a). The Barkley Deficits in Executive Functioning Scale. New York: Guilford)."

"Once we have such an operational definition of EF (i.e., utilizing the concept of an extended phenotype) and a theory of how it works in major life activities, one can see how current EF tests fall short. Now one can see why: they do not assess self-regulation directly; they are conducted over incredibly short time intervals and so miss the cross-temporal aspect of EF; they ignore any social purposes for which EF may have evolved; and they utterly fail to grasp the significance of social reciprocity, cooperation, mutualism, and culture in their construction.

Dogmatic adherence to the psychometric traditon of understanding and assessing EF provides only a superficial evaluation of even the conventional phenotypic view of EF. It fails to capture entirely the multi-level, concentrically arranged, affectively/motivationally charged, socially important, and culturally facilitated nature of the extended phenotype of EF/SR (Self-Regulation) in everyday activities. Further, the evidence available on the reliability and validity of these tests does not justify their entrenched place in making important declarations about an individual's capacity for executive functioning and self-regulation or the likelihood of impairment in important domains."

"Observations and rating scales of EF collected in ecologically naturalistic settings are more likely to be evaluating the methodical (adaptive) tactical, and strategic levels of EF."

He goes on to explain in great detail how standardized psychometric tests of Both hot and cool/cold EF do not accurately assess EF in the "real world." The caveat is that NONE of these measures are ever diagnostic; in and of themselves.

Once again, I don't see us as being that far apart on this. However, without reprinting all my references in here, I would respectfully submit that this post has more than addressed the question at hand.

tc

mctavish23

(Robert)

ADDarkD@y
08-20-13, 01:47 AM
Not bad. I appreciate the referefences. However, your data are 15 years old, with what appears to be a sample size of less than one hundred. The oldest primary reference I've posted is 2005; although I recognize there are older references within those articles.

Once again, I don't think we're that far apart. You have to remember that there are NO professional associations or academies, which allow test scores to be used for the sole purpose of diagnosing. Furthermore, Chapters 8 & 9 of the ADHD Handbook, 2nd Edition, in particular Chapter 9, covers EVERY type of test category + numerous individual tests, used in addressing ADHD; NONE of which are diagnostic in nature, for the EXACT same reasons I've already stated.

My ultimate goal is to try and educate, not just the consumer, but also other clinicians, in terms of what constitutes a best standard of practice, evidence based ADHD diagnosis.

tc

mctavish23

(Robert)

I understand and agree with your points about need for careful diagnosis. I simply saw this reference in an article I was reading and went looking for it. I have no doubt I can find a wealth of material providing these findings to be consistent.

ADDarkD@y
08-20-13, 02:03 AM
...

He goes on to explain in great detail how standardized psychometric tests of Both hot and cool/cold EF do not accurately assess EF in the "real world." The caveat is that NONE of these measures are ever diagnostic; in and of themselves.

Once again, I don't see us as being that far apart on this. However, without reprinting all my references in here, I would respectfully submit that this post has more than addressed the question at hand.


I agree. The correlations are too low ~40-50% precisely because the tests as they are are NOT accurately measuring EF. They are, however, diagnostic in my view when restricted to the proper sub-tests and statistical methods (such as clustering data). This is a softening of my original and untenable position which was that EF tests were very good measures of EF.

I think we are not THAT far apart but we do disagree significantly on points (a) if they are measuring EF moderately well, or poorly, (b) that EF subtests show a consistent deficit on specific measures in ADD type disorders (c) whether or not there is a pattern between tests which we can identify as a diagnostic criterion for each disorder.

I have argued already for (a) as much as is I think is useful. I could do more for (b) but as we are concluding I won't. Further research is needed on my part to advance (c) so I will concede the null hypothesis that there is no diagnostic criteria pattern between tests (although I personally HIGHLY doubt this).

mctavish23
08-20-13, 08:52 AM
Well said. I won't argue what the research has already said. Having read much of the latest data, with the exception of Russ's newest book, I have no doubt those data continue to support this point.

One of the best examples I can site is from the Technical Manual to the WISC-IV IQ test. On page 75, the author, David Wechsler, states in regards to the use of the respective Index Scores, that they may... "never"... be used for diagnostic purposes; meaning they don't measure "real world" EF.

Once again, I originally came out in support of your post in another thread. I can only hope you'll read some of the references I've posted.

tc

mctavish23

(Robert)

Dizfriz
08-20-13, 09:54 AM
EF TESTS

I have been kinda watching this conversation and thought I might stick my foot in the water. I have read Barkley's book Executive Functions (2012) twice and wanted to read it a third time before trying much to present his ideas on the forum. What I am doing here is just focusing on the area of EF tests. I hope I do an adequate job. Note: Most but not all of the material discussed here is from his book.

There has been an ongoing conversation over EF tests and especially EF tests and ADHD for quite a while. Part of the controversy is that some say that ADHD cannot be a deficit of EF because ADHDers do not score lower on EF tests. Barkley counters by saying that the problem is that EF tests do not measure EF. Time will tell but it looks like Barkley may be winning.

One of the things to be considered is that you have to get a good operational definition of EF to measure it. This is an area of focus in my discussion and Barkley's book.

Russell Barkley has presented a theory of EF in which he focuses on EF as influencing an extended phenotype which includes getting non related individuals and groups of individuals involved in helping the person achieve goals over time. This includes society and the larger culture as part of the extended phenotype of EF. Barkley focuses a lot on the impact of EF on society and culture and frames a major result of EF as allowing these two factors to exist. In otherwords, why do we have EF and what does it do for us?" (I will go more into this when I have read the book a third time)

One definition of EF I like is "the capacities for formulating goals, planning, and carrying out plans effectively--the executive functions are essential for independent, creative, and socially constructive behavior.

My opinion is that I feel that any measure of EF would have to include these things (goal formulation, planning, carrying out goal-directed plans, and effective performance.) in some manner.

They would also have to measure the time aspect. One major key to EF is that these activities are carried out *over time* to achieve goals. With human beings, these time spans can be very long sometimes measured in centuries.

If this aspect is not dealt with, I do not think any attempt at measuring EF will be effective.

Barkley presents some problems with EF tests that he feels must be resolved. The first problem he brings out is that EF tests as consistently shown by research to be only moderate or lower reliability and not very good validity. For example, they are also not all that good at detecting PFC injuries.

The second problem he discusses is that they have little or no ecological validity . The tests do not correlate well, or at all, with means of assessing EF in everyday life

To quote:"If a primary clinical aim is to predict how well an individual will do with executive functioning in the real world of daily life activities, then EF tests are of minimal or no help." The results of these various studies usually reveal that any single EF test shares just 0-10% of its variance with EF ratings or observations of EF in daily adaptive functioning." Executive Functions-Barkley p10-11. He goes into a lot of detail backing this up (If you are familiar with Barkley, he always goes into sometime excruciating detail backing up his statements.)

Next he discusses the problem with low and often nonsignificant predictive validity of EF tests to EF ratings. If EF tests are effective, they should be able to predict impairments of various major life activities. The EF tests are reported as being very poor or not effective at all in making these kinds of predictions.

A good statement: "Something is terribly amiss here if different methods of measuring the same construct are so poorly related to each other and lead to such disparate findings and hence conclusions" P-11

And last of what I am going write on this aspect "The totality of findings to date concerning the relationship of EF tests to EF ratings and of each to daily life indicates that EF tests are largely not sampling the same constructs as are EF ratings or direct evaluations of EF in daily life.' P-11

Basically what I am saying here is that EF "Tests" as such do not seem to be measuring EF.

What does a better job are rating scales, observation scales and such that try to measure how the person's EF abilities translate into both everyday life but also in long term adaptation.

To be honest, most clinicians can tell a lot about the level of EF just from taking a good history.


To another point, you admit that this is a new subject to you and that is OK as it was at one time new to all of us. One small point though, it is not new to either Robert or myself. Just a small point but one I think needed to be made.

What often happens is that a person new to a subject such as this puts too much into individual studies. Individual studies are good but they need to be put in context of the overall scope of the subject, current and past research and the general professional consensus and discussion.

Without this context, it is difficult to put any kind of realistic perspective on a single or small group of studies. In this case, if you did not know the controversy over EF, ADHD and neuropsychology, you would miss a lot of the nuances involved.

Right now, Barkley has fired a broadside with the book Executive Functions and seem to be ahead on points but time will tell how this turns out.

I tend to agree with Barkley here but I am going to have to do some more reading and thinking on the subject before I form a firm opinion.

Later perhaps, right now I am trying to get some understanding of epigenetics and it ain't easy.


Just throwing my two nickles worth.

Dizfriz

Addersaurus
08-20-13, 03:57 PM
What Mctavish seemed to be expressing is that the more successful approach is more ethnographic in nature, akin to an environmental psychology approach.

That is the "real world" approach, and that to me makes the most sense.

But what about the connotations.

An environmental ("real world") approach supports the notion that the environmental context of these behaviours is paramount. To remove the context of the behaviours and study in an artificial environment such as in a research lab is to remove context, an aspect that is obviously so fundamentally important with regard to disorders such as ADHD...

mctavish23
08-20-13, 08:24 PM
Also well said. When it comes right down to it, all I'm really advocating for is for everyone, client's/consumer's and clinician's alike, to simply READ THE PUBLISHED GUIDELINES FOR THE DIAGNOSIS AND TREATMENT OF ADHD. Everything I'm saying is merely paraphrasing what's already in place. Period.

tc

mctavish23

(Robert)

daveddd
08-21-13, 05:52 AM
dizfriz

just finished my second time reading it

great book, hopefully he keeps the topic going like his other books

and great that he still keeps his books so cheap, under 20 (ebook at least)

there are a few important things i was beginning to think i had zero understanding of, but it turns out i had a better grasp that i even thought

but much more that i dont at all

so im anxious for your interpretation

thanks

TygerSan
08-21-13, 07:50 AM
One thing that I think makes this debate so difficult is that you have competing goals.

The goal of the clinician and the clinical psychologist is to do what Addersaurus mentions: determine how people function in the context of their environment.

The goal of the research scientist is to somehow study the atoms of what makes people able to function in their environments, but in a way that yields clear, predictable results.

The two goals are subtly different, and in both cases you need to define what it is you want to measure.

In the case of the clinician, you're talking about things like social functioning, ability to work, self-control in the context of the complex environment that exists around us.

In the case of the laboratory scientist, you're trying to break down what "self-control" means into something that can be easily measured and tested in the laboratory. Ideally, this concept maps onto --at least partially -- the real-world idea of self-control.

The problem is, if you look at the literature, you'll find a couple of different laboratry (operational) definitions of what constitutes "self-control", "impulsivity", or "executive function", none of which fully cover what is generally meant by those psychological terms as they apply to real-world functioning.

Using impulsivity as an example: what is the general definition of impulsivity. All of us probably have a vague idea of what we would consider an impulsive act, but the definition is a bit slippery.

It can mean acting without foresight, but what does foresight mean? Does that mean acting before you have all of the information you need to make an informed decision? (not waiting until you have enough information has been termed "reflection impulsivity", and there is a laboratory task for that).

But then again, a common example of impulsivity is reaching for something on a hot stove. In that case, you have all of the information you need, but you still reach for the stove. In this case, you are unable to inhibit an action, and that is a form of motor impulsivity, which can also be measured in the laboratory.

Finally, someone can be considered impulsive if they spend their money on something shiny rather than saving it. That's more of a thinking kind of impulsivity (well, I can get this small thing now, but if I wait and save up, I'll be able to get the fancy gaming console I've always wanted. Oh, screw it, that little shiny trinket is too cool, I'll buy that now!). And, yes, there are laboratory tests that get at this type of thinking as well.

The thing is, what "type" of impulsivity best correlates with ADHD symptoms? In a way, all of them correlate somewhat. If I remember correctly, the motor stuff seems to correlate the best, but each is only a tiny window into what we call "impulsivity", so each test in isolation tells us very little about a person's functioning level in the real world.

Dizfriz
08-21-13, 11:18 AM
so each test in isolation tells us very little about a person's functioning level in the real world.

This is generally the way I see it. It is not that EF tests do not have value but but it is that they do not reflect well how they operate in the individual in their life settings much less how they interact on exposure to the broader areas of society and culture. It is this broader aspect that Barkley in addressing in his recent work.

Good post,

Dizfriz

mctavish23
08-21-13, 05:14 PM
EF Tests DO have value; just NOT diagnostically.

tc

mctavish23

(Robert)

meadd823
08-22-13, 06:31 PM
The BRIEF has both 3rd party and self-report forms.

tc

Robert

Oh I thought my diagnosis of ADHD was based upon the number of times I lost the damn questionnaires or forgot to have them filled out or by the fact is was easier to have my family member called on the phone verifying they filled the questionnaire out twice but I lost one and forgot to pick up the other, with the first packet of questionnaires never making it out of my car!!!!:rolleyes: - I found them two months after my diagnosis when I finally got around to cleaning the inside of my car.:o

mctavish23
08-22-13, 10:53 PM
Perfect example of "Real Life" Behavior superseding Test Scores. :p


u r welcome :cool: