Reverse Discrimination Case

[jillio]

Can all tests be valid in terms of race only? What if the culture bias affects the person too much and there is no way to take it out from the test?

Then you use another instrument that has been shown to be valid, or you normalize the scores to provide an accurate representation. But if the test had of been validated prior to its use, rather than simply assuming validity because some professional testing organization designed and charged for it, this situation probably never would have occurred.

 

[Beowulf]

How often is the test normally given?

If the firefighters hadn't sued, how soon would they be able to take another test?

I was wondering that as well. And also, it may seem like wishful thinking, but if the test participants' race and gender were kept secret until AFTER the testing, there might not be any lawsuits, lol.

 

[jillio]

I was wondering that as well. And also, it may seem like wishful thinking, but if the test participants' race and gender were kept secret until AFTER the testing, there might not be any lawsuits, lol.

I'm wondering...how do you keep race and gender secret? Those are pretty easily discernable characteristics.

 

[Beowulf]

I'm wondering...how do you keep race and gender secret? Those are pretty easily discernable characteristics.

I am assuming that a group of people take the test in a room, and each participant will merely have a number or some mark on the test paper. That way, those who grade the test will have no idea of the test taker's identity. That make sense?

 

[Daredevel7]

Then you use another instrument that has been shown to be valid, or you normalize the scores to provide an accurate representation. But if the test had of been validated prior to its use, rather than simply assuming validity because some professional testing organization designed and charged for it, this situation probably never would have occurred.

How do you validate a test? Performing the said test on a population and seeing if the distribution is normal? Isn't that a catch 22?

 

[Daredevel7]

I am assuming that a group of people take the test in a room, and each participant will merely have a number or some mark on the test paper. That way, those who grade the test will have no idea of the test taker's identity. That make sense?

That still wouldn't work. The results would be the same.

 

[jillio]

I am assuming that a group of people take the test in a room, and each participant will merely have a number or some mark on the test paper. That way, those who grade the test will have no idea of the test taker's identity. That make sense?

Well that is how it is done when the test is being used for experimental purposes. But this test was used for promotional purposes. Are you thinking that the tests were graded differently depending on the test takers race? I'm just not sure what you are trying to say.

 

[darkdog]

If you can't infer how the distribution affects issues of validity, I can only assume that you have never had a research methods course in your life. And most especially not one specialized in testing and assessment.

Yeah, as I said, this is not an area I know much about, which is why I'm asking questions. I get that the distribution affects validity. I'm trying to get down to specifics- i.e. how close the distribution must be to normal.

Funny. Statistical analysis for the behavioral sciences, which assessment and test validity fall into, only carries out 2 decimal places. If you are using 6 places, you have done the calculations incorrectly.

I'll show you what I did. Tell me what's wrong with it. First, I took the probability distribution function for a normal (aka Gaussian) distribution:

Then I took the standard normal distribution (mean=0, SD=1):

Then I took the cumulative distribution function.

Of course, that's an unreasonable integral, so I used the error function, which is defined as follows.

I plugged the error function into the cdf and got the following.

You can look up the values for the error function on a table or go here: Scientific Calculator
The percentage in 1 SD is 100*(Fx(1)-Fx(-1))=200Fx(1) (since erf(x) has odd symmetry).

In that calculator, you can type in 100*(Erf(1/Sqrt(2))) and hit Tab to get the answer. For the population percentage in 2 SDs, type in 100*(Erf(2/Sqrt(2))), and so on.

And what you referred me to is not applicable to the issue of validity of testing and assessment instruments.

As I said, I know that, but at least I'm at the level where I can understand this material pretty easily. That is, if it is explained clearly.

Perhaps it will become clearer as you continue your education.

I think you're trying to say that a distribution of test results should approximate the probability distribution function for a normal distribution (as described in my first equation). If so, you picked a weird way to say it.

That's because 22/7 is NOT 3.17. Check your calculation.

Check my number. It's not 3.17. It's 3 1/7, or three and one seventh. That's exactly 22/7 and approximately pi.

"Student-t is not a distribution." -jillio, April 27, 2009
"Student t is not a distribution." - jillio, April 28, 2009

I'm pretty sure William S. Gosset would disagree with you. Just sayin'. (BTW that photo is from this site: Student's t-Distribution -- from Wolfram MathWorld. And as I mentioned before, some texts call it the student-t distribution and others call it the student's t-distribution. Same things.)

T-scores are used for comparing scores to peer group scores. T-scores have a mean of 50 and an SD of 10. Z-scores have a mean of zero and an SD of 1. That is why raw scores must be converted to z-scores and t-scores for interpretation of what a raw score actually means in comparison.

I'm with you on that.

Check my posts. I have already done so. You, as well as naisho, are attempting to compute based on methods used for hypothesis testing. That is the wrong statistical application for looking at frequency distribution to determine disparities that are indicative of problems with validity in a testing instrument.

I haven't tried to compute anything. I'm just asking the questions that naturally pop up. From what I understand so far, you convert all the scores to T-scores and you compare that to a normal with mean 50, SD 10. You see if there's any skew and you see if the percentages of the population within the various SDs are approximately correct. That much I get. I'm trying to figure out the specific criteria you use for the comparison.

And I have explained again and again why the things you were assuming were incorrect. I have explained in my posts to naisho why he keeps coming up with wrong figures. Go back and check those posts.

I haven't read those posts yet, so I'll go do that and respond to those if I have any questions. Deal?

 

[jillio]

How do you validate a test? Performing the said test on a population and seeing if the distribution is normal? Isn't that a catch 22?

There are numerous ways to validate a testing instrument, and they differ depending on the type of validity being determined. The test can be correlated to another instument in which validity has already been determined, and then statistically analyze the shared variance. There are 4 steps used to determine content validity. For criterion related validity, the test is compared to an external source at the same time the test is being given to a sample population. For predictive validity, test scores are related to a future standard. Construct validity is the scientific basis for showing that a construct is actually being measured by the test, and is done empirically. To demonstrate construct validity it is useful to provide multiple sources of evidence, including experimental design validity, factor analysis, convergence with other instuments, and discrimination with other measures.

When a test is given to a specific population for purposes of demonstrating validity for that particular population, it is called norm referencing.

 

[jillio]

That still wouldn't work. The results would be the same.

Exactly.

 

[jillio]

Yeah, as I said, this is not an area I know much about, which is why I'm asking questions. I get that the distribution affects validity. I'm trying to get down to specifics- i.e. how close the distribution must be to normal.


I'll show you what I did. Tell me what's wrong with it. First, I took the probability distribution function for a normal (aka Gaussian) distribution:

Then I took the standard normal distribution (mean=0, SD=1):

Then I took the cumulative distribution function.

Of course, that's an unreasonable integral, so I used the error function, which is defined as follows.

I plugged the error function into the cdf and got the following.

You can look up the values for the error function on a table or go here: Scientific Calculator
The percentage in 1 SD is 100*(Fx(1)-Fx(-1))=200Fx(1) (since erf(x) has odd symmetry).

In that calculator, you can type in 100*(Erf(1/Sqrt(2))) and hit Tab to get the answer. For the population percentage in 2 SDs, type in 100*(Erf(2/Sqrt(2))), and so on.


As I said, I know that, but at least I'm at the level where I can understand this material pretty easily. That is, if it is explained clearly.


I think you're trying to say that a distribution of test results should approximate the probability distribution function for a normal distribution (as described in my first equation). If so, you picked a weird way to say it.


Check my number. It's not 3.17. It's 3 1/7, or three and one seventh. That's exactly 22/7 and approximately pi.

"Student-t is not a distribution." -jillio, April 27, 2009
"Student t is not a distribution." - jillio, April 28, 2009

I'm pretty sure William S. Gosset would disagree with you. Just sayin'. (BTW that photo is from this site: Student's t-Distribution -- from Wolfram MathWorld. And as I mentioned before, some texts call it the student-t distribution and others call it the student's t-distribution. Same things.)


I'm with you on that.


I haven't tried to compute anything. I'm just asking the questions that naturally pop up. From what I understand so far, you convert all the scores to T-scores and you compare that to a normal with mean 50, SD 10. You see if there's any skew and you see if the percentages of the population within the various SDs are approximately correct. That much I get. I'm trying to figure out the specific criteria you use for the comparison.


I haven't read those posts yet, so I'll go do that and respond to those if I have any questions. Deal?

In short, you are using the wrong statistics. But yes, go back and read those posts, and then I will answer any quesions you may have. It's a deal.

What you are referring to as a student T distribution is simply the curve obtained when raw scores are converted to t-scores. T-scores carry a mean of 50 with an SD of 10. Z-scores carry a mean of -0- with an SD of 1. Raw scores vary on both the mean and the SD, which is why it is necessary to convert to Z-scores or t-scores to interpret what the raw scores actually represent. That is what we are attempting to do here. Statistically analyze the scores to derive meaning. All we need do for those purposes is to convert raw scores to z-scores and then plot them cumulatively, and sorted by race (or gender, or whatever we are using to identify the test takers.) That will indicate whether the scores are falling on a normal curve (which can be expected if the test is valid for the population it has been given to) or if it indicates skews in the scores based on an identifying variable. The comparison is based on in-group and out-group diffferences in converted scores. You can take it down to T-scores if you choose, but to determine if there is a greater skew based on the race (or gender, or whatever) of the test taker, you can discover that with Z-scores. This is called norm referencing.

 

[darkdog]

In short, you are using the wrong statistics. But yes, go back and read those posts, and then I will answer any quesions you may have. It's a deal.

Cool. No questions so far, but the nerd in me can't help it. I got my Excel spreadsheet open and I'm plotting out all the scores among the various groups and converting them to z-scores and t-scores. I shall return.

 

[jillio]

Cool. No questions so far, but the nerd in me can't help it. I got my Excel spreadsheet open and I'm plotting out all the scores among the various groups and converting them to z-scores and t-scores. I shall return.

Ifyou are doing it by hand, it will take awhile! BTW...I added some to the previous post. You might want to check that.

 

[Jiro]

They can't. The white firefighters filed a lawsuit. That means that they cannot issue another test until this case is decided. That, as well, has been mentioned many times. I don't know how you continue to miss it.

You forgot somebody - a hispanic firefighter. He was part of the group who filed the lawsuit as well.

And what scientific field would that be? A student? I find it ironic indeed that you claim to be in a field when you are not practicing in that field. I also find it ironic that you even mention "real" answers when you do not appear to have the ability to distinquish "real" answers, or "real" questions.

I'm sorry but aren't you a student as well?

Again, the test has already been determined to be flawed. And again, they cannot give them another test until this case has been decided. Talk to your white firefighters. They are the ones that are preventing logical resolution. And the city was concerned with far more than "superficial" appearances. You simply do not see it because of your superficial understanding.

I'm sorry but this case being heard at U.S. Supreme Court is not to question if the test was racially-biased or not. Again - my post #378 -

"May a municipal employer disregard the results of a qualifying examination, which was carefully constructed to ensure race-neutrality, on the ground that the results of that examination yielded too many qualified applicants of one race and not enough of another?"

Like what lfoster said previously - this is a case of possible discrimination and that is for Supreme Court to determine.

 

[naisho]

Honestly, I personally think the test is crappy..

Hey DD, check this out. I had a sudden idea to go search around for example tests.

Obviously, these aren't exactly the New Haven CT promotional exams. However I would not be too surprised if they are in some ways similar in questions covered there.

The site is probably made to earn some cash along with help train people free, it states that it is based off Battalion Chief Paul Lepore who has 20 years of experience. This is what the site says about him:

Paul Lepore is a Battalion Chief with the City of Long Beach, California Fire Department. He entered the fire service as a civilian Paramedic for the Los Angeles City Fire Department in 1985. After completing his education at the Los Angeles County Paramedic Training Institute, Lepore worked in the high impact area of South Central, Los Angeles. Lepore was hired by the Long Beach Fire Department in 1986 and completed their 12-week academy. He spent the next two years working as a firefighter until he was promoted to firefighter/ paramedic. Lepore was promoted to Fire Captain in 1998 and is currently a Battalion Chief.

So with that in mind, he probably has tests based on western USA - California if there's much difference between CA and other states for promotional exams.

Become a Fireman, Firefighter Exam Training Book Written by a Battalion Chief

You can find the promotional exams on the side that says "Promotional Exams", then a link down there that says Free Promotional Downloads right before the customer satisfaction posts. It's really interesting, have a look around. I wonder what in these tests make it just so hard to understand?

Here's some interesting examples in em:
Oral Interview:
Fire Extinguishers
You are a Captain of an Engine company. Your assignment is to teach the new firefighter how to use a fire extinguisher. You have been provided with a fire extinguisher.
You have 15 minutes to prepare a 5 minute presentation on how to properly use the guidebook.

Tactical Management - Hazardous Materials Algorithm
1. Establish Command
2. Safe Approach and Position
3. Establish Staging
4.Establish Security Perimeter
5.Establish Hazard Zones
6.Rescue and Protective Actions
Staging Areas 3 min from scene
LEVEL I: Initial Arriving Units
LEVEL II: Base
Public Protection Actions - For Incoming Resources “Protecting in Place” or “Evacuation”

Tactical Development
Engine 1 reports that their supply line from the hydrant has been run over by a
vehicle.
Supply line run over:
• Notify anyone who is on a hoseline that is being pumped by Engine 1
• Have another engine lay a line to E1
• Engineer hand jacks a supply line to hydrant with help
• Request PD for street closure

The school to the North states they have smoke on the playground.
Smoke on playground:
• Direct dispatch to tell the school there is a structure fire in the area and to shelter
the children in the classroom.
• No immediate danger.
• Send a PD unit to the school for P.R.

Assessment Centers:
You are a captain of an engine company who has been dispatched to a structure with
numerous calls and a report of people trapped. It’s in your first due. As your engineer
pulls out of the station he strikes a car. What would you do and why?
Here is a checklist of what the evaluators expect:
• I would immediately notify dispatch that we have been involved in a traffic
accident and have them replace us with another engine. I would ensure that the
Battalion Chief knew we were no longer responding (this is particularly true if
your department dispatches on one channel and has a separate tactical frequency).
• I would check the condition of my firefighters and the condition of the people in
the other vehicle.
• I would request PD for traffic control and fire and EMS units for the injured.
• I would ensure my uninjured firefighters or I were treating any victims.
• I would have my firefighters place out road cones or flares to prevent a second
accident.
• After the injured were treated, I would remind my firefighters not to admit fault or
make statements to anyone.
• I would get names and contact information of any witnesses.
• I would make sure the appropriate vehicle accident and injury paperwork was
completed.
• I would make sure the fire chief and city attorney were notified.
• I would make sure that the families of any injured firefighters were notified.
• If necessary, I would facilitate critical incident stress debriefing
• I would log it in the station logbook.

It's worth a read, for interesting general firefighting questions. Now if we can find the exact exam we are golden.

 

[Jiro]

No offense but would you rather take the word of a college degree police/firefighter over a high school graduate police/firefighter?

on what subject? they both are qualified and sworn officers & professionals who abide by same laws & regulations and receive same trainings. so I'm not sure what you're getting at.

The city didn't throw out the results. The fire department threw out the results. The firefighters are only suing themselves dragging the city along for the ride.

uh yes city did. See my posts and lfoster's posts.

I disagree. Afterall if the fire department hires new recruits, do you think that they will send them into a burning building without the proper tools and mechanics of how to fight a fire? :roll:

that's why they have highly-qualified captains/LT to give instructions to firefighters. That's why I believe that to say this test is culturally-biased is absolutely nonsense! The fire discerns nothing....

 

[Jiro]

What's the matter? Can't keep up?

I have to kindly ask you to keep your condescending behavior in check. It's because I have something called job duties. Before I comment on it, I wanted to do some research on my own free time. Naisho's post regarding firefighter exam just actually saved my time.

But let's look at it this way:

You are required to take a test in order to be employed in the IT job that you now have. On that test is a subset that tests writing skills; specifically, ability to use English grammar. Also on the test is a subset that asks specific questions regarding knowledge of computer applications. The subset on computer applications is weighted to 20% and the subset checking grammar usage is weighted to 80%. Therefore, this particular test actually gives more weight (e.g. importance) to your ability to correctly use English grammar than it does to your knowledge of computer applications. Would you agree that this test is a valid assessment of your ability to do a job that is more concerned with computer application than with writing skills?

I'm sorry but no I will not look at it this way because this hypothetical example is absolutely fallacious and absurd!!!! Again - it is very apparent that you have a superficial understanding in office politic and how the business works. In case you didn't know, it does not really revolve around on some computation to determine if it's biased in any way.

 

[Jiro]

Still missing the point. If the test actually tested for this person's "people skills" then disparity in scores based on race would not be an issue. You are ignoring the fact that the professionals cited problems with the weight in items that compromised both content and construct validity.

and do you have any idea what the questions look like to make such statement?

You guys really do need to stop focusing on race, and look at the fact that the test was not valid for either the white firefighters not the minority firefighters. Some scores were artificially inflated, meaning that they did not provide an accurate representation of qualification, and some scores were artificially deflated, meaning that they did not provide an accurate representation of qualification. ALL scores were thrown out...both those that were artificially inflated and those that were artificially deflated, because none of the scores were contained predictive ability regarding job skills necessary for the job being tested for.

and you need to stop focusing on scores, statistics, or whatsoever. What you need to do is to actually research on what the questions are and how the questions are constructed (see naisho's post above). and you also need to look into "office politic" (*hint - something that your psychology graduate study does not cover*)

1. All firefighters passed SAME standard at fire academy
2. All firefighters are EQUALLY trained
3. All firefighters (for those who want to do promotion test) are given SAME studying materials

 

[lsfoster]

Is it possible to ever have a VALID test given to a population of 25% blacks and none of the blacks made it to the top 25%?

It seems like each individual's amount of knowledge and qualification would make it pretty easy to have that happen. If those particular men were less qualified, they could easily not be in the top 25%, similarly, if they were exceptionally qualified, they would make up a majority of the top scorers.

Honestly, I personally think the test is crappy. I can tell by just looking at the scores of the oral vs written in the link I gave. However, I am not sure if you were so sure that the test is invalid JUST because of the disparity in scores or because it has already gone to court twice plus the professionals already looked at it. I don't think it's right to throw away test results just because it doesn't give a distribution that you'd like, especially in a world where people are so afraid of lawsuits. I just don't think that a disparity among race alone is EVIDENT enough to warrant an automatic dismission of test scores. Delay promotions and warrant an investigation? Yes. Dismission? No, especially when a "normal distribution" would only need 2-3 black people to be promoted.

Thank you, that is exactly what I have been trying to get at. This is not about the test at all, because the city wasn't concerned with the test when the threw out the results. According to their own quotes, they weren't even worried about any implications of the scores other than "no black firefighters would be promoted". That is not a good enough reason to handle the situation the way they did.

They can't. The white firefighters filed a lawsuit. That means that they cannot issue another test until this case is decided. That, as well, has been mentioned many times. I don't know how you continue to miss it.

I don't continue to miss it, I keep explaining why it's not the issue. I'll say it again: The city would have had plenty of time to administer another test if they had actually questioned the test's validity at the time that they threw out the scores. If that had been their concern, they could easily have said that, explained what they were doing, and obtained another test from any other firehouse. The reason that they "didn't have enough time" is that they weren't concerned with the validity of the test until after the complaints and lawsuits had been filed and they needed a legal justification for their actions. Might they still be right? Sure, I've never said anything otherwise, but that does not justify the way they handled it, and it does not make it anyone's fault but their own that they never gave another test.

Why weren't the black firefighters eligible for immediate promotion? That is the question. One that you are continuing to ignore.

I also haven't ignored that question, I've answered it many times, if you read my posts. It's very simple: we don't know. There is absolutely no way for us to know if they weren't eligible because of the test, or because they were less qualified. You are of the opinion that they were ineligible simply because the test might have been flawed. I happen to disagree, I think that they did poorly on an exam, and others did better because they were more qualified. Both are opinions, and neither of us can say anything more. The whole point is that you have been arguing definitive statements that you make that you can't possible back up since we have no information either way.

No, you simply have not answered them, perhaps because you don't understand them. I don't know at this point.

Please point out which questions I have not answered, and I will either do so, or refer you to the posts where I did.

That would depend on why only black firefighters were promoted. That is the whole point.

Exactly, it is the whole point. You question the reasoning behind that decision, and not this one. Interesting...

And what scientific field would that be? A student? I find it ironic indeed that you claim to be in a field when you are not practicing in that field. I also find it ironic that you even mention "real" answers when you do not appear to have the ability to distinquish "real" answers, or "real" questions.

I'm not in my field because I want to become even more proficient in it? I don't see how that makes any sense. I could have left college with my bachelor's and gotten plenty of jobs in the field of chemistry. Instead, I chose to continue to learn more in that field while at the same time doing research and contributing to it. I don't see how that means that I'm not "in the field of chemistry" still. I am practicing, and researching, and learning, and have been published and recognized by the ACS for my work. If you don't consider that being in a field, we'll just have to agree to disagree.

Again, the test has already been determined to be flawed. And again, they cannot give them another test until this case has been decided. Talk to your white firefighters. They are the ones that are preventing logical resolution. And the city was concerned with far more than "superficial" appearances. You simply do not see it because of your superficial understanding.

If you can provide anything other than a cheap shot to back that statement up, I would like to see it. There are multiple quotes in the articles provided about the city feeling like they were in a "damned if you do, damned if you don't" situation because racially, things would look bad either way. That indicates a concern about nothing more than appearance.

Can all tests be valid in terms of race only? What if the culture bias affects the person too much and there is no way to take it out from the test?

Or any other number of factors that were clearly not included in jillio's "analysis". I should also say that we have no indication of what the board that is questioning the test included in their analysis, either. Without knowing what they did or did not consider, it is very hard to simply accept their opinion as hard fact.

Then you use another instrument that has been shown to be valid, or you normalize the scores to provide an accurate representation. But if the test had of been validated prior to its use, rather than simply assuming validity because some professional testing organization designed and charged for it, this situation probably never would have occurred.

Again, you can't "normalize" anything to provide an accurate representation without have some idea of the people's knowledge and qualification.

I am assuming that as soon as another testing instrument had been designed and validated.

I have no idea how often the test for promotions is given. This, however, was the first time that this particular test had been used.

There are tons of firehouses in CT, and I doubt that every single one of them was using the same test, or using a brand new test all at the same time. It should not be hard to contact another firehouse and request to give your firefighters the exam that they use and have been using for however long.

Well that is how it is done when the test is being used for experimental purposes. But this test was used for promotional purposes. Are you thinking that the tests were graded differently depending on the test takers race? I'm just not sure what you are trying to say.

He's trying to say that if you stopped focusing so much on race and instead just considered the range of scores on the exam, there probably would not have been any issues since the range accurately reflected what would be expected from any average group of men taking that exam. The only issue came up when they looked at race (and nothing else).

 

[jillio]

You forgot somebody - a hispanic firefighter. He was part of the group who filed the lawsuit as well.
Check the OP. He was IDed on the test as Hispanic, but was in fact, bi-racial.
I'm sorry but aren't you a student as well?

Uh, nope.


I'm sorry but this case being heard at U.S. Supreme Court is not to question if the test was racially-biased or not. Again - my post #378 -

Right. The test has already been determined to be biased. That answers the "why" question for the courts regarding the score distribution. Now the question is, is the city justified in throwing out the scores based on the test results.
Like what lfoster said previously - this is a case of possible discrimination and that is for Supreme Court to determine.

Discrimination against whom? And for what purpose? Is refusing to permit one group advantage on a testing instrument discrimination? Two lower courts have already rule that it isn't.