Fairness, Accountability and the Limits of a Model
Models trained to imitate past decisions may repeat discrimination. Compare outcomes and errors by group. Demographic parity and equalized odds can conflict when eligible shares differ and the shared true positive rate exceeds the shared false positive rate. Teams should state their criteria, remain responsible and provide explanations and ways to challenge decisions.
What a learner can do afterwards
- Show how a model can be accurate overall and much worse for one group
- Define demographic parity and equalized odds, and explain their conflict when eligible shares differ and the shared true positive rate exceeds the shared false positive rate
- Describe ethical responsibility, explanations, challenge routes and meaningful human review, while distinguishing these standards from legal rights that vary by place and use
1 · Read
Regression checks numerical predictions with residuals. Here we audit loan approvals, rejections and their errors by group. Models trained to imitate past decisions may repeat unfair patterns. Redlining is discriminatory restriction of loans or insurance by neighbourhood, often excluding racial or ethnic minority communities. Location can carry group information even after group labels are removed, so removing a label alone may leave bias. Earlier approval does not prove eligibility.
In an invented audit, a model gets 800 of 800 cases correct in group A and 100 of 200 in group B. Overall accuracy is (800 + 100) / 1,000 = 90%. Group accuracies are 100% and 50%. Divide each group's correct cases by its own total. The larger group dominates the overall score. Also separate incorrect approvals from incorrect rejections, since they can cause different harms.
For a loan audit, define eligibility on a justified basis. Approval is the positive prediction. True positive rate (t) is approvals among eligible applicants divided by all eligible applicants. False positive rate (f) is approvals among ineligible applicants divided by all ineligible applicants. Demographic parity requires equal overall approval rates across groups. Equalized odds requires equal t and equal f across groups.
Let p be the eligible share, t the true positive rate and f the false positive rate. Approval share is tp + f(1 - p) = f + (t - f)p; tp means t times p. Equalized odds requires shared t and f. If p differs and t > f, approval shares differ, so demographic parity (equal approval shares) fails. For p = 0.2 and 0.6, t = 0.8 and f = 0.1 give 24% and 52%. Equal p or shared t = f allows both criteria. Neither settles every fairness question.
Accountability means a named person or team remains responsible for decisions. An audit checks evidence against stated rules; it does not prove every decision is fair. As an ethical standard, affected people should get an understandable reason, a way to challenge it and human review with authority to change it. Legal rights vary by place and use. Report the fairness criteria chosen, the measured rates and any criteria not met.
Use each rate's correct denominator. Name the fairness criteria and their limits. Keep people responsible, with explanations and meaningful review.
Lesson, questions and study cards written for this dataset. Audit numbers and lending scenarios are hypothetical. Background study included the supplied textbook records on accountability, redlining, stakeholders and ethical values.
Readings and study cards written for this dataset. Background study included textbook material on accountability, redlining, stakeholders and ethical values. Scenarios and numerical examples are hypothetical.
More to read
Keep the evidence behind an audit
Start an audit with a question that can be checked. For a loan model, one question is whether eligible applicants are approved at the same rate across groups. Another asks how often ineligible applicants are approved. These questions use different denominators.
Keep the counts behind each percentage. Ten errors among one hundred cases give a 10% error rate. Twenty errors among one thousand cases give a 2% rate. More errors in number can mean fewer errors as a share of cases.
Record the model version, evaluation data, group sizes and definitions used. Explain how eligibility was established. Historical approvals may contain discrimination, so agreement with those decisions is not automatically a sound target.
An audit compares evidence with stated requirements. It can reveal a gap and help check a repair. It does not establish that every decision is fair or that future cases will match the measured results.
A group label can be absent while its pattern remains
Imagine a training file containing addresses and past loan decisions. Some neighbourhoods were unfairly excluded from lending. A model trained to imitate those decisions may learn to associate an address with rejection.
The file need not name a racial or ethnic group for address information to carry information about group membership. A feature used in place of, or carrying information about, another characteristic is called a proxy. Removing an explicit group label may therefore leave a route for repeating discrimination.
This invented example illustrates a risk. It does not mean every use of location is discriminatory. Nor does a measured difference between groups identify its cause by itself.
Investigate the training target, the features and the decisions produced. Ask whether the reference labels are justified. A record that someone was rejected tells you what happened; it does not by itself establish that the person was ineligible.
Make a challenge capable of changing a decision
People affected by a decision have a stake in how it is made. Responsibility remains with the organization and its people when software recommends or makes that decision.
Consider an invented applicant whose income was entered incorrectly. A useful explanation helps the applicant understand why the application was rejected and identify the data problem. A complaint form achieves little if every reply repeats the same prediction.
Meaningful human review needs relevant evidence and authority to correct the decision when appropriate. That might include correcting the income record and reconsidering the application. It does not mean promising approval before examining the case.
This lesson presents understandable reasons, challenge routes and meaningful review as ethical standards. Their legal status varies by location and application. Ethical discussion and checks of applicable law are separate tasks; neither should be substituted for the other.
Check the conditions behind a fairness conflict
Demographic parity compares the overall approval rates of groups. Equalized odds compares approval rates separately among eligible and ineligible applicants. Matching those two conditional rates does not always match the overall rates.
Let p be the eligible share, t the true positive rate and f the false positive rate. Approvals among eligible applicants contribute tp of the whole group. Approvals among ineligible applicants contribute f(1 - p). Adding them gives the approval share: tp + f(1 - p) = f + (t - f)p.
Under equalized odds, groups share t and f. If their eligible shares differ and t > f, their approval shares differ. Demographic parity is then not met. The conflict depends on these conditions, rather than on the claim that every fairness rule always clashes with every other rule.
If eligible shares are equal, the two criteria can coexist. They can also coexist when shared t = f. In that case the approval share equals f regardless of eligibility share, and the positive prediction rate does not distinguish eligible from ineligible applicants.
Neither criterion checks everything. The eligibility definition may still need scrutiny, and affected people may still need explanations and review.
2 · Watch
به انگلیسیTake it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.