notenchange Sincerely, did anyone that knows how LLMs are trained
expected otherwise?LLMs are reflexion of their training
data, and their training data reflects mostly the current
societal norms, with a bias (not sure how heavy) to the
societal norms supported by the ones involved in training
the LLMs.What's interesting for me to see, is that Chinese
models are as biased - if not more - than ChatGPT. This is
probably because a big part of their training comes from
distilation of US models.It would be interesting to see
Anthropic, Mistral and Grok in the comparison as well.
|
> rustyhancock I think the unstated part of your comment is important
- increasingly acceptable especially in professional
settings societal norms are female orientated.I don't
think this is particularly malign simply the net
effect of complaint-adverse working cultures.The net
effect is that the accepted/promoted guidelines that
get brought into the training data is a rejection of
toxic masculinity and becomes "harsher" to men.
|
> > boelboel Remember noticing in college and late high school
that women were the better students (by a large
margin when it came to group projects where form
made up for +-50% of the grade). Whenever a sort
of competition came up though men would rank at
the top. Who's to say that the form portion of an
assignment should account for that much of the
grade.
|
> > znpy > especially in professional settings societal
norms are female orientated.I've been noticing
this for a few years now.Particularly when it
comes to managing issues, a lot more "work" is
done on handling conflicts and feelings rather
than addressing the actual issue at the root
cause.I know it sounds like a meme, but I've
personally observed a bunch of colleagues and
sometimes hr involved as well having meetings on
meetings on how some system not working correctly
and how that imposes a toll on work/life balance,
mental health and stuff... rather than
(trivially?) asking: "alright, why isn't this
working and how do we fix this?".
|
> > > rustyhancock I've seen this too, sometimes as a follow up
meeting to address feelings about an issue
that was resolved that few people go
to.Specifically the people who go are those
most emotionally affected and unlikely to be
men.An effect of this is having meetings to
address feelings scheduled before
troubleshooting sessions! Opening with things
like this is not a "fix it" meeting. Quite
surreal!
|
> > > ramblerman Fwiw, I see the same behavior in male
colleagues <30y old at my new place.I've had
to learn to manage the emotional state, and
then the issue. So this might also be a
generational / gender thing
|
> > cyanydeez I think it has more to do with the "patriarchy"
than some other sinister "liberal" plot.If you're
going to argue men are suppose to be in charge of
everything, of course the poor male beggar is
going to get more blame for their
circumstances.This stuff isn't hard to understand.
|
> thenthenthen There is a meme that Deepseek tells woman to break up
with their boy friends, and when asked the same to
males, it tells them not to break up
|
> > Planktonne Who spreads the meme, and how did this paper
control for their influence?
|
> > mft_ Trained on /r/relationship_advice then.
|
> gmbhgmbhgmbh I don't see how you can conclude that Chinese models
as a whole are as biased or more biased than chatgpt
from the results of the study. Some Chinese models
were less biased than chatgpt, some models were more
biased. The newer Chinese models in this study seemed
to be amongst the less biased models in this study.
The gemini models here seemed to be the more biased
models in this study.
|
> > notenchange It;'s right there on the website with the code
used for the paper:
https://github.com/xufeiqiong/GAMA-BenchIn order
of bias favorable to women:1.
Doubao-Seed-2.0-Pro2. Qwen33. Gemini-2.5-Pro4.
Gemini-3-Pro5. DeepSeek-V4-Pro
|
> > > gmbhgmbhgmbh I don't see the specific metrics or variables
that you pulled from the study to create that
list? The "bias favorable to women" doesn't
even exist as a variable or metric in the link
that you provided or the published paper.
|
> > > > notenchange It's the `Emp.-Agg. Δ`. Results for both
the `Intimate Track` and `Public Track`
are mostly the same. The orange marking
(that ends up showing up on every single
one of them), indicates `stronger framing
toward female actors`.
|
> stymaar I don't think anyone expected otherwise, but it's good
to have this kind of benchmarks since model makers can
now correct these bias during post-training and
measure how well they do.Not unlike how the ARC-AGI
benchmark family helped improve reasoning.
|
> > benterix Not sure if it's sarcasm or not, but if you are
being serious, there is no way this would ever
happen. There is no body who would take care,
nobody who would support it. The opposite could
even be true: if someone cared enough to come up
with such an initiative, they might get heavily
criticized.
|
> > catlifeonmars > correct these biasAre you assuming that there is
some correct bias here? Wouldn't that be highly
dependent on the cultural and personal values of
the users?
|
> Lerc I did not expect otherwise based upon model
pre-training.Later training can be quite
significant.I'd like to see more benchmarks in this
area, it is one thing to expect things to be like
that, it's another to actually measure it.Measuring it
tells you something about the scale, and gives you a
baseline for improvement.
|
> RAWGOD >LLMs are reflexion of their training dataPeople don't
understand machine learning. People seem to believe
that with enough training data and parameters, a
sudden switch happens that ignites a spark of pure
intellect and turns a pile of matrices into a magic
knowledge machine.Unfortunately (or perhaps
fortunately) our world is not magical, there are no
Platonic Forms, and no Infinite Wisdom spells.
|
> riedel What one can theoretically do is to carefully remove
gender signals from training data in a processing run.
My guess is that you would not lose all to much
knowledge, however you might lose the ability to
recall facts about fictional and non-fictional
persons. Wonder if anyone would go through that hassle
though.
|
> > Planktonne Gender signals are baked deeply into our culture
and language. It's not just the direct mentions,
but the way people are described, the roles they
take, the positioning of actors in a complex
space."Remove gender signals carefully" is an
incredibly difficult challenge.
|
> > > turbo-outrun >"Remove gender signals carefully" is an
incredibly difficult challenge.Not to mention
that, given that gender-signals have been
around since the dawn of human history, you
would remove them in a small test-region
first; for safety reasons.Assess the social
consequences in a small region. Did it improve
things?Universal instantaneous-removal of any
major socio-cognitive-structure is akin to
vandalism because it might turn out to be,
architecturally, vital.
|
> > > > Planktonne We already know what happens to some
extent; think about gender-non-conforming
people. That's a (relatively small)
personal blurring of signals that
theoretically doesn't affect much outside
the person themselves.Yet it's been a
mainstay of our politics for the last few
decades, and we've seen it used as a
weapon for centuries. Our societies have
gender embedded very, very deep.
|
> > > > > riedel The question if we can or even want to
try. To me it is is rather a thought
experiment than anything realistic
(particularly it would cost quite a
lot of money with very uncertain
outcome). Speaking of embedded: an
alternative approach would be the
opposite to contrastive learning: put
some losses into the system that
encourage gender concepts to stay more
close. The political perspective is
totally different: there is always a
difference what a society wants to do
or what generally could be done.
|
> > > > > graemep its political because it does affect
much outside the person themselves.
Most of the argument is about other
people's use of language, about
systems. Some of the most hotly
debated topics here in the UK are
around health (e.g. should everyone be
asked whether they might be pregnant
if women are, should I have got a
letter asking whether I wanted a
cervical smear) and government systems
(what are the criteria for changing
passport details? should a transwoman
criminal go to a male or female
prison?).Gender is so deeply embedded
that it is impossible for it to be
just personal. It ceases to be
political where there is a social
consensus, which is true for most
societies most of the time (not
necessarily the same consensus), but
not true for western society right
now.
|
> > > > > > Planktonne That's what I'm saying; even a
relatively small change (short
hair on a woman, for example) can
kick off a lot of responses, and
ripples only spread further as
people "deviate" further from the
norms.The idea that you can
measure cultural bias purely by
switching pronouns/nouns in some
sentences is incredibly
short-sighted; we know it doesn't
work like that.
|
> > > > > > turbo-outrun > Most of the argument is about
other people's use of
language.Bingo. Because, at a
State-level, control-of-speech is
the dictatorial
dream-ticket.Government was
supposed to be
facilities-management (power and
water), but it's morphed into
top-down social-engineering
projects that are rolled out
across all 'live servers' without
any small-scale testing.
|
> > > > > turbo-outrun > Yet it's been a mainstay of our
politics for the last few
decades...The 1980s definitely felt a
lot more fun to navigate, so I'm
considering a roll-back on some more
recent features...
|
> > > riedel I would guess one could at least give
adversarial signals like changing names in
literature. Reading a lot of children books at
the moment I feel it would be not so difficult
disassociating social roles from gender.
Wonder however also how much prompt rewriting
could do: similarly to anonzmized CVs. Problem
however is probably deeper: certain
biographies will keep being discriminated.
|
> > > > Planktonne You could absolutely muddy the waters, but
that's a first step and far more easily
disregarded in context.Modern children's
books are much better about this, but
consider all of the fairy tales; you need
to swap not just names but descriptions
and roles, and then you need to rework the
way the characters respond to those roles
before you have a hope of disguising the
original genders.
|
> > > > > turbo-outrun > you need to rework the way the
characters respond to those roles
before you have a hope of disguising
the original genders.Absolutely.
There's the additional problem that
English is unusually flexible: Many
other human languages collapse when
grammatically stripped of gender.Try
de-genderizing Polish; Arabic;
Russian; Zulu, etc. etc.The current
ongoing-battle around
gender-deconstruction could be framed
as: All-Of-Human-History vs
The-Last-Thirty-Years-Of-English-Speak
ing-Politicians. Who will be the
winner?
|
> > catlifeonmars I said this elsewhere but what's appropriate is
going to vary quite a bit based on the cultural
and personal values of the users of the model.
Having a model with one particular set of
standards is likely to leave a lot of people out.
This isn't something that you can just erase and
assume that what you replace it with will be
categorically better
|
> > throw9398922 Let's keep gender roles, but we should respects
people preferences.Like I am total pussy, I
identify as a woman! Do not expect me to sacrifice
myself for some random people, just because I
happen to have a penis!
|
zdc1 I'd be curious about how the bias looks in the context of
positive situations. Is the bias that LLMs hold men more
directly accountable for any situation, and thus both
punish negative and praise positive outcomes more
strongly? Or is this bias only observable in negative
situations?To be clear: it's bad either way, since women
should be seen as having the same levels of agency and
accountability as men, but I am curious nonetheless...
|
stephbook Wasn't there a recent article on HN regarding race,
gender, age and nationality bias in LLMs? I think I read
it before it was flagged.The results were similar.
|
zug_zug I did a similar microstudy on this:
https://think-twice.me/is-ai-biased-or-sexist-making-chatg
pt...
|
> Bolwin There's a mistake in your prompt. I hope you didn't
run them all with "she"
|
macic I'm not surprised, but I don't believe it's merely a
result of the training data. I would expect the alignment
phase to have a significant impact on it as well.
|
goodcake3963 One of the worst societal bias ever
|
jrrc_dev Seems LLMs copy human behaviour pretty well
|
cyanydeez These people know they're doing anthropology right,
sociology. LLMs are a raw dump of western civilization.
|
> maxldn Not exactly raw because of the fine tuning they do to
try and avoid bias (but in a biased way). And not
exactly civilization so much as what is readily
available online, which is in its self biased in many
ways, e.g toward flame wars, via troll farms, etc.I
saw a paper on here a while back that said more or
less the same thing, but didn't frame it in a
bro-friendly manner so it got flagged into oblivion
|
> > cyanydeez Your realize all those caveats exist in every type
of anthropolgy class?
|
> inglor_cz Now imagine how the LLMs would look like if they were
a raw dump of the Islamic civilization instead.I
suspect we will soon know. This won't stay a
Western/Chinese technology forever and there will
eventually be some robotic fatwa issuer based in
Tehran or Kabul.
|
> > Bolwin What is "The" Islamic civilization?Anyway, feel
free to try the K2 horizon models, which are by a
university in the UAE, and see how they hold up to
your claims
|
radiospiel Poor males.Now, seriously: studies like this tend to
ignore that every interaction between people is shaped by
societal context. The same behaviour can mean something
different depending on who does it. This is not double
standard; what reads as patriarchal dominance from a man
might reasonably read as assertiveness from a woman,
because they're acting from different positions of power.
Asking a model to grade isolated interactions at this
small scale, stripped of that context, won't tell you
anything useful.Personally, I'm glad LLMs still account
for the reality of male privilege.(context:
https://github.com/xufeiqiong/GAMA-Bench/blob/main/figures
/o... from the linked repo)BTW: I am not saying that the
example response from the female in this example is
acceptable, just saying that this "study" is basically
pointless.
|
> danielxt > basically pointlessI found it very interesting, not
sure why you think it's pointless - I think it
highlights something interesting in society
|
> > cat-snatcher They didn't like the implication of the study and
want to keep believing that men are wrong and
women are right even when they display the exact
same behavior because of patriarchy.
|
> > > jadamson I doubt they truly believe it - much of this
kind of male feminism is performative and
rooted in insecurity.
|
> jadamson To be clear, you read your own link and think that a
man locking an irresponsible woman out of a system is
punitive, but a woman locking an irresponsible man out
of a system isn't?What is the actual reality, and why
wouldn't you want LLMs to reflect it?
|