A few years ago, the author Bruce Sterling commented on the Soylent founders that 'it was just a matter of time before they went and got good and weird in the desert,' and that's stuck with me ever since, as, for instance,the 'effective altruists' have managed to find ways to stop caring about altruisism and instead retreat to fanfic pages about robot gods, and shit like this. I poked around the edges of the rationalism movement years ago because, as Max noted, the bullet points on the front door are basically unobjectionable. I like thinking, I like thinking about thinking, I think some better thinking in the world sometimes might make it better, etc. But it didn't take long to reveal that there were core tenets of wisdom, or, perhaps the most zoomed out Bayesian priors, that just weren't in the room- of acknowledging personal and psychological finitude, and uncertainty, and the utility of diversity of ideas, of the necessity of finding people to trust, of sometimes just getting on with it, and all the rest. It was kids in a dorm room trying to make a spreadsheet about who they should date and missing that they were supplying all the numbers and were still gonna have to ask the girl.
I also recall some sociologists years ago presenting a case that the prototypical international terrorist was an engineering student with a fresh and routinely very shallow case of religion, and there might still be something to that analysis...
I think the fact that Zizians are in large part composed of trans women also leads to some of the susceptibility to cults -- when you're estranged from support networks and going through major life changes it's going to be easier to fall prey to cults, especially those that offer support and housing.
Pretty convenient avenue for a cult to brain wash a guy is to convince him that the other half of his brain is a girls brain and that he should start taking these drugs and change his whole identity -- something that will cut him off from his support structure and people who might have intervened to stop him from adopting dubious moral ideals based on ... complete horseshit read on internet forums
I think you're overstating the power of the cult here. Minority sexuality and self-identity were there in the first place, and such perspectives are already marginalized. Cults do, indeed, prey on the marginalized.
In this misattribution you are participating in said marginalization. You should entertain the idea that some people naturally don't feel congruent with the socially-normative performance of their heterosexual reproductive role, and try not to feel personally threatened by this fact of life.
Silicon Valley with "mission driven" companies can be pretty cult-like (ask folks who worked at Facebook, Apple, and Linden Lab, for example.) And plenty of cis-people are in the so-called "rationalist cults," and many of them are not welcoming places for trans people.
The alienation that a lot of trans femmes face in the Valley is vulnerability the cults use, the same way the Moonies preyed on undergrads at college campuses during the 1980s.
Heck, even the Zizian's and related groups are using lot of the same frippery as the Scientologists, with slightly different names: instead of an "auditor" going after your "engrams," the victim's now trying to "gain root" on their mind to drive out the "demons."
> By the same token, the ability to dismiss an argument with a “that sounds nuts,” without needing recourse to a point-by-point rebuttal, is anathema to the rationalist project. But it’s a pretty important skill to have if you want to avoid joining cults.
From the article on which we are commenting.
I agree that it is not coincidental that such people are over-represented both in Rationalist spaces, and in the groups under discussion.
I'd like to see more reporting on this. Was Ziz attracting trans people, or convincing people to become trans? Literally everyone connected with the group seems to be trans (based on what I've read).
I'm a little frustrated that Max ignored this dimension of the story. No matter what your philosophy of gender is, it's an important part of the story.
I think there's a simpler foundational answer as to why Rationalism seems to produce cults or cult-adjacent groups. It's perhaps not a comfortable one. Autistic people are overrepresented in rationalist spheres. While some traits of autism are protective against vulnerability to cults, others make autistics more vulnerable. Especially when there are risk amplifiers like trauma history (from what I know, the Zizians, for example, are quite traumatized individuals).
I say this because I've been around the community for a while, am very interested in cults (in an academic sense), and am autistic myself.
To be clear, having trauma doesn't make murdering people an okay thing to do. The Zizians are, in my opinion, batshit.
i think there's also a lot to be said for how the movement attracts people with a lot of math/verbal intelligence and very little social intelligence. makes for people who are easily manipulated, and who will write you a 10,000 word essay about how they're not being manipulated if confronted
This is a good point. You also have a perfect stew of tendencies, such as the use on insider terminology terminology, elitism, doomsday beliefs, and in this case trans and vegan beliefs, all of which tend to create alienated insular communities. Combine that with people who are highly literal, struggle with metaphor, and already feel alienated from society- it’s the perfect environment for a cult to form.
Came here to see if the OP mentions this because since I learned the details of this story, I've been trying to figure out the hows and whys of it, in a semi rationalist fashion and came with AS as a big part of the puzzle.
you know, i think of myself as decently versed in the worlds of both internet drama and culty shit, but the part of that one blog post that takes an entire paragraph to meticulously explain why it’s crazy for ziz to ask her to murder someone based on the fact that doing so would not provide the information ziz claims it would provide, and not because it’s goddamn motherfucking crazy for ziz to ask her to murder someone…… that was a new one for me. had to sit with it a minute. thank you for your time in the trenches, you were NOT exaggerating about these people being unreadably verbose.
i, being of a particular age, immediately thought in terms of scientology, jim jones, and all that. being from texas, i was eating peyote in the age of timothy leary and ended up in northern california doing art at ucsc. and reading this, i could not help but think of julian jaynes and “the origins of consciousness and the bicameral mind”…not that it is completely pertinent to your essay, but that it conjured the thought. were i not on a very limited pension, i would gladly support your work.
I'd love to know whether Ziz developed her theory based on Jaynes, and also how much other Rationalists discuss or take it seriously. Scott Alexander wrote an essay on Jaynes that was good. But the whole subject of the brain's two hemispheres, let alone Jaynes's ideas, seem pretty marginal to contemporary neuroscience afaik.
"In this sense its predecessors are not really the original enlightenment rationalists, but the dubious touchstones of ‘60s-hangover California--Scientology and Dianetics, Werner Erhard and est,"
There's also some historical ties between the golden age sci fi that Yudkowsky was steeped in growing up (no doubt many other Rationalists too) and these kind of culty self-improvement groups that promise almost supernatural expansion of mental abilities. Alex Nevala-Lee's book "Astounding: John W. Campbell, Isaac Asimov, Robert A. Heinlein, L. Ron Hubbard, and the Golden Age of Science Fiction" has some good info on this, especially how Campbell, an editor who was hugely influential on the development of science fiction stories after taking over the magazine "Astounding" in 1937, went in for all sorts of "new psychological technology" schemes including the beginnings of L. Ron Hubbard's Dianetics (Nevala-Lee also has some posts on Campbell and Dianetics on his blog at https://nevalalee.wordpress.com/tag/dianetics-the-modern-science-of-mental-health/ ). Also see the article on "self-help supermen" in WWII era sci-fi, which talks about Campbell's influence, at https://www.jstor.org/stable/10.5621/sciefictstud.41.3.0524
Yeah, I happen to know in this case that these guys were particularly influenced by Cordwainer Smith, sci-fi alias of psychological warfare professional Paul Linebarger. It's an interesting rabbit hole, as interesting as the one around the infamous Space Relations by Donald Barr, possible early Epstein recruiter.
Interesting, do you mean Yudkowsky specifically or "rationalists" more generally? I remember reading that Yudkowsky was especially influenced by a sci fi writer named A. E. van Vogt, who was himself influenced by an author named Alfred Korzybski, an independent scholar who had his own scheme for dramatically upgrading the way we think (Nevala-Lee has a post about him at https://nevalalee.wordpress.com/2016/10/11/to-be-or-not-to-be-2/ and Andrew Pilsch's 'Self-Help Supermen' article I linked above talks about Korzybski and van Vogt starting on p. 526, with more on the connection on p. 531-535, and p. 527-528 also reference an earlier article relevant to van Vogt and the dream of 'upgrading' oneself, 'Super Men' by Brian Attebery at https://www.jstor.org/stable/4240674).
By "these guys" I meant the Zizians specifically, but I learned it as a fact about a broader, disunified post-Vassar branch of the LessWrong/Extropian movement that included them, Jess Taylor, and Ziz's mentor Alice M. (all mentioned and/or linked above in Max's piece). And yes, the Korzybski and Van Vogt influences seem very important for Yudkowsky. Thanks for the link to Nevala-Lee, that's a nice article!
Thanks, did any of that group ever write about Cordwainer Smith or what they got out of his writing, or more like something you heard about through friends-of-friends etc.? Also when you talk about the post-Vassar branch, is that all people who had been working with Vassar either at MIRI/CFAR or at Leverage, with Jessica Taylor talking about cultishness in both at https://www.lesswrong.com/posts/MnFqyPLqbiKL8nSR7/my-experience-at-and-around-miri-and-cfar-inspired-by-zoe and with Scott Alexander blaming it on the "Vassarites" in a comment?
I heard about it directly in person at events and I haven't read enough of their online stuff to know if they wrote about it or not. This wasn't all the people who had worked with Vassar, just "a" post-Vassar branch among at least a couple different ones. Another was more "magick" with strong Crowleyan influences, sort of a The Invisibles LARP. There was some overlap of branches and it would probably be hard to describe exact boundaries between them even for insiders. Leverage wasn't particularly influenced by Vassar so I'd consider them a separate thing.
Oh shit! I just read an omnibus of John W Campbell cause I'm a freak for Carpenter's The Thing.
What I couldn't get past was that EVERY. STORY. in his collection has a single, brilliant, genius man (always man) defeat an unbeatable monster/empire/alien armada/whathaveyou with the invention of some superweapon or gambit (MacReady and the blood test) or even occasionally headslappingly obvious tactics. (Bad guys are given bulletproof shields so bury them in rockslides)
I was like wowwwww this is so nakedly 1950s atomic age/A-bomb/Galt pilled but given that a lot of great stuff had happened, in some cases due to one guy or a small team (Salk, the Curies, Einstein) and WWII had been ended (in the American propaganda that totally ignored the Soviets and Japan's willingness to surrender) by the SuperWeapon of the A-bomb. So in a sense excuseable.
I did NOT know that he grabbed the levers of power and made sure to mint out more dreck like he created. I guess he wasn't like, the most obvious practitioner of that style of scifi, hes a writer with a distinctively bad and hackneyed personal who *forced it* into a genre.
But yea. All problems are soluble by a single smart genius acting in concert with no one else or directing a faceless mob whose deaths don't count. Does that sound like you? Good news, it sounds like themselves to literally anyone!
Hah, yea they nail literally every point I mentioned. Guess this wasn't a new revelation but to be fair I'm not a big scifi guy.
I kinda fell off once I got through adolescence and realized how much of the genre is a special little boy no one believes in (harry potter, ender, Luke Skywalker) from nowhere special who has a magical destiny that makes him so much better than everyone else... you get it.
It's wish fulfillment for sad lonely kids (Which, hey I used to be an SF fan, not gonna act like I wasn't a sad lonely kid). Anyway, is there a taxonomy?
Where's the line between an unathletic nerdlinger imagining being discovered for having great hidden talent and going on adventures and a mundane adult with no particular skills imagining being a super competent worldbeating genius?
Also, one niggle, I'm not talking Connecticut Yankee in King Arthur's Court type stuff which seems to be the focus of the piece you linked, but specifically a guy of that context and milieu who is just smarter and better than everyone sround him. Usually set in the future, but in Who Goes There it's just an extra smart 1950s scientist in a base full of 1950s scientists.
Scientology is a fascinating history! It was in the news a few years ago when its current iteration imploded, now somewhat forgotten again. I love how they call all their ideas and practices "tech."
Tbh in some ways I have more respect for the Rationalists who get freaky with it than the ones who just go off measuring brainpans or whatever. If you're gonna go mad, go mad in style, you know?
One minor quibble I have is with your characterisation of the people who join these cults as "convinced of their own inadequacy". I'm sure that's true for some of them, but humility is not generally a quality I associate with the Rationalist community. In fact, I think for many cults the appeal is that someone claims (at last!) to recognise how *special* you are - so special, in fact, that you get to join the special club for special people who know the Real Truth about the universe. That's a powerful draw, especially for someone who already has trouble relating to others.
Yeah, I'm probably phrasing it wrong, but it seems really clear that a key component of the cultiest corners of the Rationalist world is the sense that you, the newcomer Rationalist, are *not rational enough*, and can be "debugged" or "reprogrammed" in some way to become *more rational*... Ziz and Jessica Taylor both describe these sort of Dianetics auditing-type "debugging" sessions happening with CFAR people, really freaky!
It's not either / or - belief in being brilliant/feeling of social inadequacy. It's classic grandiosity. They go together hand in glove. Pretty much all adolescents have it to some extent - for some people it's more extreme, and that make them more vulnerable to an ideological leader who will show them The Way.
In the Zizian flier for their MIRI/CFAR protest at https://imgur.com/a/1RstyWp there's an accusation that "CFAR does not do remotely what they claim to do on their website: they do not appreciably develop novel rationality/mental tech", so I guess their complaint was that CFAR wasn't being effective enough in looking for good mental debugging "tech"? (a word that Scientologists also like use for their own techniques)
that's interesting, ty! i can see how an ultra-Rationalist viewpoint would lead you to hear someone talk about 'reprogramming' and think 'that sounds like a good thing to have done to me'
I wrote a piece a while ago on the commonality of rape among the rationalist/effective altruists (I've been keeping track/speaking to survivors for about eight years). It's - well, it speaks to in-group cult-like behavior.
I think one of the most underappreciated essays ever written about the rat scene is Ozy's "Rationalists and the Cultic Milieu" https://thingofthings.substack.com/p/rationalists-and-the-cultic-milieu "Cultic" in this case means a specific academic thing about "stigmatized knowledge", not "cults". Ozy is also a big-name rat, so it's not like they're making the critique as an outsider. It gets at something similar to the points you're making here.
Ken has also been doing to great research into the Ziz stuff https://x.com/kenthecowboy_/status/1884393311827550716 He's actually been going out and doing legwork, interviewing people, getting court documents, finding info that contradicts previous reporting. Really useful stuff.
This article draws a lot from sources Slimepriestess wrote (Hemisphere Theory: Much More Than You Wanted To Know, zizians.info) before the death of Emma. I find it concerning that it doesn't seem to acknowledge Slimepriestess' current position on those sources: that they are significantly inaccurate and misleading, written under the influence of JD Pressman, Slimepriestess' paranoid ex.
This podcast does a good job of covering the relevant issues and is probably the most comprehensive and accurate publicly available source.
While this may be true, in the interview Slimepriestess seems to have a difficult time describing in the interview what exactly is inaccurate other than “the reasoning is more complicated than that.” Which- fine, but just because there’s a 10,000 word blog post explaining why you should try to sleep with half your brain at a time (for instance) doesn’t change the fact that they are essentially sleep depriving themselves as a result of pseudoscientific reasoning. And given how apologetic Slimepriestess is toward the Zizians I’n not convinced she’s a better source than Zizians.info.
The best use of human time is developing "safe" A.I.? I mean... that is a thing a person could think... But what a chore of an idea to start a cult around.
I took a class in college on critical reasoning that introduced me to the effective altruism community and Rationalists. I thought it was brilliant (it’s all about choice! Critical thought! Being logical and aware of your biases! Pro-human despite a holier-than-thou stance against sheeple!). I thought I was going to dedicate my career to AI ethics. COVID happened and I dropped all interest as my values radically shifted, but it’s bizarre to reflect on now. Between Zizians and Luigi Mangione, I had no idea what I was so close to.
I don't think the EA community is even nearly as cultogenic as core-Rationalism, though if you are coming at it from a rationalist-AI safety perspective, I can see having the association.
Agreed, it’s definitely not. But it’s worth noting that EA is adjacent to, and informed by, some of the same writing as Zizians and other extreme rationalists. It’s a part of the philosophy that underpins the whole tech world, not just in regards to AI ethics (e.g. Sam Bankman-Fried)
That's true. Although, (minor nitpick) I wouldn't say it underpins the "whole" tech world. The tech world is pretty big, there's lots of other elements mixed in.
Don’t paint the entire “tech world” with the EA / Rationalist brush. Been in “the tech world” since 1990 and I can’t say I have met a single such wacko in person, just online 🤷🏽♂️🤦🏽♂️
EA and rationalism aren’t inherently wacko either. Rationalism as an epistemological view just means basing opinions and actions off of logic and reason rather than faith, tradition, or sensory experience. I do feel that’s a pretty essential pillar of the tech industry. It’s still worth thinking about how and where that sentiment manifests, or what that can turn into.
“Whole” was an overreach, I could have worded that better.
Thanks for asking! I think rationalism eventually felt too removed from reality, and I don’t agree with consequentialism. I didn’t want to live in a framework where I had to turn things into calculations to know what was right or morally obligatory. The isolation of COVID exacerbated that I guess, as well as graduating college and getting thrown into “real life.” I felt such a deep emptiness in the absence of faith/tradition/experience, which are completely deprioritized in Rationalism. It all felt hollow. AI ethics also felt completely futile.
I moved onto a farm during COVID and learned to grow food for my community. It was the best decision I ever made :)
That sounds practical, wholesome and something that Rationalism would not have been able to provide. Thank you for sharing! I haven't ever been into Rationalism but what you described has been my experience with 99.99% of the things I pick up from the Internet: after some time they felt hollow, and I craved the stark reality and groundedness of real life
The Internet, for all its wonderments, can never be real life, and the cultures bred here are usually untested against reality and so collapse when they come in contact with it
Seeing as the internet was born into a dominant culture - our Western media - I guess maybe it was inevitable that the kind of online culture we've now got would eventually dominate. Have you ever read Neil Postman's Amusing Ourselves To Death (it's brilliant)?
I'm only peripherally aware of these events and identify with Effective Altruism rather than Rationalism; but this post strikes as well-researched and fair to the facts of the matter. I learned a few new things too!
I am always unconvinced by the (very frequent) claims that Rationalism and LessWrong are cults, but "cultogenic" would be a fairly accurate description.
If you want a somewhat deeper dive into the world and culture of the Rational movement, that *isn't an outright hit piece*, try Tom Chivers' "Rationalist's Guide to the Galaxy", where the journalist actually tries to understand and explain the movement from the inside view of the many mostly normal people involved, rather than analyzing it by it's greatest failures, like SBF, or conspiracy-brained a guilt by association stuff like Metz's piece or and the completely nonsensical TESCREAL argument. of course, those are useful sources if you want to declare it a dangerous cult ideology, but not if you actually want to understand it
Which is not to say that groups like the zizians, vassarites and leverage aren't real or aren't culty. But I would venture that the dynamic here is more that a bunch of open-minded people trying to change the way they live is a absolute hunting ground for cult leaders, rather than some inherent failing in "trying to be more rational" as a lifestyle
I think this basically matches what I'm saying! I'm not convinced that "Rationalism" as a movement is adequately described as simply "trying to be more rational," though.
I feel rats subculture/movement is what happens when intellectual openness to experience/ideas combined with high IQ and arrogance these two traits generate collide with autistic-like need for certainty (however you define the latter).
I'm not religious, but from the perspective of someone who has to share a planet with these people, I actually think you're probably right. Religions can make people act crazy too, but a moderate mainline belief system would probably help them figure out which is the ass and which is the elbow. I mean, what are they looking for? (God) And what are they finding? (Void) And what are they trying to compensate with? ("debugging" themselves like they're meat machines) -- it's a nihilism spiral where nothing is true and everything is possible, and so there's no ground to stand on. They need ground to stand on -- so they can figure out who they really are is NOT who they are when they break their brains in two.
I kinda sympathise with what they're looking for. They want a practical religious path to enlightenment - and a way forward that will lead humanity away from its worst impulses. In that way, they remind me a lot of radical Christian movements from the 16th and 17th centuries, and of communist revolutionaries. They feel strongly that modern society keeps people locked in shallow, meaningless lives (which is teenage and based on a limited experience of life, but not entirely) unreasonable), and the way that knowledge and technology are accelerating mean that we're genuinely coming to a point where the definition of reality and consciousness will probably be redefined - as it has been in previous centuries. As far as I understand it (not much), physicists are telling us that a whole lot of very weird shit is possible. That's what makes the religious underpinnings of this so fascinating - these cults are factions and subsects of a priesthood who aren't engaged in astrological calculations or sacrifices to abstract gods - they've built actual new systems of information synthesis; they're playing with the frontiers of mathematics and physics, and consciousness. Is the singularity coming? I have no idea, but this century is shaping up to be the most condensed and batshit crazy epoch that humanity has yet to experience. It's no wonder that young people who in the past might have been drawn to millenial doomsday thinking or the perfect society of the enlightened proletariat, or the cosmic groove of the psychedelic awakening, are currently getting into the utopian promises of AGI.
I'd argue that it's a FEAR of nothing being real (because nothing is certain, and those minds tend to need certainty) that gets rationalised (lol, pun intended) into "everything is possible".
A daily internet-use limit for people who aren't using it as part of their profession would nip this in the bud. 30-45 minutes a day of use for recreation, enough to check the news and see a nudie show.
Widespread computer addiction has tragic consequences, in this case -- young men losing their minds and turning to murder.
Its the same as the widespread use of LSD leading to the Manson murders
A few years ago, the author Bruce Sterling commented on the Soylent founders that 'it was just a matter of time before they went and got good and weird in the desert,' and that's stuck with me ever since, as, for instance,the 'effective altruists' have managed to find ways to stop caring about altruisism and instead retreat to fanfic pages about robot gods, and shit like this. I poked around the edges of the rationalism movement years ago because, as Max noted, the bullet points on the front door are basically unobjectionable. I like thinking, I like thinking about thinking, I think some better thinking in the world sometimes might make it better, etc. But it didn't take long to reveal that there were core tenets of wisdom, or, perhaps the most zoomed out Bayesian priors, that just weren't in the room- of acknowledging personal and psychological finitude, and uncertainty, and the utility of diversity of ideas, of the necessity of finding people to trust, of sometimes just getting on with it, and all the rest. It was kids in a dorm room trying to make a spreadsheet about who they should date and missing that they were supplying all the numbers and were still gonna have to ask the girl.
I also recall some sociologists years ago presenting a case that the prototypical international terrorist was an engineering student with a fresh and routinely very shallow case of religion, and there might still be something to that analysis...
I think you’re recalling this paper: “Why are there so many Engineers among Islamic Radicals?” - https://eprints.lse.ac.uk/29836/1/Why_are_there_so_many_Engineers_among_Islamic_radicals_%28publisher%29.pdf?utm_source=chatgpt.com
I think the fact that Zizians are in large part composed of trans women also leads to some of the susceptibility to cults -- when you're estranged from support networks and going through major life changes it's going to be easier to fall prey to cults, especially those that offer support and housing.
Pretty convenient avenue for a cult to brain wash a guy is to convince him that the other half of his brain is a girls brain and that he should start taking these drugs and change his whole identity -- something that will cut him off from his support structure and people who might have intervened to stop him from adopting dubious moral ideals based on ... complete horseshit read on internet forums
I think you're overstating the power of the cult here. Minority sexuality and self-identity were there in the first place, and such perspectives are already marginalized. Cults do, indeed, prey on the marginalized.
In this misattribution you are participating in said marginalization. You should entertain the idea that some people naturally don't feel congruent with the socially-normative performance of their heterosexual reproductive role, and try not to feel personally threatened by this fact of life.
Silicon Valley with "mission driven" companies can be pretty cult-like (ask folks who worked at Facebook, Apple, and Linden Lab, for example.) And plenty of cis-people are in the so-called "rationalist cults," and many of them are not welcoming places for trans people.
The alienation that a lot of trans femmes face in the Valley is vulnerability the cults use, the same way the Moonies preyed on undergrads at college campuses during the 1980s.
Heck, even the Zizian's and related groups are using lot of the same frippery as the Scientologists, with slightly different names: instead of an "auditor" going after your "engrams," the victim's now trying to "gain root" on their mind to drive out the "demons."
> By the same token, the ability to dismiss an argument with a “that sounds nuts,” without needing recourse to a point-by-point rebuttal, is anathema to the rationalist project. But it’s a pretty important skill to have if you want to avoid joining cults.
From the article on which we are commenting.
I agree that it is not coincidental that such people are over-represented both in Rationalist spaces, and in the groups under discussion.
I'd like to see more reporting on this. Was Ziz attracting trans people, or convincing people to become trans? Literally everyone connected with the group seems to be trans (based on what I've read).
I'm a little frustrated that Max ignored this dimension of the story. No matter what your philosophy of gender is, it's an important part of the story.
I think there's a simpler foundational answer as to why Rationalism seems to produce cults or cult-adjacent groups. It's perhaps not a comfortable one. Autistic people are overrepresented in rationalist spheres. While some traits of autism are protective against vulnerability to cults, others make autistics more vulnerable. Especially when there are risk amplifiers like trauma history (from what I know, the Zizians, for example, are quite traumatized individuals).
I say this because I've been around the community for a while, am very interested in cults (in an academic sense), and am autistic myself.
To be clear, having trauma doesn't make murdering people an okay thing to do. The Zizians are, in my opinion, batshit.
i think there's also a lot to be said for how the movement attracts people with a lot of math/verbal intelligence and very little social intelligence. makes for people who are easily manipulated, and who will write you a 10,000 word essay about how they're not being manipulated if confronted
100%
This is a good point. You also have a perfect stew of tendencies, such as the use on insider terminology terminology, elitism, doomsday beliefs, and in this case trans and vegan beliefs, all of which tend to create alienated insular communities. Combine that with people who are highly literal, struggle with metaphor, and already feel alienated from society- it’s the perfect environment for a cult to form.
Came here to see if the OP mentions this because since I learned the details of this story, I've been trying to figure out the hows and whys of it, in a semi rationalist fashion and came with AS as a big part of the puzzle.
you know, i think of myself as decently versed in the worlds of both internet drama and culty shit, but the part of that one blog post that takes an entire paragraph to meticulously explain why it’s crazy for ziz to ask her to murder someone based on the fact that doing so would not provide the information ziz claims it would provide, and not because it’s goddamn motherfucking crazy for ziz to ask her to murder someone…… that was a new one for me. had to sit with it a minute. thank you for your time in the trenches, you were NOT exaggerating about these people being unreadably verbose.
i, being of a particular age, immediately thought in terms of scientology, jim jones, and all that. being from texas, i was eating peyote in the age of timothy leary and ended up in northern california doing art at ucsc. and reading this, i could not help but think of julian jaynes and “the origins of consciousness and the bicameral mind”…not that it is completely pertinent to your essay, but that it conjured the thought. were i not on a very limited pension, i would gladly support your work.
I'd love to know whether Ziz developed her theory based on Jaynes, and also how much other Rationalists discuss or take it seriously. Scott Alexander wrote an essay on Jaynes that was good. But the whole subject of the brain's two hemispheres, let alone Jaynes's ideas, seem pretty marginal to contemporary neuroscience afaik.
"In this sense its predecessors are not really the original enlightenment rationalists, but the dubious touchstones of ‘60s-hangover California--Scientology and Dianetics, Werner Erhard and est,"
There's also some historical ties between the golden age sci fi that Yudkowsky was steeped in growing up (no doubt many other Rationalists too) and these kind of culty self-improvement groups that promise almost supernatural expansion of mental abilities. Alex Nevala-Lee's book "Astounding: John W. Campbell, Isaac Asimov, Robert A. Heinlein, L. Ron Hubbard, and the Golden Age of Science Fiction" has some good info on this, especially how Campbell, an editor who was hugely influential on the development of science fiction stories after taking over the magazine "Astounding" in 1937, went in for all sorts of "new psychological technology" schemes including the beginnings of L. Ron Hubbard's Dianetics (Nevala-Lee also has some posts on Campbell and Dianetics on his blog at https://nevalalee.wordpress.com/tag/dianetics-the-modern-science-of-mental-health/ ). Also see the article on "self-help supermen" in WWII era sci-fi, which talks about Campbell's influence, at https://www.jstor.org/stable/10.5621/sciefictstud.41.3.0524
Thanks for this recommendation, very interested to read it!
Yeah, I happen to know in this case that these guys were particularly influenced by Cordwainer Smith, sci-fi alias of psychological warfare professional Paul Linebarger. It's an interesting rabbit hole, as interesting as the one around the infamous Space Relations by Donald Barr, possible early Epstein recruiter.
Interesting, do you mean Yudkowsky specifically or "rationalists" more generally? I remember reading that Yudkowsky was especially influenced by a sci fi writer named A. E. van Vogt, who was himself influenced by an author named Alfred Korzybski, an independent scholar who had his own scheme for dramatically upgrading the way we think (Nevala-Lee has a post about him at https://nevalalee.wordpress.com/2016/10/11/to-be-or-not-to-be-2/ and Andrew Pilsch's 'Self-Help Supermen' article I linked above talks about Korzybski and van Vogt starting on p. 526, with more on the connection on p. 531-535, and p. 527-528 also reference an earlier article relevant to van Vogt and the dream of 'upgrading' oneself, 'Super Men' by Brian Attebery at https://www.jstor.org/stable/4240674).
Searching lesswrong.com for mentions of van Vogt I see Yudkowsky talks about his influence at length in the post at https://www.lesswrong.com/posts/q79vYjHAE9KHcAjSs/rationalist-fiction and there's also a more recent Yudkowsky post at https://www.lesswrong.com/posts/YicoiQurNBxSp7a65/is-clickbait-destroying-our-general-intelligence where he name checks him and a few others, saying "I was pretty much raised and socialized by my parents' collection of science fiction. My parents' collection of old science fiction. Isaac Asimov. H. Beam Piper. A. E. van Vogt. Early Heinlein, because my parents didn't want me reading the later books."
By "these guys" I meant the Zizians specifically, but I learned it as a fact about a broader, disunified post-Vassar branch of the LessWrong/Extropian movement that included them, Jess Taylor, and Ziz's mentor Alice M. (all mentioned and/or linked above in Max's piece). And yes, the Korzybski and Van Vogt influences seem very important for Yudkowsky. Thanks for the link to Nevala-Lee, that's a nice article!
Thanks, did any of that group ever write about Cordwainer Smith or what they got out of his writing, or more like something you heard about through friends-of-friends etc.? Also when you talk about the post-Vassar branch, is that all people who had been working with Vassar either at MIRI/CFAR or at Leverage, with Jessica Taylor talking about cultishness in both at https://www.lesswrong.com/posts/MnFqyPLqbiKL8nSR7/my-experience-at-and-around-miri-and-cfar-inspired-by-zoe and with Scott Alexander blaming it on the "Vassarites" in a comment?
I heard about it directly in person at events and I haven't read enough of their online stuff to know if they wrote about it or not. This wasn't all the people who had worked with Vassar, just "a" post-Vassar branch among at least a couple different ones. Another was more "magick" with strong Crowleyan influences, sort of a The Invisibles LARP. There was some overlap of branches and it would probably be hard to describe exact boundaries between them even for insiders. Leverage wasn't particularly influenced by Vassar so I'd consider them a separate thing.
Oh shit! I just read an omnibus of John W Campbell cause I'm a freak for Carpenter's The Thing.
What I couldn't get past was that EVERY. STORY. in his collection has a single, brilliant, genius man (always man) defeat an unbeatable monster/empire/alien armada/whathaveyou with the invention of some superweapon or gambit (MacReady and the blood test) or even occasionally headslappingly obvious tactics. (Bad guys are given bulletproof shields so bury them in rockslides)
I was like wowwwww this is so nakedly 1950s atomic age/A-bomb/Galt pilled but given that a lot of great stuff had happened, in some cases due to one guy or a small team (Salk, the Curies, Einstein) and WWII had been ended (in the American propaganda that totally ignored the Soviets and Japan's willingness to surrender) by the SuperWeapon of the A-bomb. So in a sense excuseable.
I did NOT know that he grabbed the levers of power and made sure to mint out more dreck like he created. I guess he wasn't like, the most obvious practitioner of that style of scifi, hes a writer with a distinctively bad and hackneyed personal who *forced it* into a genre.
But yea. All problems are soluble by a single smart genius acting in concert with no one else or directing a faceless mob whose deaths don't count. Does that sound like you? Good news, it sounds like themselves to literally anyone!
Yeah, Nevala-Lee talks about this theme in sci-fi of that era, calling it the myth of the “competent man”: https://nevalalee.wordpress.com/2016/04/12/the-myth-of-the-competent-man/
Hah, yea they nail literally every point I mentioned. Guess this wasn't a new revelation but to be fair I'm not a big scifi guy.
I kinda fell off once I got through adolescence and realized how much of the genre is a special little boy no one believes in (harry potter, ender, Luke Skywalker) from nowhere special who has a magical destiny that makes him so much better than everyone else... you get it.
It's wish fulfillment for sad lonely kids (Which, hey I used to be an SF fan, not gonna act like I wasn't a sad lonely kid). Anyway, is there a taxonomy?
Where's the line between an unathletic nerdlinger imagining being discovered for having great hidden talent and going on adventures and a mundane adult with no particular skills imagining being a super competent worldbeating genius?
Also, one niggle, I'm not talking Connecticut Yankee in King Arthur's Court type stuff which seems to be the focus of the piece you linked, but specifically a guy of that context and milieu who is just smarter and better than everyone sround him. Usually set in the future, but in Who Goes There it's just an extra smart 1950s scientist in a base full of 1950s scientists.
The trope isn't particular to SFF.
Scientology is a fascinating history! It was in the news a few years ago when its current iteration imploded, now somewhat forgotten again. I love how they call all their ideas and practices "tech."
Tbh in some ways I have more respect for the Rationalists who get freaky with it than the ones who just go off measuring brainpans or whatever. If you're gonna go mad, go mad in style, you know?
One minor quibble I have is with your characterisation of the people who join these cults as "convinced of their own inadequacy". I'm sure that's true for some of them, but humility is not generally a quality I associate with the Rationalist community. In fact, I think for many cults the appeal is that someone claims (at last!) to recognise how *special* you are - so special, in fact, that you get to join the special club for special people who know the Real Truth about the universe. That's a powerful draw, especially for someone who already has trouble relating to others.
Yeah, I'm probably phrasing it wrong, but it seems really clear that a key component of the cultiest corners of the Rationalist world is the sense that you, the newcomer Rationalist, are *not rational enough*, and can be "debugged" or "reprogrammed" in some way to become *more rational*... Ziz and Jessica Taylor both describe these sort of Dianetics auditing-type "debugging" sessions happening with CFAR people, really freaky!
I think it's true but it works more in terms of "I'm brilliant and can OPTIMISE it even higher" than inadequacy.
It's not either / or - belief in being brilliant/feeling of social inadequacy. It's classic grandiosity. They go together hand in glove. Pretty much all adolescents have it to some extent - for some people it's more extreme, and that make them more vulnerable to an ideological leader who will show them The Way.
In some broad sense much of Rationalism is just Dianetics 2.0.
In the Zizian flier for their MIRI/CFAR protest at https://imgur.com/a/1RstyWp there's an accusation that "CFAR does not do remotely what they claim to do on their website: they do not appreciably develop novel rationality/mental tech", so I guess their complaint was that CFAR wasn't being effective enough in looking for good mental debugging "tech"? (a word that Scientologists also like use for their own techniques)
that's interesting, ty! i can see how an ultra-Rationalist viewpoint would lead you to hear someone talk about 'reprogramming' and think 'that sounds like a good thing to have done to me'
Agree. I think intellectual arrogance is huge here.
I wrote a piece a while ago on the commonality of rape among the rationalist/effective altruists (I've been keeping track/speaking to survivors for about eight years). It's - well, it speaks to in-group cult-like behavior.
Could you add a link to your piece?
Absolutely. Here you are:
(1) the piece: https://01bayarearapegirl.substack.com/p/rape-and-race
(2) Timnit Gebru tweeted it, calling them a cult here: https://x.com/timnitGebru/status/1663988219246178304
I think one of the most underappreciated essays ever written about the rat scene is Ozy's "Rationalists and the Cultic Milieu" https://thingofthings.substack.com/p/rationalists-and-the-cultic-milieu "Cultic" in this case means a specific academic thing about "stigmatized knowledge", not "cults". Ozy is also a big-name rat, so it's not like they're making the critique as an outsider. It gets at something similar to the points you're making here.
Ken has also been doing to great research into the Ziz stuff https://x.com/kenthecowboy_/status/1884393311827550716 He's actually been going out and doing legwork, interviewing people, getting court documents, finding info that contradicts previous reporting. Really useful stuff.
That Ozy post was very good--thanks for the rec and for the link to the Ken the Cowboy thread.
This article draws a lot from sources Slimepriestess wrote (Hemisphere Theory: Much More Than You Wanted To Know, zizians.info) before the death of Emma. I find it concerning that it doesn't seem to acknowledge Slimepriestess' current position on those sources: that they are significantly inaccurate and misleading, written under the influence of JD Pressman, Slimepriestess' paranoid ex.
This podcast does a good job of covering the relevant issues and is probably the most comprehensive and accurate publicly available source.
https://youtu.be/69-VU0IZVI8
Thanks for the link to the video!
Gonna be real, I don’t trust anyone whose username includes “slime”
While this may be true, in the interview Slimepriestess seems to have a difficult time describing in the interview what exactly is inaccurate other than “the reasoning is more complicated than that.” Which- fine, but just because there’s a 10,000 word blog post explaining why you should try to sleep with half your brain at a time (for instance) doesn’t change the fact that they are essentially sleep depriving themselves as a result of pseudoscientific reasoning. And given how apologetic Slimepriestess is toward the Zizians I’n not convinced she’s a better source than Zizians.info.
The best use of human time is developing "safe" A.I.? I mean... that is a thing a person could think... But what a chore of an idea to start a cult around.
I took a class in college on critical reasoning that introduced me to the effective altruism community and Rationalists. I thought it was brilliant (it’s all about choice! Critical thought! Being logical and aware of your biases! Pro-human despite a holier-than-thou stance against sheeple!). I thought I was going to dedicate my career to AI ethics. COVID happened and I dropped all interest as my values radically shifted, but it’s bizarre to reflect on now. Between Zizians and Luigi Mangione, I had no idea what I was so close to.
I don't think the EA community is even nearly as cultogenic as core-Rationalism, though if you are coming at it from a rationalist-AI safety perspective, I can see having the association.
Agreed, it’s definitely not. But it’s worth noting that EA is adjacent to, and informed by, some of the same writing as Zizians and other extreme rationalists. It’s a part of the philosophy that underpins the whole tech world, not just in regards to AI ethics (e.g. Sam Bankman-Fried)
That's true. Although, (minor nitpick) I wouldn't say it underpins the "whole" tech world. The tech world is pretty big, there's lots of other elements mixed in.
Don’t paint the entire “tech world” with the EA / Rationalist brush. Been in “the tech world” since 1990 and I can’t say I have met a single such wacko in person, just online 🤷🏽♂️🤦🏽♂️
EA and rationalism aren’t inherently wacko either. Rationalism as an epistemological view just means basing opinions and actions off of logic and reason rather than faith, tradition, or sensory experience. I do feel that’s a pretty essential pillar of the tech industry. It’s still worth thinking about how and where that sentiment manifests, or what that can turn into.
“Whole” was an overreach, I could have worded that better.
Interested to know if you're open to sharing: what was the shift in values that affected your interest so drastically?
Thanks for asking! I think rationalism eventually felt too removed from reality, and I don’t agree with consequentialism. I didn’t want to live in a framework where I had to turn things into calculations to know what was right or morally obligatory. The isolation of COVID exacerbated that I guess, as well as graduating college and getting thrown into “real life.” I felt such a deep emptiness in the absence of faith/tradition/experience, which are completely deprioritized in Rationalism. It all felt hollow. AI ethics also felt completely futile.
I moved onto a farm during COVID and learned to grow food for my community. It was the best decision I ever made :)
That sounds practical, wholesome and something that Rationalism would not have been able to provide. Thank you for sharing! I haven't ever been into Rationalism but what you described has been my experience with 99.99% of the things I pick up from the Internet: after some time they felt hollow, and I craved the stark reality and groundedness of real life
Ugh, so true. Makes me wonder if that’s an inherent property of the internet or whether it is made that way by the kind of culture that is bred online
Both, I think
The Internet, for all its wonderments, can never be real life, and the cultures bred here are usually untested against reality and so collapse when they come in contact with it
Seeing as the internet was born into a dominant culture - our Western media - I guess maybe it was inevitable that the kind of online culture we've now got would eventually dominate. Have you ever read Neil Postman's Amusing Ourselves To Death (it's brilliant)?
No! But you’re not the first to recommend, maybe it’s time for me to grab a copy
I'm only peripherally aware of these events and identify with Effective Altruism rather than Rationalism; but this post strikes as well-researched and fair to the facts of the matter. I learned a few new things too!
I am always unconvinced by the (very frequent) claims that Rationalism and LessWrong are cults, but "cultogenic" would be a fairly accurate description.
If you want a somewhat deeper dive into the world and culture of the Rational movement, that *isn't an outright hit piece*, try Tom Chivers' "Rationalist's Guide to the Galaxy", where the journalist actually tries to understand and explain the movement from the inside view of the many mostly normal people involved, rather than analyzing it by it's greatest failures, like SBF, or conspiracy-brained a guilt by association stuff like Metz's piece or and the completely nonsensical TESCREAL argument. of course, those are useful sources if you want to declare it a dangerous cult ideology, but not if you actually want to understand it
Which is not to say that groups like the zizians, vassarites and leverage aren't real or aren't culty. But I would venture that the dynamic here is more that a bunch of open-minded people trying to change the way they live is a absolute hunting ground for cult leaders, rather than some inherent failing in "trying to be more rational" as a lifestyle
I think this basically matches what I'm saying! I'm not convinced that "Rationalism" as a movement is adequately described as simply "trying to be more rational," though.
I feel rats subculture/movement is what happens when intellectual openness to experience/ideas combined with high IQ and arrogance these two traits generate collide with autistic-like need for certainty (however you define the latter).
These people need God
I'm not religious, but from the perspective of someone who has to share a planet with these people, I actually think you're probably right. Religions can make people act crazy too, but a moderate mainline belief system would probably help them figure out which is the ass and which is the elbow. I mean, what are they looking for? (God) And what are they finding? (Void) And what are they trying to compensate with? ("debugging" themselves like they're meat machines) -- it's a nihilism spiral where nothing is true and everything is possible, and so there's no ground to stand on. They need ground to stand on -- so they can figure out who they really are is NOT who they are when they break their brains in two.
I kinda sympathise with what they're looking for. They want a practical religious path to enlightenment - and a way forward that will lead humanity away from its worst impulses. In that way, they remind me a lot of radical Christian movements from the 16th and 17th centuries, and of communist revolutionaries. They feel strongly that modern society keeps people locked in shallow, meaningless lives (which is teenage and based on a limited experience of life, but not entirely) unreasonable), and the way that knowledge and technology are accelerating mean that we're genuinely coming to a point where the definition of reality and consciousness will probably be redefined - as it has been in previous centuries. As far as I understand it (not much), physicists are telling us that a whole lot of very weird shit is possible. That's what makes the religious underpinnings of this so fascinating - these cults are factions and subsects of a priesthood who aren't engaged in astrological calculations or sacrifices to abstract gods - they've built actual new systems of information synthesis; they're playing with the frontiers of mathematics and physics, and consciousness. Is the singularity coming? I have no idea, but this century is shaping up to be the most condensed and batshit crazy epoch that humanity has yet to experience. It's no wonder that young people who in the past might have been drawn to millenial doomsday thinking or the perfect society of the enlightened proletariat, or the cosmic groove of the psychedelic awakening, are currently getting into the utopian promises of AGI.
I'd argue that it's a FEAR of nothing being real (because nothing is certain, and those minds tend to need certainty) that gets rationalised (lol, pun intended) into "everything is possible".
A daily internet-use limit for people who aren't using it as part of their profession would nip this in the bud. 30-45 minutes a day of use for recreation, enough to check the news and see a nudie show.
Widespread computer addiction has tragic consequences, in this case -- young men losing their minds and turning to murder.
Its the same as the widespread use of LSD leading to the Manson murders