In graduate school, in my lab there was a grad student who was kind of an unlikely "professor's pet". He was tall and had surfer's long hair with a bit of a hippie aesthetic. Anyways, he was also really completely clueless about how to do science correctly, but also, I guess, really good about playing politics (there was a time when he asked me to put some bacterial plasmid DNA on my mammalian cells. I told him "it doesn't work that way", but I did it anyways and handed over the cells, and he got the observation he was expecting). On his main project he was teamed up with a super sketchy foreign postdoc that I was convinced would say anything to get high profile papers out.
So they did a series of experiments and reported results that screamed "artefact". On one of them, for example, the postdoc got trained to use the electron microscope and they went through thousands and thousands of images to pick out the one that had "just the right morphology" (I am pretty sure they were snapping photos of salt crystals). On another, they reported that their research subject protein was so fast at the process we were studying that everything occurred IN MIXING TIME. That to me, screams "you are not doing your experiments carefully".
Meanwhile I was sweating balls working on a very careful preparation of similarly finicky proteins (you agitate them and they do bad things since they're metastable) and finally got it to produce reproducible results. I suggested they adapt my preparation to their protein but they couldn't give a damn, they had already published their paper and had moved on to sexier proteins.
But then an intern was put on the project, and she could not reproduce their results, after working on it for six months (she is careful and honest). At the end, I felt so bad for her, I offered to train her on my technique, but she passed. I think she was burned out on the project. I asked if I could get a sample of the protein that she had prepped, and she agreed.
I ran the protein through my preparatory technique and observed that there was a contamination that could have seeded the kinetics of their process. Upon isolating an uncontaminated sample, I carefully but briskly rushed the sample over to the machine. Nothing. Curious, I jacked the temperature up to get it going faster. Nothing. I left it in the machine overnight. Nothing. Finally, convinced that I had likely done something wrong, I dropped the sample in a shaker at temperature, came back the next day and recorded amazingly high signal. In short, the observation that it was "super fast" was entirely an artefact.
As I, too, was trained on the Electron Microscope, I quickly spotted my sample onto an EM disc, reserved some time and hopped on the 'scope. The first grid sector I looked at, there was literally TEXTBOOK morphology in front of my eyes.
I stapled together my results, gave it to the grad student, and told him that the general gist of his paper was probably still correct, but that he should be careful about characterizing his protein as exceptional. I then said it was in his hands to do the right thing.
What do you think he did? Nothing, of course. He kept on the talks circuit, still talking about how exceptional his discovery was, and to date there have been no retractions. He even won the NIH grad student of the year award.
The epilog is that after a decade of floundering I realized that even though I am pretty good at science, I was no good at playing academic politics and quit the pursuit; I drove for lyft/uber for a bit, and now I'm a backend dev. I am certain that my experiences are not unique. Amazingly the intern returned to our lab, and had her own three-year stint chasing ghosts that turned out to be overoptimistic interpretation of results reported by a postdoc.
Oh. What happened to the grad student? He's a professor in the genomics department at UW.
A friend joined a group studying some cell behaviour. They had previously had a big result that they could stimulate this behaviour in defined, serum-free culture by adding a specific factor.
Friend was to work on characterising this effect, so his first job was to reproduce the result as a base case. He couldn't. The factor didn't stimulate the behaviour.
He asked around, comparing his execution of the protocol with that of the the postdoc who had done the original work.
The method involved growing a feeder layer of cells, in serum, then lysing them and washing the plate, leaving a serum-free layer of extracellular matrix behind, as a foundation for the serum-free cell culture (this is a pretty standard technique).
Turns out the previous postdoc's idea of washing a plate was a lot less thorough than my friend's. Couple of quick changes of PBS. So they were almost certainly leaving a lot of serum factors behind on the matrix. Their serum-free culture was nothing of the sort.
The supervisor insisted that the previous postdoc's work was fine, and that my friend just didn't have good technique. The supervisor had him repeat this work for months in an attempt to make it work. But he's a careful worker, so it never did.
This is the worst situation when the supervisor (professor) “sees no evil, hears no evil”.
In a similar situation a prior students work couldn’t be repeated and it was pretty clear the student made up the results. “Water under the bridge, let’s move on”. Of course the publication still counted for the prof.
This feels like automation would have great benefits for these types of things.
Instead of relying on people getting the right technique, you load in their program, dump chemicals into the right vials, then let it run and check the results
Well, a lot of these tasks are already automated (ie, shakers), but most bench workers have their own quirks on existing protocols. Most labs have their own 'dialect' of common mol bio techniques that 'work' in their particular setup. Perhaps the reagent from their particular supplier requires a longer incubation time, or the enzymes are wonky and you need to add more. Everybody I know does washing steps their own way - they say the "official" protocol is too long/cumerbersome/wasteful. More often than not, their own variant of the protocol is documented in their lab books, but not in the publications, where it cites the original protocol.
I imagine the postdoc would have a negative control of not adding the vector? Otherwise its hard to convince people the effect was coming from the vector.
Having worked in both academia and industry in biotech field, I have to say that the bar of reproducibility is a lot higher in industry.
In academia, the goal is to publish. The peer-review process won't care to repeat your experiments. And the chance that other lab repeating your experiments was slim -- why spending time repeating other people's success?
In contrast, in industry, an experiment has to be bullet-proof reproducible in order to be ending up in a product. That includes materials from multiple manufacturing batches of reagents, at multiple customer sites with varying environmental conditions, and operator with vastly different skills.
I can second this. Working in industry, the bar is quite high for rigor. The general attitude of industrial researchers is to be very very skeptical of academia, since a lot of things just don't reproduce (cherry-picked data, p-hacking, only work in a narrow domain, etc., etc.). These researchers are almost all people with PhDs in various science fields, so not exactly skeptics.
Industry works solely on stuff that's reproducible because it wants to put these things into practice. That makes for an admirable level of rigor, but constrains their freedom to look at unprofitable and unlikely ideas. That inevitably results in inadvertent p-hacking. The first attempt to look at something unexpected is always "This might be nothing, but..."
They call in other people earlier because they're not protecting trade secrets or trying to get an advantage. They do want priority, and arguably it would be better if they could wait longer and do more work first, but the funding goes to the ones who discover it first.
So there's no real reason for either academics or industry scientists to look askance at each other. They're doing different things, with standards that differ because they're pursuing different goals. They both need each other: applications result in money that pushed for new ideas, and ideas result in new applications.
I agree with you, and I don't want my comment to be read as an indictment of academia exactly - we couldn't live without it, it has huge returns on investment, etc. It's worth reading 10 bad papers to find 1 with a kernel of a good idea (and worth spending research funding on 100 bad experiments to get 1 useful result).
I think what I mean to say is that the skills required in industrial research (which can be quite speculative in well-funded companies, by which I mean a 5% chance of success or so) are somewhat different from those required in academia.
I could sense this sort of problem even in CS and could not wait to get into an applied position as soon as possible. If you cannot build the thing you’re an expert in, you’re no kind of expert I understand.
In many academic disciplines there are no real incentives for reproducible research. On the contrary, reproducibility helps your colleagues/competitors poke holes in your papers. It is quite perverse that being secretive and sneaky is better for career advancement that being open and honest. This is the underlying root of the problem.
Well, I believe that the biggest problem is that there are very little incentives in doing that. Everybody (your university, the Government, the funding agencies...) rushes you to publish as many papers as possible, get zillions of citations, and boost your h-index; however, they do not give a damn about the reproducibility of the results you are publishing.
I'm sorry but this saddens me to no end, even I did better science during my BSc and MSc; it's not just disheartening, it's frightening. Reading this almost made me feel ill to my stomach. I don't know what else to say at the blatant disregard for scientific ethics and sense of duty.
And we complain that the public at large doesn't trust us "educated" folk, well I can't see why...
My own, rather bitter, experiences with academic research in the early 1990s led me to suspect that by trying to "manage" academic research at a large scale was utterly counter productive and was optimising for all the wrong things (publications, career progression, money, politics) and was actually dramatically reducing the amount of actual science being done.
I left, co-founded a startup and never regretted it for a moment.
Edit: The point where I was sure I had to leave was when I was actually starting to play the "publications" game too well - when you find yourself negotiating with colleagues to get your name on their paper for a bit of help I'd decided things weren't really for me.
Edit2: I'd wanted to be an academic research scientist since I was about 5 or so when I actually got what I thought was my dream job I was delighted - took me a couple of years to work out why almost nothing in the environment seemed to work in the way I expected them to ("Why is everyone so conservative?") and became, as one outsider described me, "hyper cynical".
I had the 'luck' of being a research assistant at a prestigious academic collaboration involving multiple equally prestigious universities. This was in my bachelor years, and I still hadn't decided whether to pursue a career in academia or elsewhere.
While the experience day to day was definitely fun, it destroyed any desire I had of entering the field. A lot of politics, a lot of statistically suspect stuff (even to me, in my third year of a bachelor), and a lot of busiwork.
After that experience I went into web development (full-stack). What I like about it is that even though there IS politics, even though there IS taking shortcuts, and god forgive me for some of the code I delivered, in the end whatever I work on has to actually do the thing it's supposed to do. It doesn't remove the aforementioned problems, but it grounds everything in a way that is mostly acceptable to me.
As frustrating as it can be to build some convoluted web app that feels like it's held together by scotch tape, it's nice to know that it eventually has to do whatever the client asks for, however flawed.
Apologies, I meant conservative in the sense of resistance to contemplate new ideas rather than the political sense. Somewhat naively I had assumed that academic research was where people would be most welcoming of at least discussing new ideas, whereas I found the opposite to be true.
Thanks! Could you give a few examples? About what were folks so conservative? Were they stubborn proving/supporting their own paradigm/hypothesis or ... they were just simply not open to any ideas? About methods or about theory? Both?
It was quite a long time ago (~30 years) but I suspect a lot of it was simply because senior academics didn't realise they were actually managers, had no interest in managing or even understand that there were problems.
If I had to guess, in the academic context it would mean no actual novel thinking, just churning out more papers on the same `winning` theories in the field, things where before even starting you have a clear idea of what the result would look like.
The problem with going to a startup is it is kind of like going from the frying pan into the fire. As someone who has worked in both academia and industry, while academia and its pursuit of publications leads to bad behavior, industry and its pursuit of money is even more unprincipled. While it might not be that hard to fool peer reviewers with nonsense, it is way easier to fool venture capitalists, who often know no science and and are just listening for the hot buzzwords.
Is that small-c conservative? Or do you mean rightwing? (curious, I assume the former...)
In either case pretty much all humans are profoundly small-c conservative, "big change projects" on society-scale do often end in war/death/etc. At least, it's probably 50/50 whether its a "National Health Service" or a "World War".
However the reason is deeper than that: evolution does not care if you're thriving, it cares that you are breeding. So you're optimized for "minimum safety" not "maximum flourishing".
So if things are stable then you will prefer to stay in them for as long as possible. It is why people need to "hit rock bottom" before they can be helped, often, ie., their local-minimum needs to become unstable so they will prefer the uncertainty of change.
This is true, and in my opinion there is one more tendency which you also imply.
Not only the public at large, but even University graduates start to an extent distrusting those who are "professionals" in academia. It is simply a whole other world, where you are only judged by the number of papers under your name, perhaps never having contributed to anything practical - seems so detached from real life.
It is worse. If you mention that you are not trusting every scientific results per se, you're being labeled as stupid and uneducated. This sort of absolute reasoning is making the distrust even worse. How can you have trust in a in a system that is unwilling to publicly admit its shortcomings? Trust and honesty come in pairs.
This science-bro movement scares me too. "But SCIENCE said so! You're a SCIENCE denier!"
It feels like a religion, with its own T-shirts and all. Appeals to authority, intellectual posturing… often from people with little understanding of the actual science. Honest insiders are way more careful with any absolute statements.
No wonder there's a (also scary) rise of conspiracy theories.
How do people not observe those as two sides of the same coin?
I think people have to get more serious about separating science as a procedure from scientism (that is, philosophical issues that are often discussed in tandem). When one uses the phrase, “science denier”, it often means, “you don’t agree with my philosophy/metaphysics/economic policy” rather than “you deny these particular facts”, which causes people to be rightly concerned. I’m not optimistic that this is going to change anytime soon, but this, I think, accounts for many of the issues in current discourse.
In the UK we've seen a fascinating evolution from skeptic societies to science denial conspiracy theorists. To _massively_ simplify what's a relatively complex piece of sociological weirdness: using your intuition about how the world works is a good heuristic for spotting charlatans, but it fails you badly when the science tells you something that doesn't accord with your intuition.
I tend to limit my use of "science denier" when an organization or its followers systematically deny scientific knowledge on multiple unrelated fronts.
Interestingly, I have read that in the 1920s and 30s, there was actually an organized relativity denialist movement, that wrote articles and held public protests.
Relativity was a huge philosophical shift from the comparative simplicity of Newton's laws. It's not surprising that there was resistance to it.
Tesla was famously against relativity, telling the New York Times, "Einstein’s relativity work is a magnificent mathematical garb which fascinates, dazzles and makes people blind to the underlying errors. The theory is like a beggar clothed in purple whom ignorant people take for a king".
Indeed, and the anti-relativity movement also had a very strong undercurrent of antisemitism.
Chances are, most of the people marching against relativity had no clue about Newtonian mechanics, and were told stuff such as relativity leading to moral relativism.
Since I read Seeing Like a State, I've started to think "charlatan" whenever I hear the word "science". As in "scientific forestry", "climate science" (scientists who study Earth's climate call themselves meteorologists), "scientific racism". Is "computer science" an exception? I'm not game to speculate.
Which actual scientists describe themselves that way? We're physicists, geologists, botanists, psychologists or whatever. When someone says they're a scientist, it suggests that they're not part of any actual scientific discipline, but making a false appeal to authority.
>scientists who study Earth's climate call themselves meteorologists
This is just incorrect. Meteorologists don't study Earth's climate, they study weather. Meteorologists don't use ice cores or tree rings for their research, they study much shorter-term fluid dynamics. Climate scientists do study climate, and not weather. The disciplines are related (specifically, they're under atmospheric sciences), but to dismiss either one as being less scientific is picking favorites despite all evidence to the contrary. I suppose you could use the synonym "climatology" if you want a word without "science" in it, but it seems like a pretty silly heuristic regardless.
I don't like the science-bro movement, but I also think they might fill an important niche. The anti-science movement has too many people and too much time and too high of a (answer time / question time) gish-gallop ratio for scientists to possibly engage with. If scientists try to fight the anti-science crowd, they will lose.
Science bros, for all their faults, can trade blows on more even footing, and that's something. Perhaps even a vitally important something. Even if science bros aren't great at science proper, their contribution to societal consensus formation might be as important as the underlying science itself!
Science bros often misuse "anti-science" to try to shutdown opinions they disagree with. Hence people worried about the unlikely event of being killed by a nuclear power plant are anti-science, but people worried about the even more unlikely event of being killed by a super intelligent AI aren't. Misusing the word "science" (particularly by people who don't seem to have a good grasp on it) and turning it into a rhetorical cudgel is harmful, and pushes the idea that science is ideological.
I found that the movement you talk about is more about putting your faith in the "scientist" as opposed to the actual "science".
It seems much easier to find scientists who will tow your political viewpoint and then people can use them as a resource to prove that unless you take this person's "expertise" as gospel, then it proves you are a science "denier".
My cousin was a student at a lab where a sketchy grad student doctored results too. She was majorly sketched out by the whole thing and that the PI supported the whole op. It was very painful and set her back a bit but she managed to switch to a different lab and do proper work, defend her thesis, graduate, and get far away from those people. Now we just shake our head in disbelief at them but at that point it was fairly existential. It's not easy to switch labs after some time there and people will sort of distrust you and everything.
My impression is that some large number of 'results' are fake results. I can't even imagine in non-hard sciences what the fakery is when the hard sciences have this stuff.
Attention and effort will go to “well-presented fake”.
Marie Curie believed that radioactivity might have been caused by ghosts or the paranormal because of such things.[1] While there may actually be ghosts or other things paranormal, I’d bet that Marie Curie was fooled.
The good part is that Curie’s work persists, and we think we have more understanding about radioactive substances.
I’m not sure whether she had to spend time specifically debunking the ghost-of-radioactivity theory; that just happened because of her work studying radioactive substances and their effects.
I find this fascinating. If we allow ourselves to entertain the idea of quantum time paradoxes, could it be that the radioactivity was in fact caused by the ghost of Marie Curie herself? She would have a very strong and obvious reason to haunt the science.
That sounds like self-imposed slavery: every time someone wants radioactivity, Marie Curie's ghost needs to show up and produce it. What with all the nuclear reactors and RTGs on far-flung spacecraft, she's a busy ghost.
Every now and then I go through some of wikipedias sources on certain social topics because I don't trust them at all. The amount of BS I've found in papers, even though I don't eve have any research background at all is impressive.
My favourite was probably this one paper where the author essentially made a reddit-post asking a community about themselves, then cherry-picked (the post is still up, with timestamps and all) a few comments and came to a conclusion that didn't really fit those hand-picked comments.
In conclusion: Wikipedia is a dumpster fire and shouldn't be used for anything other than hard facts like dates and for entertainment.
Wikipedia actually aims to use secondary or tertiary sources, because of the likely bias in primary (and to some extent, secondary) sources. Statements of fact shouldn’t be supported only be primary sources (publications), though they may be referenced for the historical context. However, quality control on something as big as Wikipedia is essentially impossible.
An encyclopaedia with rather low standards that many people sadly treat as an absolute source of truth.
You're right that this isn't really a wikipedia problem though. It's a matter of education because an overwhelming majority of the population isn't competent enough to fact-check memes on facebook, let alone wikipedia, and if wikipedia doesn't do it either, then that responsibility is pushed all the way back to the scientists doing the actual research.
This is an incredible lack of redundancy if you consider how important wikipedia has become in shaping public opinion. It's a system where the scientific publication process is the single point of failure and this article clearly shows that it does fail rather often.
So what way is there to make this process safer? There needs to be at least another link in the chain that confirms information, preferably two or three.
... that somehow manages to have articles on proeminent subjects that are more in depth and factual than any competing encyclopedic endeavor, while, at the same time, far surpassing them by orders of magnitude on breadth for obscure and less academic topics.
Wikipedia is not an encyclopedia in the traditional sense, and can't be judged on the same standards. It is simply in a league of its own, it fails in different ways than traditional editor-controlled projects and is a fantastic repository of human knowledge and educational resource.
Wikipedia is overall great, though highly politicized in some topics, for science/engineering topics there's actual review by other Wikipedians and sourcing is solid.
Moreover, the talk page always has anything that might be controversial about the article that you might be interested in.
Sure, it will rarely display incorrect data, but it happens less and less as antivandalism bots become smarter.
Wikipedia itself has descended into the same pathologies you see in 'science' today: a bunch of gatekeepers who, by account of having been there the longest, have set up a moat of rules and 'culture' and such, to the point where newcomers are shut down or drowned out. I'm not saying it's impossible to get in; but only those that sufficiently mould themselves to the existing people and structures will last long enough to become fully accepted. And so the system sustains itself.
I wanted to edit an article about an obscure religious group that included some blatantly wrong statements about the espoused ideology. They had an academic tertiary source making these claims extrapolated from reaearch by the same author that made both plausible claims but also included similar inferences. Being a very obscure group there aren't many other academic sources discussing it. All literature that could disprove these claims comes from non-academics affiliated with the group which are a no-go.
As per Wikipedia rules (which took hours to figure out), there's not much one can do short of getting some impartial or friendly academic to publish a more reasonable article.
I already spend a lot of time trying to "fix the internet" and just don't have the stamina to also start fixing wikipedia now. I'm also being turned away by the constant stories of edit-wars that tend to happen about certain controversial topics.
Do you mind telling us which paper it was? I have a faint idea which one you mean, because I've read a bunch about reddit, and I would love to know if it's the same one or something else.
Also describes medical research. Only it's even worse. Problem is, you have no choice if you want to work in a university hospital. The system essentially tells you "you'll be doing shit science... or you'll leave!". Been at it for more than 10y, and no hint of change in sight. This is going to be really, really hard to change unfortunately.
At least my research is. Mainly due to hierarchical pressure. And from what I see around me, most medical papers must be read with a healthy dose of skepticism. I've personally witnessed incredible feats of dishonesty that I won't describe here.
There are multiple reasons degrading research quality. An important one is spreadsheet incompetence. Another one is that medical research goes hand in hand with academic achievement, which in medicine also means money and power (probably more than in most other fields). I guess we have the same kind of problems as everyone else, overall.
One thing people often miss is that clinical data is of abysmal quality and reliability, so honest analysis is really difficult.
I'm a postdoc at a medical school, and this hasn't been my experience. At least in our setup, clinical data tends to be channeled into a collaboration with a computational lab who are better stewards of data handling. Is there cherry picking and over selling the results? Sure. Outright dishonesty is something I have yet to see in my current institution (I did see a fair deal of fudging in my graduate institute, though)
Well, I think things are beginning to be better managed in some centers. If that's your case, then good for you. In my center, it's basically the wild west and data management is a catastrophe.
But are you working the clinical wards? Because things are definitely much better managed in places such as epidemiology units. The true horrors mostly come from clinical researchers digging into excel spreadsheets without knowing a mean from a median.
I'm in a computational lab, but I think I understand what you're describing. My medical school was acquired a few years ago by a hospital network, encouraging us to collaborate with our new clinical researchers. The medical school itself had a strong background in rigorous basic research with animal models, and the clinical samples are a relatively smooth transition. The data is obviously nowhere as clean or plentiful as with animal models, but that's to be expected.
So for example, my lab's expertise was in single cell developmental models, primarily for organ development in mice. Extended that to tumors from clinical samples was relatively straightforward. One of my colleagues is working on an autism dataset, but I wouldn't expect that to be nowhere nearly as clean.
I am also from a molecular biology background and saw this often. We call these guys the "Golden Boys". They are super successful, but completely useless. If you still believe live is fair, wake up sunshine.
Dont worry about it. It becomes a trap and many of them loose their minds as them dig themselves into deeper and deeper holes not knowing what else to do. Source: shrink in the family at a univ counselling center. At the end of the day they are misguided people.
Even though there is a high price, their function is to train the survival skills of the honest folk who rise up the food chain. And dont have any doubt they have survived these type of people (usually thanks to the right networks and mentors), have developed their own tricks and exist in large numbers.
I don't particularly care about him per se (though i'm sorry to burst your model of society, from what I hear over my residual science network, I'm pretty sure he's oblivious or doesn't care), I'm a bay area dev, and I'm making enough money and have made good investments in friend's startups that my only regret is not having started sooner. Hopefully I'll be able to cash out with enough to do my own biotech, so I'm just biding my time for now. But what does concern me is that this is endemic in chemistry. It's not talked about much outside.... Which makes me wonder if other sciences are just as bad, "we just don't hear it". The incentive structure and nearly nonexistent self-reporting accountabilily is just the same; and meanwhile everything operates under a general social deification of the sciences.
We hear it all the time. Its an ancient story. The history of science is full of these stories.
Misguided/driven/ambitious people are always looking for shortcuts and they will find them. Its like dealing with mosquitos, cockroaches, weeds, software bugs and cancer. It never ends.
I think the point of GP's story is that this isn't just one or two bad apples here and there, but it's endemic in that domain - and most likely in others too (I'm leaning to believe that; it's not the first story like this I've read in recent years).
Being an endemic problem means you have to switch your assumptions; when reading a random scientific paper, you're no longer thinking, "this is probably right, but I must be wary of mistakes" - you're thinking, "this is most likely utter bullshit, but maybe there's some salvageable insight in it".
I think once you've seen a few papers in high-tier journals that turn out to be bullshit once you start to dig a bit deeper, there is not other choice than to adopt this harsh stance on random scientific papers. Especially if you want to do work with that expands on findings on other papers that roughly look good "trust but verify" seems to be the way to go.
I've only recently dipped by toes into academic life in a lab, but it very much seems that PIs generally know which are the bad apples. E.g. when discussing whether some data is good enough to be publishable the PIs reaction was something along the lines of "If we were FAMOUS_LAB_NAME it would be, but we want to do it in a way that holds up". So it seems like there are at least some barriers to how incompetence would hurt the whole field.
I'm also surprised that there is no mention of the PI in GP's story. As it's a paper published by the lab, it's not just on the grad student "to do the right thing", but even more on the more senior scientist, whose reputation is also at stake.
> I think once you've seen a few papers in high-tier journals that turn out to be bullshit once you start to dig a bit deeper, there is not other choice than to adopt this harsh stance on random scientific papers. Especially if you want to do work with that expands on findings on other papers that roughly look good "trust but verify" seems to be the way to go.
Yeah, but I meant that in general case, you no longer "trust but verify", but "assume bullshit and hope there's a nugget of truth in the paper".
This has interesting implications for consuming science for non-academic use, too. I've been accused of being "anti-science" when I said this before, but I no longer trust arguments backed by citations around soft-ish fields like social sciences, dietetics or medicine. Even if the person posting a claim does good work of selecting citations (so they're not all saying something tangentially related, or much more specific, or "in mice!"), if the claim is counterintuitive and papers seem complex enough, I mentally code this as very weak evidence - i.e. most likely bullshit, but there were some papers claiming it, so if that comes up again, many times in different contexts, I may be willing to entertain the claim being true.
And stories like this make me extend this principle to biology and chemistry in general as well. I've burned myself enough times, getting excited about some result, only to later learn it was bunk.
The same pattern of course repeats outside academia, but more overtly - you can hardly trust any commercial communication either. At this point, I'm wondering how are we even managing to keep a society running? It's very hard work to make progress and contribute, if you have to assume everyone is either bullshitting, or repeating bullshit they've heard elsewhere.
Funny story, PI noticed an error in one of my papers and I (happily) issued a very minor retraction. Also in one of the threads I talked about how he did retract several year's worth of work done on a different project by the intern when she joined later. So he was alright. Plus, as a junior (2nd year grad student) you really don't want to tattle on the NIH grad student of the year. Who do you think wrote the recommendation?
it's endemic in biology, and it's endemic in chemistry (I had feet in both sides). The sentiment you wrote in the last sentence is exactly what I feel whenever I read a paper, hit it on the nail.
The crazy thing, is that the honest scientists are working at middling university. It is worse the higher up you go. I have had the opportunity to work in a upper-midrange research university [time-] sandwiched between two very high profile institutes. The institutes were way more corrupt. Like inviting the lab and the DARPA PM to hors d'oeuvres and cocktails at the institute leader's private mansion type of stuff (it turned out that that DARPA PM also had some wierd scientific overinterpretation skeletons / PI railroading the whistleblower stuff in her closet, and for a stint was the CTO of a microsample blood diagnostics company, I can't make this shit up, I guess after Theranos it got too wierd, she's now the CEO of another biotech startup -- how TF do people like this get VC money, and yet I can't get people to raise for some buddies with a growth industry company, and had to make the entire first investment myself?).
Of course working at a upper-midrange university sucks for other reasons. Especially at state universities, the red tape is astounding. And support staff is truly incompetent. Orders would fail to get placed or would arrive and disappear (not even theft, just incompetence) all the time.
While the "host" (people who pay, often with minimal decision power over their resources) turns a blind eye, "parasites" (cheaters who profit disproportionately) proliferate. Is that really so surprising?
When somebody else foots the bill, it's feast time!
To be clear, I'm with you. Also a PhD-turned-industry, for much the same reasons. But I realize what you describe is a completely rational strategy. The options always come down to:
1) Try not to be a host – if you have the wherewithal
2) Try to be a parasite – if you have the stomach
3) Suck it up & stay salty – otherwise. You can call it a balance, equilibrium, natural order of things – whatever helps you sleep at night.
Take your pick and then choose appropriate means. Romantic resolutions and wishful thinking – kinda like Atlas Shrugged solution for option 1) – rarely work.
There is a reason there is a huge replication crisis in academia and it’s exactly what you say above. When folks in industry need to develop a product based a published paper more often then not it’s bullshit.
Yup, I was taught this as part of graduate school.
Nobody ever said it was fraud, they said things like they wouldn't share the data and I couldn't replicate.
In general, the incentives for shoddy science (get Nature papers or find a new career) tend to reward bad behaviour, and I just wasn't able to find something unexpected and pretend it had been my hypothesis all along (it's almost impossible to publish a social science paper where you disconfirm your major hypothesis).
The problem isn't that such people are getting away with unearned good feelings and so the fact that some may feel bad later isn't a solution or a reason not to worry. The problem is that they are wasting scientific resources (e.g. the time of the careful intern trying to reproduce flawed results), polluting research by publishing misleading findings, and discouraging legitimate research.
The problem is that there is no working system in place that makes such abuses of scientific truth visible.
We would need to get away from inefficient communication via publications and set a system in place that tracks findings in detail, and whether they can be replicated first.
But there is no willingness to do so after the US of A deeply harmed the scientific mission and academics by introducing infuriatingly dumb economical incentives into science.
> But there is no willingness to do so after the US of A deeply harmed the scientific mission and academics by introducing infuriatingly dumb economical incentives into science.
The number of incentives driving this kind of activity in science is disheartening.
So many papers get published, few are read widely, and even fewer are replicated, they'll still get citations if the talk circuit is played right. Citations are what advance a scientist in their career, and anything that could be tossed off as an unfortunate statistical anomaly or error is unlikely to end a career.
In such a world, "optimal play" would be to intentionally or unintentionally P-hack, or just slightly embellish results such that the work is interesting to cite, but not interesting enough to replicate. People who do this will eventually move up ahead of everyone else, ultimately favoring incremental but bogus resuls.
The thing I find disheartening is that if fraudulent results are being cited, it must mean that the mechanism of "standing on the shoulders of giants" is not working. One would expect that these papers would be contributions that citing scientists could benefit from and use in their own work with impact. For example if scientist A truly developed an O(N) sorting algorithm, then a scientist B might use it in their work to derive some other result.
I guess in some fields of science the effective dependency graph of academic work is very flat, and the true results get plucked and developed by industry (being true results it is actually possible to meet the higher reproducibility bar there). And the citations don't actually reflect the true dependencies, but some political/social graph instead. Too bad.
> And the citations don't actually reflect the true dependencies, but some political/social graph instead. Too bad.
I think this gets to the major concern with Academia today, as it becomes somewhat of a self-reinforcing feedback loop. Curry citations with political savvy, get awarded grants due to citations and political savvy, show that you are productive due to citations, grants, and political savvy - earning yet more political capital.
This will probably become my go to explanation for why Academic CS research has largely become decoupled from industrial application and industrial research. While political savvy is important in a large corporation, eventually you need to produce results.
this is incredible. And I thought I had it bad with politics in tech companies... this is some next level not giving a fuck right there - people who cheat like that should be punished severely, and work as supermarket cashiers, not become fricking professors. Unfortunately, I too, were I in your shoes, wouldn't pursue it much further past filing a formal complaint or two: the game is asymmetrical, it's much harder to nail someone for wrongdoing than it is for them to fudge up some lab results. Not to mention the emotional toil and waste of time and potential political blowback the would be whistle blower would suffer...
About half of my friends in grad school have had their careers damaged to varying extent by academic fraud, some have wasted lives chasing bad results (one friend lost several years chasing a bad result by Homme Hellinga), some have had bad stuff perpetrated upon them by bad actors with big names (one had her result we suspect - stolen by Carolyn Bertozzi via the review process, luckily her boss was a member of NAS and PNAS track-III'd the paper ahead of Bertozzi's publication).
Just be aware that it's not always like this, and that some fields are less prone to it than others.
In my 8 years in research mathematics, I didn't see a single case that would come close to this horror show (not that mathematics is free of unethical behavior, of course). Collaborating with biologists, however, I got exposed to a world far more backstabby than I've since experienced in the corporate world.
I think this is largely because results in math are easily verifiable compared to chemistry, or as an even worse example, the social sciences. The latter are also suffering from the replication crisis the most.
Math has a different problem. Because of the wide breadth of the field, and highly specialized nature of problems, it can take a very long time for anyone to actually verify a result with confidence. If ever. Unless you’re doing something famous like P!=NP, there might not be many people capable of checking your work in a reasonable amount of time.
The story of Fermat’s Last is a great example, what would have happened if that wasn’t a famous problem?
I agree, even proofs are wrong more often than you’d think, but I’m not sure whether math is actually so uniquely broad that other fields don’t suffer from this problem.
Maybe it's not its breadth, but its depth. That isn't to say that other fields aren't deep, don't get me wrong. But the more tightly coupled with the high-level physical world a field is (think for example medicine or biology), the more it is prone to having technologoical advances from the outside make new sub-fields crop up and old ones die. Think of for example the multitude of research areas made possible by gene editing, or high-resolution NMR imaging.
Of course this happens to some extent in math too, but a lot of subfields aren't killed or born due to outside technological changes. Number theory remains number theory, and still builds directly on centuries of work, even if computer verification has helped in some cases (disclaimer: I'm not a number theorist).
For most subfields of mathematics, you have a lot of depth to cover before you get to the forefront of research. That isn't to say that it's by any means easy to get to the forefront of more high-level physical sciences, but there are certainly subfields in biology or medicine that didn't exist a mere 40 years ago (also true in math, but in general far more rare there).
Also math can hinge on small technicalities. Like Andrew Wiles got a pre review for the proof of fermat conjecture and all was well. I depth review later found a serious gap/flaw which fortunately with hard work and luck he could plug. In contrast if you invent the electron microskop and you get images, you still made the invention (even if small or even big details might be wrong, and the result could be better). In other science often the gist is not effected.
I moved from Physics to Biology. It was quite a culture shock and I ended up leaving after a few years. The amount of shady practices, outright data manipulation, PIs ignoring students just making up stuff was just hard to be in on.
The modern financial system is making a butchery out of honest people. I've seen it happen over and over at many different companies and industries.
Educational institutions are rotting from the inside. Idiots were being rewarded at the expense of intelligent people and now the idiots have taken over control and rewarding other idiots.
If you want to know what happens next, watch 'Idiocracy' or 'Planet of the apes'. At this rate, it will certainly take less than 500 years to get there.
You can see it based on how slow scientific development has gotten; there are very few major new breakthroughs compared to before... Most of the ones that get attention are BS.
Tardigrade DNA is a new one, so popular that it became a major plot point in Star Trek. Turned out it was probably just a sloppy grad student not being careful with their samples/not taking into account microbes physically hitching a ride on the tardigrade
> You can see it based on how slow scientific development has gotten
I feel that the cause and effect are reverse; while the low hanging fruit was available and getting discovered it was a lot harder to get away with fraudulent results. But now that we're facing diminishing returns and more fish in the pond due to years of overtraining fraud is easier to sell.
I just wanted to say that this resonated with me, as a former grad student involved in protein research who is now doing dev work. I hope you're doing well these days!
I'm doing pretty good, thank you! I enjoy coding, I've always been a better coder than a biochemist (likely on account of started coding at age 5, started biochemistry at age 19). I'm still doing garage science here and there, and being a programmer affords me the capability to both afford that and have some time to do it.
i have been thinking a lot about this problem. society has innumerable unsolved problems in healthcare, and many talented people who would like to contribute but cannot.
in software, the open-source model allows people to advance critical initiatives without quitting their day jobs or making onerous commitments.
how can we achieve the same in healthcare, that is let outsiders contribute and advance the state of the art?
I don't know if it will "allow outsiders to contribute" but I would like to see a biotech that makes patent-free drugs. I tried to make nonprofit out of that but there was a lot I didn't understand about how I work, how the world works, and how to get things done, so I will take another crack at it in 5-10 years.
i believe this is not only possible, but will happen sooner rather than later because of advancing capabilities in software, machine learning, and collaboration. we simply need the right people providing capital and launching these biotechs.
re patents, the key is to drive down costs for research and testing. research seems like the low-hanging fruit, comparatively speaking, but it's unclear how to reduce the costs of clinical trials in an uncontroversial way.
> how can we achieve the same in healthcare, that is let outsiders contribute and advance the state of the art?
The Biohacking community is actually really adept, and had made a lot of progress in making Science accessible, prior to COVID you had teams already working together across continents and different time zones. So when someone like Josiah Zayner wanted to tackle a COVID vaccination trial on himself and other biohackers they already had the means and methods ready to go.
The problem is if you want to play by their (academia) rules you're never going to making any inroads, you can't publish and no one will give you a grant for your work, and you're not going to be a chair of anything for your work even if it pans out: but, certain therapies are in development that started off as Biopunk/Biohacker projects.
It's super exciting and hard but also way more work than just BSing your way in academia into a professor role as its all too common occurrence. Professional students becoming mediocre professors was a far worse problem in the Sciences than I could have ever imagined, the one's I really felt bad for were the post docs with actual meaningful research, often with severe social anxiety and poor speaking skills, but were forced to teach undergrad and simply just read the book aloud as 'lecture.' My Organic Chem professor comes to mind, my inorganic professor (did his MSc at Cambridge!) was a rockstar to us undergrads and would do office hours during his lunch hour between lab research and the university made him protest before they'd release back pay during the cuts and layoffs.. it was pathetic and I felt so bad for him, my review was scathing of the University as I left and I've never really forgiven them for that.
Obviously with no VC model in Science to follow for anything but the most brazen outliers (theranos) it's unlikely to happen. Personally I'd volunteer to help middle school or HS kids get involved in plant and Ag science and take some on in culinary if such an Industry still exists in the US after COVID and help them bypass the University track altogether. That is what I focused on after I left working in a lab, but there aren't many avenues for this model to scale to take on massive projects due to a lack of funding. And the money and stability is abysmal, but the Science and fraternity of actual Scientists doing meaningful work is probably more than half of the reason most of us decided to study it in the first place.
Chamath needs to stop pretending to care about politics and solve real problems like funding Community Science wet-labs next to libraries to help the youth care about Science in a meaningful way instead of wasting their time on tik-tok or Instagram with his billions.
Josiah's videos about demystifying the COVID vaccine all got taken down, then the entire channel got shut down. Super disappointing move by YouTube. It was definitely one of my favorite channels for science education.
(squinting suspiciously) ... exactly why did they get taken down? The Algorithm has a well-earned reputation for being capricious, but there's also a ton of good-sounding bullshit out there.
> (squinting suspiciously) ... exactly why did they get taken down? The Algorithm has a well-earned reputation for being capricious, but there's also a ton of good-sounding bullshit out there.
Theories abound, but most/all of these platforms don't have to provide an explanation, and the end-user has little to no recourse on the natter: so far its been youtube, patreon and facebook none of which have followed up. Here is Josiah on an alt platform (odysee) explaining the situation through his eyes [0].
It's sad to see a pioneer of Biohacking dismissing p2p solutions like torrenting and even Bitcoin in order to bypass the censorship, but I think a lot of this just has to do with the clunky nature of its former or perhaps even current UI/UX for people with limited time or attention or familiarity with tech solutions, especially since it was so easy to use Youtube to distribute your content with just a simple click.
I honestly could have him up and running in a day or two with a solution just in case Paypal does in fact shut him down, that would interface with Fiat/CC payment upfront and convert into BTC if needed: the reason BTC is needed is because paypal or bank accounts can shut you out of your funds if you are already a target. It would mainly be a settlements network and only be slightly more steps than what he is used to, as well. But he is right about volatility as that cannot be helped as of right now.
I kind of want to reach out, but I'm dealing with more than I want at the moment due to COVID in my family, but its something I'm considering because Josiah is such a massive inspiration to us Biohackers that deplatforming from the big platforms should be the canary in the coal mine. They even shut down his Patreon!
> Josiah's videos about demystifying the COVID vaccine all got taken down, then the entire channel got shut down. Super disappointing move by YouTube. It was definitely one of my favorite channels for science education.
It seems from his twitter that he even left Oakland for Austin since December when that all went down.
But then look at how WSB was shutdown when it presented a real threat to the establishment. I think this is the same thing happening, but Josiah and the CDC were actually just informing people how gene therapies work in the most biohacker/biopunk way, which is near and dear to my heart for reasons I already explained.
I am not doubting your story, but I know plenty of solid careful scientists doing honest work and being successful. One of these successful scientists self-retracted a high impact paper after he discovered that he had made a data coding mistake. It was painful, but he did the right thing of his own accord, even though it halted work on several follow-up papers that he was drafting.
I am simply providing a counter-example of academic integrity to make the point that one's personal experiences, good or bad, may not reflect what is generally true of academica/science.
I think that probably most people show integrity... but it's a problem if review processes, editorial mechanisms, and culture reward those who don't show it.
The problem is of a lack of data publishing. All data should be published; all conclusions published (preferably with the code that generated the conclusions) so corrections can be made and improved conclusions drawn easily.
Agreed. If you have any recommendations for long-term public data archival they would be greatly appreciated. OSF recently instituted a 50 GB cap which rules out publishing many types of raw data, and subscription options (AWS, Dropbox, etc.) will lead to link rot when the uploading author changes jobs or retires, or the project's money runs out. Sure, publishing summary spreadsheets is a good first step, but there should be a public place for video and other large data files. IPFS was previously suggested but the data still needs to be hosted somewhere. Maybe YouTube is the best option, despite transcoding?
I have no answers. If the scientific community were cooperative enough it could perhaps come up with a shared platform, but it'd be hard to associate budgets with shared resources.
The exact same thing happen to me with a high profile professor who sits on editorial boards of top conferences. He was interested in shilling his dataset far and wide, and will do what ever it takes to show the "value add" of his stuff to other tasks. The smart ones just gave him the number the wanted. I did the work and told him the value add is dubious at best. He found someone else to gave him the number he wanted to tell the story he pre-determined, and made claims in the papers that didn’t even line up with the published results in that very paper. The music goes on and the whole experience is a complete waste of life.
the irony is that the intern after three years of suffering, twisted the arm of our PI and convinced him to do the right thing and got an 11-page retraction identifying and confirming the source of her artifact. This diligence got her a job at a big pharma company.
Of course her paper should have been a cautionary tale, but there are still people using the flawed technique for high-throughput studies to this day.
I don't know anything about this field and I certainly don't know any people involved, but this comment made me curious and after a few google searches I think I can tell who it is too.
Which leads me into some thoughts about not rushing to judgement. I believe the commenter above is doing his best to be a reliable narrator, but it's always possible there was more to this story that was not visible to him at the time, that might exculpate a bit. It's also notable that people change over time, can improve on their faults, and might have learned something in the years since. Best not to view their past mistakes as forever damaging.
I also did a few Google searches and am no closer to figuring out who this is supposed to be :(
For what it's worth, I agree with you that we shouldn't rush to judgement. While its certainly possible in this particular case that there was genuine misconduct, quite often there is a simple misunderstanding.
As an anecdote, during my graduate work I had a fellow PhD candidate convinced his guide was out to sabotage his work, because it 'threatened' to overthrow the guide's long established model. He was convinced that the work of the prior student's work that clashed with his was fudged, and that the PI was covering it up. It is possible? Sure, but not very likely. It's a tad convenient when the people you disagree with also happen to be mustache twirling villains.
I've seen a general trend with young academics at the beginning of their scientific career. They tend to be exuberant, convinced of their own superiority. Until that point, they've tended to be the smartest person in the room, the pick of the lot from among their peers. Hit graduate school, and suddenly everybody around you is just as smart as you, but that appreciation takes a few years to sink in. When your experiments don't work, it's hard to digest and easy to imagine the other guy cutting corners. I'm not suggesting that this is what happened with the top level comment, but could explain many of the other comments I see here.
When there is an open question, with important consequences but unclear resolution, it is hard to know the right answer. Somehow, it is easier to know the wrong answer, and that person will reach for it immediately. So, watch him and choose the opposite.
In any group there is such a person, called the Oracle of Wrong, and almost anybody can tell you who it is. He is the one most likely to wear a trilby, and no wrong choice he has made has ever caused him any personal discomfort.
What an intresting but sadly somewhat common story. Thanks for sharing!
Im a undergrad electronics student so basically a world apart, in terms of skill and department, but this is one of the reasons i do not wish to pursue academia and instead focus on intresting jobs
> In graduate school, in my lab there was a grad student who was kind of an unlikely "professor's pet". He was tall and had surfer's long hair with a bit of a hippie aesthetic. Anyways, he was also really completely clueless about how to do science correctly, but also, I guess, really good about playing politics (there was a time when he asked me to put some bacterial plasmid DNA on my mammalian cells. I told him "it doesn't work that way", but I did it anyways and handed over the cells, and he got the observation he was expecting). On his main project he was teamed up with a super sketchy foreign postdoc that I was convinced would say anything to get high profile papers out.
God damn, just this paragraph alone made me remember why I ran like hell after my undergrad even during the financial crisis of 2008's horrible job market and being up to my eyeballs in debt; I saw the politicking behind what it took just to get a department to give a nod to a tenured professor's peer reviewed paper.
It was fucking pathetic and I've never been more ashamed of my what would be my profession than that but it set the tone for what to expect and made me realize just how irreparably marred that system is. It was followed by a sense of dread that nothing I could do would ever change that and I turned down the offer to work in said professor's lab to carry things on into grad school (MS) and just worked as hard as possible to pay off my debts and pivot my Life entirely. I'd rather sweep and clean floors helping a small business grow into something real than ever go back to that despicable environment.
Academia is definitely a mind-prison, and a trap for so many brilliant minds that may not have ability or wherewithal to try their hand a startup or have the necessary paperwork (citizenship) to take on private sector work, which itself carries a ton of pitfalls.
There are some benefits to the University model but I really hope COVID disrupts the monopoly Universities have over this domain for good! Ed-tech really should be much bigger source of funding and development, but FAANG just keeps suckering in people that could otherwise do something actually useful for Society.
> What do you think he did? Nothing, of course. He kept on the talks circuit, still talking about how exceptional his discovery was, and to date there have been no retractions. He even won the NIH grad student of the year award.
> Oh. What happened to the grad student? He's a professor in the genomics department at UW.
He is literately the academic 'Big Head' character from Silicon Valley that every lab/department has. I'd speak of my own experiences further, nothing as bad as yours, but I really don't feel like ruining my evening any further.
> I am also from a molecular biology background and saw this often. We call these guys the "Golden Boys". They are super successful, but completely useless. If you still believe live is fair, wake up sunshine.
Same, I should have made the leap to Microbiology in JR year, but I just wanted to GTFO and even abandoned by double major (Biochemistry) work just to speed up the process.
> The epilog is that after a decade of floundering I realized that even though I am pretty good at science, I was no good at playing academic politics and quit the pursuit; I drove for lyft/uber for a bit, and now I'm a backend dev.
I'm really sorry. It seems a lot of people are hit by a wall of cruelty. More is less in our lives.
Have you thought of joining some biohacklab to keep enjoying your talent and curiosity on your original field ?
This is fundamentally because the risk-reward incentive structure is absolutely broken right now. If your lab expects productivity and transformative science (basically if you get a Nature paper every month that will be good!) then something will give.
And this rot starts all the way from funding agencies (NIH/NSF/DOE) who have become hardcore bean counters.
I am taking up masters(Materials Science) after a spell in Corporate. I was really hoping to go into research and academia. This is quite disappointing to hear.
Then again, it helps to remove any expectations that any field would be devoid of politics in general. Bit relieved to be disillusioned now, rather than much later.
His work at Scripps matches the same research group and timeframe of when dnautics was there, and he's now a professor in UW's genomics lab. The topic described seems to fit what he was researching then, and he received a prestigious grad student award for it.
Yes, universities will always protect insiders over outsiders. You can go on Twitter or something and post but it will likely just be seen as a ranting by someone who couldn't make it and most outsiders don't understand the system anyway.
What obligation does UW have to listen to him? Better to make a fuss on Twitter and let this get to donors who will have a thing or two to say to the department chair and head of school/deans/provost/president.
As a public institution, UW has an obligation to take seriously and investigate allegations of academic fraud. Besides that they also have a reputation to maintain.
The reputation to maintain is why they will protect the insider over the outsider. Unless it is really blatant or the insider is really junior or new, universities will do everything in their power to silence the allegations.
I'm unsure of how the term "foreign" is being used above. Is it implied as a pejorative there? For example, if OP had written "a super sketchy white postdoc", or "a super sketchy black postdoc", would the HN community tolerate that?
I think in this case it's probably relevant as it does not exactly make fixing these things easier. For example, later he points out he can't talk the language of the institution this person works at. I don't think it's meant to accuse foreigners of fraudulent science here.
Sure, I guess i should have been more specific, he was a postdoc who was from a country where getting one really awesome paper in a lab with a moderately good name (which ours was) would be an instant ticket to tenured professorship at the top academic facility in the country. That should give you an idea of the incentives at play. That doesn't necessarily make him sketchy. But he was also a sketchy human.
It's not about race, it's about the quality of the academic system, which is bad in many countries. I suppose GP intended it to compound - as in the guy was sketchy per se, and from a sketchy place.
> the academic system, which is bad in many countries. I suppose GP intended it to compound - as in the guy was sketchy per se, and from a sketchy place.
All you're arguing is that it's "not the most sketchy" not that it's less sketchy than the average. And you're offering no proof or example of that either.
If you're going to get hung up on technical details, note that GP didn't ask whether it's less sketchy than average, just whether it's less sketchy.
Finding concrete proof or examples is obviously hard in this subject matter (how are you going to prove something as abstract as sketchiness), but here's one observation: predatory conferences mostly only exist outside the West. To be even more concrete, two of the most infamous predatory publishers (WASET and OMICS) are based in Turkey and India respectively. You generally won't find something nearly as sketchy in the West.
>two of the most infamous predatory publishers (WASET and OMICS) are based in Turkey and India respectively
Well, as an Indian postdoc working in the US, I can speak to some of these sketchy behaviours. In terms of the predatory publishers, my Indian institution had its own filters, and most labs have their own as well. For example, for a while we had an institutional restriction on submitting manuscripts to conference proceedings, with the justification that the hard time limit equals substandard peer review. In addition, for the longest time we were not allowed to submit anything to open source journals, with similar justifications. Publishing in a journal with an IF < 8 was also frowned upon, and the institution would not cover publication expenses. AFAIK other institutions had similar filters for publications. I would regard my institution as a decent, but nowhere near the best in my field in India.
Who does publish in these predatory journals? Smaller, less well funded universities with desperate students, ever since the government mandated first author publications as a requirement for receiving PhD degrees.
So they did a series of experiments and reported results that screamed "artefact". On one of them, for example, the postdoc got trained to use the electron microscope and they went through thousands and thousands of images to pick out the one that had "just the right morphology" (I am pretty sure they were snapping photos of salt crystals). On another, they reported that their research subject protein was so fast at the process we were studying that everything occurred IN MIXING TIME. That to me, screams "you are not doing your experiments carefully".
Meanwhile I was sweating balls working on a very careful preparation of similarly finicky proteins (you agitate them and they do bad things since they're metastable) and finally got it to produce reproducible results. I suggested they adapt my preparation to their protein but they couldn't give a damn, they had already published their paper and had moved on to sexier proteins.
But then an intern was put on the project, and she could not reproduce their results, after working on it for six months (she is careful and honest). At the end, I felt so bad for her, I offered to train her on my technique, but she passed. I think she was burned out on the project. I asked if I could get a sample of the protein that she had prepped, and she agreed.
I ran the protein through my preparatory technique and observed that there was a contamination that could have seeded the kinetics of their process. Upon isolating an uncontaminated sample, I carefully but briskly rushed the sample over to the machine. Nothing. Curious, I jacked the temperature up to get it going faster. Nothing. I left it in the machine overnight. Nothing. Finally, convinced that I had likely done something wrong, I dropped the sample in a shaker at temperature, came back the next day and recorded amazingly high signal. In short, the observation that it was "super fast" was entirely an artefact.
As I, too, was trained on the Electron Microscope, I quickly spotted my sample onto an EM disc, reserved some time and hopped on the 'scope. The first grid sector I looked at, there was literally TEXTBOOK morphology in front of my eyes.
I stapled together my results, gave it to the grad student, and told him that the general gist of his paper was probably still correct, but that he should be careful about characterizing his protein as exceptional. I then said it was in his hands to do the right thing.
What do you think he did? Nothing, of course. He kept on the talks circuit, still talking about how exceptional his discovery was, and to date there have been no retractions. He even won the NIH grad student of the year award.
The epilog is that after a decade of floundering I realized that even though I am pretty good at science, I was no good at playing academic politics and quit the pursuit; I drove for lyft/uber for a bit, and now I'm a backend dev. I am certain that my experiences are not unique. Amazingly the intern returned to our lab, and had her own three-year stint chasing ghosts that turned out to be overoptimistic interpretation of results reported by a postdoc.
Oh. What happened to the grad student? He's a professor in the genomics department at UW.