View Full Version : article about verifiable study results/medical journals/open access information.


sarahsweets
10-15-13, 05:35 AM
http://abcnews.go.com/blogs/health/2013/10/14/online-medical-journals-accept-fake-study-could-the-public-be-hurt/

TygerSan
10-15-13, 08:59 AM
When this study was published, I knew it was going to be misinterpreted by the press, and I was right. Even NPR, who generally have decent science reporters, seemed a bit confused about the point of the study.

Ironically, since the study in question is published in Science and is therefore not open access, I can't currently read it to comment on the merits of the study directly.

That said, what isn't being made clear in all this is that peer review - that is the process by which an research article written by a team of scientists is read by some other scientists in the same field and evaluated on its scientific merit- *isn't designed to sniff out outright fraud*. Peer review is just like what it sounds like- your article is read by people who have similar interests and research programs and they evaluate your methods, your analysis and whether your conclusions follow logically from the former. The assumption from the get go, though, is that your research is real.

Not only that, but the major issue being examined by the article in the original post is not that a fake study got accepted into so-called reputable journals, but that a study that was designed specifically with a major logical flaw that should have been red-flagged by reviewers was not and the papers were accepted anyways, specifically by journals that were open access.

Frankly, delving deeper into this issue I'm not sure that the fact these journals were open access is the main part of the problem. Peer-review is kind of a crapshoot anyways. it really depends on who you get is your reviewers and how much time they have. Some more senior researchers who are pressed for time often delegate that task to people who have less experience like postdocs.

The difference between open access journals and the more traditional journals, is that if you can read an article without a subscription, it is probably an open access journal. If you need a subscription to read then it probably is not. Personally, I like to be able to read articles without having to pay $40. This is why open access is not really such a bad thing.

The flipside to that, though is that open access journals charge the authors of their papers to publish in the journal. This can seem dangerously close to a self-publishing model, where money trumps quality. The better journals still do have peer review, though. The quality of that can vary similarly as in the traditional model.

The study that this news article was based on was published in science. Science is one of the big players in the traditional publishing industry. I hate to question motives, But they do have a stake in maligning the open access publishing industry.

If I had to drive parallel between science and nature and a similarly positioned open access journal, I would probably say that journal would be PLOS-one.

Amtram
10-15-13, 10:16 AM
I think the point of the study (I don't have access, either, but I've been reading the blogs of many scientists who do) was to show that universal open access is not the laudable goal that some of its proponents espouse. PLoS has a peer review system, and they were pretty much the only journal to catch the scam. They're also protesting the loudest, because they really don't like the fact that the low-quality and pay-to-publish journals were included under the term "open access," but I think they're ignoring the reality there.

While peer review isn't perfect, it serves a very important purpose. Anyone who gets information on anything scientific using a google search or a popular science aggregator or a commercial media outlet gets exposed to science as interpreted by people who don't know squat about science. Even after a finding has gone through peer review and publication, even when the authors and perhaps other experts in the same field are interviewed, the findings still get twisted and messed around and the general public has no idea what was actually discovered.

Throw in "open access," in the inherently flawed model that exists with these thousands of journals that don't have stringent standards for research or peer review, and the situation is ripe for the public to be bombarded with stuff that started off as misinformation before journalists and bloggers even got a hold of it.

PLoS's version of open access is everyone being able to read directly from reputable, high Impact Factor journals without having to pay hundreds of dollars for a subscription and proof of your educational background, or $30-50 to Scripps or Elsevier for each full paper. They want open access to valid, well-designed, impactful research. Problem is, "open access" would work for all the awful research, too. And the awful research is understandable to anyone with an 8th grade education (while the good research often makes little sense to people who aren't specialists in the field.)

This mega-Sokal (http://en.wikipedia.org/wiki/Sokal_affair) was meant to point out these shortcomings in open-access advocacy. I think it was important. It seems to me that a lot of scientists who are in favor of open access are unaware or even deliberately ignorant of the negative potential that needs to be thought through before open access can be a reality.

TygerSan
10-15-13, 10:34 AM
I would emphasize also, that just as there are crappy open access journals, there are crappy traditional journals. Neither model has a monopoly on poor quality.

I've gotten some really perfunctory and poorly thought out reviews for a paper we were submitting to a top-tier specialty journal and got reviews from people that were perfunctory and clearly missed the point.

Amtram
10-15-13, 01:16 PM
Absolutely. However, a crappy piece in a good journal is going to be spotted and called out by scientists who know it's bad much more quickly.

dvdnvwls
10-15-13, 01:54 PM
Any journal can be good, open-access or closed-access. I think an important part of this discussion is "What incentives or requirements need to be present in all scientific journals in order to make sure that most of the low-quality research never reaches the eyes of the public?" Stating it positively (publish only the good) wouldn't have the same effect - I think each publication does need to have people who are specifically hunting for bad pieces to throw out.

SB_UK
10-15-13, 02:17 PM
Publish or Perish

- the problem is funding.

You won't get any if you don't -- forcing all people into a publication-fest regardless of merit.

It's not about science - it's about survival.

If you can do some research easily and publish a decent paper on it you will.

From yesterday 'well of course they're saying that gene xyz is important in eosinophils - they're eosinophil experts (have all the assays up and running) and the gene is hot'

- it's opportunism
- and they published in Nature ?? communications whatever that new 'stretching the brand' journal is.

Permits scientists to boast about their Nature publication ? I don't know.

TygerSan
10-15-13, 02:25 PM
Publish or Perish

- the problem is funding.

You won't get any if you don't -- forcing all people into a publication-fest regardless of merit.

It's not about science - it's about survival.

If you can do some research easily and publish a decent paper on it you will.

From yesterday 'well of course they're saying that gene xyz is important in eosinophils - they're eosinophil experts (have all the assays up and running) and the gene is hot'

- it's opportunism
- and they published in Nature ?? communications whatever that new 'stretching the brand' journal is.

Permits scientists to boast about their Nature publication ? I don't know.

and what SB says is spot on. It's why studies published in Nature, Science, and other prestigious journals are often rushed and not of the highest quality: you have a sexy result, you have to get it out *now* not wait for the replication or confirmation that will strengthen your case.

A lot of times it's more about the story and interpretation than the results anyway. The salesman rises faster than the worker bee, and in this climate, you have to stand out to get funded.

SB_UK
10-15-13, 03:00 PM
The salesman rises faster than the worker bee, and in this climate, you have to stand out to get funded.

That's it - there's also a shift to - if you can pay for the latest technology - you'll get a bigger publication - get more funding

- which is selecting for salesmen scientists.

The pace of change currently is dizzying - this new machine - the HiSeq is incredible and very very very expensive to buy and run.

It's almost as though scientists have become microbes - and the opportunist pathogen with most virulent growth characteristics (ruthless salesmen) are overgrowing the community it formerly! shared a Petri dish with.

If you owned a car dealership - would you prefer people who lied effectively to clinch the deal - or people who told the truth ?
If you wanted to buy a car - would you prefer people who lied effectively to clinch the deal - or people who told the truth ?

SB_UK
10-15-13, 03:15 PM
wow! wikiP has it all ... ... ...
http://en.wikipedia.org/wiki/Publish_or_perish

SB_UK
10-15-13, 03:16 PM
Frequent publication is one of few methods at scholars' disposal to demonstrate academic talent. Successful publications bring attention to scholars and their sponsoring institutions, which can facilitate continued funding and an individual's progress through their field. In popular academic perception, scholars who publish infrequently, or who focus on activities that do not result in publications, such as instructing undergrads , may find themselves out of contention for available tenure-track positions.The pressure to publish has been cited as a cause of poor work being submitted to academic journals.Too funny! (in an entirely black humour sense)

namazu
10-15-13, 03:25 PM
So, I thought it might be useful to give some background for people who aren't intimately familiar with the world of academic publishing in science. There's a lot of terminology that gets thrown around with respect to publications, and it can get confusing (and is often conflated by the media).

There are a few different issues at play here, that have gotten mixed up (in part, because they are related in real life, and in part because the terms don't always get used in the way they were originally meant):

https://lh3.googleusercontent.com/-TDYy5FOP6do/Ul2NETc8oWI/AAAAAAAAG2E/fy-lCllCKTk/s582/review.jpg
*Note that the quality of the study (how well it was designed and carried out) and its importance/attractiveness (in terms of how ground-breaking the research is -- or sounds) are separate issues! SB_UK, TygerSan, and Dvdnvwls have all mentioned this above, but it's worth repeating.


https://lh6.googleusercontent.com/-s0CpG_Ygfv8/Ul2NDhimy1I/AAAAAAAAG14/TwOlTZo5UNQ/s619/charges.jpg
*Note: some journals require this fee only from authors at institutions in developed countries, or with grant money that covers it, but may waive the charges for authors in developing countries, students, people without big grants…

https://lh4.googleusercontent.com/-zvGDFhCyIzE/Ul2ND5vv0-I/AAAAAAAAG18/qu2RNvpo_DY/s581/publisher.jpg

https://lh5.googleusercontent.com/-XD71aIJJ9JY/Ul2NDDKjP8I/AAAAAAAAG1w/r4MgqvJG70g/s576/access.jpg
* Many journals keep the articles behind a paywall by default, but then after a certain period of time, often 1 year, the articles revert to “open access”. New U.S. regulations requiring researchers who get federal grant funding to make their results available to the public have resulted in many journals adopting this “'embargoed' for a period of time but then opened up” policy.


Now, there are all different kinds of combinations of these models -- pick an option from each chart and put them together, and there's probably a journal like that. And lots of gray areas open to interpretations, and things that don't quite fit these models. (I'll spare you the Venn diagrams!) And you can't necessarily tell how good a journal is, or how good any individual paper within a journal is -- i.e. how well-done and how valuable the science is -- based solely on which boxes are ticked.


Although the coverage of the Science paper referenced in the article SarahSweets posted spawned many headlines casting aspersions on "open access" journals, the problem really has to do with the combination of poor (or nonexistent) peer review and publication charges.


There are a lot of new publications (or "publications", cough cough), made possible by the internet, that are run by less-than-scrupulous groups or individuals, which will basically accept anything submitted to them in exchange for a fee. And there are lots of scientists (or "scientists", cough cough) whose career advancement depends on publications (and some institutions focus on "how many" rather than "how good").


So it's "win-win" (not really) in the sense that some desperate researcher can add a line to his/her CV (for an outrageous fee and without necessarily having produced anything of value), and some publisher (possibly a scammer or close to it) makes a buck off this person.

But it's lose-lose for the public, because it floods the internet with "peer-reviewed science" (often in name only) that doesn't have any of the quality control that's supposed to go along with it, and skews the metrics by which researchers who do good, honest, valuable work get judged.


Now, I should reiterate that peer review isn't perfect or a guarantee of quality, there are sometimes legitimate reasons publishers charge for publication, and sometimes really good research unfortunately gets shafted by traditional review systems.


But adding profit motives tends to increase the likelihood of problems. In addition to those "desperate researchers" stuck in the churn of "publish or perish", there have been instances where parties with vested monetary interests (e.g. trade groups, corporations) can use these less-than-scrupulous journals to publish results that help support their bottom line. (And this isn't limited to new/small publishers, either; there was a scandal not too long ago where Elsevier, I think it was -- a major for-profit publisher that has some good-quality, respected journals -- was caught publishing some journals that were basically subsidized by, and mouthpieces for, pharmaceutical companies...)


It's true that many of the sketchy/"predatory" journals, including the ones called on the carpet by this paper, are "open-access", because open-access is trendy (and, in principle, a good thing), and easy to do when you are publishing only online. Also, many institutions now provide incentives to publish in open-access journals (again, on principle). But the problems with these journals arise not *because* they are open-access -- they just drag the "open-access" label into the mud with them.


Anyway...there are lots of related issues tied into the ones raised by this article -- benefits/flaws of the traditional peer-review system as "gatekeeper", and alternatives to that system; the unpaid labor that goes into peer-review, even when the publisher is for-profit; the responsibility of researchers and publishers to make science (publicly-funded or otherwise) available to the public; etc.


But at heart, the problem isn't open-access, which is (in my view) the "right" and "socially-responsible" way to publish.


It's the perverse financial and career incentives that allow publishers and researchers to get away with profiting from dreck that doesn't advance science, and the institutions that are complicit in this.


Full disclosure: I have published in
- non-profit, open-access, peer-reviewed, no-fee journals; (my favorite kind)
- non-profit, paywalled, peer-reviewed, no-fee journals;
- non-profit, hybrid access, peer-reviewed, fee-to-publish journals;
and
- for-profit, paywalled, peer-reviewed, no-fee journals.
and reviewed papers (for free, on my own time) for the first two kinds.

Amtram
10-15-13, 04:23 PM
That was excellent. I think you pretty much covered all the places where the system can fail (or succeed) when it comes to publication. Unfortunately, there's no way to cover all the bases - and even if you did, there would be people out there misinterpreting even the best, most solid research!

SB_UK
10-16-13, 04:52 AM
~

Impressive post.

Nothing to add ... ...

qinkin
10-17-13, 09:11 PM
But adding profit motives tends to increase the likelihood of problems. In addition to those "desperate researchers" stuck in the churn of "publish or perish", there have been instances where parties with vested monetary interests (e.g. trade groups, corporations) can use these less-than-scrupulous journals to publish results that help support their bottom line. (And this isn't limited to new/small publishers, either; there was a scandal not too long ago where Elsevier, I think it was -- a major for-profit publisher that has some good-quality, respected journals -- was caught publishing some journals that were basically subsidized by, and mouthpieces for, pharmaceutical companies...)


From what I read about this in a sizable article published in the news section of Science, was that various journals' lack of scrutiny towards what was published despite appearing morally superior on the outside. With the worthless journals, money is made in the process, but this happens in everything I can think of. However, can we expect the Journals that have been exposed to publish BS to last long with practices like that or make anywhere near the amount of money the good journals do? "Nonprofits" still make money and the staff still indeed profit, don't they?

http://www.sciencemag.org/content/342/6154/60.full

It's the perverse financial and career incentives that allow publishers and researchers to get away with profiting from dreck that doesn't advance science, and the institutions that are complicit in this.


Well it's a combination of lacking responsibility with probably varying incentives. Should one not have an incentive to further their careers or finances? I'm not seeing where incentives allow people to get away with bad science, though. I think Open Access is better for everyone, including the journal itself. I think Open Access can work and has worked in a market setting.

The rejections tell a story of their own. Some open-access journals that have been criticized for poor quality control provided the most rigorous peer review of all. For example, the flagship journal of the Public Library of Science, PLOS ONE, was the only journal that called attention to the paper's potential ethical problems, such as its lack of documentation about the treatment of animals used to generate cells for the experiment. The journal meticulously checked with the fictional authors that this and other prerequisites of a proper scientific study were met before sending it out for review. PLOS ONE rejected the paper 2 weeks later on the basis of its scientific quality.


Other publishers are glad to have dodged the bullet. "It is a relief to know that our system is working," says Paul Peters, chief strategy officer of Hindawi, an open-access publisher in Cairo. Hindawi is an enormous operation: a 1000-strong editorial staff handling more than 25,000 articles per year from 559 journals. When Hindawi began expanding into open-access publishing in 2004, Peters admits, "we looked amateurish." But since then, he says, "publication ethics" has been their mantra. Peer reviewers at one Hindawi journal, Chemotherapy Research and Practice, rejected my paper after identifying its glaring faults. An editor recommended I try another Hindawi journal, ISRN Oncology; it, too, rejected my submission.



Ironically, since the study in question is published in Science and is therefore not open access, I can't currently read it to comment on the merits of the study directly.


Not sure if whatever study they are referring to is published in Science, I found a link to BMJ published on same day as the journal article found at the bottom of the link I posted. John Bohannon is a correspondent for Science, tho.

namazu
10-17-13, 10:10 PM
From what I read about this in a sizable article published in the news section of Science, was that various journals' lack of scrutiny towards what was published despite appearing morally superior on the outside.
As far as I can tell, these journals don't claim to be "morally superior" in terms of anything except providing access to the final published work. (And some of them don't even hint at that; they just use "open-access" as a buzzword, without regard for real concern about information being available to the public.)

As some of the commentary on Bohannon's work noted, he didn't submit to any non-open-access journals, but it's likely that some of them might have been fleeced (some willingly) in exchange for publication fees.

With the worthless journals, money is made in the process, but this happens in everything I can think of. However, can we expect the Journals that have been exposed to publish BS to last long with practices like that or make anywhere near the amount of money the good journals do?
Probably not so long. But as long as there are unscrupulous people, and people not paying attention to what's happening, there will be opportunities for fraudulent operations to make a buck.

"Nonprofits" still make money and the staff still indeed profit, don't they?
Sure, they tend to make some money -- at least one hopes they make enough to keep their operations running, and perhaps to subsidize worthwhile, but lower-profile journals/projects in their portfolio.

And we could argue about what's fair compensation for employees/directors of nonprofits, and so on...

My criticism is not solely of the profit-generation motives; as I noted above, there are good-quality journals in both the non-profit and for-profit categories.

But I do see as problematic a few things:

1) People/organizations that are essentially scammers (as some of the more egregious fake-journals are) cashing in on the academic publishing market. Some of these new-on-the-scene "journals" (but not all!) are basically websites with a bank account attached, and little else -- "Nigerian spam" disguised as academic publishing.

2) The concept of "open access" being equated -- unfairly, in my opinion, even if the term has been coopted by some of these scammy outfits -- with "low-quality junk".

3) Larger, more established publishers relying on scientists (many of whom are supported by public funds in the form of grants, etc.) to supply articles and to do most of the review/editing/etc. for them, uncompensated by those for-profit journals, and yet turning around and charging the public to read the final work, and sometimes making it difficult even for authors to reuse their own material as they see fit.


Well it's a combination of lacking responsibility with probably varying incentives. Should one not have an incentive to further their careers or finances? I'm not seeing where incentives allow people to get away with bad science, though.
I don't have a problem with people being motivated to support themselves or to further their careers.

What I have a problem with is the institutions that demand that people tick X number of "publications" boxes as a benchmark for hiring or promotion, without regard to the quality of those publications, and the people who play that game without compunction. As long as merely "having a publication" (never mind if it's junk) counts for something somewhere (which it most certainly does), people will have incentives to publish junk.

And profit motives for the unscrupulous publishers -- i.e., the more articles we publish, the more publication fees we rake in, quality be damned! -- certainly allow people to get away with bad science.

(The same could be said of various forms of research or other business misconduct -- they may be means to an end, but I sure as heck don't think it's right for people to commit fraud simply because it will further their careers! I wouldn't equate publishing in a bottom-of-the-barrel journal with these forms of misconduct as a rule, because it's certainly possible for very good research to find its way to not-very-good journals. But there are some parallels, especially when the person placing the article is aware that the peer review is nominal but nonexistent, or that regardless of the quality of the science, it will be published for a fee...)

Not everyone will voluntarily engage in these practices -- there are plenty of scientists -- the majority, I hope and believe -- who pride themselves on doing good work that advances knowledge in small or large ways.

But if there's a finite number of jobs available, and increasing competition for the available pool of funding, there will be people (and institutions) who get caught up in numbers games and trying to play the system. And some of them succeed, at least for a while, and to the detriment of honest (and equally in need of supporting themselves) researchers who are driven more by "scientific" motivation.

This (various forms of playing publications games) happens with traditional journals as well, as there's a range of quality and selectivity -- but once you add honest-to-goodness "we'll publish anything if you pay us!" operations, the seamy side of publishing gets that much seamier.

To the extent that poorly-done meta-analyses fail to filter out some of this poorly-done (or funded-by-special-interests) work, the problem of some of this stuff working its way into the "scientific literature" (cough, cough) can really become a problem and a threat to health -- because decisions about policies and insurance coverage and so on are made based on this "evidence".

Good meta-analyses tend to have quality controls or quality ratings that help reduce that problem (though other types of publication biases may still be present) -- but the proliferation of junk journals makes it easier to disseminate just about anything with the cachet of "science".

I think Open Access is better for everyone, including the journal itself. I think Open Access can work and has worked in a market setting.
I wholeheartedly agree!

My point certainly wasn't to bash the Open Access model -- actually, I thought I was defending it. I would love it if all journals were OA.

What I disapprove of is the proliferation of crappy journals with less-than-ethical and less-than-transparent operating procedures, as well as those who would imply that these lapses in quality and/or ethics are somehow a flaw of the OA model.

Not sure how coherent this post is -- I suspect "not very" -- but I guess I'd summarize by saying:

- I think open access is the way of the future, and the socially responsible way to publish.

- I despise those people and systems that have no qualms about corrupting science in the name of personal or corporate profit, or because they are too wrapped up in "prestige metrics" to actually care whether they're fulfilling their ostensibly-noble missions.

- I recognize there are tons of gray areas all around and it's hard to make blanket statements about any model without oversimplifying things.


Incidentally, it's not just written publications that are like this. There are plenty of "vanity" conferences that I see as morally equivalent to some of the "we don't care what you wrote as long as you pay" type journals. Again -- the problem isn't open-access on the far side! It's poor quality control and those who enable it because it serves their less-than-noble ends. (Trip to Hawaii paid for by taxpayers... Line on CV suggesting prestigious talk... Conference fees lining someone's pockets... Never mind that content and value to the public may be minimal.)

Amtram
10-18-13, 09:52 AM
I found a thought-provoking article from an open-access publisher who had rejected the fake paper, but whose rejection was not included in Bohannon's statistics. It also has links and references to a number of journals that have highly questionable reputations and publish as "open access" that were included in the study.

Unscientific spoof paper accepted by 157 "black sheep" open access journals - but the Bohannon study has severe flaws itself (http://gunther-eysenbach.blogspot.ca/2013/10/unscientific-spoof-paper-accepted-by.html)


He makes some interesting points, and it's certainly nice to have a list of journals that are worthless for future reference.

qinkin
10-19-13, 04:18 PM
1) People/organizations that are essentially scammers (as some of the more egregious fake-journals are) cashing in on the academic publishing market. Some of these new-on-the-scene "journals" (but not all!) are basically websites with a bank account attached, and little else -- "Nigerian spam" disguised as academic publishing.

2) The concept of "open access" being equated -- unfairly, in my opinion, even if the term has been coopted by some of these scammy outfits -- with "low-quality junk".

3) Larger, more established publishers relying on scientists (many of whom are supported by public funds in the form of grants, etc.) to supply articles and to do most of the review/editing/etc. for them, uncompensated by those for-profit journals, and yet turning around and charging the public to read the final work, and sometimes making it difficult even for authors to reuse their own material as they see fit.



You can be OA and be perfectly unethical. U can be egalitarian and be perfectly unethical as well. Being OA or whatever can be it seem as if those sources are conscientious, but it's just not as black and white as that.

Well it's true however, open access doesn't always equate to high quality and one can guess that there'd still be problems regardless of monetary incentive or being closed access. I think the study was a great thing and needs to be done on a regular basis and improved upon as well. I don't have much reason to believe it was anti-open access based on Bohannon's piece. I'm not going to speak for others about their counter blogs or whatever. Open Access is gaining more influence and needs to be monitored. Not sure we can depend on just one source for it all. I did not get the same impressions as a few authors that were made to counter it, but they have some decent points, though I wouldn't agree that Bohannon's piece gave the impression that Open Access journals are **** in general. It's actually and clearly stated in the piece, that it is not anti-OA journals.

https://petitions.whitehouse.gov/petition/require-free-access-over-internet-scientific-journal-articles-arising-taxpayer-funded-research/wDX82FLQ

Link is directed towards 3)



I found a thought-provoking article from an open-access publisher who had rejected the fake paper, but whose rejection was not included in Bohannon's statistics. It also has links and references to a number of journals that have highly questionable reputations and publish as "open access" that were included in the study.

Unscientific spoof paper accepted by 157 "black sheep" open access journals - but the Bohannon study has severe flaws itself


He makes some interesting points, and it's certainly nice to have a list of journals that are worthless for future reference.


Yeah I dunno about "severe flaws" or even if it was "flawed" in the sense of being invalidated. I noticed that Bohannon made some replies in the comments section. And yeah, I mentioned above that it wasn't really an attack on OA. My default position, I would say, is that the Bohannon related study underwent more rigorous scrutiny than the counter blogs before being published. The blog was informative, but I'm not sure i agree with the premise.

namazu
10-19-13, 04:48 PM
You can be OA and be perfectly unethical. U can be egalitarian and be perfectly unethical as well. Being OA or whatever can be it seem as if those sources are conscientious, but it's just not as black and white as that.
I don't think anyone said it was black or white...so I'm not sure what you're arguing against (if anything...maybe you were just seconding what others have said upthread?).

In my opinion, "open-access" -- meaning, articles are available for free to the public -- is broadly a desirable and ethically-appropriate thing, for reasons I've noted above, and which I think you're referring to with the petition you linked.

Whether any given journal that labels itself "open-access" is a good journal (in terms of quality and/or ethics) is another matter.

The "author pays to publish" aspect is common but not universal among open-access journals, and it is present in some non-open-access journals as well. Although it is tied closely to issues of open-access (because if the journal isn't getting revenue from subscription or article fees, it probably has to make up that revenue somehow), it's really a separate thing. And those charges may be legitimate (if they cover actual expenses in producing and distributing the article) or they may be excessive.

I don't have much reason to believe it was anti-open access based on Bohannon's piece.

I don't know what Bohannon's intent was, but I think if his interest was primarily in exposing less-than-adequate peer review, it would have been worthwhile to send his paper to a wider range of journals, not only "open access" ones. By limiting his experiment to open-access journals, he gives the impression that such journals are more likely to have inadequate review processes. This may be true, and Bohannon may not have directly made such an argument, but that's the overall impression that he (or, more accurately, media coverage of the story) gives.


My default position, I would say, is that the Bohannon related study underwent more rigorous scrutiny than the counter blogs before being published.
It's hard to say -- I don't think Bohannon's "study" was peer-reviewed, either. I don't know who may have scrutinized it prior to publication.

Amtram
10-19-13, 05:54 PM
Even if Bohannon's study was biased or that bias somehow made his numbers flawed, the fact that it shed some light on the shortcomings of "open access" so that people who can influence the accountability of the publishing process can fix the problems it found. . .well, sunshine is the best disinfectant.

qinkin
10-19-13, 07:01 PM
I don't think anyone said it was black or white...so I'm not sure what you're arguing against (if anything...maybe you were just seconding what others have said upthread?).


Oops made a typo.. o well I think you understood what I was saying.. I'm really just trying to make my point clear enough to be understood.

I guess it still is unclear whether this is a study or what, the BMJ link is just to another news article.

In my opinion, "open-access" -- meaning, articles are available for free to the public -- is broadly a desirable and ethically-appropriate thing, for reasons I've noted above, and which I think you're referring to with the petition you linked.


I'm referring to the fact that it's been an issue that has been improved upon.. Meaning, when research is funded by the public, open access is increasingly becoming a requirement.

Anyway, I'm not everyone, I'm not the media, and I will trust my interpretation as accurate. I really can't do anything for people that read into things, what isn't there to begin with. There's no real breach of morality going on in the article, is my take.

namazu
10-20-13, 12:52 AM
sunshine is the best disinfectant.
I agree with this.



Still, I'd caution against viewing this as a "shortcoming of open-access", rather than as a problem of proliferation of poor-quality or even fraudulent "journals". I see it as more of the latter. That some of these journals use their "open-access" status as a guise to charge questionable publication fees is a problem, for sure, but the actions of some sketchy journals shouldn't tarnish the whole "open-access" enterprise.



Meaning, when research is funded by the public, open access is increasingly becoming a requirement.
Many journals keep the articles behind a paywall by default, but then after a certain period of time, often 1 year, the articles revert to “open access”. New U.S. regulations requiring researchers who get federal grant funding to make their results available to the public have resulted in many journals adopting this “'embargoed' for a period of time but then opened up” policy.
I don't disagree with you there... ;)

SB_UK
10-20-13, 02:57 AM
This is just a debate about the 'rights and wrongs' of money.
It's occurring in parallel in nearly each and every domain of human endeavour.

There's no point in trying to solve the specific problem we're facing here - and need to tackle the wider problem.

Money.

As stated many times over - money post-WWII with decreasing rate of increase of global population [economic growth] has made money itself - a 'sinking' ship.

Money is the root problem described on this thread.
Money has a limited shelf life (and will collapse at some point in the near future).

Perhaps we should just put in place a replacement - and then watch as problems such as the one described on this thread - are eliminated.

It's going to happen anyway.

What's the mechanism ?
Simply co-operative structures for life's essentials - eliminating need for money - at which point it'll be OK for it (that system) to follow the path of exponential decrease in worth.

Have a look at the rate of increase in American debt - and you'll see - that we're on the exponential phase uppawards.

Exponential increase in consumption is all that'll keep the economic system in place.
https://fbcdn-sphotos-d-a.akamaihd.net/hphotos-ak-ash4/p480x480/1380243_531501856932474_808183173_n.jpg
Now - do you want exponential increase in consumption ?

How do we do it - in a soon-to-be reducing global population ?

And it's well known that over-consumption (obesity, factory pollution etc) leads to a terrible life.

-*-

Time to cut back.

SB_UK
10-20-13, 03:06 AM
In my opinion, "open-access" -- meaning, articles are available for free to the public -- is broadly a desirable and ethically-appropriate thing, for reasons I've noted above, and which I think you're referring to with the petition you linked.

I guess I'm suggesting that "open-access" to EVERYTHING is a good thing.

But only prevented by money.

:-) Sure - in a world without money - we wouldn't all be able to have a big Mercedes Benz - but we could all share one - if people still wanted a thing which all people had.
I wonder whether the 'cachet' of all of the things people covet - lies in their exclusivity ie if we all could have a go in a Merc, try on a Gucci and then use an Apple computer
- then people 'd just get bored of 'em and turn to self-service bikes, quick drying clothes that can be washed in the shower and some cloud based computer which folds into your pocket.

IE wonderful techs which're liberating - no more loss of bikes handing them into to be repaired and coming back broken [yes - happened many times], no more washing/ironing and having to reinstall operating systems/software when the whole computer grinds to a halt. Who has the patience to wait that long ?

Oooooo nice ! Really nice (http://en.wikipedia.org/wiki/Cloud_gaming) and it's best friend (http://www.zeitnews.org/applied-sciences/computer-science/world-record-wireless-data-transmission-100-gbits).

The best thing (open access to all) and what we currently do (exclusivity) are NOT compatible.

I could specify the 'best thing' in 5 minutes - it could be circulated and amended (in its entirety possibly) by the planet via facebook in 5 mintues - and we'd be off.

Bottom line - survival essentials - all for themselves - co-operative structure.
Yes - the Queen, George Clooney and Jennifer Lopez could be next to you on a building site exposing their nether regions as they reach for mortar.

And that'd be true equality.

-*-

As of nerve - it became all about information.

SB_UK
10-20-13, 03:46 AM
“Hence, radio systems having a data rate of 1 terabit per second appear to be feasible.”We've rendered medical research as it stands - obsolete through community formation by virtue of information technology.

What will be published in the future - will tend towards open access art - which'll be delivered for free to you.

In that happy, unstressful world - we'll find the diseases of Western living (diseases of di stress) eliminated.

Or from years back now - 'a hobbit village sitting atop unimaginable wireless bandwidth'

- quality of sound/music activates our dopaminergic system.

MX2012
10-20-13, 10:56 AM
While other posters may have noted this but many studies are influenced by where the funding comes from, by who requests the study, and who reviews the study. Today, many peer review panels include a "business" perspective, so this tends to diminish negative results that reflect on business practices.

namazu
10-20-13, 06:52 PM
While other posters may have noted this but many studies are influenced by where the funding comes from, by who requests the study, and who reviews the study. Today, many peer review panels include a "business" perspective, so this tends to diminish negative results that reflect on business practices.
Are you talking about review for grant funding, or for publications?

I've not heard of business perspectives being included as a criterion in peer review for journal articles -- in fact, when I've reviewed articles, the journals have required me to disclose any potential conflicts of interest -- but I could imagine some funding providers being interested in potential "return on investment".

There is publication bias, both because negative results tend not to be "sexy", and because, as you noted, they can impinge on corporate interests' bottom lines. But that's usually prior to the peer review stage -- many such results aren't submitted for publication in the first place. There have been efforts/rules to combat this, such as registration of clinical trials prior to their conduct, so that they can't simply be swept under the rug.

A peer reviewer who recommends against publication simply because results may threaten the commercial viability of a product, and not because of problems in study methodology or interpretation, is acting unethically. (That's not to say it never happens, but it's not generally encouraged.)

Amtram
10-20-13, 08:22 PM
Researchers are required to disclose conflicts of interest, and studies need to disclose funding sources. It doesn't take a PhD to discover whether a study was funded by or is biased by an industry. That's kind of a straw man at this point.

SB_UK
10-21-13, 11:20 AM
If really poor studies are published in journals by authors - then the authors (who will be listed on the article) will *forever* have their names associated with open access nonsense.
This'll do them no good.

It may or may not be easier to publish nonsense - but who wants one's name publically associated with it ?

Open access means just that - some guy can write a blog post slamming your paper - with the ability to reference it - in all its gory nonsense detail.

I think the majority doing science are honest.

One place I see for open access journals which aren't so demanding ... ... is to allow those of us with not particularly earth shattering results to get some kinda' publication out of 'em - and thereby not to fall victim to 'publish or perish'

- it's generally very hard to publish 'negative' studies (no statistically significant data found) - even though they may have been immaculately well undertaken - just without any significant hook to base publication on

- in many ways - I'm suggesting lower quality journals as a way of salvaging careers out of experimentation/results which have been perfectly executed
- but which haven't lead anywhere useful.

It's REALLY stressful when one begins a speculative project - worrying about whether the luck that is required to find something of interest - comes your way; LUCK is definitely involved.

So - imagine you've a dataset which can be used as the 34th failed association to some factor - where 23 previous studies have confirmed association
- well - we'd like the data out there - if only for future meta-analyses -

- but it's so old hat - that finding a higher quality journal to allow publication ? might ? prove difficult.

Don't know -- open access journals as a way of people (students etc) firing off publications.
As a way of people being sure that when they undertake a study - they WILL get something out of it regardless.

Sure it might be JMIR as opposed to JAMA - but that'd mean a lot to little people.

namazu
10-21-13, 12:03 PM
If really poor studies are published in journals by authors - then the authors (who will be listed on the article) will *forever* have their names associated with open access nonsense.
This'll do them no good.

It may or may not be easier to publish nonsense - but who wants one's name publically associated with it ?
You'd be surprised! There are a lot of institutions with aspirations of being "research universities", but which are going about establishing their "research cred" (or so they think/claim) in ways that prioritize easy metrics (e.g. "our researchers have on average x number of publications in scientific journals") rather than quality of work.

And there are plenty of people in want/need of employment who play this game either willingly or under duress.

I think the majority doing science are honest.
I agree.

The small minority make me uneasy, though, as they can end up fouling things up for everyone.

I recently did a review, and turned up a highly-cited paper that --upon closer scrutiny -- appeared to be largely plagiarized/fabricated. And yet the "results" obtained from it -- which many others (including, to my dismay, the authors whose work had been ripped off in the first place!) were happy to repeat as fact -- remain part of the literature.

One place I see for open access journals which aren't so demanding ... ... is to allow those of us with not particularly earth shattering results to get some kinda' publication out of 'em - and thereby not to fall victim to 'publish or perish'

- it's generally very hard to publish 'negative' studies (no statistically significant data found) - even though they may have been immaculately well undertaken - just without any significant hook to base publication on

- in many ways - I'm suggesting lower quality journals as a way of salvaging careers out of experimentation/results which have been perfectly executed
- but which haven't lead anywhere useful.

It's REALLY stressful when one begins a speculative project - worrying about whether the luck that is required to find something of interest - comes your way; LUCK is definitely involved.

So - imagine you've a dataset which can be used as the 34th failed association to some factor - where 23 previous studies have confirmed association
- well - we'd like the data out there - if only for future meta-analyses -

- but it's so old hat - that finding a higher quality journal to allow publication ? might ? prove difficult.

Don't know -- open access journals as a way of people (students etc) firing off publications.
As a way of people being sure that when they undertake a study - they WILL get something out of it regardless.

Sure it might be JMIR as opposed to JAMA - but that'd mean a lot to little people.
Absolutely -- and in fact, I think even some of the better-established journals would do well to entertain more "negative results" papers, at least when the negative results aren't simply the outcome of an "underpowered"* study.

It's a real problem when good work with "disappointing" or "not what we expected" results gets passed over in favor of "sexier"/"punchier", but not necessarily as high-quality, work.

And you're absolutely right about the potential utility for metanalyses (and even some of those under-powered studies could be useful there, when pooled appropriately), and the value to students and others engaged in the inherently-risky enterprise of research. (After all, if we already knew how everything was going to turn out, there wouldn't be any point in doing any of the research...)

But I guess I'd make the distinction that "lower-impact" journals aren't necessarily -- and shouldn't be! -- "lower-quality" journals. They may have papers with negative results, or work that doesn't make for splashy headlines, but which is well-done and still of value.

What concerns me are those journals (such as those identified by Bohannon's project) whose review processes -- for whatever reasons -- ignore methodological soundness/validity/etc. and let through studies that are really poorly done.


*For the uninitiated: "power" is a technical term in study design/statistics, related to being assured that if there were some actual difference in outcomes resulting from a drug, exposure, etc., your study is designed properly and with a big enough population that you'd be more likely to pick up on that difference instead of missing it.

SB_UK
10-22-13, 07:32 AM
Yes - you're right - lower impact should not be conflated with lower quality
- though that's the mindset in science - gotta' getta' higher Impact factor publication.

Apparently there's a Chinese group we're working alongside - which gets a bonus dependent on the Impact factor of the journal their work is accepted into.

Never heard of that before - it doesn't really seem right ... ... though will doubtless accomplish what it sets out to do.

Though as you know - I believe that it's impossible to do meaningful science and have any relationship with money/power etc
- different minds are required for science and personal wealth, respectively.

For good science - the mind need be logically consistent -
For power/money - one need deform the logical consistency of mind - in order to comply.

You need the freedom to reject your own position.

-*-

Anyway - I'm giving up on science in two days ... ... and so will watch what happens from the side-lines.
We're in for one final dash for data, in parallel with the final house price bubble which is under way - (easy money as 'help to buy' and zero % interest rates bed in) - 100,000+ genomes sequenced, millions to be studies in epidemiology etc etc - anyway one final mad dash for data

- and at the end of it all - the conclusion will be that all of our problems go away in a fair (no social division) society.

See epidemiologists Marmot and Wilkinson for more.

Human beings are not designed to operate in a social hierarchy, worse still, those who climb in a social hierarchy (and this is simply the argument used above wrt science) - are required to break their own minds in order to comply.

It's better without name, title, power, money etc etc
- because that way one encounters the freedom to think freely.

As Jung stated - I do not wish any school to be made out of my findings - because I require that all people have the same freedom to discover for themselves, just as I did.
Or as the Monty Python team wrote - 'I came here to teach you to think for yourselves'.

Nobody can think clearly - if they desire one outcome over another.

TygerSan
10-22-13, 09:08 PM
Food for thought on impact factor (a metric based on how many citations a journal gets per recently cited article)

The highest impact factor journal for general science is probably around 50. In mathematics, it's probably around 1 or less, simply because the number of articles each article cites varies greatly by field.

So impact factor is a quick and dirty way of looking at journal quality, but it doesn't tell the whole story.

Amtram
10-22-13, 09:23 PM
However, when you get to either the highest end or the lowest end, it's fairly likely that the journal did something to earn its IF!

MX2012
10-23-13, 03:45 PM
Are you talking about review for grant funding, or for publications?

I've not heard of business perspectives being included as a criterion in peer review for journal articles -- in fact, when I've reviewed articles, the journals have required me to disclose any potential conflicts of interest -- but I could imagine some funding providers being interested in potential "return on investment".

There is publication bias, both because negative results tend not to be "sexy", and because, as you noted, they can impinge on corporate interests' bottom lines. But that's usually prior to the peer review stage -- many such results aren't submitted for publication in the first place. There have been efforts/rules to combat this, such as registration of clinical trials prior to their conduct, so that they can't simply be swept under the rug.

A peer reviewer who recommends against publication simply because results may threaten the commercial viability of a product, and not because of problems in study methodology or interpretation, is acting unethically. (That's not to say it never happens, but it's not generally encouraged.)

In many cases, the "review" comes before publication. For example, go to the National Academies of Science website, then go to the National Academy Press and select a publication. Look at the committee that generated the report. Often you will find a member that represents the industry that the report covers. Also the Academy has a Business Roundtable committee. These are changes made to the Academy over the last 15 years that reflects the influence of business money and influence to affect study outcomes.

Peer reviewers "proof" the final report but the report was tailored by the committee.

namazu
10-23-13, 04:43 PM
In many cases, the "review" comes before publication. For example, go to the National Academies of Science website, then go to the National Academy Press and select a publication. Look at the committee that generated the report. Often you will find a member that represents the industry that the report covers. Also the Academy has a Business Roundtable committee. These are changes made to the Academy over the last 15 years that reflects the influence of business money and influence to affect study outcomes.

Peer reviewers "proof" the final report but the report was tailored by the committee.
Yes -- thanks for clarifying. You're right about that angle -- "consensus statements", "practice guidelines", etc. issued by panels of varying composition may reflect a variety of interests besides merely "describing the science in the most accurate way possible".

Similar concerns about industry influence were certainly raised/expressed about work by the DSM committee as well (despite rules that are supposed to reduce/air conflicts of interest).

In my mental taxonomy, I guess I kind of think of these types of documents as being a different animal from the regular peer-reviewed literature. Still, they are usually even more influential than "regular" scientific publications (because they tend to be commissioned to respond to specific policy or practice questions, and translated into policies and standards of practice and funding allocations, rather than simply incorporated into the evolving scientific literature).

It's a good point you bring up there, and I think the concerns about entrenched ideas, vested interests, etc. are quite important. Here again, I'd like to believe that esteemed experts (regardless of affiliation) are honest and ethical, but I think transparency in the process is important to allow for appropriate scrutiny/verification/questions.

Amtram
10-23-13, 05:33 PM
It's also very interesting to read the thoughts of people who have done peer review themselves - they usually take their responsibility very seriously and review the submissions as meticulously as possible, but they also know that there are people who are more into the power trip than the science.

Despite research being submitted anonymously, some scientists' work is so specialized that peer reviewers know whose it is. There are scientists who request that specific reviewers not look at their work, because there are reviewers who will reject work just knowing who conducted the research. There are reviewers who are known to farm out their review work but cover up their tracks enough so that they don't get barred from reviewing - it also lessens the quality of review.

Unfortunately, many of the problems with peer review stem from the fact that human beings are doing it. But without human beings, there is no peer review at all. There are coalitions of scientists who are lobbying for changes to peer review to make it more objective and less subject to human error and ego, but they're battling an entrenched regime. Fortunately, they're not giving up. . .

namazu
10-23-13, 06:56 PM
Despite research being submitted anonymously, some scientists' work is so specialized that peer reviewers know whose it is. There are scientists who request that specific reviewers not look at their work, because there are reviewers who will reject work just knowing who conducted the research.
The conventions of the process are also field- and sometimes even journal-dependent.

Many journals solicit names of potential reviewers from the authors of an article as part of the submission process, in large part because the authors are very likely to know (or know of) other scientists who have adequate expertise to review their paper. A journal editor may have broad knowledge in the field, but may not have enough specific expertise in a particular subfield to know who would be a qualified reviewer.

They do also ask if there are people to exclude -- sometimes this may be because of interpersonal conflicts or professional grudges. Sometimes it's for other reasons. For example, I excluded a paper from a review because I found substantial evidence of plagiarism/fabrication. Had my paper been sent to the author of the dodgy paper, I'd have gotten roundly trashed. As it was, one of the reviewers commented on the omission of the paper -- but we explained the situation to the editors, and they accepted our explanation.

The reviews I've done recently have not been anonymous on the submitter side -- that is, the authors' names were listed on the paper I received to review -- but my comments were returned anonymously to the authors.

Similarly, when I've submitted papers and gotten comments and suggestions for revision, I did not know who had reviewed my paper, but they obviously knew who I was (because they said things like "Namazu et al. misspelled 'research' on line 329 and their paper would benefit from more thorough proofreading..." :o).

In other fields, the names of the authors may be concealed from the reviewers, though as you noted, depending on the size of the field and the nature of the work, it's not always possible to have completely anonymous review.

Unfortunately, many of the problems with peer review stem from the fact that human beings are doing it. But without human beings, there is no peer review at all. There are coalitions of scientists who are lobbying for changes to peer review to make it more objective and less subject to human error and ego, but they're battling an entrenched regime. Fortunately, they're not giving up. . .
Another problem, in my view, is that peer review is something that active scientists are expected to do, but they're not often rewarded for doing it well (if at all).

I'm not saying we should bribe people to review -- that would create other issues -- but peer review, along with other forms of service, tends to "count" for less in terms of job evaluation at many institutions (especially, and somewhat paradoxically, the high-powered research institutions) than publications and sometimes teaching.

The flip-side is that there's kind of an unstated quid pro quo -- we all know that if papers don't get reviewed, they don't get published (well, excepting all those places that accepted Bohannon's paper, and others of their ilk!), or they get reviewed by people with less expertise, so there's a sense of shared responsibility to review.

It's just difficult, because to do it well takes a fair amount of time and effort, and that time and effort has to come from somewhere, and that cuts into time that people might otherwise spend doing the work they are actually employed to do, or their evenings/weekends/etc.

TygerSan
10-23-13, 07:35 PM
In my field, most journals make it so that you don't know who reviewed your paper, but they know you (though you can often specify a list of people you'd like to get the paper for review,making with any people you want to avoid).

The thought is that between the subject matter and the papers you cite (a number of which are likely to be from your group or close collaborators) it's pretty obvious who you are.

MX2012
10-24-13, 01:37 AM
Yes -- thanks for clarifying. You're right about that angle -- "consensus statements", "practice guidelines", etc. issued by panels of varying composition may reflect a variety of interests besides merely "describing the science in the most accurate way possible".

Similar concerns about industry influence were certainly raised/expressed about work by the DSM committee as well (despite rules that are supposed to reduce/air conflicts of interest).

In my mental taxonomy, I guess I kind of think of these types of documents as being a different animal from the regular peer-reviewed literature. Still, they are usually even more influential than "regular" scientific publications (because they tend to be commissioned to respond to specific policy or practice questions, and translated into policies and standards of practice and funding allocations, rather than simply incorporated into the evolving scientific literature).

It's a good point you bring up there, and I think the concerns about entrenched ideas, vested interests, etc. are quite important. Here again, I'd like to believe that esteemed experts (regardless of affiliation) are honest and ethical, but I think transparency in the process is important to allow for appropriate scrutiny/verification/questions.

I do appreciate your clarifications. Much appreciated.

qinkin
10-24-13, 09:57 PM
Many journals keep the articles behind a paywall by default, but then after a certain period of time, often 1 year, the articles revert to “open access”. New U.S. regulations requiring researchers who get federal grant funding to make their results available to the public have resulted in many journals adopting this “'embargoed' for a period of time but then opened up” policy.


O wow, you said that originally? oops. Delayed in getting back with you guys because I never got any notifications of replies lol.

------

SB, I have difficulty seeing where you end and Peter Joseph Merola (age 30s) /Jacque Fresco (age 90s) begins, at times. I mean come on, pretty crappy documentaries/books/talks (currently ongoing even, pretty horrible stuff) by scientific standards. Having had much experience in this, and furthermore, genuinely in favor of their ideas at one point, these guys have unfortunately lost their credibility. I can say this with confidence. If you don't read this, o well, others will and what you are saying will make more sense to them--- the only reason y I bother replying.

---

Been thinking about the reason why the type of journals they studied was very specific and limited.. They submitted, if I'm not mistaken, to only pay to be published, not pay to submit, journals. This would significantly reduce the costs of the endeavor, for what it's worth. I found some of the dirty details through a linked-to blog that was posted here, please see link if interested. Still somewhat of a mystery why Science chose to undertake this avenue of publication. I'm not sure what the precedent and such for this mode of publication is. But not sure what other questions haven't been answered to already..

One could say it gives the wrong impression, but upon reading the article, it clearly doesn't, to me. The report makes sense to me for the most part.

http://www.sciencemag.org/content/342/6154/60/suppl/DC1