guest post by Christopher Lee
A long time ago in a galaxy far, far away, scientists (and mathematicians) simply wrote letters to each other to discuss their findings.
In cultured cities, they formed clubs for the same purpose; at club meetings, particularly juicy letters might be read out in their entirety. Everything was informal (bureaucracy to-science ratio around zero), individual (each person spoke only for themselves, and made up their own mind), and direct (when Pierre wrote to Johan, or Nikolai to Karl, no one yelled “Stop! It has not yet been blessed by a Journal!”).
To use my nomenclature, it was a selected-papers network. And it worked brilliantly for hundreds of years, despite wars, plagues and severe network latency (ping times of 109 msec).
Even work we consider “modern” was conducted this way, almost to the twentieth century: for example, Darwin’s work on evolution by natural selection was “published” in 1858, by his friends arranging a reading of it at a meeting of the Linnean Society. From this point of view, it’s the current journal system that’s a historical anomaly, and a very recent one at that.
I’ll spare you an essay on the problems of the current system. Instead I want to focus on the practical question of how to change the system. The nub of the question is a conundrum: how is it, that just as the Internet is reducing publication and distribution costs to zero, Elsevier, the Nature group and other companies have been aggressively raising subscription prices (for us to read our own articles!), in many cases to extortionate levels?
That publishing companies would seek to outlaw Open Access rules via cynical legislation like the “Research Works” Act goes without saying; that they could blithely expect the market to buy a total divorce of price vs. value reveals a special kind of economic illogic.
That illogic has a name: the Walled Garden—and it is the immovable object we are up against. Any effort we make must be informed by careful study of what makes its iniquities so robust.
I’ll start by reviewing some obvious but important points.
A walled garden is an empty container that people are encouraged to fill with their precious content—at which point it stops being “theirs”, and becomes the effective property of whoever controls the container. The key word is control. When Pierre wrote a letter to Johan, the idea that they must pay some ignoramus $40 for the privilege would have been laughable, because there was no practical way for a third party to control that process. But when you put the same text in a journal, it gains control: it can block Pierre’s letter for any reason (or no reason); and it can lock out Johan (or any other reader) unless he pays whatever price it demands.
Some people might say this is just the “free market” at work—but that is a gross misunderstanding of the walled garden concept. Unless you can point to exactly how the “walls” lock people in, you don’t really understand it. For an author, a free market would be multiple journals competing to consider his paper (just as multiple papers compete for acceptance by a journal). This would be perfectly practical (they could all share the same set of 2-3 referee reports), but that’s not how journals decided to do it. For a reader or librarian, a free market would be multiple journals competing to deliver the same content (same articles): you choose the distributor that provides the best price and service.
Journals simply agree not to compete, by inserting a universal “non-compete clause” in their contract; not only are authors forced to give exclusive rights to one journal, they are not even permitted to seek multiple bids (let more than one journal at a time see the paper). The whole purpose of the walled garden is to eliminate the free market.
Do you want to reform some of the problems of the current system? Then you had better come to grips with the following walled garden principles:
• Walled gardens make individual choice irrelevant, by transferring control to the owner, and tying your one remaining option (to leave the container) to being locked out of your professional ecosystem.
• All the competition are walled gardens.
• Walled garden competition is winner-take-all.
• Even if the “good guys” win and become the biggest walled garden, they become “bad guys”: masters of the walled garden, whose interests become diametrically opposed to those of the people stuck in their walled garden.
To make these ideas concrete, let’s see how they apply to any
reform effort such as selectedpapers.net.
Walled gardens make individual choice irrelevant
Say somebody starts a website dedicated to such a reform effort, and you decide to contribute a review of an interesting paper. But such a brand-new site by definition has zero fraction of the relevant audience.
Question: what’s the point of writing a review, if it affects nothing and no one will read it? There is no point. Note that if you still choose to make that effort, this will achieve nothing. Individuals choosing to exile themselves from their professional ecosystem have no effect on the Walled Garden. Only a move of the whole ecosystem (a majority) would affect it.
Note this is dramatically different from a free market: even if I, a tiny flea, buy shares of the biggest, most traded company (AAPL, say), on the world’s biggest stock exchange, I immediately see AAPL’s price rise (a tiny bit) in response; when I sell, the price immediately falls in response. A free market is exquisitely sensitive to an individual’s decisions.
This is not an academic question. Many, many people have already tried to start websites with similar “reform” goals as selectedpapers.net. Unfortunately, none of them are gaining traction, for the same reasons that Diaspora has zero chance to beat Facebook.
(If you want to look at one of the early leaders, “open source”, and backed by none other than the Nature Publishing Group, check out Connotea.org. Or on the flip side, consider the fate of Mendeley.)
For years after writing the Selected-Papers Network paper, I held off from doing anything, because at that time I could not see any path for solving this practical problem.
All the competition are walled gardens
In the physical world, walls do not build themselves, and they have a distressing (or pleasing!) tendency to fall down. In the digital world, by contrast, walls are not the exception but the rule.
A walled garden is simply any container whose data do not automatically interoperate with and in the outside world. Since it takes very special design to achieve any interoperability at all, nearly all websites are walled gardens by default.
More to the point, if websites A and B are competing with each other, is website A going to give B its crown jewels (its users and data)? No, it’s going to build the walls higher. Note that even if a website is open source (anyone can grab its code and start their own site), it’s still a walled garden because its users and their precious data are only stored in its site, and cannot get out.
The significance of this for us is that essentially every “reform” solution being pushed at us, from Mendeley on out to idealistic open source sites, is unfortunately in practice a walled garden. And that means users won’t own their own content (in the crucial sense of control); the walled garden will.
Walled garden competition is winner-take-all
All this is made worse by the fact that walled garden competition has a strong tendency towards monopoly. It rewards consolidation and punishes small market players. In social networks, size matters. When a little walled garden tries to compete with a big walled garden, all advantages powerfully aid the big incumbent, even if the little one offers great new features. The whole mechanism of individuals “voting with their feet” can’t operate when the only choice available to them is to jump off a cliff: that is, leave the ecosystem where everyone else is.
Even if you win the walled garden war, the community will lose
Walled gardens intrinsically create a divergence of interests between their owners vs. their users. By giving the owner control and locking in the users, it gives the owner a powerful incentive to expand and exploit his control, at the expense of users, with very little recourse for them. For example, I think my own motivations for starting selectedpapers.net are reasonably pure, but if—for the purpose of argument—it were to grow to dominate mathematics, I still don’t think you should let me (or anyone else) own it as a walled garden.
First of all, you probably won’t agree with many of my decisions; second, if Elsevier offers me $100 million, how can you know I won’t just sell you out? That’s what the founders of Mendeley just did. Note this argument applies not just to individuals, but even to the duly elected representatives of your own professional societies. For example, in biology some professional societies have been among the most reactionary in fighting Open Access—because they make most of their money from “their” journals. Because they own a walled garden, their interests align with Elsevier, not with their own members.
Actually that’s the whole story of how we got in this mess in the first place. The journal system was started by good people with good intentions, as the “Proceedings” of their club meetings. But because it introduced a mechanism of control, it became a walled garden, with inevitable consequences. If we devote our efforts to a solution that in practice becomes a walled garden, the consequences will again be inevitable.
Why am I dwelling on all these negatives? Let’s not kid ourselves: this is a hard problem, and we are by no means the first to try to crack it. Most of the doors in this prison have already been tried by smart, hard-working people, and they did not lead out. Obviously I don’t believe there’s no way out, or I wouldn’t have started selectedpapers.net. But I do believe we all need to absorb these lessons, if we’re to have any chance of real success.
Roll these principles over in your mind; wargame the possible pathways for reform and note where they collide with one of these principles. Can you find a reliable way out?
In my next post I’ll offer my own analysis of where I think the weak link is. But I am very curious to hear what you come up with.
Azimuth 
My proposal for a way out is to bring the literature back into the hand of the researchers and their institutions: international cooperation between a crystallization point of key institutions makes everything that is openly accessible today (i.e., virtually everything published 12 months ago or older, plus OA articles, pre-prints, post-prints, etc.) accessible via a single entry-point (federated, de-centralized database). Everything not covered by this database would be covered by a ‘request e-print’ button: this formidable access alone would attract virtually every scientist, as it would provide better services than what we currently have. With such a database, these key institutions could cut subscriptions en masse, freeing millions to develop our own scholarly communication system.
I’ve been pitching this idea at several international library meetings and received very positive feedback.
[…] 2: Christopher Lee takes the view somewhat opposite to the one from this post, […]
If you publish your paper on your own blog, several search engines will pick it up. Neither the Internet, your blog, nor search engines are walled gardens.
And indeed, many papers are freely available on the world wide web today. Why do scientists still want to pay both to have their papers published and to read other papers? Perhaps there is something desirable about the walled garden after all?
I think that while people complain about the extortionate prices, they really do want to be able to pay good money to put their paper above that of the riff-raff, or to read only “the best” papers.
Finn wrote:
In Part 1, I gave two reasons for this. One involves the key function of journals: endorsement, saying a paper is ‘good’, thus giving academics authors the verifiable units of prestige that they need to get hired, tenured and promoted. I wrote:
I regret you are taking the path of “endorsement” and “prestige”, these are exactly the lures which keep us from leaving the broken system.
Or, if you want to keep the endorsement system, why not make it more robust by decentralising it? I had a proposal (well, I thought about it in relation to a specific problem), which uses three independent parts:
A 3-parts system for scientific credibility.
I think the real path is towards a job and funding; prestige is just a means toward this end. You need a way of convincing employers of your worth, which will necessarily be some widely understandable form of prestige. For niche areas this has to be some sort of reliable substitute for a genuine understanding of quality, since employers can’t actually scrutinize your work (for several reasons). There’s no getting out of the need for prestige of some form, but we could ideally alter the form that it takes.
Thanks for the link! I agree with you wholeheartedly that a transparent system of evaluation of papers is needed, and also that it has to be engineered into the very framework that people use to read papers. That is, that the mass action of individuals reading, discussing and recommending papers (by explicitly stated criteria) should directly yield the validity and impact metrics that institutions later consider when making funding and personnel decisions. In other words, the metrics should reflect a paper’s real impact on the community, after they’ve had a chance to see and test its claims. Note this “post-evaluation” is completely different from what we have now, which I would characterize as “pre-evaluation”, i.e. 2 or 3 referees’ guess about what the impact of the paper should be, and locked in before anyone else is allowed to see the paper.
Glad you like it, the “post-evaluation” is one of the reasons I support your #spnetwork.
For the readers, here is the 3-parts system: 1) the researcher, 2) the “endorser” and 3) anybody else, like an institution where the researcher applies for a job. The “endorser” can be any entity which makes public a list of criteria which are needed to endorse a researcher (there is no one-fits-all solution, there could be many endorsers) The researcher who thinks (!) she satisfy the criteria from the list of the endorser makes public this, by making a link to the endorser, along with a page with proof for this. Finally, the part 3) can look at the pages 1) and 2) and decide, independently, based on available proof. All parts live by their choices (some endorsers may have criteria considered stupid by others, some researchers might make false claims, the hiring institution, say, 3), may turn out to not take the right decisions based on the information available). In time, some endorsers gain credibility, as well as some researchers, moreover there is no police involved and there is no unique solution for all (maybe some institutions are looking for good educators, others for eccentric sociopaths).
Chorasimilarity wrote:
Right now academics need endorsement—that is, some prestige-bearing entity that says their work is “good”—to get jobs and promotions. Telling them to forsake this is telling them to commit career suicide. They won’t do it. The need for endorsement would only cease if the university system as we know it were overthrown. Perhaps you want to overthrow it. But that’s not my goal. I’m simply trying to make publication cheaper and better.
The selected papers network allows for a highly decentralized system of endorsement. Prestige-granting entities of all kinds can say papers are “good” using the selected papers network.
“Perhaps you want to overthrow it.” No, I try to help, so that my two kids still have a university system to attend. Looking at how things are going presently, perspectives are not bright. A big proportion of young, especially gifted people, just don’t see what’s the relevance of this system which is ruled by ISI counts and more and more conformity. These people, who could be the professors of my kids, choose paths outside academia.
“Telling them to forsake this is telling them to commit career suicide. They won’t do it.” Agree, but the main role of the publication system is not to be an endorsement tool for the HR departments. Instead, it is about disseminating knowledge and communicating research such that it facilitates the use of the scientific method. Having a system which is better at this (not perfect, nothing can ever be perfect, only a bit better) has direct economical implications, at least. I’m afraid that insisting on prestige, which is a matter which interest nobody else outside academia, leads us to being increasingly irrelevant. It does not look to me that the “publish or perish”, which is a manifestation of the disease of writing for getting prestige points, when counted per journal reputation points, has improved the position of the academia in the society. Rather the opposite seems to be true.
In conclusion, what I believe is not telling them to commit carreer suicide by avoiding the legacy publishing, but instead I believe that for the good of all of us, especially on the long term, it would be very helpful that influent people in the academic circles, which are not as many, could take a stronger stance in favor of going back to the joy of research and communication, instead of hunting reputation points. Let the publishers worry about too fast changes, let them damp the expectations. If researchers see that their influent, respected peers become less timid than the anonymous average one, this could make a difference. If these respected people, with accomplished carreers, take views which are at least comparable with the ones of young bright people, who try to change the system and then leave academia, this would make a difference.
Re: “…a free market would be multiple journals competing to consider his paper…”
I wanted to note that this is how academic legal papers are published. Authors submit to a clearinghouse, and journals bid on the right to publish. There is even the possibility of a bidding war!
Bobbie Spellman (Editor in Chief of Perspectives on Psychological Science) describes this, and wishes there were some way to improve the review and publishing process:
But of course this type of idea might just generate a bigger walled garden around the smaller ones.
Note that it is not a free market because of a very specific government intervention: copyright.
Do away with copyright law and you get back to a free market.
Further reading:
Do we need copyright?
http://lemire.me/blog/archives/2012/03/22/do-we-need-copyright/
@Björn Brembs, I’m trying to understand concretely how you think this would make a big change vs. what we have now, after a decade+ of Open Access efforts. Setting up some new paper repository site seems actually less powerful (because initially no one is going to be familiar with or visiting that site) than just putting the pdf links on the major search engines that everyone uses (since that’s how people find papers nowadays). The major search engine in my field (Pubmed) very much tries to provide those links whenever there is an Open Access version of the paper available. That is helpful, but it’s not in any sense revolutionizing publishing in my field. I would say there’s no discernible effect. RE: your “request eprint” button, I suspect that would just run into the apathy that many academics display vis a vis Open Access. I.e. once their paper is published in a journal, they often don’t bother to make a free copy available even if that is their right (or a preprint, which is almost always legal). While more and better access to published papers is obviously a good thing, I don’t see it as striking at the heart of what enforces the current system.
@Finn, “just saying no” to journals and consigning your papers to your blog is what I referred to as the “exile” option. Being locked out of your professional ecosystem (even by your own choice) is not going to make that ecosystem better. Scientific publication is not just a matter of putting a PDF up on the web. When people say “publish or perish” they mean it has to be officially validated and accepted, and right now that means just one thing: published in a prestigious journal. An individual can rightly criticize the many flaws of the current review system, but simply saying “everybody should stop following the system” ignores a crucial fact. Everybody is following the system because they are locked into it by constraints imposed by other people’s expectations. Wishful thinking will not defeat this system. To free people we have to break those constraints.
It’s only exile if no one else joins you.
I agree with chora: Trying to change this system, but conceding that there is such a thing as prestige and that hiring committees cannot ascertain it, is like saying you’d like to eliminate global warming, but conceding that we can’t do anything about people’s need to use fossil fuels at increasing levels. Prestige and its pursuit is exactly the leverage that the publishers use to maintain this crooked system.
There were academic jobs back in Gauss’ day – even before that. How did people get them? How did they keep them? How did people acquire a reputation for good work back then? And how might we do that today? I think the answers are roughly the same now as they were then.
Anyway, there’s a lot of dissatisfaction out there. The iron should be struck while it is still hot. We have big names willing to go off to ‘exile’. We may never have a better chance to leave the world of walled gardens.
Sorry, you misunderstood me: I was simply describing the system as it exists today, not advocating that it should be retained in its current form. I was arguing that the current reality is that most people are constrained to the current system by other people (hiring committees, grant committees) adhering to the “prestige” measures of the current system. And I was arguing that we must therefore break those measures (“pre-evaluation”) and replace them with something better (“post-evaluation”). So you and I are in agreement about goals. And I fully agree about striking while the iron is hot.
So let me ask you: are you willing to submit your next paper to a new peer review process and journal (backed by the big names you mentioned) that follow the “openness” principles you have in mind? By virtue of being new, it of course won’t (yet) have as much reputation as the established journals. Are you willing to make that small sacrifice in initial “prestige”, to leave the walled garden and help build the open system?
Thinking about what you said, I think for me the answer is YES.
John Iskra wrote:
There were very few academic jobs in Gauss’ day compared to now, because university education was only for an elite few. Jobs were handed out using the “good old boys” system: the people doing the hiring would typically know the job candidates personally. Things were more localized: often the job candidates would have studied at the universities they later went on to work in, and almost always they would come from the same country.
None of these conditions are true in academia today, especially not in the US, which gets many of its best students and professors from other countries. In the US, the big shift happened after World War II, when higher education was extended to more people and federal funding for science grew immensely.
A typical math job at my university will get several hundred applicants from countries all around the world, and there are laws requiring us to give all applicants fair consideration. Hires have to be okayed by several layers of bureaucracy beyond the department.
Tenure and promotions also go through several layers of bureaucracy, including ‘ad hoc committees’ composed of faculty from various departments. So, it’s not enough for us to know that a mathematician is good: we need to convince people from outside our department that this is true. Saying someone has classified all Calabi-Yau manifolds would not be enough, even though to a mathematician this would be the equivalent of walking on water.
In general, if people from any given academic department can’t convince outsiders who know nothing about that subject that they’re doing great things, they’ll eventually decide that department is weak and give it fewer positions and less money. This is where bean-counting becomes extremely important, and dangerous.
Prestige was just as important in Gauss’ day, and the system was just as flawed as it is now. It just worked quite differently.
Re: John Iskra, I agree. The “exile” is happening already, there are lots of talents who left or never considered to enter academia, sadly. The reponsibility is on the side of academic managers, hiding from responsibility behind numbers. There is no rule governing humans, other than belief, i.e. if more people believe it’s broken, then at some point there will be a change. Universities were places for training priests, then they changed, under the pressure coming from the spreading of the press, into places where research is welcomed. Now, they are treated as busineses, so they lose their cool factor. Next generation? Eff it! In the bean counting parallel reality things look great.
I should add that while there are lots of problems with academia, I can’t imagine any other job where I could have the same freedom to do cool stuff. I have never worried much about bean-counting aspects, and I’ve done just fine. So for me, at least, it’s an institution that I want to improve rather than toss out.
Luckily we don’t need to settle questions about the fate of academia to agree on some ways to improve communication, like the spnetwork. If one tries to settle too many big questions at once, different people come up with different answers, the community of concerned people splinters, and the number of people working on any one given solution diminishes.
The ambiguity of words like “prestige” and “cool factor” is tripping us up here… So let me clarify what I was saying:
* in my view nothing should get in the way of each individual deciding for himself what interests him, and whose work he wants to hear about. In particular, each person should speak for themselves alone and for no one else; there should be no central point of control that decides what the community should read. So if “cool” means what personally interests you, then I’m all for it; if it means what makes a Nature editor dream of press releases, then I’m against it (as a principle for how to run the distribution network).
The job of the research distribution system should be that of a search engine (i.e. enabling each individual to find what they individually want, and thus to hook up with other individuals with the same interests) not that of a TV network (to find a lowest common denominator “script” for the biggest possible audience).
* The selectedpapers.net approach definitely intends a crucial role for people who are willing to exit the current “bean counting” system. Specifically our “public data federation” open model seeks to provide them the maximum leverage possible, by ensuring that whatever they write can be visible across all the social networks, instead of being trapped within a single site as in the walled garden model.
One more point: once you have a distribution system based on each individual’s decisions about what interests them (as opposed to a central point of control), then you should provide measures of “prestige” / “cool factor” that simply reflect all those individual decisions and assessments. In other words, the *actual impact* of a paper, measured across the whole community, instead of the *predicted impact* assigned by fiat of the central point of control.
@Niles Johnson, I suggested a similar “one-stop review” framework in the original selected papers network paper. See section 5.2.4 (and previous section): http://www.frontiersin.org/Computational_Neuroscience/10.3389/fncom.2012.00001/full#h6
Right; for me it was useful to learn that there is already a such a system working in some publishing domain. It demonstrates that it is at least *possible* for those aspects of the selected papers network to be successful. Moreover, I was intrigued to learn that one-stop review was of interest to at least one other person with non-trivial influence (namely Spellman) but who appears to be unaware of your ideas.
@Finn the more I reread your comment “(scientists) want to be able to pay good money to put their paper above that of the riff-raff”, the more puzzled I get. Scientists do not pay Elsevier $$$ to get their articles *published* (as you implied). It’s the other way round: Elsevier makes its money by getting university *librarians* to pay $1 million each for subscriptions to *read* its journals. How can paying $$$ to read *other* people’s papers “put *your* paper above that of the riff-raff”?
The money which is “extorted” from librarians or scientists ultimately comes from tax-payers. It does not matter how it is channeled before it reaches Elsevier. The important thing is that Elsevier gets lots of money which it can use to create and maintain *brands*. Elsevier are not operating under “a special kind of economic illogic”, they are operating like much of the rest of the economy which sells beer and shoes and cars and so forth by brand. You can’t have free or non-exclusive brands. The best brands must be expensive and exclusive. And scientists (like everyone else) appear to want brands.
The “brand” here should be the scientist, not the journal. This is not at all like the production of a good, because journals only function as middlemen and have no intrinsic value. Publishing a paper is like producing a food product and trying to get it stocked by a popular and well-known grocery store. The store then makes you sign a contract that says you can’t sell your product anywhere else, and then drastically marks up the price thanks to their artificial monopoly. There is no sense in which that is a free market. Free markets fundamentally depend on the existence of *substitutes* for any given good, and for competition between both buyers and sellers, both of which this setup is meant to undermine. The invisible hand can’t work if there are no meaningful choices.
RE: “It does not matter how (the money) is channelled before it reaches Elsevier. The important thing is that Elsevier gets lots of money…” You’ve turned the facts of the research economy upside down. What you really mean is “It does not matter how (the money) is channelled before it reaches researchers. The important thing is that researchers get lots of money and information so they can make discoveries that help people and the economy.” It’s the researchers who produce research; Elsevier is just a middleman that interposed itself between researchers who want to communicate with each other.
RE: “Elsevier… are operating like … the economy which sells beer and shoes…” You’re trying to imply that Elsevier operates like the free market we all enjoy when we go to the supermarket and buy some beer: you can freely choose which beer(s) you want to buy and how much; you can choose what retailer you want to buy it from; any retailer can sell that beer, for any price they want; and you can buy that beer free of any other obligations. But it is precisely these free market characteristics that Elsevier has eliminated by refusing to let universities decide what journals they want. It’s as if they said “you can only buy your beer if you agree to a year’s subscription to it and our 5000 other products too…”
cjlee: “To use my nomenclature, it was a selected-papers network. And it worked brilliantly…”
Citation needed.
cjlee: “Even work we consider “modern” was conducted this way, almost to the twentieth century: for example, Darwin’s work on evolution by natural selection was “published” in 1858, by his friends arranging a reading of it at a meeting of the Linnean Society.”
Darwin sat on his work for many years. He was a gentleman scientist with no pressure on him to publish, until he received a letter from Alfred Russel Wallace containing very similar ideas to his own. When Darwin’s friends arranged the reading you mention, they also presented Wallace’s work, without his knowledge or permission. I don’t want to go back to that kind of system.
cjlee: “From this point of view, it’s the current journal system that’s a historical anomaly, and a very recent one at that.”
The public funding of science is also a historical anomaly, as is the participation of women in science. I want more such anomalies.
citation: http://www.frontiersin.org/Computational_Neuroscience/10.3389/fncom.2012.00001/abstract
The high price of journals, and the poor quality of existing peer review, is not facilitating either public funding of science or women in science (any more than the high cost of textbooks helps college students learn…). Bringing in such red herrings is not helping your case.
(e-)Published theory abundantly demonstrated “superluminal” neutrinos. It was a loose fiberoptic connector. The Walled Garden is also a closed shop wherein trusted employees’ marketable asset is loyalty (by threat of being shunned), excluding unfair competition. It nourishes a managerial class outputting quantified abstractions orthogonal to content.
Some 45 years of Standard Model and quantum gravitation fill phone book-thick journals with rigorously derived chaff. “…it is not right that matters, but victory.” Functional publication and its facilitated access value product over process. Nekulturny!
Very nice ideas.
I do not know a simple solution, of course, since no one does. I also like the idea of former times, writing letters to each other, but this probably is/was coupled to a much smaller number of scientists knowing each other “personally”, and it probably also only worked well for established fields and important problems in it. If you were working on groundbreaking stuff, you probably were very on your own and may even have great problems to make a point in the community. This was to some degree remedied by an independent publishing system, which allowed you to write your thoughts down and publish them on your own expense. So maybe, not everything was good in the old system either.
A second remark…even if one does not solve the problem of scientific publishing by inventing “something new”…one should may tackle the problem, of how scientific publishing is connected to making a career in science. That simply means…STOP THE COUNTING OF ARTICLES…in CVs and elsewhere and start to assess the quality of someones work, independent from the fact, where it was publishes (impact factor crap) or how it was published (if published in the common way at all). The power of the publishers is more or less completely to our will of giving them the power to judge our careers by them taking control. Stop this, and you possibly are in a totally different situation regarding power and control within the system.
In theory I agree with you. In practice things work a little differently. You have x professors (say 60) and a pool of money for a pay raise (this happens sometimes). You have one week to decide how to distribute the money based on their accomplishments in the last few years, say 3. What do you do? Read a hundred plus papers in the fields you know nothing about?
I fully agree that people need metrics for evaluating validity and impact. But bear in mind that the current metric for the “value” of a paper (“Name of the journal it was published in”, AKA “impact factor”) is ridiculous in multiple ways:
* the metric for evaluating a paper should be measured for the *paper*, not the journal (obvious: the variation in citation counts for papers within one journal is as wide as the differences in impact factor between journals);
* the current metric is locked in at the moment the journal accepts the paper, i.e. before everybody has had a chance to even read it, let alone evaluate its claims. I refer to this as the “pre-evaluation” principle. Unfortunately many studies have shown that pre-evaluation works badly. For example, pharmaceutical companies try rigorously to reproduce academic studies relevant to disease or drug discovery. Unfortunately, in 80% of cases they find that the reported experimental results (using the exact same reagents and protocols) are not reproducible. And those are for papers published in the *most* selective journals (Nature, Science etc.) The bottom line is that pre-evaluation (before anyone has had a chance to test the claims) is the worst time to lock in the paper’s “value” metric. Far better would be to let the community read the paper first, discuss its claims, and test them. I call that “post-evaluation”. We need to move from pre-evaluation metrics to post-evaluation metrics — and that is perfectly feasible in the Internet era. Science is complicated, and in many fields it’s just very hard to assess which claims will turn out to pass the test of time (without anybody actually trying to reproduce them), let alone to predict which will turn out to be important (high impact) in the long run. The field is littered with examples of both false positives and false negatives caused by pre-evaluation’s errors.
I could go on with lots more critiques of the current metric, but you get the idea. What we need is rigorous research on scientometrics.
Note I wasn’t saying “everything was good in the old system”, but rather that it had one key advantage we need to learn from: because it had no point a third-party could control, it was not a walled garden. That is highly relevant, because the root of all our problems is the walled garden.
In my last post, I outlined four aspects of walled gardens that make them very resistant to escape…