Help me revolutionize academic publishing!

August 20, 2009 5:07 PM

I have a friend who is launching a new peer-reviewed, all online, no-subscription academic journal. We have been kicking around ideas to make it awesome and I thought what better place to get great ideas...

For example, one idea we had was a Netflix-like rating system. You could rate the content of papers, and we could recommend papers to you based on similar users. Users interested in ferromagnetic thin film tunnel junctions also liked...

Author karma? Tag clouds? Online discussion (and rebuttal!)? Think outside the dead-tree journal box! What features make online journals great? What novel features would make online academic journals great?

[And again, let me emphasize that this is for peer-reviewed academic journals, e.g., http://apl.aip.org/ not... LiveJournal!]

posted by juliewhite to science & nature (13 comments total)

IANAA, but faceted search based on multiple aspects of studies would be nice. Methodology/reporting method, subjects, locations, populations (if social science... probably not from your ferromagnetic example), dates, n-value/p-value for statistical studies, etc. And of course full-text keywords as well. Pull the metadata out and let people drill down. Very possible with the proper metadata collection and something like Solr. Maybe journals already do this?

Once you've got specific searches down, give people customized notifications when new articles match those searches.

Tag clouds by themselves are very "eh" at best, and seem even less useful if each visitor is looking within a specialty, as they really just show off what's popular and look sort of decorative.

Maybe a degrees-of-separation widget based on co-authorship? Might work better if you had data about papers published elsewhere, though.

Depending on the type of data being presented, it would be très cool — though maybe authors would revolt — if raw data tables could be uploaded and there was some sort of visualization app that allowed readers to chart it themselves, using a variety of appropriate techniques. It could be hella complex to program, but a dedicated ActionScript programmer with dogeared copies of Tufte could have a great time.

Video of experiments, with annotations and references/links to the relevant sections of the paper.
posted by pengale at 5:30 PM on August 20 [2 favorites]

This has the potential to be really cool, but because it'd be online (and not under the wing of a university or foundation), you'd have to think up some creative ways to protect intellectual property, especially since you wouldn't even have a subscription. Could you present papers only as PDFs (a la Google Books) with some sort of watermark/intense metadata attached?

Another question is the problem of prestige. Real academics want to be able to put well-known publications (in-field, at least) on their CVs, so you might have a problem attracting people who are really good at what they do. Maybe (unless there are some legality issues) you could present this as a place to hold open peer-reviewing before submission to a mainstream journal?
posted by oinopaponton at 5:33 PM on August 20

I was in academic publishing (currently on hiatus of indeterminite length) and have thought about this a good deal. One thing I noticed there are quite a few parties attempting to integrate some of the same features you mention. So, it might be good first look at what others do, and gauge how well it has (or has not) worked for them in terms of user participation, etc. Then expand/get creative from there. For example, I would look at the widgets the PLOS journals have played with and feature on their site, since they've been pretty much a step-forward in employing some of the Web 2.0-ish things you mention.
posted by NikitaNikita at 5:47 PM on August 20

What features make online journals great?

Open access, quick review times & a transparent process, good editors, good reviewer pool, clean well organized simple web site, support for latex submissions (esp. at the proof stage), and last but not least, a prestigious editorial board and/or organization behind it so the journal is ranked well and you get initial buy-in from prominent researchers in the field. The last of these is probably the most important if you want the journal to go anywhere. That's really about it. Sorry to be negative, but honestly, if you don't have these features, no amount of novel technical features of the web interface to the journal is going to make it great.

I can give you a model in my field that seems to be going well, which basically has all of these properties: semantics and pragmatics (and none of the ones you seem interested in). They have a lot of blogging/writing about the lead up to starting the journal. There are many online-only open access journals in other fields being started these days, too.

I guess I can give you some comments on your particular ideas:

Netflix-like rating system
Author karma?

Unless you already have tenure (for that matter, unless you are already full), you _seriously_ do not want to get involved with any metric that quantifies the prominence/quality of researchers in your field. For similar reasons I doubt you will get people to use rating systems if their ratings are visible in any way. The politics are just too problematic.

Tag clouds?

This does seem cool but I'm not sure with a high quality single journal's data you'll have much for years...would work better with a preprint archive or a research aggregator. Actually I think this will be a problem with the above features too.

Online discussion (and rebuttal!)?

This may be workable and interesting. However, I think you would find that you don't necessarily want to be "publishing" research content that you don't have editorial oversight for. The normal submission/reply model involves peer review at each stage, and this is good IMO. I also would personally not be interested in having this kind of feature for my articles; if someone finds a problem, I would rather they do things like email me, try to publish a reply themselves, develop the problem into new research, etc.
posted by advil at 5:48 PM on August 20

Oh, also, kind of basic, but RSS feeds.
posted by advil at 5:49 PM on August 20

Forward references! It's easy to see what sources an article cites by looking at footnotes, but online tools have the capacity to also show what papers cite the paper that you are currently looking at. It would be great if your database could update older papers as they get cited by newer ones, so that you can look forward from one article to later articles that cite the former.
posted by philosophygeek at 5:50 PM on August 20

I like the idea of forward references. I've actually toyed with starting a list of "papers that cited my papers" on my web page, except so far there isn't much so that would just look sad.

One journal that I think does the online journal thing well is the Electronic Journal of Combinatorics (combinatorics.org). (Disclaimer: I've published a paper here.)

And I agree that a single journal won't have enough content for any of the Web 2.0-type tools you suggest. There are occasionally murmurings among mathematicians and physicists to attach that sort of thing to arxiv.org (the preprint server used by many of us) but nothing that I know of has taken off.

I suspect that most of the things that make online journals good are the same things that make print journals good, though. In the end nobody will actually look at the journal unless it contains good papers.
posted by madcaptenor at 6:52 PM on August 20

Philosophers' Imprint is an online-only open-access journal for philosophy, which publishes articles "irregularly" (whenever a good article is submitted). It tries to maintain the credibility of print journals without the restrictions or inconvenience. It doesn't have the fancy features you're suggesting, but I bet you could ask them what ideas they considered when creating it.
posted by k. at 6:57 PM on August 20

Scholarpedia does something like this in wiki format, although most of the articles are essentially reviews of a particular area. "Famous" researchers are suggested for certain categories, and the community votes for the person to write about a certain subject. This neatly sidesteps the main problem you'd face with a de novo journal, namely: credibility.

From what I understand of the genesis of new journals, there has to be a substantial community who are interested in participating in it before it publishes its first issue. This community tends to be focused on a particular area. The community lends the journal credibility by submitting, reviewing, and actually reading papers submitted to the journal.

As an extremely junior researcher, I'm interested in getting my work out in front of a large audience of people who are doing the same thing as me, so that I may be judged by my peers and awarded all due accolades. This helps my career. There is a vanishingly small chance that I would submit publishable research to an online journal that is not guaranteed to get my research in front of the "right" people.

This isn't to say your idea sucks. There's a lot wrong with the journal system as it works now - Elsevier being a good example. I think you need to narrow your goals in the short run, focus on building an online journal for a subject area in which you know that people would be willing to submit articles. That way you can build up your "name" before expanding to other (all?) academic areas. All the Web 10.0 bells and whistles won't make an academic read something without a reputation.
posted by logicpunk at 7:30 PM on August 20

Run, don't walk over to the Science 2.0 Friendfeed room. A lot of the major players in open science hang out there (PLoS people, open-notebook science, creators of online sharing tools like Mendeley and Zotero, advocates for free science). Skim the archives there and subscribe to the ongoing conversation.

I've reposted this question there, and will collate some of the responses from that room into the thread. Feel free to create your own account and ask more detailed questions.

Okay, as for my suggestions, I don't think most rating systems are very good in general. The basic problem is that science is so specialized that general up-and-down ratings are useless. You have the right idea when you talk about netflix (users who liked this, also liked this).

The big problem, as I see it, is building a critical mass of users who will actually use the ratings system. I mean, if there are only two dozen people in the world studying the same things I am, you've got to have a pretty big pool of people before it's likely that they'll be there offering useful recommendations to me.

Another challenge is to make it as dead-simple as possible to leave a rating or comment. Most scientists, especially the early adopters you'll need to attract, have little patience for jumping through pointless hoops.
posted by chrisamiller at 8:35 PM on August 20

Since open access journals are becoming/have become a well-established segment of scholarly publishing, it probably behooves you to look at some of the many open source software packages that are intended for publishing open access journals. This may give you an idea of what features your readers will expect, and suggest new ones that you haven't thought of. Also, does your institution have a Scholarly Communications or Institutional Repository Librarian? This will be right up their alley and something they can give you lots of helpful advice on.
posted by MsMolly at 9:01 PM on August 20

1. I like the ideas here. For example, I think a discussion page for each paper (with the capability for pingbacks) would be very powerful.

2. Some research aggregators do forward pointers, sometimes under the name of autonomous citation indexing. Citeseer comes to mind.

3. To be honest, I agree with the comment above that all these features would be far more useful in a research aggregator than a journal. I read, review, and submit papers to perhaps a dozen journals. If each one has its own way of providing recommendations and comments, learning each of the individual systems is a huge bother, which would make me reluctant to use them. Furthermore, an aggregator can make use of data from lots of journals, rather than just one.

4. The network effects here are a bit difficult to manage. Perhaps your friend, who may well be a more senior researcher than I, already knows what I'm about to say, but just in case: I would hazard a guess, having never started a journal, that the single most important factor in the success of a new journal is the editorial board, who need to be leaders in their field. I would absolutely not submit to a journal unless I or one of my collaborators was either familiar with the journal's reputation, with the professional reputations of its editors in my research area, or with some quality papers it had published. Basically, I want to know that my colleagues, who are after all the audience of my paper, read and respect the journal too.
posted by sesquipedalian at 10:59 PM on August 20

My repost of this to the Science 2.0 room on FF contains the following as of today. Click through to read it in context with appropriate links, etc:

-- In my opinion, it's essential that the entire peer review process is attributed to real-names and is fully published. Basically the opposite of standard anonymous peer-review. Readers can see the originally submitted manuscript; communications between editor, referees, and authors; referee reports; author responses; and all iterations of this, up through the final manuscript. In my opinion, if it doesn't do this, then it's not awesome :) - Steve Koch

-- Published yes; real names no. I sign all my reviews but I'm a failure anyway in academic terms, and I'm just plain ornery besides. You're never going to get full and frank review when the reviewer is scared of the reviewee. Give the reviewers the option to remain anonymous. - Bill Hooker

-- Hey Bill, I strongly disagree, though I do get your point. Anonymous review leads to lazy review. Attributed review leads to real & honest review. Let's consider a famous lab submitting a careless and flawed paper. I think most scientists would not choose kissing ass over pointing out errors. Instead, they will choose from: (A) having huevos to shred the paper, (rare) (B) figuring out how to tactfully point out the mistakes (sometimes) (C) declining to review at all (common / for fear of... more... - Steve Koch

-- Furthermore, I think that even if the paper ends up not being reviewed, the journal should post the original submission. This would really make authors ask the question, "is this really what I think is the final version?" I'd say 1/2 the papers I review appear to have been hastily submitted with the understanding that it's likely they'll be able to continue doing experiments and submit a major revision after review. That's a huge waste of my time. And I'd like to get credit for pointing out the necessary controls, citations, and analyses that they end up doing. - Steve Koch

-- Steve - when you propose "The editor should have a rule that if x number of reviewers decline to review, then the paper is returned to the original authors.", you add another burden on manuscripts that cross disciplinary boundaries or other lines of traditional thought. Anonymous reviews that are not made public tend to lead to laziness but if the original manuscript plus the reviews plus the reply to the reviewers are all public, anonymity does not seem to have this effect. For some journals which already use such a system, see http://www.atmos-chem-phys.net/ and http://www.biogeosciences.net/ . - Daniel Mietchen
One way to awesomeness could be to integrate the papers with existing knowledge much more than is current habit - drop the "List of references" in favour of direct hyperlinks (also to the raw data) whose targets are archived in a standard fashion (e.g. via http://www.webcitation.org/ ), integrate/ embed multimedia and 3D content, provide one version of an article "as is", while making another editable by anyone registered with their real names, such that errors can be fixed and the context can be updated as new research comes in or is replicated. Make it machine-readable. - Daniel Mietchen

-- Awesomeness? Don't call it a journal. Forget everything you know about journals. Take inspiration from non-journal websites and create something new. - Neil Saunders

-- I don't like being a wet blanket but while I applaud the enthusiasm, looking at the original question most of this has been tried in one form or another already. These things generall fail when they are trying too much to be like a journal but they also fail when they don't look enough like a journal because people can't see any value in submiting good work to them. I think there are three or four similar iniatives out there at the moment driven mostly by people who think it can be done "for free" (which it can, but only if they are willing to work full time on it without pay unfortunately) - Cameron Neylon

-- I would take this: http://sciphu.com/ and make it better :-) Best of luck ! at some point one of these initiatives needs to succeed. - Nils Reinton

-- Make sure to let authors keep the copyright and make it clear to everyone that it is indeed peer-reviewed so that no flags will be raised when listed on CVs - Jean-Claude Bradley
posted by chrisamiller at 2:21 PM on August 22