I'm trying to find a way of determining the "trustworthiness", for lack of a better term, of a scientific study. To do so, I would really like to know if there are systematic methods you can use to map out disciplines in which you are not already an expert. Help me sort this out.
Basically I'm speccing out a lit review for a research project. The review is a "scoping review", not a systematic review, and to do it I'm going to be covering a lot of different subject domains, many of which I don't have much experience in. I'm not interested in exhaustively cataloging all of the primary literature, nor do I have time to do so; instead I would like to get a sense of the expert consensus in each domain as it relates to my topic, where such a consensus exists. Right now I'm writing a proposal and I want to formalize my search process a bit and could use some help.
After some thought I've defined a paper to be "trustworthy" if it is accepted by a consensus of the appropriate community of experts. Some signifiers of this acceptance are favorable citation in other trustworthy papers, publication in a trusted platform, authorship by a trusted author, and peer review.
Evaluating these signifiers is easy if you are in your home domain, since you know who the experts are and you know what the trusted platforms are. However, if you're outside your home domain, it seems to be very difficult if not impossible. This seems to be a pretty fundamental problem in science (see here
) and something that everyone has to go through when they need to pull info from a field that's new to them.
Normally I would look for surveys and reviews to start from, but this particular topic is pretty new and there isn't much out there yet. The topic is pretty niche as well and doesn't have its own journals, per se; rather, results that are relevant to this topic are reported in a gaggle of journals from all kinds of different disciplines and fields (hence the need for this scoping review). The only good way forward I can see is to consult experts in all the relevant domains, but this may not always be possible, and in any case there is a chicken-and-egg problem in that I first have to find out who the experts are before I can consult them.
I can state it simpler.
- How can you find out who is an expert in a field?
- How can you find out what are the trusted journals (or other platforms) in a field?
- In particular, are there systematic ways of exploring literature that can help you figure these out?