Papers, BibDesk, or a maaaagical program made by fairies?

September 20, 2011 3:39 PM

My boss needs an academic reference manager (think: Papers, Sente, BibDesk) that can handle a huge existing BibTeX reference library and a huge-separate-but-sometimes-overlapping existing library of papers in PDF format. Have one you love? Tell me about it.

My boss wants a good way to organize his library of academic papers that integrates well with his brand new iPad. Basically, he wants a well-tagged library of PDFs that he can take anywhere.

We currently have a 3300-odd-entry BibDesk (i.e., BibTeX) repository of citations that our lab uses for all papers. Citekeys are extremely regular, and the repository is in good shape. This .bib file lives in an SVN repository that all lab members can check out, add to, etc.

We also have a 2.3-gigabyte folder of papers on our server that's organized by the same citekey structure. Not all papers in our BibDesk archive have PDFs, and not all PDFs have an entry in BibDesk. This folder is not on the same server as the BibDesk SVN repo. It is accessible through the web, so a hyperlink to a PDF would work.

Put simply, we want to integrate the two as easily as possible to create his publication library. Mega-bonus points if other lab members could have access to the database files.

My dream integration involving the perfect piece of Software X would look something like this:

-- I import my huge master BibTeX file into Software X, so it has entries for each paper that was in BibDesk.
-- Software X matches the citekeys in each entry with the 2.3 GB folder of PDFs and imports the ones that match.
-- For all entries without PDFs, Software X attempts to download the paper using our university library proxy.
-- For all PDFs without entries, Software X scans the PDF for title, authors, etc and uses online archives to fill in missing data.
-- Any entries or PDFs without matches get tagged as "unknown" or "missing PDF".


I know Papers does some of this, but I had a tough time on my initial import. I want to be able to tell it, "Look in THIS folder under the citekey.pdf filename, THEN look online!" It also did pretty poorly when I imported a representative sample of my PDF files; didn't tag 90% of them. I feel like Papers is better suited to building a library from scratch.

Will Papers (eventually) do everything I need once I learn how to use it? Should I stick with BibDesk and try to add links to my papers rather than dragging the files in one by one? (Can I THEN import from BibDesk into Papers for iPad integration?) Is another piece of software like Sente more suited for our purposes?

TIA.

posted by supercres to computers & internet (8 answers total)

Just realized that cross-computer syncing would be awesome. Sente is looking better and better.

Do any users of Sente think it'll do the other things I need from it?
posted by supercres at 4:02 PM on September 20, 2011

Sente does a lot of this No ezproxy as yet, but you can log into your library web page from the integrated browser and conduct new searches from there.

Importing bib desk refs with PDFs should be straightforward. It should just find the PDF if it is associated in the bib desk library. If it isn't when you import the PDF later it will normally match it to the local library ref before searching pub Ed, google scholar etc.

Matching PDfs to metadata for those not in bib desk is also easy, but each one needs to be done individually so it will take some time, although the process is semi automatic.

Getting PDFs for bib desk refs without them is a little more convoluted because of the lack of ezproxy (although it is coming). However either using the integrated browser as mentioned above, or getting the safari to Sente service installed helps.

(personally I used to use papers for this then import inti Sente, but now I just use Sente or safari)

The good news is that, once the library is up and running it's awesome. The synch between computer/iPads really works, even for highlighting in PDFs. The note taking is very powerful. The ability to use status, rating, keywords, static (not synched :() and smart (synched!) collections makes it a breeze to keep track of large numbers of references.

The bibliography functions used to get criticized compared to bookends. However, although I don't use them myself much, they have improved.

And of course you could always try the demo!

Michael

P.s. I have no affiliation with Sente. I've used Sente, bookends and papers for several years ( and endnote before that). I've switched to just using Sente in the last year or so for my 3 libraries, each with 1,000 or so medical refs.
posted by bister at 5:33 PM on September 20, 2011

Mega-bonus points if other lab members could have access to the database files.

I don't know about the integration step, or the ipad part. But if you end up sticking with bibdesk's autofile function for the pdf file management, there is nothing to stop you from putting the pdf repository into version control also. Well, I can't really speak for subversion, I don't know how well it deals with binary files. But I have my bibdesk pdf archive, about 1GB, in git, and only the initial syncs were painful.
posted by advil at 6:06 PM on September 20, 2011

I use Mendeley. I haven't used it for a *huge* database, but it has worked reasonably well for me for figuring out paper refs with a collaborator (I could share tags and notes and new references with her pretty easily).

I also have built my Mendeley library mostly from scratch, but it does have some import functions; might be worth checking it out, anyhow.
posted by nat at 9:09 PM on September 20, 2011

I agree that Mendeley is good for this particular problem.
posted by unknowncommand at 10:26 PM on September 20, 2011

Mendeley can do many of the things you are looking for, but not all, I think. It's pretty good about scraping for metadata, though it has trouble with older papers where the PDF is not OCRed. It also conveniently renames all your files according to a format you specify (for instance, I use Author_Date_Journal.pdf). When it's not confident that it's named your paper correctly, it tags it as "needs review" and you can check it yourself.

You can sync your library across computers (by way of letting Mendeley store your entire library on their servers). You can also set up shared groups so everyone can access whatever parts of the library you desire. For a large library, you'll have to pay for the online storage space. You can store however much you want locally in Mendeley Desktop, but free online storage space (and thereby syncing of PDFs) is limited; you can sync as much citation info (without attached PDFs) as you like.

Mendeley is capable of importing BibTeX (and from Papers, and from Zotero...), but I don't think you will be able to automatically match up your BibTeX library with your other PDF library and have it find/remove all the duplicates. Poor handling of duplicate papers is the biggest problem with Mendeley right now, I think.

As far as I know, you also cannot directly import new papers from the web through Mendeley Desktop. You can add an "Import to Mendeley" function to your browser, though. (I cannot tell you about how it works, since I don't use this function - I add papers manually).
posted by pemberkins at 9:26 AM on September 21, 2011

If you have a web browser open, you can drag the link to a pdf into the papers list on Mendeley Desktop and it'll add the paper.

Good for when the Import to Mendeley function isn't working (it seems to work only patchily with arxiv, which is my main resource). Also good if it's a pdf you need official access for or if you don't like the web applet for whatever reason.
posted by nat at 5:40 PM on September 22, 2011

After trying trial versions of Papers and Sente, plus the full (free) Mendeley, I got a license for Sente. The deciding factor, I think, was the cross-desktop syncing, as my boss often works from home. I'm crossing my fingers for additional improvements in the future, like:

-- Ability to easily download papers through a proxy
-- Ability to add papers on iPad version from Dropbox, e-mail, or web
-- Smarter automatic paper lookup, using title and author keywords
posted by supercres at 9:21 AM on September 23, 2011