Hundreds of characters indexed in the cloud (fictional worldbuilding data management tools)
November 27, 2012 12:41 PM   Subscribe

Us: 13 participants in a multi-year group fiction/worldbuilding project with hundreds of named characters, places, and items to keep track of. Our current system (YahooGroups databases, GDocs, and some other resources) works, but I keep thinking there's a better solution out there. More details within, but we're looking for something that does all of what we're currently doing, plus internal linkage, effective relevance-based searching, internal sorting, tagging or some other method of grouping data in multiple ways, and that is web-based, lightly password protected, accessible on a wide variety of devices, and easy for our other participants to use. (We'd also take a pony, if you had one.)

- We're looking at about 2000 individual 'entries' (where each entry is a named person, item, business, etc.) by the end of the project. We currently have somewhere over 1000.

All of them have links to specific pieces of the external project (standard web links, but they should be clickable, not require pasting.) Some entries are long (1000+ words by the time we're at end of project). Some are very short (a line or two).

- We have 13 player-authors at varying levels of technical skill using a wide variety of devices. (We also have an external readership.)

- For project timeline reasons, we'd look at switching to a new system if we could get it up and running in 3-4 months (with a fair amount of time we can throw at the move), but not something that would require extensive building. (In which case, we'd stick with what we currently have.) Assume more time/energy available on the data management side than the "fiddle with code" side.

Our resources
- Two people (I'm one of them) with reasonable technical eptness. I'm an IT Librarian, and good at making the data dance once I have somewhere to put it (more on the end-user side than the coding side, but I'm good if there's documentation) and I'm willing to come up with a tagging/indexing/etc. system to manage terminology as needed. She describes herself as "jack of all trades with reasonable sysadmin proficiency and comfortable with several different programming languages, but extremely busy and not inclined to code something from ground up".

- We're willing to consider a one-time cost or a small ongoing cost ($5ish a month) for the perfect solution, but we'd vastly prefer free and open source.

- My partner in this says: "cloud-hosting server space running Ubuntu 12.04.1 LTS, with root access and the usual LAMP stuff already installed, although for cost control purposes we would like to keep the RAM requirements under 512K."

Stuff we absolutely need (and currently have)
- Web accessible, without needing the individual users referring to the info to remember to download/upload/etc. (An exception for the people - mostly me - who enter new data is possible.)

- Free for individual users (up to about 15 people)

- Accessible from a wide variety of devices (Mac, Windows, phones, tablets, Nook, etc.) and accessible to other people on the project at varying tech levels and available time/energy to learn new things. (We're all good at web/comment/etc. stuff, but complex search methods or unfamiliar filtering tools are easier for some people than others.)

- Password protected, but with easy access once we're logged in. (This project has an external readership: we don't want our readers seeing our notes, but we also don't want to be prompted for passwords every 5 minutes or every time we change pages.)

- Easy and rapid data entry (we drop 40-150 new references into our docs every week, sometimes more) Ctrl-K in GDocs is very very handy.

- Multiple people can access without stepping on each other's feet. (Generally, I do the data entry, but other people may be reading/sorting/etc. pieces of it simultaneously. We sometimes hit problems with GDoc spreadsheet filters, for example.)

- Likely available/supported for another 3 years (duration of the project), and with some sort of export/backup method where we can get the data out if we need to.

- Ability to link to the actual content (standard weblinks) as clickable links. Ideally with very easy entry (most of what we drop into our notes are a sentence of explanation with a clickable link built into the summary).

- Ability to search (say, on a character last name) and find all references within all of our documents.

Stuff we'd really like
- Multiple sorting criteria

- True full-text searching, preferably with relevance.

- Linkage to individual entries (i.e. to a specific NPC: with GDocs, we can link to another GDoc, but not to a specific heading within it.)

- Cross-referencing (see point above). Ideally, I'd love some reliably way to do both See: and See Also: refs.

- The ability for individual pieces of data to appear in multiple heads and subheads (think employment, age data, etc.)

- The ability to browse, ideally in some sort of sortable order (i.e. everyone who works at X place, everyone who went to school in Y years.) One thing we really like about our current system (despite some of the indexing issues it raises) is being able to see everyone in a particular category at once. (That's the basis for each doc, mostly.)

- The ability to include images would be excellent, but is by no means a requirement. We're unlikely to need audio files or video.

Things we've looked at
I've looked at Omeka, which is fascinating, but I'm not happy with the password tests I've done, our metadata preferences are not theirs, and I found data input took way too long. (I'm the one likely to be putting those 40-150 pieces of data in every week.) I've also looked at various CMS tools to various degrees (Drupal, Joomla, and I do a lot with WordPress), and all of them seem to have too clunky a workflow for this use. I'm willing to consider otherwise if you've got a specific recommendation, though. My collaborator on this is not willing to touch Ruby-based solutions, and I am fine with that.

(I've also looked at TiddlyWiki: I'm not convinced it'd work on all necessary devices - see the Nook issue - and it might drive me, personally, up a wall. But if you've done a project of this complexity with it, tell me about it?)
posted by modernhypatia to Technology (10 answers total) 6 users marked this as a favorite
I hate TiddlyWiki. It uses JS and becomes very slow when there are numerous entries. Also, I'm not sure that it's feasible for multiple to edit at the same time (and certainly not easily).

This sounds like you need a more general Wiki solution. PmWiki looks like a possible candidate. Or just google search for "Password Protected Personal Wiki".
posted by ethidda at 12:52 PM on November 27, 2012

Thanks, ethidda. Your reply reminds me that I forgot to explain why we didn't go for a wiki solution in the first place: when I started pulling all the indexing together last spring, "being able to see entire category of people/entries in one place" seemed more useful than "be able to link to individual entries".

Now that I've got masses of data entered, the first part's still very useful, but the second one is getting more of my interest again. PMWiki looks like it'd do much of what we want, but we'd still be stuck with problems doing certain kinds of searches. (Though I can think of some possible category-based solutions.)
posted by modernhypatia at 1:08 PM on November 27, 2012

Yeah, this basically sounds like a wiki to me. JAMWiki is a solid re-implementation of MediaWiki (the software that runs Wikipedia) in Java, intending to eventually be 100% compatible with MediaWiki, which currently supports all of the features I use, at least, like user groups, templating, categories, and full-text search via Apache Lucene.

Your partner's talking about 512 megs of RAM, not kilobytes, right? Because 512K of RAM would be, like, a desktop calculator.

JAMWiki runs just fine for me in 512M on Ubuntu and the Resin web server for Java. There's a .deb repository for installing Resin on Ubuntu.
posted by XMLicious at 1:09 PM on November 27, 2012

Do you know about transcluding in wikis? Apart from categories you can also embed one page in another page or break up pages into pieces that appear in multiple locations.

(Like you could have each character's stats, if this is an RPG, be entered into a page at /CharacterName/stats and have that be embedded as a sidebar in the main article about the character and also in a page that's just a big table of the stats of all characters listed together.)
posted by XMLicious at 1:15 PM on November 27, 2012

XMLicious - M, not K, you're right. (Sorry, neither of us caught that.)

And I did not know about transcluding in wikis, and that might very well solve my issues with using one rather than the current system. (Journal-based RPG, more or less: no stats, but we have things like dates or ages, and various 'this person is a member of X group' kinds of data: fairly standardised, but more complicated than pure numbers.)
posted by modernhypatia at 1:24 PM on November 27, 2012

If you go with a wiki check into templating too as it's the general mechanism for doing almost anything sophisticated and for separating content from styling/formatting. (Templating is essentially a very limited static programming language for wikis, though you can usually also implement extended functionality in PHP or Java or whatever the wiki software is written in.)

"This person is a member of X group" seems like a perfect fit for wiki categories.
posted by XMLicious at 1:36 PM on November 27, 2012

Alternatively, what about a password protected blog (I'm thinking a private wordpress, though the wordpress staff would still be able to see it--or you can install it on your server). You can have multiple authors. You can link to individual posts. You can tag posts with the relevant categories, and then get all the posts on the same page for a category. You can have HTML anchors in your post to link to a specific section.

I'm not sure you can search it without making it public, though.
posted by ethidda at 1:36 PM on November 27, 2012

Much thanks for the answers.

ethidda - I'm familiar with WordPress, but for this use, I can't think of a way to make it workable at the scale we'd need. (Either we'd go with 30ish long pages - which is what we're currently doing, only I could link to anchors in other pages - or we'd end up doing individual entries for each person (aka 1500 or so), and having a really hard time wading through to find the ones we needed to edit when or working with some of the larger categories. Since that would make a massive increase in the data entry time, I'd rather not.)

We're kicking around what organisation for a wiki solution would look like, and contemplating PMWiki, though.
posted by modernhypatia at 6:54 AM on November 28, 2012

Just make sure that whichever wiki software you choose has the features you're planning on using - some actually don't implement most of the features in common usage on Wikipedia.
posted by XMLicious at 1:39 PM on November 28, 2012

Have you considered something like GitHub? GitHub has Gist pages which are sharable wiki content, you can start for free for up to 2 GB and GH has clients for nearly every platform.

The documents you currently have could be version controlled, and you can share data for free to start and create public/private areas for a minimal cost ($84/year for 5 repositories on GitHub).

GitHub has a lot of nice features, including version control, public/private access and it would invite contributors to fork your worlds and stories while retaining your right of authorship.

I think you may be looking for more of a CMS than GitHub but maybe check it out. It might be a better solution than you'd think.
posted by lon_star at 11:27 PM on January 15, 2013

« Older My processes, they are   |   My cat may have been misdiagnosed. Time to find a... Newer »
This thread is closed to new comments.