the Dr Frankenstein of collaborative software
August 27, 2008 5:18 AM Subscribe
Easiest, most effective way to stand up a collaborative portal?
I'm interested in standing up a collaborative portal for a new digital community. I want to take advantage of new advances in web software over the last 3-4 years.
I want to include as many of the following "features" as possible (in order of preference): membership profiles, social relationships, social bookmarking, search, content management, file storage and sharing, wiki, messaging, discussion boards, ratings & rankings, chat, sub-group creation, whiteboards, project management, time management, web-casts, surveys, polls, blogging, collaborative filtering, and Q&A.
Sure, I can implement these features from the bottom up myself - it would take awhile and be quite expensive. However, I really don't want to re-invent the wheel. I know most, if not all, of these features already exist in some form or another, either as either hosted services or separate web products.
So, a few questions:
1. First, is it even possible to mash up all of these various hosted services into something that looks like one cohesive site?
2. If yes to 1, are there patterns, best practices, examples, or support groups to help execute such a thing?
3. Otherwise, if I host it myself (to protect sensitive data), would it be easier to build it from scratch (assuming I'm an expert developer)? Or hack together a bunch of separate products?
I'm interested in standing up a collaborative portal for a new digital community. I want to take advantage of new advances in web software over the last 3-4 years.
I want to include as many of the following "features" as possible (in order of preference): membership profiles, social relationships, social bookmarking, search, content management, file storage and sharing, wiki, messaging, discussion boards, ratings & rankings, chat, sub-group creation, whiteboards, project management, time management, web-casts, surveys, polls, blogging, collaborative filtering, and Q&A.
Sure, I can implement these features from the bottom up myself - it would take awhile and be quite expensive. However, I really don't want to re-invent the wheel. I know most, if not all, of these features already exist in some form or another, either as either hosted services or separate web products.
So, a few questions:
1. First, is it even possible to mash up all of these various hosted services into something that looks like one cohesive site?
2. If yes to 1, are there patterns, best practices, examples, or support groups to help execute such a thing?
3. Otherwise, if I host it myself (to protect sensitive data), would it be easier to build it from scratch (assuming I'm an expert developer)? Or hack together a bunch of separate products?
It's very, very, very hard to build such a website using 1) off-the-shelf-tools, and 2) that stays cohesive.
I have to leave for work in a minute, but I have had the opposite experience of verb with Drupal. We built a huge site on top of Drupal and it resulted in months of frustration trying to impose our exact vision onto the existing modules (most of which only got us 80% of the way we needed to go, and in some cases were so brittle that we had to rewrite the functionality from scratch anyways: example, Organic Groups, which does not -- or did not -- support walled garden grouping out of the box -- which is pretty much what people consider the core competency of a grouping functionality should be).
The second challenge after wrestling with simple implementation was actual UI -- this was a more difficult task, made almost impossible by the inflexibility of the Drupal UI and lack of integration between modules. What we ended up with was wholly confusing. If you want to approach this as more than a hobby effort, I would fully recommend starting out with a UX expert.
If I had to do it again, I'd use a rapid-dev framework like ruby on rails so there's a greater ability to customize and integrate all the functionality of the website on the front-end. Other ROR bonuses are really well integrated test suites (which is HUGE), and pretty awesome deploy tools.
Don't get me wrong, you *can* build this with Drupal. In fact, you could put together a simulcrum of what you're talking about in less than a month with Drupal, but it will *definitely* be the frankenstein of your title, and will probably take a monster machine to run on (Views and OG are are massive resource hogs, imho, and from what I saw of CCK-generated content types, the generated schema was super-wasteful -- TEXT fields for string vars, unnecessary lookup tables for typing, that sort of thing). If you aren't a developer, I'm going to say that Drupal or Joomla or another CMS with module functionality is your best bet, and out of Drupal and Joomla, I'd take Drupal (version 6, for sure).
posted by fishfucker at 8:03 AM on August 27, 2008 [1 favorite]
I have to leave for work in a minute, but I have had the opposite experience of verb with Drupal. We built a huge site on top of Drupal and it resulted in months of frustration trying to impose our exact vision onto the existing modules (most of which only got us 80% of the way we needed to go, and in some cases were so brittle that we had to rewrite the functionality from scratch anyways: example, Organic Groups, which does not -- or did not -- support walled garden grouping out of the box -- which is pretty much what people consider the core competency of a grouping functionality should be).
The second challenge after wrestling with simple implementation was actual UI -- this was a more difficult task, made almost impossible by the inflexibility of the Drupal UI and lack of integration between modules. What we ended up with was wholly confusing. If you want to approach this as more than a hobby effort, I would fully recommend starting out with a UX expert.
If I had to do it again, I'd use a rapid-dev framework like ruby on rails so there's a greater ability to customize and integrate all the functionality of the website on the front-end. Other ROR bonuses are really well integrated test suites (which is HUGE), and pretty awesome deploy tools.
Don't get me wrong, you *can* build this with Drupal. In fact, you could put together a simulcrum of what you're talking about in less than a month with Drupal, but it will *definitely* be the frankenstein of your title, and will probably take a monster machine to run on (Views and OG are are massive resource hogs, imho, and from what I saw of CCK-generated content types, the generated schema was super-wasteful -- TEXT fields for string vars, unnecessary lookup tables for typing, that sort of thing). If you aren't a developer, I'm going to say that Drupal or Joomla or another CMS with module functionality is your best bet, and out of Drupal and Joomla, I'd take Drupal (version 6, for sure).
posted by fishfucker at 8:03 AM on August 27, 2008 [1 favorite]
Despite what I just typed, I'll second fishfucker's warnings. The kind of functionality you've laid out is complicated no matter how you cut it, and using a modular system like Drupal you'll almost certainly end up finding rough edges, bumping into/working around workflow assumptions, etc. The "Frankenstein" quality that is being discussed will be very, very difficult to avoid unless you commit to hiring some user experience experts, ironing out the details of how you want the site to function and how you want people to interact with it, and implementing carefully to avoid painting yourself into a corner. That warning applies whether you use an existing system, build it from scratch, or whatever.
As I said early -- I'm biased and I've spent the better part of three years working with Drupal, so stepping around its potholes and working with its (many, IMO) strengths is what I'm used to. A lot of the limitations fishfucker notes can be overcome "easily" if you spend six months learning the APIs in some depth, but that undercuts the benefit of using a simple LEGO-style tool to build the site. By all means take a look at it, but don't take my word for it.
posted by verb at 9:14 AM on August 27, 2008
As I said early -- I'm biased and I've spent the better part of three years working with Drupal, so stepping around its potholes and working with its (many, IMO) strengths is what I'm used to. A lot of the limitations fishfucker notes can be overcome "easily" if you spend six months learning the APIs in some depth, but that undercuts the benefit of using a simple LEGO-style tool to build the site. By all means take a look at it, but don't take my word for it.
posted by verb at 9:14 AM on August 27, 2008
I'm a huge fan of Drupal, so I strongly recommend you take a look. However, there are definite limitations. (My experience is that there are limitations with any approach you take, including writing it from the ground up, and that one way or another, if you want to have things exactly the way you envision them, you'll end up writing your own code to get past those limitations - but at least if you start with Drupal, all the stuff it DOES do for you is already done. You get a nice foundation to start with.)
I'd suggest picking one aspect of what you want to do and then building it using at least two toolsets. Start with the member profiles (or any other component, just pick one), and install Drupal, Ruby on Rails, and maybe one other framework or CMS or what have you. Build three starter versions of your dream portal, with a focus on member profiles (or whatever feature you picked), until you're happy with them.
That's really the only way to know which approach will work best for you.
posted by kristi at 6:40 PM on August 27, 2008
I'd suggest picking one aspect of what you want to do and then building it using at least two toolsets. Start with the member profiles (or any other component, just pick one), and install Drupal, Ruby on Rails, and maybe one other framework or CMS or what have you. Build three starter versions of your dream portal, with a focus on member profiles (or whatever feature you picked), until you're happy with them.
That's really the only way to know which approach will work best for you.
posted by kristi at 6:40 PM on August 27, 2008
A lot of the limitations fishfucker notes can be overcome "easily" if you spend six months learning the APIs in some depth, but that undercuts the benefit of using a simple LEGO-style tool to build the site.
I tend to agree with this, and I hope my un-nuanced message didn't put you off (like I said, I had to run out the door). Despite my bad experience with a couple Drupal projects where I either inherited someone else's mess or produced one (due to extremely tight timelines); there is some elegance in Drupal's API, and it's fairly developer friendly. It's actually may be a good first choice for someone looking to start practicing web development, as many of the more difficult or tedious work has already been done (menu/path system, form system, etc). I guess my major problem is I consistently see Drupal touted as an end-user swiss-army-knife, when in reality, a LOT of the third-party modules (which give you functionality beyond the basic multi-user blog that Drupal comes as stock) are often of poor quality and need a developer to seat them correctly.
To the OP; I think in my rush to advise against Drupal, I failed to adequately respond to your question:
1. First, is it even possible to mash up all of these various hosted services into something that looks like one cohesive site?
Yes and no. I'd probably start with a good book on user interface design. I think a couple important questions to ask yourself here is "Why do I need this feature?". Less is always more -- think of the best websites out there. Most of them do one thing and do it well. Flickr, Twitter, Google (which has a lot of functionality, but all carefully branded and in essence 'name-spaced'), 37signals projects, YouTube. After trying to build a project very similar to what you're talking about, I'm of the opinion that having tons of features isn't as useful as having three or four features that work well together and really NAIL the purpose and target market of the website.
2. If yes to 1, are there patterns, best practices, examples, or support groups to help execute such a thing?
I don't know. I'd recommend reading books on usability and user experience. I've heard "Don't Make Me Think" is an excellent start, but it's still sitting on my shelf.
3. Otherwise, if I host it myself (to protect sensitive data), would it be easier to build it from scratch (assuming I'm an expert developer)? Or hack together a bunch of separate products?
Easier really depends on how much custom functionality you want to add. If your interest is primarily in *building* this thing, I'd pick rails or django and have some fun. If your interest is primarily in *using* the result, I'd pick existing software that comes close and hack in anything else you need.
posted by fishfucker at 8:00 AM on August 28, 2008
I tend to agree with this, and I hope my un-nuanced message didn't put you off (like I said, I had to run out the door). Despite my bad experience with a couple Drupal projects where I either inherited someone else's mess or produced one (due to extremely tight timelines); there is some elegance in Drupal's API, and it's fairly developer friendly. It's actually may be a good first choice for someone looking to start practicing web development, as many of the more difficult or tedious work has already been done (menu/path system, form system, etc). I guess my major problem is I consistently see Drupal touted as an end-user swiss-army-knife, when in reality, a LOT of the third-party modules (which give you functionality beyond the basic multi-user blog that Drupal comes as stock) are often of poor quality and need a developer to seat them correctly.
To the OP; I think in my rush to advise against Drupal, I failed to adequately respond to your question:
1. First, is it even possible to mash up all of these various hosted services into something that looks like one cohesive site?
Yes and no. I'd probably start with a good book on user interface design. I think a couple important questions to ask yourself here is "Why do I need this feature?". Less is always more -- think of the best websites out there. Most of them do one thing and do it well. Flickr, Twitter, Google (which has a lot of functionality, but all carefully branded and in essence 'name-spaced'), 37signals projects, YouTube. After trying to build a project very similar to what you're talking about, I'm of the opinion that having tons of features isn't as useful as having three or four features that work well together and really NAIL the purpose and target market of the website.
2. If yes to 1, are there patterns, best practices, examples, or support groups to help execute such a thing?
I don't know. I'd recommend reading books on usability and user experience. I've heard "Don't Make Me Think" is an excellent start, but it's still sitting on my shelf.
3. Otherwise, if I host it myself (to protect sensitive data), would it be easier to build it from scratch (assuming I'm an expert developer)? Or hack together a bunch of separate products?
Easier really depends on how much custom functionality you want to add. If your interest is primarily in *building* this thing, I'd pick rails or django and have some fun. If your interest is primarily in *using* the result, I'd pick existing software that comes close and hack in anything else you need.
posted by fishfucker at 8:00 AM on August 28, 2008
Also, my new ambition is to do Click & Clack style commentary on web frameworks with fishfucker.
posted by verb at 11:58 AM on August 28, 2008
posted by verb at 11:58 AM on August 28, 2008
This thread is closed to new comments.
First, the freebies. Some of the features, like customizable user profiles, multi-user blogging, polls, indexed searching and discussion forums, are already built into Drupal.
Drupal also allows you to define your own 'content types' like blog posts, photos, and so on, then give different groups of users access to using/viewing them. All these content types start with basic title/body/author data, and a plugin module (the 'Content Construction Kit') allows you to customize them by adding new fields. Using that kind of mechanism, you could easily build the basic kinds of content (file uploads, wiki posts, etc) that populate the site.
The incredibly useful 'Views' module then lets you build custom listings of these types of content (as well as many other discrete data types, like user accounts or uploaded files). Want to make a sidebar block with the five most recent comments? A page that lists comments posted in reply to blog entries by the currently logged in user? An RSS feed with podcast enclosures for every MP3 file uploaded to the site? That's the kind of stuff that Views lets you build by combining filter rules and various output settings.
The bulk of your functionality will really be a combination of custom content types, permissions to control what users can create or look at them, and custom Views to control where they appear and how they're presented. These pieces form the skeleton. Other modules can be dropped in to add additional functionality or add special behaviors to those types of content as needed. For example...
Social relationships can be manged using a couple possible plugin-in modules. For simple stuff, there's a module called "Flag" that lets any user flag another user as a friend. You can then build lists of what users someone "has friended" and build lists of content that are sensitive to the "friend status" of the authors vs. the user currently viewing the listings. "Stuff recently posted by my friends" and so on. There's also a User Relationships module that is a bit more complex but allows more detailed social graph features, like constructing "boss" or "coworker" relationships based on other existing relationships. (Bob lists Ed as his brother, and Ed lists Mary as his sister, so Bob is Mary's sister.) It's not often you need that sort of thing, but it's there if you do.
For the social bookmarking, it's possible to simply create a 'bookmark' content type that includes a 'link/url' field. If you're interested in integrating with an existing service, there's a Delicious module for Drupal that lets users specify their delicious username, and auto-import their bookmarks. They then become content on your site to be listed/displayed/searched/etc.
Wiki management is generally done by setting up a content type with very permissive access control: any user can edit it regardless of who created it. Turning 'revision tracking' on for that content type, and dropping in the Diff module, means that it's easy to track who made what changes. The Freelinking module and the WikiTools module then add bits of Wikilike behavior, including automatic linking of CamelCase words or [[Wiki|yoursite.com]] style links, auto-creation of linked pages that don't exist, etc.
For chat there are many solutions floating around, but I tend to prefer the Mibbit IRC plugin -- it sets up an IRC channel on the server of your choice and gives people a customizable AJAX-powered widget on the site to chat. Since it's powered using IRC, you also have the option of letting people hop in using the IRC client of their choice. For me, that's a big benefit because I tend to hate webchat clients.
Sub-group creation is the realm of the 'Organic Groups' plugin. It lets users with the proper permissions create new groups, invite other users, manage content that's explicitly posted inside the group, and send out announcements. The 'Fivestar' module lets you add Amazon-style ratings to any content type, and its data is automatically exposed to the aforementioned Views module, so 'Most popular content' style listings are pretty straightforward.
I'm running out of steam, as I haven't yet managed to get my first cup of coffee brewed, but in my work I've found that Drupal's modular architecture and emphasis on allowing plugins to cooperate, rather than setting up functional "silos" of data inside your site, works really well. Obviously it's possible to tie multiple sites together, or to use a hosted service like Ning to achieve some of this functionality. It's also possible to roll it yourself using an existing framework like Rails, CakePHP, Django, and so on. Given the complexity of your project, though, I think you'll spend a really monstrous amount of time doing the "little bits that have to get done," like feed generation, access control, search indexing, etc., before you can even start investing time in the real functionality.
I'm really interested to see what other folks suggest -- I work a lot with Drupal and it's very easy to get tunnel vision regarding one's preferred set of tools.
posted by verb at 6:25 AM on August 27, 2008 [4 favorites]