How can I build a cloud based file server for Adobe's InCopy?

May 16, 2011 2:23 PM

How can I build a cloud based file server for Adobe's InCopy?

I'm doing some work for a client that needs a file server to use Adobe's InCopy for a geographically distributed workforce. The scenario below describes just about all aspects pretty accurately. I've managed to meet most of the requirements using Amazon's AWS, building most of this out on an Linux EC2 instance with openvpn and samba. My knowledge and expertise here is heavy in Linux, AWS is coming quickly, yet I'm clueless with Adobe products.

At this point I'm having big time trouble with network latency. I've created a Windows shared drive via Samba which is being accessed over the Internet, via the VPN. When a participant opens an InCopy project file (.icma file) it's a really long operation. There's a lock file (.idlk file) that's supposed to be created when a user "checks out" a segment of the project --yet in some cases this isn't created or doesn't actually lock the file.

We had some brief luck with something called Nomadesk. And it certaily looks like some sort of caching needs to happen...

Any ideas would be appreciated!

Scenario

Geographically distributed workforce (otherwise SOHO or mobile) needs to access hosted VPN resources to establish secure LAN over WAN (Internet). End users may be Mac or Windows based and should not be expected to own or maintain hardware, install software beyond VPN client, or perform extensive configurations. SSL (https) based VPN is acceptable and possibly preferred. The workforce will use Adobe InCopy software in a collaborative fashion. The fundamental requirement is that all members of the workforce are able to work on a single network (eg, 192.168.1/24). Additionally a hosted file server is required, preferably offering an additional layer of authentication. This file server may be cloud based and must offer transparent backup, redundancy, and D/R.

posted by ezekieldas to Technology (6 answers total)

You're missing a bunch of not-stated critical facts here.

How big are the incopy project files? If the file server is purely cloud hosted, then you are re-downloading the file across the entire internet every time you open it, which, if the client has limited network bandwidth, could be a killer. Don't forget that it's possible that the project file might reference other very large files and try to copy them into memory too.

How much real free bandwidth does the worst client location have? For example: 'mobile' could mean they use a tethering plan to pair their 3g phone, which is generally about a 1 MBps average connection if you're lucky.

How malleable are the checkin/checkout requirements; do the files need to be completely consistent at all times? Are there, perhaps, positive requirements, like there being only a few writers but many readers, or 'it's ok if there's a version clash', or 'we only have one writer per project per day' or something like that?

If the requirement is hard and fast that the users must be able to open very large files, using limited bandwidth, and that the files must be completely synchronous (a change by person A is immediately reflected upon their checkin and a subsequent checkout by person B), then the client is screwed and you should let them know that the fundamental laws of physics are not amenable to their requirements.

If the requirement can be de-strictured in some way -- e.g., the client needs huge files but is OK with them being replicated every night, and with long delays when 'true synchronicity' is overtly demanded -- then you might be able to throw something together using, e.g., rsync to branch-local caching servers.
posted by felix at 3:25 PM on May 16, 2011

Response by poster: I don't know if all of these missing points are so critical. But this may offer some insight...

1) These are typical residential connections, cable, DSL 2-6Mb
2) Checkin/Checkout needn't be so strict. For the time being I'd just like to see it work
between two clients.

You raised a good point regarding how much data is transferred with a checkout. The project file alone is 25Mb. I'm unsure if accompanying files are also transferred with a checkout.
posted by ezekieldas at 4:08 PM on May 16, 2011

In my experience, Windows file sharing just works really, really badly over slow (below 10 mbit/s) networks, especially for files over 1MB. You might well find that software designed to work collaboratively on a real LAN really does just fail miserably when you run it over a network with restricted bandwidth and higher latency, and that your clients will need to modify their workflow and/or toolset to account for the limits imposed by reality.
posted by flabdablet at 7:18 PM on May 16, 2011

There are a lot of variables and moving parts here, but key is: what bandwith do each of the remote sites (users?) have? What's the bandwith you are getting out of the Amazon service? What caliber firewalls are on the recipients' end and are they doing any filtering/QoS?

A VPN for file sharing like this really is not the best solution for any number of reasons. Doesn't matter if you're using Windows file sharing, FTP, Samba, whatever - there will be problems and it won't be a seamless expeirence for the users. Things get even worse if you are talking about databases, even simple ones, over a VPN. The problem is the network / connection quality and there are again too many possible variables to account for.

For this kind of workload you are better doing Distributed File System if the users are at remote locations, this way everything gets synced back up but they work locally on their data. The other solution is a terminal server environment running 2008 R2 using RemoteFX so the graphics performance is good.

But file sharing like you've setup, especially with anything larger than a megabyte or so is abysmal.
posted by tgrundke at 4:54 AM on May 17, 2011

It seems like you're probably in over your head a little, OP, but here's a good starting point for if you want to learn more about the centralized-desktop approach tgrundke mentions, which I agree is a better solution in practice:

http://community.citrix.com/display/cdn/Citrix+C3+Lab
posted by felix at 10:23 AM on May 17, 2011

Response by poster: These Citrix docs point to AMIs which may serve this much better. Big thanks.
posted by ezekieldas at 1:10 PM on May 17, 2011