Expected speeds of upload to Gdrive

October 29, 2018 5:27 PM

Uploading 197 GB of files. I am sitting on 442-697 Mbps on the upload side. Even with 50% overhead I should have had this done in less than 8 hours. What am I doing wrong?

So, I decided to migrate by ebook library to Gdrive away from Dropbox for a variety of reasons and consolidate my other reading assets. Anyway, assume less than a TB of data and yes, I trimmed it down. I have been uploading for WEEKS. Am I just calculating things wrong? The latest installment is just 197 GB and at that speed, my upload is I should be looking at a few hours, not a few weeks.

Relevant details:
* external hard drive SSD USB 3.0
* upload speeds at lowest are 444 Mbps with a high of 697 Mbps; download 242 Mbps
* On OSX and using gDrive's sync folder
* Since last Tuesday from this latest round I am at ~61,000 files out of 160,581
* Don't know if it is relevant at all, but I am in Australia

Where is the weak point? What am I doing wrong? Is there anything that can be done? Was there a better path?

posted by jadepearl to Computers & Internet (11 answers total)

Just a data point as a fellow Australian: Google Drive is pretty sluggish at the best of times, and there is a big big difference in expected upload/download speeds for 197GB of data in a single file (or, say, >1,000 files), and expected upload/download speeds for 197GB spread across almost 200,000 files.

What the reason is for that I can only conjecture ineptly as I've lost all of my networking knowledge and didn't have much to begin with, but perhaps it's that each file is being treated as an individual packet with a header a footer, thus contributing quite significantly to the overhead?

Plus, forgive me if I'm misunderstanding completely, are you downloading from DropBox and uploading to GDrive simultaneously, and on top of that your Mac is first caching the downloaded DropBox file, then placing it on your external SSD, and then uploading it to GDrive?
posted by turbid dahlia at 6:43 PM on October 29, 2018

I don’t know the tools are telling you these numbers, but this could be a confusion between Mbps (megabits/second) and MBps (megabytes/second). Mbps is what network engineers talk about, MBps is what normal people care about. 1 MBps is approximately 10 Mbps, so if this is the issue you’re overestimating your upload speed by a factor of ~10.
posted by russm at 6:45 PM on October 29, 2018 [2 favorites]

(Just thinking out loud about what I'd do:)

Step 1: Pull down everything from DropBox in batches of say 5,000 to store locally on my internal drive (instead of introducing the extra step of moving it via USB to the SSD)

Step 2: Upload in batches of say 5,000 to gDrive (though for uploading I'd probably do even less, say 2,000).

Sorry I have been of zero help :-P
posted by turbid dahlia at 6:47 PM on October 29, 2018

Response by poster: In reply to folks:

* all data has already been pulled from dropbox and is on an external hard drive so no download and then upload. It is all being done through the Google Sync Folder/Tool that is the most recent iteration recommended with Drive One customers
* Looking at the speed test it is Mbps or megabits. I used this calculator to figure out length of upload with a very generous overhead
posted by jadepearl at 6:58 PM on October 29, 2018

I moved about 60-70 GB to GDrive earlier this year and I found the sync tools painfully slow. It was much quicker to just drag folders onto Gdrive in the browser in small batches as I had time. For static data (mine is photos and music) I wasn't worried about maintaining any sort of sync. The data changes infrequently enough that it's just easier to manually add new files once in a while.
posted by COD at 7:02 PM on October 29, 2018 [1 favorite]

upload speeds at lowest are 444 Mbps with a high of 697 Mbps; download 242 Mbps

I am in Australia

As an Australian I'm having trouble believing that both of these statements are true, unless there is an Australian ISP offering gigabit-class upload that I'm just completely unaware of, in which case who is it because I want some.
posted by flabdablet at 9:08 PM on October 29, 2018 [1 favorite]

Response by poster: flabdablet, I am on ethernet at the University of Sydney. It was those, sweet, sweet numbers that drove me to Uni. If I was trying this on residential, I would be weeping. Curse you, Optus!
posted by jadepearl at 9:39 PM on October 29, 2018 [1 favorite]

OK. Well in that case your problem is likely just to be the Google Drive desktop client, which every time I've ever attempted to do anything useful with it has struck me as a hideous bug-ridden pile of garbage, easily confused by anything resembling subdirectories. It's probably been uploading the same small file over and over and over again for the last few weeks.

You might well get better results from uploading stuff to drive.google.com through a web browser. Bust it into folders of a few 10s of GB each so you can pick things up easily again after the inevitable process interruptions.
posted by flabdablet at 10:03 PM on October 29, 2018

Simple answer: talk to your Uni's Core/Border/External/Research networking administrators. This is right up their alley. Probably quadruply so if your issues wasn't personal and wasn't gDrive. But they'll probably tell you something like "our connection to the internet is only 10G and it's shared between us and Melbourne and all the other Uni's. You're trying to use 1/20th of all of our commodity internet connection to upload your books."

I'd still guess that 1/2 of your problem is just gDrive sync lameness. Turn off sync, copy things over once in bulk, turn sync back on.

Anyway, your network admin would know what a sane speed-to-google should be and have a better idea.
posted by zengargoyle at 1:36 AM on October 30, 2018 [1 favorite]

most universities often do significant traffic shaping so that people like you don't overwhelm their network. uploads/downloads start super fast but then slow down the longer they stay open.

i'm currently on a university connection with a similar upload speedtest and i upload 800 mb of data once a week in two files. it should take me 10 seconds based on the speedtest- it actually takes closer to a minute.
posted by noloveforned at 6:01 AM on October 30, 2018 [1 favorite]

I just uploaded 500MB from a windows laptop to my google drive. I first attempted it with the Microsoft Explorer browser, and it predicted an upload time of 60 minutes. After getting about 20 into it I decided to try it with Chrome (which was also installed on that computer) and it completed in 8 minutes.

TLDR, try a browser and specifically try using Chrome.
posted by Winnie the Proust at 6:15 AM on October 30, 2018