I want to aggregate everything of mine online
October 10, 2021 7:39 PM Subscribe
I've been on the internet since 1988. I've saved pretty much everything - email, chat transcripts, irc conversations, blog posts, browser history, personal notes, photos, etc. -
I want to put it all in a place where I can easily access it. I do not have any logical explanation for why I want this but I'm constantly stressed out about it.
Currently I add things I want to save (text, texts, email, notes, photos, my old blog posts, links, articles, stories, quizzes, webcomics, threads... everything) ad infinitum to email, Metafilter, Google docs, Stickies (OSX), Notes (OSX), DropBox, Evernote, PDFs, links in Chrome, Pocket, OneTab, Facebook, notes in paper notebooks and notes on paper that get dropped on the floor under the desk. Probably more.
I haven't found any one method to collect all this stuff or that I can consistently use every day to keep track of all of this. Scrivener is an awesome app where I put everything I can find but doesn't mitigate the problem of finding everything.
I think and worry about this all the time.
When I die, I'll have no family survivors and realistically no one will notice.
And yet I still want to organize and save my digital legacy. I've been considering a memoir which would be a collection of stories I tell and even if none of my friends or no one I'm related to is still around to read it, I'd love to throw it into the world.
Thoughts? Suggestions?
Currently I add things I want to save (text, texts, email, notes, photos, my old blog posts, links, articles, stories, quizzes, webcomics, threads... everything) ad infinitum to email, Metafilter, Google docs, Stickies (OSX), Notes (OSX), DropBox, Evernote, PDFs, links in Chrome, Pocket, OneTab, Facebook, notes in paper notebooks and notes on paper that get dropped on the floor under the desk. Probably more.
I haven't found any one method to collect all this stuff or that I can consistently use every day to keep track of all of this. Scrivener is an awesome app where I put everything I can find but doesn't mitigate the problem of finding everything.
I think and worry about this all the time.
When I die, I'll have no family survivors and realistically no one will notice.
And yet I still want to organize and save my digital legacy. I've been considering a memoir which would be a collection of stories I tell and even if none of my friends or no one I'm related to is still around to read it, I'd love to throw it into the world.
Thoughts? Suggestions?
Best answer: I think you could use Paperless, since it has OCR and tagging. It's designed for scanned documents, but it works the same on anything you can save as text or a PDF (it can also intercept email attachments). So you could just go around and "print" everything to PDFs.
posted by Snijglau at 8:11 PM on October 10, 2021 [1 favorite]
posted by Snijglau at 8:11 PM on October 10, 2021 [1 favorite]
Best answer: Elasticsearch (indexing & search), Logstash (ingestion), Grafana/Kibana (if you want visuals).
There will be other digital preservationists, but raw source material from someone's life delights historians: upload your database to the internet archive along with training material for how people can use it/create their own.
posted by k3ninho at 11:41 PM on October 10, 2021 [4 favorites]
There will be other digital preservationists, but raw source material from someone's life delights historians: upload your database to the internet archive along with training material for how people can use it/create their own.
posted by k3ninho at 11:41 PM on October 10, 2021 [4 favorites]
Response by poster: (I apologize for my previous weird comments.) Please MeMail me if you'd like.
posted by bendy at 12:11 AM on October 11, 2021 [1 favorite]
posted by bendy at 12:11 AM on October 11, 2021 [1 favorite]
Response by poster: Naw, there's no database involved - that's kind of the issue. There are a ton of sources of data all over the place - a database would be a great solution (I love MongoDB) but the question is how to gather the data from everywhere. And there is a lot of ADHD involved as well.
posted by bendy at 12:16 AM on October 11, 2021 [1 favorite]
posted by bendy at 12:16 AM on October 11, 2021 [1 favorite]
Best answer: I’m not sure this will cover some/many of the things you list but might serve as inspiration and, if you’re technically minded, might be something you’d want to expand… Twitter’s own simonw has made Dogsheep, a tool to get various sources of personal data into an SQLite database with a browsable and searchable front end.
posted by fabius at 6:07 AM on October 11, 2021 [4 favorites]
posted by fabius at 6:07 AM on October 11, 2021 [4 favorites]
Best answer: You're looking at a Data Lake. And there is open source software (primarily for large companies) that have similar problems with many different kinds of data that is somewhat related but not in explicit databases. It'll be a bunch of work to get set up as it's oriented at working software devs, but certainly doable. Here's some videos for a package called Kylo.
The data lake software is oriented towards data processing but should have tools that work with your data.
Get a easy to use scanner and get in the habit of dropping all papers into it.
Most of the online services have ways (APIs) to download your data onto a local hard drive.
Set up two or more replicas physically distant and have an automated tool like rsync to keep all the copies of everything current.
There are probably some machine learning tools from researchers working on similar problems, that scan a body (lake) of data and make inferences.
posted by sammyo at 9:00 AM on October 11, 2021 [2 favorites]
The data lake software is oriented towards data processing but should have tools that work with your data.
Get a easy to use scanner and get in the habit of dropping all papers into it.
Most of the online services have ways (APIs) to download your data onto a local hard drive.
Set up two or more replicas physically distant and have an automated tool like rsync to keep all the copies of everything current.
There are probably some machine learning tools from researchers working on similar problems, that scan a body (lake) of data and make inferences.
posted by sammyo at 9:00 AM on October 11, 2021 [2 favorites]
Response by poster: It'll be a bunch of work to get set up as it's oriented at working software devs, but certainly doable.
Fortunately I am a working software dev and have been a *nix nerd for decades. :)
One of the hundreds of projects I'm in the middle of is using Postman to authenticate with various sites' APIs and download all my data, then run it through Puppeteer to check for broken links. Still it's only part of the stack of stuff.
I appreciate all your answers and I have a lot of research to do now.
posted by bendy at 5:51 PM on October 11, 2021
Fortunately I am a working software dev and have been a *nix nerd for decades. :)
One of the hundreds of projects I'm in the middle of is using Postman to authenticate with various sites' APIs and download all my data, then run it through Puppeteer to check for broken links. Still it's only part of the stack of stuff.
I appreciate all your answers and I have a lot of research to do now.
posted by bendy at 5:51 PM on October 11, 2021
Best answer: I think DevonThink could work well here. It OCRs documents, imports RSS, emails, and lots of other file formats. It has interesting search capabilities and shows "related" items that are sometimes surprising and helpful. It also can index files, so if you have folders already created, you can add them without duplicating the files.
I also wonder if the MacSparky Paperless Field Guide might help.
posted by 10ch at 7:51 PM on October 11, 2021 [2 favorites]
I also wonder if the MacSparky Paperless Field Guide might help.
posted by 10ch at 7:51 PM on October 11, 2021 [2 favorites]
Response by poster: (I hope I'm not threadsitting.)
10ch: I've been starting to use Scrivener as a place to collect text. DevonThink looks like another option to try.
posted by bendy at 9:39 PM on October 11, 2021
10ch: I've been starting to use Scrivener as a place to collect text. DevonThink looks like another option to try.
posted by bendy at 9:39 PM on October 11, 2021
Ten years ago I would have recommended StoryTlr, a web-based “lifestreaming” package. Lifestreaming collects all your social media feeds and comes up with a sort of super blog about your activities.
But I haven’t kept up with that corner of the web so I don’t know if there’s a current software solution that meets your needs.
posted by Monochrome at 7:56 PM on October 12, 2021 [1 favorite]
But I haven’t kept up with that corner of the web so I don’t know if there’s a current software solution that meets your needs.
posted by Monochrome at 7:56 PM on October 12, 2021 [1 favorite]
« Older Favourite D&D channels & campaigns to... | How can I best remove a thin welding seam from... Newer »
This thread is closed to new comments.
posted by wesleyac at 7:47 PM on October 10, 2021 [3 favorites]