How to merge multiple text files keeping original filename and date

September 12, 2009 7:07 PM

Hi AskMeFi. Can someone tell me how to combine 350 text files into one large text file so that the original filenames and dates are included above (or below) each entry? I'm using Vista Home Premium and they are all generic .txt files. I'm moving all of my notes into Evernote but want to save the names and dates of the individual txt files.
posted by chimrichalds to computers & internet (15 comments total)

I've used TXTcollector to do this with Pidgin logs.
posted by sun-el at 7:39 PM on September 12

for %I in (*.txt) do @echo ====%~nxI >>allfiles.txt && type "%~nxI" >>allfiles.txt

"====" is just a marker to make it easier to search/page through, you can use whatever you want, or nothing at all.
posted by rhizome at 7:44 PM on September 12

Here is a simple script that does what you want:

@ECHO OFF
ECHO Creating %1...
FOR /F "usebackq tokens=1" %%i IN (`DIR /B`) DO (
  ECHO Adding %%i
  ECHO. >> %1
  ECHO === %%i: >> %1
  TYPE %%i >> %1  
) 

Here is example usage (I called the script concatem.cmd):

C:\test\files>..\concatem.cmd everything.txt
Creating everything.txt...
Adding file1.txt
Adding file2.txt
Adding file3.txt

C:\test\files>type everything.txt

=== file1.txt:
this is file one.

=== file2.txt:
this is file two.

=== file3.txt:
this is file three.

C:\test\files>dir
 Volume in drive C has no label.
 Volume Serial Number is 9840-A7ED

 Directory of C:\test\files

09/12/2009  07:47 PM              .
09/12/2009  07:47 PM              ..
09/12/2009  07:47 PM               119 everything.txt
09/12/2009  07:45 PM                19 file1.txt
09/12/2009  07:45 PM                19 file2.txt
09/12/2009  07:46 PM                21 file3.txt
               4 File(s)            178 bytes
               2 Dir(s)  22,012,522,496 bytes free

Note that if you create the script and run it from the same directory as the files, it will include the script in the output file. You could modify the script to exclude it, or just remove it by hand.
posted by jeffamaphone at 7:49 PM on September 12

Should have previewed. Owell.
posted by jeffamaphone at 7:50 PM on September 12

Thanks sun-el. I saw that program when I was searching google before I asked here at MeFi. Unfortunately it doesn't look like it has an option for putting the date of the individual text files in the final merged file. I really want to keep the dates from when the text files were created because many of these text files are personal diary entries. Thanks for the response though.
posted by chimrichalds at 7:50 PM on September 12

Aw, rhizome is snazzier than me. Oh well, this works too:

import os, time

fileList=os.listdir('.')
fileList.sort()

for fileName in fileList:
     fileStats = os.stat(fileName)
     fileDate = time.localtime(fileStats[8])
     fileDate = time.strftime("%m/%d/%y %H:%M:%S", fileDate)
     print "*** FILE: %s\t\tDATE: %s" % (fileName, fileDate)
     fileCurrent = open(fileName)
     print fileCurrent.read()

This is in Python, and should be used from the command line. Put the script in the directory with the text files and run:

c:\python26\python mefi.py

Change "c:\python26" to reflect the place where you installed Python and "mefi.py" to reflect whatever you named the script.

rhizome's is better but I always forget Windows has improved its scripting capabilities lately.
posted by sonic meat machine at 7:50 PM on September 12

Rhizome has it, but it's missing the date as it stands. To include the date it should read like this (all one line):

for %I in (*.txt) do @echo ====%~nxI >>allfiles.txt && echo %~tI >>allfiles.txt && type "%~nxI" >>allfiles.txt

posted by Hardcore Poser at 7:58 PM on September 12

oops, didn't see the date part. you can just add a date chunk to the ==== bit:

for %I in (*.txt) do @echo ====%~tnxI >>allfiles.txt && type "%~nxI" >>allfiles.txt

posted by rhizome at 8:09 PM on September 12

Thanks so much! I can't wait to try it but...I'm a bit confused.

Do I replace %I with C:\Users\user01\notes from Vista's "Start Search" (Run) box?

If so, do I need to replace anything else?

Thanks again, I really appreciate all of your help!
posted by chimrichalds at 8:16 PM on September 12

Just enter it at the command line like that. The %I is an iterator, it's internal to his script.
posted by sonic meat machine at 8:27 PM on September 12

At the command line, first enter

cd \Users\user01\notes

That puts you in the directory where all your notes are. Then enter

for %I in (*.txt) do @echo ====%~nxI >>allfiles.txt && echo %~tI >>allfiles.txt && type "%~nxI" >>allfiles.txt

The for %I in (*.txt) part doesn't need to be changed at all. It tells Windows, "Run the following commands on every .txt file in the current directory."
posted by nebulawindphone at 8:44 PM on September 12

Thanks, I got it to work and am trying to tweak it a bit to add some carriage returns before or after the @echo. I searched google and found that $_ = carrriage return so I made this one that didn't work:

for %I in (*.txt) do @echo **********************%~tnxI >>allfiles.txt && type "$_" >>allfiles.txt && type "%~nxI" >>allfiles.txt && type "$_" >>allfiles.txt

I got lots of: "The system cannot find the file specified"

I'm still playing around with it.

It would also be cool if the entries were listed inside the larger file chronologically.

But hey...even if this is the best I can do, I'm very happy!

Thanks so much!
posted by chimrichalds at 9:10 PM on September 12

Two issues (I think) in your last example:

1) You're using ******* as a separator, which I'm guessing the shell is trying to expand (* is a special character).
2) Type reads out the contents of a file; echo spits out characters directly (so you either go with 'echo Hello World' or 'type hw.txt', where hw.txt contains "Hello World", to get the same result). It looks to me like you mixed those up & you need to echo the cr/lf character.
posted by range at 10:18 PM on September 12

Thanks range, I've got one that I really like that works great:

for %I in (*.txt) do @echo ********************************************* >>allfiles.txt && @echo %~tnxI >>allfiles.txt && @echo ********************************************* >>allfiles.txt && @echo. >>allfiles.txt && type "%~nxI" >>allfiles.txt && @echo. >>allfiles.txt && @echo. >>allfiles.txt && @echo. >>allfiles.txt

I would still like to have the entries in chronological order so I'm google searching now.

Thanks everyone!
posted by chimrichalds at 11:08 PM on September 12

I did eventually get all of the text files merged chronologically into one large text file with the original file name and creation date as headers along with a few carriage returns to make it look pretty.

Here is what I did in case anyone else searches mefi for this type of thing:


After not finding an easy way to sort the files chronologically via a script, I decided to just rename all of the text files using the creation date as a prefix. I made a batch file named "rename.bat" with this inside:


@ECHO OFF
FOR %%V IN (%1) DO FOR /F "tokens=1-6 delims=/: " %%J IN ("%%~tV") DO rename "%%V" %%L-%%J-%%K_%%M%%N%%O__%%V


I dropped the batch file in the directory, opened a cmd from vista's "start/search" (run) box and navigated to the directory where my text files and batch file were and typed: rename *.txt

I worked great but unfortunately the batch file would skip any files with spaces, or odd characters like: ";@,'()," etc... so I had to go download a file renamer, "Bulk Rename Utility" (a good one!) to remove all of the offending characters and move those few straglers to a different directory along with my batch file and cmd prompt and try again.

After I finally got all of the text files renamed with the creation dates in front (e.g. "2000-08-28_0644AM__jade.txt"), the files were allready ordered chronologically when sorted by name so I just used the script from the post above minus the "t" (date) part, and everything fell into place naturally like this:



*********************************************
2000-08-28_0644AM__julie.txt
*********************************************

8 lbs 5 oz!!!!!

*********************************************
2000-10-23_0907PM__coltrane_blindfold_test.txt
*********************************************

Blindfold Test: John Coltrane
An Exclusive Online Extra

by Leonard Feather — 02/19/1959....



Here's that script again only without the "t" part:



for %I in (*.txt) do @echo ********************************************* >>allfiles.txt && @echo %~nxI >>allfiles.txt && @echo ********************************************* >>allfiles.txt && @echo. >>allfiles.txt && type "%~nxI" >>allfiles.txt && @echo. >>allfiles.txt && @echo. >>allfiles.txt && @echo. >>allfiles.txt



Thanks again to all of the script wizards that responded. As you can tell, I was pretty much just feeling my way around in the dark on this one!
posted by chimrichalds at 8:55 AM on September 13