Ask MetaFilter questions tagged with scripting

How to merge data fields into a plain text file?

geekgirl397 — Wed, 25 Nov 2009 08:04:23 -0800

Is there some way to do a "mail merge" type operation on plain text, xml or html files without using a word processor? I've been working on our company wiki and I want to bulk create a bunch of pages by merging certain field data from spreadsheets, csv files or other datasources into the template text files that I've created. The output should be XML or plain text.

I have a separate task where I have a simple html template that I'd like to populate with data pulled from a csv. The output needs to be standard, simple HTML (no funny MS Word markup)

I have access to MS Office, Open Office and Abiword but everything I've tried so far seems to "pollute" my files with word processor type formatting. (I am running KDE on Ubuntu 9.10, but I also have Win XP running as a virtual machine under Virtualbox - I have access to any version of the MS Office software, and obviously anything open source).

I'm sure there's some open source tool designed to merge text files. I'm not opposed to using a command line interface although I probably would prefer a GUI. My Google-fu is failing me on this one.

Any tips on how I should achieve this, or pointers to the tools I should be looking at?

thanks!

MS Outlook: scheduling VBA scripts?

pompomtom — Tue, 10 Nov 2009 15:45:38 -0800

Using Outlook 2007 on windows XP, how do I run a (VBA) script either daily, or on opening the application? In excel, I'd use something like an Auto_Open() sub, but I can't seem to find an analog for Outlook. Can I just refer to the script to a recurring appointment somehow?

PowerShell: recursive processing of all files and folders without OutOfMemory exception?

splice — Thu, 08 Oct 2009 08:26:22 -0800

I need to use a PowerShell script to scan through a large amount of files and folders for certain security permissions. The issue is that it seems that the "Get-ChildItem" function insists on shoving the whole folder and file structure in memory. Since the drive has over a million files in over 30,000 folders, the script runs for 30 minutes, then throws out a System.OutofMemoryException without doing anything else. The script is as follows:

$files = Get-ChildItem c:\

foreach $file in $files {
'Do something here
}

The obvious problem is the first statement. The obvious question remains: how do I walk through a drive recursively without running out of memory? Something of a Get-ChildItem that would execute code on the target without needing to first put the whole structure in memory?

DAW scripting

vanar sena — Thu, 17 Sep 2009 23:22:01 -0800

What tools and/or DAW software do I need to automate DAW actions through scripts? Digital music recording and production is very new to me, but I have some programming experience. It occurs to me that being able to write scripts to automate DAW actions (setting EQ, effects, mix levels etc) would be fun and useful. The ideal DAW would allow running of scripts just as easily as triggering other DAW events using automation tracks, or even the generation of automation tracks via scripts.

Are there any DAWs that support this out of the box? What languages and tools do I need to know/have?

For reference, I have the following software at the moment: REAPER, RiffWorks, GarageBand on OS X; Ardour and LMMS on Linux, though I find working with audio on Linux too trying to bother most of the time.

Apologies if this is an RTFM question; if this is the case, some pointers to FMs would be perfect.

ExcelFilter: Between Day 1 and Day 90, I had how many of which?

SansPoint — Wed, 02 Sep 2009 07:09:05 -0800

How can I count certain things in a range of cells in Excel that fall between two dates? I've got an Excel workbook that keeps track of every thing I produce at my job. (What the things are is irrelevant.) The date it's created in in Column A. In Column F there are criteria to determine which type of thing is is (two specific ones). I would like to count the number of each that fall between two dates, i.e. between, say 9/01/2009 and 11/30/2009.

For the life of me, I can't figger it out.

I've tried wrapping my head around dynamic ranges, but all I can see for doing that is OFFSET(), which would be great if all the date ranges had the same amount of rows, but they don't. Besides, I add new rows to this thing regularly, as each new thingie is generated. So, having, say, the last 30 of these counted does not help.

Google searching has given me info about how to count for the last x of a certain range, where x is days, weeks, months, etc. Doesn't help.

Oh, and I *really* would not like to have to go into VBA scripting for this.

So, uh, help?

Please help me make many documents much shorter.

procrastination — Wed, 29 Jul 2009 11:43:05 -0800

How can I access the "AutoSummary" feature of Microsoft Word progammatically to summarize many documents at once? Alternatively, what other programs are there that will summarize documents that can be accessed programmatically? If this isn't the right place to ask, where is? I am working on an academic research project. I have several collections of plain text documents written in English. For each I would like to produce a number of increasingly smaller summaries of each of the documents using the "AutoSummarize" feature of Microsoft Word or some other program with similar functionality. I would like each summary saved with a file name indicative of the base file name and the size of the summary. How would you suggest I go about this?

In essence, here is my desired algorithm:

// Get all files to work over
Foreach file (read all files in directory)
    // Get all the summary sizes as a percent of original size
    Foreach summary_size_percent(95 90 85 ... 25 20)
        // Do the summarization and save the file
        Autosummarize file to summary_size_percent and
        save as file_summary_percent.txt

I have access to any common operating system and multiple versions of Word (Windows and Mac) and Pages for the Mac as well, though it seems less able to specify an exact percent for the summary feature. I have some money I can use to buy almost any other document summarization software, or if there is freeware I can give that a swing too. I can buy and install scripting software if needed.

If there turn out to be multiple ways to do this, so much the better, especially if they produce different summaries. I am mostly a Unix programmer, so I am pretty familiar with Perl and Java and C on a Unix console, but I haven't done any Windows or Mac scripting.

If you know of a better place to ask this, where would it be?

Help in rendering Gmail 'Last Accounty Activity' useless.

passtehbrainz — Sat, 11 Jul 2009 21:19:08 -0800

Automatically login / logout of Gmail at a specified interval As you know, gmail lists the latest activity at the bottom of the page, and shows the last few ip's and the times they logged in by clicking details.

I need to 'fool' this log, by automating logging in, and logging out at a specified interval. Some ideas would be through a firefox extension, applescript, a mouse clicking recording program, etc, but I haven't found any such solution.

Without going into details, I need to render the gmail activity log useless.

Doing Stuff Repeatedly

Ziggy Zaga — Thu, 09 Jul 2009 09:40:02 -0800

I need to write a script of some sort to grab content from a series of web pages. How should I go about it? Here's the deal: I have a textbook that is available in electronic form online. The problem is that each page is a JPEG that is served up individually by a PHP script. I click on a link, it takes me to another web page with the embedded image on it. I want to download all of the pages and run OCR on it so I can search the book, do copy/paste for notes, etc.

I can sit there and do this manually but there's 600 pages or so and I want to use this as an opportunity to learn a new skill.

I have tried web spiders (HTTRack) and they did not work. The PHP script serves up each file with the exact same filename. The image embedded on the page also does not have a *.jpg extension, but some random PHP stuff.

I have tried to use Selenium-IDE but apparently it can't (a) do anything incrementally without the aid of another scripting language, and (b) is unable to actually download content.

I tried FlashGot's gallery feature but I think the PHP script interferes with the referrer or something because I only end up downloading the placeholder that warns against hotlinking.

So it seems I need some way of actually walking Firefox through the steps.

Here's the pseudocode of what I'm trying to do:

- Go to URL http://www.blah.edu/students/stuff.php?eh=123
- Go to URL http://www.blah.edu/students/view.php?pg=f&id=456
- Save http://www.blah.edu/students/showpage.php?pg=f&id=456 (this is actually an image) with filename page456.jpg.
- Go back to step 2, increment the id number to 457 and repeat.

What scripting language would best facilitate this? I'm on Linux but can use Windows if I must.

Thanks

Which forums do the best web programmers hang out at?

jackson5 — Fri, 26 Jun 2009 16:15:45 -0800

Which forums do the best web programmers hang out at? Long story short: I've got a web site that already does over 1 million pageviews per month. I've got a great idea to take it to the next level, but I'm looking for someone who can handle the coding and technical aspects in exchange for an equity stake.

I'm wanting to bring aboard a great programmer who doesn't necessarily need the work but will be motivated by the equity.

Which forums would you recommend I post on to find this person?

Also, if there's anyone interested here, let me know.

Need to automate a weekly task using AppleScript

joshrholloway — Sat, 28 Mar 2009 20:20:15 -0800

I need to automate the process of printing a report to PDF with the date in the file name, saving it in an archive folder, printing it, and emailing it as an attachment. I am on a Mac and using Pages, Automator, and AppleScript. Help me find the missing pieces! I have to write a report each Friday for work and send it as an attachment to a specific email address where it gets processed by an internal server. Usually my process is as follows:

1. Write report in Pages.
2. Print report as PDF to a specific folder using OS X's built-in capabilities.
3. Name the PDF file based on the date.
4. Print the PDF file to a hard copy.
5. Attach the PDF to an email using Mail and send it.

I realize that's not a ton of work, but this Friday I felt the urge to automate the process. At first I wanted to do it to save myself some time, and then once I had spent an hour trying to figure it out it became more of a challenge, and then after spending another hour on it I didn't care about solving the problem myself and just wanted a solution by any means necessary.

Most of the individual pieces of this automated task are not difficult to accomplish, but I have found it difficult to get each step to talk to the next. I have also had trouble due to the fact that I want to make the file name of the saved PDF contain the date it was saved. I'm sure this would be easy for anyone who has used AppleScript extensively, but I'm just a beginner at it so it's baffling me.

Could someone help me figure out the proper script structure to make all this work? If it matters, I am using the latest version of OS X and Pages. The files need to be saved with the name "weekly_report_MM_DD_YYYY.pdf" (it doesn't have to be exact, just something along those lines) and the email that it is attached to doesn't need anything in it except for a recipient. If I've forgotten any necessary details, please let me know and I'll provide them.

HELP!

CU: A flange.

turgid dahlia — Mon, 09 Mar 2009 18:23:04 -0800

How To Write Technical Training Films? The government department I work for is in the midst of an enormous communications restructure and, hoping to move into corporate communications, I volunteered my services to one of the managers in my business group.

One of the myriad tasks on his plate is the start-to-finish development of a number of technical training films. The nature of our work here sees people out in the field installing specialised monitoring equipment.

From the prescis:

“These films will be for internal departmental use as training aids. The primary target will be for new staff who will not have someone at their side to assist them. It is hoped that most of the tasks in the field process will be covered. They may last a few minutes only dealing with specific technical issues or of greater length if going through data entry type procedures.

“These training films are not to be seen as replacements/substitutes for the National Standard training requirements – they will at all times be complimentary.”

So, pretty straightforward (if dry) stuff, I imagine. What I have been asked to do - with the assistance of experts in the actual processes I will be documenting – is develop the “textual storyboards”, or teleplays, if you will, for these processes.

I have a fair idea of how to set about doing this but I’d like to make an extra good impression with this work, so I’d love to hear about any easily accessible resources (preferably online) on this subject, and I would similarly love to hear advice and tips from any MeFites who have done similar work.

Find it and add it once more in mysql field

dabitch — Wed, 21 Jan 2009 02:28:03 -0800

I've made a large mess and need to fix it. It's a "find this" and "add it once more, but above it, in that same field" dealio in one single tables field in mysql. "This" is a different string every time. Not sure this can be done with mysql, so I'm open to all other ideas. Ok, so I have a badly created embed and object tag code all saved in one mysql field lets call it 'body'. I'm doing the embed like the example at the bottom of this apple page.

Body has other data/info before and after the embed and object tags so 'body' can vary greatly in length.
The Object tags calls the quicktime codebase, and underneath that we have the param name= stuff but we lack param name=src linking to a file! Below the param name comes the embed src=link to file! tag. So, what we want to do is get that link from embed, in each individual 'body' field, and create a param name src with the same link and add this to this same 'body' field.

Each link is different, obviously, and we're talking 45,000 entries which is why I am scratching my head how to do this. Is there a way to, say, add (wildcard) where like% solution I could do, I'm stumped?

Stop Bash from getting ahead of itself...

tmcw — Sat, 06 Dec 2008 13:09:53 -0800

Shell scripting quandry: running apt-get and then running a command that requires a apt-get installed package fails because bash doesn't wait for apt-get to finish... My first approach is running a shell script that runs the installation, and I've had this problem. The code is like

apt-get install blah blah blah python-setuptools
easy_install libxml

However, bash runs apt-get and doesn't wait for it to finish before running the easy_install line - and the easy_install line requires setuptools to be installed. Currently I've just split the script into two scripts at that point, so that this little timing issue doesn't come up. But that's a workaround and I'm frustrated that there isn't a better way. Using 'wait' isn't working because apt-get is interactive...

Bonus points if you can point to a way that shell scripts can just install the most recent version of a package without specifying exactly what it is in the name? My issue is that this is for Ubuntu 7-8 and I don't want it to fail because of differences in repository versions...

Thanks! I'm a bit of a shell-scripting noob, and writing a robust install script for this project (Mapnik) is a big challenge.

BASH: Handling odd characters in a variable?

Ziggy Zaga — Tue, 18 Nov 2008 13:58:55 -0800

BASH Scripting. I'm trying to do a simple "mv" operation with a not-so-simple filename that contains special characters (as in brackets and hyphens). The filename itself is stored within a variable. The operation fails, and I can understand why, but I don't know how to get around it. A sample filename would be something like:

[2008] Region {XX}.pdf

...and is stored in $sFilename. Not all of the filenames are the same format although all of them start with the year in square brackets.

I've got the rest of the script set up to parse for the year in the initial set of [ ] brackets and create a folder (./2008) with that name. The second part of the script is supposed to move the aforementioned file into the created folder. I attempt to do this with the command:

mv "$sFilename" ./$sPrefix

...where $sPrefix is the parsed item from the brackets. It errs out with the message:

mv: cannot stat `[2008]': No such file or directory.

It seems to me the shell is choking on the filename. What's the best way to work around this WITHOUT renaming the files (since the program that imports them is Windows-based and requires they be intact)?

Thanks

How do I use either/or logic inside an OSX Automator action?

JPowers — Sun, 19 Oct 2008 23:03:04 -0800

How do I use either/or logic inside an OSX Automator action? This seems SO bloody simple, but I can't for the life of me figure it out.

So I've got an Automator workflow that finds all of the files with an extension of .avi inside a certain folder, then does something with them. This works great, but what I REALLY want is for the workflow to search for all files in the folder that have an .avi *or* a .wmv extension. Is this possible?

I've tried "avi OR wmv", "avi {or} wmv", "avi "or" wmv" and anything else you could possible imagine.

The specific action I want to be able to put the either/or operator into is the "Filter Finder Items" action. This is a default Automator action that comes with OSX.

*beating head against desk*

Looking for a way automate making custom resumes

Pontifex — Mon, 15 Sep 2008 19:15:15 -0800

Where can I find a service / program that would enable me to dynamically create resumes and cover letters? I use a method for fabricating cover letters and resumes.

I tailor my documents to match the jobs I'm applying for.

My method involves annotating my work history with phrases describing my work, my personal projects, my school experience, etc. I've stored common annotations in a large spreadsheet in order to reuse some of my previous work.

Everything relevant to the job is represented. This is the most successful method I've tried and I'd prefer to keep using it.

This process takes quite a long time to produce viable documents.

I'd like to automate the process.

I'm in the process of doing my own research and developing a scripting solution as an alpha method for automating the process. I want to know if this has been done before; To save me some time and trouble.

Thanks for your input.

How do I make simple dynamic web pages using perl?

zzztimbo — Fri, 22 Aug 2008 09:24:58 -0800

How should I develop simple dynamic web pages using perl? Does anyone have any suggestions on how best to produce simple web pages in perl that execute logic (e.g., opening property documents to retrieve property values and saving back changes). I am new to perl and would like some detail instructions or links to tutorials. Would a framework like Mason be overkill?

Run multiple php websites from a single server?

zingzangzung — Wed, 16 Jul 2008 11:04:41 -0800

How practical is it to run multiple php websites on multiple servers from a single script on one central server? I wrote a script which is now being used on a few websites, and it has been suggested that to make updates simpler we run it from a central server ie each website will use the script from one source to actually run. While on the surface this seems like A Very Good Ideas, I am sure there are issues which might make it a very bad idea. Can you convince me one way or the other.

"And all the bells and whistles."

Bakuun — Wed, 09 Jul 2008 23:03:11 -0800

I'm looking for a way to let members of my forum create an individual "member page" for themselves, or have it created for them automatically upon being accepted into a certain usergroup, that they could fill out with miscellaneous info. To be more precise, the forum revolves an online game, and the member page would have details about their characters, background stories, etc. It doesn't really have to be tied to their forum account, although it would be a smooth way to handle it from the user standpoint. Except I'm clueless on how to make this work. The board is hosted on space I own, under a domain I own. So theoretically anything is possible in the technical department, minus my lack of knowledge. There are so many forum softwares out there, and gallery softwares, but I don't even know where to begin to look for something that offers the functionality I want. Is this a feature I'll have to script in myself (difficulty level: probably high, but if that's the only way, point me in the right direction!), or is there another way for the php uninitiated?

Extra details: I'm using Simple Machine Forums, and ideally I'd like the pages created once someone is of the right usergroup (to differentiate between game members and visitors). Though as I said, being attached as part of the board isn't a real necessity.

All help mightily appreciated. If extra details are needed for you to answer, just poke!

Customizing a Wordpress function

adamrice — Mon, 30 Jun 2008 06:36:19 -0800

Wordpress/PHP question. I'm working on a Wordpress theme and want to use an algorithm to alter some of the HTML it emits. I've got a general idea of how to do this, and could probably manage a brutish implementation on my own, but my PHP-fu is weak and there is probably a better way. Here's what's going on. I'm using the blueprint css approach for a gridded layout. And I've got a "bottom-bar" of widgets, laid out in three columns. With blueprint css, the div for the third column needs class="last" added to it in (in addition to class declarations that apply to all divs) order for everything to line up right.

What I want to do is get the count of widgets each time a widget is emitted, and if count mod 3 = 0, insert 'last' into the class. Barring that, if I could insert class="widget-N" (where N=count) in all divs, I could manage, although it would be less elegant. Ideally, all browsers would magically support CSS3 and obviate this problem, but I'm not holding my breath.

Obviously I don't want to touch the core code. I don't mind inserting an altered version of whatever function is necessary into my functions.php file; but if there's a callback that lets me avoid doing that, so much the better.

Scripter needs to learn a "real" language.

wongcorgi — Wed, 11 Jun 2008 16:37:46 -0800

Help a Perl/Python guy learn C# I want to learn C#/ASP.NET. The problem is that I'm a scripter by heart (Perl/Python/PHP/Javascript). I learn by coming up with an idea for a program and figuring out how to do it (via docs, reference). I like the O'reilly "Cookbook" series.

Every time I read a C#/ASP.NET book, I look at the examples and think, "I could write this faster and simpler in Perl". Does anyone have ideas/resources/examples/books/projects that would be good for C# but bad in a scripting language? I need some motivation/enlightenment!

I've been trying to learn C# on and off for the past year or so, help!

Maya MEL scripting

comatose — Thu, 22 May 2008 10:07:41 -0800

How long will it take me to learn MEL scripting in Maya for the sole purpose of creating and manipulating complex geometries? Yes there are similar topics on Maya here, but I just need to be very comfortable creating and manipulating complex geometries for the sake of architecture or objects resembling architecture. I do not need to be a master in Maya by any means. This is for an upcoming masters thesis in architecture, MARCH. I would like to accomplish something along the lines of this (think only mastering 1 of the 5 approaches):

http://www.mh-portfolio.com/indexHa.html

I have a light background using Maya but have absolutely no idea how to script. Is 4 months time enough for me to learn how to script in Maya for meeting my goals? How would you recommend going about this?

I have been motivated by this book:

Algorithmic Architecture by Kostas Terzidis

Shell scripting or something better?

Blazecock Pileon — Fri, 09 May 2008 14:19:40 -0800

Faster techniques for grabbing lines of a file and doing text manipulation and arithmetic via shell (or other) scripting? 1. Are there faster ways to grab lines of a text file?

Currently I use while read line and sed -n #p filename in bash and csh scripts to grab lines of a file I'm interested in. This seems slow. Are there better (faster) ways to get the line of a file, or to iterate through specified ranges of lines in a file?

2. Are there faster ways than awk to grab values in a line?

Let's say I have the tab-delimited line:

abc 123 345 0.52

What I'd like to do is get the second and third values, or the fourth value, as quickly as possible. Is there a better way than awk? Will a perl or other interpreted language script run faster than a shell script for scraping values from a text file?

3. Arithmetic with bash?

I've been doing $((${value1}+${value2})) for integer arithmetic and calc ${value1} / ${value2} for floating point arithmetic within bash. Will I gain a performance benefit from switching over my code from bash to another shell script language, or to another interpreted language entirely?

Thanks for any and all tips and tricks.

My website's scripts eat up too much bandwidth. How do I resolve this?

goodnewsfortheinsane — Sat, 05 Apr 2008 09:21:43 -0800

My site's cron tasks eat up a lot of bandwidth. Can I change the paths from http to local? Or can I move the tasks to my home server? One of my websites (the election one) runs a truckload of PHP to parse, mesh, filter, and cache an equally large number of external RSS feeds which are then stored and included in a reasonably lean (~10k) index.html. This creates a strange situation where the bandwidth consumed by the PHP (locally, but via http) far exceeds the actual bandwidth consumed by visitors. My web hosting provider has warned me about the excessive activity, but given the limited number of visitors, upgrading to a hosting deal with a higher bandwidth limit isn't really worthwhile for me.

How do I resolve this?

Current situation: a bunch of FeedForAll PHP scripts are executed using a Ruby-as-CGI script (which wackybrit kindly helped me with) called via cron at regular intervals, varying by nature of content: none-time critical content would be updated hourly, whereas latest news etc. is updated every 10 minutes (was 2 minutes, but I took it down a notch for now to avoid further problems with my host). My monthly bandwidth limit is 8 GB; actual visitor usage is a modest fraction of that, but the scripting alone was responsible for 37 GB (!) in March.

Several options:

1) The bandwidth is measured by the host using HTTP responses. If I run the scripts using local UNIX paths, e.g. /usr/gnfti/www/ etc., the scripts generate no HTTP headers and thus don't count towards the quota. Of course they will still incur server load, but I have discussed this with the hosting company and it seems they're okay with it if I limit the use somewhat as a compromise. But then, not all of these scripts seem to be okay with local paths, most notably the keyword filter (rssFilter.php), which is responsible for 25% of the bandwidth on its own.

Any ideas how to make it work this way anyway?

2) I have a home server running WAMP on XP. Could I run the PHP on this box instead and have the output automatically upload to the remote server via FTP? Most of the bandwidth is to do with moving stuff around locally anyway; the actual output is in the order of kilobytes. I imagine this would involve running some sort of cron + FTP client/scheduler on the home box, but my knowledge in this area is limited.

Could this be done, and if so, how?

3) Lastly, if you have any suggestions on how to pull this off aside from the examples above, that too would be much appreciated.

Thanks in advance for any suggestions or insight you might have to offer, guys.

How should I convert a Thunderbird inbox into text files?

eisenkr — Fri, 14 Mar 2008 11:57:48 -0800

What's the best way to export all of the emails in a Thunderbird inbox to text files, one text file per email? I'm planning on doing some linguistic analysis of emails and need to take a Thunderbird inbox and convert it into text files before I get to work. Google searches have revealed many mbox (which I think Thunderbird uses) parsers of, I am sure, varying quality. I'd love to skip the endless trial and error stage and start with the perfect tool. What's the best way to convert a Thunderbird inbox to text files?

ps - I'm comfortable using Perl and Python if you guys think a scripting solution is best.