Mac app to slice images and re-assemble them randomly?
December 27, 2017 10:30 PM Subscribe
I am wondering if there is a Mac app (or automation) that could take a folder of images, cut the images into equally-wide strips, and then reassemble the strips next to each other randomly to make new collage-like images.
I have thousands of images that I would like to automatically cut-up and mix-up randomly.
I know in the past that web-ready graphic apps would allow you to cut images into slices, but I'm hoping there’s something (either an app, a workflow, an automation, online web app, whatever) that would do everything—the slicing, the mixing, the reassembling, and the saving as new files.
This is kind of what I’d like the end result to look like.
Any suggestions or ideas?
I have thousands of images that I would like to automatically cut-up and mix-up randomly.
I know in the past that web-ready graphic apps would allow you to cut images into slices, but I'm hoping there’s something (either an app, a workflow, an automation, online web app, whatever) that would do everything—the slicing, the mixing, the reassembling, and the saving as new files.
This is kind of what I’d like the end result to look like.
Any suggestions or ideas?
If you're working with thousands of images, the speediest tool available that can be pressed into service for this would probably have to be the complex video filter subsystem of the command-driven video processor ffmpeg, which can treat large collections of images as video streams and do moderately complicated things to them at very high speeds.
Unfortunately the ffmpeg command line syntax, though comprehensive and capable, is also massively intimidating and very fiddly; the documentation also makes a much better reference than a tutorial. So before diving in and bending it to our will, it would be helpful to know a few things.
How many output images do you envisage creating with each application of your desired tool? One? Tens? Thousands?
Roughly how many slices should be composed into a single output image?
Should the slices always be vertical, as in your linked example?
Are the slices cut from any given original source required to be tiles? That is, could they in principle be reassembled to re-form the original image, or can they just be taken from random pixel offsets like strips slashed at random from a magazine? If the latter, are they allowed to overlap?
Should all the slices be the same width?
posted by flabdablet at 9:39 AM on December 28, 2017
Unfortunately the ffmpeg command line syntax, though comprehensive and capable, is also massively intimidating and very fiddly; the documentation also makes a much better reference than a tutorial. So before diving in and bending it to our will, it would be helpful to know a few things.
How many output images do you envisage creating with each application of your desired tool? One? Tens? Thousands?
Roughly how many slices should be composed into a single output image?
Should the slices always be vertical, as in your linked example?
Are the slices cut from any given original source required to be tiles? That is, could they in principle be reassembled to re-form the original image, or can they just be taken from random pixel offsets like strips slashed at random from a magazine? If the latter, are they allowed to overlap?
Should all the slices be the same width?
posted by flabdablet at 9:39 AM on December 28, 2017
Will the output images all be the same size?
Are the input images all the same size as each other?
If yes to both, do the output images have to be the same size as the inputs?
posted by flabdablet at 9:41 AM on December 28, 2017
Are the input images all the same size as each other?
If yes to both, do the output images have to be the same size as the inputs?
posted by flabdablet at 9:41 AM on December 28, 2017
Response by poster: How many output images do you envisage creating with each application of your desired tool? One? Tens? Thousands?
Probably thousands—the idea would be to turn on the app and have it chug away, making randomly-assembled-image after randomly-assembled-image.
Depending on how long the process took, I could see starting the process and then letting my Mac chug away for a day or two, creating thousands (or tens of thousands of "collages").
I would then sort through the folder filled with the resulting assembled collage images, looking for particularly nice looking collages.
Serendipitous art arrived at through planned chaos.
Roughly how many slices should be composed into a single output image?
I was thinking that the slices would be vertical, with about a 1:15 to 1:20 ratio width-to-height.
My dSLR takes photos that are 4000×6000 pixels, so these photos would be sliced into 30 distinct vertical slices, with each slice being 4000 pixels tall and 200 pixels wide.
Ideally, (like if this were an app) it would be nice if you could introduce some minor variation in the width of the strips—
I know I refered to pixel width, but ideally, it would be better if one could specify the size of the strips by ratio or "number of strips to slice the image into" so that if I wanted to do the same with a non-standard sized image (say, a panorama), it wouldn’t be a bear to configure.
Should the slices always be vertical, as in your linked example?
My main interest right now is vertical slices, but if I could get horizontal (or even diagonal!) slices that would be cool too.
Are the slices cut from any given original source required to be tiles? That is, could they in principle be reassembled to re-form the original image, or can they just be taken from random pixel offsets like strips slashed at random from a magazine? If the latter, are they allowed to overlap?
I’m not sure I totally understand this question... they would be "tiles" (although long rectangular "tiles" like a Kit-Kat bar, not squares). The digital slices would be just like if you sliced a photo into 30 strips (or ran it through an old-school non-cross-cutting paper shredder)—each strip would be distinct, sharing no identical pixels.
Mathematically, they could wind up reassembling the original image, but I think it would be like a 1-in-900 chance (30 strips times thirty spaces, if I remember my statistics correctly). And this would be only if I ran one image through the process...
What I would want to do—again, ideally—is throw the app/workflow at a folder full of (scores/hundreds/thousands of) images, have it cut all of the images into a big grab-bag of these slices, then re-assemble randomly picked slices into new images made up of 30 non-overlapping slices. So, no overlapping of slices, and no space(s) left uncovered.
Also, I think it would be nice to be able to say
Should all the slices be the same width?
Well, the same relative width—for a 4×6 landscape formated image, about 30 slices.
Again, it would be cool to give this option some "play" for variation, but it's not necessary.
Will the output images all be the same size?
I imagine that they would remain the size of the original—like, if the process cut up a bunch of 4000×6000 pixel images, that the resulting collage image would also be 4000×6000 pixels.
Are the input images all the same size as each other?
They probably would be, for simplicity's sake, although it would be cool if I could tell the app/process
If yes to both, do the output images have to be the same size as the inputs?
They don’t have to be the same size, like I don't care if they get enlarged to say 8000×12000, but they should be the same basic height-to-width ratio
posted by blueberry at 7:03 PM on December 28, 2017
Probably thousands—the idea would be to turn on the app and have it chug away, making randomly-assembled-image after randomly-assembled-image.
Depending on how long the process took, I could see starting the process and then letting my Mac chug away for a day or two, creating thousands (or tens of thousands of "collages").
I would then sort through the folder filled with the resulting assembled collage images, looking for particularly nice looking collages.
Serendipitous art arrived at through planned chaos.
Roughly how many slices should be composed into a single output image?
I was thinking that the slices would be vertical, with about a 1:15 to 1:20 ratio width-to-height.
My dSLR takes photos that are 4000×6000 pixels, so these photos would be sliced into 30 distinct vertical slices, with each slice being 4000 pixels tall and 200 pixels wide.
Ideally, (like if this were an app) it would be nice if you could introduce some minor variation in the width of the strips—
(such as "Allow up to 20% of the slices made to vary in ratio from as skinny as 1:25 to as wide as 1:15")—but I don't know if that would be possible in a workflow situation.I know I refered to pixel width, but ideally, it would be better if one could specify the size of the strips by ratio or "number of strips to slice the image into" so that if I wanted to do the same with a non-standard sized image (say, a panorama), it wouldn’t be a bear to configure.
Should the slices always be vertical, as in your linked example?
My main interest right now is vertical slices, but if I could get horizontal (or even diagonal!) slices that would be cool too.
Are the slices cut from any given original source required to be tiles? That is, could they in principle be reassembled to re-form the original image, or can they just be taken from random pixel offsets like strips slashed at random from a magazine? If the latter, are they allowed to overlap?
I’m not sure I totally understand this question... they would be "tiles" (although long rectangular "tiles" like a Kit-Kat bar, not squares). The digital slices would be just like if you sliced a photo into 30 strips (or ran it through an old-school non-cross-cutting paper shredder)—each strip would be distinct, sharing no identical pixels.
Mathematically, they could wind up reassembling the original image, but I think it would be like a 1-in-900 chance (30 strips times thirty spaces, if I remember my statistics correctly). And this would be only if I ran one image through the process...
What I would want to do—again, ideally—is throw the app/workflow at a folder full of (scores/hundreds/thousands of) images, have it cut all of the images into a big grab-bag of these slices, then re-assemble randomly picked slices into new images made up of 30 non-overlapping slices. So, no overlapping of slices, and no space(s) left uncovered.
Also, I think it would be nice to be able to say
"If there are at least 30 images that got sliced into strips, when assembling the collages, don’t take more that one slice from each original image."FredFlintstone.jpg would be cut into strips then named FredFlintstone-strip-01.jpg, FredFlintstone-strip-02.jpg, etcetera.Should all the slices be the same width?
Well, the same relative width—for a 4×6 landscape formated image, about 30 slices.
Again, it would be cool to give this option some "play" for variation, but it's not necessary.
Will the output images all be the same size?
I imagine that they would remain the size of the original—like, if the process cut up a bunch of 4000×6000 pixel images, that the resulting collage image would also be 4000×6000 pixels.
Are the input images all the same size as each other?
They probably would be, for simplicity's sake, although it would be cool if I could tell the app/process
"Listen, I want any/all collages made this time to be 4000×6000 pixels, so if you happen to run across an image that is cropped to 5400×3600, up-size it to 4000×6000 before you slice it so that everything fits together in the end".Otherwise, I would just need to go and re-size any cropped images to all be the same size before running the task(s).
If yes to both, do the output images have to be the same size as the inputs?
They don’t have to be the same size, like I don't care if they get enlarged to say 8000×12000, but they should be the same basic height-to-width ratio
posted by blueberry at 7:03 PM on December 28, 2017
This is all very helpful clarification. And yes, the answers you gave to the tiling question do address the issue I had in my mind when I asked them.
One last thing: if we are indeed going to work with strips whose widths are allowed to vary somewhat, then there will be a good chance that assembling 30 of these at random into a final tiling will not yield an image with the same aspect ratio as the originals. On average it should be fairly close though. Would you prefer to deal with that issue by
(a) ignoring it - the output image is allowed to have whatever aspect ratio is forced by the stripes chosen to build it
(b) scaling the finished image to a standard aspect ratio
(c) restricting the choice of component stripes for any given output image to sets of stripes that do tile a rectangle the same size as a standard input image
(d) other?
In any case, I already have enough to have a crack at implementing the first part of workflow you've proposed: running a huge batch of photos through a virtual shredder to make a pile of stripe images. Let me play with that for a while and get back to you.
I'll be doing this work on a Linux box, not a Mac. The underlying tools are all available for both so this will eventually work on your system as well, though there may be a little shoehorning required to get it to work smoothly.
posted by flabdablet at 8:37 PM on December 28, 2017
One last thing: if we are indeed going to work with strips whose widths are allowed to vary somewhat, then there will be a good chance that assembling 30 of these at random into a final tiling will not yield an image with the same aspect ratio as the originals. On average it should be fairly close though. Would you prefer to deal with that issue by
(a) ignoring it - the output image is allowed to have whatever aspect ratio is forced by the stripes chosen to build it
(b) scaling the finished image to a standard aspect ratio
(c) restricting the choice of component stripes for any given output image to sets of stripes that do tile a rectangle the same size as a standard input image
(d) other?
In any case, I already have enough to have a crack at implementing the first part of workflow you've proposed: running a huge batch of photos through a virtual shredder to make a pile of stripe images. Let me play with that for a while and get back to you.
I'll be doing this work on a Linux box, not a Mac. The underlying tools are all available for both so this will eventually work on your system as well, though there may be a little shoehorning required to get it to work smoothly.
posted by flabdablet at 8:37 PM on December 28, 2017
I have some initial test results now.
My test image was a 3072x2304 pixel JPEG, and I'm scaling it to 6000x4000 and then cutting it into 30 strips of 200x4000 pixels each. The strips get saved in the lossless PNG format to avoid accumulating further JPEG artifacts.
The first tool I tried was convert, from the ImageMagick suite. I based the command I used on this recipe:
That's going to be really irritatingly slow if you intend to process thousands of these things. So I gritted my teeth and turned to ffmpeg, using a recipe based on my best current understanding of how to make it do what I want.
There is plenty of ffmpeg documentation spread around the Web, but the tool's capabilities are so vast that working out the best way to make it do anything is a bit of a haystack needle process. Anyway, here's the best I could do:
That command line is so ridiculous that the best option would be to generate it programatically rather than having it embedded in a script as-is. This would also make for good control over number of shreds and/or variable shred widths.
But it's still slower than I'd like. How committed are you to the idea of reducing all your source images to a vast pile of nicely named shreds as a first pass? If you're going to be selecting your final results by eyeballing candidates one at a time, then it seems to me that a better approach might be to include shred generation in the collage building step, generating only as many shreds as are needed to compose each output image and never actually saving the individual shreds as files. I would expect to be able to generate output images at rates roughly commensurate with those already achieved for shredding inputs.
posted by flabdablet at 1:10 AM on December 29, 2017
My test image was a 3072x2304 pixel JPEG, and I'm scaling it to 6000x4000 and then cutting it into 30 strips of 200x4000 pixels each. The strips get saved in the lossless PNG format to avoid accumulating further JPEG artifacts.
The first tool I tried was convert, from the ImageMagick suite. I based the command I used on this recipe:
time convert IMG_0981.JPG +gravity -scale '6000x4000!' -crop 200x4000 shreds/IMG_0981-%02d.pngOn my dear old 2009-vintage Core 2 Duo E8500 based computer, this took 14 seconds to scale and cut up the test image.
That's going to be really irritatingly slow if you intend to process thousands of these things. So I gritted my teeth and turned to ffmpeg, using a recipe based on my best current understanding of how to make it do what I want.
There is plenty of ffmpeg documentation spread around the Web, but the tool's capabilities are so vast that working out the best way to make it do anything is a bit of a haystack needle process. Anyway, here's the best I could do:
time ffmpeg -loglevel error -f image2 -i IMG_0981.JPG -filter_complex ' [0:v]scale=6000:4000,split=30[s0][s1][s2][s3][s4][s5][s6][s7][s8][s9][s10][s11][s12][s13][s14][s15][s16][s17][s18][s19][s20][s21][s22][s23][s24][s25][s26][s27][s28][s29]; [s0]crop=200:4000:0:0[c0]; [s1]crop=200:4000:200:0[c1]; [s2]crop=200:4000:400:0[c2]; [s3]crop=200:4000:600:0[c3]; [s4]crop=200:4000:800:0[c4]; [s5]crop=200:4000:1000:0[c5]; [s6]crop=200:4000:1200:0[c6]; [s7]crop=200:4000:1400:0[c7]; [s8]crop=200:4000:1600:0[c8]; [s9]crop=200:4000:1800:0[c9]; [s10]crop=200:4000:2000:0[c10]; [s11]crop=200:4000:2200:0[c11]; [s12]crop=200:4000:2400:0[c12]; [s13]crop=200:4000:2600:0[c13]; [s14]crop=200:4000:2800:0[c14]; [s15]crop=200:4000:3000:0[c15]; [s16]crop=200:4000:3200:0[c16]; [s17]crop=200:4000:3400:0[c17]; [s18]crop=200:4000:3600:0[c18]; [s19]crop=200:4000:3800:0[c19]; [s20]crop=200:4000:4000:0[c20]; [s21]crop=200:4000:4200:0[c21]; [s22]crop=200:4000:4400:0[c22]; [s23]crop=200:4000:4600:0[c23]; [s24]crop=200:4000:4800:0[c24]; [s25]crop=200:4000:5000:0[c25]; [s26]crop=200:4000:5200:0[c26]; [s27]crop=200:4000:5400:0[c27]; [s28]crop=200:4000:5600:0[c28]; [s29]crop=200:4000:5800:0[c29]' \ -map [c0] shreds/test-00.png \ -map [c1] shreds/test-01.png \ -map [c2] shreds/test-02.png \ -map [c3] shreds/test-03.png \ -map [c4] shreds/test-04.png \ -map [c5] shreds/test-05.png \ -map [c6] shreds/test-06.png \ -map [c7] shreds/test-07.png \ -map [c8] shreds/test-08.png \ -map [c9] shreds/test-09.png \ -map [c10] shreds/test-10.png \ -map [c11] shreds/test-11.png \ -map [c12] shreds/test-12.png \ -map [c13] shreds/test-13.png \ -map [c14] shreds/test-14.png \ -map [c15] shreds/test-15.png \ -map [c16] shreds/test-16.png \ -map [c17] shreds/test-17.png \ -map [c18] shreds/test-18.png \ -map [c19] shreds/test-19.png \ -map [c20] shreds/test-20.png \ -map [c21] shreds/test-21.png \ -map [c22] shreds/test-22.png \ -map [c23] shreds/test-23.png \ -map [c24] shreds/test-24.png \ -map [c25] shreds/test-25.png \ -map [c26] shreds/test-26.png \ -map [c27] shreds/test-27.png \ -map [c28] shreds/test-28.png \ -map [c29] shreds/test-29.pngClearly that's a ridiculous command line, but it does the same job as the convert one in just 2.5 seconds.
That command line is so ridiculous that the best option would be to generate it programatically rather than having it embedded in a script as-is. This would also make for good control over number of shreds and/or variable shred widths.
But it's still slower than I'd like. How committed are you to the idea of reducing all your source images to a vast pile of nicely named shreds as a first pass? If you're going to be selecting your final results by eyeballing candidates one at a time, then it seems to me that a better approach might be to include shred generation in the collage building step, generating only as many shreds as are needed to compose each output image and never actually saving the individual shreds as files. I would expect to be able to generate output images at rates roughly commensurate with those already achieved for shredding inputs.
posted by flabdablet at 1:10 AM on December 29, 2017
Response by poster: This is fantastic, flabdablet—thank you for your interest and your help on this!
One last thing: if we are indeed going to work with strips whose widths are allowed to vary somewhat, then there will be a good chance that assembling 30 of these at random into a final tiling will not yield an image with the same aspect ratio as the originals. On average it should be fairly close though. Would you prefer to deal with that issue by
(a) ignoring it - the output image is allowed to have whatever aspect ratio is forced by the stripes chosen to build it
Yeah, I thought about that after posting. I would choose "(a)".
I think that it would be close enough that the change in aspect ratio wouldn't be a problem. I guess, ideally, the process would know that if it had a couple of wider slices, that it should kind of back-fill somewhere else in the image with an extra thin slice or two (which would then knock the number of slices over 30).
But again, since the idea was for just a small amount of the slices to vary in size, that the collages would mostly work out to the same aspect ratio (and could always be trimmed off of the top and/or bottom if one really wanted to regain that 4×6 aspect ratio)
How committed are you to the idea of reducing all your source images to a vast pile of nicely named shreds as a first pass?
Hmm, I guess my thinking about the
(printing out each slice separately and then re-creating the collage physically. Pasting it all together on a backing, maybe playing a little with the negative space—spacing the strips out, nudging some strips higher, some lower—so that while the same basic pattern and dance of colors was there, that they played across the backing (say a nice sanded plywood) rising and falling like musical notes or like the lines of a crosswalk if it described the arc of a meandering creek...)
However, for that to be done, not only would the slices have to be saved as actual files and named, but there would also have to be a rather intensely verbose way that the component parts of each collage were logged someplace—like a long-ass text file that said
One last thing: if we are indeed going to work with strips whose widths are allowed to vary somewhat, then there will be a good chance that assembling 30 of these at random into a final tiling will not yield an image with the same aspect ratio as the originals. On average it should be fairly close though. Would you prefer to deal with that issue by
(a) ignoring it - the output image is allowed to have whatever aspect ratio is forced by the stripes chosen to build it
Yeah, I thought about that after posting. I would choose "(a)".
I think that it would be close enough that the change in aspect ratio wouldn't be a problem. I guess, ideally, the process would know that if it had a couple of wider slices, that it should kind of back-fill somewhere else in the image with an extra thin slice or two (which would then knock the number of slices over 30).
But again, since the idea was for just a small amount of the slices to vary in size, that the collages would mostly work out to the same aspect ratio (and could always be trimmed off of the top and/or bottom if one really wanted to regain that 4×6 aspect ratio)
How committed are you to the idea of reducing all your source images to a vast pile of nicely named shreds as a first pass?
Hmm, I guess my thinking about the
"slice all the images and save all of the newly-named slices in a big pot"(printing out each slice separately and then re-creating the collage physically. Pasting it all together on a backing, maybe playing a little with the negative space—spacing the strips out, nudging some strips higher, some lower—so that while the same basic pattern and dance of colors was there, that they played across the backing (say a nice sanded plywood) rising and falling like musical notes or like the lines of a crosswalk if it described the arc of a meandering creek...)
However, for that to be done, not only would the slices have to be saved as actual files and named, but there would also have to be a rather intensely verbose way that the component parts of each collage were logged someplace—like a long-ass text file that said
Collage number 10348 was created using the following 30 strips from left to right: Wilma-strip-19, Dino-strip-04, Slate-strip-26, Bedrock-strip-09, Pebbles-strip-21, ...")"Let's see, for the next piece I need to open up the Dino file and grab a slice from the fourth segment area..."posted by blueberry at 1:59 AM on December 29, 2017 [1 favorite]
Ideally, (like if this were an app) it would be nice if you could introduce some minor variation in the width of the strips—
(such as "Allow up to 20% of the slices made to vary in ratio from as skinny as 1:25 to as wide as 1:15")
—but I don't know if that would be possible in a workflow situation.
Anything is possible, given a sufficiently convoluted script. The only tricky part is working out exactly what the script is intended to to.
For example: if I'm cutting a photo into 30 strips, and the width of 20% of those is intended to be as much as 25% away from their mean width, how do I deal with the fact that the width of all the strips is still constrained to add up to the width of the original photo?
Am I allowed to end up with some number of strips other than 30?
Am I allowed to end up with a strip cut from an edge that's more than 25% different from the mean width?
For the 20% that have the inaccurate widths, how is the inaccuracy distributed? That is, within that 20%, do we want to be more likely to generate widths that are nearly accurate, or do we want a uniformly random distribution of widths between mean×0.75 and mean×1.25 for those?
The answers to these questions will determine whether the script can just work across a photo from left to right cutting off slices until it's done, or whether it needs to work by slicing the photo into possibly somewhat inaccurate halves, then the halves into somewhat inaccurate thirds, then the sixths into somewhat inaccurate fifths.
Or we could just abandon the initial cutting-into-slices model entirely, and instead compose the final collage from strips of some suitable width sliced at completely random horizontal offsets from one source image per output stripe. Visually I think you'd be hard-pressed to spot the difference.
posted by flabdablet at 2:06 AM on December 29, 2017
(such as "Allow up to 20% of the slices made to vary in ratio from as skinny as 1:25 to as wide as 1:15")
—but I don't know if that would be possible in a workflow situation.
Anything is possible, given a sufficiently convoluted script. The only tricky part is working out exactly what the script is intended to to.
For example: if I'm cutting a photo into 30 strips, and the width of 20% of those is intended to be as much as 25% away from their mean width, how do I deal with the fact that the width of all the strips is still constrained to add up to the width of the original photo?
Am I allowed to end up with some number of strips other than 30?
Am I allowed to end up with a strip cut from an edge that's more than 25% different from the mean width?
For the 20% that have the inaccurate widths, how is the inaccuracy distributed? That is, within that 20%, do we want to be more likely to generate widths that are nearly accurate, or do we want a uniformly random distribution of widths between mean×0.75 and mean×1.25 for those?
The answers to these questions will determine whether the script can just work across a photo from left to right cutting off slices until it's done, or whether it needs to work by slicing the photo into possibly somewhat inaccurate halves, then the halves into somewhat inaccurate thirds, then the sixths into somewhat inaccurate fifths.
Or we could just abandon the initial cutting-into-slices model entirely, and instead compose the final collage from strips of some suitable width sliced at completely random horizontal offsets from one source image per output stripe. Visually I think you'd be hard-pressed to spot the difference.
posted by flabdablet at 2:06 AM on December 29, 2017
But again, since the idea was for just a small amount of the slices to vary in size, that the collages would mostly work out to the same aspect ratio (and could always be trimmed off of the top and/or bottom if one really wanted to regain that 4×6 aspect ratio)
Also very very easy just to scale a final image back to an exact 4x6. If its initial aspect ratio was within a few percent of 1.5 anyway, you'd never spot the resulting distortions.
posted by flabdablet at 2:09 AM on December 29, 2017
Also very very easy just to scale a final image back to an exact 4x6. If its initial aspect ratio was within a few percent of 1.5 anyway, you'd never spot the resulting distortions.
posted by flabdablet at 2:09 AM on December 29, 2017
Also, if we're looking at a log file for possible future printed strip construction, I'd recommend logging strip width and offset in pixels rather than as any kind of strip segment number. That way, we get to play with assorted kinds of strip generation method without needing to alter the log file format.
posted by flabdablet at 2:14 AM on December 29, 2017
posted by flabdablet at 2:14 AM on December 29, 2017
Response by poster: For example: if I'm cutting a photo into 30 strips, and the width of 20% of those is intended to be as much as 25% away from their mean width, how do I deal with the fact that the width of all the strips is still constrained to add up to the width of the original photo?
If I'm understanding this, then any "extra" width would just be cut off of the final collage. Like, strip 30 may indeed be 200 pixels wide, but since there was a wider-than-normal slice used in the collage, there's only space enough for the left-most 178 pixels of slice 30 to stay on the collage.
Am I allowed to end up with some number of strips other than 30?
Yeah, I just put the "30 slices per 4×6 ratio" as more of a guideline—like I thought the strips were wide enough to show color and give a hint of subject matter, but not enough to paint the source image's whole subject.
If there are, say a bunch of skinny strips, yeah, just keep filling up the image with strips until the 4×6 area is covered. (although if only x% of the the images vary in size, the number of thiner strips seems like it would still be pretty low—although I guess there is the chance that every fifth strip could be thinner (if we were using 20% for that run of collages).
Am I allowed to end up with a strip cut from an edge that's more than 25% different from the mean width?
I think I answered this above...? Like, "if the strip image(s) run off of the table is it okay to cut off the overhang?" Yes.
For the 20% that have the inaccurate widths, how is the inaccuracy distributed? That is, within that 20%, do we want to be more likely to generate widths that are nearly accurate, or do we want a uniformly random distribution of widths between mean×0.75 and mean×1.25 for those?
If I'm understanding, I think I would want everything to be random. The 20% of the strips would be placed in the left-to-right collage sequence at random, and the 20% that varied in size would randomly be between the two width extremes. Did that answer that, or did I misinterpret the question?
The answers to these questions will determine whether the script can just work across a photo from left to right cutting off slices until it's done, or whether it needs to work by slicing the photo into possibly somewhat inaccurate halves, then the halves into somewhat inaccurate thirds, then the sixths into somewhat inaccurate fifths.
Hmm, maybe I'm misunderstanding... my thinking is that
—like a brick-layer working from left to right—
you hand the bricklayer a brick and she places it in SPACE #01,
then you hand her another brick, but
—lucky day, this one is a special brick that is only 80% of the usual width!
She places this skinnier brick in SPACE #02, then you hand her the next brick, and so on...
Well, at the end of the bricks, it seems that 30 bricks doesn't quite cover the alotted space so we add a SPACE #31 and then add a 31st brick. But, wouldn't'cha know it—that 31st brick actually goes over the alotted space available, so... we just uncerimoniously chop of the right-most part of the 31st brick that doesn't fit.
Or we could just abandon the initial cutting-into-slices model entirely, and instead compose the final collage from strips of some suitable width sliced at completely random horizontal offsets from one source image per output stripe. Visually I think you'd be hard-pressed to spot the difference.
That would work too, but I guess I liked the idea of being about to re-produce the image if I ever wanted to, and if each strip was randomly picked, it would be damned near impossible to re-create—unless the location was logged somehow
If I'm understanding this, then any "extra" width would just be cut off of the final collage. Like, strip 30 may indeed be 200 pixels wide, but since there was a wider-than-normal slice used in the collage, there's only space enough for the left-most 178 pixels of slice 30 to stay on the collage.
Am I allowed to end up with some number of strips other than 30?
Yeah, I just put the "30 slices per 4×6 ratio" as more of a guideline—like I thought the strips were wide enough to show color and give a hint of subject matter, but not enough to paint the source image's whole subject.
If there are, say a bunch of skinny strips, yeah, just keep filling up the image with strips until the 4×6 area is covered. (although if only x% of the the images vary in size, the number of thiner strips seems like it would still be pretty low—although I guess there is the chance that every fifth strip could be thinner (if we were using 20% for that run of collages).
Am I allowed to end up with a strip cut from an edge that's more than 25% different from the mean width?
I think I answered this above...? Like, "if the strip image(s) run off of the table is it okay to cut off the overhang?" Yes.
For the 20% that have the inaccurate widths, how is the inaccuracy distributed? That is, within that 20%, do we want to be more likely to generate widths that are nearly accurate, or do we want a uniformly random distribution of widths between mean×0.75 and mean×1.25 for those?
If I'm understanding, I think I would want everything to be random. The 20% of the strips would be placed in the left-to-right collage sequence at random, and the 20% that varied in size would randomly be between the two width extremes. Did that answer that, or did I misinterpret the question?
The answers to these questions will determine whether the script can just work across a photo from left to right cutting off slices until it's done, or whether it needs to work by slicing the photo into possibly somewhat inaccurate halves, then the halves into somewhat inaccurate thirds, then the sixths into somewhat inaccurate fifths.
Hmm, maybe I'm misunderstanding... my thinking is that
—like a brick-layer working from left to right—
you hand the bricklayer a brick and she places it in SPACE #01,
then you hand her another brick, but
—lucky day, this one is a special brick that is only 80% of the usual width!
She places this skinnier brick in SPACE #02, then you hand her the next brick, and so on...
Well, at the end of the bricks, it seems that 30 bricks doesn't quite cover the alotted space so we add a SPACE #31 and then add a 31st brick. But, wouldn't'cha know it—that 31st brick actually goes over the alotted space available, so... we just uncerimoniously chop of the right-most part of the 31st brick that doesn't fit.
Or we could just abandon the initial cutting-into-slices model entirely, and instead compose the final collage from strips of some suitable width sliced at completely random horizontal offsets from one source image per output stripe. Visually I think you'd be hard-pressed to spot the difference.
That would work too, but I guess I liked the idea of being about to re-produce the image if I ever wanted to, and if each strip was randomly picked, it would be damned near impossible to re-create—unless the location was logged somehow
("Collage number 10348 was created using the following 30 strips from left to right: Wilma.png 1872 pixels to 2072 pixels, Dino.png 14 pixels to 178 pixels, ...")posted by blueberry at 2:39 AM on December 29, 2017 If I'm understanding this, then any "extra" width would just be cut off of the final collage.
I was really thinking about the process of cutting up the source photos to make stripes. Hadn't got as far as collage assembly.
That would work too, but I guess I liked the idea of being about to re-produce the image if I ever wanted to, and if each strip was randomly picked, it would be damned near impossible to re-create
Computer randomness is usually not real randomness. It usually works from a starting "seed" value, transforming that over and over and over using some mathematical formula designed to make deviations from "real" randomness (whatever that is) very difficult to detect.
To reproduce the results of a script with pseudo-randomness at its heart, all one needs to do is save the seed values at appropriate times.
unless the location was logged somehow
there would also have to be a rather intensely verbose way that the component parts of each collage were logged someplace—like a long-ass text file that said
Collage number 10348 was created using the following 30 strips from left to right: Wilma-strip-19, Dino-strip-04, Slate-strip-26, Bedrock-strip-09, Pebbles-strip-21, ...")
This kind of log file is a completely reasonable thing to generate, and making it reasonably easy for a future script to parse will save work. I'd recommend a simple format like
Putting the numbers first on the tile source lines makes it possible to use spaces to separate the numbers from each other and the names while still allowing for names that contain spaces. I'm a big fan of minimally punctuated task-specific formats like this because although they're still cleanly machine readable they're also quite eyeball friendly (unlike JSON or even worse XML).
A completely reasonable approach would be to have a pair of cooperating scripts: one of them generates just the log file, and the other reads through it and makes collages as it goes. That way you're absolutely guaranteed of being able to reproduce anything you've already seen, because you know for sure that it was actually made from a log file you already have.
posted by flabdablet at 2:51 AM on December 29, 2017
I was really thinking about the process of cutting up the source photos to make stripes. Hadn't got as far as collage assembly.
That would work too, but I guess I liked the idea of being about to re-produce the image if I ever wanted to, and if each strip was randomly picked, it would be damned near impossible to re-create
Computer randomness is usually not real randomness. It usually works from a starting "seed" value, transforming that over and over and over using some mathematical formula designed to make deviations from "real" randomness (whatever that is) very difficult to detect.
To reproduce the results of a script with pseudo-randomness at its heart, all one needs to do is save the seed values at appropriate times.
unless the location was logged somehow
there would also have to be a rather intensely verbose way that the component parts of each collage were logged someplace—like a long-ass text file that said
Collage number 10348 was created using the following 30 strips from left to right: Wilma-strip-19, Dino-strip-04, Slate-strip-26, Bedrock-strip-09, Pebbles-strip-21, ...")
This kind of log file is a completely reasonable thing to generate, and making it reasonably easy for a future script to parse will save work. I'd recommend a simple format like
[collage-10348.jpg] 200 4000 3800 0 Wilma 200 4000 800 0 Dino 200 4000 5200 0 Slate 200 4000 1800 0 Bedrock 200 4000 4200 0 Pebbles ...Any line that starts with [ introduces a collage, and its name can be extracted by stripping off the surrounding []. The lines that follow, up to but not including the next [ line and ignoring blank lines, are the crop size, crop offset and original image name for the tiles that make up the collage, in left-to-right then top-to-bottom order. In the present case there's only one row of tiles but the extra flexibility we get from allowing the log file to be completely explicit about them costs very little and might be handy at some point.
Putting the numbers first on the tile source lines makes it possible to use spaces to separate the numbers from each other and the names while still allowing for names that contain spaces. I'm a big fan of minimally punctuated task-specific formats like this because although they're still cleanly machine readable they're also quite eyeball friendly (unlike JSON or even worse XML).
A completely reasonable approach would be to have a pair of cooperating scripts: one of them generates just the log file, and the other reads through it and makes collages as it goes. That way you're absolutely guaranteed of being able to reproduce anything you've already seen, because you know for sure that it was actually made from a log file you already have.
posted by flabdablet at 2:51 AM on December 29, 2017
Another amusing possibility might be to generate a log file as above, then have a script that reads that and spits out HTML and CSS that directly references the appropriate source images, allowing you to preview collages in your web browser without ever actually constructing them at all.
posted by flabdablet at 2:55 AM on December 29, 2017
posted by flabdablet at 2:55 AM on December 29, 2017
Another advantage of doing spec generation before making any actual images is that if you find a collage you like that you think would improve with a tiny bit of tweaking - perhaps that nice pink stripe could be a fraction wider, or cut from a little further left in its source so it doesn't have that unintentional dick shape a third of the way down? - then you can pull its spec out of the huge file with a text editor, mess with its numbers a little, then save it in its own individual spec file to be run through the image generator again.
So here's a script that generates collage spec files in the format I proposed above.
I just ran this against a selection of folders containing 2622 images, and it completed in one minute and twelve seconds. When I piped its output into less instead of waiting for the whole thing to complete, the first collage spec became available in under five seconds.
Note that this is doing none of the actual image processing work, just making a honking great text file (5 megabytes and 83904 lines for me) containing cropping specifications for input images for 2622 collages. The numbers in those specs assume that the input images will be scaled to 6000x4000 before being cropped.
The number of collage specifications will always be the same as the number of input images. You can run the thing repeatedly to make more specs if you want. It uses the random number generator built into bash, which I have never been able to seed reliably; you will get different output every time.
The script works by listing the names of all available files, each repeated 30 times; shuffling the resulting massive list into a random ordering; then walking through the shuffled list, writing out extraction specs for one strip from each file encountered and grouping those into collages 30 at a time.
It makes no attempt at all to ensure that the resulting collages will be close to the right width, and it also makes no attempt to ensure that strips cut from any given input file will never contain common content or that at most one strip from any given input file will show up in any given collage. I figured if you're just going to be looking for nice results by eye, those constraints are probably not terribly important and this algorithm is nice and simple.
posted by flabdablet at 7:40 AM on December 29, 2017
So here's a script that generates collage spec files in the format I proposed above.
I just ran this against a selection of folders containing 2622 images, and it completed in one minute and twelve seconds. When I piped its output into less instead of waiting for the whole thing to complete, the first collage spec became available in under five seconds.
Note that this is doing none of the actual image processing work, just making a honking great text file (5 megabytes and 83904 lines for me) containing cropping specifications for input images for 2622 collages. The numbers in those specs assume that the input images will be scaled to 6000x4000 before being cropped.
The number of collage specifications will always be the same as the number of input images. You can run the thing repeatedly to make more specs if you want. It uses the random number generator built into bash, which I have never been able to seed reliably; you will get different output every time.
The script works by listing the names of all available files, each repeated 30 times; shuffling the resulting massive list into a random ordering; then walking through the shuffled list, writing out extraction specs for one strip from each file encountered and grouping those into collages 30 at a time.
It makes no attempt at all to ensure that the resulting collages will be close to the right width, and it also makes no attempt to ensure that strips cut from any given input file will never contain common content or that at most one strip from any given input file will show up in any given collage. I figured if you're just going to be looking for nice results by eye, those constraints are probably not terribly important and this algorithm is nice and simple.
#!/bin/bash width=6000 #standard width all input images should be scaled to height=4000 #standard height all input images should be scaled to strips=30 #number of strips to cut input images into (nominal) inaccurate=10 #percentage of strips to be inaccurately sliced inaccuracy=25 #maximum percentage strip width inaccuracy # Random distribution expressions returning a number in the # range -32767 to +32767: uncomment one of these dist='(RANDOM%2?-1:1)*RANDOM' #linear #dist='RANDOM-RANDOM' #double dice: larger inaccuracies are less common let rs=32768 #0 <= RANDOM < rs let wac=width/strips #width of an accurately cut strip let sir=inaccurate*rs/100 #scaled inaccurate slice rate let dmi=rs*100/inaccuracy/wac #divisor to scale dist range to max inaccuracy let sn=0 cn=0 #reset strip and collage numbers find "$@" -type f -iname '*.jpg' | #list image files from all specified folders while read -r pathname do for i in $(seq $strips) #repeat each pathname 'strips' times do echo $RANDOM $RANDOM "$pathname" #with a couple of random numbers prepended done done | sort -n | #sorting lines with prepended random numbers shuffles them while read -r f1 f2 pathname do let 'sn==0&&++cn' && echo [collage-$cn] #write header before first strip let "sw=wac+(RANDOM<sir)*($dist)/dmi" #calculate strip width let mo=width-sw #calculate maximum offset for that width let o=RANDOM*mo/rs #choose a random offset within that range echo $sw $height $o 0 $pathname #write crop and source spec for strip let '++sn<strips||(sn=0)' || echo #write blank line after last strip done
posted by flabdablet at 7:40 AM on December 29, 2017
Response by poster: This sounds great, but can you explain to me—as if I were a child—how exactly I go about running this series of instructions?
Seeing that reference to “bash”, I Googled around seeing that it seems to be Unix stuff. Is it something I then paste into the Terminal app on my Mac?
If i do that, will it prompt me for the locations to draw-from/save-to? Or do my source files need to be in a certain, set location before I paste the script into Terminal?
I found this page, which talks about how to turn Unix shell scripts into executable files.
If I do something like that then double-click on it, will the running script prompt me for information like “source folder” and “destination folder”? Or, should I drop the executable file into the same folder as the source images?
I’m afraid that the closest thing I’ve ever done to programming was fiddling in Basic on the Apple ][+.
Thank you again for all of your interest and leg work on this—I’m excited to see what your script can do!
posted by blueberry at 3:39 AM on December 30, 2017
Seeing that reference to “bash”, I Googled around seeing that it seems to be Unix stuff. Is it something I then paste into the Terminal app on my Mac?
If i do that, will it prompt me for the locations to draw-from/save-to? Or do my source files need to be in a certain, set location before I paste the script into Terminal?
I found this page, which talks about how to turn Unix shell scripts into executable files.
If I do something like that then double-click on it, will the running script prompt me for information like “source folder” and “destination folder”? Or, should I drop the executable file into the same folder as the source images?
I’m afraid that the closest thing I’ve ever done to programming was fiddling in Basic on the Apple ][+.
Thank you again for all of your interest and leg work on this—I’m excited to see what your script can do!
posted by blueberry at 3:39 AM on December 30, 2017
I’m afraid that the closest thing I’ve ever done to programming was fiddling in Basic on the Apple ][+.
That's plenty. It's where I got started as well. And that's a command line environment similar in many respects to the one you get in Terminal, so you're already over the first few hurdles.
The first thing you do with all that stuff is select all the lines from #!/bin/bash to the final done, copy them, paste them into an empty TextEdit window, then save the result as a plain text file named make-collage-specs in your home folder.
There are single and double quotes in there, and they must not be replaced with "smart" quotes. So before you paste into TextEdit, you should find all the automatic "smart" replacement settings in its options (I don't have a Mac so I can't actually give you chapter and verse on where those are) and turn them off.
Next, open a Terminal and enter the command
Now we need to tell Unix that it's allowed to run make-collage-specs as a script. The command that does that is
When you're working in Terminal, you should find that hitting the Tab key after entering enough characters to identify a file's name unambiguously will make Terminal type the rest of the name in for you. So when you're entering that chmod command, try hitting Tab once you've got as far as the mak at the start of make-collage-specs and you'll probably find that the rest gets filled in. Tab completion is what makes using longish filenames in Terminal tolerable. It didn't exist in the very earliest versions of the Unix shell, which is why all the command names are so short and cryptic.
Next, use Finder to create a new folder called test-pics inside your home folder and copy a small handful of pictures from your collection into that. Four or five will be enough at this stage.
Back in your Terminal session, enter
Copy the first stanza of that flood of text out of Terminal and paste it into a reply here. I'm excited to see what you get as well. If it all looks good, I'll show you how to go about saving the output from make-collage-specs as a file, we can make a start on some image processing scripts that read that and do what you actually wanted to do in the first place, and we can talk about setting things up so you don't actually need to be typing stuff into Terminal every time you need to use this stuff.
posted by flabdablet at 5:45 AM on December 30, 2017
That's plenty. It's where I got started as well. And that's a command line environment similar in many respects to the one you get in Terminal, so you're already over the first few hurdles.
The first thing you do with all that stuff is select all the lines from #!/bin/bash to the final done, copy them, paste them into an empty TextEdit window, then save the result as a plain text file named make-collage-specs in your home folder.
There are single and double quotes in there, and they must not be replaced with "smart" quotes. So before you paste into TextEdit, you should find all the automatic "smart" replacement settings in its options (I don't have a Mac so I can't actually give you chapter and verse on where those are) and turn them off.
Next, open a Terminal and enter the command
ls -lYou should see a bunch of files listed, and make-collage-specs should be one of them. ls on Unix does the same job as CATALOG did in AppleDOS, if you remember that, and -l means "long form" so you get to see all the permissions, file sizes and modification dates along with the file names.
Now we need to tell Unix that it's allowed to run make-collage-specs as a script. The command that does that is
chmod ug+x make-collage-specschmod means "change mode" and ug+x means "for the user that owns the file as well as users in the group that owns the file, add execute permission to the file's existing permissions".
When you're working in Terminal, you should find that hitting the Tab key after entering enough characters to identify a file's name unambiguously will make Terminal type the rest of the name in for you. So when you're entering that chmod command, try hitting Tab once you've got as far as the mak at the start of make-collage-specs and you'll probably find that the rest gets filled in. Tab completion is what makes using longish filenames in Terminal tolerable. It didn't exist in the very earliest versions of the Unix shell, which is why all the command names are so short and cryptic.
Next, use Finder to create a new folder called test-pics inside your home folder and copy a small handful of pictures from your collection into that. Four or five will be enough at this stage.
Back in your Terminal session, enter
./make-collage-specs test-picsand you should see a massive flood of text get generated, but not so massive as to prevent you from scrolling the Terminal window back far enough to see it all. Again, note that you should be able to use Tab completion to finish both those names - you don't need to type every character by hand.
Copy the first stanza of that flood of text out of Terminal and paste it into a reply here. I'm excited to see what you get as well. If it all looks good, I'll show you how to go about saving the output from make-collage-specs as a file, we can make a start on some image processing scripts that read that and do what you actually wanted to do in the first place, and we can talk about setting things up so you don't actually need to be typing stuff into Terminal every time you need to use this stuff.
posted by flabdablet at 5:45 AM on December 30, 2017
By the way, thanks for the chance to teach a man how to fish. This is fun!
posted by flabdablet at 5:54 AM on December 30, 2017
posted by flabdablet at 5:54 AM on December 30, 2017
Response by poster: flabdablet, okay I followed your directions—which were not only great, but made me say "flabdablet is walking me through it and teaching me the History of why it's like this at this same time, this is great!"...
The only hiccup was that when I saved the TextEdit file, it of course appended a
Here's the first stanza:
posted by blueberry at 10:19 PM on December 30, 2017
The only hiccup was that when I saved the TextEdit file, it of course appended a
.txt to the end, so when I first pasted the commands Terminal couldn't find my make-collage-specs.txt file to change its mode. I figured that problem out myself and, BOOM, I got the flood of text.Here's the first stanza:
./make-collage-specs.txt test-pics
[collage-1]
200 4000 3254 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 3529 0 test-pics/29123567132_2aee13ec1c_o.jpg
178 4000 394 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 5248 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 847 0 test-pics/24803085776_5870e811fd_o.jpg
188 4000 869 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 3675 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 311 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 2453 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 989 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 752 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 1457 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 785 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 5655 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 32 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 314 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 1639 0 test-pics/29153147011_ab2abba1ae_o.jpg
218 4000 4139 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 3793 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 3225 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 2870 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 4472 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 4560 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 4423 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 2367 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 685 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 4681 0 test-pics/29153041571_acbfff3d85_o.jpg
162 4000 5643 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 229 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 4111 0 test-pics/29153169971_aca019021c_o.jpgposted by blueberry at 10:19 PM on December 30, 2017
Okay, this is really good progress.
Let's tidy up TextEdit's mess:
Now let's take an opportunity both to apply a little bug fix and make sure our choice of tools is sound. Open make-collage-specs in TextEdit again, find the line that reads
Back in Terminal, do another ls -l to make sure you now still have only make-collage-specs and not both make-collage-specs and make-collage-specs.txt. If you only have the one and its permissions still look more like -rwxr-xr-- than -rw-r--r-- , that means TextEdit is going to be good enough for the rest of this exercise and we won't have to grapple with learning another text editor just yet.
Assuming all is well, try another
I'm going to pause here for a rambling diversion on what we're actually working with here, and why that bug fix makes sense.
make-collage-specs is a bash script file, and bash is the default shell (command line interpreter) on your Mac. You could, if you chose, just enter every line from make-collage-specs into Terminal one at a time instead of invoking it as a script, and it would do the same things.
This should feel quite familiar: the bash interpreter that runs when you open Terminal works very similarly to the Applesoft BASIC interpreter that runs when you switch on an Apple ][+. Just like Applesoft, bash is both an interactive command line interpreter and a programming language, where the command lines and the programming language have the same syntax and you can try stuff out at the command line to figure out how it's going to work inside your programs.
Unlike Applesoft, bash doesn't have the idea of a current program that's in memory; you don't LOAD a program in order to edit it and then SAVE it again, you use a text editor for that (which also means your programs don't need line numbers, because keeping track of which order the lines are in is the text editor's responsibility). You also don't have a RUN command to run the program in memory because there isn't one; instead, you can just type a saved program's name as if it were a command, and bash will run it for you straight from disk.
The syntax of bash script is obviously not the same as that of BASIC, but it shares many of the fundamental ideas. You have named variables available just like you do in BASIC, and you can assign stuff to them and use the values saved in them in expressions.
Unlike Applesoft, bash uses a variable's entire name, not just its first two characters, to distinguish it from others. Also unlike Applesoft, bash variables only ever contain character strings, even though bash supports arithmetic operations for strings that consist solely of sequences of numeric digits. Internally, bash uses 64-bit signed integer arithmetic without overflow checking, meaning that all values it does arithmetic with have to fit within the range -9223372036854775808 to 9223372036854775807 to avoid bogus results and that the results of divisions have no fractional part (remainders are simply discarded).
With all the above in mind, you should now recognize the first few lines of make-collage-specs as assignment statements that set a bunch of variables to values that can be manipulated as integers. They are put there right at the beginning to make it easy for you to tweak the way the specs get generated to suit your artistic purpose. They also introduce the bash syntax for comments: anything from a # at the beginning of a word to the end of the line it occurs on gets ignored by the interpreter.
Now have a look at the line that reads
The single quotes that surround that string value tell bash not to mess with its innards in any way before assigning it to dist, just use it exactly as written (the quotes themselves do get stripped off it first).
Which brings up the first of the bash language's many interesting quirks. Because every value that bash ever deals with is handled first and foremost as a character string, there are lots of instances where you don't actually need to wrap a string in quotes to tell bash where it begins and ends. All those numeric values being assigned at the top of the script - the ones that look like numbers - are in fact just bare, unquoted strings.
In your Terminal window, type
Now try
In Applesoft you can use a PRINT statement to find out what's stored in a variable. The bash equivalent for PRINT is echo. Try this:
Each of these is triggered by some kind of punctuation character, and there are a lot of them, and they can be hard to spot especially if you're not already Super Bash Programmer. That's why it's good form to wrap any string more complicated than a simple English word in single quotes if you want to make it clear that it's not to be messed with just yet.
Just for completeness, see if you can correctly predict what you'll see after entering
Time to look at the next part of make-collage-specs: that group of five let commands.
All the strings you hand to a let command will get evaluated as arithmetic expressions. In the context of an arithmetic expression, assignment with = is just another arithmetic operator; the value of an assignment is the same as the value assigned, which means you can do things like
Using let to achieve let rs=32768 is a bit gratuitous; I only did it for prosody. But let wac=width/strips is more interesting. Inside an arithmetic expression, the values of variables get substituted for their names without any need for the $ you'd use to get at them inside a string.
Predict and compare the result of pasting
Multiplication and division operators have equal precedence, and bash evaluates operations of equal precedence in strict left-to-right order. I were to write let sir=inaccurate/100*rs instead, the intermediate result inaccurate/100 would get truncated to zero most of the time (this is integer arithmetic) and sir would usually end up at zero, completely losing the intended percentage. Doing the multiplication step first, which is mathematically equivalent, lets the integer division step chew on numbers big enough to make the discarding of fractional parts unimportant.
Similar considerations apply to the next let expression.
Now look a bit further down in the script, for the line that reads
That means that the expression actually seen by let will be either
RANDOM is a "magic" variable that appears to contain a new random value between 0 and 32767 every time it's read. That makes either of the subexpressions that could be wedged in via expansion of dist yield a random value between -32767 and +32767. They have different distributions of those values, though: RANDOM-RANDOM is much more likely to produce values near zero than values out at the edges, while the other formula yields an even spread.
There are operators inside that other formula that you won't recognize from BASIC, though they're in C if you've ever looked at that language. The % operator does remaindering, so RANDOM%2 yields the remainder of dividing a random number by 2: either 0 or 1, equally distributed. The a?b:c construction tests a, returning b if a is truthy (nonzero) or c if false (zero), so (RANDOM%2?-1:1) yields either -1 or 1, equally distributed. That then gets multiplied by another RANDOM, giving us an equal chance of ending up with either a random positive number or a random negative one. There's a very small lump in the distribution at exactly zero, but I don't care.
Whichever method is used, the resulting random number gets multiplied by (RANDOM<sir) which, because of the way sir was calculated above, has only an $inaccurate percent chance of being 1 (true); otherwise it's zero (false), multiplying by which throws all the calculated randomness away.
Any surviving randomness gets its range reduced from -32767..+32767 to the allowable range of maximum pixel count inaccuracy by being divided by the value of dmi, calculated earlier for that purpose. Adding the result to the normal width for an accurately cut strip (stored in wac) gets us the final strip width to be written as spec output.
Having worked out the width of the strip to be cut, the next step is to work out where across the photo it will be cut from. let mo=width-sw gives us the maximum (rightmost) offset where a cut could start and still achieve a strip of the required width, so we want a random offset between 0 and $mo inclusive.
And this is where the bug was. Before you fixed it, the next line read
But with any luck you've fixed it now.
Having worked through all of this for you, I've just noticed another bug. You should replace the line that reads
This got long, and I need sleep, so I'll wrap it up for tonight. Let me tease you, though, with this: assuming we didn't manage to break make-collage-specs in the process of bugfixing it, so it's still capable of generating walls of text for you, try
posted by flabdablet at 8:15 AM on December 31, 2017
Let's tidy up TextEdit's mess:
mv make-collage-specs.txt make-collage-specsmv is pronounced "move" and it means "rename". Again, Tab completion is your friend here: you can use it for both filenames in this command line, just backspace over the .txt that gets auto-typed after you Tab-complete the second one.
Now let's take an opportunity both to apply a little bug fix and make sure our choice of tools is sound. Open make-collage-specs in TextEdit again, find the line that reads
let o=RANDOM*mo/rs #choose a random offset within that rangeand replace it with this one
let 'o=mo*RANDOM/(rs-1)' #choose a random offset within that rangeand Save.
Back in Terminal, do another ls -l to make sure you now still have only make-collage-specs and not both make-collage-specs and make-collage-specs.txt. If you only have the one and its permissions still look more like -rwxr-xr-- than -rw-r--r-- , that means TextEdit is going to be good enough for the rest of this exercise and we won't have to grapple with learning another text editor just yet.
Assuming all is well, try another
./make-collage-specs test-picsand verify that you get another wall of text.
I'm going to pause here for a rambling diversion on what we're actually working with here, and why that bug fix makes sense.
make-collage-specs is a bash script file, and bash is the default shell (command line interpreter) on your Mac. You could, if you chose, just enter every line from make-collage-specs into Terminal one at a time instead of invoking it as a script, and it would do the same things.
This should feel quite familiar: the bash interpreter that runs when you open Terminal works very similarly to the Applesoft BASIC interpreter that runs when you switch on an Apple ][+. Just like Applesoft, bash is both an interactive command line interpreter and a programming language, where the command lines and the programming language have the same syntax and you can try stuff out at the command line to figure out how it's going to work inside your programs.
Unlike Applesoft, bash doesn't have the idea of a current program that's in memory; you don't LOAD a program in order to edit it and then SAVE it again, you use a text editor for that (which also means your programs don't need line numbers, because keeping track of which order the lines are in is the text editor's responsibility). You also don't have a RUN command to run the program in memory because there isn't one; instead, you can just type a saved program's name as if it were a command, and bash will run it for you straight from disk.
The syntax of bash script is obviously not the same as that of BASIC, but it shares many of the fundamental ideas. You have named variables available just like you do in BASIC, and you can assign stuff to them and use the values saved in them in expressions.
Unlike Applesoft, bash uses a variable's entire name, not just its first two characters, to distinguish it from others. Also unlike Applesoft, bash variables only ever contain character strings, even though bash supports arithmetic operations for strings that consist solely of sequences of numeric digits. Internally, bash uses 64-bit signed integer arithmetic without overflow checking, meaning that all values it does arithmetic with have to fit within the range -9223372036854775808 to 9223372036854775807 to avoid bogus results and that the results of divisions have no fractional part (remainders are simply discarded).
With all the above in mind, you should now recognize the first few lines of make-collage-specs as assignment statements that set a bunch of variables to values that can be manipulated as integers. They are put there right at the beginning to make it easy for you to tweak the way the specs get generated to suit your artistic purpose. They also introduce the bash syntax for comments: anything from a # at the beginning of a word to the end of the line it occurs on gets ignored by the interpreter.
Now have a look at the line that reads
dist='(RANDOM%2?-1:1)*RANDOM'Despite the fact that the stuff to the right of the = looks like some kind of arithmetic expression, this is actually just a variable named dist getting the string value (RANDOM%2?-1:1)*RANDOM assigned to it. The fact that the contents of that string bear a strong resemblance to an arithmetic expression will be exploited later, but doesn't matter for the purposes of this value assignment.
The single quotes that surround that string value tell bash not to mess with its innards in any way before assigning it to dist, just use it exactly as written (the quotes themselves do get stripped off it first).
Which brings up the first of the bash language's many interesting quirks. Because every value that bash ever deals with is handled first and foremost as a character string, there are lots of instances where you don't actually need to wrap a string in quotes to tell bash where it begins and ends. All those numeric values being assigned at the top of the script - the ones that look like numbers - are in fact just bare, unquoted strings.
In your Terminal window, type
foo=bar,baz,quxYou should find that bash accepts this without complaint. What you've done there is assigned the string bar,baz,qux to the variable foo, a perfectly legitimate operation.
Now try
yo=hello sailorThis time, bash will complain:
bash: sailor: command not foundIt has no problem with yo=hello - as in the foo=bar,baz,qux case, this just assigns a value to a variable. But bash interprets the line you just typed as meaning "assign the value 'hello' to the variable named 'yo', then run the command 'sailor' in a way that makes that variable available to that command if it needs it". So it complains because it can't find a command called sailor installed on your Mac.
In Applesoft you can use a PRINT statement to find out what's stored in a variable. The bash equivalent for PRINT is echo. Try this:
echo fooYou will see bash reply with
fooAgain, bash has just interpreted the bare, unquoted value foo as a meaningless character string, and obediently echoed it back to you. Which is fine - you might perhaps want to put something like
echo Donein a script - but it doesn't help with seeing inside our variables. What you actually need right now is this:
echo $fooand this should get you
bar,baz,quxRemember how I mentioned that wrapping a string in single quotes tells bash not to mess with its innards? This particular kind of messing - replacing a $ followed immediately by the name of some variable with the value of that variable - is one of the kinds I had in mind. The bash manual calls this parameter expansion and it's one of several kinds of potentially useful messing that bash can do with strings.
Each of these is triggered by some kind of punctuation character, and there are a lot of them, and they can be hard to spot especially if you're not already Super Bash Programmer. That's why it's good form to wrap any string more complicated than a simple English word in single quotes if you want to make it clear that it's not to be messed with just yet.
Just for completeness, see if you can correctly predict what you'll see after entering
echo '$foo'and then make sure you were right.
Time to look at the next part of make-collage-specs: that group of five let commands.
All the strings you hand to a let command will get evaluated as arithmetic expressions. In the context of an arithmetic expression, assignment with = is just another arithmetic operator; the value of an assignment is the same as the value assigned, which means you can do things like
let 'a=(b=1+2)+4' echo $a $bSee if you can predict what output you'll get before pasting those two lines into your Terminal. If not, see if you can work out why you get the output you do.
Using let to achieve let rs=32768 is a bit gratuitous; I only did it for prosody. But let wac=width/strips is more interesting. Inside an arithmetic expression, the values of variables get substituted for their names without any need for the $ you'd use to get at them inside a string.
Predict and compare the result of pasting
rs=32768 half=rs/2 echo $rs $halfwith that of pasting
let rs=32768 let half=rs/2 echo $rs $halflet sir=inaccurate*rs/100 exposes another subtlety: order of operations can matter. inaccurate is intended to hold a percentage, from 0 to 100. I'm trying to translate that to a scaled inaccuracy rate matched to the range of the random number generator, which returns random numbers greater than or equal to 0 and less than rs=32768.
Multiplication and division operators have equal precedence, and bash evaluates operations of equal precedence in strict left-to-right order. I were to write let sir=inaccurate/100*rs instead, the intermediate result inaccurate/100 would get truncated to zero most of the time (this is integer arithmetic) and sir would usually end up at zero, completely losing the intended percentage. Doing the multiplication step first, which is mathematically equivalent, lets the integer division step chew on numbers big enough to make the discarding of fractional parts unimportant.
Similar considerations apply to the next let expression.
Now look a bit further down in the script, for the line that reads
let "sw=wac+(RANDOM<sir)*($dist)/dmi" #calculate strip widthSee that $dist buried inside the expression string handed to let? The fact that this string is wrapped in double quotes, not single quotes, means that parameter expansion is allowed to happen inside it (though there are other kinds of possible messing that the double-quoting does prohibit). So $dist will get replaced with the value stored in dist, which will be either (RANDOM%2?-1:1)*RANDOM or RANDOM-RANDOM depending on whether you've removed the commenting-out # from the start of the "double dice" line or not.
That means that the expression actually seen by let will be either
sw=wac+(RANDOM<sir)*((RANDOM%2?-1:1)*RANDOM)/dmior
sw=wac+(RANDOM<sir)*(RANDOM-RANDOM)/dmiThe $dist expansion gets done before the expanded string is ever examined by let, and the fact that dist is full of stuff that looks like an arithmetic expression doesn't matter during the expansion process. It's only when let actually gets around to evaluating the expanded result as an arithmetic expression that all the rules about variable value for name substitution inside arithmetic expressions come into play. I did it this way so that you could, if you wanted, play with designing dist expressions yielding various kinds of random distribution without worrying too much about the details of how those expressions would end up being used.
RANDOM is a "magic" variable that appears to contain a new random value between 0 and 32767 every time it's read. That makes either of the subexpressions that could be wedged in via expansion of dist yield a random value between -32767 and +32767. They have different distributions of those values, though: RANDOM-RANDOM is much more likely to produce values near zero than values out at the edges, while the other formula yields an even spread.
There are operators inside that other formula that you won't recognize from BASIC, though they're in C if you've ever looked at that language. The % operator does remaindering, so RANDOM%2 yields the remainder of dividing a random number by 2: either 0 or 1, equally distributed. The a?b:c construction tests a, returning b if a is truthy (nonzero) or c if false (zero), so (RANDOM%2?-1:1) yields either -1 or 1, equally distributed. That then gets multiplied by another RANDOM, giving us an equal chance of ending up with either a random positive number or a random negative one. There's a very small lump in the distribution at exactly zero, but I don't care.
Whichever method is used, the resulting random number gets multiplied by (RANDOM<sir) which, because of the way sir was calculated above, has only an $inaccurate percent chance of being 1 (true); otherwise it's zero (false), multiplying by which throws all the calculated randomness away.
Any surviving randomness gets its range reduced from -32767..+32767 to the allowable range of maximum pixel count inaccuracy by being divided by the value of dmi, calculated earlier for that purpose. Adding the result to the normal width for an accurately cut strip (stored in wac) gets us the final strip width to be written as spec output.
Having worked out the width of the strip to be cut, the next step is to work out where across the photo it will be cut from. let mo=width-sw gives us the maximum (rightmost) offset where a cut could start and still achieve a strip of the required width, so we want a random offset between 0 and $mo inclusive.
And this is where the bug was. Before you fixed it, the next line read
let o=RANDOM*mo/rs #choose a random offset within that range$mo gets multiplied by a random value guaranteed to be less than $rs, if only by 1; the result then gets divided by $rs. That's an integer division and any fractional part is just thrown away, not rounded. So $o is guaranteed to end up less than $mo; it can't ever be equal to it. Which means that there is a 1-pixel stripe down the right hand side of every source photo that will never be included in a cut strip.
But with any luck you've fixed it now.
Having worked through all of this for you, I've just noticed another bug. You should replace the line that reads
echo $sw $height $o 0 $pathname #write crop and source spec for stripwith
echo $sw $height $o 0 "$pathname" #write crop and source spec for stripIf you can tell me why this matters, you grok bash. Clue: word splitting.
This got long, and I need sleep, so I'll wrap it up for tonight. Let me tease you, though, with this: assuming we didn't manage to break make-collage-specs in the process of bugfixing it, so it's still capable of generating walls of text for you, try
./make-collage-specs test-pics >spec-set-1.txtSee if you can guess what this is going to do before you try it.
posted by flabdablet at 8:15 AM on December 31, 2017
Response by poster: Whew, that took me a lot of slow reading, but I think I understood most of it. I did guess that the final command would wind up creating a log text file :)
posted by blueberry at 9:30 PM on December 31, 2017
posted by blueberry at 9:30 PM on December 31, 2017
Excellent!
Now that you have spec-set-1.txt available, we can start using it to make collages.
First we need to get ffmpeg working on your machine. Grab a self-contained OS X build of it from evermeet.cx. The one you want is the release version, packaged as a DMG. Here's the link to the DMG for version 3.4.1, just in case that's still the current release by the time you read this.
When you open that DMG after downloading it, you should see a file named ffmpeg inside it with a size of about 43 megabytes (43646516 bytes for the 3.4.1 version). Copy that into your home folder.
In Terminal, enter the command
By the way, I'm more than happy to clarify anything I went too fast on in the previous comment. There are no stupid questions when it comes to bash scripting, only incompetent tutors.
posted by flabdablet at 11:25 PM on December 31, 2017
Now that you have spec-set-1.txt available, we can start using it to make collages.
First we need to get ffmpeg working on your machine. Grab a self-contained OS X build of it from evermeet.cx. The one you want is the release version, packaged as a DMG. Here's the link to the DMG for version 3.4.1, just in case that's still the current release by the time you read this.
When you open that DMG after downloading it, you should see a file named ffmpeg inside it with a size of about 43 megabytes (43646516 bytes for the 3.4.1 version). Copy that into your home folder.
In Terminal, enter the command
./ffmpeg -versionand paste the resulting output into a reply. If it looks reasonably close to what I'm seeing when I ask my Linux box's ffmpeg for its own version number, we should be good to go.
By the way, I'm more than happy to clarify anything I went too fast on in the previous comment. There are no stupid questions when it comes to bash scripting, only incompetent tutors.
posted by flabdablet at 11:25 PM on December 31, 2017
Response by poster:
posted by blueberry at 10:26 PM on January 4, 2018
ffmpeg version 3.4.1-tessus Copyright (c) 2000-2017 the FFmpeg developers
built with Apple LLVM version 8.0.0 (clang-800.0.42.1)
configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libass --enable-libbluray --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzmq --enable-libzvbi --enable-version3 --pkg-config-flags=--static --disable-ffplay
libavutil 55. 78.100 / 55. 78.100
libavcodec 57.107.100 / 57.107.100
libavformat 57. 83.100 / 57. 83.100
libavdevice 57. 10.100 / 57. 10.100
libavfilter 6.107.100 / 6.107.100
libswscale 4. 8.100 / 4. 8.100
libswresample 2. 9.100 / 2. 9.100
libpostproc 54. 7.100 / 54. 7.100
posted by blueberry at 10:26 PM on January 4, 2018
Save this as make-collages:
posted by flabdablet at 10:39 PM on January 4, 2018
#!/bin/bash
width=6000
height=4000
ffmpeg=:
inputs=()
crop=
pads=
sn=0
output=
folder=${1:-.}
mkdir -p "$folder"
while read -r line
do
case $line in
'['*)
$ffmpeg "${inputs[@]}" -filter_complex "$crop${pads}hstack=$sn" "$folder/$output"
ffmpeg='./ffmpeg -nostdin -y -loglevel error'
inputs=()
crop=
pads=
sn=0
output=${line:1:-1}.jpg
;;
[0-9]*)
read -r w h x y pathname <<<"$line"
inputs+=(-f image2 -i "$pathname")
crop+="scale=$width:$height,crop=$w:$h:$x:$y[$sn];"
pads+="[$sn]"
let sn+=1
;;
esac
done
$ffmpeg "${inputs[@]}" -filter_complex "$crop${pads}hstack=$sn" "$folder/$output"
posted by flabdablet at 10:39 PM on January 4, 2018
Save this as make-collage-strips:
posted by flabdablet at 10:40 PM on January 4, 2018
#!/bin/bash
width=6000
height=4000
ffmpeg='./ffmpeg -nostdin -y -loglevel error'
folder=${1:-.}
mkdir -p "$folder"
while read -r line
do
case $line in
'['*)
name=${line:1:-1}
sn=0
;;
[0-9]*)
read -r w h x y pathname <<<"$line"
$ffmpeg -f image2 -i "$pathname" -vf "scale=$width:$height,crop=$w:$h:$x:$y" "$folder/$name-strip-$sn.png"
let sn+=1
;;
esac
done
posted by flabdablet at 10:40 PM on January 4, 2018
Now try
posted by flabdablet at 10:46 PM on January 4, 2018
./make-collages collage-set-1 <spec-set-1.txtIf that works: open spec-set-1 with TextEdit, copy the spec stanza for collage-3, and paste that into a new file called collage-3-spec.txt. Then try
./make-collage-strips collage-3-strips <collage-3-spec.txtIf all is working well, the next thing I'll cover is ways to avoid needing to keep untidy amounts of everything just jammed directly in your home folder.
posted by flabdablet at 10:46 PM on January 4, 2018
By the way, after saving those two scripts, don't forget to chmod them as executable.
posted by flabdablet at 10:48 PM on January 4, 2018
posted by flabdablet at 10:48 PM on January 4, 2018
Response by poster: Hmm, for some reason that I am not understanding, when I try
on my
I get:
posted by blueberry at 11:19 PM on January 4, 2018
chmod ug+x make-collages on my
make-collages text file, I get:
chmod: make-collages: No such file or directorymake-collage-strips text file (both of which the Finder says are Plain Text Document(s), and are shown in the Finder window as not having the .txt file extension)posted by blueberry at 11:19 PM on January 4, 2018
Try doing an ls -l before the chmod and making sure you can see both those files listed in its output, just to make sure that TextEdit did in fact save them directly into your home folder.
posted by flabdablet at 11:28 PM on January 4, 2018
posted by flabdablet at 11:28 PM on January 4, 2018
It's also possible that TextEdit and/or Finder has done something "smart", and therefore horrible and annoying, to the - signs inside the filenames themselves. Like changing them to em dashes or adding spaces or some bloody thing.
posted by flabdablet at 11:29 PM on January 4, 2018
posted by flabdablet at 11:29 PM on January 4, 2018
Response by poster: Both files are in my home folder.
Okay, when I did the
I pasted the names into the text command as
Yet, when I look at the files in the Finder, they appear unchanged.
In the Finder, they look like
posted by blueberry at 7:20 PM on January 5, 2018
Okay, when I did the
ls -l, I saw that the files do indeed have a .txt on the end of them (the .txt was not showing in the list view in the Finder).I pasted the names into the text command as
and Terminal takes the commands without protests, giving me thechmod ug+x make-collage-strips.txt
and
chmod ug+x make-collages.txt
jacks-mac:~ Jack$ prompt after each commandYet, when I look at the files in the Finder, they appear unchanged.
In the Finder, they look like
Plain Text Document documents, not Unix executables, and in Terminal after a ls -l command, they still have the .txt file extensions.posted by blueberry at 7:20 PM on January 5, 2018
Response by poster: flabdablet, I just MeMail'ed you a link to a screenshot showing my Terminal window and my Finder window after trying the commands—in case that might help.
posted by blueberry at 7:27 PM on January 5, 2018
posted by blueberry at 7:27 PM on January 5, 2018
which make it seem like it took the commands without any error
...which is in fact what happened.
In the Finder, they look like Plain Text Document documents, not Unix executables, and in Terminal after a ls -l command, they still have the .txt file extensions.
That will be because Finder has "smart" (i.e. irritating cheap-ass heuristic-driven) behaviour intended to reduce confusion for files moved back and forth between Mac and PC worlds, and is actually hiding the .txt extension on those filenames from you so that you don't get confuzzled.
I expect that if you were to paste the following commands into Terminal,
If it doesn't, try double-checking that the #!/bin/bash line constitutes the very first characters in each of those files, with nothing at all before it; no spaces, no newlines, no tabs, no redundant UTF-8 byte order marks, nothing. GUI tools clearly know better than you do what you actually need to see, so don't use them for this check - instead, use
For what it's worth, as long as the files are marked executable then bash should let you execute them even if they do have .txt at the end of their names. Command names having no extensions is a convention, not a requirement.
posted by flabdablet at 1:41 AM on January 6, 2018
...which is in fact what happened.
In the Finder, they look like Plain Text Document documents, not Unix executables, and in Terminal after a ls -l command, they still have the .txt file extensions.
That will be because Finder has "smart" (i.e. irritating cheap-ass heuristic-driven) behaviour intended to reduce confusion for files moved back and forth between Mac and PC worlds, and is actually hiding the .txt extension on those filenames from you so that you don't get confuzzled.
I expect that if you were to paste the following commands into Terminal,
mv make-collages.txt make-collages mv make-collage-strips.txt make-collage-stripsthen Finder would give up on believing they're text files and show them as Unix executables.
If it doesn't, try double-checking that the #!/bin/bash line constitutes the very first characters in each of those files, with nothing at all before it; no spaces, no newlines, no tabs, no redundant UTF-8 byte order marks, nothing. GUI tools clearly know better than you do what you actually need to see, so don't use them for this check - instead, use
hexdump -C -n128 make-collagesThe first two hex numbers you should see in the resulting output are 23 21, the ASCII codes for the # and the !.
For what it's worth, as long as the files are marked executable then bash should let you execute them even if they do have .txt at the end of their names. Command names having no extensions is a convention, not a requirement.
posted by flabdablet at 1:41 AM on January 6, 2018
Response by poster: Those
However, when I tried the command
mv commands worked fine, turning the files into executables.However, when I tried the command
./make-collages collage-set-1 ./make-collages: line 22: -1: substring expression < 0
[hstack @ 0x7fa5d15000e0] Value 0.000000 for parameter 'inputs' out of range [2 - 2.14748e+09]
Last message repeated 1 times
[hstack @ 0x7fa5d15000e0] Error setting option inputs to value 0.
[Parsed_hstack_0 @ 0x7fa5d1500000] Error applying options to the filter.
[AVFilterGraph @ 0x7fa5d140f340] Error initializing filter 'hstack' with args '0'
Error initializing complex filters.
Result too largeResponse by poster: Oh, and somewhere along the line I did get a
spec-set-1.txt file, the contents of which look like [collage-1]
200 4000 59 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 871 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 4241 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 1130 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 3221 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 3540 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 3904 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 3879 0 test-pics/29123567132_2aee13ec1c_o.jpg
232 4000 5428 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 3541 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 4814 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 4525 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 4346 0 test-pics/29153169971_aca019021c_o.jpg
151 4000 817 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 4333 0 test-pics/24803085776_5870e811fd_o.jpg
235 4000 887 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 3382 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 306 0 test-pics/24803085776_5870e811fd_o.jpg
152 4000 1047 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 2714 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 2563 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 2577 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 2692 0 test-pics/29231469045_509d268d83_o.jpg
241 4000 4766 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 563 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 143 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 5221 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 820 0 test-pics/24803085776_5870e811fd_o.jpg
233 4000 2687 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 2864 0 test-pics/29123567132_2aee13ec1c_o.jpg
[collage-2]
231 4000 683 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 2872 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 5729 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 5321 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 2663 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 3062 0 test-pics/24162760293_8a9f01c50a_o.jpg
152 4000 5835 0 test-pics/28611104513_c58707a48c_o.jpg
208 4000 586 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 2013 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 4674 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 599 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 5604 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 3859 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 1152 0 test-pics/29153147011_ab2abba1ae_o.jpg
169 4000 3253 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 5357 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 762 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 1466 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 3690 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 1449 0 test-pics/29153169971_aca019021c_o.jpg
240 4000 3868 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 3295 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 58 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 2678 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 4618 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 999 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 708 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 4008 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 5654 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 5193 0 test-pics/29123567132_2aee13ec1c_o.jpg
[collage-3]
200 4000 3259 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 348 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 61 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 4321 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 2556 0 test-pics/28611104513_c58707a48c_o.jpg
244 4000 323 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 2410 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 1636 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 4440 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 5260 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 4561 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 5772 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 4729 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 5540 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 1034 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 3914 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 2990 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 197 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 2786 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 4584 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 4716 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 4883 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 4587 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 5007 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 298 0 test-pics/29231469045_509d268d83_o.jpg
216 4000 1835 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 4595 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 4887 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 2002 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 82 0 test-pics/29153169971_aca019021c_o.jpg
[collage-4]
200 4000 727 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 2788 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 5543 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 2762 0 test-pics/29123567132_2aee13ec1c_o.jpg
239 4000 4411 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 4101 0 test-pics/24162760293_8a9f01c50a_o.jpg
151 4000 3895 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 3556 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 2304 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 26 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 899 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 430 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 4571 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 4543 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 2168 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 3619 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 961 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 4385 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 5333 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 3003 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 3427 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 1819 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 2383 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 5079 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 1937 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 5218 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 5007 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 336 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 1444 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 2235 0 test-pics/29123567132_2aee13ec1c_o.jpg
[collage-5]
200 4000 366 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 235 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 863 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 2577 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 5475 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 4578 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 1049 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 5385 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 5570 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 3685 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 1165 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 1403 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 2687 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 3770 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 865 0 test-pics/29153041571_acbfff3d85_o.jpg
195 4000 4163 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 438 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 115 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 2462 0 test-pics/29123567132_2aee13ec1c_o.jpg
178 4000 5757 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 385 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 2446 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 1508 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 2143 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 4212 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 2308 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 566 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 1303 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 4305 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 4238 0 test-pics/24162760293_8a9f01c50a_o.jpg
[collage-6]
200 4000 5587 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 2014 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 3324 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 4741 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 4884 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 1932 0 test-pics/28611104513_c58707a48c_o.jpg
212 4000 4768 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 267 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 307 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 5543 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 3600 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 983 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 3617 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 3139 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 5726 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 3951 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 3864 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 1994 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 1263 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 1098 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 4358 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 5338 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 1468 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 4599 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 1649 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 2928 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 325 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 2644 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 5096 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 2233 0 test-pics/29153041571_acbfff3d85_o.jpg
[collage-7]
200 4000 5715 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 2217 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 4247 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 1597 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 996 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 2271 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 12 0 test-pics/29153147011_ab2abba1ae_o.jpg
164 4000 4049 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 5172 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 1701 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 1037 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 1947 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 5476 0 test-pics/29123567132_2aee13ec1c_o.jpg
193 4000 1526 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 2681 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 2605 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 138 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 1245 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 790 0 test-pics/24162760293_8a9f01c50a_o.jpg
159 4000 1227 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 563 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 1042 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 5472 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 563 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 4505 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 3933 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 2673 0 test-pics/28611104513_c58707a48c_o.jpg
200 4000 1546 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 4044 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 557 0 test-pics/29153041571_acbfff3d85_o.jpg
[collage-8]
200 4000 2749 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 5379 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 5159 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 345 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 2045 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 4156 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 904 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 4696 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 2468 0 test-pics/29123567132_2aee13ec1c_o.jpg
200 4000 426 0 test-pics/28611104513_c58707a48c_o.jpg
168 4000 2595 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 3353 0 test-pics/29153147011_ab2abba1ae_o.jpg
200 4000 2060 0 test-pics/29153147011_ab2abba1ae_o.jpg
229 4000 3553 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 1319 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 5578 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 2978 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 2974 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 5045 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 1725 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 2065 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 5005 0 test-pics/24162760293_8a9f01c50a_o.jpg
200 4000 60 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 217 0 test-pics/29153041571_acbfff3d85_o.jpg
210 4000 4617 0 test-pics/29153041571_acbfff3d85_o.jpg
200 4000 384 0 test-pics/29231469045_509d268d83_o.jpg
200 4000 5417 0 test-pics/24803085776_5870e811fd_o.jpg
200 4000 720 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 3401 0 test-pics/29153169971_aca019021c_o.jpg
200 4000 1168 0 test-pics/29153041571_acbfff3d85_o.jpg
However, when I tried the command
That will be because make-collages is expecting to be able to read one or more collage specifications, like the ones comprising spec-set-1.txt, in order to know what collages to make and which source files to cut their strips from. The command you actually want is
We are so close to success right now. I can smell it.
posted by flabdablet at 4:08 AM on January 8, 2018 [1 favorite]
./make-collages collage-set-1
I got [errors]That will be because make-collages is expecting to be able to read one or more collage specifications, like the ones comprising spec-set-1.txt, in order to know what collages to make and which source files to cut their strips from. The command you actually want is
./make-collages collage-set-1 <spec-set-1.txt
The < is an input redirection that tells bash to make spec-set-1.txt available as the input stream for make-collages; without the redirection, make-collages would instead attempt to read collage specifications from your keyboard.We are so close to success right now. I can smell it.
posted by flabdablet at 4:08 AM on January 8, 2018 [1 favorite]
Oh, and the reason the errors are completely incomprehensible is because my cheap-ass scripts contain no attempts at error checking, and whatever has gone wrong has made it attempt to invoke ffmpeg via a command line that wasn't built properly and doesn't make sense. That, and ffmpeg's error messages are designed to be far more useful to its developers than to its end users. It's a fast tool, and a capable tool, but it's in no way a friendly tool.
posted by flabdablet at 4:12 AM on January 8, 2018
posted by flabdablet at 4:12 AM on January 8, 2018
Response by poster: Hmm, when I try the
./make-collages collage-set-1 <spec-set-1.txtjacks-mac:~ Jack$ ./make-collages collage-set-1 <spec-set-1.txt
./make-collages: line 22: -1: substring expression < 0
[hstack @ 0x7f98ef504140] Value 0.000000 for parameter 'inputs' out of range [2 - 2.14748e+09]
Last message repeated 1 times
[hstack @ 0x7f98ef504140] Error setting option inputs to value 0.
[Parsed_hstack_0 @ 0x7f98ef502ba0] Error applying options to the filter.
[AVFilterGraph @ 0x7f98ef7005a0] Error initializing filter 'hstack' with args '0'
Error initializing complex filters.
Result too largeResponse by poster: I MeMail’ed you another screenshot of the error in Terminal, if that might help.
posted by blueberry at 10:04 PM on January 8, 2018
posted by blueberry at 10:04 PM on January 8, 2018
The utility for other people of continuing this thread is pretty limited until we get this bug sorted out, so let's switch to private communication for a bit. I've sent you an email.
posted by flabdablet at 12:00 AM on January 9, 2018
posted by flabdablet at 12:00 AM on January 9, 2018
Turns out that the first of the errors is because Apple's build of bash 4.3 is apparently missing a feature that was introduced in mainline bash 4.2: negative substring lengths don't work in OS X bash.
Substituting the rather unwieldy ${line:1:${#line}-2} for ${line:1:-1} everywhere the latter occurs should get rid of it.
posted by flabdablet at 12:38 AM on January 9, 2018
Substituting the rather unwieldy ${line:1:${#line}-2} for ${line:1:-1} everywhere the latter occurs should get rid of it.
posted by flabdablet at 12:38 AM on January 9, 2018
Response by poster: I changed that string of text where I found it
(in the
and re-ran the
Yay! Thank you so much, flabdablet!
* trying to include the > symbol kept making the HTML interpreter think I was trying to type the
posted by blueberry at 12:56 AM on January 9, 2018
(in the
make-collages and make-collage-strips executable files) and re-ran the
./make-collages collage-set-1 [less-than*]spec-set-1.txt command and
Yay! Thank you so much, flabdablet!
* trying to include the > symbol kept making the HTML interpreter think I was trying to type the
spec tag.posted by blueberry at 12:56 AM on January 9, 2018
Response by poster: (Just fiddling around, but my favorite so far.)
posted by blueberry at 1:25 AM on January 9, 2018
posted by blueberry at 1:25 AM on January 9, 2018
Bonanza!
If you copy the defining stanza for your favourite collage out of spec-set-1.txt, paste it into spec-best-so-far.txt, and then run
You could also feed ./make-collage-strips directly from spec-set-1.txt rather than excerpting stanzas from it first, but strips generation would then take much longer and you'd end up with a large collection of strips you probably don't need.
If you can verify that make-collage-strips is now working as well, I'll move on to tidying things up so your home folder doesn't need to have all this guff saved directly inside it. Once that's done, we should be in good shape to add a bit of draggery and droppery that lets you achieve the original aim of bulk collage generation in the background without needing to visit Terminal to make it happen.
posted by flabdablet at 2:11 AM on January 9, 2018
If you copy the defining stanza for your favourite collage out of spec-set-1.txt, paste it into spec-best-so-far.txt, and then run
./make-collage-strips best-so-far <spec-best-so-far.txtyou should end up with a folder called best-so-far full of individual PNG files for the strips that make that collage, ready for printing and playing with.
You could also feed ./make-collage-strips directly from spec-set-1.txt rather than excerpting stanzas from it first, but strips generation would then take much longer and you'd end up with a large collection of strips you probably don't need.
If you can verify that make-collage-strips is now working as well, I'll move on to tidying things up so your home folder doesn't need to have all this guff saved directly inside it. Once that's done, we should be in good shape to add a bit of draggery and droppery that lets you achieve the original aim of bulk collage generation in the background without needing to visit Terminal to make it happen.
posted by flabdablet at 2:11 AM on January 9, 2018
By the way, the trick to making < appear in comments here is to enter it as < (leading ampersand and trailing semicolon are both required).
posted by flabdablet at 2:12 AM on January 9, 2018
posted by flabdablet at 2:12 AM on January 9, 2018
A marginal improvement in robustness for make-collages and make-collage-strips can be had by replacing the lines in both that read
As things stand at present, the ${line:1:${#line}-2} expression within that logic will barf if the input contains a line consisting of a lone [. This would make ${#line} (the length of $line) equal to 1, which would therefore make ${#line}-2 come to -1, which we've already seen that OS X bash considers illegal (and which other bashes would quietly misinterpret as a from-end offset).
With the revised case pattern in place, ${line:1:${#line}-2} will never even be evaluated unless $line is at least three characters long (and the outermost two, which ${line:1:${#line}-2} is designed to remove, are indeed the square brackets they're supposed to be).
posted by flabdablet at 4:11 AM on January 9, 2018
case $line in '['*)with
case $line in '['?*']')This means that the logic that's supposed to process the header line of a collage specification stanza will only be triggered by an input line that begins with [ and ends with ] and has at least one character in between, rather than by any input line that happens to begin with [ (as might be encountered in a spec file constructed carelessly or maliciously by hand).
As things stand at present, the ${line:1:${#line}-2} expression within that logic will barf if the input contains a line consisting of a lone [. This would make ${#line} (the length of $line) equal to 1, which would therefore make ${#line}-2 come to -1, which we've already seen that OS X bash considers illegal (and which other bashes would quietly misinterpret as a from-end offset).
With the revised case pattern in place, ${line:1:${#line}-2} will never even be evaluated unless $line is at least three characters long (and the outermost two, which ${line:1:${#line}-2} is designed to remove, are indeed the square brackets they're supposed to be).
posted by flabdablet at 4:11 AM on January 9, 2018
Response by poster: Sorry for the delay—yes, I can confirm that the make-collage-strips action is working.
Also, I have updated the make-collages and make-collage-strips scripts per your advice. :)
posted by blueberry at 11:09 PM on January 13, 2018
Also, I have updated the make-collages and make-collage-strips scripts per your advice. :)
posted by blueberry at 11:09 PM on January 13, 2018
OK. Let's pause to tidy up a little, and then if you still want to pursue draggery and droppery instead of using Terminal commands to do this stuff we can move on from there.
So far, all the user-installed commands you've been using - the scripts themselves, and the ffmpeg executable - have been saved in your home folder, and run using syntax like
Operating systems distinguish between files stored on the same device by requiring that they have unique names.
If the device in question is fairly small - the size of an Apple II floppy disk, for example - then the number of files it can hold is quite limited, a simple list of names per disk is easy to use and understand, and the requirement that each file on a given disk has a unique name is not onerous to manage.
On a computer has multiple disk drives attached to it, the operating system needs to give you some way to specify which of those disks it should search when asked to work with any given file. AppleDOS (1978) did this by letting you specify explicit hardware expansion slot and drive numbers:
AppleDOS had the implicit notion of a "current" disk drive: whichever one you'd used most recently was the one it would use again until you chose a different one by adding explicit slot and/or drive number options to some other command. In CP/M, the current disk could be chosen explicitly: entering just
These early microcomputer OS patterns echoed many of those that had been common in operating systems for larger computers of earlier decades, for much the same reason: resource constraints. But in the world of larger machines, relatively huge hard disk drives capable of containing many thousands of files were already common, and scaled-up ways of organizing them had already been developed.
A key insight of the developers of Multics (1965) was that the problem of giving user-friendly names to files on a drive was actually very similar to the problem of giving user-friendly names to the assortment of disks attached at any given time to any given system, and that a unified naming convention could be designed to cover both. This insight was incorporated into the design of Unix (1969) from which the under-the-hood parts of OS X are directly descended.
In AppleDOS, every floppy disk has a catalog: an OS-defined structure that stores a list of the filenames that exist on the disk, along with details about whereabouts on the disk those files can be found. A CP/M disk has a structure called the directory that does much the same job. In Unix, that structure is pulled to pieces: Unix disk files are numbered by the system rather than named, and given their user-visible names only by having those names associated with (linked to) their file (inode) numbers inside directory files, in and of themselves just another kind of file.
This allows a number of interesting things to be done. A disk can hold files whose names do not appear at all in its central directory file (inode 2, by convention); only in other directory files that are themselves linked (perhaps even indirectly!) from the central directory. Which means that filenames no longer need to be unique system-wide, or even unique within any given disk, in order to refer unambiguously to their underlying files: being unique within any directory they're listed in is enough.
And yes, files can be listed in and linked from more than one directory at a time. They can even be linked to more than once, under different names, inside a single directory.
Files can also exist on disks without any user-visible name assigned to them at all. A process can open a file and start working with it; then it, or even another process - perhaps a user working in a command shell - can unlink the file, an operation that on most other systems would amount to deleting it from the disk; but the original process retains full working access to that file until it closes it. Files don't actually get deleted from the disks until no directory entry links to them and no process is holding them open.
The other foundational idea is that the central directory on any given disk volume does not need to occupy a central position in the user's view of the system: rather than relying on extra command options like ,S6,D1 or device-identifying syntax like a B: prefix to distinguish one volume's central directory from another, those central directories can be referred to by filenames in exactly the same way as their own subsidiary directory files are.
Unix makes this work by supporting a mount operation allowing the central directory on any disk to overlay and substitute itself for the contents of any directory file that's already accessible, and by creating a nameless empty directory in RAM at boot time that exists for the sole purpose of allowing some disk's central directory to be mounted on it. Directories that get other disks mounted on them are called mount points, and it's conventional for them to be empty rather than having useful contents of their own; that way, mounting another disk doesn't remove anything useful from the existing filesystem.
A common metaphor for the hierarchical structure that results from files listed in directories listed in directories listed in ... listed in a central directory is that of a tree, with the central directory as its root and trunk, other directories as branches, and files as leaves. IT being a topsy turvy kind of pursuit, filesystem trees are almost always pictured upside down, with the root at the top.
A newer metaphor pictures directory files as manila folders, which can themselves be named and then nested inside other manila folders. This is compelling enough that the Mac and other systems influenced by it always refer to directory files as folders, and speak of the files listed in those directory files as being contained in those folders.
These metaphors work pretty well most of the time, but taking them too seriously will get in the way of understanding links, so don't do that.
The only character not allowed inside a Unix filename is / (slash), which serves to separate the name of a file from that of a directory the filename appears in. If a directory named foo includes an entry for a file named bar then that file can be referred to as foo/bar. If bar is itself a directory file, and includes an entry for a file named qux, then foo/bar/qux unambiguously identifies that file.
A name that includes a chain of directory name prefixes delimited by slashes is called a pathname: it explicitly specifies the path that needs to be taken through some subtree of directories to find the file it refers to. Just about any Unix facility that will accept a simple filename will also accept a pathname in its place, and pathnames can in principle involve any number of directory lookups.
Directory filenames can have a trailing / to indicate that the pathname so formed is that of a directory, in cases where this matters. So foo/bar and foo/bar/ refer to the same file, and the latter form hints that it is a directory.
Much as AppleDOS and CP/M both had a notion of a current drive, Unix has a notion of a current directory. Every process (including the command shell running in your Terminal window) always has a current directory, and files linked from it can be accessed via simple unadorned names rather than slash-containing pathnames. The current directory you will be using when you initially open a Terminal window is your home folder. This is why, when you do stuff like
Any directory you can name can be made the current directory using the cd command:
For example, after you've set collage-set-1 as your current directory, you can return to your home folder using
As a consequence of its mount point having no filename, it was never given one either; but because it is the only nameless directory file in the whole filesystem it still has an unambiguous pathname, which is just / (an empty directory name, followed by the slash that can optionally trail any directory file's pathname).
Any pathname whose first component is named in and linked from the root directory will therefore begin with a slash, as well as optionally ending with one if it's the pathname of a directory. Pathnames that begin with a slash are called absolute pathnames, and unambiguously identify the same file no matter what the current directory is set to.
More tomorrow.
posted by flabdablet at 2:49 AM on January 15, 2018
So far, all the user-installed commands you've been using - the scripts themselves, and the ffmpeg executable - have been saved in your home folder, and run using syntax like
./make-collage-specs bla bla bla ./make-collages etcetera ./ffmpeg huge incomprehensible string of command argumentsWhat all of these have in common is the ./ prefix. So it's time for a historical backgrounder on what that is and why it's been needed here.
Operating systems distinguish between files stored on the same device by requiring that they have unique names.
If the device in question is fairly small - the size of an Apple II floppy disk, for example - then the number of files it can hold is quite limited, a simple list of names per disk is easy to use and understand, and the requirement that each file on a given disk has a unique name is not onerous to manage.
On a computer has multiple disk drives attached to it, the operating system needs to give you some way to specify which of those disks it should search when asked to work with any given file. AppleDOS (1978) did this by letting you specify explicit hardware expansion slot and drive numbers:
RUN HAMURABI,S6,D2CP/M (1974) did it by assigning each disk drive a logical letter: regardless of the details of how they were attached to the machine, the drive you booted from was A: and the other one (very typical to find only two - floppy drives were expensive!) was B:. The drive letter (complete with the colon marking it as such) could be prepended to filenames, as in
MBASIC B:HAMURABI.BASThe other thing this CP/M example shows is the fairly common pattern of allowing the names of executable files to be used as commands. Unlike AppleDOS, where you had to use DOS's built-in RUN command to make a BASIC program run from disk (or BRUN for machine language programs), CP/M would react to entry of any command name it didn't immediately recognize by trying to run a program with that name from disk. The MBASIC in that example would actually have been MBASIC.EXE, an executable file stored on drive A: that contained Microsoft's BASIC interpreter.
AppleDOS had the implicit notion of a "current" disk drive: whichever one you'd used most recently was the one it would use again until you chose a different one by adding explicit slot and/or drive number options to some other command. In CP/M, the current disk could be chosen explicitly: entering just
B:as a command would make B: the current disk. Then, assuming MBASIC.EXE was on drive A: and HAMURABI.BAS was on B: you could have used
A:MBASIC HAMURABI.BASto get the same effect as before.
These early microcomputer OS patterns echoed many of those that had been common in operating systems for larger computers of earlier decades, for much the same reason: resource constraints. But in the world of larger machines, relatively huge hard disk drives capable of containing many thousands of files were already common, and scaled-up ways of organizing them had already been developed.
A key insight of the developers of Multics (1965) was that the problem of giving user-friendly names to files on a drive was actually very similar to the problem of giving user-friendly names to the assortment of disks attached at any given time to any given system, and that a unified naming convention could be designed to cover both. This insight was incorporated into the design of Unix (1969) from which the under-the-hood parts of OS X are directly descended.
In AppleDOS, every floppy disk has a catalog: an OS-defined structure that stores a list of the filenames that exist on the disk, along with details about whereabouts on the disk those files can be found. A CP/M disk has a structure called the directory that does much the same job. In Unix, that structure is pulled to pieces: Unix disk files are numbered by the system rather than named, and given their user-visible names only by having those names associated with (linked to) their file (inode) numbers inside directory files, in and of themselves just another kind of file.
This allows a number of interesting things to be done. A disk can hold files whose names do not appear at all in its central directory file (inode 2, by convention); only in other directory files that are themselves linked (perhaps even indirectly!) from the central directory. Which means that filenames no longer need to be unique system-wide, or even unique within any given disk, in order to refer unambiguously to their underlying files: being unique within any directory they're listed in is enough.
And yes, files can be listed in and linked from more than one directory at a time. They can even be linked to more than once, under different names, inside a single directory.
Files can also exist on disks without any user-visible name assigned to them at all. A process can open a file and start working with it; then it, or even another process - perhaps a user working in a command shell - can unlink the file, an operation that on most other systems would amount to deleting it from the disk; but the original process retains full working access to that file until it closes it. Files don't actually get deleted from the disks until no directory entry links to them and no process is holding them open.
The other foundational idea is that the central directory on any given disk volume does not need to occupy a central position in the user's view of the system: rather than relying on extra command options like ,S6,D1 or device-identifying syntax like a B: prefix to distinguish one volume's central directory from another, those central directories can be referred to by filenames in exactly the same way as their own subsidiary directory files are.
Unix makes this work by supporting a mount operation allowing the central directory on any disk to overlay and substitute itself for the contents of any directory file that's already accessible, and by creating a nameless empty directory in RAM at boot time that exists for the sole purpose of allowing some disk's central directory to be mounted on it. Directories that get other disks mounted on them are called mount points, and it's conventional for them to be empty rather than having useful contents of their own; that way, mounting another disk doesn't remove anything useful from the existing filesystem.
A common metaphor for the hierarchical structure that results from files listed in directories listed in directories listed in ... listed in a central directory is that of a tree, with the central directory as its root and trunk, other directories as branches, and files as leaves. IT being a topsy turvy kind of pursuit, filesystem trees are almost always pictured upside down, with the root at the top.
A newer metaphor pictures directory files as manila folders, which can themselves be named and then nested inside other manila folders. This is compelling enough that the Mac and other systems influenced by it always refer to directory files as folders, and speak of the files listed in those directory files as being contained in those folders.
These metaphors work pretty well most of the time, but taking them too seriously will get in the way of understanding links, so don't do that.
The only character not allowed inside a Unix filename is / (slash), which serves to separate the name of a file from that of a directory the filename appears in. If a directory named foo includes an entry for a file named bar then that file can be referred to as foo/bar. If bar is itself a directory file, and includes an entry for a file named qux, then foo/bar/qux unambiguously identifies that file.
A name that includes a chain of directory name prefixes delimited by slashes is called a pathname: it explicitly specifies the path that needs to be taken through some subtree of directories to find the file it refers to. Just about any Unix facility that will accept a simple filename will also accept a pathname in its place, and pathnames can in principle involve any number of directory lookups.
Directory filenames can have a trailing / to indicate that the pathname so formed is that of a directory, in cases where this matters. So foo/bar and foo/bar/ refer to the same file, and the latter form hints that it is a directory.
Much as AppleDOS and CP/M both had a notion of a current drive, Unix has a notion of a current directory. Every process (including the command shell running in your Terminal window) always has a current directory, and files linked from it can be accessed via simple unadorned names rather than slash-containing pathnames. The current directory you will be using when you initially open a Terminal window is your home folder. This is why, when you do stuff like
chmod ug+x make-collage-specsUnix knows that the file whose mode you want to change is the one named make-collage-specs in your shell process's current directory - your home folder - rather than some other file that might be named make-collage-specs in some other folder elsewhere in the filesystem.
Any directory you can name can be made the current directory using the cd command:
ls #list the contents of the current directory, your home folder cd collage-set-1 #make collage-set-1 the current directory ls #list the contents of the current directory, now collage-set-1Unix directory files contain two standard entries that can't be removed: . is always linked to the new directory itself, and .. is a link to its parent (which will be the directory it was itself linked from when first created, or the directory containing the name of its mount point if it's actually the central directory on some disk). You typically don't see . and .. when you examine the contents of a directory; in fact you don't see any name beginning with . as these are all hidden by convention (if you add the -a option to ls it will show you all the hidden names as well).
For example, after you've set collage-set-1 as your current directory, you can return to your home folder using
cd ..Because . and .. are indeed just filenames and always link to directory files, they can be used freely inside pathnames:
ls #list the contents of your home folder ls . #same again, but this time explicitly specifying the folder to list ls ./. #same again, with an extra lookup ls ././././././././. #same again - you can keep looking up your home folder inside itself forever ls collage-set-1 #list the contents of the collage-set-1 folder inside your home folder ls collage-set-1/.. #list the contents of collage-set-1's parent folder, which is your home folder ls .. #list the contents of your home folder's parent folder ls ../.. #list the contents of your home folder's parent folder's parent folder ls ../../.. #and that of your home folder's great-grandparent's folderIf you keep on with the exercise of constructing ever-longer pathnames along the ../../../.. pattern, you will eventually find yourself looking at the contents of a directory that appears to be its own parent: you get the same contents, no matter how many more ../ prefixes you add to the pathname. This is the root directory of the entire filesystem, the one mounted over the nameless directory that the system built while booting up, and it is the only directory in the whole filesystem whose . and .. entries both link to the same thing.
As a consequence of its mount point having no filename, it was never given one either; but because it is the only nameless directory file in the whole filesystem it still has an unambiguous pathname, which is just / (an empty directory name, followed by the slash that can optionally trail any directory file's pathname).
Any pathname whose first component is named in and linked from the root directory will therefore begin with a slash, as well as optionally ending with one if it's the pathname of a directory. Pathnames that begin with a slash are called absolute pathnames, and unambiguously identify the same file no matter what the current directory is set to.
More tomorrow.
posted by flabdablet at 2:49 AM on January 15, 2018
A loose thread from yesterday: paths that don't begin with a slash (which also includes plain unadorned filenames) are called relative pathnames, and their first component will always be looked up in the current directory.
Now we come to how bash deals with command lines. I'm going to tell you some lies now by glossing over details, so take what follows with a grain of salt. It will be broadly correct.
The first word on a bash command line is the name of a command you want bash to run for you. Whenever bash runs a command, it hands copies of all the words it found on the command line (including the command name itself) to the command in the form of so-called arguments, numbered from zero.
As far as bash is concerned, a single word is anything that doesn't contain any of the characters that bash uses to separate words from one another (whitespace, semicolons, parentheses, a handful of others) or pretty much anything at all whose internals are protected by quotes; as far as the commands it runs for you are concerned, a single word is whatever bash hands over and says is a single word. So for the command line
For the command line
If a command name contains a slash - that is, if it's clearly a pathname - then bash will just find the file linked to that pathname and attempt to execute it. Absolute and relative pathnames both work.
This is how we've been invoking your scripts up to this point, and it's also how the scripts themselves have been invoking ffmpeg. The point of the ./ prefix, then, is to make bash treat the name of an executable (be it script or binary) that happens to be in the current folder as an actual pathname, without requiring it to be in some other folder in order to be so recognized.
The Unix kernel, whose job it is to resolve pathnames into files, treats the relative pathnames foo and ./foo as equivalent: both resolve to the file linked to the name foo in the current directory, though the second form does involve a redundant directory lookup operation. It's only bash that requires the ./ prefix for executables in the current directory, and then only so that it can recognize their names unambiguously as pathnames by detecting the slash inside them.
If a command name doesn't contain a slash, the rules for finding the command it specifies get more complicated.
First, bash checks to see whether a shell function (whose purpose is much like that of a subroutine in AppleSoft BASIC) has been defined with that name. If it has, then bash will treat the command line as a call to that function (rather like a GOSUB in BASIC).
If it couldn't find a function with that name, it checks its own list of builtin commands. There are quite a lot of these - cd, let, echo and read, for example, are builtins used in my scripts above.
If neither of those attempts succeeds, the last thing bash will try is finding an executable file to invoke. It does this by searching a specific collection of directories, in order, for a file with a name matching the command. By safety-driven convention, that collection includes only absolute pathnames and in particular does not include the current directory. If it did, you could run executables in the current directory without using the ./ prefix. But it doesn't so you can't.
The collection of directory pathnames to be searched for command executables is called the search path, and it's kept in a bash variable named PATH. You can see your search path with the command
By default, the contents of PATH on an OS X box look like this**:
If you enter the command ls /bin you will see a whole pile of command names listed; if you do ls /usr/bin you'll see a whole pile more. ls /usr/local/bin is unlikely to show you anything at all. So by what mad method is this stuff organized?
As with just about anything really messy, the answer boils down to history.
The first version of Unix was brought up on a relatively small machine, not massively more capable than an Apple II though it did have hard disks. The files needed to build and run the OS itself pretty much filled one of the hard disks, the one initially mounted as the root filesystem. All the executable files that the system needed in order to work once booted, including init (the very first process) and sh (the command shell itself) as well as all the commands that the shell could run, were tucked away tidily in the directory at /bin (for "binaries").
In order for the system to achieve much of anything useful, it needed more disk space and to keep it maintainable, there needed to be some separation between stuff provided by system implementors and stuff provided by system users. So it quickly became customary to mount a second disk at the mount point /usr (short for "user") where all the stuff generated by system users could be stored.
Some of that stuff was executable programs and scripts written by system users for general use by other users, and to keep that tidy, those shared executables were given their own directory - functionally similar to /bin but on the disk mounted at /usr, ending up as /usr/bin. But having to enter a /usr/bin prefix for every command stored there, or to cd /usr/bin before using those commands with a ./ prefix, was annoying enough that the shell got the PATH mechanism added on pretty early, replacing the initial implicit search of /bin alone.
One of the earliest useful abuses discovered for that mechanism involved putting scripts in /usr/bin that had the same names as commands in /bin. If you didn't like the default behaviour of some command that the system implementors had provided in /bin, it was pretty easy to write a little wrapper script and save it in /usr/bin with the same name. By setting PATH up so that /usr/bin appeared before /bin, your wrapper script, rather than the implementor-provided executable from /bin, would be what the shell always invoked to perform that command; the wrapper, in turn, could invoke the implementor-provided executable via its absolute pathname.
Unix eventually escaped from Bell Labs into the wild, taking a huge collection of useful stuff in /usr/bin with it. By the time the Berkeley folks started releasing their own version, the distinction between stuff to be put in /bin and stuff to be put in /usr/bin had got quite blurry, already becoming more a matter of tradition than anything else; the only things that absolutely had to be in /bin were such commands as the system needed in order to run the init scripts responsible for finding and mounting the secondary filesystem on /usr.
As a system administrator in charge of keeping your university's shiny new Unix installation working smoothly it was a bit risky to mess too much with the internals of what Bell or Berkeley had handed you, lest all your messings disappear when they handed you something else. So it rapidly became customary for scripts and programs developed by local administrators, as opposed to system maintainers, to be stored in /usr/local/bin and for that directory to precede the others in $PATH. As before, this allows local administrators to override the behaviour of released programs that need tweaking to suit local conditions.
And this is where you come in. You are the local administrator of your own Mac, and /usr/local/bin is the natural home for scripts and programs you develop for users of that machine (i.e. you). Of course you could just decide to put them wherever you like, and arrange for the location(s) you've chosen to appear at some suitable spot in $PATH, but using the traditional spot that's already listed in it is the PATH of least resistance.
Tomorrow: moving ffmpeg and your scripts to /usr/local/bin and compensating for the breakage that results.
*bash deals with the redirection <spec-set-1.txt as two words: the redirection operator < itself is one, and the filename spec-set-1.txt is the other. bash also handles redirections internally and does not include their specifiers in the collection of words it passes on as arguments. And in fact it allows redirection specifiers to occur anywhere in a command line, not just at the end. So really it's the first word that isn't part of a redirection spec that's the name of the command you want it to run.
**Yes, the use of :, a character that could in theory be part of a filename, to separate the components in a list of pathnames is indeed a nasty little compromise. Unix is full of nasty little compromises. See also: worse is better.
posted by flabdablet at 4:49 PM on January 15, 2018
Now we come to how bash deals with command lines. I'm going to tell you some lies now by glossing over details, so take what follows with a grain of salt. It will be broadly correct.
The first word on a bash command line is the name of a command you want bash to run for you. Whenever bash runs a command, it hands copies of all the words it found on the command line (including the command name itself) to the command in the form of so-called arguments, numbered from zero.
As far as bash is concerned, a single word is anything that doesn't contain any of the characters that bash uses to separate words from one another (whitespace, semicolons, parentheses, a handful of others) or pretty much anything at all whose internals are protected by quotes; as far as the commands it runs for you are concerned, a single word is whatever bash hands over and says is a single word. So for the command line
chmod ug+x make-collage-specschmod is the first word, ug+x is the second and make-collage-specs is the third; when chmod runs, it will see chmod as argument 0, ug+x as argument 1 and make-collage-specs as argument 2.
For the command line
./make-collages collage-set-1 <spec-set-1.txt./make-collages is the first word and collage-set-1 is the second*. So when the make-collages script runs, it will see ./make-collages as argument 0 and collage-set-1 as argument 1.
If a command name contains a slash - that is, if it's clearly a pathname - then bash will just find the file linked to that pathname and attempt to execute it. Absolute and relative pathnames both work.
This is how we've been invoking your scripts up to this point, and it's also how the scripts themselves have been invoking ffmpeg. The point of the ./ prefix, then, is to make bash treat the name of an executable (be it script or binary) that happens to be in the current folder as an actual pathname, without requiring it to be in some other folder in order to be so recognized.
The Unix kernel, whose job it is to resolve pathnames into files, treats the relative pathnames foo and ./foo as equivalent: both resolve to the file linked to the name foo in the current directory, though the second form does involve a redundant directory lookup operation. It's only bash that requires the ./ prefix for executables in the current directory, and then only so that it can recognize their names unambiguously as pathnames by detecting the slash inside them.
If a command name doesn't contain a slash, the rules for finding the command it specifies get more complicated.
First, bash checks to see whether a shell function (whose purpose is much like that of a subroutine in AppleSoft BASIC) has been defined with that name. If it has, then bash will treat the command line as a call to that function (rather like a GOSUB in BASIC).
If it couldn't find a function with that name, it checks its own list of builtin commands. There are quite a lot of these - cd, let, echo and read, for example, are builtins used in my scripts above.
If neither of those attempts succeeds, the last thing bash will try is finding an executable file to invoke. It does this by searching a specific collection of directories, in order, for a file with a name matching the command. By safety-driven convention, that collection includes only absolute pathnames and in particular does not include the current directory. If it did, you could run executables in the current directory without using the ./ prefix. But it doesn't so you can't.
The collection of directory pathnames to be searched for command executables is called the search path, and it's kept in a bash variable named PATH. You can see your search path with the command
echo $PATH(note that bash variable names are case sensitive, so echo $path and echo $Path won't work).
By default, the contents of PATH on an OS X box look like this**:
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbinSo when you did chmod ug+x make-collage-specs, the files bash searched for to get the chmod job done were /usr/local/bin/chmod, /usr/bin/chmod and /bin/chmod. The last of these actually exists, so that's the one bash used. Had it got as far as needing to search for /usr/sbin/chmod and /sbin/chmod as well, those searches would have failed even if those files actually existed: as a non-administrator user, you don't have access rights to the /usr/sbin or /sbin directories.
If you enter the command ls /bin you will see a whole pile of command names listed; if you do ls /usr/bin you'll see a whole pile more. ls /usr/local/bin is unlikely to show you anything at all. So by what mad method is this stuff organized?
As with just about anything really messy, the answer boils down to history.
The first version of Unix was brought up on a relatively small machine, not massively more capable than an Apple II though it did have hard disks. The files needed to build and run the OS itself pretty much filled one of the hard disks, the one initially mounted as the root filesystem. All the executable files that the system needed in order to work once booted, including init (the very first process) and sh (the command shell itself) as well as all the commands that the shell could run, were tucked away tidily in the directory at /bin (for "binaries").
In order for the system to achieve much of anything useful, it needed more disk space and to keep it maintainable, there needed to be some separation between stuff provided by system implementors and stuff provided by system users. So it quickly became customary to mount a second disk at the mount point /usr (short for "user") where all the stuff generated by system users could be stored.
Some of that stuff was executable programs and scripts written by system users for general use by other users, and to keep that tidy, those shared executables were given their own directory - functionally similar to /bin but on the disk mounted at /usr, ending up as /usr/bin. But having to enter a /usr/bin prefix for every command stored there, or to cd /usr/bin before using those commands with a ./ prefix, was annoying enough that the shell got the PATH mechanism added on pretty early, replacing the initial implicit search of /bin alone.
One of the earliest useful abuses discovered for that mechanism involved putting scripts in /usr/bin that had the same names as commands in /bin. If you didn't like the default behaviour of some command that the system implementors had provided in /bin, it was pretty easy to write a little wrapper script and save it in /usr/bin with the same name. By setting PATH up so that /usr/bin appeared before /bin, your wrapper script, rather than the implementor-provided executable from /bin, would be what the shell always invoked to perform that command; the wrapper, in turn, could invoke the implementor-provided executable via its absolute pathname.
Unix eventually escaped from Bell Labs into the wild, taking a huge collection of useful stuff in /usr/bin with it. By the time the Berkeley folks started releasing their own version, the distinction between stuff to be put in /bin and stuff to be put in /usr/bin had got quite blurry, already becoming more a matter of tradition than anything else; the only things that absolutely had to be in /bin were such commands as the system needed in order to run the init scripts responsible for finding and mounting the secondary filesystem on /usr.
As a system administrator in charge of keeping your university's shiny new Unix installation working smoothly it was a bit risky to mess too much with the internals of what Bell or Berkeley had handed you, lest all your messings disappear when they handed you something else. So it rapidly became customary for scripts and programs developed by local administrators, as opposed to system maintainers, to be stored in /usr/local/bin and for that directory to precede the others in $PATH. As before, this allows local administrators to override the behaviour of released programs that need tweaking to suit local conditions.
And this is where you come in. You are the local administrator of your own Mac, and /usr/local/bin is the natural home for scripts and programs you develop for users of that machine (i.e. you). Of course you could just decide to put them wherever you like, and arrange for the location(s) you've chosen to appear at some suitable spot in $PATH, but using the traditional spot that's already listed in it is the PATH of least resistance.
Tomorrow: moving ffmpeg and your scripts to /usr/local/bin and compensating for the breakage that results.
*bash deals with the redirection <spec-set-1.txt as two words: the redirection operator < itself is one, and the filename spec-set-1.txt is the other. bash also handles redirections internally and does not include their specifiers in the collection of words it passes on as arguments. And in fact it allows redirection specifiers to occur anywhere in a command line, not just at the end. So really it's the first word that isn't part of a redirection spec that's the name of the command you want it to run.
**Yes, the use of :, a character that could in theory be part of a filename, to separate the components in a list of pathnames is indeed a nasty little compromise. Unix is full of nasty little compromises. See also: worse is better.
posted by flabdablet at 4:49 PM on January 15, 2018
To make sure the way is clear for tomorrow's exercise, could you please run the following commands and post their output:
posted by flabdablet at 7:41 PM on January 15, 2018
echo $PATH ls -al /usr/local/bin
posted by flabdablet at 7:41 PM on January 15, 2018
This thread is closed to new comments.
posted by spacewrench at 10:40 PM on December 27, 2017