﻿Hello!
In this video I will teach you about filename substitution.
Specifically, I will show you how to use basic wildcard meta-characters to build ambiguous filenames,
which refer to multiple files and directories at once in the Linux shell.
Here are the contents of my current directory which I will refer to in this video.
Regular files are shown in white, while directory names are blue.
Let me start by reviewing some of the terminology I will use throughout the video.
With the exception of the root directory,
each file and directory on a Linux filesystem has a name.
While regular files serve a different purpose than directory files,
their naming follows the same rules.
I like to use the generic term filename, which refers to both regular file and directory names.
Sometimes several filenames are put together to form a path (or a pathname),
where filenames are separated by forward slashes,
allowing us to access files and directories outside of the present working directory.
Commands which operate on filenames or pathnames,
receive them as parameters, provided as a space-separated list.
For example, if I would like to duplicate customer0 and customer2
files into the /tmp directory, I can do it using the cp command...
In this case, the cp command received 3 parameters (sometimes called arguments) –
2 source filenames and 1 destination directory.
The cp command (or most of the other common commands for that matter)
can handle an arbitrary number of source filenames.
Typing them by hand would be rather tedious if there were more than a couple of them.
The shell has a feature where wildcard meta-characters can
be used in an ambiguous filename to refer to multiple filenames at once.
The process of defining these wildcard patterns and using them to match existing filenames
is called globbing, while an ambiguous filename is generically called a glob.
I use the term wildcard expressions, ambiguous filenames and globs interchangeably.
The first filename-matching meta-character is the ‘?’, which matches any single character.
In a simple scenario, we can combine it with a literal match.
For example, here is how I would copy files with names that start with customer,
followed by exactly one character, into the processed/ directory:
I used the ‘-v’ verbose option with the cp command, so you can see which files were matched
and ultimately copied, but you can always use the ls command to verify the command outcome.
In this case, the shell matched multiple filenames in my present working directory, where the ‘?’
wildcard matched digits in their name such as ‘0‘ through ‘9’ and lower-case letter ‘s’.
Wildcard matching does not distinguish between regular files and directories.
In this case there were 9 regular files and 1 directory matched,
but only the regular files were copied. The customers directory was not copied,
as the cp command did not have the ‘-r’ recursive option, which is required to copy directories.
There is a way to distinguish between regular files and directories in the filename substitution,
but this requires a bit more knowledge on your end, so I will explain that in the next video.
While the substitution process is transparent to us,
we can actually see what the above command actually looks like,
after the shell performs the filename substitution
if we prefix the entire command line with echo, the shell’s print command.
Using the echo command is a common method to test or preview a command without actually running it,
which gives us a chance to fix potential problems
and diminish the risk of accidental data loss if we use globbing.
You can certainly appreciate that I did not have to type such long list of files,
but only the short glob.
Filename substitution is a feature of the shell and for that reason,
it will work with any command.
As long as a match is possible, the command does not see the wildcard expression,
only the files that were matched to it.
We can use multiple ‘?’ wildcards in the same glob.
For example, now I will move all files named customer followed by two characters.
This time 7 regular files were matched.
The first ‘?’ matched the digits ‘0’, ‘1’ and ‘2’ and ‘6’,
while the second ‘?’ matched letters ‘x’, ‘y’ and ‘z’ and a digit ‘2’.
The ability to match any character using the ‘?’ is very handy,
but sometimes we want to be more specific
and limit the matches to a narrower group of specific characters.
For that, we have the [ ] character class.
Just like the ‘?’, the [ ] class will match a single character,
but this time out of the list provided within the [ ].
We can put as many characters inside the square brackets as needed,
either on their own or as dash-separated ranges of consecutive digits or letters of the alphabet.
For example, now I will copy the following files into the /tmp directory:
customer1, customer 3, as well as files named consecutively customer6 through customer9.
I have seen some Internet tutorials on this topic which separate the [] items with commas.
Unless you are matching a literal comma (in which case you would have only one inside the []),
separating list items with commas would be incorrect.
This wildcard expression matched 5 files.
An ambiguous filename is like a wish list and the shell will do its best to satisfy it,
but in some cases there may not be even one match.
For example, the following pattern will not have any matches...
In this case the shell looked for filenames starting with customer, followed by a digit, followed by ‘.txt’.
None of the files in my present directory satisfy such requirements.
As no filenames matched, the shell did not make any substitution,
and the cp command received the ambiguous filename untouched, including the square brackets.
This is an important finding!
Ambiguous filenames are only substituted if possible,
otherwise the shell passes on the wildcard expression unchanged to the command.
If the substitution does not happen, the command will try to use the ambiguous filename literally,
which can become a problem.
In this case trying to copy a file literally named ‘customer[0-9].txt’,
which ultimately resulted in the cp command producing a ‘No such file’ error message.
This time the failed match did not cause any real problem,
as no files were created, changed or lost in the process.
The ‘cp’ command informed me that there was a problem
and I have a chance to fix the pattern and run the command again.
In other cases a failed substitution may not result in a visible error message,
possibly silently making unwanted changes to the filesystem.
Let’s say I would like to update the date stamps of files with names that start with ‘customer’
following by one more character but mistype ‘costomer’.
There were no matching filenames, so the glob was not substituted.
The touch command worked with what was provided literally,
making a file actually named “costomer?”.
Unless I checked using the ls command, I would not even know that this has happened.
The date stamp was not updated for the files I wanted to update
and now I have an extra file with a misspelled name.
I need to delete it...
The unwanted file has a special character in it - the '?',
so it need to be quoted using the backslash.
I explained in detail in my previous videos how special characters in filenames can cause problems
and how to use quoting to avoid them.
The shell has a number of environmental settings which control the wildcard substitution process.
One of the settings which I like to use is the ‘failglob’.
Here is how failglob environment setting is enabled...
This setting, when enabled, instructs the shell to produce an error message when
there is no match to an ambiguous filename and the actual command will never be run.
Let me re-run the previous command so you can see how the shell behaves now.
This time I got a ‘no match’ error from the shell.
The touch command was never run and therefore no unwanted file was created.
The ‘shopt’ command stands for ‘shell option’
and it is used to modify the default behaviour of the shell.
Environmental settings such as this one stay in effect until they are unset
or the terminal is closed.
You can make them permanent by adding the set command to the hidden .bashrc shell configuration file.
You can edit the .bashrc file using any text editor, for example nano.
I suggest placing the command just before or after any existing shopt commands.
You can also make a comment about it by adding a line
that starts with ‘#’ (hash or pound sign) before it, explaining what the setting does.
If you have not used nano before, you press Ctrl+x when you are done with the file changes.
Confirm the changes by pressing ‘y’ and Enter to exit the editor.
Sometimes it may be easier to specify which characters we do not want to match.
This is accomplished by providing the ‘!’ at the beginning of the list inside the square brackets,
which in this context means ‘not’.
This exclusion affects all character listed after the ‘!’ inside the [].
Let’s say I would like to remove all customer... files except customer1 from the /tmp/ directory.
I can use the [] for that with the ! exclusion.
Six customer files were removed with customer1 file remaining.
Multiple sets of square brackets can be used together,
each matching single characters out of the characters listed within it.
For example, the following will duplicate files from the processed directory into my current directory,
with names followed by ‘0’ or ‘2’, not followed by ‘y’.
In this case three files were matched.
Notably, files 'customer0y' and 'customer2y' did not match and they were not copied.
While the first square bracket matched them partially, the ‘y’ at the end of their names ruled them out.
Whether you use a single set of square brackets or multiples,
what is not immediately obvious to beginners, is that we need a minimum of 2 characters inside a [] group.
Beginners often do not realize that and make wildcard expressions such as this one...
This wildcard expression works fine and produces valid matches,
as I can see a matched subset of files inside of processed/ directory,
but the second set of square brackets does not do anything, as it can only match a ‘y’.
In this case a literal match of ‘y’ should be used.
Same results, but a simpler command. Simpler is better!
Different wildcards can be used in the same glob.
For example, now I will copy all files named customer,
followed by either ‘1’ or ‘2’, followed by one more character into /tmp.
This time 3 files were matched and copied where the ‘?’ matched letters ‘x’ and ‘y’.
The ‘?’ is not in square brackets here.
If I did that, the [] would act as quotes, turning off the wildcard property of the ‘?’
and the ‘?’ would be matched literally in such case.
There is often a need to wildcard-match more than just one character.
For that we use the ‘*’, which will match multiples of any characters.
For example, I have several directories with names starting with invoices in my present working directory.
Here is how I would move all of them into the processed directory...
In this case it was sufficient to start with a literal match of lower-case letter ‘i’
and then add the ‘*’ wildcard,
as there were no other filenames starting with lower-case ‘i’ in my current directory.
Such expression reads as:
filenames that start with lower-case letter ‘i’, possibly followed followed by more characters.
If there was a file name simply named “i”, it would be matched here as well.
We can combine the ‘*’ wildcard with other wildcards in the same glob.
For example, here is how I would copy filenames into /tmp/ directory, starting with ‘c’ and ending with a digit...
The matches include the ‘c5’ file, which has no characters between ‘c’ and ‘5’.
Those non-existent characters were matched by the ‘*’ wildcard.
The ‘* wildcard can match nothing when combined with literal matches or other wildcards.
That’s it for the basics of file filename matching, but there is so much more!
Go ahead and check out the more advanced video, but before you do,
please hit the like button and subscribe!
It takes only 2 seconds, it costs you nothing but it lets me know that what I do makes a difference for you.
Thanks for watching!