Hello,
This video is an intermediate-level look at special cases of filename substitution in the Linux shell.
Specifically, I will focus on wildcard matching hidden files and handling
of filenames starting with a dash.
This is not a beginner tutorial and I assume that you are already familiar with the general
concept of BASH filename expansion.
If not, strongly suggest that you watch my beginner tutorial video first, linked in the video description.
Consider the contents of my example2 directory, which has a mix of non-hidden and hidden filenames.
The ‘-a’ option of the ls command shows ‘all files’, including the default ‘.’ and ‘..’ system entries.
In this specific case, the ‘.’ refers to the ‘example2’ directory,
while ‘..’ represents the parent to ‘example2’ called ‘backup’,
which I happen to be in right now.
These two special entries are shown in each directory listing on a Linux filesystem,
where they are used to logically interconnect directories into a hierarchical tree-like structure.
The ‘.’ and ‘..’ entries can be classified as hidden, as their names start with a ‘.’
However, they are a separate category from user-created hidden files and directories,
and I will explain in a moment how this distinction matters.
First, let’s have a look at the following command line which will use a wildcard expression
to perform a common task of copying files...
This command copied a number of files from the example2 directory into /tmp/.
I used the -v (verbose) option so you can see which files were processed by the command,
but you can always use the ls command to check the command results.
This command copied 4 files, all matched by the ‘*’ wildcard inside example2 directory.
It is a common misnomer, where the ‘*’ wildcard, used on its own the way I just did,
is described as matching ‘all filenames’.
To be precise, the ‘*’ wildcard alone matches only ‘non-hidden’ filenames.
The ‘.’, if present at the beginning of a filename, is special
and normally it is not matched by wildcards under BASH,
the default Linux shell.
This is why a couple of hidden files I have in my example directory were not copied here.
Regardless of which wildcard is used – the ‘?’, the [] character class, or the ‘*,
none of them will normally match the ‘.’ at the beginning of a hidden file or directory name.
Matching hidden files using wildcard expressions is possible, however it is not handled consistently
across different shells. I am using BASH, the standard Linux shell,
so what I am about to discuss applies to that shell only, in the current version 5.
Under BASH with default settings, it is possible to wildcard-match hidden filenames
if we start the glob with a literal ‘.’.
Consider the following command...
In this case I used the [] character class followed by an ‘*’ to match user-created
hidden filenames in the example2 directory.
This command’s wildcard expression matched 6 files, which happened to be all user-created
hidden files in example2 directory.
One could be tempted to use a generic ‘.*’ expression, but this shell will include the system ‘.’ and '..' entries with it.
Let’s try it anyway and see what will happen.
What happened here exactly?
The files I wanted to copy were copied, but the ‘.’ and ‘..’ entries were also
matched by the wildcard.
Because “..” and “.” entries were matched, the command tried to copy parent
and current directories in their entirety, which is not what was intended.
Fortunately, this time nothing bad happened, since I did not include the -r recursive option with my cp command.
Unintended results of such poorly-structured command could be difficult to reverse.
Certainly the rm command with such wildcard could cause irreversible data loss,
but even a ‘cp’ command may overwrite files without prompting by default.
Using the “.*” expression is a common naive attempt to capture “all hidden filenames”.
In most cases we actually want “almost all” hidden filenames, excluding the “.” and “..” entries.
It would be very unusual to want to match the ‘.’ and ‘..’ entries using a wildcard.
In fact, you should do everything possible to exclude these entries from being matched through wildcards,
otherwise you will get undesired results as you have just seen.
As a mater of principle, I strongly advise against using the “.*” wildcard expression in BASH.
Instead, you need to craft a unique wildcard expression every time in order to match hidden filenames.
This is be relatively simple in some cases, as I did earlier using a ‘[]’ character class,
but it may be difficult or even impossible, depending on what names your hidden files have or how many there may be.
In order to match ‘almost all filenames’, excluding the ‘.’ and ‘..’ entries,
you may need to provide two wildcard expressions with a command –
one for non-hidden names and one for hidden ones.
Here is a full command, which will copy almost all files from my example2 directory into /tmp...
If you are familiar with the use of curly braces, you can also shorten the command
by employing them here like that...
Neither command is an elegant solution, but it works, and both version are fairly easy to understand.
Advanced users employ the dotglob shell option, which allows using a single wildcard expression
to match both non-hidden and hidden filenames.
Perhaps more importantly, the ‘.’ and ‘..’ default entries will no longer be matched,
simplifying the process of wildcard matching.
As demonstrated previously, it was necessary to use two wildcard expressions,
customized to specific contents of a directory in other to copy almost all files from example2 directory.
With dotglob option in place, the equivalent command will only use the ‘*’ wildcard,
resulting in a simpler command.
You can match non-hidden or hidden filenames separately with dotglob in place,
but the globs will now be different then when dotglob option was not active.
For example, if I want to match all non-hidden filenames now,
I need to exclude the leading ‘.’ using a character class...
This expression matched filenames inside example2 which do not start with a ‘.’.
Conversely, to match only hidden filenames with dotglob enabled,
the ‘.’ needs to be explicitly enclosed in square brackets.
Here I copied all hidden filenames from example2 into /tmp.
If you want to to enable the dotglob functionality permanently,
you can add it to the .bashrc shell configuration file.
I prefer to enable it only when needed.
Of course, you are not limited to the ‘*’ wildcard with dotglob enabled,
you can use the ‘?’ or the ‘[]’ as needed to build more specific and complex ambiguous pathnames.
Shell filename expansion can get complicated very quickly.
This is why I recommend to use a glob with the echo command to test it first,
and only then move on to the actual command that makes changes to the filesystem.
For example, now I would like to remove hidden files with names longer than 1 character.
I will test it first using echo...
Seeing echo output and comparing it with the actual list of files inside example2,
I can now run the remove command with confidence.
The .c file was not removed as its name is only 1 character long.
The hidden filename ‘.’ is just one of the possible cases which may require special handling
whenever we use wildcards on the Linux command line,
but even without hidden files, the filenames themselves can break a command line.
Consider the following situation...
I have the following files in my present working directory...
Let’s say I want to copy these files into the /tmp/ directory using a wildcard expression.
The command for that is very simple: cp -v * /tmp/
In this case the ‘*’ matched all 8 non-hidden filenames in my present working directory.
The command worked fine and produced an intended outcome.
The potential problem here is that starting an ambiguous name with a wildcard
or using wildcards on their own, like I just did, is actually tricky territory.
Sometimes matched filenames may result in incorrect command behaviour
or an outright data loss if unusual filenames get matched.
Particularly filenames that start with a ‘-’ cause problems.
For example, I will create and extra file called ‘-c’ here
and run the above cp command again.
This time the cp command failed, as the ‘-c’ file was understood as a command option
and not a file name.
The fact that the cp command does not have a ‘-c’ option is not that important.
What matters is that a command may not work or may work incorrectly
if a glob matches a file with a name that starts with a ‘-’.
The troublesome nature of dash-leading filenames is not limited to only commands with wilcard expressions.
Let me show you such scenario.
I will try to remove the ‘-c’ file now.
You will notice that the rm command error provides me me with a hint how to overcome this problem.
I can add a ./prefix before the filename with a leading dash.
Now the rm command worked as expected and the ‘-c’ file has been removed.
Adding a ./prefix solved the problem and this prefix is the recommended workaround for filenames with leading dashes.
As another example, now I would like to create two files with dash-leading names...
You know already that this will not work.
I need to add the ./ prefix to each of these unusual filenames.
There is an alternate method to handle filenames with dash-leading filenames,
which relies on adding a double dash to the command line.
I used that method earlier to create the original ‘-c’ file.
A double dash tells the command that anything to the right of that double dash
is not to be interpreted as a command option.
I will use that method to remove the -c file now.
Either method is fine, but the ./ prefix is preferred,
as some command lines may have wildcards in the middle or the command used may not support the double dash.
Let me show you one more example of how filenames with leading dashes can cause trouble.
Let’s say I would like to interactively clean up my current directory using the ‘rm’ command and the ‘*’ wildcard,
forgetting to use the ./ prefix.
Normally the rm command with -i option would prompt me for each filename whether I want to remove it.
Let’s see what happens in this specific situation...
Ironically, with the exception of the ‘-f’ file, everything inside my present directory is gone.
In this command, the ‘-f’ filename was interpreted as an option by the ‘rm’ command,
where it means to force the remove.
The ‘-f’ option overrides interactive options such as ‘-i’ for rm command and this is why I was not prompted.
You need to eliminate the possibility of matching the leading ‘-’ in file names when using shell wildcards.
While you may employ good file naming rules and never use such filenames,
they may exist on your filesystem anyway,
created by other, less-knowledgeable users or by poorly written programs.
Fortunately, dash-leading filenames only cause problems in relative pathnames,
and only when referred to directly or wildcard-matched in present working directory.
For example, the ./ prefix does not need be used in the following command...
Here the wildcard is at the end of a relative-to-home pathname,
so there is no danger of filenames being interpreted as command options.
Lets do one more example...
This command uses a relative pathname, but the glob does not start with a wildcard,
so ./ prefix is not used.
Expression such as ‘c*’ cannot match a file with a leading dash in their name.
The next video in this series discusses wildcard-matching regular files vs. directories,
but before you continue there, make sure to give this video thumbs up and subscribe if you are not a subscriber already.
Thanks for watching!