{"id":6457,"date":"2019-05-14T09:45:57","date_gmt":"2019-05-14T14:45:57","guid":{"rendered":"https:\/\/www.poweradmin.com\/blog\/?p=6457"},"modified":"2019-05-10T16:56:59","modified_gmt":"2019-05-10T21:56:59","slug":"identifying-duplicate-files-in-linux","status":"publish","type":"post","link":"https:\/\/www.poweradmin.com\/blog\/identifying-duplicate-files-in-linux\/","title":{"rendered":"Identifying Duplicate Files in Linux"},"content":{"rendered":"

By Des Nnochiri<\/span><\/strong><\/span><\/span><\/p>\n

\u00a0<\/p>\n

Keeping redundant copies of essential files and programs can assist in recovery when system glitches or other incidents occur. However, duplicate files also hold the potential to confuse matters and introduce errors. It\u2019s possible to have too much of a good thing, so keeping track of these duplicates is always a good idea.<\/span><\/p>\n

\u00a0<\/span><\/p>\n

Why Files Sometimes Multiply<\/span><\/h2>\n

\u00a0<\/span><\/p>\n

If you\u2019re a music, video, or graphics enthusiast, you\u2019ll understand how easy it can be for files with similar names or similar content to pile up on your storage drives. What might surprise you is that this kind of download or file-saving activity isn\u2019t the biggest cause of file replication on most systems.<\/span><\/p>\n

\u00a0<\/span><\/p>\n

Operating systems and application software are actually the biggest culprits. These programs often create duplicate files for entirely legitimate purposes.<\/span><\/p>\n

\u00a0<\/span><\/p>\n

Some programs are designed with built-in mechanisms to protect them from the malfunctioning of other, related software. One way of doing this is for an application to install its own local copy of a shared library or support files. This prevents the software from losing access to essential files or code if another program that uses the same library is uninstalled or becomes hopelessly corrupted.<\/span><\/p>\n

\u00a0<\/span><\/p>\n

Operating systems typically offer users the option of rolling back their installation to a previous version if the current set-up becomes damaged or corrupted by a virus attack or some other circumstance. This requires entire sets of OS files to be held in a designated portion of your hard drive, many of which will be exact duplicates of files on the currently active operating system.<\/span><\/p>\n

\u00a0<\/span><\/p>\n

Similarly, when some applications update themselves, they\u2019ll save copies of any files that have been changed in case the upgrade goes wrong and the previous installation needs to be restored.<\/span><\/p>\n

\u00a0<\/span><\/p>\n

When More Means Less<\/span><\/h2>\n

\u00a0<\/span><\/p>\n

Besides their potential to consume large quantities of valuable storage space, duplicate files can sometimes create problems affecting usability or operational matters.<\/span><\/p>\n

\u00a0<\/span><\/p>\n

If, for example, the information making up a duplicate file set becomes corrupted, applications depending on this data for their functionality may exhibit erratic or even destructive behaviors. Duplicate files may slow down or complicate system-wide operations, such as file indexing or database sorts and searches.<\/span><\/p>\n

\u00a0<\/span><\/p>\n

The Issue of Hard and Symbolic Links<\/span><\/h2>\n

\u00a0<\/span><\/p>\n

In a Linux environment, the issue of duplicate files may be even more pronounced. That\u2019s because two or more entities on a Linux drive can have different names yet still be recognized by the system as identical versions of the same file.<\/span><\/p>\n

\u00a0<\/span><\/p>\n

Files sharing the same disk space in a Linux installation will share the same inode, which is the data structure that stores all the information about a file except its name and its content. Two or more files with a common inode may have different names and file system locations, yet they\u2019ll still share the same content, ownership, permissions, and other characteristics.<\/span><\/p>\n

\u00a0<\/span><\/p>\n

Files like these are known as hard links. They operate in contrast to symbolic links, which point to other files by containing their names. In her <\/span>analysis of this subject for NetworkWorld<\/span><\/a>, Sandra Henry-Stocker points out that symbolic links are easy to identify in a file listing by the \u201cl\u201d in the first position and the -> symbol that refers to the file being referenced:<\/span><\/p>\n

\u00a0<\/span><\/p>\n

$ ls -l my*<\/span><\/p>\n

-rw-r\u2013r\u2013 4 shs shs\u00a0\u00a0 228 Apr 12 19:37 myfile<\/span><\/p>\n

lrwxrwxrwx 1 shs shs\u00a0\u00a0\u00a0\u00a0 6 Apr 15 11:18 myref -> myfile<\/span><\/p>\n

-rw-r\u2013r\u2013 4 shs shs\u00a0\u00a0 228 Apr 12 19:37 mytwin<\/span><\/p>\n

\u00a0<\/span><\/p>\n

Finding Duplicate Files in Linux<\/span><\/h2>\n

\u00a0<\/span><\/p>\n

To identify the hard links in a single directory, you can list the files using the ls -i command, and sort them by inode number. The inode numbers will appear in the first column of this type of output. <\/span><\/p>\n

\u00a0<\/p>\n

\"\"<\/a><\/p>\n

\u00a0<\/p>\n

(Image source: <\/span>NetworkWorld<\/span><\/a>)<\/span><\/p>\n

\u00a0<\/span><\/p>\n

Rather than scanning reams of output for identical node numbers, it\u2019s possible to find out if one particular file is hard-linked to another. In this case, you should use the -samefile option of the find command. The starting location that you provide to the find command will dictate how much of the file system is scanned for matches. In the example below, the search takes place in the current directory and its subdirectories.<\/span><\/p>\n

\u00a0<\/p>\n

\"\"<\/a><\/span><\/p>\n

\u00a0<\/p>\n

(Image source: <\/span>NetworkWorld<\/span><\/a>)<\/span><\/p>\n

\u00a0<\/span><\/p>\n

To find duplicate files and drill down to more detail on their characteristics, you can use the find command\u2019s -ls option:<\/span><\/p>\n

\u00a0<\/p>\n

\"\"<\/a><\/p>\n

\u00a0<\/p>\n

(Image source: <\/span>NetworkWorld<\/span><\/a>)<\/span><\/p>\n

\u00a0<\/span><\/p>\n

The first column of output displays the inode number. It then lists the file permissions, links, owner, file size, date information, and the names of any files that refer to the same disk content. <\/span><\/p>\n

\u00a0<\/span><\/p>\n

To locate all instances of hard links in a single directory, you can run a script like this:<\/span><\/p>\n

\u00a0<\/span><\/p>\n

\"\"<\/a><\/p>\n

\u00a0<\/p>\n

(Image source: <\/span>NetworkWorld<\/span><\/a>)<\/span><\/p>\n

\u00a0<\/span><\/p>\n

Note that using this method to scan for Linux duplicate files which contain the same content but don\u2019t share inodes (i.e., simple file copies) takes considerably more time and effort.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

By Des Nnochiri \u00a0 Keeping redundant copies of essential files and programs can assist in recovery when system glitches or other incidents occur. However, duplicate files also hold the potential to confuse matters and introduce errors. It\u2019s possible to have too much of a good thing, so keeping track of these duplicates is always a […]<\/p>\n","protected":false},"author":15,"featured_media":6469,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,447,9],"tags":[23,294,269,697,699,698,693,700,704,705,398,526,538,703,702,701],"class_list":["post-6457","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-general-it","category-linux","category-technical","tag-data","tag-data-sharing","tag-data-storage","tag-duplicate-files","tag-file-corruption","tag-file-replication","tag-file-sharing","tag-hard-link","tag-inode","tag-inode-number","tag-linux","tag-linux-command","tag-linux-directory","tag-linux-drive","tag-linux-environment","tag-symbolic-link"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.poweradmin.com\/blog\/wp-json\/wp\/v2\/posts\/6457","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.poweradmin.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.poweradmin.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.poweradmin.com\/blog\/wp-json\/wp\/v2\/users\/15"}],"replies":[{"embeddable":true,"href":"https:\/\/www.poweradmin.com\/blog\/wp-json\/wp\/v2\/comments?post=6457"}],"version-history":[{"count":5,"href":"https:\/\/www.poweradmin.com\/blog\/wp-json\/wp\/v2\/posts\/6457\/revisions"}],"predecessor-version":[{"id":6493,"href":"https:\/\/www.poweradmin.com\/blog\/wp-json\/wp\/v2\/posts\/6457\/revisions\/6493"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.poweradmin.com\/blog\/wp-json\/wp\/v2\/media\/6469"}],"wp:attachment":[{"href":"https:\/\/www.poweradmin.com\/blog\/wp-json\/wp\/v2\/media?parent=6457"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.poweradmin.com\/blog\/wp-json\/wp\/v2\/categories?post=6457"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.poweradmin.com\/blog\/wp-json\/wp\/v2\/tags?post=6457"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}