Repository navigation
Following symlinks on external storage causes loops #20197
Description
How to use GitHub
- Please use the 👍 reaction to show that you are affected by the same issue.
- Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
- Subscribe to receive notifications on status change and new comments.
Steps to reproduce
Each of the approx. 600 LDAP users has symbolic links in his home directory, depending on the group membership, as follows:
+all -> /home/all
+allteachers -> /home/groups/TEACHERS
+classes -> /home/classes
+groups -> /home/groups
+software -> /home/software
When a user uses the Nextcloud (desktop) app to synchronize external shares, there is an endless loop.
Also the Nextcloud cronjob runs without end and fills the table oc_filecache by scanning the same files over and over again. The serverload is increased by the php nextcloud cronjob and mysql.
For example a user teacher1 within the group TEACHERS accesses:
/home/teacher1/+classes/1A/student1/+groups/TEACHERS...
...
/home/teacher1/+classes/12A/student12/+groups/TEACHERS...
and also the same share:
/home/groups/TEACHERS...
etc...
Expected behaviour
- Symbolic links on external shares should not be followed, or it should be configurable.
- If several users share the same external directory, these files should only be indexed once by occ files:scan to keep the size of the database small and to reduce the server load
Actual behaviour
- Symbolic links on the external drives are followed and the same files are synchronized/indexed again and again
- The same files are indexed by the cronjob, I guess "occ files:scan" for all users, although all users, except the home directory, have the same files.
Server configuration
Operating system:
Linux cloud 4.9.0-12-amd64 #1 SMP Debian 4.9.210-1 (2020-01-20) x86_64 GNU/Linux
Web server:
apache 2.4.38-3+deb10u3
Database:
mariadb-10.3
PHP version:
7.3.14-1~deb10u1
Nextcloud version: (see Nextcloud admin page)
18.0.3
Updated from an older Nextcloud/ownCloud or fresh install:
Fresh install
Where did you install Nextcloud from:
https://download.nextcloud.com/server/releases/nextcloud-18.0.3.zip
Signing status:
Signing status
No errors have been found.List of activated apps:
App list
Enabled: - accessibility: 1.4.0 - activity: 2.11.0 - bruteforcesettings: 1.5.0 - calendar: 2.0.2 - cloud_federation_api: 1.1.0 - comments: 1.8.0 - contacts: 3.2.0 - dav: 1.14.0 - deck: 0.8.0 - federatedfilesharing: 1.8.0 - federation: 1.8.0 - files: 1.13.1 - files_antivirus: 2.2.1 - files_external: 1.9.0 - files_pdfviewer: 1.7.0 - files_rightclick: 0.15.2 - files_sharing: 1.10.1 - files_trashbin: 1.8.0 - files_versions: 1.11.0 - files_videoplayer: 1.7.0 - firstrunwizard: 2.7.0 - logreader: 2.3.0 - lookup_server_connector: 1.6.0 - nextcloud_announcements: 1.7.0 - notes: 3.2.0 - notifications: 2.6.0 - oauth2: 1.6.0 - onlyoffice: 4.1.4 - password_policy: 1.8.0 - polls: 1.3.0 - privacy: 1.2.0 - provisioning_api: 1.8.0 - serverinfo: 1.8.0 - settings: 1.0.0 - sharebymail: 1.8.0 - spreed: 8.0.5 - support: 1.1.0 - survey_client: 1.6.0 - systemtags: 1.8.0 - terms_of_service: 1.4.0 - text: 2.0.0 - theming: 1.9.0 - twofactor_backupcodes: 1.7.0 - updatenotification: 1.8.0 - user_ldap: 1.8.0 - viewer: 1.2.0 - workflowengine: 2.0.0 Disabled: - admin_audit - encryption - photos - recommendationsNextcloud configuration:
Config report
{ "system": { "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "***REMOVED SENSITIVE VALUE***" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "18.0.3.0", "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "mysql.utf8mb4": true, "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "memcache.locking": "\\OC\\Memcache\\Redis", "memcache.local": "\\OC\\Memcache\\Redis", "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": 6379 }, "mail_smtpmode": "smtp", "mail_sendmailmode": "smtp", "skeletondirectory": "", "ldapIgnoreNamingRules": false, "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory", "ldapUserCleanupInterval": 30, "loglevel": 2, "maintenance": false, "upgrade.disable-web": true, "theme": "", "lost_password_link": "disabled" } }Are you using external storage, if yes which one: local/smb/sftp/...
Details
| 1 | /all | SMB / CIFS | Log-in credentials, save in database | host: "pdc-server", share: "all", root: "", domain: "", show_hidden: false, timeout: "" | | All | | | 2 | /allteachers | SMB / CIFS | Log-in credentials, save in database | host: "pdc-server", share: "allteachers", root: "", domain: "", show_hidden: false, timeout: "" | | | TEACHERS | | 3 | /groups | SMB / CIFS | Log-in credentials, save in database | host: "pdc-server", share: "groups", root: "", domain: "", show_hidden: false, timeout: "" | | All | | | 4 | /software | SMB / CIFS | Log-in credentials, save in database | host: "pdc-server", share: "software", root: "", domain: "", show_hidden: false, timeout: "" | | All | | | 5 | /homes | SFTP | Log-in credentials, save in database | host: "schooladmin", root: "\/home\/teachers\/$user" | | | TEACHERS | | 6 | /homes | SFTP | Log-in credentials, save in database | host: "schooladmin", root: "\/home\/sysadmins\/$user" | | | SYSADMINS | | 7 | /homes | SFTP | Log-in credentials, save in database | host: "schooladmin", root: "\/home\/students\/$user" | | | STUDENTS |Are you using encryption: yes/no
no
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
LDAP
LDAP configuration (delete this part if not used)
LDAP config
+-------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Configuration | s01 | +-------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | hasMemberOfFilterSupport | 1 | | homeFolderNamingRule | attr:uid | | lastJpegPhotoLookup | 0 | | ldapAgentName | | | ldapAgentPassword | *** | | ldapAttributesForGroupSearch | | | ldapAttributesForUserSearch | | | ldapBackupHost | | | ldapBackupPort | | | ldapBase | dc=schule,dc=edu | | ldapBaseGroups | ou=group,dc=schule,dc=edu | | ldapBaseUsers | ou=people,dc=schule,dc=edu | | ldapCacheTTL | 600 | | ldapConfigurationActive | 1 | | ldapDefaultPPolicyDN | | | ldapDynamicGroupMemberURL | | | ldapEmailAttribute | | | ldapExperiencedAdmin | 0 | | ldapExpertUUIDGroupAttr | gidNumber | | ldapExpertUUIDUserAttr | uid | | ldapExpertUsernameAttr | uid | | ldapExtStorageHomeAttribute | uid | | ldapGidNumber | gidNumber | | ldapGroupDisplayName | cn | | ldapGroupFilter | (&(|(objectclass=SchoolGroup))(!(|(cn=ADMINISTRATION)(cn=DOMAINUSERS)(cn=TEMPLATES)(cn=WORKSTATIONS)))) | | ldapGroupFilterGroups | ADMINISTRATION;DOMAINUSERS;STUDENTS;TEMPLATES;WORKSTATIONS | | ldapGroupFilterMode | 1 | | ldapGroupFilterObjectclass | SchoolGroup | | ldapGroupMemberAssocAttr | member | | ldapHost | schooladmin | | ldapIgnoreNamingRules | | | ldapLoginFilter | (&(&(|(objectclass=SchoolAccount))(|(memberof=cn=STUDENTS,ou=group,dc=schule,dc=edu)(memberof=cn=SYSADMINS,ou=group,dc=schule,dc=edu)(memberof=cn=TEACHERS,ou=group,dc=schule,dc=edu)))(uid=%uid)) | | ldapLoginFilterAttributes | | | ldapLoginFilterEmail | 0 | | ldapLoginFilterMode | 0 | | ldapLoginFilterUsername | 1 | | ldapNestedGroups | 1 | | ldapOverrideMainServer | | | ldapPagingSize | 500 | | ldapPort | 389 | | ldapQuotaAttribute | | | ldapQuotaDefault | | | ldapTLS | 0 | | ldapUserAvatarRule | default | | ldapUserDisplayName | addressBookCN | | ldapUserDisplayName2 | | | ldapUserFilter | (&(|(objectclass=SchoolAccount))(|(memberof=cn=STUDENTS,ou=group,dc=schule,dc=edu)(memberof=cn=SYSADMINS,ou=group,dc=schule,dc=edu)(memberof=cn=TEACHERS,ou=group,dc=schule,dc=edu))) | | ldapUserFilterGroups | STUDENTS;SYSADMINS;TEACHERS | | ldapUserFilterMode | 0 | | ldapUserFilterObjectclass | SchoolAccount | | ldapUuidGroupAttribute | auto | | ldapUuidUserAttribute | auto | | turnOffCertCheck | 1 | | turnOnPasswordChange | 0 | | useMemberOfToDetectMembership | 1 | +-------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+Logs
Web server access log
Web server acces log
xxx.xxx.xxx.xxx - dub [26/Mar/2020:17:15:59 +0100] "PROPFIND /remote.php/dav/files/dub/homes/+classes/10AM/user1/+groups/TEACHERS/test.txt HTTP/1.1" 207 105075 "-" "Mozilla/5.0 (Windows) mirall/2.6.4stable-Win64 (build 20200303) (Nextcloud)"
Activity
flammekueche commentedon Jun 16, 2020
Same problem on NC 19.0.0 (docker version).
PHP 7.4.7, Mariadb 10.4.13
Im not using LDAP, just a symlnks outside data is enough to cause the loop.
flammekueche commentedon Jun 25, 2020
Huge increase of sql requests respond time, as well as database size.
Everywhere a symlnks exists, entries with path included "//" are recursively added.
SELECT storage,path,name FROM oc_filecache WHERE path LIKE '%//%' limit 1000;
+---------+----------------------------------------------------------------+---------------------------+
| storage | path | name |
+---------+----------------------------------------------------------------+---------------------------+
| 3 | files/Documents/Synology/Adblock//ad-blocker | ad-blocker |
| 3 | files/Documents/Synology//spamassassin | spamassassin |
| 3 | files/Documents/Synology//diskstation | diskstation |
| 3 | files/Documents/Synology//synoreport | synoreport |
...
| 3 | files/Documents/Synology/owncloud | owncloud |
| 3 | files/Documents/Synology//owncloud | owncloud |
| 3 | files/Documents/Synology///owncloud | owncloud |
| 3 | files/Documents/Synology////owncloud | owncloud |
| 3 | files/Documents/Synology/////owncloud | owncloud |
| 3 | files/Documents/Synology//////owncloud | owncloud |
| 3 | files/Documents/Synology///////owncloud | owncloud |
...
+---------+----------------------------------------------------------------+---------------------------+
Temporary workaround:
DELETE FROM oc_filecache WHERE path LIKE '%//%';
adepertat commentedon Jul 10, 2020
I have the exact same symptoms except for the fact that I do not use symlinks. Rather my files are on a an NFS mount.
Found the same workaround as @flammekueche but it is unsatisfactory because the tablespace grows anyway. I can recover the space with
ALTER TABLE oc_filecache FORCE;but if I'm not careful, the table will grow too large, the partition will fill up, and I am forced toTRUNCATE oc_filecacheto recover the space.NextCloud 19.0.0 official Docker image / MariaDB 10.4.13
adepertat commentedon Jul 20, 2020
Updated to 19.0.1 today, issue persists.
pohuing commentedon Oct 18, 2020
This also has the funny error that the scanner eventually fails and produces a log. Since we're dealing with infinite recursion here we get a logfile that eats up all the drive's space and then just stops. I've got 40+GB of trying to scan one directory right now.
3v1n0 commentedon Oct 20, 2020
Got a 122GB oc_filecache db because of this... And those are relative links that could be followed properly.
szaimen commentedon Jun 17, 2021
Is this Issue still valid in NC21.0.2? If not, please close this issue. Thanks! :)
PetitVerglas commentedon Apr 17, 2025
This issue is still valid with NC 30.0.6
fix(files): skip directory symlinks that loop back onto the scanned path