Repository navigation
Extremely long-named nested folders corrupt filecache.path on move #5477
Description
Steps to reproduce
- (Works both on shared folders and personal folders. However, shared folders seem to make the bug easier to trigger.)
0.1. (The easiest cue that the bug triggered (database file descriptors corrupted) is that the files shown on the web interface are now duplicated, and show their full path instead of filenames.) - Create a new subdirectory with an extremely long name. (Having a space or internationalized character in its name also helps trigger that bug)
1.1. Repeat step 1 until you have at least 3-4 nested directories - Create a new nested directory named "target" and one named "subdir"
- In target, create a few dummy text files. Use Lorem/Bacon Ipsum for payload.
- Move folder target into subdir
4.1. (Sometimes that is enough to trigger the bug. Have a look (on the web interface) at the contents of the target directory) - Move folder target back one level (out of subdir)
5.1. (This usually triggers the bug. Have a look (on the web interface) at the contents of the target directory) - Add a nested directory (1) and repeat steps 4-5.
6.1. Repeat until file descriptors break in database.
Expected behaviour
Moving folder moves with its contents unchanged. Mostly.
(Like we'd expect the mv command to do.)
Actual behaviour
Moving folders in a deeply-nested subfolder (as is common in an enterprise environment) can cause an error when updating the filecache.path field, which breaks the link between the database entry for a given file and the file on the filesystem. (File on filesystem is where we expect it to be.)
When browsing into the subfolder with the web interface, it looks like the system re-scans that folder on the filesystem and re-recreates an entry in the database to match the orphaned file on the filesystem. (Or something, the two database entries are identical but for fileid, path and path_hash)
On the web interface, these two entries for the same file will show a duplicated file, and will print the full filepath instead of the filename.
(It may also serve as root cause for #5449 )
Server configuration
Operating system:
Linux sleipnir 4.9.0-2-amd64 #1 SMP Debian 4.9.18-1 (2017-03-30) x86_64 GNU/Linux
Web server:
NGINX 1.10.3-1
Database:
mariadb 10.1.23-9+deb9u1
PHP version:
PHP7.0.19-1 FPM
Nextcloud version: (see Nextcloud admin page)
12.0.0.29
Updated from an older Nextcloud/ownCloud or fresh install:
Updated from NC 11 to NC12 on release date.
Where did you install Nextcloud from:
Updater Module in NC
Signing status:
Signing status
No errors have been found.List of activated apps:
App list
Enabled: - activity: 2.5.2 - bruteforcesettings: 1.0.2 - comments: 1.2.0 - dav: 1.3.0 - federatedfilesharing: 1.2.0 - federation: 1.2.0 - files: 1.7.2 - files_markdown: 1.0.1 - files_pdfviewer: 1.1.1 - files_sharing: 1.4.0 - files_texteditor: 2.4.1 - files_trashbin: 1.2.0 - files_versions: 1.5.0 - files_videoplayer: 1.1.0 - firstrunwizard: 2.1 - gallery: 17.0.0 - logreader: 2.0.0 - lookup_server_connector: 1.0.0 - nextcloud_announcements: 1.1 - notifications: 2.0.0 - oauth2: 1.0.5 - password_policy: 1.2.2 - provisioning_api: 1.2.0 - serverinfo: 1.2.0 - sharebymail: 1.2.0 - survey_client: 1.0.0 - systemtags: 1.2.0 - theming: 1.3.0 - twofactor_backupcodes: 1.1.1 - updatenotification: 1.2.0 - user_ldap: 1.2.1 - workflowengine: 1.2.0 Disabled: - admin_audit - encryption - files_external - user_externalNextcloud configuration:
Config report
{ "system": { "updatechecker": false, "instanceid": "oc6n6tqq77xl", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "tsunami.ageg.ca", "nextcloud.ageg.ca" ], "datadirectory": "\/srv\/owncloud\/data", "overwrite.cli.url": "https:\/\/tsunami.ageg.ca", "dbhost": "localhost", "dbname": "nextcloud", "dbtype": "mysql", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "dbtableprefix": "oc_", "version": "12.0.0.29", "logtimezone": "UTC", "installed": true, "theme": "", "forcessl": true, "forceSSLforSubdomains": true, "openssl": { "config": "\/etc\/ssl\/openssl.cnf" }, "memcache.local": "\\OC\\Memcache\\Redis", "memcache.locking": "\\OC\\Memcache\\Redis", "redis": { "host": "127.0.0.1", "port": 6379 }, "mail_smtpmode": "smtp", "mail_smtpsecure": "tls", "mail_from_address": "serveur", "mail_domain": "ageg.ca", "mail_smtpauthtype": "PLAIN", "mail_smtpauth": 1, "mail_smtphost": "sleipnir.ageg.ca", "mail_smtpport": "587", "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***", "ldapIgnoreNamingRules": false, "loglevel": 0, "maintenance": false, "trashbin_retention_obligation": "auto", "htaccess.RewriteBase": "\/", "ldapProviderFactory": "\\OCA\\User_LDAP\\LDAPProviderFactory", "updater.release.channel": "stable" } }Are you using external storage, if yes which one: local storage on filesystem (outside of \var\www\nextcloud folder)
Are you using encryption: not in nextcloud
Are you using an external user-backend, if yes which one: LDAP
LDAP configuration (delete this part if not used)
LDAP config
+-------------------------------+------------------------------------------------------------+ | Configuration | s01 | +-------------------------------+------------------------------------------------------------+ | hasMemberOfFilterSupport | | | hasPagedResultSupport | | | homeFolderNamingRule | | | lastJpegPhotoLookup | 0 | | ldapAgentName | cn=nextcloud,ou=services,dc=ageg,dc=ca | | ldapAgentPassword | *** | | ldapAttributesForGroupSearch | | | ldapAttributesForUserSearch | | | ldapBackupHost | | | ldapBackupPort | | | ldapBase | dc=ageg,dc=ca | | ldapBaseGroups | dc=ageg,dc=ca | | ldapBaseUsers | dc=ageg,dc=ca | | ldapCacheTTL | 600 | | ldapConfigurationActive | 1 | | ldapDefaultPPolicyDN | | | ldapDynamicGroupMemberURL | | | ldapEmailAttribute | mail | | ldapExperiencedAdmin | 0 | | ldapExpertUUIDGroupAttr | | | ldapExpertUUIDUserAttr | | | ldapExpertUsernameAttr | | | ldapGidNumber | gidNumber | | ldapGroupDisplayName | cn | | ldapGroupFilter | | | ldapGroupFilterGroups | | | ldapGroupFilterMode | 0 | | ldapGroupFilterObjectclass | | | ldapGroupMemberAssocAttr | uniqueMember | | ldapHost | ldapi:/// | | ldapIgnoreNamingRules | | | ldapLoginFilter | (&(|(objectclass=inetOrgPerson))(|(uid=%uid)(|(cn=%uid)))) | | ldapLoginFilterAttributes | cn | | ldapLoginFilterEmail | 0 | | ldapLoginFilterMode | 0 | | ldapLoginFilterUsername | 1 | | ldapNestedGroups | 0 | | ldapOverrideMainServer | | | ldapPagingSize | 500 | | ldapPort | 389 | | ldapQuotaAttribute | | | ldapQuotaDefault | | | ldapTLS | 0 | | ldapUserDisplayName | displayname | | ldapUserDisplayName2 | | | ldapUserFilter | (|(objectclass=inetOrgPerson)) | | ldapUserFilterGroups | | | ldapUserFilterMode | 0 | | ldapUserFilterObjectclass | inetOrgPerson | | ldapUuidGroupAttribute | auto | | ldapUuidUserAttribute | auto | | turnOffCertCheck | 0 | | turnOnPasswordChange | 0 | | useMemberOfToDetectMembership | 1 | +-------------------------------+------------------------------------------------------------+Client configuration
Browser:
Chromium 59.0.3071.86 (Developer Build) (64-bit)
Firefox 52.0.2 (64-bit)
Operating system:
Debian 4.9.30-2 (2017-06-12) x86_64 GNU/Linux
Activity
carboneater commentedon Jun 20, 2017
I pushed it to the extreme to trigger the bug, so when I tried to replicate it in a shared folder, I used folder names like those:
(Shared is a shared folder)
(Moving this folder into another one broke it)
(Applying steps 2-5 in that one also triggered it, badly: folders are being duplicated in that entry...)
(Working in a non-shared environment, I had to push it a little more)
Other examples from my production environment (All are shared folders):
(Like I said, including spaces/internationalized characters seemed to make this bug more likely to occur.)
We've been using ownCloud/Nextcloud with internationalized characters (french) for years without issues.
jospoortvliet commentedon Jun 20, 2017
@icewind1991 how-to-break-the-file-cache ;-)
carboneater commentedon Jun 20, 2017
For more results, I ran the following queries on my oc_filecache table to get a list of all my duplicate filecache entries:
Here are my results.
oc_filecache.xlsx
Looks like some concatenate functions are truncating the end of the path string and the beginning of the filename.
MorrisJobke commentedon Jul 18, 2017
@icewind1991 Ping
jospoortvliet commentedon Jul 18, 2017
@carboneater thanks for the debugging. Hope Icewind can figure this one out, he's working on filecache stuff atm anyhow, I'm sure he loves it 🥇
metro2030 commentedon Aug 30, 2017
@icewind1991 Any news?
blizzz commentedon Sep 8, 2017
@icewind1991 🏓
J0WI commentedon Nov 26, 2019
Duplicate of #9907?
skjnldsv commentedon Apr 10, 2020
Looks like so yeas