Skip to content

makedirs bug when multiprocessing #310

Closed
Closed
makedirs bug when multiprocessing#310

Description

@simonm3
simonm3
opened on Jul 15, 2019

I have multiple processes writing different files to the same folder. To ensure the folder hierarchy exists I call makedirs(recreate=True) before I create the files. Sometimes this fails saying folder already exists. This is because of race condition where another process created it after the check.

The pyfilesystem code in base/makedirs has thread lock which stops this happening in threads. It also has a try except to ignore the exception for the lowest subfolder. However this should be around the creation of parent folders too:

            dir_paths = tools.get_intermediate_dirs(self, path)
            for dir_path in dir_paths:
                self.makedir(dir_path, permissions=permissions)
            try:
                self.makedir(path)
            except errors.DirectoryExists:
                if not recreate:
                    raise

Activity

  1. simonm3 commented on Jul 17, 2019

    @simonm3
    simonm3
    on Jul 17, 2019
    Author

    Like this....also the code is missing a permissions parameter in the second call to makedir.

    def makedirs(
    self,
    path,  # type: Text
    permissions=None,  # type: Optional[Permissions]
    recreate=False,  # type: bool
    ):
    self.check()
    with self._lock:
        dir_paths = tools.get_intermediate_dirs(self, path)
        for dir_path in dir_paths:
            try:
                self.makedir(dir_path, permissions=permissions)
            except errors.DirectoryExists:
                if not recreate:
                    raise
        try:
            self.makedir(path, permissions=permissions)
        except errors.DirectoryExists:
            if not recreate:
                raise
        return self.opendir(path)

  2. willmcgugan commented on Jul 24, 2019

    @willmcgugan
    willmcgugan
    on Jul 24, 2019
    Member

    That looks sensible. Would you mind submitting a PR?

  3. willmcgugan commented on Jul 28, 2019

    @willmcgugan
    willmcgugan
    on Jul 28, 2019
    Member

    Fixed in v2.4.9

  4. willmcgugan
    closed this as completedon Jul 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions