You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Checking directories via match_file() does not work on Path objects #65
I am currently investigating a problem in the black formatter which uses pathspec to figure out ignored files by parsing .gitignore files. black uses this library and tries to check if directories are ignored before checking their contents.
It boils down to this minimum example:
frompathlibimportPathimportpathspecspec=pathspec.PathSpec.from_lines('gitwildmatch', """/important/""".splitlines())
assertspec.match_file("important/bar.log")
assertspec.match_file("/important/")
assertspec.match_file("important/")
assertspec.match_file("important") # does not matchassertspec.match_file(Path("important/")) # does not match as pathlib removes trailing slash
It matches the file important/bar.log as expected, however, it does not match the folder when it has no trailing slash.
This becomes especially problematic when using Path objects, as the trailing slash is removed by pathlib.
I don't know where this needs to be fixed. What do you think? Is black using it wrong? Should the match_file() method check if the given Path object is a directory and add the trailing slash on its own before checking?
It would certainly be possible to implement a workaround in black and check with Path.is_dir(), convert to str and append the slash...
FYI, the place where the check happens in black is here. relative_path is a pathlib.Path and can be a directory.
@yschroeder I'm undecided on this at the moment. I'm leaning towards supporting it but it's kind of inconsistent.
Supporting Path.is_dir() would be convenient but its behavior is not immediately apparent unless you're looking at the file-system. Path.is_dir() is only true when the directory exists on the file-system. It's false otherwise, even if constructed as Path('important/'). This makes sense when you think of it as an analog to os.path.isdir(). PurePath.is_dir() does not exist because it represents a virtual path that doesn't necessarily exist on the file-system. It would be nice if Path and PurePath maintained trailing slashes.
Does black always use Path with real file-system paths as opposed to PurePath or paths that don't exist?
Yes, I am also not too sure, as Path.is_dir() requires that the path actually exists in the file system. pathspec only works on strings and doesn't care if the path exists or not, which is a good thing. I realized this problem when I tried to create a test case for pathspec which would require a mock for is_dir() as otherwise the test would need real files and folders.
Disclaimer: I am not a developer of black but only a user, so I might be completely wrong here.
What black does is:
Find the root directory of the repository
Create a PathSpec for the found .gitignore
Iterate over all elements of the root directory via Path.iter_dir()
Check the paths from step 3 with match_file() of the spec from step 2
If step 4 did not match, format that file or decend into the directory
Repeat above steps in the subdirectory
In step 4, the match_file() fails and we do not decend into the subdirectory in step 5 although we should.
TL;DR: black uses Path with real file-system paths which really exist (and tries to check directories with match_file()).
Thanks for the summary. After thinking about this, I don't think the right behavior is to treat Path with .is_dir() specially because its behavior is so tied to the file-system. What may be useful is I can add a utility function, say append_dir_sep(), which appends/adds the trailing path separator when Path.is_dir() is True. It would essentially be:
On a side note, looking the black's usage that you pointed out, here, it looks like relative_path is already a str when it's passed to pattern.match_file() because normalize_path_maybe_ignore() returns a str. With this in mind, the normalize_path_maybe_ignore() function would have to be modified to either use append_dir_sep() or add the dir separator itself.
I've added the pathspec.util.append_dir_sep() function in the new release, v0.10.3. I believe that is all that I can address for this issue due to limitations with Path and the implementation of black.
Activity
yschroeder commentedon Nov 15, 2022
If you acknowledge that this needs to addressed by
pathspec, I think the place to fix this would beutil.normalize_file().When it creates the
str. It should check if it currently handles aPathand if yes, append the trailing slash ifPath.is_dir().cpburnz commentedon Nov 16, 2022
@yschroeder I'm undecided on this at the moment. I'm leaning towards supporting it but it's kind of inconsistent.
Supporting
Path.is_dir()would be convenient but its behavior is not immediately apparent unless you're looking at the file-system.Path.is_dir()is only true when the directory exists on the file-system. It's false otherwise, even if constructed asPath('important/'). This makes sense when you think of it as an analog toos.path.isdir().PurePath.is_dir()does not exist because it represents a virtual path that doesn't necessarily exist on the file-system. It would be nice ifPathandPurePathmaintained trailing slashes.Does black always use
Pathwith real file-system paths as opposed toPurePathor paths that don't exist?yschroeder commentedon Nov 16, 2022
Yes, I am also not too sure, as
Path.is_dir()requires that the path actually exists in the file system.pathspeconly works on strings and doesn't care if the path exists or not, which is a good thing. I realized this problem when I tried to create a test case forpathspecwhich would require a mock foris_dir()as otherwise the test would need real files and folders.Disclaimer: I am not a developer of
blackbut only a user, so I might be completely wrong here.What
blackdoes is:PathSpecfor the found.gitignorePath.iter_dir()match_file()of the spec from step 2In step 4, the
match_file()fails and we do not decend into the subdirectory in step 5 although we should.TL;DR:
blackusesPathwith real file-system paths which really exist (and tries to check directories withmatch_file()).cpburnz commentedon Nov 22, 2022
@yschroeder
Thanks for the summary. After thinking about this, I don't think the right behavior is to treat
Pathwith.is_dir()specially because its behavior is so tied to the file-system. What may be useful is I can add a utility function, sayappend_dir_sep(), which appends/adds the trailing path separator whenPath.is_dir()isTrue. It would essentially be:Then to use it you can do:
On a side note, looking the black's usage that you pointed out, here, it looks like
relative_pathis already astrwhen it's passed topattern.match_file()becausenormalize_path_maybe_ignore()returns astr. With this in mind, thenormalize_path_maybe_ignore()function would have to be modified to either useappend_dir_sep()or add the dir separator itself.Add limited feature for #65
cpburnz commentedon Dec 10, 2022
I've added the
pathspec.util.append_dir_sep()function in the new release, v0.10.3. I believe that is all that I can address for this issue due to limitations withPathand the implementation of black.