Packagedcode to handle .podspec file by rpotter12 · Pull Request #2067 · aboutcode-org/scancode-toolkit

rpotter12 · 2020-06-13T23:36:32Z

Fixes #2036

Implementation to handle .podspec files.

Created cocoapods.py which handle Cocoapods Package.
Created spec.py which parse .podspec and .gemspec files and return dictionary with data.

Tasks

Reviewed contribution guidelines
PR is descriptively titled 📑 and links the original issue above 🔗
Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
Run tests locally to check for errors.
Commits are in uniquely-named feature branch and has no merge conflicts 📁

rpotter12 · 2020-06-14T19:44:09Z

@MaJuRG This PR is not completed yet.

steven-esser

Does the gemfileparser library work on podspec files?

rpotter12 · 2020-06-23T00:05:10Z

Does the gemfileparser library work on podspec files?

@MaJuRG No that library does not work with podspec files. podspec files are very much same with the gemspec file but not exactly same. So I have edited the code of gemspec file here to get all the data of podspec file and the approach is totally different from @balasankarc of https://gitlab.com/balasankarc/gemfileparser library.

steven-esser · 2020-06-23T00:09:05Z

What exactly is different about them, from a technical side? would the parser work if you changed the filename?

Just trying to understand more to give proper advice :)

rpotter12 · 2020-06-23T00:41:27Z

What exactly is different about them, from a technical side? would the parser work if you changed the filename?

Just trying to understand more to give proper advice :)

@MaJuRG
Like in dependency is written in the format s.dependency "<dep_name>" "<dep_version" but in gemspec file it is written in the format s.add_development_dependency "<dep_name>" "<dep_version" or s.add_development_dependency(%q<minitest>, [">= 2.0.2"])
and
in gemspec authors and email are written in different line like s.authors = ["Aaron Patterson"], s.email = ["aaron@tenderlovemaking.com"] and in podspec file author and email are same line s.author = {"Aaron Patterson" => "aaron@tenderlovemaking.com"}

rpotter12 · 2020-06-25T03:29:27Z

@MaJuRG @pombredanne @JonoYang I have made all the changes. Please review this again and merge this if everything looks good :)

pombredanne

Thank you for the progress!
I have a more extra comments provided inline for your consideration

pombredanne · 2020-06-25T09:55:21Z

+    return column_list
+
+
+def parse_podspec(location):


Is this not just a variation on the .gemspec?

Yes, .gemspec and .podspec files are very similar to each other but not exactly same so I have made changes according to it :)

You should have a shared utility module and not duplicate things then.

pombredanne · 2020-06-25T10:38:44Z

+    #  the lib prefix of their name.
+    #
+
+    # spec.framework  = "SomeFramework"


No sure, but do we want to track these as deps?

frameworks are a list of system frameworks that the user’s target needs to link against. I don't think this comes under dependencies. What do you say??

rpotter12 · 2020-06-26T21:54:41Z

@MaJuRG @pombredanne @JonoYang I have done all the necessary changes. Please review this again :)

steven-esser

The url parsing is not correct at this time. See comments.

pombredanne

Thank you for the progress and updates. 👍
I am picky so please see my comments inline!

pombredanne · 2020-07-01T07:35:09Z

+
+
+"""
+Handle Podspec Package Manager


What about instead:

""" Handle cocoapods packages manifests for macOS and iOS including .podspec, Podfile and Podfile.lock files. See https://cocoapods.org """

Also this may mean you want to rename the file from podspec.py to cocoapods.py which is a bit more generic?

pombredanne · 2020-07-01T07:40:54Z

+
+def parse_line(line):
+    """
+    Parses each line and creates dependency objects accordingly


What does a "line" arg looks like here?
Also please use the imperative style for docstring, e.g. something along the lines of:

Return a Dependency by parsing a line` string`

... but then, this is misleading: this function does not return a dependency?

Also add an example of what a "line" looks like.

pombredanne · 2020-07-01T07:42:34Z

+            stripped_column = stripped_column.replace('[', "")
+            stripped_column = stripped_column.strip()
+            column_list.append(stripped_column)
+    return column_list


It looks to me like you are parsing/cleaning up something that may be Ruby code here. This is also something that I saw in the rubygems code. You should have shared utility functions to handle this for both cocoapods an rubygems and not duplicate the code.

pombredanne · 2020-07-01T08:10:05Z

        check_json_scan(expected_file, result_file, regen=False)

+    def test_package_command_scan_podspec(self):
+        test_dir = self.get_test_loc('podspec/package')


Try to avoid reusing the same test files in a different test as this creates unnecessary coupling between tests that will bite us in the future.

Here you want to have a dedicated directory for this test f=data files and its expected file.

pombredanne · 2020-07-01T08:11:30Z

@@ -0,0 +1,140 @@
+#


Is this SwiftLib.podspec the example from their doc? It feels a little contrived. Can you find a better set of data files?

pombredanne · 2020-07-01T08:13:37Z

+  "code_view_url": null,
+  "vcs_url": "https://github.com/nanopb/nanopb.git",
+  "copyright": null,
+  "license_expression": "unknown",


We definitely need to override the license detection in order to detect correctly these declared_license
Also is there a way to collect ALL the license values of ALL the podspecs? Ask around for help for this or check in https://github.com/CocoaPods/Specs ... this will be useful to properly create mapping of they license conventions to our keys (in addition to applying proper detection to their license field structure)
That should be in its own ticket though, so create one

pombredanne · 2020-07-01T08:20:51Z

+  ],
+  "purl": "pkg:pods/nanopb@1.30905.0",
+  "repository_homepage_url": "https://cocoapods.org/pods/nanopb",
+  "repository_download_url": "https://github.com/nanopb/nanopb/archive/1.30905.0.zip",


Would this be also the download URL?

pods source code can be download by this url. :)

pombredanne · 2020-07-01T08:21:26Z

+  "purl": "pkg:pods/nanopb@1.30905.0",
+  "repository_homepage_url": "https://cocoapods.org/pods/nanopb",
+  "repository_download_url": "https://github.com/nanopb/nanopb/archive/1.30905.0.zip",
+  "api_data_url": "https://github.com/nanopb/nanopb"


Is this instead something from from https://github.com/CocoaPods/Specs ? you should check in the pod Ruby code to understand how things work.

Signed-off-by: rpotter12 <rohitpotter12@gmail.com>

steven-esser

Some small nitpicks.

steven-esser · 2020-07-07T01:30:36Z

+
+def is_podspec(location):
+    """
+    Checks is the file is a podspec file or not.


This should be: Checks if the file is actually a podspec file (nitpick)

steven-esser · 2020-07-07T01:31:00Z

+
+def build_package(podspec_data):
+    """
+    Return a Pacakge object from a package data mapping or None.


Pacakge should be Package

rpotter12 · 2020-09-03T11:32:45Z

#2075 is the merged which is the extended version of this PR.

rpotter12 force-pushed the podspec-parser branch 2 times, most recently from c41b9f0 to 6e884e7 Compare June 13, 2020 23:47

steven-esser suggested changes Jun 14, 2020

View reviewed changes

Comment thread src/packagedcode/podspec.py Outdated

Comment thread src/packagedcode/podspec.py Outdated

Comment thread src/packagedcode/podspec.py Outdated

Comment thread src/packagedcode/podspec.py Outdated

rpotter12 force-pushed the podspec-parser branch from 6e884e7 to a309eb3 Compare June 22, 2020 02:06

rpotter12 requested review from pombredanne and steven-esser June 22, 2020 02:08

rpotter12 force-pushed the podspec-parser branch 4 times, most recently from ac689e2 to 4db59fe Compare June 22, 2020 03:08

steven-esser reviewed Jun 22, 2020

View reviewed changes

rpotter12 requested a review from steven-esser June 23, 2020 00:08

rpotter12 force-pushed the podspec-parser branch 2 times, most recently from 96cfb83 to ae40e74 Compare June 24, 2020 20:01

steven-esser suggested changes Jun 25, 2020

View reviewed changes

Comment thread src/packagedcode/podspec.py Outdated

Comment thread src/packagedcode/podspec.py Outdated

Comment thread src/packagedcode/podspec.py Outdated

Comment thread src/packagedcode/podspec.py Outdated

Comment thread src/packagedcode/podspec.py Outdated

rpotter12 force-pushed the podspec-parser branch from ae40e74 to cbb501d Compare June 25, 2020 03:26

rpotter12 requested a review from steven-esser June 25, 2020 03:29

pombredanne requested changes Jun 25, 2020

View reviewed changes

rpotter12 closed this Jun 26, 2020

rpotter12 reopened this Jun 26, 2020

rpotter12 force-pushed the podspec-parser branch from cbb501d to 8fa310a Compare June 26, 2020 21:49

rpotter12 requested a review from pombredanne June 26, 2020 21:51

steven-esser suggested changes Jun 29, 2020

View reviewed changes

rpotter12 force-pushed the podspec-parser branch from 8fa310a to b1e6da1 Compare June 30, 2020 03:08

rpotter12 requested a review from steven-esser June 30, 2020 03:19

pombredanne requested changes Jul 1, 2020

View reviewed changes

rpotter12 force-pushed the podspec-parser branch 3 times, most recently from cddcff3 to 66cc58f Compare July 3, 2020 01:51

rpotter12 requested a review from pombredanne July 3, 2020 01:51

rpotter12 force-pushed the podspec-parser branch from 66cc58f to 8690e7a Compare July 3, 2020 01:53

rpotter12 added 2 commits July 3, 2020 20:43

implementation to handle podspec

17d4164

Signed-off-by: rpotter12 <rohitpotter12@gmail.com>

add alphanumeric with special char regex expression

8cb166a

Signed-off-by: rpotter12 <rohitpotter12@gmail.com>

rpotter12 force-pushed the podspec-parser branch from 8690e7a to 8cb166a Compare July 3, 2020 15:13

steven-esser suggested changes Jul 7, 2020

View reviewed changes

rpotter12 closed this Sep 3, 2020

Uh oh!

Uh oh!

Conversation

rpotter12 commented Jun 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tasks

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rpotter12 commented Jun 14, 2020

Uh oh!

steven-esser left a comment

Choose a reason for hiding this comment

Uh oh!

rpotter12 commented Jun 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steven-esser commented Jun 23, 2020

Uh oh!

rpotter12 commented Jun 23, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rpotter12 commented Jun 25, 2020

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rpotter12 Jun 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rpotter12 commented Jun 26, 2020

Uh oh!

steven-esser left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rpotter12 commented Jun 13, 2020 •

edited

Loading

rpotter12 commented Jun 23, 2020 •

edited

Loading

rpotter12 Jun 25, 2020 •

edited

Loading