Expose all .whl,.ABOUT, .NOTICE,.LICENCE files and Create an archive by Abhishek-Dev09-zz · Pull Request #2117 · aboutcode-org/scancode-toolkit

Abhishek-Dev09-zz · 2020-07-16T18:10:56Z

It will upload all .whl,.ABOUT, .NOTICE,.LICENCE files of thirdparty repo to new repo as an asset of specific OS and python and also generate archive for each Oses and all python

Release them as asset on Github for #2136

Signed-off-by: Abhishek Kumar abhishek.kasyap09@gmail.com

Fixes #2068 aboutcode-org/skeleton#50 #2072

It will upload all .whl,.ABOUT, .NOTICE,.LICENCE files of thirdparty repo to github new repo as an asset of specific OS and python 3 version and also generate archive for each Oses and all python and release them an asset on github. Signed-off-by: Abhishek Kumar <abhishek.kasyap09@gmail.com>

steven-esser

Can we put these in etc/scripts/ like in #2118?

Abhishek-Dev09-zz · 2020-07-20T15:23:10Z

Can we put these in etc/scripts/ like in #2118?

Yes, you can run from anywhere no problem

pombredanne

Looking good!
I have a few nitpickings for your consideration

pombredanne · 2020-07-21T17:19:35Z

+    """
+    Generate an archive for dependencies for specific OS and
+    given version of python by taking directory as an input.
+    """


Can you remind me why and how this archive would be used?

It will generate an archive for each os and each python. For example - macOS_py36.tar.gz, win3_py36.zip, win64_py.36zip, linux_py36.tar.gz . For more info see PR description

I still do not know what these archives will be used for.... Are you saying that you plan to upload one such archive per OS/Python/scancode version to a GH release?
That archive will not be usable as a pip find-link then. And if there are dupes between OSes... that is going to be problematic. The archives we want to upload would be something built with etc/release/ scripts instead... e.g. full sdist with a streamlined thirparty

But you said earlier One archive for each OS/Python combo should be created and each should have its exact set of pinned requirements included in the built archive, no more and no less.

Yes, but I would prefer to have the sdist being use for these archives that's using a proper manifest and does not include things we do not want there. For this you should expand on the release script instead

FYI , this will give you 1 directory along with tar.gz ,if you are on linux /macOS.
if you are on window, you get 1 directory along with zip.

BTW i got directory of wheels of 30.1 MB and tar.gz of 28 MB on my macOS.

Signed-off-by: Abhishek Kumar <abhishek.kasyap09@gmail.com>

pombredanne · 2020-07-25T12:10:59Z

+from __future__ import print_function
+
+import argparse
+from commoncode.system import on_windows


Sort your imports: commoncode is not from the stdlib, but our own code and should go in a separate block.
See for instance https://github.com/nexB/scancode-toolkit/blob/develop/src/packagedcode/cargo.py#L36

pombredanne · 2020-07-25T12:13:06Z

+deps_archive.py \\
+  --input thirdparty \\
+  --req requirements.txt
+  --output_file macOS_py36 \\


Please remove that as it does help readability.

Signed-off-by: Abhishek Kumar <abhishek.kasyap09@gmail.com>

pombredanne · 2020-07-26T15:07:13Z

+    root_dir = os.path.abspath(archive_name)
+    output_dir = os.path.abspath(archive_name)
+    if on_windows:
+        make_archive(output_dir, 'zip', root_dir)


I am stil not clear about what these archives wrt. to what we need as sdist

here you get all wheel/sdist deps packed like sdist file that required for ur python/os.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Do not spawn a process but call a function instead. Clarify variable and options names and their help text. Use access token from a cli option or an environment variable. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Abhishek-Dev09-zz · 2020-07-26T23:07:06Z

 from __future__ import print_function

 import argparse
-from fnmatch import fnmatchcase


why you remove this?

Abhishek-Dev09-zz · 2020-07-26T23:09:06Z

+
 from commoncode.fileutils import resource_iter

-python_version = str(sys.version_info[0]) + str(sys.version_info[1])


Why remove this section ,it will help you filter the python 3 files if your directory contains multiple python deps.

Abhishek-Dev09-zz · 2020-07-26T23:11:07Z

-        or fnmatchcase(files, py_abi)
-        or (
-            fnmatchcase(files, '*tar.gz*')
-            and not fnmatchcase(files, '*py2-ipaddress-3.4.1.tar.gz*')


It will upload unwanted file also

Abhishek-Dev09-zz · 2020-07-26T23:13:08Z

-        '--body-string',
-        help='Required : Text describing the release. Ignored if the release already exists.',
+        '--token',
+        help=TOKEN_HELP,


great detail here.

Abhishek-Dev09-zz · 2020-07-26T23:15:03Z

-
-
-def main_with_args(args: str) -> None:
+    api = grr.GithubApi(


Either you call API or cmd . result will be same.

Abhishek-Dev09-zz · 2020-07-26T23:16:31Z

+        help='Text description for the release. Ignored if the release exists.',
        type=str,
-        default='10',
+        required=False,


no need to mention that

Abhishek-Dev09-zz · 2020-07-26T23:17:07Z

+        help=TOKEN_HELP,
        type=str,
-        required=True,
+        required=False,


no need to mention, by default it is false

Abhishek-Dev09-zz · 2020-07-26T23:18:18Z



-def main() -> None:
+def main():


Why you remove anotations?

pombredanne

I am still not 100% clear about the deps archive and these will be used. But I am merging now so we can get a clearer picture and improve from that base.

Abhishek-Dev09-zz force-pushed the #295-github_release branch 2 times, most recently from ee13ae6 to a1c0500 Compare July 16, 2020 18:16

Abhishek-Dev09-zz force-pushed the #295-github_release branch from a1c0500 to f08eacf Compare July 16, 2020 18:18

Abhishek-Dev09-zz commented Jul 18, 2020

View reviewed changes

Comment thread github_release.py Outdated

steven-esser reviewed Jul 20, 2020

View reviewed changes

pombredanne requested changes Jul 21, 2020

View reviewed changes

Move all script to etc/scripts

89e957e

Signed-off-by: Abhishek Kumar <abhishek.kasyap09@gmail.com>

Abhishek-Dev09-zz force-pushed the #295-github_release branch from e0ee198 to 89e957e Compare July 21, 2020 18:49

Use abspath instead of expanduser

90bff2b

Signed-off-by: Abhishek Kumar <abhishek.kasyap09@gmail.com>

Abhishek-Dev09-zz requested a review from pombredanne July 23, 2020 05:52

Format the script properly

2e80d15

Signed-off-by: Abhishek Kumar <abhishek.kasyap09@gmail.com>

pombredanne reviewed Jul 25, 2020

View reviewed changes

Sort imports,remove examples,use proper variable name

393fad4

Signed-off-by: Abhishek Kumar <abhishek.kasyap09@gmail.com>

Abhishek-Dev09-zz requested review from pombredanne and steven-esser July 26, 2020 05:10

pombredanne mentioned this pull request Jul 26, 2020

Automate the creation of requirement.txt and fetch deps #2118

Merged

Improve the description and documentation of function

d0e1386

Signed-off-by: Abhishek Kumar <abhishek.kasyap09@gmail.com>

pombredanne reviewed Jul 26, 2020

View reviewed changes

pombredanne added 2 commits July 26, 2020 17:16

Improve help texts

27c4e86

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Improve release creation and upload aboutcode-org#2072

c062d21

Do not spawn a process but call a function instead. Clarify variable and options names and their help text. Use access token from a cli option or an environment variable. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne force-pushed the #295-github_release branch from 2e6a01f to c062d21 Compare July 26, 2020 21:29

Abhishek-Dev09-zz commented Jul 26, 2020

View reviewed changes

Comment thread etc/scripts/github_release.py

def main() -> None:

def main():

Abhishek-Dev09-zz Jul 26, 2020

Copy link
Copy Markdown

Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why you remove anotations?

This was referenced Jul 27, 2020

Dev utility: build and download the sdist and wheels of a single package and all its dependencies #2087

Closed

Dev utility: Create release scripts to build archives #2072

Closed

End to end approach for third-party packages handlings in ScanCode #2135

Closed

pombredanne approved these changes Jul 27, 2020

View reviewed changes

pombredanne merged commit 07685a8 into aboutcode-org:develop Jul 27, 2020

Abhishek-Dev09-zz deleted the #295-github_release branch July 27, 2020 17:09


		from commoncode.fileutils import resource_iter

		python_version = str(sys.version_info[0]) + str(sys.version_info[1])

Uh oh!

Uh oh!

Conversation

Abhishek-Dev09-zz commented Jul 16, 2020 • edited by pombredanne Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

steven-esser left a comment

Choose a reason for hiding this comment

Uh oh!

Abhishek-Dev09-zz commented Jul 20, 2020

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Abhishek-Dev09-zz commented Jul 16, 2020 •

edited by pombredanne

Loading