Skip to content

New repository tools should allow hashed-bin delegations #137

Closed
Closed
New repository tools should allow hashed-bin delegations#137
Assignees
vladimir-v-diaz
Labels
documentationDocumentation of the project as well as procedural documentationenhancementrepositoryRelated to the repository implementation

Description

@trishankkarthik
trishankkarthik
opened on Nov 20, 2013

@nekt reports that the new repository tools do not allow someone to implement something like the unclaimed role which the PEP recommends.

@JustinCappos : What do you think the interface should look like?

Activity

  1. ghost assigned
    vladimir-v-diaz
    on Nov 20, 2013

  2. vladimir-v-diaz commented on Nov 20, 2013

    @vladimir-v-diaz
    vladimir-v-diaz
    on Nov 20, 2013
    Contributor

    Example of hashed-bin delegations:
    https://github.com/theupdateframework/pypi.updateframework.com/blob/master/delegate_unclaimed_targets.py

  3. JustinCappos commented on Nov 20, 2013

    @JustinCappos
    JustinCappos
    on Nov 20, 2013
    Member

    I thought it is allowed, but there should be a helper function to tell you
    what the hash of something is so you can select the right bin.

    If so, we should add that function.

    On Wed, Nov 20, 2013 at 10:37 AM, Trishank Karthik Kuppusamy <
    notifications@github.com> wrote:

    @nekt https://github.com/nekt reports that the new repository tools do
    not allow someone to implement something like the unclaimed role which the
    PEP https://github.com/theupdateframework/pep-on-pypi-with-tufrecommends.

    @JustinCappos https://github.com/JustinCappos : What do you think the
    interface should look like?

    —
    Reply to this email directly or view it on GitHubhttps://github.com//issues/137
    .

  4. trishankkarthik commented on Nov 20, 2013

    @trishankkarthik
    trishankkarthik
    on Nov 20, 2013
    ContributorAuthor

    Correct, it is technically allowed, but the entire process should be much sweeter.

  5. JustinCappos commented on Nov 20, 2013

    @JustinCappos
    JustinCappos
    on Nov 20, 2013
    Member

    I agree. Let's add the hash helper function. (It must already exist for
    client checking but let's just expose what exists.)

    On Wed, Nov 20, 2013 at 12:29 PM, Trishank Karthik Kuppusamy <
    notifications@github.com> wrote:

    Correct, it is technically allowed, but the entire process should be much
    sweeter.

    —
    Reply to this email directly or view it on GitHubhttps://github.com//issues/137#issuecomment-28909949
    .

  6. vladimir-v-diaz commented on Jan 2, 2014

    @vladimir-v-diaz
    vladimir-v-diaz
    on Jan 2, 2014
    Contributor

    As requested, hash helper method added: 2a61a80

  7. vladimir-v-diaz
    closed this as completedon Jan 2, 2014

  8. trishankkarthik commented on Jan 5, 2014

    @trishankkarthik
    trishankkarthik
    on Jan 5, 2014
    ContributorAuthor

    Hmm. This is useful, but it still does not complete the picture for developers. I vote for reopening this issue.

  9. JustinCappos commented on Jan 14, 2014

    @JustinCappos
    JustinCappos
    on Jan 14, 2014
    Member

    Can you give a before and after example code so that I can understand what is lacking?

  10. JustinCappos
    reopened this on Jan 14, 2014
  11. vladimir-v-diaz
    added a commit that references this issue on Jan 21, 2014

    Update issues #151 and #137.

    5d1906a

  12. vladimir-v-diaz commented on Jan 21, 2014

    @vladimir-v-diaz
    vladimir-v-diaz
    on Jan 21, 2014
    Contributor

    Initial implementation of issue 137.

  13. vladimir-v-diaz
    added a commit that references this issue on Jan 22, 2014

    Update issue #137.

    96f6152

  14. vladimir-v-diaz commented on Jan 22, 2014

    @vladimir-v-diaz
    vladimir-v-diaz
    on Jan 22, 2014
    Contributor

    @JustinCappos:
    Creating hashed bins with the delegate_hashed_bins() method.

    targets = repository.get_filepaths_in_directory('repository/targets/packages', recursive_walk=True)
repository.targets('unclaimed').delegate_hashed_bins(list_of_targets=targets, keys_of_hashed_bins=[targets_public], number_of_bins=32)

for delegation in repository.targets('unclaimed').delegations:
  delegation.load_signing_key(targets_private)

repository.write()
    $ ls repository/metadata/targets/unclaimed/
00-07.txt  18-1f.txt  30-37.txt  48-4f.txt  60-67.txt  78-7f.txt  90-97.txt  a8-af.txt  c0-c7.txt  d8-df.txt  f0-f7.txt
08-0f.txt  20-27.txt  38-3f.txt  50-57.txt  68-6f.txt  80-87.txt  98-9f.txt  b0-b7.txt  c8-cf.txt  e0-e7.txt  f8-ff.txt
10-17.txt  28-2f.txt  40-47.txt  58-5f.txt  70-77.txt  88-8f.txt  a0-a7.txt  b8-bf.txt  d0-d7.txt  e8-ef.txt

    Request a file stored in a hashed bin:
    Note: The client only has root.txt.

    configurations = tuf.interposition.configure()
urllib.urlopen('http://localhost:8080/packages/django/file5.txt')
tuf.interposition.deconfigure(configurations)
    [2014-01-22 22:50:04,828 UTC] [tuf.interposition] [INFO][info:38@utility.py]
Adding updater for interposed network location: localhost:8080

[2014-01-22 22:50:04,832 UTC] [tuf.interposition] [INFO][info:38@utility.py]
Refreshing top-level metadata for interposed network location: localhost:8080

[2014-01-22 22:50:04,832 UTC] [tuf.download] [INFO][_download_file:740@download.py]
Downloading: http://localhost:8001/metadata/timestamp.txt

[2014-01-22 22:50:04,840 UTC] [tuf.download] [INFO][_check_downloaded_length:671@download.py]
Downloaded 1264 bytes out of an upper limit of 16384 bytes.

[2014-01-22 22:50:04,840 UTC] [tuf.client.updater] [INFO][_get_file:1155@updater.py]
Not decompressing http://localhost:8001/metadata/timestamp.txt

[2014-01-22 22:50:04,846 UTC] [tuf.download] [INFO][_download_file:740@download.py]
Downloading: http://localhost:8001/metadata/release.txt

[2014-01-22 22:50:04,848 UTC] [tuf.download] [INFO][_check_downloaded_length:649@download.py]
Downloaded 9775 bytes out of the expected 9775 bytes.

[2014-01-22 22:50:04,848 UTC] [tuf.client.updater] [INFO][_get_file:1155@updater.py]
Not decompressing http://localhost:8001/metadata/release.txt

[2014-01-22 22:50:04,848 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha224 hash is correct: ff7b153423e35ced63e9b1823cf7cf5bd899a41cd28b5e98b329766b

[2014-01-22 22:50:04,848 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha256 hash is correct: 4975acbc63fc8a80e70d2321c113bf143abc4395821d233f82e7111d3783f8d6

[2014-01-22 22:50:04,857 UTC] [tuf.client.updater] [INFO][_update_metadata_if_changed:1465@updater.py]
'root.txt' up-to-date.

[2014-01-22 22:50:04,858 UTC] [tuf.download] [INFO][_download_file:740@download.py]
Downloading: http://localhost:8001/metadata/targets.txt

[2014-01-22 22:50:04,859 UTC] [tuf.download] [INFO][_check_downloaded_length:649@download.py]
Downloaded 2112 bytes out of the expected 2112 bytes.

[2014-01-22 22:50:04,859 UTC] [tuf.client.updater] [INFO][_get_file:1155@updater.py]
Not decompressing http://localhost:8001/metadata/targets.txt

[2014-01-22 22:50:04,860 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha224 hash is correct: bc47032e31596371966dcc6bfe0e182fa38f585daebbe188aaa01805

[2014-01-22 22:50:04,860 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha256 hash is correct: f5625cd4fdc48feb834c0a998adaa3156583cca6713d80b2bd210fd6f926f3ac

[2014-01-22 22:50:04,868 UTC] [tuf.interposition] [INFO][info:38@utility.py]
Found updater for interposed network location: localhost:8080

[2014-01-22 22:50:04,868 UTC] [tuf.interposition] [INFO][info:38@utility.py]
Interposing for http://localhost:8080/packages/django/file5.txt

[2014-01-22 22:50:04,871 UTC] [tuf.client.updater] [INFO][_update_metadata_if_changed:1465@updater.py]
'targets.txt' up-to-date.

[2014-01-22 22:50:04,872 UTC] [tuf.download] [INFO][_download_file:740@download.py]
Downloading: http://localhost:8001/metadata/targets/unclaimed.txt.gz

[2014-01-22 22:50:04,874 UTC] [tuf.download] [INFO][_check_downloaded_length:649@download.py]
Downloaded 2187 bytes out of the expected 2187 bytes.

[2014-01-22 22:50:04,874 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha224 hash is correct: 924ba00f3834f48945095816bdfac0a63c4320b68a171faf251cbc6c

[2014-01-22 22:50:04,874 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha256 hash is correct: 41d52d9c367ce921eae11d273f4c7fc26ea4765fc20daebdfb0c4bcef162e446

[2014-01-22 22:50:04,875 UTC] [tuf.client.updater] [INFO][_get_file:1152@updater.py]
Decompressing http://localhost:8001/metadata/targets/unclaimed.txt.gz

[2014-01-22 22:50:04,875 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha224 hash is correct: fa905dc53afb37df094d86e25b3098285e83f039ea797f8de8e62ad3

[2014-01-22 22:50:04,876 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha256 hash is correct: 3e18c43c7cd277fb250c0144bad5ddc8879b415af8243d81e9129d0082db2f5e

[2014-01-22 22:50:04,891 UTC] [tuf.download] [INFO][_download_file:740@download.py]
Downloading: http://localhost:8001/metadata/targets/unclaimed/f0-f7.txt

[2014-01-22 22:50:04,893 UTC] [tuf.download] [INFO][_check_downloaded_length:649@download.py]
Downloaded 1332 bytes out of the expected 1332 bytes.

[2014-01-22 22:50:04,893 UTC] [tuf.client.updater] [INFO][_get_file:1155@updater.py]
Not decompressing http://localhost:8001/metadata/targets/unclaimed/f0-f7.txt

[2014-01-22 22:50:04,893 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha224 hash is correct: cf80776de05775844ebae8f5e8a69dfe365c8608eb0e9642d91542db

[2014-01-22 22:50:04,893 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha256 hash is correct: 53588ca227dbe6d8fbbc34cef7af91e1c4abf3a1ecd5664f144d1e7b7f240051

[2014-01-22 22:50:04,899 UTC] [tuf.download] [INFO][_download_file:740@download.py]
Downloading: http://localhost:8001/targets/packages/django/file5.txt

[2014-01-22 22:50:04,900 UTC] [tuf.download] [INFO][_check_downloaded_length:649@download.py]
Downloaded 6 bytes out of the expected 6 bytes.

[2014-01-22 22:50:04,901 UTC] [tuf.client.updater] [INFO][_get_file:1155@updater.py]
Not decompressing http://localhost:8001/targets/packages/django/file5.txt

[2014-01-22 22:50:04,901 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha224 hash is correct: a8e923b5902a13ac83831ac03ba16561616b1306d36e7e54f632bbe2

[2014-01-22 22:50:04,901 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha256 hash is correct: 2dbe9e279b85ee61cab10513c39791a2ce4bad678906832551afd1446c24bbe3

[2014-01-22 22:50:04,901 UTC] [tuf.interposition] [INFO][info:38@utility.py]
Updater removed for interposed network location: localhost:8080.

  15. trishankkarthik commented on Jan 22, 2014

    @trishankkarthik
    trishankkarthik
    on Jan 22, 2014
    ContributorAuthor

    Very nice --- this is exciting!

    On Wed, Jan 22, 2014 at 5:54 PM, Vladimir Diaz notifications@github.comwrote:

    @JustinCappos https://github.com/JustinCappos:
    Creating hashed bins with the delegate_hashed_bins() method.

    targets = repository.get_filepaths_in_directory('repository/targets/packages', recursive_walk=True)repository.targets('unclaimed').delegate_hashed_bins(list_of_targets=targets, keys_of_hashed_bins=[targets_public], number_of_bins=32)
    for delegation in repository.targets('unclaimed').delegations:
    delegation.load_signing_key(targets_private)
    repository.write()

    $ ls repository/metadata/targets/unclaimed/
    00-07.txt 18-1f.txt 30-37.txt 48-4f.txt 60-67.txt 78-7f.txt 90-97.txt a8-af.txt c0-c7.txt d8-df.txt f0-f7.txt
    08-0f.txt 20-27.txt 38-3f.txt 50-57.txt 68-6f.txt 80-87.txt 98-9f.txt b0-b7.txt c8-cf.txt e0-e7.txt f8-ff.txt
    10-17.txt 28-2f.txt 40-47.txt 58-5f.txt 70-77.txt 88-8f.txt a0-a7.txt b8-bf.txt d0-d7.txt e8-ef.txt

    Request a file stored in a hashed bin:
    Note: The client only has root.txt.

    configurations = tuf.interposition.configure()urllib.urlopen('http://localhost:8080/packages/django/file5.txt')tuf.interposition.deconfigure(configurations)

    [2014-01-22 22:50:04,828 UTC] [tuf.interposition] [INFO][info:38@utility.py]
    Adding updater for interposed network location: localhost:8080
    [2014-01-22 22:50:04,832 UTC] [tuf.interposition] [INFO][info:38@utility.py]
    Refreshing top-level metadata for interposed network location: localhost:8080
    [2014-01-22 22:50:04,832 UTC] [tuf.download] [INFO][_download_file:740@download.py]
    Downloading: http://localhost:8001/metadata/timestamp.txt
    [2014-01-22 22:50:04,840 UTC] [tuf.download] [INFO][_check_downloaded_length:671@download.py]
    Downloaded 1264 bytes out of an upper limit of 16384 bytes.
    [2014-01-22 22:50:04,840 UTC] [tuf.client.updater] [INFO][_get_file:1155@updater.py]
    Not decompressing http://localhost:8001/metadata/timestamp.txt
    [2014-01-22 22:50:04,846 UTC] [tuf.download] [INFO][_download_file:740@download.py]
    Downloading: http://localhost:8001/metadata/release.txt
    [2014-01-22 22:50:04,848 UTC] [tuf.download] [INFO][_check_downloaded_length:649@download.py]
    Downloaded 9775 bytes out of the expected 9775 bytes.
    [2014-01-22 22:50:04,848 UTC] [tuf.client.updater] [INFO][_get_file:1155@updater.py]
    Not decompressing http://localhost:8001/metadata/release.txt
    [2014-01-22 22:50:04,848 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha224 hash is correct: ff7b153423e35ced63e9b1823cf7cf5bd899a41cd28b5e98b329766b
    [2014-01-22 22:50:04,848 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]The file's sha256 hash is correct: 4975acbc63fc8a80e70d2321c113bf143abc4395821d233f82e7111d3783f8d6
    [2014-01-22 22:50:04,857 UTC] [tuf.client.updater] [INFO][_update_metadata_if_changed:1465@updater.py]'root.txt' up-to-date.
    [2014-01-22 22:50:04,858 UTC] [tuf.download] [INFO][_download_file:740@download.py]
    Downloading: http://localhost:8001/metadata/targets.txt
    [2014-01-22 22:50:04,859 UTC] [tuf.download] [INFO][_check_downloaded_length:649@download.py]
    Downloaded 2112 bytes out of the expected 2112 bytes.
    [2014-01-22 22:50:04,859 UTC] [tuf.client.updater] [INFO][_get_file:1155@updater.py]
    Not decompressing http://localhost:8001/metadata/targets.txt
    [2014-01-22 22:50:04,860 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha224 hash is correct: bc47032e31596371966dcc6bfe0e182fa38f585daebbe188aaa01805
    [2014-01-22 22:50:04,860 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]The file's sha256 hash is correct: f5625cd4fdc48feb834c0a998adaa3156583cca6713d80b2bd210fd6f926f3ac
    [2014-01-22 22:50:04,868 UTC] [tuf.interposition] [INFO][info:38@utility.py]
    Found updater for interposed network location: localhost:8080
    [2014-01-22 22:50:04,868 UTC] [tuf.interposition] [INFO][info:38@utility.py]
    Interposing for http://localhost:8080/packages/django/file5.txt
    [2014-01-22 22:50:04,871 UTC] [tuf.client.updater] [INFO][_update_metadata_if_changed:1465@updater.py]'targets.txt' up-to-date.
    [2014-01-22 22:50:04,872 UTC] [tuf.download] [INFO][_download_file:740@download.py]
    Downloading: http://localhost:8001/metadata/targets/unclaimed.txt.gz
    [2014-01-22 22:50:04,874 UTC] [tuf.download] [INFO][_check_downloaded_length:649@download.py]
    Downloaded 2187 bytes out of the expected 2187 bytes.
    [2014-01-22 22:50:04,874 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha224 hash is correct: 924ba00f3834f48945095816bdfac0a63c4320b68a171faf251cbc6c
    [2014-01-22 22:50:04,874 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]The file's sha256 hash is correct: 41d52d9c367ce921eae11d273f4c7fc26ea4765fc20daebdfb0c4bcef162e446
    [2014-01-22 22:50:04,875 UTC] [tuf.client.updater] [INFO][_get_file:1152@updater.py]
    Decompressing http://localhost:8001/metadata/targets/unclaimed.txt.gz
    [2014-01-22 22:50:04,875 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha224 hash is correct: fa905dc53afb37df094d86e25b3098285e83f039ea797f8de8e62ad3
    [2014-01-22 22:50:04,876 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]The file's sha256 hash is correct: 3e18c43c7cd277fb250c0144bad5ddc8879b415af8243d81e9129d0082db2f5e
    [2014-01-22 22:50:04,891 UTC] [tuf.download] [INFO][_download_file:740@download.py]
    Downloading: http://localhost:8001/metadata/targets/unclaimed/f0-f7.txt
    [2014-01-22 22:50:04,893 UTC] [tuf.download] [INFO][_check_downloaded_length:649@download.py]
    Downloaded 1332 bytes out of the expected 1332 bytes.
    [2014-01-22 22:50:04,893 UTC] [tuf.client.updater] [INFO][_get_file:1155@updater.py]
    Not decompressing http://localhost:8001/metadata/targets/unclaimed/f0-f7.txt
    [2014-01-22 22:50:04,893 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha224 hash is correct: cf80776de05775844ebae8f5e8a69dfe365c8608eb0e9642d91542db
    [2014-01-22 22:50:04,893 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]The file's sha256 hash is correct: 53588ca227dbe6d8fbbc34cef7af91e1c4abf3a1ecd5664f144d1e7b7f240051
    [2014-01-22 22:50:04,899 UTC] [tuf.download] [INFO][_download_file:740@download.py]
    Downloading: http://localhost:8001/targets/packages/django/file5.txt
    [2014-01-22 22:50:04,900 UTC] [tuf.download] [INFO][_check_downloaded_length:649@download.py]
    Downloaded 6 bytes out of the expected 6 bytes.
    [2014-01-22 22:50:04,901 UTC] [tuf.client.updater] [INFO][_get_file:1155@updater.py]
    Not decompressing http://localhost:8001/targets/packages/django/file5.txt
    [2014-01-22 22:50:04,901 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha224 hash is correct: a8e923b5902a13ac83831ac03ba16561616b1306d36e7e54f632bbe2
    [2014-01-22 22:50:04,901 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]The file's sha256 hash is correct: 2dbe9e279b85ee61cab10513c39791a2ce4bad678906832551afd1446c24bbe3
    [2014-01-22 22:50:04,901 UTC] [tuf.interposition] [INFO][info:38@utility.py]
    Updater removed for interposed network location: localhost:8080.

    —
    Reply to this email directly or view it on GitHubhttps://github.com//issues/137#issuecomment-33078477
    .

  16. vladimir-v-diaz
    added a commit that references this issue on Jan 23, 2014

    Address Issue #137 and update repository_tool.py.

    973d3a2

  17. vladimir-v-diaz commented on Jan 23, 2014

    @vladimir-v-diaz
    vladimir-v-diaz
    on Jan 23, 2014
    Contributor

    Final updates to #137.

    Add the add_restricted_paths() method.
    Rename libtuf.py
    Update README.
    Update delegate_hashed_bins() docstring.
    More testing of hashed bins and consistent snapshots.

  18. vladimir-v-diaz
    closed this as completedon Jan 23, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

documentationDocumentation of the project as well as procedural documentationenhancementrepositoryRelated to the repository implementation

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions