Skip to content

New repository tools should provide option to write consistent snapshots #151

Closed

Description

@trishankkarthik

Observed

The new repository tools writes metadata with the assumption that the metadata will not be continuously changing.

Problem

This means that a pair of concurrent writing of metadata to the same repository may produce inconsistent metadata on disk.

Expected

The new repository tools should offer the option to produce consistent snapshots as outlined in the PEP.

Issue #152 will outline how the TUF specification should change to produce and recognize consistent snapshots.

Activity

  1. vladimir-v-diaz commented on Jan 14, 2014

    @vladimir-v-diaz
    Contributor

    Writing consistent snapshots:

    repository.write(consistent_snapshots=True)
    $ ls metadata
    0bd0704d1caf7bd62bc2780f95ffebf2afa5f51762ec332f8944cf89e8688833.targets.txt
    8579fa3ad8b1df5086b94d5cdd0c7f092fcf540ff3c32c56d91e4026ab1ce108.root.txt
    aa0cdcd23071486411935435ffa150a564f24546705028cbdc4ce4965a0b8c5f.timestamp.txt
    dda4623fef86d03ad0ef28800a7d4820321816d801177f6903462f49598fa370.release.txt
    root.txt
    targets
    timestamp.txt
    
    $ ls metadata/targets/
    148858bda6ea25a47e3ad996377fd2b4f8c23dcd77d6a00a7deaf41a9aa49fbc.django.txt
    django
  2. JustinCappos commented on Jan 14, 2014

    @JustinCappos
    Member

    Cool. So we decided to ditch the hashalg.hash.filename format?

    Justin

    On Tue, Jan 14, 2014 at 10:10 AM, Vladimir Diaz notifications@github.comwrote:

    Writing consistent snapshots:

    repository.write(consistent_snapshots=True)

    $ ls metadata
    0bd0704d1caf7bd62bc2780f95ffebf2afa5f51762ec332f8944cf89e8688833.targets.txt
    8579fa3ad8b1df5086b94d5cdd0c7f092fcf540ff3c32c56d91e4026ab1ce108.root.txt
    aa0cdcd23071486411935435ffa150a564f24546705028cbdc4ce4965a0b8c5f.timestamp.txt
    dda4623fef86d03ad0ef28800a7d4820321816d801177f6903462f49598fa370.release.txt
    root.txt
    targets
    timestamp.txt
    $ ls metadata/targets/
    148858bda6ea25a47e3ad996377fd2b4f8c23dcd77d6a00a7deaf41a9aa49fbc.django.txt
    django

    —
    Reply to this email directly or view it on GitHubhttps://github.com//issues/151#issuecomment-32272068
    .

  3. vladimir-v-diaz commented on Jan 14, 2014

    @vladimir-v-diaz
    Contributor

    Most likely. I thought about it some more and concluded it supports a lot of the "hashalg.hash.filename" behavior I imagined (e.g., change the hash alg and not require a single change to parsing code). It's nice and simple.

  4. JustinCappos commented on Jan 14, 2014

    @JustinCappos
    Member

    Okay, how does this work? What is the rationale?

    Justin

    On Tue, Jan 14, 2014 at 10:31 AM, Vladimir Diaz notifications@github.comwrote:

    Most likely. I thought about it some more and concluded it supports a lot
    of the "hashalg.hash.filename" behavior I imagined (e.g., change the hash
    alg and not require a single change to parsing code). It's nice and simple.

    —
    Reply to this email directly or view it on GitHubhttps://github.com//issues/151#issuecomment-32274157
    .

  5. added a commit that references this issue on Jan 17, 2014
  6. vladimir-v-diaz commented on Jan 17, 2014

    @vladimir-v-diaz
    Contributor

    Support multiple hash algorithms, where the generated digests of metadata and target files is included in metadata (and filenames if 'consistent_snapshots' is True). Previously, only a single hash algorithm was supported, and it was set by default to 'sha256' in code. Repository maintainers may now choose any, and/or multiple, hash algorithms from those supported by TUF. By default, 'sha256' is used when generating digests.

    Support the recent change to the TUF specification, where writing consistent snapshots may include N versions of identical metadata and targets, if N hash algorithms is used by the repository when generating metadata.

    Update code affected by the recent changes to the specification, such as targets that may include digests in their filename.

    Support consistent snapshots of compressed metadata, including repositories that provide multiple versions of metadata with different digests included in filenames.

    The repository tools can now load repositories that include consistent snapshots of metadata and targets, including those with multiple (i.e., multiple digests prepended to filenames) consistent snapshots of files.

    The client code (i.e., updater.py and interposition) may now read repositories with 'consistent_snapshots': true in Root metadata, and properly request and update files with digests included.

    Summary of previous implementation updates:
    Update Root schema
    Support writing metadata in rolename.ext and digest.rolename.ext formats.
    Modify libtuf.py methods that walk and extract metadata to recognize the digest.rolename.ext format.
    Write root.txt and timestamp.txt if consistent_snapshots is True.
    Support automatic version increments of delegated metadata.

  7. vladimir-v-diaz commented on Jan 17, 2014

    @vladimir-v-diaz
    Contributor

    Writing consistent snapshots, where metadata and target files may include the digests generated by multiple hash algorithms:

    tuf.conf.HASH_ALGORITHMS = ['sha224', 'sha256']                                 
    repository.write(consistent_snapshots=True)
    $ ls repository/metadata/
    1ae87a4ef0029c29e5c4181082d3f6437c35ec8b4b1de1ab216a13ee.targets.txt
    c52b8c34466bbdb848cad94b1512489435716d5679462bf15019c6aa4c237e50.targets.txt
    1e195f9fe80fd5946607ee637696e62721f313496e92ad4d673ce95d.timestamp.txt
    c5ef9cf43bc373c7d17631d3390b504dfcb6f7b98896994ab7358f66b14bb894.timestamp.txt
    3b54a66ed16423d84bd6f1333c1206c78b6dd256f875e7f6ec761d37.release.txt
    root.txt
    93c836cb23a3dded67eae1704443893b9d0874e46ed6a5446851572812d64979.root.txt
    targets
    9d103c5ba0c2a9582966e979284261216817e8efc6517cd478989a3f5e08ee4e.release.txt
    timestamp.txt
    aab02b2f7ba0c022e05058a116481be8a7890c3c95a7a4e113c1f29b.root.txt
    
    $ ls repository/metadata/targets/
    b269a31bb1786421734148e09b0621a8776282078e70cecec3d4c6d6.django.txt.gz
    django
    b6214e7f8909f608af3398077c62cabf2f4a9ceaf64b82a140647dcbc5394bb3.django.txt
    e5f14273949efe1ad052bd6de21f9dcaee208aa5ad9df7f9fbb23bcb.django.txt
    dbfa35c7c2be17ababc1c8fed10d043cb8cd832347579bd9cd7de3caa19eab8a.django.txt.gz
  8. vladimir-v-diaz commented on Jan 17, 2014

    @vladimir-v-diaz
    Contributor
    configurations = tuf.interposition.configure()
    
    urllib.urlopen('http://localhost/file1.txt')
    urllib.urlopen('http://localhost/file2.txt')
    
    tuf.interposition.deconfigure(configurations)

    Observed:

    localhost - - [17/Jan/2014 11:24:51] "GET /metadata/timestamp.txt HTTP/1.1" 200 -
    localhost - - [17/Jan/2014 11:24:51] "GET /metadata/9d103c5ba0c2a9582966e979284261216817e8efc6517cd478989a3f5e08ee4e.release.txt HTTP/1.1" 200 -
    localhost - - [17/Jan/2014 11:24:51] "GET /metadata/1ae87a4ef0029c29e5c4181082d3f6437c35ec8b4b1de1ab216a13ee.targets.txt HTTP/1.1" 200 -
    localhost - - [17/Jan/2014 11:24:51] "GET /targets/5df2fa9eea382d07d47dd99d6c467c71ef3c520e38c01894fd873966.file1.txt HTTP/1.1" 200 -
    localhost - - [17/Jan/2014 11:24:51] "GET /targets/8bba1f160aaed38958655319f7af17f9c1cd7844a30581596537cb13.file2.txt HTTP/1.1" 200 -
  9. vladimir-v-diaz commented on Jan 17, 2014

    @vladimir-v-diaz
    Contributor

    Logger output when 'consistent_snapshots' is True:

    [2014-01-17 16:24:51,930 UTC] [tuf.interposition] [INFO][info:38@utility.py]
    Adding updater for Configuration(netloc=localhost:80)
    
    [2014-01-17 16:24:51,931 UTC] [tuf.interposition] [INFO][info:38@utility.py]
    Refreshing top-level metadata for Configuration(netloc=localhost:80)
    
    [2014-01-17 16:24:51,931 UTC] [tuf.download] [INFO][_download_file:726@download.py]
    Downloading: http://localhost:8001/metadata/timestamp.txt
    
    [2014-01-17 16:24:51,936 UTC] [tuf.download] [WARNING][_check_content_length:589@download.py]
    reported_length (1264) < required_length (16384)
    
    [2014-01-17 16:24:51,936 UTC] [tuf.download] [WARNING][_check_downloaded_length:656@download.py]
    Downloaded 1264 bytes, but expected 16384 bytes. There is a difference of 15120 bytes!
    
    [2014-01-17 16:24:51,939 UTC] [tuf.download] [INFO][_download_file:726@download.py]
    Downloading: http://localhost:8001/metadata/9d103c5ba0c2a9582966e979284261216817e8efc6517cd478989a3f5e08ee4e.release.txt
    
    [2014-01-17 16:24:51,940 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha224 hash is correct: 3b54a66ed16423d84bd6f1333c1206c78b6dd256f875e7f6ec761d37
    
    [2014-01-17 16:24:51,940 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha256 hash is correct: 9d103c5ba0c2a9582966e979284261216817e8efc6517cd478989a3f5e08ee4e
    
    [2014-01-17 16:24:51,942 UTC] [tuf.client.updater] [INFO][_update_metadata_if_changed:1465@updater.py]
    'root.txt' up-to-date.
    
    [2014-01-17 16:24:51,942 UTC] [tuf.download] [INFO][_download_file:726@download.py]
    Downloading: http://localhost:8001/metadata/1ae87a4ef0029c29e5c4181082d3f6437c35ec8b4b1de1ab216a13ee.targets.txt
    
    [2014-01-17 16:24:51,943 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha224 hash is correct: 1ae87a4ef0029c29e5c4181082d3f6437c35ec8b4b1de1ab216a13ee
    
    [2014-01-17 16:24:51,943 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha256 hash is correct: c52b8c34466bbdb848cad94b1512489435716d5679462bf15019c6aa4c237e50
    
    [2014-01-17 16:24:51,947 UTC] [tuf.interposition] [INFO][info:38@utility.py]
    Found updater for hostname=localhost
    
    [2014-01-17 16:24:51,947 UTC] [tuf.interposition] [INFO][info:38@utility.py]
    Interposing for http://localhost/file1.txt
    
    [2014-01-17 16:24:51,949 UTC] [tuf.client.updater] [INFO][_update_metadata_if_changed:1465@updater.py]
    'targets.txt' up-to-date.
    
    [2014-01-17 16:24:51,949 UTC] [tuf.download] [INFO][_download_file:726@download.py]
    Downloading: http://localhost:8001/targets/5df2fa9eea382d07d47dd99d6c467c71ef3c520e38c01894fd873966.file1.txt
    
    [2014-01-17 16:24:51,950 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha224 hash is correct: 5df2fa9eea382d07d47dd99d6c467c71ef3c520e38c01894fd873966
    
    [2014-01-17 16:24:51,950 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha256 hash is correct: ecdc5536f73bdae8816f0ea40726ef5e9b810d914493075903bb90623d97b1d8
    
    [2014-01-17 16:24:51,950 UTC] [tuf.interposition] [INFO][info:38@utility.py]
    Found updater for hostname=localhost
    
    [2014-01-17 16:24:51,950 UTC] [tuf.interposition] [INFO][info:38@utility.py]
    Interposing for http://localhost/file2.txt
    
    [2014-01-17 16:24:51,950 UTC] [tuf.client.updater] [INFO][_update_metadata_if_changed:1465@updater.py]
    'targets.txt' up-to-date.
    
    [2014-01-17 16:24:51,950 UTC] [tuf.download] [INFO][_download_file:726@download.py]
    Downloading: http://localhost:8001/targets/8bba1f160aaed38958655319f7af17f9c1cd7844a30581596537cb13.file2.txt
    
    [2014-01-17 16:24:51,951 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha224 hash is correct: 8bba1f160aaed38958655319f7af17f9c1cd7844a30581596537cb13
    
    [2014-01-17 16:24:51,951 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha256 hash is correct: 67ee5478eaadb034ba59944eb977797b49ca6aa8d3574587f36ebcbeeb65f70e
    
    [2014-01-17 16:24:51,951 UTC] [tuf.interposition] [INFO][info:38@utility.py]
    Updater removed for Configuration(netloc=localhost:80).
  10. trishankkarthik commented on Jan 17, 2014

    @trishankkarthik
    ContributorAuthor

    Great! Could you test this on a target in a delegated targets role,
    just to be sure?

  11. vladimir-v-diaz commented on Jan 17, 2014

    @vladimir-v-diaz
    Contributor

    Good idea. I had not tested delegated roles until you mentioned it.

    configurations = tuf.interposition.configure()
    urllib.urlopen('http://localhost/django/file3.txt')
    tuf.interposition.deconfigure(configurations)

    Observed:

    localhost - - [17/Jan/2014 11:46:31] "GET /metadata/timestamp.txt HTTP/1.1" 200 -
    localhost - - [17/Jan/2014 11:46:31] "GET /metadata/targets/b269a31bb1786421734148e09b0621a8776282078e70cecec3d4c6d6.django.txt.gz HTTP/1.1" 200 -
    localhost - - [17/Jan/2014 11:46:31] "GET /targets/django/94f6e58bd04a4513b8301e75f40527cf7610c66d1960b26f6ac2e743e108bdac.file3.txt HTTP/1.1" 200 -
    [2014-01-17 16:46:31,725 UTC] [tuf.interposition] [INFO][info:38@utility.py]
    Adding updater for Configuration(netloc=localhost:80)
    
    [2014-01-17 16:46:31,727 UTC] [tuf.interposition] [INFO][info:38@utility.py]
    Refreshing top-level metadata for Configuration(netloc=localhost:80)
    
    [2014-01-17 16:46:31,727 UTC] [tuf.download] [INFO][_download_file:726@download.py]
    Downloading: http://localhost:8001/metadata/timestamp.txt
    
    [2014-01-17 16:46:31,732 UTC] [tuf.download] [WARNING][_check_content_length:589@download.py]
    reported_length (1264) < required_length (16384)
    
    [2014-01-17 16:46:31,732 UTC] [tuf.download] [WARNING][_check_downloaded_length:656@download.py]
    Downloaded 1264 bytes, but expected 16384 bytes. There is a difference of 15120 bytes!
    
    [2014-01-17 16:46:31,735 UTC] [tuf.client.updater] [INFO][_update_metadata_if_changed:1465@updater.py]
    'release.txt' up-to-date.
    
    [2014-01-17 16:46:31,735 UTC] [tuf.client.updater] [INFO][_update_metadata_if_changed:1465@updater.py]
    'root.txt' up-to-date.
    
    [2014-01-17 16:46:31,735 UTC] [tuf.client.updater] [INFO][_update_metadata_if_changed:1465@updater.py]
    'targets.txt' up-to-date.
    
    [2014-01-17 16:46:31,737 UTC] [tuf.interposition] [INFO][info:38@utility.py]
    Found updater for hostname=localhost
    
    [2014-01-17 16:46:31,737 UTC] [tuf.interposition] [INFO][info:38@utility.py]
    Interposing for http://localhost/django/file3.txt
    
    [2014-01-17 16:46:31,738 UTC] [tuf.client.updater] [INFO][_update_metadata_if_changed:1465@updater.py]
    'targets.txt' up-to-date.
    
    [2014-01-17 16:46:31,739 UTC] [tuf.download] [INFO][_download_file:726@download.py]
    Downloading: http://localhost:8001/metadata/targets/b269a31bb1786421734148e09b0621a8776282078e70cecec3d4c6d6.django.txt.gz
    
    [2014-01-17 16:46:31,740 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha224 hash is correct: b269a31bb1786421734148e09b0621a8776282078e70cecec3d4c6d6
    
    [2014-01-17 16:46:31,740 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha256 hash is correct: dbfa35c7c2be17ababc1c8fed10d043cb8cd832347579bd9cd7de3caa19eab8a
    
    [2014-01-17 16:46:31,740 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha224 hash is correct: e5f14273949efe1ad052bd6de21f9dcaee208aa5ad9df7f9fbb23bcb
    
    [2014-01-17 16:46:31,740 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha256 hash is correct: b6214e7f8909f608af3398077c62cabf2f4a9ceaf64b82a140647dcbc5394bb3
    
    [2014-01-17 16:46:31,742 UTC] [tuf.download] [INFO][_download_file:726@download.py]
    Downloading: http://localhost:8001/targets/django/94f6e58bd04a4513b8301e75f40527cf7610c66d1960b26f6ac2e743e108bdac.file3.txt
    
    [2014-01-17 16:46:31,744 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha224 hash is correct: 561bfec53d3dfda762a2431ff6de32f343e505d68829612d1842e6b4
    
    [2014-01-17 16:46:31,744 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha256 hash is correct: 94f6e58bd04a4513b8301e75f40527cf7610c66d1960b26f6ac2e743e108bdac
    
    [2014-01-17 16:46:31,744 UTC] [tuf.interposition] [INFO][info:38@utility.py]
    Updater removed for Configuration(netloc=localhost:80).
  12. vladimir-v-diaz commented on Jan 17, 2014

    @vladimir-v-diaz
    Contributor

    Still need to update logger messages (not obvious that compressed and uncompressed hashes are verified), documentation, minor issues, and refactor a few things.

  13. trishankkarthik commented on Jan 17, 2014

    @trishankkarthik
    ContributorAuthor

    On 01/17/2014 11:55 AM, Vladimir Diaz wrote:

    Good idea. I had not tested delegated roles until you mentioned it.

    Very good, thanks! Looks like everything works as it should.

  14. added a commit that references this issue on Jan 18, 2014
  15. vladimir-v-diaz commented on Jan 19, 2014

    @vladimir-v-diaz
    Contributor

    Verifying compressed and uncompressed metadata:

    [2014-01-19 01:38:36,679 UTC] [tuf.download] [INFO][_download_file:740@download.py]
    Downloading: http://localhost:8001/metadata/targets/d28735bb21e41668cf526413b2e68f683673e50060b34cd4cb53b582491117c8.django.txt.gz
    
    [2014-01-19 01:38:36,681 UTC] [tuf.download] [WARNING][_check_downloaded_length:649@download.py]
    Downloaded 1806 bytes out of the expected 1806 bytes.
    
    [2014-01-19 01:38:36,681 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha224 hash is correct: 20b109dfde808e8f10d239625a2ea7f28624bf00ea1446cac4c3c5e8
    
    [2014-01-19 01:38:36,682 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha256 hash is correct: d28735bb21e41668cf526413b2e68f683673e50060b34cd4cb53b582491117c8
    
    [2014-01-19 01:38:36,682 UTC] [tuf.client.updater] [INFO][_get_file:1152@updater.py]
    Decompressing http://localhost:8001/metadata/targets/d28735bb21e41668cf526413b2e68f683673e50060b34cd4cb53b582491117c8.django.txt.gz
    
    [2014-01-19 01:38:36,682 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha224 hash is correct: 54f988d7f25456622a1dbe771fe633b36b1445b9a3f23efe886b6393
    
    [2014-01-19 01:38:36,682 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha256 hash is correct: 4b544a0bd028153be15d93e21717361ccdc93deda21a469e06c65c5eed670ac6
  16. trishankkarthik commented on Jan 19, 2014

    @trishankkarthik
    ContributorAuthor

    Why is the second log entry a warning, when expected == observed?

    On Sat, Jan 18, 2014 at 8:46 PM, Vladimir Diaz notifications@github.comwrote:

    Verifying compressed and uncompressed metadata:

    [2014-01-19 01:38:36,679 UTC] [tuf.download] [INFO][_download_file:740@download.py]
    Downloading: http://localhost:8001/metadata/targets/d28735bb21e41668cf526413b2e68f683673e50060b34cd4cb53b582491117c8.django.txt.gz
    [2014-01-19 01:38:36,681 UTC] [tuf.download] [WARNING][_check_downloaded_length:649@download.py]
    Downloaded 1806 bytes out of the expected 1806 bytes.
    [2014-01-19 01:38:36,681 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha224 hash is correct: 20b109dfde808e8f10d239625a2ea7f28624bf00ea1446cac4c3c5e8
    [2014-01-19 01:38:36,682 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]The file's sha256 hash is correct: d28735bb21e41668cf526413b2e68f683673e50060b34cd4cb53b582491117c8
    [2014-01-19 01:38:36,682 UTC] [tuf.client.updater] [INFO][_get_file:1152@updater.py]
    Decompressing http://localhost:8001/metadata/targets/d28735bb21e41668cf526413b2e68f683673e50060b34cd4cb53b582491117c8.django.txt.gz
    [2014-01-19 01:38:36,682 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
    The file's sha224 hash is correct: 54f988d7f25456622a1dbe771fe633b36b1445b9a3f23efe886b6393
    [2014-01-19 01:38:36,682 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]The file's sha256 hash is correct: 4b544a0bd028153be15d93e21717361ccdc93deda21a469e06c65c5eed670ac6

    —
    Reply to this email directly or view it on GitHubhttps://github.com//issues/151#issuecomment-32698834
    .

  17. added a commit that references this issue on Jan 21, 2014
  18. vladimir-v-diaz commented on Jan 21, 2014

    @vladimir-v-diaz
    Contributor

    Update docstrings and comments and complete the initial implementation of issue 151.
    Adjust logger level for tuf.download._check_downloaded_length().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions