Repository navigation
client/updater design: let user download targets? #1142
Description
Activity
I had a look on the current implementation and I would vote for implementing minimal changes in the current Updater code trying to achieve "let user download targets" sooner and then transfer this to the client refactor.
I have some questions and comments for @jku but I'd appreciate general comments related to the client API:
let user handle "downloading target" and expose "verifying target"
I don't see a reason why we cannot expose a
verify_targetfunction and keep the currentdownload_target(which I would rename to a more explicitdownload_verified_targetbut this will change the current API so maybe not).
Anyway, the community may have its security reasons about not letting the client handle downloads on its own and I would like to hear them before implementing it. (@joshuagl @lukpueh @trishankatdatadog )"resolving download URLs" must be better exposed to user
I can speculate what do you mean @jku but can you be more specific :) Do you mean adding
get_list_of_mirrorsor similar to the client module or you dislike something from the function internals?"local cache check" need re-design
Currently "updated_targets" looks for targets in a path constructed by filepath and destination_directory. Do you think this logic has to be handled by user too? Maybe an example form the pip local cache that don't match tuf's expectations?
Reacted by Joshua LockSo I think I was wrong and tuf can handle the cache -- and probably needs to (pip should just disable http caching for these downloads).
- user wants to download target name, checks cache with Updater.updated_targets()
- User asks updater what needs to be downloaded, Updater returns URLs that should be tried in sequence
- user downloads URLs until one works, returns file object to Updater
- Updater verifies the file object
- if failure, returns error
- if success, inserts file object to cache and returns filepath
So the issue about "resolving URLs" is just that there are multiple potential URLs for a target name
One potential solution might be to keep the API as is, but add an optional download handler function argument to
download_target(). This user provided download handler would be called with a download URL argument (potentially multiple times) before download_target() returns.I start liking the idea of passing a download handler, or even not passing it as a parameter to the download function but adding it as a config on updater init.
In TUF the download function signature is:
def _download_file(url, required_length, STRICT_REQUIRED_LENGTH=True):I guess the requirement for a client would be to pass a download handler that can ensure the downloaded file size?
the community may have its security reasons about not letting the client handle downloads on its own and I would like to hear them before implementing it.
Protecting against endless data attacks require us to ensure we are only downloading up to a metadata defined number of bytes. It's easier for the framework to make assertions about that if the framework is controlling downloads.
If the handler implements a pre-defined interface, we can offload that responsibility to the client integrating the framework for targets downloads.
Anyway, the community may have its security reasons about not letting the client handle downloads on its own and I would like to hear them before implementing it. (@joshuagl @lukpueh @trishankatdatadog )
As long as we use 3rd-party code to download, we can never guarantee what it actually does.
Reacted by Joshua Lock
This is a future wishlist item for client API (#1135), based on working on pip integration.
Current issues with TUF from pip point of view:
Some of these could be fixed by adding more complexity to TUF... but potentially the correct fix is to separate downloading from the other TUF functionality and let the clients do that themselves.
High level functionality list of the client currently looks like this:
client.updater.Updater.refresh())client.updater.Updater.get_one_valid_targetinfo())mirrors.get_list_of_mirrors(), happens during all metadata and target downloads)client.updater.Updater.updated_targets())client.updater.Updater.download_target())client.updater.Updater.download_target())Maybe we can optionally let user handle "downloading target" and expose "verifying target" step in a more usable way to the user. This means that "resolving download URLs" must be better exposed to user. Also "local cache check" need re-design: if user downloads files, they also control caching so we can't expect tuf cache directory structure to exist.