Repository navigation
[C++] Filesystem implementation for Azure Blob Storage #18014
Description
Subissues:
- [C++] Skeleton for Azure Filesystem Implementation #35903
- [C++] Configure
azuritein preparation for testing Azure C++ filesystem #36886 - [C++] Implement file reads for Azure filesystem #37511
- [C++] Return filesystem properties not user defined metadata in Azure file reads #38330
- [C++] Implement file writes for Azure filesystem #38333
- [C++] Implement GetFileInfo for a single file in Azure filesystem #38335
- [C++] Implement
GetFileInfoselector for Azure filesystem #38597 - [C++] Implement all important authentication methods for Azure filesystem #38598
- [C++][FS][Azure] Default credential authentication #39262
- [C++][FS][Azure] Workload Identity Authentication #39318
- [C++][FS][Azure] Managed Identity authentication #39320
- [C++][FS][Azure] Client secret authentication #39343
- [C++][FS][Azure] Azure CLI authentation #39344
- [C++][FS][Azure] Environment credential authentication #39345
- [C++][FS] Use default Azure credentials implicitly and support anonymous credentials explictly #39449
- [C++][FS][Azure] Implement
CreateDir()#38699 - [C++][FS][Azure] Implement
DeleteDir()#38700 - [C++][FS][Azure] Implement
DeleteDirContents()#38701 - [C++][FS][Azure] Implement
DeleteRootDirContents()#38702 - [C++][FS][Azure] Implement
DeleteFile()#38703 - [C++][FS][Azure] Implement
DeleteFile()for flat-namespace storage accounts #40074 - [C++][FS][Azure] Implement
Move()#38704 - [C++][FS][Azure] Implement
CopyFile()#38705 - [C++][FS][Azure] Rename
AzurePathtoAzureLocation#38758 - [C++][FS][Azure] Should
GetFileInfo()against a directory always return true without hierarchical namespace support? #38772 - [C++][FS][Azure] Add allow_container_deletion option #38999
- [C++][FS][Azure] Use the generic filesystem tests #39069
- [C++][FS][Azure] Make mutations of test data explicit in azurefs_test.cc #39119
- [C++] Remove the AzureOptions::backend option #39292
- [C++] Inform caller of Azure Storage container not-existing when checking for HNS support #39297
- [C++][FS][Azure] Implement Move() for flat namespace storage accounts #40025
- [C++][FS][Azure] Include user set metadata in
ObjectInputFile::ReadMetadata#40026 - [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri() #40028
- [C++][FS][Azure] Expose parallel transfer config options available in the Azure SDK #40035
- [C++][FS][Azure] Optimise
ObjectAppendStream::DoAppendin the case of many small appends #40036 - [C++][FS][Azure] Prevent reading or writing directory marker blobs #40037
- [C++][FS][Azure] Fix CreateDir and DeleteDir on hierarchical namespace accounts where the directory has a trailing
/#40052 - [C++][FS][Azure] Consider how and if to fix disallowed characters in file metadata #40057
- ...
Reporter: Wes McKinney / @wesm
Assignee: Shefali Singh
Related issues:
- [C++] Filesystem implementation for Azure Data Lake (is cloned by)
- [R] Expose Azure Blob Storage filesystem (is depended upon by)
Original Issue Attachments:
PRs and other links:
Note: This issue was originally created as ARROW-2034. Please see the migration documentation for further details.
Activity
asfimport commentedon May 6, 2020
Wes McKinney / @wesm:
I see that TileDB (MIT license) has built a C++ wrapper for Azure
https://github.com/TileDB-Inc/TileDB/blob/dev/tiledb/sm/filesystem/azure.cc
No this is not moved to fsspec, this is a C++ ticket
asfimport commentedon Jan 28, 2021
Antoine Pitrou / @pitrou:
I'm not sure I understand the relationship between Blob Store and Data Lake. Is Data Lake a higher-level layer above Blob Store? Or are they two different services that would need separate filesystem implementations?
asfimport commentedon Jan 28, 2021
Antoine Pitrou / @pitrou:
According to https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction ,
asfimport commentedon Jan 28, 2021
Uwe Korn / @xhochy:
It is as confusing in reality, here is what they all are (I'm though already 1 year outdated on this):
asfimport commentedon Jan 28, 2021
Antoine Pitrou / @pitrou:
Ha, and here are some interesting resources:
https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-known-issues : "Blob APIs and Data Lake Storage Gen2 APIs can operate on the same data." but "You cannot use blob API and Data Lake Storage APIs to write to the same instance of a file" (sounds ok?)
https://docs.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs : three different kinds of blobs. "Block blobs" sound most useful, though "append blobs" may work too.
https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-namespace : a Data Lake admin can configure a it with a hierarchical filesystem semantics. Apparently it may enable different APIs (??).
asfimport commentedon Jan 28, 2021
Yesh:
There is also https://github.com/Azure/azure-sdk-for-cpp which I’ve tested against adl gen2 .
asfimport commentedon Jan 28, 2021
Antoine Pitrou / @pitrou:
Various pointers about the Azure C++ SDK:
https://devblogs.microsoft.com/azure-sdk/cppintro/#example-code-using-the-c-storage-blob-client-libraryhttps://devblogs.microsoft.com/azure-sdk/cppintro/
https://github.com/Azure/azure-sdk-for-cpp
https://github.com/Azure/azure-sdk-for-cpp/blob/master/sdk/storage/azure-storage-blobs/sample/blob_getting_started.cpp
https://azure.github.io/azure-sdk-for-cpp/ (note the API docs are segregated per component, with separate land pages for Core, Storage, Blog Storage, etc.)
Note that the SDK requires C++14.
asfimport commentedon Dec 2, 2021
Tom Augspurger / @TomAugspurger:
Does Arrow support C+\14 features now (or more specifically, is the SDK being C\14 a problem?) From https://issues.apache.org/jira/browse/ARROW-13744 it seems like C\14 is at least tested, but https://github.com/apache/arrow/blame/master/docs/source/developers/cpp/building.rst#L40 says a "A C\+11-enabled compiler. " is required.
asfimport commentedon Dec 2, 2021
Neal Richardson / @nealrichardson:
I'm not an expert here, but I think we could use C+14 if required and if the compiler supports it. If the compiler doesn't support C 14, we wouldn't be able to build the azure sdk. So the line would be that Arrow requires C 11 at a minimum, and some features are only available with C+14.
asfimport commentedon Dec 2, 2021
Antoine Pitrou / @pitrou:
We require only C+11 in the codebase. We might add C+14-requiring optional components if desired, but that will add complication to the build setup.
asfimport commentedon Dec 21, 2021
Shashanka Balakuntala Srinivasa:
hi @pitrou , we were looking into implementing this feature from our side. I did try to compile the whole arrow code base with c+14 and ran unit tests as well. Everything is passing in local and as mentioned before : [ARROW-13744] [CI] c+14 and 17 nightly job fails - ASF JIRA (apache.org) ticket mentions we have daily build run for validation which are passing.
Since the azure sdks depend on c+14 features, and since we have the code compiled in c 14, can we look into upgrading the c+ version to 14?
Let me know if there are any issues. I will be happy to take those and work on them.
asfimport commentedon Dec 21, 2021
Antoine Pitrou / @pitrou:
[~balakuntala]We can probably require C+\14 for the Azure filesystem implementation only, but the rest of Arrow should remain C+11-compatible.asfimport commentedon Sep 6, 2022
Dean MacGregor:
If someone wants to work on this but doesn't have an Azure account let me know. I can make a storage account for this development/testing
asfimport commentedon Oct 5, 2022
Bipin Mathew:
I have a rudimentary implementation of this that supports import and export to Azure. I tried to align as closely as possible to the s3 implementation, however as I needed it only for a specific use case and have not had a chance to implement all the same methods. Depending on bandwidth, I could probably build out the implementation further. I have never contributed to this project. Do all endpoints need to be implemented before it can be merged into the code base? Can it be built out over time? I attach what I have so far here.
azfs.hazfs.cc
asfimport commentedon Oct 5, 2022
Antoine Pitrou / @pitrou:
[~mathewb001]Well, there's a Github PR open already, did you take a look?29 remaining items
av8or1 commentedon Mar 5, 2024
kou/felipecrv - How close is this to being done? I see a few green-colored items in the list above, but they seem to be completed already (at least the ones I looked at did). Is there anything else we can work on? Will this make it into version 16.0.0? Thanks
felipecrv commentedon Mar 5, 2024
@av8or1 most of it works now.
s3fsdoesn't even supportMove.AzureFileSystemsupportsMoveon accounts with HNS enabled. You don't have to wait for this issue to be closed to start using what's already merged and ready to be part of the 16.0.0 release.av8or1 commentedon Mar 5, 2024
Hi felipe- Thanks. Well, the company can't utilize any library that isn't an official release. I suppose that I could begin writing the code that will utilize the ADLS stuff now, then when 16.0.0 is released (April?), I would be able to produce our product shortly thereafter. What remains to be completed, by the way? Anything I could help with? Thanks
felipecrv commentedon Mar 5, 2024
@av8or1 as I said above:
Movewith Blobs API (not a critical feature at all,s3fsdoesn't even supportMove). Python bindings (PR is open), and URI parsing (PR is open). Are any of these a dealbreaker for you? Everything else will be available in 16.0.0.av8or1 commentedon Mar 20, 2024
@felipecrv OK thank you. Work has been busy. Just now looking at this again. It appears that @kou has completed the URI parsing business (#40028). Thus I will prepare on my end to use the library when it is released. Hopefully in April. Thanks
wirable23 commentedon Mar 26, 2024
@felipecrv do you know when 16.0.0 would be available?
felipecrv commentedon Mar 26, 2024
In April. The time it takes for the release also depends on how smooth the packaging and publishing process goes.
Tom-Newton commentedon Mar 26, 2024
I think ideally #40036 would be taken care of before the 16.0.0 release. I don't have any real world performance numbers but I suspect write performance is currently a bit disappointing.
raulcd commentedon Mar 26, 2024
Hi @Tom-Newton there's no planned release before 16.0.0
The feature freeze for 16.0.0 is planned for the 8th of April.
felipecrv commentedon Mar 27, 2024
I opened a MINOR PR expanding some of the docstrings: #40838
raulcd commentedon Apr 8, 2024
There are some subtasks still opened. @felipecrv should I tag this umbrella issue as 17.0.0?
felipecrv commentedon Apr 8, 2024
AzureFileSystemis already usable and feature-complete even without these open issues being fixed. I'm in favor of closing (marking it as complete) making it part of the 16.0 release in the logs.kou commentedon Apr 8, 2024
Let's close this as complete.
We don't need an umbrella issue for
AzureFileSystem. We can just use separated issues like other components.kou commentedon Apr 9, 2024
Note that the current
AzureFileSystem'sCopyFile()doesn't work with Azure hierarchical namespace support. See also: #41095Some other
AzureFileSystemimplementations for Azure hierarchical namespace support have some problems: #41034I want to add the fix of this (#41068) to 16.0.0.