Skip to content

Test with all gRPC Credential classes. #137

Closed

Description

@coryan

Currently both bigtable::AdminClient and bigtable::Client use grpc::GoogleDefaultCredentials, or grpc::InsecureChannelCredentials if taking to the Cloud Bigtable emulator.

We need to support any sort of credentials, this means that the credentials should be a parameter, or set via a policy. Most likely the latter, because some credential types (notably oauth credentials) require a periodic refresh (see GoogleRefreshTokenCredentials in the docs)

One of the problems is to refresh the credentials automatically, not hard if we have an async client, or if we are willing to keep an extra thread to perform the task.

Activity

  1. added
    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
    help wantedWe'd love to have community involvement on this issue.
    on Dec 22, 2017
  2. coryan commented on Dec 22, 2017

    @coryan
    ContributorAuthor

    @sduskis I will use this bug to track the issue with supporting other credential types, including the ones that need refreshing. You should let us know if this should be part of the MVP or not.

  3. coryan commented on Dec 25, 2017

    @coryan
    ContributorAuthor

    This comment is just to record what I have discovered so far. It may seem obvious to folks with more experience with web authentication, but some of us do not know anything about web authentication (e.g., me!), so it might be useful.

    Connecting to the Cloud Bigtable Emulator

    If you are running against the Cloud Bigtable Emulator you really do not need (or want) authentication, so we need to use grpc::InsecureChannelCredentials.

    The bigtable::*Client classes already take care of that.

    Connecting to Cloud Bigtable with a Service Account

    Here, you may want to read the general intro to authentication in gRPC. My interpretation of this document is that basically grpc::GoogleDefaultCredentials works great if you are using a service account to access Cloud Bigtable. There are two sub-cases:

    • If you are running in GCE or GAE, the VM will have all the configuration you need to run with that service account.
    • If you are running outside GCE or GAE, you need to download the private keys of that service account to a file, and set the GOOGLE_APPLICATION_CREDENTIALS to the path of that file

    Connecting to Cloud Bigtable with an OAuth 2.0 Token

    That leaves basically one other case: you are running an application that cannot use a service account, for example, some kind of proxy that will access Cloud Bigtable on behalf of multiple users. I really cannot tell how common this is, but seems like something we need to support.

    In this case you may want to start with the general documentation about using OAuth 2.0 with googleapis.

    I expect this tool to be useful in this case. It can create an access (and refresh?) token for you. It needs the scopes for Cloud Bigtable, which can be found here:

    https://github.com/googleapis/googleapis/blob/master/google/bigtable/bigtable.yaml
    https://github.com/googleapis/googleapis/blob/master/google/bigtable/admin/bigtableadmin.yaml

    The workflow is to obtain a refresh token using the browser (who is supposed to ask the client "do you want to use to access Cloud Bigtable). The application uses the refresh token to get access tokens, the refresh tokens are supposed to be kept secret but last longer than access tokens. The access tokens is what we need to pass to grpc::AccessTokenCredentials. The refresh tokens are what we pass to grpc::RefreshTokenCredentials.

    Connecting to Cloud Bigtable with SSL Authentication

    This is not supported by Google Cloud Platform. The only supported authentication in GCP are user accounts (via OAuth2), service accounts, and API Keys. API Keys are not applicable for Cloud Bigtable.

    Connecting to Cloud Bigtable with Cronet

    TODO - fill out this section. Not clear how this is used, maybe from a Chromium or Android project?

    https://chromium.googlesource.com/chromium/src/+/master/components/cronet?autodive=0%2F%2F

  4. added
    api: bigtableIssues related to the Bigtable API.
    and removed
    api: bigtableIssues related to the Bigtable API.
    on Jan 21, 2018
  5. coryan commented on Jun 14, 2018

    @coryan
    ContributorAuthor

    cc: @sduskis I had completely forgotten about this one. I believe this should just work right out of the box, we just use the gRPC classes for Credentials. But somebody should check, maybe write a couple of tests.

  6. 2 remaining items

  7. manish-qlogic commented on Jul 19, 2018

    @manish-qlogic
    Contributor

    Use gRPC Credential Classes with GCP Bigtable c++ client

    Before you begin

    Set the Environment Variables

    Following environment variables must be set to connect to InstanceAdmin or TableAdmin endlpoints

    • GOOGLE_APPLICATION_CREDENTIALS
      export GOOGLE_APPLICATION_CREDENTIALS="[PATH]"
      where [PATH] is file path for JSON file that contains either user account or Service account key. Please refer this link for more details on Google Authentication.

    • GRPC_DEFAULT_SSL_ROOTS_FILE_PATH
      export GRPC_DEFAULT_SSL_ROOTS_FILE_PATH="[PATH]"
      where [PATH] is file path for gRPC Root Certificate. Please refer this link for more details on gRPC Root Certifacte Configuration.

    Use Access Token Credentials

    • Acquire Access Token
      Use gcloud to acquire an Access Token
    • Sample code to connect to InstanceAdmin endpoint
    #include "google/cloud/bigtable/instance_admin.h"
    #include "google/cloud/bigtable/instance_admin_client.h"
    
    int main(int argc, char* argv[]) try {
      std::string const project_id = argv[1];
      // Put Acquired Access Token
      grpc::string access_token("Valid Access Token");
      auto call_credentials = grpc::AccessTokenCredentials(access_token);
      auto channel_credentials = grpc::SslCredentials(grpc::SslCredentialsOptions());
      auto credentials = grpc::CompositeChannelCredentials(channel_credentials, call_credentials);
      auto instance_admin_client(
          google::cloud::bigtable::CreateDefaultInstanceAdminClient(
              project_id, google::cloud::bigtable::ClientOptions(credentials)));
      google::cloud::bigtable::InstanceAdmin instance_admin(instance_admin_client);
      auto instances = instance_admin.ListInstances();
      return 0;
    } catch (std::exception const& ex) {
      std::cerr << "Standard C++ exception raised: " << ex.what() << std::endl;
      return 1;
    }
    

    Use Refresh Token Credentials

    • Sample code to connect to InstanceAdmin endpoint
    #include "google/cloud/bigtable/instance_admin.h"
    #include "google/cloud/bigtable/instance_admin_client.h"
    
    int main(int argc, char* argv[]) try {
      std::string const project_id = argv[1];
      grpc::string refresh_token = R"""({
          "client_id": "XXXXXX.apps.googleusercontent.com",
          "client_secret": "<actual secret here>",
          "refresh_token": "<actual token value here>",
          "type": "authorized_user"
      })""";
      auto call_credentials = grpc::GoogleRefreshTokenCredentials(refresh_token);
      auto channel_credentials = grpc::SslCredentials(grpc::SslCredentialsOptions());
      auto credentials = grpc::CompositeChannelCredentials(channel_credentials, call_credentials);
      auto instance_admin_client(
          google::cloud::bigtable::CreateDefaultInstanceAdminClient(
              project_id, google::cloud::bigtable::ClientOptions(credentials)));
      google::cloud::bigtable::InstanceAdmin instance_admin(instance_admin_client);
      auto instances = instance_admin.ListInstances();
      return 0;
    } catch (std::exception const& ex) {
      std::cerr << "Standard C++ exception raised: " << ex.what() << std::endl;
      return 1;
    }
    

    Use JWT Access Token Credentials

    • Sample code to connect to InstanceAdmin endpoint
    #include "google/cloud/bigtable/instance_admin.h"
    #include "google/cloud/bigtable/instance_admin_client.h"
    
    int main(int argc, char* argv[]) try {
      std::string const project_id = argv[1];
      grpc::string json_key = R"""({
          "type": "service_account",
          "project_id": "foo-project",
          "private_key_id": "<actual private key id here>",
          "private_key": "<actual private key here>",
          "client_email": "foo-email@foo-project.iam.gserviceaccount.com",
          "client_id": "<actual client id here>",
          "auth_uri": "https://accounts.google.com/o/oauth2/auth",
          "token_uri": "https://accounts.google.com/o/oauth2/token",
          "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
          "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/foo-email%40foo-project.iam.gserviceaccount.com"
    })""";
      auto call_credentials = grpc::ServiceAccountJWTAccessCredentials(json_key, 6000);
      auto channel_credentials = grpc::SslCredentials(grpc::SslCredentialsOptions());
      auto credentials = grpc::CompositeChannelCredentials(channel_credentials, call_credentials);
      auto instance_admin_client(
          google::cloud::bigtable::CreateDefaultInstanceAdminClient(
              project_id, google::cloud::bigtable::ClientOptions(credentials)));
      google::cloud::bigtable::InstanceAdmin instance_admin(instance_admin_client);
      auto instances = instance_admin.ListInstances();
      return 0;
    } catch (std::exception const& ex) {
      std::cerr << "Standard C++ exception raised: " << ex.what() << std::endl;
      return 1;
    }
    
    

    Use Google Compute Engine Credentials

    • Sample code to connect to InstanceAdmin endpoint
    #include "google/cloud/bigtable/instance_admin.h"
    #include "google/cloud/bigtable/instance_admin_client.h"
    
    int main(int argc, char* argv[]) try {
      std::string const project_id = argv[1];
      auto call_credentials = grpc::GoogleComputeEngineCredentials();
      auto channel_credentials = grpc::SslCredentials(grpc::SslCredentialsOptions());
      auto credentials = grpc::CompositeChannelCredentials(channel_credentials, call_credentials);
      auto instance_admin_client(
    	  google::cloud::bigtable::CreateDefaultInstanceAdminClient(
    		  project_id, google::cloud::bigtable::ClientOptions(credentials)));
      google::cloud::bigtable::InstanceAdmin instance_admin(instance_admin_client);
      auto instances = instance_admin.ListInstances();
      return 0;
    } catch (std::exception const& ex) {
      std::cerr << "Standard C++ exception raised: " << ex.what() << std::endl;
      return 1;
    }
    
    

    Note :- One may face "Request had insufficient authentication scopes." error while running above example. This might be due to disabled "Cloud API access scope" for Bigtable. This error can be removed by providing sufficient access as explained here.

    Use of IAM Credentials

    These credentials are not supported.

  8. manish-qlogic commented on Jul 25, 2018

    @manish-qlogic
    Contributor

    @coryan
    An attempt with IAM Credentials resulted into "Received RST_STREAM with error code 1 [13=INTERNAL] " from gRPC server. I have updated these credentials as "not supported".
    Do we keep the bug open and change the milestone or create a new issue just for grpc::GoogleIAMCredentials()?

  9. coryan commented on Jul 26, 2018

    @coryan
    ContributorAuthor

    I just changed the milestone. Let's keep this bug open for the moment. I will unassigned it too since you have done all that can be done here.

  10. added
    type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.
    and removed
    priority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.
    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
    on Sep 12, 2018
  11. changed the title [-]Support all grpc Credential classes.[/-] [+]Test with all gRPC Credential classes.[/+] on May 2, 2019
  12. coryan commented on May 2, 2019

    @coryan
    ContributorAuthor

    After re-reading the bug, we are done here.

  13. evanj commented on Sep 18, 2019

    @evanj

    This should be documented somewhere! Like maybe in the examples, or in the gRPC documentation? This is very hard to figure out on your own.

  14. coryan commented on Sep 19, 2019

    @coryan
    ContributorAuthor

    @evanj thanks, we will lift the results from here to some examples.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: bigtableIssues related to the Bigtable API.help wantedWe'd love to have community involvement on this issue.type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions