Implement the core functionality for the connect and make_request methods of the Backend abstract class. #4447

Overview

This task involves adding core functionality for two key methods within a Backend abstract class:

• connect: Establishes a connection to our external backend service.
• make_request: Sends requests to the connected service and handles the response accordingly.

Description and outcomes

• Locate the Backend and BackendResponse classes. They reside in contentcuration/automation/utils/appnexus/base.py.

• Define variables:

  • base_url: Store the specific backend instance's base url.
  • connect_endpoint: Use this path to check backend availability.

• Sessions

  • Sessions are used to construct requests in the Backend class
  • One of the biggest benefits to using sessions is the ability to reuse the same TCP connection for multiple requests on the same host.
  • Sessions could present connectivity challenges if not handled correctly (for example stale connections). One way to ensure that connections are reliable is to implement a mechanism that sets the maximum connection age for sessions, ensuring they are only reusable for specific periods of time, after which they are recycled.
  • Below are steps on how a custom session class with maximum connection age capabilities could be implemented.
    • Create a class named SessionWithMaxConnectionAge that extends requests.Session in contentcuration/automation/utils/appnexus/base.py
    • In its constructor, accept an age argument for dynamic assignment of the maximum connection age. It has a default time of 10s.
    • Within __init__, declare and initialize a variable last_used with the current time, to keep track of when the session was last used.
    • Override request() and implement the logic below;
      • If the current_time - last_used > age, close any open connections and reinitialize session
      • Set the last_used variable to the current time.
      • Return the response, by calling super().request(...)
      • Usage:

      # Initializes a session with a maximum connection age of 60 seconds
      this.session = SessionWithMaxConnectionAge(60)
      

• Initialization

  • Backend
    • In the constructor, accept a url_prefix argument for dynamic assignment of url prefixes during instantiation(eg, version)
    • Within __init__, declare and initialize a session variable (of type SessionWithMaxConnectionAge()) that will be used to make requests.
  • BackendResponse
    • In the constructor, accept kwargs
    • Within __init__, dynamically set the class attributes using kwargs

• Construct full path

  • Create a private method named _construct_full_url.
  • This method should combine base_url, url_prefix, and path in that order, removing any trailing slashes beforehand.

  base_url = 'http://localhost:8000'
  url_prefix = '/v1'
  path = '/embed-topics'
  full_url = _construct_full_url(base_url, url_prefix, path)
  print(full_url)
  # http://localhost:8000/v1/embed-topics

• Connect to the backend

  • Attempt to make a connection in the connect method using the defined session variable, utilizing both the connect_endpoint and _construct_full_url as needed.
  • Ensure proper error handling based on the list of exceptions supported by the requests python package. Also, you can use this connect method as inspiration.
    • Recycle the session when RequestException, or ConnectionError are initially encountered and retry the request, else raise an exception.

• Make requests

  • The make_request method should use the session, connect_endpoint, _construct_full_url, and the provided request object to make requests to the backend.
  • The request object carries information like path, method, headers, params, and body, essential for invoking session.request().
  • Extract the JSON from the response above and use it to create the BackendResponse object, which is then returned.
  • Ensure proper error handling based on the list of exceptions supported by the requests python package. Also, you can use this request method as inspiration.
    • Recycle the session when RequestException, or ConnectionError are initially encountered and retry the request, else raise an exception.

• Error handling

  • The requests package raises these exceptions, that can be categorized as below;
    • Connection related
      • RequestException
      • ConnectionError
    • Timeout related
      • Timeout
    • HTTP related
      • HTTPError
      • TooManyRedirects
    • URL related
      • URLRequired
    • Response related
      • JSONDecodeError
  • Implement the error handling based on the categorizations above
  • For each category, create a new exception class descriptive of the category. Add them to contentcuration/automation/utils/appnexus/base.py
  • Raise the created exception classes for each category
  • Add error logging for each raised exception

Acceptance Criteria

• Initialization
  • The Backend class constructor accepts a url_prefix argument for dynamic assignment of url prefixes during instantiation.
  • A session variable of type SessionWithMaxConnectionAge is declared within the Backend class constructor.
  • The BackendResponse class constructor accepts the kwargs argument.
  • A private _construct_full_url method in the Backend class constructs the full url by combining base_url, url_prefix, and path.
• Connect to the backend:
  • The connect method uses the session variable and _construct_full_url to attempt a connection to the backend using the connect_endpoint.
  • Appropriate error handling is implemented based on the supported exceptions.
• Make requests
  • The make_request method uses the session variable and _construct_full_url to make requests to the backend using the specified request object.
  • The method returns a BackendResponse object constructed from the JSON response of the session.request() call.
  • Appropriate error handling is implemented based on the supported exceptions.
• The BackendResponse class constructor accepts keyword arguments and dynamically sets corresponding attributes.
• Tests are written to verify correctness of added code

Assumptions and Dependencies

• body from the request object is of type JSON, thus should be handled appropriately.
• Whereas some variable and method names already exist in the code, others are presented as suggestions. Feel free to adapt these names as necessary, for clarity.

Scope

The scope of this task is limited to;

• Implementing the core functionality of the connect and make_request methods

Accessibility Requirements

NA

Resources

• requests
• NetworkClient class