Skip to content

Custom inference session for improved ONNX model handling - #24

Merged
CodeWithKyrian merged 4 commits into
0.4.0-devfrom
modify-inference-session-tensor
Apr 23, 2024
Merged

CodeWithKyrian merged 4 commits into
0.4.0-devfrom
modify-inference-session-tensor

Conversation

@CodeWithKyrian

Copy link
Copy Markdown
Owner

What:

  • Bug Fix
  • New Feature

Description:

This PR introduces a custom inference session class that improves how ONNX inference is handled within TransformersPHP. Previously, the original inference session from ankane/onnnxruntime-php processed inputs and outputs as flat arrays. Since TransformersPHP primarily works with tensors, this created extra conversion steps.

Here's how the original process worked:

  • Convert input tensor to a standard PHP array.
  • Convert PHP array to a C array for the ONNX session.
  • Model processes the input and returns a C array.
  • Convert the C array back to a multidimensional PHP array.
  • Convert the PHP array back to a tensor.

With the new custom inference session, these unnecessary conversions and overheads are eliminated, conserving memory and improving performance. Now, the session accepts a tensor as input and returns a tensor as output, streamlining the process to:

  • Convert input tensor to a C array for the ONNX session.
  • Model processes the input and returns a C array.
  • Convert the C array back to a tensor.

Additionally, the custom inference session resolves an issue with zero-sized tensors. The previous approach struggled with zero-sized arrays, losing contextual information about their shape. By working directly with tensors, the new inference session retains shape information even for zero-sized tensors. This allows for accurate memory allocation and shape management, eliminating the need for manual adjustments for attention masks in decoder models.

In summary, this PR optimizes the handling of inputs and outputs in ONNX inference sessions, reducing conversion overhead and improving memory management and performance for larger model inputs and outputs.

@CodeWithKyrian
CodeWithKyrian merged commit 2200fa3 into 0.4.0-dev Apr 23, 2024
@CodeWithKyrian
CodeWithKyrian deleted the modify-inference-session-tensor branch April 23, 2024 07:49
@martindewawd martindewawd mentioned this pull request Mar 25, 2025
1 of 4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant