Support for Intel SHA extensions

Intel SHA instructions assist with hardware acceleration of the SHA-1 and SHA-256 hash algorithms.

Current Ryzen processors that support these instructions can reach SHA-256 speeds of around 2 GB/s, roughly 4x faster than not using SHA instructions.

Support for these instructions via .NET Core's hardware intrinsics would be useful in SHA-heavy workloads.

Proposed API

namespace System.Runtime.Intrinsics.X86
{
    public class Sha
    {
        /// <summary>
        /// __m128i _mm_sha1msg1_epu32 (__m128i a, __m128i b)
        /// SHA1MSG1 xmm, xmm/m128
        /// </summary>
        public static Vector128<byte> Sha1MessageSchedule1(Vector128<byte> a, Vector128<byte> b) => MessageSchedule1(a, b);

        /// <summary>
        /// __m128i _mm_sha1msg2_epu32 (__m128i a, __m128i b)
        /// SHA1MSG2 xmm, xmm/m128
        /// </summary>
        public static Vector128<byte> Sha1MessageSchedule2(Vector128<byte> a, Vector128<byte> b) => MessageSchedule2(a, b);

        /// <summary>
        /// __m128i _mm_sha1nexte_epu32 (__m128i a, __m128i b)
        /// SHA1NEXTE xmm, xmm/m128
        /// </summary>
        public static Vector128<byte> Sha1NextE(Vector128<byte> a, Vector128<byte> b) => NextE(a, b);

        /// <summary>
        /// __m128i _mm_sha1rnds4_epu32 (__m128i a, __m128i b, const int func)
        /// SHA1RNDS4 xmm, xmm/m128, imm8
        /// </summary>
        public static Vector128<byte> Sha1Rounds(Vector128<byte> state1, Vector128<byte> state2, byte func) => Rounds(state1, state2, func);

        /// <summary>
        /// __m128i _mm_sha256msg1_epu32 (__m128i a, __m128i b)
        /// SHA256MSG1 xmm, xmm/m128
        /// </summary>
        public static Vector128<byte> Sha256MessageSchedule1(Vector128<byte> a, Vector128<byte> b) => MessageSchedule1(a, b);

        /// <summary>
        /// __m128i _mm_sha256msg2_epu32 (__m128i a, __m128i b)
        /// SHA256MSG2 xmm, xmm/m128
        /// </summary>
        public static Vector128<byte> Sha256MessageSchedule2(Vector128<byte> a, Vector128<byte> b) => MessageSchedule2(a, b);

        /// <summary>
        /// __m128i _mm_sha256rnds2_epu32 (__m128i a, __m128i b, __m128i k)
        /// SHA256RNDS2 xmm, xmm/m128, &lt;XMM0&gt;
        /// </summary>
        public static Vector128<byte> Sha256Rounds(Vector128<byte> state1, Vector128<byte> state2, Vector128<byte> message) => Rounds(state1, state2, message);
    }
}

The naming of the functions and parameters still need work.

@Thealexbarney

@Thealexbarney thanks for the proposal. Our API review process is detailed here: https://github.com/dotnet/runtime/blob/master/docs/project/api-review-process.md

Could you please try to rewrite the original post to explicitly call out the APIs you would like to be added. We have a "good example" of a complete proposal here: https://github.com/dotnet/corefx/issues/271

Namely, we are looking for something like the Proposed API section at a minimum, but adding additional sections can help with the overall review process.

In this particular case it looks like there are 7 APIs that would be split between System.Runtime.Intrinsics.X86.Sha1 and System.Runtime.Intrinsics.X86.Sha256

Support for these instructions via .NET Core's hardware intrinsics would be useful in SHA-heavy workloads.

I'm curious what the use case is for these intrinisics? The SHA implementations provided by OpenSSL, CNG, etc should already be using these intrinisics. For example, https://github.com/openssl/openssl/blob/3c957bcd54d097167e53660fa100aa1ba85d63b5/crypto/sha/asm/sha256-mb-x86_64.pl#L539

Realistically no one should be using these intrinsics and they should only be using the known good implementations from locations like System.Security.

However, we've also said that we don't see a reason to block developers from having access to them as other languages (C, C++, Rust, etc) expose the corresponding intrinsics and there may be valid use cases for non-cryptographic scenarios (for example, a fast checksum of a downloaded web file without taking a dependency on a third party, not installed by default, library).

I'm curious what the use case is for these intrinisics? The SHA implementations provided by OpenSSL, CNG, etc should already be using these intrinisics.

The use case would be similar to the use case of AES-NI intrinsics.
Calling out to an outside implementation often has a performance impact, especially when working with small message sizes. Support for intrinsics would avoid that overhead by keeping that computation in-process. I suppose it also gives developers more freedom in creating their own implementations, although I don't know what benefit that would provide.

I agree that we should make these public for symmetry with AES, but I would strongly implore any developer not to use these APIs unless you really, really, really know what you're doing. Normally, writing secure assembly-level cryptography involves exercising tight control over register allocation, stack spilling, and knowledge about where your thread might be interrupted. The .NET Framework doesn't give developers such low-level control over program compilation and execution, and it's enormously easy to shoot yourself in the foot by making an algorithm which appears to be faster but which in actuality inadvertently leaks secrets.

I've updated the top comment with what the API would look like. The naming of the functions and parameters still need work.

And as a side-note, my current use cases for both AES and SHA intrinsics don't need to be completely secure. Access to intrinsics has greatly sped up AES modes that aren't supported by .NET, close to the performance hand-written assembly would get. XTS mode is ~3-4x faster and doesn't require any tricks to reduce overhead from the crypto API calls.

@tannergooding

@tannergooding is there anything missing from the proposal? If not I suppose we can mark it ready for review.

@tannergooding

@tannergooding, happy to assign to me. :)

Did we end up with the linked analyzer proposal mentioned above?

@tannergooding

@tannergooding You said in the Approval video that there are two different CPUID flags.

However, under x86 there is only one CPUID flag "SHA".

Also, it is difficult to generate these classes with the ThunkGenerator (see #62999 (comment)), as this is an exceptional case (for reference: C# Discord #lowlevel conversation).

Could you enlighten us? If this was a mistake, I would suggest the following:

namespace System.Runtime.Intrinsics.X86
{
    public class Sha
    {
        public static bool IsSupported { get; }
        public static Vector128<byte> Sha1MessageSchedule1(Vector128<byte> a, Vector128<byte> b);
        public static Vector128<byte> Sha1MessageSchedule2(Vector128<byte> a, Vector128<byte> b);
        public static Vector128<byte> Sha1NextE(Vector128<byte> a, Vector128<byte> b);
        public static Vector128<byte> Sha1FourRounds(Vector128<byte> state1, Vector128<byte> state2, byte func);
        public static Vector128<byte> Sha256MessageSchedule1(Vector128<byte> a, Vector128<byte> b);
        public static Vector128<byte> Sha256MessageSchedule2(Vector128<byte> a, Vector128<byte> b);
        public static Vector128<byte> Sha256TwoRounds(Vector128<byte> state1, Vector128<byte> state2, Vector128<byte> message);
    }
}

/cc @saucecontrol
/cc @MichalStrehovsky

Did we end up with the linked analyzer proposal mentioned above?

Judging from the title of Issue dotnet/roslyn-analyzers#3646, the issue is not only AES-specific but general for cryptographic types

It was likely a misstatement or misremembering, etc.

The proposed names look like what I was going to suggest, but we'll need to take it back to API review in either case.

Updated the proposal, it needs to go back through API review.

Video

Generally looks good as proposed.

Parameter names and the *Rounds method names were aligned with the previously approved version.

namespace System.Runtime.Intrinsics.X86
{
    public abstract class Sha : X86Base
    {
       public static bool IsSupported { get; }

       public static class X64 : X86Base.X64 // unless it's SSE(2).
       {
           public static bool IsSupported { get; }
       }

        /// <summary>
        /// __m128i _mm_sha1msg1_epu32 (__m128i a, __m128i b)
        /// SHA1MSG1 xmm, xmm/m128
        /// </summary>
        public static Vector128<byte> Sha1MessageSchedule1(Vector128<byte> a, Vector128<byte> b) => MessageSchedule1(a, b);

        /// <summary>
        /// __m128i _mm_sha1msg2_epu32 (__m128i a, __m128i b)
        /// SHA1MSG2 xmm, xmm/m128
        /// </summary>
        public static Vector128<byte> Sha1MessageSchedule2(Vector128<byte> a, Vector128<byte> b) => MessageSchedule2(a, b);

        /// <summary>
        /// __m128i _mm_sha1nexte_epu32 (__m128i a, __m128i b)
        /// SHA1NEXTE xmm, xmm/m128
        /// </summary>
        public static Vector128<byte> Sha1NextE(Vector128<byte> a, Vector128<byte> b) => NextE(a, b);

        /// <summary>
        /// __m128i _mm_sha1rnds4_epu32 (__m128i a, __m128i b, const int func)
        /// SHA1RNDS4 xmm, xmm/m128, imm8
        /// </summary>
        public static Vector128<byte> Sha1FourRounds(Vector128<byte> a, Vector128<byte> b, byte func) => Rounds(state1, state2, func);

        /// <summary>
        /// __m128i _mm_sha256msg1_epu32 (__m128i a, __m128i b)
        /// SHA256MSG1 xmm, xmm/m128
        /// </summary>
        public static Vector128<byte> Sha256MessageSchedule1(Vector128<byte> a, Vector128<byte> b) => MessageSchedule1(a, b);

        /// <summary>
        /// __m128i _mm_sha256msg2_epu32 (__m128i a, __m128i b)
        /// SHA256MSG2 xmm, xmm/m128
        /// </summary>
        public static Vector128<byte> Sha256MessageSchedule2(Vector128<byte> a, Vector128<byte> b) => MessageSchedule2(a, b);

        /// <summary>
        /// __m128i _mm_sha256rnds2_epu32 (__m128i a, __m128i b, __m128i k)
        /// SHA256RNDS2 xmm, xmm/m128, &lt;XMM0&gt;
        /// </summary>
        public static Vector128<byte> Sha256TwoRounds(Vector128<byte> a, Vector128<byte> b, Vector128<byte> k) => Rounds(state1, state2, message);
    }
}

The baseline should be SSE2, matching AES. This is dependent on the SIMD registers (and so therefore dependent on SSE) and is dependent on the integer based SIMD support (and so therefore dependent on SSE2).

Most ISAs in the architecture manual have documentation similar to the following:

12.13.4 Checking for AESNI Support
Before an application attempts to use AESNI instructions or PCLMULQDQ, the application should follow the steps illustrated in Section 11.6.2, “Checking for SSE/SSE2 Support.” Next, use the additional step provided below:
Check that the processor supports AESNI (if CPUID.01H:ECX.AESNI[bit 25] = 1); check that the processor supports PCLMULQDQ (if CPUID.01H:ECX.PCLMULQDQ[bit 1] = 1).

Unfortunately the SHA Extensions ISA does not have any such documentation; not even a section stating to just check CPUID.07H:EBX.SHA[bit 29] = 1. However, given when it was released, the features it uses, and other considerations SSE2 is the logical dependency.

I'll be following up with our friends at Intel/AMD to confirm it isn't something later instead; but SSE2 is the correct starting point.

For tracking: The implementation is basically ready in #62999 - only the changes to the JIT/VM are buggy (bad codegen -> SEHException). SEHException.
If someone JIT-experienced is looking for a little challenge, you are welcome to have a look.

Navigation Menu

Repository navigation

Support for Intel SHA extensions #256

Proposed API

14 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support for Intel SHA extensions #256

Description

Proposed API

Activity

tannergooding commented on Nov 26, 2019

vcsjones commented on Nov 26, 2019

tannergooding commented on Nov 26, 2019

Thealexbarney commented on Nov 26, 2019

GrabYourPitchforks commented on Nov 26, 2019

Thealexbarney commented on Nov 26, 2019

GrabYourPitchforks commented on Jan 29, 2020

14 remaining items

deeprobin commented on Dec 19, 2021

danmoseley commented on Dec 20, 2021

deeprobin commented on Jan 7, 2022

deeprobin commented on Jan 7, 2022

tannergooding commented on Jan 7, 2022

tannergooding commented on Jan 11, 2022

bartonjs commented on Jan 18, 2022

tannergooding commented on Jan 18, 2022

deeprobin commented on May 25, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions