Improve LHS tensor.pack on non-f32 types for x86

[hanhanW]

We have optimized codegen for packing on f32 types, but not int8. This is a tracking issue for int8 case. I observed that some pack ops are not vectorized. Because masking is only supported on limited ops for dynamic shapes. We should just relax the condition to using isElementwise(), so linalg.transpose op can also get vectorized. I have an easy fix locally, and will send it out for review.

With the change and better distribution logic, we can save up to 43% total dispatch sizes for int8 models, see https://gist.github.com/iree-github-actions-bot/fa5becb880b9a6afc2d362883a585d5a

The next step is having better pack codegen for non-f32 types. We need a pattern to pack innermost tile being a single element and leverage it to 16x16 transpose lowering. Looking at transpose permutation map and using vector.bitcast op should help here.

[hanhanW]

llvm/llvm-project#71454 fixes the vectorization issue.

[hanhanW]

This is the details about 16x16 transpose trick. The 4x4, 8x8, 16x16 tricks have the same idea. https://stackoverflow.com/questions/29519222/how-to-transpose-a-16x16-matrix-using-simd-instructions
Implementation: https://github.com/llvm/llvm-project/blob/main/mlir/lib/Dialect/Vector/Transforms/LowerVectorTranspose.cpp

In your prototype, I think we can add the bitcast pattern before transpose lowering, i.e., https://github.com/openxla/iree/blob/bd2c92dbb3d2109cd624fa18e75b9bf3caaa4ae5/compiler/src/iree/compiler/Codegen/LLVMCPU/LLVMCPUVectorLowering.cpp#L148

You can preset lowering_config; the 16x16 shuffle optimization should be kicked in automatically. It should give us a much better performance. If it works, then we can teach tile size selection about it at https://github.com/openxla/iree/blob/bd2c92dbb3d2109cd624fa18e75b9bf3caaa4ae5/compiler/src/iree/compiler/Codegen/LLVMCPU/KernelDispatch.cpp#L1259-L1270

Here is the benchmark data when I implemented the trick for f32 types: #13318

There can still be some performance improvements left on the table. We should be able to replace the vunpck*pd ones with a combination of shufps + blends that should be faster, in theory. The review comments in https://reviews.llvm.org/D148685 is very helpful to me.

[hanhanW]

I have some prototype in https://github.com/hanhanW/iree/tree/improve-pack, but need to structure them better. The next step in mind to try are

1. Generate vector.bitcast around vector.transpose in virtual vector lowering stage
2. Add a pattern matching for vector.transfer_read -> shape_cast -> bitcast to help flatten them correctly.
3. Similiar pattern for vector.transfer_write chain.
4. Unroll bitcast to 1d vectors, which is prototyped in the branch.

2 and 3 are needed. Otherwise we will generate bunch of scalar vector.bitcast op during 4.