error while removing epsilon from LG file

[Manjunath-mlp]

I am trying to create LG.pt from icefall code using compile_lg.py but while running k2.remove_epsilon
i am getting check failed:
Check failed: num_elems == row_splits[num_rows
] (188777998 vs. 445811431)

Any idea

[csukuangfj]

can you tell us the version of k2 and the size of your LG?

if your k2 is not.the latest one, can you first try the latest one?

[Manjunath-mlp]

k2 versin is latest only, k2 version: 1.24.4.And LG is kind of large ,lm.fst.txt is around 3GB.When i commented the remove epsilon line there is no issue,so i dont think size of lg matters in my case.

[Manjunath-mlp]

or remove epsilon operation is too big to handle in this case?

[csukuangfj]

k2 versin is latest only, k2 version: 1.24.4.

Please show the exact version. And try the latest version if you are not using it.

By latest, I mean the latest commit, not the latest tag.

[csukuangfj]

The latest version 1.24.4.dev20250304

[Manjunath-mlp]

python3 -m k2.version

k2 version: 1.24.4
Build type: Release
Git SHA1: f5a59be
Git date: Fri Nov 22 11:52:24 2024
Cuda used to build k2: 11.7
cuDNN used to build k2:
Python version used to build k2: 3.10
OS used to build k2: Ubuntu 18.04.5 LTS
CMake version: 3.31.0
GCC version: 7.5.0
CMAKE_CUDA_FLAGS: -Wno-deprecated-gpu-targets -lineinfo --expt-extended-lambda -use_fast_math -Xptxas=-w --expt-extended-lambda -gencode arch=compute_61,code=sm_61 -DONNX_NAMESPACE=onnx_c2 -gencode arch=compute_61,code=sm_61 -Xcudafe --diag_suppress=cc_clobber_ignored,--diag_suppress=integer_sign_change,--diag_suppress=useless_using_declaration,--diag_suppress=set_but_not_used,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=implicit_return_from_non_void_function,--diag_suppress=unsigned_compare_with_zero,--diag_suppress=declared_but_not_referenced,--diag_suppress=bad_friend_decl --expt-relaxed-constexpr --expt-extended-lambda -D_GLIBCXX_USE_CXX11_ABI=0 --compiler-options -Wall --compiler-options -Wno-strict-overflow --compiler-options -Wno-unknown-pragmas
CMAKE_CXX_FLAGS: -D_GLIBCXX_USE_CXX11_ABI=0 -Wno-unused-variable -Wno-strict-overflow
PyTorch version used to build k2: 2.0.1+cu117
PyTorch is using Cuda: 11.7
NVTX enabled: True
With CUDA: True
Disable debug: True
Sync kernels : False
Disable checks: False
Max cpu memory allocate: 214748364800 bytes (or 200.0 GB)
k2 abort: False
file: /miniconda3/envs/k2/lib/python3.10/site-packages/k2-1.24.4.dev20241122+cuda11.7.torch2.0.1-py3.10-linux-x86_64.egg/k2/version/version.py
_k2.file: /miniconda3/envs/k2/lib/python3.10/site-packages/k2-1.24.4.dev20241122+cuda11.7.torch2.0.1-py3.10-linux-x86_64.egg/_k2.cpython-310-x86_64-linux-gnu.so

[Manjunath-mlp]

I installed latest k2 version,still same check failed in remove epsilon

[Manjunath-mlp]

Any update on the issue,its compiling for smaller lg size but not for larger size lg

[csukuangfj]

please have a look at
https://github.com/k2-fsa/icefall/blob/master/egs%2Flibrispeech%2FASR%2Flocal%2Fcompile_hlg_using_openfst.py

it is for hlg, but you can change it to support LG.

[Manjunath-mlp]

its working now with hlg script,thanks.