tags(5) generators based on tree-sitter

[masatake]

I have spent a long time improving ctags.
However, I am also interested in activating the community developing tags(5) based software.

I found about two tree-sitter based tags generators.

• https://github.com/npezza93/ttags
• https://github.com/jha-naman/treetags

I sent a first pull request to treetags.

[masatake]

[yamato@dev64]~/var/ttags% ./target/release/ttags src/*.rs
[yamato@dev64]~/var/ttags% less tags 
[yamato@dev64]~/var/ttags% ./target/release/ttags src/*.rs
[yamato@dev64]~/var/ttags% mv tags ttags.tags
[yamato@dev64]~/var/ttags% ~/var/treetags/target/release/treetags src/*.rs
[yamato@dev64]~/var/ttags% mv tags treetags.tags
-rw-r--r--. 1 yamato yamato 7684 Oct 17 09:21 treetags.tags
-rw-r--r--. 1 yamato yamato 2165 Oct 17 09:20 ttags.tags

[masatake]

$ find ~/var/linux | xargs cat > /dev/null
$ time ./target/release/ttags ~/var/linux
./target/release/ttags ~/var/linux  388.27s user 7.90s system 93% cpu 7:04.96 total
[yamato@dev64]~/var/ttags% time ~/var/treetags/target/release/treetags -f treetags.tags --workers 16 ~/var/linux           
~/var/treetags/target/release/treetags -f treetags.tags --workers 16   584.86s user 18.07s system 687% cpu 1:27.75 total
~/var/ctags-github/ctags --options=NONE -o ctags.tags -R ~/var/linux  85.27s user 11.32s system 88% cpu 1:48.55 total
[yamato@dev64]~/var/ttags% ls -lh *.tags
-rw-r--r--. 1 yamato yamato 1.5G Oct 17 09:46 ctags.tags
-rw-r--r--. 1 yamato yamato 1.7G Oct 17 09:39 treetags.tags
-rw-r--r--. 1 yamato yamato  78M Oct 17 09:32 ttags.tags
[yamato@dev64]~/var/ttags% time ~/var/treetags/target/release/treetags --sort=no -f treetags-no-sowrt.tags --workers 16 ~/var/linux                              
~/var/treetags/target/release/treetags --sort=no -f treetags-no-sowrt.tags  1  571.17s user 17.87s system 982% cpu 59.968 total

Monitoring the output of the top command, treetags uses multiple CPUs aggressively.
Using external sort looks good idea for treetags.

I also tried ptags that runs multiple ctags processes in parallel. However, I got an error.

[yamato@dev64]~/var/ttags% ~/var/ptags/target/debug/ptags -f ptags.tags -t 1 --bin-ctags /home/yamato/bin/ctags -c --options=NONE  -- ~/var/linux  
failed to call ctags
failed to call ctags
EPERM: Operation not permitted
[yamato@dev64]~/var/ttags% strace -f -e execve ~/var/ptags/target/debug/ptags -f ptags.tags -t 1 --bin-ctags /home/yamato/bin/ctags -c --options=NONE  -- ~/var/linux  
execve("/home/yamato/var/ptags/target/debug/ptags", ["/home/yamato/var/ptags/target/de"..., "-f", "ptags.tags", "-t", "1", "--bin-ctags", "/home/yamato/bin/ctags", "-c", "--options=NONE", "--", "/home/yamato/var/linux"], 0x7ffc304b3868 /* 76 vars */) = 0
strace: Process 2150098 attached
[pid 2150098] execve("/home/yamato/.cargo/bin/git", ["git", "ls-files", "--cached", "--exclude-standard"], 0x7fffd24ebca8 /* 76 vars */) = -1 ENOENT (No such file or directory)
[pid 2150098] execve("/home/yamato/perl5/bin/git", ["git", "ls-files", "--cached", "--exclude-standard"], 0x7fffd24ebca8 /* 76 vars */) = -1 ENOENT (No such file or directory)
[pid 2150098] execve("/usr/share/Modules/bin/git", ["git", "ls-files", "--cached", "--exclude-standard"], 0x7fffd24ebca8 /* 76 vars */) = -1 ENOENT (No such file or directory)
[pid 2150098] execve("/usr/local/sbin/git", ["git", "ls-files", "--cached", "--exclude-standard"], 0x7fffd24ebca8 /* 76 vars */) = -1 ENOENT (No such file or directory)
[pid 2150098] execve("/usr/local/bin/git", ["git", "ls-files", "--cached", "--exclude-standard"], 0x7fffd24ebca8 /* 76 vars */) = -1 ENOENT (No such file or directory)
[pid 2150098] execve("/usr/sbin/git", ["git", "ls-files", "--cached", "--exclude-standard"], 0x7fffd24ebca8 /* 76 vars */) = -1 ENOENT (No such file or directory)
[pid 2150098] execve("/usr/bin/git", ["git", "ls-files", "--cached", "--exclude-standard"], 0x7fffd24ebca8 /* 76 vars */) = 0
[pid 2150098] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2150098, si_uid=1000, si_status=0, si_utime=2 /* 0.02 s */, si_stime=1 /* 0.01 s */} ---
strace: Process 2150101 attached
strace: Process 2150102 attached
[pid 2150102] execve("/home/yamato/bin/ctags", ["/home/yamato/bin/ctags", "-L -", "-f -", "--options=NONE"], 0x7fffd24ebca8 /* 76 vars */) = 0
failed to call ctags
failed to call ctags
EPERM: Operation not permitted
[pid 2150101] +++ exited with 0 +++
[pid 2150097] +++ exited with 0 +++
+++ exited with 1 +++

This is the first time I run ptags. I will inspect more.

[masatake]

[yamato@dev64]~/var/ttags%  ./target/release/ttags --help                                                                                                          
ttags 0.4.3
Nick Pezza <pezza@hey.com>

USAGE:
    ttags [FLAGS] [OPTIONS] <files>...
    ttags [FLAGS] [OPTIONS] <SUBCOMMAND>

FLAGS:
    -a, --append      Append tags to existing file
    -h, --help        Prints help information
    -r, --relative    Should paths be relative to cwd? By default, it's relative to tag-file
    -V, --version     Prints version information

OPTIONS:
    -f, --tag-file <FILE|->    File to write tags to. Use '-' to output to stdout [default: ./tags]

ARGS:
    <files>...    Specify files to parse for tags

SUBCOMMANDS:
    help    Prints this message or the help of the given subcommand(s)
    lsp     Creates lsp server

The lsp subcommand looks interesting.

I also find lsp using ctags as a backend.

• https://github.com/netmute/ctags-lsp
• https://github.com/yowu/ctags_ls

[masatake]

Other interesting items

• https://github.com/gladiatr72/mcp-ctags
• https://github.com/osinmv/function-lookup-mcp

[y9san9]

@masatake found this Discussion and wanted to ask something. Right now I am in search for a ctags solution that generates tags using tree-sitter. Tree-sitter becomes a de-facto standard for editors (nvim, helix, VS Code, GitHub Syntax Highlighting), so a lot of people are going to make tree-sitter grammars and it's going to be more and more maintained.

For me universal ctags work good, but for some languages like TypeScript I have to grep for things. I don't use LSP because it's unpredictable and I would like to use ctags for that matter. ttags is unmaintained as from what I see.

Have you thought about adding at least a fallback backend that uses tree-sitter? As a user I would love to say "I want ctags with this tree-sitter-kotlin repo". Because now even some new languages also start to provide tree-sitter grammars. And I can't write parsers myself, so I don't think I can contribute a new language to ctags. But other people already did write a parser for that language, so it might be useful to reuse that

[masatake]

@y9san9 See https://github.com/jha-naman/treetags . I think the maintainer wants to get more feedback from a person like you.

[y9san9]

I will contact them via issues. However, I wanted to hear your opinion your think it worth it to support parallel parsers as there is a shiny parser lib that is becoming a standard. Do you think ctags could evolve into something that utilize these parsers?

[y9san9]

Btw thank you for the fantastic work you've done, I feel that this is so complex piece of software

[y9san9]

The current problem I am having is that typescript declarations are randomly not parsed. And it's the primary case for ctags for me cuz typescript declarations are not greppable :(

[y9san9]

And unless it's a proper parsing it's going to be very hard to parse things correctly