Enry changes the input byte slice

[vmarkovtsev]

enry.GetLanguage changes the contents of its second argument - the byte slice. How to reproduce:

reproduce.go

package main

import (
    "io/ioutil"
    "os"

    "gopkg.in/src-d/enry.v1"
)

func main() {
    data, _ := ioutil.ReadAll(os.Stdin)
    language := enry.GetLanguage("ParabolicPointer.cs", data)
    os.Stdout.Write(data)
    println(language)
}

cat ParabolicPointer.cs | go run reproduce.go > new.cs
diff ParabolicPointer.cs new.cs

You should see

50c50
<         for(int i=0; i<points; i++)
---
>         for(int i=0; i<points;>i++)

ParabolicPointer.cs: ParabolicPointer.zip

This is the root cause of src-d/hercules#178

[creachadair]

$ go run repro.go < ParabolicPointer.cs 
C#
before eb7239ae3fc314c45ac58bf03abbd19aff2a0243
 after 37a2b661c1e43cb2ed34b51c924e424278233fb0

I haven't yet figured out which strategy is the culprit, but I'm working on narrowing it down.

[bzz]

Final decision is made by the DefaultClassifier which seems to be cause of this.

[creachadair]

Specifically, the problem happens in the guts of the tokenizer package. It takes slices into the original input, and then makes replacements into them. Since the slices are views, this winds up changing the content.

The tokens should probably have been strings instead of slices, since that's never really safe when you're making changes. But for now I think we can make a reasonable fix of this by having the tokenizer operate on a copy.

[creachadair]

I've started a fairly straightforward test+fix in #197.