DAAC-521: GitHub REST single-repo template and docs by DavidDwyer87 · Pull Request #14 · lucidworks/rest-configurations

DavidDwyer87 · 2026-03-26T16:49:31Z

Summary

Adds github/github-repo-v1.json and github/github-v1.asciidoc — a REST connector template and documentation for crawling a single GitHub repository.

The template targets /repos/{owner}/{repo} and indexes 15 entity types as separate Solr documents: REPOSITORY, ISSUE, PULL_REQUEST, BRANCH, COMMIT, COMMIT_DIFF, TAG, MILESTONE, COLLABORATOR, RELEASE, COMMENT, COMMIT_COMMENT, CONTENT, FOLDER, BLOB.

To crawl multiple repositories, create one datasource per repository.

Design decisions

COMMIT per-branch crawling — COMMIT is a child of BRANCH (not REPOSITORY). The sha=${LW_PARENT_DATA_KEY} query parameter receives the branch name via COMMIT's parentIdKey=name, so commits are crawled across all branches. A commit reachable from multiple branches produces one Solr document per branch (known delta vs classic connector which deduplicates by SHA).
COMMIT_DIFF — child of COMMIT, re-fetches each commit via /commits/{sha} and uses dataPath=files to index one Solr document per changed file with per-file diff fields (filename, status, patch, additions, deletions).
File content via Contents API — three-stage chain: CONTENT (root listing) → FOLDER (recursive traversal via recursiveRequest=true) → BLOB (binary download). CONTENT and FOLDER use skipIndexation=true for discovery only. BLOB uses binaryResponse=true with Accept: application/vnd.github.raw+json for Tika-based text extraction.

Test Results (github-rest vs github-classic)

Entity	github-rest	github-classic	Match
repository	1	1	✅
pull_request	15	15	✅
branch	7	7	✅
commit	49	46	~+3 (shared commits across branches)
commit_diff	90	80	~+10 (from extra per-branch commits)
collaborator	23	23	✅
tag	1	1	✅
blob	11	4	REST=file content, Classic=tree nodes
issue_comment	4	4	✅
Total	201	190

Known Feature Parity Gaps (vs GitHub Classic V1 Connector)

Feature	Status	Details
Per-branch commit deduplication	Known delta	Commits reachable from multiple branches are indexed per-branch; classic deduplicates by SHA
ETag incremental crawling	Not supported	Requires persisting ETags per resource across crawls. Deferred.

Related docs:

REST Connector - GitHub Features
Jira: DAAC-521, DAAC-804

Test plan

Validate github-repo-v1.json is valid JSON
Deploy repo template to Fusion and test crawl against a single repo
Verify all 14 entity types are indexed (including COMMIT_DIFF, CONTENT/FOLDER/BLOB)
Verify COMMIT per-branch crawling produces commits across all branches
Verify COMMIT_DIFF extracts per-file change details with dataPath=files
Verify Contents API crawling produces no 404 errors
Compare document counts against classic connector
Review asciidoc for accuracy and completeness

- Update v1 API sunset date from March 31, 2025 to August 5, 2026 - Fix typo: "obtains" → "obtain" - Normalize COMMENT_FOOTER_BLOG limit from 25 to 50 for consistency - Add missing trailing newlines to both files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Replace github-v1.json with github-repo-v1.json for single-repo crawling. Update asciidoc documentation to focus on single-repo template only - users should create one datasource per repository for multi-repo crawling. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add two new child request blocks for repository labels and issue events, bringing the total entity types from 13 to 15. Update documentation with new endpoint rows, permissions, and entity type references. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Revert confluence directory to match main — these changes belong on a separate branch. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Hardcode owner/repo in all child endpoints of the repo template, add BLOB child request under CONTENT to download file content via /git/blobs/{sha} with binaryResponse=true for Tika text extraction. parentIdKey=full_name is retained on all children (connector requires it). CONTENT sets parentIdKey=sha to pass blob SHA to BLOB child. Document expected 404 errors for directory entries in git tree. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…template Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…plate Add COMMIT_DIFF child request (parent: COMMIT) that fetches single-commit detail endpoint to get file-level diff info (files array and stats object). Update AsciiDoc documentation across all sections: entity list, permissions table, rate limiting count (13→14), pagination, variables, endpoints table, parent-child hierarchy notes, and response parsing references. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Remove github-classic comparison sections and internal references that don't belong in recipe docs. Fix BLOB request to use raw Accept header instead of base64. Remove numFetchThreads/retryCount/maxDelayTime from JSON properties level. Update pagination, rate limiting, and parsing descriptions for accuracy.

The /contents endpoint only returns root-level items and is not recursive. Restored /git/trees/HEAD?recursive=1 which returns the full file tree in a single call. Also restored WARNING and NOTE about directory 404 errors when BLOB requests encounter tree entries. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…crawling Switch from /git/trees/HEAD + /git/blobs/{sha} to /contents/{path} using a recursive CONTENT → FOLDER → BLOB pattern (similar to Alfresco connector). This eliminates 404 errors from blob requests hitting directory SHAs and matches the classic GitHub connector approach. Updated both the JSON template and AsciiDoc documentation.

…er-branch crawling - Removed ISSUE and PR_REVIEW_COMMENT entity types from template and docs - Changed COMMIT parent from REPOSITORY to BRANCH with sha=${LW_PARENT_DATA_KEY} - Updated COMMIT_DIFF: dataId to blob_url, dataPath to files - Updated asciidoc docs: entity list, endpoints table, pagination, variables, notes, known limitations, response parsing, and terminology sections - Entity count reduced from 16 to 14

…_COMMENT Renamed ISSUE_COMMENT objectType to COMMENT and removed the separate PR_REVIEW_COMMENT entity since the /issues/comments endpoint already captures both issue and PR comments. Simplified documentation throughout.

…html_url Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

mcondo and others added 12 commits February 10, 2025 11:48

DAAC-38: Confluence Recipe

1ad8999

DAAC-38: Confluence Recipe

3e9e707

DAAC-38: Confluence Recipe

a9df59b

DAAC-38: Confluence Recipe

e3efebe

DAAC-38: Confluence Recipe

2d5312c

update to deprication notice

8f50882

Remove confluence v1 datasource and docs

6edd060

POS for github datasource

ca9eb8b

WIP

011e195

DavidDwyer87 marked this pull request as ready for review March 26, 2026 19:14

DavidDwyer87 and others added 5 commits March 26, 2026 18:27

DAAC-521: Remove confluence changes from branch

c691476

Revert confluence directory to match main — these changes belong on a separate branch. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

DAAC-521: Align confluence files with remote main

4d092c2

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

DAAC-521: Remove LABEL and ISSUE_EVENT entity types from GitHub REST …

485d9fc

…template Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

mcondo reviewed Apr 2, 2026

View reviewed changes

mcondo reviewed Apr 8, 2026

View reviewed changes

Comment thread github/github-repo-v1.json Outdated

Comment thread github/github-v1.asciidoc Outdated

Comment thread github/github-v1.asciidoc Outdated

Comment thread github/github-repo-v1.json

Comment thread github/github-v1.asciidoc Outdated

mcondo reviewed Apr 13, 2026

View reviewed changes

Comment thread github/github-repo-v1.json Outdated

DavidDwyer87 added 2 commits April 14, 2026 11:54

mcondo reviewed Apr 15, 2026

View reviewed changes

Comment thread github/github-v1.asciidoc Outdated

mcondo reviewed Apr 15, 2026

View reviewed changes

Comment thread github/github-repo-v1.json Outdated

mcondo reviewed Apr 15, 2026

View reviewed changes

Comment thread github/github-repo-v1.json Outdated

DavidDwyer87 added 2 commits April 15, 2026 17:39

DAAC-521: Fix BLOB header property from headerName to headerKey

636e2a0

DAAC-521: Add ISSUE and PR_REVIEW_COMMENT entities back to repo template

312223e

DavidDwyer87 and others added 2 commits April 17, 2026 12:04

DAAC-521: Fix table format. Minor adjustment

7ac66bc

mcondo approved these changes Apr 20, 2026

View reviewed changes

DavidDwyer87 and others added 2 commits April 23, 2026 21:15

DAAC-521: Update dataId values to use cleaner identifiers instead of …

9c28b5a

…html_url Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Change DataID from blob_url to filename

2d19fe0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DAAC-521: GitHub REST single-repo template and docs#14

DAAC-521: GitHub REST single-repo template and docs#14
DavidDwyer87 wants to merge 27 commits intomainfrom
DAAC-521-Github-rest

DavidDwyer87 commented Mar 26, 2026 •

edited by mcondo

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

DavidDwyer87 commented Mar 26, 2026 • edited by mcondo Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Design decisions

Test Results (github-rest vs github-classic)

Known Feature Parity Gaps (vs GitHub Classic V1 Connector)

Test plan

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

DavidDwyer87 commented Mar 26, 2026 •

edited by mcondo

Loading