New tool: ingest_email.py #111
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum
reviewed
Dec 3, 2025
Collaborator
gvanrossum
left a comment
gvanrossum
left a comment
There was a problem hiding this comment.
Let's treat this as a draft PR
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
gvanrossum-ms
temporarily deployed
to
build-pipeline
GitHub Actions
Inactive
robgruen
approved these changes
Dec 3, 2025
| """Ingest email files into a database.""" | ||
|
|
||
| # Collect all .eml files | ||
| if verbose: |
Collaborator
There was a problem hiding this comment.
Is there a cleaner way to do this verbose printing? Could there be a printVerbose() method that we call instead of print() and then it decides to print the message or not based on the verbose flag?
Collaborator
There was a problem hiding this comment.
It would feel a little "clever", so I'll skip it.
| print(f" {preview}") | ||
|
|
||
| # Pass source_id to mark as ingested atomically with the message | ||
| source_ids = [email_id] if email_id else None |
Collaborator
There was a problem hiding this comment.
maybe emit a warning when the email id doesn't exist.
Do we try to create one based on the from/to/timestamp/subject hash if we don't have one otherwise?
Collaborator
There was a problem hiding this comment.
We could. I'll defer doing that until there's demand.
| def mark_source_ingested(self, source_id: str) -> None: | ||
| """Mark a source as ingested. | ||
| This performs an INSERT but does NOT commit. It should be called within |
Collaborator
There was a problem hiding this comment.
I haven't tried this, but the docs say you can do cursor.in_transaction to see if there's an active transaction. That could keep someone from calling this if there's no active transaction.
Collaborator
There was a problem hiding this comment.
Meh, this function only exists so add_messages_with_indexing can call it. I'm not too worried about users calling it wrongly.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This does the same as
@add_messagesin test_gmail.py does, but now the storage provider interface has changed to allow storing the "is it ingested" flag per message id in the database, so it is transactionally safe.(Almost all by Claude Opus 4.5 Preview.)
I'm not in a hurry to remove test_email.py -- we need to make sure that all its use cases are now incorporated into either ingest_email.py ro query.py. I think @parse_messages is the only one left.