Support for diary sexp timestamps

[unhammer]

I'm guessing this is a long shot, but would it be possible to include events specified with diary sexps in org-ql-select calls, e.g.

** Lunch on last Thursdays of each month 11:00-12:00
     SCHEDULED: <%%(diary-float t 4 (- 1))%>

[alphapapa]

Yes, that should be possible. It would probably be best for it to be optional and disabled by default, because it would preclude some regexp-based optimizations. I probably won't work on this soon, but I'll put it on the list. Thanks.

[alhassy]

@alphapapa Since this is marked as a good first issue, I'd like to try to take it on. Any guidance on where to get started would be appreciated, thanks!

[alphapapa]

@alhassy Cool. I would probably start with the form (org-ql-defpred scheduled ...), copy it, rename it, and start developing it from there. You could omit the "preamble" regexp optimization for now. You'll probably need to refer to some code in the diary or Org libraries that are related.

By the way, I don't recall if you have already, but in order to contribute substantial features to this library, you'll need to have done FSF copyright assignment for Emacs.

[alhassy]

I spent my evening on an orthogonal approach:

;; Get all entries with sexprs, and execute them for the given time range.
(org-ql-view--display
  :header "All dynamically generated sexprs"
  :strings (-flatten (org-ql-query
		       :select '(my/execute-sexpr-in-subtree :from "today" :to "+100d")
		       :from "~/.emacs.d/demo.org"
		       :where '(regexp "%%"))))

Helper Utils

(cl-defun my/execute-sexpr-in-subtree (&key from to)
  "Get a list of all dynamically generated entries from date FROM to date TO for the sexpr entries in the current subtree"
  (cl-loop for sexpr-string in (save-excursion (my/extract-diary-sexprs-in-subtree))
	   for sexpr = (cl-first (read-from-string sexpr-string))
	   append
	   (thread-last
	     (cl-loop  for date in (dates :from from :to to)
		       for org-agenda-current-date = date
		       for entry = "" ;; Placeholder, FIXME. This should be the text /after/  %%(org-calendar-holiday), if any!
		       for result = (eval sexpr)
		       when result
		       collect result))))
  
(defun my/extract-diary-sexprs-in-subtree ()
  "Extract just the S-expressions inside %%(...) in the current Org buffer."
  (interactive)
  (let ((results '())
	(end-point-of-search (save-excursion (org-end-of-subtree t) (point))))
    (save-excursion
      (goto-char (point-min))
      (while (re-search-forward "^%%\\((.*)\\)" end-point-of-search t)
        (push (substring-no-properties (match-string 1)) results)))
    (nreverse results)))


(cl-defun dates (&key from to)
  "Return a list of (month day year) dates from FROM to TO.
FROM and TO are Org-style date strings like \"today\", \"+4d\", \"2025-04-30\"."
  (let* ((from-str (or from "today"))
	 (to-str   (or to "+0d"))
	 (from-time (apply #'encode-time (org-parse-time-string (org-read-date nil nil from-str))))
	 (to-time   (apply #'encode-time (org-parse-time-string (org-read-date nil nil to-str))))
	 (results '()))
    (while (not (time-less-p to-time from-time))
      (let* ((decoded (decode-time from-time))
	     (month (nth 4 decoded))
	     (day   (nth 3 decoded))
	     (year  (nth 5 decoded)))
        (push (list month day year) results))
      (setq from-time (time-add from-time (days-to-time 1))))
    (nreverse results)))

---

@alhassy Cool. I would probably start with the form (org-ql-defpred scheduled ...)

I thought about declaring a new predicate, but I was concerned about how to act on the resulting entries. Namely, I want to run my/execute-sexpr-in-subtree (from the details above) on each Org entry that contains a sexpr. So I spliced the above code into org-ql-search: If query looks like (%% :from x :to y) then do a (regexp "%%") search and perform my/execute-sexpr-in-subtree on each entry, else behave as usual. This approach seems hacky.

Details

(cl-defun my/org-ql-search (buffers-files query &key narrow super-groups sort title
					  (buffer org-ql-view-buffer))
  "Search for QUERY with `org-ql'. QUERY may be of the form `(%% :from x :to y)`."
  (declare (indent defun))
  (interactive (list (org-ql-view--complete-buffers-files)
                     (read-string "Query: " (when org-ql-view-query
                                              (format "%S" org-ql-view-query)))
                     :narrow (or org-ql-view-narrow (equal current-prefix-arg '(4)))
                     :super-groups (org-ql-view--complete-super-groups)
                     :sort (org-ql-view--complete-sort)))
  (block nil
    (with-temp-buffer
      (let* ((query (cl-etypecase query
                      (string (if (or (string-prefix-p "(" query)
                                      (string-prefix-p "\"" query))
                                  ;; Read sexp query.
                                  (read query)
				;; Parse non-sexp query into sexp query.
				(org-ql--query-string-to-sexp query)))
                      (list query)))
	     ;; BEGIN MODIFICATION ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
             (results (pcase query
			(`(%% :from ,from :to ,to)
			 (progn
			   (org-ql-view--display
			     :buffer buffer
			     :header (org-ql-view--header-line-format :buffers-files buffers-files :query query :title title)
			     :strings
			     (-uniq (-flatten (org-ql-query
						:select `(my/execute-sexpr-in-subtree :from ,from :to ,to)
						:from buffers-files
						:narrow narrow
						:order-by sort
						:where '(regexp "%%")))))
			   ;; NOTE: I couldn't get these strings to work with #'org-ql-view--format-element, so returning early.
			   (cl-return)))
			(t (org-ql-select buffers-files query
			     :action 'element-with-markers
			     :narrow narrow
			     :sort sort))))
	     ;; END MODIFICATION ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
	     (strings (-map #'org-ql-view--format-element results))
	     (buffer (or buffer (format "%s %s*" org-ql-view-buffer-name-prefix (or title query))))
	     (header (org-ql-view--header-line-format
		      :buffers-files buffers-files :query query :title title))
	     ;; Bind variables for `org-ql-view--display' to set.
	     (org-ql-view-buffers-files buffers-files)
	     (org-ql-view-query query)
	     (org-ql-view-sort sort)
	     (org-ql-view-narrow narrow)
	     (org-ql-view-super-groups super-groups)
	     (org-ql-view-title title))	
	(when super-groups
	  (let ((org-super-agenda-groups (cl-etypecase super-groups
					   (symbol (symbol-value super-groups))
					   (list super-groups))))
	    (setf strings (org-super-agenda--group-items strings))))
	(org-ql-view--display :buffer buffer :header header :strings strings)))))


;; Example use
(my/org-ql-search (org-agenda-files) '(%% :from "-1d" :to "+10d") :title "Holidays")

---

By the way, I don't recall if you have already, but in order to contribute substantial features to this library, you'll need to have done FSF copyright assignment for Emacs.

I'm happy to do so. Do I just need to follow the instructions at https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=doc/Copyright/request-assign.changes;h=a430efff118f267119234142b21f8fd5f3a8b6ba;hb=HEAD ?

[alphapapa]

I'm not exactly sure what you're trying to do. To be clear, the purpose of Org QL predicates is to return non-nil if an entry matches the arguments to the predicate. What the user does with matching entries is up to the user.

I'm not certain if support for diary sexps can be added to the existing timestamp-based predicates without harming performance, so I suggest starting with a new, diary sexp-specific predicate first. I'd get it to match reliably and safely (remembering that you're calling eval--and from looking at diary-lib, it appears that that's what's intended), then start looking at performance as compared to the existing timestamp predicates. If support for diary sexps can be integrated into the existing timestamp predicates safely and without harming performance, great; if not, they can live as separate predicates for the users who need them.

I'd also suggest using some test-driven development for the matching part, to ensure that the predicate is matching exactly what it's supposed to.

Yes, you can follow those instructions; you can also request the form on emacs-devel, but AFAIK it's the same content.

[alhassy]

To be clear, the purpose of Org QL predicates is to return non-nil if an entry matches the arguments to the predicate.

Entries that have diary sexps can already be obtained, more or less, using the regexp predicate.

I'm not exactly sure what you're trying to do.

I considered the original question posted in this thread to mean If I write an org-ql query for items scheduled in a certain time range, then the following event should be present if it falls within that range.

** Lunch on last Thursdays of each month 11:00-12:00
     SCHEDULED: <%%(diary-float t 4 (- 1))%>

Generalizing on this idea, what if you want to see all of your holidays for a specific time range? Say you have

** Holy Days & Holidays                          

%%(org-calendar-holiday)

With this understanding in mind, I came up with the (%% :from x :to y) selector.

• This selector is more general than simply looking at scheduled entries and evaluating their sexp forms.