15 minute implementation in TXR Lisp.

Background: TXR already Lisp has quasi-string-literals, which are template strings that do implicit interpolation when evaluated. They do not produce an object where you can inspect the values and fixed strings and do things with these before the merge.

  1> (let ((user "Bob") (greeting "how are you?"))
       `Hello @user, @greeting`)
  "Hello Bob, how are you?"

The underlying syntax behind the `...` notation is the sys:quasi expression. We can quote the quasistring and look at the car (head symbol) and cdr (rest of the list):

  2> (car '`Hello @user, @greeting`)
  sys:quasi
  3> (cdr '`Hello @user, @greeting`)
  ("Hello " @user ", " @greeting)

So that is a bit like f-strings.

OK, now with those pieces, I just right now made a macro te that gives us a template object.

  4> (load "template")
  nil

You invoke it with one argument as (te <quasistring>)

  5> (let ((user "Bob") (greeting "how are you?"))
       (te `Hello @user, @greeting`))
  #S(template merge #<interpreted fun: lambda (#:self-0073)> strings #("Hello " ", ")
              vals #("Bob" "how are you?"))
  6> *5.vals
  #("Bob" "how are you?")
  7> *5.strings
  #("Hello " ", ")
  8> *5.(merge)
  "Hello Bob, how are you?"
  9> (set [*5.vals 0] "Alice")
  "Alice"
  10> *5.(merge)
  "Hello Alice, how are you?"

You can see the object captured the values from the lexical variables, and we can rewrite them, like changing Bob to Alice. When we call the merge method on the object, it combines the template and the values.

(We cannot alter the strings in this implementation; they are for "informational purposes only").

Here is how the macro expands:

  11> (macroexpand-1 '(te `Hello @user, @greeting`))
  (new template
    merge (lambda (#:self-0073)
            (let* ((#:vals-0074
                    #:self-0073.vals)
                   (#:var-0075
                    [#:vals-0074
                      0])
                   (#:var-0076
                    [#:vals-0074
                      1]))
              `Hello @{#:var-0075}, @{#:var-0076}`))
    strings '#("Hello " ", ")
    vals (vec user greeting))

It produces a constructor invocation (new template ...) which specifies values for the slots merge, strings and vals.

The initialization of strings is trivial: just a vector of the strings pulled from the quasistring.

The vals slot is initialized by a `(vec ...)` call whose arguments are the expressions from the quasistring. This gets evaluated in the right lexical scope where the macro is expanded. This is how we capture those values.

The most complicated part is the lambda expression that initializes merge. This takes a single argument, which is the self-object, anonymized by a gensym variable for hygiene. It binds the .vals slot of the object to another gensym lexical. Then a genyms local variable is bound for each value, referencing into consecutive elements of the value vector. E.g. #:var-0075 is bound to [#:vals-0074 0], the first value.

The body of the let is a transformed version of the original template, in which the interpolated expressions are replaced by gensyms, which reference the bindings that index into the vector.

The complete implementation in template.tl (referenced by (load "template") in command line 4) is:

  (defstruct template ()
    merge
    strings
    vals)

  (defun compile-template (quasi)
    (match (@(eq 'sys:quasi) . @args) quasi
      (let ((gensyms (build-list))
            (exprs (build-list))
            (strings (build-list))
            (xquasi (build-list '(sys:quasi)))
            (self (gensym "self-"))
            (vals (gensym "vals-")))
        (while-true-match-case (pop args)
          ((@(eq 'sys:var) @(bindable @sym))
           exprs.(add sym)
           (let ((g (gensym "var-")))
             gensyms.(add g)
             xquasi.(add g)))
          ((@(eq 'sys:expr) @expr)
           exprs.(add expr)
           (let ((g (gensym "expr-")))
             gensyms.(add g)
             xquasi.(add g)))
          (@(stringp @str)
           strings.(add str)
           xquasi.(add str))
          (@else (compile-error quasi
                                "invalid expression in template: ~s" else)))
        ^(new template
              merge (lambda (,self)
                      (let* ((,vals (qref ,self vals))
                             ,*[map (ret ^(,@1 [,vals ,@2])) gensyms.(get) 0])
                        ,xquasi.(get)))
              strings ',(vec-list strings.(get))
              vals (vec ,*exprs.(get))))))

  (defmacro te (quasi)
    (compile-template quasi))

We can see an expansion:

That Lisp Curse document, though off the mark in general, was right the observation that social problems in languages like Python are just technical problems in Lisp (and often minor ones).

In Python you have to wait for some new PEP to be approved in order to get something that is like f-strings but gives you an object which intercepts the interpolation. Several proposals are tendered and then one is picked, etc. People waste their time producing rejected proposals, and time on all the bureucracy in general.

In Lisp land, oh we have basic template strings already, let's make template objects in 15 minutes. Nobody else has to approve it or like it. It will backport into older versions of the language easily.

P.S.

I was going to have the template object carry a hash of those values that are produced by variables; while coding this, I forgot. If we know that an interpolation is @greeting, we'd like to be access something using the greeting symbol as a key.

(I don't see any of this is as useful, so I don't plan on doing anything more to it. It has no place in Lisp, because for instance, we would not take anything resembling this approach for HTML generation, or anything else.)

PEP 750 – Template Strings