It's fascinating how differently languages approach the string formatting design space.
- Java's been trying to add f/t-strings, but its designers appear to be perfectionists to a fault, unable to accept anything that doesn't solve every single problem possible to imagine: [1].
- Go developers seem to have taken no more than 5 minutes considering the problem, then thoughtlessly discarded it: [2]. A position born from pure ignorance as far as I'm concerned.
- Python, on the other hand, has consistently put forth a balanced approach of discussing each new way of formatting strings for some time, deciding on a good enough implementation and going with it.
In the end, I find it hard to disagree with Python's approach. Its devs have been able to get value from first the best variant of sprintf in .format() since 2008, f-strings since 2016, and now t-strings.
> Go developers seem to have taken no more than 5 minutes considering the problem, then thoughtlessly discarded it: [2]. A position born from pure ignorance as far as I'm concerned
There are a million things in go that could be described this way.
Looking at the various conversations involving string interpolation, this characterization is extremely unkind. They've clearly spent a lot more than 5 minutes thinking about this, including writing their own mini-proposals[1].
Are they wrong about this issue? I think they are. There is a big difference in ergonomics between String interpolation and something like fmt.Sprintf, and the performance cost of fmt.Sprintf is non-trivial as well. But I can't say they didn't put any thought into this.
As we've seen multiple times with Go generics and error handling before, their slow progress on correcting serious usability issues with the language stem from the same basic reasons we see with recent Java features: they are just being quite perfectionist about it. And unlike Java, the Go team would not even release an experimental feature unless they feel quite good about it.
> There is a big difference in ergonomics between String interpolation and something like fmt.Sprintf
On the other hand, there’s a difference in localizability as well: the latter is localizable, the former isn’t. (It also worries me that I see no substantive discussion of localization in PEP 750.)
I just expect better from professional language designers. To me, the blindingly obvious follow up to the thought "We understand that people familiar with other languages would like to see string interpolation in Go." [1] is to research how said other languages have gone about implementing this and to present a brief summary of their findings. This is table stakes stuff.
Then there's "You can [get] a similar effect using fmt.Sprint, with custom functions for non-default formatting." [2]:
- Just the fact that "you can already do this" needs to be said should give the designers pause. Clearly you can't already do this if people are requesting a new feature. Indeed, this situation exactly mimics the story of Go's generics - after all, they do not let you do anything you couldn't do before, and yet they got added to Go. It's as if ergonomics matter, huh.
Another way to look at this: if fmt.Sprint is so good it should be used way more than fmt.Sprintf right? Should be easy to prove :)
- The argument crumbles under the load-bearing "similar effect". I already scratched the surface of why this is wrong in a sibling post: [3].
I suspect the reason for this shallow dismissal is the designers didn't go as far as to A/B test their proposal themselves, so their arguments are based on their gut feel instead of experience. That's the only way I can see someone would come up with the idea that fmt.Sprint and f-strings are similar enough. They actually are if all you do is imagine yourself writing the simplest case possible:
fmt.Sprint("This house is ", measurements(2.5), " tall")
f"This house is {measurements(2.5)} tall"
Similar enough, so long as you're willing to handwave away the need to match quotation marks and insert commas and don't spend time coding using both approaches. If you did, you'd find that writing brand new string formatting statements is much rarer than modifying existing ones. And that's where the meat of the differences is buried. Modifying f-strings is trivial, but making any changes to existing fmt.Sprint calls is painful.
P.S. Proposing syntax as noisy as:
fmt.Println("This house is \(measurements(2.5)) tall")
is just another sign the designers don't get it. The entire point is to reduce the amount of typing and visual noise.
Value types anyone? I have zero doubt it is tough to add and get right, esp. to retrofit, but it has been so many years that I have learned/discarded several new languages since Java... and they STILL aren't launched yet.
A format function that arbitrarily executes code from within a format string sounds like a complete nightmare. Log4j as an example.
The rejection's example shows how that arbitrary code within the string could instead be fixed functions outside of a string. Safer, easier for compilers and programmers; unless an 'eval' for strings is what was desired. (Offhand I've only seen eval in /scripted/ languages; go makes binaries.)
In case there was confusion: Python's f-string functionality in particular is specific to string literals. The f prefix doesn't create a different data type; instead, the contents of the literal are parsed at compile time and the entire thing is rewritten into equivalent string concatenation code (although IIRC it uses dedicated bytecodes, in at least some versions).
The t-string proposal involves using new data types to abstract the concatenation and formatting process, but it's still a compile-time process - and the parts between the braces still involve code that executes first - and there's still no separate type for the overall t-string literal, and no way to end up eval'ing code from user-supplied data except by explicitly requesting to do so.
Python source code is translated into bytecode for a VM just like in Java or C#, and by default it's cached in .pyc files. It's only different in that you can ask to execute a source code file and the compilation happens automatically before the bytecode-interpretation.
`SyntaxError` is fundamentally different from other exceptions because it can occur during compilation, and only occurs at run-time if explicitly raised (or via explicit invocation of another code compilation, such as with `exec`/`eval`, or importing a module). This is also why you can't catch a `SyntaxError` caused by the invalid syntax of your own code, but only from such an explicit `raise` or a request to compile a source code string (see https://stackoverflow.com/questions/1856408 ).
My reply was to the parent post's SPECIFIC example of Golang's rejected feature request. Please go read that proposal.
It is NOT about the possibility of referencing existing / future (lazy / deferred evaluation) string literals from within the string, but about a format string that would literally evaluate arbitrary functions within a string.
The proposal doesn't say anything about executing code in user-supplied strings. It only talks about a string literal that is processed by the compiler (at which point no user-supplied string can be available).
On the other hand, the current solution offered by Go (fmt.Sprintf) is the one who supports a user-supplied format String. Admittedly, there is a limited amount of damage that could be done this well, but you can at the very least cause a program to panic.
The reason for declining this feature[1] has nothing to do with what you stated. Ian Lance Taylor simply said: "This doesn't seem to have a big advantage over calling fmt.Sprintf" and "You can a similar effect using fmt.Sprint". He conceded that there are performance advantages to string interpolation, but he doesn't believe there are any gains in usability over fmt.Sprintf/fmt.Sprint and as is usual with Go (compared to other languages), they're loathe to add new features to the compiler[2].
No, the format function doesn't "arbitrarily execute code."
An f/t string is syntax not runtime.
Instead of
"Hello " + subject + "!"
you write
f"Hello {subject}!"
That subject is simple an normal code expression, but one that occurs after the opening quote of the literal and before the ending quote of the literal.
And instead of
query(["SELECT * FROM account WHERE id = ", " AND active"], [id])
you write
query(t"SELECT * FROM account WHERE id = {id} AND active")
It's a way of writing string literals that if anything makes injection less likely.
Thanks for this example - it makes it clear it can be a mechanism for something like sqlc/typed sql (my go-to with python too, don't like orms) without a transpilation step or arguably awkward language API wrappers to the SQL. We'll need linters to prevent accidentally using `f` instead of `t` but I guess we needed that already anyways. Great to be able to see the actual cost in the DB without having to actually find the query for something like `typeddb.SelectActiveAccount(I'd)`. Good stuff.
The PEP says these return a new type `Template`, so you should be able to both type and/or duck type for these specifically and reject non-Template inputs.
The Rejected Golang proposal cited by the post I'm replying to. NOT Python's present PEP or any other string that might resolve magic variables (just not literally eval / exec functions!).
As far as I can tell from the linked proposal, it wouldn't have involved such evaluation either. It seems like it was intended to work fundamentally the same way as it currently does in Python: by analyzing the string literal ahead of time and translating into equivalent explicit formatting code, as syntactic sugar. There seem to have been many misunderstandings in the GitHub discussion.
The inline {variable} reference suffix format would be less confusing for situations that involve _many_ variables. Though I'm a bit more partial to this syntax with an immediately trailing %{variable} packet since my gut feeling is that special case would be cleaner in a parser.
No, it's exactly the opposite--f-strings are, roughly, eval (that is, unsanitary string concatenation that is presumptively an error in any nontrivial use) to t-strings which are just an alternative expression syntax, and do not even dereference their arguments.
D had a big blow up over string interpolation. Walter wanted something simple and the community wanted something more like these template ones from Python (at least from scanning the first little bit of the PEP). Walter eventually went with what the community wanted.
This led to the OpenD language fork (https://opendlang.org/index.html) which is led by some contributors who had other more general gripes with D. The fork is trying to merge in useful stuff from main D, while advancing the language. They have a Discord which unfortunately is the main source of info.
Superficially f-strings reminds you of php and everyone remembers how awful that was. But Python's implementation is leagues better and we also have better tooling (ie smart parsers) for handling fstrings.
I promise, no trolling from me in this comment. I never understood the advantage of Python f-strings over printf-style format strings. I tried to Google for pros and cons and didn't find anything very satisfying. Can someone provide a brief list of pros and cons? To be clear, I can always do what I need to do with both, but I don't know f-strings nearly as well as printf-style, because of my experience with C programming.
Sure, here are the two Go/C-style formatting options:
fmt.Sprintf("This house is %s tall", measurements(2.5))
fmt.Sprint("This house is ", measurements(2.5), " tall")
And the Python f-string equivalent:
f"This house is {measurements(2.5)} tall"
The Sprintf version sucks because for every formatting argument, like "%s", we need to stop reading the string and look for the corresponding argument to the function. Not so bad for one argument but gets linearly worse.
Sprint is better in that regard, we can read from left to right without interruptions, but is a pain to write due to all the punctuation, nevermind refactor. For example, try adding a new variable between "This" and "house". With the f-string you just type {var} before "house" and you're done. With Sprint, you're now juggling quotation marks and commas. And that's just a simple addition of a new variable. Moving variables or substrings around is even worse.
Summing up, f-strings are substantially more ergonomic to use and since string formatting is so commonly done, this adds up quickly.
Nick Humrich here, the author who helped rewrite PEP 501 to introduce t-strings, which was the foundation for this PEP. I am not an author on this accepted PEP, but I know this PEP and story pretty well. Let me know if you have any questions.
I am super excited this is finally accepted. I started working on PEP 501 4 years ago.
I often read concerns that complexity keeps being added to the language with yet another flavour of string or whatnot. Given that those who author and deliberate on PEPs are, kind of by definition, experts who spend a lot of time with the language, they might struggle to grok the Python experience from the perspective of a novice or beginner. How does the PEP process guard against this bias?
There are many long-term users of Python who participate in PEP discussion who argue for beginners[0], often because they professionally are teaching Python.
There are also loads of people basically defaulting to "no" on new features, because they understand that there is a cost of supporting things. I will often disagree about the evaluation of that cost, but it's hard to say there is no cost.
Nobody wants a system that is unusable, slow, hard to implement for, or hard to understand. People sometimes just have different weights on each of these properties. And some people are in a very awkward position of overestimating costs due to overestimating implementation effort. So you end up in discussions like "this is hard to understand!" "No it isn't!"
Hard to move beyond, but the existence of these kinds of conversations serve, in a way, as proof that people aren't jumping on every new feature. Python is still a language that is conservative in what it adds.
This should actually inspire more confidence in people that features added to Python are _useful_, because there are many people who are defaulting to not adding new features. Recent additions to Python speeding up is more an indicator of the process improving and identifying the good stuff rather than a lowering of the bar.
[0]: I often think that these discussions often get fairly intense. Understandability is definitely a core Python value, but I Think sometimes discussions confuse "understandability" with "amount of things in the system". You don't have to fully understand pervasive hashing to understand Python's pervasive value equality semantics! A complex system is needed to support a simple one!
All discussion on PEP's happens in public forums where anyone can opine on things before they are accepted. I agree that the experts are more likely to participate in this exchange. And while this is wish-washy, I feel like the process is really intended to benefit the experts more than the novices anyways.
There have been processes put into place in recent years to try to curb the difficulty of things. One of those is that all new PEPs have to include a "how can you teach this to beginers" section, as seen here on this pep: https://peps.python.org/pep-0750/#how-to-teach-this
I think "how can you teach this to beginners?" is a fantastic, low-hanging fruit option for encouraging the wizards to think about that very important type of user.
Other than a more broad "how is the language as a whole faring?" test, which might be done through surveys or other product-style research, I think this is just plainly a hard problem to approach, just by the nature that it's largely about user experience.
The average Python developer does not even know what a "PEP" is. Open discussion is good yes, but no one really knows what the average developer wants because they simply does not care if its Python or Java or whatever else.
"Some hammers are just shaped weird, oh well, just make do with it."
For example, some people that I interview does not "get" why you have to initialize the dict before doing dict[k] += 1. They know that they have to do some ritual of checking for k in dict and dict[k] = 0. But they don't get that += desugars into dict[k] = dict[k] + 1.
You might find the Python discussion forums ([0] and [1]) interesting; conversation that guides the evolution of PEPs happens there.
As Nick mentioned, PEP 750 had a long and winding road to its final acceptance; as the process wore on, and the complexities of the earliest cuts of the PEPs were reconsidered, the two converged.
Building off this question, it's not clear to me why Python should have both t-strings and f-strings. The difference between the two seems like a stumbling block to new programmers, and my "ideal python" would have only one of these mechanisms.
f-strings immediately become a string, and are "invisible" to the runtime from a normal string. t-strings introduce an object so that libraries can do custom logic/formatting on the template strings, such as decided _how_ to format the string.
My main motivation as an author of 501 was to ensure user input is properly escaped when inserting into sql, which you cant enforce with f-strings.
> ensure user input is properly escaped when inserting into sql
I used to wish for that and got it in JS with template strings and libs around it. For what it’s worth (you got a whole PEP done, you have more credibility than I do) I ended up changing my mind, I think it’s a mistake.
It’s _nice_ from a syntax perspective. But it obscures the reality of sql query/parameter segregation, it builds an abstraction on top of sql that’s leaky and doesn’t even look like an abstraction.
And more importantly, it looks _way too close_ to the wrong thing. If the difference between the safe way to do sql and the unsafe way is one character and a non-trivial understanding of string formatting in python… bad things will happen. In a one-person project it’s manageable, in a bigger one where people have different experiences and seniority it will go wrong.
It’s certainly cute. I don’t thing it’s a good thing for sql queries.
I understand your concern, and I think the PEP addresses it. Quite bluntly, t"foo" is not a string, while f"foo" is. You'll get a typecheck error if you run a typechecker like any reasonable developer, and will get a runtime error if you ignore the type mismatch, because t"foo" even lacks a __str__() method.
One statement the PEP could put front and center in the abstract could be "t-strings are not strings".
For one thing, `f"something"` is of type `str`; `t"something"` is of type `string.templatelib.Template`. With t-strings, your code can know which parts of the string were dynamically substituted and which were not.
The types aren't so important. __call__ or reference returns type string, an f and a t will be interchangeable from the consumer side.
Example, if you can go through (I'm not sure you can) and trivially replace all your fs with ts, and then have some minor fixups where the final product is used, I don't think a migration from one to the other would be terribly painful. Time-consuming, yes.
Give it a few years to when f-string usage has worn off to the point that a decision can be made to remove it without breaking a significant number of projects in the wild.
The linter is a big deal, actually. I've worked with Python off and on during the past few decades; I just recently moved onto a project that uses Python with a bunch of linters and autoformatters enabled. I was used to writing my strings ('foo %s % bar), and the precommit linter told me to write f'foo %{bar}'. Easy enough!
printf-style formatting ("foo %s" % "bar") feels the most ready to be retired (except insofar as it probably never will, because it's a nice shortcut).
The other ones at least are based on the same format string syntax.
"foo {}".format("bar") would be an obvious "just use f-string" case, except when the formatting happens far off. But in that case you could "just" use t-strings? Except in cases where you're (for example) reading a format string from a file. Remember, t- and f- strings are syntactic elements, so dynamism prevents usage of it!
So you have the following use cases:
- printf-style formatting: some C-style string formatting is needed
- .format: You can't use an f- string because of non-locality in data to format, and you can't use a t- string due to dynamism in
- f-string: you have the template and the data in the same spot lexicographically, and you just want string concatenation (very common!)
- t-string: you have the template and the data in the same spot lexicogrpahically, but want to use special logic to actually build up your resulting value (which might not even be a string!)
The last two additions being syntax makes it hard to use them to cover all use cases of the first two.
But in a specific use case? It's very likely that there is an exact best answer amongst these 4.
> printf-style formatting ("foo %s" % "bar") feels the most ready to be retired (except insofar as it probably never will, because it's a nice shortcut).
It’s also the only one which is anything near safe for being user provided.
`str.format` allows the format string to navigate through indexes, entries, and attributes. If the result of the formatting is echoed back and any non-trivial object it passed in, it allows for all sorts of introspection.
printf-style... does not support any of that. It can only format the objects passed in.
Very good point. While I think we could do away with the syntactic shorthand, definitely would want to keep some function/method around with the capabilities.
And if it's being used, and isn't considered problematic, then it should remain. I've found use for all the current ones:
(1) for text that naturally has curlies,
(2) for templating
(3) for immediate interpolation, and improved at-site readability
I see (4) being about the flexibility of (2) and readability of (3). Maybe it'll eventually grow to dominate one or both, but it's also fine if it doesn't. I don't see (1) going away at all since the curly collision still exists in (4).
Putting down my marker on the opposite. Once you're targeting a version of python that has t-strings, decent linters/libraries have an excuse to put almost all uses of f-strings in the ground.
If the usage of a feature is significantly close enough to 0 because there is a well used alternative, what need is there for backward compatibility? If anything, it can be pushed to a third party package on PyPI.
My memory is that ES6's template strings preceded f-strings. If that is correct, do you happen to know why python was saddled with f-strings, which seem like an obviously inferior design, in the first place? We are now at five largely redundant string interpolation systems (%, .format, string.Template, f-string, t-string).
PEP 501 when originally written (not by me) was intended to be the competing standard against f-strings, and to have been more inline with ES6's template strings. There was debate between the more simple f-string PEP (PEP 498) and PEP 501. Ultimately, it was decided to go with f-strings as a less confusing, more approachable version (and also easier to implement) and to "defer" PEP 501 to "see what happens". Since then, the python internal have also changed, allowing t-strings to be even easier to implement (See PEP 701). We have seen what happens, and now its introduced. f-strings and t-strings are not competing systems. They are different. Similar to ES6 templates and namedTaggedTemplates, they are used for different things while API feels similar intentionally.
f-strings are not inferior to t-strings, they are better for most use cases of string templating where what you really want, is just a string.
> they are better for most use cases of string templating where what you really want, is just a string.
I think use cases where you want to unconditionally bash a string together are rare. I'd bet that in > 80% of cases the "just a string" really is just a terrible representation for what really is either some tree (html, sql, python, ...) structure or at least requires lazy processing (logging, where you only want to pay for the expensive string formatting and generation if you run at the log level or higher that the relevant logging line is meant to operate).
I'm not familiar with ES6 template strings, but why are they better than f-strings? F-strings just work, and work well, in my experience so I'm wondering what I'm missing out on. Especially since the language I use the most is c++... So I guess I don't expect much out of string manipulation lol.
The problem with f-strings is that they make an extremely limited use case convenient (bashing unstructured text) and thus people people invariably use them for the less limited use case for which no such covenient mechanism exists. Constructing ASTs (including html and SQL). Or logging (where you want to avoid unconditionally computing some expensive string represenation).
I do this myself. I basically always use the subtl wrong log.warning(f"Unexpected {response=} encountered") and not the correct, and depending on the loglevel cheaper log.warning("Unexpected respone=%s encountered", repsonse). The extra visual noise is typically not worth the extra correctness and performance (I'd obviously not do this in some publically exposed service receiving untrusted inputs).
I'd argue these use cases are in fact more prevalent then the bashing unstructured text use case.
Encouraging people to write injection vulnerabilities or performance and correcness bugs isn't great language design.
ES2015 template strings from the beginning supported "tagged template literals" where the tag is a function that gets the template itself and the objects passed to the "holes" in the template as separate arguments. From there that function can do things like turn the holes themselves into something like SQL Parameter syntax and wrap the things that go in those holes in properly escaped SQL Parameters.
`some template {someVar}` was f-strings and someFunction`some template {someVar}` was more like what these t-strings provide to Python. t-strings return an object (called Template) with the template and the things that go into the "holes", versus tagged templates are a function calling pattern, but t-strings are still basically the other, richer half of ES2015+ template strings.
Thank you for your work on this topic and for answering questions here. I have a question: is there a way to avoid the security issues with string formatting described here? It seems like all (most?) string formatting options suffer from the same issue.
Thank you! Curious what options for deferred evalution were considered and rejected? IMHO, the main benefit of deferred evaluation isn't in the saving of a bit of code to define a deferred evaluation class, but in standardazing the API so that anyone can read the code without having to learn what it means in each project.
Also: were prompt templates for LLM prompt chaining a use case that influenced the design in any way (examples being LangChain and dozens of other libraries with similar functionlity)?
One solution that existed for a while was using the `!` operator for deferred. `t!'my defered {str}'`
The main reason for non having deferred evaluation was that it over-complicated the feature quite a bit and introduces a rune. Deferred evaluation also has the potential to dramatically increase complexity for beginners in the language, as it can be confusing to follow if you dont know what is going on. Which means "deferred by default" wasnt going to be accepted.
As for LLM's, it was not the main consideration, as the PEP process here started before LLM's were popular.
The expression (x) is eagerly evaluated in both cases, cuz that's how Python works. You can defer the format call but Python fundamentally doesn't have an equivalent of lazy/compile time flag argument evaluation and this doesn't change that.
For a logger t-strings are mostly just a more pleasant and less bug-prone syntax for #2
Not quite; the interpolations are not eagerly stringified which is the potentially expensive part. In this sense it's kind of a middle ground between the two approaches.
My hope is to write some new documentation as 3.14 nears release that explains the (growing) constellation of string formatting mechanisms in Python and describes when they might each be useful. They overlap to some degree, but each has a unique twist that makes them useful in different situations. PEP 292 is going nowhere and is used, for instance, in really powerful libraries like `flufl.i18n`
Does Python really need yet another type of string literal? I feel like while templating is a good addition to the standard library, it's not something that needs syntactic support. t"blah blah" is just an alias for Template("blah blah", context), isn't it?
yes, it does actually need syntax support. In order for it to work, you need to preserve which parts of the string are static (hard coded) and which parts are dynamic (likely user input). Which you can only do at a syntax level. You could potentially do it by hand, using placeholders, like with `%`, but as we now live in f-string world, we have something better. The syntax highlighting and ergonomics of f-strings are so good, devs prefer it in most cases. The idea is to make the most ergonomic thing, also the safest thing. By decreasing ergonomics, you reduce the adoption of safer symantics.
That's why I specified the context argument. Something like Template("{name} {address}", dict(name = ..., address = ...)) would be exactly equivalent to t"{name} {address}" assuming those variables are fetched from the local scope.
Yes, which is essentially how SQLalchemy works today. You can still put strings in the context though, so for more complex things, it's turtles all the way down. Also, as f-srrings are more ergonomic, people now reach for them, even when they shouldn't
So now you are repeating the name of each interpolated value three times (once in the template string, once for the key in the dict, once for the actual value).
Yes, you can do that, but a t-string is much more ergonomic, and IMO, more readable .
Yes, of course. Any function with syntactic support will be more "ergonomic" than one without. But t-strings are for a relatively niche use case and can't even fully replace f-strings since they aren't actually strings. Even for input sanitizing they seem insufficient since you cant use them to create pre-compiled statements/templates.
No, you don't need pre-compilation, assuming by pre-compilation you mean compiling a template that you pass values to later[1]
t-strings allow a library to perform transformations on the values, such as escaping them, or passing them as separate values to a parameterized query. Escaping html and parameterizing sql queries were the first two example use cases given in the PEP.
And I disagree that such use cases are niche. In my experience, needing to sanitize user input is an extremely common thing to need to do, and having the language and library make it as easy as possible to do it correctly is a very good thing.
[1]: I do wish they hadn't called these Templates, because it's not really a template so much as an intermediate representation of an interpolated value.
Yes, web is a niche. Outside of web there is simply no possibility of turning untrusted input into executable code so sanitation isn't needed. In web development you already have two dozen templating libraries that offer much more comprehensive safe and fast text-generation solutions than what t-strings do.
Pre-compilation means that you first compile the template, then you supply the template with values when you render it multiple times. This is not possible with t-strings since the values are bound when the t-string is created.
Most excellent! I love f-strings and replaced all the various other string interpolation instances in my code with them, but they have the significant issue that you can't defer evaluating them. For instance, you can write:
Until this, there wasn't a way to use f-strings formatting without interpolating the results at that moment:
>>> template = f'Hello, {name}'
Traceback (most recent call last):
File "<python-input-5>", line 1, in <module>
template = f'Hello, {name}'
^^^^
NameError: name 'name' is not defined
It was annoying being able to use f-strings almost everywhere, but str.format in enough odd corners that you have to put up with it.
Delayed execution is basically equivalent to a function call, which is already a thing. It also has basically the same API as point of use and requires maybe 1 extra line.
I'm not sure if t-strings help here? unless I misread the PEP, it seems like they still eagerly evaluate the interpolations.
There is an observation that you can use `lambda` inside to delay evaluation of an interpolation, but I think this lambda captures any variables it uses from the context.
> There is an observation that you can use `lambda` inside to delay evaluation of an interpolation, but I think this lambda captures any variables it uses from the context.
And that actually makes "Template Strings" a misnomer in my mind. I mean, deferred (and repeated) evaluation of a template is the thing that makes template a template.
Kinda messy PEP, IMO, I'm less excited by it than I'd like to be. The goal is clear, but the whole design feels backwards.
There was a very very long discussion on this point alone, and there are a lot of weird edge cases, and led to weird syntax things. The high level idea was to defer lazy eval to a later PEP if its still needed enough.
There are a lot of existing workarounds in the discussions if you are interested enough in using it, such as using lambdas and t-strings together.
Yeah, to be honest, every time this comes to mind I think “wow, this would be really neat!”, then realize just using .format() explicitly is way easier to read.
I’m not convinced that a language level feature is worth it for this. You could achieve the same thing with a function returning an f-string no? And if you want injection safety, just use a tag type and a sanitisation function that takes a string and returns the type. Then the function returning the f-string could take the Sanitised string as an argument to prevent calling it with unsanitised input.
I guess it’s more concise, but differentiating between eager and delayed execution with a single character makes the language less readable for people who are not as familiar with Python (especially latest update syntax etc).
EDIT: to flesh out with an example:
class Sanitised(str):
# init function that sanitises or just use as a tag type that has an external sanitisation function.
def sqltemplate(name: Sanitised) -> str:
return f”select * from {name}”
> You could achieve the same thing with a function returning an f-string no
no.
> just use a tag type and a sanitisation function that takes a string and returns the type
Okay, so you have a `sqlstring(somestring)` function, and the dev has to call it. But... what if they pass in an f-string?
`sqlstring(f'select from mytable where col = {value}')`
You havent actually prevented/enforced anything. With template strings, its turtles all the way down. You can enforce they pass in a template and you can safely escape anything that is a variable because its impossible to have a variable type (possible injection) in the template literal.
This example still works, the entire f-string is sanitised (including whatever the value of name was). Assuming sqlstring is the sanitisation function.
The “template” would be a separate function that returns an f-string bound from function arguments.
Yes. Only if your dev remembers to use sanatized all the time. This is how most SQL works today. You could also forget and accidentally write a f-string, or because you dont know. But with t-strings you can actually prevent unsanatized inputs. With your example, you need to intentionally sanitize still.
You cant throw an error on unsanitized because the language has no way to know if its sanitized or not. Either way, its just a string. "returning an f-string" is equivalent to returning a normal string at runtime.
Well you enforce this with types. That’s how every other language does it. By specifying that the type of the function has to be a sanitised string, it will reject unsanitised string with the type checker.
> it has no way of knowing if it’s sanitised or not
It does. You define the SanitisedString class. Constructing one sanitises the string. Then when you specify that as the argument, it forces the user to sanitise the string.
If you want to do it without types, you can check with `isinstance` at runtime, but that is not as safe.
Your example is a bit too simple. What I mean by that is, you have hardcoded your function to inject a specific part of your string. But t-strings allow you to write the full query `t'select * from table where name = {name}'` directly, without have to use a function. This matters because the SQL connection library itself can enforce templates.
SQL libraries can NOT enforce "sanitized types" because then you couldnt write raw sql without problems. They have to know the difference between "this is hard coded" and "this is a dynamic user variable". And the libraries can't know that, without t-strings.
Does it make it easier? The “escape” for both is to just use unsafe version of the Template -> string function or explicitly mark an unsafe string as sanitised. Both seem similar in (un)safety
There is no such function; Template.__str__() returns Template.__repr__() which is very unlikely to be useful. You pretty much have to process your Template instance in some way before converting to a string.
Right, but it is possible to write a template -> string function that doesn’t sanitise and use it (or more realistically use the wrong one). Just as it’s possible to unsafely cast an unsafe string to a sanitised one and use it (rather than use a sanitise function that returns the wrapper type).
It's worse than function returning an f-string. Template type is very flat, you won't know which arguments are left unbound.
modules, classes, protocols, functions returning functions, all options in Python, each work well for reuse, no need to use more than 2 at once, yet the world swims upstream.
I’m not sure you understood my example. The f-string is within a function. The function argument only accepts sanitised input type.
If you create a subclass of str which has an init function that sanitises, then you can’t create a Sanitised type by casting right?
And even if you could, there is also nothing stopping you from using a different function to “html” that just returns the string without sanitising. They are on the same relative level of safety.
Oh, I'm pretty sure I didn't understand your example and am probably missing something obvious. That's why I'm here asking dumb questions!
I think I'm following more, and I see how you can accomplish this by encapsulating the rendering, but I'm still not seeing how this is possible with user facing f-strings. Think you can write up a quick example?
template1 is a function that takes in a parameter evil (with a SanitisedString type that wraps a regular str) and returns the fully expanded str. It is implemented by just returning an f-string equivalent to the t-string in your example. Same with template2.
Using the SanitisedString type forces the user to explicitly call a sanitiser function that returns a SanitisedString and prevents them from passing in an unsanitised str.
You're just handing off responsibility for sanitization to the user instead of the library author.
With t-strings the rendering function is responsible for sanitization, and users can pass unrendered templates to it.
With f-strings there's no concept of an unrendered template, it just immediately becomes a string. Whoever is creating the template therefore has to be careful what they put in it.
I'm excited about this. I really like how JavaScript's tagged template literals https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe... can help handle things like automatic HTML escaping or SQL parameterization, it looks like these will bring the same capability to Python.
Yes! PEP 750 landed exactly there: as a pythonic parallel to JavaScript's tagged template strings. I'm hopeful that the tooling ecosystem will catch up soon so we see syntax coloring, formatting of specific t-string content types, etc. in the future.
I just wish it didn't get Pythonified in the process, e.g. needing to be a function call because backtick is hard to type on some keyboards, nearly having a completely separate concept of evaluating the arguments, etc. x`` vs x(t'') is a 2x blowup in terms of line-noise at worst.
> Backticks are traditionally banned from use in future language features, due to the small symbol. No reader should need to distinguish ` from ' at a glance. It’s entirely possible that the prevailing opinion on this has changed, but it’s certainly going to be easier to stick to the letter prefixes and regular quotes.
and earlier in the thread the difficulty in typing it for some is cited as another reason
Maintainer of lit-html here, which uses tagged template literals in JavaScript extensively.
This looks really great! It's almost exactly like JavaScript tagged template literals, just with a fixed tag function of:
(strings, ...values) => {strings, values};
It's pretty interesting how what would be the tag function in JavaScript, and the arguments to it, are separated by the Template class. At first it seems like this will add noise since it takes more characters to write, but it can make nested templates more compact.
Take this type of nested template structure in JS:
Python's unfortunate lambda syntax aside, not needing html() around nested template could be nice (assuming an html() function would interpret plain Templates as HTML).
In JavaScript reliable syntax highlighting and type-checking are keyed off the fact that a template can only ever have a single tag, so a static analyzer can know what the nested language is. In Python you could separate the template creation from the processing possibly introduce some ambiguities, but hopefully that's rare in practice.
I'm personally would be interested to see if a special html() processing instruction could both emit server-rendered HTML and say, lit-html JavaScript templates that could be used to update the DOM client-side with new data. That could lead to some very transparent fine-grained single page updates, from what looks like traditional server-only code.
Perhaps more idiomatically: html(t"<ul>{(t"<li>{i}</li>" for i in items)}</ul>")
> syntax highlighting and type-checking are keyed off the fact that a template can only ever have a single tag
Yes, this is a key difference and something we agonized a bit over as the PEP came together. In the (very) long term, I'm hopeful that we see type annotations used to indicate the expected string content type. In the nearer term, I think a certain amount of "clever kludginess" will be necessary in tools like (say) black if they wish to provide specialized formatting for common types.
> a special html() processing instruction could both emit server-rendered HTML and say, lit-html JavaScript templates that could be used to update the DOM client-side with new data
I'd love to see this and it's exactly the sort of thing I'm hoping emerges from PEP 750 over time. Please do reach out if you'd like to talk it over!
Python already has built-in data structure literals that allow you to express lispy DSLs:
html(['ul', {'class': 'foo'}, *(['li', item] for item in items)])
I guess template strings do make it more concise. Kind of like Racket's "#lang at-exp racket".
The benefit of lisp-like representation is you have the entire structure of the data, not just a sequence of already-serialized and not-yet-serialized pieces.
I'd like to add, after the first publication for discussion, we got some wonderful involvement from Andrea Giammarchi who brought his deep JS libraries and tools experience into the PEP. In fact, he's deeply involved in the next steps, with some forthcoming demos and libraries that will make a real difference. Exciting times.
So we are well on our way to turning python to PHP.
Edit: Sorry I was snarky, its late here.
I already didn't like f-strings and t-strings just add complexity to the language to fix a problem introduced by f-strings.
We really don't need more syntax for string interpolation, in my opinion string.format is the optimal. I could even live with % just because the syntax has been around for so long.
I'd rather the language team focus on more substantive stuff.
What's the issue with f-strings? I'm wondering because I thought they basically had no downside versus using the older alternatives. I use them so often that they are very substantive to me. If anything, this is exactly what python should be focusing on, there really isn't a lot more that they can do considering the design, expectations, and usage of python.
On a more philosophical level php is this feature. At least as it was used originally and how it's mostly used today. PHP was and is embedded in html code. If you have a look at a wordpress file you are going to see something like this:
string.format and string substitution are bloat and annoying to use, while f-strings makes it very easy to improve readability. So in the end, they remove big complexity in usage, by adding very little and straightforward complexity in syntax.
string.Template does fundamentally the same sort of thing as str.format, or str.__mod__ (i.e. the % operator on strings). It was in a sense a prototype for str.format, introduced way back in Python 2.4 (https://peps.python.org/pep-0292/). It does create an instance of a separate type, but it neither pre-parses the string (the substitution is implemented using regexes that reprocess the original string each time) nor eagerly evaluates the values to substitute. Further, in order to change how the formatting is done, you'd have to subclass (with the new system, you just write a function that accepts the Template as a parameter).
I like this but am still not a fan of the constant adding of things to the language. It’s starting to feel like a language designed by committee, which it basically is.
It's a feature that removes the need for hundreds of specialized template classes in various frameworks and is a generalization of an existing feature in a way that doesn't add much in the way of grammar. The net complexity reduction is a win here.
Putting aside template strings themselves for the moment, I'm stunned by some of the code in this PEP. It's so verbose! For example, "Implementing f-strings with t-strings":
def f(template: Template) -> str:
parts = []
for item in template:
match item:
case str() as s:
parts.append(s)
case Interpolation(value, _, conversion, format_spec):
value = convert(value, conversion)
value = format(value, format_spec)
parts.append(value)
return "".join(parts)
Is this what idiomatic Python has become? 11 lines to express a loop, a conditional and a couple of function calls? I use Python because I want to write executable pseudocode, not excessive superfluousness.
We wanted at least a couple examples that showed use of Python's newer pattern matching capabilities with the new Interpolation type. From this outsider's perspective, I'd say that developer instincts and aesthetic preferences are decidedly mixed here -- even amongst the core team! You can certainly write this as:
def f(template: Template) -> str:
return "".join(
item if isinstance(item, str) else
format(convert(item.value, item.conversion), item.format_spec)
for item in template
)
Or, y'know, several other ways that might feel more idiomatic depending on where you're coming from.
Your example is interesting - `join`, a generator expression and a ternary, and Python requires us to write all of them inside-out. It's a shame we can't write that in a more natural order, something like:
def f(template):
return (for item in template:
isinstance(item, str) then item else
format(convert(item.value, item.conversion), item.format_spec)
).join('')
I think it's very deliberate that these sorts of expressions are inside-out from the corresponding statements. It mixes and matches modes of thinking in my mind - the same kind of mistake as command-query separation violations. When I read a list comprehension, I can read it left to right: "A list (`[`) of each `item` that I get, if I look `for` each `sublist` that is `in l`, then `for` each `item` which is `in` that `sublist`." And the entire description is - well - descriptive, not imperative. With the other order, I have to think in terms of explicitly following the steps, and then mentally translating the last `item` into an imperative "... and put it into the result list".
Nah, idiomatic Python always used to prefer comprehensions over explicit loops. This is just the `match` statement making code 3x longer than it needs to be.
You can express the loop as a list comprehension, and I would too.
As for the logic, I would still use pattern matching for branching and destructuring, but I’d put it in a helper. More lines is not a negative in my book, though I admit the thing with convert and format is weird.
> I would still use pattern matching for branching and destructing, but I’d put it in a helper
Yeah, using a helper function makes things much clearer. To be honest though, I'm not a huge fan of using either `isinstance` (which is generally a sign of a bad design) nor `match/case` (which is essentially a "modern" way to write `isinstance`).
I can't help but think that a better design could avoid the need for either of those (e.g. via polymorphism).
What I would use is an ADT and I make a wish for that every day in every language that’s not Haskell :D
Haskell also has operator overloading on steroids so you could use the (|>) operator from Flow and write transformations the same as you would shell pipes. I’d love to whip up an example but it’s difficult on this tiny screen. Will try to remember when I’m on my computer.
Before someone chimes in with ML propaganda, I warn you that I’m going to exercise my rights under the Castle Doctorine the moment you say “of”.
You can access .strings and .interpolations directly and avoid type checks. There is always one more string than interpolation, of which there may be zero.
def _f_part(item) -> str:
match item:
case str() as s:
return s
case Interpolation(value, _, conversion, format_spec):
return format(convert(value, conversion), format_spec)
def f(template: Template) -> str:
return ''.join(map(_f_part, template))
The `match` part could still be written using Python's if-expression syntax, too. But this way avoids having very long lines like in the Ruby example, and also destructures `item` to avoid repeatedly writing `item.`.
I very frequently use this helper-function (or sometimes a generator) idiom in order to avoid building a temporary list to `.join` (or subject to other processing). It separates per-item processing from the overall algorithm, which suits my interpretation of the "functions should do one thing" maxim.
The Python version is straightforward to read and understand to a programmer of any language. The Ruby version is an idiosyncratic symbol soup.
If I were tasked to modify the Python version to say, handle the case where `item` is an int, it would be immediately obvious to me that all I need to do is modify the `match` statement with `case int() as i:`, I don't even need to know Python to figure that out. On the other hand, modifying the Ruby version seems to require intimate knowledge of its syntax.
I think for someone with a basic knowledge of both languages, the Ruby version is more understandable than the Python. It's a combination of basic Ruby features, whereas Python's `match` statement is much more obscure - it isn't really Python at all, it's "a DSL contrived to look like Python [...] but with very different semantics" [0].
I don't particularly love the Ruby code either, though - I think the ideal implementation would be something like:
What do you mean? Python has always been that way. "Explicit is better than implicit. [..] Readability counts." from the Zen of python.
> By contrast, here's the equivalent Ruby:
Which is awful to read. And of course you could write it similar short in python. But it is not the purpose of a documentation to write short, cryptic code.
Almost all Python programmers should be familiar with list comprehensions - this should be easy to understand:
parts = [... if isinstance(item, Interpolation) else ... for item in template]
Instead the example uses an explicit loop, coupled with the quirks of the `match` statement. This is much less readable IMO:
parts = []
for item in template:
match item:
case str() as s:
parts.append(...)
case Interpolation(value, _, conversion, format_spec):
parts.append(...)
> [Ruby] is awful to read
I think for someone with a basic knowledge of Ruby, it's more understandable than the Python. It's a combination of basic Ruby features, nothing advanced.
I don't particularly love Ruby's syntax either, though - I think the ideal implementation would be something like:
Could someone explain more why this should be a language feature?
My understanding of template strings is they are like f-strings but don't do the interpolation bit. The name binding is there but the values are not formatted into the string yet. So effectively this provides a "hook" into the stringification of the interpolated values, right?
If so, this seems like a very narrow feature to bake into the language... Personally, I haven't had issues with introducing some abstraction like functions or custom types to do custom interpolation.
The best use case I know of for these kinds of things is as a way to prevent sql injection. SQL injection is a really annoying attack because the "obvious" way to insert dynamic data into your queries is exactly the wrong way. With a template string you can present a nice API for your sql library where you just pass it "a string" but it can decompose that string into query and arguments for proper parameterization itself without the caller having to think about it.
You can see some motivations for it further down the document. Basically it allows libraries to access the ease of use of f-strings for more than just formatting a string according to what the language allows. Structured logging is one area where I would like to use this.
It's useful for doing things like writing parameterized SQL queries that avoid injection attacks. Or anything that quacks like that proverbial duck, where you are going to emit some plaintext and need to validate it as it is filled.
Because python dominates AI, and python is dominating because of AI. And prompting really, really benefits from f-strings and templated strings. LLMs as a whole means the rise of unstructured data, and flexible string manipulation is really important for handling that.
I like the implementation, but it looks like nobody has pointed out that Python already has a built-in string template system [1]. Granted, it's not quite as simple to use, but I have been using it for a while.
In the past when I needed this I just made a function that processed the f string, often enough a simple lambda function would do. This looks like additional complexity for not a lot of gain.
You cant process an f-string the same way you can process a t-string. An f-string does not preserve which parts of it are static and dynamic, which you need to know to properly escape user input.
I've been toying with the idea of having arbitrary string types, like
sql"SELECT FROM ..."
or
re"\d\d[abc]"
that the development environment could highlight properly, that would ... I don't know. In the end t and f string don't do anything that a t() and f() function couldn't have done, except they are nice. So it would be nice to have more.
IIRC, previous versions of the PEP wanted things to work this way. You might also be interested in the "syntactic macros" PEP (https://peps.python.org/pep-0638/).
> If a single interpolation is expensive to evaluate, it can be explicitly wrapped in a lambda in the template string literal
I’m having trouble understanding this - Can someone please help out with an example use case for this? It seems like before with an f string we had instant evaluation, now with a t string we control the evaluation, why would we further delay evaluation - Is it just to utilise running a function on a string first (i.e. save a foo = process(bar) line?)
> It seems like before with an f string we had instant evaluation, now with a t string we control the evaluation
You don't completely control the evaluation.
From the PEP:
> Template strings are evaluated eagerly from left to right, just like f-strings. This means that interpolations are evaluated immediately when the template string is processed, not deferred or wrapped in lambdas.
If one of the things you are interpolating is, as a silly example, an invocation of a slow recursive fibonacci function, the template string expression itself (resulting in a Template object) will take a long while to evaluate.
> the template string expression itself (resulting in a Template object) will take a long while to evaluate
Are you saying that calling:
template = t”{fib_slow()}”
Will immediately run the function, as opposed to when the __str__ is called (or is it because of __repr__?) - Apparent I might just have to sit down with the code and grok it that way, but thanks for helping me understand!
Yes. Just like with f-strings, the call to `fib_slow()` is extracted at compile-time, and occurs eagerly at run-time.
What t-strings offer over f-strings is the ability to control how the final string is put together from the calculated results. And the t-string doesn't have `__str__` - you have to explicitly pass it to a named formatting function to get a string. (There is `__repr__, but that's for debugging, of course.) So you can potentially reuse the same Template instance (created from the t-string) multiple times in different contexts to format the same information different ways.
This is super exciting but I wish they added a function to render the template to a string rather then having everyone write their own version of basic rendering.
One place we landed with PEP 750 is that Template instances have no natural string rendering.
That is, you must process a Template in some way to get a useful string out the other side. This is why Template.__str__() is spec'd to be the same as Template.__repr__().
I totally agree with `Template.__str__()` == `Template.__repr__()`, but I would strongly be for adding a `Template.format()`. I expect that over time I'll be passed template strings where I expected normal strings and adding that example `f(template: Template)` everywhere to my code bases is just a lot of repeated code for no real value.
That said, I think this is a great bit of work and I look forward to getting to use it! Thank you!
> and adding that example `f(template: Template)` everywhere to my code bases is just a lot of repeated code for no real value.
I'd agree that it should be somewhere within the library, even if it's just a separate top-level function in the described `string.templatelib`. If it isn't, I'm sure someone will make a PyPI package for it right away.
PEP 501 allowed `.format()` to do this, but discussions on PEP 750 felt that format shouldn't be needed, because when you need a basic format, you should probably just use f-strings. But maybe `__format__` can be added in in a future version of python if this is really wanted.
Can't think of a good reason now on why I would need this rather than just a simple f-string.
Any unsafe string input should normally be sanitized before being added in a template/concatenation, leaving the sanitization in the end doesn't seem like the best approach, but ok.
Yeah, I generally like most of the stuff that goes into Python, but I have to admit I can't really see the usefulness in this (or, at least, enough usefulness to warrant making it standard).
One of the PEP's developers, Lysandros, presented this in our local meetup, so I am passingly familiar with it, but still, I might be missing something.
I guess the crux of it is that I don't understand why it's `t"some string"` instead of `Template("some string")`. What do we gain by the shorthand?
Ah OK, I see what you mean, so they're basically an f-string that hasn't been "baked in" yet, and you can still access all its parameters. That's pretty cool.
This could be a game-changer for applications that involve lots of HTML, SQL, or other languages that tend to be embedded within Python programs. Frankly, HTML templating is the worst part of Python webdev DX for me.
Excited to see what libraries and tooling comes out of this.
Agreed, although this doesn't work as well for HTML or SQL that doesn't fit on one line. Since having everything in a string literal is a little messy, and doesn't have the benefit of autoindent, syntax coloring, etc.
This is one place where s-expressions of Lisp make embedding these DSLs syntactically easier.
You can see how the parentheses syntax will practically scale better, to a larger and more complex mix of HTML and host language expressions. (A multi-line example using the normal text editor autoindent is on "https://docs.racket-lang.org/html-template/".)
My hope is that we'll quickly see the tooling ecosystem catch up and -- just like in JavaScript-land -- support syntax coloring and formatting specific types of content in t-strings, like HTML.
Yeah, editor support will help a lot. Though the PEP's way of doing things doesn't specify the language of the template where the literal occurs, so detection of that might have to be kludged.
(Or, in a sufficiently working program, an editor with semantic analysis access could use something like type inference in the Python side, to determine the language in a less-kludgey way.)
> doesn't specify the language of the template where the literal occurs
Yeah, something we spent a bunch of time considering. In the end, we decided it probably needed to stay out of scope for the PEP.
You're right that JavaScript has an easier time here. Most of the JS tools we looked at simply inspect the name of the tag and if it's (say) html, they attempt to color/format string content as HTML regardless of what the html() function actually does or the string's contents.
Currently, tools like black have no knowledge of types. I'm guessing some amount of kludging is to be expected on day one. But my hope is over the long term, we'll see a story emerge for how annotations can indicate the expected content type.
lot_of_html isn't a string literal with dangerous elements smooshed into the trusted scaffolding like an f-string would do.
It's a template instance that still needs to be safely processed into a renderable string, e.g. by escaping whatever `evil` evaluates to and even validating the final HTML syntax.
I can easily end up unsafely processed. It's a footgun.
And why would you be validating HTML on the fly, when it's coming from your program, not as an input into it. Even if you can do it at program startup once for each template, it's still pointless overhead.
The whole thing is wrongheaded; exactly the kind of stove-pipe people end up inventing when they don't have metaprogramming.
No, that isn't how it works. The unprocessed version is not a `str` instance and doesn't implement `__str__`:
> This is because Template instances are intended to be used by template processing code, which may return a string or any other type. There is no canonical way to convert a Template to a string.
If you tried to use the Template directly as if it were a string, you'd get either a TypeError or completely malformed HTML (the `repr` of the Template instance, which would look very different).
>And why would you be validating HTML on the fly
You wouldn't be; you'd be escaping user-generated content that tries to break a page by including HTML markup.
Well, no; the entire point is that there are multiple ways to interpret the Template in context. One possible application is for localization (supposing that the interpolations are objects that can give a localized string on demand, or e.g. integers to be formatted in a locale-aware manner). It should probably be possible to do that with .format anyway, but just as an example. Perhaps someone out there wants to store fragments of SQL within a JSON document, too. The point is to maintain that flexibility, by separating the parsing and formatting steps.
On the contrary, I implemented a substantial facsimile of it in a Lisp dialect yesterday; see my comment history.
1> (load "template")
nil
2> (let ((field "name") (table "customers"))
(te `SELECT @field FROM @table`))
#S(template merge #<interpreted fun: lambda (#:self-0073)> strings #("SELECT " " FROM ")
vals #("name" "customers"))
3> *2.(merge)
"SELECT name FROM customers"
4> [*2.vals 0]
"name"
5> (set [*2.vals 0] "id")
"id"
6> *2.(merge)
"SELECT id FROM customers"
7> (upd *2.vals (map upcase-str))
#("ID" "CUSTOMERS")
8> *2.(merge)
"SELECT ID FROM CUSTOMERS"
Unlike template strings, it was done in the language. There already are quasi-strings in the form of `...` syntax. We can quote that syntax (e.g. implicitly as a macro argument) and then pull apart its pieces to construct an object. It should work even in a years-out-of-date installation of the language. No new tooling is required; no changes to syntax highlighting in the editor, nothing.
It's a parlor trick that doesn't have any uses. The structured log messages use case is the most promising, because it has a consumer which actually wants the interpolated pieces that it would otherwise have to hackily parse out.
I predict that Python will eventually get dedicated HTML syntax: perhaps something that uses indentation to indicate element nesting. Let's "vibe pseudocode" a sketch:
html:
div (class='main' id='1'):
p:
"paragraph text"
> Unlike template strings, it was done in the language.
This is a disingenuous way to compare a LISP with a language that values syntactic convention and strives to be readable and maintainable by more than one person.
> I predict that Python will eventually get dedicated HTML syntax:
How do you assign this result to a variable? How is this any better than
It's a true statement; I made a usable construct that causes template strings to be turned into objects, rather than to perform the string substitution. I did it without changing any syntax, or have to ship a new version of the language.
Python has to ship a new version for this; there is no way for existing installations to use the code.
I don't have to change anything in my editor setup.
So who is it that values syntactic convention?
t"abc" is a syntax errror in vast numbers of existing Python installations; how can we call it convention?
> How do you assign this result to a variable?
That's to be worked out. The "html:" fantasy construct could have a way to do that. When you think of assigning to a variable, the first thing that comes to mind is "a = b" syntax. But, look, the define construct in Python also assigns to a variable:
define fun(arg):
...
The imagined html construct could have arguments like to assign an object to a variable, or just spin out textual HTML into a specified stream. I don't want to get into it, but there are obvious ways it could be done such that it hits all sorts of requirements.
> How is it any better
It renders all of the following problems unrepresentable: mismatched angle brackets, mismatched or unclosed tags, bad attribute syntax, injection.
Jinja et al. are fine. But they still forces me to choose Python OR HTML. I have to jump between files and LSPs. t-strings pave the way for true Python+HTML seamless DX.
So does this mean that any place where code exists that looks for `type(someinstance) == str` will break because the type will be `Template` even though `someinstance` could still be used in the following code?
Good addition. Java has done a few previews of this and is still trying to figure out the best way. Maybe the feedback on the Python version will help.
No. Not really intended to be a replacement for jinja. But it also depends on how you use jinja. If you use only very basic functionality of jinja, maybe.
For a quick glance, I didn't see the corner case where the old-style `%` interpolation (may) shine over PEP750: building up a JSON document with a dictionary:
As a python meh-er, this is actually good design. Everyone is jumping on C that it has no strings, but then other languages throw raw strings at you with some interpolation and call it a day. Also it's 2025 and people will still comment "do we need such a bloated string mechanism" and then watch new devs produce bulks of injectionable strings.
I'm quite disappointed with the pace of development of the Python language. Five years of support for a particular version seems far too short.
I mostly use Python in scientific contexts, and hitting end-of-life after five years means that for a lot project, code needs to transition language versions in the middle of a project. Not to mention the damage to reproducibility. Once something is marked "end of life" it means that future OS versions are going to have a really good reason to say "this code shouldn't even be able to run on our new OS."
Template strings seem OK, but I would give up all new language features in a heartbeat to get a bit of long term support.
> hitting end-of-life after five years means that for a lot project, code needs to transition language versions in the middle of a project.
Your Python interpreter will not spontaneously combust due to being end-of-life. It just eventually won't be able to run new versions of tools; but your existing tool versions should also work fine for a very long time. All you're missing out on is bugfixes, which third parties (such as a Linux distro) are often willing to provide.
When a language like Python doesn't innovate at this rate, eventually people will get annoyed about how primitive it ends up feeling compared to languages that have been around for less than half as long. The devs' time and resources are limited and they've clearly advertised their time preference and committed to a reliable schedule - this is an explicit attempt to accommodate users like you better, compared to the historical attitude of releasing the next minor version "when it's done". It also means that they're locked in to supporting five versions at a time while developing a sixth. There's only so much that can reasonably be expected here.
Seriously, what you're getting here is well above the curve for open-source development.
You can use uv to pin, download, and use a specific version of a Python interpreter on a per project basis. You shouldn't use your OS provided Python interpreter for long running projects for all the reasons you mentioned, including reproducibility. If you insist on using the vendor provided interpreter, then use RHEL (or a free clone) or any other long suppported Linux distro (there are many). Five years is a very long time in technology. It is not reasonable to expect more from core maintainers IMO. Especially considering we have better options.
But it's not a long time in the OP's field of science. Unfortunately despite a strong preference for Python in the scientific community, the language's design team seem to ignore that community's needs entirely, in favour of the needs of large technology companies.
I was hopeful that in the transition from a BDFL-based governance system to a Steering Council, we would see a larger variety of experience and opinions designing the language. Instead, I don't think there has ever been a single scientist, finance worker etc on the Steering Council - it's always software developers, almost always employees of large software companies.
You can pay for long term support from private companies.
And your scientific context is a distinct minority for python now. Most new development for python is for data/AI. Considering LLMs get updated every quarter, and depreciated every year, there is no appetite for code that doesn't get updated for 5 years.
Honestly it's very weird to have data/AI be outside of the scientific context! They both used to be a subset of scientific Python.
The code will be updated over five years, but there's no need to be on continual version churn on the underlying language. And frankly I'm surprised that it's tolerated so widely in the community. Trying to run a Node project from 5 years ago is often an exercise in futility, and it will be a big shame when/if that happens to Python.
can't wait to have my linter tell me I should be using t-strings instead of f-strings. apologies for not being patient enough to read through this to find it, but I hope they can remove f-strings:
> There should be one-- and preferably only one --obvious way to do it.
I wonder what fun The Amazing David Beazley thinks PEP 750 t-strings are?
I recently asked him:
--
Hi David! I am a huge long time fan of SWIG and your numerous epic talks on Python.
I remember watching you give a kinda recent talk where you made the point that it’s a great idea to take advantage of the latest features in Python, instead of wasting your time trying to be backwards compatible.
I think you discussed how great f-strings were, which I was originally skeptical about, but you convinced me to change my mind.
I’ve googled around and can’t find that talk any more, so maybe I was confabulating, or it had a weird name, or maybe you’ve just given so many great talks I couldn’t find the needle in the haystack.
What made me want to re-watch and link my cow-orkers to your talk was the recent rolling out of PEP 701: Syntactic formalization of f-strings, which makes f-strings even better!
Oh by the way, do you have any SWIG SWAG? I’d totally proudly wear a SWIG t-shirt!
The f-strings section starts at 10:24 where he's live coding Python on a tombstone with a dead parrot. But the whole talk is well worth watching, like all his talks!
Background: TXR already Lisp has quasi-string-literals, which are template strings that do implicit interpolation when evaluated. They do not produce an object where you can inspect the values and fixed strings and do things with these before the merge.
1> (let ((user "Bob") (greeting "how are you?"))
`Hello @user, @greeting`)
"Hello Bob, how are you?"
The underlying syntax behind the `...` notation is the sys:quasi expression. We can quote the quasistring and look at the car (head symbol) and cdr (rest of the list):
OK, now with those pieces, I just right now made a macro te that gives us a template object.
4> (load "template")
nil
You invoke it with one argument as (te <quasistring>)
5> (let ((user "Bob") (greeting "how are you?"))
(te `Hello @user, @greeting`))
#S(template merge #<interpreted fun: lambda (#:self-0073)> strings #("Hello " ", ")
vals #("Bob" "how are you?"))
6> *5.vals
#("Bob" "how are you?")
7> *5.strings
#("Hello " ", ")
8> *5.(merge)
"Hello Bob, how are you?"
9> (set [*5.vals 0] "Alice")
"Alice"
10> *5.(merge)
"Hello Alice, how are you?"
You can see the object captured the values from the lexical variables, and we can rewrite them, like changing Bob to Alice. When we call the merge method on the object, it combines the template and the values.
(We cannot alter the strings in this implementation; they are for "informational purposes only").
It produces a constructor invocation (new template ...) which specifies values for the slots merge, strings and vals.
The initialization of strings is trivial: just a vector of the strings pulled from the quasistring.
The vals slot is initialized by a `(vec ...)` call whose arguments are the expressions from the quasistring. This gets evaluated in the right lexical scope where the macro is expanded. This is how we capture those values.
The most complicated part is the lambda expression that initializes merge. This takes a single argument, which is the self-object, anonymized by a gensym variable for hygiene. It binds the .vals slot of the object to another gensym lexical. Then a genyms local variable is bound for each value, referencing into consecutive elements of the value vector. E.g. #:var-0075 is bound to [#:vals-0074 0], the first value.
The body of the let is a transformed version of the original template, in which the interpolated expressions are replaced by gensyms, which reference the bindings that index into the vector.
The complete implementation in template.tl (referenced by (load "template") in command line 4) is:
That Lisp Curse document, though off the mark in general, was right the observation that social problems in languages like Python are just technical problems in Lisp (and often minor ones).
In Python you have to wait for some new PEP to be approved in order to get something that is like f-strings but gives you an object which intercepts the interpolation. Several proposals are tendered and then one is picked, etc. People waste their time producing rejected proposals, and time on all the bureucracy in general.
In Lisp land, oh we have basic template strings already, let's make template objects in 15 minutes. Nobody else has to approve it or like it. It will backport into older versions of the language easily.
P.S.
I was going to have the template object carry a hash of those values that are produced by variables; while coding this, I forgot. If we know that an interpolation is @greeting, we'd like to be access something using the greeting symbol as a key.
(I don't see any of this is as useful, so I don't plan on doing anything more to it. It has no place in Lisp, because for instance, we would not take anything resembling this approach for HTML generation, or anything else.)
The stated use case is to avoid injection attacks. However the primary reason why injection attacks work is that the easiest way to write the code makes it vulnerable to injection attacks. This remains true, and so injection attacks will continue to happen.
Templates offer to improve this by adding interpolations, which are able to do things like escaping. However the code for said interpolations is now located at some distance from the template. You therefore get code that locally looks good, even if it has security mistakes. Instead of one source of error - the developer interpolated - you now have three. The developer forgot to interpolate, the developer chose the wrong interpolation, or the interpolation itself got it wrong. We now have more sources of error, and more action at a distance. Which makes it harder to audit the code for sources of potential error.
This is something I've observed over my life. Developers don't notice the cognitive overhead of all of the abstractions that they have internalized. Therefore over time they add more. This results in code that works "by magic". And serious problems if the magic doesn't quite work in the way that developers are relying on.
Templates are yet another step towards "more magic". With predictable consequences down the road.
Template.__str__() is equivalent to Template.__repr__(), which is to say that these aren't f-strings in an important sense: you can't get a useful string out of them until you process them in some way.
The expectation is that developers will typically make use of well-established libraries that build on top of t-strings. For instance, developers might grab a package that provides an html() function that accepts Template instances and returns some Element type, which can then be safely converted into a string.
Stepping back, t-strings are a pythonic parallel to JavaScript's tagged template strings. They have many of the same advantages and drawbacks.
In my experience it was precisely when it wasn't abstracted properly that injection attacks became more prevalent, not the other way around.
In PHP, people used to just call mysql_query on a string and all the escaping was done with mysql_escape_string. According to you that nice locality of query construction and sanitization that should've improved security, but my god did it ever not do that.
It was exactly layers of abstractions, moving things far away from the programmer, with prepared statements to ORMs, that meaningfully reduced the number of SQL injection vulnerabilities.
Another example is JavaScript, how many XSS vulnerabilities never happened because of all the shadow dom frameworks? Layers of abstractions like these (JSX,etc) are a major reason we don't see many XSS vulnerabilities nowadays.
> You therefore get code that locally looks good, even if it has security mistakes. Instead of one source of error - the developer interpolated - you now have three.
Could you give examples of this?
> The developer forgot to interpolate
What would this look like? The only way to get dynamic/user input into a template is either through interpolation or concatenation.
Before:
f"..html_str..." + user_provided_str # oops! should have: html_str + sanitize(user_provided_str)
After:
t"...html_template..." + user_provided_str # oops! should have: t"...html_template...{user_provided_str}"
Does this really leave us worse off?
Unless you're referring to something like this:
Before:
html = "love > war" # oops! should have been: html = "love > war"
After:
html = "love > war" # oops! should have been: html = t"love > war"
But then the two scenarios are nearly identical.
> the developer chose the wrong interpolation
What kind of interpolation would be the "wrong interpolation"?
> or the interpolation itself got it wrong.
Isn't that analogous to sanitize(user_provided_str) having a bug?
> Instead of one source of error - the developer interpolated - you now have three. The developer forgot to interpolate, the developer chose the wrong interpolation, or the interpolation itself got it wrong.
The idea is for the interpolation to be provided by the library - just as the library is expected to provide a quoting/escaping/sanitization function today. But now the interpolation function can demand to receive an instance of the new Template type, and raise a `TypeError` if given a pre-formatted string. And that work can perhaps also be rolled into the same interface as the actual querying command. And manually creating a Template instance from a pre-formatted string is difficult and sticks out like a sore thumb (and it would be easy for linters to detect the pattern).
> This is something I've observed over my life. Developers don't notice the cognitive overhead of all of the abstractions that they have internalized. Therefore over time they add more. This results in code that works "by magic". And serious problems if the magic doesn't quite work in the way that developers are relying on.
By this logic, we couldn't have languages like Python at all.
Libraries can enforce only template strings, and properly escape the output. This is already possible in Javascript, and you can completely prevent injection attacks using it.
> The developer forgot to interpolate
not possible if you enforce only templates
> the developer chose the wrong interpolation
Not possible if the library converts from template to interpolation itself
> or the interpolation itself got it wrong
Sure, but that would be library code.
That's such a small scope of an addition I don't see the comparison. I suppose overloading arbitrary types with it can sometimes make the actions performed opaque.
It's fascinating how differently languages approach the string formatting design space.
- Java's been trying to add f/t-strings, but its designers appear to be perfectionists to a fault, unable to accept anything that doesn't solve every single problem possible to imagine: [1].
- Go developers seem to have taken no more than 5 minutes considering the problem, then thoughtlessly discarded it: [2]. A position born from pure ignorance as far as I'm concerned.
- Python, on the other hand, has consistently put forth a balanced approach of discussing each new way of formatting strings for some time, deciding on a good enough implementation and going with it.
In the end, I find it hard to disagree with Python's approach. Its devs have been able to get value from first the best variant of sprintf in .format() since 2008, f-strings since 2016, and now t-strings.
[1]: https://news.ycombinator.com/item?id=40737095
[2]: https://github.com/golang/go/issues/34174#issuecomment-14509...
> Go developers seem to have taken no more than 5 minutes considering the problem, then thoughtlessly discarded it: [2]. A position born from pure ignorance as far as I'm concerned
There are a million things in go that could be described this way.
Looking at the various conversations involving string interpolation, this characterization is extremely unkind. They've clearly spent a lot more than 5 minutes thinking about this, including writing their own mini-proposals[1].
Are they wrong about this issue? I think they are. There is a big difference in ergonomics between String interpolation and something like fmt.Sprintf, and the performance cost of fmt.Sprintf is non-trivial as well. But I can't say they didn't put any thought into this.
As we've seen multiple times with Go generics and error handling before, their slow progress on correcting serious usability issues with the language stem from the same basic reasons we see with recent Java features: they are just being quite perfectionist about it. And unlike Java, the Go team would not even release an experimental feature unless they feel quite good about it.
[1] https://github.com/golang/go/issues/57616
> There is a big difference in ergonomics between String interpolation and something like fmt.Sprintf
On the other hand, there’s a difference in localizability as well: the latter is localizable, the former isn’t. (It also worries me that I see no substantive discussion of localization in PEP 750.)
I just expect better from professional language designers. To me, the blindingly obvious follow up to the thought "We understand that people familiar with other languages would like to see string interpolation in Go." [1] is to research how said other languages have gone about implementing this and to present a brief summary of their findings. This is table stakes stuff.
Then there's "You can [get] a similar effect using fmt.Sprint, with custom functions for non-default formatting." [2]:
- Just the fact that "you can already do this" needs to be said should give the designers pause. Clearly you can't already do this if people are requesting a new feature. Indeed, this situation exactly mimics the story of Go's generics - after all, they do not let you do anything you couldn't do before, and yet they got added to Go. It's as if ergonomics matter, huh.
Another way to look at this: if fmt.Sprint is so good it should be used way more than fmt.Sprintf right? Should be easy to prove :)
- The argument crumbles under the load-bearing "similar effect". I already scratched the surface of why this is wrong in a sibling post: [3].
I suspect the reason for this shallow dismissal is the designers didn't go as far as to A/B test their proposal themselves, so their arguments are based on their gut feel instead of experience. That's the only way I can see someone would come up with the idea that fmt.Sprint and f-strings are similar enough. They actually are if all you do is imagine yourself writing the simplest case possible:
Similar enough, so long as you're willing to handwave away the need to match quotation marks and insert commas and don't spend time coding using both approaches. If you did, you'd find that writing brand new string formatting statements is much rarer than modifying existing ones. And that's where the meat of the differences is buried. Modifying f-strings is trivial, but making any changes to existing fmt.Sprint calls is painful.P.S. Proposing syntax as noisy as:
is just another sign the designers don't get it. The entire point is to reduce the amount of typing and visual noise.[1]: https://github.com/golang/go/issues/57616#issuecomment-14509...
[2]: https://github.com/golang/go/issues/34174#issuecomment-14509...
[3]: https://news.ycombinator.com/item?id=43651419
> Proposing syntax as noisy as […] is just another sign the designers don't get it.
Are you objecting to the use of `\(…)` here instead of `{…}`? Because of the extra character or because of the need to nest parentheses?
Value types anyone? I have zero doubt it is tough to add and get right, esp. to retrofit, but it has been so many years that I have learned/discarded several new languages since Java... and they STILL aren't launched yet.
Go(lang)'s rejection makes sense.
A format function that arbitrarily executes code from within a format string sounds like a complete nightmare. Log4j as an example.
The rejection's example shows how that arbitrary code within the string could instead be fixed functions outside of a string. Safer, easier for compilers and programmers; unless an 'eval' for strings is what was desired. (Offhand I've only seen eval in /scripted/ languages; go makes binaries.)
In many languages, f-strings (or f-string like constructs) are only supported for string literals, not user-supplied strings.
When compiling, those can be lowered to simple string concatenation, just like any for loop can be lowered to and represented as a while.
In case there was confusion: Python's f-string functionality in particular is specific to string literals. The f prefix doesn't create a different data type; instead, the contents of the literal are parsed at compile time and the entire thing is rewritten into equivalent string concatenation code (although IIRC it uses dedicated bytecodes, in at least some versions).
The t-string proposal involves using new data types to abstract the concatenation and formatting process, but it's still a compile-time process - and the parts between the braces still involve code that executes first - and there's still no separate type for the overall t-string literal, and no way to end up eval'ing code from user-supplied data except by explicitly requesting to do so.
There is no compile time in python
Yes, there is.
Python source code is translated into bytecode for a VM just like in Java or C#, and by default it's cached in .pyc files. It's only different in that you can ask to execute a source code file and the compilation happens automatically before the bytecode-interpretation.
`SyntaxError` is fundamentally different from other exceptions because it can occur during compilation, and only occurs at run-time if explicitly raised (or via explicit invocation of another code compilation, such as with `exec`/`eval`, or importing a module). This is also why you can't catch a `SyntaxError` caused by the invalid syntax of your own code, but only from such an explicit `raise` or a request to compile a source code string (see https://stackoverflow.com/questions/1856408 ).
Yes there is, when it compiles source code to bytecode.
My reply was to the parent post's SPECIFIC example of Golang's rejected feature request. Please go read that proposal.
It is NOT about the possibility of referencing existing / future (lazy / deferred evaluation) string literals from within the string, but about a format string that would literally evaluate arbitrary functions within a string.
The proposal doesn't say anything about executing code in user-supplied strings. It only talks about a string literal that is processed by the compiler (at which point no user-supplied string can be available).
On the other hand, the current solution offered by Go (fmt.Sprintf) is the one who supports a user-supplied format String. Admittedly, there is a limited amount of damage that could be done this well, but you can at the very least cause a program to panic.
The reason for declining this feature[1] has nothing to do with what you stated. Ian Lance Taylor simply said: "This doesn't seem to have a big advantage over calling fmt.Sprintf" and "You can a similar effect using fmt.Sprint". He conceded that there are performance advantages to string interpolation, but he doesn't believe there are any gains in usability over fmt.Sprintf/fmt.Sprint and as is usual with Go (compared to other languages), they're loathe to add new features to the compiler[2].
[1] https://github.com/golang/go/issues/34174#issuecomment-14509...
[2] https://github.com/golang/go/issues/34174#issuecomment-53013...
What's the risk of user supplied strings? Surely you know their size. What else is there to worry about?
No, the format function doesn't "arbitrarily execute code."
An f/t string is syntax not runtime.
Instead of
you write That subject is simple an normal code expression, but one that occurs after the opening quote of the literal and before the ending quote of the literal.And instead of
you write It's a way of writing string literals that if anything makes injection less likely.Thanks for this example - it makes it clear it can be a mechanism for something like sqlc/typed sql (my go-to with python too, don't like orms) without a transpilation step or arguably awkward language API wrappers to the SQL. We'll need linters to prevent accidentally using `f` instead of `t` but I guess we needed that already anyways. Great to be able to see the actual cost in the DB without having to actually find the query for something like `typeddb.SelectActiveAccount(I'd)`. Good stuff.
The PEP says these return a new type `Template`, so you should be able to both type and/or duck type for these specifically and reject non-Template inputs.
It is a different type.
You can verify that either via static typechecking, or at runtime.
Please read the context of my reply again.
The Rejected Golang proposal cited by the post I'm replying to. NOT Python's present PEP or any other string that might resolve magic variables (just not literally eval / exec functions!).
As far as I can tell from the linked proposal, it wouldn't have involved such evaluation either. It seems like it was intended to work fundamentally the same way as it currently does in Python: by analyzing the string literal ahead of time and translating into equivalent explicit formatting code, as syntactic sugar. There seem to have been many misunderstandings in the GitHub discussion.
In that case, I might have misunderstood the intent of those examples.
However the difficulty of understanding also illustrates the increased maintenance burden and language complexity.
Unless workarounds to a missing feature have a higher maintenance burden like in this case, and you can't avoid it via learning
Go's preferred way would probably be something like compute the aliased operations on the line(s) before, then reference the final values.
E.G. Adapting https://github.com/golang/go/issues/34174
The inline {variable} reference suffix format would be less confusing for situations that involve _many_ variables. Though I'm a bit more partial to this syntax with an immediately trailing %{variable} packet since my gut feeling is that special case would be cleaner in a parser.The proposal cited Swift, Kotlin, and C# which have similar syntax sugar.
The proposal was for the same.
> A format function that arbitrarily executes code from within a format string
So, a template? I certainly ain't gonna be using go for its mustache support.
No, it's exactly the opposite--f-strings are, roughly, eval (that is, unsanitary string concatenation that is presumptively an error in any nontrivial use) to t-strings which are just an alternative expression syntax, and do not even dereference their arguments.
f-strings are not eval. It's not dynamic. It's simply an expression that is ran just like every other expression.
Then there is Ruby, which just has beautiful string formatting without strange decorators.
That tracks. Ruby followed in the footsteps of Perl, which had string manipulation as a main priority for the language.
That issue has a link to another Issue with more discussion: https://github.com/golang/go/issues/57616.
But as is all too common in the go community, there seems to be a lot of confusion about what is proposed, and resistance to any change.
D had a big blow up over string interpolation. Walter wanted something simple and the community wanted something more like these template ones from Python (at least from scanning the first little bit of the PEP). Walter eventually went with what the community wanted.
This led to the OpenD language fork (https://opendlang.org/index.html) which is led by some contributors who had other more general gripes with D. The fork is trying to merge in useful stuff from main D, while advancing the language. They have a Discord which unfortunately is the main source of info.
It's especially weird how hard people have to fight for string interpolation given it has had implementations since the 1970s.
Even PEP 498 (fstrings) was a battle.
Superficially f-strings reminds you of php and everyone remembers how awful that was. But Python's implementation is leagues better and we also have better tooling (ie smart parsers) for handling fstrings.
I promise, no trolling from me in this comment. I never understood the advantage of Python f-strings over printf-style format strings. I tried to Google for pros and cons and didn't find anything very satisfying. Can someone provide a brief list of pros and cons? To be clear, I can always do what I need to do with both, but I don't know f-strings nearly as well as printf-style, because of my experience with C programming.
Sure, here are the two Go/C-style formatting options:
And the Python f-string equivalent: The Sprintf version sucks because for every formatting argument, like "%s", we need to stop reading the string and look for the corresponding argument to the function. Not so bad for one argument but gets linearly worse.Sprint is better in that regard, we can read from left to right without interruptions, but is a pain to write due to all the punctuation, nevermind refactor. For example, try adding a new variable between "This" and "house". With the f-string you just type {var} before "house" and you're done. With Sprint, you're now juggling quotation marks and commas. And that's just a simple addition of a new variable. Moving variables or substrings around is even worse.
Summing up, f-strings are substantially more ergonomic to use and since string formatting is so commonly done, this adds up quickly.
For all its other problems, f-strings make Python such a pleasure to work with. C# has something similar IIRC.
Looks great - unlike java which is somehow recommending the format:
STR."Hello \{this.user.firstname()}, how are you?\nIt's \{tempC}°C today!"
compared to scala
s"Hello ${this.user.firstname()}, how are you?\nIt's ${tempC}°C today!"
STR."" ? really?
Yeah, I hate to bikeshed, but this is the worst syntax possible without being a full-out prank.
also a syntax for braces that looks like escaping
>Go developers seem to have taken no more than 5 minutes considering the problem, then thoughtlessly discarded it
The issue you linked was opened in 2019 and closed with no new comments in 2023, with active discussion through 2022.
Nick Humrich here, the author who helped rewrite PEP 501 to introduce t-strings, which was the foundation for this PEP. I am not an author on this accepted PEP, but I know this PEP and story pretty well. Let me know if you have any questions.
I am super excited this is finally accepted. I started working on PEP 501 4 years ago.
I often read concerns that complexity keeps being added to the language with yet another flavour of string or whatnot. Given that those who author and deliberate on PEPs are, kind of by definition, experts who spend a lot of time with the language, they might struggle to grok the Python experience from the perspective of a novice or beginner. How does the PEP process guard against this bias?
There are many long-term users of Python who participate in PEP discussion who argue for beginners[0], often because they professionally are teaching Python.
There are also loads of people basically defaulting to "no" on new features, because they understand that there is a cost of supporting things. I will often disagree about the evaluation of that cost, but it's hard to say there is no cost.
Nobody wants a system that is unusable, slow, hard to implement for, or hard to understand. People sometimes just have different weights on each of these properties. And some people are in a very awkward position of overestimating costs due to overestimating implementation effort. So you end up in discussions like "this is hard to understand!" "No it isn't!"
Hard to move beyond, but the existence of these kinds of conversations serve, in a way, as proof that people aren't jumping on every new feature. Python is still a language that is conservative in what it adds.
This should actually inspire more confidence in people that features added to Python are _useful_, because there are many people who are defaulting to not adding new features. Recent additions to Python speeding up is more an indicator of the process improving and identifying the good stuff rather than a lowering of the bar.
[0]: I often think that these discussions often get fairly intense. Understandability is definitely a core Python value, but I Think sometimes discussions confuse "understandability" with "amount of things in the system". You don't have to fully understand pervasive hashing to understand Python's pervasive value equality semantics! A complex system is needed to support a simple one!
All discussion on PEP's happens in public forums where anyone can opine on things before they are accepted. I agree that the experts are more likely to participate in this exchange. And while this is wish-washy, I feel like the process is really intended to benefit the experts more than the novices anyways.
There have been processes put into place in recent years to try to curb the difficulty of things. One of those is that all new PEPs have to include a "how can you teach this to beginers" section, as seen here on this pep: https://peps.python.org/pep-0750/#how-to-teach-this
I think "how can you teach this to beginners?" is a fantastic, low-hanging fruit option for encouraging the wizards to think about that very important type of user.
Other than a more broad "how is the language as a whole faring?" test, which might be done through surveys or other product-style research, I think this is just plainly a hard problem to approach, just by the nature that it's largely about user experience.
"How does this fit with everything else beginners have to learn to understand basic code?" is sorely needed.
The average Python developer does not even know what a "PEP" is. Open discussion is good yes, but no one really knows what the average developer wants because they simply does not care if its Python or Java or whatever else.
"Some hammers are just shaped weird, oh well, just make do with it."
For example, some people that I interview does not "get" why you have to initialize the dict before doing dict[k] += 1. They know that they have to do some ritual of checking for k in dict and dict[k] = 0. But they don't get that += desugars into dict[k] = dict[k] + 1.
defaultdict is so regularly useful for just this reason!
You might find the Python discussion forums ([0] and [1]) interesting; conversation that guides the evolution of PEPs happens there.
As Nick mentioned, PEP 750 had a long and winding road to its final acceptance; as the process wore on, and the complexities of the earliest cuts of the PEPs were reconsidered, the two converged.
[0] The very first announcement: https://discuss.python.org/t/pep-750-tag-strings-for-writing...
[1] Much later in the PEP process: https://discuss.python.org/t/pep750-template-strings-new-upd...
Building off this question, it's not clear to me why Python should have both t-strings and f-strings. The difference between the two seems like a stumbling block to new programmers, and my "ideal python" would have only one of these mechanisms.
f-strings immediately become a string, and are "invisible" to the runtime from a normal string. t-strings introduce an object so that libraries can do custom logic/formatting on the template strings, such as decided _how_ to format the string.
My main motivation as an author of 501 was to ensure user input is properly escaped when inserting into sql, which you cant enforce with f-strings.
> ensure user input is properly escaped when inserting into sql
I used to wish for that and got it in JS with template strings and libs around it. For what it’s worth (you got a whole PEP done, you have more credibility than I do) I ended up changing my mind, I think it’s a mistake.
It’s _nice_ from a syntax perspective. But it obscures the reality of sql query/parameter segregation, it builds an abstraction on top of sql that’s leaky and doesn’t even look like an abstraction.
And more importantly, it looks _way too close_ to the wrong thing. If the difference between the safe way to do sql and the unsafe way is one character and a non-trivial understanding of string formatting in python… bad things will happen. In a one-person project it’s manageable, in a bigger one where people have different experiences and seniority it will go wrong.
It’s certainly cute. I don’t thing it’s a good thing for sql queries.
I understand your concern, and I think the PEP addresses it. Quite bluntly, t"foo" is not a string, while f"foo" is. You'll get a typecheck error if you run a typechecker like any reasonable developer, and will get a runtime error if you ignore the type mismatch, because t"foo" even lacks a __str__() method.
One statement the PEP could put front and center in the abstract could be "t-strings are not strings".
> "t-strings are not strings"
t-string is an unfortunate name for something that is not a string.
I wish it were called "string templates" instead, with t"whatever" form being called a "template literal".
Simpson's Individual Stringettes!
https://www.youtube.com/watch?v=7qNj-QFZbew
Away with floods! Away with workaday tidal waves!
That all make senses to me. But it definitely won't make sense to my intro to programming students. They already have enough weird syntax to juggle.
Then dont teach them t-strings
For one thing, `f"something"` is of type `str`; `t"something"` is of type `string.templatelib.Template`. With t-strings, your code can know which parts of the string were dynamically substituted and which were not.
The types aren't so important. __call__ or reference returns type string, an f and a t will be interchangeable from the consumer side.
Example, if you can go through (I'm not sure you can) and trivially replace all your fs with ts, and then have some minor fixups where the final product is used, I don't think a migration from one to the other would be terribly painful. Time-consuming, yes.
Not sure that's true. `Template`s don't provide a `__str__` function, so you need to pass them to a processing function to get a `str` back.
https://peps.python.org/pep-0750/#no-template-str-implementa...
Give it a few years to when f-string usage has worn off to the point that a decision can be made to remove it without breaking a significant number of projects in the wild.
That will never happen.
Well if it continues to be popular then that is all good. Just keep it. What matters is that usage isn't complex for anyone.
Well now we'll have four different ways to format strings, since removing old ones is something that doesn't actually happen:
This is where an opinionated linter comes in handy. Ensures people gradually move to the “better” version while not breaking backwards compatibility.
It does suck for beginners who end up having to know about all variations until their usage drops off.
The linter is a big deal, actually. I've worked with Python off and on during the past few decades; I just recently moved onto a project that uses Python with a bunch of linters and autoformatters enabled. I was used to writing my strings ('foo %s % bar), and the precommit linter told me to write f'foo %{bar}'. Easy enough!
printf-style formatting ("foo %s" % "bar") feels the most ready to be retired (except insofar as it probably never will, because it's a nice shortcut).
The other ones at least are based on the same format string syntax.
"foo {}".format("bar") would be an obvious "just use f-string" case, except when the formatting happens far off. But in that case you could "just" use t-strings? Except in cases where you're (for example) reading a format string from a file. Remember, t- and f- strings are syntactic elements, so dynamism prevents usage of it!
So you have the following use cases:
- printf-style formatting: some C-style string formatting is needed
- .format: You can't use an f- string because of non-locality in data to format, and you can't use a t- string due to dynamism in
- f-string: you have the template and the data in the same spot lexicographically, and you just want string concatenation (very common!)
- t-string: you have the template and the data in the same spot lexicogrpahically, but want to use special logic to actually build up your resulting value (which might not even be a string!)
The last two additions being syntax makes it hard to use them to cover all use cases of the first two.
But in a specific use case? It's very likely that there is an exact best answer amongst these 4.
> printf-style formatting ("foo %s" % "bar") feels the most ready to be retired (except insofar as it probably never will, because it's a nice shortcut).
It’s also the only one which is anything near safe for being user provided.
I don’t think I’ve ever used % formatting in Python - what makes it safer than `format`?
`str.format` allows the format string to navigate through indexes, entries, and attributes. If the result of the formatting is echoed back and any non-trivial object it passed in, it allows for all sorts of introspection.
printf-style... does not support any of that. It can only format the objects passed in.
Very good point. While I think we could do away with the syntactic shorthand, definitely would want to keep some function/method around with the capabilities.
.format is also nice because you can have more complex subexpressions broken over multiple lines instead of having complex expressions inside the {}.
And if it's being used, and isn't considered problematic, then it should remain. I've found use for all the current ones: (1) for text that naturally has curlies, (2) for templating (3) for immediate interpolation, and improved at-site readability
I see (4) being about the flexibility of (2) and readability of (3). Maybe it'll eventually grow to dominate one or both, but it's also fine if it doesn't. I don't see (1) going away at all since the curly collision still exists in (4).
Five, if you count the log module. I hope t-strings will come there soon.
log.error("foo happend %s", reason)
Don't forget string.Template:
Putting down my marker on the opposite. Once you're targeting a version of python that has t-strings, decent linters/libraries have an excuse to put almost all uses of f-strings in the ground.
No backward compatibility?!
If the usage of a feature is significantly close enough to 0 because there is a well used alternative, what need is there for backward compatibility? If anything, it can be pushed to a third party package on PyPI.
My memory is that ES6's template strings preceded f-strings. If that is correct, do you happen to know why python was saddled with f-strings, which seem like an obviously inferior design, in the first place? We are now at five largely redundant string interpolation systems (%, .format, string.Template, f-string, t-string).
PEP 501 when originally written (not by me) was intended to be the competing standard against f-strings, and to have been more inline with ES6's template strings. There was debate between the more simple f-string PEP (PEP 498) and PEP 501. Ultimately, it was decided to go with f-strings as a less confusing, more approachable version (and also easier to implement) and to "defer" PEP 501 to "see what happens". Since then, the python internal have also changed, allowing t-strings to be even easier to implement (See PEP 701). We have seen what happens, and now its introduced. f-strings and t-strings are not competing systems. They are different. Similar to ES6 templates and namedTaggedTemplates, they are used for different things while API feels similar intentionally. f-strings are not inferior to t-strings, they are better for most use cases of string templating where what you really want, is just a string.
Thanks!
> they are better for most use cases of string templating where what you really want, is just a string.
I think use cases where you want to unconditionally bash a string together are rare. I'd bet that in > 80% of cases the "just a string" really is just a terrible representation for what really is either some tree (html, sql, python, ...) structure or at least requires lazy processing (logging, where you only want to pay for the expensive string formatting and generation if you run at the log level or higher that the relevant logging line is meant to operate).
I'm not familiar with ES6 template strings, but why are they better than f-strings? F-strings just work, and work well, in my experience so I'm wondering what I'm missing out on. Especially since the language I use the most is c++... So I guess I don't expect much out of string manipulation lol.
The problem with f-strings is that they make an extremely limited use case convenient (bashing unstructured text) and thus people people invariably use them for the less limited use case for which no such covenient mechanism exists. Constructing ASTs (including html and SQL). Or logging (where you want to avoid unconditionally computing some expensive string represenation).
I do this myself. I basically always use the subtl wrong log.warning(f"Unexpected {response=} encountered") and not the correct, and depending on the loglevel cheaper log.warning("Unexpected respone=%s encountered", repsonse). The extra visual noise is typically not worth the extra correctness and performance (I'd obviously not do this in some publically exposed service receiving untrusted inputs).
I'd argue these use cases are in fact more prevalent then the bashing unstructured text use case.
Encouraging people to write injection vulnerabilities or performance and correcness bugs isn't great language design.
ES2015 template strings from the beginning supported "tagged template literals" where the tag is a function that gets the template itself and the objects passed to the "holes" in the template as separate arguments. From there that function can do things like turn the holes themselves into something like SQL Parameter syntax and wrap the things that go in those holes in properly escaped SQL Parameters.
`some template {someVar}` was f-strings and someFunction`some template {someVar}` was more like what these t-strings provide to Python. t-strings return an object (called Template) with the template and the things that go into the "holes", versus tagged templates are a function calling pattern, but t-strings are still basically the other, richer half of ES2015+ template strings.
Thank you for your work on this topic and for answering questions here. I have a question: is there a way to avoid the security issues with string formatting described here? It seems like all (most?) string formatting options suffer from the same issue.
https://lucumr.pocoo.org/2016/12/29/careful-with-str-format/
Thank you! Curious what options for deferred evalution were considered and rejected? IMHO, the main benefit of deferred evaluation isn't in the saving of a bit of code to define a deferred evaluation class, but in standardazing the API so that anyone can read the code without having to learn what it means in each project.
Also: were prompt templates for LLM prompt chaining a use case that influenced the design in any way (examples being LangChain and dozens of other libraries with similar functionlity)?
One solution that existed for a while was using the `!` operator for deferred. `t!'my defered {str}'`
The main reason for non having deferred evaluation was that it over-complicated the feature quite a bit and introduces a rune. Deferred evaluation also has the potential to dramatically increase complexity for beginners in the language, as it can be confusing to follow if you dont know what is going on. Which means "deferred by default" wasnt going to be accepted.
As for LLM's, it was not the main consideration, as the PEP process here started before LLM's were popular.
Ah interesting, so the complexity wasn't in the API design or implementation, but only in the additional rune? Is that really such a big cost?
> were prompt templates for LLM prompt chaining a use case that influenced the design in any way
Maybe not directly, but the Python community is full of LLM users and so I think there's a general awareness of the issues.
Is there an example of how these could be used in LLM prompting?
As I understand, it may help a bit with logging performance, not sure, still trying to understand the template abilities.
So, right now, you have two options to log:
1. `logger.debug(f'Processing {x}')` - looks great, but evaluates anyway, even if logging level > `logging.DEBUG`;
2. `logger.debug('Processing %s', x)` - won't evaluate till necessary.
What would be the approach with t-strings in this case? Would we get any benefits?
The expression (x) is eagerly evaluated in both cases, cuz that's how Python works. You can defer the format call but Python fundamentally doesn't have an equivalent of lazy/compile time flag argument evaluation and this doesn't change that.
For a logger t-strings are mostly just a more pleasant and less bug-prone syntax for #2
T-strings, like f-strings, are eagerly evaluated -- so in this sense, no, there's no benefit.
Not quite; the interpolations are not eagerly stringified which is the potentially expensive part. In this sense it's kind of a middle ground between the two approaches.
Sure, good point; I generally think of evaluation, not stringification, as the likely dominant expense. But maybe it’s sometimes the other way around?
Nice work on PEP 501! Probably a silly question, but how comes PEP 292 isn't mentioned anywhere in PEP 750?
My hope is to write some new documentation as 3.14 nears release that explains the (growing) constellation of string formatting mechanisms in Python and describes when they might each be useful. They overlap to some degree, but each has a unique twist that makes them useful in different situations. PEP 292 is going nowhere and is used, for instance, in really powerful libraries like `flufl.i18n`
Is a PEP 750 Template entirely different from a PEP 292 Template? I’m a bit confused about the relationship.
Yeah, they’re unrelated. (PEP 750 places its Template and Interpolation classes in the new string.templatelib)
Does Python really need yet another type of string literal? I feel like while templating is a good addition to the standard library, it's not something that needs syntactic support. t"blah blah" is just an alias for Template("blah blah", context), isn't it?
yes, it does actually need syntax support. In order for it to work, you need to preserve which parts of the string are static (hard coded) and which parts are dynamic (likely user input). Which you can only do at a syntax level. You could potentially do it by hand, using placeholders, like with `%`, but as we now live in f-string world, we have something better. The syntax highlighting and ergonomics of f-strings are so good, devs prefer it in most cases. The idea is to make the most ergonomic thing, also the safest thing. By decreasing ergonomics, you reduce the adoption of safer symantics.
That's why I specified the context argument. Something like Template("{name} {address}", dict(name = ..., address = ...)) would be exactly equivalent to t"{name} {address}" assuming those variables are fetched from the local scope.
Yes, which is essentially how SQLalchemy works today. You can still put strings in the context though, so for more complex things, it's turtles all the way down. Also, as f-srrings are more ergonomic, people now reach for them, even when they shouldn't
So now you are repeating the name of each interpolated value three times (once in the template string, once for the key in the dict, once for the actual value).
Yes, you can do that, but a t-string is much more ergonomic, and IMO, more readable .
Yes, of course. Any function with syntactic support will be more "ergonomic" than one without. But t-strings are for a relatively niche use case and can't even fully replace f-strings since they aren't actually strings. Even for input sanitizing they seem insufficient since you cant use them to create pre-compiled statements/templates.
Preventing injection attacks in sql queries, html, etc. is a niche use case?
Yes. And it also relies on pre-compilation, which t-strings do not support.
No, you don't need pre-compilation, assuming by pre-compilation you mean compiling a template that you pass values to later[1]
t-strings allow a library to perform transformations on the values, such as escaping them, or passing them as separate values to a parameterized query. Escaping html and parameterizing sql queries were the first two example use cases given in the PEP.
And I disagree that such use cases are niche. In my experience, needing to sanitize user input is an extremely common thing to need to do, and having the language and library make it as easy as possible to do it correctly is a very good thing.
[1]: I do wish they hadn't called these Templates, because it's not really a template so much as an intermediate representation of an interpolated value.
Yes, web is a niche. Outside of web there is simply no possibility of turning untrusted input into executable code so sanitation isn't needed. In web development you already have two dozen templating libraries that offer much more comprehensive safe and fast text-generation solutions than what t-strings do.
Pre-compilation means that you first compile the template, then you supply the template with values when you render it multiple times. This is not possible with t-strings since the values are bound when the t-string is created.
A library can't capture interpolated variables
[flagged]
Most excellent! I love f-strings and replaced all the various other string interpolation instances in my code with them, but they have the significant issue that you can't defer evaluating them. For instance, you can write:
Until this, there wasn't a way to use f-strings formatting without interpolating the results at that moment: It was annoying being able to use f-strings almost everywhere, but str.format in enough odd corners that you have to put up with it.Delayed execution is basically equivalent to a function call, which is already a thing. It also has basically the same API as point of use and requires maybe 1 extra line.
I'm not sure if t-strings help here? unless I misread the PEP, it seems like they still eagerly evaluate the interpolations.
There is an observation that you can use `lambda` inside to delay evaluation of an interpolation, but I think this lambda captures any variables it uses from the context.
> There is an observation that you can use `lambda` inside to delay evaluation of an interpolation, but I think this lambda captures any variables it uses from the context.
Actually lambda works fine here
> I'm not sure if t-strings help here?
That's correct, they don't. Evaluation of t-string expressions is immediate, just like with f-strings.
Since we have the full generality of Python at our disposal, a typical solution is to simply wrap your t-string in a function or a lambda.
(An early version of the PEP had tools for deferred evaluation but these were dropped for being too complex, particularly for a first cut.)
And that actually makes "Template Strings" a misnomer in my mind. I mean, deferred (and repeated) evaluation of a template is the thing that makes template a template.
Kinda messy PEP, IMO, I'm less excited by it than I'd like to be. The goal is clear, but the whole design feels backwards.
Naming is hard; for instance, JavaScript also has its own template strings, which are also eagerly evaluated.
> An early version of this PEP proposed that interpolations should be lazily evaluated. [...] This was rejected for several reasons [...]
Bummer. This could have been so useful:
(Though str.format isn’t really that bad here either.)There was a very very long discussion on this point alone, and there are a lot of weird edge cases, and led to weird syntax things. The high level idea was to defer lazy eval to a later PEP if its still needed enough.
There are a lot of existing workarounds in the discussions if you are interested enough in using it, such as using lambdas and t-strings together.
Would be useful in that exact case, but would be an absolute nightmare to debug, on par with using global variables as function inputs
Yeah, to be honest, every time this comes to mind I think “wow, this would be really neat!”, then realize just using .format() explicitly is way easier to read.
I do think that people are far too hesitant to bind member functions sometimes:
(it's likely that typecheckers suck at this like they suck at everything else though)You could do something like t”Hello, {“name”}” (or wrap “name” in a class to make it slightly less hacky).
Lambda only captures variables which haven't been passed in as argument.
PEP 750 doesn't directly address this because it's straightforward to simply wrap template creation in a function (or a lambda, if that's your style):
The issue was solved by having a rich and Turing complete language. I am not huge on adding language features. This seems like userland stuff.
I tend to agree. I think it's easy enough to use a lambda in this case
You gravely misunderstand.
The point of evaluation of the expressions is the same.
is still an error if you haven't defined name.BUT the result of a t-string is not a string; it is a Template which has two attributes:
So you can then operate on the parts separately (HTML escape, pass to SQL driver, etc.).I read 'most excellent' in the Bill and Ted voice
Then it worked.
[dead]
I’m not convinced that a language level feature is worth it for this. You could achieve the same thing with a function returning an f-string no? And if you want injection safety, just use a tag type and a sanitisation function that takes a string and returns the type. Then the function returning the f-string could take the Sanitised string as an argument to prevent calling it with unsanitised input.
I guess it’s more concise, but differentiating between eager and delayed execution with a single character makes the language less readable for people who are not as familiar with Python (especially latest update syntax etc).
EDIT: to flesh out with an example:
class Sanitised(str): # init function that sanitises or just use as a tag type that has an external sanitisation function.
def sqltemplate(name: Sanitised) -> str: return f”select * from {name}”
# Usage sqltemplate(name=sanitise(“some injection”))
# Attempt to pass unsanitised sqltemplate(name=“some injection”) # type check error
> You could achieve the same thing with a function returning an f-string no no.
> just use a tag type and a sanitisation function that takes a string and returns the type
Okay, so you have a `sqlstring(somestring)` function, and the dev has to call it. But... what if they pass in an f-string?
`sqlstring(f'select from mytable where col = {value}')`
You havent actually prevented/enforced anything. With template strings, its turtles all the way down. You can enforce they pass in a template and you can safely escape anything that is a variable because its impossible to have a variable type (possible injection) in the template literal.
Added example to parent comment.
This example still works, the entire f-string is sanitised (including whatever the value of name was). Assuming sqlstring is the sanitisation function.
The “template” would be a separate function that returns an f-string bound from function arguments.
Yes. Only if your dev remembers to use sanatized all the time. This is how most SQL works today. You could also forget and accidentally write a f-string, or because you dont know. But with t-strings you can actually prevent unsanatized inputs. With your example, you need to intentionally sanitize still.
You cant throw an error on unsanitized because the language has no way to know if its sanitized or not. Either way, its just a string. "returning an f-string" is equivalent to returning a normal string at runtime.
Well you enforce this with types. That’s how every other language does it. By specifying that the type of the function has to be a sanitised string, it will reject unsanitised string with the type checker.
> it has no way of knowing if it’s sanitised or not
It does. You define the SanitisedString class. Constructing one sanitises the string. Then when you specify that as the argument, it forces the user to sanitise the string.
If you want to do it without types, you can check with `isinstance` at runtime, but that is not as safe.
Your example is a bit too simple. What I mean by that is, you have hardcoded your function to inject a specific part of your string. But t-strings allow you to write the full query `t'select * from table where name = {name}'` directly, without have to use a function. This matters because the SQL connection library itself can enforce templates. SQL libraries can NOT enforce "sanitized types" because then you couldnt write raw sql without problems. They have to know the difference between "this is hard coded" and "this is a dynamic user variable". And the libraries can't know that, without t-strings.
No, most SQL today uses placeholders and has since circa 2008. If you are sanitizing you are doing it wrong to begin with.
If its only use is to make injecton safety a bit easier to achieve, it's worth it to me.
Does it make it easier? The “escape” for both is to just use unsafe version of the Template -> string function or explicitly mark an unsafe string as sanitised. Both seem similar in (un)safety
> the Template -> string function
There is no such function; Template.__str__() returns Template.__repr__() which is very unlikely to be useful. You pretty much have to process your Template instance in some way before converting to a string.
Right, but it is possible to write a template -> string function that doesn’t sanitise and use it (or more realistically use the wrong one). Just as it’s possible to unsafely cast an unsafe string to a sanitised one and use it (rather than use a sanitise function that returns the wrapper type).
They are both similar in their unsafety.
It's worse than function returning an f-string. Template type is very flat, you won't know which arguments are left unbound.
modules, classes, protocols, functions returning functions, all options in Python, each work well for reuse, no need to use more than 2 at once, yet the world swims upstream.
How do you leave arguments unbound?
Yes, exactly. T-string arguments can't be unbound; they are eagerly evaluated.
I don't see how this prevents calling your returned f-string with unsensitized inputs.
I’m not sure you understood my example. The f-string is within a function. The function argument only accepts sanitised input type.
If you create a subclass of str which has an init function that sanitises, then you can’t create a Sanitised type by casting right?
And even if you could, there is also nothing stopping you from using a different function to “html” that just returns the string without sanitising. They are on the same relative level of safety.
Oh, I'm pretty sure I didn't understand your example and am probably missing something obvious. That's why I'm here asking dumb questions!
I think I'm following more, and I see how you can accomplish this by encapsulating the rendering, but I'm still not seeing how this is possible with user facing f-strings. Think you can write up a quick example?
Added example to parent comment.
Thanks mate! (BTW: Indenting code with four spaces makes HN format it like code.)
So the thing I'm still not getting from your example is allowing the template itself to be customized.
template1 is a function that takes in a parameter evil (with a SanitisedString type that wraps a regular str) and returns the fully expanded str. It is implemented by just returning an f-string equivalent to the t-string in your example. Same with template2.
Using the SanitisedString type forces the user to explicitly call a sanitiser function that returns a SanitisedString and prevents them from passing in an unsanitised str.
You're just handing off responsibility for sanitization to the user instead of the library author.
With t-strings the rendering function is responsible for sanitization, and users can pass unrendered templates to it.
With f-strings there's no concept of an unrendered template, it just immediately becomes a string. Whoever is creating the template therefore has to be careful what they put in it.
I'm excited about this. I really like how JavaScript's tagged template literals https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe... can help handle things like automatic HTML escaping or SQL parameterization, it looks like these will bring the same capability to Python.
Yes! PEP 750 landed exactly there: as a pythonic parallel to JavaScript's tagged template strings. I'm hopeful that the tooling ecosystem will catch up soon so we see syntax coloring, formatting of specific t-string content types, etc. in the future.
I just wish it didn't get Pythonified in the process, e.g. needing to be a function call because backtick is hard to type on some keyboards, nearly having a completely separate concept of evaluating the arguments, etc. x`` vs x(t'') is a 2x blowup in terms of line-noise at worst.
I don't think that's the reason.
https://peps.python.org/pep-0750/#arbitrary-string-literal-p...
that gets solved by using a different quoting character for template literals.
https://discuss.python.org/t/pep-750-tag-strings-for-writing...
> Backticks are traditionally banned from use in future language features, due to the small symbol. No reader should need to distinguish ` from ' at a glance. It’s entirely possible that the prevailing opinion on this has changed, but it’s certainly going to be easier to stick to the letter prefixes and regular quotes.
and earlier in the thread the difficulty in typing it for some is cited as another reason
That's fascinating, TIL backticks are banned from Python.
Maintainer of lit-html here, which uses tagged template literals in JavaScript extensively.
This looks really great! It's almost exactly like JavaScript tagged template literals, just with a fixed tag function of:
It's pretty interesting how what would be the tag function in JavaScript, and the arguments to it, are separated by the Template class. At first it seems like this will add noise since it takes more characters to write, but it can make nested templates more compact.Take this type of nested template structure in JS:
With PEP 750, I suppose this would be: Python's unfortunate lambda syntax aside, not needing html() around nested template could be nice (assuming an html() function would interpret plain Templates as HTML).In JavaScript reliable syntax highlighting and type-checking are keyed off the fact that a template can only ever have a single tag, so a static analyzer can know what the nested language is. In Python you could separate the template creation from the processing possibly introduce some ambiguities, but hopefully that's rare in practice.
I'm personally would be interested to see if a special html() processing instruction could both emit server-rendered HTML and say, lit-html JavaScript templates that could be used to update the DOM client-side with new data. That could lead to some very transparent fine-grained single page updates, from what looks like traditional server-only code.
> assuming an html() function would interpret plain Templates as HTML
Agreed; it feels natural to accept plain templates (and simple sequences of plain templates) as HTML; this is hinted at in the PEP.
> html(t"<ul>{map(lambda i: t"<li>{i}</li>", items)}</ul>")
Perhaps more idiomatically: html(t"<ul>{(t"<li>{i}</li>" for i in items)}</ul>")
> syntax highlighting and type-checking are keyed off the fact that a template can only ever have a single tag
Yes, this is a key difference and something we agonized a bit over as the PEP came together. In the (very) long term, I'm hopeful that we see type annotations used to indicate the expected string content type. In the nearer term, I think a certain amount of "clever kludginess" will be necessary in tools like (say) black if they wish to provide specialized formatting for common types.
> a special html() processing instruction could both emit server-rendered HTML and say, lit-html JavaScript templates that could be used to update the DOM client-side with new data
I'd love to see this and it's exactly the sort of thing I'm hoping emerges from PEP 750 over time. Please do reach out if you'd like to talk it over!
Python already has built-in data structure literals that allow you to express lispy DSLs:
I guess template strings do make it more concise. Kind of like Racket's "#lang at-exp racket".The benefit of lisp-like representation is you have the entire structure of the data, not just a sequence of already-serialized and not-yet-serialized pieces.
Generally a generator expression would be more idiomatic in Python than map/lambda.
I'd like to add, after the first publication for discussion, we got some wonderful involvement from Andrea Giammarchi who brought his deep JS libraries and tools experience into the PEP. In fact, he's deeply involved in the next steps, with some forthcoming demos and libraries that will make a real difference. Exciting times.
> not needing html() around nested template could be nice
One possibility would be to define __and__ on html so that you can write e.g. html&t"<b>{x}</b>" (or whichever operator looks the best).
So we are well on our way to turning python to PHP.
Edit: Sorry I was snarky, its late here.
I already didn't like f-strings and t-strings just add complexity to the language to fix a problem introduced by f-strings.
We really don't need more syntax for string interpolation, in my opinion string.format is the optimal. I could even live with % just because the syntax has been around for so long.
I'd rather the language team focus on more substantive stuff.
> turning python to PHP.
Why stop there? Go full Perl (:
I think Python needs more quoting operators, too. Maybe qq{} qq() q// ...
[I say this as someone who actually likes Perl and chuckles from afar at such Python developments. May you get there one day!]
Quoting operators are something I actually miss in Python whereas t-strings are something I have never wanted in 17 years of writing Python.
What's the issue with f-strings? I'm wondering because I thought they basically had no downside versus using the older alternatives. I use them so often that they are very substantive to me. If anything, this is exactly what python should be focusing on, there really isn't a lot more that they can do considering the design, expectations, and usage of python.
In the motivation for the t-string types, their gripe is that f-strings are not templates.
My issue with them is that you have to write your syntax in the string complex expressions dictionary access and such become awkward.
But, this whole thing is bike-shedding in my opinion, and I don't really care about the color of the bike shed.
Pretty sure PHP does not have this feature. Can you give me an example?
I believe that jab was that PHP has a bunch of ways to do similar things and Python, in their view, is turning out that way, too.
On a more philosophical level php is this feature. At least as it was used originally and how it's mostly used today. PHP was and is embedded in html code. If you have a look at a wordpress file you are going to see something like this:
<?php ... ?><some_markup>...<? php ... ?><some_more_markup here>...
string.format and string substitution are bloat and annoying to use, while f-strings makes it very easy to improve readability. So in the end, they remove big complexity in usage, by adding very little and straightforward complexity in syntax.
Am I missing something, or is this a fancier string.Template [0]? Don't get me wrong, it looks very useful, especially the literals.
[0]: https://docs.python.org/3/library/string.html#template-strin...
string.Template does fundamentally the same sort of thing as str.format, or str.__mod__ (i.e. the % operator on strings). It was in a sense a prototype for str.format, introduced way back in Python 2.4 (https://peps.python.org/pep-0292/). It does create an instance of a separate type, but it neither pre-parses the string (the substitution is implemented using regexes that reprocess the original string each time) nor eagerly evaluates the values to substitute. Further, in order to change how the formatting is done, you'd have to subclass (with the new system, you just write a function that accepts the Template as a parameter).
I like this but am still not a fan of the constant adding of things to the language. It’s starting to feel like a language designed by committee, which it basically is.
It's a feature that removes the need for hundreds of specialized template classes in various frameworks and is a generalization of an existing feature in a way that doesn't add much in the way of grammar. The net complexity reduction is a win here.
> It’s starting to feel like a language designed by committee, which it basically is.
That's exactly what it is. It's just that they use the word "council" instead of "committee".
Python is released annually and having some new juice helps advertise the language.
Whether or not this is technically a swift call is in the eye of the beholder.
[dead]
If you want to see a usage for this I've built, and use, [SQL-tString](https://github.com/pgjones/sql-tstring) as an SQL builder.
Putting aside template strings themselves for the moment, I'm stunned by some of the code in this PEP. It's so verbose! For example, "Implementing f-strings with t-strings":
Is this what idiomatic Python has become? 11 lines to express a loop, a conditional and a couple of function calls? I use Python because I want to write executable pseudocode, not excessive superfluousness.By contrast, here's the equivalent Ruby:
We wanted at least a couple examples that showed use of Python's newer pattern matching capabilities with the new Interpolation type. From this outsider's perspective, I'd say that developer instincts and aesthetic preferences are decidedly mixed here -- even amongst the core team! You can certainly write this as:
Or, y'know, several other ways that might feel more idiomatic depending on where you're coming from.Your example is interesting - `join`, a generator expression and a ternary, and Python requires us to write all of them inside-out. It's a shame we can't write that in a more natural order, something like:
Yeah. `join` has forever been backwards to me in Python and I still sometimes get it wrong the first time out.
Comprehensions, though -- they are perfection. :-)
IMO comprehensions like `[x**2 for x in range(4)]` would be better written as `[for x in range(4): x**2]`.
That would make the mapping between a comprehension and the equivalent loop much clearer, especially once you use nested loops and/or conditionals.
For example, to flatten a list of lists `l = [[1, 2], [3], [4, 5, 6]]`:
vsI think it's very deliberate that these sorts of expressions are inside-out from the corresponding statements. It mixes and matches modes of thinking in my mind - the same kind of mistake as command-query separation violations. When I read a list comprehension, I can read it left to right: "A list (`[`) of each `item` that I get, if I look `for` each `sublist` that is `in l`, then `for` each `item` which is `in` that `sublist`." And the entire description is - well - descriptive, not imperative. With the other order, I have to think in terms of explicitly following the steps, and then mentally translating the last `item` into an imperative "... and put it into the result list".
That would make dict comprehensions confusing due to two occurrences of colons with different semantics.
Ah, yes - good point!
This is how Python has always been. It's more verbose and IMO easier to grok, but it still lets you create expressive DSLs like Ruby does.
Python has always been my preference, and a couple of my coworkers have always preferred Ruby. Different strokes for different folks.
> This is how Python has always been.
Nah, idiomatic Python always used to prefer comprehensions over explicit loops. This is just the `match` statement making code 3x longer than it needs to be.
You can express the loop as a list comprehension, and I would too.
As for the logic, I would still use pattern matching for branching and destructuring, but I’d put it in a helper. More lines is not a negative in my book, though I admit the thing with convert and format is weird.
> I would still use pattern matching for branching and destructing, but I’d put it in a helper
Yeah, using a helper function makes things much clearer. To be honest though, I'm not a huge fan of using either `isinstance` (which is generally a sign of a bad design) nor `match/case` (which is essentially a "modern" way to write `isinstance`).
I can't help but think that a better design could avoid the need for either of those (e.g. via polymorphism).
What I would use is an ADT and I make a wish for that every day in every language that’s not Haskell :D
Haskell also has operator overloading on steroids so you could use the (|>) operator from Flow and write transformations the same as you would shell pipes. I’d love to whip up an example but it’s difficult on this tiny screen. Will try to remember when I’m on my computer.
Before someone chimes in with ML propaganda, I warn you that I’m going to exercise my rights under the Castle Doctorine the moment you say “of”.
You can access .strings and .interpolations directly and avoid type checks. There is always one more string than interpolation, of which there may be zero.
> As for the logic, I would still use pattern matching for branching and destructuring, but I’d put it in a helper.
I wrote it up (https://news.ycombinator.com/item?id=43650001) before reading your comment :)
A compromise version:
The `match` part could still be written using Python's if-expression syntax, too. But this way avoids having very long lines like in the Ruby example, and also destructures `item` to avoid repeatedly writing `item.`.I very frequently use this helper-function (or sometimes a generator) idiom in order to avoid building a temporary list to `.join` (or subject to other processing). It separates per-item processing from the overall algorithm, which suits my interpretation of the "functions should do one thing" maxim.
The Python version is straightforward to read and understand to a programmer of any language. The Ruby version is an idiosyncratic symbol soup.
If I were tasked to modify the Python version to say, handle the case where `item` is an int, it would be immediately obvious to me that all I need to do is modify the `match` statement with `case int() as i:`, I don't even need to know Python to figure that out. On the other hand, modifying the Ruby version seems to require intimate knowledge of its syntax.
I think for someone with a basic knowledge of both languages, the Ruby version is more understandable than the Python. It's a combination of basic Ruby features, whereas Python's `match` statement is much more obscure - it isn't really Python at all, it's "a DSL contrived to look like Python [...] but with very different semantics" [0].
I don't particularly love the Ruby code either, though - I think the ideal implementation would be something like:
[0] https://discuss.python.org/t/gauging-sentiment-on-pattern-ma...> Is this what idiomatic Python has become?
What do you mean? Python has always been that way. "Explicit is better than implicit. [..] Readability counts." from the Zen of python.
> By contrast, here's the equivalent Ruby:
Which is awful to read. And of course you could write it similar short in python. But it is not the purpose of a documentation to write short, cryptic code.
> Readability counts
Almost all Python programmers should be familiar with list comprehensions - this should be easy to understand:
Instead the example uses an explicit loop, coupled with the quirks of the `match` statement. This is much less readable IMO: > [Ruby] is awful to readI think for someone with a basic knowledge of Ruby, it's more understandable than the Python. It's a combination of basic Ruby features, nothing advanced.
I don't particularly love Ruby's syntax either, though - I think the ideal implementation would be something like:
> Almost all Python programmers should be familiar with list comprehensions
Being familiar doesn't mean it's readable. They can be useful, but readability is usually not on that list.
> I think for someone with a basic knowledge of Ruby, it's more understandable than the Python.
I know both, and still consider it awful. Readability is not about making it short or being able to decipher it.
That Ruby code is clever and concise, and terrible to read and extend
IMO it's much closer to the ideal way to write the function, which would be something like:
By what definition of "ideal"? You just hid all the complexity in those undefined `convert` and `format` methods.
The original code does the same! All the real work is inside `convert` and `format`, it just adds another layer of complexity on top.
The ideal version has the same behaviour and shows that the extra complexity is unnecessary.
This is a great PEP. Very similar to what I wanted to achieve with my P1819 for C++, back in 2019: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p18...
Could someone explain more why this should be a language feature?
My understanding of template strings is they are like f-strings but don't do the interpolation bit. The name binding is there but the values are not formatted into the string yet. So effectively this provides a "hook" into the stringification of the interpolated values, right?
If so, this seems like a very narrow feature to bake into the language... Personally, I haven't had issues with introducing some abstraction like functions or custom types to do custom interpolation.
It reminds me of javascripts template literals (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...) and .nets FormattableString (https://learn.microsoft.com/en-us/dotnet/api/system.formatta...)
The best use case I know of for these kinds of things is as a way to prevent sql injection. SQL injection is a really annoying attack because the "obvious" way to insert dynamic data into your queries is exactly the wrong way. With a template string you can present a nice API for your sql library where you just pass it "a string" but it can decompose that string into query and arguments for proper parameterization itself without the caller having to think about it.
You can see some motivations for it further down the document. Basically it allows libraries to access the ease of use of f-strings for more than just formatting a string according to what the language allows. Structured logging is one area where I would like to use this.
It's useful for doing things like writing parameterized SQL queries that avoid injection attacks. Or anything that quacks like that proverbial duck, where you are going to emit some plaintext and need to validate it as it is filled.
Because python dominates AI, and python is dominating because of AI. And prompting really, really benefits from f-strings and templated strings. LLMs as a whole means the rise of unstructured data, and flexible string manipulation is really important for handling that.
I like the implementation, but it looks like nobody has pointed out that Python already has a built-in string template system [1]. Granted, it's not quite as simple to use, but I have been using it for a while.
[1] https://docs.python.org/3/library/string.html#template-strin...
edit: this was mentioned by milesrout in https://news.ycombinator.com/item?id=43649607
In the past when I needed this I just made a function that processed the f string, often enough a simple lambda function would do. This looks like additional complexity for not a lot of gain.
You cant process an f-string the same way you can process a t-string. An f-string does not preserve which parts of it are static and dynamic, which you need to know to properly escape user input.
I've been toying with the idea of having arbitrary string types, like
or that the development environment could highlight properly, that would ... I don't know. In the end t and f string don't do anything that a t() and f() function couldn't have done, except they are nice. So it would be nice to have more.IIRC, previous versions of the PEP wanted things to work this way. You might also be interested in the "syntactic macros" PEP (https://peps.python.org/pep-0638/).
This appears similar to Java's String Template preview feature that was withdrawn:
https://openjdk.org/jeps/465
This is probably the best overview of why it was withdrawn:
https://mail.openjdk.org/pipermail/amber-spec-experts/2024-A...
> If a single interpolation is expensive to evaluate, it can be explicitly wrapped in a lambda in the template string literal
I’m having trouble understanding this - Can someone please help out with an example use case for this? It seems like before with an f string we had instant evaluation, now with a t string we control the evaluation, why would we further delay evaluation - Is it just to utilise running a function on a string first (i.e. save a foo = process(bar) line?)
> It seems like before with an f string we had instant evaluation, now with a t string we control the evaluation
You don't completely control the evaluation.
From the PEP:
> Template strings are evaluated eagerly from left to right, just like f-strings. This means that interpolations are evaluated immediately when the template string is processed, not deferred or wrapped in lambdas.
If one of the things you are interpolating is, as a silly example, an invocation of a slow recursive fibonacci function, the template string expression itself (resulting in a Template object) will take a long while to evaluate.
> the template string expression itself (resulting in a Template object) will take a long while to evaluate
Are you saying that calling:
Will immediately run the function, as opposed to when the __str__ is called (or is it because of __repr__?) - Apparent I might just have to sit down with the code and grok it that way, but thanks for helping me understand!Yes. Just like with f-strings, the call to `fib_slow()` is extracted at compile-time, and occurs eagerly at run-time.
What t-strings offer over f-strings is the ability to control how the final string is put together from the calculated results. And the t-string doesn't have `__str__` - you have to explicitly pass it to a named formatting function to get a string. (There is `__repr__, but that's for debugging, of course.) So you can potentially reuse the same Template instance (created from the t-string) multiple times in different contexts to format the same information different ways.
Yes, fib_slow() will be evaluated immediately, just like it would be if you were using an f-string.
Awesome, that makes sense now - thanks all!
So that you can calculate the string in A.py and then use it over in B.py .
This is super exciting but I wish they added a function to render the template to a string rather then having everyone write their own version of basic rendering.
One place we landed with PEP 750 is that Template instances have no natural string rendering.
That is, you must process a Template in some way to get a useful string out the other side. This is why Template.__str__() is spec'd to be the same as Template.__repr__().
If you want to render a Template like an f-string for some reason, the pep750 examples repo contains an implementation of an `f(template: Template) -> str` method: https://github.com/davepeck/pep750-examples/blob/main/pep/fs...
This could be revisited, for instance to add `Template.format()` in the future.
I totally agree with `Template.__str__()` == `Template.__repr__()`, but I would strongly be for adding a `Template.format()`. I expect that over time I'll be passed template strings where I expected normal strings and adding that example `f(template: Template)` everywhere to my code bases is just a lot of repeated code for no real value.
That said, I think this is a great bit of work and I look forward to getting to use it! Thank you!
> and adding that example `f(template: Template)` everywhere to my code bases is just a lot of repeated code for no real value.
I'd agree that it should be somewhere within the library, even if it's just a separate top-level function in the described `string.templatelib`. If it isn't, I'm sure someone will make a PyPI package for it right away.
PEP 501 allowed `.format()` to do this, but discussions on PEP 750 felt that format shouldn't be needed, because when you need a basic format, you should probably just use f-strings. But maybe `__format__` can be added in in a future version of python if this is really wanted.
These templates don’t seem to be semantically aware like Go’s html/template that takes care of mitigating XSS for you, among other things.
Correct. Intended for library authors to do that. A SQL library, for example, could accept a template type and mitigate against SQL injection for you.
> If a single interpolation is expensive to evaluate, it can be explicitly wrapped in a lambda in the template string literal
> https://peps.python.org/pep-0750/#approaches-to-lazy-evaluat...
Hmm, I have a feeling there's a pitfall.
Feels unnecessary...
Can't think of a good reason now on why I would need this rather than just a simple f-string.
Any unsafe string input should normally be sanitized before being added in a template/concatenation, leaving the sanitization in the end doesn't seem like the best approach, but ok.
Yeah, I generally like most of the stuff that goes into Python, but I have to admit I can't really see the usefulness in this (or, at least, enough usefulness to warrant making it standard).
One of the PEP's developers, Lysandros, presented this in our local meetup, so I am passingly familiar with it, but still, I might be missing something.
I guess the crux of it is that I don't understand why it's `t"some string"` instead of `Template("some string")`. What do we gain by the shorthand?
> What do we gain by the shorthand?
Because it's new syntax, it allows for parsing the literal ahead of time and eagerly evaluating the substitutions. Code like
essentially gets translated into That is: subsequent changes to `bar` won't affect the result of evaluating the template, but that evaluation can still apply custom rules.Ah OK, I see what you mean, so they're basically an f-string that hasn't been "baked in" yet, and you can still access all its parameters. That's pretty cool.
Yes, that's also a fine way to describe it IMO.
I guess this could be useful in SQL for escaping values, I saw a project on the comments here.
With templates:
Without templates: So one less argument to pass if we use templates.But yeah it does seem a bit confusing, and maybe kinda not pythonic? Not sure.
SQL templating is one of the explicitly cited motivating use cases.
This could be a game-changer for applications that involve lots of HTML, SQL, or other languages that tend to be embedded within Python programs. Frankly, HTML templating is the worst part of Python webdev DX for me.
Excited to see what libraries and tooling comes out of this.
Agreed, although this doesn't work as well for HTML or SQL that doesn't fit on one line. Since having everything in a string literal is a little messy, and doesn't have the benefit of autoindent, syntax coloring, etc.
This is one place where s-expressions of Lisp make embedding these DSLs syntactically easier.
To borrow the PEP's HTML examples:
You can see how the parentheses syntax will practically scale better, to a larger and more complex mix of HTML and host language expressions. (A multi-line example using the normal text editor autoindent is on "https://docs.racket-lang.org/html-template/".)> Agreed, although this doesn't work as well for HTML or SQL that doesn't fit on one line.
PEP 750 t-strings literals work with python's tripe-quote syntax (and its lesser-used implicit string concat syntax):
My hope is that we'll quickly see the tooling ecosystem catch up and -- just like in JavaScript-land -- support syntax coloring and formatting specific types of content in t-strings, like HTML.Yeah, editor support will help a lot. Though the PEP's way of doing things doesn't specify the language of the template where the literal occurs, so detection of that might have to be kludged.
(Or, in a sufficiently working program, an editor with semantic analysis access could use something like type inference in the Python side, to determine the language in a less-kludgey way.)
> doesn't specify the language of the template where the literal occurs
Yeah, something we spent a bunch of time considering. In the end, we decided it probably needed to stay out of scope for the PEP.
You're right that JavaScript has an easier time here. Most of the JS tools we looked at simply inspect the name of the tag and if it's (say) html, they attempt to color/format string content as HTML regardless of what the html() function actually does or the string's contents.
Currently, tools like black have no knowledge of types. I'm guessing some amount of kludging is to be expected on day one. But my hope is over the long term, we'll see a story emerge for how annotations can indicate the expected content type.
But that will not apply proper HTML escaping to the evil script element, allowing injection, and allows tags not to be closed:
lot_of_html isn't a string literal with dangerous elements smooshed into the trusted scaffolding like an f-string would do.
It's a template instance that still needs to be safely processed into a renderable string, e.g. by escaping whatever `evil` evaluates to and even validating the final HTML syntax.
I can easily end up unsafely processed. It's a footgun.
And why would you be validating HTML on the fly, when it's coming from your program, not as an input into it. Even if you can do it at program startup once for each template, it's still pointless overhead.
The whole thing is wrongheaded; exactly the kind of stove-pipe people end up inventing when they don't have metaprogramming.
> I can easily end up unsafely processed.
No, that isn't how it works. The unprocessed version is not a `str` instance and doesn't implement `__str__`:
> This is because Template instances are intended to be used by template processing code, which may return a string or any other type. There is no canonical way to convert a Template to a string.
If you tried to use the Template directly as if it were a string, you'd get either a TypeError or completely malformed HTML (the `repr` of the Template instance, which would look very different).
>And why would you be validating HTML on the fly
You wouldn't be; you'd be escaping user-generated content that tries to break a page by including HTML markup.
> There is no canonical way to convert a Template to a string.
... but let me assure you it's never the wrong one!
Well, no; the entire point is that there are multiple ways to interpret the Template in context. One possible application is for localization (supposing that the interpolations are objects that can give a localized string on demand, or e.g. integers to be formatted in a locale-aware manner). It should probably be possible to do that with .format anyway, but just as an example. Perhaps someone out there wants to store fragments of SQL within a JSON document, too. The point is to maintain that flexibility, by separating the parsing and formatting steps.
> I can easily end up unsafely processed
I’m curious how?
You don't have to add HTML validation to your processing func, but you brought up invalid syntax up as an issue with string templating.
> The whole thing is wrongheaded; exactly the kind of stove-pipe people end up inventing when they don't have metaprogramming.
Python has many metaprogramming features. I don't think you understand this feature much less its motivation.
How else would you go about adding language support for e.g. HTML and SQL within Python?
On the contrary, I implemented a substantial facsimile of it in a Lisp dialect yesterday; see my comment history.
Unlike template strings, it was done in the language. There already are quasi-strings in the form of `...` syntax. We can quote that syntax (e.g. implicitly as a macro argument) and then pull apart its pieces to construct an object. It should work even in a years-out-of-date installation of the language. No new tooling is required; no changes to syntax highlighting in the editor, nothing.It's a parlor trick that doesn't have any uses. The structured log messages use case is the most promising, because it has a consumer which actually wants the interpolated pieces that it would otherwise have to hackily parse out.
I predict that Python will eventually get dedicated HTML syntax: perhaps something that uses indentation to indicate element nesting. Let's "vibe pseudocode" a sketch:
or whatever.> Unlike template strings, it was done in the language.
This is a disingenuous way to compare a LISP with a language that values syntactic convention and strives to be readable and maintainable by more than one person.
> I predict that Python will eventually get dedicated HTML syntax:
How do you assign this result to a variable? How is this any better than
It's a true statement; I made a usable construct that causes template strings to be turned into objects, rather than to perform the string substitution. I did it without changing any syntax, or have to ship a new version of the language.
Python has to ship a new version for this; there is no way for existing installations to use the code.
I don't have to change anything in my editor setup.
So who is it that values syntactic convention?
t"abc" is a syntax errror in vast numbers of existing Python installations; how can we call it convention?
> How do you assign this result to a variable?
That's to be worked out. The "html:" fantasy construct could have a way to do that. When you think of assigning to a variable, the first thing that comes to mind is "a = b" syntax. But, look, the define construct in Python also assigns to a variable:
The imagined html construct could have arguments like to assign an object to a variable, or just spin out textual HTML into a specified stream. I don't want to get into it, but there are obvious ways it could be done such that it hits all sorts of requirements.> How is it any better
It renders all of the following problems unrepresentable: mismatched angle brackets, mismatched or unclosed tags, bad attribute syntax, injection.
These developer ergonomics reduce injection 90% in my experience.
When concatenating strings is the harder approach, it is really beautiful.
Whats wrong with jinja?
Jinja et al. are fine. But they still forces me to choose Python OR HTML. I have to jump between files and LSPs. t-strings pave the way for true Python+HTML seamless DX.
So does this mean that any place where code exists that looks for `type(someinstance) == str` will break because the type will be `Template` even though `someinstance` could still be used in the following code?
yes. t-strings are not `str`
With the proliferation of string type prefixes will they add one that represents a r#""# real raw string capable of ending with a backslash?
Why should I use this instead of Template class from string
Good addition. Java has done a few previews of this and is still trying to figure out the best way. Maybe the feedback on the Python version will help.
Making t-strings not actual strings but a different type is the genius here.
It is now be a generic expression evaluator and a template rendered!
Is it a replacement for Jinja2 templates? I use them a lot in LLM pipelines (e.g., to fill in the system prompt and provide more context).
If you just need value replacement yes. But you could also already do that with str.format.
I think this gives you slightly more control before interpolating.
If you want control flow inside a template, jinja and friends are probably still useful.
don't see any looping expression functionality in the PEP
No. Not really intended to be a replacement for jinja. But it also depends on how you use jinja. If you use only very basic functionality of jinja, maybe.
For a quick glance, I didn't see the corner case where the old-style `%` interpolation (may) shine over PEP750: building up a JSON document with a dictionary:
>>> hello_world = {"hello":"HELL" ,"world":"O'WORLD"}
>>> json_template='{"hello":"%(hello)s","world":"%(world)s"}'
>>> print(json_template % hello_world)
{"hello":"HELL","world":"O'WORLD"}
Wait, I wasn't supposed to be using f strings in SQL queries?! (don't @ me)
What kind of special string will be added next? We already have f-strings, .format, %s ...
str.format was introduced in 2008. String formatting conventions can reasonably change in the span of 18 years.
Aside from that, the format method and the % operator don't represent separate types. - they're just methods on `str`.
Aside from sanitization, this also allows replication of Ruby’s %W[…] syntax
As a python meh-er, this is actually good design. Everyone is jumping on C that it has no strings, but then other languages throw raw strings at you with some interpolation and call it a day. Also it's 2025 and people will still comment "do we need such a bloated string mechanism" and then watch new devs produce bulks of injectionable strings.
I always think the coming of this feature is inevitable.
Are these less useless for translation than f-strings?
I'm quite disappointed with the pace of development of the Python language. Five years of support for a particular version seems far too short.
I mostly use Python in scientific contexts, and hitting end-of-life after five years means that for a lot project, code needs to transition language versions in the middle of a project. Not to mention the damage to reproducibility. Once something is marked "end of life" it means that future OS versions are going to have a really good reason to say "this code shouldn't even be able to run on our new OS."
Template strings seem OK, but I would give up all new language features in a heartbeat to get a bit of long term support.
> hitting end-of-life after five years means that for a lot project, code needs to transition language versions in the middle of a project.
Your Python interpreter will not spontaneously combust due to being end-of-life. It just eventually won't be able to run new versions of tools; but your existing tool versions should also work fine for a very long time. All you're missing out on is bugfixes, which third parties (such as a Linux distro) are often willing to provide.
When a language like Python doesn't innovate at this rate, eventually people will get annoyed about how primitive it ends up feeling compared to languages that have been around for less than half as long. The devs' time and resources are limited and they've clearly advertised their time preference and committed to a reliable schedule - this is an explicit attempt to accommodate users like you better, compared to the historical attitude of releasing the next minor version "when it's done". It also means that they're locked in to supporting five versions at a time while developing a sixth. There's only so much that can reasonably be expected here.
Seriously, what you're getting here is well above the curve for open-source development.
You can use uv to pin, download, and use a specific version of a Python interpreter on a per project basis. You shouldn't use your OS provided Python interpreter for long running projects for all the reasons you mentioned, including reproducibility. If you insist on using the vendor provided interpreter, then use RHEL (or a free clone) or any other long suppported Linux distro (there are many). Five years is a very long time in technology. It is not reasonable to expect more from core maintainers IMO. Especially considering we have better options.
> Five years is a very long time in technology.
But it's not a long time in the OP's field of science. Unfortunately despite a strong preference for Python in the scientific community, the language's design team seem to ignore that community's needs entirely, in favour of the needs of large technology companies.
I was hopeful that in the transition from a BDFL-based governance system to a Steering Council, we would see a larger variety of experience and opinions designing the language. Instead, I don't think there has ever been a single scientist, finance worker etc on the Steering Council - it's always software developers, almost always employees of large software companies.
You can pay for long term support from private companies.
And your scientific context is a distinct minority for python now. Most new development for python is for data/AI. Considering LLMs get updated every quarter, and depreciated every year, there is no appetite for code that doesn't get updated for 5 years.
Honestly it's very weird to have data/AI be outside of the scientific context! They both used to be a subset of scientific Python.
The code will be updated over five years, but there's no need to be on continual version churn on the underlying language. And frankly I'm surprised that it's tolerated so widely in the community. Trying to run a Node project from 5 years ago is often an exercise in futility, and it will be a big shame when/if that happens to Python.
Thanks but I still use "%s" % (a,) the way I learned a dozen years ago and I'll keep doing it until the day I die.
Good for you.
can't wait to have my linter tell me I should be using t-strings instead of f-strings. apologies for not being patient enough to read through this to find it, but I hope they can remove f-strings:
> There should be one-- and preferably only one --obvious way to do it.
They do different things. You can implement f-strings in t-strings, but it's extra work. The obvious way is therefore:
Use f-strings if you can, otherwise use t-strings.
Just use ChatGPT to change it.
I wonder what fun The Amazing David Beazley thinks PEP 750 t-strings are?
I recently asked him:
--
Hi David! I am a huge long time fan of SWIG and your numerous epic talks on Python.
I remember watching you give a kinda recent talk where you made the point that it’s a great idea to take advantage of the latest features in Python, instead of wasting your time trying to be backwards compatible.
I think you discussed how great f-strings were, which I was originally skeptical about, but you convinced me to change my mind.
I’ve googled around and can’t find that talk any more, so maybe I was confabulating, or it had a weird name, or maybe you’ve just given so many great talks I couldn’t find the needle in the haystack.
What made me want to re-watch and link my cow-orkers to your talk was the recent rolling out of PEP 701: Syntactic formalization of f-strings, which makes f-strings even better!
Oh by the way, do you have any SWIG SWAG? I’d totally proudly wear a SWIG t-shirt!
-Don
--
He replied:
Hi Don,
It was probably the "Fun of Reinvention".
https://www.youtube.com/watch?v=js_0wjzuMfc
If not, all other talks can be found at:
https://www.dabeaz.com/talks.html
As for swag, I got nothing. Sorry!
Cheers, Dave
--
Thank you!
This must be some corollary of rule 34:
https://www.swigwholesale.com/swig-swag
(Don’t worry, sfw!)
-Don
--
The f-strings section starts at 10:24 where he's live coding Python on a tombstone with a dead parrot. But the whole talk is well worth watching, like all his talks!
https://youtu.be/js_0wjzuMfc?t=624
prob dont need jinja anymore then
15 minute implementation in TXR Lisp.
Background: TXR already Lisp has quasi-string-literals, which are template strings that do implicit interpolation when evaluated. They do not produce an object where you can inspect the values and fixed strings and do things with these before the merge.
The underlying syntax behind the `...` notation is the sys:quasi expression. We can quote the quasistring and look at the car (head symbol) and cdr (rest of the list): So that is a bit like f-strings.OK, now with those pieces, I just right now made a macro te that gives us a template object.
You invoke it with one argument as (te <quasistring>) You can see the object captured the values from the lexical variables, and we can rewrite them, like changing Bob to Alice. When we call the merge method on the object, it combines the template and the values.(We cannot alter the strings in this implementation; they are for "informational purposes only").
Here is how the macro expands:
It produces a constructor invocation (new template ...) which specifies values for the slots merge, strings and vals.The initialization of strings is trivial: just a vector of the strings pulled from the quasistring.
The vals slot is initialized by a `(vec ...)` call whose arguments are the expressions from the quasistring. This gets evaluated in the right lexical scope where the macro is expanded. This is how we capture those values.
The most complicated part is the lambda expression that initializes merge. This takes a single argument, which is the self-object, anonymized by a gensym variable for hygiene. It binds the .vals slot of the object to another gensym lexical. Then a genyms local variable is bound for each value, referencing into consecutive elements of the value vector. E.g. #:var-0075 is bound to [#:vals-0074 0], the first value.
The body of the let is a transformed version of the original template, in which the interpolated expressions are replaced by gensyms, which reference the bindings that index into the vector.
The complete implementation in template.tl (referenced by (load "template") in command line 4) is:
We can see an expansion:That Lisp Curse document, though off the mark in general, was right the observation that social problems in languages like Python are just technical problems in Lisp (and often minor ones).
In Python you have to wait for some new PEP to be approved in order to get something that is like f-strings but gives you an object which intercepts the interpolation. Several proposals are tendered and then one is picked, etc. People waste their time producing rejected proposals, and time on all the bureucracy in general.
In Lisp land, oh we have basic template strings already, let's make template objects in 15 minutes. Nobody else has to approve it or like it. It will backport into older versions of the language easily.
P.S.
I was going to have the template object carry a hash of those values that are produced by variables; while coding this, I forgot. If we know that an interpolation is @greeting, we'd like to be access something using the greeting symbol as a key.
(I don't see any of this is as useful, so I don't plan on doing anything more to it. It has no place in Lisp, because for instance, we would not take anything resembling this approach for HTML generation, or anything else.)
Yet another way to do strings in Python, I was more than happy with the original way with tupple parameters.
Aw sweet, a third way to do string interpolation in Python.
I'm really loving this lovecraftian space the "batteries included" and "one obvious way to do it" design philosophy brought us!
I dislike this feature.
The stated use case is to avoid injection attacks. However the primary reason why injection attacks work is that the easiest way to write the code makes it vulnerable to injection attacks. This remains true, and so injection attacks will continue to happen.
Templates offer to improve this by adding interpolations, which are able to do things like escaping. However the code for said interpolations is now located at some distance from the template. You therefore get code that locally looks good, even if it has security mistakes. Instead of one source of error - the developer interpolated - you now have three. The developer forgot to interpolate, the developer chose the wrong interpolation, or the interpolation itself got it wrong. We now have more sources of error, and more action at a distance. Which makes it harder to audit the code for sources of potential error.
This is something I've observed over my life. Developers don't notice the cognitive overhead of all of the abstractions that they have internalized. Therefore over time they add more. This results in code that works "by magic". And serious problems if the magic doesn't quite work in the way that developers are relying on.
Templates are yet another step towards "more magic". With predictable consequences down the road.
I'm not sure I agree with this analysis.
Template.__str__() is equivalent to Template.__repr__(), which is to say that these aren't f-strings in an important sense: you can't get a useful string out of them until you process them in some way.
The expectation is that developers will typically make use of well-established libraries that build on top of t-strings. For instance, developers might grab a package that provides an html() function that accepts Template instances and returns some Element type, which can then be safely converted into a string.
Stepping back, t-strings are a pythonic parallel to JavaScript's tagged template strings. They have many of the same advantages and drawbacks.
In my experience it was precisely when it wasn't abstracted properly that injection attacks became more prevalent, not the other way around.
In PHP, people used to just call mysql_query on a string and all the escaping was done with mysql_escape_string. According to you that nice locality of query construction and sanitization that should've improved security, but my god did it ever not do that.
It was exactly layers of abstractions, moving things far away from the programmer, with prepared statements to ORMs, that meaningfully reduced the number of SQL injection vulnerabilities.
Another example is JavaScript, how many XSS vulnerabilities never happened because of all the shadow dom frameworks? Layers of abstractions like these (JSX,etc) are a major reason we don't see many XSS vulnerabilities nowadays.
> You therefore get code that locally looks good, even if it has security mistakes. Instead of one source of error - the developer interpolated - you now have three.
Could you give examples of this?
> The developer forgot to interpolate
What would this look like? The only way to get dynamic/user input into a template is either through interpolation or concatenation.
Before:
f"..html_str..." + user_provided_str # oops! should have: html_str + sanitize(user_provided_str)
After:
t"...html_template..." + user_provided_str # oops! should have: t"...html_template...{user_provided_str}"
Does this really leave us worse off?
Unless you're referring to something like this:
Before:
html = "love > war" # oops! should have been: html = "love > war"
After:
html = "love > war" # oops! should have been: html = t"love > war"
But then the two scenarios are nearly identical.
> the developer chose the wrong interpolation
What kind of interpolation would be the "wrong interpolation"?
> or the interpolation itself got it wrong.
Isn't that analogous to sanitize(user_provided_str) having a bug?
> Instead of one source of error - the developer interpolated - you now have three. The developer forgot to interpolate, the developer chose the wrong interpolation, or the interpolation itself got it wrong.
The idea is for the interpolation to be provided by the library - just as the library is expected to provide a quoting/escaping/sanitization function today. But now the interpolation function can demand to receive an instance of the new Template type, and raise a `TypeError` if given a pre-formatted string. And that work can perhaps also be rolled into the same interface as the actual querying command. And manually creating a Template instance from a pre-formatted string is difficult and sticks out like a sore thumb (and it would be easy for linters to detect the pattern).
> This is something I've observed over my life. Developers don't notice the cognitive overhead of all of the abstractions that they have internalized. Therefore over time they add more. This results in code that works "by magic". And serious problems if the magic doesn't quite work in the way that developers are relying on.
By this logic, we couldn't have languages like Python at all.
Libraries can enforce only template strings, and properly escape the output. This is already possible in Javascript, and you can completely prevent injection attacks using it. > The developer forgot to interpolate not possible if you enforce only templates
> the developer chose the wrong interpolation Not possible if the library converts from template to interpolation itself
> or the interpolation itself got it wrong Sure, but that would be library code.
It’s kinda like saying += is magic. Yeah the best kind.
That's such a small scope of an addition I don't see the comparison. I suppose overloading arbitrary types with it can sometimes make the actions performed opaque.