PII tokenization for LLMs: tokenization vs masking, with code

PII tokenization replaces each piece of personal data in a prompt with a stand-in token before the request leaves your network, and puts the real value back after the model replies. The provider sees the shape of the record, one customer, one account, one phone number, and never the values. Your users still get a complete answer.

Plenty of AI projects stall at the legal review, when someone asks what leaves the network and the answer is everything: the support ticket, the customer's email, the account number. We build the layer that changes that answer: a small piece of code in the request path that adds single-digit milliseconds to a call. This guide covers the design choices in it, the Python, where an entity recognizer such as Presidio fits, what the tokens mean under GDPR, and the CI test that turns "we tokenize PII" into something a compliance team can check.

PII tokenization vs masking: which keeps the answer useful?

Tokenize when the model has to reason about who is who, and mask only when it does not. Masking replaces every email with [EMAIL] and every number with ****. It is simple, and it makes the answers worse, as soon as a prompt holds more than one of anything.

Say a ticket mentions two customers and two accounts. After masking, the model sees [EMAIL] twice and **** twice. It cannot tell which account belongs to whom, so it cannot write "refund account X for customer Y", because the structure it needed has gone.

Reversible tokenization keeps that structure. Each sensitive value gets its own token, and the same value always gets the same token within a request, so two different emails get two different tokens and the model can still say "this customer, that account".

Two properties make tokenization work in a prompt. It is reversible: a private map from token to value lives for the life of the request, so the answer can be restored. And it is format-preserving: an email token still looks like an email, and a 16-digit account number becomes another 16-digit number, so prompts, parsers and validators downstream keep working and the model treats the field as it would treat the real one.

Is tokenized data still personal data under GDPR?

For you, yes, and for the provider possibly not, which is the argument to bring to legal review. A token map you keep is pseudonymization in GDPR's own terms: the data can no longer be tied to a person without additional information held separately. Pseudonymized data stays personal data for whoever holds that additional information, so tokenizing does not take your own processing out of scope.

The provider's position is different. In EDPS v SRB (C-413/23 P, 4 September 2025), the EU Court of Justice held that pseudonymized data passed to a recipient is not always personal data from that recipient's side, when the recipient cannot reverse the pseudonymization or identify people by other means. A provider that receives user_1ec32386@example.invalid, under a key that changes every request and never leaves your process, is the setup that ruling speaks to.

Describe the layer to your counsel as pseudonymization, and have them read the ruling against your setup. Anonymization is the wrong word for it, because the map you keep can reverse every token.

Which fields should you tokenize?

Decide with whoever owns the risk, from your real data, before writing code. Not every field is personal data, and encoding everything destroys the answer as surely as masking does. Over-redaction is its own failure.

The output is a short list of data classes, such as email, account number and phone, with a rule per class and per jurisdiction. On the telecom marketing platform we are building agents for, subscriber data spans several jurisdictions, and the analytics agent works through a governed data-access layer that masks, aggregates or excludes PII by policy before the agent sees a row. That policy is enforced in the access layer, never in the prompt, because a prompt is a request the model can ignore.

Structured classes such as emails, account numbers and phone numbers can start with pattern rules. Names, addresses and free-text identifiers need entity recognition, covered further down.

How do you tokenize PII before the provider call?

Create one boundary object per request that detects sensitive values, swaps each for a token, remembers the mapping and writes a record of what it encoded.

import hmac, hashlib, re
from dataclasses import dataclass, field
 
# Pattern rules for the fields your risk owner marked sensitive.
# In production, add an NER pass for names and addresses.
PATTERNS = {
    "EMAIL":   re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b"),
    "ACCOUNT": re.compile(r"\b\d{10,16}\b"),
    "PHONE":   re.compile(r"\+?\d[\d -]{8,}\d"),
}
 
@dataclass
class Boundary:
    key: bytes                                   # per-request secret, never logged
    vault: dict = field(default_factory=dict)    # token -> raw value
    seen: dict = field(default_factory=dict)     # raw value -> token
    record: list = field(default_factory=list)   # what was encoded, without the values
 
    def _digest(self, kind, value):
        return hmac.new(self.key, f"{kind}:{value}".encode(), hashlib.sha256).hexdigest()
 
    def _surrogate(self, kind, value):
        h = self._digest(kind, value)
        digits = "".join(str(int(c, 16) % 10) for c in h)
        if kind == "ACCOUNT":
            # Same length, all digits. The "99" prefix marks it as a stand-in.
            return "99" + digits[: len(value) - 2]
        if kind == "EMAIL":
            return f"user_{h[:8]}@example.invalid"
        if kind == "PHONE":
            return "+00 " + digits[:10]
        return f"{kind}_{h[:8].upper()}"
 
    def encode(self, text: str) -> str:
        for kind, pattern in PATTERNS.items():
            def swap(m):
                raw = m.group(0)
                if raw in self.vault:            # already a token, leave it
                    return raw
                if raw not in self.seen:
                    tok = self._surrogate(kind, raw)
                    self.seen[raw] = tok
                    self.vault[tok] = raw
                    self.record.append({"kind": kind, "token": tok})
                return self.seen[raw]
            text = pattern.sub(swap, text)
        return text

Three choices in it are deliberate.

A fresh key for every request#

Tokens come from an HMAC of the value under a random key. The same email gives the same token inside one request, so the model can connect two mentions of one customer. Across requests the key changes, so the provider cannot link tokens between calls and build a profile. The changing key also puts the provider in the position the EDPS v SRB ruling describes.

Tokens that cannot be real#

example.invalid is a reserved domain that never resolves, and +00 is not a country code. If a token ever escapes into an outgoing email or a log, it is visibly a stand-in and not someone else's real address.

A record of tokens, not values#

The per-request record lists the kinds of data encoded and the tokens used, and never the values. You show that record in the review afterwards.

Run on a support ticket:

import os
b = Boundary(key=os.urandom(32))
ticket = ("Customer priya.shah@acme.in says account 4111222233334444 "
          "was charged twice. Call back on +91 98765 43210.")
print(b.encode(ticket))
# Customer user_1ec32386@example.invalid says account 9955708008262310
# was charged twice. Call back on +00 0573126352.

Your tokens will differ, because the key is random, but the model still knows there is one customer, one account and one phone number, and what happened to them. It does not know who they are.

How do you restore the real values after the reply?

Decode with a lookup in the other direction, longest token first so one token never clips another:

    def decode(self, text: str) -> str:
        # Longest tokens first, so one token never clips another.
        for tok in sorted(self.vault, key=len, reverse=True):
            text = text.replace(tok, self.vault[tok])
        return text

If the model replies "Email user_1ec32386@example.invalid to confirm the refund on 9955708008262310", your user sees the real email and account number, using values your own systems already hold.

Models sometimes reformat what they copy, so a phone token can come back with different spacing. Tell the model in the system prompt to copy identifiers exactly. Then, after decoding, look for anything still shaped like a token: example.invalid, a +00 number, a 99-prefixed account. A leftover token means the model changed or invented it. Treat that response as failed and retry it or route it to a person, rather than sending a half-decoded answer.

Where does Presidio fit for names and addresses?

Use an entity recognizer for detection and keep your own tokens for the replacement. Patterns catch emails and account numbers, and they miss names, street addresses and identifiers written into free text. Presidio, the open-source PII toolkit Microsoft published, pairs named-entity recognition with pattern recognizers in its analyzer, and its anonymizer offers replace, redact, mask, hash and encrypt operators, with a decrypt operator that reverses encryption.

Presidio's encrypt and decrypt pair is reversible, and it is not format-preserving: an encrypted name comes back as ciphertext, which a model reads as noise rather than as a person. Feed the analyzer's results into the boundary above, so a detected name gets a name-shaped token from the same per-request map and decodes with everything else. Every detector misses something, so the leakage test below sits behind all of it.

How do you prove no PII reached the model provider?

Write a test that fails the build when a raw value reaches the provider. "We tokenize PII" is a claim, and this test is the evidence. Put a fake provider behind your real code path, capture everything that would have been sent, and assert that none of your known sensitive fixtures appear in it:

import os, json
from boundary import Boundary
 
RAW_FIXTURES = ["priya.shah@acme.in", "4111222233334444", "+91 98765 43210"]
 
class CapturingProvider:
    def __init__(self):
        self.sent = []
    def complete(self, payload: dict) -> str:
        self.sent.append(json.dumps(payload))
        return "ok"
 
def handle_ticket(text, provider):
    b = Boundary(key=os.urandom(32))
    payload = {"messages": [{"role": "user", "content": b.encode(text)}]}
    return b.decode(provider.complete(payload))
 
def test_no_raw_value_reaches_provider():
    provider = CapturingProvider()
    handle_ticket("From priya.shah@acme.in, card 4111222233334444, "
                  "phone +91 98765 43210", provider)
    for body in provider.sent:
        for raw in RAW_FIXTURES:
            assert raw not in body, f"raw value leaked: {raw!r}"

Run it in CI on every pull request, and grow the fixtures. Each time someone finds a format the rules missed, such as an email with a plus sign or an account number with spaces, add it as a fixture before fixing the rule. The suite becomes the list of everything you have promised not to send, and we hand it to a compliance team instead of a policy document.

Test the whole path, not just the boundary class. Leaks rarely start in the tokenizer. They start when someone later adds a helpful field to the prompt, the raw customer record for context or the user's profile in the system message, and it skips the encoder. A test on the real request path catches that on the pull request that adds it.

Where does the tokenization layer sit?

Put it in the request path, directly in front of the provider call, so every call goes through it. A boundary that each feature calls when it remembers to will be skipped by the feature written in a hurry. Keep it fast as well. In our builds the encode and decode step adds single-digit milliseconds, so there is no latency argument for switching it off under load.

Three limits shape the rest of the design:

• It protects the next request. Data already sent to a provider is a separate conversation with that provider.
• Detection misses things. Patterns and entity recognition both need the leakage suite behind them.
• The token map is now the sensitive object. It lives for one request, in your process, and never goes into logs or traces.

Credentials follow the same rule. On Solarpunk, the desktop agent we built, secrets sit in an encrypted vault on the user's machine and are injected into the tool adapter when a tool runs, so they never travel into the prompt. Whatever the model never receives, a prompt injection cannot ask it to repeat.

We build this layer for teams whose AI work is waiting on that legal review. PII redaction and data boundaries covers how an engagement runs, from classifying fields in week one to the leakage suite your compliance team reviews.

Common questions

What is PII tokenization?#

PII tokenization replaces each piece of personal data in a prompt with a stand-in token before the request goes to the model provider, and restores the real values after the reply. The provider sees the structure of the record and never the values, and the user still gets a complete answer.

What is the difference between tokenization and masking?#

Masking replaces every email with the same placeholder, so the model cannot tell which account belongs to which customer, and the original cannot be restored. Reversible tokenization gives each value its own stand-in within a request, so the structure survives and the real values go back in after the reply.

Is tokenized data still personal data under GDPR?#

For the organization holding the token map, yes: it is pseudonymized data, which stays personal data. For a recipient that cannot reverse the tokens or identify people by other means, the EU Court of Justice held in EDPS v SRB (C-413/23 P, 4 September 2025) that it is not always personal data. A fresh key per request that never leaves your process puts the model provider in that second position.

Can Microsoft Presidio do reversible PII tokenization?#

Presidio's anonymizer has an encrypt operator and a decrypt operator that reverses it, so its output is reversible. It is not format-preserving, because an encrypted value comes back as ciphertext. Bigcircle recommends an entity recognizer such as Presidio's analyzer for detection, with format-preserving tokens generated from your own per-request map.

Does PII tokenization slow down the LLM call?#

In Bigcircle's builds the encode and decode step adds single-digit milliseconds, which is small next to the model call itself. Keeping it that fast matters, because a boundary that teams switch off under load protects nothing.

How do you prove no personal data reached the provider?#

Put a fake provider behind the real code path, capture everything that would have been sent, and fail the build if a known sensitive fixture appears in it. Run the test in CI on every pull request, and add a fixture each time someone finds a format the rules missed.

Further reading

• PII redaction and data boundaries. How Bigcircle builds and proves the layer.
• AI agent permissions. What an agent may do once the data it sees is controlled.
• Prompt injection in multi-agent systems. Why what never reaches the model cannot be extracted from it.
• Court of Justice of the EU, EDPS v SRB, C-413/23 P, judgment of 4 September 2025. Pseudonymized data from the recipient's side.
• Presidio documentation. The analyzer's recognizers and the anonymizer's operators, including encrypt and decrypt.