The road to dkjson 3.0 (dkolf.de)

Introduction

dkjson, my Lua library for encoding and decoding data structures in JSON, will soon be 16 years old. I am documenting both my past decisions and my thoughts for breaking changes in a future version.

Author: David Heiko Kolf, 2026-06-05.

Origin and Goals

In 2010 I was playing text-based online games (multi-user dungeons). Some game clients used for MUDs (as well as my own self-coded client) are using Lua as their script language. A proposed extension was based on JSON and I wanted to be prepared for it. I wanted a JSON library with the following properties:

• It should be possible to use it in sandboxed environments (such as computer games) where not even the default module system is available.
• To achieve that, it should be contained in a single file.
• It cannot contain any dependencies beyond the basic Lua functions available in Lua 5.1.
• It must correctly parse any valid JSON (which could also be parsed by the default JavaScript implementations).
• The output should escape any problematic code points.

In 2010, the Lua libraries for JSON I found did not meet my requirements and so I started writing yet another JSON library.

Widely deployed: VLC media player

It is often hard to know how many people are using a library, but in one program I did notice that my library got quite a wide distribution: The VLC media player uses dkjson for a few web APIs. VLC is a very popular video player and so there is a significant amount of computers where a file named "dkjson.luac" can be found on the hard drive. (Even though most users probably do not use those web functions).

Image: VLC on the Artemis II moon mission (bottom-left corner).

Stable, but not abandoned

Open-Source libraries are often judged by the number of recent releases and the number of contributors. dkjson has very few releases and is only developed by myself. But my library is not abandoned, I just try to keep it as stable as possible. To me "perfect software" would be software that is "completed" and that does not require any changes. This is not possible for every application, but for JSON this should be an achievable goal.

I still make sure to fix any bugs that are known to me and to make sure the library also runs with future versions of Lua. I even fix bugs in the 1.x version, despite doubting that anyone still uses that version — this is just training for myself on supporting long-term-stable versions.

Is JSON a minefield?

In 2016 an article critical about JSON parsers (including mine) was published: Parsing JSON is a Minefield. Despite its age it is still often cited, so now is probably still a good time to document my thoughts on this topic.

When developing dkjson I was following Postel's principle, which is compatible with RFC 8259:

A JSON parser MUST accept all texts that conform to the JSON grammar. A JSON parser MAY accept non-JSON forms or extensions.

But I do admit that this has downsides: A file can appear valid in one context and invalid in another. This can have security implications. Additionally, users can start to depend on non-recommended behavior (see Hyrum's Law).

There are problems with the JSON standard itself:

• The original JSON site does not explicitly mention UTF-16 surrogate pairs, but supporting them is necessary in order to permit all valid (JavaScript) escape sequences. In 2010 when I was starting with dkjson, many libraries were missing them. RFC 8259 mentions surrogate pairs.
• The JSON standard does not define an encoding for the floating point values "infinity" and "NaN". This was often a common fault for encoders.
• The standard does not place limits on the numbers and a user of dkjson reported that they are dealing with at least one API based on JSON which encodes 64-bit numbers — something even the standard JavaScript implementation could not parse without corruption.
• The standard allows duplicated keys, even while discouraging their use. Duplicated keys can be very misleading.

The tests do show that most libraries are able to parse valid JSON. I am not aware of any non-standard encoding that has become expected in the ecosystem. So I disagree that parsing JSON is a minefield — but I can understand the criticism that most parsers (including mine) are bad at validating JSON.

However, I have realized that dkjson does indeed have an avoidable flaw in regards to trailing whitespace and trailing invalid content that follows an otherwise valid JSON encoding. Fixing this would be a breaking change and so I have started designing dkjson 3.0.

{"real_key":"real_value"},"fake_key":"fake_value"}

Breaking changes for dkjson 3.0

End-of-content handling

In order fix the above mentioned handling of the end of valid content I consider to use the start-position parameter as the toggle: If the user does not specify a start position, then the entire string must be decoded or else it is an error. However, this requires a further change even when the user does specify the start position: trailing white space must be consumed before reporting the final position.

r, pos, err = json.decode([=[true ]=], 1)
2.10: pos == 5
3.0: pos == 6

r, pos, err = json.decode[=[{"real_key":"real_value"},"fake_key":"fake_value"}]=]
2.10: r ~= nil, err == nil
3.0: r == nil, err ~= nil

r, pos, err = json.decode([=[{"real_key":"real_value"},"fake_key":"fake_value"}]=], 1)
2.10: r ~= nil, pos == 26, err == nil
3.0: r ~= nil, pos == 26, err == nil -- unchanged

Array detection

Currently I am using a rather complex logic to tell whether a table should be encoded as an object or an array. In hindsight I am not quite happy about it: The encoder should not try to make some best effort to encode an arbitrary data structure, but it can expect that the programmer provides a well-crafted data structure intended for JSON.

So I would change it to treat every object where the Lua length operator returns a length greater than zero as an array, encode all the integer keys from one to the length and just ignore any other keys. Empty tables would handled the same as before (arrays per default, can be changed with the metatable).

json.encode {"a","b",x="y"}
2.10: {"1":"a","2":"b","x":"y"}
3.0: ["a","b"]

Reporting invalid escape codes

Rather than producing invalid Unicode code points in the UTF-8 encoding, a partial encoding of a UTF-16 surrogate pair should result in a decoding error.

str, pos, err = json.decode[=["\uD834"]=]
2.10: str == "\237\160\180", err == nil
3.0: str == nil, err ~= nil --2026-07-10: I changed my mind on this, read explanation below

str, pos, err = json.decode[=["\uGG"]=]
2.10: str == "uGG", err == nil
3.0: str == nil, err ~= nil

str, pos, err = json.decode[=["\Y"]=]
2.10: str == "Y", err == nil
3.0: str == nil, err ~= nil

Version format

The version will be in a proper Semantic Versioning schema:

json.version
2.10: "2.10"
3.0: "3.0.0"

What I will keep

I will still support all Lua versions since 5.1. I will also avoid making too many changes and additions in order to maintain the stability of dkjson. I probably also will not fix the problem with duplicated keys and just continue to follow the example of JSON.parse in web browsers.

json.encode({{a = 1}, {b = 2}}, { indent = true })

2.10:
[{
    "a":1
  },{
    "b":2
  }]

3.0:
[{
  "a":1
},{
  "b":2
}]

json.encode({{a = 1}, {b = 2}}, { indent = "\t" })

[{
	"a":1
},{
	"b":2
}]

Feedback welcome

Besides the above mentioned VLC, I have little knowledge about where and how my library is actually used at all. So I have got some questions and in case you are using dkjson, maybe you could help me with them:

• For what kind of application are you using dkjson?
• Are you using the LPeg-based decoder?
• Do you use partial matching (setting a start position, reading the end position)? Would my proposed change break anything for you?
• Are you setting your custom meta-tables for arrays and objects?
• Are you explicitly deactivating the standard meta-tables?
• Are you decoding input with non-standard features?
• Which options are you using for encoding?

I would be very happy to receive your feedback at david@dkolf.de!