Internet-Draft | Packed CBOR | July 2022 |
Bormann | Expires 25 January 2023 | [Page] |
The Concise Binary Object Representation (CBOR, RFC 8949 == STD 94) is a data format whose design goals include the possibility of extremely small code size, fairly small message size, and extensibility without the need for version negotiation.¶
CBOR does not provide any forms of data compression. CBOR data items, in particular when generated from legacy data models, often allow considerable gains in compactness when applying data compression. While traditional data compression techniques such as DEFLATE (RFC 1951) can work well for CBOR encoded data items, their disadvantage is that the receiver needs to decompress the compressed form to make use of the data.¶
This specification describes Packed CBOR, a simple transformation of a CBOR data item into another CBOR data item that is almost as easy to consume as the original CBOR data item. A separate decompression step is therefore often not required at the receiver.¶
The present version (-07) adds the concept of Tag Equivalence as initially discussed at the CBOR Interim meeting 12 in 2022 and further in PR #6, for discussion before and at IETF 114.¶
This note is to be removed before publishing as an RFC.¶
Status information for this document may be found at https://datatracker.ietf.org/doc/draft-ietf-cbor-packed/.¶
Discussion of this document takes place on the CBOR Working Group mailing list (mailto:cbor@ietf.org), which is archived at https://mailarchive.ietf.org/arch/browse/cbor/. Subscribe at https://www.ietf.org/mailman/listinfo/cbor/.¶
Source for this draft and an issue tracker can be found at https://github.com/cbor-wg/cbor-packed.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 25 January 2023.¶
Copyright (c) 2022 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
The Concise Binary Object Representation (CBOR, [STD94]) is a data format whose design goals include the possibility of extremely small code size, fairly small message size, and extensibility without the need for version negotiation.¶
CBOR does not provide any forms of data compression. CBOR data items, in particular when generated from legacy data models, often allow considerable gains in compactness when applying data compression. While traditional data compression techniques such as DEFLATE [RFC1951] can work well for CBOR encoded data items, their disadvantage is that the receiver needs to decompress the compressed form to make use of the data.¶
This specification describes Packed CBOR, a simple transformation of a CBOR data item into another CBOR data item that is almost as easy to consume as the original CBOR data item. A separate decompression step is therefore often not required at the receiver.¶
This document defines the Packed CBOR format by specifying the transformation from a Packed CBOR data item to the original CBOR data item; it does not define an algorithm for a packer. Different packers can differ in the amount of effort they invest in arriving at a minimal packed form; often, they simply employ the sharing that is natural for a specific application.¶
Packed CBOR can make use of two kinds of optimization:¶
A specific application protocol that employs Packed CBOR might allow both kinds of optimization or limit the representation to item sharing only.¶
Packed CBOR is defined in two parts: Referencing packing tables (Section 2) and setting up packing tables (Section 3).¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
A shared item reference or an argument reference.¶
A reference to a shared item as defined in Section 2.2.¶
A reference that combines a shared argument with a rump item as defined in Section 2.3.¶
Prefix or suffix, used as an argument in an argument reference employing the default function "concatenation".¶
An argument reference that uses a tag for argument, rump, or both, causing the application of a function to reconstruct the data item.¶
The pair of a shared item table and an argument table.¶
The packing tables in effect at the data item under consideration.¶
The result of applying a packed reference in the context of given Packing tables.¶
The definitions of [STD94] apply. Specifically: The term "byte" is used in its now customary sense as a synonym for "octet"; "byte strings" are CBOR data items carrying a sequence of zero or more (binary) bytes, while "text strings" are CBOR data items carrying a sequence of zero or more Unicode code points (more precisely: Unicode scalar values), encoded in UTF-8 [STD63].¶
Where bit arithmetic is explained, this document uses the notation familiar from the programming language C (including C++14's 0bnnn binary literals), except that, in the plain text form of this document, the operator "^" stands for exponentiation, and, in the HTML and PDF versions, subtraction and negation are rendered as a hyphen ("-", as are various dashes).¶
This section describes the packing tables, their structure, and how they are referenced.¶
At any point within a data item making use of Packed CBOR, there is a Current Set of packing tables that applies.¶
There are two packing tables in a Current Set:¶
Without any table setup, these two tables are empty arrays. Table setup can cause these arrays to be non-empty, where the elements are (potentially themselves packed) data items. Each of the tables is indexed by an unsigned integer (starting from 0). Such an index may be derived from information in tags and their content as well as from CBOR simple values.¶
The argument table serves as a common table that can be used for argument references, i.e., for concatenation as well as references involving a function tag.¶
When referencing an argument, a distinction is made between straight and inverted references; if no function tag is involved, a straight reference combines a prefix out of the argument table with the rump data item, and an inverted reference combines a rump data item with a suffix out of the argument table.¶
straight reference | table index |
---|---|
Tag 6(straight rump) | 0 |
Tag 224-255(straight rump) | 0-31 |
Tag 28704-32767(straight rump) | 32-4095 |
Tag 1879052288-2147483647(straight rump) | 4096-268435455 |
inverted reference | table index |
---|---|
Tag 216-223(inverted rump) | 0-7 |
Tag 27647-28671(inverted rump) | 8-1023 |
Tag 1811940352-1879048191(inverted rump) | 1024-67108863 |
Argument data items are referenced by using the reference data items in Table 2 and Table 3.¶
The tag number of the reference indicates a table index (an unsigned integer) leading to the "argument"; the tag content of the reference is the "rump item".¶
When reconstructing the original data item, such a reference is replaced by a data item constructed from the argument data item found in the table (argument, which might need to be recursively unpacked first) and the rump data item (rump, again possibly recursively unpacked).¶
Separate from the tag used as a reference, a tag ("function tag") may be involved to supply a function to be used in resolving the reference. It is crucial not to confuse reference tag and, if present, function tag.¶
A straight reference uses the argument as the provisional left hand side and the rump data item as the right hand side. An inverted reference uses the rump data item as the provisional left hand side and the argument as the right hand side.¶
In both cases, the provisional left hand side is examined. If it is a tag ("function tag"), it is "unwrapped": The function tag's tag number is established as the function to be applied, and the tag content is kept as the unwrapped left hand side. If the provisional left hand side is not a tag, it is kept as the unwrapped left hand side, and the function to be applied is concatenation, as defined below. The right hand side is taken as is as the unwrapped right hand side.¶
If a function tag was given, the reference is replaced by the result of applying the unpacking function to be computed to the left and right hand sides. The unpacking function is defined by the definition of the tag number supplied. If that definition does not define an unpacking function, the result of the unpacking is not valid.¶
If no function tag was given, the reference is replaced by the left hand side "concatenated" with the right hand side, where concatenation is defined as in Section 2.4.¶
As a contrived (but short) example, if the argument table is
["foobar", h'666f6f62', "fo"]
, each of the following straight (prefix)
references will unpack to "foobart"
: 6("t")
, 225("art")
,
226("obart")
(the byte string h'666f6f62' == 'foob' is concatenated
into a text string, and the last example is not an optimization).¶
Note that table index 0 of the argument table can be referenced both with tag 6 and tag 224, however tag 6 with an integer content is used for shared item references (see Table 1), so to combine index 0 with an integer rump, tag 224 needs to be used.¶
Taking into account the encoding and ignoring the less optimal tag 224, there is one single-byte straight (prefix) reference, 31 (25-20) two-byte references, 4064 (212-25) three-byte references, and 26843160 (228-212) five-byte references for straight references. 268435455 (228) is an artificial limit, but should be high enough that there, again, is no practical limit to how many prefix items might be used in a Packed CBOR item. The numbers for inverted (suffix) references are one quarter of those, except that there is no single-byte reference and 8 two-byte references.¶
The concatenation function is defined as follows:¶
This specification uses up a large number of Simple Values and Tags, in particular one of the rare one-byte tags and two thirds of the one-byte simple values. Since the objective is compression, this is warranted only based on a consensus that this specific format could be useful for a wide area of applications, while maintaining reasonable simplicity in particular at the side of the consumer.¶
A maliciously crafted Packed CBOR data item might contain a reference loop. A consumer/decompressor MUST protect against that.¶
The packing references described in Section 2 assume that packing tables have been set up.¶
By default, both tables are empty (zero-length arrays).¶
Table setup can happen in one of two ways:¶
By one or more tags enclosing the packed content. Each tag is usually defined to build an augmented table by adding to the packing tables that already apply to the tag, and to apply the resulting augmented table when unpacking the tag content. Usually, the semantics of the tag will be to prepend items to one or more of the tables. (The specific behavior of any such tag, in the presence of a table applying to it, needs to be carefully specified.)¶
Note that it may be useful to leave a particular efficiency tier alone and only prepend to a higher tier; e.g., a tag could insert shared items at table index 16 and shift anything that was already there further down in the array while leaving index 0 to 15 alone. Explicit additions by tag can combine with application-environment supplied tables that apply to the entire CBOR data item.¶
Packed item references in the newly constructed (low-numbered) parts of the table are usually interpreted in the number space of that table (which includes the, now higher-numbered, inherited parts), while references in any existing, inherited (higher-numbered) part continue to use the (more limited) number space of the inherited table.¶
For table setup, the present specification only defines a single tag, which operates by prepending to the (by default empty) tables.¶
A predefined tag for packing table setup is defined in CDDL [RFC8610] as in Figure 1:¶
(This assumes the allocation of tag number 113 ('q') for this tag.)¶
The arrays given as the first and second element of the content of the tag 113 are prepended to the tables for shared items and arguments that apply to the entire tag (by default empty tables). As discussed in the introduction to this section, references in the supplied new arrays use the new number space (where inherited items are shifted by the new items given), while the inherited items themselves use the inherited number space (so their semantics do not change by the mere action of inheritance).¶
The original CBOR data item can be reconstructed by recursively replacing shared and argument references encountered in the rump by their expansions.¶
In Section 5.3.2 of [STD94], the validity of tags is defined in terms of type and value of their tag content. The CBOR Tag registry [IANA.cbor-tags] Section 9.2 of [STD94] allows recording the "data item" for a registered tag, which is usually an abbreviated description of the top-level data type allowed for the tag content.¶
In other words, in the registry, the validity of a tag of a given tag number is described in terms of the top-level structure of the data carried in the tag content. The description of a tag might add further constraints for the data item. But in any case, a tag definition can only specify validity based on the structure of its tag content.¶
In Packed CBOR, a reference tag might be "standing in" for the actual tag content of an outer tag, or for a structural component of that. In this case, the formal structure of the outer tag's content before unpacking usually no longer fulfills the validity conditions of the outer tag.¶
The underlying problem is not unique to Packed CBOR. For instance, [RFC8746] describes tags 64..87 that "stand in" for CBOR arrays (the native form of which has major type 4). For the other tags defined in this specification, which require some array structure of the tag content, a footnote was added:¶
[...] The second element of the outer array in the data item is a native CBOR array (major type 4) or Typed Array (one of tag 64..87)¶
The top-down approach to handle the "rendezvous" between the outer and inner tags does not support extensibility: any further Typed Array tags being defined do not inherit the exception granted to tag number 64..87; they would need to formally update all existing tag definitions that can accept typed arrays or be of limited use with these existing tags.¶
Instead, the tag validity mechanism needs to be extended by a bottom-up component: A tag definition needs to be able to declare that the tag can "stand in" for, (is, in terms of tag validity, equivalent to) some structure.¶
E.g., tag 64..87 could have declared their equivalence to the CBOR major type 4 arrays they stand in for.¶
A tag definition MAY declare Tag Equivalence to some existing structure for the tag, under some conditions defined by the new tag definition. This, in effect, extends all existing tag definitions that accept the named structure to accept the newly defined tag under the conditions given for the Tag Equivalence.¶
A number of limitations apply to Tag Equivalence, which therefore should be applied deliberately and sparingly:¶
In the registry "CBOR Simple Values" [IANA.cbor-simple-values], IANA is requested to allocate the simple values defined in Table 5.¶
Value | Semantics | Reference |
---|---|---|
0-15 | Packed CBOR: shared | draft-ietf-cbor-packed |
The security considerations of [STD94] apply.¶
Loops in the Packed CBOR can be used as a denial of service attack, see Section 2.5.¶
As the unpacking is deterministic, packed forms can be used as signing inputs. (Note that if external dictionaries are added to cbor-packed, this requires additional consideration.)¶
The (JSON-compatible) CBOR data structure depicted in Figure 2, 400 bytes of binary CBOR, could lead to a packed CBOR data item depicted in Figure 3, ~309 bytes. Note that this particular example does not lend itself to prefix compression.¶
The (JSON-compatible) CBOR data structure below has been packed with shared item and (partial) prefix compression only.¶
CBOR packing was originally invented with the rest of CBOR, but did not make it into [RFC7049], the predecessor of [STD94]. Various attempts to come up with a specification over the years didn't proceed. In 2017, Sebastian Käbisch proposed investigating compact representations of W3C Thing Descriptions, which prompted the author to come up with what turned into the present design.¶