tools/net/sunrpc/xdrgen/README

xdrgen - Linux Kernel XDR code generator

Introduction
------------

SunRPC programs are typically specified using a language defined by
RFC 4506. In fact, all IETF-published NFS specifications provide a
description of the specified protocol using this language.

Since the 1990's, user space consumers of SunRPC have had access to
a tool that could read such XDR specifications and then generate C
code that implements the RPC portions of that protocol. This tool is
called rpcgen.

This RPC-level code is code that handles input directly from the
network, and thus a high degree of memory safety and sanity checking
is needed to help ensure proper levels of security. Bugs in this
code can have significant impact on security and performance.

However, it is code that is repetitive and tedious to write by hand.

The C code generated by rpcgen makes extensive use of the facilities
of the user space TI-RPC library and libc. Furthermore, the dialect
of the generated code is very traditional K&R C.

The Linux kernel's implementation of SunRPC-based protocols hand-roll
their XDR implementation. There are two main reasons for this:

1. libtirpc (and its predecessors) operate only in user space. The
   kernel's RPC implementation and its API are significantly
   different than libtirpc.

2. rpcgen-generated code is believed to be less efficient than code
   that is hand-written.

These days, gcc and its kin are capable of optimizing code better
than human authors. There are only a few instances where writing
XDR code by hand will make a measurable performance different.

In addition, the current hand-written code in the Linux kernel is
difficult to audit and prove that it implements exactly what is in
the protocol specification.

In order to accrue the benefits of machine-generated XDR code in the
kernel, a tool is needed that will output C code that works against
the kernel's SunRPC implementation rather than libtirpc.

Enter xdrgen.


Dependencies
------------

These dependencies are typically packaged by Linux distributions:

- python3
- python3-lark
- python3-jinja2

These dependencies are available via PyPi:

- pip install 'lark[interegular]'


XDR Specifications
------------------

When adding a new protocol implementation to the kernel, the XDR
specification can be derived by feeding a .txt copy of the RFC to
the script located in tools/net/sunrpc/extract.sh.

   $ extract.sh < rfc0001.txt > new2.x


Operation
---------

Once a .x file is available, use xdrgen to generate source and
header files containing an implementation of XDR encoding and
decoding functions for the specified protocol.

   $ ./xdrgen definitions new2.x > include/linux/sunrpc/xdrgen/new2.h
   $ ./xdrgen declarations new2.x > new2xdr_gen.h

and

   $ ./xdrgen source new2.x > new2xdr_gen.c

The files are ready to use for a server-side protocol implementation,
or may be used as a guide for implementing these routines by hand.

By default, the only comments added to this code are kdoc comments
that appear directly in front of the public per-procedure APIs. For
deeper introspection, specifying the "--annotate" flag will insert
additional comments in the generated code to help readers match the
generated code to specific parts of the XDR specification.

Because the generated code is targeted for the Linux kernel, it
is tagged with a GPLv2-only license.

The xdrgen tool can also provide lexical and syntax checking of
an XDR specification:

   $ ./xdrgen lint xdr/new.x


How It Works
------------

xdrgen does not use machine learning to generate source code. The
translation is entirely deterministic.

RFC 4506 Section 6 contains a BNF grammar of the XDR specification
language. The grammar has been adapted for use by the Python Lark
module.

The xdr.ebnf file in this directory contains the grammar used to
parse XDR specifications. xdrgen configures Lark using the grammar
in xdr.ebnf. Lark parses the target XDR specification using this
grammar, creating a parse tree.

xdrgen then transforms the parse tree into an abstract syntax tree.
This tree is passed to a series of code generators.

The generators are implemented as Python classes residing in the
generators/ directory. Each generator emits code created from Jinja2
templates stored in the templates/ directory.

The source code is generated in the same order in which they appear
in the specification to ensure the generated code compiles. This
conforms with the behavior of rpcgen.

xdrgen assumes that the generated source code is further compiled by
a compiler that can optimize in a number of ways, including:

 - Unused functions are discarded (ie, not added to the executable)

 - Aggressive function inlining removes unnecessary stack frames

 - Single-arm switch statements are replaced by a single conditional
   branch

And so on.


Pragmas
-------

Pragma directives specify exceptions to the normal generation of
encoding and decoding functions. The directives are described
below.

Pragma aggregate
------ ---------

  pragma aggregate <struct> <member> ;

A variable-length array member is normally encoded and decoded by
staging it in a C array and walking that array. When the application
object is not that array -- for instance when the member is a wire
projection of a different in-kernel structure -- building it is
wasted work. This directive marks such a member so that xdrgen emits a
codec that owns only the wire framing (the length prefix, the bound
check, and the per-element codec) and drives the application
through begin/item/end hooks, processing one element at a time with
no staged array. The marked member must be a struct member
declared either as a variable-length array or as an optional-data
list ("type *name"); xdrgen rejects a directive naming anything
else, and generates the codec for the server side only.

The two forms differ only in framing. A variable-length array
carries a u32 element count, so the framing writes that count and
bound-checks it. Its generated decoder hands each element to the
ordinary element decoder from a local variable, so the element
type may not contain a variable-length array or an optional-data
member at any depth; xdrgen rejects a directive whose element type
does. The array must declare a maximum size: the decoder hands the
wire count to the begin hook before it decodes any element, and
only the declared bound limits that count. xdrgen rejects a
directive naming an unbounded array. An optional-data list carries
no count: each element is prefixed by a value-follows TRUE and the
sequence is closed by a FALSE, which the framing writes after the
last element. The element type must therefore be a pointer type,
whose codec owns that framing; xdrgen rejects any other element
type. That form has no count to loop on, so its encoder pulls
elements until the item hook returns false. False ends the list
rather than failing the encode: a producer stops when its source
is exhausted or when a reply budget fills. A producer that stops
for an error records it in its own state and either fails the
encode from the end hook or reports it in the procedure status.
For the optional-data form xdrgen generates a working
encoder only; the decoder it emits fails without consuming the
list, so xdrgen also rejects the directive when the struct is
reachable from an RPC argument.

For example:

  pragma aggregate secattr aclent;
  pragma aggregate secattr dfaclent;

The hooks are supplied by hand. Their names derive from the pragma
header and the containing struct, so that two programs sharing a
type name resolve to distinct symbols. All marked members of one
struct share a hook set, so they share one element type as well;
xdrgen rejects a marked member whose element type differs from
that of a member already marked in the same struct. The cursor's
member_id field selects the member:

  nfs_acl2_secattr_encode_begin()
  nfs_acl2_secattr_encode()
  nfs_acl2_secattr_encode_end()
  nfs_acl2_secattr_decode_begin()
  nfs_acl2_secattr_decode()
  nfs_acl2_secattr_decode_end()

For a variable-length array, a decoder fills the cursor's count from
the wire before calling the begin hook, and an encoder's begin hook
sets that count for the framing to bound-check and emit. An
optional-data list's encoder does not read the count and its
decoder never fills it. Once a begin hook
has succeeded its end hook runs, so that it can release what begin
took; the end hook receives the running success flag.

For each marked member xdrgen emits an enumeration constant -- the
pragma header, struct, and member joined by underscores and
upper-cased -- in the generated header, and the generated framing
initializes the cursor's member_id with it. The hooks compare
member_id against these constants rather than bare integers:

  NFS_ACL2_SECATTR_ACLENT
  NFS_ACL2_SECATTR_DFACLENT

The hooks reach their application state through the xdr_stream's
xdrgen_ctx pointer: svcxdr_init_decode() and svcxdr_init_encode()
bind it to the svc_rqst, and the generated framing copies it into
the cursor's ctx field, from which a hook reaches rq_argp or
rq_resp. xdrgen emits the hook prototypes in the generated header.

Pragma big_endian
------ ----------

  pragma big_endian <enum> ;

For variables that might contain only a small number values, it
is more efficient to avoid the byte-swap when encoding or decoding
on little-endian machines. Such is often the case with error status
codes. For example:

  pragma big_endian nfsstat3;

In this case, when generating an XDR struct or union containing a
field of type "nfsstat3", xdrgen will make the type of that field
"__be32" instead of "enum nfsstat3". XDR unions then switch on the
non-byte-swapped value of that field.

Pragma exclude
------ -------

  pragma exclude <RPC procedure> ;

In some cases, a procedure encoder or decoder function might need
special processing that cannot be automatically generated. The
automatically-generated functions might conflict or interfere with
the hand-rolled function. To avoid editing the generated source code
by hand, a pragma can specify that the procedure's encoder and
decoder functions are not included in the generated header and
source.

For example:

  pragma exclude NFSPROC3_READDIRPLUS;

Excludes the decoder function for the READDIRPLUS argument and the
encoder function for the READDIRPLUS result.

Note that because data item encoder and decoder functions are
defined "static __maybe_unused", subsequent compilation
automatically excludes data item encoder and decoder functions that
are used only by excluded procedure.

Pragma header
------ ------

  pragma header <string> ;

Provide a name to use for the header file. For example:

  pragma header nlm4;

Adds

  #include "nlm4xdr_gen.h"

to the generated source file.

Pragma public
------ ------

  pragma public <XDR data item> ;

Normally XDR encoder and decoder functions are "static". In case an
implementer wants to call these functions from other source code,
s/he can add a public pragma in the input .x file to indicate a set
of functions that should get a prototype in the generated header,
and the function definitions will not be declared static.

For example:

  pragma public nfsstat3;

Adds these prototypes in the generated header:

  bool xdrgen_decode_nfsstat3(struct xdr_stream *xdr, enum nfsstat3 *ptr);
  bool xdrgen_encode_nfsstat3(struct xdr_stream *xdr, enum nfsstat3 value);

And, in the generated source code, both of these functions appear
without the "static __maybe_unused" modifiers.

Pragma pages
------ -----

  pragma pages <type> <member> ;

A variable-length opaque or string member is normally encoded by
copying its content into the Reply buffer. When the member carries
a bulk payload that a server-side read operation has already
deposited in the pages of the Reply buffer, that copy is wasted
work, and it prevents a transport capable of direct data placement
from conveying the payload separately from the rest of the Reply.
This directive marks such a member so that the emitted encoder
inserts the payload pages into the encoding stream by reference and
marks the byte range as a result payload.

The marked member may be a member of a struct or an arm of a union,
and its type may be named directly or reached through a typedef. A
union arm declared directly as a string is the one exception: xdrgen
generates such an arm as a char *, which carries no length field, so
the directive is rejected there and the arm has to be declared
through a typedef. For example:

  pragma pages READ3resok data;
  pragma pages readlinkres data;

The first marks the opaque data member of struct READ3resok; the
second marks the path arm of union readlinkres, where path is
"typedef string path<NFS_MAXPATHLEN>".

Whether the directive drives the encoder or the decoder follows
from how the containing type is used. A server decodes arguments
and encodes results, so the directive on a member of a result type
(such as READ3resok) marks a payload to encode by reference, while
the directive on a member of an argument type (such as the data
member of an NFS WRITE argument) marks a payload to decode in place.

An argument type here is the struct named as an RPC procedure
argument, any aggregate reachable from it, and any typedef alias of
either; a result type is defined the same way. A decoded member
and an encoded one take different C representations, so a marked
type that is reachable from both an argument and a result is
rejected. The directive on a union arm of an argument type is
rejected as well:
the union generator emits only the ordinary copying decoder, so a
page-resident union arm needs generator support that does not yet
exist.

The emitted encoder reads only the marked member's length field to
determine how many octets of page content to insert; the member's
data pointer is not read. The emitted decoder reads the length
prefix and captures the payload octets by reference, leaving them in
the pages of the Receive buffer. A decoded page-resident member is
represented as a struct xdr_buf so that a later stage can hand the
payload to the server's file system without a copy.

The server and client manage payload pages differently, so this
directive applies only to server-side code generation. Client-side
handling of a page-resident member remains future work.


Future Work
-----------

Finish implementing XDR pointer and list types.

Generate client-side procedure functions

Expand the README into a user guide similar to rpcgen(1)

Add more pragma directives:

  * @skip -- do not decode, but rather skip, the specified argument
    field

Extend pragma pages to client-side code generation, and to union
arms of RPC procedure arguments

Enable something like a #include to dynamically insert the content
of other specification files

Build a unit test suite for verifying translation of XDR language
into compilable code

Add a command-line option to insert trace_printk call sites in the
generated source code, for improved (temporary) observability

Generate kernel Rust code as well as C code