Crate gutenberg_post_parser[−][src]

The Gutenberg post parser

Gutenberg is a new post editor for the WordPress ecosystem. A post has always been HTML, and it continues to be. The difference is that the HTML is now annotated. Like most annotation languages, it is located in comments, like this:

<h1>Famous post</h1>

<!-- wp:component {attributes: "as JSON"} -->
lorem ipsum
<!-- /wp:component -->

The parser analyses a post and generates an Abstract Syntax Tree (AST) of it. The AST is then accessible to many languages through bindings.

Platforms and bindings, aka targets

The parser aims at being used on different platforms, such as: the Web (within multiple browsers), Web applications like Electron, native applications like macOS, iOS, Windows, Linux etc.

Thus, the parser can be compiled as:

A binary,
A static library,
Can be embedded in any Rust projects,
A WebAssembly binary,
An ASM.js module,
A NodeJS native module,
A C header,
A PHP extension,
And soon more.

This project uses Justfile as an alternative to Makefile. Every following command will use just, you might consider to install it. To learn about all the commands, just just --list.

Note: Right now, this project needs rustc nightly to compile the WASM target. This target should switch to stable in a couple of months. Since then, be sure to run the latest nightly version with rustup update nightly.

Binary

To compile the parser to a binary, run:

$ just build-binary
$ ./target/release/gutenberg-post-parser --emit-json tests/fixtures/gutenberg-demo.html

Static library

To compile the parser to a static library, run:

$ just build-library
$ ls target/release/

WebAssembly

To compile the parser to a WebAssembly binary, run:

$ just build-wasm
$ ./bindings/wasm/bin/gutenberg-post-parser --emit-json tests/fixtures/gutenberg-demo.html

If you would like to test directly in your browser, run:

$ just build-wasm
$ just start-wasm-server
$ open localhost:8888

Learn more about the WebAssembly binding.

ASM.js

To compile the parser to an ASM.js module, run:

$ just build-asmjs
$ just start-asmjs-server
$ open localhost:8888

The ASM.js module is slower than the WebAssembly binary, but it is useful for Internet Explorer compatibility, or any browser that does not support WebAssembly. Remember that ASM.js is just a JavaScript file.

Learn more about the ASM.js binding.

NodeJS

To compile the parser to a NodeJS native module, run:

$ just build-nodejs
$ ./bindings/nodejs/bin/gutenberg-post-parser --emit-json tests/fixtures/gutenberg-demo.html

Learn more about the NodeJS binding.

C

To compile the parser to a C header, run:

$ just build-c
$ ./bindings/c/bin/gutenberg-post-parser tests/fixtures/gutenberg-demo.html

PHP

To compile the parser to a PHP extension, run:

$ just build-php
$ ./bindings/php/bin/gutenberg-post-parser --emit-debug tests/fixtures/gutenberg-demo.html

To load the extension, add extension=gutenberg_post_parser in the php.ini file (hint: Run php --ini to locate this configuration file), or run PHP such as php -d extension=gutenberg_post_parser file.php.

Learn more about the PHP binding.

Performance and guarantee

The parser guarantees to never copy the data in memory while analyzing, which makes it fast and memory efficient.

WASM binary

A yet-to-be-official benchmark is used to compare the performance of the actual Javascript parser against the Rust parser compiled as a WASM binary so that it can run in the browser. Here are the results:

file	Javascript parser (ms)	Rust parser as a WASM binary (ms)	speedup
`demo-post.html`	13.167	0.43	× 31
`shortcode-shortcomings.html`	26.784	0.476	× 56
`redesigning-chrome-desktop.html`	75.500	1.811	× 42
`web-at-maximum-fps.html`	88.118	1.334	× 66
`early-adopting-the-future.html`	201.011	3.171	× 63
`pygmalian-raw-html.html`	311.416	2.894	× 108
`moby-dick-parsed.html`	2,466.533	23.62	× 104

The WASM binary of the Rust parser is in average 67 times faster than the actual Javascript implementation.

PHP native extension

Another benchmark has been used to compare the performance of the actual PHP parser against the Rust parser compiled as a PHP native extension. Here are the results:

file	PHP parser (ms)	Rust parser as a PHP native extension (ms)	speedup
`demo-post.html`	14.656	0.019	× 771
`shortcode-shortcomings.html`	38.624	0.022	× 1756
`redesigning-chrome-desktop.html`	109.377	0.090	× 1215
`web-at-maximum-fps.html`	87.345	0.061	× 1432
`early-adopting-the-future.html`	137.056	0.074	× 1852
`pygmalian-raw-html.html`	165.072	0.009	× 18341
`moby-dick-parsed.html`	2,681.936	0.993	× 2701

The PHP extension of the Rust parser is in average 4010 times faster than the actual PHP implementation.

Note that memory limit has been hit very quickly with the PHP parser, while the Rust parser as a PHP native extension has a small memory footprint.

License

The license is a classic BSD-3-Clause:

New BSD License

Copyright © Ivan Enderlin. All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

Neither the name of this project nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS AND CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Modules

ast	The Abstract Syntax Tree (AST), i.e. the output of the parser.
combinators	Additional combinators specifically tailored for this parser.
parser	The Gutenberg post parser.

Macros

fold_into_vector_many0	`fold_into_vector_many0!(I -> IResult<I,O>, R) => I -> IResult<I, R>` is a wrapper around `fold_many0!` specifically designed for vectors.
take_until_terminated	`take_until_terminated(S, C)` is like `take_until` but with a lookahead combinator `C`. It's not similar to `terminated!(take_until(S, peek!(C)))` because it loops over the input until `C` is true.
take_until_terminated_and_consume	`take_until_terminated_and_consume(S, C)` is similar to `take_until_terminated` but it consumes `S`.

Functions

root	The `root` function represents the axiom of the grammar, i.e. the top rule.

Type Definitions

Input	Represent the type of a parser input.
InputElement	Represent the type of a parser input element. See `Input` for more information.