Asynchronous Nunjucks with Filters and Extensions

The Problem

Nunjucks was developed synchronous. The architecture behind is synchronous.

So if you use nunjucks everything is fine until you hit an asynchronous problem. One easy example is:
  • You want to have a method to download a data as json from a server
That is easy to solve, normally. You download the data before creating the nunjucks context and pass the data to nunjucks after download.

The problem arises when you don't know the source of the data in front. An example:

 {% set dataSource = userSelection %}
 {% set data = loadData('https://mydomain/sources/' + dataSource + '.json') %}

The userSelection is e.g. a value from a dropdown on an html page. This can have multiple values.
Now you have to download the data while rendering the nunjucks template, because you don't know the value of the "dataSource" variable upfront.

Nunjucks Filters

Nunjucks Filters are making the language more flexibel. Filters are nothing else then a function. In Nunjucks you can apply a filter with a pipe "|". The value in front of the pipe is passed as the first argument to the Nunjucks filter function. You can also combine multiple filters after each other


{{ "hello" | title }} // => "Hello"
{{ ['apple', 'banana', 'cherry'] | join(",") }} // => "apple,banana,cherry"
{{ "foo" | replace("foo", "bar") | capitalize }} // => "BAR"


How to solve the async problem with Nunjucks Filters

Nunjucks Filter support asynchronous calls. The easiest way is to make an await filter which gets a Promise as an argument. So imagine loadData('https://...') returns a Promise. Now you can do this.
We rename loadData to loadDataAsync so it fits to the most languange syntax


  {{ loadDataAsync('https://...') | await }}

An example on how you can implement the await filter:

  nunjucksEnvironment.addFilter(
    'await',
    (promise, callback) => promise.then(result => callback(null, result)),
    true,
  );
Because asynchronousity is added later to the language, you have to change the renderString function to make the await filter work. You add a third argument which is a callback that returns the rendered template.

nunjucksEnvironment.renderString(
  template,
  nunjucksContext,
  (error, result) => {
    console.log(result);
  }
);


Nunjucks Extensions

Nunjucks Extensions are Nujucks Filters on stereoids. You can really define your own syntax here.
The problem here is that you will not find any documentation on how to implement a nunjucks extension. They advise was to read the nunjucks code 🥴. The sparse nunjucks documentation for creating custom extensions is here.

An example of a nunjucks extension:

{% sum 5,6 %}

Where "sum" is the "Nunjucks Tag" and 5 and 6 are the arguments. Lets implement this extension to see how it works.


function SumExtension() {
  this.tags = ['sum'];

  this.parse = function (parser, nodes) {
    // The first node is the sum node.
    // With nextToken we "jump over" sum node to 5,6
    const sumNode = parser.nextToken();

    // Nujucks parser offers a methode parseSignature which helps to parse arguments of a tag
    // 5 and 6 are now parsed as argumen nodes.
    const argNodes = parser.parseSignature(null, true);

    // We say to the parser that he can move to the end of the line to parse next things.
    parser.advanceAfterBlockEnd(sumNode.value);

    // The CallExtension method on the nodes is a helper which compiles
    // the argument nodes. The second argument is the name of the method to
    // pass the compiled code.
    return new nodes.CallExtension(this, 'run', argNodes, []);
  };

  this.run = function (environment, leftOperand, rightOperand) {
    // We now return a safe string with the sum of the operation.
    return new nunjucks.runtime.SafeString(`${leftOperand + rightOperand}`);
  };
}

const nunjucksEnvironment = nunjucks.configure([], {
  autoescape: false,
  throwOnUndefined: false,
});

nunjucksEnvironment.addExtension('SumExtenstion', new SumExtension());

const template = '{% sum 5,6 %}';
const result = nunjucksEnvironment.renderString(template, {});


Lets make another Extension which has a body component:

{% set name = "Homer" %}
{% uppercase %} Hello {{name}} {%enduppercase%}


function UppercaseExtension() {
  this.tags = ['uppercase'];

  this.parse = function (parser, nodes) {
    // The first node is the uppercase node.
    // With nextToken we "jump over" uppercase to the blockend node
    parser.nextToken();

    // We skip over the %} end-block token
    // If we encounter another symbol then an end-block we fail.
    if (!parser.skip(lexer.TOKEN_BLOCK_END)){
      parser.fail('uppercase: end of block expected');
      return;
    };

    // parse everything between uppercase and enduppercase blocks
    const bodyNode = parser.parseUntilBlocks('enduppercase');

    // move forward after the %} endblock. We are finished.
    parser.advanceAfterBlockEnd();

    // The CallExtension method is a helper which compiles
    // the nodes we pass. The second argument is the name of the method to
    // pass the compiled code.
    return new nodes.CallExtension(this, 'run', undefined, [bodyNode]);
  };

  this.run = function (environment, body) {
    // body is a function and can return the compiled value
    const str = body();
    // We now return a safe string with the sum of the operation.    
    return new nunjucks.runtime.SafeString(str.toUpperCase());
  };
}

const nunjucksEnvironment = nunjucks.configure([], {
  autoescape: false,
  throwOnUndefined: false,
});

nunjucksEnvironment.addExtension('UppercaseExtension', new UppercaseExtension());

const template = `
{% set name = "Baris" %}
{% uppercase %} 
  hello {{name}} 
{% enduppercase %}`;

const result = nunjucksEnvironment.renderString(template, {});

How about a new Extension which works like const. You can set the variable just once.

function ConstExtension() {
  this.tags = ['const'];

  this.parse = function (parser, nodes) {
    // The first node is the const node.
    // With nextToken we "jump over" the const node
    const token = parser.nextToken();

    // parse the signature var = value
    const args = parser.parseSignature(false, true);

    // move after the %} endblock. We are finished
    parser.advanceAfterBlockEnd(token.value);

    // The CallExtension method on the nodes is a helper which compiles
    // the body nodes. The second argument is the name of the method to
    // pass the compiled code.
    return new nodes.CallExtension(this, 'run', args);
  };

  this.run = function (environment, args) {
    // args is an object with key-value pair
    //e.g. { name: "Baris" }
    const varName = Object.keys(args)[0]; // "name"
    const varValue = args[varName]; // "Baris"

    const constVars = environment.ctx['@CONST_VARS'] || [];

    // check if we already added the variable name to our array 
    // that hold the const vars
    if (constVars.includes(varName)) {
      throw Error(`${varName} already declared`);
    }

    // Add variable name to the array so that we know
    // next time it was used.
    constVars.length
      ? environment.ctx['@CONST_VARS'].push(varName)
      : (environment.ctx['@CONST_VARS'] = [varName]);

    // set the nunjucks context so
    // name is now "Baris"
    environment.ctx[varName] = varValue;
  };
}

const nunjucksEnvironment = nunjucks.configure([], {
  autoescape: false,
  throwOnUndefined: false,
});

nunjucksEnvironment.addExtension('ConstExtension', new ConstExtension());

const template = `
{% set surname = "Bikmaz" %}
{% const name = "Baris " + surname %}
hello {{name}} 
{% const name = "Elvis" %} // ===> This line should throw an error.
hello {{name}} `;

const result = nunjucksEnvironment.renderString(template, {});

Asynchronous Extensions

Now lets do something asynchronous. How about showing a text after a delay.

export function DelayExtension() {
  this.tags = ['delay'];

  this.parse = function (parser, nodes) {
    // The first node is the delay node.
    // With nextToken we "jump over" over it to the next node
    const delayNode = parser.nextToken();

    // Nujucks parser offers a methode parseSignature which helps to parse arguments of a tag
    // 2 and the text are now parsed as argument nodes.
    const argNodes = parser.parseSignature(null, true);

    // Go until block-end "%}" node. We are finished.
    parser.advanceAfterBlockEnd(delayNode.value);

    // Note that we call CallExtensionAsync now instead of CallExtension.
    // This will add a callback argument the "runDelay" method
    return new nodes.CallExtensionAsync(this, 'runDelay', argNodes, []);
  };

  // Note the callback function. We must call it to tell nunjucks
  // that the extension has been executed.
  this.runDelay = function (_, secs: number, text: string, callback: () => void) {    
    const milisecs = secs * 1000;

    window.setTimeout(() =>{
      new nunjucks.runtime.SafeString(text);
      callback();
    }, milisecs);
  };
}

const nunjucksEnvironment = nunjucks.configure([], {
  autoescape: false,
  throwOnUndefined: false,
});

nunjucksEnvironment.addExtension('DelayExtension', new DelayExtension());

const template = 'Hello {% delay 2, "World" %}';

// We must use now the renderString method with a 
// callback as we have implemented an async Extension
nunjucksEnvironment.renderString(template, {}, (error, result)=>{
  const appDiv: HTMLElement = document.getElementById('app');
  appDiv.innerHTML = result;
});

Parser Methods

  • advanceAfterBlockEnd(name?: string): Node
    Moves to the next node and checks if it's a block-end token like "%} else it fails. If you have a closing tag like {% endif %} then you don't need to pass a name. If not you need to pass a name.

  • advanceAfterVariableEnd(): Node
    Moves to the next node and checks if it's a variable-end token else it fails.

  • fail(msg: string, lineNr: number, colNr: number)
    Throws an error message of the error on the current node.

  • parseExpression(): Node
    Parses expressions like r=5 or 2+2-3.

  • parseSignature(tolerant?: boolean, noParanthesis?: boolean): Node
    Parses a signature. A signature is a comma seperated expression list like 6, "Hello", 8+1, as="World". The noParanthesis argument tells the parser that the arguments are not within paranthesis. If tolereant is set the method does not throw.

  • parseUntilBlocks(...blockNames: string | string[]): NodeList
    If you have a block expression like {% if(..) %} {%else%} {% endif %} and you are at the block-end of the first if as an example you can call parseUntilBlocks('else', 'endif') to parse all the block between.

  • peekToken(): Node
    Takes the current node but does not move next node.

  • skip(type: string): boolean
    Moves to the next node and returns true if the node is of the given type, e.g. skip(lexer.TOKEN_SYMBOL)

  • skipValue(type: string, val: string): boolean
    Moves to the next node and returns true if the node is of the given type and value, e.g. skipValue('operator', '+')

Conclusion

It is possible to do asyncronous work within nunjucks but especially the Extensions are not really documented.

Comments

Post a Comment

Popular posts from this blog

What is Base, Local & Remote in Git merge

Image
When you use a merge tool for Git and you have conflicts you often see these three versions of a file. This is called the three-way-merge. BASE LOCAL REMOTE So what are these files exactly? Imagine you have a branch. While you're working on the code, someone else has already changed the code in the main branch. Now you'll have a conflict if you merge your code into the main branch. The picture above emphasizes the points that are involved into the merge. GREEN : The common anchestor of the change. This is the BASE. RED : Your changes to the code. This is the REMOTE. ORANGE : The code change of someone else. This is the LOCAL. BLUE : The end result file after the merge. Here a more detailed illustration: Sometimes you will fallback to a two-way merge e.g. in cases where you or the other part deleted a file. Some tools refer them as "ours" or "theirs" ours => LOCAL theirs => REMOTE CONFISUNG NAMES? I personally find the naming confusing because why the h
Read more

Debug Azure Function locally with https on a custom domain

Image
Description If you want to deploy an Azure Function normally this endpoint is publicly available. Therefore you need to secure the connection to this endpoint at least with ssl. Before you deploy you also want to test the connection probably locally. The following steps should help you to setup a https connection to a local Azure Function. 1 STEP: Create a Self Signed Certificate via PowerShell Open Windows Terminal or PowerShell as an Administrator The following script will create a self signed certificate in the "Users" own certificate store with the password 12345 The script will also export the certificate as PFX to C:\server.pfx The domain for the certificate will be "debug-local.com" $cert = New-SelfSignedCertificate -DnsName "debug-local.com", "debug-local.com" -CertStoreLocation "cert:\LocalMachine\My" $myPwd = ConvertTo-SecureString -String "12345" -Force -AsPlainText Get-ChildItem -Path "cert:\localMachine\My
Read more