Erlang-RED – A Node-RED backend coded in Erlang
An experiment to replace Node-REDs existing NodeJS backend with an Erlang equivalent that is 100% compatible[1] to existing flow code.
The goal is bring the advantages of low-code visual flow-based programming to a programming language that is designed for message passing and concurrency from the ground up, hence Erlang. More details described in the corresponding blog post.
[1] = 100% won’t be possible since function nodes that are coded in Javascript aren’t supported (or aren’t intended to be supported – unless someone has a workaround)
Why?
Node-RED is a amazing[*] tool for creating flows that describe concurrent processing, it is just a shame the NodeJS is single threaded. So why not use something that is multi-process from the ground up? Concurrency is guaranteed and included.
Also Erlang isn’t the most understandable of programming language – unless one has fallen into in a cauldron of Prolog, spiced with Lisp.
So won’t it be great to have the simplicity of low-code visual flow based programming and the performance (and concurrency) of Erlang?
[*] = amazing in the sense of extendability (Nodes can easily be added and developed by third parties), understandability (Node-RED terminology is straightforward) and usability (once a rectangle becomes a unit of computation and a line becomes a pathway for data flow)
Development Strategy
My development process is best described as flow driven development based around a set of test flows to ensure that node functionality is implemented correctly – meaning that it matches the existing Node-RED functionality.
Test flows are mirrored in a separate repository for better maintainability and also integration with existing Node-RED installations.
Architecture
Erlang architecture is best described by describing various use cases:
Deploying flows to Erlang-Red. Explains the start up process and how Erlang processes are started for nodes.
Workings of a supervisor node supervising a function node.
The challenges of function node which must support timeouts, sub-processes and being supervised by a supervisor.
Inner workings of link nodes and how to deal with dynamic link calls.
Supported Nodes & Features
This is a non-complete list of nodes that partially or completely work:
Node
Comment
Example Flow
catch
catches exception of selected nodes and of entire flows but not groups
Flow
change
supports many operators but not all. JSONata in basic form is also supported.
Flow
complete
is available and can be used on certain nodes, not all
Flow
csv
initial RFC4180 decoder working, supports only comma separator
Flow
debug
only debugs the entire message, individal msg properties aren’t supported. msg count as status is supported.
Flow
delay
supported static delay not dynamic delay set via msg.delay
Flow
event handler
Erlang-Red node for the Erlang gen_event behaviour. Supports both dynamic and static configuration of the event handler.
Flow
exec
executing and killing commands is supported but only for commands in spawn mode and set on the node. Appending arguments to commands isn’t supported. Timeouts are supported. Kill messages are also supported.
Flow
file in
working for files located in /priv
Flow
function
working for any Erlang. Stop and start also respected. Timeout and more than one output port isn’t supported.
Flow
http in
working for GET and POST, not available for PUT,DELETE etc
Flow
http request
basic support for doing rrequests, anything complex probably won’t work
Flow
http response
working
Flow
inject
working for most types except for flow, global …
Flow
join
manual arrays of count X is working, parts isn’t supported
Flow
json
working
Flow
junction
working
Flow
link call
working – dynamic & static calls and timeout is respected
Flow
link in
working
Flow
link out
working
Flow
markdown
working and supports whatever earmark supports.
Flow
module
Erlang module for defining Erlang modules that can be used with the function, event handler and statemachine nodes.
Flow
mqtt in
should be working
Flow
mqtt out
should be working
Flow
noop
doing nothing is very much supported
Flow
split
splitting arrays into individual messages is supported, string, buffers and objects aren’t.
Flow
statemachine
Implements the gen_statem behaviour. Requires a module node to define the actions of the statemachine.
Flow
status
working
Flow
supervisor
Erlang-only node that implements the supervisor behaviour. Supports supervising supervisors and ordering of processes (i.e. nodes) to ensure correct restart and shutdown sequences.
Flow
switch
most operators work along with basic JSONata expressions
Flow
template
mustache templating is working but parsing into JSON or YAML isn’t supported
Flow
trigger
the default settings should work
Flow
Contexts are not supported, so there is no setting things on flow, node or global.
JSONata has been partially implemented by the Erlang JSONata Parser.
Elixir & Erlang-RED
Elixir helpers can be added to erlang-red-elixir-helpers repository.
There is nothing stopping anyone from creating a complete node in Elixir provided there is a Erlang “node-wrapper”, i.e., a bit of Erlang code in the src/nodes directory that references the Elixir node.
The initial example markdown node is an Erlang node that references Elixir code. I also wrote an Elixir wrapper function whereby I could have just as easily referenced Earmark directly from the Erlang code. That was a stylist choice.
I intend to use Elixir code for importing Elixir libraries to the project and less coding nodes in Elixir. I simply prefer Erlang syntax. But each to their own 🙂
Build
$ rebar3 get-deps && rebar3 compile
Test
Development
$ rebar3 shell –apps erlang_red
Open the Node-RED visual flow editor in a browser:
$ open -a Firefox http://localhost:9090/node-red
Docker
I use docker to develop this so for me, the following works:
prompt$ git clone [email protected]:gorenje/erlang-red.git
prompt$ cd erlang-red
prompt$ docker run -it -v $(pwd)/erlang-red:/code -p 9090:8080 -w /code –rm erlang bash
docker> rebar3 shell –apps erlang_red
Then from the docker host machine, open a browser:
prompt$ open -a Firefox http://localhost:9090/node-red
That should display the Node-RED visual editor.
Release
A release can be bundled together:
$ rebar3 as prod release -n erlang_red
All static frontend code (for the Node-RED flow editor) and the test flow files in priv/testflows are bundled into the release.
Cowboy server will started on port 8080 unless the PORT env variable is set.
Fly.io Deployment
A sample Dockerfile Dockerfile.fly is provided to allow for easy launching of an instance as a fly application.
The provided shell script (fly_er.sh) sets some common expected parameters for the launch.
Advanced users may wish to examine the fly launch line therein and adjust for their requirements.
Heroku Deployment
Using the container stack at heroku, deployment becomes a git push heroku after the usual heroku setup:
heroku login –> heroku git:remote -a –> heroku stack:set container –> git push heroku
However the Dockerfile.heroku does not start the flow editor, the image is designed to run a set of flows, in this case (at time of writing) a simple website with a single page.
Basically this flow is the red-erik.org site.
The image does this by setting the following ENV variables:
COMPUTEFLOW=499288ab4007ac6a – flow to be used. This can also be a comma separated list of flows that are all started.
DISABLE_FLOWEDITOR=YES – any value will do, if set the flow editor is disabled.
Also be aware that Erlang-RED supports a PORT env variable to specifying the port upon which Cowboy will listen on for connections. The default is 8080.
Heroku uses this to specify the port to connect for a docker image so that its load balancer can get it right.
Example
What the gif shows is executing a simple flow using Erlang as a backend. The flow demonstrates the difference in the switch node of ‘check all’ or ‘stop at first match’.
All nodes are are processes- that is shown on the left in the terminal window.
This example is extremely trivial but it does lay the groundwork for expansion.
Testing
To create unit tests for this, Node-RED frontend has been extended with a
“Create Test Case” button on the export dialog:
Test flows are stored in the testflows directory and will be picked up the next time make eunit-test is called. In this way it is possible to create unit tests visually.
Flow tests can also be tested within the flow editor, for more details see below.
The flow test suite is now maintained in a separate repository but is duplicated here.
Assert Nodes
To better support testing of flows, two new nodes have been created:
“Assert Failed” node cases unit tests to fail if a message reaches it, regardless of any message values. It’s basically the same as a assert(false) call. The intention is to ensure that specific parts of a flow aren’t reached.
The second node (in green) is an equivalent to a change node except it contains test on attributes of the message object. Possible tests include ‘equal’, ‘match’, ‘unset’ and the respective inverses. Here the intention is that a message passes through is tested for specific values else the unit test fails.
These nodes are necessary since there is no other way to test whether flow is working or not.
Also remember these flow tests are designed to ensure the Erlang backend is correctly implementing node functionality. The purpose of these nodes is not to ensure that a flow is correct, rather that the functionality of implemented nodes works and continues to work correctly.
Visual Unit Testing
My plan is to create test flows that represent specific NodeRED functionality that needs to be implemented by Erlang-RED. This provides regression testing and todos for the implementation.
I created a keyboard shortcut for creating and storing these test flows from the flow editor directly. However I was still use the terminal to execute tests make eunit-test – which became painful. So instead I pulled this testing into Node-RED, as the gif demonstrates:
What the gif shows is my list of unit tests, which at the press of a button, can all be tested. Notifications for each test shows the result. In addition, the tree list shows which tests failed/succeed (red ‘x’ or green check). Also tests can be executed individually so that failures can be checked individually.
The best bit though is that all errors are pushed to the debug panel and from there I get directly to the node causing the error. Visual unit testing is completely integrated into Erlang-RED.
My intention is to create many small flows that represent functionality that needs to be implemented by Erlang-RED. These unit tests shows the compatibility to Node-RED and less the correctness of the Erlang code.
Contributing
Contributions very much welcome in the form of Erlang code or as Node-RED test-flows, ideally with the Erlang implementation. Elixir code is also welcome, only it has its own home.
Each test flow should test exactly one feature and use the assert nodes to check correctness of expected results. Tests can also be pending to indicate that the corresponding Erlang functionality is still missing.
Sibling Repos
An overview of the sibling projects for both the reader and me:
Unit test flow suite provides visual unit tests that verify the functionality being implemented here is the same as in Node-RED. Those test flows are designed to be executed in both Node-RED and Erlang-RED. FlowHub.org maintains the repository and is used to synchronise flow tests between Erlang-RED and Node-RED. These tests can also be used for other projects that aim to replicate Node-RED functionality in an alternative programming language.
Node-RED and Erlang-RED unit testing nodes are used to define and automatically ensure the correct functionality. These nodes are embedded in test flows and ensure that test flows are correct. This makes testing repeatable and reliable and fast! As an aside, these nodes are maintained in an Node-RED flow.
JSONata support for Erlang-RED is implemented by an Erlang parser with a grammer that covers most of JSONata syntax, no guarantees made. Support of JSONata functionality is limited to what the test flows require. Nothing prevents others from extending the functionality themselves, it is not a priority of mine.
Elixir helper library allows Elixir code to be also part of Erlang-RED. Erlang-RED is not intended to be a pure Erlang project, it is intended to be a pure BEAM project. Anything that compiles down to the BEAM VM, why not include it?
Supervisor nodes and other Erlang behaviours as Node-RED nodes. Node package includes gen_statem and gen_event as nodes that can be used with Erlang-Red flows. These nodes can also be installed into Node-RED but there they do nothing.
FAQs
Questions and Answers at either the Erlang Forum or the Node-RED Forum.
Also for more details, there was also a discussion on Hacker News.
Branch Technology
To branch or not to branch, that isn’t really a question. I’m currently working directly on main but ensure that all tests succeed before pushing, so main branch will always work. Locally I work with branches but have no desire to make those branches public since I’m working on my own.
Versioning is completely random and has little or no meaning at the moment. I prefer to use Milestones since these are arrived at and are not planned, I don’t know when the next milestone will be reached.
If this project becomes more collaborative or a “production ready piece of software”, more certainty will be applied to the development process, i.e., semantic version numbers will introduced.
I’m more than happy to deal with conflicts if someone developed something on a branch and it doesn’t merge – I understand that multiple direct pushes to main everyday isn’t the done thing but I don’t like to have code lying around for weeks on end, not being merged because it’s not on a release schedule.
Coding is a creative process, creativity cannot be planned. Imagine Van Gogh working according to a release plan.
Acknowledgement
Nick and Dave for bring Node-RED to live – amazing quality and flexibility and the entire Node-RED community.
Much thanks to
@mwmiller for providing a fly server for running a live version of Erlang-RED,
@joaohf many tips on coding Erlang and structuring an Erlang project, and
@Maria-12648430 for debugging my initial attempt to create a gen_server for nodes.
Disclaimer
No Artificial Intelligence was harmed in the creation of this codebase. This codebase is old skool search engine (ddg), stackoverflow, blog posts and RTFM technology.
Also be aware that this project partly uses the Don’t do Evil un-enforceable license. The point of the license is not to be enforceable but to make the reader think about what is evil. After all, Pope Leo (the new one) did say “evil will not prevail” – what does that even mean?
