dotifi: Generating DOT files from the Apache Nifi canvas

Something I really enjoy is when someone who is using one of the open source projects that I contribute to  has a cool idea or need that inspires me.  Sometimes people have needs or ideas and I may think, well that would be cool, but I have no idea how you would do that.  But other times, they get you thinking and you do have some idea, and that can lead to some fun things.

This happened recently with a thread on the Apache NiFi Slack, where someone posed a simple question:

That is pretty interesting.  How to document a Nifi flow?

That really got me thinking, and brought something to mind.  https://diagrams.mingrammer.com.

Diagrams lets you generate pretty stylish diagrams in python code.  I had recently discovered it and used it for a couple of diagrams to use in a presentation and liked it a lot.

Along with NiPyApi, the idea would be to traverse a NiFi Canvas and generate a diagram from that.

That is a pretty cool idea, and diagrams is definitely pretty neat.. but diagrams is not 'universal'.  In other words that tool, if it existed would generate diagrams, but what if you needed more flexibility?

Then, the idea changed : 

By Plant UML files, I was actually saying DOT files.  A standard for generating graphs in multiple formats, used by a great many tools.

Now this was an idea.  What if we had a tool that could generate both a DOT file definition of a directed graph from NiFi and produce an image?  What if those DOT files could be used in other standard tools?

That may be useful to someone ( even if I personally don't have use for it ).

Another bonus is that while I have enough knowledge of Python to work on existing code... I hadn't ever written any significant Python code from scratch or published a package as I had say published jars in Apache projects or with my palindromicity github org.  So here is the second bonus.  An interesting project that gives me a chance to learn a ton about Python the languages, the tools, and the packaging and distribution side of it.

So I wrote dotifi, which is now available on pypi.

dotifi is a utility to generate DOT files and images based on graphviz from the canvas of an Apache NiFi instance.

dotifi supports:

• Connecting to Apache NiFi using nipyapi in the background and generating a graph from the canvas items
• Process Groups
• Remote Process Groups
• Input and Output Ports
• Processors
• Connections
• Creating DOT files using graphviz and pygraphviz
• Creating a graphic file in Graphviz supported formats and using supported generators
• Starting with an existing dot file as opposed to building from a live NiFi instance, so that you can generate a dot file, then run dotifi multiple times over it to generate multiple outputs without having to have NiFi available.
• Starting from a specific process group, such that the output is scoped to that group
• Controlling the depth of recursion
• Supplying DOT files as templates for configuring the graph, edge and node attributes for:
• the root graph ( and all children)
• Any process group by ID
• Supplying name value pairs ( in yaml configuration file) to override and set node attributes
• Processors by id
• Remote Process Groups by id
• Commandline arguments and yaml file configuration ( overloading supported )
• Configuration file supports environmental variables

A quick example, running dotifi against a NiFi instance with some nested program groups, with the default  formatting and parameters:

DOT file:

sample-output/nifi-canvas.dot

PNG:

graphviz-visual-editor:

This is not the prettiest graph ever, but you can override the attributes of the graph objects in the configuration file, and come up with customized output.

dotifi is new, but hopefully useful.  I look forward to improving it with some ideas I have, and hopefully with feedback from someone who uses it :)

https://pypi.org/project/dotifi/

pip install dotifi