Sending Multipart / Form Data with InvokeHTTP in Apache NiFi 1.12

Apache NiFi InvokeHTTP processor is described as:

An HTTP client processor which can interact with a configurable HTTP Endpoint. The destination URL and HTTP Method are configurable. FlowFile attributes are converted to HTTP headers and the FlowFile contents are included as the body of the request (if the HTTP Method is PUT, POST or PATCH).

Before today, it did not support the POST method with multipart/form data. But with the landing of a pull request today, it does.  This feature is now on master, and should be available in the next release (1.12).

You can now send:

• Just form data ( name / value pairs )
• Form data and the FlowFile content ( optionally setting the file name attribute )
• Just the FlowFile content

Since NiFi has processors to handle multipart/form data requests already, we'll be able to test it out completely in NiFi.

Our simple flow

* it is assumed that the reader is familiar with Apache Nifi enough to add processors and configure connections etc.  You may start with a blank canvas, or create a new Program Group to work in.

Describing the flow

The goal of the flow is to generate a FlowFile, and send it to an HandleHttpRequest processor, and then log its contents and send a reply. 

When posting multipart/form data, you generally need to post "parts" which can have general data as name/value pairs.  It is also possible to post a name value pair with a filename set to signify that you are uploading a file.  InvokeHTTP now supports posting an arbitrary number of regular name/value pairs drawn from FlowFile dynamic properties that you add, and one "part" with the FlowFile content.  Setting a filename on the content is optional, and the filename will be the $filename attribute of the FlowFile. 

Add and configure the GenerateFlowFile processor

Add a GenerateFlowFile processor to the canvas, we will use this to ... generate a FlowFile that we will send in the post.

Modify the Run Schedule so that we don't flood the HandleHttpRequest processor

Configure the properties to send a simple bit of content. Be sure to set the Mime Type to text/plain

Add and configure the InvokeHTTP processor

Add an InvokeHTTP processor to the canvas and configure it.

We'll be automatically terminating all the relationships

Configure for the POST method, and set the Remote URL, we will use this later when setting up the HandleHttpRequest processor

This is where we will configure the multipart/form data properties.

• FlowFile Form Data Name: this property sets the "name" part of the name/value pair, for the "part" that will hold the Flowfile content
• Set FlowFile Form Data File Name: if set to true, the FlowFile's $fileName attribute will be set as the File Name for the "part" that will hold the FlowFile content.  This is optional.  You would set this if you where posting to a specific file upload endpoint that required a file name.
• Add Dynamic Properties using the (+)
• The properties must be named in a special way, so that the processor knows they are multipart/form data.
• The name MUST start with "post:form:"
• Everything after "post:form:" will be used as the "name" part of that name/value pair
• The value will be used as the "value" part of that name/value pair.  This supports NiFi Evaluation Language.

Our simple configuration, we do not set the file name, and we post the flowfile and 2 form data parts.

Configure the success relationship of the GenerateFlowFile processor to the InvokeHTTP processor

Add and configure the HandleHttpRequest processor

Add the HandleHttpRequest processor to the canvas and configure it.

Set the port, the allowed path, and configure so that only POST is allowed.  We must also add the HTTP Context Map Service ( click the arrow, take the defaults etc )

Add and configure LogAttributes processor

Add a LogAttributes processor to the canvas and configure it.

Connect the Success relationship of HandleHttpRequest to the LogAttributes processor.  Also Automatically Terminate all LogAttribute relationships.

Add and configure RouteOnAttribute processor

Add a RouteOnAttribute processor to the canvas and configure it.

When HandleHttpRequest handles multipart/form data, it will output a FlowFile for each "part".  So our configuration will result in 3 outputs.  This is all well and good, but it confuses the HttpContext service and HandleHttpResponse, because they won't expect the 2nd and 3rd FlowFiles.  To get past this, we'll add a RouteOnAttribute processor, and only route the "part" FlowFile with the GenerateFlowFile produced content.

The Routing strategy is the name of the new route, I've set it to the attribute name we are checking. The Route to Property name is an Expression Language statement that checks the attribute that holds the part name.  *I ran the flow after LogAttributes to see the output and get the right attribute to check.

Automatically Terminate the unmatched relationship for the RouteOnAttribute processor.

Add and configure the HandleHttpResponse processor

Add the HandleHttpResponse processor to the canvas and configure it.

Set the HTTP Status Code to 200, configure the HTTP Context Map to the existing service.

Automatically Terminate the HandleHttpResponse relationships, and configure the RouteOnAttribute's http.multipart.name relationship to the HandleHttpResponse processor.

Run the flow

Select the 2 HandleHttp* processors, along with the LogAttributes and RouteOnAttribute processors and start them.

Then Select the GenerateFlowfile and InvokeHttpProcessor and start them.

Everything will be running then, and you should see the numbers start to update across the flow.  Remember we are sending a new request every 10 seconds.

At this point, you can with wait a bit, and open the nifi-app.log file, or if you are on a *nix (macOS is a *nix ;) ) tail it.

Review the nifi-app.log

The log is found in the logs directory of your nifi installation.

You will see three outputs per run ( every 10 seconds ), one for each "part".