Camel Route if Empty Try Again

Both Camel and NiFi are Apache projects. Their lawmaking is mostly written in Java and both are targeting data processing and integration. However, there are as well many differences. One deviation is that NiFi is a platform and Camel a framework.

For NiFi it means that information technology's a software solution where you centrally build dataflows. The concept of dataflows let you lot take multiple processors to process data. Together processors form a dataflow. NiFi has around 200 processors, most of which are congenital-in.

Camel is mostly used at a code level. On the top of the framework companies have build platforms like for example Talend ESB or Red Chapeau Fuse. Nonetheless, y'all can just as easily utilize it in your own application code or build an API, an integration or a microservice.

Camel supports all kind of integration patterns and components. A developer tin can take the core-engine of the framework and can add more than than 300 components to it. The components and patterns together form a route.

Combining superpowers

Both projects provide a lot of powerful concepts, patterns and processors. Between the 200 processors and 300 components there is a quite some overlap. Still, no software in the globe back up all libraries, protocols and technologies. What if these superpowers could exist combined?

There is currently no NiFi component in Camel and no Camel processor in NiFi. The difficulty is that both accept implemented lots of protocols, but they don't provide 1 for external parties. It's similar having a gasoline and an electrical engine. They can piece of work together in all kind of hybrid means, but information technology's not like shooting fish in a barrel to combine them.

The example

Let's explore a hybrid solution for NiFi and Camel. As an instance we apply a very simple, hello world-like, use case:

Moving files betwixt directories.

Every bit source directory we employ C:\in and equally destination C:\out

How would one create a pure NiFi solution? Well, just use the GetFile and PutFile processor:

And how would this work in Camel? This can be done by using the Camel DSL:

          from(file://C:/in).to(file://C:/out);        

Both provide a simple and sufficient solution. Nobody would complicate things by using multiple technologies. But to go along things simple this is exactly what we will do 🙂

Keep in mind that in that location are many more than complex situations where it makes sense to utilize both. Nosotros'll come back to that later. First we volition create a demo to combine Camel and NiFi in i solution. We practise this on a software (tooling) and a code level.

I solution on software level

We'll continue with our simple example of moving files between ii directories. To investigate a solution on a software level we shall not code. Only software components are being used.

In NiFi the normal approach is creating flow with the user interface. Equally we don't want to lawmaking our Camel Route likewise, we use Assimbly Gateway to configure the route in a browser. Assimbly allows to create connections with Camel and ActiveMQ.

Next step is to find a matching protocol to connect both technologies. A good candidate is to employ JMS. Both are well-supported by both Camel and NiFi. Here is the combined period:

Let's bank check the JMS example in more detail.

1. Camel

The Camel route running in Assimbly picks upwardly a file from C:\in and puts it as a message on the JMS Queue "in" on ActiveMQ (likewise running in Assimbly).

You can detect out how to run this Assimbly menstruation with Camel and ActiveMQ on the Assimbly wiki. There is a quick-first and also a tuturial on message queueing.

two. NiFi

Apache NiFi gets the message from the queue 'in' with the ConsumeJMS processor and publishes it on queue 'out' with PublishJMS processor.

To accomplish this, nosotros commencement create a controller service for JMS:

The ActiveMQ Artemis client library (JMS Client Libraries) is downloaded directly from Maven.

Next pace is to configure the ConsumeJMS processor:

And the PublishJMS processor:

Last, just non least, nosotros start the menses:

three. Camel

Another Assimbly catamenia let Camel consumes the file from the queue 'out' and saves it into the directory C:\out. For this flow we clone the first flow and configure it in reverse:

When testing the flow it nevertheless functions the same way every bit NiFi or Camel did on their own, but now combined in one solution.

More complex stuff

Y'all tin choose this setup in more complex situations because of:

• Separation of Concerns: allow NiFi run menstruation logic and Camel run the connections (without the demand of applications doing a lot of integration).
• Let NiFi piece of work centrally and Camel distributed.
• Enhances functionality: NiFi processors and Camel's components.
• Accept a clear send layer (MQ).

It's possible that completely different teams or engineers piece of work on either of those tools.

Other options

Our example used JMS, it's of course possible to use other protocols. For example let Camel mail a bulletin with the HTTP component and let the "HandleHttpRequest" processor of NiFi handle this request. Then NiFi posts the bulletin with the invokeHttp processor to the "Jetty component" hosted past Camel that saves the message to file.

In that location are many other possibilities to apply both NiFi and Camel (through Assimbly Gateway) together. For example apply Apache Kafka banker with topics as broker instead of ActiveMQ. Or to employ their Residue interfaces. The cardinal point to take is that here you can have split-of-concerns and this setup support all kinds of employ cases.

One solution on code level

Like Camel, NiFi can be extended with Java code. This is washed by creating a custom processor or service controller. In that location has been some give-and-take in the NiFi community to utilize Camel lawmaking within NiFi processors. This is reflected in the Jira issues:

nifi-924 and nifi-1842

and as well on the mailing list

This has non been materialized yet and there is not a lot of code to find on this topic. Therefore, I created ii experimental custom NiFi processors which combines NiFi and Camel code.

How do they work?

As a starting time stride nosotros create a custom NiFi processor. There is an splendid guide for this written by Shubam Gupta. This takes a maven archetype to generate a default custom processor.

With this guide we create a new 'ConsumeWithCamel' processor. We add the following properties:

1. From URI (the URI of the Camel component for consuming)
2. Error URI (The URI of the Camel component for errors)
3. LogLevel (The loglevel to the NiFi log of the Camel component).

Then we add together the Camel code that:

1. Starts a CamelContext
2. Configures the route
3. Creates a consumer template

We let Assimbly Connector handle the Camel code. This API is used in Assimbly Gateway equally well. It uses a convention over configuration approach and already has a lot of Camel components (like the File component) born.

Hither is the code used when starting the NiFi processor:

          @OnScheduled public void onScheduled(last ProcessContext context) {          //Utilise Assimbly Connector to manage Apache Camel (https://github.com/assimbly/connector)      getLogger().info("Starting Apache Camel");                  //Start Apache camel         endeavor {           startCamelConnector();         } catch (Exception e2) {           getLogger().fault("Can't start Apache Camel.");           e2.printStackTrace();         }                      //Create an Assimbly flow ID   UUID uuid = UUID.randomUUID();   flowId = context.getName() + uuid.toString();                      //configure the catamenia (Camel road)         try {            configureCamelFlow(context);         } catch (Exception e1) {           getLogger().error("Can't configure Apache Camel route.");           e1.printStackTrace();         }                 //start the flow (Camel route)         attempt {           connector.startFlow(flowId);         } catch (Exception e1) {          getLogger().error("Can't start Apache Camel.");          e1.printStackTrace();         }                       //Create the endpoint      endeavor {         template = connector.getConsumerTemplate();                      } catch (Exception eastward) {          getLogger().error("Can't create Apache Camel endpoint.");         e.printStackTrace();       }            }        

Last step is to become the messages from Camel with the help of the consumerTemplate and pass information technology through to the NiFi Processor.

The lawmaking to process a message:

          @Override public void onTrigger(last ProcessContext context, terminal ProcessSession session) throws ProcessException {          //Get the message from the Camel road Object output = template.receiveBody("direct:nifi-" + flowId);          if ( output == naught ) {             return; }       FlowFile flowfile = session.create();          // To write the results dorsum out to menstruum file flowfile = session.write(flowfile, new OutputStreamCallback() {          @Override public void process(OutputStream out) throws IOException {            out.write(output.toString().getBytes());        } });          session.transfer(flowfile, SUCCESS);          }        

You can find the complete lawmaking on Github.

ProduceWithCamel

Next we create another custom Nifi Processor: "ProduceWithCamel". This is similar to the consume processor, but it works in the contrary direction. For this nosotros'll utilize a producerTemplate to produce the messages. The code for this processor you tin can find here.

Note: These are experimental processors created only for this demo.

Testing the code

To examination the code you tin download the ConsumeWithCamel processor and besides the ProduceWithCamel processor. Both NAR files are put into the lib directory of NiFi.

Now we can utilize the new Consume processor and configure it:

The Error URI is empty, which hateful errors will be logged to the NiFi log file.

Secondly we configure the produce processor:

Finally we connect both processors with each other and start the flow.

The file will be picked up and stored simply similar in all other examples.

More than possibilities

Merely like using tools, the code solution also creates all kinds of possibilities. For case dataflows aren't loosely coupled. So you lot always need to connect processors or procedure groups. When creating two process groups now you loosely couple with Camels' VM component.

The first process grouping uses the producewithcamel with the URI vm://secondProcessGroup

The second procedure group consumes this message:

At present both flows move the file from one directory to another, but the procedure groups aren't continued as usual. The new solution acts similar a 'wormhole'.

Though every example had the same result, at that place were many paths. Within integration information technology'southward good to apply open source also equally an open mind. Together they're unstoppable on whatsoever path you are on.

adkinswithems1988.blogspot.com

Source: https://caesarexperts.nl/blog/using-camel-and-nifi-in-one-solution/

Share this post