Understanding Oracle's Integration Cloud Service and how to get the most out of your implementation

Category: Definitive Tips Page 1 of 2

Short tips on ICS. Covering things that have been added to ICS since the book was publisahed or couldn’t be fitted in.

Beating OIC’s FTP File Limits

OIC has for some time now provided an FTP adaptor and more recently included a full FTP server capability. But both have limits on the file size and capacity. The file constraints (1GB for a file and 500GB for the FTP server) shouldn’t be an issue for day to day activities. But OIC is often used to support SaaS Financials and other cloud solutions which do have monthly process cycles which can generate significant data volumes, for example, payroll data. The question is how to handle such data with such constraints?

Data Integration?

There is a school of thought that points to the possibility when handling such large data volumes we should consider using Data Integration rather than a more event-centric integration tool. Personally, I think there is a lot of validity in the argument, and anyone dealing with such bulky data activities should review and question if it is a better answer.

That said, there are cases where it does stand-up. For example:

  • If an organization is transitioning to a more event-driven or at least micro-batch model, you have to start the transition somewhere, but trying to line up changes everywhere can be problematic, so we have to start somewhere. Building an integration process so you have an event model developed, but in the interim, you need to take that bulk mechanism and convert it to a small stream of events.
  • You may be working with a bulk data extract and only need a small subset of the data provided, it won’t help if the data is also represented using a verbose notation such as XML.

Other Approaches

How to overcome the constraint? Oracle databases aren’t so constrained, and SQLLoader can provide an easy means to ingest the data into a staging table. The benefit of this is:

  • if you’re only needing a subset of the data you can pull just those columns from the table.
  • the bulk of the use of XML to be self-describing can be shed through using the DB schema as being more prescriptive.
  • SQL scripts can handle the checksum records removing that data and overhead from the integration process, leaving you to concentrate more on the business process.

If the data is still substantial once in the database there are a number of strategies to consume the data in more manageable chunks, such as

  • Running SQL script on the database that takes each row and calls the OIC as a restful API point. This approach is potentially very interesting as it may then mean if you’re moving towards an event process in the future the API endpoint represents the future state and the database stored procedure is mimicking the future client behaviour.
  • Use polling strategies and result set limits to control how much data is processed in a single execution of the integration. This approach does mean the integration needs to tag which records have been processed to avoid re-reading them.

Useful Links

Integration with APIless Systems – Leveraging RPA – Definitive Tip #10

Traditionally integrating with systems that don’t offer APIs or a shared storage mechanism (such as open tables) has been something of a headache often resulting in the ‘last mile’ of the integration process being manual. The manual steps often come because the cost of building and maintaining the means to integrate has not been cost effective or even an option (vendor has end of lifted a product, and not willing to add an integration mechanism).

The idea of ‘screen scraping’ isn’t new, but the cost of implementing such mechanisms has dropped and the new generation of Robotic Process Automation (RPA) tools such those provided by UiPath have made it significantly easier to integrate and automate UI driven processes. Whilst UI based integration isn’t recommended as a first option for integration, it shouldn’t be ruled out particularly as it gets easier and easier to generate and maintain the UI automation. There are several factors that need to be considered as to whether such an approach is appropriate, for example:

  • the ability to run a robot to execute the UI interaction,
  • the volume of data needing to be moved through the UI – you wont escape the latency issues that may exist with UI steps,
  • is the UI being automated changing rapidly (is there enough cost benefit for automating)

1Oracle have been working in partnership with one of the leading RPA product vendors – UiPath, which has resulted in an adaptor for Oracle Integration Cloud.  The adaptor allows you to pass data to the UiPath Orchestrator component which will run the processes in an unattended mode.  In the adaptor configuration you provide information about how many resources you want the Orchestrator to apply to the task, the queuing of the job and so on.

UIP_4698_WEBSITE-GRAPHICS_WEB_100518

UiPath’s Architecture

for more information on RPA and the adaptor the following links maybe of help:

Whilst Oracle’s roadmap in the RPA space is not entirely clear we have heard indications that Oracle are limiting themselves to just UiPath (this is what UiPath say about the partnership).

Regardless of the approach you’ll see that the adoption of RPA is important in Oracle’s vision, with their Agile Finance making a clear indication of its view (see the paper here).

Deploying OIC – Definitive Tip #9

cloud-stack_CloudStack-cloud-270When ICS was launched it delivered on of the values of PaaS namely you didn’t need to worry about setting up storage, database and compute, it was all sorted out for you. Admittedly it come with a lack of elasticity when it came to resilience and scaling. In contrast Services such as SOA CS which required you to go through each of the layers, but gave you a degree of flexibility. The whilst simpler than building SOA on-premises it still represents a laborious and fiddly process that took time.

When OIC (Oracle Integration Cloud) arrived and the introduction Universal Credits we had a pricing model that made it a lot easier to be elastic in terms of approach to resourcing, but a deployment model that following SOA CS rather than ICS. To an extend, one step forward, and another back.

Fortunately we are seeing head way that means we have recovered that backward step.

Using REPL to Simplify ICS Mgmt Tasks – Definitive Tip 10

Earlier in 2017 the Oracle A-Team released a Python library that abstracts the ICS REST API to make some management tasks that you may wish to perform easier. For example identifying a set of integrations and activating or deactivating them, or simply importing and exporting them. In this Definitive Tip we will look at the REPL tool and its benefits. The REPL tool can be downloaded here.

ICS Pricing – ICS Definitive Tip #9

Cloud Costs

ICS pricing is based on two aspects – the number of connections and the number of messages processed.  But what constitutes a connection?  What happens if I exceed the number of messages or connections?

Connecting directly to DBaaS – ICS Definitive Tip #7

DBaaSEnriching an integration from data in a database or DBaaS (Database as a Service) is not an unusual requirement. Many integration use cases today need to access a database that is on-premises. The means to connect to the database is fairly obvious – the connection agent. Our book goes into a lot more detail as to why that is, and the implications of using database connections.

However when it comes to Oracle’s DBaaS a service it would be very easy to assume that given that you’re using two different parts of Oracle’s PaaS that it would be straight forward to connect the two together without an agent. However, at least today whether its on-premises or DBaaS you need to use a connection agent. This does mean that you’ll need an IaaS node to host the connection agent.

This quirk is driven by the fact that there are some scenarios that this does actually make sense. For example – the Oracle domains need to have a high level of isolation, so when the DBaaS is in another domain then the decoupling via the agent makes sense. When your database is in a different zone of the cloud – then you’re running DB calls across what is effectively a wide are network – not good.

Conditional Mapping – ICS Definitive Tip #6

The question of conditional mapping comes up regularly on the ICS part of community.oracle.com which prompted us to write this blog. So if you want the output of an integration in element X to contain value from element A or B depending upon element C there are a number of answers, each having pros and cons. So let’s look at them before we offer an example of how we would solve the simpler more common problem.. Your choices are ….

  • Use the power of XSLT within the mapping,
  • Use the conditional option Within a Basic Map Data or an Orchestration pattern,
  • Use a custom piece of JavaScript
XLST Condition in the flow Custom Javascript
Single path through the integration Simpler to read and see visually Can be used either in the condition oer the mapping. Easier to apply than XSLT directly.
ICS doesnt make it the easiest to write this, but reading is fairly easy Easy to visually understand, but means more maintenance effort Easy to incorporate, but makes the development process multistep

We have talked about the condition/filtering in the book, and we will tackle the use of JavaScript in its own Definitive Guide, so let’s look at the XSL mapping option, and probably the route that will cover most cases. In many respects this is nothing to do with ICS, and is all about the use of XSLT so you may find it helpful to have an XSLT reference to hand. To this end the following maybe useful:

  • W3C – perhaps not the easiest to read, but definitely definitive,
  • XMLSchool far more readable but keeps things simple, and focuses on XSLT to create (X)HTML content

Differentiating the ICS Agent Types – Definitive Tip #5

In our book we talked about the difference between the agents offered by ICS, namely the Connection  and Execution agents. Whilst we differentiated the two, we did focus on the connection agent as this is the type we expect to see used in most cases. However the execution agent still suffers from a level of confusion, and it has been helped by being called  ‘ICS on-premises’.

As part of a number of recent conversations the questions and confusion of what the execution agent is and how it works has come up. There is the well known saying ‘a picture is as good as a 1000 words’ which prompted us to develop the diagram below as a power point slide – in its power point form much of the detail is used as an animated build up.

ICS Agent Comparison

Selecting which iPaaS Product(s) to use – ICS Definitive Tip #4

In the last few years Oracle’s iPaaS offerings have progressed significantly from one or two solutions such as Mobile Cloud Service to in excess of 35 offerings. This makes it challenging to decide which product(s) to use. The means for selection depends upon a range of factors including:

  • Most suitable licensing model (ICS provides a lower cost of entry than SOA CS as you’re paying by message volumes)
  • Development skills and desire to resilience or invest in people (SOA on-premises to SOA CS is easier than a technology change such as ICS)
  • The amount of control you have in an environment – JCS offers more control than ACCS for example

Using iCal in Scheduling – ICS Definitive Tip #3

In ICS Definitive Guide #2 we looked at the scheduling of integrations and mentioned the possibility of using iCal. In this tip we shall explore advanced iCal facility for defining schedules within ICS.

icalThe iCal format is probably best known for sharing meeting events in tools such as outlook. The format is actually a standard defined by the IETF originally in 1998 (https://www.ietf.org/rfc/rfc2445.txt)  and updated several times the latest being IETF Logo - owner of iCal std 2015 (https://tools.ietf.org/html/rfc7529). The goal of the definition has been for the purpose of driving standardisation across PIM tools (Personal Information Management) tools which includes Outlook and Thunderbird etc. It is a substantial standard the original document ran to 150 pages, but the section supported by the schweduler’s advanced option purely relates to the reoccurence definition covered by section 4.3.10 of the original RFC, and only takes five pages. If you want to know lots about iCal and the varying revisions then you may find https://icalendar.org/ helpful.

It is worth mentioning that today ICS under the hood exploits Oracle’s Enterprise Scheduler Service (ESS). ESS documentation identifies that it works to the 2445 version of the specification (see here). ESS became a 1st class citizen component with the start of the 12c middleware product stack. As a result, if you see any errors processing an iCal configuration the error codes will reference ESS does.

Page 1 of 2

Powered by WordPress & Theme by Anders Norén