Netflix Conductor DO_WHILE loop implementation with task retry logic

  1. Significantly fast and lightweight than Airflow
  2. Its distributed architecture like Airflow, but provides client libraries in many languages for workers to be implemented in JAVA, Python, Node JS and C#
  3. Gives control to you to choose your own database persistence layer and queuing service
  4. Open source similar to Airflow
  1. System Tasks: executed on conductor server
  2. Worker Tasks: executed by custom workers (You have to register the worker task first)
  1. https://dummy.restapiexample.com/api/v1/employees — This API returns a LIST of all the employees
  2. https://dummy.restapiexample.com/api/v1/employee/1 — This API returns the details of one employee for which ID is provided
[{
"name": "get_employees",
"taskReferenceName": "get_employees",
"retryCount": 10,
"retryLogic": "FIXED",
"retryDelaySeconds": 2,
"ownerEmail": "example@email.com"
},
{
"name": "get_name",
"taskReferenceName": "get_name",
"retryCount": 10,
"retryLogic": "FIXED",
"retryDelaySeconds": 2,
"ownerEmail": "example@email.com"
}
]
{
"name": "Do_While_Workflow",
"description": "Workflow to demo DO_WHILE",
"version": 1,
"tasks": [{
"name": "get_employees",
"taskReferenceName": "get_employees",
"inputParameters": {
"http_request": {
"uri": "${workflow.input.baseUrl}/employees",
"method": "GET",
"contentType": "application/json",
"readTimeOut": 5000
}
},
"type": "HTTP"
}, {
"name": "get_names_print_names",
"taskReferenceName": "get_names_print_names",
"inputParameters": {
"employee_count": "${get_employees.output.response.body.data.length()}"
},
"type": "DO_WHILE",
"loopCondition": "$.get_names_print_names['iteration'] < $.employee_count",
"loopOver": [{
"name": "get_item_at_index",
"taskReferenceName": "get_item_at_index",
"inputParameters": {
"items": "${get_employees.output.response.body.data}",
"iterator": "${get_names_print_names.output.iteration}",
"evaluatorType": "javascript",
"expression": "$.items.get(($.iterator || 1) -1)"
},
"type": "INLINE"
}, {
"name": "get_name",
"taskReferenceName": "get_name",
"inputParameters": {
"http_request": {
"uri": "${workflow.input.baseUrl}/employee/${get_item_at_index.output.result.id}",
"method": "GET",
"contentType": "application/json",
"readTimeOut": 5000
}
},
"type": "HTTP"
}
]
}
],
"inputParameters": [],
"outputParameters": {},
"schemaVersion": 2,
"restartable": true,
"ownerEmail": "example@email.com"
}
  1. Get all the employees first. The API will return a ArrayList under — get_employees.output.response.body.data
  2. For DO_WHILE inputParameters, one can calculate the count of ArrayList using length() function
  3. DO_WHILE saves the number of loop in a output parameter “iteration”. You can refer it as taskName[‘iteration’] or taskName.output.iteration
  4. For DO_WHILE loopCondition is simply a LAMBDA call as: `$.get_names_print_names[‘iteration’] < $.employee_count`. This will ensure loop will run till iterations are less than count
  5. Finally DO_WHILE loopOver, where you can specify the tasks you want to loop over.
  6. Here the 1st task I am doing is to get the data element by each iteration using a INLINE operator by executing a javascript expression — “expression”: “$.items.get(($.iterator || 1) -1)”
  7. Note the syntax here a little bit. $.iterator gives 1 for the 1st iteration, but because $.items is an array, we need to get the 1st one. So I am subtracting 1 to get $.items(0) which is my first element
  8. Once you have the item, the same is used in the next step to do the HTTP call by extracting the id from the employee object

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Paras Bansal

Paras Bansal

Solutions Architect, Cloud Enthusiast