references/common-schemas/ai-description.yml
AIDescription:
type: object
description: |
AI-generated descriptions and documentation for the resource.
This object contains automatically generated content that helps users
understand the purpose, behavior, and configuration of the resource without
requiring them to analyze the technical details. The AI-generated content
is sanitized and safe for display in the UI.
properties:
summary:
type:
- string
- "null"
description: |
Brief AI-generated summary of the resource's purpose and functionality.
This concise description provides a quick overview of what the resource does,
what systems it interacts with, and its primary role in the integration.
The summary is suitable for display in list views, dashboards, and other
contexts where space is limited.
Maximum length: 10KB
examples: ["AI-generated overview of what this resource does and the systems it interacts with."]
detailed:
type:
- string
- "null"
description: |
Comprehensive AI-generated description of the resource's functionality.
This detailed explanation covers the resource's purpose, configuration details,
data flow patterns, filtering logic, and other technical aspects. It provides
in-depth information suitable for documentation, tooltips, or detailed views
in the administration interface.
The content may include HTML formatting for improved readability.
Maximum length: 10KB
examples: ["<p>A detailed AI-generated explanation of this resource's purpose, configuration, and behavior.</p><p>The text is regenerated when the resource's configuration changes.</p>"]
generatedOn:
type:
- string
- "null"
format: date-time
description: |
Timestamp indicating when the AI description was generated.
This field helps track the freshness of the AI-generated content and
determine when it might need to be regenerated due to changes in the
resource's configuration or behavior.
The timestamp is recorded in ISO 8601 format with UTC timezone (Z suffix).
examples: ["2023-05-10T13:25:42Z"]references/common-schemas/apim.yml
APIM:
type: array
description: |
Read-only field that stores information about the integration resources
published in the API Management (APIM) platform.
This field tracks the relationship between integrator.io resources and their
published counterparts in the APIM platform, which is
tightly integrated with the Celigo UI. When resources are "pushed" to APIM,
this field is populated with the relevant identifiers and statuses.
items:
type: object
properties:
apiId:
type: string
description: |
Identifier for the API where this integrator.io resource is published in the APIM.
This is an APIM resource identifier (not prefixed with underscore like Celigo IDs)
that uniquely identifies the API in the API Management platform.
flowId:
type: string
description: |
Identifier for the flow within the API where this integrator.io resource is linked.
When an API has multiple integrator.io resources linked, each resource is associated
with a specific flow in the API, identified by this field. This is an APIM
resource identifier.
status:
type: string
description: |
Indicates the publishing stage of the integrator.io resource in APIM.
Possible values:
- 'oaspending': The resource is published but the OpenAPI Specification (OAS) is not
yet published. The apiId will be updated with the API ID created in APIM.
- 'published': The OpenAPI Specification for the integrator.io resource has been
successfully uploaded to APIM.
enum:
- oaspending
- published
x-enumDescriptions:
oaspending: The resource is published to APIM but its OpenAPI Specification has not yet been uploaded.
published: The OpenAPI Specification has been successfully uploaded to APIM.references/common-schemas/filter.yml
Filter:
type: object
description: |
Configuration for selectively processing records based on specified criteria. This object enables
precise control over which items are included or excluded from processing operations.
**Filter behavior**
When configured, the filter is applied before processing begins:
- Items that match the filter criteria are processed
- Items that don't match are completely skipped
- No partial processing is performed
Send the whole object. A rules array sent alone as the `filter` value is not
stored, and a full update of an export that carries one removes the filter the
export had, with no error.
**Implementation approaches**
There are two distinct filtering mechanisms available:
**Rule-Based Filtering (`type: "expression"`)**
- **Best For**: Common filtering patterns based on standard attributes
- **Capabilities**: Filter by names, values, dates, numerical ranges, text patterns
- **Advantages**: Declarative, no coding required, consistent performance
- **Configuration**: Define rules in the `expression` object
- **Use When**: You have clear, static criteria for selection
**Script-Based Filtering (`type: "script"`)**
- **Best For**: Complex logic, dynamic criteria, or business rules
- **Capabilities**: Full programmatic control, access to complete metadata
- **Advantages**: Maximum flexibility, can implement any filtering logic
- **Configuration**: Reference a script in the `script` object
- **Use When**: Simple rules aren't sufficient or logic needs to be dynamic
properties:
type:
type: string
description: |
Determines which filtering mechanism to use. This choice affects which properties
must be configured and how filtering logic is implemented.
**Available types**
**Rule-Based Filtering (`"expression"`)**
- **Required Config**: The `expression` object with rule definitions
- **Behavior**: Evaluates declarative rules against item attributes
- **Best For**: Common patterns like name matching, date ranges, value limits
- **Advantages**: Simpler to configure, no custom code required
**Script-Based Filtering (`"script"`)**
- **Required Config**: The `script` object with _scriptId and function
- **Behavior**: Executes custom JavaScript to determine which items to process
- **Best For**: Complex conditions, business logic, dynamic criteria
- **Advantages**: Maximum flexibility, can implement any logic
**Implementation guidance**
1. For standard filtering needs (name, size, date), use `"expression"`
2. For complex logic or conditions not covered by expressions, use `"script"`
3. When selecting a type, you must configure the corresponding object:
- `type: "expression"` requires the `expression` object
- `type: "script"` requires the `script` object
enum: ["expression", "script"]
x-enumDescriptions:
expression: Rule-based filtering using declarative conditions.
script: Script-based filtering using custom JavaScript logic.
examples: ["expression", "script"]
expression:
type: object
description: |
Configuration for declarative rule-based filtering. This object enables filtering
items based on common attributes without requiring custom code.
**Usage context**
This object is REQUIRED when `filter.type` is set to "expression" and should not be
configured otherwise. It provides a standardized way to define filtering rules that
can match against item attributes like name, type, value, date, and other properties.
**Implementation guidance**
The expression system uses a rule-based approach where:
- Rules can be combined with AND/OR logic
- Each rule can check a specific attribute
- Multiple conditions can be applied (ranges, pattern matching, exact matches)
**Common filter patterns**
1. **Pattern matching**: Regular expressions with `matches`
2. **Value range filtering**: Numbers between min and max values
3. **Date range filtering**: Items created/modified within specific time ranges
4. **Status checking**: Items with specific status values or properties
For AI agents: Rule-based filtering should be your first choice when the filtering criteria
can be expressed in terms of standard attributes. Only use script-based filtering when
more complex logic is required.
properties:
version:
type: string
description: |
Version identifier for the expression format. Currently only version "1" is supported.
This field ensures future compatibility if the expression format evolves. Always set to "1"
for current implementations.
enum: ["1"]
x-enumDescriptions:
"1": Current and only supported filter expression version.
examples: ["1"]
rules:
type: array
description: |
Expression array defining filter conditions using prefix notation. The first element is the operator,
followed by its operands which may themselves be nested expression arrays.
The rule expression follows this pattern:
- First element: Operator name (string)
- Remaining elements: Operands for that operator (values or nested expressions)
**Expression structure**
Filter expressions use a prefix notation where operators appear before their operands:
```
[operator, operand1, operand2, ...]
```
**Comparison Operators**
- `"equals"`: Exact match (equals)
- `"notequals"`: Not equal to value (not equals)
- `"greaterthan"`: Value is greater than specified value (is greater than)
- `"greaterthanequals"`: Value is greater than or equal to specified value (is greater than or equals)
- `"lessthan"`: Value is less than specified value (is less than)
- `"lessthanequals"`: Value is less than or equal to specified value (is less than or equals)
- `"startswith"`: String starts with specified prefix (starts with)
- `"notstartswith"`: String does not start with specified prefix (does not start with)
- `"endswith"`: String ends with specified suffix (ends with)
- `"notendswith"`: String does not end with specified suffix (does not end with)
- `"contains"`: String contains specified substring (contains)
- `"doesnotcontain"`: String does not contain specified substring (does not contain)
- `"empty"`: Field is missing or empty (is empty)
- `"notempty"`: Field contains a value (is not empty)
- `"matches"`: Matches a regular expression, ignoring case (matches)
Operator names are case-insensitive: `NOTEMPTY` and `endsWith` work and are stored
as sent. `isempty`, `isnotempty`, `anyof`, `notcontains`, `doesnotstartwith`,
`doesnotendwith`, `notmatches`, `doesnotmatch` and `notequal` are not operators:
they are refused with 422 "Invalid operation: <name>", with the name in lowercase,
both when the filter is saved and by `POST /v1/processors/filter`. To negate a
condition, wrap it in `not`.
The string comparisons (`equals`, `contains`, `startswith`, `endswith` and their
negations) are case-sensitive; wrap the field in `lowercase` to compare without case.
`matches` is not: every pattern is compiled with the flags `gi`, so `^paid$` also
matches `PAID` and `[a-z]` matches capital letters. An inline flag such as `(?i)`
makes the pattern invalid. The filter still saves, but in a run every record fails
with `FILTER_EVALUATION_FAILED` ("Invalid regular expression: /(?i)…/gi: Invalid
group"), and `POST /v1/processors/filter` refuses the call with 422 (live-verified
2026-10-22).
**Logical Operators**
- `"and"`: All conditions must be true
- `"or"`: At least one condition must be true
- `"not"`: Negates the condition
**Field Access and Type Conversion**
- `"extract"`: Access a field from the item by name
- `"settings"`: Access a custom setting from the flow, flow step, or integration configuration
- `"boolean"`: Convert value to Boolean type
- `"epochtime"`: Convert value to Epoch Time (Unix timestamp)
- `"number"`: Convert value to Number type
- `"string"`: Convert value to String type
**Field Access Details**
**Using `extract` to access record fields:**
- Retrieves values from the current record being processed
- Can access nested properties using dot notation (e.g., `"customer.email"`)
- Paths are relative to the record itself: `["extract", "record.status"]`
matches nothing
- In a record filter, must be wrapped in a type conversion (`string`, `number`,
`boolean` or `epochtime`): a bare `["extract", ...]` operand is refused with 422
"Field operation extract must be called by a Type Conversion Operation but was called
by <operator>", both when an import's filter is saved and by
`POST /v1/processors/filter`
**Using `settings` to access configuration values:**
- Retrieves values from the integration's configuration settings
- Supports different scopes with prefix notation:
- `flow.settingName`: Access flow-level settings
- `flowGrouping.settingName`: Access the flow group's settings
- `export.settingName`: Access the export's (or lookup's) own settings
- `import.settingName`: Access the import's own settings
- `integration.settingName`: Access integration-level settings
- `connection.settingName`, `iClient.settingName`: Access the step's
connection and iClient settings
- A scope whose resource has no settings can be absent; a rule that reads
it matches nothing
- Must be wrapped in a type conversion like `extract`: a bare
`["settings", ...]` operand is refused with 422 "Field operation settings
must be called by a Type Conversion Operation"
- Useful for dynamic filtering based on configuration
**Field Transformations**
- `"lowercase"`: Convert string to lowercase
- `"uppercase"`: Convert string to uppercase
- `"ceiling"`: Round number up to the nearest integer
- `"floor"`: Round number down to the nearest integer
- `"abs"`: Get absolute value of a number
**Arithmetic** (number operands)
- `"add"`, `"subtract"`, `"multiply"`, `"divide"`, `"modulo"`: for example
`["subtract", ["number", ["extract", "total"]], ["number", ["extract", "discount"]]]`.
`mod` is not an operator (422 "Invalid operation: mod.").
**Dates**
- `epochtime` reads ISO 8601 strings and epoch-millisecond numbers. An empty
string, a string that isn't a date, or epoch milliseconds sent as a string
fail it with "Invalid date. Cannot convert given string or number to date
time.", followed by `; record=` and the record as JSON: in a run that record fails (`FILTER_EVALUATION_FAILED`, source
`output_filter` on an export), and `POST /v1/processors/filter` and an
export preview refuse the whole call with 422. `and` and `or` evaluate
every operand, so a `notempty` test beside it doesn't prevent the failure.
A missing or null field doesn't match.
- `datetime`, `date` and `timestamp` are not type conversions: 422
"Invalid operation: <name>."
- `number` reads a missing or empty field as 0.
**Handlebars operands**
- An operand can also be a string holding exactly one Handlebars expression, such as
`"{{record.status}}"` or `"{{lowercase record.code}}"`, on either side of the
operator. It renders against the record (`record.<field>`; an unprefixed
`{{status}}` renders empty) before the comparison.
- The rendered value is a string: wrap it in `["number", ...]` to compare it with a
number, or the rule is refused with 422.
- `epochtime` does not render Handlebars: `["epochtime", "{{timestamp}}"]` is refused
with 422 "invalid datetime format".
- A string with text around the expression, or with more than one expression, is
compared as a literal.
- `["number", "{{timestamp \"x\"}}"]` is the current time in epoch milliseconds,
so `["greaterthanequals", ["epochtime", ["extract", "created"]], ["subtract",
["number", "{{timestamp \"x\"}}"], 604800000]]` keeps records created in the
last 7 days.
- An unknown helper fails the rule with 422 "Failed to evaluate handlebars
expression".
Choose the type conversion that matches the comparison — `number` for numeric
comparisons, `epochtime` for dates, `string` for text. For example:
```json
[
"equals",
[
"number", // Convert to number before comparison
[
"extract",
"quantity"
]
],
100
]
```
Example with datetime conversion:
```json
[
"greaterthan",
[
"epochtime", // Convert to Unix timestamp before comparison
[
"extract",
"createdDate"
]
],
1609459200000 // January 1, 2021 as Unix timestamp in milliseconds
]
```
Example with transformations:
```json
[
"and",
[
"matches",
[
"lowercase", // Convert to lowercase before matching
[
"string",
[
"extract",
"categories"
]
]
],
"netsuite"
],
[
"notequals",
[
"string",
[
"extract",
"recurrence.pattern.type"
]
],
""
]
]
```
Example comparing a record field with a flow setting:
```json
[
"equals",
[
"string",
[
"extract",
"trantype"
]
],
[
"string",
[
"settings",
"flow.trantype"
]
]
]
```
**Examples**
Example 1: Status field is not equal to "cancelled"
```json
[
"notequals",
[
"string",
[
"extract",
"status"
]
],
"cancelled"
]
```
Example 2: Filename starts with "HC"
```json
[
"startswith",
[
"string",
[
"extract",
"filename"
]
],
"HC"
]
```
Example 3: Amount is greater than 100
```json
[
"greaterthan",
[
"number",
[
"extract",
"amount"
]
],
100
]
```
Example 4: Order date is after January 1, 2023 (UTC). Compare dates as `epochtime`
against epoch milliseconds: a `string` comparison of ISO timestamps orders values with
different UTC offsets wrongly.
```json
[
"greaterthan",
[
"epochtime",
[
"extract",
"orderDate"
]
],
1672531200000
]
```
x-celigo-ai-guidance:
- |-
To test whether a field has a value, use `notempty` on the wrapped extract, not a
`matches` pattern: a record without the field matches `.*\S.*`.
items:
oneOf:
- title: String
type: string
- title: Number
type: number
- title: Boolean
type: boolean
- title: Object
type: object
- title: Array
type: array
- title: 'Null'
type: 'null'
script:
type: object
description: |
Configuration for programmable script-based filtering. This object enables complex, custom
filtering logic beyond what expression-based filtering can provide.
**Usage context**
This object is REQUIRED when `filter.type` is set to "script" and should not be configured
otherwise. It provides a way to execute custom JavaScript code to determine which items
should be processed.
**Implementation approach**
Script-based filtering works by:
1. Executing the specified function from the referenced script
2. Passing item data to the function
3. Using the function's return value (true/false) to determine inclusion
**Common use cases**
Script filtering is ideal for:
- Complex business logic that can't be expressed as simple rules
- Dynamic filtering criteria that change based on external factors
- Content-based filtering that requires deep inspection
- Advanced pattern matching beyond simple wildcards
- Multi-stage filtering with intermediate logic
For AI agents: Only use script-based filtering when expression-based filtering is insufficient.
Script filtering requires maintaining custom code, which adds complexity to the integration.
properties:
_scriptId:
type: string
x-celigo-refModel: scripts
description: |
Reference to the Script resource that contains the filtering logic. This must be a valid
ObjectId of a Script resource in this account: any other id fails the save with 422
("Only a script owned by the user or a script shared by the Integration App can be used
at filter.script._scriptId.").
The referenced script must contain the function specified in the `function` field
and must be written to handle filtering specifically. The script receives
item data as its input and must return a boolean value indicating whether
to process the item (true) or skip it (false).
Format: 24-character hexadecimal string (MongoDB ObjectId)
examples: ["60a2c4e6f321d800129a1a3c", "5f8d43a1b9e5a80011a35f2c"]
function:
type: string
description: |
Name of the function within the script to execute for filtering decisions. This function
must exist in the script referenced by _scriptId.
**Function requirements**
The specified function must:
- Accept item data as its first parameter
- Return a boolean value (true to process the item, false to skip it)
- Handle errors gracefully
- Execute efficiently (as it may run for many items)
**Function signature**
```javascript
function filterItems(itemData) {
// itemData contains properties of the item being evaluated
// Custom logic here
return true; // or false to skip the item
}
```
For AI agents: Ensure the function name exactly matches a function defined in the
referenced script, as mismatches will cause the filter to fail.
examples: ["filterItems", "shouldProcess", "evaluateMatch"]references/common-schemas/form.yml
Form:
type: object
description: |
Configuration for creating user-friendly settings forms that make it easier for less technical users
to configure integration resources.
**Settings form builder**
The Settings Form Builder allows you to create or edit user-friendly fields that prompt for text entry
or selections that will be returned as settings applied to this resource. Your forms can include any
field types that you see elsewhere in integrator.io, such as:
- Text fields
- Dropdown selections
- Checkboxes
- Radio buttons
- Date pickers
- Multi-select fields
- Search fields
Form fields make it much easier for less technical users to work with your integration settings by:
- Providing clear labels and help text
- Enforcing validation rules
- Offering pre-defined selection options
- Grouping related settings logically
- Supporting conditional visibility
- Creating a consistent user experience
properties:
form:
type: object
description: |
Configuration that defines the structure, fields, and behavior of the settings form.
This object contains the complete definition of the form's layout, fields, validation rules,
and interactive behaviors. The specific structure depends on the form complexity and can include
field definitions, sections, conditional display logic, and default values.
The form configuration is typically created and managed through the visual Form Builder interface
rather than edited directly as JSON.
properties:
fieldMap:
type: object
description: |
A mapping of field identifiers to their configuration objects.
Each key in this object represents a unique field ID, and the value contains
all the configuration settings for that specific form field.
additionalProperties:
type: object
description: |
Configuration for an individual form field.
properties:
id:
type: string
description: |
Unique identifier for this field within the form: layout containers and conditions
refer to the field by it. Use the same string as the field's fieldMap key and its `name`.
examples: ["url", "mode", "body", "keyValue", "text", "multiline"]
name:
type: string
description: |
Key the field's value is saved under: one top-level key of the resource's `settings`,
which the form also reads the value back from when it opens. Nested objects inside
`settings` are written to it directly, not through a form field's `name`.
examples: ["url", "mode", "body", "keyValue", "text", "multiline"]
type:
type: string
description: |
The type of form control to render for this field.
enum: [
"text", "checkbox", "radiogroup", "relativeuri", "editor", "keyvalue",
"select", "multiselect", "toggle", "datetime", "date",
"exportSelect", "staticMap"
]
x-enumDescriptions:
text: Single-line or multi-line text input field.
checkbox: Boolean checkbox control for true/false values.
radiogroup: Group of mutually exclusive radio button options.
relativeuri: Text input for entering a relative URI path.
editor: Code or content editor with syntax highlighting.
keyvalue: Dynamic list of key-value pair inputs.
select: Dropdown menu for selecting a single option.
multiselect: Dropdown menu that allows selecting multiple options.
toggle: On/off toggle switch control.
datetime: Date and time picker control.
date: Date-only picker control without a time component.
exportSelect: Dropdown whose options are records returned by an export.
staticMap: Static two-column mapping table for pairing source and destination values.
examples: ["text", "checkbox", "radiogroup", "editor", "keyvalue", "select", "multiselect"]
label:
type: string
description: |
Display label shown next to the field in the form.
examples: ["Some url with handlebars support.", "Mode of operation", "Key-value pairs", "Text"]
description:
type: string
description: |
Detailed explanation text that appears below the field, providing more context
than the label or helpText.
examples: ["This input is used to collect a set of key-value pairs. The item delete action is optional and set using th showDelete prop. Also note that he key and value names can be configured as well."]
helpText:
type: string
description: |
Explanatory text that appears when hovering over the help icon next to the field.
Used to provide additional guidance on how to use the field.
examples: ["example of a custom input.", "This is the most basic input. It models an http form input"]
required:
type: boolean
description: |
When true, the field must have a value before the form can be submitted. Only the form
enforces it: an API write of `settings` is stored without the value.
default: false
examples: [true, false]
defaultValue:
description: |
Value the form shows while the resource's `settings` holds no value under the field's
`name`.
examples: ["us", true]
multiline:
type: boolean
description: |
For text fields, determines whether the input should be a multi-line text area
instead of a single-line input.
default: false
examples: [true, false]
rowsMax:
type: integer
description: |
For multiline text fields, specifies the maximum number of visible rows.
examples: [5]
inputType:
type: string
description: |
For text fields, specifies the HTML input type attribute to apply additional
validation or specialized input behavior.
enum: ["text", "number", "email", "password", "tel", "url"]
x-enumDescriptions:
text: Standard text input with no special validation.
number: Numeric input that restricts entry to numbers.
email: Email address input with email format validation.
password: Masked text input that hides the entered characters.
tel: Telephone number input optimized for phone number entry.
url: URL input with web address format validation.
examples: ["number"]
delimiter:
type: string
description: |
For text fields, specifies a character to use for splitting the input into an array.
Used for collecting multiple values in a single text field.
examples: [","]
mode:
type: string
description: |
For editor fields, specifies the type of content being edited for syntax highlighting.
enum: ["json", "xml", "csv", "text"]
x-enumDescriptions:
json: JSON syntax highlighting and validation.
xml: XML syntax highlighting and validation.
csv: CSV format highlighting for comma-separated data.
text: Plain text mode with no syntax highlighting.
examples: ["json"]
keyName:
type: string
description: |
For keyvalue and staticMap fields, names the first column: each saved row holds that
column's value under this key. A keyvalue field without it uses `key`.
examples: ["theKey", "source"]
valueName:
type: string
description: |
For keyvalue and staticMap fields, names the second column: each saved row holds that
column's value under this key. A keyvalue field without it uses `value`.
examples: ["theValue", "target"]
keyLabel:
type: string
description: |
For staticMap fields, heading of the first column.
examples: ["Source status"]
valueLabel:
type: string
description: |
For staticMap fields, heading of the second column.
examples: ["Destination status"]
keyOptions:
type: array
description: |
For staticMap fields, suggestions offered in the first column's cells. A cell still
accepts a typed value.
items:
type: object
properties:
label:
type: string
description: |
Display text for the suggestion.
value:
type: string
description: |
Value saved when the suggestion is chosen.
examples: [[{"label": "Pending", "value": "pending"}, {"label": "Shipped", "value": "shipped"}]]
valueOptions:
type: array
description: |
For staticMap fields, suggestions offered in the second column's cells. A cell still
accepts a typed value.
items:
type: object
properties:
label:
type: string
description: |
Display text for the suggestion.
value:
type: string
description: |
Value saved when the suggestion is chosen.
examples: [[{"label": "Open", "value": "open"}, {"label": "Closed", "value": "closed"}]]
keyResource:
type: object
description: |
For staticMap fields, fills the first column's suggestions from an export the form
carries inline in `virtual`, run the way an exportSelect's `resource` is. With
`keyResource` or `valueResource` set, the form requires both columns in every row
unless `keyValueRequired` is false.
properties:
virtual:
type: object
description: |
Export document to run, inline, shaped like an exportSelect's `resource.virtual`.
additionalProperties: true
valueResource:
type: object
description: |
For staticMap fields, fills the second column's suggestions from an export the form
carries inline in `virtual`, run the way an exportSelect's `resource` is.
properties:
virtual:
type: object
description: |
Export document to run, inline, shaped like an exportSelect's `resource.virtual`.
additionalProperties: true
keyValueRequired:
type: boolean
description: |
For staticMap fields with `keyResource` or `valueResource`, when false, saves rows
that fill only one column; by default the form drops them.
default: true
examples: [false]
resource:
type: object
description: |
For exportSelect fields, the export whose records become the options. When the form
opens, the UI previews the export in `virtual` (with `test.limit` 100 unless the export
sets its own) and lists the preview's `data`: each record needs a `label` and a `value`
(a string or number serves as both), so shape them with the export's `transform`. A
saved export can't be referenced, and an HTTP export that finds no records leaves the
field showing an error instead of options.
properties:
virtual:
type: object
description: |
Export document to run, inline: `adaptorType`, its adaptor block, and an optional
`transform`. `_connectionId` defaults to the connection of the export or import the
form belongs to; set it on any other resource's form.
additionalProperties: true
examples:
- virtual:
adaptorType: HTTPExport
http:
relativeURI: /v1/warehouses
method: GET
formType: http
response:
resourcePath: warehouses
transform:
type: expression
expression:
version: "2"
rulesTwoDotZero:
mode: create
mappings:
- generate: label
extract: $.name
dataType: string
status: Active
- generate: value
extract: $.code
dataType: string
status: Active
multiselect:
type: boolean
description: |
For exportSelect fields, when true, the user can choose several options and the value
is saved as an array.
default: false
examples: [true]
supportHandlebars:
type: boolean
description: |
For exportSelect fields, when true, renders the `resource.virtual` export as a Handlebars
template before each preview (all but its `transform`, `hooks`, and `filter`). On an
integration, flow group, flow, export, or import form, the form's current values are
under that form's scope name (`integration`, `flowGrouping`, `flow`, `export`, or
`import`), beside the saved settings of the scopes above it, with no `settings.`
prefix; on a connection or iClient form the template has no context. The options then
load through `refreshOptionsOnChangesTo` or `typeAheadSearch`, or when the user
refreshes the field.
default: false
examples: [true]
refreshOptionsOnChangesTo:
type: array
items:
type: string
description: |
For exportSelect fields with `supportHandlebars`, ids of the fields whose values the
templated export uses: the options load once one of them has a value and reload whenever
one changes.
examples: [["region"]]
typeAheadSearch:
type: boolean
description: |
For exportSelect fields, when true, re-runs the templated export as the user types, with
the typed text as this field's own value; an empty box clears the options, and typed
text is accepted as the value.
default: false
examples: [true]
showDelete:
type: boolean
description: |
For keyvalue fields, determines whether to show a delete button for each key-value pair.
examples: [true, false]
doNotAllowFutureDates:
type: boolean
description: |
For date and datetime fields, restricts selection to dates not in the future.
examples: [true, false]
skipTimezoneConversion:
type: boolean
description: |
For datetime fields, prevents automatic timezone conversion of the date value.
examples: [true, false]
options:
type: array
description: |
For fields that present choices (select, multiselect, radiogroup, toggle), defines
the available options.
items:
anyOf:
- title: Option group
type: object
required: [items]
properties:
items:
type: array
items:
oneOf:
- title: String value
type: string
- title: Label-value pair
type: object
properties:
label:
type: string
description: |
Display text for the option.
examples: ["United States", "Canada", "India"]
value:
type: string
description: |
Value to store when this option is selected.
examples: ["us", "cdn", "in"]
description: |
Array of option values/labels to display in the selection control.
examples: [
["Create", "Update", "Delete"],
[{"label": "United States", "value": "us"}, {"label": "Canada", "value": "cdn"}]
]
- title: Label-value pair
type: object
required: [label]
properties:
label:
type: string
description: |
Display text for the option.
examples: ["US", "Canada", "India"]
value:
type: string
description: |
Value to store when this option is selected.
examples: ["us", "cdn", "in"]
visibleWhen:
type: array
description: |
Conditional display rules that determine when this field should be visible.
If empty or not provided, the field is always visible.
items:
type: object
properties:
field:
type: string
description: |
The ID of another field whose value controls the visibility of this field. The
save doesn't check it: a rule naming an ID that isn't in the form counts as met,
so the field stays visible.
examples: ["mode"]
is:
type: array
items:
type: [string, boolean, number, 'null']
description: |
Array of values - if the referenced field has any of these values,
this field will be visible. Values may be strings, booleans (for
checkbox/toggle fields), numbers (for numeric inputs), or null.
examples: [["Update"], [true]]
layout:
type: object
description: |
Defines how the form fields are arranged and grouped in the UI.
The layout can organize fields into columns, sections, or other visual groupings.
properties:
type:
type: string
description: |
The type of layout to use for the form.
enum: ["column", "collapse", "box", "indent", "tabWithoutSave", "verticalTabWithoutSave"]
x-enumDescriptions:
column: Places the containers side by side, one column each.
collapse: Groups fields into collapsible/expandable sections.
box: Wraps field groups in bordered boxes with headings.
indent: Indents field groups to show hierarchy.
tabWithoutSave: Organizes field groups into horizontal tabs (no per-tab save).
verticalTabWithoutSave: Organizes field groups into vertical tabs (no per-tab save).
examples: ["column", "collapse"]
fields:
type: array
items:
type: string
description: |
Field IDs shown above the containers, without a heading.
examples: [["region", "vatNumber"]]
containers:
type: array
description: |
Array of container objects that group fields or contain nested containers.
Each container can represent a column, box, indented section, or collapsible section.
items:
type: object
properties:
type:
type: string
description: |
The visual style of the container.
enum: ["indent", "box", "collapse"]
x-enumDescriptions:
indent: Indents the contained fields to show hierarchy.
box: Wraps the contained fields in a bordered box with a heading.
collapse: Wraps the contained fields in a collapsible/expandable section.
examples: ["indent", "box", "collapse"]
label:
type: string
description: |
The heading text displayed for this container.
examples: ["Basic fields", "Advanced fields", "Indented fields", "Boxed fields"]
fields:
type: array
items:
type: string
description: |
Array of field IDs that should be displayed in this container.
Each ID must correspond to a key in the fieldMap object: the save doesn't check it,
and an ID that doesn't renders nothing. A field the layout doesn't list isn't shown.
examples: [["A", "url"], ["mode", "body"], ["keyValue", "checkbox"]]
containers:
type: array
description: |
Nested containers within this container. Allows for hierarchical organization
of fields with different visual styles.
items:
type: object
properties:
label:
type: string
description: |
The heading text displayed for this nested container.
examples: ["Indented fields", "Boxed fields", "Collapsed fields"]
fields:
type: array
items:
type: string
description: |
Array of field IDs that should be displayed in this nested container.
examples: [["keyValue", "checkbox"]]
additionalProperties: true
init:
type: object
description: |
Configuration for custom JavaScript initialization that executes when the form is first loaded.
This object defines a JavaScript hook that prepares the form for use, sets initial field values,
performs validation, or otherwise customizes the form behavior before it is displayed to the user.
**Function signature**
The initialization function is invoked with a single 'options' argument containing contextual information:
```javascript
function formInit(options) {
// Process options and return the form object
return options.resource.settingsForm.form;
}
```
**Available context**
The 'options' argument provides access to:
- `options.resource` - The current resource being configured
- `options.parentResource` - The parent of the current resource
- `options.grandparentResource` - The grandparent of the current resource
- `options.license` - For integration apps, the license provisioned to the integration
- `options.parentLicense` - For integration apps, the parent of the license
- `options.sandbox` - Boolean flag indicating whether the script is running in a sandbox environment
**Common uses**
- Dynamically generate field options based on resource configuration
- Pre-populate default values from related resources
- Apply conditional logic that depends on resource properties
- Add, remove, or modify form fields based on user permissions or account settings
- Fetch external data to populate selection options
- Implement complex validation rules that depend on resource context
- Create branching form experiences based on user selections
**Return value**
The function must return a valid form object that the UI can render.
Throwing an exception will signal an error to the user.
The UI runs the function each time it opens the form and renders the returned form instead of
`form`, then fills in the values from `settings`; a thrown error's message replaces the form.
properties:
function:
type: string
description: |
The name of the function to execute within the referenced script.
This property specifies which function to invoke from the script
referenced by _scriptId. The function will be called when the form
is initialized and should handle any custom setup logic.
The function must follow the expected signature and return a valid form object.
examples: ["initializeForm", "setupFormState", "prepareFormData"]
_scriptId:
type: string
format: objectId
x-celigo-refModel: scripts
description: |
Reference to a predefined script resource containing the initialization function.
The referenced script should contain the function specified in the
'function' property. On an integration, a flow group, a flow, an export, an import, or a
connection, an id that isn't a script in the same account (or, on an Integration App's
resources, one the app shares) is refused with 422 `invalid_ref`. An iClient save
stores the id without that check.
examples: ["60a2c4e6f321d800129a1a3c", "5f8d43a1b9e5a80011a35f2c"]references/common-schemas/hooks.yml
Hook:
type: object
description: |-
A single lifecycle hook — a JavaScript function invoked from a script (or hosted on a
stack) at a fixed point in an import or AI-agent run. Shared by every hook slot; the
slot's own description says when in the lifecycle it fires.
properties:
function:
type: string
description: |-
Function to invoke: one in the script named by `_scriptId`, or one the code deployed on
the stack named by `_stackId` exposes.
examples: ["transformRecords"]
_scriptId:
type: string
format: objectId
x-celigo-refModel: scripts
description: |-
Script containing the hook function named in `function`. Send this or `_stackId`, not both.
Must be a script in this account: any other id fails the save with 422, naming the hook,
such as "Only a script owned by the user or a script shared by the Integration App can
be used at hooks.preMap._scriptId." on an import's `preMap`.
_stackId:
type: string
format: objectId
x-celigo-refModel: stacks
description: |-
Stack that hosts the hook logic, used instead of a script for stack-based deployments.
The API doesn't enforce the either/or: a save carrying both `_scriptId` and `_stackId`
answers 200 and stores both, and the script and the stack then each list the step as a
dependent.
x-celigo-ai-guidance:
- When moving a hook onto a stack, remove `_scriptId` and set `function` to the stack's function in the same update.
configuration:
type: [object, 'null']
description: |-
Static parameters passed to the hook function at runtime, letting one script be
reused with different settings. Null when the hook carries no static parameters.
references/common-schemas/ia-resource-response.yml
IAResourceResponse:
type: object
description: Integration app response fields for resources that are part of integration apps
properties:
_integrationId:
type: string
format: objectId
x-celigo-refModel: integrations
readOnly: true
description: |
Reference to the specific integration instance that contains this resource.
This field is only populated for resources that are part of an integration app
installation. It contains the unique identifier (_id) of the integration
resource that was installed in the account.
The integration instance represents a specific installed instance of an
integration app, with its own configuration, settings, and runtime environment.
This reference enables:
- Tracing the resource back to its parent integration instance
- Permission and access control based on integration ownership
- Lifecycle management (enabling/disabling, updating, or uninstalling)
examples: ["5f9a7b2c3d4e5f6a7b8c9d0e"]
_connectorId:
type: string
format: objectId
x-celigo-refModel: connectors
readOnly: true
description: |-
Integration App (connector) that defines this resource. Present only on resources an
Integration App owns. The app locks parts of these documents, and a PUT that changes a
locked field answers 200 but keeps the stored value, so read the document back to confirm
a change.
examples: ["5e8d43a1b9e5a80011a35f1b"]
x-celigo-ai-guidance:
- |-
Read-only connector (Integration App template) reference; present only on
connector-owned resources. Never set or modify it when authoring resources.
references/common-schemas/lookups.yml
Lookups:
type: array
description: |
Configuration for value-to-value transformations using lookup tables.
**Purpose**
Lookups provide a way to translate values from one system to another. They transform
input values into output values using either static mapping tables or
dynamic lookup caches.
**Lookup mechanisms**
There are two distinct lookup mechanisms available:
1. **Static Lookups**: Define a simple key-value map object and store it as part of your resource
- Best for: Small, fixed sets of values that rarely change
- Implementation: Configure the `map` object with input-to-output value mappings
- Example: Country codes, status values, simple translations
2. **Dynamic Lookups**: Reference an existing 'Lookup Cache' resource in your Celigo account
- Best for: Large datasets, frequently changing values, or complex reference data
- Implementation: Configure `_lookupCacheId` to reference cached data maintained independently
- Example: Product catalogs, customer databases, pricing information
**Property usage**
There are two mutually exclusive ways to configure lookups, depending on which mechanism you choose:
1. **For Static Mappings**: Configure the `map` property with a direct key-value object
```json
"map": {"US": "United States", "CA": "Canada"}
```
2. **For Dynamic Lookups**: Configure the following properties:
- `_lookupCacheId`: Reference to the lookup cache resource
- `extract`: JSON path to extract specific value from the returned lookup object
**When to use**
Lookups are ideal for:
1. **Value Translation**: Mapping codes or IDs to human-readable values
2. **Data Enrichment**: Adding related information to records during processing
3. **Normalization**: Ensuring consistent formatting of values across systems
**Implementation details**
Lookups can be referenced in:
1. **Field Mappings**: Direct use in field transformation configurations
2. **Handlebars Templates**: Use within templates with the syntax:
```
{{lookup 'lookupName' record.fieldName}}
```
**Example usage**
```json
"lookups": [
{
"name": "countryCodeToName",
"map": {
"US": "United States",
"CA": "Canada",
"UK": "United Kingdom"
},
"default": "Unknown Country",
"allowFailures": true
},
{
"name": "productDetails",
"_lookupCacheId": "60a2c4e6f321d800129a1a3c",
"extract": "$.details.price",
"allowFailures": false
}
]
```
items:
type: object
properties:
name:
type: string
description: |
Unique identifier for the lookup table within this configuration.
This name must be unique within the scope where the lookup is defined and is used to reference
the lookup in handlebars templates with the syntax {{lookup 'name' value}}.
Choose descriptive names that indicate the transformation purpose, such as:
- "countryCodeToName" for country code to full name conversion
- "statusMapping" for status code translations
- "departmentCodes" for department code to name mapping
examples: ["countryCodeToName", "statusMapping", "departmentCodes"]
map:
type: [object, 'null']
description: |
The lookup mapping table as key-value pairs. The platform stores `null`
here on dynamic lookups, which resolve values at runtime instead of
from a static table.
This object contains the input values as keys and their corresponding
output values. When a input value matches a key in this object,
it will be replaced with the corresponding value.
The map should be kept to a reasonable size (typically under 100 entries)
for optimal performance. For larger mapping requirements, consider using
dynamic lookups instead.
Maps can include:
- Simple code to name conversions: {"US": "United States"}
- Status transformations: {"A": "Active", "I": "Inactive"}
- ID to name mappings: {"100": "Marketing", "200": "Sales"}
Values can be strings, numbers, or booleans, but all are stored as strings
in the configuration.
examples: [{"US": "United States", "UK": "United Kingdom", "CA": "Canada"}, {"A": "Active", "I": "Inactive", "P": "Pending"}, {"true": "Yes", "false": "No"}]
_lookupCacheId:
type: string
description: |
Reference to a LookupCache resource that contains the reference data for the lookup.
**Purpose**
This field connects the lookup to an external data source that has been cached in the system.
Unlike static lookups that use the `map` property, dynamic lookups can reference large datasets
or frequently changing information without requiring constant updates to the integration.
**Implementation details**
The LookupCache resource referenced by this ID contains:
- The data records to be used as a reference source
- Configuration for how the data should be indexed and accessed
- Caching parameters to balance performance with data freshness
**Usage patterns**
Commonly used to reference:
- Product catalogs or SKU databases
- Customer or account information
- Pricing tables or discount rules
- Complex business logic lookup tables
Format: 24-character hexadecimal string (MongoDB ObjectId)
format: objectid
x-celigo-refModel: lookupcaches
examples: ["60a2c4e6f321d800129a1a3c", "5f8d43a1b9e5a80011a35f2c"]
extract:
type: string
description: |
JSON path expression that extracts a specific value from the cached lookup object.
**Purpose**
When using dynamic lookups with a LookupCache, this JSON path identifies which field to extract
from the cached object after it has been retrieved using the lookup key.
**Implementation details**
- Must use JSON path syntax (similar to mapping extract fields)
- Operates on the cached object returned by the lookup operation
- Examples:
- "$.name" - Extract the name field from the top level
- "$.details.price" - Extract a nested price field
- "$.attributes[0].value" - Extract a value from the first element of an array
**Usage scenario**
When a lookup cache contains complex objects:
```json
// Cache entry for key "PROD-123":
{
"id": "PROD-123",
"name": "Premium Widget",
"details": {
"price": 99.99,
"currency": "USD",
"inStock": true
}
}
```
Setting extract to "$.details.price" would return 99.99 as the lookup result.
If no extract is provided, the entire cached object is returned as the lookup result.
examples: ["$.name", "$.details.price", "$.attributes.color", "$.category.name"]
default:
type: [string, 'null']
description: |
Default value to use when the source value is not found in the lookup map.
The platform stores `null` here when no default is configured.
This value is used as a fallback when:
1. The source value doesn't match any key in the map
2. allowFailures is true or omitted
Setting an appropriate default helps prevent flow failures due to unexpected
values and provides predictable behavior for edge cases.
Common default patterns include:
- Descriptive unknowns: "Unknown Country", "Unspecified Status"
- Original value indicators: "{Original Value}", "No mapping found"
- Neutral values: "Other", "N/A", "Miscellaneous"
A save refuses a default together with an explicit `allowFailures: false`
(422 `Lookups with default set can not have allowFailures explicitly set to
false.`): a lookup either fails a miss (`allowFailures: false`, no default) or
falls back to its default.
examples: ["Unknown", "Other", "N/A", "Unmapped Value", "{Original Value}"]
x-celigo-ai-guidance:
- >-
Omit this field entirely when no fallback value is intended: a
miss the lookup lets through then leaves the mapped field out,
and `allowFailures: false` (which a save refuses alongside a
default) fails it. An empty string is a value, not "no default" — the
save accepts `default: ""`, and an allowed miss writes `""`.
- >-
A default sends every unmapped value with no error, so add the
missing key to the map when the destination needs the real value;
keys match the extracted value exactly (case, spacing and suffixes
count), and a save replaces the whole map.
allowFailures:
type: [boolean, 'null']
description: |
When true, missing lookup values will use the default value rather than causing an error.
**Behavior control**
This field determines how the system handles source values that don't exist in the map:
- true: Use the default value for missing mappings and continue processing
- false: Treat missing mappings as errors, failing the record (`lookup_error` for a
static map, `lookupcache_evaluation_failed` for a lookup cache)
- omitted or null: the record doesn't fail either. On an import a static lookup writes
its default if it has one, and a static or cache lookup without one leaves the
mapped field out of the output, with no error
**Recommendation**
Set this to true when:
- New source values might appear over time
- Data quality issues could introduce unexpected values
- Processing should continue even with imperfect mapping
Set this to false when:
- Complete data accuracy is critical
- All possible source values are known and controlled
- Missing mappings indicate serious data problems that should be addressed
A default raises no error: records that miss go out with the default value and
nothing reports them. Use true only when that fallback is acceptable at the
destination, and set false explicitly when a missing value must stop the record.
examples: [true, false]
x-celigo-ai-guidance:
- >-
Never leave this field out to mean "fail on a miss": on an import, an omitted
or null value lets a missing key through without an error (with the default,
or without the mapped field when there is none). Only an explicit false fails
the record.
references/common-schemas/mappings.yml
Mappings:
type: array
description: |
Array of field mapping configurations for transforming data from one format into another.
**Guidance**
This schema is designed around RECURSION as its core architectural principle. Understanding this recursive
nature is essential for building effective mappings:
1. The schema is self-referential by design - a mapping can contain nested mappings of the same structure
2. Complex data structures (nested objects and arrays of objects, to any depth) are ALL
handled through this recursive pattern; an array of arrays is not (build one in a script hook)
3. Each mapping handles one level of the data structure; deeper levels are handled by nested mappings
When generating mappings programmatically:
- For simple fields (string, number, boolean): Create single mapping objects
- For objects: Create a parent mapping with nested 'mappings' array containing child field mappings
- For arrays: Use 'buildArrayHelper' with extract paths defining array inputs and
recursive 'mappings' to define object structures
The system will process these nested structures recursively during runtime, ensuring proper construction
of complex hierarchical data while maintaining excellent performance.
On save, a leaf row with neither a source (`extract`, `hardCodedValue`, `lookupName`) nor a
destination (`generate`) is dropped silently, at every nesting level, and so is an object or
object-array row left with no source, no destination and no remaining child `mappings` or
`buildArrayHelper` entries; a row with only one of the two, and a source-less, destination-less
wrapper that still has children, are kept.
The shape of the array is the mapper's **JSON structure**. Named top-level items produce an object
record (`record { }`). Exactly one top-level item with `dataType: objectarray` and no `generate` is
the `rows [ ]` structure: the mapped record is an array — usually one element per element of the
source array that root's `buildArrayHelper` iterates — Handlebars templates such as `http.body`
bind it as `rows` instead of `record`, and the mapping editor shows that row's destination disabled.
x-celigo-ai-guidance:
- |-
Rows that exist in the saved document — including `Draft` and `Inactive` rows and
rows with only a source or only a destination — are the user's state: an edit that
re-sends the tree must carry them through unchanged unless the user asked to change
them. Only a row with neither source nor destination — and, for an object or
object-array row, no remaining children — can be left out; the save would drop it
anyway. A wrapper with children but no `generate` is kept by the save, so keep it.
- |-
A single unnamed top-level `objectarray` item is the `rows [ ]` JSON structure, not a
missing destination: keep it unnamed when editing (naming it turns the output into an
object, and a template that iterates `rows` then renders nothing), and author that
shape only when the request body must be an array or a template iterates `rows`.
The editor's save check accepts the unnamed root only in the `rows [ ]` structure;
any other row with a source and no `generate` fails with "One or more destination
field values not entered".
items:
type: object
properties:
generate:
type: string
description: |
**Purpose**
Defines the target field name in the output object/record.
**Guidance**
This is the PRIMARY FIELD that identifies the output property being created:
- For regular fields: Set to the exact property name (e.g., "firstName", "price", "isActive")
- For object fields: Set to the object property name, then add child mappings in the 'mappings' array
- For array fields: Set to the array property name, then configure 'buildArrayHelper'
- One structural blank: the single top-level `objectarray` item of a `rows [ ]` mapping (the
array output structure — see the array's description). A child-bearing wrapper and a source-only row that
already exist in a saved document also stay unnamed when the tree is re-sent — filling in a
name changes the output; it does not complete the row
IMPORTANT: Do NOT use dot notation (e.g., "customer.firstName") in this field. Instead, create proper
hierarchical structure with nested mappings:
```json
{
"generate": "customer",
"dataType": "object",
"status": "Active",
"mappings": [
{"generate": "firstName", "dataType": "string", "extract": "$.name.first", "status": "Active"}
]
}
```
When parsing existing mappings, an empty `generate` on a row inside an `arrayarray` row's
`buildArrayHelper` entry is that row's inner array (`arrayarray` fails at run time; see
`dataType`); other unnamed rows are the wrappers and source-only rows described above; an
empty `generate` on the ONLY top-level item, with `dataType: objectarray`, is the `rows [ ]` JSON
structure — array output, bound as `rows` in templates — and is intentional, not a missing name.
examples: ["name", "address", "items"]
dataType:
type: string
description: |
**Purpose**
Explicitly declares the data type of the output field, controlling how data is processed and structured.
**Guidance**
This is a REQUIRED field that fundamentally determines mapping behavior:
**Simple Types (direct value mapping)**
- `string`: Text values, converts other types to string representation
- `number`: Numeric values. Converts a numeric string (`"24"` → `24`,
`"007"` → `7`) and leaves the field out for an empty string; a value
holding letters or a thousands separator (`"1,250.00"`) fails the
record with `non_parsable_string` (`<generate>: String contains alpha
characters`)
- `boolean`: True/false values, converts truthy/falsy values
Dates are represented as strings — use `string` for date fields and
drive the parsing/formatting through the `extractDateFormat` /
`generateDateFormat` / `extractDateTimezone` / `generateDateTimezone`
fields. There is no separate `date` enum value.
**Complex Types (require additional configuration)**
- `object`: Creates a nested object from child mappings in the 'mappings' array, or
copies a whole object with `extract` (one or the other, not both)
**Array Types**
- `stringarray`: Array of strings
- `numberarray`: Array of numbers
- `booleanarray`: Array of booleans
- `objectarray`: Array of objects (most common array type)
`arrayarray` is not a type the mapper recognizes: a save accepts it, but every record
then fails with `Destination datatype is a required field.` Build an array of arrays
in a script hook instead.
Array dataTypes are populated two ways: construct or iterate through
`buildArrayHelper` entries, or set a constant with `hardCodedValue`. The
row's own `extract` is never read on an array type: a primitive-array row
that has only an `extract` (JSONPath, Handlebars or a literal) yields
nothing, with no error, and an `objectarray` row without `buildArrayHelper`
or `hardCodedValue` is refused with 422 ("buildArrayHelper" or
"hardCodedValue" must be present for object array datatype.). To copy a
source array through unchanged, use one entry that iterates it
(`buildArrayHelper: [{"extract": "$.items[*]"}]`).
IMPORTANT: The dataType controls which additional fields are relevant:
- For date-like string fields: extractDateFormat, generateDateFormat, etc. become relevant
- For object types: 'mappings' array becomes relevant
- For array types: `buildArrayHelper` or `hardCodedValue` (see above)
When analyzing existing mappings or generating new ones, always check dataType first
to understand what additional fields should be present.
x-celigo-ai-guidance:
- |-
Missing `dataType` is rejected at save with validation error
`[mandatory_destination_datatype] Destination datatype is a
required field.`
- |-
Give each row the type the destination field expects; `"string"` is
not a safe default. It accepts any source value, but sends a numeric
source as text (`"24"`), which a destination that checks JSON types
refuses. Map numeric destination fields as `number`, which fails a
record (`non_parsable_string`) only when the value holds letters or a
thousands separator.
enum:
- string
- number
- boolean
- object
- stringarray
- numberarray
- booleanarray
- objectarray
- arrayarray
x-celigo-deprecated-enum-values:
- arrayarray
x-enumDescriptions:
string: Text values; converts other types to their string representation.
number: Numeric values; converts numeric strings, and fails the record (`non_parsable_string`) on letters or a thousands separator.
boolean: True/false values; converts truthy/falsy values.
object: Nested object built from child `mappings`, or copied whole with `extract` (not both).
stringarray: Array of string values.
numberarray: Array of numeric values.
booleanarray: Array of boolean values.
objectarray: Array of objects, the most common array type.
arrayarray: "Deprecated: not a type the mapper recognizes; a row that uses it fails"
examples: ["string", "number", "object", "objectarray"]
extract:
type: string
description: |
**Purpose**
Defines how to retrieve data from the input record to populate the output field.
**Guidance**
This field supports THREE DISTINCT PATTERNS that are easily distinguished:
**1. json Path Syntax**
- MUST start with '$' — the record root. An object record is addressed as
'$.field'; a grouped (array) record as '$[0].field' / '$[*].field'
- Used for precisely targeting data in structured JSON objects
- Examples: '$.customer.firstName', '$.items[0].price', '$.addresses[*].street',
'$[*].Tax' (every row of a grouped record)
- Wildcards like [*] extract multiple values/objects
- A key that contains a dot needs bracket notation: `$['P.O. Number']` (or
`$["P.O. Number"]`, `$.[P.O. Number]`). `$.P.O. Number` is read as a nested
path: it maps nothing and the record still succeeds. Spaces, hyphens and `#`
need no brackets (`$.Ship To Name`, `$.Tracking #`)
- Filter expressions such as `$.items[?(@.active)]` fail the record with
"Eval [?(expr)] prevented in JSONPath expression."
```json
"extract": "$.customer.addresses[*]" // Extracts all addresses
```
**2. Handlebars Template Syntax**
- Contains '{{' and '}}' pattern
- Evaluated by the AFE 2.0 handlebars template engine
- Can include logic, formatting, and computation
- Access input record fields with {{record.fieldName}} notation; a grouped
(array) record binds as `rows` instead — {{rows.0.fieldName}} / {{#each rows}}
- Examples: "{{record.firstName}} {{record.lastName}}", "{{#if record.isActive}}Active{{else}}Inactive{{/if}}"
- Valid on scalars and on `object` rows, where the rendered output must
parse into an object. On an array row the template goes in a
`buildArrayHelper` entry instead: on an `objectarray` row's own
`extract` it is refused with 422 ("buildArrayHelper" or "hardCodedValue"
must be present for object array datatype.), and on a primitive-array
row's own `extract` it yields nothing (the field is left out) without an
error
```json
"extract": "{{record.price}} {{record.currency}}" // Combines two fields
```
**3. Hard-Coded Value (literal string)**
- Does NOT start with '$'
- Does NOT contain handlebars '{{' syntax
- System treats it as a literal string value
- VERY COMMON for setting static/constant values
- Examples: "Active", "USD", "Completed", "true"
```json
"extract": "primary" // Sets field value to the literal string "primary"
"extract": "true" // Sets field value to the literal string "true"
"extract": "N/A" // Sets field value to the literal string "N/A"
```
This third pattern is the simplest and most efficient way to set hard-coded values in your mappings.
AI agents should use this pattern whenever a field needs a static value that doesn't come from
the input record or require computation.
**Important implementation details**
- JSON path patterns ALWAYS execute from the TOP-LEVEL root of the input record
- The system maintains this context even in deeply nested mappings
- For object mappings without child mappings, extract should return a complete object
- An `object` row takes `extract` (to copy a whole object) or child `mappings`, not
both: a row with both fails with 422 (`"extract" should not be present if
"mappings" field is present`). A `buildArrayHelper` entry is different: there
`extract` and `mappings` work together.
- The root is the record as it arrives: an object record roots at `$.field`; a
grouped record (an array of rows — grouped exports, file key columns, NetSuite
grouped saved searches) roots at the array, so its rows are addressed as
`$[0].field` (one row) or `$[*].field` (every row). A root that contradicts the
record's shape resolves to nothing without an error.
For most simple field-to-field mappings, prefer JSON path syntax for its clarity and performance.
For hard-coded values, simply use the literal string as the extract value.
x-celigo-ai-guidance:
- |-
## GROUPED (ARRAY) RECORDS
Decide the JSONPath root from the record's shape before writing
any extract. When the record is an array of rows, `$.field`,
`$.properties.field` and `$.rows[*].field` all resolve to
nothing — the processor still reports success and the request
body ships as `{}`. Address rows as `$[0].field` (that row's
value, typed) or `$[*].field` (every row: joined with commas
into a `string` destination; rejected with
`incompatible_destination_conversion` into a `number`
destination). Column labels with spaces work as
`$[*][Order Number]`, `$[*].[Order Number]`, or
`$[*]['Order Number']`. For one destination element per row
the two array kinds take different helper forms: a primitive
array (`stringarray` / `numberarray`) uses `buildArrayHelper:
[{"extract": "$[*].field"}]`; an `objectarray` uses
`buildArrayHelper: [{"extract": "$[*]", "mappings": [...]}]`
with the children addressing the row relatively (`$.field`,
`{{rows.field}}`). An `extract` placed directly on the
array-typed field yields nothing. Handlebars
extracts bind the array record as `rows` (`{{rows.0.field}}`),
an object record as `record`. The reverse holds too: `$[*]` on
an object record iterates the object's values, not rows.
- |-
## SIZE LIMIT
The platform's import save validator caps this field at **1024
characters**. Saves with a longer `extract` are rejected
with `[size_limit_exceeded] The maximum size of the field:
mappings.<index>.extract in the Import document should be
1024 characters`. This typically only matters for
Handlebars templates that build complex strings inline -- if
a template is approaching the cap, prefer one of:
- **Decompose into nested mappings** (set `dataType:
"object"` and use child `mappings` -- each child gets
its own 1024-char budget).
- **Move the logic to a `preMap` hook** so the heavy
transformation runs in script and the mapping just reads
`record.<computed>`.
- **Use a lookup** (`lookupName` + a small `lookups` entry)
for value translation that would otherwise need a long
`{{#if ...}}{{else if ...}}` chain.
examples: ["$.fullName", "{{record.fullName}}", "primary", "$.lineItems[*].unitPrice"]
extractDateFormat:
type: string
description: |
Specifies the format pattern of the input date string to ensure proper parsing.
Used on string-typed mappings whose `extract` yields a date. Uses
Moment.js-compatible formatting tokens to describe how the incoming date
string is structured.
examples: ["YYYY-MM-DD", "MM/DD/YYYY", "DD-MMM-YYYY HH:mm:ss"]
extractDateTimezone:
type: string
description: |
Specifies the timezone of the input date string using Olson/IANA timezone identifiers.
Used on string-typed mappings whose `extract` yields a date; tells the system
how to interpret timestamp values from the input system.
examples: ["UTC", "America/New_York", "Europe/London"]
generateDateFormat:
type: string
description: |
Specifies the output format pattern when generating a date string or converting
from a Date type to String type.
Uses Moment.js-compatible formatting tokens to define the structure of the resulting
date string.
examples: ["YYYY-MM-DD", "MM/DD/YYYY", "DD-MMM-YYYY HH:mm:ss"]
generateDateTimezone:
type: string
description: |
Specifies the timezone to apply when generating or converting timestamp values
using Olson/IANA timezone identifiers.
Controls timezone conversion when producing date output.
examples: ["UTC", "America/New_York", "Europe/London"]
default:
type: [string, 'null']
description: |
Specifies a fallback value to use when extract returns empty/null or when conditional
logic fails and no other mapping supplies a value.
Explicit JSON `null` is itself a valid fallback: the destination field is written as
JSON null when the extract yields nothing (the Mapper UI's "Use null as default value"
action). Omitting the key entirely means no fallback — the field is left out of the
output when the extract is missing or an empty string (the UI's "Do nothing" action),
while a null source is still written as null unless `conditional.when` is
`extract_not_empty`. Failing the record instead is `conditional.when:
fail_record_when_empty`, which excludes a default.
x-celigo-ai-guidance:
- |-
`"default": null` and `"default": ""` are distinct saved values, not
absent keys: null renders the destination field as JSON null and the
empty string renders it as `""` when the extract yields nothing.
Preserve them verbatim when editing or migrating a mapping — dropping
the key changes runtime behavior from "use null/empty string as
default value" to "do nothing" (a missing or `""` source is left out
of the output, while a null source is written as null unless
`conditional.when` is `extract_not_empty`).
- |-
The four "Action to take if source field has no value" choices map to:
do nothing → no `default` key (add `conditional.when:
extract_not_empty` to leave a null source out too; without it a null
source is written as null); use null → `"default": null`; use empty string or
a custom value → `"default": "<value>"`; cause the record to fail →
`conditional.when: fail_record_when_empty` with no `default` key.
examples: ["N/A", "0", "false"]
lookupName:
type: string
description: |
**Purpose**
References a lookup table for transforming values during the mapping process.
**Usage**
The lookupName refers to a named lookup defined in the lookups array of the same resource.
On an HTTP import that is `http.lookups`: a `lookups` array sent only at the top level
is dropped on save. On an import, a name that matches no lookup fails every record
with "Unable to find lookup in mapping configs - <name>", and a key the lookup
doesn't hold follows the lookup's `allowFailures`: false fails the record; true, or
the field left out, lets the record succeed with the lookup's `default`, or without
the mapped field when there is none.
```json
{
"generate": "countryName",
"dataType": "string",
"extract": "$.countryCode",
"lookupName": "countryCodeToName"
}
```
During processing, the system:
1. Extracts the value from the input record (e.g., "US")
2. Finds the lookup table with the specified name
3. Uses the extracted value as a key in the lookup
4. Returns the corresponding value (e.g., "United States")
**Benefits**
- **Standardization**: Ensures consistent value translation across mappings
- **Centralization**: Define translations once and reference them in multiple places
- **Maintainability**: Update all mappings by changing the lookup definition
- **Readability**: Makes mappings more descriptive and self-documenting
The specific lookup capabilities depend on the context where mappings are used.
examples: ["countryCodeToName", "statusMapping", "departmentCodes"]
hardCodedValue:
type: string
description: |
Constant written to the destination instead of an extracted value. On a primitive
array (`stringarray`, `numberarray`, `booleanarray`) the string is split on commas,
verbatim and untrimmed, into elements of the array's type (`"A,B"` gives `["A", "B"]`).
On an `object` it must be a JSON object string and on an `objectarray` a JSON array
string (an object string becomes one element); the save parses it and refuses non-JSON
or the wrong shape with 422, and an array row with a constant needs no `buildArrayHelper`.
examples: ["USD", "A,B", "[{\"code\": \"A\"}, {\"code\": \"B\"}]"]
description:
type: string
description: |
Optional free-text annotation that appears in the Mapper sidebar to provide context about
the mapping's purpose for collaboration and documentation.
Has no functional impact on the mapping behavior.
examples: ["Maps customer name", "Converts price to target currency", "Formats date for target system"]
sourceDataType:
type: string
description: |
Declares the JSON type of the value produced by `extract`, before any
conversion to `dataType`. Same enum as `dataType`. Set on leaf mappings
only — parent mappings (with child `mappings` or `buildArrayHelper`)
have no extracted value of their own; the children carry their own
`sourceDataType`.
For date fields use `string` (JSON represents dates as strings); the
parsing/formatting lives in `extractDateFormat` / `generateDateFormat` /
`extractDateTimezone` / `generateDateTimezone`.
enum:
- string
- number
- boolean
- object
- stringarray
- numberarray
- booleanarray
- objectarray
- arrayarray
x-enumDescriptions:
string: Source value is a text string.
number: Source value is numeric.
boolean: Source value is true or false.
object: Source value is a JSON object.
stringarray: Source value is an array of strings.
numberarray: Source value is an array of numbers.
booleanarray: Source value is an array of booleans.
objectarray: Source value is an array of objects.
arrayarray: Source value is an array of arrays.
examples: ["string", "number", "boolean", "object"]
mappings:
type: array
description: |
**Purpose**
Enables recursive definition of nested object structures through child mapping objects.
**Guidance**
This is the KEY FIELD that implements the recursive pattern at the core of this schema:
**When to Use**
- REQUIRED when dataType = "object" (unless you are copying an entire object from the input record)
- REQUIRED in buildArrayHelper.mappings when defining complex object array elements
- NEVER used with simple types (string, number, boolean)
**Behavior**
- Each mapping in this array becomes a property of the parent object
- The full Mappings schema is repeated recursively at each level
- Can be nested to any depth for complex hierarchical structures
**Context Handling**
- Each level of nesting changes the mapping CONTEXT for 'generate'
- The extraction CONTEXT always remains the original input record
- This means child mappings can pull data from anywhere in the input record
**Common Patterns**
**Nested Objects**
```json
{
"generate": "customer",
"dataType": "object",
"mappings": [
{
"generate": "contact",
"dataType": "object",
"mappings": [
{"generate": "email", "dataType": "string", "extract": "$.customerEmail"}
]
}
]
}
```
**Multiple Fields in Object**
```json
{
"generate": "address",
"dataType": "object",
"mappings": [
{"generate": "street", "dataType": "string", "extract": "$.address.line1"},
{"generate": "city", "dataType": "string", "extract": "$.address.city"},
{"generate": "country", "dataType": "string", "extract": "$.address.country"}
]
}
```
IMPORTANT: When analyzing or generating mappings, ALWAYS check if parent.dataType = "object"
or if you're inside buildArrayHelper.mappings for objectarray elements. These are the only
valid contexts for the mappings array.
items:
$ref: '#/Mappings/items'
examples: [[{"generate": "firstName", "extract": "$.name.first", "dataType": "string", "status": "Active"}, {"generate": "lastName", "extract": "$.name.last", "dataType": "string", "status": "Active"}]]
buildArrayHelper:
type: array
x-celigo-ai-guidance:
- |-
Entry-level `extract` accepts an iterating array path
(`$.items[*]`), `"$"` (the record root), or omission — a
constructed entry (child `mappings` with the extract omitted or
`"$"`; the two spellings are equivalent) contributes exactly one
element, and entries append in order. On a grouped (array)
record the record itself is the array to iterate: `"$[*]"`
yields one `objectarray` element per row (children address the
row as `$.field`), and `"$[*].field"` yields one primitive-array
value per row. Indexed or comma-joined
entry extracts (`"$[0]"`, `"$[0],$[0]"`) are not valid: the
processor renders the entire destination array as null with no
validation error. To emit a fixed number of static elements, add
that many sibling constructed entries — never index the extract.
- |-
An entry's `extract` can be a Handlebars template. On an
`objectarray` it must render a JSON object (one element) or a JSON
array (one element per item). On a primitive array each entry
renders one element, so a serialized array becomes one string
element. A bare literal entry extract is refused with 422 ("extract
only supports json path or handlebars expression"); set a constant
with the row's `hardCodedValue` instead.
description: |
**Purpose**
Configures how to construct arrays in the output record, handling various array types and inputs.
**Guidance**
This is the mechanism for building array data types:
**When to Use**
- REQUIRED when dataType ends with "array" (stringarray, objectarray, etc.), unless the
row sets `hardCodedValue`: an array row's own `extract` is never read (see `dataType`)
- Each entry in this array contributes elements to the output array
- Multiple entries allow combining data from different input arrays
**Array Type Handling**
**For Simple Arrays (stringarray, numberarray, booleanarray)**
- Only the 'extract' field is used to pull values
- JSON path with wildcards (e.g., $.items[*].name) returns multiple values;
on a grouped (array) record the record itself is the array, so the
path roots at it ($[*].name — one value per row)
- Each result is converted to the appropriate primitive type
```json
{
"generate": "productNames",
"dataType": "stringarray",
"buildArrayHelper": [
{"extract": "$.products[*].name"}
]
}
```
**For Object Arrays (objectarray) - three patterns**
1. Extract Only (existing objects):
```json
{
"generate": "contacts",
"dataType": "objectarray",
"buildArrayHelper": [
{"extract": "$.account.primaryContacts[*]"}, // Pull primary contact objects
{"extract": "$.account.secondaryContacts[*]"}, // Pull secondary contact objects
{"extract": "$.vendor.contactPersons[*]"}, // Pull vendor contact objects
{"extract": "$.subsidiaries[*].mainContact"} // Pull main contact from each subsidiary
]
}
```
2. Mappings Only (constructed object):
```json
{
"generate": "contactInfo",
"dataType": "objectarray",
"buildArrayHelper": [
{
"mappings": [ // Creates one object in the array
{"generate": "type", "dataType": "string", "extract": "primary"},
{"generate": "email", "dataType": "string", "extract": "$.primaryEmail"}
]
},
{
"mappings": [ // Creates another object in the array
{"generate": "type", "dataType": "string", "extract": "secondary"},
{"generate": "email", "dataType": "string", "extract": "$.secondaryEmail"}
]
}
]
}
```
Each constructed entry contributes exactly one element, in entry
order; writing the entry with `"extract": "$"` is equivalent to
omitting the extract (both anchor the element at the record root).
A fixed number of static elements is built from that many sibling
constructed entries.
3. Extract AND Mappings (transform input arrays):
```json
{
"generate": "lineItems",
"dataType": "objectarray",
"buildArrayHelper": [
{
"extract": "$.order.items[*]", // For each item in the array
"mappings": [ // Transform to this structure using the composite object
{"generate": "sku", "dataType": "string", "extract": "$.order.items.productId"}, // Notice: items is singular
{"generate": "quantity", "dataType": "number", "extract": "$.order.items.qty"}, // Notice: items is singular
{"generate": "orderNumber", "dataType": "string", "extract": "$.order.id"}, // Access parent data
{"generate": "customerName", "dataType": "string", "extract": "$.customerName"} // Access root data
]
}
]
}
```
**Arrays of Arrays**
- The mapper has no working type for them: `arrayarray` saves, but every record
fails with `Destination datatype is a required field.` Build an array of arrays
in a script hook instead.
**Important details**
- When both extract and mappings are provided, the system creates special composite objects
that maintain hierarchical context during processing
- This enables accessing both the current array element AND its parent context
- An ITERATING entry's extract MUST be a JSON path that iterates an array:
'$.items[*]' when the array is a field of an object record, or '$[*]'
(objectarray) / '$[*].field' (primitive arrays) when the record itself is a
grouped array of rows. A CONSTRUCTED entry (child mappings building one
element) instead omits extract or sets it to exactly "$" — never rewrite "$"
into an indexed or comma-joined form ('$[0]', '$[0],$[0]'): those render the
entire destination array as null without raising a validation error
- Each array helper entry acts independently, potentially adding multiple elements
The buildArrayHelper is the most complex part of the mappings system - always analyze the
dataType first to understand which pattern is appropriate.
items:
type: object
properties:
extract:
type: string
description: |
JSON path expression that identifies the input array or values to extract.
For objectarray with mappings, this defines which input objects to iterate through.
The JSON path must return either a single object or an array of objects.
For a CONSTRUCTED entry (child mappings building exactly one element from
record-root fields and constants), omit this field or set it to exactly "$" —
the two spellings are equivalent. Indexed or comma-joined forms ('$[0]',
'$[0],$[0]') are not valid at the entry level: the processor renders the
entire destination array as null without raising a validation error.
The system creates special composite objects during processing to maintain
hierarchical relationships, allowing easy access to both the current array item
and its parent contexts.
An iterating entry keeps the `[*]`: `$.line_items` without it yields one empty
element per item, with no error. Child extracts drop the `[*]` but keep the
array's path (`$.line_items.sku`); `$.line_items[*].sku` maps nothing, and an
element-relative `$.sku` reads the record root instead of the element.
examples: ["$.items[*]", "$.order.lineItems[*]", "$.parents[*].children[*]", "$"]
name:
type: string
maxLength: 200
description: |
Display label for this source object in the mapper, so the sources of
an object array can be told apart instead of all showing as `$`.
Has no effect at runtime; stored only when non-empty.
x-celigo-ai-guidance:
- |-
Carry an existing `name` through every edit — it is the user's label
for the tab, and an entry sent without it loses the label. Duplicate
names across entries are allowed. Set one only when the user names
the source object; never invent labels.
examples: ["Billing address", "Shipping address"]
sourceDataType:
type: string
description: |
Declares the JSON type of the input array being iterated, to ensure
proper type handling during array construction. Same enum as `dataType`.
enum:
- string
- number
- boolean
- object
- stringarray
- numberarray
- booleanarray
- objectarray
- arrayarray
x-enumDescriptions:
string: Input array contains string elements.
number: Input array contains numeric elements.
boolean: Input array contains boolean elements.
object: Input array contains object elements.
stringarray: Input array contains arrays of strings.
numberarray: Input array contains arrays of numbers.
booleanarray: Input array contains arrays of booleans.
objectarray: Input array contains arrays of objects.
arrayarray: Input array contains arrays of arrays.
examples: ["string", "number", "boolean", "object"]
default:
type: [string, 'null']
description: |
Specifies a fallback value when the extracted array element is empty or
not found in the input data.
Explicit JSON `null` is itself a valid fallback (the element is written as
JSON null); omitting the key means no fallback is applied.
examples: ["N/A", "0", "false"]
conditional:
type: object
description: |
Defines conditional rules for including each array element in the result.
x-celigo-ai-guidance:
- |-
Judged against this entry's own `extract` (the array source). The
design scopes the UI's tab-level "Action to take if source field has
no value" to copy-as-is tabs (no child `mappings`); the API accepts
`fail_record_when_empty` on a mapped entry as well, and the mapper
processor honors it there — the record fails when that array is
missing or empty (`errors[]` entry `code: extract_empty`, `fatal:
true`, message `<generate>: Source field has no value`; probed
2026-10-09). Say so when writing it on a mapped entry: the user may
not find the setting in the mapper's tab. To fail on an empty field
inside each element, set the action on that child mapping instead.
properties:
when:
type: string
description: |
Condition judged on the entry's extract — skip the elements the
extract does not return, or fail the whole record when the source
array is missing or empty.
enum:
- extract_not_empty
- fail_record_when_empty
x-enumDescriptions:
extract_not_empty: Include an array element only when the extract returns a non-empty value for it
fail_record_when_empty: Fail the record when the entry's source array is missing or empty
examples: ["extract_not_empty"]
mappings:
type: array
description: |
Contains recursive mapping definitions for complex array element transformations.
**Composite object mechanism**
When both 'extract' and 'mappings' are used together, the system implements a sophisticated
"composite object" approach that is crucial for AI agents to understand:
1. The system starts with the complete input record
2. For each array element matched by the extract path, it creates a modified version of
the input record where:
- Array paths in the extract JSON path are REPLACED with single objects
- Each array ([]) in the path is converted to a single object ({})
- This preserves the hierarchical relationship between nested arrays
**Example**
Given an input record:
```json
{
"customer": {
"name": "John Doe",
"orders": [
{
"id": "O-001",
"items": [
{"sku": "ABC", "qty": 2},
{"sku": "XYZ", "qty": 1}
]
},
{
"id": "O-002",
"items": [
{"sku": "DEF", "qty": 3}
]
}
]
}
}
```
With extract path: `$.customer.orders[*].items[*]`
For each item, the system creates a composite object like:
```json
{
"customer": {
"name": "John Doe",
"orders": { // Note: Array replaced with single object
"id": "O-001",
"items": { // Note: Array replaced with single object
"sku": "ABC",
"qty": 2
}
}
}
}
```
Then in your mappings, you can access:
- The current item: `$.customer.orders.items.sku`
- The parent order: `$.customer.orders.id`
- Top-level data: `$.customer.name`
This approach allows for precise mapping from deeply nested structures while maintaining
access to all contextual parent data, without requiring complex array index management.
**Implementation guidance**
When implementing the composite object mechanism:
1. Analyze the extract path to identify all array patterns (`[*]` or `[number]`)
2. For each array in the path, understand that it will be replaced with a single object
3. In the mappings, use paths that reference these arrays as if they were objects
4. Remember that every mapping still has access to the full input record context
5. This mechanism is especially powerful when mapping hierarchical data like:
- Order → Line Items → Taxes/Discounts
- Customer → Addresses → Address Lines
- Invoice → Line Items → Serial Numbers
The extract path effectively tells the system "iterate through these arrays",
while the composite object mechanism ensures you can still access both the
current array item AND its parent context during mapping.
items:
$ref: '#/Mappings/items'
examples: [[{"generate": "itemName", "extract": "$.name", "dataType": "string", "status": "Active"}, {"generate": "itemPrice", "extract": "$.price", "dataType": "number", "status": "Active"}]]
# The save rejects the fail-record action paired with a default on an
# entry ([unsupported_mapping]); the entry schema has no lookupName or
# hardCodedValue, so this is the only pair at this level.
if:
properties:
conditional:
properties:
when:
const: fail_record_when_empty
required: [when]
required: [conditional]
then:
not:
required: [default]
status:
type: string
description: |
**Purpose**
Controls whether the mapping is applied. Send it on every mapping entry.
**Guidance**
Only `Active` rows run. `Draft` and `Inactive` rows keep their
configuration, are saved with minimal validation, and are skipped at
runtime and in preview; a non-Active parent skips its whole subtree
whatever its children say. Imports (on save and in the mapper processor)
and API response mappings reject a row missing this field (validation
error: "Mapping object must have status field present."); a Tool's
output mapping saves and runs one, which is why `required` doesn't
list it.
x-celigo-ai-guidance:
- |-
Emit `"Active"` for mappings that should run; `"Draft"` is also
accepted and saves the mapping while bypassing the Active-only
field validations. The value is case-sensitive. The API rejects:
- Missing field: `[status_field_absent] Mapping object must have status field present.`
- Wrong case (e.g. `"active"`): `[invalid_status_enums] active is not a valid enum for status field.`
- |-
`"Inactive"` is the user's "Set inactive" on a row: use it only when
asked to disable or park a row, never to mean incomplete (that is
`"Draft"`). Setting a parent `Inactive` means no `Active` row may
remain under it — the save rejects an `Active` row under an
`Inactive` parent (`[inactive_parent_active_child]` on `mappings`,
no message text) and never cascades or promotes on its own; `Draft`
descendants are kept as they are, and a `Draft` parent does not gate
its children. The UI's Set inactive on a parent marks the parent and
every child `Inactive`. Re-enabling a child
means re-enabling its ancestors in the same payload. Edits that keep
existing rows must carry their stored status forward — rewriting a
`Draft` or `Inactive` row to `Active` turns a row the user parked
back on.
enum:
- Active
- Draft
- Inactive
x-enumDescriptions:
Active: The mapping is enabled and applied during processing
Draft: In-progress mapping; saved without the Active-only field validations and skipped at runtime
Inactive: Disabled by the user; configuration kept, skipped at runtime and in preview
examples: ["Active"]
conditional:
type: object
description: |
**Purpose**
Defines conditional processing rules for the entire mapping.
**Guidance**
These conditions determine whether the mapping is applied based on record
state or field content:
**When to Use**
- When a mapping should only be applied in specific circumstances
- To implement conditional logic without using complex handlebars expressions
- For creating mappings that only run during create or update operations
**Available Conditions**
- `record_created`: Apply only when creating a new record
Useful for setting initial values that should not be overwritten during updates
- `record_updated`: Apply only when updating an existing record
Useful for transformation logic that should only run during updates
- `extract_not_empty`: Apply only when the extract field returns a value
Useful for conditional mapping based on input data availability
- `fail_record_when_empty`: Fail the record when the extract field returns
no value (the UI's "Cause record to fail" action when the source field
has no value)
**Example**
```json
{
"generate": "statusMessage",
"dataType": "string",
"status": "Active",
"extract": "$.status.message",
"conditional": {
"when": "extract_not_empty" // Only map when status.message exists
}
}
```
x-celigo-ai-guidance:
- |-
`when` holds one value, and two different settings share it: the
composite-import operation gate (`record_created` / `record_updated`,
the UI's "Only perform mapping when") and the empty-source action
(`extract_not_empty` / `fail_record_when_empty`, the UI's "Action to
take if source field has no value"). A row cannot carry both — writing
one replaces the other and the save succeeds, so a composite import
row that needs both has to choose. Never author `fail_record_when_empty`
on a row that already carries `record_created` or `record_updated`.
properties:
when:
type: string
description: |
Condition that gates this mapping — either an operation gate on a
composite (create-or-update) import or the action to take when the
extract returns no value. Empty means null, undefined or an empty
string; `0`, `false` and whitespace are values.
x-celigo-ai-guidance:
- |-
`fail_record_when_empty` is the fourth "Action to take if source field
has no value" and is mutually exclusive with the other three: send it
with no `default` key. A `default`, a `lookupName` or a `hardCodedValue`
on the same row — or a `default` on a `buildArrayHelper` entry — is
rejected at save with `[unsupported_mapping] "<key>" and conditional.when
"fail_record_when_empty" cannot be present in mapping simultaneously.`
(the lookup variant adds `Use lookup "allowFailures" instead.`);
lookups fail through `allowFailures: false`, and a hard-coded row
always has a value. At runtime an empty extract fails the record with
an `errors[]` entry `{code: "extract_empty", fatal: true, source:
"mapping", message: "<generate>: Source field has no value"}`
("Input field has no value" in a Transformation 2.0); `0` and `false`
are values and map through.
enum:
- record_created
- record_updated
- extract_not_empty
- fail_record_when_empty
x-enumDescriptions:
record_created: Apply this mapping only when creating a new record
record_updated: Apply this mapping only when updating an existing record
extract_not_empty: Apply this mapping only when the extract field returns a non-empty value
fail_record_when_empty: Fail the record when the extract field returns no value
examples: ["record_created", "record_updated", "extract_not_empty", "fail_record_when_empty"]
required:
- dataType
if:
properties:
conditional:
properties:
when:
const: fail_record_when_empty
required: [when]
required: [conditional]
then:
not:
anyOf:
- required: [default]
- required: [lookupName]
- required: [hardCodedValue]
references/common-schemas/one-to-many.yml
OneToMany:
type: boolean
description: |
When true, the step runs once per child record instead of once per incoming record.
`pathToMany` names the array field that holds the children inside an object record; when
the incoming record is itself an array (grouped or row-based data), leave `pathToMany` blank
and each element becomes a record. The fan-out is scoped to this step: afterwards the children
are re-joined into the record's original shape — the object with its array, or the array of rows —
carrying any response-mapping enrichment, and that re-joined record is what the next step
receives. Not for locating records in an export's HTTP response; use
`http.response.resourcePath` for that. Applies to steps that receive a record — imports and
lookups (an export referenced as a page processor of a flow, API, or Tool). A standalone export
(a flow's page generator) has no incoming record, so the fields are saved but have no effect on
the records it produces; to emit one record per array element from a standalone export, use a
`hooks.preSavePage` script, or for a file-definition export make the repeating segment the
record boundary in the definition's rules.
x-celigo-ai-guidance:
- |-
Only imports and lookups (exports placed as page processors) fan out. On a standalone export
(a flow's page generator) the API saves ``oneToMany`` / ``pathToMany`` and applies neither:
the records reach the next step unchanged and no error is raised. Live-verified 2026-09-18 on
a flow test run — an order record with three ``lines`` reached the import as ONE record with
the setting on the export, and as THREE child records with the same setting on the import.
Never propose these fields as the way to split a standalone export's records; that is
``hooks.preSavePage`` work (or, for a file definition, the definition's record boundary).
The rules below apply only on a step that receives a record.
- |-
Two modes, decided by the SHAPE of the record entering the step. Object record with a child
array under a field → set ``pathToMany`` to that field. Array record (a ``groupByFields``
export, a CSV/file group, a script that returns an array) → leave ``pathToMany`` blank; each
element becomes one record. Live-verified 2026-09-10 on flows, Tools, and APIs.
- |-
Blank ``pathToMany`` on an OBJECT record fails every record with code ``oneToMany_error``:
"Failed to extract child records from oneToMany import data. Record at index 0 was not an
array, and no PathToMany was provided". Builder previews return only the code, with no
message. The fix is never "turn oneToMany off" by default — first determine whether the
record arriving at the step is an object (set a path) or an array (blank is correct; a
preview error on an array-fed step is the preview panel re-applying one-to-many to the
already-fanned-out record — verify with a test run or a real run).
- |-
Child record shape differs by mode. Path mode: each child is the array element plus
``_PARENT`` (the parent record minus the array). Blank/row mode: each child is the element
itself, with no ``_PARENT``. In both modes templates on the step reference the child as
``{{record.<field>}}``; a response mapping on a one-to-many lookup merges into each child and
the downstream step receives the re-joined parent (path mode) or the re-joined array (row
mode) with every element enriched.
- |-
One-to-many never batches by itself: ``batch_of_records`` appears as the HTTP body template
root only when ``http.batchSize`` is greater than 1 (children are batched after fan-out).
With the default batch size the template root is ``record``.
default: false
examples: [true, false]
references/common-schemas/path-to-many.yml
PathToMany:
type: string
description: |
Path to the array of child records inside an object record when `oneToMany` is true, in dot
notation (`items`, `lines.lineItems`). Leave blank when the incoming record is itself an array
(grouped or row-based data) — each element is then a child record. A path that does not resolve
to an array processes zero records and reports success. Read only on steps that receive a
record (imports, lookup exports); on a standalone export it is saved and ignored — see `oneToMany`.
x-celigo-ai-guidance:
- |-
Inert on a standalone export (a flow's page generator) exactly like ``oneToMany`` — the value
is saved, the export's records reach the next step unchanged, and nothing errors
(live-verified 2026-09-18). Splitting a standalone export's records into one per element is
``hooks.preSavePage`` work, or the file definition's record boundary; never this field.
- |-
Set this ONLY when the record entering the step is an object whose children sit under a field.
On an array record (a ``groupByFields`` export, a CSV/file group, a script returning an array)
any path resolves to nothing: the step processes ZERO children, builds no request, and reports
success with no error. Live-verified 2026-09-10 — the failure is silent, so confirm the arriving
record's shape (test run stage data, or the flow builder's input panel: ``rows`` means array).
- |-
Blank with an object record is the ``oneToMany_error`` failure ("Record at index 0 was not an
array, and no PathToMany was provided"); blank with an array record is the correct row-based
configuration. An empty string behaves exactly like an absent field.
- |-
Use the bare field name or dotted path (``lineItems``, ``details.items``), not a JSONPath
expression. Each child in path mode carries ``_PARENT`` (the parent minus the array), so
parent-level fields are read as ``{{record._PARENT.<field>}}``.
examples: ["items", "line_items", "orderLines", "lines.lineItems"]
references/common-schemas/pre-save.yml
PreSave:
type: object
description: |
Defines a JavaScript hook that executes before the resource is saved.
This hook allows for programmatic validation, transformation, or enrichment of the
resource itself before it is persisted. It can be used to enforce business rules,
set derived properties, or implement cross-field validations that can't be expressed
through the standard UI.
**Function signature**
The preSave function is invoked with a single 'options' argument containing:
```javascript
function preSave(options) {
// Process options and return the modified resource
return options.newResource;
}
```
**Available context**
The 'options' argument provides access to:
- `options.newResource` - The resource being saved (with pending changes)
- `options.oldResource` - The previous version of the resource (before changes)
- `options.sandbox` - Boolean flag indicating whether the script is running in a sandbox environment
**Common uses**
- Enforcing complex business rules across multiple fields
- Automatically deriving field values based on other configuration
- Performing validation that depends on external systems or data
- Normalizing or standardizing configuration values
- Adding computed or derived properties
- Implementing versioning or change tracking
- Dynamically looking up data using the Celigo API module to enrich configuration
**Return value**
The function must return the newResource object (potentially modified) to be saved.
Throwing an exception will prevent saving and signal an error to the user.
properties:
function:
type: string
description: |
The name of the function to execute within the referenced script.
This property specifies which function to invoke from the script
referenced by _scriptId. The function will be called just before
the resource is saved.
The function must follow the expected signature and return the resource object.
examples: ["preSave", "validateResource", "prepareResourceForSave"]
_scriptId:
type: string
format: objectId
x-celigo-refModel: scripts
description: |
Reference to a predefined script resource containing the preSave function.
The referenced script should contain the function specified in the
'function' property. This script must be accessible within the user's account
and have appropriate permissions.
examples: ["60a2c4e6f321d800129a1a3c", "5f8e7d2c1b54e6003a9c7e12"]references/common-schemas/resource-response.yml
ResourceResponse:
type: object
description: Response
x-celigo-ai-guidance:
- Core response fields shared by all Celigo resources
properties:
_id:
type: string
format: objectId
readOnly: true
description: Unique identifier for the resource. Format is a 24-character hexadecimal string.
x-celigo-ai-guidance:
- |-
The _id is used in:
- API endpoints that operate on a specific resource (e.g., GET, PUT, DELETE)
- References from other resources (e.g., flows that use this resource)
- Job history and error tracking
examples: ["5f8d43a1b9e5a80011a35f2c"]
createdAt:
type: string
format: date-time
readOnly: true
description: Timestamp when the resource was created. Set automatically and cannot be modified.
x-celigo-ai-guidance:
- Recorded in ISO 8601 with a UTC timezone (Z suffix).
examples: ["2023-04-01T09:15:32Z"]
lastModified:
type: string
format: date-time
readOnly: true
description: Timestamp when the resource was last updated. Changes whenever any property is modified.
x-celigo-ai-guidance:
- Recorded in ISO 8601 with a UTC timezone (Z suffix); always equal to or later than `createdAt`.
- |-
Records that something was written, not what — a settings edit, a flag flip, or a
platform workflow all move it. What changed and when is in the audit log: the
per-resource route `GET /v1/{resourceType}/{_id}/audit` for the types its
`resourceType` enum lists, or the account-wide
`GET /v1/audit?resourceType=<singular>&_resourceId={_id}` for the further types that
endpoint's enum accepts (users, environments, notifications among them). A type absent
from both enums — jobs, connectors, templates — has only this timestamp.
examples: ["2023-04-15T14:30:15Z"]
deletedAt:
type: [string, 'null']
format: date-time
readOnly: true
description: Timestamp when the resource was soft-deleted. When null or absent, the resource is active.
x-celigo-ai-guidance:
- Non-null means the resource is soft-deleted (in the recycle bin) — recoverable within the retention period before permanent removal.
examples: ["2023-05-20T11:45:32Z"]
required:
- _id
references/common-schemas/settings.yml
Settings:
type: object
description: |
Configuration settings that can be accessed by hooks, filters, mappings and handlebars templates at runtime.
It enables customization of the resource's logic, allowing hooks, mappings, filters, and
handlebars to access and apply the settings at runtime.
**Usage**
The settings object can store arbitrary JSON data that you want to save with the resource.
While it's often populated through a form defined in the `settingsForm` field, you can also:
- Directly provide JSON settings without using a form
- Store configuration values used by hooks and templates
- Create resource-specific constants and parameters
- Maintain lookup tables or mapping structures
- Define conditional logic parameters
**Accessibility**
Settings are available in:
- All handlebars fields for building dynamic payloads
- Field mapping expressions
- JavaScript hooks via the options object
- Filters and transformations
At runtime, the settings objects in the step's execution chain are gathered into a
single `settings` context keyed by fixed scope keys. Flow runs populate the full
chain: `settings.integration.*`, `settings.flowGrouping.*` (the flow group the flow
belongs to), `settings.flow.*`, `settings.connection.*`, `settings.iClient.*`, and
the running step's own scope — `settings.export.*` on exports/lookups,
`settings.import.*` on imports (the key matches the step type). Steps executing
inside a My API or a Tool receive only the step's own scope plus
`settings.connection.*` / `settings.iClient.*`; the integration, flowGrouping, and
flow scopes resolve empty there. The scope key is literal — resource display names
and settingsForm section labels are never part of the path. The segments after the
scope mirror the stored settings JSON key path exactly: a top-level field is
`settings.<scope>.<fieldId>`, and nested objects add one segment per JSON key
(e.g. `settings.flowGrouping.Customer.region` when the group's settings JSON nests
`region` under `Customer`). References without a scope key
(e.g. `{{settings.myField}}`) resolve to empty strings.
**Best practices**
For non-technical users, create a custom form instead of editing the JSON directly.
This provides a user-friendly interface for updating settings without requiring JSON knowledge.
additionalProperties: true
references/common-schemas/transform.yml
Transform:
type: object
description: |
Configuration for transforming data during processing operations. This object enables
reshaping of records.
**Transformation capabilities**
Celigo's transformation engine offers powerful features for data manipulation:
- Precise field mapping with JSONPath expressions
- Support for any level of nested arrays
- Formula-based field value generation
- Dynamic references to flow and integration settings
**Implementation approaches**
There are two distinct transformation mechanisms available:
**Rule-Based Transformation (`type: "expression"`)**
- **Best For**: Most transformation scenarios from simple to complex
- **Capabilities**: Field mapping, formula calculations, lookups, nested data handling
- **Advantages**: Visual configuration, no coding required, intuitive interface
- **Configuration**: Define rules in the `expression` object
- **Use When**: You have clear mapping requirements or need to reshape data structure
**Script-Based Transformation (`type: "script"`)**
- **Best For**: Extremely complex logic or proprietary algorithms
- **Capabilities**: Full programmatic control, custom processing, complex business rules
- **Advantages**: Maximum flexibility, can implement any transformation logic
- **Configuration**: Reference a script in the `script` object
- **Use When**: Visual transformation tools aren't sufficient for your use case
properties:
type:
type: string
description: |
Determines which transformation mechanism to use. This choice affects which properties
must be configured and how transformation logic is implemented.
**Available types**
**Rule-Based Transformation (`"expression"`)**
- **Required Config**: The `expression` object with mapping definitions
- **Behavior**: Applies declarative rules to reshape data
- **Best For**: Most transformation scenarios from simple to complex
- **Advantages**: Visual configuration, no coding required
**Script-Based Transformation (`"script"`)**
- **Required Config**: The `script` object with _scriptId and function
- **Behavior**: Executes custom JavaScript to transform data
- **Best For**: Extremely complex logic or proprietary algorithms
- **Advantages**: Maximum flexibility, can implement any logic
**Implementation guidance**
1. For standard data transformations, use `"expression"`
2. For complex logic or specialized processing, use `"script"`
3. When selecting a type, you must configure the corresponding object:
- `type: "expression"` requires the `expression` object
- `type: "script"` requires the `script` object
enum: ["expression", "script"]
x-enumDescriptions:
expression: Rule-based transformation using declarative mapping definitions.
script: Script-based transformation using custom JavaScript logic.
examples: ["expression", "script"]
expression:
type: object
description: |
Configuration for declarative rule-based transformations. This object enables reshaping data
without requiring custom code.
**Usage context**
This object is REQUIRED when `transform.type` is set to "expression" and should not be
configured otherwise. It provides a standardized way to define transformation rules that
can map, modify, and generate data elements.
**Implementation guidance**
The expression system uses a rule-based approach where:
- Field mappings define how input data is transformed to target fields
- Formulas can be used to calculate or generate new values
- Lookups can enrich data by fetching related information
- Mode determines how records are processed (create new or modify existing)
properties:
version:
type: string
description: |
Version of the expression format. Determines which rules
property contains the transformation logic.
enum: ["1", "2"]
x-enumDescriptions:
"1": "Legacy format. Rules are in the `rules` array (array of arrays of extract/generate objects)."
"2": "Current format. Rules are in the `rulesTwoDotZero` object with mode, mappings, and lookups."
examples: ["1", "2"]
rules:
type: array
description: |
Transformation rules for version 1 expressions. An array of
rule groups; each group is an array of field-mapping objects.
Most transforms have a single group. Present when `version`
is `"1"`. The output record contains ONLY the generated
fields — every unmapped field is dropped (v1 has no
equivalent of Transform 2.0's `modify` mode), and the
record's trace key does not survive the rebuild.
x-celigo-ai-guidance:
- |-
v1 rules REPLACE the record: a one-field rule group drops
every other field, silently breaking downstream filters,
router branch rules, and templates that referenced the
dropped fields. Either map every field downstream logic
needs forward, or author Transform 2.0 with
``mode: "modify"`` instead.
- |-
An import's ``responseTransform`` is the exception: it accepts
version ``"1"`` only (``"2"`` is refused with 422, verified live
2026-10-21), so there, map every response field that response
mapping needs.
items:
type: array
items:
type: object
properties:
extract:
type: string
description: |
Source field path to read from. Supports multiple
syntaxes: bare field names (`id`), dot notation
(`fulfillment.shipment_id`), slash-prefixed paths
for XML (`/FeedProcessingStatus`), wildcards (`*.id`,
`*.[Internal ID]`), and array indexing (`SDF[0]`).
generate:
type: string
description: |
Target field name to write to. Typically a bare name
(`id`) or dot path (`SDF.Filter.ID`).
key:
type: string
description: |
Auto-generated identifier for this rule, used by the
UI to track individual rules for editing and reordering.
required:
- extract
- generate
rulesTwoDotZero:
type: object
description: |
Configuration for version 2 transformation rules. This object contains the core logic
for how data is mapped, enriched, and transformed.
**Capabilities**
Transformation 2.0 provides:
- Precise field mapping with JSONPath expressions
- Support for deeply nested data structures
- Formula-based field generation
- Dynamic lookups for data enrichment
- Multiple operating modes to fit different scenarios
properties:
mode:
type: string
description: |
Transformation mode that determines how records are handled during processing.
**Available modes**
**Create Mode (`"create"`)**
- **Behavior**: Builds entirely new output records from inputs
- **Use When**: Output structure differs significantly from input
- **Advantage**: Clean slate approach, no field inheritance
**Modify Mode (`"modify"`)**
- **Behavior**: Makes targeted edits to existing records
- **Use When**: Output structure should remain similar to input
- **Advantage**: Preserves unmapped fields from the original record
enum: ["create", "modify"]
x-enumDescriptions:
create: Builds entirely new output records from inputs, ignoring unmapped fields.
modify: Makes targeted edits to existing records, preserving unmapped fields.
x-celigo-ai-guidance:
- |-
On a grouped record (the export groups rows, so the record is an
array) only `create` can author output — build it from the rows with
`$[0].field`, `$[*].field`, or a `buildArrayHelper` over `$[*]`.
`modify` has no object to overlay onto an array: the record passes
through untouched and the processor reports success. Either mode
reshapes the inside of one record; neither changes how many records
the page carries (that is the export's `preSavePage` hook) or whether
the export groups at all (that is its grouping setting).
examples: ["create", "modify"]
mappings:
$ref: './mappings.yml#/Mappings'
lookups:
allOf:
- description: |
Shared lookup tables used across all mappings defined in the transformation rules.
**Purpose**
Lookups provide centralized value translation that can be referenced from any mapping
in your transformation configuration. They enable consistent translation of codes, IDs,
and values between systems without duplicating translation logic.
**Usage in transformations**
Lookups are particularly valuable in transformations for:
- **Data Normalization**: Standardizing values from diverse source systems
- **Code Translation**: Converting between different coding systems (e.g., status codes)
- **Field Enrichment**: Adding descriptive values based on ID or code lookups
- **Cross-Reference Resolution**: Mapping identifiers between integrated systems
**Implementation**
Lookups are defined once in this array and referenced by name in mappings:
```json
"lookups": [
{
"name": "statusMapping",
"map": {
"A": "Active",
"I": "Inactive",
"P": "Pending"
},
"default": "Unknown Status"
}
]
```
Then referenced in mappings using the lookupName property:
```json
{
"generate": "status",
"dataType": "string",
"extract": "$.statusCode",
"lookupName": "statusMapping"
}
```
The system automatically applies the lookup during transformation processing.
For complete details on lookup properties and behavior, see the Lookups schema.
- $ref: './lookups.yml#/Lookups'
inputContext:
type: string
enum:
- record
- envelope
x-enumDescriptions:
record: The transform processor evaluates `mappings[].extract` JSONPath against the flat record itself (`$.field`, no `record.` segment) — the default when absent.
envelope: The transform processor evaluates JSONPath against the wrapped runtime envelope `{record, job, testMode, settings, headers, queryParams}` (file-provider steps also carry `fileMeta`) — record fields need the `record.` segment (`$.record.field`), while `headers` and `queryParams` sit beside `record` (`$.headers['X-Id']`, `$.queryParams.src`).
description: |
Controls the JSON shape the transformTwoDotZero processor
evaluates `mappings[].extract` JSONPath values against at
flow runtime. Applies only to Transform 2.0 (v2,
`rulesTwoDotZero`); v1 transforms (the `rules` array on
`transform.expression.rules`) and script-mode transforms
ignore this field.
x-celigo-ai-guidance:
- |-
## Extract anchoring by mode
- `record` (default — absent / empty / undefined): the
processor sees the FLAT record as its evaluation root.
`mappings[].extract` JSONPath MUST anchor directly at
`$` with NO `record.` segment — e.g. `"$.orderId"`,
`"$.customer.email"`; PLACEHOLDER extracts use
`"$.PLACEHOLDER_<field>"`.
This is the dominant Celigo idiom and the right choice for new transforms.
- `envelope`: the processor sees a wrapped object
`{record: {...}, job, testMode, settings, headers,
queryParams}` (plus `fileMeta` on file-provider steps).
`mappings[].extract` JSONPath MUST include the
`record.` segment for record fields — e.g.
`"$.record.orderId"` — and reads `headers` and
`queryParams` beside it (`$.headers['X-Id']`);
PLACEHOLDER extracts use `"$.record.PLACEHOLDER_<field>"`.
Pick this only when the transform was authored against
the envelope shape on purpose.
- |-
Handlebars expressions inside `extract`
(`{{record.field}}`, `{{dateFormat ... record.field}}`)
ALWAYS reference the `record` root regardless of
`inputContext` — only JSONPath anchoring flips.
- |-
In `envelope` mode the output is still a bare record: a
`modify` transform returns the record with its mapped
fields merged in (no `record` wrapper, no `job` or
`settings` keys), and a `create` transform returns only
the mapped fields.
- |-
Emit `inputContext` explicitly on new v2 transforms:
unset is read as `record`, but stating it makes the
extract-anchoring contract self-documenting and immune
to a platform default change.
- |-
## Allowed values
- ``record`` (default — emitted when the field is absent
/ empty / undefined): the transform processor sees
the FLAT record itself as its evaluation root.
``mappings[].extract`` JSONPath values MUST anchor
directly at ``$`` with NO ``record.`` segment — for
example ``"$.orderId"`` or
``"$.customer.email"``. PLACEHOLDER extracts use
``"$.PLACEHOLDER_<field>"``. This is the dominant
Celigo idiom and the right choice for new transforms.
- ``envelope``: the transform processor sees a wrapped
object of the shape
``{record: {...}, job, testMode, settings, headers,
queryParams}`` (plus ``fileMeta`` on file-provider
steps; ``headers`` and ``queryParams`` sit beside
``record``) — the
same runtime-context envelope Handlebars / SQL /
GraphQL templates render against. ``mappings[].extract``
JSONPath values for record fields MUST include the
``record.`` segment — for example ``"$.record.orderId"``
or ``"$.record.customer.email"`` — while ``headers`` and
``queryParams`` are read beside it
(``"$.headers['X-Id']"``). PLACEHOLDER extracts
use ``"$.record.PLACEHOLDER_<field>"``. Pick this
mode only when the export was authored against the
envelope shape on purpose (typically because an
upstream step intentionally feeds the wrapped
runtime context through to the transform).
## Handlebars expressions are NOT affected
Handlebars expressions inside ``extract``
(``{{record.field}}``,
``{{dateFormat ... record.field}}``, etc.) ALWAYS
reference the runtime context's ``record`` root
regardless of ``inputContext`` — the Handlebars runtime
always sees the wrapped envelope. Only JSONPath
behavior flips with this field.
## Agent guidance
Set this field explicitly on every new v2 transform
the agent authors. Leaving it unset is interpreted as
``record`` by the platform, but emitting
``inputContext: record`` makes the contract
self-documenting and prevents the transform from
drifting into envelope mode if an upstream change ever
changes the platform default. Read this value before
writing ``mappings[].extract`` so the JSONPath form
matches the runtime root.
- |-
## fileMeta on file-provider steps
File-provider exports and lookups populate a
``fileMeta`` envelope key describing each record's
source file: ``fileName``, ``fileSize``, and
``lastModifiedTime`` (epoch milliseconds). Reach it
as ``$.fileMeta.<field>`` in envelope mode, or as
``{{fileMeta.<field>}}`` from a Handlebars extract in
either mode. The keys are exactly these three —
lowercase variants such as ``filesize`` or
``filename`` resolve empty. The standalone transform
processor substitutes sample fileMeta values when the
posted data lacks them, so a resolving editor render
is not proof the step provides fileMeta — only
file-provider steps populate it at flow runtime.
examples: ["record", "envelope"]
script:
type: object
description: |
Configuration for programmable script-based transformations. This object enables complex, custom
transformation logic beyond what expression-based transformations can provide.
**Usage context**
This object is REQUIRED when `transform.type` is set to "script" and should not be configured
otherwise. It provides a way to execute custom JavaScript code to transform data according to
specialized business rules or complex algorithms.
**Implementation approach**
Script-based transformation works by:
1. Executing the specified function from the referenced script
2. Passing input data to the function
3. Using the function's return value as the transformed output
**Common use cases**
Script transformation is ideal for:
- Complex business logic that can't be expressed through mappings
- Algorithmic transformations requiring computation
- Dynamic transformations based on external factors
- Legacy system data format compatibility
- Multi-stage processing with intermediate steps
Only use script-based transformation when expression-based transformation is insufficient.
Script transformation requires maintaining custom code, which adds complexity to the integration.
properties:
_scriptId:
type: string
description: |
Reference to a predefined script resource containing the transformation logic.
The referenced script should contain the function specified in the
'function' property. It must be a script in this account: any other id fails the save
with 422 ("Only a script owned by the user or a script shared by the Integration App can
be used at transform.script._scriptId.").
format: objectid
x-celigo-refModel: scripts
examples: ["60a2c4e6f321d800129a1a3c", "5f8d43a1b9e5a80011a35f2c"]
function:
type: string
description: |
Name of the function within the script to execute for transformation. This function
must exist in the script referenced by _scriptId.
examples: ["transformData", "convertRecord", "applyMapping"]
references/guardrail-schemas/guardrail-config.yml
# Guardrail-owned contract for the `guardrail` object. It mirrors the import spec's
# GuardrailConfig field for field with one deliberate difference: the AI-agent branch
# (GuardrailAiAgentConfig, below) is composed without the `tools` arrays that the server
# rejects on guardrails, instead of $ref-ing the tool-bearing AiAgentConfig. Keep the shared
# fields (type, confidenceThreshold, pii, moderation, and the provider blocks) in step with
# specs/v1/import/components/schemas/{guardrail,aiagent}.yml when either side changes.
GuardrailConfig:
type: object
description: |-
Selects the check a guardrail applies and carries its settings. `type` picks the check
and the matching sub-object (`pii`, `moderation`, or `aiAgent`) configures it.
Responses echo the active sub-object with `confidenceThreshold` defaulted and also
return the inactive `pii` / `moderation` siblings as empty stubs
(`pii: {entities: [], mask: false}`, `moderation: {categories: []}`) — only the active
type's sub-object is meaningful. Guardrails created before the AI-agent stub was
stripped on write may still return an inert `aiAgent` on `pii` / `moderation` guardrails.
x-celigo-ai-guidance:
- |-
Pick exactly one type. Prefer the deterministic local classifiers (`pii`, then
`moderation`) whenever the brief's intent fits; use `ai_agent` only for custom
judgement that has no `pii.entities` / `moderation.categories` analogue — the full
decision rule is on `type`.
- |-
Guardrails never invoke tools. A non-empty `tools` array anywhere under `aiAgent`
(`aiAgent.tools`, `aiAgent.openai.tools`, `aiAgent.litellm._overrides.gemini.tools`,
`aiAgent.litellm._overrides.anthropic.tools`) fails the save with 422
`guardrail_tools_not_allowed`. MCP `prompts` and `resources` are still accepted on
save, but a guardrail's run never uses them: it gives the model only its instructions
and input, whether the MCP server can be reached or not. Put the policy the guardrail
must apply in its instructions.
properties:
type:
type: string
enum: ["ai_agent", "pii", "moderation"]
x-enumDescriptions:
ai_agent: Evaluate data using an AI model with custom instructions.
pii: Detect personally identifiable information in data fields.
moderation: Check content for harmful or inappropriate categories.
description: |-
The type of guardrail to apply. Each type requires its corresponding
sub-configuration object (`aiAgent`, `pii`, or `moderation`).
x-celigo-ai-guidance:
- |-
The type of guardrail to apply.
Pick exactly one — `pii` / `moderation` / `ai_agent` are mutually exclusive at
the type level (a brief that wants both runs as two guardrails in sequence,
not one).
- |-
## How to choose
Apply these rules **in order** and pick the first match. Do
NOT skip ahead to `ai_agent` just because the brief is in
natural language — `pii` and `moderation` are also expressed
as natural-language briefs. The decision is about WHAT the
brief asks the guardrail to detect, not how the brief is
worded.
1. **`pii`** — pick this when the brief names ANY personally
identifiable information, OR when the brief names entity
categories (SSN, email, phone, credit card, address, name,
passport, license, account number, etc.) that match
`pii.entities`. Examples that route here:
"detect SSN and credit card numbers", "mask customer
emails and phone numbers", "scan for PII".
2. **`moderation`** — pick this when the brief asks the
guardrail to detect harmful, inappropriate, unsafe, or
policy-violating CONTENT (as opposed to PII data fields)
AND the intent maps to any value in `moderation.categories`
(`sexual`, `hate`, `harassment`, `self_harm`, `violence`,
`illicit`, etc.) — including via close synonyms. Apply
the synonym mapping liberally:
- "explicit", "obscene", "vulgar", "profane",
"inappropriate language", "swearing", "NSFW" → typically
maps to `sexual` and/or `harassment`
- "abusive", "insulting", "bullying", "toxic" → `harassment`
- "hateful", "discriminatory", "racist", "sexist" → `hate`
- "threatening", "intimidating" → `harassment_threatening`
or `violence`
- "graphic violence", "gore" → `violence_graphic`
- "suicide", "self-injury" → `self_harm`
- "drugs", "weapons", "illegal activity" → `illicit`
Briefs like "screen email body for explicit language",
"block toxic chat messages", "flag harassment in support
tickets", "moderate user-generated content for unsafe
material" all route to `moderation`. When the brief uses
a synonym, populate `moderation.categories` with the
closest enum value(s) — do NOT invent new category
strings; the enum is fixed.
3. **`ai_agent`** — pick this ONLY when the brief asks for
judgement that doesn't fit `pii` or `moderation`. Typical
triggers: domain-specific compliance checks (HIPAA / SOX
rule evaluation), business-policy validation (price
bounds, approval thresholds), data-quality assertions
(required fields populated, values plausible), or any
"evaluate against these custom rules" framing where the
rules are domain-specific and have no `pii.entities` /
`moderation.categories` analogue. If you can map the
brief's intent to a moderation category (even via a
synonym above), use `moderation` instead — `ai_agent` is
the catch-all, not the default.
Tie-breaker for borderline cases: prefer the more specific
type (`pii` > `moderation` > `ai_agent`). `pii` and
`moderation` are local deterministic classifiers with fixed
enums — they're cheaper, faster, and more predictable than
`ai_agent` (which calls a real LLM per record). Default to
the cheaper path whenever the intent fits.
- |-
## Sub-configuration requirements (one per type)
- **ai_agent** requires the `aiAgent` sub-configuration.
- **pii** requires the `pii` sub-configuration with at least
one entity from `pii.entities`.
- **moderation** requires the `moderation` sub-configuration
with at least one category from `moderation.categories`.
confidenceThreshold:
type: number
minimum: 0
maximum: 1
default: 0.7
x-celigo-ui-override: >-
Required by the guardrail form (confidenceThreshold is required:true once a type is
selected, default 0.7). Encoded to mirror the form so builders produce connectable
configurations.
description: |-
Confidence threshold (0 to 1). Detections below this threshold are
ignored. Lower values catch more issues but increase false positives.
x-celigo-ai-guidance:
- Confidence threshold for guardrail detection (0 to 1).
- |-
Only detections with confidence at or above this threshold will be flagged.
Lower values catch more potential issues but may increase false positives.
examples: [0.7, 0.5, 0.9]
aiAgent:
type: object
description: |-
LLM configuration for the check; set when `type` is `ai_agent` and validated
against `GuardrailAiAgentConfig`. On `pii` / `moderation` guardrails a legacy
server-written stub may appear here — it is inert, and current servers strip it on
write.
pii:
type: object
required: [entities]
description: |-
PII detection configuration. Required when `type` is `pii`.
x-celigo-ai-guidance:
- Configuration for PII (Personally Identifiable Information) detection.
- |-
Required when `guardrail.type` is "pii".
At least one entity type must be specified.
properties:
entities:
type: array
description: |-
PII entity types to detect. When `type` is `pii`, at least one
entry is required; the inactive sibling on other guardrail types
may be served with an empty list.
x-celigo-ai-guidance:
- PII entity types to detect in the data.
- At least one entity must be specified when using PII guardrails.
items:
type: string
enum:
- credit_card_number
- card_security_code_cvv_cvc
- cryptocurrency_wallet_address
- date_and_time
- email_address
- iban_code
- bic_swift_bank_identifier_code
- ip_address
- location
- medical_license_number
- national_registration_number
- persons_name
- phone_number
- url
- us_bank_account_number
- us_drivers_license
- us_itin
- us_passport_number
- us_social_security_number
- uk_nhs_number
- uk_national_insurance_number
- spanish_nif
- spanish_nie
- italian_fiscal_code
- italian_drivers_license
- italian_vat_code
- italian_passport
- italian_identity_card
- polish_pesel
- finnish_personal_identity_code
- singapore_nric_fin
- singapore_uen
- australian_abn
- australian_acn
- australian_tfn
- australian_medicare
- indian_pan
- indian_aadhaar
- indian_vehicle_registration
- indian_voter_id
- indian_passport
- korean_resident_registration_number
x-enumDescriptions:
credit_card_number: Credit or debit card number (e.g., Visa, Mastercard, Amex).
card_security_code_cvv_cvc: Card security code printed on credit or debit cards (CVV/CVC).
cryptocurrency_wallet_address: Cryptocurrency wallet address (e.g., Bitcoin, Ethereum).
date_and_time: Date and/or time values that could identify an individual.
email_address: Email address in standard format.
iban_code: International Bank Account Number used for cross-border payments.
bic_swift_bank_identifier_code: BIC/SWIFT code identifying a specific bank for international transfers.
ip_address: IPv4 or IPv6 network address.
location: Physical location or geographic coordinates.
medical_license_number: Medical professional license or registration number.
national_registration_number: National identification or registration number.
persons_name: Full or partial name of a person.
phone_number: Telephone number in any format.
url: Web URL or URI.
us_bank_account_number: United States bank account number.
us_drivers_license: United States driver's license number.
us_itin: United States Individual Taxpayer Identification Number.
us_passport_number: United States passport number.
us_social_security_number: United States Social Security Number (SSN).
uk_nhs_number: United Kingdom National Health Service number.
uk_national_insurance_number: United Kingdom National Insurance number.
spanish_nif: Spanish tax identification number (NIF).
spanish_nie: Spanish foreigner identification number (NIE).
italian_fiscal_code: Italian fiscal code (Codice Fiscale).
italian_drivers_license: Italian driver's license number.
italian_vat_code: Italian VAT identification number (Partita IVA).
italian_passport: Italian passport number.
italian_identity_card: Italian national identity card number.
polish_pesel: Polish national identification number (PESEL).
finnish_personal_identity_code: Finnish personal identity code (henkilotunnus).
singapore_nric_fin: Singapore National Registration Identity Card or Foreign Identification Number.
singapore_uen: Singapore Unique Entity Number for business registration.
australian_abn: Australian Business Number.
australian_acn: Australian Company Number.
australian_tfn: Australian Tax File Number.
australian_medicare: Australian Medicare card number.
indian_pan: Indian Permanent Account Number for tax purposes.
indian_aadhaar: Indian Aadhaar unique identity number.
indian_vehicle_registration: Indian vehicle registration number.
indian_voter_id: Indian voter identification card number.
indian_passport: Indian passport number.
korean_resident_registration_number: South Korean resident registration number.
examples:
- ["email_address", "phone_number", "persons_name"]
- ["us_social_security_number", "credit_card_number"]
mask:
type: boolean
default: false
description: |-
When true, the result also carries `masked`: the evaluated input with each detected
value replaced by asterisks. The record itself is unchanged. With whole-record input
(no `mapping`, and no top-level `text` field on the record), `masked` is the record
serialized as one JSON string. When false, PII is flagged only.
x-celigo-ai-guidance:
- |-
Masking doesn't rewrite the record. To pass masked text on, select the field as the
`text` input (`mapping`) and response-map `masked` back onto that field on the
guardrail's flow step (`{"extract": "masked", "generate": "<field>"}`).
moderation:
type: object
required: [categories]
description: |-
Content moderation configuration. Required when `type` is `moderation`.
x-celigo-ai-guidance:
- |-
Required when `guardrail.type` is "moderation".
At least one category must be specified.
properties:
categories:
type: array
description: |-
Content moderation categories to check. When `type` is
`moderation`, at least one entry is required; the inactive
sibling on other guardrail types may be served with an empty
list.
x-celigo-ai-guidance:
- At least one category must be specified when using moderation guardrails.
items:
type: string
enum:
- sexual
- sexual_minors
- hate
- hate_threatening
- harassment
- harassment_threatening
- self_harm
- self_harm_intent
- self_harm_instructions
- violence
- violence_graphic
- illicit
- illicit_violent
x-enumDescriptions:
sexual: Content depicting sexual activity or explicit sexual material.
sexual_minors: Sexual content involving minors.
hate: Content expressing hatred toward a group based on protected characteristics.
hate_threatening: Hateful content that includes threats of violence or serious harm.
harassment: Content that targets, intimidates, or bullies an individual.
harassment_threatening: Harassment content that includes threats of violence or serious harm.
self_harm: Content that promotes or depicts self-harm behaviors.
self_harm_intent: Content expressing intent to engage in self-harm.
self_harm_instructions: Content providing instructions for self-harm methods.
violence: Content depicting or promoting physical violence.
violence_graphic: Graphic or gory depictions of violence or injury.
illicit: Content promoting illegal activities or unlawful behavior.
illicit_violent: Content promoting illegal activities that involve violence.
examples:
- ["hate", "violence", "harassment"]
- ["sexual", "self_harm", "illicit"]
required:
- type
- confidenceThreshold
if:
required: [type]
properties:
type:
const: pii
then:
required: [pii]
properties:
pii:
required: [entities]
properties:
entities:
minItems: 1
else:
if:
required: [type]
properties:
type:
const: moderation
then:
required: [moderation]
properties:
moderation:
required: [categories]
properties:
categories:
minItems: 1
else:
required: [aiAgent]
properties:
aiAgent:
$ref: '#/GuardrailAiAgentConfig'
GuardrailAiAgentConfig:
type: object
description: |
LLM configuration for an `ai_agent` guardrail — which provider and model evaluate each
record, with instructions, parameter tuning, and output format. Guardrails never invoke
tools, so unlike an AI agent import no block here accepts a `tools` array. MCP `prompts`
and `resources` are accepted on save but never used at run time, so put the policy the
guardrail must apply in its instructions. Providers come in two families with two
configuration formats:
- **Built-in providers** — each has its own configuration block.
- **openai**: OpenAI models (GPT-4.1, GPT-5, etc.). Configure via the `openai` object.
- **gemini**: Google Gemini models via the LiteLLM proxy. Configure via `litellm` with overrides in `litellm._overrides.gemini`.
- **anthropic**: Anthropic Claude models via the LiteLLM proxy. Configure via `litellm` with overrides in `litellm._overrides.anthropic`.
- **Catalog providers** — `mistral`, `xai`, `huggingface`, `deepseek`, `cohere`, `groq`. Added
as catalog data rather than platform code, they share one generic flat format:
`model`, `modelOptions`, `instructions`, `output`, and the `prompts` / `resources`
arrays directly on this object.
A `_connectionId` on the parent import selects bring-your-own-key (BYOK)
credentials. Without one, built-in providers run on platform-managed
credentials; catalog providers have no platform-managed credentials, so the
save requires one — without it the guardrail fails to save with 422 on
`guardrail.aiAgent.model`: `ai_model_not_available` for a catalog-listed model,
`ai_model_not_in_catalog` for one the catalog does not list (the model
catalog check answers before any `connection_required`).
**Model lifecycle.** Every model the platform offers carries a lifecycle in
the platform's model catalog: `active`, `deprecated` (still usable; a
retirement date and suggested replacement models may be published), or
`retired`, plus an availability switch. The save and every run of an
`ai_agent` guardrail check the configured model against it exactly as for an
AI agent import: a model that is not in the catalog, switched off, not offered
for the guardrail's credential type, or retired (by status, or once its
retirement date has passed) is refused — the save with 422
`ai_model_not_in_catalog`, `ai_model_disabled`, `ai_model_not_available`, or
`ai_model_retired` on the model field, the run with a `Run blocked:` error
carrying the same code before any model call (`Run blocked: model
"gpt-4.1-nano" is retired on "Oct 23, 2026". Please choose a supported model
and test your agent configuration.` — the run-time message names the date
only). A deprecated model saves and runs until
its retirement date. The platform never substitutes a model: once a model is
retired, every save of the guardrail is refused until the model changes.
Guardrails on affected models are listed by `GET /v1/aiagents/aiModelHealth`.
With a `_connectionId`, a model the catalog does not list is passed through to
the vendor as-is; a catalog-listed model remains governed.
x-celigo-ai-guidance:
- |-
Do not send `tools` on any block. A non-empty array at `tools`, `openai.tools`,
`litellm._overrides.gemini.tools`, or `litellm._overrides.anthropic.tools` fails the
save with 422 `guardrail_tools_not_allowed` naming that path. Responses still echo
`openai.tools: []` on OpenAI guardrails; do not copy it back into a write.
required: [provider]
properties:
provider:
type: string
enum: ["openai", "gemini", "anthropic", "mistral", "xai", "huggingface", "deepseek", "cohere", "groq"]
x-enumDescriptions:
openai: Use OpenAI models via the OpenAI Responses API (built-in; `openai` block).
gemini: Use Google Gemini models via the LiteLLM proxy (built-in; `litellm` block).
anthropic: Use Anthropic Claude models via the LiteLLM proxy (built-in; `litellm` block).
mistral: Use Mistral models (catalog provider; generic flat format, BYOK).
xai: Use xAI Grok models (catalog provider; generic flat format, BYOK).
huggingface: Use Hugging Face inference endpoint models (catalog provider; generic flat format, BYOK).
deepseek: Use DeepSeek models (catalog provider; generic flat format, BYOK).
cohere: Use Cohere models (catalog provider; generic flat format, BYOK).
groq: Use Groq-hosted models (catalog provider; generic flat format, BYOK).
description: AI provider to use.
x-celigo-ai-guidance:
- |-
The provider decides the configuration format, and the server rejects a
mismatch with 422 `invalid_provider_config_format`: `openai` needs the
`openai` block, `gemini` and `anthropic` need the `litellm` block (Gemini- or
Claude-specific settings under `litellm._overrides`), and the catalog
providers (`mistral`, `xai`, `huggingface`, `deepseek`, `cohere`, `groq`) need the
generic flat format — `model` and `modelOptions` directly on this object,
never an `openai` or `litellm` block.
- |-
Catalog providers are BYOK-only, and the save enforces it through the model
catalog: without a `_connectionId` the guardrail fails to save with 422
`ai_model_not_available` on `guardrail.aiAgent.model` ("Model \"grok-4.6\" is
not supported. Please choose a supported model and test your agent
configuration.") — the model is a catalog row that is not offered on
platform-managed credentials. Attach an HTTP connection to the vendor's API
(token auth) before saving a catalog-provider guardrail.
model:
type: string
description: |-
Model identifier for a catalog provider (generic flat format) — for example
`grok-4.6`. Built-in providers set the model inside their own block
(`openai.model`, `litellm.model`) instead.
x-celigo-ai-guidance:
- |-
Required for every catalog provider — omitting it or sending an empty string
fails the save with 422 `missing_required_field`.
- |-
Without a `_connectionId` on the parent import the model check answers first:
a catalog-provider guardrail fails to save with 422 on `guardrail.aiAgent.model` —
`ai_model_not_available` for a catalog-listed model (catalog providers are not
offered on platform-managed credentials), `ai_model_not_in_catalog` for an id
the catalog does not list.
With a `_connectionId` (BYOK) a model the catalog does not list saves as-is —
the server does not check it against the catalog or the vendor; a
catalog-listed model is still governed by its lifecycle row (see the
schema description — a retired or switched-off model is refused on save).
modelOptions:
type: object
description: |-
Model tuning for a catalog provider (generic flat format). Which keys, values, and
ranges are accepted is defined per catalog model and validated on save when the
model is a catalog entry.
properties:
maxOutputTokens:
type: number
description: Maximum number of tokens the model may generate.
temperature:
type: number
description: Sampling temperature.
topP:
type: number
description: Nucleus sampling threshold.
reasoning_effort:
type: string
enum: ["low", "medium", "high", "xhigh"]
x-enumDescriptions:
low: Minimal reasoning before answering.
medium: Moderate reasoning.
high: Extensive reasoning.
xhigh: Maximum reasoning.
description: Reasoning depth for models that expose it.
x-celigo-ai-guidance:
- |-
When the model is a catalog entry, the server validates this object against
that model's schema and fails the save with 422 `invalid_model_options`
naming the key: unknown keys are rejected, `reasoning_effort` must be one of
the enum values where the model accepts it at all (xAI's `grok-4.6` does;
DeepSeek's `deepseek-chat` rejects the key), and numeric bounds apply —
on `grok-4.6`, `temperature` ≤ 2, `topP` ≤ 1, `maxOutputTokens` ≥ 100.
- |-
With a `_connectionId` and a model the catalog does not list, the object is
passed through unvalidated — send only options the vendor's API accepts.
instructions:
type: string
description: |-
System prompt for a catalog provider (generic flat format). Built-in providers
set it inside their own block (`openai.instructions`,
`litellm._overrides.anthropic.systemInstruction`).
output:
type: object
description: |-
Output format for a catalog provider (generic flat format). Built-in providers
configure it inside their own block (`openai.output`, `litellm.responseFormat`).
properties:
format:
type: object
description: Controls the structure of the model's output.
properties:
type:
type: string
enum: ["text", "json_schema"]
x-enumDescriptions:
text: Free-form text.
json_schema: Structured JSON conforming to `jsonSchema`.
description: Output type.
x-celigo-ai-guidance:
- |-
`blob` is not accepted here — the save fails with 422
`unsupported_output_format` naming the model (verified on `grok-4.6`).
name:
type: string
description: Name of the JSON schema, for `json_schema` output.
strict:
type: boolean
description: When true, the model must conform exactly to `jsonSchema`.
jsonSchema:
type: object
additionalProperties: true
description: JSON Schema the structured output must conform to, for `json_schema` output.
verbose:
type: string
description: Level of detail in the model's response, for models that expose it.
tools:
type: array
deprecated: true
description: |-
Not available on guardrails: a non-empty array fails the save with 422
`guardrail_tools_not_allowed`. Kept in source only because the field exists on the
shared import model.
items:
type: object
additionalProperties: true
prompts:
type: array
description: |-
MCP prompt entries saved with a catalog-provider guardrail (generic flat format).
Each entry references one MCP connection and prompt names; a guardrail's run never fetches them.
items:
type: object
properties:
type:
type: string
enum: ["mcp"]
x-enumDescriptions:
mcp: Reference prompts exposed by an MCP server.
description: Type of prompt entry. Always "mcp".
mcp:
type: object
properties:
_mcpConnectionId:
type: string
format: objectId
x-celigo-refModel: connections
description: Connection to the MCP server that exposes the prompts.
allowedPrompts:
type: array
description: Prompt names saved with the entry; a guardrail's run never fetches them.
items:
type: string
x-celigo-ai-guidance:
- |-
An unknown `_mcpConnectionId` fails the save with 422 `invalid_ref` naming
the array index.
resources:
$ref: '../schemas/aiagent.yml#/McpResources'
openai:
type: object
description: |
OpenAI-specific configuration. Used when `provider` is "openai".
required: [model, instructions]
properties:
instructions:
type: string
maxLength: 1000000
description: |
System prompt that defines the check's behavior, goals, and constraints.
examples: ["You are a data validation agent. Check each record for completeness."]
x-celigo-ui-override: >-
Required by the AI agent configuration form (the model's system prompt). Encoded
here to mirror the form so builders produce connectable configurations.
model:
type: string
description: |-
OpenAI model identifier. Open string (not an enum) — model names change frequently.
examples: ["gpt-5.4", "gpt-5.4-pro", "gpt-5-mini", "gpt-5-nano", "gpt-4.1-mini"]
x-celigo-ui-override: >-
Required by the AI agent configuration form. Encoded here to mirror the form so
builders produce connectable configurations.
x-celigo-ai-guidance:
- |-
With platform-managed credentials (no `_connectionId` on the parent
import), the model must be a catalog row the platform offers on its own
credentials. The save checks this — a model that is not in the catalog
fails with 422 `ai_model_not_in_catalog` on
`guardrail.aiAgent.openai.model` ("Model \"gpt-4\" is not supported.
Please choose a supported model and test your agent configuration."). A
retired model is refused the same way with `ai_model_retired` — the
message names the retirement date and, when any remain eligible, the
suggested next models ("Model \"gpt-4.1-nano\" is retired on \"Oct 23,
2026\" and cannot be used. Suggested next models: gpt-5.6-luna. Please
test and update your agent configuration.") — and so is a switched-off
one with `ai_model_disabled`; a deprecated model saves. Never swap the
model on the user's behalf — relay the message and let them choose.
- |-
With a `_connectionId` (BYOK) a model the catalog does not list is passed
through to the provider as-is; a catalog-listed model is still governed by
its lifecycle row.
reasoning:
type: object
description: |-
Controls depth of reasoning for complex tasks.
x-celigo-ai-guidance:
- |-
Supported on reasoning-capable OpenAI models only (currently the GPT-5 family and the o-series).
Setting `reasoning` on a non-reasoning model (e.g.
`gpt-4.1-mini`) is rejected by the provider at runtime — omit the field on those models.
properties:
effort:
type: string
enum: ["none", "minimal", "low", "medium", "high", "xhigh", "max"]
x-enumDescriptions:
none: No additional reasoning effort.
minimal: Least reasoning effort for simple, straightforward tasks.
low: Light reasoning for moderately simple tasks.
medium: Balanced reasoning effort for typical tasks.
high: Extensive reasoning effort for complex, multi-step tasks.
xhigh: Extended reasoning effort beyond high for demanding tasks.
max: Highest available reasoning effort.
description: How much reasoning effort the model should invest
x-celigo-ai-guidance:
- |-
How much reasoning effort the model should invest.
Reasoning- capable models only — see `reasoning` for the gating rule.
summary:
type: string
enum: ["concise", "auto", "detailed"]
x-enumDescriptions:
concise: Brief, high-level reasoning summary.
auto: Let the model decide the appropriate level of detail.
detailed: Comprehensive reasoning summary with full explanations.
description: Level of detail in reasoning summaries
x-celigo-ai-guidance:
- |-
Level of detail in reasoning summaries.
Reasoning-capable models only — see `reasoning` for the gating rule.
temperature:
type: number
minimum: 0
maximum: 2
description: |
Sampling temperature. Higher values (e.g. 1.5) produce more creative output,
lower values (e.g. 0.2) produce more focused and deterministic output.
topP:
type: number
minimum: 0.1
maximum: 1
description: Nucleus sampling parameter
topLogprobs:
type: number
minimum: 0
maximum: 20
description: Number of most likely tokens to return log probabilities for at each output position.
maxOutputTokens:
type: number
minimum: 100
maximum: 128000
description: Maximum number of tokens in the model's response. Unlike an AI agent import, a guardrail does not receive a server-written value when this is omitted.
serviceTier:
type: string
enum: ["auto", "default", "priority"]
x-enumDescriptions:
auto: Let OpenAI automatically select the appropriate service tier.
default: Standard service tier with normal rate limits and latency.
priority: Premium tier with higher rate limits and lower latency at increased cost.
description: |-
OpenAI service tier. "priority" provides higher rate limits and
lower latency at increased cost.
x-celigo-ai-guidance:
- |-
BYOK only.
Platform-managed credentials (no `_connectionId` on the parent import) always
run at `default` regardless of what's set here.
output:
type: object
description: |-
Output format configuration. A guardrail stores `format` but doesn't use it: the
request to the model always carries the guardrail's own `validation_result` JSON
schema (`flagged`, `reasoning`, `confidence`), and the result returns `flagged` and
`reasoning` whatever `format` holds.
properties:
format:
type: object
description: |
Stored output structure. The request to the model uses the guardrail's own
verdict schema instead (see `output`).
properties:
type:
type: string
enum: ["text", "json_schema", "blob"]
x-enumDescriptions:
text: Stored format type; the model request still uses the guardrail's verdict schema.
json_schema: A stored JSON schema (`jsonSchema`); the model request still uses the guardrail's verdict schema.
blob: Stored format type; the model request still uses the guardrail's verdict schema.
description: Output format type, as stored.
schemaMode:
type: string
enum: ["manual", "json"]
x-enumDescriptions:
manual: The schema editor opens `jsonSchema` in the visual field-by-field view
json: The schema editor opens `jsonSchema` in the raw JSON view
description: |-
Which view a schema editor opens `jsonSchema` in. Stored and echoed on
guardrails for parity with the AI-agent format block — the guardrail form
does not set it today, and neither view changes the model request.
x-celigo-ai-guidance:
- |-
Resend the stored value on every update — a full PUT that omits the field
clears it — and never invent one: a value outside the enum fails the save with
422 `enum` on `guardrail.aiAgent.openai.output.format.schemaMode`.
name:
type: string
description: Name for the output format (used with json_schema)
x-celigo-ui-override: >-
Required by the AI agent form when output format type is json_schema. Encoded
here to mirror the form so builders produce connectable configurations.
strict:
type: boolean
description: Stored strict-validation flag for `jsonSchema`. A guardrail's model request uses its own verdict schema, so this doesn't apply there.
x-celigo-ai-guidance:
- Stored only on a guardrail; it doesn't change the guardrail's verdict.
jsonSchema:
type: object
description: |
Stored JSON Schema. Required when `format.type` is "json_schema". A guardrail
doesn't send it to the model; the request carries the `validation_result`
verdict schema instead.
x-celigo-ui-override: >-
Required by the AI agent form when output format type is json_schema. Encoded
here to mirror the form so builders produce connectable configurations.
properties:
type:
type: string
description: Root type of the stored JSON Schema.
enum: ["object", "array", "string", "number", "integer", "boolean"]
x-enumDescriptions:
object: JSON object with named properties.
array: Ordered list of values.
string: Text string value.
number: Numeric value including decimals.
integer: Whole number value without decimals.
boolean: True or false value.
properties:
type: object
additionalProperties: true
description: Field definitions in the stored JSON Schema.
required:
type: array
description: Property names the stored JSON Schema marks as required.
items:
type: string
additionalProperties:
type: boolean
description: Whether the stored JSON Schema allows properties beyond those in `properties`.
schema:
type: object
readOnly: true
additionalProperties: true
description: |-
Server-written copy of `jsonSchema`, present in responses once `jsonSchema`
is set. Send `jsonSchema`, not this field.
if:
properties:
type:
const: json_schema
required: [type]
then:
required: [name, jsonSchema]
verbose:
type: string
enum: ["low", "medium", "high"]
x-enumDescriptions:
low: Minimal detail in the model's response.
medium: Moderate detail in the model's response, the default level.
high: Maximum detail and verbosity in the model's response.
description: Level of detail in the model's response
x-celigo-ai-guidance:
- |-
Supported on the GPT-5 family only. Other OpenAI models (e.g.
`gpt-4.1`, `gpt-4.1-mini`,
`gpt-4.1-nano`) accept only `medium` (the no-op default) — `low` and `high` are
rejected by the provider at runtime.
Omit the field entirely on non-GPT-5 models rather than relying on the default.
tools:
type: array
deprecated: true
description: |-
Always an empty array on guardrails saved today: the server writes `[]` and
rejects a non-empty array with 422 `guardrail_tools_not_allowed`. Guardrails
saved before the restriction may still return their stored entries.
items:
type: object
additionalProperties: true
prompts:
type: array
description: |-
MCP prompt entries saved with the guardrail. Each item references one
MCP connection and prompt names; a guardrail's run never fetches them.
items:
type: object
properties:
type:
type: string
enum: ["mcp"]
x-enumDescriptions:
mcp: Connect to an MCP server to use its exposed prompts.
description: Type of prompt entry. Always "mcp".
mcp:
type: object
description: MCP server prompt configuration.
properties:
_mcpConnectionId:
type: string
format: objectId
x-celigo-refModel: connections
description: Connection to the MCP server.
allowedPrompts:
type: array
description: Prompt names saved with the entry; a guardrail's run never fetches them.
items:
type: string
resources:
$ref: '../schemas/aiagent.yml#/McpResources'
litellm:
type: object
description: |
LiteLLM proxy configuration. Used when `provider` is "gemini" or "anthropic".
LiteLLM provides a unified interface to multiple AI providers. Gemini-specific
settings are in `_overrides.gemini`; Claude-specific settings are in
`_overrides.anthropic`.
`model` is required when litellm is the active provider path.
properties:
model:
type: string
description: |-
LiteLLM model identifier. For Gemini, models are stored without the `gemini/`
prefix; for Anthropic, use the Claude model id (e.g. `claude-sonnet-4-6`).
examples: ["gemini-2.5-flash", "gemini-2.5-pro", "gemini-2.5-flash-lite", "claude-sonnet-4-6", "claude-opus-4-7"]
x-celigo-ui-override: >-
Required by the AI agent configuration form. Encoded here to mirror the form so
builders produce connectable configurations.
x-celigo-ai-guidance:
- |-
With platform-managed credentials (no `_connectionId` on the parent
import), the model must be a catalog row the platform offers for the
provider on its own credentials — Gemini and Anthropic Claude each have
their own rows. The save checks this: a model that is not in the catalog
fails with 422 `ai_model_not_in_catalog` on
`guardrail.aiAgent.litellm.model`, a retired one with `ai_model_retired`
(the message names the retirement date and, when any remain eligible, the
suggested next models), a switched-off one with `ai_model_disabled`; a
deprecated model saves. Never
swap the model on the user's behalf — relay the message and let them choose.
- |-
With a `_connectionId` (BYOK) a model the catalog does not list is passed
through to the provider as-is; a catalog-listed model is still governed by
its lifecycle row.
temperature:
type: number
minimum: 0
maximum: 2
description: Sampling temperature
x-celigo-ai-guidance:
- |-
When `provider` is "anthropic", the value must be between 0 and 1 — a higher value
fails the save with 422 `invalid_field_value`. Gemini accepts the full 0-2 range.
maxCompletionTokens:
type: number
minimum: 100
maximum: 128000
description: Maximum number of tokens in the response
topP:
type: number
minimum: 0.1
maximum: 1
description: Nucleus sampling parameter
seed:
type: number
description: Random seed for reproducible outputs
responseFormat:
type: object
description: Output format configuration
x-celigo-ai-guidance:
- |-
When `provider` is "anthropic", `type` must be "text" or "json_schema" — "blob" is
rejected with 422 `invalid_response_format`. As with the other providers, "json_schema"
requires `name` and `jsonSchema`.
properties:
type:
type: string
description: Output format type.
enum: ["text", "json_schema", "blob"]
x-enumDescriptions:
text: Free-form text response from the model.
json_schema: Structured JSON output conforming to a defined schema.
blob: Binary data output for non-text content.
schemaMode:
type: string
enum: ["manual", "json"]
x-enumDescriptions:
manual: The schema editor opens `jsonSchema` in the visual field-by-field view
json: The schema editor opens `jsonSchema` in the raw JSON view
description: |-
Which view a schema editor opens `jsonSchema` in; the schema sent to the provider
is the same either way. Stored and echoed on guardrails for parity with the
AI-agent format block — the guardrail form does not set it today.
x-celigo-ai-guidance:
- |-
Resend the stored value on every update — a full PUT that omits the field clears
it — and never invent one: a value outside the enum fails the save with 422 `enum`
on `guardrail.aiAgent.litellm.responseFormat.schemaMode`.
name:
type: string
description: Name for the output format (used with json_schema).
x-celigo-ui-override: >-
Required by the AI agent form when output format type is json_schema. Encoded
here to mirror the form so builders produce connectable configurations.
strict:
type: boolean
description: When true, enforces strict schema validation on output.
jsonSchema:
type: object
description: JSON Schema for structured output. Required when `responseFormat.type` is "json_schema".
x-celigo-ui-override: >-
Required by the AI agent form when output format type is json_schema. Encoded
here to mirror the form so builders produce connectable configurations.
properties:
type:
type: string
description: Root JSON Schema type of the structured output; use "object" for record-shaped results.
enum: ["object", "array", "string", "number", "integer", "boolean"]
x-enumDescriptions:
object: JSON object with named properties.
array: Ordered list of values.
string: Text string value.
number: Numeric value including decimals.
integer: Whole number value without decimals.
boolean: True or false value.
properties:
type: object
additionalProperties: true
description: JSON Schema definitions for each field the structured output may contain.
required:
type: array
description: Property names the model must include in the structured output.
items:
type: string
additionalProperties:
type: boolean
description: When true, the structured output may include properties beyond those defined in `properties`.
if:
properties:
type:
const: json_schema
required: [type]
then:
required: [name, jsonSchema]
_overrides:
type: object
description: Provider-specific overrides
properties:
gemini:
type: object
description: |
Gemini-specific configuration overrides.
required: [systemInstruction]
properties:
systemInstruction:
type: string
maxLength: 1000000
description: |
System instruction for Gemini models. Equivalent to OpenAI's `instructions`.
Maximum 1,000,000 characters.
x-celigo-ui-override: >-
Required by the AI agent configuration form (the model's system prompt, the
Gemini equivalent of `instructions`). Encoded here to mirror the form so
builders produce connectable configurations.
tools:
type: array
deprecated: true
description: |-
Not available on guardrails: a non-empty array fails the save with 422
`guardrail_tools_not_allowed`.
items:
type: object
additionalProperties: true
prompts:
type: array
description: |-
MCP prompt entries saved with a Gemini guardrail. Each item references one
MCP connection and prompt names; a guardrail's run never fetches them.
items:
type: object
properties:
type:
type: string
enum: ["mcp"]
x-enumDescriptions:
mcp: Connect to an MCP server to use its exposed prompts.
description: Type of prompt entry. Always "mcp".
mcp:
type: object
description: MCP server prompt configuration.
properties:
_mcpConnectionId:
type: string
format: objectId
x-celigo-refModel: connections
description: Connection to the MCP server.
allowedPrompts:
type: array
description: Prompt names saved with the entry; a guardrail's run never fetches them.
items:
type: string
resources:
$ref: '../schemas/aiagent.yml#/McpResources'
responseModalities:
type: array
description: Response output modalities
x-celigo-ai-guidance:
- |-
`["image"]` requires an image-capable Gemini model (currently `gemini-2.5-flash-image`).
Selecting it on a text-only model is rejected by the provider — pair image
output with the right model or omit.
items:
type: string
enum: ["text", "image"]
x-enumDescriptions:
text: Generate text content in the response.
image: Generate image content in the response.
topK:
type: number
description: Top-K sampling parameter for Gemini
thinkingConfig:
type: object
description: Controls Gemini's extended thinking capabilities
x-celigo-ai-guidance:
- |-
Supported on reasoning-capable Gemini models only (currently `gemini-2.5-pro`
and `gemini-2.5-flash`).
Setting `thinkingConfig` on a non-thinking model (e.g.
`gemini-2.5-flash-lite`,
`gemini-2.5-flash-image`) is rejected by the provider at runtime — omit the
field on those models.
properties:
includeThoughts:
type: boolean
description: When true, includes the model's thinking steps in the response.
x-celigo-ai-guidance:
- Whether to include thinking steps in the response
thinkingBudget:
type: number
minimum: 100
maximum: 4000
description: Maximum tokens allocated for thinking
thinkingLevel:
type: string
description: Controls how much thinking effort the model applies; use higher levels for complex, multi-step tasks at the cost of latency and tokens.
enum: ["minimal", "low", "medium", "high"]
x-enumDescriptions:
minimal: Least thinking effort for simple tasks.
low: Light thinking for moderately simple tasks.
medium: Balanced thinking effort for typical tasks.
high: Maximum thinking effort for complex, multi-step tasks.
imageConfig:
type: object
description: Gemini image generation configuration
properties:
aspectRatio:
type: string
description: Aspect ratio of generated images; choose a ratio matching the intended display format.
enum: ["1:1", "2:3", "3:2", "3:4", "4:3", "4:5", "5:4", "9:16", "16:9", "21:9"]
x-enumDescriptions:
"1:1": Square aspect ratio.
"2:3": Portrait aspect ratio (2 wide by 3 tall).
"3:2": Landscape aspect ratio (3 wide by 2 tall).
"3:4": Portrait aspect ratio (3 wide by 4 tall).
"4:3": Standard landscape aspect ratio (4 wide by 3 tall).
"4:5": Portrait aspect ratio (4 wide by 5 tall).
"5:4": Landscape aspect ratio (5 wide by 4 tall).
"9:16": Tall portrait aspect ratio for mobile screens.
"16:9": Widescreen landscape aspect ratio.
"21:9": Ultra-widescreen cinematic aspect ratio.
imageSize:
type: string
description: Output resolution of generated images; higher resolutions increase detail and file size.
enum: ["1K", "2K", "4K"]
x-enumDescriptions:
"1K": Standard resolution output around 1024 pixels.
"2K": High resolution output around 2048 pixels.
"4K": Ultra-high resolution output around 4096 pixels.
mediaResolution:
type: string
enum: ["low", "medium", "high"]
x-enumDescriptions:
low: Low resolution for faster processing and reduced token usage.
medium: Balanced resolution for typical media processing.
high: High resolution for maximum detail in media inputs.
description: Resolution for media inputs (images, video)
anthropic:
type: object
description: |
Claude-specific configuration overrides. Used when `provider` is "anthropic".
required: [systemInstruction]
properties:
systemInstruction:
type: string
maxLength: 1000000
description: |
System instruction for Claude models. Equivalent to OpenAI's `instructions`.
Maximum 1,000,000 characters.
x-celigo-ui-override: >-
Required by the AI agent configuration form (the model's system prompt, the
Claude equivalent of `instructions`). Encoded here to mirror the form so
builders produce connectable configurations.
topK:
type: number
minimum: 0
description: |-
Top-K sampling parameter. Deprecated on Claude models released after Claude Opus
4.6, which reject any value at runtime; set it only on older models.
thinkingConfig:
type: object
description: Controls Claude's extended thinking capabilities.
properties:
type:
type: string
enum: ["enabled", "disabled", "adaptive"]
x-enumDescriptions:
enabled: Turn on extended thinking with an explicit token budget.
disabled: Turn off extended thinking.
adaptive: Let the model decide how much to think based on task complexity.
description: Extended thinking mode.
budgetTokens:
type: number
minimum: 1024
description: |-
Maximum tokens allocated for thinking. Required when `type` is "enabled" —
omitting it fails the save with 422 `invalid_thinking_config`.
x-celigo-ui-override: >-
Required by the AI agent form when thinkingConfig.type is "enabled". Encoded
here to mirror the form so builders produce connectable configurations.
display:
type: string
enum: ["summarized", "omitted"]
x-enumDescriptions:
summarized: Include a summarized view of the model's thinking in the response.
omitted: Exclude the model's thinking from the response.
description: How thinking output is surfaced in the response.
effort:
type: string
enum: ["low", "medium", "high", "xhigh", "max"]
x-enumDescriptions:
low: Light thinking effort for moderately simple tasks.
medium: Balanced thinking effort for typical tasks.
high: Maximum standard thinking effort for complex tasks.
xhigh: Extended thinking effort beyond high for demanding tasks.
max: Highest available thinking effort.
description: |-
How much thinking effort the model applies; use with `type` "adaptive". Higher
values (`xhigh`, `max`) may be gated to specific Claude models by the provider.
if:
properties:
type:
const: enabled
required: [type]
then:
required: [budgetTokens]
serviceTier:
type: string
enum: ["auto", "standard_only"]
x-enumDescriptions:
auto: Let Anthropic select the appropriate service tier.
standard_only: Restrict processing to the standard service tier.
description: Anthropic service tier for the request.
tools:
type: array
deprecated: true
description: |-
Not available on guardrails: a non-empty array fails the save with 422
`guardrail_tools_not_allowed`.
items:
type: object
additionalProperties: true
prompts:
type: array
description: |-
MCP prompt entries saved with a Claude guardrail. Each item references one
MCP connection and prompt names; a guardrail's run never fetches them.
items:
type: object
properties:
type:
type: string
enum: ["mcp"]
x-enumDescriptions:
mcp: Connect to an MCP server to use its exposed prompts.
description: Type of prompt entry. Always "mcp".
mcp:
type: object
description: MCP server prompt configuration.
properties:
_mcpConnectionId:
type: string
format: objectId
x-celigo-refModel: connections
description: Connection to the MCP server.
allowedPrompts:
type: array
description: Prompt names saved with the entry; a guardrail's run never fetches them.
items:
type: string
resources:
$ref: '../schemas/aiagent.yml#/McpResources'
# Same provider→block chain as the AI agent import's AiAgentConfig: the instance-level
# `required` is what forces the provider's block to exist.
if:
properties:
provider:
const: openai
required: [provider]
then:
required: [openai]
properties:
openai:
required: [model, instructions]
else:
if:
properties:
provider:
enum: [gemini, anthropic]
required: [provider]
then:
required: [litellm]
properties:
litellm:
required: [model]
else:
if:
properties:
provider:
enum: [mistral, xai, huggingface, deepseek, cohere, groq]
required: [provider]
then:
required: [model]
references/schemas/aiagent.yml
AiAgentConfig:
type: object
description: |
AI agent configuration for an AiAgentImport. An `ai_agent` guardrail is configured by
the guardrail spec's `GuardrailAiAgentConfig` instead, which has no `tools` arrays.
Configures which AI provider and model to use, along with instructions, parameter
tuning, output format, and available tools. Providers come in two families with
two configuration formats:
- **Built-in providers** — each has its own configuration block.
- **openai**: OpenAI models (GPT-4.1, GPT-5, etc.). Configure via the `openai` object.
- **gemini**: Google Gemini models via the LiteLLM proxy. Configure via `litellm` with overrides in `litellm._overrides.gemini`.
- **anthropic**: Anthropic Claude models via the LiteLLM proxy. Configure via `litellm` with overrides in `litellm._overrides.anthropic`.
- **Catalog providers** — `mistral`, `xai`, `huggingface`, `deepseek`, `cohere`, `groq`. Added
as catalog data rather than platform code, they share one generic flat format:
`model`, `modelOptions`, `instructions`, `output`, and the `tools` / `prompts` /
`resources` arrays directly on this object.
A `_connectionId` on the parent import selects bring-your-own-key (BYOK)
credentials. Without one, built-in providers run on platform-managed
credentials; catalog providers have no platform-managed credentials, so the
save requires one — without it the agent fails to save with 422 on
`aiAgent.model`: `ai_model_not_available` for a catalog-listed model,
`ai_model_not_in_catalog` for one the catalog does not list (the model
catalog check answers before any `connection_required`).
**Model lifecycle.** Every model the platform offers — built-in and catalog
providers alike — carries a lifecycle in the platform's model catalog:
`active`, `deprecated` (still usable; a retirement date and suggested
replacement models may be published), or `retired`, plus an availability
switch. The save and every run check the configured model against it: a
model that is not in the catalog, switched off, not offered for the agent's
credential type, or retired (by status, or once its retirement date has
passed) is refused — the save with 422 `ai_model_not_in_catalog`,
`ai_model_disabled`, `ai_model_not_available`, or `ai_model_retired` on the
model field, the run with a `Run blocked:` error carrying the same code before
any model call (`Run blocked: model "gpt-4.1-nano" is retired on "Oct 23,
2026". Please choose a supported model and test your agent configuration.` —
the run-time message names the date only; the chat route streams it as an
`error` event). A deprecated model saves and runs until its retirement date. The
platform never substitutes a model: once a model is retired, every save of
the agent is refused until the model changes. Agents and guardrails on
affected models are listed by `GET /v1/aiagents/aiModelHealth`. With a
`_connectionId`, a model the catalog does not list is passed through to the
vendor as-is; a catalog-listed model remains governed.
required: [provider]
properties:
provider:
type: string
enum: ["openai", "gemini", "anthropic", "mistral", "xai", "huggingface", "deepseek", "cohere", "groq"]
x-enumDescriptions:
openai: Use OpenAI models via the OpenAI Responses API (built-in; `openai` block).
gemini: Use Google Gemini models via the LiteLLM proxy (built-in; `litellm` block).
anthropic: Use Anthropic Claude models via the LiteLLM proxy (built-in; `litellm` block).
mistral: Use Mistral models (catalog provider; generic flat format, BYOK).
xai: Use xAI Grok models (catalog provider; generic flat format, BYOK).
huggingface: Use Hugging Face inference endpoint models (catalog provider; generic flat format, BYOK).
deepseek: Use DeepSeek models (catalog provider; generic flat format, BYOK).
cohere: Use Cohere models (catalog provider; generic flat format, BYOK).
groq: Use Groq-hosted models (catalog provider; generic flat format, BYOK).
description: AI provider to use.
x-celigo-ai-guidance:
- |-
The provider decides the configuration format, and the server rejects a
mismatch with 422 `invalid_provider_config_format`: `openai` needs the
`openai` block, `gemini` and `anthropic` need the `litellm` block (Gemini- or
Claude-specific settings under `litellm._overrides`), and the catalog
providers (`mistral`, `xai`, `huggingface`, `deepseek`, `cohere`, `groq`) need the
generic flat format — `model` and `modelOptions` directly on this object,
never an `openai` or `litellm` block.
- |-
Catalog providers are BYOK-only, and the save enforces it through the model
catalog: without a `_connectionId` the agent fails to save with 422
`ai_model_not_available` on `aiAgent.model` ("Model \"grok-4.6\" is not
supported. Please choose a supported model and test your agent
configuration.") — the model is a catalog row that is not offered on
platform-managed credentials. Attach an HTTP connection to the vendor's API
(token auth) before saving a catalog-provider agent.
model:
type: string
description: |-
Model identifier for a catalog provider (generic flat format) — for example
`grok-4.6`. Built-in providers set the model inside their own block
(`openai.model`, `litellm.model`) instead.
x-celigo-ai-guidance:
- |-
Required for every catalog provider — omitting it or sending an empty string
fails the save with 422 `missing_required_field`.
- |-
Without a `_connectionId` on the parent import the model check answers first:
a catalog-provider agent fails to save with 422 on `aiAgent.model` —
`ai_model_not_available` for a catalog-listed model (catalog providers are not
offered on platform-managed credentials), `ai_model_not_in_catalog` for an id
the catalog does not list.
With a `_connectionId` (BYOK) a model the catalog does not list saves as-is —
the server does not check it against the catalog or the vendor; a
catalog-listed model is still governed by its lifecycle row (see the
schema description — a retired or switched-off model is refused on save).
modelOptions:
type: object
description: |-
Model tuning for a catalog provider (generic flat format). Which keys, values, and
ranges are accepted is defined per catalog model and validated on save when the
model is a catalog entry.
properties:
maxOutputTokens:
type: number
description: Maximum number of tokens the model may generate.
temperature:
type: number
description: Sampling temperature.
topP:
type: number
description: Nucleus sampling threshold.
reasoning_effort:
type: string
enum: ["low", "medium", "high", "xhigh"]
x-enumDescriptions:
low: Minimal reasoning before answering.
medium: Moderate reasoning.
high: Extensive reasoning.
xhigh: Maximum reasoning.
description: Reasoning depth for models that expose it.
x-celigo-ai-guidance:
- |-
When the model is a catalog entry, the server validates this object against
that model's schema and fails the save with 422 `invalid_model_options`
naming the key: unknown keys are rejected, `reasoning_effort` must be one of
the enum values where the model accepts it at all (xAI's `grok-4.6` does;
DeepSeek's `deepseek-chat` rejects the key), and numeric bounds apply —
on `grok-4.6`, `temperature` ≤ 2, `topP` ≤ 1, `maxOutputTokens` ≥ 100.
- |-
With a `_connectionId` and a model the catalog does not list, the object is
passed through unvalidated — send only options the vendor's API accepts.
instructions:
type: string
description: |-
System prompt for a catalog provider (generic flat format). Built-in providers
set it inside their own block (`openai.instructions`,
`litellm._overrides.anthropic.systemInstruction`).
output:
type: object
description: |-
Output format for a catalog provider (generic flat format). Built-in providers
configure it inside their own block (`openai.output`, `litellm.responseFormat`).
properties:
format:
type: object
description: Controls the structure of the model's output.
properties:
type:
type: string
enum: ["text", "json_schema"]
x-enumDescriptions:
text: Free-form text.
json_schema: Structured JSON conforming to `jsonSchema`.
description: Output type.
x-celigo-ai-guidance:
- |-
`blob` is not accepted here — the save fails with 422
`unsupported_output_format` naming the model (verified on `grok-4.6`).
name:
type: string
description: Name of the JSON schema, for `json_schema` output.
strict:
type: boolean
description: When true, the model must conform exactly to `jsonSchema`.
jsonSchema:
type: object
additionalProperties: true
description: JSON Schema the structured output must conform to, for `json_schema` output.
verbose:
type: string
description: Level of detail in the model's response, for models that expose it.
tools:
type: array
description: |-
Tools available to a catalog-provider agent (generic flat format). Entries mirror
the built-in providers' tool entries: a Celigo Tool by `_toolId`, an MCP server by
`_mcpConnectionId`, or a vendor web-search tool.
items:
type: object
properties:
type:
type: string
enum: ["tool", "mcp", "web_search"]
x-enumDescriptions:
tool: Reference a reusable Celigo Tool resource.
mcp: Connect to an MCP server for additional external tools.
web_search: Search the web for real-time information, where the vendor's model supports it.
description: Type of tool entry.
tool:
type: object
description: Reference to a Celigo Tool resource, used when type is "tool".
properties:
_toolId:
type: string
format: objectId
x-celigo-refModel: tools
description: The Celigo Tool to call.
mcp:
type: object
description: MCP server connection, used when type is "mcp".
properties:
_mcpConnectionId:
type: string
format: objectId
x-celigo-refModel: connections
description: Connection to the MCP server.
allowedTools:
type: array
description: |-
Specific tools to allow from the MCP server (all if omitted). Each entry
is either a plain tool name or an object carrying display metadata.
items:
type: [string, object]
config:
type: object
additionalProperties: true
description: Tool-specific options for vendor-native tools (for example web-search settings).
x-celigo-ai-guidance:
- |-
References are checked on save: an unknown `_toolId` or `_mcpConnectionId`
fails with 422 `invalid_ref` naming the array index.
prompts:
type: array
description: |-
MCP prompt entries available to a catalog-provider agent (generic flat format).
Each entry references one MCP connection and the prompt names allowed from it.
items:
type: object
properties:
type:
type: string
enum: ["mcp"]
x-enumDescriptions:
mcp: Reference prompts exposed by an MCP server.
description: Type of prompt entry. Always "mcp".
mcp:
type: object
properties:
_mcpConnectionId:
type: string
format: objectId
x-celigo-refModel: connections
description: Connection to the MCP server that exposes the prompts.
allowedPrompts:
type: array
description: Prompt names the agent may fetch from the server.
items:
type: string
x-celigo-ai-guidance:
- |-
An unknown `_mcpConnectionId` fails the save with 422 `invalid_ref` naming
the array index.
resources:
$ref: '#/McpResources'
openai:
type: object
description: |
OpenAI-specific configuration. Used when `provider` is "openai".
required: [model, instructions]
properties:
instructions:
type: string
maxLength: 1000000
description: |
System prompt that defines the AI agent's behavior, goals, and constraints.
examples: ["You are a data validation agent. Check each record for completeness."]
x-celigo-ui-override: >-
Required by the AI agent configuration form (the model's system prompt). Encoded
here to mirror the form so builders produce connectable configurations.
model:
type: string
description: |-
OpenAI model identifier. Open string (not an enum) — model names change frequently.
examples: ["gpt-5.4", "gpt-5.4-pro", "gpt-5-mini", "gpt-5-nano", "gpt-4.1-mini"]
x-celigo-ui-override: >-
Required by the AI agent configuration form. Encoded here to mirror the form so
builders produce connectable configurations.
x-celigo-ai-guidance:
- |-
With platform-managed credentials (no `_connectionId` on the parent
import), the model must be a catalog row the platform offers on its own
credentials. The save checks this — a model that is not in the catalog
fails with 422 `ai_model_not_in_catalog` on `aiAgent.openai.model`
("Model \"gpt-4\" is not supported. Please choose a supported model and
test your agent configuration."), and a dry-run answers the same
`errors` with `valid: false`. A retired model is refused the same way with
`ai_model_retired` — the message names the retirement date and, when any
remain eligible, the suggested next models ("Model \"gpt-4.1-nano\" is
retired on \"Oct 23, 2026\" and cannot be used. Suggested next models:
gpt-5.6-luna. Please test and update your agent configuration.") — and so
is a switched-off one with `ai_model_disabled`; a deprecated model saves.
Never swap the model on the user's behalf — relay the message and let
them choose.
- |-
With a `_connectionId` (BYOK) a model the catalog does not list is passed
through to the provider as-is; a catalog-listed model is still governed by
its lifecycle row.
reasoning:
type: object
description: |-
Controls depth of reasoning for complex tasks.
x-celigo-ai-guidance:
- |-
Supported on reasoning-capable OpenAI models only (currently the GPT-5 family and the o-series).
Setting `reasoning` on a non-reasoning model (e.g.
`gpt-4.1-mini`) is rejected by the provider at runtime — omit the field on those models.
properties:
effort:
type: string
enum: ["none", "minimal", "low", "medium", "high", "xhigh", "max"]
x-enumDescriptions:
none: No additional reasoning effort.
minimal: Least reasoning effort for simple, straightforward tasks.
low: Light reasoning for moderately simple tasks.
medium: Balanced reasoning effort for typical tasks.
high: Heavy reasoning effort for complex, multi-step tasks.
xhigh: Very high reasoning effort for the most demanding, multi-step tasks.
max: Highest available reasoning effort, above xhigh.
description: How much reasoning effort the model should invest
x-celigo-ai-guidance:
- |-
How much reasoning effort the model should invest.
Reasoning- capable models only — see `reasoning` for the gating rule.
- |-
Any other value fails the save with 422 `enum` on `aiAgent.openai.reasoning.effort`.
summary:
type: string
enum: ["concise", "auto", "detailed"]
x-enumDescriptions:
concise: Brief, high-level reasoning summary.
auto: Let the model decide the appropriate level of detail.
detailed: Comprehensive reasoning summary with full explanations.
description: Level of detail in reasoning summaries
x-celigo-ai-guidance:
- |-
Level of detail in reasoning summaries.
Reasoning-capable models only — see `reasoning` for the gating rule.
temperature:
type: number
minimum: 0
maximum: 2
description: |
Sampling temperature. Higher values (e.g. 1.5) produce more creative output,
lower values (e.g. 0.2) produce more focused and deterministic output.
topP:
type: number
minimum: 0.1
maximum: 1
description: Nucleus sampling parameter
topLogprobs:
type: number
minimum: 0
maximum: 20
description: Number of most likely tokens to return log probabilities for at each output position.
maxOutputTokens:
type: number
minimum: 100
maximum: 128000
default: 5000
description: |-
Caps the length of the model's response, in tokens. An agent created without it is
saved with 5000; raise it for long outputs such as large extractions.
serviceTier:
type: string
enum: ["auto", "default", "priority"]
x-enumDescriptions:
auto: Let OpenAI automatically select the appropriate service tier.
default: Standard service tier with normal rate limits and latency.
priority: Premium tier with higher rate limits and lower latency at increased cost.
default: "default"
description: |-
OpenAI service tier. "priority" provides higher rate limits and
lower latency at increased cost.
x-celigo-ai-guidance:
- |-
BYOK only.
Platform-managed credentials (no `_connectionId` on the parent import) always
run at `default` regardless of what's set here.
output:
type: object
description: Output format configuration
properties:
format:
type: object
description: |
Controls the structure of the model's output.
properties:
type:
type: string
enum: ["text", "json_schema", "blob"]
x-enumDescriptions:
text: Free-form text response from the model.
json_schema: Structured JSON output conforming to a defined schema.
blob: Image generation — the run calls the image generation tool and returns the stored image's `blobKey`, not `_text` or `_json`.
default: "text"
description: Output format type.
schemaMode:
type: string
enum: ["manual", "json"]
x-enumDescriptions:
manual: The UI opens `jsonSchema` in the visual field-by-field editor
json: The UI opens `jsonSchema` in the raw JSON editor
description: |-
Which editor the UI opens `jsonSchema` in; the schema sent to the provider
is the same either way, and the field is only meaningful when `type` is
"json_schema". Unset opens the visual editor, and an edit there rebuilds
`jsonSchema` from the field list, keeping only `type`, `properties`,
`required`, `items`, `description`, `enum`, `pattern`, length and numeric
bounds, `multipleOf`, `minItems`/`maxItems` and a `"null"` type — other
keywords are dropped without a prompt, and `integer` becomes `number`.
x-celigo-ai-guidance:
- |-
Set `json` on a schema written as raw JSON; keep a user's `manual` and stay
inside the keywords the visual editor keeps. A full PUT that omits the field
clears it, and a value outside the enum fails the save with 422 `enum` on
`aiAgent.openai.output.format.schemaMode`.
name:
type: string
description: Name for the output format (used with json_schema)
x-celigo-ui-override: >-
Required by the AI agent form when output format type is json_schema. Encoded
here to mirror the form so builders produce connectable configurations.
strict:
type: boolean
default: false
description: When true, enforces strict schema validation on output.
x-celigo-ai-guidance:
- Whether to enforce strict schema validation on output
jsonSchema:
type: object
description: |
JSON Schema for structured output. Required when `format.type` is "json_schema".
x-celigo-ui-override: >-
Required by the AI agent form when output format type is json_schema. Encoded
here to mirror the form so builders produce connectable configurations.
properties:
type:
type: string
description: Root JSON Schema type of the structured output; use "object" for record-shaped results.
enum: ["object", "array", "string", "number", "integer", "boolean"]
x-enumDescriptions:
object: JSON object with named properties.
array: Ordered list of values.
string: Text string value.
number: Numeric value including decimals.
integer: Whole number value without decimals.
boolean: True or false value.
properties:
type: object
additionalProperties: true
description: JSON Schema definitions for each field the structured output may contain.
required:
type: array
description: Property names the model must include in the structured output.
items:
type: string
additionalProperties:
type: boolean
description: When true, the structured output may include properties beyond those defined in `properties`.
schema:
type: object
readOnly: true
additionalProperties: true
description: |-
Server-written copy of `jsonSchema`, returned whenever `jsonSchema` is set — on
AI agent imports and on guardrail agents alike. Send `jsonSchema`, not this field.
if:
properties:
type:
const: json_schema
required: [type]
then:
required: [name, jsonSchema]
verbose:
type: string
enum: ["low", "medium", "high"]
x-enumDescriptions:
low: Minimal detail in the model's response.
medium: Moderate detail in the model's response, the default level.
high: Maximum detail and verbosity in the model's response.
default: "medium"
description: Level of detail in the model's response
x-celigo-ai-guidance:
- |-
Supported on the GPT-5 family only. Other OpenAI models (e.g.
`gpt-4.1`, `gpt-4.1-mini`,
`gpt-4.1-nano`) accept only `medium` (the no-op default) — `low` and `high` are
rejected by the provider at runtime.
Omit the field entirely on non-GPT-5 models rather than relying on the default.
tools:
type: array
description: |
Tools available to the AI agent during processing.
items:
type: object
properties:
type:
type: string
enum: ["web_search", "mcp", "image_generation", "code_execution", "tool"]
x-enumDescriptions:
web_search: Search the web for real-time information.
mcp: Connect to an MCP server for additional external tools.
image_generation: Generate images using an AI image model.
code_execution: "Enable OpenAI's built-in code-execution tool; carries no configuration block, send `{type: code_execution}` only"
tool: Reference a reusable Celigo Tool resource.
description: |-
Type of tool. Every type except `code_execution` is configured through the
sibling object of the same name; OpenAI spells this one `code_execution`, while
the Gemini and Anthropic overrides spell theirs `codeExecution` — the other spelling
fails with 422 `enum`.
webSearch:
type: object
description: Web search configuration (empty object to enable)
imageGeneration:
type: object
description: Image generation configuration
properties:
background:
type: string
description: Controls whether generated images have a transparent or opaque background; use transparent only with output formats that support it (png, webp).
enum: ["transparent", "opaque"]
x-enumDescriptions:
transparent: Generate an image with a transparent background.
opaque: Generate an image with a solid, non-transparent background.
quality:
type: string
description: Rendering quality of generated images, trading detail for generation speed and file size.
enum: ["low", "medium", "high"]
x-enumDescriptions:
low: Lower quality for faster generation and smaller file size.
medium: Balanced quality and generation speed.
high: Highest quality output with more detail.
size:
type: string
description: Pixel dimensions of generated images; choose square, portrait, or landscape to match the intended use.
enum: ["1024x1024", "1024x1536", "1536x1024"]
x-enumDescriptions:
1024x1024: Square image at 1024 by 1024 pixels.
1024x1536: Portrait image at 1024 by 1536 pixels.
1536x1024: Landscape image at 1536 by 1024 pixels.
outputFormat:
type: string
description: File format of generated images; use png or webp when transparency is needed.
enum: ["png", "webp", "jpeg"]
x-enumDescriptions:
png: PNG format with lossless compression, supports transparency.
webp: WebP format with efficient compression for web use.
jpeg: JPEG format with lossy compression for smaller file sizes.
mcp:
type: object
description: MCP server tool configuration
properties:
_mcpConnectionId:
type: string
format: objectId
x-celigo-refModel: connections
description: Connection to the MCP server
allowedTools:
type: array
description: |-
Specific tools to allow from the MCP server (all if
omitted). Each entry is either a plain tool name (legacy
form) or an object carrying display metadata.
items:
type: [string, object]
properties:
name:
type: string
maxLength: 256
description: Tool name as exposed by the MCP server.
title:
type: string
maxLength: 300
description: Display title shown for the tool.
description:
type: string
maxLength: 1000
description: Display description shown for the tool.
required: [name]
x-celigo-ai-guidance:
- |-
Object entries accept only name/title/description — any
other key fails validation with "unknown field(s)", and
an object without a non-empty name fails with a
missing-required-field error. String entries are the
legacy form and stay valid.
allowedPrompts:
type: array
deprecated: true
description: Not stored on a tool entry — a save drops it. Prompt names go in a `prompts[]` entry's own `mcp.allowedPrompts`.
items:
type: string
tool:
type: object
description: |
Reference to a Celigo Tool resource.
properties:
_toolId:
type: string
format: objectId
x-celigo-refModel: tools
description: Reference to the Tool resource
overrides:
type: object
description: Per-agent overrides for the tool's internal resources
properties:
connections:
type: array
description: |
Remaps the tool's abstract connections for this agent. Each entry pairs
the tool's abstract connection placeholder (`_abstractId`) with the
concrete connection (`_id`) to use for this agent; entries without
`_id` keep the tool's own default connection.
items:
type: [object, 'null']
required: [_abstractId]
properties:
_abstractId:
type: string
format: objectId
description: The tool's abstract connection placeholder being overridden.
_id:
type: string
format: objectId
x-celigo-refModel: connections
description: Concrete connection to use in place of the abstract placeholder.
prompts:
type: array
description: |
MCP prompt entries available to the agent. Each item references one MCP connection
and the prompt names allowed from it. Configured alongside `tools` in the form but
stored separately; an entry's `allowedPrompts` is what distinguishes a prompt entry
from an MCP tool entry (which carries `allowedTools`).
items:
type: object
properties:
type:
type: string
enum: ["mcp"]
x-enumDescriptions:
mcp: Connect to an MCP server to use its exposed prompts.
description: Type of prompt entry. Always "mcp".
mcp:
type: object
description: MCP server prompt configuration.
properties:
_mcpConnectionId:
type: string
format: objectId
x-celigo-refModel: connections
description: Connection to the MCP server.
allowedPrompts:
type: array
description: Prompt names to allow from the MCP server.
items:
type: string
resources:
$ref: '#/McpResources'
litellm:
type: object
description: |
LiteLLM proxy configuration. Used when `provider` is "gemini" or "anthropic".
LiteLLM provides a unified interface to multiple AI providers. Gemini-specific
settings are in `_overrides.gemini`; Claude-specific settings are in
`_overrides.anthropic`.
`model` is required when litellm is the active provider path.
properties:
model:
type: string
description: |-
LiteLLM model identifier. For Gemini, models are stored without the `gemini/`
prefix; for Anthropic, use the Claude model id (e.g. `claude-sonnet-4-6`).
examples: ["gemini-2.5-flash", "gemini-2.5-pro", "gemini-2.5-flash-lite", "claude-sonnet-4-6", "claude-opus-4-7"]
x-celigo-ui-override: >-
Required by the AI agent configuration form. Encoded here to mirror the form so
builders produce connectable configurations.
x-celigo-ai-guidance:
- |-
With platform-managed credentials (no `_connectionId` on the parent
import), the model must be a catalog row the platform offers for the
provider on its own credentials — Gemini and Anthropic Claude each have
their own rows. The save checks this: a model that is not in the catalog
fails with 422 `ai_model_not_in_catalog` on `aiAgent.litellm.model`, a
retired one with `ai_model_retired` (the message names the retirement
date and, when any remain eligible, the suggested next models), a
switched-off one with `ai_model_disabled`; a deprecated model saves. Never
swap the model on the user's behalf — relay the message and let them choose.
- |-
With a `_connectionId` (BYOK) a model the catalog does not list is passed
through to the provider as-is; a catalog-listed model is still governed by
its lifecycle row.
temperature:
type: number
minimum: 0
maximum: 2
description: Sampling temperature
x-celigo-ai-guidance:
- |-
When `provider` is "anthropic", the value must be between 0 and 1 — a higher value
fails the save with 422 `invalid_field_value`. Gemini accepts the full 0-2 range.
maxCompletionTokens:
type: number
minimum: 100
maximum: 128000
default: 5000
description: Maximum number of tokens in the response
topP:
type: number
minimum: 0.1
maximum: 1
description: Nucleus sampling parameter
seed:
type: number
description: Random seed for reproducible outputs
responseFormat:
type: object
description: Output format configuration
x-celigo-ai-guidance:
- |-
When `provider` is "anthropic", `type` must be "text" or "json_schema" — "blob" is
rejected with 422 `invalid_response_format`. As with the other providers, "json_schema"
requires `name` and `jsonSchema`.
properties:
type:
type: string
description: Output format type.
enum: ["text", "json_schema", "blob"]
x-enumDescriptions:
text: Free-form text response from the model.
json_schema: Structured JSON output conforming to a defined schema.
blob: Image generation — the run returns the stored image's `blobKey`, not `_text` or `_json`.
default: "text"
schemaMode:
type: string
enum: ["manual", "json"]
x-enumDescriptions:
manual: The UI opens `jsonSchema` in the visual field-by-field editor
json: The UI opens `jsonSchema` in the raw JSON editor
description: |-
Which editor the UI opens `jsonSchema` in; the schema sent to the provider is
the same either way, and the field is only meaningful when `type` is
"json_schema". Unset opens the visual editor, and an edit there rebuilds
`jsonSchema` from the field list, keeping only `type`, `properties`, `required`,
`items`, `description`, `enum`, `pattern`, length and numeric bounds,
`multipleOf`, `minItems`/`maxItems` and a `"null"` type — other keywords are
dropped without a prompt, and `integer` becomes `number`.
x-celigo-ai-guidance:
- |-
Set `json` on a schema written as raw JSON; keep a user's `manual` and stay inside
the keywords the visual editor keeps. A full PUT that omits the field clears it,
and a value outside the enum fails the save with 422 `enum` on
`aiAgent.litellm.responseFormat.schemaMode`.
name:
type: string
description: Name for the output format (used with json_schema).
x-celigo-ui-override: >-
Required by the AI agent form when output format type is json_schema. Encoded
here to mirror the form so builders produce connectable configurations.
strict:
type: boolean
description: When true, enforces strict schema validation on output.
default: false
jsonSchema:
type: object
description: JSON Schema for structured output. Required when `responseFormat.type` is "json_schema".
x-celigo-ui-override: >-
Required by the AI agent form when output format type is json_schema. Encoded
here to mirror the form so builders produce connectable configurations.
properties:
type:
type: string
description: Root JSON Schema type of the structured output; use "object" for record-shaped results.
enum: ["object", "array", "string", "number", "integer", "boolean"]
x-enumDescriptions:
object: JSON object with named properties.
array: Ordered list of values.
string: Text string value.
number: Numeric value including decimals.
integer: Whole number value without decimals.
boolean: True or false value.
properties:
type: object
additionalProperties: true
description: JSON Schema definitions for each field the structured output may contain.
required:
type: array
description: Property names the model must include in the structured output.
items:
type: string
additionalProperties:
type: boolean
description: When true, the structured output may include properties beyond those defined in `properties`.
if:
properties:
type:
const: json_schema
required: [type]
then:
required: [name, jsonSchema]
_overrides:
type: object
description: Provider-specific overrides
properties:
gemini:
type: object
description: |
Gemini-specific configuration overrides.
required: [systemInstruction]
properties:
systemInstruction:
type: string
maxLength: 1000000
description: |
System instruction for Gemini models. Equivalent to OpenAI's `instructions`.
Maximum 1,000,000 characters.
x-celigo-ui-override: >-
Required by the AI agent configuration form (the model's system prompt, the
Gemini equivalent of `instructions`). Encoded here to mirror the form so
builders produce connectable configurations.
tools:
type: array
description: Gemini-specific tools
items:
type: object
properties:
type:
type: string
enum: ["googleSearch", "urlContext", "fileSearch", "codeExecution", "mcp", "tool"]
x-enumDescriptions:
googleSearch: Use Google Search for real-time information grounding.
urlContext: Retrieve and use content from specified URLs.
fileSearch: Search through previously uploaded files.
codeExecution: "Enable Gemini's built-in code-execution tool; carries no configuration block, send `{type: codeExecution}` only"
mcp: Connect to an MCP server for additional external tools.
tool: Reference a reusable Celigo Tool resource.
description: |-
Type of Gemini tool. `codeExecution` is the Gemini spelling — `code_execution`
fails with 422 `enum`.
googleSearch:
type: object
description: Google Search configuration (empty object to enable)
urlContext:
type: object
description: URL context configuration (empty object to enable)
fileSearch:
type: object
description: File search configuration, used when type is "fileSearch".
properties:
fileSearchStoreNames:
type: array
description: Names of the file search stores the model can query.
items:
type: string
mcp:
type: object
description: MCP server tool configuration, used when type is "mcp".
properties:
_mcpConnectionId:
type: string
format: objectId
x-celigo-refModel: connections
description: Connection to the MCP server.
allowedTools:
type: array
description: |-
Specific tools to allow from the MCP server (all if
omitted). Each entry is either a plain tool name
(legacy form) or an object carrying display metadata
— same contract as the OpenAI `allowedTools`.
items:
type: [string, object]
properties:
name:
type: string
maxLength: 256
description: Tool name as exposed by the MCP server.
title:
type: string
maxLength: 300
description: Display title shown for the tool.
description:
type: string
maxLength: 1000
description: Display description shown for the tool.
required: [name]
allowedPrompts:
type: array
deprecated: true
description: Not stored on a tool entry — a save drops it. Prompt names go in a `prompts[]` entry's own `mcp.allowedPrompts`.
items:
type: string
tool:
type: object
description: Reference to a Celigo Tool resource, used when type is "tool".
properties:
_toolId:
type: string
format: objectId
x-celigo-refModel: tools
description: Reference to the Tool resource.
overrides:
type: object
description: Per-agent overrides for the tool's internal resources.
properties:
connections:
type: array
description: |
Remaps the tool's abstract connections for this agent. Each entry
pairs the tool's abstract connection placeholder (`_abstractId`)
with the concrete connection (`_id`) to use for this agent;
entries without `_id` keep the tool's own default connection.
items:
type: [object, 'null']
required: [_abstractId]
properties:
_abstractId:
type: string
format: objectId
description: The tool's abstract connection placeholder being overridden.
_id:
type: string
format: objectId
x-celigo-refModel: connections
description: Concrete connection to use in place of the abstract placeholder.
prompts:
type: array
description: |
MCP prompt entries available to the Gemini agent. Each item references one
MCP connection and the prompt names allowed from it. The presence of
`allowedPrompts` distinguishes a prompt entry from an MCP tool entry.
items:
type: object
properties:
type:
type: string
enum: ["mcp"]
x-enumDescriptions:
mcp: Connect to an MCP server to use its exposed prompts.
description: Type of prompt entry. Always "mcp".
mcp:
type: object
description: MCP server prompt configuration.
properties:
_mcpConnectionId:
type: string
format: objectId
x-celigo-refModel: connections
description: Connection to the MCP server.
allowedPrompts:
type: array
description: Prompt names to allow from the MCP server.
items:
type: string
resources:
$ref: '#/McpResources'
responseModalities:
type: array
description: Response output modalities
x-celigo-ai-guidance:
- |-
`["image"]` requires an image-capable Gemini model (currently `gemini-2.5-flash-image`).
Selecting it on a text-only model is rejected by the provider — pair image
output with the right model or omit.
items:
type: string
enum: ["text", "image"]
x-enumDescriptions:
text: Generate text content in the response.
image: Generate image content in the response.
default: ["text"]
topK:
type: number
description: Top-K sampling parameter for Gemini
thinkingConfig:
type: object
description: Controls Gemini's extended thinking capabilities
x-celigo-ai-guidance:
- |-
Supported on reasoning-capable Gemini models only (currently `gemini-2.5-pro`
and `gemini-2.5-flash`).
Setting `thinkingConfig` on a non-thinking model (e.g.
`gemini-2.5-flash-lite`,
`gemini-2.5-flash-image`) is rejected by the provider at runtime — omit the
field on those models.
properties:
includeThoughts:
type: boolean
description: When true, includes the model's thinking steps in the response.
x-celigo-ai-guidance:
- Whether to include thinking steps in the response
thinkingBudget:
type: number
minimum: 100
maximum: 4000
description: Maximum tokens allocated for thinking
thinkingLevel:
type: string
description: Controls how much thinking effort the model applies; use higher levels for complex, multi-step tasks at the cost of latency and tokens.
enum: ["minimal", "low", "medium", "high"]
x-enumDescriptions:
minimal: Least thinking effort for simple tasks.
low: Light thinking for moderately simple tasks.
medium: Balanced thinking effort for typical tasks.
high: Maximum thinking effort for complex, multi-step tasks.
imageConfig:
type: object
description: Gemini image generation configuration
properties:
aspectRatio:
type: string
description: Aspect ratio of generated images; choose a ratio matching the intended display format.
enum: ["1:1", "2:3", "3:2", "3:4", "4:3", "4:5", "5:4", "9:16", "16:9", "21:9"]
x-enumDescriptions:
"1:1": Square aspect ratio.
"2:3": Portrait aspect ratio (2 wide by 3 tall).
"3:2": Landscape aspect ratio (3 wide by 2 tall).
"3:4": Portrait aspect ratio (3 wide by 4 tall).
"4:3": Standard landscape aspect ratio (4 wide by 3 tall).
"4:5": Portrait aspect ratio (4 wide by 5 tall).
"5:4": Landscape aspect ratio (5 wide by 4 tall).
"9:16": Tall portrait aspect ratio for mobile screens.
"16:9": Widescreen landscape aspect ratio.
"21:9": Ultra-widescreen cinematic aspect ratio.
imageSize:
type: string
description: Output resolution of generated images; higher resolutions increase detail and file size.
enum: ["1K", "2K", "4K"]
x-enumDescriptions:
"1K": Standard resolution output around 1024 pixels.
"2K": High resolution output around 2048 pixels.
"4K": Ultra-high resolution output around 4096 pixels.
mediaResolution:
type: string
enum: ["low", "medium", "high"]
x-enumDescriptions:
low: Low resolution for faster processing and reduced token usage.
medium: Balanced resolution for typical media processing.
high: High resolution for maximum detail in media inputs.
description: Resolution for media inputs (images, video)
anthropic:
type: object
description: |
Claude-specific configuration overrides. Used when `provider` is "anthropic".
required: [systemInstruction]
properties:
systemInstruction:
type: string
maxLength: 1000000
description: |
System instruction for Claude models. Equivalent to OpenAI's `instructions`.
Maximum 1,000,000 characters.
x-celigo-ui-override: >-
Required by the AI agent configuration form (the model's system prompt, the
Claude equivalent of `instructions`). Encoded here to mirror the form so
builders produce connectable configurations.
topK:
type: number
minimum: 0
description: |-
Top-K sampling parameter. Deprecated on Claude models released after Claude Opus
4.6, which reject any value at runtime; set it only on older models.
thinkingConfig:
type: object
description: Controls Claude's extended thinking capabilities.
properties:
type:
type: string
enum: ["enabled", "disabled", "adaptive"]
x-enumDescriptions:
enabled: Turn on extended thinking with an explicit token budget.
disabled: Turn off extended thinking.
adaptive: Let the model decide how much to think based on task complexity.
default: "disabled"
description: Extended thinking mode.
budgetTokens:
type: number
minimum: 1024
description: |-
Maximum tokens allocated for thinking. Required when `type` is "enabled" —
omitting it fails the save with 422 `invalid_thinking_config`.
x-celigo-ui-override: >-
Required by the AI agent form when thinkingConfig.type is "enabled". Encoded
here to mirror the form so builders produce connectable configurations.
display:
type: string
enum: ["summarized", "omitted"]
x-enumDescriptions:
summarized: Include a summarized view of the model's thinking in the response.
omitted: Exclude the model's thinking from the response.
default: "summarized"
description: How thinking output is surfaced in the response.
effort:
type: string
enum: ["low", "medium", "high", "xhigh", "max"]
x-enumDescriptions:
low: Light thinking effort for moderately simple tasks.
medium: Balanced thinking effort for typical tasks.
high: Maximum standard thinking effort for complex tasks.
xhigh: Extended thinking effort beyond high for demanding tasks.
max: Highest available thinking effort.
description: |-
How much thinking effort the model applies; use with `type` "adaptive". Higher
values (`xhigh`, `max`) may be gated to specific Claude models by the provider.
if:
properties:
type:
const: enabled
required: [type]
then:
required: [budgetTokens]
serviceTier:
type: string
enum: ["auto", "standard_only"]
x-enumDescriptions:
auto: Let Anthropic select the appropriate service tier.
standard_only: Restrict processing to the standard service tier.
default: "auto"
description: Anthropic service tier for the request.
tools:
type: array
description: Claude-specific tools.
items:
type: object
properties:
type:
type: string
enum: ["tool", "mcp", "webSearch", "codeExecution"]
x-enumDescriptions:
tool: Reference a reusable Celigo Tool resource.
mcp: Connect to an MCP server for additional external tools.
webSearch: Search the web for real-time information.
codeExecution: "Enable Anthropic's built-in code-execution tool; carries no configuration block, send `{type: codeExecution}` only"
description: |-
Type of Claude tool. `codeExecution` is the Anthropic spelling — `code_execution`
fails with 422 `enum`.
tool:
type: object
description: Reference to a Celigo Tool resource, used when type is "tool".
properties:
_toolId:
type: string
format: objectId
x-celigo-refModel: tools
description: Reference to the Tool resource.
overrides:
type: object
description: Per-agent overrides for the tool's internal resources.
properties:
connections:
type: array
description: |
Remaps the tool's abstract connections for this agent. Each entry
pairs the tool's abstract connection placeholder (`_abstractId`)
with the concrete connection (`_id`) to use for this agent;
entries without `_id` keep the tool's own default connection.
items:
type: [object, 'null']
required: [_abstractId]
properties:
_abstractId:
type: string
format: objectId
description: The tool's abstract connection placeholder being overridden.
_id:
type: string
format: objectId
x-celigo-refModel: connections
description: Concrete connection to use in place of the abstract placeholder.
mcp:
type: object
description: MCP server tool configuration, used when type is "mcp".
properties:
_mcpConnectionId:
type: string
format: objectId
x-celigo-refModel: connections
description: Connection to the MCP server.
allowedTools:
type: array
description: |-
Specific tools to allow from the MCP server (all if omitted). Each
entry is either a plain tool name or an object carrying display
metadata — same contract as the OpenAI and Gemini `allowedTools`.
items:
oneOf:
- title: name
type: string
description: Tool name as exposed by the MCP server.
- title: tool
type: object
required: [name]
properties:
name:
type: string
maxLength: 256
description: Tool name as exposed by the MCP server.
title:
type: string
maxLength: 300
description: Display title shown for the tool.
description:
type: string
maxLength: 1000
description: Display description shown for the tool.
x-celigo-ai-guidance:
- |-
Object entries accept only name/title/description — any other key
fails the save with 422 `invalid_field_value` ("unknown field(s): …"),
and an object without `name` fails with 422 `missing_required_field`
("Expected field: name to be present.").
examples: [["hub_repo_search", {"name": "hub_repo_info", "title": "Repository info", "description": "Look up a repository by name"}]]
webSearch:
type: object
description: Web search configuration, used when type is "webSearch".
x-celigo-ai-guidance:
- |-
`allowedDomains` and `blockedDomains` are mutually exclusive — set at most
one. Sending both fails with 422 `invalid_hosted_tool`.
- |-
When `userLocation` is present, set at least one of `city`, `country`,
`region`, or `timezone`; an empty `userLocation` fails with 422
`missing_required_field`.
properties:
version:
type: string
pattern: "^\\d{8}$"
description: |-
Anthropic web search tool version (YYYYMMDD). Selects the tool version sent
on the wire; unsupported versions surface as an Anthropic 400.
examples: ["20250305"]
allowedDomains:
type: array
description: Domains the search may return results from. Mutually exclusive with `blockedDomains`.
items:
type: string
blockedDomains:
type: array
description: Domains to exclude from search results. Mutually exclusive with `allowedDomains`.
items:
type: string
userLocation:
type: object
description: |-
Approximate user location used to localize search results. When present, at
least one of `city`, `country`, `region`, or `timezone` must be set.
properties:
type:
type: string
enum: ["approximate"]
x-enumDescriptions:
approximate: Localize results to an approximate user location.
description: Location type. Always "approximate".
city:
type: string
maxLength: 256
description: City name for localizing search results.
country:
type: string
maxLength: 8
description: ISO 3166-1 alpha-2 country code for localizing search results.
region:
type: string
maxLength: 256
description: Region or state for localizing search results.
timezone:
type: string
maxLength: 64
description: IANA timezone for localizing search results.
prompts:
type: array
description: |
MCP prompt entries available to the Claude agent. Each item references one MCP
connection and the prompt names allowed from it. The presence of `allowedPrompts`
distinguishes a prompt entry from an MCP tool entry.
items:
type: object
properties:
type:
type: string
enum: ["mcp"]
x-enumDescriptions:
mcp: Connect to an MCP server to use its exposed prompts.
description: Type of prompt entry. Always "mcp".
mcp:
type: object
description: MCP server prompt configuration.
properties:
_mcpConnectionId:
type: string
format: objectId
x-celigo-refModel: connections
description: Connection to the MCP server.
allowedPrompts:
type: array
description: Prompt names to allow from the MCP server.
items:
type: string
resources:
$ref: '#/McpResources'
# Instance-level `required: [openai|litellm|model]` is what actually forces the
# provider's format to exist — nesting requireds under `properties` alone does not (a
# bare `{provider: openai}` would validate). The server 422s
# ("`model` is required for AI Agent import") when the block — or, for a catalog
# provider, the top-level `model` — is missing. `model` is server-required;
# `instructions` is a form-mirror (server accepts its omission — see the
# x-celigo-ui-override on that field).
if:
properties:
provider:
const: openai
required: [provider]
then:
required: [openai]
properties:
openai:
required: [model, instructions]
else:
if:
properties:
provider:
enum: [gemini, anthropic]
required: [provider]
then:
required: [litellm]
properties:
litellm:
required: [model]
else:
if:
properties:
provider:
enum: [mistral, xai, huggingface, deepseek, cohere, groq]
required: [provider]
then:
required: [model]
McpResources:
type: array
description: |-
Governed MCP resources — read-only reference content (policies, schemas, documentation)
pulled from connected MCP servers and made available to the agent as a consistent source
of truth. Each entry references one MCP connection and the specific resources allowed from it.
items:
type: object
required: [type, mcp]
properties:
type:
type: string
enum: ["mcp"]
x-enumDescriptions:
mcp: Reference read-only resources exposed by an MCP server.
description: Type of resource entry. Always "mcp".
mcp:
type: object
required: [_mcpConnectionId, allowedResources]
description: MCP server resource configuration.
properties:
_mcpConnectionId:
type: string
format: objectId
x-celigo-refModel: connections
description: Connection to the MCP server that exposes the resources.
allowedResources:
type: array
minItems: 1
description: |-
Resources to expose to the agent from the MCP server. Must contain at least one
entry; each entry identifies one resource by name and URI.
items:
type: object
required: [name, uri]
properties:
name:
type: string
description: Display name of the MCP resource.
examples: ["Data Retention Policy"]
uri:
type: string
description: URI that identifies the resource on the MCP server.
examples: ["resource://policies/data-retention"]
references/schemas/as2.yml
As2:
type: object
description: |-
Configures how outbound AS2 messages are built for this import. Required when the
_connectionId field references an AS2 connection. AS2 (Applicability Statement 2) transmits
EDI and other data securely over HTTP/S using S/MIME encryption and digital signatures; this
object names the payload file, the message identifier, retry behavior, and any extra headers.
required:
- fileNameTemplate
properties:
_tpConnectorId:
type: string
format: objectId
x-celigo-refModel: tradingpartnerconnectors
description: |-
Trading Partner Connector that supplies the partner-specific EDI and AS2 configuration.
When set, the import inherits the connector's pre-configured settings; omit to use only
the AS2 connection details.
examples: ["60a2c4e6f321d800129a1a3c"]
fileNameTemplate:
type: string
x-celigo-agent:
- handlebars
x-celigo-ui-override: >-
Required by the AS2 import form (fieldDefinitions/resources/imports/as2.js:
as2.fileNameTemplate is required:true, default 'file-{{timestamp}}'). Encoded to mirror
the form so builders produce connectable configurations.
description: |-
Handlebars template that names the file carrying each AS2 message payload. Include a
uniqueness token such as {{timestamp}} so concurrent or repeated sends do not collide;
defaults to file-{{timestamp}} in the form.
examples: ["file-{{timestamp}}", "orders-{{timestamp}}.edi", "{{record.documentNumber}}.edi"]
messageIdTemplate:
type: string
x-celigo-agent:
- handlebars
description: |-
Handlebars template that generates the Message-ID header for each AS2 message. Placeholders
are replaced at runtime; the resulting value must be globally unique and follow RFC 5322
header formatting. Leave unset to let the platform generate the Message-ID.
examples: ["{{timestamp}}-{{record.id}}@celigo.com", "<{{timestamp}}@example.com>"]
maxRetries:
type: number
default: 0
description: |-
Number of times a failed transmission is retried after transient errors such as network
failures or timeouts; does not affect the initial send. The form offers 1–5; set to 0
(the default) to report failure immediately without retrying.
examples: [0, 3, 5]
headers:
type: array
items:
type: object
properties:
name:
type: string
description: Name of the HTTP header sent with the AS2 message (for example, `content-disposition`). Header names are case-insensitive.
value:
type: string
description: Value sent for the header. Supports handlebars expressions rendered against the record.
x-celigo-agent:
- handlebars
description: |-
HTTP headers sent with the AS2 message transmission.
Supports both standard AS2 headers and custom headers required by trading partner agreements.
examples:
- [{ "name": "content-disposition", "value": "attachment; filename=order-{{record.id}}.edi" }]
references/schemas/base.yml
ImportBase:
type: object
description: Writable import fields shared by the request and response schemas.
properties:
_connectionId:
type: string
format: objectId
x-celigo-refModel: connections
description: |-
Connection this import uses to reach the destination system. The connection's type must
be compatible with the import's `adaptorType` (e.g. an `HTTPImport` needs an `http`
connection; `NetSuiteDistributedImport` and `NetSuiteHTTPImport` both need a `netsuite`
connection). Server-required — POST without it fails with 422 "Expected field:
_connectionId to be present" — except for the connection-less flavors (`ToolImport`,
`AiAgentImport`, `GuardrailImport`, `TopicImport`), which the server creates without
one. An Event Streams publisher (`TopicImport`) must omit it: with a connection set the
server validates the document as that connection's adaptor instead.
examples: ["60a2c4e6f321d800129a1a3c"]
_integrationId:
type: [string, 'null']
format: objectId
x-celigo-refModel: integrations
description: |-
Not stored on an import: a value sent on create or on a full update is dropped, and a GET
doesn't return the field. An import isn't owned by an integration; to find the flows and
other resources that use it, call `GET /v1/imports/{_id}/dependencies`.
_connectorId:
type: string
format: objectId
x-celigo-refModel: connectors
description: |-
Connector this import was created from, set when the import is part of an installed
integration app. The app's install sets it: a create that sends it is refused with 403
("Cannot create an Integration App resource").
examples: ["60a2c4e6f321d800129a1a3c"]
adaptorType:
type: string
description: |-
Selects the adaptor technology that executes this import, which determines the compatible
connection types and which adaptor-specific configuration object must also be supplied
(e.g. set `salesforce` when using `SalesforceImport`).
x-celigo-ai-guidance:
- |-
Use HTTPImport for all REST/SOAP APIs unless a specialized adaptor exists for the
target system; always pair the adaptorType with its matching configuration object.
- |-
For NetSuite, use NetSuiteDistributedImport (with the netsuite_da object) when the
Celigo SuiteApp or bundle is installed in the account — check with
`GET /v1/connections/{_id}/distributedApps` (`suiteapp.success` / `bundle.success`).
Use NetSuiteHTTPImport (with `nsDomainType` and the http object) when neither is
installed or the user asks for NetSuite's REST APIs; the platform's SuiteTalk connector
catalogs `POST /services/rest/record/v1/<type>` for inserts and
`PATCH /services/rest/record/v1/<type>/{id}` for updates, and saving links the step to
them. NetSuiteImport is a legacy adaptor for accounts without the Celigo SuiteApp 2.0 —
do not use it unless explicitly requested.
- |-
Use RDBMSImport (with the rdbms object) for built-in database types such as SQL Server,
MySQL, PostgreSQL, Snowflake, Oracle, and MariaDB. Use JDBCImport (with the jdbc
object) only for generic JDBC connections that are not built-in RDBMS types.
enum:
- HTTPImport
- FTPImport
- AS2Import
- S3Import
- NetSuiteImport
- NetSuiteDistributedImport
- NetSuiteHTTPImport
- SalesforceImport
- JDBCImport
- RDBMSImport
- MongodbImport
- DynamodbImport
- WrapperImport
- AiAgentImport
- GuardrailImport
- FileSystemImport
- ToolImport
- TopicImport
- RESTImport
x-enumDescriptions:
HTTPImport: Generic REST or SOAP API import via HTTP connections.
FTPImport: File import via FTP or SFTP connections.
AS2Import: File import via AS2 (Applicability Statement 2) protocol.
S3Import: File import to Amazon S3 storage.
NetSuiteImport: Legacy NetSuite import for accounts without the Celigo SuiteApp 2.0.
NetSuiteDistributedImport: NetSuite import using the distributed SuiteApp 2.0 adaptor (default for NetSuite when the SuiteApp or bundle is installed).
NetSuiteHTTPImport: NetSuite import over its REST APIs — SuiteTalk REST record endpoints or a custom RESTlet — on a netsuite connection, without the Celigo SuiteApp or bundle.
SalesforceImport: Salesforce-specific import using SOAP, REST, Metadata, or Composite APIs.
JDBCImport: Database import via a generic JDBC connection (not for built-in RDBMS types).
RDBMSImport: Database import for built-in RDBMS types such as SQL Server, MySQL, PostgreSQL, Snowflake, and Oracle.
MongodbImport: MongoDB database import for document-based operations.
DynamodbImport: Amazon DynamoDB import for NoSQL table operations.
WrapperImport: Custom adaptor import built outside of Celigo (rarely used).
AiAgentImport: AI agent import using OpenAI, Gemini, Anthropic Claude, or a BYOK catalog provider (Mistral, xAI, Hugging Face, DeepSeek, Cohere, Groq) for data processing.
GuardrailImport: Safety and compliance check import for PII detection, moderation, or AI evaluation.
FileSystemImport: Local or network file system import.
ToolImport: Import that executes a reusable Celigo Tool resource.
TopicImport: Publishes each record as a message to an Event Streams topic, configured through `topic`; needs no connection.
RESTImport: Legacy REST adaptor, superseded by HTTPImport — still accepted and present on older imports.
examples: ["HTTPImport", "NetSuiteDistributedImport", "NetSuiteHTTPImport", "SalesforceImport", "FTPImport", "AiAgentImport", "GuardrailImport", "TopicImport"]
nsDomainType:
type: string
enum:
- suitetalk
- restlet
x-lowercase: true
x-enumDescriptions:
suitetalk: NetSuite SuiteTalk REST web services — record endpoints under /services/rest/record/v1
restlet: A custom RESTlet script deployed in the NetSuite account, called at /app/site/hosting/restlet.nl?script=<id>&deploy=<n>
description: |-
Selects which NetSuite REST host a `NetSuiteHTTPImport` calls. The server derives the
full base URL from the connection's NetSuite account
(`https://<account>.suitetalk.api.netsuite.com` or `https://<account>.restlets.api.netsuite.com`)
and signs each request with the connection's token-based credentials, so each
`http.relativeURI` entry carries the complete path starting at `/services/rest/...` or
`/app/site/hosting/restlet.nl`. Ignored on other adaptor types.
x-celigo-ai-guidance:
- |-
Required on `NetSuiteHTTPImport`: omitting it fails the save with 422 "Path `nsDomainType`
is required." The import's `_connectionId` must be a netsuite connection — an http
connection fails with 422 invalid_connection_type.
- |-
With `suitetalk`, saving links `http._httpConnectorVersionId`, `_httpConnectorResourceId`
and `_httpConnectorEndpointIds` to the platform's SuiteTalk connector from each
`http.method` + `http.relativeURI` pair (for example `POST /services/rest/record/v1/customer`
links to that record type's insert endpoint), so the UI's Simple view renders an
API-authored step; `restlet` steps are not linked.
externalId:
type: [string, 'null']
description: External identifier for correlating the import with a record in another system.
examples: ["shopify-customer-import"]
as2:
$ref: './as2.yml#/As2'
dynamodb:
$ref: './dynamodb.yml#/Dynamodb'
http:
$ref: './http.yml#/Http'
ftp:
$ref: './ftp.yml#/Ftp'
jdbc:
$ref: './jdbc.yml#/Jdbc'
mongodb:
$ref: './mongodb.yml#/Mongodb'
netsuite:
$ref: './netsuite.yml#/NetSuite'
netsuite_da:
$ref: './netsuitedistributed.yml#/NetsuiteDistributed'
rdbms:
$ref: './rdbms.yml#/Rdbms'
s3:
$ref: './s3.yml#/S3'
wrapper:
$ref: './wrapper.yml#/Wrapper'
salesforce:
$ref: './salesforce.yml#/Salesforce'
tool:
$ref: './tool.yml#/Tool'
topic:
$ref: './topic.yml#/Topic'
file:
$ref: './file.yml#/File'
filesystem:
$ref: './filesystem.yml#/FileSystem'
aiAgent:
$ref: './aiagent.yml#/AiAgentConfig'
guardrail:
$ref: './guardrail.yml#/GuardrailConfig'
name:
type: string
minLength: 1
maxLength: 100
description: |-
Display name for the import, shown in the flow builder, job history, and error logs.
Descriptive, unique names indicating the destination system and purpose make large
accounts easier to manage.
examples: ["Shopify Customer Import", "NetSuite Sales Order Import"]
description:
type: [string, 'null']
description: |-
Free-text summary of what the import writes and why. Shown in the UI and available to
AI agents for context; has no effect on execution.
maxLength: 5120
unencrypted:
type: object
description: Custom configuration values stored without encryption and returned in API responses.
sampleData:
type: [object, array, string]
description: Sample input record used to preview and build the import's mappings.
distributed:
type: boolean
description: |-
When true, the import uses a distributed adaptor (such as `NetSuiteDistributedImport`)
that executes inside the target application rather than on Celigo's servers.
maxAttempts:
type: number
description: Maximum number of attempts made to deliver a record before it is marked as failed.
ignoreExisting:
type: boolean
description: |-
When true, records that already exist in the destination system are silently skipped
instead of being created or updated — used for create-only operations that must avoid
duplicates. Existing records are identified by the import's lookup configuration or by
a populated `ignoreExtract` field on the incoming record.
x-celigo-ai-guidance:
- |-
Set true for create/insert-only operations when the user asks to skip, ignore,
or avoid duplicating existing records.
Leave false for update, sync, or upsert operations.
Requires a configured lookup (or ignoreExtract) to identify existing records.
- |-
## EXAMPLES:
### Should set ignoreExisting: true
- "Create customer records in Shopify while ignoring existing customers" → ignoreExisting: true
- "Create vendor records while ignoring existing vendors" → ignoreExisting: true
- "Import new products only, skip existing ones" → ignoreExisting: true
- "Add orders to the system, ignore any that already exist" → ignoreExisting: true
- "Create accounts but don't update existing ones" → ignoreExisting: true
- "Insert new contacts, skip duplicates" → ignoreExisting: true
- "Create records, skipping any that match existing data" → ignoreExisting: true
### Should NOT set ignoreExisting: true
- "Update existing customer records" → ignoreExisting: false (update operation)
- "Create or update records" → ignoreExisting: false (upsert operation)
- "Sync all records" → ignoreExisting: false (sync/upsert operation)
- |-
## WHEN TO SET THIS TO TRUE (MUST DETECT THESE PATTERNS):
**ALWAYS set ignoreExisting to true** when the user's prompt includes ANY of these patterns:
### Primary Patterns (very common):
- "ignore existing" / "ignoring existing" / "ignore any existing"
- "skip existing" / "skipping existing" / "skip any existing"
- "while ignoring existing" / "while skipping existing"
- "skip duplicates" / "avoid duplicates" / "ignore duplicates"
### Secondary Patterns:
- "only create new" / "only add new" / "create new only"
- "don't update existing" / "avoid updating existing" / "without updating"
- "create if doesn't exist" / "add if doesn't exist"
- "insert only" / "create only"
- "skip if already exists" / "ignore if exists"
- "don't process existing" / "bypass existing"
### Context Clues:
- User says "create" or "insert" AND mentions checking/matching against existing data
- User wants to add records but NOT modify what's already there
- User is concerned about duplicate prevention during creation
- |-
## WHEN TO LEAVE THIS FALSE/UNDEFINED:
Leave this false or undefined when:
- The prompt doesn't mention skipping or ignoring existing records
- The import is only performing updates (no inserts)
- The prompt says "sync", "update", or "upsert"
- |-
## IMPORTANT:
- This field is typically used with INSERT operations
- Common pattern: "Create X while ignoring existing X" → MUST set to true
- |-
## IMPORTANT NOTES:
- This flag requires a properly configured lookup to identify existing records, or have an ignoreExtract field configured to identify the field that is used to determine if the record already exists.
- The lookup typically checks a unique identifier field (id, email, external_id, etc.)
- When true, existing records are silently skipped (not updated, not errored)
- Use this for insert-only or create-only operations
- Do NOT use this if the user wants to update existing records
- |-
## NOT FOR COMPOSITE UPSERTS
``ignoreExisting`` is for single-operation imports (CREATE-only)
that want to **skip** records that already exist. It is NOT
the field for composite upsert imports (those with both CREATE
and UPDATE in ``requestType``) — those route records between two
write paths instead of skipping, and use
``http.existingExtract`` / ``http.existingLookupName`` as their
decider field. ``ignoreExisting`` and ``existingExtract`` /
``existingLookupName`` are mutually exclusive — never set both
on the same import.
- |-
## ADAPTOR-LEVEL FIELDS YOU PAIR WITH THIS
Once ``ignoreExisting`` is enabled you must also set the field
that identifies the existence-check key. Two patterns,
mutually exclusive — pick exactly one based on whether the
incoming record already carries the destination id:
- ``http.ignoreExtract`` (or the adaptor's equivalent) — a
field name on the incoming record. Zero per-record HTTP
overhead. Use when an upstream step or source-system field
already populates the id on each record.
- ``http.ignoreLookupName`` (or the adaptor's equivalent) —
the name of a lookup in ``http.lookups`` that probes the
destination per record. Adds one HTTP request per record.
Use when the record has no field to inspect.
ignoreMissing:
type: boolean
description: |-
When true, records that do not already exist in the destination system are silently
skipped instead of producing errors — used for update-only operations. Existing records
are identified the same way as for `ignoreExisting`: the adaptor's ignore lookup
(`ignoreLookupName`) or a populated `ignoreExtract` field on the incoming record.
idLockTemplate:
type: [string, 'null']
description: |-
Handlebars template that generates a concurrency lock key for each record: records that
render the same key are never submitted at the same time, while records with different
keys are processed in parallel. When omitted, the record's trace key is the lock key (a
blank trace key means no lock; a custom `traceKeyTemplate` upstream arrives on the record
as `io_custom_trace_key`); a template replaces that default rather than adding to it.
The template is rendered from the record as it arrives at the import — before the import's
own lookups and mapping run — so a key built from one of the import's lookup results
(`{{lookup.<name>}}`) or from a mapped field renders empty, and an empty key is no lock at
all; a field added upstream by a lookup step's response mapping is on the record and works.
x-celigo-ai-guidance:
- |-
The lock is scoped to this import resource across every page and every concurrent run
that submits through it — including other flows sharing the import — not to one page,
and not to the connection: two different imports rendering the same key on the same
connection still run in parallel. Records within a single page are
sent one at a time regardless, so the lock matters when same-key records sit in different
pages or runs.
- |-
With `ignoreExisting` and an ignore lookup, the existence lookup runs inside the lock:
lookup-then-write for one key does not interleave with another record's lookup-then-write
for the same key. That pairing is the duplicate-prevention pattern for destinations that
can be queried but do not reject duplicates — key the lock on the business identifier the
write is keyed on. With the check but no lock, concurrent same-key pages all read a miss
before any write lands and all create. Connection `concurrencyLevel: 1` also closes that
race for the import (measured on a single import: its pages ran one at a time and the
same-key pages created once) — the lock is the per-key tool that leaves unrelated records
parallel.
- |-
Key it on a field the record carries when it reaches this import (`{{{record.externalId}}}-GR`,
or a field an upstream lookup step added through its response mapping), never on one of
this import's own lookup results; the key must render non-empty for the records that
matter, or those records have no lock.
examples: ["{{record.customerId}}", "{{{record.externalId}}}-GR"]
dataURITemplate:
type: [string, 'null']
x-celigo-agent:
- handlebars
description: |-
Handlebars template that builds a link back to each record in the destination
application's UI. The resolved URL is stored with error records in job history so users
can jump straight to the record.
examples: ["https://admin.shopify.com/customers/{{record.id}}"]
oneToMany:
$ref: '../common-schemas/one-to-many.yml#/OneToMany'
pathToMany:
$ref: '../common-schemas/path-to-many.yml#/PathToMany'
blobKeyPath:
type: string
x-celigo-excludeAdaptorTypes:
- AiAgentImport
- GuardrailImport
- ToolImport
description: |-
Path in the input record that holds the blob key identifying the file content to import.
At send time the platform follows this path, retrieves the referenced file from integrator.io
storage, and streams its bytes into the outgoing request. Not accepted on `AiAgentImport`,
`GuardrailImport`, or `ToolImport` — those adaptors process records only and reject the
field with 422 `not_supported`.
x-celigo-ai-guidance:
- |-
On AiAgentImport, GuardrailImport and ToolImport the save fails with 422 `not_supported`
("Field "blobKeyPath" is not supported for this import type."); route file content
through a blob-capable import (HTTP, FTP, S3) instead.
- |-
Optional on a `blob: true` file-provider import: when omitted the platform reads the
record's root `blobKey` — the field a blob export emits on its `{blobKey, fileMeta}`
record — so a file-transfer flow needs no `blobKeyPath`. Observed on an FTP import
(recorded runs writing files with no `blobKeyPath`) and on a Google Drive folder
upload (`http.type: file`, the marketplace file-transfer template's shape); other
file providers were not probed. Set it only when a response mapping placed the key
elsewhere; on those two providers its absence is not a cause of upload failures.
- |-
The value at this path must be a blobKey produced by an upstream blob
export or blob lookup and carried onto the record via that step's
response mapping (idiom: extract ``data.0.blobKey`` → generate
``blobKey``). A record field holding base64 file *content* is not a
blobKey — inline content is record data, not a storage reference, so
it is typically staged into blob storage first (blob steps support a
base64 ``blobFormat``) or sent as ordinary record data instead.
- |-
For multipart/form-data uploads, pair this field with an ``http.body``
parts array whose attachment part has ``"value": "{{blob}}"`` — the
platform substitutes the file bytes at that part. (APIs expecting
``multipart/related`` instead — e.g. Google Drive
``uploadType=multipart`` — take a hand-authored MIME body that
references ``{{blob}}`` directly; see the ``http.body`` guidance.)
The assembled multipart body is not available in audit logs, so
verify delivery from job results or the destination side rather
than log output.
examples: ["attachment.blobKey"]
blob:
type: boolean
x-celigo-excludeAdaptorTypes:
- AiAgentImport
- GuardrailImport
- ToolImport
description: |-
When true, this import transfers raw file content (blobs) to the destination rather than
structured records, reading each file through `blobKeyPath` (optional on FTP and Google
Drive file-provider imports, which fall back to the record's root `blobKey`). Not accepted on
`AiAgentImport`, `GuardrailImport`, or `ToolImport` — those adaptors process records only
and reject the flag with 422 `not_supported`.
x-celigo-ai-guidance:
- |-
On AiAgentImport, GuardrailImport and ToolImport the save fails with 422 `not_supported`
("Field "blob" is not supported for this import type."); never set it on those adaptors.
examples: [true, false]
assistant:
type: string
description: |-
Identifier for the connector assistant used to configure this import. For the cloud
file providers it also selects the provider-specific location rules —
`azuredatalakestoragegen2` (container in `file.directory.name`, file path split into
`http.relativeURI` and `file.fileName`) and `googlecloudstorage` (bucket in
`file.directory.storageRootId`).
examples: ["shopify", "azuredatalakestoragegen2", "googlecloudstorage"]
deleteAfterImport:
type: boolean
description: When true, the source file is deleted after it is successfully imported.
assistantMetadata:
type: object
description: Metadata associated with the connector assistant configuration.
useTechAdaptorForm:
type: boolean
description: |-
When true, the UI presents the full technical adaptor form for this import instead of
the simplified assistant form.
distributedAdaptorData:
type: object
description: Internal state stored by distributed adaptors (such as the NetSuite SuiteApp) for this import.
filter:
x-celigo-agent:
- filter
description: |-
Filter applied to incoming records before they are sent to the destination system.
Records that match continue through the import; records that don't skip this import,
counted as ignored, and continue to the next step. Filter expressions reference the
incoming record's fields.
allOf:
- $ref: '../common-schemas/filter.yml#/Filter'
x-celigo-ai-guidance:
- |-
## IMPORT-SPECIFIC BEHAVIOR
**Pre-Import Filtering**: Filters are applied to incoming records before they are sent to the destination system. A record that doesn't match is not sent, but it is not removed from the flow: it continues to the next step without this import's response-mapped fields, so a later step that writes those fields needs its own filter.
traceKeyTemplate:
type: string
x-celigo-agent:
- handlebars
description: |-
Handlebars template that overrides how each record's unique trace key is generated,
used to track records through the flow and match errors to records. Trace keys are
capped at 256 characters; a longer key is stored truncated from the middle, keeping
the beginning and end of the value. The trace key is also the import's default
concurrency lock key when `idLockTemplate` is not set; it never de-duplicates — a record
that arrives twice with the same key is processed twice.
x-celigo-ai-guidance:
- |-
## CRITICAL: ONLY VALID WHEN ``oneToMany: true``
On import resources, the platform REJECTS this field unless
``oneToMany`` is also ``true``. Validation error code:
``[cannot_set_traceKeyTemplate] The traceKeyTemplate field on
the import document can only be set in case of oneToMany.``
Default behavior (one-to-one imports) does NOT need a trace
key template -- omit the field entirely. Only emit it when
the import is genuinely fan-out (one source record produces
many destination calls) and you are explicitly setting
``oneToMany: true`` in the same payload.
examples: ["{{record.orderNumber}}"]
mockResponse:
type: array
description: |-
Predefined response records used in place of calling the destination system when
testing the import, so flows can run without writing real data. Records must be in
integrator.io canonical format — the server rejects any other shape with a 422.
x-celigo-ai-guidance:
- |-
Flow test runs and flow-builder previews NEVER execute imports —
they substitute this mockResponse, or the platform placeholder
``{id: "1234567890", mockResponse: "replace me"}`` when none is
saved. With the placeholder, downstream response mappings extract
nothing real and downstream steps' sample panels go empty — the
fix for "no data reaches the next step" in the builder is saving
a realistic mockResponse here, not editing downstream mappings.
- |-
``POST /v1/imports/preview`` is the exception: it executes the
composed destination request for real — mockResponse does not
protect that surface.
items:
type: object
properties:
statusCode:
type: integer
description: HTTP status code the mock returns (e.g. 200, 403).
id:
type: [string, integer]
description: Identifier echoed for the mocked record.
ignored:
type: boolean
description: When true, the mocked record reports as ignored rather than imported.
dataURI:
type: string
description: Data URI echoed for the mocked record.
errors:
type: array
description: Mock error entries returned with the record.
_json:
type: [object, array]
description: Mock response body.
_headers:
type: object
description: Mock response headers.
_ediProfileId:
type: string
format: objectId
x-celigo-refModel: ediprofiles
description: |-
EDI profile this import uses to generate outbound X12 or EDIFACT documents — it supplies
the envelope qualifiers, delimiters, version, and validation rules. Set it when the
import produces EDI output; omit it otherwise. Accepted on the file-writing imports —
FTP, AS2, S3 and HTTP file mode (`http.type: "file"`).
x-celigo-ai-guidance:
- >-
On an HTTP import set `http.type: "file"` (the folder mode of file-provider connectors)
and pair this field with `file.type: "filedefinition"` and `file.skipAggregation: true`
or `file.aggregation.type: "key"` — the platform saves EDI files per page or per key,
never aggregated per run.
parsers:
type: array
description: |-
Legacy parser configuration slot. The server initializes this field to an empty array
and current API writes never populate it; treat it as server bookkeeping rather than a
setting to configure.
hooks:
type: object
description: |-
Custom JavaScript hooks that run at fixed points in the import lifecycle for
transformations, validation, and custom result handling beyond what configuration alone
can express. A hook name not listed here (an export's `preSavePage`, for example) is
dropped on save without an error, so read `hooks` back after saving.
x-celigo-ai-guidance:
- |-
JavaScript hook references attached to lifecycle stages of the import.
Engage hooks ONLY when the user explicitly wants scripting OR a requirement
cannot be expressed through declarative configuration (mappings / filters /
responseTransform).
Hooks bypass the platform's declarative validation, so use them sparingly.
properties:
preMap:
type: object
x-celigo-agent:
- script
description: |-
Hook that runs on each page of records before the import's mappings are applied.
Commonly used to reshape or filter records ahead of mapping. Its changes shape only
the request this import sends: the next step receives the record as it arrived here,
plus whatever this step's response mapping merges back. A value later steps need
belongs in the source export's `preSavePage`, in this step's `postResponseMap` hook on
the flow, or in the later import's own `preMap`.
The function returns one element per record in `options.data`, in order:
`{ data: <record> }` sends the record and `{ errors: [{ code, message }] }` fails it.
An element without `data` — `{}`, or the record itself returned unwrapped — skips that
record with no error: nothing is sent and the record is reported `ignored: true` at
stage `preMapHook`. A return with more or fewer elements fails every record on the page
with `invalid_extension_response` ("Extension result doesn't contain the same number
of elements as the request object. Expected 2, got 1. Extension returned statusCode
200"), source `pre_map_hook`.
x-celigo-ai-guidance:
- |-
Pre-map JavaScript hook — runs once per page BEFORE the
mapping step, on the raw upstream records. Engage when the
user needs to reshape, validate, filter, or annotate records
before the mapping sees them (drop archived records,
normalize date strings). Don't
engage for static value substitution or simple field
renames — mappings handle those.
Runtime contract: ``options.data[]`` (the page of upstream
records) → array of the same length, each element
``{data: record}`` or ``{errors: [...]}``. A bare record in
the returned array is skipped without an error, so
``options.data.map(r => { r.total = t; return r })`` sends
nothing.
- |-
Verified live 2026-10-22 by invoking an HTTP import with a
Mapper 2.0 number mapping: one ``{data: record}`` per record
sent both records; the bare records sent nothing and both
came back ``ignored: true``, ``stage: preMapHook``, with no
error; one element for two records failed both with 422
``invalid_extension_response``.
allOf:
- $ref: '../common-schemas/hooks.yml#/Hook'
postMap:
type: object
x-celigo-agent:
- script
description: |-
Hook that runs after the import's mappings are applied but before records are sent
to the destination system. Commonly used for adjustments that need the mapped
(destination-shaped) record. Like `preMap`, its changes shape only the request this
import sends; the next step never sees them.
x-celigo-ai-guidance:
- |-
Post-map JavaScript hook — runs once per page AFTER the
mapping step but BEFORE the destination call. Engage when
the user needs to adjust the mapped output's shape in ways
the mapping schema can't express (XML/SOAP envelope
wrapping, conditional payload composition, collapsing
multiple mapped records into one batch payload). Engage
rarely — declarative mappings cover most cases.
Runtime contract: ``options.preMapData[]`` (the page as it
entered the mapping step) + ``options.postMapData[]`` (the
mapping output) → array same length as ``postMapData``.
allOf:
- $ref: '../common-schemas/hooks.yml#/Hook'
postSubmit:
type: object
x-celigo-agent:
- script
description: |-
Hook that runs after the destination system responds, with access to both the
submitted records and the destination's responses. Commonly used to inspect results
or adjust the response data passed downstream. Each `options.responseData` entry is
one submitted record's result, in order: `statusCode`, `errors` (`{code, message,
source}`, empty on success), `ignored`, `id`, and `_json` — the destination's parsed
response body, where its own error detail sits (`_headers` holds the response
headers).
x-celigo-ai-guidance:
- |-
Post-submit JavaScript hook — runs AFTER each batch of
destination writes, on the destination's response.
Runtime contract: ``options.responseData[]`` (one per
submitted record; ``{statusCode, errors, _json, ...}`` —
``id`` is the destination record's on real runs; flow test
runs substitute the saved ``mockResponse`` or the placeholder
response, whose envelope carries the mock's id, such as
``1234567890``) → array of the same length, typically mutated
to annotate or augment per-record errors.
allOf:
- $ref: '../common-schemas/hooks.yml#/Hook'
postAggregate:
type: object
x-celigo-agent:
- script
description: |-
Hook that runs after an aggregated file has been submitted to the destination, for
file-based imports that combine records into a single file.
x-celigo-ai-guidance:
- |-
Post-aggregate JavaScript hook — runs ONCE after a
file-aggregating import (FTP/S3-style) submits its
aggregated file. Only legal on those imports: the API
rejects it on HTTP records imports with a 422
``invalid_hook`` ("unsupported postAggregate hook") —
never emit it for non-file adaptors.
Runtime contract: ``options.postAggregateData``
(``{success, _json, code, message, source}``) → no return
value; side effects only (logging, notifying, downstream
triggers).
allOf:
- $ref: '../common-schemas/hooks.yml#/Hook'
sampleResponseData:
type: [object, array, string]
description: |-
Sample destination response kept with the import for authoring its response mapping.
Flow test runs and previews don't read it; they substitute `mockResponse`, or a
placeholder when none is saved.
responseTransform:
x-celigo-agent:
- transform
description: |-
Transformation that reshapes the destination system's response after records are
imported: it runs after the postSubmit hook, on each record's response body (`_json`),
and its output replaces that body, so response mapping can't read a field the
transform drops. Expression rules use version `"1"` (`rules`); version `"2"` is refused
on save. Commonly used to extract relevant fields from verbose API responses.
allOf:
- $ref: '../common-schemas/transform.yml#/Transform'
- type: object
properties:
expression:
type: object
properties:
version:
type: string
enum: ["1"]
x-enumDescriptions:
"1": Transformation 1.0 rules in `rules`, the only version an import's response transform accepts.
description: Must be `"1"` on an import's response transform; `"2"` is refused with 422.
x-celigo-ai-guidance:
- |-
Verified live 2026-10-21 on an HTTP import in a flow run: the postSubmit hook's changes
to `responseData[i]._json` were in the transform's input, the transform's output replaced
`_json` (only the mapped fields were kept), and response mapping then found nothing at a
dropped field's path. The execution-log stage is `responseTransformation`, and
`POST /v1/imports/{_id}/invoke` returns the transformed body.
- |-
A save with `expression.version: "2"` (`rulesTwoDotZero`) fails with 422 "`2` is not a
valid enum value for path `responseTransform.expression.version`." Write `version: "1"`
rules (`[[{extract, generate}, ...]]`).
- |-
## IMPORT-SPECIFIC BEHAVIOR
**Response Transformation**: Transforms the response data returned by the destination system after records have been imported. This allows you to reshape, filter, or enrich the response before response mapping reads it; the record passed downstream is the incoming record plus what response mapping copies out.
**Common Use Cases**:
- Extracting relevant fields from verbose API responses
- Normalizing response formats across different destination systems
- Adding computed fields based on response data
- Filtering out sensitive information before logging or further processing
- |-
## Transformation 1.0 rules only, and what response mapping reads
Expression rules here are Transformation 1.0 only (`expression.version: "1"`, the
`rules` array) or a script: a version "2" expression is refused with 422
(`` `2` is not a valid enum value for path `responseTransform.expression.version`. ``).
The rules read the parsed response body, and their output replaces `_json`. v1 rules
keep only the fields they generate, so the step's response mapping reads those
names (at the envelope root or under `_json`) and nothing the rules dropped; `id`
still resolves.
modelMetadata:
type: object
description: Metadata about the destination data model captured for this import. Rarely set.
mapping:
type: object
x-celigo-agent:
- mapping
description: |-
Mapper 1.0 mapping configuration. `fields` maps individual fields; `lists` maps
record lists, each with its own `fields` array. Superseded by Mapper 2.0
(`mappings`) but still widely used.
x-celigo-ai-guidance:
- |-
## Adaptor routing
REQUIRED for ``SalesforceImport`` and ``WrapperImport``: these
adaptors must use Mapper 1.0; Mapper 2.0 is not supported for
them. ``NetSuiteDistributedImport`` is also V1-required
but uses a NetSuite-specific variant at its adaptor-scoped copy
``netsuite_da.mapping`` (richer sublist / subRecord vocabulary) —
not this field.
Optional on other import adaptors (``HTTPImport``, ``RDBMSImport``,
``FTPImport``, ``JDBCImport``, etc.) when the user explicitly asks
for Mapper 1.0 — e.g. when the import wraps a legacy v1 connector
or is being migrated from an older integration. Those adaptors
default to Mapper 2.0 at the top-level ``mappings`` (plural) field.
Do not set this field unsolicited; the v2 default produces better
output for modern connectors.
An import that carries both this field and ``mappings`` runs only
``mappings``: the Mapper 1.0 rows are ignored, with no error. A
conversion to Mapper 2.0 therefore translates every row and removes
this field in the same save.
- |-
Allowed (but not the default) on other import adaptors such as
``HTTPImport``, ``RDBMSImport``, ``FTPImport``, and
``JDBCImport`` when the user explicitly asks for Mapper 1.0 —
for example when wrapping a legacy v1 connector or migrating
from an older integration. Those adaptors default to Mapper 2.0
at ``mappings.yml#/Mappings``; emit this shape only when V1 is
intended.
Shape is a flat ``{fields, lists}`` pair: ``fields`` maps body-level
destination columns; ``lists`` maps source arrays into destination
line-item / sublist arrays. Unlike Mapper 2.0 there is no recursive
``buildArrayHelper`` — arrays are expressed only through ``lists``.
Lookups referenced by ``lookupName`` live on the top-level
``resource.lookups`` array (mirrored under the adaptor key for
adaptors that maintain their own copy; on an HTTP import that
copy, ``http.lookups``, is the one a save keeps — see the mirror
invariant on ``lookups``). Lookups are independent of this
mapping: they are neither created nor cleared automatically when
this mapping changes.
properties:
fields:
type: array
description: Field-level mapping entries applied to each record.
items:
$ref: '#/MappingField'
x-celigo-ai-guidance:
- |-
Body-level field mappings. Each entry writes one destination
column from one source expression. Order has no semantic
meaning; entries are applied independently per record.
lists:
type: array
description: List-level mappings, each generating a sublist on the target record.
items:
type: object
properties:
generate:
type: string
description: Target sublist or array path to generate.
x-celigo-ai-guidance:
- |-
Destination array / line-item id (e.g. Salesforce child
relationship name, HTTP request sublist key, NetSuite
sublist name like ``item`` or ``addressbook``).
This identifies the array itself — do NOT add bracket
notation here. Per-element semantics belong on the inner
``fields[].extract`` paths (which path-walk from the source
record root and use ``[*]`` to drive iteration). Dot
notation IS allowed here when the destination places the
array inside a parent object (e.g. ``order.lines``).
fields:
type: array
description: Field mappings applied within each generated list item.
items:
$ref: '#/MappingField'
x-celigo-ai-guidance:
- |-
Per-element field mappings applied to each item in the extracted
array. Uses the same shape as top-level ``fields`` — same
extract / generate / lookupName / conditional semantics.
x-celigo-ai-guidance:
- |-
One source-array → destination-array projection. The destination is
identified by ``generate`` (e.g. a line-item collection name on the
target record). Each element of the source array is walked through
``fields`` to build the corresponding destination element.
- |-
Line-item / sublist mappings. Each entry projects an array from
the source record into a destination array, applying its own
``fields`` sub-mapping to each element.
mappings:
x-celigo-agent:
- mapping
description: |-
Field mappings that transform incoming records into the destination system's field
structure — renaming fields, converting types, building nested objects, and applying
formulas or lookups to derive values. The mapped output builds only the request this
import sends: the next step receives the record as it arrived here, plus whatever this
step's response mapping merges back.
x-celigo-ai-guidance:
- |-
## Extract evaluation root
``mappings[].extract`` JSONPath values are interpreted against
the root selected by the sibling ``inputContext`` field on this
import — ``record`` mode (the default) evaluates JSONPath
against the flat record itself (``"$.field"``), and ``envelope``
mode evaluates against the wrapped runtime envelope
(``"$.record.field"``). See ``inputContext`` for the full
per-mode contract. Handlebars expressions in ``extract`` always
reference ``{{record.field}}`` regardless — only JSONPath
behavior flips with ``inputContext``.
- |-
## JSON structure — record { } or rows [ ]
The shape of this array is the mapper's JSON structure. Named
top-level items produce an object record; ``http.body`` and the
other templates bind it as ``record``. Exactly one top-level item
with ``dataType: objectarray`` and no ``generate`` is the
``rows [ ]`` structure: the mapped record is an array — one
element per element the item's ``buildArrayHelper`` iterates —
and templates bind it as ``rows`` (``{{#each rows}}``), with
``record.*`` undefined. The mapping editor shows that row's
destination disabled; its save check accepts the unnamed root
only in the ``rows [ ]`` structure and fails any other row that
has a source and no ``generate`` with "One or more destination
field values not entered". Keep the root unnamed when editing a
``rows [ ]`` mapping — naming it turns the output into an object
and a ``{{#each rows}}`` body renders nothing.
- |-
## IMPORT-SPECIFIC BEHAVIOR
**Data Transformation**: Mappings define how fields from incoming records are transformed and mapped to the destination system's field structure. This enables data normalization, field renaming, value transformation, and complex nested object construction.
**Common Use Cases**:
- Mapping source field names to destination field names
- Transforming data types and formats
- Building nested object structures required by the destination
- Applying formulas and lookups to derive field values
allOf:
- $ref: '../common-schemas/mappings.yml#/Mappings'
lookups:
x-celigo-agent:
- lookup
description: |-
Top-level mirror of the adaptor-specific ``lookups`` array. Celigo persists lookups
in TWO locations on every import resource: ``<adaptor_key>.lookups`` (the typed,
adaptor-specific definition) and this top-level ``lookups`` array (a permissive copy).
x-celigo-ai-guidance:
- |-
## Mirror invariant
Both arrays MUST hold byte-for-byte identical content: write
both on every save and treat them as a single logical field.
On an HTTP import the adaptor copy decides: a save keeps
``http.lookups`` and mirrors it here, overwriting a copy here
that differs, and an array sent only here is dropped on create
and on PUT with no error. A mapping row that names one of its
lookups then fails every record with ``Unable to find lookup in
mapping configs - <name>``.
- |-
## Shape
The shape of each item depends on the adaptor. See the
``lookups`` property on the adaptor sub-schema for the canonical
definition:
- ``salesforce.lookups`` — SOQL-based (sObjectType, whereClause,
resultField, ...)
- ``netsuite_da.lookups`` — NetSuite search (recordType,
searchField, operator, ...)
- ``http.lookups`` — HTTP request (method, relativeURI, body,
extract, ...)
- ``jdbc.lookups`` / ``rdbms.lookups`` — SQL query (query,
extract, ...)
- ``wrapper.lookups`` — generic (map, _lookupCacheId, extract,
default)
Agents SHOULD generate and validate lookups against the adaptor
schema, then mirror the result here unchanged. The platform
ignores extraneous fields on this top-level array.
allOf:
- $ref: '../common-schemas/lookups.yml#/Lookups'
inputContext:
type: string
enum: [record, envelope]
x-enumDescriptions:
record: The import receives each record directly (default behavior).
envelope: The import receives records wrapped in an envelope with metadata.
description: Controls the shape of the input passed to the import's processing pipeline.
x-celigo-ai-guidance:
- |-
## v1 mappings ignore this field
Mapper 1.0 (``mapping`` field) and the NetSuite DA variant
(``netsuite_da.mapping``) use dot-paths instead of JSONPath,
and the v1 runtime always evaluates those dot-paths against
the flat record regardless of ``inputContext``. The v1
mapperProcessor does not traverse wrapped-envelope samples,
so envelope-mode framing is meaningless for v1 — agents
authoring v1 mappings emit bare dot-paths (``email``,
``customer.first_name``, ``line_items[*].sku``) in both
modes.
- |-
## Allowed values
- ``record`` (default — emitted when the field is absent /
empty / undefined): the mapper sees the FLAT record itself as
its evaluation root. ``mappings[].extract`` JSONPath values
MUST anchor directly at ``$`` with NO ``record.`` segment —
for example ``"$.orderId"`` or ``"$.customer.email"``.
PLACEHOLDER extracts use ``"$.PLACEHOLDER_<field>"``. This is
the dominant Celigo idiom and the right choice for new
imports.
- ``envelope``: the mapper sees a wrapped object of the shape
``{record: {...}, settings: ..., job: ...}`` — the same
runtime-context envelope Handlebars / SQL / GraphQL
templates render against. ``mappings[].extract`` JSONPath values
MUST include the ``record.`` segment — for example
``"$.record.orderId"`` or ``"$.record.customer.email"``.
PLACEHOLDER extracts use ``"$.record.PLACEHOLDER_<field>"``.
Pick this mode only when the import was authored against the
envelope shape on purpose (typically because an upstream step
intentionally feeds the wrapped runtime context through to
the import, or the customer is migrating a legacy config that
assumes the envelope).
- |-
## Handlebars expressions are NOT affected
Handlebars expressions inside ``extract`` (``{{record.field}}``,
``{{dateFormat ... record.field}}``, etc.) ALWAYS reference the
runtime context's ``record`` root regardless of
``inputContext`` — the Handlebars runtime always sees the
wrapped envelope. Only JSONPath behavior flips with this field.
- |-
## Agent guidance
Set this field explicitly on every new import the agent
authors. Leaving it unset is interpreted as ``record`` by the
platform, but emitting ``inputContext: record`` makes the
contract self-documenting and prevents the mapper from drifting
into envelope mode if an upstream change ever changes the
platform default. Read this value before writing
``mappings[].extract`` so the JSONPath form matches the runtime
root.
rest:
type: object
deprecated: true
additionalProperties: true
description: |-
Legacy REST adaptor configuration, present on imports created before the REST adaptor
was folded into `http`. Superseded by the `http` object.
settingsForm:
x-celigo-agent:
- settings_form
$ref: '../common-schemas/form.yml#/Form'
preSave:
$ref: '../common-schemas/pre-save.yml#/PreSave'
settings:
$ref: '../common-schemas/settings.yml#/Settings'
MappingField:
type: object
description: One Mapper 1.0 field-mapping entry.
properties:
generate:
type: string
description: Target field path to write on the destination record.
examples: ["billingAddress.city"]
x-celigo-ai-guidance:
- |-
Destination field path on the target record. Use the target
system's raw field id — for Salesforce the API name (e.g.
``AccountId``, ``CustomField__c``); for HTTP/REST the JSON
key the endpoint expects; for NetSuite the internal field id
(lowercase, no spaces).
Mapper 1.0 expands two notations into the destination
structure at write time:
- **Dot notation** — ``customer.email`` builds nested
objects, writing ``{customer: {email: value}}``. Use it
freely for any depth of nested object on adaptors whose
payloads are JSON (HTTP, REST, Shopify-style, etc.).
NetSuite body fields are flat — do not use dot notation
there.
- **Bracket-star notation** — ``addresses[*].company`` is the
idiomatic V1 expression for "this scalar belongs to every
element of the destination array as it is being
constructed." Two or more ``[*]``-suffixed fields with the
same array prefix collaborate to populate a single element
per source record, e.g. pairing
``customer.addresses[*].company`` with
``customer.addresses[*].city`` produces one address object
carrying both fields.
- **Numeric bracket notation** — ``addresses[0].company``
writes to a specific index. It works, but hardcodes the
slot and is only appropriate when the destination genuinely
targets a fixed position (rare). Prefer ``[*]`` whenever
the user describes "an address" / "a line" / "an entry";
reserve ``[N]`` for "the second address" and similar
explicit-index intents.
customer.email → {"customer": {"email": ...}}
customer.addresses[*].company → {"customer": {"addresses": [{"company": ...}]}}
customer.addresses[*].city ↑ adds "city" to the same address element
extract:
type: string
description: |
Source expression. Accepts:
- A dot path from the record root (e.g. `email`, `address.city`), read from the
incoming record. A path that matches no field writes nothing.
- Handlebars template (contains `{{`) for concatenation /
formatting / conditional logic (e.g. `{{firstName}} {{lastName}}`,
`{{#if mobile}}{{mobile}}{{else}}{{phone}}{{/if}}`). The template reads the
record's fields by bare name: `{{record.firstName}}` and `{{data.firstName}}`
render nothing, so the field is left out even when the record has the value.
A `$.` JSONPath (the Mapper 2.0 `mappings` syntax) reads nothing here, and a bare
string is read as a field path, not a constant: set constants in `hardCodedValue`.
Omit when using `hardCodedValue`.
examples: ["address.city"]
hardCodedValue:
type: [string, 'null']
description: Static value written to `generate` instead of extracting from the source record.
x-celigo-ai-guidance:
- |-
Static value written verbatim. Use it for every constant: a bare
string in ``extract`` is read as a field path. Mutually exclusive
with ``extract``.
dataType:
type: string
enum: [string, number, boolean, numberarray, stringarray, json]
x-lowercase: true
x-enumDescriptions:
string: Text string data type.
number: Numeric data type including integers and decimals.
boolean: True or false data type.
numberarray: Array of numeric values.
stringarray: Array of text string values.
json: Arbitrary JSON object or structure.
description: Data type coercion applied to the mapped value.
discardIfEmpty:
type: boolean
description: When true, the field is omitted from the output when the extracted value is empty.
x-celigo-ai-guidance:
- |-
When true, skip this field entirely when ``extract`` / lookup
produces an empty string, null or nothing. Without this flag an
empty string is written as ``""`` and null as null, either of which
may overwrite existing data; a missing source is left out either way.
immutable:
type: boolean
description: When true, the mapped value cannot be overwritten by later mapping steps.
lookupName:
type: string
description: Name of an entry in the import's `lookups` array (`http.lookups` on an HTTP import) used to translate the value.
x-celigo-ai-guidance:
- |-
Name of a lookup defined on ``resource.lookups`` whose result
replaces the extracted value. The looked-up record's
``resultField`` (or full record, depending on lookup shape)
becomes the mapped value.
default:
type: [string, 'null']
description: |-
Fallback value written when the source value is an empty string, null or
missing. Without a default, an empty string and null are written as they are
and a missing field is left out.
x-celigo-ai-guidance:
- |-
Use ``default``, not ``hardCodedValue``, for a fallback on a field that is
sometimes empty: ``hardCodedValue`` is written on every record and replaces
the values that exist.
extractDateFormat:
type: string
description: Date format of the source value, used to parse it before conversion.
x-celigo-ai-guidance:
- |-
Moment.js-style format describing how the source date string is
structured (e.g. ``MM/DD/YYYY``, ``YYYY-MM-DDTHH:mm:ssZ``).
extractDateTimezone:
type: string
description: Timezone applied when parsing the source date value.
x-celigo-ai-guidance:
- |-
IANA timezone of the source value (e.g. ``America/New_York``).
Set together with ``generateDateTimezone`` only when a real
timezone conversion is required.
generateDateFormat:
type: string
description: Date format applied to the value written to the destination.
generateDateTimezone:
type: string
description: Timezone applied when formatting the destination date value.
conditional:
type: object
description: |
Only apply this mapping entry when the given condition is satisfied.
properties:
when:
type: string
enum:
- record_created
- record_updated
- extract_not_empty
- lookup_not_empty
- lookup_empty
- expression
x-enumDescriptions:
record_created: Apply only when creating a new record.
record_updated: Apply only when updating an existing record.
extract_not_empty: Apply only when `extract` produces a non-empty value.
lookup_not_empty: Apply only when the paired lookup returns a value.
lookup_empty: Apply only when the paired lookup returns no value.
expression: Apply only when the accompanying `expression` evaluates truthy.
description: |
Condition that gates whether this mapping entry is applied.
`lookup_not_empty` / `lookup_empty` evaluate the lookup named by the
sibling `lookupName`; `expression` evaluates the sibling `expression`.
lookupName:
type: string
description: |
Lookup to evaluate for the `lookup_not_empty` / `lookup_empty`
conditions.
expression:
type: string
description: |
Expression evaluated when `when` is `expression`.
x-celigo-ai-guidance:
- |-
One Mapper 1.0 body field assignment. Exactly one of ``extract`` or
``hardCodedValue`` should carry the value; setting both is
rejected by the mapper processor.
references/schemas/clone-request.yml
CloneRequest:
type: object
description: Request body for cloning an import.
properties:
name:
type: string
description: Optional name for the cloned resource. If omitted, the server may generate a default clone name.
examples: ["Clone - Import Orders"]
connectionMap:
type: object
description: |
Mapping of original connection ids to replacement connection ids.
Keys are source connection ids on the original resource; values are target connection ids.
For an import that uses a connection, an empty map fails with 400
`invalid_connectionMap`, as it does for an export; map the connection to itself to reuse
it in the clone.
additionalProperties:
type: string
example:
5f7c579b6411271af4e7cefa: 64a1234567890abcdef12345
5f7c579b6411271af4e7cefb: 64a1234567890abcdef12346
_stackId:
type: string
format: objectId
x-celigo-refModel: stacks
description: |-
Stack that every stack reference in the cloned resources (hooks and wrapper
connections running on a stack) points at. Required when the clone preview reports
`stackRequired: true`: without it the clone fails with 400 `stack_not_found`. The
stack must belong to the target environment or be shared with it (404
`stack_not_found` for an unknown id, 400 `invalid_stack` otherwise).
examples: ["5f8d43a1b9e5a80011a35f2c"]
sandbox:
type: boolean
deprecated: true
description: Optional flag copied from the original resource when applicable.
additionalProperties: true
references/schemas/clone-response.yml
CloneResponse:
description: Response body for a clone operation. Some clone endpoints return the cloned resource, while others may return a list of related created resources.
oneOf:
- $ref: "./response.yml#/Import"
- type: array
items:
type: object
properties:
model:
type: string
description: Model name of the created resource (e.g., Flow, Export, Import).
_id:
type: string
format: objectId
description: Unique id of the created resource.
name:
type: string
description: Optional name of the created resource.
references/schemas/dynamodb.yml
Dynamodb:
type: object
description: |-
Configuration for DynamoDB imports. Required when the _connectionId field references a
DynamoDB connection; must not be included for other connection types. putItem writes a full
item document; updateItem modifies named attributes of an existing item identified by its
key.
required:
- region
- method
- tableName
properties:
region:
type: string
x-lowercase: true
enum: ["us-east-1", "us-east-2", "us-west-1", "us-west-2", "af-south-1", "ap-east-1",
"ap-south-1", "ap-northeast-1", "ap-northeast-2", "ap-northeast-3", "ap-southeast-1",
"ap-southeast-2", "ca-central-1", "eu-central-1", "eu-west-1", "eu-west-2",
"eu-west-3", "eu-south-1", "eu-north-1", "me-south-1", "sa-east-1"]
x-enumDescriptions:
us-east-1: US East (N. Virginia).
us-east-2: US East (Ohio).
us-west-1: US West (N. California).
us-west-2: US West (Oregon).
af-south-1: Africa (Cape Town).
ap-east-1: Asia Pacific (Hong Kong).
ap-south-1: Asia Pacific (Mumbai).
ap-northeast-1: Asia Pacific (Tokyo).
ap-northeast-2: Asia Pacific (Seoul).
ap-northeast-3: Asia Pacific (Osaka).
ap-southeast-1: Asia Pacific (Singapore).
ap-southeast-2: Asia Pacific (Sydney).
ca-central-1: Canada (Central).
eu-central-1: Europe (Frankfurt).
eu-west-1: Europe (Ireland).
eu-west-2: Europe (London).
eu-west-3: Europe (Paris).
eu-south-1: Europe (Milan).
eu-north-1: Europe (Stockholm).
me-south-1: Middle East (Bahrain).
sa-east-1: South America (Sao Paulo).
default: "us-east-1"
x-celigo-ui-override: >-
Required by the DynamoDB import form (dynamodb.region is required:true, default
us-east-1).
Encoded to mirror the form so builders produce connectable configurations.
description: |-
AWS region hosting the DynamoDB table. Determines the service endpoint used for all
requests and must match the region where the table is deployed.
x-celigo-ai-guidance:
- |-
Specifies the AWS region where the DynamoDB service is hosted and where all
database operations will be executed.
This property defines the precise geographical location of the DynamoDB instance,
which is essential for optimizing application latency,
ensuring compliance with data residency and sovereignty regulations,
and aligning with organizational infrastructure strategies.
Selecting the appropriate region helps minimize network latency by placing data
physically closer to end-users or application servers,
thereby improving performance and responsiveness.
Additionally, the chosen region influences service availability,
pricing structures, feature sets, and legal compliance requirements,
making it a critical configuration parameter.
The region value must be a valid AWS region identifier (such as "us-east-1" or
"eu-west-1") and is used to construct the correct service endpoint URL for all
DynamoDB interactions.
Proper configuration of this property is vital for seamless integration with AWS SDKs,
authentication mechanisms, and IAM policies that may be region-specific,
ensuring secure and efficient access to DynamoDB resources.
- |-
## FIELD BEHAVIOR:
- Determines the AWS data center location for all DynamoDB operations.
- Directly impacts network latency, data residency, and compliance adherence.
- Must be specified as a valid and supported AWS region code.
- Influences the endpoint URL, authentication endpoints, and service routing.
- Affects service availability, pricing, feature availability, and latency.
- Remains consistent throughout the lifecycle of the application unless explicitly changed.
- |-
## IMPLEMENTATION GUIDANCE:
- Validate the region against the current list of supported AWS regions to prevent misconfiguration and runtime errors.
- Use the region value to dynamically construct the DynamoDB service endpoint URL and configure SDK clients accordingly.
- Allow the region to be configurable and overrideable to support testing environments, multi-region deployments, disaster recovery, or failover scenarios.
- Ensure consistency of the region setting across all AWS service configurations within the application to avoid cross-region conflicts and data inconsistency.
- Consider the implications of region selection on data sovereignty laws, compliance requirements, and organizational policies.
- Monitor AWS announcements for new regions or changes in service availability to keep configurations up to date.
- |-
## EXAMPLES:
- "us-east-1" (Northern Virginia, USA)
- "eu-west-1" (Ireland, Europe)
- "ap-southeast-2" (Sydney, Australia)
- "ca-central-1" (Canada Central)
- "sa-east-1" (São Paulo, Brazil)
examples: ["us-east-1", "eu-west-1", "ap-southeast-2", "ca-central-1", "sa-east-1"]
method:
type: string
enum:
- putItem
- updateItem
x-enumDescriptions:
putItem: Create a new item or replace an existing item entirely in the DynamoDB
table.
updateItem: Modify specific attributes of an existing item in the DynamoDB
table without replacing it.
x-celigo-ui-override: >-
Required by the DynamoDB import form (dynamodb.method is required:true). Encoded
to mirror
the form so builders produce connectable configurations.
description: Write operation performed against the DynamoDB table for each record.
tableName:
type: string
x-celigo-ui-override: >-
Required by the DynamoDB import form (dynamodb.tableName is required:true).
Encoded to
mirror the form so builders produce connectable configurations.
description: |-
DynamoDB table that receives the imported records.
Must match an existing table in the target AWS account and region; table names are case-sensitive.
x-celigo-ai-guidance:
- |-
The name of the DynamoDB table to be accessed or manipulated,
serving as the primary identifier for routing API requests to the correct data store.
This string must exactly match the name of an existing DynamoDB table within the
specified AWS account and region,
including case sensitivity.
It is essential for performing operations such as reading, writing, updating,
or deleting items within the table,
ensuring that all data interactions target the intended resource accurately.
- |-
## FIELD BEHAVIOR:
- Specifies the exact DynamoDB table targeted for all data operations.
- Must be unique within the AWS account and region to prevent ambiguity.
- Directs API calls to the appropriate table for the requested action.
- Case-sensitive and must strictly adhere to DynamoDB naming conventions.
- Acts as a critical routing parameter in API requests to identify the data source.
- |-
## EXAMPLES:
- "Users"
- "Orders2024"
- "Inventory_Table"
- "CustomerData"
- "Sales-Data.2024"
- |-
## IMPORTANT NOTES:
- DynamoDB table names can be up to 255 characters in length.
- Table names are case-sensitive and must be used consistently across all API calls.
- The specified table must exist in the targeted AWS region before any operation is performed.
- Changing the table name requires updating all dependent code and redeploying applications as necessary.
- Incorrect table names will result in API errors such as ResourceNotFoundException.
- |-
## DEPENDENCY CHAIN:
- Dependent on the AWS account and region context where DynamoDB is accessed.
- Requires appropriate IAM permissions to perform operations on the specified table.
- Works in conjunction with other DynamoDB parameters such as key schema, attribute definitions, and provisioned throughput settings.
- Relies on network connectivity and AWS service availability for successful API interactions.
- |-
## TECHNICAL DETAILS:
- Represented as a string value.
examples: ["Customers", "Orders", "Product-Catalog", "Users", "Orders2024",
"Inventory_Table", "CustomerData", "Sales-Data.2024"]
partitionKey:
type: string
x-celigo-agent:
- handlebars
x-celigo-ui-override: >-
Required by the DynamoDB import form when method is updateItem (dynamodb.partitionKey
required, visibleWhen method=updateItem).
description: |-
Value of the target item's partition key for the updateItem operation; together with
sortKey (when the table defines one) it identifies the item each record writes to.
Supports handlebars to reference the incoming record's fields.
examples: ["{{record.id}}", "{{record.customerId}}"]
sortKey:
type: string
x-celigo-agent:
- handlebars
description: |-
Value of the target item's sort key; supports handlebars. Omit when the table defines only
a partition key.
examples: ["{{record.timestamp}}", "{{record.lineId}}"]
itemDocument:
type: string
x-celigo-agent:
- handlebars
description: |-
JSON document written to the table for the putItem operation, containing all attribute
names and values for the item, including its key attributes. Supports handlebars.
x-celigo-ai-guidance:
- |-
## EXAMPLES:
- `{ "UserId": "12345", "Name": "John Doe", "Age": 30, "Preferences": { "Language": "en", "Notifications": true } }`
examples: ['{"id": "{{record.id}}", "name": "{{record.name}}", "total": {{record.total}} }']
updateExpression:
type: string
x-celigo-ui-override: >-
Required by the DynamoDB import form when method is updateItem (dynamodb.updateExpression
required, visibleWhen method=updateItem).
description: |-
DynamoDB update expression naming which attributes to set, add, or remove on the item for
the updateItem operation. Only the named attributes change; reference attribute names and
values through the expressionAttributeNames and expressionAttributeValues placeholders.
x-celigo-ai-guidance:
- |-
A string that defines the specific attributes to be updated, added,
or removed within a DynamoDB item using DynamoDB's UpdateExpression syntax.
This expression provides precise control over how item attributes are modified
by including one or more of the following clauses:
SET (to assign or overwrite attribute values), REMOVE (to delete attributes),
ADD (to increment numeric attributes or add elements to sets),
and DELETE (to remove elements from sets).
The updateExpression enables atomic, conditional,
and efficient updates without replacing the entire item,
ensuring data integrity and minimizing write costs.
It must be carefully constructed using placeholders for attribute names and
values—ExpressionAttributeNames and ExpressionAttributeValues—to avoid conflicts
with reserved keywords and prevent injection vulnerabilities.
This expression is essential for performing complex update operations in a single request,
supporting both simple attribute changes and advanced manipulations like
incrementing counters or modifying sets.
- |-
## FIELD BEHAVIOR:
- Specifies one or more atomic update operations on a DynamoDB item's attributes.
- Supports combining multiple update actions (SET, REMOVE, ADD, DELETE) within a single expression, separated by spaces.
- Requires strict adherence to DynamoDB's UpdateExpression syntax and semantics.
- Works in conjunction with ExpressionAttributeNames and ExpressionAttributeValues to safely reference attribute names and values.
- Only affects the attributes explicitly mentioned; all other attributes remain unchanged.
- Enables conditional updates when combined with ConditionExpression for safe concurrent modifications.
- |-
## IMPLEMENTATION GUIDANCE:
- Construct the updateExpression using valid DynamoDB syntax, incorporating clauses such as SET, REMOVE, ADD, and DELETE as needed.
- Always use placeholders (e.g., '#attrName' for attribute names and ':value' for attribute values) to avoid conflicts with reserved words and enhance security.
- Validate the expression syntax and ensure all referenced placeholders are defined before executing the update operation.
- Combine multiple update actions by separating them with spaces within the same expression string.
- Test expressions thoroughly to prevent runtime errors caused by syntax issues or missing placeholders.
- Use SET for assigning or overwriting attribute values, REMOVE for deleting attributes, ADD for incrementing numbers or adding set elements, and DELETE for removing set elements.
- Avoid using reserved keywords directly; always substitute them with ExpressionAttributeNames placeholders.
- |-
## EXAMPLES:
- "SET #name = :newName, #age = :newAge REMOVE #obsoleteAttribute"
- "ADD #score :incrementValue DELETE #tags :tagToRemove"
- "SET #status = :statusValue"
examples: ["SET #name = :name, #total = :total", "SET #status = :status REMOVE
#temp", "SET #name = :newName, #age = :newAge REMOVE #obsoleteAttribute",
"ADD #score :incrementValue DELETE #tags :tagToRemove", "SET #status = :statusValue"]
conditionExpression:
type: string
description: |-
Condition evaluated against the existing item before the write executes.
When the condition evaluates to false, the operation is aborted and no changes are made.
x-celigo-ai-guidance:
- |-
A string that defines the conditional logic to be applied to a DynamoDB operation,
specifying the precise criteria that must be met for the operation to proceed.
This expression leverages DynamoDB's condition expression syntax to evaluate
attributes of the targeted item,
enabling fine-grained control over operations such as PutItem, UpdateItem,
DeleteItem, and transactional writes.
It supports a comprehensive set of logical operators (AND, OR, NOT),
comparison operators (=, <, >, BETWEEN, IN), functions (attribute_exists,
attribute_not_exists, begins_with, contains, size),
and attribute path notations to construct complex, nested conditions.
This ensures data integrity by preventing unintended modifications and enforcing
business rules at the database level.
- |-
## FIELD BEHAVIOR:
- Determines whether the DynamoDB operation executes based on the evaluation of the condition against the item's attributes.
- Evaluated atomically before performing the operation; if the condition evaluates to false, the operation is aborted and no changes are made.
- Supports combining multiple conditions using logical operators for complex, multi-attribute criteria.
- Utilizes built-in functions to inspect attribute existence, value patterns, and sizes, enabling sophisticated conditional checks.
- Applies exclusively to the specific item targeted by the operation, including nested attributes accessed via attribute path notation.
- Influences transactional operations by ensuring all conditions in a transaction are met before committing.
- |-
## IMPLEMENTATION GUIDANCE:
- Always use ExpressionAttributeNames and ExpressionAttributeValues to safely reference attribute names and values, avoiding conflicts with reserved keywords and improving readability.
- Validate the syntax of the condition expression prior to request submission to prevent runtime errors and failed operations.
- Carefully construct expressions to handle edge cases, such as attributes that may be missing or have null values.
- Combine multiple conditions logically to enforce precise constraints and business logic on the operation.
- Test condition expressions thoroughly across diverse data scenarios to ensure expected behavior and robustness.
- Keep expressions as simple and efficient as possible to optimize performance and reduce request complexity.
- |-
## EXAMPLES:
- "attribute_exists(#name) AND #age >= :minAge"
- "#status = :activeStatus"
- "attribute_not_exists(#id)"
- "#score BETWEEN :low AND :high"
- "begins_with(#title, :prefix)"
- "contains(#tags, :tagValue) OR size(#comments) > :minComments"
- |-
## IMPORTANT NOTES:
- The conditionExpression is optional but highly recommended to safeguard against unintended overwrites, deletions, or updates.
examples: ["attribute_not_exists(#pk)", "#version = :expectedVersion", "attribute_exists(#name)
AND #age >= :minAge", "#status = :activeStatus", "attribute_not_exists(#id)",
"#score BETWEEN :low AND :high", "begins_with(#title, :prefix)", "contains(#tags,
:tagValue) OR size(#comments) > :minComments"]
expressionAttributeNames:
type: string
description: |-
JSON placeholder-to-name map for attribute names referenced in expressions, for example {"#N": "Name"}.
Use it when an attribute name is a reserved word or contains special characters; placeholder keys start with #.
x-celigo-ai-guidance:
- |-
expressionAttributeNames is a map of substitution tokens used to safely
reference attribute names within DynamoDB expressions.
This property allows you to define placeholder tokens for attribute names that
might otherwise cause conflicts due to being reserved words,
containing special characters, or having names that are dynamically generated.
By using these placeholders,
you can avoid syntax errors and ambiguities in expressions such as ProjectionExpression,
FilterExpression, UpdateExpression, and ConditionExpression,
ensuring your queries and updates are both valid and maintainable.
- |-
## FIELD BEHAVIOR:
- Functions as a dictionary where each key is a placeholder token starting with a '#' character, and each value is the actual attribute name it represents.
- Enables the use of reserved words, special characters, or otherwise problematic attribute names within DynamoDB expressions.
- Supports complex expressions by allowing multiple attribute name substitutions within a single expression.
- Prevents syntax errors and improves clarity by explicitly mapping placeholders to attribute names.
- Applies only within the scope of the expression where it is defined, ensuring localized substitution without affecting other parts of the request.
- |-
## EXAMPLES:
- `{"#N": "Name", "#A": "Age"}` to substitute attribute names "Name" and "Age" in an expression.
- `{"#status": "Status"}` when "Status" is a reserved word or needs escaping in an expression.
- `{"#yr": "Year", "#mn": "Month"}` for expressions involving multiple date-related attributes.
- `{"#addr": "Address", "#zip": "ZipCode"}` to handle attribute names with special characters or spaces.
- `{"#P": "Price", "#Q": "Quantity"}` used in an UpdateExpression to increment numeric attributes.
examples: ['{"#name": "Name", "#status": "status"}']
expressionAttributeValues:
type: string
x-celigo-agent:
- handlebars
description: |-
JSON placeholder-to-value map for attribute values referenced in expressions; placeholder
keys start with a colon (e.g. ":val") and each value is a DynamoDB attribute-value object
such as {"S": "text"}. Supports handlebars.
x-celigo-ai-guidance:
- |-
expressionAttributeValues is a map of substitution tokens for attribute values
used within DynamoDB expressions.
These tokens serve as placeholders that safely inject values into expressions
such as ConditionExpression,
FilterExpression, UpdateExpression, and KeyConditionExpression.
By separating data from code,
this mechanism prevents injection attacks and syntax errors,
ensuring secure and reliable query and update operations.
Each key in this map is a placeholder string that begins with a colon (":"),
and each corresponding value is a DynamoDB attribute value object representing
the actual data to be used in the expression.
This design enforces that user input is never directly embedded into expressions,
enhancing both security and maintainability.
- |-
## FIELD BEHAVIOR:
- Contains key-value pairs where keys are placeholders prefixed with a colon (e.g., ":val1") used within DynamoDB expressions.
- Each key maps to a DynamoDB attribute value object specifying the data type and value, supporting all DynamoDB data types including strings, numbers, binaries, lists, maps, sets, and booleans.
- Enables safe and dynamic substitution of attribute values in expressions, avoiding direct string concatenation and reducing the risk of injection vulnerabilities.
- Mandatory when expressions reference attribute values to ensure correct parsing and execution of queries or updates.
- Placeholders are case-sensitive and must exactly match those used in the corresponding expressions.
- Supports complex data structures, allowing nested maps and lists as attribute values for advanced querying scenarios.
- |-
## IMPLEMENTATION GUIDANCE:
- Always prefix placeholder keys with a colon (":") to clearly identify them as substitution tokens within expressions.
- Ensure every placeholder referenced in an expression has a corresponding entry in expressionAttributeValues to avoid runtime errors.
- Format each value according to DynamoDB's attribute value specification, such as { "S": "string" }, { "N": "123" }, { "BOOL": true }, or complex nested structures.
- Avoid using reserved words, spaces, or invalid characters in placeholder names to prevent syntax errors and conflicts.
- Validate that the data types of values align with the expected attribute types defined in the table schema to maintain data integrity.
- When multiple expressions are used simultaneously, maintain unique and consistent placeholder names to prevent collisions and ambiguity.
- Use expressionAttributeValues in conjunction with expressionAttributeNames when attribute names are reserved words or contain special characters to ensure proper expression parsing.
- Consider the size limitations of expressionAttributeValues to avoid exceeding DynamoDB's request size limits.
examples: ['{":name": {"S": "{{record.name}}"}, ":total": {"N": "{{record.total}}"}}']
ignoreExtract:
type: string
x-celigo-agent:
- handlebars
description: |-
Path to the field in the source record used to determine whether the record already exists.
Only used when ignoreExisting is set on the import.
examples: ["{{record.id}}"]
if:
properties:
method:
const: updateItem
required:
- method
then:
required:
- partitionKey
- updateExpression
references/schemas/file.yml
File:
type: object
x-celigo-excludeAdaptorTypes:
- RDBMSImport
- JDBCImport
- SalesforceImport
- NetSuiteImport
- MongodbImport
x-celigo-ai-guidance:
- |-
Never author or patch `file` on an `RDBMSImport`. On a bulk load the API echoes a
platform-derived `file` object whose values a save cannot change: the write is accepted,
the change is discarded, and a read-back shows the original values. Shape the records
before the write (a preMap/postMap hook, the mapping, a transform) or correct the source
data instead.
description: |-
**CRITICAL: This object is REQUIRED for all file-based import adaptor types.**
**When to include this object**
✅ **MUST SET** when `adaptorType` is one of:
- `S3Import`
- `FTPImport`
- `AS2Import`
- `HTTPImport` in file mode (`http.type: "file"` — the folder mode of the cloud file
providers such as Google Drive, Box, Dropbox, Azure Blob, Google Cloud Storage and Celigo
Storage); the same `file` object, including `type: filedefinition` with `_ediProfileId`
for EDI, applies as on S3 / FTP
❌ **DO NOT SET** for non-file-based imports like:
- `SalesforceImport`
- `NetSuiteImport`
- `HTTPImport` in records mode (`http.type` unset — a REST endpoint receiving records)
- `MongodbImport`
- `RDBMSImport`
- `JDBCImport`
**Read-only on database bulk loads**
An `RDBMSImport` whose `rdbms.queryType` is `bulk_load` returns a `file` object anyway: the
platform's description of the staging file it generates for the load (delimiter, quoting,
newline handling, and how the load is batched into files). It is derived on every read,
never stored — a save that changes it is accepted and the change discarded (the audit log
still records the request), and the import editor has no control for it. Change what
reaches the destination with a preMap/postMap hook, the mapping, or a transform, or correct
the source data.
**Minimum required fields**
For most file imports, you need at minimum:
- `fileName`: The output file name (supports Handlebars like `{{timestamp}}`)
- `type`: The file format (json, csv, xml, xlsx)
- `aggregation.type`: How the run's records are batched into files (`all` for one file per run)
**Example (S3 Import)**
```json
{
"file": {
"fileName": "customers-{{timestamp}}.json",
"aggregation": { "type": "all" },
"type": "json"
},
"s3": {
"region": "us-east-1",
"bucket": "my-bucket",
"fileKey": "customers-{{timestamp}}.json"
},
"adaptorType": "S3Import"
}
```
properties:
fileName:
type: string
description: |-
**REQUIRED for file-based imports.**
The name of the file to be created/written. Supports Handlebars expressions for dynamic naming,
rendered against the incoming record — a file-transfer import keeps the source file's name from
the record's `fileMeta`, whose field names are the exporting provider's
(`{{record.fileMeta.fileName}}` from an FTP source, `fileMeta.name` from Google Drive).
**Default behavior**
- When the user does not specify a file naming convention, **always default to timestamped filenames** using `{{timestamp}}` (e.g., `"items-{{timestamp}}.csv"`). This ensures each run produces a unique file and avoids overwriting previous exports.
- Only use a fixed filename (without timestamp) if the user explicitly requests overwriting or a fixed name.
**Common patterns**
- `"data-{{timestamp}}.json"` - Timestamped JSON file (DEFAULT — use this pattern when not specified)
- `"export-{{timestamp 'YYYY-MM-DD'}}.csv"` - Date-stamped CSV file
- `"{{recordType}}-backup.xml"` - Dynamic record type naming
**Examples**
- `"customers-{{timestamp}}.json"`
- `"orders-export.csv"`
- `"inventory-{{timestamp 'YYYY-MM-DD'}}.xlsx"`
**Important**
- For S3 imports, this should typically match the `s3.fileKey` value
- For FTP imports, this should typically match the `ftp.fileName` value; the one blob file-transfer run on record set only this field and left `ftp.fileName` unset
x-celigo-ai-guidance:
- |-
`fileMeta` is not a root key of this template's context (it is one only on
file-provider export and lookup steps): `{{fileMeta.fileName}}` renders empty and the
file has no name; on an FTP import the record then fails with `missing_s3_key` "Can not
process upload file without computedObject.fileName.", labeled source `internal`
although the template is the cause. The incoming record carries the source file's
metadata — on an FTP blob import `{{record.fileMeta.fileName}}` keeps the source file's
name at run time (the only spelling proven by a run). The field's editor preview
(data version 1, its default) exposes the record as `data` — the `{blobKey, fileMeta}`
record on a `blob: true` import, the page's record array otherwise — so `record.`
previews empty there; judge by a run. The metadata field names are the exporting
provider's — read them off the incoming sample record.
examples: ["customers-{{timestamp}}.json", "orders-export.csv"]
skipAggregation:
type: boolean
description: |-
Legacy companion of `aggregation.type`: `true` when the run's records are written one
file per page, `false` when they are combined across the run or grouped by key.
Validation reads the value sent — an EDI file definition needs `true` unless
`aggregation.type` is `key`, and `batchSize` needs `true` — then the save rewrites it
from `aggregation.type`, so send the two as a consistent pair.
default: false
x-celigo-ai-guidance:
- |-
Send `skipAggregation` and `aggregation.type` as a consistent pair, exactly as the UI
does: `page` with `true`; `all` or `key` with `false`. Validation reads the SENT flag —
a `file.type: filedefinition` import saved with `page` and no `skipAggregation: true`
fails with 422 `not_supported` ("Aggregation is not supported for EDI files, please set
skipAggregation to true in Imports"), and `batchSize` fails with 422 `not_supported`
("File imports do not support batching with aggregation") unless the sent flag is
`true`. The SAVED flag is then derived from `aggregation.type`, so this flag alone never
says which mode is active — read `aggregation.type`. It does not mean one file per record.
type:
type: string
enum:
- json
- csv
- xml
- xlsx
- filedefinition
x-enumDescriptions:
json: JSON format, best for structured or nested data and API payloads; most common for API data
csv: Comma-separated values format for flat tabular data.
xml: XML format for XML-based integrations and SOAP services.
xlsx: Microsoft Excel spreadsheet format.
filedefinition: Custom file definition format for complex, proprietary, or EDI file structures.
description: |-
**REQUIRED for file-based imports.**
The format of the output file.
Select the format that matches the target system's requirements.
x-celigo-ai-guidance:
- |-
## SELECTION GUIDANCE:
- Use `"json"` for structured/nested data, API payloads
- Use `"csv"` for flat tabular data, spreadsheet-like records
- Use `"xml"` for XML-based integrations, SOAP services
- Use `"xlsx"` for Excel-compatible outputs
encoding:
type: string
enum:
- utf8
- win1252
- utf-16le
- gb18030
- macroman
- iso88591
- shiftjis
x-enumDescriptions:
utf8: UTF-8 Unicode encoding, the universal default for most systems.
win1252: Windows-1252 Western European encoding for legacy Windows systems.
"utf-16le": UTF-16 Little Endian encoding with byte order mark support.
gb18030: Chinese National Standard encoding for Chinese character sets.
macroman: Mac OS Roman encoding for legacy Macintosh systems.
iso88591: ISO 8859-1 Latin-1 encoding for Western European languages.
shiftjis: Shift JIS encoding for Japanese character sets.
description: |-
Character encoding for the output file.
**Default:** `"utf8"`
Change from the default only when the target system requires a specific encoding.
default: utf8
delete:
type: boolean
description: |-
Whether to delete the source file after successful import.
**Values**
- `true`: Delete source file after processing
- `false`: Keep source file
**Default:** `false`
default: false
compressionFormat:
type: string
enum:
- gzip
- zip
x-enumDescriptions:
gzip: GZIP compression producing a single compressed .gz file.
zip: ZIP compression producing a .zip archive that can contain multiple files.
description: |-
Compression format for the output file.
**When to use**
- Large files that benefit from compression
- When target system expects compressed files
backupPath:
type: string
description: |-
Path where backup copies of files should be stored.
**Examples**
- `"backup/"` - Relative backup folder
- `"/archive/2024/"` - Absolute backup path
examples: ["backup/", "/archive/2024/"]
purgeInternalBackup:
type: boolean
description: |-
Whether to purge internal backup copies after successful processing.
**Default:** `false`
default: false
batchSize:
type: integer
description: |-
Number of records to include per batch/file when processing large datasets.
**When to use**
- Large imports that need to be split into multiple files
- When target system has file size limitations
**Note**
- Offered when records are written one file per page (`aggregation.type: page` with
`skipAggregation: true`)
x-celigo-ai-guidance:
- |-
Probed for `all` and `page`: the save rejects `batchSize` with 422 `not_supported`
("File imports do not support batching with aggregation") when the sent
`skipAggregation` is not `true`, and accepts it with `page` + `skipAggregation: true`.
The `key` + `batchSize` combination has not been probed (it needs a B2B Manager
account) — treat it as unverified, neither allowed nor forbidden.
encrypt:
type: boolean
description: |-
Whether to encrypt the output file.
**Values**
- `true`: Encrypt the file (requires PGP configuration)
- `false`: No encryption
**Default:** `false`
default: false
csv:
type: object
description: |-
CSV-specific configuration. Only used when `type` is `"csv"`.
properties:
rowDelimiter:
type: string
description: Character(s) used to separate rows. Default is newline.
default: "\n"
columnDelimiter:
type: string
description: Character(s) used to separate columns. Default is comma.
default: ","
includeHeader:
type: boolean
description: When true, includes a header row with column names in the output.
default: true
wrapWithQuotes:
type: boolean
description: When true, wraps field values in quotes.
x-celigo-ai-guidance:
- Whether to wrap field values in quotes.
default: false
replaceTabWithSpace:
type: boolean
description: Replace tab characters with spaces.
default: false
replaceNewlineWithSpace:
type: boolean
description: Replace newline characters with spaces within fields.
default: false
truncateLastRowDelimiter:
type: boolean
description: Remove trailing row delimiter from the file.
default: false
json:
type: object
description: |-
JSON-specific configuration. Only used when `type` is `"json"`.
properties:
resourcePath:
type: string
description: JSONPath expression to locate records within the JSON structure.
xml:
type: object
description: |-
XML-specific configuration. Only used when `type` is `"xml"`.
properties:
resourcePath:
type: string
description: XPath expression to locate records within the XML structure.
fileDefinition:
type: object
description: |
Configuration settings for parsing files using a predefined file definition. This object enables processing of complex, non-standard, or proprietary file formats that require specialized parsing logic beyond what the standard parsers (CSV, JSON, XML, etc.) can handle.
**When to use**
Configure this object when the `type` field is set to "filedefinition". This approach is required for properly handling:
- Legacy or proprietary file formats with complex structures
- Fixed-width text files where field positions are defined by character positions
- Electronic Data Interchange (EDI) documents (X12, EDIFACT, etc.)
- Multi-record type files where different lines have different formats
- Files requiring complex preprocessing or custom parsing logic
**File definition characteristics**
- **Custom Parsing Rules**: Applies predefined parsing logic to complex file formats
- **Reusable Configurations**: References externally defined parsing rules that can be reused
- **Complex Format Support**: Handles formats that standard parsers cannot process
- **Specialized Processing**: Often used for industry-specific or legacy formats
**Implementation strategy for ai agents**
1. **Format Analysis**:
- Determine if the file format is standard (CSV, JSON, XML) or requires custom parsing
- Check if the format follows industry standards like EDI, SWIFT, or fixed-width
- Assess if there are multiple record types within the same file
- Identify if specialized logic is needed to interpret the file structure
2. **File Definition Selection**:
- Verify that a suitable file definition has already been created in the system
- Check if existing file definitions match the format requirements
- Confirm the file definition ID from system administrators if needed
- Ensure the file definition is compatible with the export's needs
properties:
_fileDefinitionId:
type: string
format: objectId
x-celigo-refModel: filedefinitions
description: Reference to the file definition resource.
examples: ["66a1f2c3b4d5e6f7a8b9c0d1"]
aggregation:
type: object
description: |-
Controls how a flow run's records are batched into output files — the transfer form's
"How many files would you like to generate?" setting. The legacy `skipAggregation` flag
mirrors `type` and is maintained by the platform.
properties:
type:
type: string
enum: ["all", "page", "key"]
x-enumDescriptions:
all: One file per flow run — every record from the run (per export, or per request on real-time flows) in a single file
page: One file per page of records
key: One file per aggregation key — EDI file definitions only; records sharing the key derived from the file definition's document type are combined into one document
description: |-
Which batches of a run's records become separate output files. EDI file definitions
(`file.type: filedefinition`) are written per page or per key, never per run; `key`
is accepted only for them and only when the account has B2B Manager. The grouping
key and the line-item loop it merges come from the EDI file definition's document
type, so there is no separate key field to configure.
examples: ["all", "page", "key"]
x-celigo-ai-guidance:
- |-
Validated on save: an unknown value fails with 422 `enum`; `key` fails with 422
`not_supported` unless `file.type` is `filedefinition` ("Aggregation type key is
only supported for file type filedefinition") and the account has B2B Manager
("Aggregation type key is only supported for B2B Manager").
- |-
For `file.type: filedefinition` send `page` (with `skipAggregation: true`) or `key`
(with `false`); `all` is not an EDI option — the validator enforces this through the
`skipAggregation` rule (see that field). For every other file type the choices are
`all` and `page`. The UI offers exactly these splits.
- |-
A user asking where the "aggregation key" is configured is asking about this
field: the key itself is derived from the file definition's document type, never
entered.
lookups:
type: array
description: |-
Named value-substitution maps available to this import's field mappings; each lookup
translates source values to target values, with an optional default for unmatched values.
Referenced by name from mapping rules.
items:
type: object
properties:
name:
type: string
description: Identifier used to reference this lookup from mapping rules.
examples: ["status_Lookup"]
map:
type: object
description: Source-value to target-value pairs applied during mapping.
additionalProperties:
type: string
default:
type: [string, 'null']
description: Value substituted when a source value is not found in map; empty string
when omitted. May be stored as null (no default).
allowFailures:
type: boolean
description: When true, a value missing from map does not fail the record and the default (or original value) is used.
examples: [true, false]
pgp:
type: object
description: |-
PGP encryption settings applied when encrypt is true; selects the symmetric cipher and the
signing hash. The connection must hold the recipient's public key.
properties:
symmetricKeyAlgorithm:
type: string
enum: ["twofish", "cast5", "3des", "aes128", "aes192", "aes256"]
x-enumDescriptions:
twofish: Twofish block cipher.
cast5: CAST5 (CAST-128) block cipher.
3des: Triple DES block cipher (legacy).
aes128: AES with a 128-bit key.
aes192: AES with a 192-bit key.
aes256: AES with a 256-bit key (default, recommended).
default: aes256
description: Symmetric cipher used to encrypt the file body; aes256 unless the recipient requires another.
examples: ["aes256", "3des"]
hashAlgorithm:
type: string
enum: ["sha256", "sha384", "sha512", "sha224"]
x-enumDescriptions:
sha256: SHA-256 signing hash.
sha384: SHA-384 signing hash.
sha512: SHA-512 signing hash.
sha224: SHA-224 signing hash.
description: Hash algorithm used when signing the encrypted file; set only when the recipient requires a specific one.
examples: ["sha256", "sha512"]
skipRename:
type: boolean
description: |-
When true, the import writes directly to the final file name instead of writing to a
temporary name and renaming on completion. Use only when the destination does not support
atomic rename; the default (false) uses the safe write-then-rename behavior.
examples: [true, false]
appendToExistingFile:
type: boolean
default: false
x-celigo-ui-override: >-
Defaulted by the Azure Data Lake Storage Gen2 import form
(fieldDefinitions/resources/imports/azuredatalakestoragegen2.js: 'file.appendToExistingFile'
defaultValue r?.file?.appendToExistingFile === true, shown only for file.type csv in
records mode). When it is on, preSave clears file.compressionFormat, file.encrypt,
file.pgp and the Content-Type header — append mode and compress/encrypt are exclusive
on the form. The API stores the flag on any file import.
description: |-
When true, an Azure Data Lake Storage Gen2 import (`assistant: azuredatalakestoragegen2`)
appends each run's CSV output to the file named by `file.fileName` instead of replacing
it. Appended content cannot be compressed or encrypted, so leave `compressionFormat`,
`encrypt` and `pgp` unset when this is on.
examples: [true, false]
contentType:
type: string
x-celigo-agent:
- handlebars
description: |-
MIME type recorded on the file written to a cloud file provider such as Azure Data Lake
Storage Gen2. Accepts a Handlebars expression so the type can follow the record (e.g.
`{{record.mimeType}}`). Omit to let the destination assign its default.
examples: ["text/csv", "application/json", "{{record.mimeType}}"]
directory:
type: object
required:
- pathMode
description: |-
Structured destination location for cloud file-provider imports (Google Drive shared
drives, Box, Dropbox, Google Cloud Storage, Azure Data Lake Storage Gen2), replacing the
flat path used by classic FTP/S3 destinations. Select the location by folder ID or by a
path relative to a configured storage root via pathMode. For the drive-style providers,
directoryId mode addresses the folder by id alone — the flat destination-path field is
not consulted at runtime. Google Cloud Storage puts the bucket in `storageRootId`;
Azure Data Lake Storage Gen2 puts the container in `name` and the directory prefix of the
file path in `http.relativeURI`.
x-celigo-ai-guidance:
- >-
Save-time validation accepts any string for id without checking it against the
provider — set it only from a folder ID the user supplied or the UI folder browser
stored; never fabricate one. To address a folder by path instead, set pathMode to
relativePath and put the path in the adaptor's flat path field.
properties:
pathMode:
type: string
enum: ["relativePath", "directoryId"]
x-enumDescriptions:
relativePath: Address the location by a path relative to the storage root identified by storageRootId.
directoryId: Address the location directly by the provider's folder ID set in id.
description: How the directory is addressed — by provider folder ID, or by a path relative to a configured storage root.
examples: ["relativePath", "directoryId"]
id:
type: string
description: |-
Provider-native folder ID of the destination directory (e.g. a Google Drive folder ID);
set when pathMode is directoryId. This is the file provider's own ID, not a Celigo resource ID.
examples: ["1KppvTTr4jKhJqwPAhGWN6SRwk5m45DL2"]
name:
type: string
description: |-
Display name of the folder identified by id, kept for readability in the UI; the
folder browser fills it when the location is picked visually. For Azure Data Lake
Storage Gen2 it is the authoritative container name instead — the form saves the
container here and no id.
examples: ["Invoices - Outbound", "invoices-outbound"]
storageRootId:
type: string
description: |-
Provider-native ID of the storage root the relative path resolves against; set when
pathMode is relativePath. The provider's own ID, not a Celigo resource ID. For Google
Cloud Storage it is the bucket name.
examples: ["1qU1hvFarBFMb4-twqKm1rR5o_6j9vs9g", "erp-exports"]
storageRootName:
type: string
description: Display name of the storage root identified by storageRootId, retained for reference in the UI.
examples: ["Shared drive - Finance"]
backupDirectory:
type: object
required:
- pathMode
description: |-
Structured location for backup copies on cloud file providers — the file-provider
counterpart to backupPath. Addressed the same way as directory.
properties:
pathMode:
type: string
enum: ["relativePath", "directoryId"]
x-enumDescriptions:
relativePath: Address the backup location by a path relative to the storage root identified by storageRootId.
directoryId: Address the backup location directly by the provider's folder ID set in id.
description: How the backup location is addressed — by provider folder ID, or by a path relative to a configured storage root.
examples: ["relativePath", "directoryId"]
id:
type: string
description: Provider-native folder ID of the backup directory; set when pathMode is directoryId.
examples: ["1KppvTTr4jKhJqwPAhGWN6SRwk5m45DL2"]
name:
type: string
description: |-
Display name of the backup folder identified by id, kept for readability in the UI;
the folder browser fills it when the location is picked visually.
examples: ["Processed archive"]
storageRootId:
type: string
description: Provider-native ID of the storage root the backup relative path resolves against; set when pathMode is relativePath.
examples: ["1qU1hvFarBFMb4-twqKm1rR5o_6j9vs9g"]
references/schemas/filesystem.yml
FileSystem:
type: object
description: |-
Defines where files are written to a local or mounted folder on the host running the
on-premise agent. Required when the _connectionId field references a file-system (on-premise)
connection; must not be included for other connection types.
required:
- directoryPath
properties:
directoryPath:
type: string
x-celigo-agent:
- handlebars
description: |-
Folder on the on-premise agent's host where generated files are written; the agent's OS
account must have write permission on it. Accepts a local OS path or a UNC network share,
and supports handlebars templates for dynamic folders.
examples: ["C:\\Celigo\\Outbound", "D:\\imports\\incoming", "\\\\NAS\\Shared\\Drop"]
references/schemas/ftp.yml
Ftp:
type: object
description: |-
Defines where files are written on an FTP, FTPS, or SFTP server. Required when the
_connectionId field references an FTP/SFTP connection; must not be included for other
connection types. directoryPath selects the target folder and the file naming comes from
the import's file configuration.
required:
- directoryPath
properties:
_tpConnectorId:
type: string
format: objectId
x-celigo-refModel: tradingpartnerconnectors
description: |-
Trading Partner Connector that supplies partner-specific B2B settings for this import.
When set, the import inherits the connector's pre-configured settings; omit to use only
the FTP connection details.
x-celigo-ai-guidance:
- |-
Identifier for the third-party connector utilized in the FTP integration,
serving as a unique and immutable reference that links the FTP configuration to
an external connector service.
This identifier is essential for accurately associating the FTP connection with
the intended third-party system,
enabling seamless and secure data transfer, synchronization,
and management through the specified connector.
It ensures that all FTP operations are correctly routed and managed via the external service,
supporting reliable integration workflows and maintaining system integrity
throughout the connection lifecycle.
- |-
## FIELD BEHAVIOR:
- Uniquely identifies the third-party connector associated with the FTP configuration.
- Acts as a key reference to bind FTP settings to a specific external connector service.
- Required when creating, updating, or managing FTP connections via third-party integrations.
- Typically immutable after initial assignment to preserve connection consistency.
- Used by system components to route FTP operations through the correct external connector.
- |-
## IMPLEMENTATION GUIDANCE:
- Should be a string or numeric identifier that aligns with the third-party connector's identification scheme.
- Must be validated against the system's registry of available connectors to ensure validity.
- Should be set securely and protected from unauthorized modification.
- Updates to this identifier should be handled cautiously, with appropriate validation and impact assessment.
- Ensure the identifier complies with any formatting or naming conventions imposed by the third-party service.
- Transmit and store this identifier securely, using encryption where applicable.
- |-
## EXAMPLES:
- "connector_12345"
- "tpConnector-67890"
- "ftpThirdPartyConnector01"
- "extConnector_abcde"
- "thirdPartyFTP_001"
- |-
## IMPORTANT NOTES:
- This field is critical for correctly mapping FTP configurations to their corresponding third-party services.
- An incorrect or missing _tpConnectorId can lead to connection failures, data misrouting, or integration errors.
- Synchronization between this identifier and the third-party connector records must be maintained to avoid inconsistencies.
- Changes to this field may require re-authentication or reconfiguration of the FTP integration.
- Proper error handling should be implemented to manage cases where the identifier does not match any registered connector.
- |-
## DEPENDENCY CHAIN:
- Depends on the existence and registration of third-party connectors within the system.
- Referenced by FTP connection management modules, authentication services, and integration workflows.
- Interacts with authorization mechanisms to ensure secure access to third-party services.
- May influence logging, monitoring, and auditing processes related to FTP integrations.
examples: ["60a2c4e6f321d800129a1a3c"]
directoryPath:
type: string
x-celigo-agent:
- handlebars
x-celigo-ui-override: >-
Required by the FTP import form (fieldDefinitions/resources/imports/ftp.js: ftp.directoryPath
is required:true). Encoded to mirror the form so builders produce connectable configurations.
description: |-
Directory on the server where imported files are written, either absolute or relative to
the login directory; the FTP user must have write permission on it. Use forward slashes
regardless of server OS — paths are case-sensitive on UNIX/Linux servers. Supports
handlebars templates.
x-celigo-ai-guidance:
- |-
A handlebars template here is rendered against the same context as `fileName`
(`fileMeta` is not a root key). An expression that renders empty fails the record
with `missing_s3_key` "Can not process upload file without
computedObject.fileLocation." — labeled source `internal` although the template is
the cause.
- |-
The path to the directory on the FTP server where files will be accessed,
stored, or managed.
This path can be specified either as an absolute path starting from the root
directory of the FTP server or as a relative path based on the user's initial
directory upon login,
depending on the server's configuration and permissions.
It is essential that the path adheres to the FTP server's directory structure,
naming conventions,
and access controls to ensure successful file operations such as uploading,
downloading, listing, or deleting files.
The directoryPath serves as the primary reference point for all file-related
commands during an FTP session,
determining the scope and context of file management activities.
- |-
## FIELD BEHAVIOR:
- Defines the target directory location on the FTP server for all file-related operations.
- Supports both absolute and relative path formats, interpreted according to the FTP server's setup.
- Must comply with the server's directory hierarchy, naming rules, and access permissions.
- Used by the FTP client to navigate and perform operations within the specified directory.
- Influences the scope of accessible files and subdirectories during FTP sessions.
- Changes to this path affect subsequent file operations until updated or reset.
- |-
## IMPLEMENTATION GUIDANCE:
- Validate the directory path format to ensure compatibility with the FTP server, typically using forward slashes (`/`) as separators.
- Normalize the path to remove redundant elements such as `.` (current directory) and `..` (parent directory) to prevent errors and security vulnerabilities.
- Handle scenarios where the specified directory does not exist by either creating it if permissions allow or returning a clear, descriptive error message.
- Properly encode special characters and escape sequences in the path to conform with FTP protocol requirements and server expectations.
- Provide informative and user-friendly error messages when the path is invalid, inaccessible, or lacks sufficient permissions.
- Sanitize input to prevent directory traversal attacks or unauthorized access to restricted areas.
- Consider server-specific behaviors such as case sensitivity, symbolic links, and virtual directories when interpreting the path.
- Ensure that path changes are atomic and consistent to avoid race conditions during concurrent FTP operations.
- |-
## EXAMPLES:
- `/uploads/images`
- `documents/reports/2024`
- `/home/user/data`
- `./backup`
- `../shared/resources`
- `/var/www/html`
- `projects/current`
- |-
## IMPORTANT NOTES:
- Access to the specified directory depends on the credentials used during FTP authentication.
examples: ["users/dave", "incoming/orders", "/var/ftp/upload", "EDI/850"]
fileName:
type: string
x-celigo-agent:
- handlebars
description: |-
Name of the file written to the server, including its extension; do not include directory
separators — the location comes from directoryPath. Supports handlebars placeholders such
as items-{{timestamp}}.csv to generate a unique name per run; `{{timestamp}}` alone renders
the full ISO-8601 time, colons included (`2026-10-23T08:25:21.743Z`), and a format gives
a date stamp instead: `orders-{{timestamp 'YYYYMMDD'}}.csv`. There is no `date` helper. The
template is rendered
against the incoming record: a file-transfer import from an FTP source kept the source
file's name with `{{record.fileMeta.fileName}}` set on `file.fileName` and this field left
unset — other source providers expose the name under their own `fileMeta` field.
x-celigo-ai-guidance:
- |-
When the user does not specify a file naming convention,
default to a timestamped filename such as "items-{{timestamp}}.csv" so each run
produces a unique file and avoids overwriting previous files.
Use a fixed filename only when the user explicitly requests overwriting or a fixed name.
- |-
`fileMeta` is not a root key of this template's context (it is one only on
file-provider export and lookup steps): `{{fileMeta.fileName}}` renders empty and the
upload fails with `missing_s3_key` "Can not process upload file without
computedObject.fileName." — an error the platform labels source `internal` although the
template is the cause. The run-time-proven way to keep the source file's name on a
`blob: true` import is `{{record.fileMeta.fileName}}` on `file.fileName` with this field
omitted (recorded runs of an FTP blob import set exactly that); the same template
on this field has not been run. The field's editor preview (data version 1, its default)
exposes the record as `data` —
the `{blobKey, fileMeta}` record on a blob import, the page's record array otherwise —
so `record.` previews empty there; judge by a run.
- |-
The name of the file to be accessed, transferred,
or manipulated via FTP (File Transfer Protocol).
This property specifies the exact filename, including its extension,
uniquely identifying the file within the FTP directory.
It is critical for accurately targeting the file during operations such as upload,
download, rename, or delete.
The filename must be precise and comply with the naming conventions and
restrictions of the FTP server's underlying file system to avoid errors.
Case sensitivity depends on the server's operating system,
and special attention should be given to character encoding,
especially when handling non-ASCII or international characters.
The filename should not include any directory path separators,
as these are managed separately in directory or path properties.
- |-
## FIELD BEHAVIOR:
- Identifies the specific file within the FTP server directory for various file operations.
- Must include the file extension to ensure precise identification.
- Case sensitivity is determined by the FTP server's operating system.
- Excludes directory or path information; only the filename itself is specified.
- Used in conjunction with directory or path properties to locate the full file path.
- |-
## IMPLEMENTATION GUIDANCE:
- Validate that the filename contains only characters allowed by the FTP server's file system.
- Ensure the filename is a non-empty, well-formed string without path separators.
- Normalize case if required to match server-specific case sensitivity rules.
- Properly handle encoding for special or international characters to prevent transfer errors.
- Combine with directory or path properties to construct the complete file location.
- Avoid embedding directory paths or separators within the filename property.
- |-
## EXAMPLES:
- "report.pdf"
- "image_2024.png"
- "backup_2023_12_31.zip"
- "data.csv"
- "items-{{timestamp}}.csv"
- |-
## IMPORTANT NOTES:
- The filename alone may not be sufficient to locate the file without the corresponding directory or path.
- Different FTP servers and operating systems may have varying rules regarding case sensitivity.
- Path separators such as "/" or "\" must not be included in the filename.
- Adherence to FTP server naming conventions and restrictions is essential to prevent errors.
- Be mindful of potential filename length limits imposed by the FTP server or protocol.
- |-
## DEPENDENCY CHAIN:
- Depends on FTP directory or path properties to specify the full file location.
- Utilized alongside FTP server credentials and connection settings.
- May be influenced by file transfer mode (binary or ASCII) based on the file type.
examples: ["items-{{timestamp}}.csv", "orders-{{timestamp 'YYYYMMDD'}}.xml", "export.json"]
inProgressFileName:
type: string
x-celigo-agent:
- handlebars
x-celigo-ui-override: >-
The FTP import form requires this once its "Use temp file" toggle is on
(fieldDefinitions/resources/imports/ftp.js: 'ftp.inProgressFileName' requiredWhen
ftp.useTempFile is true; blob mode 'ftp.blobInProgressFileName' requiredWhen
ftp.blobUseTempFile is true). The toggles are form-only and are not saved — the form
derives them from whether this field is set — so the API has no field to condition on
and saves the import with or without the name.
description: |-
Temporary name the file carries while its upload is in progress, preventing other systems
from processing a partially transferred file; it is renamed to its final name once the
upload completes. Its presence is what the UI shows as "Use temp file while upload in
progress": set it to turn the temporary name on, omit it to turn it off. Include a
handlebars uniqueness token (e.g. {{timestamp}}) when one file is written per flow run.
x-celigo-ai-guidance:
- |-
inProgressFileName is the designated temporary filename assigned to a file
during the FTP upload process to clearly indicate that the file transfer is
currently underway and not yet complete.
This naming convention serves as a safeguard to prevent other systems or
processes from prematurely accessing,
processing, or locking the file before the upload finishes successfully.
Typically, once the upload is finalized,
the file is renamed to its intended permanent filename,
signaling that it is ready for further use or processing.
Utilizing an in-progress filename helps maintain data integrity,
avoid conflicts,
and streamline workflow automation in environments where files are continuously
transferred and monitored.
It plays a critical role in managing file state visibility,
ensuring that incomplete or partially transferred files are not mistakenly processed,
which could lead to data corruption or inconsistent system behavior.
- |-
## FIELD BEHAVIOR:
- Acts as a temporary marker for files actively being uploaded via FTP or similar protocols.
- Prevents other processes, systems, or automated workflows from accessing or processing the file until the upload is fully complete.
- The file is renamed atomically from the inProgressFileName to the final intended filename upon successful upload completion.
- Helps manage concurrency, avoid file corruption, partial reads, or race conditions during file transfer.
- Indicates the current state of the file within the transfer lifecycle, providing clear visibility into upload progress.
- Supports cleanup mechanisms by identifying orphaned or stale temporary files resulting from interrupted uploads.
- |-
## IMPLEMENTATION GUIDANCE:
- Choose a unique, consistent, and easily identifiable naming pattern distinct from final filenames to clearly denote the in-progress status.
- Commonly use suffixes or prefixes such as ".inprogress", ".tmp", "_uploading", or similar conventions that are recognized across systems.
- Ensure the FTP server and client support atomic renaming operations to avoid partial file visibility or race conditions.
- Verify that the chosen temporary filename does not collide with existing files in the target directory to prevent overwriting or confusion.
- Maintain consistent naming conventions across all related systems, scripts, and automation tools for clarity and maintainability.
- Implement robust cleanup routines to detect and remove orphaned or stale in-progress files from failed or abandoned uploads.
- Coordinate synchronization between upload completion and renaming to prevent premature processing or file locking issues.
- |-
## EXAMPLES:
- "datafile.csv.inprogress"
- "upload.tmp"
- "report_20240615.tmp"
- "file_upload_inprogress"
- "transaction_12345.uploading"
examples: ["items-{{timestamp}}.csv.inprogress", "upload.tmp"]
backupDirectoryPath:
type: string
x-celigo-agent:
- handlebars
description: |-
Directory on the same server where a copy of each written file is retained after a
successful import; if omitted, no server-side backup is kept. Supports static paths or
handlebars templates.
x-celigo-ai-guidance:
- |-
backupDirectoryPath specifies the file system path to the directory where backup
files are stored during FTP operations.
This directory serves as the designated location for saving copies of files
before they are overwritten or deleted,
ensuring that original data can be recovered if necessary.
The path can be either absolute or relative, depending on the system context,
and must conform to the operating system's file path conventions.
It is essential that this path is valid, accessible,
and writable by the FTP service or application performing the backups to
guarantee reliable backup creation and data integrity.
Proper configuration of this path is critical to prevent data loss and to
maintain a secure and organized backup environment.
- |-
## FIELD BEHAVIOR:
- Specifies the target directory for storing backup copies of files involved in FTP transactions.
- Activated only when backup functionality is enabled within the FTP process.
- Requires the path to be valid, accessible, and writable to ensure successful backup creation.
- Supports both absolute and relative paths, with relative paths resolved against a defined base directory.
- If unset or empty, backup operations may be disabled or fallback to a default directory if configured.
- |-
## IMPLEMENTATION GUIDANCE:
- Validate the existence of the directory and verify write permissions before initiating backups.
- Normalize paths to handle trailing slashes, redundant separators, and platform-specific path formats.
- Provide clear and actionable error messages if the directory is invalid, inaccessible, or lacks necessary permissions.
- Support environment variables or placeholders for dynamic path resolution where applicable.
- Enforce security best practices by restricting access to the backup directory to authorized users or processes only.
- Ensure sufficient disk space is available in the backup directory to accommodate backup files without interruption.
- Implement concurrency controls such as file locking or synchronization if multiple backup operations may occur simultaneously.
- |-
## EXAMPLES:
- "/var/ftp/backups"
- "C:\\ftp\\backup"
- "./backup_files"
- "/home/user/ftp_backup"
- |-
## IMPORTANT NOTES:
- The backup directory must have adequate storage capacity to prevent backup failures due to insufficient space.
- Permissions must allow the FTP process or service to create and modify files within the directory.
- Backup files may contain sensitive or confidential data; appropriate security measures should be enforced to protect them.
- Path formatting should align with the conventions of the underlying operating system to avoid errors.
examples: ["processed", "archive", "backup/{{timestamp 'YYYY-MM'}}"]
references/schemas/guardrail.yml
GuardrailConfig:
type: object
description: |-
Configuration for GuardrailImport adaptor type.
Guardrails evaluate data flowing through integrations for safety and
compliance. The `type` field selects which check to apply, and the
corresponding sub-object (`aiAgent`, `pii`, or `moderation`) provides
the configuration.
A `_connectionId` on the parent import is only needed for BYOK
`ai_agent` guardrails. In responses the server echoes the active type's
sub-object and applies the `confidenceThreshold` default; it also
returns inactive sibling sub-objects (e.g. `moderation: {categories: []}`
on a `pii` guardrail, or a populated `pii` left over from a type switch),
but only the active type's sub-object is meaningful. Legacy documents
may carry a server-written default `aiAgent` stub on `pii`/`moderation`
guardrails; current servers strip the inactive `aiAgent` on write.
x-celigo-ai-guidance:
- |-
Guardrails are safety and compliance checks that can be applied
to data flowing through integrations. Three mutually-exclusive
types are supported — see `type` below for the full decision
rule on which to pick:
- **pii**: Detect (and optionally mask) personally identifiable
information. Local deterministic classifier; fixed enum of
entity types (`pii.entities`).
- **moderation**: Detect harmful / inappropriate / policy-
violating CONTENT. Local deterministic classifier; fixed enum
of categories (`moderation.categories` — `sexual`, `hate`,
`harassment`, `self_harm`, `violence`, `illicit`, etc.). Use
category names exactly as listed in the enum — e.g. `hate`
(not "hate speech"), `violence` (not "violent content"). Map
natural-language synonyms ("explicit" → `sexual`, "toxic" →
`harassment`, etc.) to the closest enum value rather than
falling back to `ai_agent`.
- **ai_agent**: LLM-evaluated judgement against custom
instructions. Catch-all for intents that don't fit `pii` or
`moderation` — domain-specific compliance, business-policy
checks, data-quality assertions, etc. Pick this LAST, not as
the default.
- |-
Prefer the cheaper / more deterministic types (`pii` > `moderation` >
`ai_agent`) whenever the brief's intent fits.
- Guardrail imports do not require a `_connectionId` (unless using BYOK for `ai_agent` type).
- |-
Never send `tools` anywhere under `aiAgent` on an `ai_agent` guardrail (`aiAgent.tools`,
`aiAgent.openai.tools`, `aiAgent.litellm._overrides.gemini.tools`, `….anthropic.tools`) —
any non-empty array fails the save with 422 `guardrail_tools_not_allowed` ("Tools aren't
supported on a guardrail AI agent configuration…"), on create and on update. `prompts` and
`resources` are accepted on save but never used at run time, so put the policy in the
instructions. The server still echoes
`openai.tools: []` on OpenAI guardrails.
properties:
type:
type: string
enum: ["ai_agent", "pii", "moderation"]
x-enumDescriptions:
ai_agent: Evaluate data using an AI model with custom instructions.
pii: Detect personally identifiable information in data fields.
moderation: Check content for harmful or inappropriate categories.
description: |-
The type of guardrail to apply. Each type requires its corresponding
sub-configuration object (`aiAgent`, `pii`, or `moderation`).
x-celigo-ai-guidance:
- |-
The type of guardrail to apply.
Pick exactly one — `pii` / `moderation` / `ai_agent` are mutually exclusive at
the type level (a brief that wants both runs as two guardrails in sequence,
not one).
- |-
## How to choose
Apply these rules **in order** and pick the first match. Do
NOT skip ahead to `ai_agent` just because the brief is in
natural language — `pii` and `moderation` are also expressed
as natural-language briefs. The decision is about WHAT the
brief asks the guardrail to detect, not how the brief is
worded.
1. **`pii`** — pick this when the brief names ANY personally
identifiable information, OR when the brief names entity
categories (SSN, email, phone, credit card, address, name,
passport, license, account number, etc.) that match
`pii.entities`. Examples that route here:
"detect SSN and credit card numbers", "mask customer
emails and phone numbers", "scan for PII".
2. **`moderation`** — pick this when the brief asks the
guardrail to detect harmful, inappropriate, unsafe, or
policy-violating CONTENT (as opposed to PII data fields)
AND the intent maps to any value in `moderation.categories`
(`sexual`, `hate`, `harassment`, `self_harm`, `violence`,
`illicit`, etc.) — including via close synonyms. Apply
the synonym mapping liberally:
- "explicit", "obscene", "vulgar", "profane",
"inappropriate language", "swearing", "NSFW" → typically
maps to `sexual` and/or `harassment`
- "abusive", "insulting", "bullying", "toxic" → `harassment`
- "hateful", "discriminatory", "racist", "sexist" → `hate`
- "threatening", "intimidating" → `harassment_threatening`
or `violence`
- "graphic violence", "gore" → `violence_graphic`
- "suicide", "self-injury" → `self_harm`
- "drugs", "weapons", "illegal activity" → `illicit`
Briefs like "screen email body for explicit language",
"block toxic chat messages", "flag harassment in support
tickets", "moderate user-generated content for unsafe
material" all route to `moderation`. When the brief uses
a synonym, populate `moderation.categories` with the
closest enum value(s) — do NOT invent new category
strings; the enum is fixed.
3. **`ai_agent`** — pick this ONLY when the brief asks for
judgement that doesn't fit `pii` or `moderation`. Typical
triggers: domain-specific compliance checks (HIPAA / SOX
rule evaluation), business-policy validation (price
bounds, approval thresholds), data-quality assertions
(required fields populated, values plausible), or any
"evaluate against these custom rules" framing where the
rules are domain-specific and have no `pii.entities` /
`moderation.categories` analogue. If you can map the
brief's intent to a moderation category (even via a
synonym above), use `moderation` instead — `ai_agent` is
the catch-all, not the default.
Tie-breaker for borderline cases: prefer the more specific
type (`pii` > `moderation` > `ai_agent`). `pii` and
`moderation` are local deterministic classifiers with fixed
enums — they're cheaper, faster, and more predictable than
`ai_agent` (which calls a real LLM per record). Default to
the cheaper path whenever the intent fits.
- |-
## Sub-configuration requirements (one per type)
- **ai_agent** requires the `aiAgent` sub-configuration.
- **pii** requires the `pii` sub-configuration with at least
one entity from `pii.entities`.
- **moderation** requires the `moderation` sub-configuration
with at least one category from `moderation.categories`.
confidenceThreshold:
type: number
minimum: 0
maximum: 1
default: 0.7
x-celigo-ui-override: >-
Required by the guardrail form (confidenceThreshold is required:true once a type is
selected, default 0.7). Encoded to mirror the form so builders produce connectable
configurations.
description: |-
Confidence threshold (0 to 1). Detections below this threshold are
ignored. Lower values catch more issues but increase false positives.
x-celigo-ai-guidance:
- Confidence threshold for guardrail detection (0 to 1).
- |-
Only detections with confidence at or above this threshold will be flagged.
Lower values catch more potential issues but may increase false positives.
examples: [0.7, 0.5, 0.9]
aiAgent:
type: object
description: |-
AI agent check configuration; set when `type` is `ai_agent`. On
`pii`/`moderation` guardrails a legacy server-written stub may
appear here — it is inert, and current servers strip it on write.
pii:
type: object
required: [entities]
description: |-
PII detection configuration. Required when `type` is `pii`.
x-celigo-ai-guidance:
- Configuration for PII (Personally Identifiable Information) detection.
- |-
Required when `guardrail.type` is "pii".
At least one entity type must be specified.
properties:
entities:
type: array
description: |-
PII entity types to detect. When `type` is `pii`, at least one
entry is required; the inactive sibling on other guardrail types
may be served with an empty list.
x-celigo-ai-guidance:
- PII entity types to detect in the data.
- At least one entity must be specified when using PII guardrails.
items:
type: string
enum:
- credit_card_number
- card_security_code_cvv_cvc
- cryptocurrency_wallet_address
- date_and_time
- email_address
- iban_code
- bic_swift_bank_identifier_code
- ip_address
- location
- medical_license_number
- national_registration_number
- persons_name
- phone_number
- url
- us_bank_account_number
- us_drivers_license
- us_itin
- us_passport_number
- us_social_security_number
- uk_nhs_number
- uk_national_insurance_number
- spanish_nif
- spanish_nie
- italian_fiscal_code
- italian_drivers_license
- italian_vat_code
- italian_passport
- italian_identity_card
- polish_pesel
- finnish_personal_identity_code
- singapore_nric_fin
- singapore_uen
- australian_abn
- australian_acn
- australian_tfn
- australian_medicare
- indian_pan
- indian_aadhaar
- indian_vehicle_registration
- indian_voter_id
- indian_passport
- korean_resident_registration_number
x-enumDescriptions:
credit_card_number: Credit or debit card number (e.g., Visa, Mastercard, Amex).
card_security_code_cvv_cvc: Card security code printed on credit or debit cards (CVV/CVC).
cryptocurrency_wallet_address: Cryptocurrency wallet address (e.g., Bitcoin, Ethereum).
date_and_time: Date and/or time values that could identify an individual.
email_address: Email address in standard format.
iban_code: International Bank Account Number used for cross-border payments.
bic_swift_bank_identifier_code: BIC/SWIFT code identifying a specific bank for international transfers.
ip_address: IPv4 or IPv6 network address.
location: Physical location or geographic coordinates.
medical_license_number: Medical professional license or registration number.
national_registration_number: National identification or registration number.
persons_name: Full or partial name of a person.
phone_number: Telephone number in any format.
url: Web URL or URI.
us_bank_account_number: United States bank account number.
us_drivers_license: United States driver's license number.
us_itin: United States Individual Taxpayer Identification Number.
us_passport_number: United States passport number.
us_social_security_number: United States Social Security Number (SSN).
uk_nhs_number: United Kingdom National Health Service number.
uk_national_insurance_number: United Kingdom National Insurance number.
spanish_nif: Spanish tax identification number (NIF).
spanish_nie: Spanish foreigner identification number (NIE).
italian_fiscal_code: Italian fiscal code (Codice Fiscale).
italian_drivers_license: Italian driver's license number.
italian_vat_code: Italian VAT identification number (Partita IVA).
italian_passport: Italian passport number.
italian_identity_card: Italian national identity card number.
polish_pesel: Polish national identification number (PESEL).
finnish_personal_identity_code: Finnish personal identity code (henkilotunnus).
singapore_nric_fin: Singapore National Registration Identity Card or Foreign Identification Number.
singapore_uen: Singapore Unique Entity Number for business registration.
australian_abn: Australian Business Number.
australian_acn: Australian Company Number.
australian_tfn: Australian Tax File Number.
australian_medicare: Australian Medicare card number.
indian_pan: Indian Permanent Account Number for tax purposes.
indian_aadhaar: Indian Aadhaar unique identity number.
indian_vehicle_registration: Indian vehicle registration number.
indian_voter_id: Indian voter identification card number.
indian_passport: Indian passport number.
korean_resident_registration_number: South Korean resident registration number.
examples:
- ["email_address", "phone_number", "persons_name"]
- ["us_social_security_number", "credit_card_number"]
mask:
type: boolean
default: false
description: |-
When true, the result also carries `masked`: the evaluated input with each detected
value replaced by asterisks. The record itself is unchanged. With whole-record input
(no `mapping`, and no top-level `text` field on the record), `masked` is the record
serialized as one JSON string. When false, PII is flagged only.
x-celigo-ai-guidance:
- |-
Masking doesn't rewrite the record. To pass masked text on, select the field as the
`text` input (`mapping`) and response-map `masked` back onto that field on the
guardrail's flow step (`{"extract": "masked", "generate": "<field>"}`).
moderation:
type: object
required: [categories]
description: |-
Content moderation configuration. Required when `type` is `moderation`.
x-celigo-ai-guidance:
- |-
Required when `guardrail.type` is "moderation".
At least one category must be specified.
properties:
categories:
type: array
description: |-
Content moderation categories to check. When `type` is
`moderation`, at least one entry is required; the inactive
sibling on other guardrail types may be served with an empty
list.
x-celigo-ai-guidance:
- At least one category must be specified when using moderation guardrails.
items:
type: string
enum:
- sexual
- sexual_minors
- hate
- hate_threatening
- harassment
- harassment_threatening
- self_harm
- self_harm_intent
- self_harm_instructions
- violence
- violence_graphic
- illicit
- illicit_violent
x-enumDescriptions:
sexual: Content depicting sexual activity or explicit sexual material.
sexual_minors: Sexual content involving minors.
hate: Content expressing hatred toward a group based on protected characteristics.
hate_threatening: Hateful content that includes threats of violence or serious harm.
harassment: Content that targets, intimidates, or bullies an individual.
harassment_threatening: Harassment content that includes threats of violence or serious harm.
self_harm: Content that promotes or depicts self-harm behaviors.
self_harm_intent: Content expressing intent to engage in self-harm.
self_harm_instructions: Content providing instructions for self-harm methods.
violence: Content depicting or promoting physical violence.
violence_graphic: Graphic or gory depictions of violence or injury.
illicit: Content promoting illegal activities or unlawful behavior.
illicit_violent: Content promoting illegal activities that involve violence.
examples:
- ["hate", "violence", "harassment"]
- ["sexual", "self_harm", "illicit"]
required:
- type
- confidenceThreshold
if:
required: [type]
properties:
type:
const: pii
then:
required: [pii]
properties:
pii:
required: [entities]
properties:
entities:
minItems: 1
else:
if:
required: [type]
properties:
type:
const: moderation
then:
required: [moderation]
properties:
moderation:
required: [categories]
properties:
categories:
minItems: 1
else:
required: [aiAgent]
properties:
aiAgent:
# Guardrails reject every `tools` array (422 guardrail_tools_not_allowed), so the
# ai_agent branch uses the guardrail spec's tools-free copy of the provider blocks
# rather than the tool-bearing AiAgentConfig used by AiAgentImport.
$ref: "../guardrail-schemas/guardrail-config.yml#/GuardrailAiAgentConfig"
references/schemas/http.yml
Http:
type: object
description: |-
Configuration for HTTP imports.
Required whenever the connection referenced by _connectionId has type http, and on
`NetSuiteHTTPImport` (NetSuite's REST APIs on a `netsuite` connection, where `nsDomainType`
supplies the host).
x-celigo-ai-guidance:
- |-
IMPORTANT:
When the _connectionId field points to a connection where the type is http,
this object MUST be populated for the import to function properly.
This is a required configuration for all HTTP based imports,
as determined by the connection associated with the import.
- |-
A `NetSuiteHTTPImport` uses this same object on a netsuite connection: each `relativeURI`
entry carries the full path from `/services/rest/record/v1/<type>` (SuiteTalk) or
`/app/site/hosting/restlet.nl?script=<id>&deploy=<n>` (custom RESTlet); the host and the
token signature come from the connection.
properties:
sendPostMappedData:
type: boolean
default: true
description: |-
When true, the request body is the record as reshaped by the import's mapping step. When false, the original pre-mapped record is sent instead, bypassing the mappings for the payload.
isRest:
type: boolean
description: |-
Internal flag indicating the import was built with the legacy REST form rather than the unified HTTP framework. Set by the form; not configured directly.
strictHandlebarEvaluation:
type: boolean
description: |-
Intended to make handlebars expressions that reference a missing field raise an error instead of rendering an empty string. In live runs on an HTTP import it does not have that effect on `http.body` templates or mapping expressions — an undefined path still renders empty and the record succeeds with the flag true — and `relativeURI` / `headers[].value` templates already fail on an undefined path regardless of this flag. Not a safeguard against a wrong template root or a misspelled path.
x-celigo-ai-guidance:
- |-
Do not rely on this flag to catch a mis-rooted or misspelled path. Live-verified
2026-09-24 on an HTTP import in real flow runs with the flag ``true``: an undefined
``{{record.<field>}}`` in ``http.body`` and an undefined ``{{<field>}}`` in a mapping
expression both rendered empty and the record succeeded — the same outcome as with
the flag ``false`` or unset. Relative URI and header templates fail on an undefined
path regardless of this flag. To catch a wrong root, check the template against the
record's shape (``record`` for an object, ``rows`` for a grouped record,
``batch_of_records`` when ``batchSize`` exceeds 1) before saving.
formType:
type: string
enum:
- assistant
- http
- rest
- graph_ql
- assistant_graphql
x-enumDescriptions:
assistant: Assistant-guided form for systems with a Celigo connector available.
http: Standard HTTP import form for generic HTTP endpoints.
rest: Legacy REST form type for older imports not supported by the new HTTP framework.
graph_ql: GraphQL import form for GraphQL API endpoints.
assistant_graphql: Assistant-guided GraphQL form for systems with a connector that supports GraphQL.
description: Determines the UI form and configuration experience for this import.
x-celigo-ai-guidance:
- Use `assistant` as the default value if there is a connector available for the system.
- |-
Use `assistant_graphql` as the default value if there is a connector available
for the system and the connector supports GraphQL.
- |-
The value selects the editor and changes one thing in a GraphQL request: under
`graph_ql` and `assistant_graphql`, a `variables` member stored as a JSON string is
parsed and sent as an object, while under `http` it is sent as the string; an object
`variables` goes out unchanged under `graph_ql` and `http`. Live-verified 2026-10-22 with import
previews against an echo endpoint: a string `variables` under `graph_ql`,
`assistant_graphql` and `http`, and an object `variables` under `graph_ql` and `http`.
- |-
Each `http.body[]` entry stays a template STRING — a
stringified `{"query", "operationName", "variables"}` envelope, one per operation for the
update/create pair (`http.body[0]` / `http.body[1]`); never an object entry, which is stored
as-is and does not render. The GraphQL form parses that string to show each member in its
own field and re-stringifies it on save; an entry that is not parseable JSON at rest renders
those fields empty, so keep templated variables inside a JSON string value. When
`variables` is itself a JSON string, the string values inside it sit two JSON levels
deep: write them as `\"{{jsonEncode record.x}}\"` with double braces, because a plain
`{{record.x}}`, or a triple-brace `jsonEncode`, fails a record whose value holds a quote
with 422 `invalid_json` before the request is sent. An object or array there is
`{{jsonSerialize record.x}}`, also with double braces; `{{{jsonSerialize record.x}}}`
fails the same way (live-verified 2026-10-22).
- |-
Create GraphQL imports on a `graph_ql` connection (or a connector connection bound to its
GraphQL API): that setup offers the Form view toggle between the GraphQL editor and the
raw HTTP form, and the GraphQL form has no relative URI field — the connection `baseURI`
is expected to be the GraphQL endpoint.
examples:
- assistant
- http
- rest
- graph_ql
- assistant_graphql
type:
type: string
enum:
- file
- records
x-enumDescriptions:
file: Import handles raw file content such as binary data, PDFs, or images.
records: Import handles structured record data such as JSON objects or XML records.
description: |-
Controls whether the import sends structured record data or raw file content.
Most HTTP imports use records; use file only when importing raw file content that will be processed downstream.
x-celigo-ai-guidance:
- Specifies the type of data being imported via HTTP.
examples:
- records
- file
requestMediaType:
type: string
enum:
- xml
- json
- csv
- urlencoded
- form-data
- octet-stream
- plaintext
x-enumDescriptions:
xml: XML request body sent as application/xml.
json: JSON request body sent as application/json, the most common format.
csv: CSV data sent as text/csv.
urlencoded: URL-encoded form data sent as application/x-www-form-urlencoded.
"form-data": Multipart form data for file uploads and mixed content; the body is authored as a JSON array of parts and the platform generates the MIME boundary at send time.
"octet-stream": Raw binary data sent as application/octet-stream.
plaintext: Plain text content sent as text/plain.
description: |-
Content type used to serialize the request body sent to the target API.
Most REST APIs use json.
x-celigo-ai-guidance:
- |-
``form-data`` changes the body contract entirely: author ``http.body``
as a JSON array of ``{name, value, type}`` parts (see the multipart
guidance on ``body``) instead of a Handlebars record template. Do not
include a MIME boundary — integrator.io adds it automatically when
the data is sent.
examples:
- json
- xml
_httpConnectorEndpointIds:
type: array
items:
type: string
format: objectId
x-celigo-refModel: httpconnectorendpoints
description: |-
The connector endpoints this import calls, one per entry of `relativeURI` / `method` (for
example a CREATE and an UPDATE endpoint), shown in the UI as the selected endpoints. On save the
platform derives them (with the version and resource ids) from each `relativeURI` / `method`
pair that matches an endpoint of the connector the connection is bound to, and replaces ids
that belong to another connector; ids for the connection's own connector are kept as sent, and
when nothing matches, or the connection has no connector, whatever was sent is kept.
x-celigo-ai-guidance:
- |-
To build an import on a prebuilt connector, bind the CONNECTION (`http._httpConnectorId`) and
write each operation's `relativeURI` and `method` into the positional arrays; the platform links
the import to those endpoints on save, so the pointer fields may be omitted. Read the saved
import back: missing entries mean a URI matched no endpoint of the connection's connector. Set
`_httpConnectorVersionId`, `_httpConnectorResourceId` and `_httpConnectorEndpointIds` together
only to pin same-connector endpoints the URIs would not match. Imports use this plural field;
the singular `_httpConnectorEndpointId` is dropped on write. Endpoint discovery through the
published operations: `GET /v1/httpconnectors/{_id}`, `…/{_id}/httpconnectorresources`,
`…/{_id}/{_httpConnectorResourceId}/httpconnectorendpoints`, and
`…/{_id}/{_httpConnectorResourceId}/{_id}` for an endpoint's `method` and `relativeURI`.
Verified against the live API 2026-09-21.
blobFormat:
type: string
enum:
- utf8
- ucs2
- utf-16le
- ascii
- binary
- base64
- hex
x-enumDescriptions:
utf8: UTF-8 Unicode encoding, the universal default.
ucs2: UCS-2 encoding for 16-bit Unicode characters.
"utf-16le": UTF-16 Little Endian encoding.
ascii: ASCII encoding for basic Latin characters (7-bit).
binary: Raw binary encoding with no character conversion.
base64: Base64 encoding for representing binary data as ASCII text.
hex: Hexadecimal encoding representing each byte as two hex digits.
description: |
Character encoding format for blob/binary data imports.
Only relevant when type is "file" or when handling binary content.
examples:
- utf8
- base64
batchSize:
type: integer
description: |-
Maximum number of records submitted per HTTP request, which affects throughput and API rate limiting.
REST services typically send one record per request; raise it only for batch endpoints or RPC/XML services that accept multiple records, consulting the target API documentation for the optimal value.
Above 1, the body, relative URI, and header templates render once per request against `batch_of_records` — each entry carrying that record as `record` (or `rows` for a grouped record) — and a root-level `record` or `rows` is undefined.
The default varies by API.
x-celigo-ai-guidance:
- |-
A batched import's templates iterate ``{{#each batch_of_records}}`` and address
``this.record.<field>`` (``this.rows.0.<field>`` for grouped records), or iterate
the bare records as ``{{#each data}}``; a ``{{record.<field>}}`` left at the root
renders empty in the body and fails the request in a relative URI or header. With
the default batch size the shape root is ``record`` / ``rows``, ``data`` holds that
one record, and ``batch_of_records`` is undefined.
examples:
- 1
- 10
- 100
successMediaType:
type: string
enum:
- xml
- json
- plaintext
x-enumDescriptions:
xml: Expect XML-formatted success responses.
json: Expect JSON-formatted success responses, the most common format.
plaintext: Expect plain text success responses.
description: Media type expected in successful responses from the target API. Most APIs return json.
examples:
- json
- xml
requestType:
type: array
items:
type: string
enum:
- CREATE
- UPDATE
x-enumDescriptions:
CREATE: Create new records in the destination system.
UPDATE: Update existing records in the destination system.
description: |-
Operation type each request performs; for composite (upsert) imports, include both values.
The array is positionally aligned with the relativeURI and method arrays — each index maps to one operation — and the existingExtract field determines which operation runs at runtime.
x-celigo-ai-guidance:
- Specifies the type of operation(s) this import performs.
- |-
Example composite upsert:
```json
"requestType": ["UPDATE", "CREATE"],
"method": ["PUT", "POST"],
"relativeURI": ["/customers/{{{data.0.customerId}}}.json", "/customers.json"],
"existingExtract": "customerId"
```
examples:
- [CREATE]
- [UPDATE]
- [UPDATE, CREATE]
errorMediaType:
type: string
enum:
- xml
- json
- plaintext
x-enumDescriptions:
xml: Expect XML-formatted error responses.
json: Expect JSON-formatted error responses, the most common format.
plaintext: Expect plain text error messages.
description: Media type expected in error responses from the target API. Most APIs return json.
examples:
- json
- xml
followRedirects:
type: boolean
default: true
description: |-
When explicitly false, 3xx responses are not followed — the redirect
response itself (status code, `Location` header, body) is what the
import records as the result. Omitted or true follows allowed
redirects (the default behavior).
examples: [false]
maxRedirects:
type: integer
minimum: 1
maximum: 10
description: |-
Caps how many consecutive 3xx redirects are followed. Only applies
when `followRedirects` is not false; to not follow at all, set
`followRedirects: false` rather than `maxRedirects: 0`. Decimal
values are rejected on save.
examples: [3]
relativeURI:
type: array
items:
type: string
x-celigo-agent:
- handlebars
x-celigo-handlebars-context: pre-mapped
description: |-
Relative URI path for each import request, as an array of strings (never an object); values can include Handlebars expressions for dynamic segments.
Templates render against the pre-mapped record — the original input before the Import's mapping step — because URI construction usually needs business identifiers that mappings may rename or remove.
For composite (upsert) imports, the array is positionally aligned with method and requestType, and existingExtract decides the index used at runtime: the UPDATE index when its field has a value, the CREATE index when it is empty or missing.
Reference the existing record ID in the UPDATE URI with `{{{data.0.fieldName}}}` (triple braces with the data.0 prefix).
Use separate array elements per operation rather than `{{#if}}` conditionals inside a single URI.
URI templates render strictly: a path that is undefined on the record — such as `record.<field>` on a grouped record, whose root is `rows` — fails the request instead of rendering empty.
A field the record lacks, or a `{{lookup.<name>}}` whose lookup found nothing, fails that record before anything is sent, with `handlebars_template_parse_error` (source `application`, classification `parse`): `Failed to generate request url from template: <URL>. Details: "<field>" not defined in the model.` That is missing data, not template syntax, and a retry of the same record fails the same way: send records that may lack the destination id through a composite UPDATE/CREATE import (`existingExtract`), or skip them with `ignoreMissing` and `ignoreExtract`.
A field that holds an empty string renders empty, and the request is sent without the value. A field that holds null renders the text `null` (a URI of `/orders/{{record.id}}` is sent as `/orders/null`), and a field under an object that is null fails the record like a missing one (`"<field>" not defined in null`).
x-celigo-ai-guidance:
- '**CRITICAL: This MUST be an array of strings, NOT an object.**'
- |-
## ARRAY SHAPE IS REQUIRED EVEN FOR FULLY-STATIC URLs
The array wrap is a structural requirement of the import
document, not a Handlebars/templating concern. The platform
rejects scalar strings here regardless of whether the URI
uses ``{{...}}`` templating:
✅ ``"relativeURI": ["/fba/outbound/2020-07-01/fulfillmentOrders"]``
✅ ``"relativeURI": ["/customers/{{{record.customerId}}}"]``
❌ ``"relativeURI": "/fba/outbound/2020-07-01/fulfillmentOrders"`` ← scalar string
❌ ``"relativeURI": "/customers/{{{record.customerId}}}"`` ← scalar string
The array shape exists to support the composite create+update
pattern (see "Composite/upsert imports" below), where the
``relativeURI``, ``method``, and ``requestType`` arrays are
positionally aligned. Single-operation imports still use the
same shape -- a 1-element array -- so the runtime never has
to special-case the scalar form.
- |-
## Single-operation imports (one element):
```json
"relativeURI": ["/api/v1/customers"]
```
- |-
## Composite/upsert imports (multiple elements — positionally aligned):
For imports with both CREATE and UPDATE in requestType, relativeURI, method,
and requestType arrays are **positionally aligned**. Each index corresponds
to one operation. The existingExtract field determines which index is used at
runtime: if the existingExtract field has a value → use the UPDATE index;
if empty/missing → use the CREATE index.
Example of a composite upsert:
```json
"relativeURI": ["/customers/{{{data.0.shopifyCustomerId}}}.json", "/customers.json"],
"method": ["PUT", "POST"],
"requestType": ["UPDATE", "CREATE"],
"existingExtract": "shopifyCustomerId"
```
Here index 0 is the UPDATE operation (PUT to a specific customer) and index 1 is
the CREATE operation (POST to the collection endpoint).
**IMPORTANT**: For composite imports, use the `{{{data.0.fieldName}}}` handlebars
syntax (triple braces with data.0 prefix) to reference the existing record ID in
the UPDATE URI. Do NOT use `{{#if}}` conditionals in a single URI — use separate
array elements instead.
- |-
## WRONG FORMAT (DO NOT DO THIS):
```json
"relativeURI": {"/api/v1/customers": "CREATE"}
```
```json
"relativeURI": ["{{#if id}}/customers/{{id}}.json{{else}}/customers.json{{/if}}"]
```
```json
// STRINGIFIED ARRAY — the multi-entry array got JSON-encoded
// into a single string and re-wrapped in a 1-element array.
// The save-time normalizer detects and unwraps this exact
// shape; do not emit it deliberately.
"relativeURI": ["[\"/customers/{{{data.0.id}}}\",\"/customers\"]"]
```
⚠️ On save the platform derives the connector pointer fields
(``_httpConnectorVersionId`` / ``_httpConnectorResourceId`` /
``_httpConnectorEndpointIds``) from each ``relativeURI`` /
``method`` pair that matches an endpoint of the connector the
connection is bound to (path parameters and query strings still
match); pointers you send for that connector are kept, pointers
from another connector are replaced, and a URI that matches
nothing leaves whatever you sent in place — nothing is unset
(verified 2026-09-21 on create, on PUT, and after a re-save). Keep
each entry equal to the endpoint you mean so the pointer and the
request agree. Get the URI shape right.
- |-
On a GraphQL-form import (`formType` `graph_ql` / `assistant_graphql`) the form owns
this field: it shows no relative-URI field and writes `["/"]` back on every save in the
editor, because the request path is expected to come from the connection's `baseURI` —
a `graph_ql` connection, or a connector connection bound to its GraphQL API. A path set
here through the API on such an import does reach the wire, but it is invisible in the
editor and reverts to `["/"]` the next time the import is saved there; when the
connection's base URI stops at an API root, move the import to a connection that
supplies the GraphQL endpoint instead of patching this field.
examples:
- ["/api/v1/customers"]
- ["/customers/{{{data.0.shopifyCustomerId}}}.json", "/customers.json"]
method:
type: array
items:
type: string
enum:
- GET
- PUT
- POST
- PATCH
- DELETE
x-enumDescriptions:
GET: Retrieve data from the server (rarely used for imports).
PUT: Replace an existing resource entirely.
POST: Create a new resource, the most common method for imports.
PATCH: Partially update an existing resource.
DELETE: Remove a resource from the server.
description: |-
HTTP method used for each import request.
For composite (upsert) imports, the array is positionally aligned with relativeURI and requestType — e.g. `["PUT", "POST"]` with `requestType: ["UPDATE", "CREATE"]` uses PUT for updates and POST for creates.
x-celigo-ai-guidance:
- |-
For composite/upsert imports,
specify multiple methods positionally aligned with relativeURI and requestType arrays.
Each index maps to one operation.
examples:
- [POST]
- [PUT, POST]
_httpConnectorVersionId:
type: string
format: objectId
description: |-
Pins the import to a version of the published HTTP connector its connection is bound to
(`connection.http._httpConnectorId`), so the UI renders that connector's endpoint forms instead
of hand-authored requests. Linked on save together with `_httpConnectorResourceId` and
`_httpConnectorEndpointIds` — see that field for the rule.
_httpConnectorResourceId:
type: string
format: objectId
x-celigo-refModel: httpconnectorresources
description: |-
The connector resource (a group of related endpoints, such as Orders) that this import's
endpoints belong to. Linked on save together with `_httpConnectorEndpointIds` — see that field
for the rule.
_httpConnectorEndpointId:
type: string
format: objectId
readOnly: true
x-celigo-refModel: httpconnectorendpoints
description: |-
Not used by imports: values sent on create or update are dropped, and the platform does not
populate it. Imports bind connector endpoints per operation through `_httpConnectorEndpointIds`.
body:
type: array
items:
type: [string, 'null']
description: |-
Request body template for the import — an array of Handlebars template strings
positionally aligned with the `method` array (e.g. `["{{{record}}}"]`). A `null`
entry means no template for the method at that position. Each entry is a string:
write a JSON body as a template string (`["{\"id\": \"{{record.id}}\"}"]`), not as
a JSON object, because an object entry does not render as JSON at runtime.
Leave undefined for standard imports — the Import's mapping step transforms the source record and the mapped data is sent as the request body automatically.
Set it only when the target API requires XML, SOAP, or a custom structure that mappings cannot produce, or when the user explicitly specifies a request body.
When set, the mapping step still runs first and the template renders against the post-mapped record — use `{{record.<mappedField>}}` to reference mapping outputs.
Hand-templated bodies are harder to maintain and debug than mappings; when in doubt, leave this field undefined.
Custom envelopes can combine static header or account fields with per-message
field references inside a loop.
Static wrappers around the mapped record are appropriate for message-bus class envelopes,
HMAC signature wrappers, integration batch headers,
named single-record array wrappers, and similar destination-required structures.
Batch templates can wrap an envelope, such as a header, version, or marketplace,
around serialized records.
Use this pattern when each message needs a small number of fields rather than the whole record.
x-celigo-ai-guidance:
- |-
Handlebars template that produces the HTTP request body.
Renders against the POST-mapped record at runtime:
the Import's mapping step runs first and shapes the source record,
then this template renders against that mapped record.
Reference the mapped record with the ``record`` root when wrapping it (e.g.
``{"requestData": {{{jsonSerialize record}}} }``).
Keep a literal ``}`` apart from a closing ``}}`` or ``}}}`` (a space is enough): the save
accepts ``{{{jsonSerialize record}}}}``, but every record then fails with
``handlebars_template_parse_error``.
Avoid bare ``{{{jsonSerialize record}}}`` with no wrapper — leave the field undefined instead.
Also avoid wrapping a single post-mapped scalar field in ``jsonSerialize`` (e.g.
``{{{jsonSerialize record.<scalar>}}}``) — that pattern is brittle.
- |-
In a JSON body, write a string value with double braces inside the quotes
(``"{{record.name}}"``): double braces JSON-escape the value at run time, with no HTML
escaping. Triple braces insert the raw text, so a value holding a double quote,
backslash, newline or tab fails that record with 422 ``invalid_json`` before anything
is sent. Double braces write a null value -- a null field, or any field under an
object that is null -- as the text ``null``, so ``"{{record.customer.name}}"`` sends
the string "null" (triple braces rendered ``{{{record.customer.name}}}`` empty,
whether ``customer`` or its ``name`` was null). Guard a nullable field with
``{{#if record.customer.name}}{{record.customer.name}}{{/if}}`` or
``{{#if record.customer}}{{record.customer.name}}{{/if}}``: both render empty for a
null field and for a null object. The Handlebars processor renders the unguarded null
empty, so judge it by an import preview or a run.
- |-
Every request context carries two data roots: ``data`` — the list of records in
the request, always present (Pattern B) — and ONE shape-specific root that follows
the record entering the step. An object record is ``record``; a GROUPED record (an
array of rows — a ``groupByFields`` export, file key columns, a NetSuite grouped
saved search) is ``rows`` (``{{rows.0.<field>}}``, ``{{#each rows}}``,
``{{{jsonSerialize rows}}}``) and ``record.*`` is undefined on it; when
``batchSize`` exceeds 1 the shape root is ``batch_of_records`` and neither ``record``
nor ``rows`` exists at the top level. Inside ``data`` a grouped record is itself an
array element (``{{#each data}}{{#each this}}…{{/each}}{{/each}}``). An object record
that carries its own ``rows`` field is still ``record`` — ``{{#each record.rows}}``.
Here an undefined path renders empty — and ``strictHandlebarEvaluation: true`` does
not change that (live-verified 2026-09-24 in real flow runs: an undefined
``{{record.<field>}}`` in the body rendered empty and the record succeeded); in
``relativeURI`` and ``headers[].value`` templates an undefined path always fails the
request — a wrong root there is a runtime failure, not a blank. Verify the body's
root against the record shape; no flag will catch it.
- |-
## LEAVE UNDEFINED for the most common case
When in doubt, leave this field undefined. The platform sends
the post-mapped record as JSON automatically — the body template
only adds value when there's something static around the record
(a message envelope, signature wrap, batch header, etc.) that
the mapper shouldn't have to know about. If your template
would be bare ``{{{jsonSerialize record}}}`` with nothing
around it, you don't need a template at all.
Leave undefined for:
- REST APIs accepting JSON CRUD payloads (the destination
already wants exactly the post-mapped record shape).
- Any case where the destination wants the mapped record as-is.
GraphQL destinations are different: the body is the stringified
``{query, operationName, variables}`` envelope described under
``formType``, with the post-mapped record passed in
``variables``.
- |-
## SET THIS FIELD when one of the patterns below applies
Choose the pattern that fits the destination's shape.
### Pattern A — ``{{{jsonSerialize record}}}`` inside a static wrapper
Use this ONLY when the destination requires a small static
structure around the mapped record that the user shouldn't have
to author in the mapper itself — a message-bus class envelope,
an HMAC signature wrap, an integration batch header, a named
single-record array wrap, etc. The mapper stays focused on the
destination's field shape; the body owns the transport-level
scaffolding.
Rare for non-GraphQL destinations, but real; the use cases
that fit are obvious when the user describes them ("wrap each
request in a SQS-style message envelope", "add a batch header
with timestamp + interface name", etc.).
Typical shapes:
- Message-bus envelope:
``{"MessageBody": {"class": "EntityUpdate", "payload": {{{jsonSerialize record}}} }}``
- Metadata wrap:
``{"batchNumber": "{{timestamp dateTimeFormat CST}}", "interfaceType": "...", "requestData": {{{jsonSerialize record}}} }``
- Named single-record array wrap (when destination wants ``[record]`` but each request sends one):
``{"products": [{{{jsonSerialize record}}}]}``
NEVER emit BARE ``{{{jsonSerialize record}}}`` with no wrapper.
The platform sends the post-mapped record as JSON automatically
when body is undefined; bare ``jsonSerialize record`` produces
identical wire output, so it is redundant config the user has
to maintain for zero benefit.
### Pattern B — Batch / page wrapper around mapped records
Wrap an envelope (header, version, marketplace, etc.) around
a serialized batch. The template root depends ONLY on
``batchSize`` (live-verified 2026-09-10): with ``batchSize`` 1
(the default) the root carries ``record`` (an object record —
or the fanned-out child on a ``oneToMany`` import) or ``rows``
(a grouped/array record), and ``data`` is the one-element list
of records in the request (a grouped record is that element,
itself an array); with ``batchSize`` > 1 the root carries
``batch_of_records`` (one ``{record: …}`` item per object
record, ``{rows: […]}`` for a grouped record) and ``data`` (the
bare records), and ``record`` / ``rows`` are ABSENT. ``oneToMany`` does not make
``batch_of_records`` appear — children are batched after
fan-out under the same rule.
Example shape:
``{"values": {{#each data}} {{{jsonSerialize this}}} {{/each}} }``
Example with comma-separation (``batchSize`` > 1):
``[{{#each batch_of_records}}{{{jsonSerialize this.record}}}{{#unless @last}},{{/unless}}{{/each}}]``
### Pattern C — Custom envelope with per-message fields
Static envelope (header, account-id, etc.) with per-message
field references inside a loop. Use when each message needs
a small number of fields rather than the whole record.
``{{sku}}`` / ``{{quantity}}`` / etc. reference fields on the
current ``{{#each data}}`` item.
Example shape:
``{"header": {"sellerId": "{{connection.http.unencrypted.sellerId}}"}, "messages": [{{#each data}}{"sku": "{{sku}}", "qty": {{quantity}} }{{/each}}]}``
- |-
## Anti-pattern — DO NOT wrap a post-mapped scalar in jsonSerialize
Do NOT use ``{{{jsonSerialize record.<scalar_field>}}}`` or
``{{{jsonSerialize (getValue "record.<scalar_field>")}}}`` where
the path resolves to a scalar string / number. The exact path
depends on the destination's mapped shape — flat fields like
``record.sku`` or ``record.total``, or nested fields like
``record.line.qty`` if the mapping nests under ``line``. The
anti-pattern is the ``jsonSerialize`` wrap, not any particular
path shape.
- Essentially never needed.
- Failure mode: the body validator sees the PRE-mapped sample
shape, can't resolve the post-mapped path during validation,
and ``jsonSerialize`` on an unresolved path raises
``Helper {{jsonSerialize}} expects an argument`` — the
template fails validation even when it would render at
runtime. Variations (``getValue`` wrappers, alternate paths)
fail the same way.
If you find yourself wanting to reference a per-field
post-mapped scalar wrapped in ``jsonSerialize``, your situation
is almost certainly the "leave body undefined" case (the
destination just wants the mapped record as JSON) or Pattern C
(use ``{{sku}}`` / ``{{quantity}}`` directly without
``jsonSerialize`` inside a per-message envelope). To embed a
scalar in a JSON string literal, use double braces inside the
quotes (``"{{record.<field>}}"``) WITHOUT ``jsonSerialize``;
triple braces there fail on a quote, backslash, newline or tab
in the value. The anti-pattern is specifically the
``jsonSerialize`` wrap.
- |-
## Multipart/form-data uploads are a different body contract
When the request media type is ``form-data`` (on the connection
or via ``requestMediaType``), this field is NOT a Handlebars
record template — it is a JSON array of multipart parts, each
``{name, value, type}`` with optional ``filename`` (include the
file extension) and ``mime-headers``:
- The file part is ``"type": "attachment"`` with
``"value": "{{blob}}"`` — ``{{blob}}`` / ``{{{blob}}}`` are the
only documented values for an attachment part. Import saves do
NOT validate this (live-verified: a wrong value saves cleanly),
so it must be right at authoring time. One file reference per
import.
- The file bytes come from integrator.io storage via the
import's ``blobKeyPath``, which points at the record's
blobKey from an upstream blob export/lookup.
- Plain text fields the API wants alongside the file ride as
additional ``"type": "inline"`` parts; an inline part whose
value is a JSON object must be serialized:
``{{{jsonEncode (jsonSerialize field)}}}``.
- Do not include a MIME boundary — integrator.io assembles the
final multipart body and adds the boundary automatically at
send time. That assembled body is not available in audit
logs.
- |-
## Hand-authored multipart bodies (multipart/related APIs)
Some upload APIs expect a multipart flavor other than
form-data — Google Drive's
``/upload/drive/v3/files?uploadType=multipart`` expects
``multipart/related`` — and the form-data parts-array
machinery does not produce that flavor. For those APIs,
author this field as the LITERAL MIME document instead: an
explicit boundary, a metadata part (for Drive, JSON carrying
``name`` / ``parents``), and a content part whose body is
``{{blob}}`` (double braces) — the platform streams the
staged file bytes at that reference.
- Keep ``blob: true`` + ``blobKeyPath`` on the import; the
blob reference still comes from an upstream blob
export/lookup's response mapping.
- Do NOT set ``http.type: "file"`` alongside an API-endpoint
URL. That switches the step into file-provider folder mode,
which treats the relativeURI as a folder path — the mixed
config saves cleanly and fails every record at runtime with
"Directory is not found."
- Match the flavor to the destination API's documentation
before choosing: form-data APIs get the parts array above;
multipart/related APIs get the hand-authored document. Never
mix the two shapes in one body.
examples:
- ['{"name": "{{record.name}}", "email": "{{record.email}}"}']
- ['{"requestData": {{{jsonSerialize record}}} }']
- ['{"values": [{{#each batch_of_records}}{{{jsonSerialize this.record}}}{{#unless @last}},{{/unless}}{{/each}}]}']
x-celigo-agent:
- handlebars
x-celigo-handlebars-context: post-mapped
ignoreEmptyNodes:
type: boolean
description: |-
When true, fields with empty values are stripped from the request body before it is sent,
so the target API receives only populated fields. Shown as "Remove empty fields from HTTP
request body" in the UI.
examples: [true, false]
existingExtract:
type: string
description: |-
Field name or JSON path that drives the upsert decision for composite imports with both CREATE and UPDATE in requestType.
When the field has a value in the incoming record, the UPDATE operation's relativeURI and method are used; when it is empty or missing, the CREATE operation runs.
Must match a field populated by an upstream lookup or response mapping (e.g. a destination system ID like "shopifyCustomerId"); omit for single-operation imports.
x-celigo-ai-guidance:
- |-
Used together with ``requestType: ["UPDATE", "CREATE"]`` (or the
reverse order). The runtime evaluates this field on every
incoming record:
- If the field has a non-empty value → use the UPDATE operation
(and the corresponding ``relativeURI`` / ``method`` at the
matching positional index).
- If the field is empty or missing → use the CREATE operation.
- |-
The field must be populated by an upstream step (a prior lookup node,
a response mapping from an earlier write,
or a source- system field maintained as a cross-reference like ``shopifyCustomerId``).
This pattern adds **zero** per-record HTTP overhead because the existence check
happens against a field that's already on the record.
- |-
## Mutually exclusive with ``existingLookupName``
``existingExtract`` and ``existingLookupName`` are two ways to
answer the same question ("does this record already exist in the
destination?") for composite imports — set EXACTLY ONE. Use
``existingExtract`` when the incoming record already carries
the destination id; use ``existingLookupName`` (with a named
entry in ``http.lookups``) when the import needs to probe the
destination per record because the record has no id.
- |-
## When NOT to set this field
Only valid for composite imports with BOTH ``CREATE`` and
``UPDATE`` in ``requestType``. Single-operation imports (pure
create or pure update) use ``ignoreExtract`` / ``ignoreLookupName``
instead — those work together with ``ignoreExisting`` /
``ignoreMissing`` to skip records rather than route them.
examples:
- "shopifyCustomerId"
- "netsuite_id"
- "existingRecordId"
existingLookupName:
type: string
description: |-
Name of an entry in `lookups` that resolves whether each record already exists in the destination, driving the composite upsert decision when `existingExtract` alone is insufficient (e.g. the check requires a call to the target system). Paired with composite CREATE/UPDATE imports.
Use `existingLookupName` when the incoming record has no destination ID and
needs a per-record probe to discover one;
use `existingExtract` when the record already carries the ID.
For single-operation imports,
use `ignoreExtract` or `ignoreLookupName` with `ignoreExisting` or
`ignoreMissing` to skip records instead of routing them.
The lookup pattern adds one HTTP request per record at runtime,
making it better suited to one-off migrations and low-volume reconciliation than
high-volume recurring syncs.
x-celigo-ai-guidance:
- |-
Used together with ``requestType: ["UPDATE", "CREATE"]`` (or the
reverse order). The runtime runs the named lookup against every
incoming record:
- If the lookup returns a non-empty value → use the UPDATE
operation (and the corresponding ``relativeURI`` / ``method``
at the matching positional index). The lookup's extracted
value is what the UPDATE URL handlebars resolve to via
``{{lookup.<name>}}`` (e.g. ``PATCH /persons/{{lookup.<name>}}``).
- If the lookup returns empty / null → use the CREATE operation.
- |-
The value MUST match the ``name`` of a lookup entry in ``http.lookups``.
The lookup config itself (HTTP path, method, extract,
fallback chain) is defined separately, in that entry.
- |-
## Mutually exclusive with ``existingExtract``
``existingExtract`` and ``existingLookupName`` are two ways to
answer the same question ("does this record already exist in
the destination?") for composite imports — set EXACTLY ONE.
Use ``existingLookupName`` when the incoming record has no
destination id and needs a per-record probe to discover one;
use ``existingExtract`` when the record already carries the id.
- |-
## When NOT to set this field
Only valid for composite imports with BOTH ``CREATE`` and
``UPDATE`` in ``requestType``. Single-operation imports (pure
create or pure update) use ``ignoreExtract`` / ``ignoreLookupName``
instead — those work together with ``ignoreExisting`` /
``ignoreMissing`` to skip records rather than route them.
examples:
- "lookupCustomerByEmail"
- pipedrive_person_lookup
- netsuite_internal_id_lookup
ignoreExtract:
type: string
description: |-
Field name on the incoming record that drives the SKIP-vs-PROCESS
decision when ``ignoreExisting`` or ``ignoreMissing`` is set.
Pairs with either of the two skip-mode flags:
- With ``ignoreExisting: true`` (CREATE-only imports that should
avoid duplicates): if ``ignoreExtract`` has a value on the
record, the record is treated as already-existing and is
skipped; if empty, the record is created.
- With ``ignoreMissing: true`` (UPDATE-only imports that should
avoid creating new records): if ``ignoreExtract`` has a value,
the record is treated as existing and is updated; if empty,
the record is skipped.
``ignoreExtract`` checks a field that's already on the record —
zero per-record HTTP overhead. Use ``ignoreLookupName`` instead
when the record has no field to inspect and the existence check
needs a per-record probe against the destination.
**Mutually exclusive with ``ignoreLookupName``**
Set EXACTLY ONE of ``ignoreExtract`` / ``ignoreLookupName`` when
``ignoreExisting`` or ``ignoreMissing`` is enabled.
**When not to set this field**
Only valid for single-operation imports with either
``ignoreExisting: true`` or ``ignoreMissing: true`` set.
Composite (upsert) imports — those with both ``CREATE`` and
``UPDATE`` in ``requestType`` — use ``existingExtract`` /
``existingLookupName`` instead, because they route records
between two write paths rather than skipping them.
examples:
- "id"
- "shopifyCustomerId"
- "$.customer.email"
endPointBodyLimit:
type: integer
description: |
Maximum size limit for the request body in bytes.
Used to enforce API-specific size constraints.
headers:
type: array
items:
type: object
properties:
name:
type: string
description: Name of the HTTP header to include with each import request.
value:
type: string
description: |-
Value sent for the header. Supports handlebars expressions rendered against the pre-mapped record, with the same data roots as the body (`record` / `rows` / `batch_of_records`, plus `data`); an undefined path fails the request rather than rendering empty.
x-celigo-agent:
- handlebars
x-celigo-handlebars-context: pre-mapped
description: |-
Custom HTTP headers included with import requests; values can contain Handlebars expressions.
Value templates render against the pre-mapped record (the original input before the Import's mapping step) — use `{{record.<field>}}` to reference fields as they appear in the upstream source.
x-celigo-ai-guidance:
- Each header has a name and value, which can include handlebars expressions.
examples:
- [{"name": "X-Custom-Header", "value": "custom-value"}]
response:
type: object
properties:
resourcePath:
type: array
items:
type: string
description: |-
Per-operation JSON path to the resource collection in each
response. Outer-array length MUST match ``requestType``
length: 1 entry for single-op imports, N entries for an
N-operation composite import (duplicate the value when
operations agree). See the ``response`` object docs
above for the full positional-array contract.
Required when ``batchSize > 1`` because the platform needs
to know where the array of result records lives inside the
response envelope. When the API returns a bare array at
the top level, leave this field unset.
Each entry is a DOT-PATH STRING (e.g. ``"data.results"``,
``"items"``) — do NOT split paths into segments across
array elements.
examples:
- ["data.results"] # single-op
- ["data.results", "data.results"] # composite, both agree (duplicated)
- ["data.updated", "data.created"] # composite, different per op
resourceIdPath:
type: array
items:
type: [string, 'null']
description: |-
Per-operation JSON path to the unique ID field within each
record in the response. Outer-array length MUST match
``requestType`` length — duplicate the value when both
operations agree (the common case) rather than collapsing
to a 1-entry array.
Each entry is a DOT-PATH STRING, or null for an operation
with no ID path configured. When not specified, the platform
looks for standard `id` or `_id` fields automatically. When
set, a response with no value at the path fails the record
with `missing_id`, quoting the response.
x-celigo-ai-guidance:
- |-
Common LLM mistake:
splitting a single dot-path like ``"data.id"`` into array segments ``["data",
"id"]`` — DO NOT do this.
See the ``response`` object docs above.
- |-
For a GraphQL mutation, point it at the id only a successful call returns
(``["data.customerCreate.customer.id"]``): a 200 that carries top-level ``errors``
or the mutation's ``userErrors`` then fails the record with ``missing_id`` instead
of counting as written. Live-verified 2026-10-22 against an echo endpoint.
examples:
- ["id"] # single-op
- ["data.id"] # single-op, nested response
- ["data.id", "data.id"] # composite, both agree (duplicated)
successPath:
type: array
items:
type: string
description: |-
Per-operation JSON path to a field that indicates whether
the API call succeeded. Outer-array length MUST match
``requestType`` length — duplicate the value when both
operations agree rather than collapsing to a 1-entry
array on a composite import.
Use this when the API returns HTTP 200 for everything and
signals success / failure through a body field. Pairs with
``successValues`` to define which values at this path mean
success.
Each entry is a DOT-PATH STRING.
examples:
- ["success"] # single-op
- ["status"] # single-op, status field
- ["success", "success"] # composite, both agree (duplicated)
successValues:
type: array
items:
type: array
items:
type: string
description: |-
Per-operation list of values at ``successPath`` that mean
success. TWO levels of array nesting:
- Outer array: one entry per ``requestType`` index.
Length MUST match ``requestType`` length — duplicate
the inner array when both operations agree rather than
collapsing to a 1-entry outer array on a composite
import.
- Inner array: the list of acceptable success values for
that operation.
All comparisons are STRING comparisons regardless of the
actual response value type — the platform stringifies
numbers, booleans, etc. before comparing.
examples:
- [["true"]] # single-op: success when path == "true"
- [["ok", "OK"]] # single-op: multiple acceptable values
- [["true"], ["true"]] # composite, both agree (duplicated)
- [["true", "1"], ["true"]] # composite, UPDATE accepts more values
failPath:
type: array
items:
type: string
description: |-
Per-operation JSON path to a field that signals failure
even when HTTP returns 200. Outer-array length MUST
match ``requestType`` length — duplicate when both
operations agree. Same array semantics as ``successPath``.
Some APIs use only ``successPath`` (failure = absence of
success); others publish both an explicit success indicator
and an explicit failure indicator. When both are
available, set both — the platform evaluates ``successPath``
first and falls through to ``failPath`` only when the
success check is inconclusive.
Without ``failValues``, a truthy value at the path fails
the response. A path that resolves to an array matches only
when the array has exactly one element.
x-celigo-ai-guidance:
- |-
A GraphQL API usually reports a failed mutation with HTTP 200 and an ``errors`` array, which
counts as written when no rule catches it. ``failPath: ["errors"]`` fails the record
for one error and lets two or more through; use ``["errors[0].message"]`` with
``errorPath: "errors[0].message"`` so the record fails with the server's message.
Live-verified 2026-10-22 with import runs against an echo endpoint.
examples:
- ["error"] # single-op
- ["success"] # single-op, same path as successPath, opposite values
- ["error", "error"] # composite, both agree (duplicated)
- ["errors[0].message"] # single-op, GraphQL
failValues:
type: array
items:
type: array
items:
type: string
description: |-
Per-operation list of values at ``failPath`` that mean
failure. Same two-level array shape and length contract
as ``successValues`` — outer length MUST match
``requestType`` length, duplicate the inner array on
composite imports when both operations agree.
Often used as the inverse of ``successValues`` (e.g.
``successValues: [["true"]]`` paired with
``failValues: [["false"]]``).
examples:
- [["false"]] # single-op
- [["false"], ["false"]] # composite, both agree (duplicated)
- [["false", "error"], ["false"]] # composite, UPDATE has more failure values
errorPath:
type: string
description: |-
JSON path to the error message inside the response body.
**Note**: this field is NOT a per-operation array — it's a
single string applied to every operation. Most APIs use
the same error-message field shape for both UPDATE and
CREATE responses, so the platform doesn't expose a per-op
override here.
examples:
- error
- errors[0].message
- "data.error"
allowArrayforSuccessPath:
type: boolean
description: When true, allows array values at successPath during success evaluation.
x-celigo-ai-guidance:
- |-
Whether to evaluate ``successPath`` when its value is an array (rather than a single scalar).
When ``true``,
the success check passes if ANY element in the array matches ``successValues``.
Defaults to ``false`` (single-value comparison only).
hasHeader:
type: boolean
description: When true, treats the first record in the response as a header row (for CSV responses).
x-celigo-ai-guidance:
- |-
For CSV responses:
whether the first record is a header row that names the columns rather than carrying data.
Ignored for JSON / XML responses.
description: |-
Configuration for parsing and interpreting HTTP responses returned
from each write request the import issues.
**Critical: positional per-operation arrays**
Every field inside ``response`` (``resourcePath``, ``resourceIdPath``,
``successPath``, ``successValues``, ``failPath``, ``failValues``)
is an ARRAY whose outer length is positionally aligned with
``requestType`` / ``method`` / ``relativeURI``. This is the
SAME positional-alignment contract as those three sibling
arrays — see ``relativeURI`` docs for the full explanation.
Each outer-array index describes how to parse the response from
the operation at that same index in ``requestType``:
- **Single-operation import** (``requestType: ["CREATE"]``):
every response field is a 1-element array.
``successPath: ["ok"]`` means "for the one operation,
look at path ``ok`` to detect success".
- **Composite upsert** (``requestType: ["UPDATE", "CREATE"]``):
every response field is a 2-element array. Index 0 parses
the UPDATE response, index 1 parses the CREATE response.
**successValues / failValues — array of arrays**
``successValues`` and ``failValues`` carry an extra array level
because each operation can have MULTIPLE acceptable values:
- Outer array: one entry per operation (positional, as above).
- Inner array: the list of acceptable values for that operation.
Example:
```json
"requestType": ["UPDATE", "CREATE"],
"successPath": ["ok", "ok"],
"successValues": [["true", "1"], ["true"]]
```
Means: the UPDATE response is successful when ``ok`` is either
``"true"`` or ``"1"``; the CREATE response is successful only
when ``ok`` is ``"true"``.
**Always duplicate when operations agree (REQUIRED for composite)**
The outer-array length MUST match ``requestType`` length on
every response field. When the UPDATE and CREATE responses
of the same connector look the same (which is the common
case — most APIs return the same shape from both endpoints),
DUPLICATE the entry so the array has the same length as
``requestType``. Do NOT collapse to a 1-entry array on a
composite import.
```json
// ✅ Correct — explicit per-operation entries, even when identical
"requestType": ["UPDATE", "CREATE"],
"resourceIdPath": ["data.id", "data.id"],
"successPath": ["success", "success"],
"successValues": [["true"], ["true"]]
```
```json
// ❌ Wrong — 1-entry array on a composite import
"requestType": ["UPDATE", "CREATE"],
"resourceIdPath": ["data.id"],
"successPath": ["success"],
"successValues": [["true"]]
```
Why duplicate even when they agree:
- **UI parity.** The Celigo UI renders one response-handling
form per ``requestType`` index. A 1-entry array on a
composite import leaves the second operation's form panel
empty, which looks like a half-built config to the user
reviewing the saved import — even if it works at runtime.
- **Symmetric shape rule.** ``relativeURI`` / ``method`` /
``requestType`` REQUIRE the entry count to match across
all three (the runtime decider uses positional alignment).
Making ``response.*`` follow the same "outer length ==
requestType length" rule means there's ONE shape rule to
remember, not two.
- **Unambiguous intent.** A 2-entry duplicated array
documents in the saved config that the developer
considered both operations. A 1-entry array could mean
"both ops use this" or could mean "I forgot to fill in
the second op." Duplication eliminates the ambiguity.
When operations DIFFER (rare — different response shapes
per endpoint), set distinct values at each index. Either
way the outer length always matches ``requestType``.
**Common llm mistake (DO not do THIS)**
Do NOT split a single dot-path into array segments thinking the
outer array represents path segments:
```json
// WRONG — the LLM expressed "data.id" as path segments
"requestType": ["UPDATE", "CREATE"],
"resourceIdPath": ["data", "id"]
```
This is interpreted by the platform as: UPDATE response's id is
at path ``data``, CREATE response's id is at path ``id`` — almost
never what the user wants. The CORRECT shape is the single
dot-path expressed as ONE string per operation, duplicated to
match ``requestType`` length:
```json
"requestType": ["UPDATE", "CREATE"],
"resourceIdPath": ["data.id", "data.id"]
```
_asyncHelperId:
type: string
format: objectId
x-celigo-refModel: asynchelpers
description: |-
Reference to an AsyncHelper resource that polls an asynchronous
destination API on this import's behalf.
Set this ONLY when the destination API is genuinely asynchronous — it
acknowledges the submitted payload (HTTP 202, a job ticket) and ingests
it in the background, so completion must be polled for (e.g. bulk
image/file ingestion, large bulk-load endpoints). Most imports are
synchronous and need NO async helper; adding one to a synchronous
destination just adds polling overhead plus a status and result export
to maintain. When in doubt, leave it unset.
The referenced helper bundles the polling config plus a required status
export (polled to check progress) and a result export (fetches the
final payload).
examples:
- 60a2c4e6f321d800129a1a3c
x-celigo-agent:
- async_helper
ignoreLookupName:
type: string
description: |-
Name of an entry in `lookups` used to check whether each record already exists, so `ignoreMissing`/`ignoreExisting` can skip it. Use it instead of `ignoreExtract` when existence must be resolved by a call to the target system rather than read from a field on the incoming record.
x-celigo-ai-guidance:
- |-
## When NOT to set this field
Only valid for single-operation imports with either
``ignoreExisting: true`` or ``ignoreMissing: true`` set.
Composite (upsert) imports — those with both ``CREATE`` and
``UPDATE`` in ``requestType`` — use ``existingExtract`` /
``existingLookupName`` instead, because they route records
between two write paths rather than skipping them.
- |-
Name of a lookup defined in ``http.lookups`` that drives the SKIP-vs-PROCESS
decision when ``ignoreExisting`` or ``ignoreMissing`` is set.
- |-
The value MUST match the ``name`` of a lookup entry in ``http.lookups``.
This pattern adds **one HTTP request per record** at runtime — use
``ignoreExtract`` instead when the record already carries the field that signals
existence.
- |-
Pairs with either of the two skip-mode flags:
- With ``ignoreExisting: true`` (CREATE-only imports that should
avoid duplicates): the named lookup probes the destination
per record; if it returns a value, the record is treated as
already-existing and is skipped; if it returns empty, the
record is created.
- With ``ignoreMissing: true`` (UPDATE-only imports that should
avoid creating new records): same probe — a non-empty result
means the record exists and gets updated; empty means skip.
- |-
## Mutually exclusive with ``ignoreExtract``
Set EXACTLY ONE of ``ignoreExtract`` / ``ignoreLookupName`` when
``ignoreExisting`` or ``ignoreMissing`` is enabled.
examples:
- "lookupExistingOrder"
- shopify_customer_lookup
- netsuite_internal_id_lookup
lookups:
x-celigo-agent:
- lookup
description: |-
Lookups referenced by `existingLookupName`/`ignoreLookupName` to resolve record existence against the destination system at runtime.
x-celigo-ai-guidance:
- |-
Lookup definitions executed over the HTTP connection.
Each entry issues an HTTP request (or reads from cache / static map) and
extracts a value that downstream layers reference by ``name``.
- |-
## How callers reference a lookup by name
Lookups are addressable by their ``name`` from three places:
- **Field mappings** — a mapping entry sets
``lookupName: "<name>"`` to resolve a field's value through
the lookup. A ``lookupName`` that matches no entry in this
array fails every record with ``Unable to find lookup in
mapping configs - <name>``.
- **Handlebars expressions** — anywhere handlebars is
evaluated (relativeURI, headers, body), ``{{lookup "name"
value}}`` calls the lookup with ``value`` as input.
- **Top-level decider fields** — ``http.existingLookupName`` /
``http.ignoreLookupName`` set to ``"<name>"`` make a lookup
drive the import's UPDATE-vs-CREATE (composite upsert) or
SKIP-vs-PROCESS (``ignoreExisting`` / ``ignoreMissing``)
decision, respectively. See those fields for the full
decision semantics.
- |-
## Mirror invariant
The Celigo platform persists lookups in TWO locations on an HTTP
import: top-level ``lookups`` and ``http.lookups``, and
``http.lookups`` is the copy that counts. A save keeps
``http.lookups`` and mirrors it to the top level, overwriting a
top-level copy that differs; a ``lookups`` array sent only at the
top level is dropped with no error, on create and on PUT, and both
arrays read back null. Write ``http.lookups`` on every save;
sending the same array at the top level too is harmless.
- |-
## Static vs dynamic vs cached
- **Static** — only ``map`` + optional ``default``. No HTTP
request issued. Use for small controlled vocabularies.
- **Dynamic** — ``method`` + ``relativeURI`` (+ ``body``,
``headers``, ``extract``). Issues a live HTTP request per
lookup. Adds one HTTP request per record at runtime — factor
this into flow design for high-volume syncs.
- **Cached** — ``_lookupCacheId`` + ``extract``. Reads from a
pre-populated LookupCache resource.
items:
type: object
properties:
name:
type: string
description: |
Unique identifier for this lookup. Mapping fields reference it
via ``lookupName`` and Handlebars expressions reference it via
``{{lookup.name}}``. Must be unique within this
resource's ``lookups`` array.
method:
type: string
enum:
- GET
- POST
- PUT
- PATCH
- DELETE
description: |
HTTP method used to issue the lookup request. ``GET`` and
``POST`` are most common. Omit when using a static ``map`` or
``_lookupCacheId``.
relativeURI:
type: string
x-celigo-agent:
- handlebars
description: |
HTTP path (relative to the connection's base URL) that
resolves the lookup. Use Handlebars with triple braces to
inject values from the incoming record — e.g.
``/customers?email={{{email}}}`` or
``/accounts/{{{accountId}}}``.
body:
type: string
x-celigo-agent:
- handlebars
description: |
Request body template for POST / PUT / PATCH lookup methods, typically
a JSON string with Handlebars placeholders (``{"email":"{{email}}"}``);
ignored for GET / DELETE. The record's fields sit at the top level:
``{{email}}`` renders the record's value, while ``{{record.email}}``
renders empty.
x-celigo-ai-guidance:
- >-
Put the lookup's request template here, not in ``postBody``: a
``postBody`` saves without an error but is not kept under
``http.lookups``, so the lookup goes out with no body and the run
reports no error.
- >-
Write string values with double braces inside the quotes
(``"{{email}}"``): they are JSON-escaped at runtime. With triple
braces a value holding a double quote breaks the request body, and
the lookup quietly finds nothing, so an existence check writes the
record as new.
headers:
type: array
items:
type: object
properties:
name:
type: string
value:
type: string
description: |
Optional custom headers to attach to the lookup request.
Useful for endpoints that need a different Accept header or
an auxiliary auth token separate from the connection-level
headers.
extract:
type: string
description: |
Path to the lookup value in the HTTP response body, written
without a ``$.`` prefix: ``id``, ``results[0].id``, or ``ids``
for an array of ids. A ``$.``-prefixed path, a ``[*]``
wildcard, or a dot path through an array (``results.id``)
resolves nothing, so the lookup misses: with
``allowFailures: false`` the record fails
(``batch_lookup_evaluation_failed``); otherwise it goes on with
``default`` or without the value, and nothing errors. A
composite import whose ``existingLookupName`` names such a
lookup sends every record to CREATE. Mapper 2.0 mapping
extracts are the opposite: they take the ``$.``.
_lookupCacheId:
type: string
format: objectId
x-celigo-refModel: lookupcaches
description: |
Optional reference to a LookupCache resource. When set, the
lookup reads from the cache instead of issuing a live HTTP
request — useful for stable reference data that should be
pre-loaded.
map:
type: [object, 'null']
description: |
Optional static key→value object evaluated BEFORE the HTTP
request. When the incoming value matches a key, the mapped
value is returned without hitting the remote endpoint. Null
on dynamic lookups with no static map.
default:
type: [string, 'null']
description: |
Value written when the lookup finds no match and the miss is
allowed (``allowFailures`` true, or omitted on a static
``map``). A save refuses a ``default`` together with an
explicit ``allowFailures: false`` (422 "Lookups with default
set can not have allowFailures explicitly set to false."), so
a lookup either fails a miss (``allowFailures: false``, no
default) or falls back to its default. Stored as null when no
fallback is configured.
x-celigo-ai-guidance:
- >-
Omit this field entirely when no fallback value is
intended. An empty string is a value, not "no default":
the save accepts `default: ""`, and a miss the lookup lets
through writes `""`.
- >-
A default raises no error: every unmatched value goes out
as the default with a 200. Set one only when a single
fallback is right for every value the lookup can't match;
when a miss must stop the record, set allowFailures: false
and leave the default out. An ambiguous answer (two or
more matches) doesn't use the default either: see
useDefaultOnMultipleMatches.
allowFailures:
type: [boolean, 'null']
description: |
When true, a lookup miss uses ``default`` (or leaves the mapped
field out when there is none) and the record continues. When
false, a miss fails the record: ``lookup_error`` ("Could not find
value for static lookup: …") for a static ``map``,
``lookupcache_evaluation_failed`` for a cache, and
``batch_lookup_evaluation_failed`` (source ``internal``, "<field>:
The batch lookup could not be evaluated. Check the value in the
key '<value>', and try again.") for a live request that matches
nothing or whose ``extract`` resolves nothing. Omitted or null,
a miss doesn't fail the record either: a static lookup writes
its ``default`` if it has one, and a lookup without one (static,
cache, or a live request that matches nothing or fails) leaves
the mapped field out.
False is also the only setting under which an ambiguous answer
fails the record: two or more values at ``extract`` (an array of
ids or of result objects) fail it with ``lookup_conflict``
("Multiple matches found for lookup."). Omitted, null or true,
the first value is sent with no error, and an array of result
objects is written as the text ``[object Object]``. An indexed
``extract`` (``ids[0]``) takes the first match whatever this
field says, so it hides an ambiguous search rather than fixing
it.
x-celigo-ai-guidance:
- >-
Set false explicitly when a miss must stop the record. Leaving
the field out lets the record through with the default or
without the looked-up value, so a destination that needs the
real value gets a request without it and nothing errors.
- >-
Fix an ambiguous lookup in its search, not here: query on a
key that identifies one record (an email, an external id)
and keep false, so a search that still finds two fails
instead of writing against the first match.
useDefaultOnMultipleMatches:
type: boolean
description: |
Meant to return ``default`` when the response holds more than
one match. On an HTTP import it has not been seen to take
effect: with ``allowFailures`` omitted or true the first match
is sent whatever this flag says, with or without a
``default``, and with ``allowFailures: false`` the record
fails with ``lookup_conflict``. Two or more matches fail the
record only when ``allowFailures`` is false (see
``allowFailures``), so make the search match one record
rather than relying on this flag.
type: array
references/schemas/jdbc.yml
Jdbc:
type: object
description: |-
Configuration for JDBC import operations. Defines how data is written to a database
via a JDBC connection.
**Query type determines which fields are required**
| queryType | Required fields | Do NOT set |
|------------------|------------------------------|-------------------|
| ["per_record"] | query (array of SQL strings) | bulkInsert |
| ["per_page"] | query (array of SQL strings) | bulkInsert |
| ["bulk_insert"] | bulkInsert object | query |
| ["bulk_load"] | bulkLoad object | query |
**Critical:** query IS AN ARRAY OF STRINGS
The query field must be an array of plain strings, NOT a single string and NOT an array of objects.
Correct: ["INSERT INTO users (name) VALUES ('{{{name}}}')"]
Wrong: "INSERT INTO users ..."
Wrong: [{"query": "INSERT INTO users ..."}]
required:
- queryType
properties:
query:
type: array
items:
type: string
x-celigo-agent:
- sql
description: |-
Array of SQL query strings to execute. REQUIRED when queryType is ["per_record"] or ["per_page"].
Each element is a complete SQL statement as a plain string. Typically contains a single query.
Use Handlebars {{fieldName}} syntax to inject values from incoming records.
**Format — array of strings**
- CORRECT: ["INSERT INTO users (name, email) VALUES ('{{{name}}}', '{{{email}}}')"]
- WRONG: "INSERT INTO users ..." (not an array)
- WRONG: [{"query": "INSERT INTO users ..."}] (objects are invalid — causes Cast error)
**Examples**
- INSERT: ["INSERT INTO users (name, email) VALUES ('{{{name}}}', '{{{email}}}')"]
- UPDATE: ["UPDATE inventory SET qty = {{{quantity}}} WHERE sku = '{{{sku}}}'"]
- MERGE/UPSERT: ["MERGE INTO target USING (SELECT CAST(? AS VARCHAR) AS email) AS src ON target.email = src.email WHEN MATCHED THEN UPDATE SET name = ? WHEN NOT MATCHED THEN INSERT (name, email) VALUES (?, ?)"]
**String vs numeric values in handlebars**
- Use triple braces so the raw value is emitted; double braces emit the value already single-quoted, so wrapping them in quotes double-quotes it.
- Strings: supply the quotes yourself — '{{{name}}}'
- Numbers: no quotes — {{{quantity}}}
queryType:
type: array
items:
type: string
enum:
- bulk_insert
- per_record
- per_page
- bulk_load
x-enumDescriptions:
bulk_insert: Batch INSERT via multi-row VALUES clause for efficient bulk data loading.
per_record: Execute one SQL statement per record, giving full control over individual record SQL.
per_page: Execute one SQL statement per page (batch) of records.
bulk_load: Stage data as a file and load via database-native COPY/bulk mechanism for maximum throughput.
x-celigo-ui-override: >-
Required by the JDBC import form (jdbc.queryType is required:true). Encoded to mirror the
form so builders produce connectable configurations.
description: |
Execution strategy for the SQL operation. REQUIRED. Must be an array with one value.
**Decision tree**
1. If UPDATE or UPSERT/MERGE → ["per_record"] (set query field)
2. If INSERT with "ignore existing" / "skip duplicates" / match logic → ["per_record"] (set query field)
3. If pure INSERT with no duplicate checking → ["bulk_insert"] (set bulkInsert object)
4. If high-volume bulk load → ["bulk_load"] (set bulkLoad object)
**Critical relationship to other fields**
| queryType | REQUIRES | DO NOT SET |
|------------------|-----------------------|--------------|
| ["per_record"] | query (array) | bulkInsert |
| ["per_page"] | query (array) | bulkInsert |
| ["bulk_insert"] | bulkInsert object | query |
| ["bulk_load"] | bulkLoad object | query |
**Examples**
- Per-record upsert: ["per_record"]
- Bulk insert: ["bulk_insert"]
- Bulk load: ["bulk_load"]
bulkInsert:
type: object
description: |
Bulk insert configuration. REQUIRED when queryType is ["bulk_insert"]. DO NOT SET when queryType is ["per_record"].
Enables efficient batch insertion of records into a database table without per-record SQL.
properties:
tableName:
type: string
description: |
Target database table name for bulk insert. REQUIRED.
Can include schema qualifiers (e.g., "schema.tableName").
examples: ["public.orders", "dbo.Customers"]
batchSize:
type: string
description: |
Number of records per batch during bulk insert.
Larger values improve throughput but use more memory.
Common values: "1000", "5000".
examples: ["1000", "5000"]
bulkLoad:
type: object
description: |
Bulk load configuration. REQUIRED when queryType is ["bulk_load"]. Uses database-native bulk loading for maximum throughput.
properties:
tableName:
type: string
description: |
Target database table name for bulk load. REQUIRED.
Can include schema qualifiers (e.g., "schema.tableName").
examples: ["public.orders", "ANALYTICS.STG_ORDERS"]
primaryKeys:
type: [array, 'null']
items:
type: string
description: |
Primary key column names for upsert/merge during bulk load.
When set, existing rows matching these keys are updated; non-matching rows are inserted.
Example: ["id"] or ["order_id", "product_id"] for composite keys.
overrideMergeOrInsertQuery:
type: boolean
description: |-
When true, a custom merge/insert query replaces the auto-generated statement for the
bulk load, enabling ignore-existing logic or conditional updates.
lookups:
type: array
items:
type: object
properties:
name:
type: string
description: |-
Unique identifier for this lookup. Mapping fields reference it
via `lookupName` and Handlebars expressions reference it via
`{{lookup "name" value}}`. Must be unique within this
resource's `lookups` array.
examples: ["lookupUserByEmail"]
query:
type: string
x-celigo-agent:
- sql
- handlebars
description: SQL query for the lookup (e.g., "SELECT id FROM users WHERE email = '{{{email}}}'").
x-celigo-ai-guidance:
- |-
SQL `SELECT` statement that resolves the lookup value.
Inject values from the incoming record with Handlebars — JDBC queries use triple braces:
`SELECT id FROM users WHERE email = '{{{email}}}'`.
Returns a single row;
use `extract` to pick a column when multiple are selected.
examples: ["SELECT id FROM users WHERE email = '{{{email}}}'"]
extract:
type: string
description: |-
JSONPath-style expression that selects the value from the
query result row (e.g. `id`, `details.price`). Omit when
the query returns a single scalar.
examples: ["id"]
_lookupCacheId:
type: string
format: objectId
x-celigo-refModel: lookupCaches
description: |-
Optional reference to a LookupCache resource. When set, the
lookup reads from the cache instead of issuing a live SQL
query — useful for stable reference data that should be
pre-loaded.
allowFailures:
type: boolean
description: |-
When true, a lookup miss (no matching row and no `default`)
resolves to `null` and the import continues. When false
(default), a miss halts the record.
map:
type: object
description: |-
Optional static key→value object evaluated BEFORE the SQL
query. When the incoming value matches a key, the mapped
value is returned without querying the database.
default:
type: string
description: |-
Value returned when the query matches no rows. When omitted
and `allowFailures` is false, an unmatched lookup halts the
record.
description: |-
Lookup definitions executed against the JDBC connection. Each entry
runs a `SELECT` and extracts a value used by field mappings
(referenced via `lookupName`) or Handlebars expressions
(referenced via `{{lookup "name" value}}`).
x-celigo-ai-guidance:
- |-
## Mirror invariant
The Celigo platform persists lookups in TWO locations on a JDBC
import: top-level `lookups` and `jdbc.lookups`. Both arrays
must hold byte-for-byte identical content, so write both on every
save.
x-celigo-agent:
- lookup
references/schemas/mongodb.yml
Mongodb:
type: object
description: |-
Configuration for MongoDB imports. Contains only the MongoDB-specific properties; import-level fields such as ignoreExisting, ignoreMissing, name, and description are set on the parent import, not here.
required:
- method
- collection
properties:
method:
type: string
enum:
- insertMany
- updateOne
x-enumDescriptions:
insertMany: Insert multiple documents into the MongoDB collection in a single batch operation.
updateOne: Update a single existing document in the MongoDB collection that matches the filter criteria.
x-celigo-ui-override: >-
Required by the MongoDB import form (mongodb.method is required:true). Encoded to mirror
the form so builders produce connectable configurations.
description: Write operation performed against the MongoDB collection for each record.
collection:
type: string
x-celigo-ui-override: >-
Required by the MongoDB import form (mongodb.collection is required:true). Encoded to
mirror the form so builders produce connectable configurations.
description: |-
MongoDB collection that receives the imported records.
Collection names are case-sensitive and must not start with the reserved "system." prefix.
filter:
type: string
x-celigo-agent:
- handlebars
x-celigo-ui-override: >-
Required by the MongoDB import form when method is updateOne (mongodb.filter is
required:true, visibleWhen method=updateOne).
description: |-
MongoDB query filter that selects the document each updateOne applies to; required when
method is updateOne. Supports standard MongoDB query operators and dot notation, and
handlebars placeholders to reference the incoming record's fields.
examples: ['{"id": "{{record.NSRecordId}}"}', '{"email": "{{record.email}}"}']
document:
type: string
x-celigo-agent:
- handlebars
description: Content of the document written to the collection (used when method is insertMany), as a JSON string that can include nested documents and arrays and handlebars placeholders.
examples: ['{"name": "{{record.name}}", "email": "{{record.email}}"}']
update:
type: string
x-celigo-agent:
- handlebars
description: |-
Modifications applied to matching documents (used when method is updateOne), using MongoDB update operators such as $set, $inc, or $push.
A complete document without operators replaces the matched document entirely.
examples: ['{"$set": {"status": "{{record.status}}"}}']
upsert:
type: boolean
description: |-
When true, inserts a new document built from the update criteria if no existing document matches the filter.
When false, records with no matching document are skipped and nothing is inserted.
ignoreExtract:
type: string
description: |-
Path to the field in the source record that identifies existing records. When that field has a value, the record is treated as existing and skipped.
Only used when the import-level ignoreExisting flag is true; do not set it otherwise.
Enclose field names containing special characters in square brackets (for example, [vendor-code]), and reference array items with an index, such as items[0].id.
x-celigo-ai-guidance:
- Set this together with the import-level ignoreExisting flag; both must be set for skip-existing behavior to work, and ignoreExisting is not part of this mongodb object.
- Use the field path from the incoming source data, not the target MongoDB field. When the prompt names an identifier field (for example, "ignore existing customers indicated by the id field"), use that field.
ignoreLookupFilter:
type: string
description: |-
JSON-stringified MongoDB query filter used to find existing documents in the target collection when the import-level ignoreExisting flag is true.
If the query matches a document, the incoming record is treated as existing and skipped.
Use Handlebars placeholders to reference values from the incoming record, for example "{\"email\":\"{{email}}\"}".
Unlike ignoreExtract, which only checks whether a field on the incoming record has a value, this filter queries the MongoDB collection itself.
x-celigo-ai-guidance:
- The value must be a JSON string (stringify the filter object); filter keys must match field names in the MongoDB collection, and Handlebars variables must match fields in the incoming data.
examples: ['{"id": "{{data.id}}"}', '{"email": "{{email}}"}']
if:
properties:
method:
const: updateOne
required:
- method
then:
required:
- filter
references/schemas/netsuite.yml
NetSuite:
type: object
description: Configuration for NetSuite imports (legacy adaptor; use netsuite_da / NetSuiteDistributedImport for SuiteApp 2.0).
# Requireds are conditional (see the if/then at the end of this schema): the server
# writes stub `netsuite` objects ({preferences:{}} / {lookups:[]}) onto imports of
# every adaptorType as bookkeeping, and file-provider configs (isFileProvider:true)
# have no recordType.
properties:
lookups:
type: array
items:
type: object
properties:
name:
type: string
description: Unique name for this lookup, referenced from mappings and handlebars templates with `{{lookup 'name' value}}`.
recordType:
type: string
description: NetSuite record type searched by a dynamic lookup (for example, `customer`, `salesOrder`). Set with `searchField` or `expression`; omit for a static `map` lookup.
searchField:
type: string
description: NetSuite field the dynamic lookup searches on (for example, `externalid`). Must be a searchable field on `recordType`. Use this or `expression`.
expression:
type: string
description: NetSuite search expression evaluated at runtime to select matching records (for example, `["recipient","is","123"]`). Use this or `searchField`.
resultField:
type: string
description: |-
Field on the matched NetSuite record whose value the lookup returns (for example, `internalid`).
Supports dot notation for nested fields.
map:
type: [object, 'null']
description: Static lookup table with input values as keys and their corresponding output values. Use instead of a NetSuite search for fixed value translations. The platform stores `null` here on search-driven lookups.
default:
type: [string, 'null']
description: Fallback value returned when the lookup finds no match. Used together with `allowFailures`. Stored as `null` when not configured.
allowFailures:
type: boolean
description: When true, an unmatched lookup falls back to `default` and processing continues; when false, an unmatched lookup fails the record.
_id:
type: object
description: Server-assigned unique identifier for this lookup entry.
readOnly: true
description: Lookup definitions used to resolve reference values during the import, using either a static `map` or a dynamic NetSuite record search.
operation:
type: string
enum:
- add
- update
- addupdate
- attach
- detach
- delete
x-lowercase: true
x-enumDescriptions:
add: Create a new record in NetSuite.
update: Update an existing record in NetSuite, identified via internalIdLookup.
addupdate: Upsert — update the record if found, otherwise create a new one.
attach: Attach one NetSuite record to another record.
detach: Detach a previously attached NetSuite record from another record.
delete: Delete an existing record from NetSuite, identified via internalIdLookup.
x-celigo-ui-override: >-
Required by the NetSuite import form (operation is required:true). Encoded to mirror the
form so builders produce connectable configurations.
description: |-
Operation to perform on NetSuite records, as a plain string (the API lowercases the value).
For `update`, `addupdate`, and `delete`, also set internalIdLookup so existing records can be found; for `add` with ignoreExisting, set internalIdLookup to check for duplicates before creating.
Default to `addupdate` when the intent is "sync", "upsert", or "create or update"; use `add` for plain "create" or "insert".
Do not wrap the value in an object like `{"type": "addupdate"}` — that fails validation.
isFileProvider:
type: boolean
description: |-
When true, enables file operations against NetSuite — browsing, uploading, downloading, updating, and deleting files.
When false or omitted, file-related operations are unavailable through this import.
customFieldMetadata:
type: object
description: Metadata describing the custom fields defined on the target NetSuite record type, used to validate and process custom field data during the import.
recordType:
type: string
x-celigo-ui-override: >-
Required by the NetSuite import form (recordType is required:true) for record imports.
File-provider imports (isFileProvider true) are built by the file-provider form, which
sets `file` instead of recordType — the schema's if/then/else mirrors that split.
description: |-
NetSuite record type this import writes to (e.g. customer, salesOrder, invoice).
Use the exact NetSuite internal record type identifier; both standard and custom record types are supported.
examples: ["customer", "salesOrder", "customrecord_user"]
recordTypeId:
type: string
description: The unique identifier specifying the record type within NetSuite. Determines the applicable schema, fields, validation rules, and processing logic. Use exact NetSuite record type identifiers such as standard types or custom record IDs.
retryUpdateAsAdd:
type: boolean
description: |-
When true, an update that fails because the target record does not exist is automatically retried as an add.
When false or unset, update failures error immediately without retrying.
May create duplicate records if the update failed for a reason other than a missing record.
batchSize:
type: number
description: |-
Number of records sent to NetSuite per API call.
Larger batches reduce the number of calls but increase memory use and timeout risk per call.
internalIdLookup:
type: object
properties:
extract:
type: string
description: Path used to extract the value for the internal ID lookup.
examples: ["externalId"]
searchField:
type: string
description: The NetSuite record field used as the key attribute for the internal ID lookup search. Should be a unique or indexed field to ensure accurate matches. Must correspond to a valid, searchable field on the target record type.
examples: ["externalid"]
expression:
type: string
description: |-
NetSuite search expression that filters which records match the lookup.
Supports logical operators (AND, OR, NOT), comparison operators, and nested conditions, evaluated at runtime against current data.
description: |-
Configuration for locating existing NetSuite records by internal ID.
Required when operation is `update`, `addupdate`, or `delete`; also used with `add` plus ignoreExisting to check for duplicates.
preferences:
type: object
properties:
ignoreReadOnlyFields:
type: boolean
description: When true, silently skips read-only fields during update operations instead of raising errors.
warningAsError:
type: boolean
description: When true, treats NetSuite warnings as errors, causing the operation to fail immediately.
skipCustomMetadataRequests:
type: boolean
description: When true, skips custom metadata requests during NetSuite API operations to improve performance.
description: Preferences that control how NetSuite handles the import operation.
file:
type: object
properties:
name:
type: string
description: |-
Filename for the file in the NetSuite File Cabinet, without any folder or path information.
Must be unique within its folder; include the file extension (e.g. .pdf, .csv).
fileType:
type: string
description: |-
NetSuite file type that governs how the file is processed, stored, and displayed.
Must match the actual content format of the file.
folder:
type: string
description: |-
Folder in the NetSuite File Cabinet where the file is stored, as a numeric folder ID or a folder path.
Updating this on an existing file moves the file to the specified folder.
folderInternalId:
type: string
description: |-
Internal ID of the File Cabinet folder where the file is stored.
Must reference an existing folder; changing it moves the file to a different folder.
internalId:
type: string
description: |-
NetSuite internal ID of an existing file, used to target it for retrieval, updates, or deletion.
Assigned by NetSuite when the file is created and cannot be changed.
backupFolderInternalId:
type: string
description: Internal ID of the File Cabinet folder where backup files are stored.
description: Configuration for the file to upload to or update in the NetSuite File Cabinet.
# Requireds apply only to real configs, not server-written bookkeeping stubs whose
# keys are all in {preferences, lookups}. File-provider configs (isFileProvider:true)
# carry operation + file but no recordType. Stack-style conditional (no allOf).
if:
not:
propertyNames:
enum: [preferences, lookups]
then:
required:
- operation
if:
properties:
isFileProvider:
const: true
required:
- isFileProvider
else:
required:
- recordType
references/schemas/netsuitedistributed.yml
NetsuiteDistributed:
type: object
description: |-
Configuration for NetSuite Distributed (SuiteApp 2.0) import operations — the primary sub-schema for the NetSuiteDistributedImport adaptorType. The API field name is "netsuite_da".
operation and recordType are required. The server may attach empty bookkeeping stubs of
this object (empty lookups/mapping, or missingOrCorruptedDAConfig) to imports of other
adaptor types; stubs omit these fields. Set internalIdLookup so existing records
can be found when operation is update, addupdate, or delete.
x-celigo-ai-guidance:
- |-
## KEY FIELDS
- operation: REQUIRED — the NetSuite operation (add, update, addupdate, etc.)
- recordType: REQUIRED — the NetSuite record type (e.g., "customer", "salesorder")
- mapping: field mappings from source data to NetSuite fields
- internalIdLookup: how to find existing records for update/addupdate/delete
- restletVersion: defaults to "suiteapp2.0"; fixed once the step is created (recreate/clone to change)
properties:
operation:
type: string
enum:
- add
- update
- addupdate
- attach
- detach
- delete
x-lowercase: true
x-enumDescriptions:
add: Create a new record in NetSuite.
update: Update an existing record in NetSuite, requires internalIdLookup.
addupdate: Upsert — update the record if found, otherwise create a new one.
attach: Attach one NetSuite record to another record.
detach: Detach a previously attached NetSuite record from another record.
delete: Delete an existing record from NetSuite, requires internalIdLookup.
x-celigo-ui-override: >-
Required by the NetSuite import form (operation is required:true). Encoded to mirror the
form so builders produce connectable configurations.
description: |-
Operation to perform on the target NetSuite record, as a plain string. Required.
For `update`, `addupdate`, and `delete`, also set internalIdLookup so existing records can be found.
x-celigo-ai-guidance:
- |-
## GUIDANCE
- Default to "addupdate" when user says "sync", "upsert", or "create or update".
- Use "add" when user says "create" or "insert" without mentioning updates.
- For "update", "addupdate", and "delete", you MUST also set internalIdLookup.
- |-
## IMPORTANT
This field is a string, NOT an object. Set it directly: "operation": "addupdate"
Do NOT send {"type": "addupdate"} — that causes a Cast error.
recordType:
type: string
x-celigo-ui-override: >-
Required by the NetSuite import form (recordType is required:true). Encoded to mirror the
form so builders produce connectable configurations.
description: |-
NetSuite record type to import into (e.g. "customer", "salesorder", "customrecord_myrecord" for custom record types). Required.
Must match a valid NetSuite record type identifier.
x-celigo-ai-guidance:
- |-
Use the NetSuite internal record-type id: lowercase, no spaces,
no camelCase (e.g.
`salesorder` — NOT `salesOrder` — and `itemfulfillment` — NOT `itemFulfillment`).
Custom records use `customrecord_<id>`.
- |-
Common standard record types in active integrations (ordered most → least frequent):
`salesorder`, `customer`, `invoice`, `itemfulfillment`,
`customerrefund`, `returnauthorization`, `creditmemo`, `inventoryitem`,
`customerpayment`, `customerdeposit`, `journalentry`, `itemreceipt`,
`cashrefund`, `cashsale`, `purchaseorder`, `deposit`, `contact`,
`transferorder`, `inventoryadjustment`, `vendorbill`, `vendor`, `job`,
`noninventoryitem`, `employee`.
examples: ["customer", "salesorder", "customrecord_user"]
recordIdentifier:
type: string
description: |
Custom record identifier, used to identify the specific record type when
importing into custom record types.
restletVersion:
type: string
enum:
- suitebundle
- suiteapp1.0
- suiteapp2.0
x-lowercase: true
x-enumDescriptions:
suitebundle: Legacy SuiteBundle RESTlet version; slated for deprecation — NetSuite no longer updates SuiteScript 1.0.
suiteapp1.0: SuiteApp 1.0 RESTlet version using SuiteScript 1.0.
suiteapp2.0: SuiteApp 2.0 RESTlet version using SuiteScript 2.0, the modern default.
description: |-
Version of the NetSuite RESTlet to use, as a plain string.
Defaults to "suiteapp2.0" when useSS2Restlets is true, "suitebundle" otherwise; rarely needs to be set explicitly for modern integrations.
The version is fixed when the step is created — the Advanced selector is disabled on existing steps, and migrating an existing step to a different version means recreating or cloning it.
x-celigo-ai-guidance:
- |-
Do not wrap the value in an object like `{"type": "suiteapp2.0"}` — that causes a Cast error.
- |-
The NetSuite API version cannot be changed on an existing step — the UI
disables the selector once the step is saved. When asked to migrate an
existing step off the SuiteBundle (or between versions), do not update
this field in place; the migration path is recreating or cloning the
step with the new version.
- |-
SuiteScript hooks are a separate setting from the step's API version:
hook code must be written for the same SuiteScript version as the step
(1.0 and 2.x are not interchangeable), and migrating a hook is its own
task — flag it whenever migrating a step that carries hooks.
- |-
Author new steps with suiteapp2.0 unless the user explicitly requires a
legacy version — NetSuite no longer updates SuiteScript 1.0 and Celigo
plans to deprecate the SuiteBundle option.
useSS2Restlets:
type: boolean
description: |-
When true, uses SuiteScript 2.0 RESTlets and restletVersion defaults to "suiteapp2.0".
Defaults to true for modern integrations.
Set at step creation together with restletVersion — the same creation-time-only constraint applies (see restletVersion).
missingOrCorruptedDAConfig:
type: boolean
description: |-
When true, the Distributed Adaptor configuration is missing or corrupted.
Set by the system — do not set manually.
batchSize:
type: number
description: |
Number of records to process per batch. Controls how many records are sent
to NetSuite in a single API call. Typical values: 50-200.
x-celigo-ai-guidance:
- |-
Batching applies to the record writes. The `internalIdLookup` searches for a
page of records are issued ahead of the writes and independently of `batchSize`,
so `batchSize: 1` does not make the existing-record lookup run one record at a
time.
internalIdLookup:
type: object
description: |-
Configuration for looking up existing NetSuite records to match against incoming data.
Required when operation is "update", "addupdate", or "delete".
x-celigo-ai-guidance:
- |-
Two ways to specify the lookup, picked per import:
- **Expression form** — the default choice: set `expression` to a
NetSuite search-expression literal (the canonical patterns are on
the `expression` field — `["internalid","is","{{{id}}}"]` for a
record that carries the internal id, `externalid` and `cust*_*`
variants, `anyof` for multi-match, AND/OR compounds). When
`expression` is set, `searchField` / `operator` / `extract` are
ignored.
- **Simple form** — set `searchField` + `operator` (+ optional
`extract`) for a single-criterion lookup against a built-in
NetSuite field; the platform builds the equivalent search
expression at runtime. For the internal id itself the expression
form above is the preferred path, and `anyof` is available only
through `expression`.
- |-
The lookup result is what identifies the target record; on `update` a source
field mapped to `internalid` is overridden by it. When the incoming record
already carries the NetSuite internal ID, point the lookup at that field with
the expression form — `["internalid","is","{{{id}}}"]` — rather than removing
the lookup. Only when every record carries it: on `restletVersion:
suiteapp2.0` a record whose field is empty does not fall through to
"create" on `addupdate` — the empty value is searched as `isempty` and
can match an existing record (see `expression`; SuiteBundle and SuiteApp
1.0 create in that case).
properties:
extract:
type: string
description: Path in the source record to extract the lookup value from (e.g. "internalId" or "externalId").
x-celigo-ai-guidance:
- |-
Path on the incoming source record (NOT the post-mapped record) whose value drives the lookup.
Used with the simple form (`searchField` + `operator`).
Common values: `"externalId"`, `"id"`.
Ignored when `expression` is set.
examples: ["externalId"]
searchField:
type: string
description: NetSuite field to search against (e.g. "externalId", "email", "name", "tranId").
x-celigo-ai-guidance:
- |-
NetSuite field the simple-form lookup queries against.
Common values: `"externalId"`, `"email"`, `"tranId"`, `"name"`.
Matching on the record's internal id is the expression form —
`["internalid","is","{{{id}}}"]` — not a `searchField`.
Ignored when `expression` is set.
examples: ["externalId"]
operator:
type: string
description: Comparison operator for the lookup (e.g. "is", "contains", "startswith").
x-celigo-ai-guidance:
- |-
Comparison operator for the simple-form lookup.
Common values: `"is"` (exact match), `"contains"`, `"startswith"`,
`"equalto"`. Multi-match (`anyof`) is available only through
`expression` (`["internalid","anyof","{{{id}}}"]`).
Ignored when `expression` is set.
expression:
type: string
description: NetSuite search expression for complex lookup conditions, used for multi-field or conditional lookups.
x-celigo-ai-guidance:
- |-
NetSuite search-expression literal as a JSON string.
The expression is evaluated at runtime — Handlebars renders `{{...}}`
substitutions FIRST against the incoming source record (pre-mapped),
then the substituted expression is sent to NetSuite as a saved-search query.
- |-
## Handlebars conventions
- **No `record.` prefix** — substitutions reference top-level
source-record fields directly. `{{{id}}}` / `{{{Id}}}` /
`{{{externalid}}}` / `{{{[Internal ID]}}}` — NOT
`{{{record.id}}}`. Top-level paths are the established
convention.
- **Triple braces `{{{...}}}`** are the preferred form;
double-brace `{{...}}` also works for cases where you want
the platform's default HTML-escape (rare in this field).
- **Bracket notation for field names with spaces or special
chars**: `{{{[Field With Spaces]}}}`, `{{{[Internal ID]}}}`.
- |-
## Canonical patterns
### Simple lookup by internal ID (most common)
`["internalid","is","{{{id}}}"]`
Use when the source record carries the destination NetSuite
internal-id directly (e.g., from a prior lookup, response
mapping, or cross-reference field).
### Multi-match by internal ID
`["internalid","anyof","{{{id}}}"]`
Same as above but uses `anyof` to match an internal-id from a
comma-separated list value. `anyof` is only available via the
expression form (the simple `operator` field doesn't carry it).
### Lookup by external ID
`["externalid","is","{{{Id}}}"]` /
`["externalid","is","{{{id}}}"]`
Use when the source record's primary key is stored in NetSuite as
an `externalid` (cross-reference scenarios — Shopify, SFCC,
Amazon, etc., where the platform-specific id is the externalid).
### Lookup by custom-record field
`["custbody_celigo_etail_order_id","is","{{{id}}}"]`
`["custentity_celigo_sfio_sf_id","is","{{{Id}}}"]`
`["custrecord_celigo_shopify_payout_id","is","{{{id}}}"]`
Use when the destination NetSuite record carries a custom field
holding the source-system id. The `cust*_*` field id MUST
match the script-id of an existing custom field on the target
record type (e.g., `custbody_*` on transactions,
`custentity_*` on entities, `custrecord_*` on custom records
and lists).
### Compound AND lookup (multiple criteria)
`[["custbody_celigo_amzio_settlement_id","is","{{{[Settlement Id]}}}"],"AND",["custbody_celigo_amzio_accounts","is","{{{[Amazon Account Internal Id]}}}"]]`
Use when the destination record needs more than one field to
uniquely identify it (composite-key scenarios — Amazon
settlements, multi-store integrations). Each criterion is a
`[field, operator, value]` triple; the outer array alternates
triples and operators (`"AND"` / `"OR"`).
### Compound with OR / nested grouping
`[["custbody_celigo_etail_order_id","is","{{{order_id}}}"],"AND",["custbody_celigo_etail_order_fulfilled","is","T"],"AND",[["custbody_celigo_etail_fulfillment_id","is","{{{id}}}"],"OR",["custbody_celigo_etail_fulfillment_id","isempty",""]]]`
`OR` works the same as `AND`; nesting groups via inner
arrays. Use `"isempty"` (with empty-string value) to match
unset fields.
- |-
## Anti-patterns
- DO NOT prefix substitutions with `record.` — top-level paths
are the convention.
- DO NOT use `jsonSerialize` here — search expressions take
scalar literals, not serialized JSON. Use `{{{id}}}` or
`{{{[Field Name]}}}` directly.
- |-
## A substitution that renders empty is not "no match"
When the source field is absent or empty, `["internalid","is","{{{id}}}"]`
renders `["internalid","is",""]`, and the SuiteApp rewrites an empty
`is` / `equalto` / `internalid anyof` value to `isempty` before the
search. On `restletVersion: suiteapp2.0` that search is not treated as
"no match": it can return existing records, and `addupdate` then
updates one of them — an unrelated record receives the mapping (for
example, a `cant_modify_sub` error on a customer the record never
named). SuiteBundle and SuiteApp 1.0 discard a blank `internalid`
match and create. Design so the operational lookup never renders
empty: key it on a value every record carries (the source id stored
as `externalid`), or route records without a destination id to a
separate `add` import.
- |-
## Body-level only
This lookup identifies the record being imported, so bare
body fields are the right paths here. A per-field lookup
attached to a sublist row (`lookups[]` referenced from
`mapping.lists[].fields[].lookupName`) is different: its
expression addresses the current line with
`<array>[*].<field>` — see `lookups[].expression`.
hooks:
type: object
description: |
Script hooks for custom processing at different stages of the import.
Each hook references a SuiteScript file and function.
properties:
preMap:
type: object
description: Runs before field mapping is applied.
properties:
fileInternalId:
type: [string, 'null']
description: NetSuite internal ID of the SuiteScript file. Null on hook
stubs where no SuiteScript file has been selected.
function:
type: string
description: Name of the function to execute.
configuration:
type: [object, 'null']
description: Configuration object passed to the hook function. Null when
the hook carries no static parameters.
postMap:
type: object
description: Runs after field mapping, before submission to NetSuite.
properties:
fileInternalId:
type: [string, 'null']
description: NetSuite internal ID of the SuiteScript file. Null on hook
stubs where no SuiteScript file has been selected.
function:
type: string
description: Name of the function to execute.
configuration:
type: [object, 'null']
description: Configuration object passed to the hook function. Null when
the hook carries no static parameters.
postSubmit:
type: object
description: Runs after the record is submitted to NetSuite.
properties:
fileInternalId:
type: [string, 'null']
description: NetSuite internal ID of the SuiteScript file. Null on hook
stubs where no SuiteScript file has been selected.
function:
type: string
description: Name of the function to execute.
configuration:
type: [object, 'null']
description: Configuration object passed to the hook function. Null when
the hook carries no static parameters.
mapping:
type: object
description: |-
NetSuite-specific Mapper 1.0 field mappings (the "DA" dialect).
Used by `NetSuiteDistributedImport` exclusively — NetSuite's
Distributed Adaptor cannot consume generic v1 mappings or the
Mapper 2.0 shape because it needs sublist, subrecord, and
conditional lookup semantics the generic schemas don't model.
Field-level shape: `fields` for body columns, `lists` for
sublists / line-item arrays.
x-celigo-ai-guidance:
- |-
Validated via the dedicated `PUT /v1/netsuiteDA/previewImportMappingFields`
endpoint (NOT `/v1/processors/mapperProcessor`).
properties:
fields:
type: array
items:
type: object
properties:
extract:
type: [string, 'null']
description: |
Path in the source record to extract the value from.
Use dot notation for nested fields (e.g., "address.city").
The platform stores `null` here on fields driven by
`hardCodedValue` instead of an extract path.
examples: ["address.city"]
generate:
type: string
description: |
The NetSuite field ID to write the value to (e.g., "companyname", "email", "subsidiary").
examples: ["companyname"]
hardCodedValue:
type: string
description: |
A static value to always use instead of extracting from source data.
Mutually exclusive with extract.
lookupName:
type: string
description: Reference to a lookup defined in netsuite_da.lookups by name.
dataType:
type: string
description: |
Data type hint for the field value (e.g., "string", "number", "date", "boolean").
internalId:
type: boolean
description: When true, the value is a NetSuite internal ID reference.
immutable:
type: boolean
description: When true, this field is only set on record creation, not on updates.
discardIfEmpty:
type: boolean
description: When true, skip this field mapping if the extracted value is empty.
extractDateFormat:
type: string
description: |
Date format of the extracted value (e.g., "MM/DD/YYYY", "ISO8601").
Used to parse date strings from source data.
extractDateTimezone:
type: [string, 'null']
description: Timezone of the extracted date value (e.g., "America/New_York"). Stored as `null` when not configured.
subRecordMapping:
type: object
description: |-
Nested Mapper 1.0 mapping for a NetSuite subrecord
embedded in this field. A subrecord is a child
record that only exists inside a parent (e.g. the
Inventory Detail on a serialized-item line of a
sales order); NetSuite reaches it through the
parent.
**Recursive structure.** The nested `mapping`
object has the same `{fields, lists}` shape as the
top-level `netsuite_da.mapping`. Recursion is
allowed but rare.
x-celigo-ai-guidance:
- |-
**Do not over-emit.** Per the Celigo help center:
"Unconfigured subrecord mappings can cause your NetSuite import to fail.
Only include subrecord mappings on imports that require subrecord data." Emit
this block ONLY when the destination record type genuinely requires the
subrecord.
- |-
**Common `recordType` values** (most common
first):
* `inventorydetail` — by far the most common.
Serial / lot / bin assignments on sales-order,
item-fulfillment, purchase-order, and inventory-
adjustment line items. Nested `mapping`
typically has empty `fields` and a single
`inventoryassignment` sublist.
* `componentinventorydetail` — same concept on
work-order / assembly-build component lines.
* `itemvendorprice` — vendor-specific price tiers
on item-vendor relationships.
* `landedcost` — landed-cost allocations.
* `itempricing` — quantity-based price tiers on
contracts.
* `mainaddress` / `shippingaddress` /
`returnaddress` — address subrecords on
transactions.
* `countdetail` — per-line detail on physical-
inventory counts.
- |-
**Where this block lives.** The enclosing field's `generate` name is a
synthetic trigger (typically `celigo_*`) whose purpose is to carry this
`subRecordMapping` block — it is NOT a real NetSuite column.
The trigger → `recordType` correspondence is mostly `celigo_<recordType>`
but NOT a strict rule:
`celigo_inventorydetail` covers BOTH `inventorydetail` and
`componentinventorydetail` depending on the parent record,
and address triggers use the `celigo_address_<qualifier>` ↔ `<qualifier>address` flip (e.g.
`celigo_address_main` ↔ `mainaddress`).
Match trigger names on existing resources rather than synthesizing new ones.
properties:
recordType:
type: string
description: |
NetSuite's internal record-type id of the
SUBRECORD (not the parent). E.g.
`inventorydetail`, `itempricing`,
`mainaddress`.
jsonPath:
type: string
description: |
JSONPath into the source record that selects the
data feeding this subrecord. Semantics depend
on placement:
* Line-level (on `lists[].fields[].`
`subRecordMapping`) — often `$` to reuse
the enclosing sublist line's row, or a path
into a nested array on that row.
* Body-level (on `fields[].subRecordMapping`)
— a path into the source record root.
Defaults to `$`.
mapping:
type: object
description: |
Nested `{fields, lists}` mapping for the
subrecord. Identical shape to the top-level
`netsuite_da.mapping`. Subrecords often have
empty `fields` and only a sublist (e.g.
`inventorydetail` → `inventoryassignment`).
lookups:
type: array
description: |
Subrecord-scoped lookups. Same shape as the
top-level `netsuite_da.lookups`. Fields inside
this `subRecordMapping.mapping` reference them
via `lookupName`. Often `[]`.
referenceFieldId:
type: string
description: |
Secondary reference-field identifier used on a
small fraction of subrecord blocks. Exact
semantics are not publicly
documented by Celigo — match the value on
existing resources when present; do NOT
synthesize one. Leave unset unless a template
clearly requires it.
conditional:
type: object
description: |
Conditional logic for when to apply this field mapping
("Only perform mapping when" in the UI). Lookup-based
conditions evaluate the lookup named by `lookupName`,
which must be defined on this import.
x-celigo-ai-guidance:
- |-
A `conditional.lookupName` that matches no lookup defined on the
import fails every record at runtime with
`Cannot set properties of undefined (setting '_hasStarNotation')`
(source `application`) — the message names neither the mapping row
nor the lookup. Diagnose by cross-referencing every body and sublist
row's `conditional.lookupName` against the import's defined lookups;
a lookup renamed or removed while a condition still references the
old name is the typical cause. Fix the reference or restore the
lookup — changing the RESTlet / SuiteScript version does not resolve
this error.
- |-
`when: record_updated` requires an operation that can update an
existing record: on an `add`-only operation every record fails with
`The mapping condition:"record_updated" not supported for the
operation`. It works on `addupdate` and `update`.
- |-
The condition is one of the fixed `when` values — there is no
predicate over a source field ("when `customer.id` is not empty",
"when `channel.name` is Point of Sale" cannot be written here).
`extract_not_empty` is the only source-driven gate: on a lookup row
it reads the row's `extract`, and when that is empty the row is
skipped and the field is left unset — nothing substitutes a value.
Value-dependent behaviour (a fallback record when the key is
missing) belongs inside the lookup's `expression` (see
`lookups[].expression`).
- |-
Two rows with the same `generate` are not a fallback pair: the
last row for a destination decides it alone — when its condition
skips it the field stays empty even if an earlier row produced a
value, and when it applies it overwrites the earlier row's value
(`ignore_if_set` reads the target record, not an earlier row).
properties:
lookupName:
type: string
description: |-
Lookup to evaluate for the condition. Must exactly match the
`name` of a lookup defined on this import.
when:
type: string
enum:
- record_created
- record_updated
- extract_not_empty
- lookup_not_empty
- lookup_empty
- ignore_if_set
x-enumDescriptions:
record_created: Apply this mapping only when a new record is being created.
record_updated: Apply this mapping only when an existing record is being updated.
extract_not_empty: Apply this mapping only when the extracted source value is non-empty.
lookup_not_empty: Apply this mapping only when the lookup returns a value.
lookup_empty: Apply this mapping only when the lookup returns no value.
ignore_if_set: Skip this mapping if the target field already has a value.
description: Condition that controls when this field mapping is applied.
description: |
Body-level field mappings. Each entry maps a source field to a NetSuite body field.
lists:
type: array
items:
type: object
properties:
generate:
type: string
description: |
The NetSuite sublist ID (e.g., "item" for sales order line items,
"addressbook" for address sublists).
jsonPath:
type: string
description: |-
LEGACY — do not set. Nominally the JSON path to the array of sublist records, but NetSuite DA ignores a top-level jsonPath here.
x-celigo-ai-guidance:
- |-
NetSuite DA ignores a top-level `jsonPath` on sublist entries and will break
iteration when one is present (only the first source element renders).
Drive iteration from each child field's `extract` instead (see `fields[*].extract` below).
fields:
type: array
items:
type: object
properties:
extract:
type: [string, 'null']
description: |-
Path from the source record root that produces
the value for this sublist column. Paths are
always anchored at the source record root —
never relative to a pre-iterated row. The
platform stores `null` here on fields driven by
`hardCodedValue` instead of an extract path.
x-celigo-ai-guidance:
- |-
Two extract patterns, often mixed in the same
sublist:
1. `parent_array[*].field_name` (bracket-star)
— iterates the source array and produces one
sublist line per element. Use for values that
vary per line (line-item sku, quantity, price,
etc.):
line_items[*].netsuite_item_id → item
line_items[*].quantity → quantity
line_items[*].price → rate
2. Bare or dotted path (`field_name` or
`parent.field`) — pulls a source-root scalar
and repeats it on every generated sublist
line. Use for values that don't vary per line
(e.g. a shared shipping carrier, order-level
metadata that decorates every package):
Tracking[*].TrackingNumber → packagetrackingnumber
ShippingCarrier → packagedescr
Here every package line gets the same
`ShippingCarrier` value; iteration is
driven by `Tracking[*]`.
- |-
The mapper drives iteration from the `[*]` segments among the child extracts;
scalar extracts tag along.
If no child uses `[*]` and none set `extract` at all (every child is a `hardCodedValue`),
the sublist renders a single static line.
- |-
A row that resolves its value through `lookupName`
(typically `extract: ""`) is driven by the sibling
rows' `[*]` extracts, and the lookup's `expression`
addresses the current line with the same
`<array>[*].<field>` path — see
`netsuite_da.lookups[].expression`.
generate:
type: string
description: NetSuite sublist field ID to write to.
hardCodedValue:
type: string
description: Static value for this sublist field.
lookupName:
type: string
description: Reference to a lookup by name.
dataType:
type: string
description: Data type hint for the field value.
internalId:
type: boolean
description: When true, the value is a NetSuite internal ID reference.
isKey:
type: boolean
description: |-
When true, this field is a key for matching the record's existing sublist lines: a
mapped line updates the one existing line whose key fields hold the same values. `lists`
says which records match lines, the keys used when no field sets `isKey`, and what a
line that matches no line, or several, does.
immutable:
type: boolean
description: When true, this field is only set on record creation, not on updates.
discardIfEmpty:
type: boolean
description: When true, skips this field mapping if the extracted value is empty.
extractDateFormat:
type: string
description: Date format of the extracted value.
extractDateTimezone:
type: [string, 'null']
description: Timezone of the extracted date value. Stored as `null` when not configured.
subRecordMapping:
type: object
description: |-
Nested Mapper 1.0 mapping for a NetSuite
subrecord embedded in this sublist-field. A
subrecord is a child record that only exists
inside a parent (e.g. the Inventory Detail on
a serialized-item line of a sales order);
NetSuite reaches it through the parent.
**Recursive structure.** The nested
`mapping` object has the same
`{fields, lists}` shape as the top-level
`netsuite_da.mapping`. Recursion is allowed
but rare.
x-celigo-ai-guidance:
- |-
**Do not over-emit.** Per the Celigo help center:
"Unconfigured subrecord mappings can cause your NetSuite import to fail.
Only include subrecord mappings on imports that require subrecord data." Emit
this block ONLY when the destination record type genuinely requires the
subrecord.
- |-
**Common `recordType` values** (most
common first):
* `inventorydetail` — by far the most
common. Serial / lot / bin assignments on
sales-order, item-fulfillment, purchase-
order, and inventory-adjustment line items.
Nested `mapping` typically has empty
`fields` and a single
`inventoryassignment` sublist.
* `componentinventorydetail` — same concept
on work-order / assembly-build component
lines.
* `itemvendorprice` — vendor-specific price
tiers on item-vendor relationships.
* `landedcost` — landed-cost allocations.
* `itempricing` — quantity-based price tiers
on contracts. Nested `mapping` typically
carries a `discount` sublist with
`fromquantity` and `rate` rows.
* `countdetail` — per-line detail on
physical-inventory counts.
- |-
**Where this block lives.** The enclosing sublist-field's `generate` name is a
synthetic trigger (typically `celigo_*`) whose purpose is to carry this
`subRecordMapping` block — it is NOT a real NetSuite column.
The trigger → `recordType` correspondence is mostly `celigo_<recordType>`
but NOT a strict rule:
`celigo_inventorydetail` covers BOTH `inventorydetail` and
`componentinventorydetail` depending on the parent record.
Match trigger names on existing resources rather than synthesizing new ones.
properties:
recordType:
type: string
description: |
NetSuite's internal record-type id of the
SUBRECORD (not the parent). E.g.
`inventorydetail`, `itempricing`,
`componentinventorydetail`.
jsonPath:
type: string
description: |
JSONPath (relative to the current sublist
line) to the rows that populate the
subrecord. Typically `$` when the
subrecord data is carried on the same
line item (iteration over the sublist
has already selected the line).
Defaults to `$`.
mapping:
type: object
description: |
Nested `{fields, lists}` mapping for the
subrecord. Identical shape to the
top-level `netsuite_da.mapping`.
Subrecords often have empty `fields` and
only a sublist (e.g. `inventorydetail` →
`inventoryassignment`).
lookups:
type: array
description: |
Subrecord-scoped lookups. Same shape as
the top-level `netsuite_da.lookups`.
Fields inside this
`subRecordMapping.mapping` reference
them via `lookupName`. Often `[]`.
referenceFieldId:
type: string
description: |
Secondary reference-field identifier used
on a small fraction of subrecord blocks.
Exact semantics are not publicly
documented by Celigo — match
the value on existing resources when
present; do NOT synthesize one. Leave
unset unless a template clearly requires
it.
conditional:
type: object
description: |-
Conditional logic for when to apply this sublist field mapping
("Only perform mapping when" in the UI). Lookup-based conditions
evaluate the lookup named by `lookupName`, which must be defined
on this import.
x-celigo-ai-guidance:
- |-
Same dangling-reference failure as body-field conditions: a
`lookupName` that matches no lookup defined on the import breaks
the run — on sublist rows this has been observed to stall the
flow job (it never completes) instead of producing the
`Cannot set properties of undefined (setting '_hasStarNotation')`
error record. Cross-reference every row's `conditional.lookupName`
against the import's defined lookups.
- |-
The operation contract differs from body-field conditions:
`when: record_updated` on an `add`-only operation is rejected
per-record on body rows (`The mapping condition:"record_updated"
not supported for the operation`) but was NOT rejected on sublist
rows in live verification. Do not rely on that tolerance — use
`when: record_updated` only on operations that can update
records (`addupdate` / `update`).
- |-
The condition is one of the fixed `when` values — the same
closed set as body rows, with no predicate over a source
field; `extract_not_empty` skips the row on the lines whose
extract is empty and leaves that line's field unset. Two
sublist rows with the same `generate` are not a fallback
pair either, and the order rule is not the body rows': among
extract rows the last row decides each line alone (a skipped
last row leaves the line's field unset even when an earlier
row produced a value), while a hard-coded row's value lands
on every line whatever its position — a later conditional
extract row does not override it.
properties:
lookupName:
type: string
description: |-
Lookup to evaluate for the condition. Must exactly match the
`name` of a lookup defined on this import.
when:
type: string
description: Condition that controls when this field mapping is applied.
enum:
- record_created
- record_updated
- extract_not_empty
- lookup_not_empty
- lookup_empty
- ignore_if_set
x-enumDescriptions:
record_created: Apply this mapping only when a new record is being created.
record_updated: Apply this mapping only when an existing record is being updated.
extract_not_empty: Apply this mapping only when the extracted source value is non-empty.
lookup_not_empty: Apply this mapping only when the lookup returns a value.
lookup_empty: Apply this mapping only when the lookup returns no value.
ignore_if_set: Skip this mapping if the target field already has a value.
description: Field mappings for each column in the sublist.
description: |
Sublist (line-item) mappings. Each entry maps source data to a NetSuite sublist.
Each mapped line updates the one existing line whose key fields hold the same values: its
`isKey` fields, or, with none, the sublist's default key. The existing lines are the stored
record's on an update, and on a transform (`Created From`: the body field
`celigo_nlobjTransformId` holds the source transaction's internal id) the lines the transform
fills in. Default keys:
- the item (`item`, else `itemkey` or `itemname`) on item fulfillments, item receipts,
invoices, cash sales, cash refunds, return authorizations, bin worksheets, inbound shipment
receipts and inventory counts, and on an assembly build's `component` sublist;
- the applied transaction (`refnum`) on the `apply`, `credit` and `deposit` sublists of
customer and vendor payments, credit memos, vendor credits and customer refunds;
- `docnumber` on a deposit's `payment` sublist;
- `vendor` and `language` on an item record's `itemvendor` and `translations` sublists.
A sublist with neither, such as a sales order's or purchase order's `item`, matches nothing, so
an update adds every mapped line after the existing ones. Two matching lines fail the record
with `MULTIPLE_LINES_WITH_SAME_KEYS` ("Multiple lines found with keys item=<value> ..."). A
line that matches none is added as a new line, which an item fulfillment's `item` sublist
refuses with `SSS_INVALID_SUBLIST_OPERATION`: its lines are its order's.
x-celigo-ai-guidance:
- |-
To make an update replace a sublist's lines, also map the body field
`celigo_replaceAllLines_<sublist id>` (for example `celigo_replaceAllLines_item`) to `true`:
the existing lines no mapped line matched are removed, then the mapped lines are written. Where
lines can't be removed it clears them instead: `itemreceive` false on an item fulfillment or
item receipt, `apply` false on an `apply` sublist. On a sublist with no `isKey` field and no
default key every existing line is unmatched, so all are replaced; with a default key, the lines
it matches stay.
- |-
Key an update on a value each line holds once: a line id the record's create wrote to a line
field, or, on a fulfillment, the order line's id mapped to `orderline`. Keying on the item fails
with `MULTIPLE_LINES_WITH_SAME_KEYS` when an order holds the same item on two lines, and records
that already hold repeated lines fail the same way once a key is set.
x-celigo-agent:
- mapping
lookups:
type: array
items:
type: object
properties:
name:
type: string
description: |-
Unique identifier for this lookup. Mapping fields reference it
via `lookupName` and Handlebars expressions reference it via
`{{lookup "name" value}}`. Must be unique within this
resource's `lookups` array.
x-celigo-ai-guidance:
- |-
Mapping rows also reference lookups by this exact name in
`conditional.lookupName` ("Only perform mapping when"). When renaming
or removing a lookup, update every referencing row: a stale field-level
`lookupName` fails records with
`Unable to find lookup in mapping configs - <name>`; a stale
`conditional.lookupName` on a body row fails them with the unnamed
`Cannot set properties of undefined (setting '_hasStarNotation')` error
(see `mapping.fields[].conditional`), and on a sublist row has been
observed to stall the flow job with no error record
(see `mapping.lists[].fields[].conditional`).
examples: ["lookupCustomerByEmail"]
recordType:
type: string
description: |-
NetSuite record type to search. Use the internal record-type
id (lowercase, underscored), e.g. `customer`, `item`,
`salesorder`, `customlist_xxx`.
examples: ["customer"]
searchField:
type: string
description: |-
NetSuite field on `recordType` to match against. Use the
internal field id — e.g. `email`, `externalid`,
`internalid`, `entityid`, `custitem_xxx`.
examples: ["email"]
resultField:
type: string
description: |-
NetSuite field whose value is returned when a record matches.
Typically `internalid` when the lookup resolves to a record
reference, or any queryable scalar field.
examples: ["internalid"]
expression:
type: string
x-celigo-agent:
- handlebars
description: |-
Optional advanced-search expression (NetSuite saved-search
DSL) for multi-criterion lookups. When present, supersedes
`searchField` / `operator` — those fields are ignored.
Handlebars inside the expression is rendered against the
whole source record before the search runs; a lookup on a
sublist row reaches the current line through
`<array>[*].<field>` (for example `line_items[*].sku`).
x-celigo-ai-guidance:
- |-
## Evaluation context — the whole source record, no `record.` prefix
The expression's Handlebars renders against the incoming
source record as the context root (the record is NOT
wrapped under `record`), then the rendered JSON array is
sent to NetSuite as the search criteria. Body-level fields
are addressed bare: `{{{id}}}`, `{{{email}}}`.
- |-
## Sublist-row lookups address the line with `<array>[*].<field>`
A lookup referenced from a sublist mapping row
(`mapping.lists[].fields[].lookupName`) still receives the
whole record; the current line's fields are reached with the
same bracket-star path the sublist extracts use —
`{{{line_items[*].sku}}}` resolves the line's item (one line
per sublist row). A bare line field (`{{sku}}`,
`{{#contains sku "_"}}`) is undefined at the record root: a
helper that receives it fails every record with
`INVALID_HANDLE_BAR_EXPRESSION` — `Could not compile handle
bar "<expression>" because "Cannot call method "indexOf" of
undefined"` — before anything reaches NetSuite. When an
audit diff shows `<array>[*].` removed from the expression
right before such errors began, that edit is the cause; do
not look for a line missing the field.
- |-
## Rendered unmatched
A rendered value NetSuite does not match fails the record
with `VALUE_LOOKUP_FAILED` — `Could not find a match for
"<rendered criteria>" for the record type "<recordType>"`
— quoting the criteria exactly as rendered, so the message
shows what the expression produced for that line. With
`allowFailures: true` the `default` value is used instead
and no error is raised. Give every branch an output.
- |-
## An empty substitution searches `isempty`
When the source path a criterion reads is null or absent
(`["custentity_celigo_etail_cust_id","is","{{{customer.id}}}"]`
on an order with no customer), the engine does not fail on
the missing input: it searches the field with an `isempty`
criterion, so the lookup now matches every target record
whose field is BLANK. Two or more such records fail the
record with `VALUE_LOOKUP_FAILED` — `More than one match
for "[<field>,"isempty"]" for the record type "<recordType>"`
(the message quotes the search field, never the empty
source path); none fail it as `Could not find a match`;
exactly one silently resolves to that record. Keep an empty
key out of the search inside the expression — branch on
the key: `{{#if customer.id}}[["custentity_celigo_etail_cust_id",
"is","{{{customer.id}}}"]]{{else}}[["internalid","anyof",
"<fallback internal id>"]]{{/if}}` routes records without a
key to a fallback record while records with a key keep
fail-on-miss; or render a value that can never match in the
else branch and set `allowFailures: true` with `default`.
A `searchField` / `operator` lookup (no `expression`) never
reaches this state: an empty extract fails the record with
`STATIC_MAPPING_ERROR` — `Exported value is empty/undefined
for the field: $.<path>` — and resolves to `default` when
`allowFailures` is true.
- |-
## The mapping preview cannot evaluate `[*]` inside a block-helper argument
`PUT /v1/netsuiteDA/previewImportMappingFields` (the mapping
editor's Preview) evaluates lookups for real — resolved
internal ids appear in `nlobjSublistIds`, misses and
compile errors in `data.mappingErrors[]` — and handles
`[*]` in plain substitutions and inside a branch's output.
It does NOT apply the per-line evaluation to `[*]` used as a
block-helper ARGUMENT (`{{#if line_items[*].x}}`,
`{{#contains line_items[*].sku "_"}}`): the sublist comes
back EMPTY with no error, whatever the expression would do
in a run. An empty preview is therefore no evidence about
such an expression — never report it as "renders nothing"
or as invalid on that basis; validate it with a run.
`<array>.@index.<field>` is likewise not evaluable in the
preview (the record is dropped with no error).
operator:
type: string
description: |-
Comparison operator for the `searchField` match. Common
values: `is`, `contains`, `startswith`, `anyof`,
`equalto`, `greaterthan`.
includeInactive:
type: boolean
description: When true, includes inactive records in lookup results.
x-celigo-ai-guidance:
- |-
When true, the search includes inactive NetSuite records.
Default false — inactive records are excluded.
useDefaultOnMultipleMatches:
type: boolean
description: |-
When true and `allowFailures` is also true, a search that
matches more than one record resolves to `default` instead
of failing the record. Has no effect while `allowFailures`
is false — an ambiguous match then fails the record whatever
this flag says.
x-celigo-ai-guidance:
- |-
Set it together with `allowFailures: true` whenever the
search value can be empty or duplicated: with
`allowFailures: true` alone an ambiguous match resolves to
ONE of the matching records with no error, and with this
flag alone the record still fails `More than one match`.
Every ambiguous match then becomes `default`, a genuinely
duplicated key included — when that must still fail, keep
`allowFailures` false and handle the empty key in the
`expression` instead (see `expression`).
allowFailures:
type: [boolean, 'null']
description: |-
When true, a lookup miss (no matching record and no
`default`) resolves to `null` and the import continues, and
a search matching more than one record resolves to one of
them unless `useDefaultOnMultipleMatches` routes it to
`default`. When false (default), a miss and an ambiguous
match both halt the record. May be stored as null (treated
as unset).
map:
type: [object, 'null']
description: |-
Optional static key→value object evaluated BEFORE the
NetSuite search. When the incoming value matches a key, the
mapped value is returned without hitting NetSuite. Null on
dynamic (search-based) lookups with no static map.
default:
type: [string, 'null']
description: |-
Value returned when the search matches no records. When
omitted and `allowFailures` is false, an unmatched lookup
halts the record. May be stored as null (no default).
description: |-
Lookup definitions that resolve reference values from NetSuite.
Referenced by name from field mappings via `lookupName` and from
Handlebars expressions via `{{lookup "name" value}}`.
x-celigo-ai-guidance:
- |-
## Mirror invariant
The Celigo platform persists NetSuite Distributed Import lookups in
TWO locations: `netsuite_da.lookups` and top-level `lookups`.
Both arrays must hold byte-for-byte identical content, so write
both on every save.
- |-
## One context for body and sublist lookups
Every lookup's `expression` renders against the whole source
record. A lookup referenced from a sublist row
(`mapping.lists[].fields[].lookupName`) is evaluated once per
line and addresses the current line with `<array>[*].<field>`
— never a bare line field. See `expression` for the failure
modes and the preview caveat.
x-celigo-agent:
- lookup
rawOverride:
type: object
description: |
Raw override object for advanced use cases. When useRawOverride is true,
this object is sent directly to the NetSuite API, bypassing normal mapping.
useRawOverride:
type: boolean
description: |
When true, uses rawOverride instead of the normal mapping configuration.
isMigrated:
type: boolean
description: When true, this import was migrated from a legacy format. Set by the system.
retryUpdateAsAdd:
type: boolean
description: |
When true, if an update fails because the record doesn't exist, automatically
retry as an add operation. Useful for initial syncs where records may not exist yet.
customFieldMetadata:
type: object
description: |
Metadata about custom fields on the target record type.
Populated by the system from NetSuite metadata — do not set manually.
file:
type: object
description: |
File cabinet configuration for file-based imports into NetSuite.
properties:
name:
type: string
description: Filename for the file in NetSuite File Cabinet.
fileType:
type: string
description: |
NetSuite file type (e.g., "PDF", "CSV", "PLAINTEXT", "EXCEL", "XML").
folder:
type: string
description: Folder path or name in the NetSuite File Cabinet.
folderInternalId:
type: string
description: Internal ID of the target folder in NetSuite File Cabinet.
internalId:
type: string
description: Internal ID of an existing file to update.
backupFolderInternalId:
type: string
description: Internal ID of a backup folder for file versioning.
isFileProvider:
type: boolean
description: |
Whether this import handles files in the NetSuite File Cabinet.
preferences:
type: object
properties:
ignoreReadOnlyFields:
type: boolean
description: When true, silently skip read-only fields instead of raising errors.
warningAsError:
type: boolean
description: When true, treat NetSuite warnings as errors that stop the import.
skipCustomMetadataRequests:
type: boolean
description: When true, skip fetching custom field metadata to improve performance.
description: |
Import behavior preferences that control how NetSuite handles the import operation.
recordTypeId:
type: string
description: |
The internal record type ID. Used for custom record types in NetSuite where
the numeric ID is needed in addition to the recordType string.
# Requireds apply only to real configs, not server-written bookkeeping stubs whose
# keys are all in {lookups, mapping, missingOrCorruptedDAConfig} — e.g.
# {"lookups":[],"mapping":{"fields":[],"lists":[]}} or {"missingOrCorruptedDAConfig":true},
# written onto imports of every adaptorType. Stack-style conditional (no allOf).
if:
not:
propertyNames:
enum: [lookups, mapping, missingOrCorruptedDAConfig]
then:
required:
- operation
- recordType
if:
properties:
operation:
enum: [update, addupdate, delete]
required:
- operation
then:
required:
- internalIdLookup
references/schemas/rdbms.yml
Rdbms:
type: object
description: |-
Configuration for RDBMS imports into SQL Server, MySQL, PostgreSQL, Snowflake, Oracle, MariaDB, and other relational databases.
queryType determines which companion field is required: per_record, per_page, and first_page use the query field; bulk_insert uses the bulkInsert object; bulk_load uses the bulkLoad object — never set query and bulkInsert together.
(The server itself stores an empty `query: []` next to bulk_insert/bulk_load configs and may leave an empty `bulkInsert: {}` stub next to others — empty companions are normal in responses; only a populated conflicting companion is invalid.)
The query field is an array of plain SQL strings, not a single string and not an array of objects.
x-celigo-ai-guidance:
- |-
## CRITICAL: query IS AN ARRAY OF STRINGS
The query field must be an array of plain strings, NOT a single string and NOT an array of objects.
Correct: ["MERGE INTO target USING ..."]
Wrong: "MERGE INTO target ..."
Wrong: [{"query": "MERGE INTO target ..."}]
# queryType is required on real configs only (see the conditional at the end of this
# schema): the server writes stub `rdbms` objects ({} / {lookups:[]}) onto imports of
# other adaptorTypes as bookkeeping.
properties:
lookups:
# Live: array of lookup configs (matches common Lookups), not a string.
$ref: '../common-schemas/lookups.yml#/Lookups'
description: |
Lookup definitions executed against the RDBMS connection. Each
entry runs a `SELECT` and extracts a value used by field mappings
(referenced via `lookupName`) or Handlebars expressions
(referenced via `{{lookup "name" value}}`).
**Mirror invariant**
The Celigo platform persists lookups in TWO locations on an RDBMS
import: top-level `lookups` and `rdbms.lookups`. Both arrays
must hold byte-for-byte identical content, so write both on every
save.
items:
type: object
properties:
name:
type: string
description: |
Unique identifier for this lookup. Mapping fields reference it
via `lookupName` and Handlebars expressions reference it via
`{{lookup "name" value}}`. Must be unique within this
resource's `lookups` array.
query:
type: string
x-celigo-agent:
- sql
- handlebars
description: |
SQL `SELECT` statement that resolves the lookup value.
Typically returns a single row with a single column; use
`extract` to pick the column when the query returns multiple.
Inject values from the incoming record with Handlebars and the
`record.` prefix — double braces for strings, which the platform
quotes, escaping their apostrophes, and triple braces for numbers:
`SELECT id FROM users WHERE email = {{record.email}}`. On Snowflake
the quoting leaves backslashes, which Snowflake reads as escape
sequences (see `query`).
extract:
type: string
description: |
JSONPath-style expression that selects the value from the query
result row (e.g. `id`, `details.price`). Omit when the
query returns a single scalar — the first column is used
automatically.
_lookupCacheId:
type: string
format: objectId
x-celigo-refModel: lookupcaches
description: |
Optional reference to a LookupCache resource. When set, the
lookup reads from the cache instead of issuing a live SQL
query — useful for stable reference data that should be
pre-loaded.
map:
type: object
description: |
Optional static key→value object evaluated BEFORE the SQL query.
When the incoming value matches a key, the mapped value is
returned without querying the database. Useful for small
controlled enumerations.
default:
type: string
description: |
Value returned when the query matches no rows. When omitted and
`allowFailures` is false, an unmatched lookup halts the
record.
allowFailures:
type: boolean
description: |
When true, a lookup miss (no matching row and no `default`)
resolves to `null` and the import continues. When false
(default), a miss halts the record.
type: array
x-celigo-agent:
- lookup
query:
type: array
items:
type: string
x-celigo-agent:
- sql
description: |-
SQL statements to execute; required when queryType is `["per_record"]`, `["per_page"]`, or `["first_page"]`, and typically contains a single statement.
Each element is a complete SQL statement as a plain string — never a single bare string and never an array of objects (objects cause a Cast error).
Inject values from the incoming record with Handlebars using the `record.` prefix: double braces for strings and dates (`{{record.name}}`; the platform quotes the value and escapes its apostrophes) and triple braces for numbers (`{{{record.quantity}}}`).
Rendering is strict: a reference the render model cannot resolve (a bare `{{name}}` without prefix, or `record.*` when the arriving record is a grouped array) fails the step with `cannot_evaluate_handlebars` — it does not render an empty value.
`record` and `rows` are aliases chosen by the arriving data's shape: an object record binds `record` (`rows` undefined); a grouped array-record (e.g. upstream `groupByFields`) binds `rows` (`record` undefined) — address grouped rows as `{{rows.0.field}}` or iterate `{{#each rows}}`.
Double braces (`{{record.field}}`) output a string value wrapped in single quotes with embedded quotes doubled, while triple braces output it raw; block helpers such as `{{#each}}` and `{{#if}}` use double braces as normal. In Snowflake queries a JSON number or boolean renders bare under double braces too (observed in export queries).
On Snowflake, the quoting leaves backslashes, which Snowflake reads as escape sequences: a double-brace value holding a backslash arrives changed, and `PARSE_JSON({{jsonSerialize …}})` fails on a double quote or line break in the data (observed in export queries). Pass such values in a dollar-quoted constant with triple braces (`$${{{record.field}}}$$`), which Snowflake reads as is, but only when the value can't contain `$$`: a `$$` in it ends the constant, and the rest is read as SQL.
x-celigo-ai-guidance:
- |-
Use Handlebars to inject values from incoming records.
RDBMS queries REQUIRE the `record.` prefix on every field substitution.
- |-
## FORMAT — ARRAY OF STRINGS
This field is an ARRAY OF STRINGS, not a single string and not an array of objects.
- CORRECT: ["INSERT INTO users (name, email) VALUES ({{record.name}}, {{record.email}})"]
- WRONG: "INSERT INTO users ..." (not an array)
- WRONG: [{"query": "INSERT INTO users ..."}] (objects are invalid — causes Cast error)
- WRONG: ["INSERT INTO users (name) VALUES ({{name}})"] (missing record. prefix — rendering is strict, so the step fails with cannot_evaluate_handlebars)
- |-
## RDBMS HANDLEBARS SYNTAX
- **MUST use `record.` prefix** on per-record field references:
`{{record.fieldName}}` or `{{{record.fieldName}}}`, NOT
`{{fieldName}}` — an unprefixed reference is undefined in the
render model and fails the step (strict rendering).
- **Grouped input binds `rows`, not `record`.** When the arriving
record is a grouped array (upstream `groupByFields` or grouped
file rows), `record` is undefined and the step fails; reference
`{{rows.0.fieldName}}` or iterate `{{#each rows}}` instead. The
two names are shape-dependent aliases over the same data.
- **Double-brace `{{record.field}}` vs. triple-brace `{{{record.field}}}`
is a QUOTING CHOICE, not a syntax rule.** In RDBMS context the
platform's Handlebars renderer treats double-brace specially: it
wraps a string value as a SQL string literal (single-quoted, with
its apostrophes escaped), and in Snowflake export queries a number
or boolean rendered bare, whether a JSON value or a numeric
helper's result (`{{add 2 3}}`). Triple-brace emits the raw value
with no quoting. In the same queries the wrapping was also seen on
string helper output (`dateFormat`, `jsonSerialize`) and on
`{{this}}` inside `{{#each}}`.
- `{{record.name}}` → `'Alice'` (auto-single-quoted, embedded `'` doubled to `''`)
- `{{{record.name}}}` → `Alice` (raw — never wrap it in single quotes of your own: the first apostrophe in the data ends the literal)
- `{{record.qty}}` → `'5'` when `qty` arrives as the string `"5"`; a JSON number rendered bare in Snowflake export queries
- `{{{record.qty}}}` → `5` (numeric literal either way — what you usually want for INT columns)
Pick by what the SQL needs at that slot:
| Slot type | Use |
|------------------------------------------|---------------------------|
| String column value | `{{record.field}}` |
| Numeric column value (INT, DECIMAL, ...) | `{{{record.field}}}` |
| Stored procedure raw arg (already typed) | `{{{record.field}}}` |
| Identifier (table/column name) | `{{{record.field}}}` |
| JSON literal, Snowflake (`PARSE_JSON`) | `PARSE_JSON($${{{jsonSerialize record.subobject}}}$$)` (a dollar-quoted constant, read as is; only for data that can't contain `$$`, which ends it) |
| JSON literal, PostgreSQL (`JSONB`) | `{{jsonSerialize record.subobject}}` (renders the JSON as a quoted string; not yet verified on PostgreSQL) |
- **Snowflake reads backslashes as escapes.** The double-brace
quoting doubles embedded apostrophes but leaves backslashes, and
Snowflake interprets backslash sequences inside a single-quoted
literal: a value holding `C:\temp` arrives with a tab in it, with
no error, and `PARSE_JSON({{jsonSerialize …}})` fails with
`Error parsing JSON` whenever a string in the object holds a double
quote or a line break, because `jsonSerialize` escapes those with
backslashes (observed in export queries). For values that can hold
these characters, use a dollar-quoted constant with triple braces
(`$${{{record.field}}}$$`), which Snowflake reads as is, but only
when the value can't contain `$$`: a `$$` in it ends the constant,
and the rest is read as SQL.
- **Block helpers** (`{{#each}}`, `{{#if}}`, `{{/each}}`,
`{{#unless}}`) use double braces as normal — they emit no value.
- **Nested fields**: `{{record.properties.email}}` (auto-quoted)
or `{{{record.properties.email}}}` (raw).
- |-
## EXAMPLES
- INSERT (string columns, auto-quoted): `["INSERT INTO customers (name, email) VALUES ({{record.name}}, {{record.email}})"]`
- INSERT (mixed numeric + string): `["INSERT INTO orders (qty, sku) VALUES ({{{record.qty}}}, {{record.sku}})"]`
- UPDATE (numeric SET, string WHERE): `["UPDATE inventory SET qty = {{{record.qty}}} WHERE sku = {{record.sku}}"]`
- UPSERT (MySQL): `["INSERT INTO users (id, name) VALUES ({{{record.id}}}, {{record.name}}) ON DUPLICATE KEY UPDATE name = {{record.name}}"]`
- MERGE (Snowflake): `["MERGE INTO target USING (SELECT {{record.email}} AS email) AS src ON target.email = src.email WHEN MATCHED THEN UPDATE SET name = {{record.name}}"]`
- Stored procedure (raw args): `["call netsuite.transformation_data_import({{record.record_type}}, 20)"]`
- JSON literal column (Snowflake VARIANT; the data must not contain `$$`): `["INSERT INTO events (payload) SELECT PARSE_JSON($${{{jsonSerialize record.financialEvents}}}$$)"]`
- Per-page batch (different Handlebars context — see `queryType: per_page` below): `["{{#each batch_of_records}}UPDATE staging SET done=1 WHERE id={{{this.record.id}}};{{/each}}"]`
examples:
- ["INSERT INTO orders (id, total) VALUES ({{{record.id}}}, {{{record.total}}})"]
- ["{{#each batch_of_records}}UPDATE customers SET name={{record.name}} WHERE id={{{record.id}}};{{/each}}"]
queryType:
type: [array, 'null']
items:
type: string
enum:
- INSERT
- UPDATE
- bulk_insert
- per_record
- per_page
- first_page
- bulk_load
x-enumDescriptions:
INSERT: Legacy value for insert operations; use bulk_insert or per_record instead.
UPDATE: Legacy value for update operations; use per_page or per_record instead.
bulk_insert: Batch INSERT via multi-row VALUES clause for efficient bulk data loading.
per_record: Execute one SQL statement per record, giving full control over individual record SQL.
per_page: Execute one SQL statement per page (batch) of records.
first_page: Execute the SQL statement once for the first page only, typically for setup or cleanup scripts.
bulk_load: Stage data as a file and load via database-native COPY/bulk mechanism for maximum throughput.
x-celigo-ui-override: >-
Required by the RDBMS import form (rdbms.queryType is required:true). The form defaults a
new import to bulk_load wherever it offers that option — Snowflake always; Google BigQuery,
Microsoft SQL Server (cloud connections only), Azure Synapse and Amazon Redshift when the
account's enableSizeBasedAggregation feature flag is on for that database — and to
bulk_insert otherwise. Encoded to mirror the form so builders produce connectable
configurations.
description: |-
Execution strategy for the SQL operation, which determines the required companion field: per_record, per_page, and first_page require `query`; bulk_insert requires `bulkInsert`; bulk_load requires `bulkLoad`.
Prefer the highest-performance type the operation supports: bulk_load (Snowflake, Google BigQuery, Microsoft SQL Server on a cloud connection, Azure Synapse and Amazon Redshift — handles INSERT, upsert via `bulkLoad.primaryKeys`, and custom merge or ignore-existing logic via `bulkLoad.overrideMergeQuery`), then bulk_insert for pure INSERTs, then per_page, then per_record only when the logic cannot be expressed as a batch operation.
For UPDATE, UPSERT, or ignore-existing logic on databases without bulk_load support, use per_page or per_record with `query` — bulk_insert has no duplicate-checking logic. Prefer per_page when the statement can loop `batch_of_records`, and use per_record when the SQL can't be written as one batch statement.
Set a single value; do not combine types or use the legacy INSERT and UPDATE values.
per_page statements render against the whole batch (`batch_of_records`) rather than a single record — loop with `{{#each batch_of_records}}...{{record.fieldName}}...{{/each}}`, with the same quoting as `query` (double braces for strings, triple for numbers).
`bulk_load` — Stages the records as files and loads them with the database's native COPY / bulk mechanism for maximum throughput; supports INSERT, upsert via bulkLoad.primaryKeys, and custom merge or ignore-existing logic via bulkLoad.overrideMergeQuery; available on Snowflake, Google BigQuery, Microsoft SQL Server (cloud connections), Azure Synapse and Amazon Redshift (NetSuite Analytics Warehouse has it through the JDBC import). The import form calls it "Use optimized bulk load (recommended for larger imports)" and, beside it, calls bulk_insert "Use batch insert SQL query".
`bulk_insert` — Batch INSERT via multi-row VALUES clause for efficient bulk data loading; has no upsert or ignore-existing logic and is available for all RDBMS types.
`per_page` — Executes one SQL statement per page (batch) of records; available for all RDBMS types.
`per_record` — Executes one SQL statement per record, giving full control over individual record SQL; available for all RDBMS types.
x-celigo-ai-guidance:
- '**CRITICAL: This field DETERMINES whether `query` or `bulkInsert` is required.**'
- |-
## DECISION TREE (Follow in order):
**STEP 0: Is this an update of an existing import?** Keep its query type unless the user asks to change it — an import set to bulk_insert stays bulk_insert when the request is about something else (the query type is additive and opt-in; nothing migrates). The steps below are for a NEW import or an explicit request to change the type.
**STEP 1: Does the database support bulk_load?**
- Snowflake, Google BigQuery, Microsoft SQL Server (a cloud connection — not an on-premise agent connection), Azure Synapse and Amazon Redshift do (NetSuite Analytics Warehouse does through the JDBC import):
→ **USE `["bulk_load"]`** with:
- `bulkLoad.tableName` (required — omitting it fails the save with 422 `missing_required_field` "rdbms imports require a tableName.")
- `bulkLoad.primaryKeys` for upsert/merge (auto-generates MERGE)
- `bulkLoad.overrideMergeQuery: true` for custom SQL (ignore-existing, conditional updates, multi-table ops). The override SQL goes in `query` and references `{{preMergeTemporaryTable}}` (also written `{{import.rdbms.bulkLoad.preMergeTemporaryTable}}`) for the staging table and `{{tableName}}` for the destination.
- No `primaryKeys` for pure INSERT (auto-generates INSERT)
- No `bulkInsert` beside it (the server drops a `bulkInsert` sent with bulk_load)
- **Amazon Redshift needs no AWS setup for bulk_load**: Celigo stages the files in Celigo-managed encrypted storage and Redshift loads them with COPY — no S3 bucket, IAM role, cluster role attachment or extra connection field. The save is accepted on either `redshift.authType` (accesskey and awsIam) and against a provisioned cluster or a Serverless workgroup alike. Never steer a Redshift import to bulk_insert for lack of setup.
- The import form shows the bulk_load option only when the account's `enableSizeBasedAggregation` feature flag is on for that database (BigQuery, SQL Server, Synapse, Redshift; Snowflake always shows it). The flag is the form's switch, not the API's — a Redshift bulk_load import saves on an account whose flag has no redshift entry (off, to the form) — so author it, and tell the user the option appears on their import form once the flag is on for the account.
**STEP 2: Database does NOT support bulk_load**
- **Pure INSERT**: → **USE `["bulk_insert"]`** (requires `bulkInsert` object)
- **UPDATE / UPSERT / custom matching logic**: → **USE `["per_page"]`** (requires `query` field). Only use `["per_record"]` if the logic truly cannot be expressed as a batch operation.
- |-
## DO NOT USE:
- `["INSERT"]` or `["UPDATE"]` as standalone values
- Multiple types combined
- |-
## LEGACY VALUES ARE WRITE-REJECTED:
Saving an import whose queryType contains INSERT (or the retired MERGE /
COPY / "BULK INSERT" values) fails with 422 `Import query types "BULK
INSERT", "INSERT", "MERGE", "COPY" are deprecated. Please change your
Import to use new query types "bulk_insert", "per_record", "per_page",
"first_page" instead`. Strings outside the enum entirely (e.g. lowercase
"insert") fail with 422 `is not a valid enum value for path
rdbms.queryType.0`.
- |-
## EXAMPLES:
### INSERT with ignoreExisting (MUST use per_record)
Prompt: "Import customers, ignore existing based on email"
```json
"queryType": ["per_record"],
"query": ["INSERT INTO customers (name, email) VALUES ({{record.name}}, {{record.email}})"]
```
### Pure INSERT (no duplicate checking)
Prompt: "Insert all records into users table"
```json
"queryType": ["bulk_insert"],
"bulkInsert": { ... }
```
### UPDATE Operation
Prompt: "Update inventory counts"
```json
"queryType": ["per_page"],
"query": ["{{#each batch_of_records}}UPDATE inventory SET qty = {{{record.qty}}} WHERE sku = {{record.sku}};{{/each}}"]
```
Use `"queryType": ["per_record"]` with `"query": ["UPDATE inventory SET qty = {{{record.qty}}} WHERE sku = {{record.sku}}"]` when the SQL can't be written as one batch statement.
### Run Once Per Page
Prompt: "Execute this stored procedure for each page of results"
```json
"queryType": ["per_page"]
```
**per_page uses a different Handlebars context than per_record.** The context is the entire batch (`batch_of_records`), not a single `record.` object. Loop through with `{{#each batch_of_records}}...{{record.fieldName}}...{{/each}}`, quoting as in per_record: double braces for strings, triple braces for numbers.
### Run Once at Start
Prompt: "Run this cleanup script before processing"
```json
"queryType": ["first_page"]
```
bulkInsert:
type: object
description: |-
Configuration for batch INSERTs via a multi-row VALUES clause; required when queryType is `["bulk_insert"]` and must not be set otherwise.
Only suited to pure INSERTs — for UPDATE or UPSERT, use queryType `["per_page"]` (a statement that loops `batch_of_records`) or `["per_record"]` with the `query` field instead, and for ignore-existing/skip-duplicates logic `["per_record"]`.
x-celigo-ai-guidance:
- |-
**CRITICAL: This object is ONLY used when `queryType` is `["bulk_insert"]`.**
✅ **SET THIS** when:
- `queryType` is `["bulk_insert"]`
- Pure INSERT operation with NO ignoreExisting/duplicate checking
❌ **DO NOT SET THIS** when:
- `queryType` is `["per_page"]` or `["per_record"]` (use the `query` field instead)
- Operation involves UPDATE, UPSERT, or ignoreExisting logic
If the prompt mentions "ignore existing", "skip duplicates", or "match on field",
use `queryType: ["per_record"]` with the `query` field instead of this object.
properties:
tableName:
type: string
description: |-
The name of the database table into which the bulk insert operation will be executed. This value must correspond to a valid, existing table within the target relational database management system (RDBMS). It serves as the primary destination for inserting multiple rows of data efficiently in a single operation. The table name can include schema or namespace qualifiers if supported by the database (e.g., "schemaName.tableName"), allowing precise targeting within complex database structures. Proper validation and sanitization of this value are essential to ensure the operation's success and to prevent SQL injection or other security vulnerabilities.
x-celigo-ai-guidance:
- |-
**Field behavior**
- Specifies the exact destination table for the bulk insert operation.
- Must correspond to an existing table in the database schema.
- Case sensitivity and naming conventions depend on the underlying RDBMS.
- Supports schema-qualified names where applicable.
- Used directly in the SQL `INSERT INTO` statement.
- Influences how data is mapped and inserted during the bulk operation.
- |-
**Dependency chain**
- Relies on the database connection and authentication configuration.
- Interacts with other bulkInsert parameters such as column mappings and data payload.
- May be influenced by transaction management, locking mechanisms, and database constraints during the insert.
- Dependent on the database's metadata for validation and existence checks.
- |-
## IMPLEMENTATION GUIDANCE:
- Verify the existence and accessibility of the table in the target database before execution.
- Ensure compliance with the RDBMS naming rules, including reserved keywords, allowed characters, and maximum length.
- Support and correctly handle schema-qualified table names, respecting database-specific syntax.
- Sanitize and validate input rigorously to prevent SQL injection and other security risks.
- Apply appropriate quoting or escaping mechanisms based on the RDBMS (e.g., backticks for MySQL, double quotes for PostgreSQL).
- Consider the impact of case sensitivity, especially when dealing with quoted identifiers.
- |-
## EXAMPLES:
- "users"
- "public.orders"
- "inventory.products"
- "sales_data_2024"
- "dbo.CustomerRecords"
- |-
## IMPORTANT NOTES:
- The target table must exist and have the appropriate schema and permissions to accept bulk inserts.
- Incorrect, misspelled, or non-existent table names will cause the operation to fail.
- Case sensitivity varies by database; for example, PostgreSQL treats quoted identifiers as case-sensitive.
- Avoid using unvalidated dynamic input to mitigate security vulnerabilities.
- Schema qualifiers should be used consistently to avoid ambiguity in multi-schema environments.
examples: ["public.orders", "dbo.Customers"]
batchSize:
type: string
description: |-
The number of records to be inserted into the database in a single batch during a bulk insert operation. This parameter is crucial for optimizing the performance and efficiency of bulk data loading by controlling how many records are grouped together before being sent to the database. Proper tuning of batchSize balances memory consumption, transaction overhead, and throughput, enabling the system to handle large volumes of data efficiently without overwhelming resources or causing timeouts. Adjusting batchSize directly impacts transaction size, network utilization, error handling granularity, and recovery strategies, making it essential to tailor this value based on the specific database capabilities, system resources, and workload characteristics.
x-celigo-ai-guidance:
- |-
**Field behavior**
- Specifies the exact count of records processed and committed in a single batch during bulk insert operations.
- Determines the frequency and size of database transactions, influencing overall throughput, latency, and system responsiveness.
- Larger batch sizes can improve throughput but may increase memory usage, transaction duration, and risk of timeouts or locks.
- Smaller batch sizes reduce memory footprint and transaction time but may increase the total number of transactions and associated overhead.
- Defines the scope of error handling, as failures typically affect only the current batch, allowing for partial retries or rollbacks.
- Controls the granularity of commit points, impacting rollback, recovery, and consistency strategies in case of failures.
- |-
## IMPLEMENTATION GUIDANCE:
- Choose a batch size that respects the database's transaction limits, available system memory, and network conditions.
- Conduct benchmarking and load testing with different batch sizes to identify the optimal balance for your environment.
- Monitor system performance continuously and adjust batchSize dynamically if supported, to adapt to varying workloads.
- Implement comprehensive error handling to manage partial batch failures, including retry logic or compensating transactions.
- Verify compatibility with database drivers, ORM frameworks, or middleware, which may impose constraints or optimizations on batch sizes.
- Consider the size, complexity, and serialization overhead of individual records, as larger or more complex records may necessitate smaller batches.
- Factor in network latency and bandwidth to optimize data transfer efficiency and reduce potential bottlenecks.
- |-
## EXAMPLES:
- 1000: Suitable for moderate bulk insert operations, balancing speed and resource consumption effectively.
- 50000: Ideal for high-throughput environments with ample memory and finely tuned database configurations.
- 100: Appropriate for systems with limited memory or where minimizing transaction size and duration is critical.
- 5000: A common default batch size providing a good compromise between performance and resource usage.
bulkLoad:
type: object
properties:
tableName:
type: string
description: |-
Target table for the bulk load.
Must reference an existing table accessible with the connection's credentials; schema-qualified names are supported.
x-celigo-ai-guidance:
- |-
The name of the target table within the relational database management system
(RDBMS) where the bulk load operation will insert,
update, or merge data.
This property specifies the exact destination table for the bulk data operation
and must correspond to a valid,
existing table in the database schema.
It can include schema qualifiers if supported by the database (e.g.,
schema.tableName),
and must adhere to the naming conventions and case sensitivity rules of the target RDBMS.
Proper specification of this property is critical to ensure data is loaded into
the correct location without errors or unintended data modification.
Accurate definition of the table name helps maintain data integrity and prevents
operational failures during bulk load processes.
- |-
## FIELD BEHAVIOR:
- Identifies the specific table in the database to receive the bulk-loaded data.
- Must reference an existing table accessible with the current database credentials.
- Supports schema-qualified names where applicable (e.g., schema.tableName).
- Used as the target for insert, update, or merge operations during bulk load.
- Case sensitivity and naming conventions depend on the underlying database system.
- Determines the scope and context of the bulk load operation within the database.
- Influences transactional behavior, locking, and concurrency controls during the operation.
- |-
## IMPLEMENTATION GUIDANCE:
- Verify the target table exists and the executing user has sufficient permissions for data modification.
- Ensure the table name respects the case sensitivity and naming rules of the RDBMS.
- Avoid reserved keywords or special characters unless properly escaped or quoted according to database requirements.
- Support fully qualified names including schema or database prefixes if required to avoid ambiguity.
- Validate compatibility between the table schema and the incoming data to prevent type mismatches or constraint violations.
- Consider transactional behavior, locking, and concurrency implications on the target table during bulk load.
- Test the bulk load operation in a development or staging environment to confirm correct table targeting.
- Implement error handling to catch and report issues related to invalid or inaccessible table names.
- |-
## EXAMPLES:
- "customers"
- "sales_data_2024"
- "public.orders"
- "inventory.products"
- "dbo.EmployeeRecords"
- |-
## IMPORTANT NOTES:
- Incorrect or misspelled table names will cause the bulk load operation to fail.
- The target table schema must be compatible with the incoming data structure to avoid errors.
- Bulk loading may overwrite, append, or merge data depending on the load mode; ensure this aligns with business requirements.
- Some databases require quoting or escaping of table names, especially if they
examples: ["public.orders", "ANALYTICS.STG_ORDERS"]
primaryKeys:
type: array
items:
type: string
description: |-
Columns that uniquely identify each record in the target table, used to match existing rows during the load.
Set for upsert behavior (a MERGE is auto-generated); omit (or empty) for a pure INSERT.
Composite keys are supported — list every key column, and names must exactly match the target schema.
x-celigo-ai-guidance:
- |-
primaryKeys specifies the list of column names that uniquely identify each
record in the target relational database table during a bulk load operation.
These keys are essential for maintaining data integrity by enforcing uniqueness
constraints and enabling precise identification of rows for operations such as
inserts,
updates, upserts, or conflict resolution.
Properly defining primaryKeys ensures that duplicate records are detected and
handled appropriately,
preventing data inconsistencies and supporting efficient data merging processes.
This property supports both single-column and composite primary keys,
requiring the exact column names as defined in the target schema,
and plays a critical role in guiding the bulk load mechanism to correctly match
and manipulate records based on their unique identifiers.
- |-
## FIELD BEHAVIOR:
- Defines one or more columns that collectively serve as the unique identifier for each row in the table.
- Enforces uniqueness constraints during bulk load operations to prevent duplicate entries.
- Facilitates detection and resolution of conflicts or duplicates during data insertion or updating.
- Influences the behavior of upsert, merge, or conflict resolution mechanisms in the bulk load process.
- Ensures that each record can be reliably matched and updated based on the specified keys.
- Supports both single-column and composite keys, maintaining the order of columns as per the database schema.
- |-
## IMPLEMENTATION GUIDANCE:
- Include all columns that form the complete primary key, especially for composite keys, maintaining the correct order as defined in the database schema.
- Ensure column names exactly match those in the target database, respecting case sensitivity where applicable.
- Validate that all specified primary key columns exist in both the source data and the target table schema before initiating the bulk load.
- Avoid leaving the primaryKeys list empty when uniqueness enforcement or conflict resolution is required.
- Consider the immutability and stability of primary key columns to prevent inconsistencies during repeated load operations.
- Confirm that the primary key columns are indexed or constrained appropriately in the target database to optimize performance.
- |-
## EXAMPLES:
- ["id"] — single-column primary key.
- ["order_id", "product_id"] — composite primary key consisting of two columns.
- ["user_id", "timestamp"] — composite key combining user identifier and timestamp for uniqueness.
- ["customer_id", "account_number", "region"] — multi-column composite primary key.
- |-
## IMPORTANT NOTES:
- Omitting primaryKeys when required can result in data duplication, failed loads, or incorrect conflict handling.
examples:
- ["id"]
- ["INTERNAL_ID", "LOCATION_INTERNAL_ID"]
overrideMergeQuery:
type: boolean
description: |-
When true, a custom merge query replaces the auto-generated MERGE statement, enabling ignore-existing logic, conditional updates, or multi-table operations ("Override merge/insert statement" on the import form).
The override SQL lives in `query` and references `{{preMergeTemporaryTable}}` (also written `{{import.rdbms.bulkLoad.preMergeTemporaryTable}}`) for the staging table and `{{tableName}}` for the destination table.
The SQL itself in this field fails the save: 422 `invalid_field` "Invalid value '<the statement>' for field 'rdbms.bulkLoad.overrideMergeQuery'. Reason: CastError."
x-celigo-ai-guidance:
- |-
This is the switch, not the SQL. Set it to true and put the statement in `query` (one string); the import form shows the SQL box only while "Override merge/insert statement" is checked. With it false or absent the platform generates the INSERT / MERGE from `bulkLoad.tableName` and `bulkLoad.primaryKeys`, and `query` stays empty.
- |-
The statement runs after the files are loaded into the staging table: read from `{{preMergeTemporaryTable}}` (also written `{{import.rdbms.bulkLoad.preMergeTemporaryTable}}`) and write to `{{tableName}}`; the form warns when the staging-table placeholder is missing. Use it for ignore-existing, conditional updates or multi-table work the generated MERGE cannot express — for example `INSERT INTO {{tableName}} (id, total) SELECT id, total FROM {{preMergeTemporaryTable}}`.
aggregation:
type: object
description: |-
How the staged files are cut and handed to the database. The import form writes it for a new bulk-load import when the account's `enableSizeBasedAggregation` flag is on for that database (Snowflake included — its form offers bulk_load without the flag but writes `aggregation` only with it); the server fills `type` and `maxFileSizeInMB` when they are omitted, so `loadDataAsAvailable` is the only value to author.
properties:
type:
type: string
enum: [size]
x-enumDescriptions:
size: Files are cut by size
description: Filled by the server on save; the only mode is size-based cutting.
maxFileSizeInMB:
type: integer
description: Size of each staged file in MB; the server fills 250 when the field is omitted and the import form never sets it — leave it out.
examples: [250]
loadDataAsAvailable:
type: boolean
default: false
description: |-
When true, each staged file is sent for loading as soon as it is written, so the dataset arrives as several smaller loads ("Load data as available" on the import form). When false, every file is held until all are ready and they are loaded together.
x-celigo-ai-guidance:
- |-
Send a real boolean and default to false unless the user asks for files to load as they are ready: a non-boolean value (for example the string "yes") makes the server drop the whole `aggregation` block without an error.
- |-
The block outlives a query-type change — after a PUT that switches the import to bulk_insert and omits `bulkLoad`, the read-back still carries `bulkLoad.aggregation`. That is server behavior, not a failed write; do not re-send the document to clear it.
description: |-
Configuration for bulk loading: the records are staged as files and loaded with the database's native COPY / bulk mechanism for maximum throughput.
Required when queryType is `["bulk_load"]`; available on Snowflake, Google BigQuery, Microsoft SQL Server (cloud connections), Azure Synapse and Amazon Redshift.
x-celigo-ai-guidance:
- |-
Specifies whether bulk loading is enabled for relational database management
system (RDBMS) operations,
allowing for the efficient insertion of large volumes of data through a single,
optimized operation.
Enabling bulkLoad significantly improves performance during data import,
migration,
or batch processing by minimizing the overhead associated with individual row
inserts and leveraging database-specific bulk insert mechanisms.
This feature is particularly beneficial for initial data loads,
large-scale migrations, or periodic batch updates where speed,
resource efficiency, and reduced transaction time are critical.
Bulk loading may temporarily alter database behavior—such as disabling indexes,
constraints, or triggers—to maximize throughput,
and often requires elevated permissions and careful management of transactional
integrity to ensure data consistency.
Proper use of bulkLoad can lead to substantial reductions in processing time and
system resource consumption during large data operations.
- |-
## FIELD BEHAVIOR:
- When set to true, the system uses optimized bulk loading techniques to insert data in large batches, greatly enhancing throughput and efficiency.
- When false or omitted, data insertion defaults to standard row-by-row operations, which may be slower and consume more resources.
- Typically enabled during scenarios involving initial data population, large batch imports, or data migration processes.
- May temporarily disable or defer enforcement of indexes, constraints, and triggers to improve performance during the bulk load operation.
- Can affect database locking and concurrency, potentially locking tables or partitions for the duration of the bulk load.
- Bulk loading operations may bypass certain transactional controls, affecting rollback and error recovery behavior.
- |-
## IMPLEMENTATION GUIDANCE:
- Verify that the target RDBMS supports bulk loading and understand its specific syntax, capabilities, and limitations.
- Assess transaction management implications, as bulk loading may alter or bypass triggers, constraints, and rollback mechanisms.
- Implement robust error handling and post-load validation to ensure data integrity and consistency after bulk operations.
- Monitor system resources such as CPU, memory, and I/O throughput during bulk load to avoid performance bottlenecks or outages.
- Plan for potential impacts on database availability, locking behavior, and concurrent access during bulk load execution.
- Ensure data is properly staged and preprocessed to meet the format and requirements of the bulk loading mechanism.
- Coordinate bulk loading with maintenance windows or low-traffic periods to minimize disruption.
updateLookupName:
type: [string, 'null']
description: Name of the lookup used for update operations in the legacy composite (queryType1) flow.
x-celigo-ai-guidance:
- |-
Specifies the exact name of the lookup table or entity within the relational
database management system (RDBMS) that is targeted for update operations.
This property serves as a precise identifier to determine which lookup data set
should be modified,
ensuring accurate and efficient targeting within the database schema.
It typically corresponds to a physical table name or a logical entity name
defined in the database and must strictly align with the existing schema to
enable successful updates without errors or unintended side effects.
Proper use of this property is critical for maintaining data integrity during
update operations on lookup data,
as it directly influences which records are affected.
Accurate specification helps prevent accidental data corruption and supports clear,
maintainable database interactions.
- |-
## FIELD BEHAVIOR:
- Defines the specific lookup table or entity to be updated during an operation.
- Acts as a key identifier for the update process to locate the correct data set.
- Must be provided when performing update operations on lookup data.
- Influences the scope and effect of the update by specifying the target entity.
- Changes to this value directly affect which data is modified in the database.
- Invalid or missing values will cause update operations to fail or target unintended data.
- |-
## IMPLEMENTATION GUIDANCE:
- Ensure the value exactly matches the existing lookup table or entity name in the database schema.
- Validate against the RDBMS naming conventions, including allowed characters, length restrictions, and case sensitivity.
- Maintain consistent naming conventions across the application to prevent ambiguity and errors.
- Account for case sensitivity based on the underlying database system's configuration and collation settings.
- Avoid using reserved keywords, special characters, or whitespace that may cause conflicts or syntax errors.
- Implement validation checks to catch misspellings or invalid names before executing update operations.
- Incorporate error handling to manage cases where the specified lookup name does not exist or is inaccessible.
- |-
## EXAMPLES:
- "country_codes"
- "user_roles"
- "product_categories"
- "status_lookup"
- "department_list"
- |-
## IMPORTANT NOTES:
- Providing an incorrect or misspelled name will result in failed update operations or runtime errors.
- This field must not be left empty when an update operation is intended.
- It is only relevant and required when performing update operations on lookup tables or entities.
- Changes to this value should be carefully managed to avoid unintended data modifications or corruption.
- Ensure appropriate permissions exist for updating the specified lookup entity to prevent authorization failures.
- Consistent use of this property across environments (development, staging, production) is essential
updateExtract:
type: [string, 'null']
description: Path used to extract the value that drives update operations in the legacy composite flow.
ignoreLookupName:
type: [string, 'null']
description: |-
Name of an entry in `lookups` that probes the database per record to decide whether it
already exists, so `ignoreExisting` can skip it (or `ignoreMissing` can skip records with
no match). Use it when existence must be read from the database; use `ignoreExtract` when
the incoming record already carries the field that signals existence.
ignoreExtract:
type: [string, 'null']
description: |-
Path to a field on the incoming record whose presence decides the skip: with
`ignoreExisting`, a populated value marks the record as existing and it is skipped; with
`ignoreMissing`, an empty value marks it as missing and it is skipped. Zero per-record
overhead; set exactly one of `ignoreExtract` / `ignoreLookupName`.
# Shape → requireds. Nested if/else-if (no allOf), stack-style.
# Outer gate: queryType is required only on real configs — the server writes stub
# `rdbms` objects ({} / {lookups:[]}) onto imports of every adaptorType.
# Inner chain: queryType → required companion field. The `type: array` guard keeps a
# null queryType from vacuously matching `contains` (JSON Schema `contains` passes on
# non-arrays), which would force a companion field the nullable type deliberately
# leaves optional.
# Mutual exclusion is expressed as "the other companion must be EMPTY if present"
# (maxItems/maxProperties 0), NOT as `not: required`: the server itself writes empty
# bookkeeping companions — POSTing {queryType:["bulk_insert"], bulkInsert:{...}} with
# no query stores query:[] alongside, and legacy docs
# carry bulkInsert:{} stubs next to per_record/bulk_load configs. A REAL companion
# (non-empty query with bulk_insert, populated bulkInsert with per_record) still fails.
if:
not:
propertyNames:
enum: [lookups]
then:
required:
- queryType
if:
required:
- queryType
properties:
queryType:
type: array
contains:
enum: [per_record, per_page]
then:
required:
- query
properties:
bulkInsert:
maxProperties: 0
else:
if:
required:
- queryType
properties:
queryType:
type: array
contains:
const: first_page
then:
required:
- query
else:
if:
required:
- queryType
properties:
queryType:
type: array
contains:
const: bulk_insert
then:
required:
- bulkInsert
properties:
query:
maxItems: 0
else:
if:
required:
- queryType
properties:
queryType:
type: array
contains:
const: bulk_load
then:
required:
- bulkLoad
# The one populated query that belongs beside bulk_load is the override
# statement: allowed only while bulkLoad.overrideMergeQuery is true.
if:
required:
- bulkLoad
properties:
bulkLoad:
type: object
required:
- overrideMergeQuery
properties:
overrideMergeQuery:
const: true
then:
properties:
query:
type: array
else:
properties:
query:
maxItems: 0
references/schemas/request.yml
Request:
type: object
description: |-
Fields that can be sent when creating or updating an import. Set the adaptor-specific
configuration object matching `adaptorType` (e.g. `netsuite_da` for `NetSuiteDistributedImport`).
`_connectionId` is required except for the connection-less flavors (`ToolImport`,
`AiAgentImport`, `GuardrailImport`, `TopicImport`): a `ToolImport` binds connections through
the referenced tool's `overrides`, AI agent / guardrail imports only use a connection for
BYOK, and an Event Streams publisher (`TopicImport`) is delivered by the platform itself.
required:
- name
allOf:
- $ref: './base.yml#/ImportBase'
# _connectionId is waived for the connection-less flavors, recognized either by an explicit
# adaptorType or by the presence of the flavor's config subdoc (the server infers the
# adaptorType from that subdoc when it is omitted). Every other import still requires it.
if:
anyOf:
- required: [adaptorType]
properties:
adaptorType:
enum: [ToolImport, AiAgentImport, GuardrailImport, TopicImport]
- required: [tool]
- required: [aiAgent]
- required: [guardrail]
- required: [topic]
then:
# Event Streams publisher: the API saves a TopicImport with an empty `topic` (batchSize
# default only); the topic binding is the form's rule.
if: { properties: { adaptorType: { const: TopicImport } }, required: [adaptorType] }
then:
x-celigo-ui-override: >-
Required by the Event Streams publisher form (fieldDefinitions/resources/imports/import.js:
'topic._topicId' is required:true). Encoded to mirror the form so builders produce
publishers that are bound to a topic.
required: [topic]
properties:
topic:
required: [_topicId]
else:
required: [_connectionId]
# NetSuite REST APIs reuse the `http` object on a netsuite connection; `nsDomainType` picks
# the host and is 422-required ("Path `nsDomainType` is required.").
if: { properties: { adaptorType: { const: NetSuiteHTTPImport } }, required: [adaptorType] }
then: { required: [http, nsDomainType] }
else:
# Cloud file providers add a location rule keyed on `assistant` (the import-level field
# the provider forms set). Form-derived: the API saves either shape without a directory.
if: { properties: { assistant: { const: azuredatalakestoragegen2 } }, required: [assistant] }
then:
x-celigo-ui-override: >-
Required by the Azure Data Lake Storage Gen2 import form
(fieldDefinitions/resources/imports/azuredatalakestoragegen2.js: container
'file.directory.storageRootId_adls' is required:true and preSave persists it as
file.directory.name; the full file path is required:true and preSave splits it into
http.relativeURI (directory prefix) and file.fileName (basename)). Encoded to mirror
the form so builders produce connectable configurations.
required: [file]
properties:
file:
required: [directory, fileName]
properties:
directory:
required: [name]
else:
if: { properties: { assistant: { const: googlecloudstorage } }, required: [assistant] }
then:
x-celigo-ui-override: >-
Required by the Google Cloud Storage import form
(fieldDefinitions/resources/imports/googlecloudstorage.js: bucket
'file.directory.storageRootId_gcs' is required:true). Encoded to mirror the form so
builders produce connectable configurations.
required: [file]
properties:
file:
required: [directory]
properties:
directory:
required: [storageRootId]
references/schemas/response.yml
Import:
type: object
required:
- _id
- name
- adaptorType
- apiIdentifier
- createdAt
- lastModified
description: Import object as returned by the API.
allOf:
- $ref: './base.yml#/ImportBase'
- $ref: '../common-schemas/resource-response.yml#/ResourceResponse'
- $ref: '../common-schemas/ia-resource-response.yml#/IAResourceResponse'
- type: object
properties:
aiDescription:
$ref: '../common-schemas/ai-description.yml#/AIDescription'
apim:
$ref: '../common-schemas/apim.yml#/APIM'
apiIdentifier:
type: string
readOnly: true
description: API identifier assigned to this import.
sandbox:
type: boolean
deprecated: true
readOnly: true
description: When true, this import belongs to a sandbox account.
rest:
type: object
deprecated: true
readOnly: true
additionalProperties: true
description: |-
Legacy REST adaptor configuration, still returned on imports created before
the REST-to-HTTP migration (`adaptorType: RESTImport`). Mirrors the shape of
`http` with per-operation arrays (`method`, `relativeURI`, `body`,
`responseIdPath`). On write the platform maintains the equivalent `http`
configuration — configure new imports through `http` instead.
_sourceId:
type: string
format: objectId
readOnly: true
description: Reference to the source resource this import was created from.
_templateId:
type: string
format: objectId
readOnly: true
x-celigo-refModel: templates
description: Template this import was created from.
draft:
type: boolean
readOnly: true
description: When true, this import is in draft state and has not been confirmed.
draftExpiresAt:
type: string
format: date-time
readOnly: true
description: Timestamp when the draft version of this import expires.
debugUntil:
type: string
format: date-time
readOnly: true
description: |-
Turns on step debug capture until this time: while it lies in the future, every
request this import sends — or, for a `ToolImport`, every record passing through
it with the tool's output — is recorded in the step's debug log
(`GET /v1/{flows|apis|tools}/{containerId}/{_id}/requests`). Read-only on PUT —
written only by `PATCH /v1/imports/{_id}` on `/debugUntil`, at most 3600 seconds
ahead (`422` `update_not_allowed` beyond that); `remove` stops capture.
references/schemas/s3.yml
S3:
type: object
description: |-
Defines where files are written to an Amazon S3 bucket. Required when the _connectionId
field references an AWS S3 connection; must not be included for other connection types.
region and bucket locate the destination, fileKey sets the object key, and backupBucket
retains a copy after a successful import.
x-celigo-ai-guidance:
- |-
## KEY FIELDS
- region: REQUIRED — the AWS region of the target bucket (e.g. us-east-1)
- bucket: REQUIRED — the destination S3 bucket name
- fileKey: the S3 object key (path + filename) for the written file
- backupBucket: optional — bucket where a copy is retained after a successful import
required:
- region
- bucket
properties:
region:
type: string
default: us-east-1
x-lowercase: true
x-celigo-ui-override: >-
Required by the S3 import form (fieldDefinitions/resources/imports/s3.js: s3.region is
required:true, default us-east-1). Encoded to mirror the form so builders produce
connectable configurations.
description: |-
AWS region where the S3 bucket resides. Must match the bucket's actual location to avoid
connectivity errors and misrouted requests.
x-celigo-ai-guidance:
- |-
The AWS region where the S3 bucket is physically located,
representing the specific geographical area within AWS's global infrastructure
that hosts the bucket.
This designation directly affects data access latency, availability, redundancy,
and compliance with regional data governance, privacy laws,
and residency requirements.
Selecting the appropriate region is critical for optimizing performance,
minimizing costs related to data transfer and storage,
and ensuring adherence to legal and organizational policies.
The region must be specified using a valid AWS region identifier that accurately
corresponds to the bucket's actual location to avoid connectivity issues,
authentication failures, and improper routing of API requests.
Proper region selection also influences disaster recovery strategies,
service availability, and integration with other AWS services,
making it a foundational parameter in S3 bucket configuration and management.
- |-
## FIELD BEHAVIOR:
- Defines the physical and logical location of the S3 bucket within AWS's global network.
- Directly impacts data access speed, latency, and throughput.
- Determines compliance with regional data sovereignty, privacy, and security regulations.
- Must be a valid AWS region code that matches the bucket's deployment location.
- Influences availability zones, fault tolerance, and disaster recovery planning.
- Affects endpoint URL construction and API request routing.
- Plays a role in cost optimization by affecting storage pricing and data transfer fees.
- Governs integration capabilities with other AWS services available in the selected region.
- |-
## IMPLEMENTATION GUIDANCE:
- Use official AWS region codes such as "us-east-1", "eu-west-2", or "ap-southeast-2" as defined by AWS.
- Validate the region value against the current list of supported AWS regions to prevent misconfiguration.
- Ensure the specified region exactly matches the bucket's physical location to avoid access and authentication errors.
- Consider cost implications including storage pricing, data transfer fees, and cross-region replication expenses.
- When migrating or replicating buckets across regions, update this property accordingly and plan for data transfer and downtime.
- Verify service availability and feature support in the chosen region before deployment.
- Regularly review AWS region updates and deprecations to maintain compatibility.
- Use region-specific endpoints when constructing API requests to ensure proper routing.
- |-
## EXAMPLES:
- us-east-1 (Northern Virginia, USA)
- eu-central-1 (Frankfurt, Germany)
- ap-southeast-2 (Sydney, Australia)
- sa-east-1 (São Paulo, Brazil)
- ca-central-1 (Canada Central)
examples: ["us-east-1", "eu-west-1", "ap-southeast-2", "eu-central-1", "sa-east-1", "ca-central-1"]
bucket:
type: string
x-celigo-ui-override: >-
Required by the S3 import form (fieldDefinitions/resources/imports/s3.js: s3.bucket is
required:true). Encoded to mirror the form so builders produce connectable configurations.
description: |-
S3 bucket that receives the imported files. The bucket must already exist and the
connection's AWS credentials must have s3:PutObject permission on it.
x-celigo-ai-guidance:
- |-
The name of the Amazon S3 bucket where data or objects are stored or will be stored.
This bucket serves as the primary container within the AWS S3 service,
uniquely identifying the storage location for your data across all AWS accounts globally.
The bucket name must strictly comply with AWS naming conventions,
including being DNS-compliant, entirely lowercase, and free of underscores,
uppercase letters, or IP address-like formats,
ensuring global uniqueness and compatibility.
It is essential that the specified bucket already exists and that the user or
service has the necessary permissions to access,
upload, or modify its contents.
Selecting the appropriate bucket, particularly with regard to its AWS region,
can significantly impact application performance, cost efficiency,
and adherence to data residency and regulatory requirements.
- |-
## FIELD BEHAVIOR:
- Specifies the target S3 bucket for all data storage and retrieval operations.
- Must reference an existing bucket with proper access permissions.
- Acts as a globally unique identifier within the AWS S3 ecosystem.
- Immutable after initial assignment; changing the bucket requires specifying a new bucket name.
- Influences data locality, latency, cost, and compliance based on the bucket's region.
- |-
## IMPLEMENTATION GUIDANCE:
- Validate bucket names against AWS S3 naming rules: lowercase letters, numbers, and hyphens only; no underscores, uppercase letters, or IP address formats.
- Confirm the bucket's existence and verify access permissions before performing operations.
- Consider the bucket's AWS region to optimize latency, cost, and compliance with data governance policies.
- Implement comprehensive error handling for scenarios such as bucket not found, access denied, or insufficient permissions.
- Align bucket selection with organizational policies and regulatory requirements concerning data storage and residency.
- |-
## EXAMPLES:
- "my-app-data-bucket"
- "user-uploads-2024"
- "company-backups-eu-west-1"
- |-
## IMPORTANT NOTES:
- Bucket names are globally unique across all AWS accounts and cannot be reused or renamed once created.
- Access control is managed through AWS IAM roles, policies, and bucket policies, which must be correctly configured.
- The bucket's region selection is critical for optimizing performance, minimizing costs, and ensuring legal compliance.
- Bucket naming restrictions prohibit uppercase letters, underscores, and IP address-like formats to maintain DNS compliance.
- |-
## DEPENDENCY CHAIN:
- Depends on the availability and stability of the AWS S3 service.
- Requires valid AWS credentials with appropriate permissions to access or modify the bucket.
examples: ["my-company-imports", "customer-data-integration", "celigo-etl-files", "my-app-data-bucket", "user-uploads-2024", "company-backups-eu-west-1"]
fileKey:
type: string
x-celigo-agent:
- handlebars
description: |-
Object key under which each file is stored in the bucket. Include slashes to organize
files in a folder-style hierarchy (for example, imports/2023/12/orders.json); keys are
case-sensitive. Supports handlebars placeholders such as orders-{{timestamp}}.json to
generate a unique key per run.
x-celigo-ai-guidance:
- |-
The unique identifier or path string that specifies the exact location of a file
stored within an Amazon S3 bucket.
This key acts as the primary reference for locating, accessing, managing,
and manipulating a specific file within the storage system.
It can be a simple filename or a hierarchical path that mimics folder structures
to logically organize files within the bucket.
The fileKey is case-sensitive and must be unique within the bucket to prevent
conflicts or overwriting.
It supports UTF-8 encoded characters and can include delimiters such as slashes
("/") to simulate directories,
enabling structured file organization.
Proper encoding and adherence to AWS naming conventions are essential to ensure
seamless integration with S3 APIs and web requests.
Additionally, the fileKey plays a critical role in access control,
lifecycle policies, and event notifications within the S3 environment.
- |-
## FIELD BEHAVIOR:
- Uniquely identifies a file within the S3 bucket namespace.
- Serves as the primary reference in all file-related operations such as retrieval, update, deletion, and metadata management.
- Case-sensitive and must be unique to avoid overwriting or conflicts.
- Supports folder-like delimiters (e.g., slashes) to simulate directory structures for logical organization.
- Used as a key parameter in all relevant S3 API operations, including GetObject, PutObject, DeleteObject, and CopyObject.
- Influences access permissions and lifecycle policies when used with prefixes or specific key patterns.
- |-
## IMPLEMENTATION GUIDANCE:
- Use a UTF-8 encoded string that accurately reflects the file's intended path or name.
- Avoid unsupported or problematic characters; strictly follow S3 key naming conventions to prevent errors.
- Incorporate logical folder structures using delimiters for better organization and maintainability.
- Ensure URL encoding when the key is included in web requests or URLs to handle special characters properly.
- Validate that the key length does not exceed S3's maximum allowed size (up to 1024 bytes).
- Consider the impact of key naming on access control policies, lifecycle rules, and event triggers.
- |-
## EXAMPLES:
- "documents/report2024.pdf"
- "images/profile/user123.png"
- "backups/2023/12/backup.zip"
- "videos/2024/events/conference.mp4"
- "logs/2024/06/15/server.log"
- |-
## IMPORTANT NOTES:
- The fileKey is case-sensitive; for example, "File.txt" and "file.txt" are distinct keys.
examples: ["orders-{{timestamp}}.json", "imports/2023/12/data.csv", "upload.xml", "documents/report2024.pdf", "images/profile/user123.png", "backups/2023/12/backup.zip", "videos/2024/events/conference.mp4", "logs/2024/06/15/server.log"]
backupBucket:
type: string
description: |-
S3 bucket where a copy of each imported file is retained after a successful import; if
omitted, no backup copy is kept. Must reference an existing bucket the connection's
credentials have s3:PutObject permission on.
x-celigo-ai-guidance:
- |-
The name of the Amazon S3 bucket designated for securely storing backup data.
This bucket serves as the primary repository for saving copies of critical information,
ensuring data durability, integrity,
and availability for recovery in the event of data loss, corruption,
or disaster.
It must be a valid, existing bucket within the AWS environment,
configured with appropriate permissions and security settings to facilitate
reliable and efficient backup operations.
Proper configuration of this bucket—including enabling versioning to preserve
historical backup states,
applying encryption to protect data at rest,
and setting lifecycle policies to manage storage costs and data retention—is
essential to optimize backup management,
maintain compliance with organizational and regulatory requirements,
and control expenses.
The bucket should ideally reside in the same AWS region as the backup source to
minimize latency and data transfer costs,
and may incorporate cross-region replication for enhanced disaster recovery capabilities.
Additionally, the bucket should be monitored regularly for access patterns,
storage usage,
and security compliance to ensure ongoing protection and operational efficiency.
- |-
## FIELD BEHAVIOR:
- Specifies the exact S3 bucket where backup data will be written and stored.
- Must reference a valid, existing bucket within the AWS account and region.
- Used exclusively during backup processes to save data snapshots or incremental backups.
- Requires appropriate write permissions to allow backup services to upload data.
- Typically remains static unless backup storage strategies or requirements change.
- Supports integration with backup scheduling and monitoring systems.
- Facilitates data recovery by maintaining backup copies in a secure, durable location.
- |-
## IMPLEMENTATION GUIDANCE:
- Verify that the bucket name adheres to AWS S3 naming conventions and is DNS-compliant.
- Confirm the bucket exists and is accessible by the backup service or user.
- Set up IAM roles and bucket policies to grant necessary write and read permissions securely.
- Enable bucket versioning to maintain historical backup versions and support data recovery.
- Implement lifecycle policies to manage storage costs by archiving or deleting old backups.
- Apply encryption (e.g., AWS KMS) to protect backup data at rest and meet compliance requirements.
- Consider enabling cross-region replication for disaster recovery scenarios.
- Regularly audit bucket permissions and access logs to ensure security compliance.
- Monitor storage usage and costs to optimize resource allocation and prevent unexpected charges.
- Align bucket configuration with organizational policies for data retention, security, and compliance.
- |-
## EXAMPLES:
- "my-app-backups"
- "company-data-backup-2024"
- "prod-environment-backup-bucket"
examples: ["my-company-archive", "integration-backup", "my-app-backups", "company-data-backup-2024", "prod-environment-backup-bucket"]
serverSideEncryptionType:
type: string
description: |-
Server-side encryption applied to uploaded objects. The connection form sets `AES256`
(SSE-S3, S3-managed keys) when encryption is enabled; leave unset to use the bucket's
default encryption.
x-celigo-ai-guidance:
- |-
serverSideEncryptionType specifies the type of server-side encryption applied to
objects stored in the S3 bucket,
determining how data at rest is securely protected by the storage service.
This property defines the encryption algorithm or method used by the server to
automatically encrypt data upon storage,
ensuring confidentiality, integrity, and compliance with security standards.
It accepts predefined values that correspond to supported encryption mechanisms,
influencing how data is encrypted, accessed,
and managed during storage and retrieval operations.
By configuring this property,
users can enforce encryption policies that safeguard sensitive information
without requiring client-side encryption management,
thereby simplifying security administration and enhancing data protection.
- |-
## FIELD BEHAVIOR:
- Defines the encryption algorithm or method used by the server to encrypt data at rest.
- Controls whether server-side encryption is enabled or disabled for stored objects.
- Accepts only the value AES256 (SSE-S3, S3-managed keys); leave unset to use the bucket's default encryption.
- Influences data handling during storage and retrieval to maintain data confidentiality and integrity.
- Applies encryption automatically without requiring client-side encryption management.
- Determines compliance with organizational and regulatory encryption requirements.
- Affects how encryption keys are managed, accessed, and audited.
- Impacts access control and permissions related to encrypted objects.
- May affect performance and cost depending on the encryption method chosen.
- |-
## IMPLEMENTATION GUIDANCE:
- Validate the input value against the supported encryption type: only "AES256".
- Ensure the selected encryption type is compatible with the S3 bucket's configuration, policies, and permissions.
- Gracefully handle scenarios where encryption is not specified, disabled, or set to an empty value.
- Provide clear and actionable error messages if an unsupported or invalid encryption type is supplied.
- Consider the impact on performance and cost when enabling encryption.
- Document encryption settings clearly for users to understand the security posture of stored data.
- Implement fallback or default behaviors when encryption settings are omitted.
- Ensure that encryption settings align with audit and compliance reporting requirements.
- Test encryption settings thoroughly to confirm data is encrypted and accessible as expected.
- |-
## EXAMPLES:
- "AES256" — Server-side encryption using Amazon S3-managed encryption keys (SSE-S3).
examples: ["AES256"]
references/schemas/salesforce.yml
Salesforce:
type: object
description: |-
Salesforce-specific configuration for the import: the operation to perform, the API to use, and object-level settings such as `sObjectType`, `idLookup`, and `upsert`. Properties like `ignoreExisting`, `ignoreMissing`, `name`, and `description` are not part of this object — they belong at the import level.
x-celigo-ai-guidance:
- |-
Configuration for Salesforce imports containing operation type, API selection,
and object-specific settings.
- |-
When asked to generate this configuration, return an object with these properties:
- `operation` (REQUIRED): The Salesforce operation type (insert, update, upsert, etc.)
- `api` (REQUIRED): The Salesforce API to use (default: "soap")
- `sObjectType`: The Salesforce object type (e.g., "Account", "Contact", "Vendor__c")
- `idLookup`: Configuration for looking up existing records
- `upsert`: Configuration for upsert operations (when operation is "upsert")
- And other operation-specific properties as needed
# Requireds are conditional (see the if/then chain at the end of this schema):
# blob configs (attachment / contentVersion) carry operation but no api and skip the
# operation→idLookup/upsert rules; server bookkeeping also writes stub `salesforce`
# objects ({lookups:[]} / {}) onto imports of every adaptorType.
properties:
lookups:
description: |
Lookup definitions used to resolve Salesforce record references or
static value mappings. Each lookup has a unique ``name`` that mapping
fields reference via ``lookupName`` and that Handlebars expressions
reference via ``{{lookup "name" value}}``.
**Mirror invariant**
The Celigo platform stores lookups in TWO locations on a Salesforce
import resource: top-level ``lookups`` and ``salesforce.lookups``.
Both arrays must hold byte-for-byte identical content, so write
both on every save.
**Static vs dynamic lookups**
- **Static** — only ``map`` + optional ``default``. No Salesforce
query runs. Use for small controlled vocabularies (country codes,
status labels, etc.).
- **Dynamic** — ``sObjectType`` + ``resultField`` + ``whereClause``.
Issues a SOQL query at runtime. Use ``map`` alongside the query to
short-circuit known values before hitting Salesforce.
items:
type: object
properties:
name:
type: string
description: |
Unique identifier for this lookup entry. Mapping fields reference
it via ``lookupName``, and Handlebars expressions reference it
via ``{{lookup "name" value}}``. Must be unique within this
resource's ``lookups`` array. Conventionally a short slug or a
generated hash — the value is only meaningful as a reference
token.
sObjectType:
type: string
description: |
Salesforce sObject API name to query (e.g. ``Account``,
``Contact``, ``Solution__c``). For custom objects include the
``__c`` suffix. Case-sensitive — must match exactly as declared
in the target Salesforce org.
resultField:
type: string
description: |
sObject field whose value is returned when the SOQL ``whereClause``
matches a record. Typically ``Id`` when the lookup resolves to a
Salesforce record reference, but any queryable field is valid.
whereClause:
type: string
x-celigo-agent:
- handlebars
description: |
SOQL ``WHERE`` clause that identifies the matching Salesforce
record. Use Handlebars expressions (triple braces recommended:
``{{{field}}}``) to inject values from the incoming source record.
Wrap the whole expression in parentheses.
Examples:
- ``(Id = {{{id [Customer SFDC ID]}}})``
- ``(Email = '{{{email}}}')``
- ``(External_Id__c = {{{string externalId}}} AND IsActive = true)``
Celigo's Handlebars helpers (``string``, ``id``, ``double``,
``bool``, ``date``) produce correctly-typed SOQL literals —
prefer them over manual quoting. String values wrap in single
quotes; numeric/id values do not.
map:
type: [object, 'null']
description: |
Optional static key→value map evaluated BEFORE the SOQL query.
When the incoming field value matches a key, the mapped value
is returned without hitting Salesforce. Useful for small
controlled enumerations where a full query would be overkill.
default:
type: [string, 'null']
description: |
Value returned when the SOQL query matches no records. When
omitted and ``allowFailures`` is false, an unmatched lookup
fails the record.
allowFailures:
type: boolean
description: |
When true, a lookup miss (no matching record and no ``default``)
is tolerated — the mapped field resolves to ``null`` and the
import continues. When false (default), a miss halts processing
for the record.
useDefaultOnMultipleMatches:
type: boolean
description: |
Controls behaviour when the SOQL query matches more than one
record. When true, the ``default`` value is returned instead of
raising an ambiguous-match error. When false (default), multiple
matches are treated as a failure.
type: array
x-celigo-agent:
- lookup
operation:
type: string
enum:
- insert
- update
- upsert
- upsertpicklistvalues
- delete
- addupdate
x-lowercase: true
x-enumDescriptions:
insert: Create new records in Salesforce.
update: Modify existing records in Salesforce, identified by record ID.
upsert: Create or update records based on an External ID field match.
upsertpicklistvalues: Create or update picklist values on a Salesforce field.
delete: Remove records from Salesforce, moving them to the recycle bin.
addupdate: Create new or update existing records using Celigo-specific matching logic.
x-celigo-ui-override: >-
Required by the Salesforce import form (operation is required:true). Encoded to mirror the
form so builders produce connectable configurations.
description: |-
Controls how records are written to Salesforce. Each value pairs with a lookup strategy: `upsert` requires `upsert.externalIdField` plus `idLookup.extract`, while `update`, `delete`, `addupdate`, and `insert` with `ignoreExisting` require `idLookup.whereClause`. Values are converted to lowercase automatically.
x-celigo-ai-guidance:
- |-
Specifies the type of Salesforce operation to perform on records.
This determines how data is created, modified, or removed in Salesforce.
- |-
## FIELD BEHAVIOR:
- Defines the specific Salesforce action to execute (insert, update, upsert, etc.)
- Dictates required fields and validation rules for the operation
- Determines API endpoint and request structure
- Automatically converted to lowercase before processing
- Influences response format and error handling
- |-
## OPERATION TYPES:
### **insert**
- **Purpose:** Create new records in Salesforce
- **Behavior:** Creates new records
- **Use when:** Creating brand new records
- **With ignoreExisting: true:** Checks for existing records and skips them (requires idLookup.whereClause)
- **Without ignoreExisting:** Creates all records without checking for duplicates
- **Example:** "Insert new leads from marketing campaign"
- **Example with ignoreExisting:** "Create vendors while ignoring existing vendors" → operation: "insert" + ignoreExisting: true
### **update**
- **Purpose:** Modify existing records in Salesforce
- **Behavior:** Requires record ID; fails if record doesn't exist
- **Use when:** Updating known existing records
- **Example:** "Update customer status based on order completion"
### **upsert**
- **Purpose:** Create or update records based on external ID
- **Behavior:** Updates if external ID matches, creates if not found
- **Use when:** Syncing data from external systems with external ID field
- **Example:** "Upsert customers by External_ID__c field"
- **REQUIRES:**
- `idLookup.extract` field (maps incoming field to External ID)
- `upsert.externalIdField` field (specifies Salesforce External ID field)
- Both fields MUST be set for upsert to work
### **upsertpicklistvalues**
- **Purpose:** Create or update picklist values dynamically
- **Behavior:** Manages picklist value creation/updates
- **Use when:** Maintaining picklist values from external data
- **Example:** "Sync product categories to Salesforce picklist"
### **delete**
- **Purpose:** Remove records from Salesforce
- **Behavior:** Requires record ID; records moved to recycle bin
- **Use when:** Removing obsolete or invalid records
- **Example:** "Delete expired opportunities"
### **addupdate**
- **Purpose:** Create new or update existing records (Celigo-specific)
- **Behavior:** Similar to upsert but uses different matching logic
- **Use when:** Upserting without requiring external ID field
- **Example:** "Sync contacts, create new or update existing by email"
- |-
## IMPLEMENTATION GUIDANCE:
- Choose operation based on whether records are new, existing, or unknown
- For **insert**: Ensure records are truly new to avoid errors
- For **insert with ignoreExisting: true**: MUST set `idLookup.whereClause` to check for existing records
- For **update/delete**: MUST set `idLookup.whereClause` to identify records
- For **upsert**: MUST also include the `upsert` object with `externalIdField` property
- For **upsert**: MUST also set `idLookup.extract` to map incoming field
- For **addupdate**: MUST set `idLookup.whereClause` to define matching criteria
- Use **upsertpicklistvalues** only for picklist management scenarios
- |-
## COMMON PATTERNS:
- **"Create X while ignoring existing X"** → operation: "insert" + api: "soap" + idLookup.whereClause
- **"Create or update X"** → operation: "upsert" + api: "soap" + upsert.externalIdField + idLookup.extract
- **"Update X"** → operation: "update" + api: "soap" + idLookup.whereClause
- **"Sync X"** → operation: "addupdate" + api: "soap" + idLookup.whereClause
- **ALWAYS include** api field (default to "soap" if not specified in prompt)
- |-
## EXAMPLES:
### Insert new records:
```json
{
"operation": "insert",
"api": "soap",
"sObjectType": "Lead"
}
```
Prompt: "Create new leads in Salesforce"
### Insert with ignore existing:
```json
{
"operation": "insert",
"api": "soap",
"sObjectType": "Vendor__c",
"idLookup": {
"whereClause": "Name = '{{{VendorName}}}'"
}
}
```
Prompt: "Create vendor records while ignoring existing vendors"
**NOTE:** For "ignore existing" prompts, also remember that `ignoreExisting: true` belongs at a different level (not part of this salesforce configuration).
### Update existing records:
```json
{
"operation": "update",
"api": "soap",
"sObjectType": "Account",
"idLookup": {
"whereClause": "Id = '{{{AccountId}}}'"
}
}
```
Prompt: "Update existing accounts in Salesforce"
### Upsert by external ID:
```json
{
"operation": "upsert",
"api": "soap",
"sObjectType": "Contact",
"idLookup": {
"extract": "ExternalId"
},
"upsert": {
"externalIdField": "External_ID__c"
}
}
```
Prompt: "Create or update contacts by External_ID__c"
### Addupdate (create or update):
```json
{
"operation": "addupdate",
"api": "soap",
"sObjectType": "Customer__c",
"idLookup": {
"whereClause": "Name = '{{{CustomerName}}}'"
}
}
```
Prompt: "Sync customers, create new or update existing"
- |-
## IMPORTANT NOTES:
- Operation value is case-insensitive (automatically converted to lowercase)
- Invalid operation values will cause API validation errors
- Each operation has different requirements for record identification
- **CRITICAL for upsert**: MUST include the `upsert` object with `externalIdField` property
- **CRITICAL for upsert**: MUST set `idLookup.extract` to map incoming field to External ID
- **upsert** requires external ID field to be configured in Salesforce
- **addupdate** is Celigo-specific and may not work with all Salesforce APIs
- Operations must align with user's Salesforce permissions
- Some operations support bulk processing more efficiently than others
api:
type: string
enum:
- soap
- rest
- metadata
- compositerecord
x-lowercase: true
x-enumDescriptions:
soap: Salesforce SOAP API for transactional or legacy imports; supports AllOrNone transaction control and XML-based requests, with more overhead than REST
rest: Standard Salesforce REST API for real-time single-record and small-batch imports; supports CRUD, flexible querying, rich error messages, and up to 2,000 records per API call
metadata: Salesforce Metadata API for deploying configuration and setup data between orgs, such as custom objects, fields, and workflows; not for data records
compositerecord: Salesforce Composite Graph API for creating hierarchical related record trees, such as parent-child relationships, atomically in one request
x-celigo-ui-override: >-
Required by the Salesforce import form (api is required:true, default soap). Encoded to
mirror the form so builders produce connectable configurations.
description: |-
Selects the Salesforce API used to execute the import. Use `soap` for most imports and another API only when its specific capability is needed; default to `soap` when no API is specified. Values are converted to lowercase automatically.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Required by the import form for record imports; legacy attachment
imports are stored without it and the platform falls back to soap
- Determines the protocol and endpoint used for Salesforce communication
- Influences request format, batch sizes, and response handling
- Automatically converted to lowercase before processing
- Affects available features and operation types
- Impacts performance and error handling behavior
- |-
## IMPLEMENTATION GUIDANCE:
- **MUST ALWAYS SET THIS FIELD** - Never leave it undefined
- If prompt doesn't specify API type, use **"soap"**
- Use **"soap"** only when AllOrNone transaction control is explicitly required
- Use **"metadata"** only for configuration deployment, never for data
- Use **"compositerecord"** for complex parent-child relationship imports
- Consider API limits and performance characteristics for your data volume
- Ensure the Salesforce connection supports the chosen API type
- |-
## EXAMPLES:
### REST API:
```json
{
"operation": "insert",
"api": "rest",
"sObjectType": "Lead"
}
```
Prompt: "Import leads into Salesforce"
### SOAP API with transaction control:
```json
{
"operation": "insert",
"api": "soap",
"sObjectType": "Invoice__c",
"soap": {
"headers": {
"allOrNone": true
}
}
}
```
Prompt: "Import invoices with all-or-none guarantee"
### Composite Record API:
```json
{
"operation": "insert",
"api": "compositerecord",
"sObjectType": "Account"
}
```
Prompt: "Import accounts with related contacts and opportunities"
- |-
## IMPORTANT NOTES:
- API value is case-insensitive (automatically converted to lowercase)
- SOAP API has slightly more overhead but provides transaction guarantees
- Metadata API is for configuration only, not data records
- Composite Record API requires careful relationship mapping
- Each API has different rate limits and performance characteristics
- Ensure your Salesforce connection and permissions support the chosen API
- |-
**REQUIRED FIELD:** Specifies which Salesforce API to use for the import operation.
Different APIs have different capabilities, performance characteristics,
and use cases.
- |-
**CRITICAL:** This field MUST ALWAYS be set.
If no specific API is mentioned in the prompt, **DEFAULT to "soap"**.
- |-
## DEFAULT CHOICE:
When in doubt or when no API is mentioned in the prompt → **USE "soap"**
soap:
type: object
properties:
headers:
type: object
properties:
allOrNone:
type: boolean
description: |-
When true, the batch is processed atomically: if any record fails, the entire transaction is rolled back and nothing is committed. When false or omitted, records that succeed are committed even if others in the batch fail.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- When true, enforces atomicity: all records in the batch must succeed or the entire operation is rolled back with no changes applied.
- When false, permits partial success: successfully processed records are committed even if some records fail.
- Applies mainly to batch DML operations including create, update, delete, and upsert.
- Helps maintain data consistency by preventing partial updates when strict transactional integrity is required.
- Influences how errors are reported and handled in the API response.
- Affects transaction commit behavior and rollback mechanisms within the Salesforce platform.
- |-
## IMPLEMENTATION GUIDANCE:
- Set to true to guarantee strict data consistency and avoid partial data changes that could lead to inconsistent states.
- Set to false to allow higher throughput and partial processing when some failures are acceptable or expected.
- Default value is false if the property is omitted, allowing partial success by default.
- When false, implement robust error handling to process partial success responses and handle individual record errors appropriately.
- Verify that the target Salesforce API operation supports the allOrNone header before use.
- Consider the impact on transaction duration and system resources when enabling atomic operations.
- Use in scenarios where data integrity is paramount, such as financial or compliance-related data updates.
- |-
## EXAMPLES:
- allOrNone = true: Attempting to update 10 records where one record fails validation results in no records being updated, preserving atomicity.
- allOrNone = false: Attempting to update 10 records where one record fails validation results in 9 records updated successfully and one failure reported.
- allOrNone = true: Deleting multiple records in a batch will either delete all specified records or none if any deletion fails.
- |-
allOrNone specifies whether the Salesforce API operation should be executed atomically,
ensuring that either all parts of the operation succeed or none do.
When set to true, if any record in a batch operation fails,
the entire transaction is rolled back and no changes are committed,
preserving data integrity across the dataset.
When set to false, the operation allows partial success,
meaning some records may be processed and committed even if others fail,
which can improve performance but requires careful error handling to manage partial failures.
This property is primarily applicable to data manipulation operations such as create,
update, delete, and upsert,
and is critical for maintaining consistent and reliable data states in
multi-record transactions.
description: SOAP headers included in requests sent to Salesforce, such as the `allOrNone` transaction-control header.
batchSize:
type: number
description: |-
Number of records sent to Salesforce in each SOAP API batch. Larger batches reduce the number of API calls; smaller batches reduce per-call load and the chance of timeouts.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Determines the count of records included in each batch request to the Salesforce SOAP API.
- Directly influences the total number of API calls required when handling large datasets.
- Affects processing throughput, latency, and memory usage during data operations.
- Can be dynamically adjusted to optimize performance based on system capabilities and network conditions.
- Impacts error handling and retry mechanisms by controlling batch granularity.
- Influences network load and the likelihood of encountering API limits or timeouts.
- |-
## IMPLEMENTATION GUIDANCE:
- Choose a batch size that adheres to Salesforce API limits and recommended best practices.
- Balance between larger batch sizes (which reduce API calls but increase memory and processing load) and smaller batch sizes (which increase API calls but reduce resource consumption).
- Ensure the batchSize value is a positive integer within the allowed range for the specific Salesforce operation.
- Continuously monitor API response times, error rates, and resource utilization to fine-tune the batch size for optimal performance.
- Default to Salesforce-recommended batch sizes when uncertain or when starting configuration.
- Consider network stability and latency when selecting batch size to minimize timeouts and failures.
- Validate batch size compatibility with other integration components and middleware.
- Test batch size settings in a staging environment before deploying to production to avoid unexpected failures.
- |-
## EXAMPLES:
- `batchSize: 200` — Processes 200 records per batch, a common default for many Salesforce operations.
- `batchSize: 500` — Processes 500 records per batch, often the maximum allowed for certain Salesforce SOAP API calls.
- `batchSize: 50` — Processes smaller batches suitable for environments with limited memory or network bandwidth.
- `batchSize: 100` — A moderate batch size balancing throughput and resource usage in typical scenarios.
- |-
The number of records to be processed in each batch during Salesforce SOAP API operations.
This property controls the size of data chunks sent or retrieved in a single API call,
optimizing performance and resource utilization by balancing throughput and
system resource consumption.
Adjusting the batch size directly impacts the number of API calls made,
the processing speed, memory usage, and network load,
enabling fine-tuning of data operations to suit specific environment constraints
and performance goals.
Proper configuration of batchSize helps prevent API throttling,
reduces the risk of timeouts,
and ensures efficient handling of large datasets by segmenting data into manageable portions.
description: Settings that apply when `api` is `soap`, including SOAP request headers and batch size.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Defines all connection, authentication, and operational parameters necessary for Salesforce SOAP API interactions.
- Facilitates the construction and transmission of properly structured SOAP requests and the parsing of SOAP responses.
- Manages the session lifecycle, including token acquisition, refresh, and timeout handling to maintain persistent connectivity.
- Supports comprehensive error detection and handling of SOAP faults, API exceptions, and network issues to maintain integration stability.
- Enables customization of SOAP headers, namespaces, and envelope structures as required by specific Salesforce API calls.
- Allows configuration of retry policies and backoff strategies in response to transient errors or rate limiting.
- Supports environment-specific configurations, such as production, sandbox, or custom Salesforce domains.
- |-
## IMPLEMENTATION GUIDANCE:
- Configure endpoint URLs and authentication credentials accurately based on the Salesforce environment (production, sandbox, or custom domains).
- Validate SOAP envelope structure, namespaces, and compliance with the latest Salesforce WSDL specifications to prevent communication errors.
- Implement robust error handling to capture, log, and respond appropriately to SOAP faults, API limit exceptions, and network failures.
- Securely store and transmit sensitive information such as session IDs, OAuth tokens, passwords, and certificates, following best security practices.
- Monitor Salesforce API usage limits and implement throttling or queuing mechanisms to avoid exceeding quotas and service disruptions.
- Set timeout values thoughtfully to balance between preventing hanging requests and allowing sufficient time for valid operations to complete.
- Regularly update the SOAP configuration to align with Salesforce API version changes, WSDL updates, and evolving security requirements.
- Incorporate logging and monitoring to track SOAP request and response cycles for troubleshooting and performance optimization.
- Ensure compatibility with Salesforce security protocols, including TLS versions and encryption standards.
- |-
Specifies the comprehensive configuration settings required for integrating with
the Salesforce SOAP API.
This property includes all essential parameters to establish, authenticate,
and manage SOAP-based communication with Salesforce services,
enabling a wide range of operations such as creating, updating, deleting,
querying, and executing actions on Salesforce objects.
It encompasses detailed configurations for endpoint URLs,
authentication credentials (such as session IDs or OAuth tokens), SOAP headers,
session management details, timeout settings,
and error handling mechanisms to ensure reliable, secure,
and efficient interactions.
The configuration must strictly adhere to Salesforce's WSDL definitions and security standards,
supporting robust SOAP API usage within the application environment.
sObjectType:
type: string
description: |-
Salesforce object the import writes to, such as `Account`, `Contact`, or a custom object like `Vendor__c`. Must exactly match the object's case-sensitive API name; custom objects include the `__c` suffix.
x-celigo-ai-guidance:
- |-
sObjectType specifies the exact Salesforce object type that the API operation will target,
such as standard objects like Account, Contact, Opportunity,
or custom objects like CustomObject__c.
This property is essential for defining the schema context of the API request,
determining which fields are relevant,
and directing the operation to the correct Salesforce data entity.
It plays a critical role in data validation, processing logic,
and endpoint routing within the Salesforce environment,
ensuring that operations such as create, update, delete,
or query are executed against the intended object type with precision.
Accurate specification of sObjectType enables dynamic adjustment of request
payloads and response handling based on the object’s schema and enforces
compliance with Salesforce’s API naming conventions and permissions model.
- |-
## FIELD BEHAVIOR:
- Identifies the specific Salesforce object (sObject) type targeted by the API operation.
- Determines the applicable schema, field availability, and validation rules based on the selected object.
- Must exactly match a valid Salesforce object API name recognized by the connected Salesforce instance, including case sensitivity.
- Routes the API request to the appropriate Salesforce object endpoint for the intended operation.
- Influences dynamic field mapping, data transformation, and response parsing processes.
- Supports both standard and custom Salesforce objects, with custom objects requiring the “__c” suffix.
- |-
## IMPLEMENTATION GUIDANCE:
- Validate sObjectType values against the current Salesforce object metadata in the target environment to prevent invalid requests.
- Enforce exact case-sensitive matching with Salesforce API object names (e.g., "Account" not "account").
- Support and recognize both standard objects and custom objects, ensuring custom objects include the “__c” suffix.
- Implement robust error handling for invalid, unsupported, or deprecated sObjectType values to provide clear feedback.
- Regularly update the list of allowed sObjectType values to reflect changes in the Salesforce schema or environment.
- Use sObjectType to dynamically tailor API request payloads, field selections, and response parsing logic.
- Consider object-level permissions and authentication scopes when processing requests involving specific sObjectTypes.
- |-
## EXAMPLES:
- "Account"
- "Contact"
- "Opportunity"
- "Lead"
- "CustomObject__c"
- "Case"
- "Campaign"
- |-
## IMPORTANT NOTES:
- Custom Salesforce objects must be referenced using their exact API names, including the “__c” suffix.
examples: ["Account", "Contact", "Vendor__c"]
idLookup:
type: object
properties:
extract:
type: string
description: |-
Field in the incoming data whose value is matched against the Salesforce External ID field named in `upsert.externalIdField`. Set only when `operation` is `upsert`; all other operations use `whereClause` instead. Choose a field that is unique and always populated — missing or null values cause the upsert to fail. If both `extract` and `whereClause` are set, `extract` takes precedence for upsert.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Specifies which field from the incoming data contains the External ID value
- **Used for upsert operations ONLY**
- Maps the incoming data field to the Salesforce External ID field specified in `upsert.externalIdField`
- Must reference a field that exists in the incoming data payload
- The field value must be unique and always populated to ensure reliable matching
- Generates field mapping at runtime to match incoming records with Salesforce records
- |-
## IMPLEMENTATION GUIDANCE:
- Choose a field from incoming data that contains unique, non-null values
- Ensure the field is always populated in the incoming records to prevent operation failures
- This field's value will be matched against the Salesforce External ID field specified in `upsert.externalIdField`
- Use the exact field name as it appears in the incoming data (case-sensitive)
- If sample data is available, validate that the field exists and has appropriate values
- Test with sample data to ensure matching works correctly before production deployment
- |-
## IMPORTANT NOTES:
- **ONLY use for upsert operations**
- For all other operations (update, delete, addupdate, insert with ignoreExisting), use `whereClause` instead
- Field must exist in incoming data and be consistently populated
- Works with `upsert.externalIdField` to enable External ID matching
- Missing or null values will cause the upsert operation to fail
- Field name is case-sensitive and must match exactly
- If both `extract` and `whereClause` are set, `extract` takes precedence for upsert
- |-
Please specify which field from your export data should map to External ID.
It is very important to pick a field that is both unique,
and guaranteed to always have a value when exported (otherwise the Salesforce
Upsert operation will fail).
If sample data is available then a select list should be displayed here to help
you find the right field from your export data.
If sample data is not available then you will need to manually type the field id
that should map to External ID.
- |-
## WHEN TO USE:
- **upsert operations ONLY** — REQUIRED when operation is "upsert"
- Works together with `upsert.externalIdField` to enable External ID matching
- |-
## WHEN NOT TO USE:
- **update** — use `whereClause` instead
- **delete** — use `whereClause` instead
- **addupdate** — use `whereClause` instead
- **insert with ignoreExisting** — use `whereClause` instead
- |-
## EXAMPLE:
### Upsert by External ID:
```json
{
"operation": "upsert",
"idLookup": {
"extract": "CustomerNumber"
},
"upsert": {
"externalIdField": "External_ID__c"
}
}
```
Maps incoming field "CustomerNumber" → Salesforce field "External_ID__c"
**How it works:**
- Incoming record has `{"CustomerNumber": "CUST-12345", "Name": "John"}`
- System extracts "CUST-12345" from CustomerNumber
- Searches Salesforce External_ID__c field for "CUST-12345"
- If found: updates the existing record
- If not found: creates a new record with External_ID__c = "CUST-12345"
- |-
## COMMON FIELD NAMES:
- "ExternalId" — incoming field containing external identifier
- "CustomerId" — incoming field with unique customer number
- "AccountNumber" — incoming field with account number
- "SourceSystemId" — incoming field from source system
examples: ["externalId"]
whereClause:
type: string
x-celigo-agent:
- handlebars
description: |-
SOQL condition (without the `WHERE` keyword) that finds existing Salesforce records for each incoming record. Required when `operation` is `update`, `delete`, or `addupdate`, and for `insert` with `ignoreExisting`; for `upsert`, use `extract` instead. Reference incoming fields with triple-brace Handlebars and wrap string values in single quotes (e.g. `Email = '{{{Email}}}'`) — triple braces emit the raw value without HTML-escaping, which would corrupt characters like `&` or `'`. Combine conditions with `AND`/`OR`.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Specifies a SOQL conditional expression to match existing records in Salesforce
- Omit the "WHERE" keyword - only provide the condition expression
- Supports Handlebars {{{fieldName}}} syntax to reference incoming data fields
- Supports SOQL operators: =, !=, >, <, >=, <=, LIKE, IN, NOT IN
- Supports logical operators: AND, OR, NOT
- Can use parentheses for grouping complex conditions
- The clause is evaluated for each incoming record to find matching Salesforce records
- |-
## IMPLEMENTATION GUIDANCE:
- Write the clause following SOQL syntax rules, omitting the "WHERE" keyword
- Use {{{fieldName}}} syntax to reference values from incoming records
- Wrap string values containing spaces in single quotes: `Name = '{{{FullName}}}'`
- Numeric and boolean values don't need quotes: `Age = {{{Age}}}`
- Use indexed fields (Email, External ID) for better performance
- Keep the clause selective to minimize records returned
- Validate syntax before deployment to prevent runtime errors
- |-
## EXAMPLES:
### Update by Email:
```json
{
"operation": "update",
"idLookup": {
"whereClause": "Email = '{{{Email}}}'"
}
}
```
Finds records with matching Email for update
### Insert with ignore existing by Email:
```json
{
"operation": "insert",
"ignoreExisting": true,
"idLookup": {
"whereClause": "Email = '{{{Email}}}'"
}
}
```
Checks if a record with matching Email exists; if found, skips creation
### Addupdate (create or update) by Email:
```json
{
"operation": "addupdate",
"idLookup": {
"whereClause": "Email = '{{{Email}}}'"
}
}
```
Updates if Email matches, creates new record if not found
### Delete by Account Number:
```json
{
"operation": "delete",
"idLookup": {
"whereClause": "AccountNumber = '{{{AcctNum}}}'"
}
}
```
Finds records with matching AccountNumber for deletion
### Complex AND logic:
```json
{
"operation": "update",
"idLookup": {
"whereClause": "Name = '{{{ProductName}}}' AND ProductFamily__c = '{{{Family}}}'"
}
}
```
Matches records where both Name AND ProductFamily match
### Complex OR logic:
```json
{
"operation": "update",
"idLookup": {
"whereClause": "Email = '{{{Email}}}' OR Phone = '{{{Phone}}}'"
}
}
```
Matches records where Email OR Phone matches
### String with spaces:
```json
{
"operation": "update",
"idLookup": {
"whereClause": "Name = '{{{FullName}}}'"
}
}
```
Note: Single quotes around {{{FullName}}} handle names with spaces like "John Smith"
- |-
## IMPORTANT NOTES:
- **REQUIRED** for update, delete, addupdate, and insert (with ignoreExisting)
- DO NOT use for upsert operations - use `extract` instead
- Use {{{fieldName}}} to reference incoming data fields
- String values with spaces must be wrapped in single quotes
- Numeric and boolean values don't need quotes
- Poor clause design can impact performance and hit governor limits
- Always test in sandbox before production deployment
- |-
## WHEN NOT TO USE:
- **upsert** — DO NOT use; use `extract` instead
- **insert without ignoreExisting** — Not needed for simple inserts
- |-
To determine if a record exists in Salesforce,
define a SOQL query WHERE clause for the sObject type selected above.
For example, if you are importing contact records with a unique email,
your WHERE clause should identify contacts with the same email.
- |-
Handlebars statements are allowed,
and more complex WHERE clauses can contain AND and OR logic.
For example, when the product name alone is not unique,
you can define a WHERE clause to find any records where the product name exists
AND the product also belongs to a specific product family,
resulting in a unique product record that matches both criteria.
- |-
To find string values that contain spaces, wrap them in single quotes,
for example:
- Name = 'John Smith'
- |-
## WHEN TO USE (REQUIRED FOR):
- **update** — REQUIRED to identify which records to update
- **delete** — REQUIRED to identify which records to delete
- **addupdate** — REQUIRED to identify existing records (update if found, insert if not)
- **insert with ignoreExisting: true** — REQUIRED to check if records exist (skip if found)
examples: ["Email = '{{{Email}}}'", "External_Id__c = '{{{externalId}}}'"]
description: |-
Controls how existing Salesforce records are matched before writing. Set exactly one of `extract` (for `upsert`) or `whereClause` (for `update`, `delete`, `addupdate`, and `insert` with `ignoreExisting`); omit for plain inserts.
Required when operation is addupdate, delete, update, or upsert.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Contains two strategies for finding existing records: `extract` and `whereClause`
- **`extract`**: Used ONLY for upsert operations
- **`whereClause`**: Used for update, delete, addupdate, and insert (with ignoreExisting)
- Generates field mapping or SOQL queries at runtime to match records
- Critical for preventing duplicate record creation and enabling record updates
- |-
## IMPLEMENTATION GUIDANCE:
- Use `extract` ONLY for upsert operations (with `upsert.externalIdField`)
- Use `whereClause` for update, delete, addupdate, and insert with ignoreExisting
- Do not set both `extract` and `whereClause` for the same operation
- **ONLY set extract or whereClause - do not nest operation or other properties**
- Ensure the lookup strategy matches the operation type
- Test thoroughly in sandbox before production deployment
- |-
## IMPORTANT NOTES:
- **This object ONLY contains `extract` or `whereClause`**
- **`extract`** is for upsert ONLY - maps to External ID field
- **`whereClause`** is for all other operations requiring lookups
- Do NOT nest `operation`, `upsert`, or other properties here - those belong at the parent salesforce level
- For upsert: `extract` + `upsert.externalIdField` work together (both at salesforce level)
- For other operations: `whereClause` enables SOQL-based record matching
- Incorrect configuration can cause duplicates or failed operations
- |-
Configuration for looking up existing Salesforce records. **This object should
ONLY contain either `extract` OR `whereClause` - NOT both,
and NOT operation or other properties.**
- |-
## WHAT THIS OBJECT CONTAINS:
This object should contain ONLY ONE of these properties:
- `extract` — for upsert operations ONLY
- `whereClause` — for update, delete, addupdate, and insert (with ignoreExisting)
**DO NOT include operation, upsert, or any other properties here** - those belong at the parent salesforce level.
- |-
## WHEN TO USE EACH FIELD:
### Use `extract` (upsert ONLY):
- **upsert** — REQUIRED: Maps incoming field to External ID field
### Use `whereClause` (all other operations):
- **update** — REQUIRED: Identifies records to update via SOQL query
- **delete** — REQUIRED: Identifies records to delete via SOQL query
- **addupdate** — REQUIRED: Identifies existing records via SOQL query
- **insert with ignoreExisting: true** — REQUIRED: Checks if records exist via SOQL query
### Leave empty:
- **insert without ignoreExisting** — No lookup needed for simple inserts
- |-
## WHAT TO SET (idLookup object content):
### For upsert:
```json
{
"extract": "CustomerNumber"
}
```
### For update/delete/addupdate:
```json
{
"whereClause": "Email = '{{{Email}}}'"
}
```
- |-
## PARENT CONTEXT EXAMPLES (for reference):
These show the complete parent-level structure. **idLookup itself should only contain extract or whereClause.**
### Upsert (parent context):
```json
{
"operation": "upsert",
"idLookup": {
"extract": "CustomerNumber"
},
"upsert": {
"externalIdField": "External_ID__c"
}
}
```
### Update (parent context):
```json
{
"operation": "update",
"idLookup": {
"whereClause": "Email = '{{{Email}}}'"
}
}
```
upsert:
type: object
properties:
externalIdField:
type: string
description: |-
API name of the Salesforce field, marked as External ID, that upsert operations match against. This is the target side of the match; `idLookup.extract` names the source field in the incoming data. If the value matches multiple Salesforce records, the upsert fails.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Identifies which Salesforce field serves as the external ID for record matching
- Must be a field marked as "External ID" in Salesforce object configuration
- Works together with `idLookup.extract` to enable upsert matching
- The field must be indexed and unique in Salesforce
- Supports text, number, and email field types configured as external IDs
- Required when `operation: "upsert"`
- |-
## IMPLEMENTATION GUIDANCE:
- Use the exact Salesforce API name of the field (case-sensitive)
- Verify the field is marked as "External ID" in Salesforce object settings
- Ensure the field is indexed for optimal performance
- Confirm the integration user has read/write access to this field
- The field must have unique values to prevent ambiguous matches
- Test in sandbox before production to verify configuration
- |-
## EXAMPLES:
### Standard External ID field:
```
"AccountNumber"
```
Standard Account field marked as External ID
### Custom External ID field:
```
"Custom_External_Id__c"
```
Custom field marked as External ID
### Email as External ID:
```
"Email"
```
Email field configured as External ID on Contact
### NetSuite ID as External ID:
```
"netsuite_conn__NetSuite_Id__c"
```
NetSuite connector External ID field
- |-
## IMPORTANT NOTES:
- Field must be explicitly marked as "External ID" in Salesforce
- Not all fields can be External IDs (must be text, number, or email type)
- Missing or null values in this field will cause upsert to fail
- If multiple records match the External ID, upsert will fail
- Always use with `idLookup.extract` for upsert operations
- |-
## HOW IT WORKS WITH idLookup.extract:
- **`idLookup.extract`**: Specifies which incoming data field contains the matching value (e.g., "CustomerNumber")
- **`externalIdField`**: Specifies which Salesforce field to match against (e.g., "External_ID__c")
- Together they create the match: incoming "CustomerNumber" → Salesforce "External_ID__c"
- |-
## HOW IT WORKS (Parent Context):
At the parent `salesforce` level, the complete configuration looks like:
```json
{
"operation": "upsert",
"idLookup": {
"extract": "CustomerNumber"
},
"upsert": {
"externalIdField": "External_ID__c"
}
}
```
Where:
- `operation: "upsert"` — at salesforce level
- `idLookup.extract: "CustomerNumber"` — incoming field, at salesforce level
- `upsert.externalIdField: "External_ID__c"` — **THIS field**, inside upsert object
- |-
## COMMON EXTERNAL ID FIELDS:
- "AccountNumber" — standard Account field
- "Email" — when configured as External ID on Contact
- "Custom_External_Id__c" — custom field on any object
- "External_ID__c" — common naming convention
- "Source_System_Id__c" — for multi-system integrations
examples: ["External_Id__c"]
description: |-
Upsert matching configuration, required when `operation` is `upsert` and omitted for all other operations. Contains only `externalIdField`, which pairs with `idLookup.extract` at the parent level.
Required when operation is upsert.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- **REQUIRED** when `operation: "upsert"`
- Contains configuration specific to upsert behavior
- **ONLY contains ONE property: `externalIdField`**
- Works in conjunction with `idLookup.extract` (which is at the SAME level as this upsert object, NOT nested within it)
- |-
## IMPLEMENTATION GUIDANCE:
- Always set `externalIdField` when using upsert
- Ensure the specified External ID field exists in Salesforce
- Verify the field is marked as "External ID" in Salesforce
- **ONLY set the externalIdField property - nothing else**
- |-
## IMPORTANT NOTES:
- **This object ONLY contains `externalIdField`**
- Do NOT nest `operation`, `idLookup`, or other properties here
- Those properties belong at the parent salesforce level, not nested in this upsert object
- May be extended with additional upsert-specific options in the future
- Only relevant when operation is "upsert"
- |-
## WHEN TO USE:
- **REQUIRED** when `operation` is "upsert" - MUST be included
- Leave undefined for insert, update, delete, or addupdate operations
- Without this object, upsert operations will fail
- |-
Configuration for Salesforce upsert operations.
This object contains ONLY the `externalIdField` property.
- '**CRITICAL: This component is REQUIRED when `operation: "upsert"`. Always include this component for upsert operations.**'
- |-
## WHY IT'S REQUIRED:
- Upsert operations need to know which Salesforce field to use for matching
- The `externalIdField` property specifies the Salesforce External ID field
- This works with `idLookup.extract` to enable record matching
- Missing this object will cause the import to fail
- |-
## DECISION:
**Return true/include this component if `operation: "upsert"` is set. This is REQUIRED for all upsert operations.**
- |-
## WHAT TO SET:
**This object should ONLY contain:**
```json
{
"externalIdField": "External_ID__c"
}
```
**DO NOT include operation, idLookup, or any other properties here** - those belong at the parent salesforce level.
- |-
## PARENT CONTEXT (for reference only):
At the parent `salesforce` level, you will also set:
- `operation: "upsert"` (at salesforce level)
- `idLookup.extract` (at salesforce level)
- `upsert.externalIdField` (THIS object)
upsertpicklistvalues:
type: object
properties:
type:
type: string
enum:
- picklist
- multipicklist
x-lowercase: true
x-enumDescriptions:
picklist: Standard single-select picklist allowing one value at a time; the most common picklist type
multipicklist: Multi-select picklist allowing multiple values separated by semicolons; use for categories, tags, or multi-value selections such as Tags, Skills, or Interests fields
description: Picklist field type being managed. Values are converted to lowercase automatically.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Determines whether the picklist allows single or multiple value selection
- Affects UI rendering and validation in Salesforce
- Automatically converted to lowercase before processing
- |-
Specifies the type of picklist field being managed.
Salesforce supports single-select picklists and multi-select picklists.
fullName:
type: string
description: |-
Fully qualified API name of the picklist field, in `ObjectName.FieldName__c` form (for example, `Account.MyPicklist__c`). Must match the case-sensitive Salesforce API name.
x-celigo-ai-guidance:
- |-
## EXAMPLES:
- "Account.MyPicklist__c"
- "Contact.Status__c"
- "CustomObject__c.Category__c"
examples: ["Account.MyPicklist__c"]
label:
type: string
description: Display label for the picklist field in the Salesforce UI. Can contain spaces and does not need to match the API name.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Human-readable label shown in Salesforce interface
- Can contain spaces and special characters
- Does not need to match API name
- |-
## EXAMPLES:
- "My Picklist"
- "Product Category"
- "Customer Status"
visibleLines:
type: number
description: |-
Number of values visible without scrolling in the Salesforce UI selection box. Only applies when `type` is `multipicklist`.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Only applies to multipicklist fields
- Controls height of the picklist selection box
- Default is typically 4-5 lines
- |-
## EXAMPLES:
- 3 — shows 3 values at once
- 5 — shows 5 values at once
- 10 — shows more values for large lists
- |-
For multi-select picklists,
specifies how many lines are visible in the UI without scrolling.
description: |-
Picklist field definition managed when `operation` is `upsertpicklistvalues`; omit for all other operations. Manages the field's metadata — name, label, and display settings — and requires Salesforce Metadata API permissions.
Required when operation is upsertpicklistvalues.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Used ONLY when `operation: "upsertpicklistvalues"`
- Manages picklist field metadata, not picklist values themselves
- Enables programmatic picklist field management
- Requires appropriate Salesforce metadata permissions
- |-
## IMPORTANT NOTES:
- This manages the FIELD itself, not the individual picklist VALUES
- Requires Salesforce API and Metadata API permissions
- Should only be set when operation is "upsertpicklistvalues"
- Most imports won't use this - it's for metadata management
- |-
## WHEN TO USE:
- When operation is set to "upsertpicklistvalues"
- For managing picklist field definitions programmatically
- When syncing picklist configurations between orgs
- For automated picklist field deployment
- |-
Configuration for managing Salesforce picklist field metadata when using the
`upsertpicklistvalues` operation.
This enables dynamic creation or updates to picklist field definitions.
removeNonSubmittableFields:
type: boolean
description: |-
When true, strips fields Salesforce will not accept — read-only, system-generated, and formula fields — from the payload before submission, preventing rejection errors. When false or omitted, all fields are submitted as-is. Only the outgoing payload is affected; the source data is unchanged.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- When enabled (set to true), the system analyzes the outgoing data payload and removes any fields identified as non-submittable based on Salesforce metadata, field-level security, and API constraints.
- When disabled (set to false) or omitted, all fields present in the payload are submitted as-is, which may lead to errors if restricted, read-only, or system-managed fields are included.
- Enhances data integrity by proactively filtering out fields that could cause rejection or failure during Salesforce data operations.
- Applies only to the outgoing submission payload, ensuring the original source data remains unchanged and intact.
- Dynamically adapts to different Salesforce objects, API versions, and user permissions by referencing up-to-date metadata and security settings.
- |-
## IMPLEMENTATION GUIDANCE:
- Enable this flag in integration scenarios where the data schema may include fields that Salesforce does not accept for the target object or API version.
- Maintain an up-to-date cache or real-time access to Salesforce metadata to accurately reflect field-level permissions, submittability, and API changes.
- Implement logging or reporting mechanisms to track which fields are removed, aiding in auditing, troubleshooting, and transparency.
- Use in conjunction with other data validation, transformation, and permission-checking steps to ensure clean, compliant data submissions.
- Thoroughly test in development and staging environments to understand the impact on data flows, error handling, and downstream processes.
- Consider user roles and permissions, as submittability may vary depending on the authenticated user's access rights and profile settings.
- |-
## EXAMPLES:
- `removeNonSubmittableFields: true` — Automatically removes read-only or system-managed fields such as CreatedDate, LastModifiedById, or formula fields before submission.
- `removeNonSubmittableFields: false` — Submits all fields, potentially causing errors if restricted or read-only fields are included.
- |-
Indicates whether fields that are not eligible for submission to Salesforce—such as read-only,
system-generated, formula,
or otherwise restricted fields—should be automatically excluded from the data
payload before it is sent.
This feature helps prevent submission errors by ensuring that only valid,
submittable fields are included in the API request,
thereby improving data integrity and reducing the likelihood of operation failures.
It operates exclusively on the outgoing submission payload without modifying the
original source data,
making it a safe and effective way to sanitize data before interaction with Salesforce APIs.
document:
type: object
properties:
id:
type: string
description: |-
Salesforce-assigned ID of the Document record. The 15-character form is case-sensitive; the 18-character form adds a case-insensitivity checksum.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Acts as the primary key uniquely identifying a document record within Salesforce.
- Immutable after creation; should never be changed or reassigned.
- Used consistently to reference the document across API calls, queries, and integrations.
- Case-sensitive and may vary in length (15 or 18 characters).
- Serves as a critical reference point for establishing relationships with other records.
- |-
## IMPLEMENTATION GUIDANCE:
- Must be generated exclusively by Salesforce or the managing system to guarantee uniqueness.
- Should be handled as a string to accommodate Salesforce’s ID formats.
- Validate incoming IDs against Salesforce ID format standards to ensure correctness.
- Avoid manual generation or modification of IDs to prevent data inconsistencies.
- Use the 18-character format when possible to leverage the checksum for case-insensitive environments.
- |-
## EXAMPLES:
- "0151t00000ABCDE"
- "0151t00000ABCDEAAA"
- |-
## IMPORTANT NOTES:
- IDs are case-sensitive; 15-character IDs are case-sensitive, while 18-character IDs include a case-insensitive checksum.
- Do not manually create or alter IDs; always use Salesforce-generated values.
- This ID is critical for performing document updates, deletions, retrievals, and establishing relationships.
- Using the correct ID format is essential for API compatibility and data integrity.
- |-
Unique identifier for the document within the Salesforce system,
serving as the primary key that distinctly identifies each document record.
This identifier is immutable once assigned and must not be altered or reused to
maintain data integrity.
It is essential for referencing the document in API calls, queries,
integrations, and any operations involving the document.
The ID is a base-62 encoded string that can be either 15 characters long,
which is case-sensitive, or 18 characters long,
which includes a case-insensitive checksum to enhance reliability in external systems.
Proper handling of this ID ensures accurate linkage and manipulation of document
records across various Salesforce processes and integrations.
- |-
## DEPENDENCY CHAIN:
- Required for all operations on salesforce.document entities, including CRUD actions.
- Other fields and processes may reference this ID to link related records or trigger actions.
- Integral to workflows, triggers, and integrations that depend on document identification.
- |-
## TECHNICAL DETAILS:
- Salesforce record IDs are base-62 encoded strings.
name:
type: string
description: Display name of the document in Salesforce, used to identify it in folders, search, and API operations.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Acts as the primary human-readable identifier for the document.
- Used in UI elements, search queries, and API calls to locate or reference the document.
- Typically mandatory when creating or updating document records.
- Should be unique within the scope of the relevant Salesforce object, folder, or library to avoid ambiguity.
- Influences sorting, filtering, and organization of documents within lists or folders.
- Changes to the name may trigger updates or validations in dependent systems or references.
- Supports international characters to accommodate global users.
- |-
## IMPLEMENTATION GUIDANCE:
- Choose descriptive yet concise names to enhance clarity, usability, and discoverability.
- Validate input to exclude unsupported special characters and ensure compliance with Salesforce naming rules.
- Enforce uniqueness constraints within the relevant context to prevent conflicts and confusion.
- Consider the impact of renaming documents on existing references, hyperlinks, integrations, and audit trails.
- Account for potential length restrictions and implement truncation or user prompts as necessary.
- Support internationalization by allowing UTF-8 encoded characters where appropriate.
- Implement validation to prevent use of reserved words or prohibited characters.
- |-
## EXAMPLES:
- "Quarterly Sales Report Q1 2024"
- "Employee Handbook"
- "Project Plan - Alpha Release"
- "Marketing Strategy Overview"
- "Client Contract - Acme Corp"
- |-
## IMPORTANT NOTES:
- The maximum length of the name may be limited (commonly up to 255 characters), depending on Salesforce configuration.
- Renaming a document can affect external references, hyperlinks, or integrations relying on the original name.
- Case sensitivity in name comparisons may vary based on Salesforce settings and should be considered when implementing logic.
- Special characters and whitespace handling should align with Salesforce’s accepted standards to avoid errors.
- |-
The name of the document as stored in Salesforce,
serving as a unique and human-readable identifier within the Salesforce environment.
This string value represents the document's title or label,
enabling users and systems to easily recognize, retrieve,
and manage the document across user interfaces, search functionalities,
and API interactions.
It is a critical attribute used to reference the document in various operations
such as creation,
updates, searches, display contexts, and integration workflows,
ensuring clear identification and organization within folders or libraries.
The name plays a pivotal role in document management by facilitating sorting,
filtering, and categorization,
and it must adhere to Salesforce’s naming conventions and constraints to
maintain system integrity and usability.
folderId:
type: string
description: |-
Salesforce folder that stores the document. The document inherits the folder's sharing and visibility settings; changing this value moves the document to a different folder.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Identifies the exact folder containing the document.
- Associates the document with a designated folder for structured organization.
- Must reference a valid, existing folder ID within Salesforce.
- Typically represented as a string conforming to Salesforce ID formats (15- or 18-character).
- Influences document visibility and access based on folder permissions.
- Changing the folderId moves the document to a different folder, affecting its context and access.
- Inherits folder-level sharing and security settings automatically.
- |-
## IMPLEMENTATION GUIDANCE:
- Verify that the folderId exists and is accessible in the Salesforce instance before assignment.
- Validate the folderId format to ensure it matches Salesforce’s 15- or 18-character ID standards.
- Use this field to filter, query, or categorize documents by their folder location in API operations.
- When creating or updating documents, setting this field assigns or moves the document to the specified folder.
- Handle permission checks to ensure the user or integration has rights to assign or access the folder.
- Consider the impact on sharing rules, workflows, and automation triggered by folder changes.
- Ensure case sensitivity is preserved when handling folder IDs.
- |-
## EXAMPLES:
- "00l1t000003XyzA" (example of a Salesforce folder ID)
- "00l5g000004AbcD"
- "00l9m00000FghIj"
- |-
## IMPORTANT NOTES:
- The folderId must be valid and the folder must be accessible by the user or integration performing the operation.
- Using an incorrect or non-existent folderId will cause errors or result in improper document categorization.
- Folder-level permissions in Salesforce can restrict the ability to assign documents or access folder contents.
- Changes to folderId may affect document sharing, visibility settings, and trigger related workflows.
- Folder IDs are case-sensitive and must be handled accordingly.
- |-
The unique identifier of the folder within Salesforce where the document is stored.
This ID is essential for organizing, categorizing,
and managing documents efficiently within the Salesforce environment.
It enables precise association of documents to specific folders,
facilitating streamlined retrieval, access control,
and hierarchical organization of document assets.
By linking a document to a folder,
it inherits the folder’s permissions and visibility settings,
ensuring consistent access management and supporting structured document workflows.
Proper use of this identifier allows for effective filtering, querying,
and management of documents based on their folder location,
which is critical for maintaining an organized and secure document repository.
contentType:
type: string
description: MIME type of the document content, such as `application/pdf` or `image/png`.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Defines the media type of the document content to guide processing, rendering, and handling.
- Identifies the specific file format (e.g., PDF, JPEG, DOCX) for compatibility and validation purposes.
- Facilitates content negotiation and appropriate response handling between systems and applications.
- Typically formatted as a standard MIME type string (type/subtype), optionally including parameters such as charset.
- Influences document handling in workflows, security scans, user interfaces, and API interactions.
- Serves as a key attribute for determining how the document is stored, transmitted, and displayed.
- |-
## IMPLEMENTATION GUIDANCE:
- Always use valid MIME types conforming to official standards (e.g., "application/pdf", "image/png") to ensure broad compatibility.
- Verify that the contentType accurately reflects the actual file content to prevent processing errors or misinterpretation.
- Update the contentType promptly if the document format changes to maintain consistency and reliability.
- Utilize this field to enable correct content handling in APIs, integrations, client applications, and automated workflows.
- Include relevant parameters (such as charset for text-based documents) when applicable to provide additional context.
- Consider validating the contentType against the file extension and content during upload or processing to enhance data integrity.
- |-
## EXAMPLES:
- "application/pdf" for PDF documents.
- "image/jpeg" for JPEG image files.
- "application/vnd.openxmlformats-officedocument.wordprocessingml.document" for Microsoft Word DOCX files.
- "text/plain; charset=utf-8" for plain text files with UTF-8 encoding.
- "application/json" for JSON formatted documents.
- "audio/mpeg" for MP3 audio files.
- |-
The MIME type of the document,
specifying the exact format of the file content stored within the Salesforce document.
This field enables systems and applications to accurately identify, process,
render, or handle the document by indicating its media type, such as PDF, image,
audio, or text formats.
It adheres to standard MIME type conventions (type/subtype) and may include
additional parameters like character encoding when necessary.
Properly setting this field ensures seamless integration, correct display,
appropriate application usage,
and reliable content negotiation across diverse platforms and services.
It plays a critical role in workflows, security scanning, compliance validation,
and API interactions by guiding how the document is stored, transmitted,
and interpreted.
developerName:
type: string
description: |-
Unique API name used to reference the document in code, metadata, and integrations. Distinct from the display name; uses only letters, numbers, and underscores with no spaces.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Serves as a unique and immutable identifier for the document within the Salesforce org.
- Widely used in code, APIs, metadata, deployment processes, and integrations to reference the document reliably.
- Should remain stable and not be altered frequently to maintain integration and system stability.
- Enforces naming conventions that disallow spaces and special characters, favoring camelCase or underscores.
- Case-insensitive in many contexts but consistent casing is recommended for clarity and maintainability.
- Acts as a key reference point in metadata relationships and dependency mappings.
- |-
## IMPLEMENTATION GUIDANCE:
- Ensure the developerName is unique within the Salesforce environment to prevent naming collisions.
- Follow Salesforce naming rules: start with a letter, use only alphanumeric characters and underscores.
- Avoid reserved keywords, spaces, special characters, and punctuation marks.
- Keep the identifier concise yet descriptive enough to clearly convey the document’s purpose or function.
- Validate the developerName against Salesforce’s character set, length restrictions, and naming policies before deployment.
- Plan for the developerName to remain stable post-deployment to avoid breaking references and integrations.
- Use consistent casing conventions (camelCase or underscores) to improve readability and reduce errors.
- Incorporate meaningful terms that reflect the document’s role or content for easier identification.
- |-
## EXAMPLES:
- "InvoiceTemplate"
- "CustomerAgreementDoc"
- "SalesReport2024"
- "ProductCatalog_v2"
- "EmployeeOnboardingGuide"
- |-
## IMPORTANT NOTES:
- Changing the developerName after deployment can disrupt integrations, metadata references, and dependent components.
- It is distinct from the document’s display name, which can include spaces and be more user-friendly.
- While Salesforce treats developerName as case-insensitive in many scenarios, maintaining consistent casing improves readability and maintenance.
- |-
developerName is a unique,
developer-friendly identifier assigned to the document within the Salesforce environment.
It acts as a stable and consistent reference used primarily in programmatic
contexts such as Apex code,
API calls, metadata configurations, deployment scripts,
and integration workflows.
This identifier ensures reliable access, manipulation,
and management of the document across various Salesforce components and external systems.
The value should be concise, descriptive,
and strictly adhere to Salesforce naming conventions to prevent conflicts and
ensure maintainability.
Typically, it employs camelCase or underscores,
excludes spaces and special characters,
and is designed to remain unchanged over time to avoid breaking dependencies or integrations.
isInternalUseOnly:
type: boolean
description: When true, marks the document for internal use only — it must not be shared with parties outside the organization.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- When set to true, the document is strictly restricted to internal users and must not be accessible or shared with any external entities.
- When false or omitted, the document may be accessible to external users, subject to other access control mechanisms.
- Commonly used to flag documents containing sensitive business strategies, internal communications, proprietary data, or confidential policies.
- Influences user interface elements and API responses to hide, restrict, or clearly label document visibility accordingly.
- Changes to this flag can trigger notifications, workflow actions, or audit events to ensure proper handling and compliance.
- May affect document indexing, search results, and export capabilities to prevent unauthorized dissemination.
- |-
## IMPLEMENTATION GUIDANCE:
- Always verify this flag before granting document access in both user interfaces and backend APIs to ensure compliance with internal use policies.
- Use in conjunction with role-based access controls, group memberships, document classification labels, and data sensitivity tags to enforce comprehensive security.
- Carefully manage updates to this flag to prevent inadvertent exposure of confidential information; consider implementing approval workflows and multi-level reviews for changes.
- Implement audit logging for all changes to this flag to support compliance, traceability, and forensic investigations.
- Ensure all integrated systems, third-party applications, and data export mechanisms respect this flag to maintain consistent access restrictions.
- Provide clear UI indicators, warnings, or access restrictions to users when accessing documents marked as internal use only.
- Regularly review and validate the accuracy of this flag to maintain up-to-date security postures.
- |-
## EXAMPLES:
- true: A confidential internal report on upcoming product launches accessible only to employees.
- false: A publicly available user manual or product datasheet intended for customers and partners.
- true: Internal HR policies and procedures documents restricted to company staff.
- false: Press releases or marketing materials distributed externally.
- true: Financial forecasts and budget plans shared exclusively within the finance department.
- true: Internal audit findings and compliance reports
- |-
isInternalUseOnly indicates whether the document is intended exclusively for
internal use within the organization and must not be shared with external
parties.
This flag is critical for protecting sensitive, proprietary,
or confidential information by restricting access strictly to authorized internal users.
It serves as a definitive marker within the document’s metadata to enforce
internal visibility policies,
prevent accidental or unauthorized external distribution,
and support compliance with organizational security standards.
Proper handling of this flag ensures that internal communications,
strategic plans, financial data,
and other sensitive materials remain protected from exposure outside the company.
isPublic:
type: boolean
description: When true, the document is accessible to all users; when false, access is limited to authorized users by Salesforce permissions and sharing settings.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- When true, the document is accessible by anyone, including unauthenticated users.
- When false, access is restricted to users with explicit permissions or roles.
- Directly influences the document’s visibility in search results, sharing interfaces, and external indexing.
- Changes to this property take immediate effect, altering who can view the document.
- Modifications may trigger updates in access control enforcement mechanisms.
- |-
## IMPLEMENTATION GUIDANCE:
- Use this property to clearly define and enforce document sharing and visibility policies.
- Always set to false for sensitive, confidential, or proprietary documents to prevent unauthorized access.
- Implement strict validation to ensure only users with appropriate administrative rights can modify this property.
- Consider organizational compliance requirements, privacy regulations, and data security standards before making documents public.
- Regularly monitor and audit changes to this property to maintain security oversight and compliance.
- Plan changes carefully, as toggling this setting can have immediate and broad impact on document accessibility.
- |-
## EXAMPLES:
- true: A publicly available product catalog, marketing brochure, or press release intended for external audiences.
- false: Internal project plans, financial reports, HR documents, or any content restricted to company personnel.
- |-
## IMPORTANT NOTES:
- Making a document public may expose it to indexing by search engines if accessible via public URLs.
- Changes to this property have immediate effect on document accessibility; ensure appropriate change management.
- Ensure alignment with corporate governance, legal requirements, and data protection policies when toggling public access.
- Public documents should be reviewed periodically to confirm that their visibility remains appropriate.
- Unauthorized changes to this property can lead to data leaks or compliance violations.
- |-
## DEPENDENCY CHAIN:
- Access control depends on user roles, profiles, and permission sets configured within the system.
- |-
Indicates whether the document is publicly accessible to all users or restricted
exclusively to authorized users within the system.
This property governs the document's visibility and access permissions,
determining if authentication is required to view its content.
When set to true,
the document becomes openly available without any access controls,
allowing anyone—including unauthenticated users—to access it freely.
Conversely,
setting this flag to false enforces security measures that limit access based on user roles,
permissions, and organizational sharing rules,
ensuring that only authorized personnel can view the document.
This setting directly impacts how the document appears in search results,
sharing interfaces, and external indexing services,
making it a critical factor in managing data privacy, compliance,
and information governance.
description: |-
Salesforce Document record written by the import, combining the file content with metadata such as name, folder, and content type.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Contains both the binary content (usually base64-encoded) and descriptive metadata of a file linked to a Salesforce record.
- Supports a diverse array of file types including PDFs, images, reports, contracts, spreadsheets, and other common document formats.
- Facilitates operations such as uploading new files, retrieving existing files, updating metadata, deleting documents, and managing versions within Salesforce.
- May be required or optional depending on the specific API operation, object context, or organizational business rules.
- Changes to the document content or metadata can trigger Salesforce workflows, triggers, validation rules, or automation processes.
- Maintains versioning and audit trails when integrated with Salesforce Content or Files features.
- Respects Salesforce security and sharing settings to control access and protect sensitive information.
- |-
## IMPLEMENTATION GUIDANCE:
- Encode the document content in base64 format when transmitting via API to ensure data integrity and compatibility.
- Validate file size and type against Salesforce platform limits and organizational policies before upload to prevent errors.
- Provide comprehensive metadata including file name, description, content type (MIME type), file extension, and relevant tags or categories to facilitate search, classification, and governance.
- Manage user permissions and access controls in accordance with Salesforce security and sharing settings to safeguard sensitive data.
- Use appropriate Salesforce API endpoints (e.g., /sobjects/Document, Attachment, ContentVersion) and HTTP methods aligned with the document type and intended operation.
- Implement robust error handling to manage responses related to file size limits, unsupported formats, permission denials, or network issues.
- |-
The 'document' property represents a digital file or record associated with a Salesforce entity,
encompassing a wide range of file types such as attachments, reports, images,
contracts, spreadsheets, and other files stored within the Salesforce platform.
It acts as a comprehensive container that includes both the file's binary
content—typically encoded in base64—and its associated metadata,
such as file name, description, MIME type, and tags.
This property enables seamless integration, retrieval, upload, update,
and management of files within Salesforce-related workflows and processes.
By linking relevant documents directly to Salesforce records,
it enhances data organization, accessibility, collaboration,
and compliance across various business functions.
Additionally, it supports versioning, audit trails,
and can trigger automation processes like workflows, validation rules,
and triggers,
ensuring robust document lifecycle management within the Salesforce ecosystem.
attachment:
type: object
properties:
id:
type: string
description: |-
Salesforce-assigned ID of the Attachment record. Prefer the 18-character case-insensitive form over the 15-character case-sensitive form in integrations.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Acts as the unique and immutable primary key for the attachment record.
- Required for all API operations involving the attachment, including retrieval, updates, and deletions.
- Ensures consistent and precise identification of the attachment in all Salesforce API interactions.
- Remains constant and unchangeable throughout the lifecycle of the attachment once created.
- |-
## IMPLEMENTATION GUIDANCE:
- Must strictly adhere to Salesforce ID formats: either 15-character case-sensitive or 18-character case-insensitive alphanumeric strings.
- Should be dynamically obtained from Salesforce API responses when creating or querying attachments to ensure accuracy.
- Avoid hardcoding or manually entering IDs to prevent data integrity issues and API errors.
- Validate the ID format programmatically before use to ensure compatibility with Salesforce API requirements.
- Prefer using the 18-character ID in integration scenarios to minimize case sensitivity issues.
- |-
## EXAMPLES:
- "00P1t00000XyzAbCDE"
- "00P1t00000XyzAbCDEAAA"
- "00P3t00000LmNOpQR1"
- |-
## IMPORTANT NOTES:
- The 15-character ID is case-sensitive, whereas the 18-character ID is case-insensitive and recommended for integration scenarios.
- Using an incorrect, malformed, or outdated ID will result in API errors or failed operations.
- The ID does not convey any descriptive information about the attachment’s content or metadata.
- Always retrieve and use the ID directly from Salesforce API responses to ensure validity and prevent errors.
- The ID is unique within the Salesforce org and cannot be reused or duplicated.
- |-
## DEPENDENCY CHAIN:
- Generated automatically by Salesforce upon creation of the attachment record.
- Utilized by other API calls and properties that require precise attachment identification.
- |-
Unique identifier for the attachment within the Salesforce system,
serving as the immutable primary key for the attachment object.
This ID is essential for accurately referencing, retrieving, updating,
or deleting the specific attachment record through API calls.
It must conform to Salesforce's standardized ID format,
which can be either a 15-character case-sensitive or an 18-character
case-insensitive alphanumeric string,
with the 18-character version preferred for integrations due to its case
insensitivity and reduced risk of errors.
While the ID itself does not contain metadata about the attachment's content,
it is crucial for linking the attachment to related Salesforce records and
ensuring precise operations on the attachment resource across the platform.
name:
type: string
description: Filename of the attachment, including the file extension (for example, `contract_agreement.pdf`).
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Stores the filename of the attachment as a string value.
- Acts as the key identifier for the attachment within the Salesforce environment.
- Displayed prominently in the Salesforce UI and API responses when viewing or managing attachments.
- Should be unique within the context of the parent record to avoid confusion.
- Includes file extensions (e.g., .pdf, .jpg, .docx) to denote the file type and facilitate correct processing.
- Case sensitivity may apply depending on the context, affecting retrieval and display.
- Used in URL generation and API endpoints, impacting accessibility and referencing.
- |-
## IMPLEMENTATION GUIDANCE:
- Validate filenames to exclude prohibited characters (such as \ / : * ? " < > |) and other special characters that could cause errors or security vulnerabilities.
- Ensure the filename length does not exceed Salesforce’s maximum limit, typically 255 characters.
- Always include the appropriate and accurate file extension to enable correct file type recognition.
- Use clear, descriptive, and consistent naming conventions that reflect the content or purpose of the attachment.
- Perform validation checks before upload to prevent failures or inconsistencies.
- Avoid renaming attachments post-upload to maintain integrity of references and links.
- Consider localization and encoding standards to support international characters where applicable.
- |-
## EXAMPLES:
- "contract_agreement.pdf"
- "profile_picture.jpg"
- "sales_report_q1_2024.xlsx"
- "meeting_notes.txt"
- "project_plan_v2.docx"
- |-
## IMPORTANT NOTES:
- Filenames are case-sensitive in certain contexts, which can impact retrieval and display.
- Avoid special characters that may interfere with URLs, file systems, or API processing.
- Renaming an attachment after upload can disrupt existing references or links to the file.
- |-
The name property represents the filename of the attachment within Salesforce
and serves as the primary identifier for the attachment.
It is crucial for enabling easy recognition, retrieval,
and management of attachments both in the user interface and through API operations.
This property typically includes the file extension,
which specifies the file type and ensures proper handling, display,
and processing of the file across different systems and platforms.
The filename should be meaningful, descriptive,
and concise to provide clarity and ease of use,
especially when managing multiple attachments linked to a single parent record.
Adhering to proper naming conventions helps maintain organization,
improves searchability,
and prevents conflicts or ambiguities that could arise from duplicate or unclear filenames.
parentId:
type: string
description: |-
Salesforce record the attachment is linked to, such as an Account, Contact, or Opportunity. Required when creating an attachment; access to the attachment follows the parent record's permissions.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Represents the Salesforce record ID that the attachment is associated with.
- Mandatory when creating or updating an attachment to specify its parent record.
- Must correspond to an existing Salesforce record ID of a valid object type that supports attachments.
- Enables querying, filtering, and reporting of attachments based on their parent record.
- Enforces referential integrity between attachments and parent records.
- Changes to the parent record, including deletion or modification, can affect the visibility, accessibility, and lifecycle of the attachment.
- Access to the attachment is governed by the permissions and sharing settings of the parent record.
- |-
## IMPLEMENTATION GUIDANCE:
- Ensure the parentId is a valid 15- or 18-character Salesforce record ID, with preference for 18-character IDs to avoid case sensitivity issues.
- Verify the existence and active status of the parent record before assigning the parentId.
- Confirm that the parent object type supports attachments to prevent errors during attachment operations.
- Handle user permissions and sharing settings to ensure appropriate access rights to the parent record and its attachments.
- Use Salesforce APIs or metadata services to validate the parentId format, object type, and existence.
- Implement robust error handling for invalid, non-existent, or unauthorized parentId values during attachment creation or updates.
- Consider the impact of parent record lifecycle events (such as deletion or archiving) on associated attachments.
- |-
## EXAMPLES:
- "0011a000003XYZ123" (Account record ID)
- "0031a000004ABC456" (Contact record ID)
- "0061a000005DEF789" (Opportunity record ID)
- |-
The unique identifier of the parent Salesforce record to which the attachment is linked.
This field establishes a direct and essential relationship between the
attachment and its parent object,
such as an Account, Contact, Opportunity,
or any other Salesforce record type that supports attachments.
It ensures that attachments are correctly associated within the Salesforce data model,
enabling accurate retrieval, management,
and organization of attachments in relation to their parent records.
By linking attachments to their parent records,
this field facilitates data integrity, efficient querying,
and seamless navigation between related records.
It is a mandatory field when creating or updating attachments and must reference a valid,
existing Salesforce record ID.
Proper use of this field guarantees referential integrity,
enforces access controls based on parent record permissions,
and impacts the visibility and lifecycle of the attachment within the Salesforce environment.
contentType:
type: string
description: MIME type of the attachment content, such as `application/pdf` or `image/png`.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Defines the media type of the attachment to guide processing, rendering, and handling decisions.
- Used by clients, servers, and intermediaries to determine appropriate actions such as inline display, download prompts, or specialized processing.
- Supports validation of file format during upload, storage, and download operations to ensure data integrity.
- Influences security measures including filtering, scanning for malware, and enforcing content policies.
- Affects user interface elements like preview generation, icon selection, and content categorization.
- May be used in logging, auditing, and analytics to track content types handled by the system.
- |-
## IMPLEMENTATION GUIDANCE:
- Must strictly adhere to the standard MIME type format as defined by IANA (e.g., "type/subtype").
- Should accurately reflect the actual content format to prevent misinterpretation or processing errors.
- Use generic types such as "application/octet-stream" only when the specific type cannot be determined.
- Include optional parameters (e.g., charset, boundary) when relevant to fully describe the content.
- Ensure the contentType is set or updated whenever the attachment content changes to maintain consistency.
- Validate the contentType against the file extension and actual content where feasible to enhance reliability.
- Consider normalizing the contentType to lowercase to maintain consistency across systems.
- Avoid relying solely on contentType for security decisions; combine with other metadata and content inspection.
- |-
## EXAMPLES:
- "image/jpeg"
- "application/pdf"
- "text/plain; charset=utf-8"
- "application/vnd.ms-excel"
- "audio/mpeg"
- "video/mp4"
- "application/json"
- "multipart/form-data; boundary=something"
- "application/zip"
- |-
The MIME type of the attachment content,
specifying the exact format and nature of the file (e.g., "image/png",
"application/pdf", "text/plain").
This information is critical for correctly interpreting, processing,
and displaying the attachment across different systems and platforms.
It enables clients and servers to determine how to handle the file,
whether to render it inline, prompt for download,
or apply specific processing rules.
Accurate contentType values facilitate validation, security scanning,
and compatibility checks,
ensuring that the attachment is managed appropriately throughout its lifecycle.
Properly defined contentType values improve interoperability, user experience,
and system security by enabling precise content handling and reducing the risk
of errors or vulnerabilities.
isPrivate:
type: boolean
description: |-
When true, the attachment is visible only to users explicitly authorized to access it; when false or omitted, it inherits the visibility of its parent record. This setting only restricts access further — it never grants access beyond the parent record's permissions.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- When set to true, the attachment is accessible only to users with explicit authorization, effectively hiding it from unauthorized users.
- When set to false or omitted, the attachment inherits the visibility of its parent record and is accessible to all users who have access to that record.
- Directly affects the attachment’s visibility scope and access control within Salesforce.
- Changes to this field immediately impact user access permissions for the attachment.
- Does not override broader organizational sharing settings but further restricts access.
- The privacy setting applies in conjunction with existing record-level and object-level security controls.
- |-
## IMPLEMENTATION GUIDANCE:
- Use this field to enforce strict data privacy and protect sensitive attachments in compliance with organizational policies and regulatory requirements.
- Ensure that user roles, profiles, and sharing rules are properly configured to respect this privacy setting.
- Validate that the value is a boolean (true or false) to prevent data inconsistencies.
- Evaluate the implications on existing sharing rules and record-level security before modifying this field.
- Consider implementing audit logging to track changes to this privacy setting for security and compliance purposes.
- Test changes in a sandbox environment to understand the impact on user access before deploying to production.
- Communicate changes to relevant stakeholders to avoid confusion or unintended access issues.
- |-
## EXAMPLES:
- `true` — The attachment is private and visible only to users explicitly authorized to access it.
- `false` — The attachment is public within the context of the parent record and visible to all users who have access to that record.
- |-
## IMPORTANT NOTES:
- Setting this field to true does not grant access to users who lack permission to view the parent record; it only further restricts access.
- Modifying this setting can significantly affect user access and should be managed carefully to avoid unintended data exposure or access denial.
- Some Salesforce editions or configurations may have limitations or specific behaviors regarding attachment privacy
- |-
Indicates whether the attachment is designated as private,
restricting its visibility exclusively to users who have been explicitly
authorized to access it.
This setting is critical for maintaining confidentiality and ensuring that
sensitive or proprietary information contained within attachments is accessible
only to appropriate personnel.
When marked as private,
the attachment’s visibility is constrained based on the user’s permissions and
the sharing settings of the parent record,
thereby reinforcing data security within the Salesforce environment.
This field directly controls the scope of access to the attachment,
dynamically influencing who can view or interact with it,
and must be managed carefully to align with organizational privacy policies and
compliance requirements.
description:
type: string
description: Free-text note describing the attachment's content or purpose, shown alongside the attachment details in Salesforce.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Optional and editable field used to describe the attachment’s nature or relevance.
- Supports plain text input including special characters, punctuation, and Unicode.
- Displayed prominently in user interfaces where attachment details appear.
- Utilized in search, filtering, sorting, and reporting functionalities to enhance data retrieval.
- Does not contain or affect the actual attachment content or file storage.
- |-
## IMPLEMENTATION GUIDANCE:
- Encourage concise yet informative descriptions to aid quick identification.
- Validate and sanitize input to prevent injection attacks and ensure data integrity.
- Support full Unicode to accommodate international characters and symbols.
- Enforce a reasonable maximum length (commonly 255 characters) to maintain performance and UI consistency.
- Avoid inclusion of sensitive or confidential information unless appropriate security controls are applied.
- Implement trimming or sanitization routines to maintain clean and consistent data.
- |-
## EXAMPLES:
- "Quarterly financial report for Q1 2024."
- "Signed contract agreement with client XYZ."
- "Product brochure for the new model launch."
- "Meeting notes and action items from April 15, 2024."
- "Technical specification document for project Alpha."
- |-
## IMPORTANT NOTES:
- This field contains descriptive metadata only, not the attachment content.
- Sensitive information should be excluded or protected according to security policies.
- Changes to this field do not impact the attachment file or its storage.
- Consistent and accurate descriptions improve attachment management and user experience.
- Plays a key role in enhancing searchability and filtering within Salesforce.
- |-
## DEPENDENCY CHAIN:
- Directly associated with the attachment entity in Salesforce.
- Often used in conjunction with other metadata fields such as attachment name, type, owner, size, and creation date.
- Supports comprehensive context when combined with related records and metadata.
- |-
A textual description providing additional context, detailed information,
and clarifying the content, purpose,
or relevance of the attachment within Salesforce.
This field enhances user understanding and facilitates easier identification,
organization,
and management of attachments by serving as descriptive metadata that
complements the attachment’s core data without containing the attachment content
itself.
It supports plain text with special characters, punctuation, and full Unicode,
enabling expressive, internationalized, and accessible descriptions.
Typically displayed in user interfaces alongside attachment details,
this description also plays a crucial role in search, filtering, sorting,
and reporting operations, improving overall data discoverability and usability.
- |-
## TECHNICAL DETAILS:
- Data type: string.
- Maximum length: typically 255 characters, configurable
description: |-
Salesforce Attachment record written by the import — a file linked to a parent record via `parentId`, with metadata such as name and content type.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Stores the binary content or a reference to a file linked to a specific Salesforce record.
- Supports multiple file formats including but not limited to documents, images, PDFs, spreadsheets, and email files.
- Enables attaching additional context, documentation, or evidence to Salesforce objects.
- Allows full lifecycle management: create, read, update, and delete operations on attachments.
- Contains metadata such as file name, MIME content type, file size, and the ID of the parent Salesforce record.
- Reflects changes immediately in the associated Salesforce record’s file attachments.
- Maintains a direct association with the parent record through the ParentId field to ensure accurate linkage.
- Supports integration with Salesforce Files (ContentVersion and ContentDocument) for advanced file management features.
- |-
## IMPLEMENTATION GUIDANCE:
- Encode file content using Base64 encoding when transmitting via Salesforce APIs to ensure data integrity.
- Validate file size against Salesforce limits (commonly up to 25 MB per attachment) and confirm supported file types before upload.
- Use the ParentId field to correctly associate the attachment with the intended Salesforce record.
- Implement robust permission checks to ensure only authorized users can upload, view, or modify attachments.
- Consider leveraging Salesforce Files (ContentVersion and ContentDocument objects) for enhanced file management capabilities, versioning, and sharing features, especially for new implementations.
- Handle error responses gracefully, including those related to file size limits, unsupported formats, or permission issues.
- Ensure metadata fields such as file name and content type are accurately populated to facilitate proper handling and display.
- When updating attachments, confirm that the new content or metadata aligns with organizational policies and compliance requirements.
- |-
The attachment property represents a file or document linked to a Salesforce record,
encompassing both the binary content of the file and its associated metadata.
It serves as a versatile container for a wide variety of file types—including documents,
images, PDFs, spreadsheets,
and emails—allowing users to enrich Salesforce records such as emails, cases,
opportunities,
or any standard or custom objects with supplementary information or evidence.
This property supports comprehensive lifecycle management,
enabling operations such as uploading new files,
retrieving existing attachments, updating file details,
and deleting attachments as needed.
By maintaining a direct association with the parent Salesforce record through
identifiers like ParentId,
it ensures accurate linkage and seamless integration within the Salesforce ecosystem.
The attachment property facilitates enhanced collaboration, documentation,
and data completeness by providing a centralized and manageable way to handle
files related to Salesforce data.
contentVersion:
type: object
properties:
contentDocumentId:
type: string
description: |-
ContentDocument this version belongs to; all versions of the same document share this ID. Assigned by Salesforce when the content version is created and immutable afterward.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Represents the unique ID of the parent Content Document to which the Content Version belongs.
- Acts as the primary link connecting multiple versions of the same document.
- Immutable after the Content Version record is created, preserving data integrity.
- Enables navigation from a specific Content Version to its parent Content Document.
- Automatically assigned and managed by Salesforce during Content Version creation.
- Essential for maintaining consistent version histories and document relationships.
- |-
## IMPLEMENTATION GUIDANCE:
- Use this ID to reference or retrieve the entire document across all its versions.
- Verify that the ID corresponds to an existing Content Document before usage.
- Utilize in SOQL queries and API calls to associate Content Versions with their parent document.
- Avoid modifying this field directly; it is managed internally by Salesforce.
- Employ this ID to support document lifecycle management, version tracking, and collaboration features.
- Leverage this field to ensure accurate linkage in integrations and custom development.
- |-
## EXAMPLES:
- "0691t00000XXXXXXAAA"
- "0692a00000YYYYYYBBB"
- "0693b00000ZZZZZZCCC"
- |-
## IMPORTANT NOTES:
- Distinct from the Content Version ID, which identifies a specific version rather than the entire document.
- Mandatory for any valid Content Version record to ensure proper document association.
- Cannot be null or empty for Content Version records.
- Critical for preserving document integrity, version tracking, and enabling collaborative document management.
- Changes to this ID are not permitted post-creation to avoid breaking version linkages.
- Plays a key role in Salesforce’s document sharing, access control, and audit capabilities.
- |-
## DEPENDENCY CHAIN:
- Requires the existence of a valid Content Document record in Salesforce.
- Directly related to Content Version records representing different iterations of the same document.
- |-
contentDocumentId is the unique identifier for the parent Content Document
associated with a specific Content Version in Salesforce.
It serves as the fundamental reference that links all versions of the same document,
enabling comprehensive version control and efficient document management within
the Salesforce ecosystem.
This ID is immutable once the Content Version is created,
ensuring a stable and consistent connection across all iterations of the document.
By maintaining this linkage,
Salesforce facilitates seamless navigation between individual document versions
and their overarching document entity,
supporting streamlined retrieval, updates, collaboration,
and organizational workflows.
This field is automatically assigned by Salesforce during the creation of a
Content Version and is critical for maintaining the integrity and traceability
of document versions throughout their lifecycle.
title:
type: string
description: Display title of this content version, shown in content libraries, search results, and version history.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Functions as the main identifier or label for the content version in user interfaces, listings, and reports.
- Enables users to distinguish between different versions of the same content efficiently.
- Reflects the content’s subject, purpose, key attributes, or version status.
- Should be concise but informative to facilitate quick recognition and comprehension.
- Commonly displayed in search results, filters, version histories, and content libraries.
- Supports sorting and filtering operations to streamline content management.
- |-
## IMPLEMENTATION GUIDANCE:
- Use clear, descriptive titles that accurately represent the content version’s focus or changes.
- Avoid special characters or formatting that could cause display, parsing, or processing issues.
- Keep titles within a reasonable length (typically up to 255 characters) to ensure readability across various UI components and devices.
- Update the title appropriately when creating new versions to reflect significant updates or revisions.
- Consider including version numbers, dates, or status indicators to enhance clarity and version tracking.
- Follow organizational naming conventions and standards to maintain consistency.
- |-
## EXAMPLES:
- "Q2 Financial Report 2024"
- "Product Launch Presentation v3"
- "Employee Handbook - Updated March 2024"
- "Marketing Strategy Overview"
- "Website Redesign Proposal - Final Draft"
- |-
## IMPORTANT NOTES:
- Titles do not need to be globally unique but should be meaningful and contextually relevant.
- Changing the title does not modify the underlying content data or its integrity.
- Titles are frequently used in search, filtering, sorting, and reporting operations within Salesforce.
- Consistent naming conventions and formatting improve content management efficiency and user experience.
- Consider organizational standards or guidelines when defining title formats.
- Titles should avoid ambiguity to prevent confusion between similar content versions.
- |-
## DEPENDENCY CHAIN:
- Part of the ContentVersion object within Salesforce’s content management framework.
- Often associated with other metadata fields such as description
- |-
The title of the content version serves as the primary name or label assigned to
this specific iteration of content within Salesforce.
It acts as a key identifier that enables users to quickly recognize,
differentiate, and manage various versions of content.
The title typically summarizes the subject matter, purpose,
or significant details about the content,
making it easier to locate and understand at a glance.
It should be concise yet sufficiently descriptive to effectively convey the
essence and context of the content version.
Well-crafted titles improve navigation, searchability,
and version tracking within content management workflows,
enhancing overall user experience and operational efficiency.
pathOnClient:
type: string
description: |-
Original file path on the client machine before upload. Used for reference and display only; it does not affect how the file is stored or accessed in Salesforce.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Represents the full absolute or relative file path on the client device where the file was stored prior to upload.
- Automatically populated during file upload processes when the client environment provides this information.
- Used mainly for reference, auditing, and user interface display to provide context about the file’s origin.
- May be empty, null, or omitted if the file was not uploaded from a client device or if the path information is unavailable or restricted.
- Does not influence file storage, retrieval, or access permissions within Salesforce.
- Retains the original formatting and syntax as provided by the client system at upload time.
- |-
## IMPLEMENTATION GUIDANCE:
- Capture the file path exactly as provided by the client system at the time of upload, preserving the original format and case sensitivity.
- Handle differences in file path syntax across operating systems (e.g., backslashes for Windows, forward slashes for Unix/Linux/macOS).
- Treat this property strictly as metadata for informational purposes; avoid using it for security, access control, or file retrieval logic.
- Sanitize or mask sensitive directory or user information when displaying or logging this path to protect user privacy and comply with data protection regulations.
- Ensure compliance with relevant privacy laws and organizational policies when storing or exposing client file paths.
- Consider truncating, normalizing, or encoding paths if necessary to conform to platform length limits, display constraints, or to prevent injection vulnerabilities.
- Provide clear documentation to users and administrators about the non-security nature of this property to prevent misuse.
- |-
## EXAMPLES:
- "C:\\Users\\JohnDoe\\Documents\\Report.pdf"
- "/home/janedoe/projects/presentation.pptx"
- "Documents/Invoices/Invoice123.pdf"
- "Desktop/Project Files/DesignMockup.png"
- "../relative/path/to/file.txt"
- |-
The pathOnClient property specifies the original file path of the content as it
exists on the client machine before being uploaded to Salesforce.
This path serves as a precise reference to the source location of the file on
the user's local system,
facilitating identification, auditing, and tracking of the file's origin.
It captures either the full absolute path or a relative path depending on the
client environment and upload context,
accurately reflecting the exact location from which the file was sourced.
This information is primarily used for informational, diagnostic,
and user interface purposes within Salesforce and does not influence how the file is stored,
accessed, or secured in the system.
tagCsv:
type: string
description: |-
Comma-separated tags applied to the content version for categorization and search. Updating this field replaces the entire tag set, so include every tag the version should keep.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Accepts multiple tags separated by commas, typically without spaces, though trimming whitespace is recommended.
- Tags are used to categorize, organize, and improve the discoverability of content versions.
- Tags can be added, updated, or removed by modifying the entire string value.
- Duplicate tags within the string should be avoided to maintain clarity and effectiveness.
- Tags are generally treated as case-insensitive for search purposes but stored exactly as entered.
- Changes to this field immediately affect how content versions are indexed and retrieved in search operations.
- |-
## IMPLEMENTATION GUIDANCE:
- Ensure individual tags do not contain commas to prevent parsing errors.
- Validate the tagCsv string for invalid characters, excessive length, and proper formatting.
- When updating tags, replace the entire string to accurately reflect the current tag set.
- Trim leading and trailing whitespace from each tag before processing or storage.
- Utilize this field to enhance search filters, categorization, and content management workflows.
- Implement application-level checks to prevent duplicate or meaningless tags.
- Consider standardizing tag formats (e.g., lowercase) to maintain consistency across records.
- |-
## EXAMPLES:
- "marketing,Q2,presentation"
- "urgent,clientA,confidential"
- "releaseNotes,v1.2,approved"
- "finance,yearEnd,reviewed"
- |-
## IMPORTANT NOTES:
- The maximum length of the tagCsv string is subject to Salesforce field size limitations.
- Tags should be meaningful, consistent, and standardized to maximize their utility.
- Modifications to tags can impact content visibility and access if used in filtering or permission logic.
- This field does not enforce uniqueness of tags; application-level logic should handle duplicate prevention.
- Tags are stored as-is; any normalization or case conversion must be handled externally.
- Improper formatting or invalid characters may cause errors or unexpected behavior in search and filtering.
- |-
## DEPENDENCY CHAIN:
- Closely related to contentVersion metadata and search/filtering functionality.
- |-
A comma-separated string representing the tags associated with the content
version in Salesforce.
This field enables the assignment of multiple descriptive tags within a single string,
facilitating efficient categorization, organization,
and enhanced searchability of the content.
Tags serve as metadata labels that help users filter, locate,
and manage content versions effectively across the platform by grouping related
items and supporting advanced search and filtering capabilities.
Proper use of this field improves content discoverability,
streamlines content management workflows,
and supports dynamic content classification.
contentLocation:
type: string
description: Specifies where the content file for this ContentVersion record is stored — "S" for Salesforce internal storage or "E" for an external system (the two values the import form offers). Salesforce also defines "L" for content shared via a link, but the form does not set it. Typically assigned automatically based on the upload method.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR:
- Determines the origin and storage context of the content file linked to the ContentVersion record.
- Influences the mechanisms and protocols used for accessing, retrieving, and managing the content.
- Typically assigned automatically by Salesforce based on the upload method or integration configuration.
- Common values include 'S' for Salesforce internal storage, 'E' for external storage systems, and 'L' for linked external locations.
- Often read-only to preserve data integrity and prevent unauthorized or unintended modifications.
- Alterations to this field can impact content visibility, sharing settings, and delivery workflows.
- |-
## IMPLEMENTATION GUIDANCE:
- Utilize this field to programmatically distinguish between internally stored content and externally managed files.
- When integrating with external repositories, ensure contentLocation accurately reflects the external storage to enable correct content handling and retrieval.
- Avoid manual updates unless supported by Salesforce documentation and accompanied by a comprehensive understanding of the consequences.
- Validate the contentLocation value prior to processing or delivering content to guarantee proper routing and access control.
- Confirm that external storage systems are correctly configured, authenticated, and authorized to maintain uninterrupted access.
- Monitor this field closely during content migration or integration activities to prevent data inconsistencies or access disruptions.
- |-
## EXAMPLES:
- 'S' — Content physically stored within Salesforce’s internal storage infrastructure.
- 'E' — Content residing in an external system integrated with Salesforce, such as a third-party content repository.
- 'L' — Content located in a linked external location, representing specialized or less common storage configurations.
- |-
## IMPORTANT NOTES:
- Manual modifications to contentLocation can lead to broken links, access failures, or data inconsistencies.
- Salesforce generally manages this field automatically to uphold consistency and reliability.
- The contentLocation value directly influences content delivery methods and access control policies.
- External content locations require proper configuration, authentication, and permissions to function correctly
- |-
contentLocation specifies the precise storage location of the content file
associated with the Salesforce ContentVersion record.
It identifies where the actual file data resides—whether within Salesforce's
internal storage infrastructure,
an external content management system,
or a linked external repository—and governs how the content is accessed,
retrieved, and managed throughout its lifecycle.
This field plays a crucial role in content delivery, access permissions,
and integration workflows by clearly indicating the source location of the file data.
Properly setting and interpreting contentLocation ensures that content is
handled correctly according to its storage context,
enabling seamless access when stored internally or facilitating robust
integration and synchronization with third-party external systems.
description: |-
Salesforce ContentVersion record written by the import — a specific version of a content item in Salesforce Files. Each modification produces a new version, and all versions of a document are linked through `contentDocumentId`.
# Shape → requireds. Nested if/else-if (no allOf), stack-style.
# Branch 1: blob configs (attachment / contentVersion) — operation only; no api, and
# the operation→idLookup/upsert rules do not apply (the blob record carries its own
# identifiers; the current server accepts blob upserts without them).
# Branch 2: server-written bookkeeping stubs — no requireds. Stub keys are {lookups}
# plus `api`, which the server defaults to "rest" on some stubs, so its presence
# alone does not indicate a user-authored config.
# Branch 3: real record configs — operation + api, then operation → matching config
# (update/delete/addupdate share one branch because their requirements are identical).
if:
anyOf:
- required: [attachment]
- required: [contentVersion]
then:
required:
- operation
else:
if:
not:
propertyNames:
enum: [lookups, api]
then:
required:
- operation
- api
if:
properties:
operation:
const: upsert
required:
- operation
then:
required:
- idLookup
- upsert
properties:
idLookup:
required:
- extract
upsert:
required:
- externalIdField
else:
if:
properties:
operation:
enum: [update, delete, addupdate]
required:
- operation
then:
required:
- idLookup
properties:
idLookup:
required:
- whereClause
else:
if:
properties:
operation:
const: upsertpicklistvalues
required:
- operation
then:
required:
- upsertpicklistvalues
properties:
upsertpicklistvalues:
required:
- fullName
references/schemas/tool.yml
Tool:
type: object
description: |-
Configuration for a `ToolImport`, which executes a reusable Celigo Tool resource as the
import action. The referenced tool defines its own input contract, processing pipeline, and
output; this object binds the tool to the import, along with the connections the tool's steps
use for this invocation.
x-celigo-ai-guidance:
- |-
A ToolImport is the **wrapper** that lets a flow / API / another Tool consume a
Celigo Tool resource as a step.
It carries a pointer to the underlying Tool plus per-consumer connection bindings.
- |-
On the wire,
the wrapper IS a regular import — the pageProcessor entry that references it is just `{type:
"import", _importId: <wrapper>}`.
The `adaptorType:
"ToolImport"` discriminator and this `tool` block are how the platform knows to
invoke the underlying Tool at runtime instead of running a normal import.
- |-
The Celigo Tool itself is a separate resource (lives at `/v1/tools/{_toolId}`)
and can be reused by many wrappers — the wrapper is consumer-specific (which
connections to bind,
which upstream-record-to-input mappings to apply); the Tool is shared.
Modify the Tool through `/v1/tools` directly;
modify the wrapper through `/v1/imports`.
required:
- _toolId
properties:
_toolId:
type: string
format: objectId
x-celigo-refModel: tools
description: |-
Tool resource this import executes. The tool supplies the input schema, routing, page
processors, and output mapping; the import runs that tool when records reach it. Required —
a `ToolImport` has no behavior without a tool to run.
x-celigo-ai-guidance:
- Reference to the Celigo Tool resource this wrapper invokes.
- |-
Required.
The Tool must already exist;
the wrapper has nothing to do without a real `_toolId` to point at.
To change which Tool the wrapper invokes,
update this field — the wrapper itself stays the same shape,
just points at a different Tool.
- |-
Writing the wrapper does not create or change the Tool itself;
author the Tool as its own resource through `/v1/tools`.
examples: ["69ce8ae7cbbb51405ae42ae5"]
overrides:
type: object
description: |-
Per-import overrides applied to the referenced tool at execution time. Tools use a
bring-your-own-keys connection model: the tool declares abstract connection slots, and the
consumer binds concrete connections here so one tool definition can run against different
systems without being modified. Bind every connection the tool's steps use; to keep the
connection a step was authored against, bind it to itself.
x-celigo-ai-guidance:
- |-
Per-consumer overrides applied when the wrapper invokes the underlying Tool.
Currently only carries connection bindings — the underlying Tool's lookups and
imports each reference a `_connectionId`,
and consumers bind real account connections to those abstract requirements here.
- |-
The save accepts a wrapper without bindings, but when the Tool's steps use connections
its run then fails with `connection_not_overridden`. Write one `connections` entry per
connection the Tool's steps use, even when the wrapper keeps the Tool's authored
connection (`_id` equal to `_abstractId`); a Tool whose steps use no connection needs none.
properties:
connections:
type: array
description: |-
Connection bindings that replace the tool's abstract connection slots for this import.
Each entry maps one of the tool's abstract connections to the concrete connection the
import should use.
x-celigo-ai-guidance:
- |-
Connection-binding overrides.
One entry per abstract connection requirement the Tool exposes.
- |-
**Discovering abstract requirements.** The Tool resource does not carry an
explicit "abstract requirements" block.
Walk the Tool's `routers[].branches[].pageProcessors[]` array,
fetch each underlying export / import resource,
and collect the unique set of `_connectionId` values.
Each unique connection id is one abstract requirement;
the connection's `type` field is its adaptor type (`salesforce`, `netsuite`,
`http`, etc.).
- |-
**Resolving each binding.** For each abstract id,
find a connection in the consumer's account whose adaptor type matches.
When multiple match,
the user prompt / context disambiguates (production vs. sandbox, etc.).
When no match is found,
fall back to reusing the abstract id as the bound id — that means "use the
connection the Tool was authored against,
as-is" and is the right fallback when the Tool author and the consumer share a connection.
items:
type: object
required:
- _id
- _abstractId
properties:
_id:
type: string
format: objectId
x-celigo-refModel: connections
description: Concrete connection bound to the abstract slot for this
import's tool run.
x-celigo-ai-guidance:
- |-
The real account connection the wrapper binds to this abstract requirement.
May equal `_abstractId` when the wrapper inherits the Tool's authored connection.
examples: ["698af29c387b80d0b91659ad"]
_abstractId:
type: string
format: objectId
x-celigo-refModel: connections
description: |-
Abstract connection slot, declared by the tool, that this binding replaces. Matches
the abstract connection id defined on the referenced tool.
x-celigo-ai-guidance:
- |-
The abstract connection id the Tool's underlying exports / imports reference.
Discovered by walking the Tool's pageProcessors as described above.
examples: ["698af29c387b80d0b91659ad"]
_borrowConcurrencyFromConnectionId:
type: string
format: objectId
x-celigo-refModel: connections
description: |
Optional pointer to another connection whose
concurrency budget this binding shares. Use
when several wrappers / steps should share rate
limits against the same downstream system but
use different credentials. Mutually exclusive
with `_id` on the same entry.
references/schemas/topic.yml
Topic:
type: object
description: |-
Publishing settings for an Event Streams publisher (`adaptorType: TopicImport`). Each
incoming record is published as a message to the topic; the platform delivers it
internally, so the import carries no `_connectionId`. Present when `adaptorType` is
`TopicImport`; the server fills `batchSize` with its default when it is omitted.
x-celigo-ai-guidance:
- |-
Sending this object without `adaptorType` is enough — the server infers `TopicImport`
from it. Never add a `_connectionId`: the server then validates the document as that
connection's adaptor instead (an http connection fails with 422 missing_required_field
"http subschema not defined").
- |-
Without a mapping, each message is the record as the step received it. With
`oneToMany: true` and `pathToMany`, every element of that array is published as its own
message and the rest of the parent record rides inside each one under `_PARENT` (the parent
without the array); a mapping reshapes the message instead.
- |-
Publishing is not limited to flows. Verified surfaces: a flow step; an API step (the API
invoked at `POST /apis/{version}{relativeURI}`); a Tool reached through a `ToolImport`
step in a flow; and `POST /v1/imports/{_id}/invoke` with `{"data": [record, ...]}`. A flow
or API test run (`/test/run`) publishes nothing — import steps are mocked there.
properties:
_topicId:
type: string
format: objectId
x-celigo-refModel: topics
x-celigo-ui-override: >-
Required by the Event Streams publisher form (fieldDefinitions/resources/imports/import.js:
'topic._topicId' is required:true, validWhen isNot ''). The API saves a TopicImport
without it, so the requirement is encoded on the request schema to mirror the form and
keep publishers bound to a topic.
description: |-
Topic the records are published to. Create or look it up with `GET /v1/topics`; the
topic must belong to the same account. A clone of the import made in the same
environment keeps `_topicId`.
x-celigo-ai-guidance:
- |-
An id that is not a topic in the account fails the save with 422 `invalid_ref`
("<id> is not a valid topic._topicId").
examples: ["6ab1f48d38d7733b4599875c"]
batchSize:
type: integer
minimum: 1
default: 1000
description: |-
Number of records published per request to the topic. Lower it when individual
messages are large; the default keeps publishing efficient for typical record sizes.
x-celigo-ai-guidance:
- |-
Zero and negative values fail the save with 422 `min` ("Path `topic.batchSize` (0) is
less than minimum allowed value (1)").
examples: [1000, 100]
references/schemas/wrapper.yml
Wrapper:
type: object
description: |-
Configuration for Wrapper imports, which delegate writing records to custom connector code
(typically a stack-hosted function) rather than a built-in adaptor. Required when the
_connectionId field references a wrapper connection.
x-celigo-ai-guidance:
- |-
Configuration for WrapperImport resources — imports backed by a user-supplied
"wrapper" JavaScript module that invokes a custom function (typically to
integrate with a system Celigo doesn't have a first-class connector for).
- |-
Wrapper imports use Mapper 1.0 for field mappings and the generic Celigo lookup shape:
mappings live in `resource.mapping` (fields/lists) and lookups mirror at
`resource.lookups` and `resource.wrapper.lookups`.
# `function` is required on real configs only (see the if/then at the end of this
# schema): the server writes stub `wrapper` objects ({lookups:[]} / {}) onto imports
# of other adaptorTypes as bookkeeping.
properties:
function:
type: string
x-celigo-ui-override: >-
Required by the wrapper import form (wrapper.function is required:true). Encoded
to mirror
the form so builders produce connectable configurations.
description: |-
Name of the function the wrapper invokes to process records.
Must match a callable function in the wrapper's execution context; names are case-sensitive.
x-celigo-ai-guidance:
- |-
Name of the function to invoke on the wrapper module.
Must match an exported function on the connection's wrapper script.
The wrapper receives the import's `configuration` object and the mapped
record(s) and returns the response the import records downstream.
examples: ["importItem", "createFulfillmentsForPending", "sendMessage"]
configuration:
type: object
additionalProperties: true
description: |-
Free-form settings passed to the wrapper function at runtime (connector-specific keys such
as method, apiVersion, headers, or handler). Structure is defined by the wrapper code, not
by this schema.
x-celigo-ai-guidance:
- |-
Opaque configuration object passed to the wrapper `function` at runtime.
Shape is defined entirely by the wrapper author — Celigo does not interpret or
validate these fields.
Use for endpoint URLs, feature flags, batch-size hints,
or any other settings the wrapper needs beyond the connection-level credentials.
lookups:
x-celigo-agent:
- lookup
$ref: ../common-schemas/lookups.yml#/Lookups
description: |-
Lookup definitions for this wrapper import. Wrapper lookups use
the generic Celigo shape: either a static `map` (key→value
object) or a dynamic `_lookupCacheId` + `extract` pair. See
`common/schemas/lookups.yml` for the full item shape.
x-celigo-ai-guidance:
- |-
## Mirror invariant
The Celigo platform persists lookups in TWO locations on a
Wrapper import: top-level `lookups` and `wrapper.lookups`.
Both arrays must hold byte-for-byte identical content, so write
both on every save.
# Requireds apply only to real configs, not server-written bookkeeping stubs whose
# keys are all in {lookups}. Stack-style conditional (no allOf).
if:
not:
propertyNames:
enum: [lookups]
then:
required:
- function
SKILL.md
---
name: configuring-imports
description: Configure Celigo imports -- the destination step that writes records to external systems. Use when creating or editing imports, choosing the adaptor type, setting up lookups, upsert or exactly-once writes, or file-based imports. For mapping rules, use writing-mappings; for AI agent steps, configuring-ai-agents.
---
<!-- TIER:1 -->
# Configuring Imports
An import is the **data destination** in a Celigo integration. It takes records from an upstream step and writes them to an external system -- REST APIs, databases, ERPs, file servers, or AI models. Every import has one adaptor type and, apart from the connection-less kinds in [Minimum Required Fields](#minimum-required-fields), is bound to exactly one connection.
Imports handle six concerns:
- **Field mapping** -- transforming source fields into the destination system's expected format (including value resolution via static maps and lookup tables). Uses Mapper 2.0 (`mappings[]` array) by default, including `NetSuiteHTTPImport`. `SalesforceImport` and `WrapperImport` support only Mapper 1.0 (`mapping.fields[]` / `mapping.lists[]`), and `NetSuiteDistributedImport` uses its own Mapper 1.0 dialect at `netsuite_da.mapping`
- **Operation logic** -- create, update, upsert, delete, attach/detach
- **Hooks** -- JavaScript pre/post processing at various pipeline stages (preMap, postMap, postSubmit). File-based imports that generate files from records also support postAggregate; an HTTP records import refuses it (422 `invalid_hook`)
- **One-to-many** -- fan out a child array already on the input record. Set `oneToMany: true` and `pathToMany` to that path (e.g., `"lineItems"`) when one source record should create one import operation per child. After fan-out the child is `record` (`{{record.sku}}`) and the parent (without the array) is `record._PARENT` (`{{record._PARENT.id}}`). Leave `pathToMany` blank when the arriving record is itself an array (grouped rows from a `groupByFields` export or a file group) -- each element then becomes the record, with no `_PARENT`. A path on an array record processes zero children and still reports success; a blank path on an object record fails with `oneToMany_error`. `oneToMany` doesn't batch by itself: templates see `batch_of_records` only when `http.batchSize` is above 1 (the default varies by API) or an RDBMS query runs `per_page`
- **Response mapping** -- extract fields from the import's API response back into the record for downstream steps. Configured on the flow's `pageProcessors[]` entry, but planned when building the import. Extracts read the response envelope: `id` (the destination-assigned record ID), `statusCode`, `ignored`, `errors`, `_headers`, `dataURI` (the record's link, when the import sets `dataURITemplate`), and `_json` (the response body after any response transform, whose top-level fields are also flattened onto the envelope). Use `id` for a created record's ID and `_json.fieldName` for other response fields (e.g., `_json.output.1.content.0.text` for OpenAI responses) -- `_json.id` works only when the body itself has a top-level `id`. Flow test runs never send imports, so `id` is empty there. Response mapping uses Transformation 1.0 syntax (extract/generate pairs), not the newer expression-based transforms
- **postResponseMap hook** -- JavaScript processing after response mapping merges the response back into the record. Configured on the flow's `pageProcessors[]` entry, but planned when building the import. Use to transform or enrich the merged record before downstream steps
Imports are used across flows, APIs, and tools.
## Import Execution Pipeline
When records arrive at an import step, this pipeline executes in strict order:
1. **Input filter** (optional) -- records that don't match skip this import (see [Gotchas](#gotchas)). Configured as `filter` on the import resource itself -- an `inputFilter` key on an import is silently discarded. See [configuring-filters](../configuring-filters/SKILL.md#import-filter-import-filter)
2. **preMap hook** (optional) -- JavaScript processing before field mapping
3. **Field mapping** -- Mapper 2.0 or 1.0 maps source fields to destination fields, including lookups and hardcoded values
4. **postMap hook** (optional) -- JavaScript processing after field mapping, before submission
5. **Submit to destination** -- writes the mapped record to the external system
6. **postSubmit hook** (optional) -- JavaScript processing after the destination responds (access response data, log results, trigger side effects)
7. **Response transform** (optional, `responseTransform`) -- reshapes the response body (`_json`) after postSubmit. Its expression rules are Transformation 1.0 only (`expression.version: "1"`; `"2"` is refused with 422), and it keeps only the fields it maps, so a field it drops is gone for response mapping. Its execution-log stage is `responseTransformation`
8. **Response mapping** (optional) -- carries data from the destination response back into the record for downstream steps. Configured on the flow's `pageProcessors[]` entry, not on the import itself
9. **postResponseMap hook** (optional) -- JavaScript processing after response mapping merges response data back into the record
**Key distinction:** Response mapping lives on the flow's `pageProcessors[]` entry, not on the import resource. preMap, the mapping and postMap shape only the request this import sends: the next step receives the record as it arrived plus what response mapping and postResponseMap add, so a field added in steps 2-4 never reaches it. When building an import that needs to pass data downstream, plan the response mapping at flow design time.
**What an HTTP import's templates see.** `http.relativeURI` and header values render against the record before mapping, and an undefined path fails the request: a field the record lacks in the URI -- or a `{{lookup.<name>}}` whose lookup found nothing -- fails the record before anything is sent, with `handlebars_template_parse_error` (`Failed to generate request url from template: … Details: "<field>" not defined in the model.`). That is missing data, not a template bug, so a retry fails the same way; an update import that some records reach without the destination id needs a composite UPDATE/CREATE or `ignoreMissing` (Common Errors). A field holding `""` is defined: it renders empty, and the request goes out without the id. A field holding null renders the text `null` instead (the request goes to `…/null`), and a field under an object that is null fails the record like a missing one (`"<field>" not defined in null`) -- see writing-handlebars, gotcha 16. `http.body` renders against the mapped record, where an undefined path renders empty and the record still succeeds (`strictHandlebarEvaluation` doesn't change that). Leave `http.body` unset to send the mapped record; when you template it, use the mapped (destination) field names, and double braces for string values in a JSON body (gotcha 11).
## Categories of Import
### Record-Based Imports
Submit structured records to APIs, databases, or ERPs. The vast majority of imports.
- `NetSuiteDistributedImport` -- high-performance SuiteApp writes (add, update, addupdate, delete, attach, detach). The default for NetSuite when the Celigo SuiteApp or bundle is installed. `netsuite_da.restletVersion` is fixed when the step is created, so author new steps on `suiteapp2.0`; a hook's SuiteScript version must match the step's
- `NetSuiteHTTPImport` -- NetSuite's REST APIs on a `netsuite` connection (`nsDomainType: "suitetalk"`: `POST /services/rest/record/v1/<type>` to insert, `PATCH .../<type>/{id}` to update; `"restlet"` for a custom RESTlet). Use it when neither the SuiteApp nor the bundle is installed, or the user asks for REST
- `HTTPImport` -- REST/GraphQL APIs (POST, PUT, PATCH, DELETE). Supports connector-assisted (`formType: "assistant"`) and GraphQL (`graph_ql`, `assistant_graphql`) modes; see [GraphQL imports](#graphql-imports)
- `SalesforceImport` -- Salesforce writes through `salesforce.api`: `soap` (the default), `rest`, or `compositerecord` (Composite Graph, a tree of related records in one request); `metadata` deploys configuration, not records. `salesforce.api` has no Bulk value
- `RDBMSImport` -- SQL databases (Snowflake, PostgreSQL, MySQL, SQL Server, Oracle). `rdbms.queryType` is a one-element array whose value fixes the companion field: `per_record`, `per_page` (renders against `batch_of_records`) and `first_page` (runs once, on the first page) take `rdbms.query`, an array of SQL strings; `bulk_insert` takes `rdbms.bulkInsert` (pure INSERT, with no upsert or ignore-existing logic); `bulk_load` takes `rdbms.bulkLoad` (a staged load that becomes a MERGE with `primaryKeys`, or runs your own SQL with `overrideMergeQuery`). Prefer `bulk_load` where the database supports it, then `bulk_insert` for pure INSERTs, then `per_page`, and `per_record` only when the logic can't be batched. The legacy uppercase `INSERT` (and the retired `MERGE`, `COPY` and `BULK INSERT`) fail any save with 422 "Import query types … are deprecated"; `UPDATE` is legacy too, so don't use it. Other strings, such as lowercase `insert`, fail as an invalid enum value. A `bulk_load` import's GET also returns a `file` block describing its staging file; the platform derives it on every read, so a save that changes it is accepted and discarded. The SQL itself: [writing-sql](../writing-sql/SKILL.md)
- `MongodbImport`, `DynamodbImport`, `JDBCImport` -- other databases
#### GraphQL imports
A GraphQL import is an `HTTPImport` with `http.formType` `graph_ql` (or `assistant_graphql`, the connector-assisted GraphQL form). Each `http.body[]` entry is a string holding the `{"query", "operationName", "variables"}` envelope, one per operation on an update/create pair -- never an object, which is stored as-is and doesn't render -- and the mutation's input comes from the record through Handlebars in `variables`.
- **Endpoint.** The import posts to the connection's base URI -- its form has no relative-URI field and writes `relativeURI: ["/"]` on every save -- so the connection must supply the GraphQL endpoint: a `graph_ql` connection, or a connector connection bound to the app's GraphQL API. On a connection bound to the app's REST API every record fails (HTTP 406 on Shopify); add a second connection bound to the GraphQL API, re-point the GraphQL steps to it, and leave the shared REST connection unchanged. The form type is set on the step, so a plain `http` connection whose base URI is the endpoint works too.
- **Variables.** Under `graph_ql` or `assistant_graphql`, a `variables` value stored as a JSON string is sent as an object; under `http` it goes out as a string. An object takes expressions inside string values only: `"variables": {"input": {"email": "{{record.email}}"}}`. A string can also carry bare values, but its string values sit two JSON levels deep, so give each one `jsonEncode` in double braces: `"variables": "{\"input\": {\"note\": \"{{jsonEncode record.note}}\", \"quantity\": {{record.qty}} } }"`. An object or array goes in as `{{jsonSerialize record.items}}`, also in double braces. With `{{record.note}}`, or triple braces on either helper, a value holding a quote fails its record with 422 `invalid_json` before anything is sent.
- **Errors come back as HTTP 200.** A GraphQL server usually answers a failed mutation with 200 and an `errors` array, and with no response settings the import counts the record as written. Set `http.response.resourceIdPath` to the id only a successful mutation returns (`["data.customerCreate.customer.id"]`): any other answer then fails the record with `missing_id`, top-level `errors` and the mutation's own `userErrors` alike. For the server's message on top-level errors, also set `failPath: ["errors[0].message"]` and `errorPath: "errors[0].message"`; `failPath: ["errors"]` catches a response with exactly one error and lets two or more through.
### File-Based Imports
Write or upload files to remote storage. Require the `file{}` configuration block. Two modes:
- **Record-to-file** -- aggregates incoming records into a file (CSV, JSON, XML, XLSX). The `file{}` block defines the output format, and `file.aggregation.type` how a run's records become files: `all` (one file per run), `page` (one per page) or `key` (EDI file definitions only, with B2B Manager). Send the legacy `file.skipAggregation` as its pair -- `true` with `page`, `false` with `all` or `key` -- because validation reads the value you send: an EDI file definition on `page` without `true`, or a `file.batchSize` without it, fails with 422 `not_supported`. The format is `file.type` (`csv`, `json`, `xml`, `xlsx`), with its settings under that name: a CSV's `file.csv.includeHeader` defaults to `true`, so set it `false` for a file with no header row. The columns are the import's `mappings` rows, in order, each row's `generate` naming its column; there is no `file.columns`.
- **Blob passthrough** -- transfers a file as-is from an upstream blob export or blob lookup. Set `blob: true`, and `blobKeyPath` to the record field holding the `blobKey` that step put there (FTP and Google Drive imports fall back to the record's root `blobKey` when it's unset). Base64 content in a record is record data, not a blobKey. AI agent, guardrail and tool imports reject both fields with 422 `not_supported`.
Adaptor types:
- `HTTPImport` with `http.type: "file"` -- the folder mode of the cloud file providers (Google Drive, Box, Dropbox, Azure Blob Storage, Google Cloud Storage, Azure Data Lake Storage Gen2, Celigo Storage). Address the folder by path in `http.relativeURI` (a folder, never an API endpoint) or by the provider's folder id in `file.directory` (`pathMode: "directoryId"`), and name the file in `file.fileName`; provider-specific location rules are in [file.yml](references/schemas/file.yml)
- `FTPImport` -- CSV, XML, JSON, XLSX, EDI files to FTP/SFTP. The folder goes in `ftp.directoryPath` and the file's name in `ftp.fileName`, which takes no directory separators (the UI stores the same name in `file.fileName` too); a dated name is a Handlebars template ([writing-handlebars > File paths and names](../writing-handlebars/SKILL.md#file-paths-and-names))
- `S3Import` -- objects to Amazon S3
- `AS2Import` -- AS2 EDI file transmission
- `FileSystemImport` -- local/on-premise filesystem writes
### AI Imports
Invoke AI models for classification, extraction, or safety checks. No `_connectionId` required unless using BYOK.
- `AiAgentImport` -- LLM invocations (any provider in the `aiAgent.provider` enum) with structured output, tool use, and reasoning
- `GuardrailImport` -- PII detection, content moderation, or custom AI-based validation
### Stack and Tool Imports
- `WrapperImport` -- custom connector code, typically a function on a stack (`wrapper.function`)
- `ToolImport` -- runs the Celigo Tool named by `tool._toolId`. It needs no `_connectionId`; instead it must bind each connection the tool's steps use in `tool.overrides.connections` as `{_abstractId, _id}` -- set `_id` equal to `_abstractId` to keep the tool's own connection. A connection left unbound fails the run with `connection_not_overridden`
### Topic Imports (Event Streams)
- `TopicImport` -- publishes each record as a message to an Event Streams topic, which topic listeners (`TopicExport`) read. Set `topic._topicId` to the topic's `_id`, never its name (`celigo topics list`; an unknown id fails the save with 422 `invalid_ref`), and optionally `topic.batchSize` (default 1000). There's no connection and no create/update logic: each record becomes one message, reshaped by the mapping if there is one; with one-to-many, each element is its own message carrying the parent under `_PARENT`. Register the topic on the flow's integration so its flows can bind to it (`celigo integrations register-topics <integrationId> <topicId>`); the account needs the Event Streams entitlement. Test runs publish nothing; an invoke publishes for real
## Import Matching (create / update / upsert)
The core decision on most record-based imports is the **operation** -- what happens to each record at the destination. Create requires no match key; update and delete require one; upsert checks first and does whichever applies. Users describe the intent in business terms ("look up the customer and update them", "match by email and upsert", "skip the ones that already exist") that resolve into five behaviors:
- **Always create** -- every record is submitted as new. No matching, no checks.
- **Create only if missing** -- match-check first; submit as create if not found, skip silently if found.
- **Update only if exists** -- match-check first; submit as update if found, skip silently if not.
- **Update only if exists, fail if missing** -- strict variant; no-match records error out instead of skipping.
- **Upsert** -- submit a create when no match is found, an update when matched. The catch-all, and the right default when the user is vague.
When matching applies, decide four things: the **matching behavior**, the **match key field(s)** (`email`, `external_id`, `customer.id` -- required for anything other than always-create), the **on-match action** (update, skip, fail), and the **on-no-match action** (create, skip, fail). A lookup on a cross-reference field (NetSuite `externalid`, or a custom field that holds the source id) finds only records whose field holds that id, so the same import must write the source id into the field when it creates them; without that mapping no later run finds them, and each run adds the record again.
How a destination implements matching is adaptor-specific -- there is no single "matching mode" field. A destination might expose: a native **upsert** keyed off an external ID (Salesforce upsert, NetSuite upsert, RDBMS `ON CONFLICT`); a distinct `addupdate` operation that handles both paths in one call; an `ignoreExisting` flag paired with a lookup that probes before writing (lock it when repeats can arrive concurrently -- see [Writing Exactly Once](#writing-exactly-once)); two separate create and update endpoints with no upsert variant (see composite imports below); or a lookup endpoint plus separate create and update endpoints, where the lookup runs pre-write to drive the create-vs-update decision. Some destinations have no matching concept at all -- writing a CSV to FTP, sending an email, posting to a webhook -- so every record goes out as-is.
**Prefer import-level matching over a separate lookup step.** Imports natively support this pre-write check, so a single import that does the matching and the write together means fewer steps, fewer round-trips, and no glue logic to maintain. A standalone lookup earns its place only when the looked-up data has a consumer *beyond* the write -- a router branching on something other than "does this exist", an AI agent reasoning over the result, or multiple downstream steps reading different fields. If the only consumer is the destination call itself, the work belongs inside the import. It's also the only check that runs under the import's concurrency lock, which matters whenever the same record can arrive twice ([Writing Exactly Once](#writing-exactly-once)). An import's own lookups refuse an ambiguous match only when told to: an HTTP lookup that finds more than one match fails the record only with `allowFailures: false`, and otherwise sends the first match with no error (gotcha 14); a NetSuite DA lookup with `allowFailures: true` also resolves to one of the matching records unless `useDefaultOnMultipleMatches` sends it to the `default` ([netsuitedistributed.yml](references/schemas/netsuitedistributed.yml)). So search on a key that identifies one record. When an update's key must come from a separate lookup that can match several records, guard it -- fail or skip on multiple matches, or check for exactly one ([configuring-lookup-caches > Safe Lookup-to-Update](../configuring-lookup-caches/SKILL.md#safe-lookup-to-update----never-write-off-an-unguarded-first-match)).
### Composite (two-endpoint) imports
When an HTTP destination has no native upsert but exposes separate create and update endpoints, a **composite import** pins both endpoints on a single import node, role-tagged create and update. The runtime picks per record via a match-key check -- the same way a native upsert would -- so the flow stays one step. Prefer this over two separate imports driven by an upstream lookup or router; reach for separate imports only when the create and update paths must diverge beyond endpoint selection (different mappings, different downstream consumers, or different hook chains).
The operations sit in positional arrays, one entry per operation: `http.requestType` (`["UPDATE", "CREATE"]`), `http.method` and `http.relativeURI`. Exactly one decider picks per record: `http.existingExtract`, a field the record already carries (non-empty means UPDATE, whose URI reads it as `{{{data.0.<field>}}}`), or `http.existingLookupName`, a lookup in `http.lookups` that probes the destination (the UPDATE URI reads `{{lookup.<name>}}`). Never add `ignoreExisting` to a composite import, and give every `http.response.*` array one entry per operation.
`existingExtract` must hold the destination's id (put there by an upstream step's response mapping, or kept as a cross-reference), never the source record's own id: every record carries one, so every record would go to UPDATE. With `existingLookupName`, a lookup that comes back empty sends the record to CREATE, with no error -- and so does one whose `extract` resolves nothing, so a `$.`-prefixed `extract` sends every record to CREATE. A templated `http.body` has one entry per operation too; left unset, both requests send the mapped record. A search-driven upsert against an API that finds contacts by email:
```json
"http": {
"requestType": ["UPDATE", "CREATE"],
"method": ["PATCH", "POST"],
"relativeURI": ["/contacts/{{lookup.findContact}}", "/contacts"],
"existingLookupName": "findContact",
"lookups": [{
"name": "findContact",
"method": "POST",
"relativeURI": "/contacts/search",
"body": "{\"email\": \"{{email}}\"}",
"extract": "results[0].id"
}]
}
```
The lookup sits in `http.lookups` (a `lookups` array sent only at the top level is dropped on save), its request template is in `body`, and the template names the record's fields at the top level (`{{email}}`, never `{{record.email}}`, which renders empty, so every search misses and every record is created again; gotchas 10 and 11). Its `extract` is the id itself, because that is what the UPDATE URI receives. An HTTP lookup's `extract` is a path into the response written without `$.` -- `results[0].id`, or `ids` for an array of ids -- unlike a Mapper 2.0 row's `extract`, which needs `$.`. A `$.` prefix, `[*]`, or a dot path through an array (`results.id`) resolves nothing, so the lookup always comes back empty.
### Writing Exactly Once
Every source can hand an import the same record twice: a sender fires a webhook again, a failed or canceled run is re-run over the same delta window, an error is retried, a record that wrote this step but failed a later one comes back on the next run, or another flow writes the same destination. **A trace key never de-duplicates** -- it correlates a record's errors and is the default lock key below; a record that arrives twice is written twice. Make the write itself idempotent, in this order of preference:
| Rank | Pattern | Why it holds under concurrency |
|---|---|---|
| 1 | **A key the destination enforces** -- an upsert or insert on a primary key or unique constraint (`ON CONFLICT`, `MERGE` -- see [writing-sql](../writing-sql/SKILL.md)), a unique external ID, an API that rejects a duplicate reference | The destination refuses the second copy |
| 2 | **`ignoreExisting: true` + the existence check on this import + `idLockTemplate` on the same business key.** `ignoreExisting` (like `ignoreMissing`) is a top-level import field, not part of `http`. The check is `http.ignoreLookupName` (a lookup on the key) or `http.ignoreExtract` (the record already carries the destination ID) -- exactly one -- or the adaptor's equivalent (NetSuite `internalIdLookup`, Salesforce `idLookup.whereClause`) | The existence check runs inside the per-key lock |
| 3 | **A ledger step** in front of a destination that can't be queried -- [configuring-lookup-caches > Deduplication Ledger](../configuring-lookup-caches/SKILL.md#deduplication-ledger) | The ledger's claim is itself rank 2 |
An insert on a unique key refuses a repeat as an error (Salesforce `DUPLICATE_VALUE`, a unique-constraint violation). That keeps a repeated record out, but it refuses a changed one too: when changes to existing records should arrive, upsert on that key.
A match that is really a search run before the write -- a lookup-driven composite create/update, an `addupdate` found by search -- is a check-then-write, not rank 1, unless the destination also rejects the duplicate key. A natural id is unique only within its source: when two sources can produce the same one (two stores, two accounts), namespace the key per source (`{{record.storeId}}-{{record.orderId}}`), or one source's record collides with or overwrites the other's. Rank 2 is only as good as its lookup: a lookup that fails counts as "not found", so the record is written (gotcha 12).
**The concurrency ID lock.** `idLockTemplate` renders a key per record; records that render the same key are never submitted at the same time, and different keys run in parallel.
- **Default:** the record's trace key (a source's `traceKeyTemplate` arrives on the record as `io_custom_trace_key`). Without a template, the trace key is the first top-level id field the record has -- `id`, `internalId`, `_id`, `internal_id` or `identifier`, in a few spellings each -- and a nested id is never used. A blank trace key means **no lock**, so set `idLockTemplate` when the identity is nested or composite. A template replaces the default rather than adding to it.
- **Render time:** from the record as it arrives, before this import's own lookups and mapping. A key built from `{{lookup.<name>}}` or a mapped field renders empty -- and an empty key is no lock at all. Key it on a field the record carries (`{{{record.orderId}}}`), or one an upstream lookup step added through response mapping.
- **Scope:** this import across every page, every concurrent run, and every flow that uses it (`celigo imports dependencies <id>`) -- not the connection: two imports rendering the same key still run in parallel. So two imports that both create the same entity (an order flow that creates missing customers, and a separate customer sync) can each miss the other's write and create it twice: create the entity through one import, or schedule the flows so they don't overlap. Records within one page are sent one at a time anyway, so a one-page test never shows the race.
- **Coverage:** the `ignoreExisting` existence check runs inside the lock. Nothing documents the same for `ignoreMissing`, the composite create/update decision, or one-to-many children -- where those must write once, connection `concurrencyLevel: 1` closes the check-then-write race (the import's pages run one at a time), at the cost of serializing everything on that connection.
## Quick Reference
### Adaptor Decision Matrix
| Your data goes to... | Use adaptorType | Category | Read schema |
|---|---|---|---|
| REST or GraphQL API | `HTTPImport` | Record-based | [http.yml](references/schemas/http.yml) |
| NetSuite, Celigo SuiteApp or bundle installed | `NetSuiteDistributedImport` | Record-based | [netsuitedistributed.yml](references/schemas/netsuitedistributed.yml) |
| NetSuite, neither installed (or REST requested) | `NetSuiteHTTPImport` + `nsDomainType` | Record-based | [http.yml](references/schemas/http.yml) |
| Salesforce objects | `SalesforceImport` | Record-based | [salesforce.yml](references/schemas/salesforce.yml) |
| SQL database (Snowflake, PostgreSQL, etc.) | `RDBMSImport` | Record-based | [rdbms.yml](references/schemas/rdbms.yml) |
| MongoDB | `MongodbImport` | Record-based | [mongodb.yml](references/schemas/mongodb.yml) |
| DynamoDB | `DynamodbImport` | Record-based | [dynamodb.yml](references/schemas/dynamodb.yml) |
| JDBC database (non-built-in) | `JDBCImport` | Record-based | [jdbc.yml](references/schemas/jdbc.yml) |
| Files to cloud storage (Google Drive, Box, Dropbox, Azure Blob, Google Cloud Storage, Celigo Storage) | `HTTPImport` with `http.type: "file"` | File-based | [http.yml](references/schemas/http.yml) |
| Files to FTP/SFTP | `FTPImport` | File-based | [ftp.yml](references/schemas/ftp.yml) |
| Files to S3 | `S3Import` | File-based | [s3.yml](references/schemas/s3.yml) |
| AS2 EDI transmission | `AS2Import` | File-based | [as2.yml](references/schemas/as2.yml) |
| Local filesystem | `FileSystemImport` | File-based | [filesystem.yml](references/schemas/filesystem.yml) |
| An LLM (any provider in the `aiAgent.provider` enum) | `AiAgentImport` | AI | [aiagent.yml](references/schemas/aiagent.yml) |
| PII detection / content moderation | `GuardrailImport` | AI | [guardrail.yml](references/schemas/guardrail.yml) |
| Celigo Tool | `ToolImport` | Tool | [tool.yml](references/schemas/tool.yml) |
| Custom connector code on a stack | `WrapperImport` | Stack | [wrapper.yml](references/schemas/wrapper.yml) |
| An Event Streams topic | `TopicImport` | Topic | [topic.yml](references/schemas/topic.yml) |
**Raw HTTP is the fallback, not the default.** Pick the most specific match, in order:
1. **Native adaptor** -- if the application has its own row (NetSuite, Salesforce, databases, FTP/S3), use it. Do not build an `HTTPImport` against that app's REST API.
2. **Pre-built HTTP connector** -- for any other REST/GraphQL app, check the 550+ connector catalog before writing HTTP config (see [Check for a pre-built connector](#3-check-for-a-pre-built-connector)). The step is still an `HTTPImport`, but it runs on a connector-backed connection and takes its endpoint config from the connector.
3. **Manual HTTP** -- hand-write the config from public API docs only when no connector exists or it doesn't cover the operation you need.
For NetSuite, run `celigo connections suiteapp-status <connectionId>`: `suiteapp.success` or `bundle.success` true means `NetSuiteDistributedImport`; both false, each with `statusCode` 404 and an install `URL`, means neither is installed, so use `NetSuiteHTTPImport`. A different `statusCode` means the check itself failed, not that the components are missing: fix the connection and re-run it before choosing. `NetSuiteImport` is legacy -- don't use it unless asked.
`adaptorType` is **case-sensitive**: `NetSuiteDistributedImport`, not `netsuitedistributedimport`.
### Minimum Required Fields
Every import needs at minimum: `name`, `adaptorType`, and the adaptor config block (`http{}`, `netsuite_da{}`, `salesforce{}`, `tool{}`, `topic{}`, etc.). `_connectionId` is required except on the connection-less imports: `ToolImport` (connections are bound per tool step in `tool.overrides`), `TopicImport` (must not carry one -- with a connection, the save is validated as that connection's adaptor and fails), and `AiAgentImport` / `GuardrailImport` unless they use a BYOK connection.
### Which Schemas to Read
1. Always: [request.yml](references/schemas/request.yml) (base fields)
2. Plus: the adaptor-specific file from the decision matrix
3. If file-based: also [file.yml](references/schemas/file.yml)
4. If cloning: [clone-request.yml](references/schemas/clone-request.yml), [clone-response.yml](references/schemas/clone-response.yml)
### Schema Index
All schemas are in [references/schemas/](references/schemas/):
- **Base fields (all imports):** [request.yml](references/schemas/request.yml)
- **Response shape:** [response.yml](references/schemas/response.yml)
- **Adaptor-specific config:**
- [http.yml](references/schemas/http.yml) -- HTTP/REST/GraphQL (methods, URIs, headers, response parsing, lookups, composite upsert via `existingExtract` or `existingLookupName`)
- [netsuitedistributed.yml](references/schemas/netsuitedistributed.yml) -- NetSuite SuiteApp (operation, recordType, internalIdLookup, mapping, lookups)
- [netsuite.yml](references/schemas/netsuite.yml) -- NetSuite legacy
- [salesforce.yml](references/schemas/salesforce.yml) -- Salesforce (sObjectType, operation, api, idLookup)
- [rdbms.yml](references/schemas/rdbms.yml) -- SQL databases (queryType, query, bulkInsert, bulkLoad)
- [ftp.yml](references/schemas/ftp.yml) -- FTP/SFTP
- [s3.yml](references/schemas/s3.yml) -- Amazon S3
- [mongodb.yml](references/schemas/mongodb.yml) -- MongoDB (method, collection, filter, upsert)
- [dynamodb.yml](references/schemas/dynamodb.yml) -- DynamoDB
- [jdbc.yml](references/schemas/jdbc.yml) -- JDBC databases
- [as2.yml](references/schemas/as2.yml) -- AS2 EDI
- [wrapper.yml](references/schemas/wrapper.yml) -- custom stack connectors
- [tool.yml](references/schemas/tool.yml) -- Tool imports (`_toolId`, connection overrides)
- [topic.yml](references/schemas/topic.yml) -- Event Streams publisher (`_topicId`, `batchSize`)
- [filesystem.yml](references/schemas/filesystem.yml) -- local filesystem
- **AI config:**
- [aiagent.yml](references/schemas/aiagent.yml) -- AI agent (provider, model, instructions, tools, structured output)
- [guardrail.yml](references/schemas/guardrail.yml) -- guardrails (PII, moderation, AI-agent validation)
- **File output:** [file.yml](references/schemas/file.yml) (CSV, XML, JSON, XLSX config for file-based imports)
- **Clone:** [clone-request.yml](references/schemas/clone-request.yml), [clone-response.yml](references/schemas/clone-response.yml)
## Related Skills
- [configuring-connections > Quick Reference](../configuring-connections/SKILL.md#quick-reference) -- connection types and auth for import destinations
- [managing-file-storage > Use stored files in a flow](../managing-file-storage/SKILL.md#8-use-stored-files-in-a-flow) -- Celigo Storage folders as a file destination
- [writing-mappings > Mapper 2.0 Workflow](../writing-mappings/SKILL.md#mapper-20-workflow) -- field mappings on imports
- [writing-scripts > Data Pipeline Hooks](../writing-scripts/SKILL.md#data-pipeline-hooks) -- preMap, postMap, postSubmit, postAggregate hooks
- [writing-handlebars > Quick Reference](../writing-handlebars/SKILL.md#quick-reference) -- dynamic values in URIs, HTTP bodies, SQL queries
- [building-flows > How to Build a Flow](../building-flows/SKILL.md#how-to-build-a-flow) -- wiring imports into flow pipelines as page processors
- [configuring-topics > How to Build an Event Stream](../configuring-topics/SKILL.md#how-to-build-an-event-stream) -- the topic a publisher writes to: registration, retention, messages
- [troubleshooting-flows > Diagnostic Workflow](../troubleshooting-flows/SKILL.md#diagnostic-workflow) -- diagnosing import-related failures
<!-- TIER:2 -->
## How to Build an Import
### 1. Identify the target application
What system are you writing data to? This determines adaptor type, connection type, and configuration shape.
### 2. Check for existing patterns
Before building from scratch, look at what already exists:
```bash
# Search your account (fast, uses local index)
celigo account search "<keyword>"
# Show what an existing import uses (connection) and what uses it (flows)
celigo account dependencies import <id>
# Find orphaned imports not referenced by any flow
celigo account lint
# Search marketplace templates
celigo templates marketplace
# Extract just imports from a template
celigo templates preview <id> --model Import
celigo templates preview <id> --summary
```
The account index auto-refreshes when stale (older than 15 minutes by default). Force a fresh snapshot with `celigo account snapshot`.
### 3. Check for a pre-built connector
**Always run this check before writing any HTTP config.** Celigo maintains 550+ HTTP connectors with pre-configured auth, endpoints, and resources. Hand-write a manual `HTTPImport` from public API docs only when this search comes up empty or the connector doesn't cover the operation you need.
```bash
# Search HTTP connectors
celigo http-connectors list | grep -i "<application-name>"
celigo http-connectors get <id> --full # see endpoints, resources, auth config
# Drill into the endpoints the connector defines for imports
celigo http-connectors catalog <id> --resource-type import --published-only
celigo http-connectors endpoint-detail <id> --resource-type import --resource-id <rid> --endpoint-id <epid>
celigo http-connectors endpoint-detail <id> --resource-type import --resource-id <rid> --endpoint-ids <createId>,<updateId> # a create/update pair
# Search trading partner connectors (EDI, AS2)
celigo tp-connectors list
```
If a connector exists, create the connection from it (`http._httpConnectorId` -- see [configuring-connections > Check for a pre-built connector and global iClient](../configuring-connections/SKILL.md#4-check-for-a-pre-built-connector-and-global-iclient)) and take the import's `relativeURI`, method, and body/response shapes from the connector's endpoint metadata rather than reconstructing them from public API docs. Write each operation's `relativeURI` and `method` exactly as the endpoint defines them: saving links the import to those endpoints by setting `http._httpConnectorVersionId`, `http._httpConnectorResourceId` and `http._httpConnectorEndpointIds[]` (one per operation; imports drop the singular `_httpConnectorEndpointId`), but pointers you send for the same connector are kept as sent. Read the import back: missing pointers mean a URI matched no endpoint. When retargeting an import, update or drop the old pointers in the same save -- a GET-modify-PUT that changes only the URI leaves the import naming the old endpoint.
`endpoint-detail` on the import plane returns the resource, whose `resourceFields` catalog the record's fields, and each endpoint's `supportedBy`: what the connector's own form applies. `preConfiguredFields` are values it sets, `fieldsToUnset` are fields it clears, and `fieldsUserMustSet` is what it asks for -- on Shopify's create-customer endpoint, `ignoreExisting` and the mapping targets `mapping.fields.generate.customer.email`, `…first_name` and `…last_name`. The paths are the form's, not the import's. On create and update endpoints, `lookupToIdentifyExisting` is the existence lookup to build (the endpoint to call and the `extract` that reads the existing id, such as `customers[0].id`), and `pathParameterToIdentifyExisting` names the update URI's id parameter.
### 4. Query metadata for the target system
For NetSuite, Salesforce, and RDBMS connections, discover available record types and fields:
```bash
celigo metadata types <connectionId> # List record types / sObjects / tables
celigo metadata fields <connectionId> <type> # List fields for an entity type
celigo metadata fields <connectionId> <type> --refresh # Bypass the cache after a field is created in the app
```
Metadata is cached on the server, so a field or record type created since the last read stays missing until you re-read with `--refresh`.
- **NetSuite:** `metadata fields` returns field IDs, types, and groups. On `NetSuiteDistributedImport`, use the IDs in `netsuite_da.mapping.fields[].generate` and `netsuite_da.mapping.lists[].fields[].generate`; sublist names (groups such as `"item"`, `"addressbook"`) go in `netsuite_da.mapping.lists[].generate`. `NetSuiteHTTPImport` maps with Mapper 2.0 `mappings[]`. Lookup field IDs here are the `searchField`/`resultField` values for `netsuite_da.lookups[]`.
- **Salesforce:** `metadata fields` returns field API names, types, and relationship info. Use field API names for `salesforce.sObjectType` lookups and for discovering which fields are createable/updateable.
- **RDBMS:** `metadata fields` returns column names and types for a table — use these to write SQL queries (see `writing-sql`) and verify column names before building `bulkInsert.tableName` or `bulkLoad.tableName`.
### 5. Determine the category
Is this a **record-based import** (submit records to an API/database/ERP), a **file-based import** (write files to storage), or an **AI import** (invoke a model)?
### 6. Choose the right adaptor type
Refer to the [Adaptor Decision Matrix](#adaptor-decision-matrix) in the Quick Reference above. For NetSuite, [Diagnosing NetSuite Steps](../configuring-exports/references/netsuite.md) has the account-side checks (the SuiteApp and bundle, the account's concurrency limit) and the errors NetSuite steps return.
### 7. Build the import JSON
Reference the [Schema Index](#schema-index) for the exact fields needed. Use the [Which Schemas to Read](#which-schemas-to-read) decision rule to determine which files to consult.
## File Uploads over HTTP (multipart/form-data)
Some destination APIs accept files only as `multipart/form-data` POSTs (Jira attachments, QuickBooks attachables, OpenAI file uploads). This is an `HTTPImport` with `blob: true` and `blobKeyPath`, posting to the API's upload endpoint, and it works nothing like a JSON record write. Do **not** set `http.type: "file"`: that is the folder mode of file-storage connectors (Google Drive, Box, Dropbox, Azure Blob), which treats `relativeURI` as a folder path -- combined with an API endpoint URL it saves cleanly and fails every record with "Directory is not found." Four pieces have to line up:
1. **Where the bytes come from.** The import never carries the file itself. A preceding blob export or blob lookup pulls the file into blob storage, and that step's response mapping puts the reference onto the record -- the idiom is `{"extract": "data.0.blobKey", "generate": "blobKey"}`. The record then carries a `blobKey` (a storage pointer, not content).
2. **The media type.** The connection's media type (or the import's request media type override) is `multipart/form-data`; success/error response media types usually override to JSON so replies parse normally.
3. **The request body.** Not raw MIME and not a normal handlebars payload -- a JSON array of parts, each `{name, value, type}` plus optional `filename` (include the extension) and `mime-headers`. The file part is `"type": "attachment"` with `"value": "{{blob}}"` -- `{{blob}}` is the only accepted value for an attachment (anything else 422s). Fields the API wants alongside the file ride as `"type": "inline"` parts; an inline part whose value is a JSON object must be serialized. One file reference per import.
4. **`blob: true` + `blobKeyPath`** (Advanced settings) -- `blobKeyPath` is the JSON path in the record where the blobKey lives (`blobKey`, or `file.blobKey` if nested). At send time the platform follows it into blob storage and streams the real bytes into the attachment part.
The platform assembles the final MIME body itself -- it generates the `boundary` (never hardcode one), writes each part's `Content-Disposition`, and substitutes the attachment part with the raw file bytes. The parts array is a build recipe, not the payload.
**Not every multipart API is form-data.** Some upload endpoints expect `multipart/related` instead (e.g. Google Drive's `/upload/drive/v3/files?uploadType=multipart`), and the parts-array machinery does NOT apply. There the request body is the literal MIME document: explicit boundary, a JSON metadata part, and a content part referencing `{{blob}}` (double braces). Check which flavor the destination API documents before building -- mixing them produces an import that saves cleanly and fails at runtime.
## Async Destinations (submit, poll, confirm)
Some destination APIs only acknowledge a write (an HTTP 202, a job ticket) and finish it in the background -- bulk loads, file ingestion, document conversion. Attach an **async helper** to the import (`http._asyncHelperId`) so the step submits, polls a status export until the external work completes, and only then resolves. The mechanics and constraints (a status export with done/error value lists and poll intervals; no transform, output filter, or hook on the async-configured step; helpers cannot nest) are identical to the export side -- see [configuring-exports > Async APIs (submit, poll, fetch)](../configuring-exports/SKILL.md#async-apis-submit-poll-fetch). Only add one when the API genuinely cannot confirm the write synchronously.
## Testing an Import Without Writing
Most ways of trying an import send the request to the destination for real. Know which is which before you point one at a production system:
| What you run | Reaches the destination? |
|---|---|
| `celigo imports invoke <id>` with records on stdin | **Yes -- a real write.** One result per record, every adaptor; no job is created |
| `celigo imports invoke` with an import document on stdin (the preview route) | **Yes -- a real write**, despite the name. Only the first record of `data` runs, so try one record per call. NetSuite, Salesforce, RDBMS, and FTP imports are rejected with 422 `invalid_adaptor`; a bare document piped without `data` runs against one empty record |
| `celigo imports invoke [<id>] --connection <connectionId>` (a virtual import) | **Yes -- a real write.** Runs the saved import, or a piped `{"import": {...}, "data": [...]}` document, through that connection, NetSuite and RDBMS imports included; one result per record |
| `celigo flows test-run <flowId> --export <exportId>` | **No.** Imports never execute in a test run -- each returns its saved `mockResponse`. A NetSuite DA import is the exception: the page still goes to NetSuite, where the SuiteApp or bundle runs the lookups and SuiteScript hooks and builds each record without saving it. Its stages are `mapping` and `lookup` (plus one per SuiteScript hook), not `request` and `response`, and an error NetSuite raises while building a record is real and counts on the job |
| A real flow run, or `celigo flows retry-errors` | Yes |
Invoking by id or as a virtual import runs the step with the import's own settings only: hooks get no integration or flow settings, and `{{settings.integration.*}}` and `{{settings.flow.*}}` render empty. To run it as a step of its flow, tool or API, add `--integration <id>` and `--flow`, `--tool` or `--api <id>` to the `--connection` form.
In the flow builder's import editor, **Preview** builds the request without sending it (HTTP imports only), while **Send** is a real write. No CLI command offers the compose-only preview. From the CLI, test-run the flow and read the import's stages with `celigo flows test-run-step-results`: the `mapping` stage's output is the record after mapping, while the `request` stage shows the body template unrendered, and its input is the record before mapping. Database imports have no compose-only surface at all; the SQL editor's evaluation is their only preview.
**Save a realistic `mockResponse`.** In test runs and flow-builder previews, an import with no saved mock answers with the placeholder `{"id": "1234567890", "mockResponse": "replace me"}` (numbered per record on a page: `1234567890-1`, `1234567890-2`), so response mappings extract nothing real and downstream steps see empty data. Save an array of records shaped like the destination's answer -- `statusCode`, the created record's `id`, and the response body under `_json`:
```bash
celigo imports set <importId> 'mockResponse=[{"statusCode": 200, "id": "SO-1001", "_json": {"id": "SO-1001", "status": "created"}}]'
```
A saved `mockResponse` substitutes only in test runs and flow-builder previews -- `imports invoke`, real runs, and retries always send. `sampleResponseData` only seeds the response-mapping editor; test runs and previews never use it.
## CLI Commands
```bash
# CRUD
celigo imports list
celigo imports get <id>
celigo imports create < import.json
celigo imports update <id> < import.json
celigo imports set <id> key=value [key2=value2 ...]
celigo imports delete <id> [-y]
celigo imports audit <id> # change history
celigo imports dependencies <id> # the flows, APIs and tools that use it
# Invoke -- a real write: each record is sent to the destination (no job is created).
# HTTP 200 even when records fail; check each result's statusCode and errors
echo '[{"name":"test"}]' | celigo imports invoke <id>
# Virtual import: an unsaved or modified document through a connection (also a real write)
echo '{"import": {...}, "data": [{"name":"test"}]}' | celigo imports invoke --connection <connectionId>
# Mock response for test runs and flow-builder previews (never used by invoke or real runs)
celigo imports set <id> mockResponse=file://./mock-response.json
# Clone and connection management
echo '{"connectionMap":{"oldConnId":"newConnId"}}' | celigo imports clone <id>
celigo imports replace-connection <id> <newConnectionId>
# Discovery
celigo templates marketplace
celigo http-connectors list
celigo http-connectors catalog <id> --resource-type import --published-only
celigo http-connectors endpoint-detail <id> --resource-type import --resource-id <rid> --endpoint-id <epid>
celigo tp-connectors list
celigo metadata types <connectionId>
celigo metadata fields <connectionId> <entityType>
# Debug -- a window of at most 60 minutes (debugUntil, set by PATCH; a PUT doesn't set it).
# It records each request the import sends while open: not earlier ones, and nothing from test runs
celigo imports enable-debug <id> [--duration <minutes>]
celigo imports disable-debug <id>
celigo flows debug-requests <flowId> <importId> # read them from the consumer (or apis / tools debug-requests)
```
<!-- TIER:3 -->
## Pre-Submit Checklist
### Required (all imports)
- [ ] `name` is set
- [ ] `adaptorType` exact case matches connection type (request.yml > adaptorType)
- [ ] `_connectionId` references a valid, online connection (skip for `ToolImport`, `TopicImport`, and AI imports without BYOK)
- [ ] Adaptor config block name matches adaptorType (`http{}` for HTTPImport, `netsuite_da{}` for NetSuiteDistributedImport, etc.)
### Adaptor-specific
- [ ] HTTP: pre-built connector was checked (`celigo http-connectors list`) -- hand-written config only because no connector covers the app or operation
- [ ] HTTP: `http.method` and `http.relativeURI` are set (http.yml)
- [ ] HTTP: string values in a JSON `http.body` or lookup `body` are double-braced inside quotes; triple braces only for numbers, booleans and serialized JSON (gotcha 11)
- [ ] HTTP lookups: every lookup is in `http.lookups` (the save keeps that array and mirrors it to the top-level `lookups`; one sent only at the top level is dropped), each request template is in `body`, not `postBody`, and names the record's fields without `record.` (gotcha 10), and each `extract` is a path without `$.` to the id itself or an array of ids, from a search on a key that matches one record (gotcha 14)
- [ ] NetSuite DA: `netsuite_da.operation` and `netsuite_da.recordType` are set, and `update`, `addupdate` and `delete` carry `netsuite_da.internalIdLookup` -- the lookup, not a mapped `internalid`, picks the record (on `update` it overrides a mapped one). When the record already carries the internal id, use the expression form `["internalid","is","{{{id}}}"]` rather than dropping the lookup (netsuitedistributed.yml)
- [ ] NetSuite DA fields set at record initialization (NetSuite preloads them when it creates or transforms the record, before the other fields): one body row `{"generate": "celigo_initializeValues", "hardCodedValue": "entity,subsidiary"}` listing their ids, comma-separated and without `.internalid`, next to the rows that map those fields. It's what the mapping editor's "Use this field for NetSuite record initialization" saves; a `useAsAnInitializeValue` key on a row isn't stored
- [ ] NetSuite DA serialized items: the SuiteApp and bundle take a line's `serialnumbers` as `<serial>(<quantity>)` text when the account's Advanced Bin / Numbered Inventory Management feature is off; with it on, give the serials through the line's inventory detail subrecord (`celigo_inventorydetail`, [netsuitedistributed.yml](references/schemas/netsuitedistributed.yml))
- [ ] NetSuite DA keyed on `externalid` (`["externalid","is","{{{id}}}"]`): a mapping writes that same source id to `externalid` (`{"extract": "id", "generate": "externalid"}`). The lookup finds only records whose `externalid` was set, so without the mapping an `addupdate` adds the record again on every run
- [ ] RDBMS: `rdbms.queryType` is one current value (`bulk_load`, `bulk_insert`, `per_page`, `per_record` or `first_page`) with its companion field set -- never the legacy uppercase `INSERT` or `UPDATE` (rdbms.yml)
- [ ] Salesforce: `salesforce.sObjectType` and `salesforce.operation` are set. `upsert` also needs `salesforce.upsert.externalIdField` (the Salesforce external id field) and `salesforce.idLookup.extract` (the record field matched against it); `update`, `delete`, `addupdate` and an `insert` with `ignoreExisting` need `idLookup.whereClause` instead (salesforce.yml). The mapping is Mapper 1.0: `mapping.fields[]` rows of `{extract, generate}` (or `hardCodedValue`), never `mappings[]`
### Cross-resource consistency
- [ ] Connection `type` matches the import's `adaptorType`
- [ ] If response mapping needed: configured on the flow's `pageProcessors[]` entry, not on the import itself
- [ ] If one-to-many: `oneToMany: true`, with `pathToMany` set to the child array path for an object record (child fields then use `record.<field>` and parent fields `record._PARENT.<field>`) and left blank for an array (grouped) record
- [ ] If downstream steps read this import's response: a realistic `mockResponse` is saved, so test runs carry real-looking data past this step
- [ ] If a repeated record would duplicate data: the write is keyed (a destination-enforced key, or `ignoreExisting` + an existence check) and `idLockTemplate` renders a non-empty key from a field the record carries on arrival -- never `{{lookup.*}}` ([Writing Exactly Once](#writing-exactly-once)), and its existence lookup has been checked against a record that exists (gotcha 12)
- [ ] Mapper 1.0 only where required: `mapping.fields[]` / `mapping.lists[]` on Salesforce and wrapper imports, `netsuite_da.mapping` on NetSuite DA -- never `mappings[]` there. `NetSuiteHTTPImport` uses `mappings[]`
## Gotchas
1. **PUT erases omitted fields.** Always GET first, modify, then PUT. The `set` command handles this. The import PATCH route accepts only `debugUntil` (any other path is refused with 422), so a change to a mapping, body or lookup is always this full PUT, never a reason to send the user to the UI. Leave the read-only fields a GET adds (`_id`, `lastModified`, `createdAt`) out of the body.
2. **Including a `rest:` block creates a legacy RESTImport.** Use only `http:` for new imports.
3. **Input filter skips that import, not the record.** Filtered records skip the current import step but continue to subsequent steps in the flow, without this import's response-mapped fields, so a later step that writes those fields back needs its own filter on them. Check `numIgnore` on the job if records seem to bypass a step.
4. **Multipart file parts only accept `{{blob}}`.** In a `multipart/form-data` parts array, the file part must be `"type": "attachment"` with `"value": "{{blob}}"` -- any other value 422s. Never hardcode the MIME `boundary`; the platform generates it. Fields the API wants alongside the file ride as `"type": "inline"` parts.
5. **`bodyKey`/`blobKey` in logs is an artifact, not a payload field.** Audit and debug logs never show the assembled multipart body -- an internal storage pointer appears where the body would be. But if the destination actually received the literal string `bodyKey` or `blobKey`, the file part is misconfigured (an `inline` part where an `attachment` belongs, or a `blobKeyPath` that doesn't resolve).
6. **`imports invoke` is a real write.** "No job created" doesn't mean nothing is sent: every form executes the request against the live destination, and a saved `mockResponse` doesn't stop them. It also runs the import's hooks, postSubmit included, and its response transform (the editor's **Send** stops before postSubmit). A 200 can still carry per-record failures in each result's `errors`, and on a file import (FTP, S3, file-mode HTTP) it means the file was composed, not delivered: an FTP import to an unreachable host still answers `{name, fileLocation}`. Point it at a sandbox, or test-run the flow when you only need to check mapping output.
7. **A trace key is not duplicate detection.** Repeated keys are normal (delta re-emits, retries) and never skip, merge, or reject a record. Only a keyed write stops the second copy -- see [Writing Exactly Once](#writing-exactly-once).
8. **A lock keyed on this import's own lookup is no lock.** `idLockTemplate` renders before the import's lookups and mapping run, so `{{lookup.<name>}}` renders empty. Check the key against a real record (an error's retry data: `celigo flows error <flowId> <importId> <errorId> --retry-data`), not a sample that may lack the field.
9. **Lookups can't chain.** A lookup's own URI, body and headers render before any lookup has run, so `{{lookup.<name>}}` there is undefined: it either fails the render (`handlebars_template_parse_error`: `Unable to read "<name>" from undefined`) or renders empty, and either way that lookup fails or asks the wrong question for every record. Test runs never execute imports, so the first real run is where it shows. Resolve the value in one query, or add an upstream lookup step and reference the field it adds to the record.
10. **An HTTP import's lookups send `body`, not `postBody`, and read the record's fields at the top level.** Put a lookup's request template in `http.lookups[].body`. A template saved as `postBody`, the field [http.yml](references/schemas/http.yml) documents, saves without complaint, but the lookup goes out with an empty body, so it asks the wrong question for every record and nothing errors. The CLI's schema check calls `body` undocumented; it's still the field that's sent. Inside that template the record's fields sit at the top level: `{{email}}` renders the record's value, and `{{record.email}}` renders empty.
11. **In a JSON body, string values take double braces.** `{"name": "{{record.name}}"}` is JSON-escaped at runtime, so a quote, backslash, newline or tab in the value arrives intact. Triple braces insert the raw text: such a value fails its record with 422 `invalid_json` before anything is sent, and in a lookup's `body` the lookup quietly finds nothing (gotcha 12). Keep triple braces for numbers, booleans and serialized JSON (`{{{jsonSerialize record}}}`).
12. **A failing existence lookup counts as "not found".** With `ignoreExisting`, an `http.ignoreLookupName` lookup that fails -- a wrong URI answering 404, a body broken by a raw quote -- extracts nothing, so the record is written as new and no error is raised. Before trusting the check, invoke the import in a sandbox with a record that already exists there and confirm the result comes back `ignored`.
13. **A 200 isn't always success.** When an API answers 200 for a failed write, set `http.response.successPath` + `successValues` or `failPath` + `failValues` (one entry per operation; values compare as strings). A record that trips them fails: an invoke reports `statusCode` 422, and a run records an error whose code is the destination's status (`200`), source `application`, with the response body as its message -- or `response_failure` ("response failed using path") when a `successPath` value doesn't match.
14. **An ambiguous HTTP lookup fails only with `allowFailures: false`.** When a lookup's `extract` resolves to two or more values (an array of ids or of result objects), the record fails with `lookup_conflict` ("Multiple matches found for lookup.") only if the lookup sets `allowFailures: false`. With it left out or `true`, the first value is sent and nothing errors -- from an array of objects, the literal text `[object Object]` -- and `useDefaultOnMultipleMatches` hasn't been seen to send it to the `default` instead. An indexed `extract` (`results[0].id`, `ids[0]`) takes the first match silently, whatever `allowFailures` says: right for a search that can match only one record, and never a fix for `lookup_conflict`, which it turns into writes against whichever record comes first. Fix ambiguity in the search: query on a key that identifies one record (an email, an external id), taken from a field the record carries -- a failed record's retry data shows them -- and keep `allowFailures: false`, so a search that still finds two fails rather than writing against the wrong record.
## Common Errors
| Error | Cause | Fix |
|-------|-------|-----|
| 422 "Invalid value '<value>' for field 'adaptorType'" | Wrong case (`httpimport`) or an unknown adaptor | Use the exact case from the decision matrix: `HTTPImport`, `NetSuiteDistributedImport`, etc. |
| 422 "Expected field: _connectionId to be present" | No connection on an adaptor that needs one | Set `_connectionId`; only the connection-less imports in [Minimum Required Fields](#minimum-required-fields) omit it |
| 422 `invalid_ref` ("connection not found.; <id> is not a valid _connectionId") | The connection id doesn't exist in this environment | Use an id from `celigo connections list` |
| Copying an Integration App's import fails: 403 `create_not_allowed` ("Cannot create an Integration App resource"), 422 `invalid_ref` ("The referenced Connection document ... belongs to different Integration App installation"), or 400 from `imports clone` ("Cloning can't be performed for ... that were installed from an Integration App.") | The body still carries the app's `_connectorId`, or the copy keeps the app's connection, which only the app's own resources can use; a clone of an app's import is refused outright | Build the copy as your own import, on a connection of your own, with `_connectorId`, `_id` and the timestamps left out |
| 422 "http subschema not defined" (or another connection type's name) | The `adaptorType` doesn't match the connection: the save validates the import as the connection's adaptor | Pick the adaptor for the connection's type from the decision matrix, with its config block |
| 422 "`<value>` is not a valid enum value for path `<field>`" | A value outside the field's enum, such as `http.method: ["PUSH"]` | Use a value the adaptor's schema lists |
| 422 `Import query types … are deprecated` | `rdbms.queryType` holds the legacy uppercase `INSERT` (or a retired `MERGE`, `COPY` or `BULK INSERT`) | Use one current value -- `bulk_load`, `bulk_insert`, `per_page`, `per_record` or `first_page` -- and set its companion field |
| 422 `… is not a valid enum value for path rdbms.queryType.0` | A `queryType` string outside the enum, such as lowercase `insert` | Use one of the current values above |
| 422 `distributed required` | Missing NetSuite flag | Use `NetSuiteDistributedImport` with `distributed: true` on the import (there's no such connection flag); check the SuiteApp with `celigo connections suiteapp-status <connectionId>` |
| 422 `mapping invalid` | Wrong mapper version | Salesforce uses Mapper 1.0 (`mapping.fields[]`) and NetSuite DA uses `netsuite_da.mapping`, not Mapper 2.0 (`mappings[]`); `NetSuiteHTTPImport` uses `mappings[]` |
| `Unable to find lookup in mapping configs - <name>` on every record (source `mapping`) | A mapping row's `lookupName` names a lookup the import doesn't define -- on an HTTP import, often one sent only in the top-level `lookups`, which the save drops; the save doesn't check it | Define the lookup under that exact name (in `http.lookups` on an HTTP import), or fix the row |
| Every record fails with `Cannot set properties of undefined (setting '_hasStarNotation')` | A NetSuite DA mapping condition (`conditional.lookupName`) names a lookup the import doesn't define; on a sublist row the job can stall instead | Fix the name or restore the lookup -- changing the SuiteScript version doesn't help |
| `INVALID_HANDLE_BAR_EXPRESSION` on a NetSuite DA sublist lookup | The lookup's `expression` reads a bare line field, which is undefined at the record root | Address the line as `<array>[*].<field>` (`{{{line_items[*].sku}}}`) |
| `VALUE_LOOKUP_FAILED` "Could not find a match for …" | The rendered criteria, quoted in the message, matched no NetSuite record; inactive records don't count unless the lookup sets `includeInactive` | Fix the criteria or the data, or set `allowFailures: true` with a `default` to fall back |
| `VALUE_LOOKUP_FAILED` "More than one match for …" (`MORE_THAN_ONE_MATCH_FOUND` on a `suiteapp2.0` step) | The criteria matched several records; an empty source value searches `["<field>","isempty"]`, which matches every blank one | Keep an empty key out of the search (branch in the `expression`), or set `useDefaultOnMultipleMatches` with `allowFailures: true` and a `default` |
| NetSuite `INVALID_KEY_OR_REF` ("Invalid <field> reference key <value>.") on some records | The field references another NetSuite record, and the value NetSuite received isn't a key it accepts. When the quoted value is a name or code and the row writing that field sets `internalId: true` (its value must be the referenced record's internal id), those records carry the name where the id belongs ([Diagnosing NetSuite Steps](../configuring-exports/references/netsuite.md) has the other causes) | Resolve the name to the id on that row with a lookup: a static `map` from each name to its internal id (the ids are the user's to supply), or a NetSuite lookup that searches the name field and returns `resultField: "internalid"` ([netsuitedistributed.yml](references/schemas/netsuitedistributed.yml)). Then retry the failed records; a retry before the change fails again |
| NetSuite `USER_ERROR` (`source: application`) whose message states a business rule ("PO # is required for customers in the Distributor category") | A rule in the NetSuite account -- often its own user event script or workflow -- refused the record. Celigo's hooks report their own `source` (`pre_map_hook`, `post_submit_hook_ss` and the like), so `application` means NetSuite raised it | Read a failing record's retry data for the value the rule asks for and check whether a mapping row writes it: map it, or have a NetSuite administrator change the rule. Don't edit a Celigo hook for it ([Diagnosing NetSuite Steps](../configuring-exports/references/netsuite.md)) |
| NetSuite `INVALID_FLD_VALUE` "You have entered an Invalid Field Value <value> for the following field: <field>" on some records, on a list or select field | The row sends text, and a NetSuite DA import sets a select field's text by the option's display name: the list has no option named exactly that value -- often a new value the source started sending. On a date field the same code means a value the import can't read as a date ([Diagnosing NetSuite Steps](../configuring-exports/references/netsuite.md)) | Compare the quoted value with what the records that succeed send. Add the option to the NetSuite list with that exact name, or translate the values with a static lookup on the row (to the options' internal ids, with `internalId: true`). Then retry the failed records |
| NetSuite `USER_ERROR` "Please enter value(s) for: <field label>" on some records only | The field is mandatory for those records and the mapping leaves it empty for them: their source value is blank, or another field makes NetSuite require it -- a customer whose `isperson` is `F` (a company) needs Company Name, while an individual (`isperson` `T`) needs first and last name instead | Read a failing record's retry data next to one that succeeded. Map the field from a value those records have, give the row a default, or set the deciding field from the data (`isperson` `T` when the company is empty). Then retry the failed records |
| NetSuite `SSS_INVALID_SUBLIST_OPERATION` ("…trying to add or remove lines from a static sublist", often ending "Tip: Please make sure that all the line items that you're trying to use exist on the record.") on an item fulfillment | A mapped line matched none of the fulfillment's lines, so the import tried to add one, and a fulfillment can't add item lines: its lines are its order's. Lines match on the sublist's `isKey` columns, or on the item when none is marked ([writing-mappings > Mapper 1.0](../writing-mappings/SKILL.md#mapper-10-reference-netsuite-and-salesforce)) | Find the line whose key value none of the order's lines holds -- in its retry data, or from a change to the row or lookup that resolves the item -- and fix the mapping or the data. A retry before that fails the same way |
| NetSuite `MULTIPLE_LINES_WITH_SAME_KEYS` "Multiple lines found with keys item=<id> …" | Two of the record's lines hold the mapped line's key values: on a fulfillment, receipt or invoice keyed on the item (no column is `isKey`), an order with that item on two lines | Key each line on what only it holds: map the order line's id (its `line` on the order) to `orderline` and mark that row `isKey` -- a mapped `orderline` without `isKey` isn't a key, so the item still is. Then retry the failed records; a retry before the change fails the same way |
| A NetSuite update repeats the record's lines on every run, with no error | The sublist has no `isKey` column, so no mapped line matches an existing line and each is added; only sublists with a default key fall back to it -- the item on fulfillment-style transactions, the applied transaction on a payment's or credit's | Mark the row that identifies a line `isKey` -- a line id the record's create wrote to a line field -- so each mapped line updates its own. To also drop lines the source no longer has, map `celigo_replaceAllLines_<sublist id>` to `true`. Records already updated keep their repeated lines, and since the repeats carry the same id, their next update with the key set fails with `MULTIPLE_LINES_WITH_SAME_KEYS`: clean those up first |
| Records the destination already has show as ignored, and their changes never arrive | `ignoreExisting` skips each record its existence check finds; it never updates one | To update them, make the import a composite UPDATE/CREATE that decides on the same lookup (`http.existingLookupName`; [Composite imports](#composite-two-endpoint-imports)) and drop `ignoreExisting` |
| `The mapping condition:"record_updated" not supported for the operation` | A `record_updated` condition on a NetSuite DA `add` import | Use `addupdate` or `update`, or drop the condition |
| A record fails with code `200`, source `application`, and the response body as its message (or `response_failure`) | The import's `failPath` / `successPath` rules judged a 200 a failure | Expected when the API reports failures in a 200 body (gotcha 13); fix the record, or the rule if it misfires |
| 422 attachment `value` invalid | Multipart file part is not `{{blob}}` | Set the file part to `"type": "attachment"`, `"value": "{{blob}}"`; let the platform generate the `boundary` |
| 422 `invalid_json` ("An invalid JSON is sent in the request.body") on some records only | A triple-braced string in a JSON `http.body` met a value holding a quote, backslash, newline or tab; in a GraphQL `variables` string, any string value without `jsonEncode` | Double-brace string values inside quotes (gotcha 11); inside a GraphQL `variables` string, `{{jsonEncode record.x}}` ([GraphQL imports](#graphql-imports)) |
| A GraphQL import reports every record written, and the destination has none | The server answered 200 with an `errors` array or the mutation's `userErrors`, and the import sets no response rules | Set `resourceIdPath` to the created object's id, and `failPath`/`errorPath` to `errors[0].message` ([GraphQL imports](#graphql-imports)) |
| `missing_id` ("could not extract id using path: …") | No id at `resourceIdPath`: the write failed (the response, quoted in the message, carries the server's errors) or the path is wrong | Read the quoted response; fix the data, or the path |
| `handlebars_template_parse_error`: "Failed to generate request url from template: … Details: "<field>" not defined in the model." (source `application`, classification `parse`) on some records | Those records lack the field the `relativeURI` names, or its `{{lookup.<name>}}` found nothing -- missing data, not template syntax. Nothing was sent, and a retry fails the same way | When the records may not exist at the destination yet (new items, no destination id), make the import a composite UPDATE/CREATE that decides on the field (`http.existingExtract`; [Composite imports](#composite-two-endpoint-imports)), or skip them with `ignoreMissing: true` and `http.ignoreExtract` set to the field. Otherwise get the id onto the record upstream, then retry |
| 422 `unsupported_type` ("NetSuiteHTTPImport does not support type: file") | `http.type: "file"` on a `NetSuiteHTTPImport`: the NetSuite REST import writes records, and the save refuses file mode | Remove `http.type` |
| Every record fails with "Directory is not found" | `http.type: "file"` (file-storage folder mode) combined with an API upload endpoint | Remove `http.type: "file"`; keep `blob: true` + `blobKeyPath` and post to the endpoint |
| 422 `invalid_adaptor` ("Import preview is not supported for adaptor type") | `celigo imports invoke` with a document on stdin, for a NetSuite, Salesforce, RDBMS, or FTP import | Save the import and invoke it by ID, or run the document as a virtual import with `--connection` (both real writes); or test-run the flow to check mapping output without sending |
| 422 `Connection is offline` from a mapping preview or a NetSuite field check | The destination connection is marked offline (its `offline` flag), so the platform doesn't call it; nothing is wrong with the import or its mapping | Fix the connection (test it, re-authorize), then retry |
| A Salesforce import fails with `response stream exceeded limit of 5242880 bytes.` | The platform reads at most 5 MB of Salesforce's answer to a request, and this batch's results were larger | Send fewer records per request |
| Salesforce `UNABLE_TO_LOCK_ROW` on some records -- on the `rest` and `compositerecord` APIs the platform replaces Salesforce's message with "Salesforce creates a lock on a record while updating it. If another request is trying to update the same record which is already locked, Salesforce would throw this error." and a help link | Another write held the Salesforce record this one needed: this import's own pages or runs submitting in parallel, or another import or flow writing the same records | Set `idLockTemplate` to the key of the record the writes contend for -- the record's own id, or the parent id they share when they update children of one record, taken from a field the incoming record carries (`{{{record.AccountId}}}`; gotcha 8) -- so pages carrying the same key never submit together through this import. The lock doesn't reach other imports: when another flow writes the same records, lower the connection's `concurrencyLevel` instead. Then retry the failed records |
| A Salesforce import reports a field error for a record that Salesforce saved | On the `soap` API (the default), when Salesforce refuses a value in a field the import mapped (the error names the field), the platform drops that field and sends the record again, up to `maxAttempts` more times (default and maximum 3; `maxAttempts: 1` turns it off). A resend that succeeds saves the record without the field, and its error still counts against the job | Look the record up in Salesforce before retrying: a retry of an `insert` creates it a second time. Fix the value at the source, update the saved record, and resolve the error |
| A `compositerecord` import fails a record with `Composite Operation failed due to the following error(s)`, then one line per failure (`<message> for <object>[<index>]`) | One object in the record's tree failed. The platform sends a record and its related records as one all-or-none Composite API request, so Salesforce saved none of them; the rollback notices it returns for the other objects (`PROCESSING_HALTED`, `ALL_OR_NONE_OPERATION_ROLLED_BACK`) are left out, and the error's code is the first real failure's | Fix the field each line names on that object, then retry the whole record |
| Salesforce `ENTITY_IS_DELETED` on an `update` or `delete` | The record was deleted in Salesforce | With `ignoreMissing: true` the import counts such records as ignored instead of failing them; without it, resolve the errors once the deletions are confirmed as intended |
| Salesforce `DUPLICATE_VALUE` ("duplicate value found: <field> duplicates value on record with id: <id>") on some records | Salesforce's meaning: the record set a unique or External ID field to a value another record already holds. On an `insert` import those records already exist in Salesforce -- sent again by an overlapping window, a re-run, or because they changed | To create new records and update existing ones, make the import an `upsert` on that External ID field (`upsert.externalIdField` with the `idLookup.extract` that carries the value; [salesforce.yml](references/schemas/salesforce.yml)), then retry the failed records. `insert` with `ignoreExisting` and an `idLookup.whereClause` skips them instead, and their changes never arrive |
| The destination refuses the text `Invalid date` in a date field (its error quotes that value) | A date mapping (`extractDateFormat`, `extractDateTimezone` or `generateDateFormat` set, Mapper 1.0 or 2.0) couldn't read the source value with its `extractDateFormat` -- `YYYY-MM-DD HH:mm:ss` when unset -- so it wrote the text `Invalid date` | Set `extractDateFormat` to the format the source actually sends (a failed record's retry data shows the value), then retry the failed records |
| Downstream steps see `1234567890` or `"replace me"` in a test run | No `mockResponse` saved on the upstream import -- test runs never call the destination | Save a realistic `mockResponse` on that import |
| The destination receives the same record twice | A bare create; `ignoreExisting` with no lock on records whose trace key is blank, or with an existence lookup that fails (gotcha 12); or a lock keyed on the import's own lookup | Key the write and lock it on the business key ([Writing Exactly Once](#writing-exactly-once)); find which runs wrote it first ([building-flows > Duplicate Records](../building-flows/SKILL.md#duplicate-records)) |
| A NetSuite `addupdate` adds the same record again on every run | `internalIdLookup` searches `externalid`, but no mapping writes `externalid`, so the lookup never finds what earlier runs added | Map the source id to `externalid` and keep the lookup; the duplicates already added stay until someone merges them |
| `lookup_error`: "Could not find value for static lookup: <name> {Lookup Key: <value>} …" on some records | The value isn't a key of that lookup's `map`, and the lookup has `allowFailures: false`; the other records succeed | Add each missing value as a key, exactly as the message quotes it (case, spacing and suffixes count), with the destination's value (ask the user -- a guessed code sends the record wrong), keeping every existing key in the save, then retry the failed records. A retry runs the step as it is now, so one before the map changes fails again. A `default` would instead send every unmapped value as the default, with no error |
| `lookup_conflict`: "Multiple matches found for lookup." (source `lookup`) on some records | An HTTP lookup with `allowFailures: false` got two or more values at its `extract`: the search key isn't unique (a name several records share) | Search on a key that identifies one record and keep `allowFailures: false`. Don't index the `extract` (`ids[0]`) or allow failures to get past it: either one writes against whichever match comes first (gotcha 14) |
| `batch_lookup_evaluation_failed`: "<field>: The batch lookup could not be evaluated. Check the value in the key '<value>', and try again." (source `internal`) | An HTTP lookup with `allowFailures: false` got nothing usable for that value: the search matched nothing, or the `extract` resolves nothing (a `$.` prefix, `[*]`, a dot path through an array) or points at a result object rather than an id | Check the `extract` against a real response -- a path without `$.` to the id (`results[0].id`) or an array of ids; if the value truly has no match, fix the data or the search, then retry |
| 422 `cannot_set_traceKeyTemplate` | `traceKeyTemplate` set on a one-to-one import | Set it on the source export or listener -- on an import it is valid only with `oneToMany: true`. To change the lock key, set `idLockTemplate` |
| `oneToMany_error` | A blank `pathToMany` on an object record ("Record at index 0 was not an array, and no PathToMany was provided"). The editor's Preview can also show it falsely on a step fed an array, because the preview re-splits the record it displays | Object record: set `pathToMany` to the child array. Array (grouped) record: blank is right, and each row is the record, with no `_PARENT` -- confirm with a test run before changing the configuration |