references/schemas/as2.yml
AS2:
type: object
description: |-
Configures AS2 exports and listeners. Optional — the transport settings live on the AS2
connection, and an AS2 export is valid with only `file` (parse settings); set this object
to link a Trading Partner Connector or to transfer raw files as blobs. An AS2 listener
acts as the flow's source, receiving trading-partner transmissions in near real-time and
handling decryption, signature verification, and MDN generation; AS2 (Applicability
Statement 2) transmits EDI and other data securely over HTTP/S using S/MIME encryption
and digital signatures.
x-celigo-ai-guidance:
- |-
## WHAT IS AS2?
Applicability Statement 2 (AS2) is a widely adopted protocol for securely and reliably transmitting
EDI and other data types over the internet using HTTP/S, S/MIME encryption, and digital signatures.
AS2 provides:
- **Message integrity** through digital signature validation
- **Confidentiality** via encryption with X.509 certificates
- **Non-repudiation** via Message Disposition Notifications (MDNs)
- |-
## AS2 EXPORT CONFIGURATION
Do not populate this object on every AS2 export — the transport settings live on the
AS2 connection, and an export with only `file` (parse settings) is valid. Set it only
to link a Trading Partner Connector (`_tpConnectorId`) or to enable raw-file blob
transfer.
- |-
## AS2 LISTENER FUNCTIONALITY
An AS2 listener is a flow step in Celigo designed to receive incoming AS2 transmissions
and deliver them into a defined integration flow. It acts as the "source" of a flow—similar to
how a webhook listener works—except it specifically handles AS2 protocol requirements, including
decryption, signature verification, and MDN generation.
Unlike periodic polling or scheduled exports, an AS2 listener functions in near real-time—when
a trading partner pushes an AS2 message, Celigo's listener step processes it instantly,
generating an MDN in response to acknowledge receipt. This ensures low-latency, event-driven
processing where each inbound AS2 transmission triggers the integration flow automatically.
properties:
_tpConnectorId:
type: string
format: objectId
x-celigo-refModel: tradingpartnerconnectors
description: |-
Trading Partner Connector that supplies the partner-specific EDI configuration —
communication protocol, document schemas, mappings, validation rules, and endpoint
details. Set this to link the export to all settings required for AS2 communication
with that partner.
x-celigo-ai-guidance:
- Reference to a TradingPartnerConnector document.
- |-
## TRADING PARTNER CONNECTOR OVERVIEW
A Trading Partner Connector in Celigo's integrator.io is a prebuilt, partner-specific integration
template that streamlines the setup and management of Electronic Data Interchange (EDI) transactions
with a designated trading partner. It encapsulates all requisite configurations:
- Communication protocol (e.g., AS2, FTP/SFTP, VAN)
- Document schemas (such as ANSI X12 or EDIFACT)
- Mappings
- Validation rules
- Endpoint details
- |-
## BENEFITS
By referencing a Trading Partner Connector through this field, organizations:
- Reduce manual setup time
- Ensure compliance with specific partner requirements
- Take advantage of Celigo's out-of-the-box EDI capabilities
- Process transactions reliably and securely
- Onboard new partners rapidly without building flows from scratch
This field is crucial for AS2 configurations as it links the export to all partner-specific
settings required for successful AS2 communication.
examples: ["60a2c4e6f321d800129a1a3c"]
blob:
type: boolean
description: |-
When true, retrieves raw files without parsing them into structured records (rendered as
a "Transfer" step in the flow UI). Use only when the file contents are not needed in
subsequent steps — for binary files or when parsing is handled downstream. Only
available on AS2 and VAN exports.
x-celigo-ai-guidance:
- |-
- **Behavior**: Retrieves raw files without parsing them into structured data records. Should only be used when the contents of the file will not be used in subsequent steps.
- **UI Appearance**: "Transfer" flow step
- **Required Config**: Configuration only available on AS2 and VAN exports (as2.blob = true)
- **Use Case**: Raw file transfers for binary files or when parsing is handled downstream
- **Important Note**: Use this when you want to handle the file as a raw blob without automatic parsing
required: []
references/schemas/clone-request.yml
CloneRequest:
type: object
description: Request body for cloning an export.
properties:
name:
type: string
description: Optional name for the cloned resource. If omitted, the server may generate a default clone name.
examples: ["Clone - Get Accounts"]
connectionMap:
type: object
description: |
Optional mapping of original connection ids to replacement connection ids.
Keys are source connection ids on the original resource; values are target connection ids.
additionalProperties:
type: string
example:
5f7c579b6411271af4e7cefa: 64a1234567890abcdef12345
5f7c579b6411271af4e7cefb: 64a1234567890abcdef12346
sandbox:
type: boolean
deprecated: true
description: Optional flag copied from the original resource when applicable.
additionalProperties: true
references/schemas/clone-response.yml
CloneResponse:
description: Response body for a clone operation. Some clone endpoints return the cloned resource, while others may return a list of related created resources.
oneOf:
- $ref: "./response.yml#/Export"
- type: array
items:
type: object
properties:
model:
type: string
description: Model name of the created resource (e.g., Flow, Export, Import).
_id:
type: string
format: objectId
description: Unique id of the created resource.
name:
type: string
description: Optional name of the created resource.
required:
- _id
references/schemas/delta.yml
Delta:
type: object
description: |-
Configures incremental exports that retrieve only records created or modified since the last
successful run. Required when the export's type is "delta"; omit for other export types.
When no cutoff is supplied, the platform-managed last-successful-run timestamp (exposed as
{{lastExportDateTime}}) is the lower bound — the first run behaves like a full export, and
after a failed run the next run reuses the last successful timestamp so changed records are
not missed.
x-celigo-ai-guidance:
- |-
This object is REQUIRED when the export's type field is set to "delta" and
should not be included for other export types.
Delta exports are designed for efficient synchronization by retrieving only
records that have been created or modified since the last execution.
- |-
## REQUIRED FIELDS
- ``startDate`` -- the initial cutoff timestamp the platform uses on the
first run (and as the floor for ``{{lastExportDateTime}}`` thereafter).
Adapter-specific validators (e.g. NetSuite's ``transformTwoDotZero``)
reject delta configs that omit ``startDate`` with
``missing_delta_startDate``, so this is effectively required for every
delta export.
- |-
## DEFAULT CUTOFF BEHAVIOR (NO USER-SUPPLIED CUTOFF)
When the user prompt does not specify an explicit cutoff timestamp,
default ``startDate`` to **today at midnight UTC** in ISO-8601 format
(e.g. ``"2026-05-14T00:00:00.000Z"``). This means the first run picks
up records changed today; subsequent runs use the platform-managed
``{{lastExportDateTime}}`` (the last successful run timestamp).
- First run: filters records changed since ``startDate``.
- Subsequent runs: uses ``{{lastExportDateTime}}`` as the lower bound;
``startDate`` becomes the floor when the platform timestamp is absent
(e.g. after a reset).
For optimal AI agent implementation, consider these guidelines:
1. Primary configuration method depends on adapter type:
- For HTTP exports: Use {{lastExportDateTime}} variable in relativeURI or body
- For specific application adapters: Use dateField to specify timestamp fields
2. The system automatically maintains the last successful run timestamp
- No need to store or manage timestamps in your own code
- First run fetches all records (equivalent to a standard export)
- Subsequent runs use this timestamp as the starting point
3. Error handling and recovery:
- If an export fails, the next run uses the last successful timestamp
- Records created/modified during a failed run will be included in the next run
- The lagOffset field can be used to handle edge cases
- |-
This object is REQUIRED when the export's type field is set to "delta" and
should not be included for other export types.
Delta exports are designed for efficient synchronization by retrieving only
records that have been created or modified since the last execution.
properties:
dateField:
type: string
x-celigo-excludeAdaptorTypes:
- http
x-celigo-ui-override: >-
Required by the export form when the export type is "delta" (delta.dateField is
required:true, visibleWhen type=delta). Encoded to mirror the form so builders produce
connectable configurations.
description: |-
Record timestamp field(s) compared against the last successful run time to identify
changed records. Accepts a single field or multiple comma-separated fields, processed
sequentially — useful when different operations update different timestamp fields.
If the flow's own downstream steps update the exported records, add export criteria
that exclude already-processed records (or use a creation-time field for process-once
flows) so each run doesn't re-export what the previous run wrote. Not supported on
HTTP exports: embed {{lastExportDateTime}} in the relativeURI or body instead;
including dateField there makes the configuration invalid. For Salesforce this
field is required and defaults to the standard timestamp fields (LastModifiedDate,
CreatedDate, SystemModstamp, LastActivityDate, LastViewedDate, LastReferencedDate) plus
any custom timestamp fields.
x-celigo-ai-guidance:
- Specifies one or more timestamp fields to filter records by modification date.
- |-
## FIELD BEHAVIOR
This field determines which record timestamp(s) are compared against the last successful run time
to identify changed records. Key characteristics:
- REQUIRED for most adapter types (except HTTP and REST where this field is not supported)
- Can reference a single field or multiple comma-separated fields
- Field(s) must exist in the source system and contain valid date/time values
- When multiple fields are specified, they are processed sequentially
- |-
## IMPLEMENTATION PATTERNS
### Single Field Pattern
```
"dateField": "lastModifiedDate"
```
- Records where lastModifiedDate > last run time are exported
- Most common pattern, suitable for most applications
- Works when a single field reliably tracks all changes
### Multiple Field Pattern
```
"dateField": "createdAt,lastModified"
```
- First exports records where createdAt > last run time
- Then exports records where lastModified > last run time
- Useful when different operations update different timestamp fields
- Handles cases where some records only have creation timestamps
###CRITICAL ADAPTOR-SPECIFIC INSTRUCTION:
- The adaptor type is HTTP. For HTTP exports, the "dateField" property MUST NOT be included in the delta configuration.
- HTTP exports use the {{lastExportDateTime}} variable directly in the relativeURI or body instead of dateField.
- DO NOT include "dateField" in your response. If you include it, the configuration will be invalid.
Example HTTP query with implicit delta:
```
"/api/v1/users?modified_since={{lastExportDateTime}}"
```
Example (nested resource with OData-style $filter):
```
"/api/v1/parents/{{record.parentId}}/children?$filter=<modifiedTimestampField> gt {{lastExportDateTime}}"
```
Substitute ``<modifiedTimestampField>`` with the actual
timestamp field the target entity exposes in its response. Do
NOT invent or copy field names from an example authored for a
different endpoint -- an unknown field name causes a runtime
"unknown property" / "invalid column" error the build cannot
catch. See the engine's filter-field grounding principle.
For Salesforce, this field is required and has the following default values:
- LastModifiedDate
- CreatedDate
- SystemModstamp
- LastActivityDate
- LastViewedDate
- LastReferencedDate
Also, any custom fields that are not listed above but are timestamp fields will be added to the default values.
```
- |-
## SELF-TRIGGERING FLOWS (WRITE-BACK TO SOURCE)
When a downstream step in the same flow saves the exported records
(status stamps, processed flags, enrichment writes), a
modification-time dateField re-qualifies every record the flow
touches — each run re-exports everything the previous run
processed, duplicating downstream effects and re-billing
per-record steps such as AI agents. Remedies, in order of
preference:
1. Keep the modification-time field and add export criteria that
exclude records already carrying the flow's own write — usually
the very field the write-back sets (e.g. "sync status is
empty"). This preserves update-capture semantics while making
the flow idempotent.
2. Switch to a creation-time field ONLY when the intent is to
process each record exactly once — this changes semantics, and
genuine edits to records will no longer be picked up.
Writes that do not save the exported record do not re-qualify it —
for example, attaching a NetSuite User Note through the Celigo
bundle's usernotes mapping does not save the parent record, so its
modification timestamp is unchanged.
examples: ["lastModifiedDate", "updatedAt", "modifiedOn", "createdAt,lastModified"]
dateFormat:
type: string
description: |-
Moment.js format string applied to the cutoff timestamp, including {{lastExportDateTime}}
when used in HTTP requests. Leave unset unless the source system requires a non-ISO8601
format; ISO 8601 is used by default. Date-only formats truncate the time portion,
widening the filter window.
x-celigo-ai-guidance:
- Defines the date/time format expected by the source system's API.
- |-
## FIELD BEHAVIOR
This field controls how the system formats the timestamp used for filtering:
- OPTIONAL: Only needed when the source system doesn't support ISO8601
- Default: ISO8601 format (YYYY-MM-DDTHH:mm:ss.sssZ)
- Uses Moment.js formatting tokens
- Directly affects the format of {{lastExportDateTime}} when used in HTTP requests
- |-
## IMPLEMENTATION PATTERNS
### Standard Date Format
```
"dateFormat": "YYYY-MM-DD" // 2023-04-15
```
- For APIs that accept date-only values
- Will truncate time portion (potentially creating a wider filter window)
### Custom DateTime Format
```
"dateFormat": "MM/DD/YYYY HH:mm:ss" // 04/15/2023 14:30:00
```
- For APIs with specific formatting requirements
- Especially common with older or proprietary systems
### Localized Format
```
"dateFormat": "DD-MMM-YYYY HH:mm:ss" // 15-Apr-2023 14:30:00
```
- For systems requiring locale-specific representations
- Often needed for ERP systems or regional applications
Leave this field unset unless the source system explicitly requires a non-ISO8601 format.
examples: ["YYYY-MM-DD", "MM/DD/YYYY", "DD-MMM-YYYY HH:mm:ss"]
lagOffset:
type: integer
description: |-
Buffer in milliseconds subtracted from the last successful run timestamp, creating an
overlapping window that catches records still propagating when the previous run executed.
Set it when records created or modified near the run time are occasionally skipped due to
replication or indexing delays. Keep it as low as possible — larger values reprocess
redundant records. A negative value shifts the window forward (look-ahead) instead of back.
x-celigo-ai-guidance:
- Specifies a time buffer (in milliseconds) to account for system data propagation delays.
- |-
## FIELD BEHAVIOR
This field addresses synchronization issues caused by replication or indexing delays:
- OPTIONAL: Only needed for systems with known data visibility delays
- Value is SUBTRACTED from the last successful run timestamp
- Creates an overlapping window to catch records that were being processed
during the previous export
- Measured in milliseconds (1000ms = 1 second)
- |-
## IMPLEMENTATION PATTERN
The formula for the effective filter date is:
```
effectiveFilterDate = lastSuccessfulRunTime - lagOffset
```
- |-
## COMMON VALUES
- 15000 (15 seconds): Typical for systems with short indexing delays
- 60000 (1 minute): Common for systems with moderate replication lag
- 300000 (5 minutes): For systems with significant processing delays
- |-
## DIAGNOSIS
This field should be configured when you observe:
- Records occasionally missing from delta exports
- Records created/modified near the export run time being skipped
- Inconsistent results between runs with similar data changes
IMPORTANT: Setting this value too high decreases efficiency by processing
redundant records. Set only as high as needed to avoid missed records.
examples: [15000, 30000, 60000]
startDate:
type: [string, 'null']
format: date-time
description: |-
Explicit lower-bound cutoff for the first run, overriding the default of starting from the
beginning of time. Subsequent runs use the platform-managed last-successful-run timestamp.
Set it to backfill from a specific point rather than exporting all history.
x-celigo-ai-guidance:
- |-
Initial cutoff timestamp for the first export run, in ISO-8601 UTC format (e.g.
``"2026-05-14T00:00:00.000Z"``).
REQUIRED for every delta export -- adapter-specific validators reject delta
configs without it (e.g.
NetSuite returns ``missing_delta_startDate`` from the ``transformTwoDotZero`` processor).
- |-
## DEFAULT
When the user prompt does not specify an explicit cutoff, default
to **today at midnight UTC** -- i.e. the current date with the time
portion zeroed out:
``"YYYY-MM-DDT00:00:00.000Z"`` where ``YYYY-MM-DD`` is the date the
flow is being authored.
- |-
## SUBSEQUENT RUNS
After the first successful run, the platform tracks the actual
last-run timestamp and exposes it as ``{{lastExportDateTime}}``;
``startDate`` becomes the floor used only when the platform
timestamp is absent (e.g. after a reset).
- |-
## EXAMPLES
- ``"2026-05-14T00:00:00.000Z"`` -- today at midnight UTC.
- ``"2025-01-01T00:00:00.000Z"`` -- explicit cutoff supplied by
the user (e.g. "only export sales orders created since 2025").
- |-
Initial cutoff timestamp for the first export run, in ISO-8601 UTC format (e.g.
``"2026-05-14T00:00:00.000Z"``).
REQUIRED for every delta export -- adapter-specific validators reject delta
configs without it (e.g.
NetSuite returns ``missing_delta_startDate`` from the ``transformTwoDotZero`` processor).
examples:
- "2024-01-01T00:00:00.000Z"
- '2026-05-14T00:00:00.000Z'
- '2025-01-01T00:00:00.000Z'
references/schemas/distributed.yml
Distributed:
type: object
description: Authentication settings for distributed (real-time listener) exports.
x-celigo-ai-guidance:
- Configuration object for distributed exports that require authentication.
- This object contains authentication credentials needed for distributed processing.
properties:
bearerToken:
type: string
description: |-
Bearer token used to authenticate inbound distributed export requests. Optional — most
distributed listeners (NetSuite, Salesforce real-time) use no token. Write-only and stored
encrypted; treat it as a sensitive credential.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR
This token provides authentication for the distributed export:
- Required for secure access to distributed endpoints
- Must be a valid bearer token format
- Used in Authorization header as "Bearer {token}"
- Should be kept secure and rotated regularly
- |-
## IMPLEMENTATION GUIDANCE
### Token management
- Store tokens securely (encrypted at rest)
- Implement token rotation policies
- Monitor token expiration dates
- Use environment variables for token storage
### Security considerations
- Never log bearer tokens in plain text
- Implement proper access controls
- Use HTTPS for all token transmissions
- Validate tokens on each request
IMPORTANT: Bearer tokens provide full access to the distributed export.
Treat them as sensitive credentials.
format: password
examples: ["eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...", "sk-1234567890abcdef"]
examples: [
{
"bearerToken": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c"
}
]
references/schemas/dynamodb.yml
DynamoDB:
type: object
description: |-
Defines how records are queried from DynamoDB tables. Required when the _connectionId field
references a DynamoDB connection; must not be included for other connection types. Basic
exports need region, method, tableName, keyConditionExpression, expressionAttributeNames,
and expressionAttributeValues; once exports (export type "once") additionally need
onceExportPartitionKey, plus onceExportSortKey for composite-key tables.
x-celigo-ai-guidance:
- Configuration object for Amazon DynamoDB data integration exports.
- |-
This object is REQUIRED when the _connectionId field references a DynamoDB
connection and must not be included for other connection types.
It defines how data is extracted from DynamoDB tables,
using query operations against NoSQL data structures.
required:
- region
- tableName
- keyConditionExpression
- expressionAttributeNames
- expressionAttributeValues
properties:
region:
type: string
enum: ["us-east-1", "us-east-2", "us-west-1", "us-west-2", "af-south-1", "ap-east-1", "ap-south-1", "ap-northeast-1", "ap-northeast-2", "ap-northeast-3", "ap-southeast-1", "ap-southeast-2", "ca-central-1", "eu-central-1", "eu-west-1", "eu-west-2", "eu-west-3", "eu-south-1", "eu-north-1", "me-south-1", "sa-east-1"]
x-enumDescriptions:
us-east-1: US East (N. Virginia).
us-east-2: US East (Ohio).
us-west-1: US West (N. California).
us-west-2: US West (Oregon).
af-south-1: Africa (Cape Town).
ap-east-1: Asia Pacific (Hong Kong).
ap-south-1: Asia Pacific (Mumbai).
ap-northeast-1: Asia Pacific (Tokyo).
ap-northeast-2: Asia Pacific (Seoul).
ap-northeast-3: Asia Pacific (Osaka).
ap-southeast-1: Asia Pacific (Singapore).
ap-southeast-2: Asia Pacific (Sydney).
ca-central-1: Canada (Central).
eu-central-1: Europe (Frankfurt).
eu-west-1: Europe (Ireland).
eu-west-2: Europe (London).
eu-west-3: Europe (Paris).
eu-south-1: Europe (Milan).
eu-north-1: Europe (Stockholm).
me-south-1: Middle East (Bahrain).
sa-east-1: South America (Sao Paulo).
description: |-
AWS region hosting the DynamoDB table. Must match the region where the table is
deployed so the integration can reach it.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR
This field determines where to connect to DynamoDB:
- REQUIRED for all DynamoDB exports
- Must match the region where your DynamoDB table is deployed
- Select the same AWS region used in your database configuration
- Ensures the integration can access your table
default: "us-east-1"
examples: ["us-east-1", "eu-west-1", "ap-southeast-2"]
method:
type: string
enum: ["query"]
x-enumDescriptions:
query: Retrieves items from DynamoDB using a key condition expression against a partition key.
description: |-
DynamoDB operation used to retrieve items. Only "query" is currently supported.
x-celigo-ai-guidance:
- Defines the DynamoDB operation method used to retrieve data.
- |-
## FIELD BEHAVIOR
- REQUIRED for all DynamoDB exports
- Currently only supports "query" operations
- Always set this value to "query"
- Additional methods may be supported in future versions
examples: ["query"]
tableName:
type: string
x-celigo-ui-override: >-
Required by the DynamoDB export form (dynamodb.tableName is required:true). Encoded to
mirror the form so builders produce connectable configurations.
description: |-
Name of the DynamoDB table to query. Must exactly match an existing table (case-sensitive).
x-celigo-ai-guidance:
- Specifies the DynamoDB table from which to retrieve data.
- |-
## FIELD BEHAVIOR
This field identifies the data source:
- REQUIRED for all DynamoDB exports
- Must be an exact match to an existing table name
- Case-sensitive as per AWS naming conventions
- Cannot be changed without recreating the export
- |-
## IMPLEMENTATION PATTERNS
### Standard Table Names
```
"tableName": "Customers"
```
examples: ["Customers", "Orders", "Product-Catalog", "Transactions-2023"]
keyConditionExpression:
type: string
description: |-
Key condition determining which items the query retrieves. Must include a condition on
the partition key and may add sort-key conditions (equality, BETWEEN, begins_with).
Reference attribute names with "#" placeholders defined in expressionAttributeNames and
values with ":" placeholders defined in expressionAttributeValues.
x-celigo-ai-guidance:
- Defines the search criteria to determine which items to retrieve from DynamoDB.
- |-
## FIELD BEHAVIOR
- REQUIRED when method="query"
- Must include a condition on the partition key
- Can optionally include conditions on the sort key
- Uses placeholders defined in expressionAttributeNames and expressionAttributeValues
- |-
## COMMON PATTERNS
```
"#pk = :pkValue" // Partition key only
"#pk = :pkValue AND #sk = :skValue" // Exact match on partition and sort key
"#pk = :pkValue AND #sk BETWEEN :start AND :end" // Range query on sort key
"#pk = :pkValue AND begins_with(#sk, :prefix)" // Prefix match on sort key
```
Placeholders with '#' reference attribute names, while ':' reference values.
examples: ["#pk = :customerId", "#pk = :orderId AND #sk BETWEEN :startDate AND :endDate", "#pk = :productId AND begins_with(#sk, :prefix)"]
filterExpression:
type: string
description: |-
Filters query results on non-key attributes after the key condition is applied; omit to
return all items matching the key condition. Uses the same "#" and ":" placeholders
defined in expressionAttributeNames and expressionAttributeValues.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR
- OPTIONAL: If omitted, all items matching the key condition are returned
- Applied after the key condition but before returning results
- Can reference any non-key attributes to further refine results
- Uses placeholders defined in expressionAttributeNames and expressionAttributeValues
- |-
## EXAMPLES
```
"#status = :active"
"#status = :active AND #price > :minPrice"
"contains(#tags, :tagValue)"
```
Refer to the DynamoDB documentation for the complete list of valid operators and syntax.
examples: ["#status = :active AND #price > :threshold", "attribute_exists(#timestamp) AND #quantity > :minQuantity", "contains(#categories, :category)"]
projectionExpression:
type: array
items:
type: string
description: |-
Attributes to return from each item, reducing data transfer; omit to return all
attributes. Each array element is one field, referenced via "#" placeholders defined in
expressionAttributeNames.
x-celigo-ai-guidance:
- Specifies which fields to return from each item in the results.
- |-
## FIELD BEHAVIOR
- OPTIONAL: If omitted, all fields are returned
- Each array element represents a field to include
- References attribute names defined in expressionAttributeNames
- Reduces data transfer by returning only needed fields
- |-
## EXAMPLES
```
["#id", "#name", "#email"] // Basic fields
["#id", "#profile.#firstName"] // Nested fields
["#id", "#items[0]", "#items[1]"] // List elements
```
Refer to the DynamoDB documentation for more details on projection syntax.
examples: [["#id", "#name", "#email"], ["#customerId", "#orderDate", "#items"], ["#partitionKey", "#sortKey", "#lastModified"]]
expressionAttributeNames:
type: string
description: |-
JSON string mapping "#" placeholders to actual attribute names, e.g.
{"#pk": "customerId"}. Placeholders defined here are used in keyConditionExpression,
filterExpression, and projectionExpression.
x-celigo-ai-guidance:
- Defines placeholders for attribute names used in expressions.
- |-
## FIELD BEHAVIOR
- REQUIRED when using expressions that reference attribute names
- Must be a valid JSON string mapping placeholders to actual attribute names
- Each placeholder must begin with a pound sign (#) followed by alphanumeric characters
- Used in keyConditionExpression, filterExpression, and projectionExpression
- |-
## EXAMPLE
```
"{\"#pk\": \"customerId\", \"#status\": \"status\"}"
```
This maps the placeholder #pk to the actual attribute name "customerId" and #status to "status".
Refer to the DynamoDB documentation for more details.
examples: ["{\"#pk\": \"id\", \"#sk\": \"timestamp\"}", "{\"#name\": \"name\", \"#price\": \"price\"}", "{\"#status\": \"status\", \"#date\": \"date\"}"]
expressionAttributeValues:
type: string
description: |-
JSON string mapping ":" placeholders to comparison values, e.g. {":status": "ACTIVE"}.
Values can be static or dynamic handlebars expressions such as {{lastExportDateTime}},
and are referenced from keyConditionExpression and filterExpression.
x-celigo-ai-guidance:
- Defines placeholder values used in expressions for comparison.
- |-
## FIELD BEHAVIOR
- REQUIRED when using expressions that compare attribute values
- Must be a valid JSON string mapping placeholders to actual values
- Each placeholder must begin with a colon (:) followed by alphanumeric characters
- Used in keyConditionExpression and filterExpression
- Can contain static values or dynamic values with handlebars syntax
- |-
## EXAMPLE
```
"{\":customerId\": \"12345\", \":status\": \"ACTIVE\"}"
```
This maps the placeholder :customerId to the value "12345" and :status to "ACTIVE".
Refer to the DynamoDB documentation for more details.
examples: ["{\":val\": \"12345\"}", "{\":start\": \"2023-01-01\", \":end\": \"{{lastExportDateTime}}\"}", "{\":status\": \"ACTIVE\", \":minValue\": 100}"]
onceExportPartitionKey:
type: string
x-celigo-ui-override: >-
Required by the DynamoDB export form when the export type is "once" (dynamodb.onceExportPartitionKey
is required, visibleWhen type=once). Encoded to mirror the form so builders produce
connectable configurations.
description: |-
Partition key attribute that uniquely identifies items when the export's type is "once".
Celigo uses it to mark items as processed after a successful export, preventing the same
items from being exported again on subsequent runs.
x-celigo-ai-guidance:
- Specifies the partition key attribute for identifying items in once exports.
- |-
## FIELD BEHAVIOR
- REQUIRED when export.type="once"
- Must specify the primary key that uniquely identifies each item in the table
- Celigo uses this to track which items have been processed
- After successful export, Celigo updates a tracking field in the database
This is needed for once exports to prevent duplicate processing of the same items
in subsequent runs by marking them as processed.
Refer to the DynamoDB documentation for more details on partition keys.
examples: ["id", "customerId", "orderNumber", "eventId"]
onceExportSortKey:
type: string
description: |-
Sort key attribute used together with onceExportPartitionKey to identify processed items
when the table has a composite primary key. Omit for tables keyed by a partition key
alone.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR
- OPTIONAL: Only needed for tables with composite primary keys
- Used together with onceExportPartitionKey for tables where items are identified by both keys
- Celigo uses both keys to uniquely identify items that have been processed
- For tables with only a partition key (simple primary key), leave this empty
This is only required if your DynamoDB table uses a composite primary key
(partition key + sort key) to uniquely identify items.
Refer to the DynamoDB documentation for more details on sort keys.
examples: ["timestamp", "email", "version", "sequenceNumber"]
pathToRecords:
$ref: './record-extraction.yml#/PathToRecords'
includeParentData:
$ref: './record-extraction.yml#/IncludeParentData'
references/schemas/file.yml
File:
type: object
description: |-
Controls how files are parsed, filtered, and processed across all file-based exports
(FTP/SFTP, Amazon S3, simple file uploads, and other file sources). The type field selects
the file format and determines which format-specific object (csv, json, xlsx, xml, or
fileDefinition) must be configured; the output field selects whether files are parsed into
records, transferred as blobs, or listed as metadata only. The filter object selectively
skips files before processing.
x-celigo-ai-guidance:
- |-
Configuration for file processing in exports.
This object defines how files are parsed, filtered,
and processed across all file-based export operations within Celigo.
- |-
## EXPORT CONTEXTS
This schema applies to multiple file-based export scenarios:
1. **Source System Types**:
- Simple exports with file uploads through the UI
- HTTP exports retrieving files from web sources
- FTP/SFTP exports downloading files from servers
- Amazon S3
- Azure Blob Storage
- Google Cloud Storage
- And other file-based source systems
- |-
## IMPLEMENTATION GUIDELINES
AI agents should consider these key decision points when configuring file processing for exports:
1. **File Format Selection**: Set the `type` field to match the format of the files being processed
(csv, json, xlsx, xml). This determines which format-specific configuration object to populate.
2. **Processing Mode**: Set the `output` field based on whether you need to:
- Parse file contents into records (`"records"`)
- Transfer files without parsing (`"blobKeys"`)
- Only retrieve metadata about files (`"metadata"`)
3. **File Filtering**: Use the `filter` object to selectively process files based on criteria
like file names, sizes, or custom logic.
4. **Format-Specific Configuration**: Configure the corresponding object (csv, json, xlsx, xml)
based on the selected file type.
- |-
## EXPORT-SPECIFIC CONSIDERATIONS
While the file processing configuration remains consistent, different export types may have
additional requirements:
- **HTTP Exports**: May need authentication and specific endpoint configurations
- **FTP/SFTP Exports**: Require server credentials and path information
- **Cloud Storage Exports**: Need bucket/container details and access credentials
The File schema focuses specifically on how files are processed once they are
retrieved from the source system, regardless of which export type is used.
properties:
encoding:
type: string
description: |-
Character encoding used to read and parse file content. If the encoding is unknown, try
utf8 first (the default), then win1252 for Western-language files with garbled
characters; consider the geographic origin of the data when selecting.
x-celigo-ai-guidance:
- |-
Character encoding used for reading and parsing file content.
This setting is critical for ensuring proper character interpretation,
especially for international data and special characters.
- |-
## ENCODING OPTIONS AND USAGE GUIDANCE
### UTF-8 (`"utf8"`)
- **Default Setting**: Used if no encoding is specified
- **Best For**: Modern text files, international character sets, XML/JSON files
- **Compatibility**: Universally supported; standard for web applications
- **When to Use**: First choice for most new integrations; handles most languages
### Windows-1252 (`"win1252"`)
- **Best For**: Legacy Windows system files, older Western European data
- **Compatibility**: Common in Windows-based exports, especially older systems
- **When to Use**: When files originate from older Windows systems or contain certain special characters not rendering properly in utf8
### UTF-16LE (`"utf-16le"`)
- **Best For**: Unicode text with extensive character requirements
- **Compatibility**: Microsoft Word documents, some database exports
- **When to Use**: When files have Byte Order Mark (BOM) or are known to be 16-bit Unicode
### GB18030 (`"gb18030"`)
- **Best For**: Chinese character sets
- **Compatibility**: Official character set standard for China
- **When to Use**: For files containing simplified or traditional Chinese characters
### Mac Roman (`"macroman"`)
- **Best For**: Legacy Mac system files (pre-OS X)
- **Compatibility**: Older Apple systems and applications
- **When to Use**: For older files created on Apple systems
### ISO-8859-1 (`"iso88591"`)
- **Best For**: Western European languages
- **Compatibility**: Widely supported in older systems
- **When to Use**: For legacy European language content
### Shift JIS (`"shiftjis"`)
- **Best For**: Japanese character sets
- **Compatibility**: Common in Japanese Windows and older systems
- **When to Use**: For files containing Japanese text
- |-
## IMPLEMENTATION GUIDANCE FOR AI AGENTS
1. **Detection Strategy**: If encoding is unknown, first try utf8 (default), then try win1252 for Western language files with errors
2. **Encoding Selection Process**:
- Check source system documentation for encoding specifications
- For files with corrupt/missing characters, test alternative encodings
- Consider geographic origin of data (Asian languages often require specific encodings)
3. **Common Issues to Watch For**:
- Mojibake (garbled text): Indicates wrong encoding selection
- Question marks or boxes: Character conversion failures
- BOM markers appearing as visible characters: Consider utf-16le
enum: ["utf8", "win1252", "utf-16le", "gb18030", "macroman", "iso88591", "shiftjis"]
x-enumDescriptions:
utf8: Universal character encoding supporting all languages (default, recommended for most files).
win1252: Windows-1252 encoding for legacy Western European text files from Windows systems.
utf-16le: UTF-16 Little Endian encoding for Unicode text, common in some Microsoft file exports.
gb18030: Chinese national standard encoding supporting simplified and traditional Chinese characters.
macroman: Mac Roman encoding for legacy files created on pre-OS X Apple systems.
iso88591: ISO 8859-1 encoding for Western European languages in older systems.
shiftjis: Shift JIS encoding for Japanese text, common in Japanese Windows systems.
examples: ["utf8", "win1252"]
type:
type: string
description: |-
Format of the files being processed; determines which format-specific configuration
object (csv, json, xlsx, xml, or fileDefinition) must be populated — other format
objects are ignored. Required for all file-based exports except blob exports (export
type "blob" or output "blobKeys").
x-celigo-ai-guidance:
- |-
This field creates a critical dependency that determines which format-specific
configuration object must be populated.
- |-
## FORMAT OPTIONS AND REQUIREMENTS
### CSV Files (`"csv"`)
- **Use For**: Tabular data with delimiter-separated values
- **Required Config**: The `csv` object with settings like delimiters and header options
- **Best For**: Simple tabular data, exports from spreadsheets, flat data structures
- **Example Sources**: Exported reports, data extracts, simple database dumps
### JSON Files (`"json"`)
- **Use For**: Hierarchical data in JavaScript Object Notation
- **Required Config**: The `json` object, especially the `resourcePath` to locate records
- **Best For**: Nested data structures, API responses, complex object representations
- **Example Sources**: REST APIs, document databases, configuration files
### Excel Files (`"xlsx"`)
- **Use For**: Microsoft Excel spreadsheets
- **Required Config**: The `xlsx` object with Excel-specific settings
- **Best For**: Business reports, formatted tabular data, multi-sheet documents
- **Example Sources**: Financial reports, manually created spreadsheets
### XML Files (`"xml"`)
- **Use For**: Extensible Markup Language documents
- **Required Config**: The `xml` object, critically the `resourcePath` using XPath
- **Best For**: Document-oriented data, SOAP responses, EDI formats
- **Example Sources**: SOAP APIs, legacy system exports, industry standard formats
### File Definition (`"filedefinition"`)
- **Use For**: Complex proprietary formats requiring custom parsing logic
- **Required Config**: The `fileDefinition` object with the _fileDefinitionId
- **Best For**: Legacy formats, fixed-width files, complex multi-record formats
- **Example Sources**: Mainframe exports, proprietary formats, EDI documents
- |-
## IMPLEMENTATION GUIDANCE
1. Determine the file format from the source system or documentation
2. Select the matching type from the enum values
3. Configure ONLY the corresponding format-specific object
4. Other format-specific objects will be ignored
For AI agents: This field creates a critical dependency chain - selecting a type
commits you to using the corresponding configuration object.
enum: ["csv", "json", "xlsx", "xml", "filedefinition"]
x-enumDescriptions:
csv: Parses delimiter-separated text files (CSV, TSV, pipe-delimited) into records.
json: Parses JSON files containing hierarchical or array-based data into records.
xlsx: Parses Microsoft Excel Open XML (.xlsx) spreadsheet files into records.
xml: Parses XML documents using XPath expressions to extract records.
filedefinition: Parses complex or proprietary file formats using a predefined file definition resource.
examples: ["csv", "json", "xlsx", "filedefinition"]
output:
type: string
description: |-
Processing mode for retrieved files: parse contents into records, transfer files as
unparsed blobs, or return only file metadata. Determines what data is passed to
subsequent flow steps.
x-celigo-ai-guidance:
- |-
Defines the fundamental processing mode for file data.
This critical field determines how files are handled and what data is passed to
subsequent flow steps.
- |-
## PROCESSING MODES
### Content Processing (`"records"`)
- **Behavior**: Files are parsed into structured records based on their format
- **Use When**: You need to access and manipulate the data inside files
- **Output**: Array of record objects reflecting the file's content
- **Example Flow**: CSV files → Parse into records → Transform → Import to target system
- **Best For**: Data synchronization, ETL processes, content-based workflows
- **Technical Impact**: Requires format-specific parsing; higher processing overhead
### File Transfer (`"blobKeys"`)
- **Behavior**: Files are treated as binary objects and transferred without parsing
- **Use When**: You need to move files between systems without modifying content
- **Output**: References to the binary file objects (blobKeys)
- **Example Flow**: Image files → Transfer as blobs → Upload to cloud storage
- **Best For**: Binary files, images, documents, any non-textual content
- **Technical Impact**: Lower processing overhead; maintains file integrity
### File Discovery (`"metadata"`)
- **Behavior**: Only file metadata is retrieved (name, size, dates) without content
- **Use When**: You need to inventory files before deciding which to process
- **Output**: Array of file metadata objects
- **Example Flow**: Scan FTP folder → Get metadata → Filter by date → Process selected files
- **Best For**: File inventory, selective processing, large directory scanning
- **Technical Impact**: Minimal processing overhead; fastest operation mode
- |-
## IMPLEMENTATION GUIDANCE
This setting profoundly affects flow architecture:
1. For data integration: Use `"records"` to work with the file contents
2. For file movement: Use `"blobKeys"` to preserve binary integrity
3. For file discovery: Use `"metadata"` as a first step before selective processing
AI agents should select this value based on whether the integration needs to
work with the file's content or just move/manage the files themselves.
enum: ["records", "metadata", "blobKeys"]
x-enumDescriptions:
records: Parses file contents into structured data records for processing in the flow.
metadata: Retrieves only file metadata (name, size, dates) without downloading or parsing content.
blobKeys: Transfers files as binary objects without parsing, preserving the original file content.
examples: ["records", "blobKeys", "metadata"]
skipDelete:
type: boolean
description: |-
When true, source files remain on the file server after processing; when false (the
default), files are deleted after successful processing. Files that fail processing are
never deleted, regardless of this setting. Enable retention when files must be processed
by other flows or kept for compliance.
x-celigo-ai-guidance:
- |-
Controls whether source files are retained or deleted after successful processing.
This setting has significant implications for data lifecycle management and system storage.
- |-
## BEHAVIOR
- **When true**: Source files remain on the file server after processing
- **When false** (default): Source files are automatically deleted after successful processing
- **Error Handling**: Files are only deleted after SUCCESSFUL processing; failed files remain intact
- |-
## DECISION FACTORS FOR AI AGENTS
Consider recommending `skipDelete: true` when:
1. **Compliance Requirements**:
- Regulatory frameworks require source file retention (GDPR, HIPAA, SOX)
- Audit trails need to maintain original file evidence
- Data retention policies mandate preserving source files
2. **Operational Needs**:
- Files need to be processed by multiple different flows
- Source files serve as disaster recovery backups
- Re-processing might be required (for testing or validation)
- Source systems do not maintain their own copy of the files
Consider recommending `skipDelete: false` (default) when:
1. **Storage Optimization**:
- Working with large files that would consume significant storage
- High volume of files processed frequently
- Files are already backed up elsewhere
- Storage costs are a concern
2. **Security Considerations**:
- Files contain sensitive data that should be minimized
- "Clean workspace" policies are in place
- Source files represent a potential security liability
- |-
## IMPLEMENTATION GUIDANCE
- **Storage Planning**: When `skipDelete: true`, ensure sufficient storage is available for file accumulation
- **File Organization**: Consider implementing an archiving strategy for retained files
- **Monitoring**: Set up space monitoring when retaining files to prevent storage exhaustion
- **Cleanup Automation**: If files must be retained but eventually deleted, consider a separate cleanup job
- |-
## INTEGRATION PATTERNS
- **Multi-stage Processing**: Set to `true` for files that need multi-step processing in separate flows
- **Extract-Transform-Archive**: Set to `true` when original files need archiving after extraction
- **Single-use Import**: Set to `false` for one-time imports where originals have no further value
- |-
## TECHNICAL CONSIDERATIONS
This setting only affects the source file server. Records extracted from the files and processed through the flow are not affected by this setting - they continue through your integration regardless of this value.
examples: [true, false]
compressionFormat:
type: string
description: |-
Compression format of incoming files, which are decompressed before any other processing
(parsing, filtering). Set this only when the source always delivers compressed files —
if a file marked as compressed is not actually compressed, processing fails. Leave unset
when files arrive uncompressed or only sometimes compressed.
x-celigo-ai-guidance:
- |-
Specifies the compression format of the files being processed.
This setting enables the system to automatically decompress files before parsing their contents.
- |-
## COMPRESSION OPTIONS
### GZIP (`"gzip"`)
- **File Extension**: Typically .gz, .gzip
- **Characteristics**: Single-file compression, maintains original file name in metadata
- **Compression Ratio**: Moderate to high, depends on file type (5-75% size reduction)
- **Common Sources**: Linux/Unix systems, database exports, API response payloads
- **Use Cases**: Individual file transfers, API response handling, log files
### ZIP (`"zip"`)
- **File Extension**: .zip
- **Characteristics**: Archive format that can contain multiple files/directories
- **Compression Ratio**: Moderate (usually 30-60% size reduction)
- **Common Sources**: Windows systems, manual exports, email attachments
- **Use Cases**: Multi-file packages, email attachments, mixed-format content
- |-
## IMPLEMENTATION GUIDANCE FOR AI AGENTS
### When to Configure Compression
1. **Source System Behavior**:
- Set when the source system always delivers compressed files
- Leave blank when files are delivered uncompressed
- NEVER set when files are sometimes compressed, sometimes not (this will cause errors)
2. **Selection Criteria**:
- Examine file extensions (.zip, .gz) in the source system
- Check source system documentation for compression specifications
- Consider typical OS of the source (.zip for Windows, .gz for Unix/Linux)
3. **Multi-file Considerations**:
- For .zip files containing multiple files, all files will be processed individually
- For nested compression (e.g., .gz files inside .zip), only the outer compression is handled
### Technical Implementation Notes
- **Processing Flow**: Files are decompressed before any other processing (parsing, filtering)
- **Filename Handling**: After decompression, the contained file name(s) are used for subsequent operations
- **Error Conditions**: If a file is marked as compressed but isn't actually compressed, processing will fail
- **Mixed Formats**: If files might arrive in different compression formats, use separate exports for each format
### Performance Considerations
- **Network Efficiency**: Compressed files reduce transfer time from source to integration platform
- **Processing Overhead**: Decompression adds slight processing time but saves network time
- **Storage Impact**: Compressed files use less storage in transit but are decompressed for processing
Leave this field blank if files are not compressed. Setting an incorrect compression format will cause processing errors.
enum: ["gzip", "zip"]
x-enumDescriptions:
gzip: Decompresses single-file gzip (.gz) compressed files before parsing.
zip: Decompresses zip (.zip) archive files, processing each contained file individually.
examples: ["gzip", "zip"]
purgeInternalBackup:
type: boolean
description: |-
When true, Celigo keeps no internal backup copies of files processed by this export;
when false (the default), copies are retained for your account's retention period and
are available for reprocessing or troubleshooting. Applies only to this export and only
to Celigo's internal copies — source files are governed by skipDelete. Enable for highly
sensitive data or zero-retention policies; without backups, recovery may require
re-obtaining files from the source system.
x-celigo-ai-guidance:
- |-
Controls whether Celigo's internal backup system retains copies of processed files.
This setting affects data retention, recovery capabilities,
and compliance posture.
- |-
## BEHAVIOR
- **When false** (default): Celigo maintains copies of all processed files for the duration of your account's retention policy
- **When true**: Celigo will NOT keep internal backup copies of files processed by this specific export
- **Scope**: This setting applies only to this specific export configuration; other exports are unaffected
- |-
## DECISION FACTORS FOR AI AGENTS
### Scenarios to Recommend `purgeInternalBackup: true`
1. **Data Sensitivity Requirements**:
- Files contain highly sensitive information (PII, PHI, financial, etc.)
- Data residency/sovereignty requirements prohibit additional copies
- Zero-retention policies mandate immediate deletion after processing
- Compliance frameworks require minimizing data copies (GDPR, HIPAA)
2. **Technical Considerations**:
- Very large files where storage costs are significant
- Files that are already reliably backed up in source systems
- Files with very short-lived relevance (e.g., temporary processing files)
- Processing of non-production/test data that doesn't require retention
### Scenarios to Recommend `purgeInternalBackup: false` (Default)
1. **Recovery Requirements**:
- Files represent critical business data with recovery needs
- Source systems don't maintain reliable backups
- Reprocessing capabilities are needed for disaster recovery
- Audit trails require evidence of processed files
2. **Operational Benefits**:
- Troubleshooting integration issues requires access to source files
- Files might need reprocessing in case of downstream errors
- Historical analysis or validation may be required
- Protection against source system data loss
- |-
## IMPLEMENTATION GUIDANCE
### Governance Considerations
- **Data Lifecycle**: Setting to `true` permanently removes files from Celigo after processing
- **Recovery Impact**: Without backups, recovery from certain errors may require re-obtaining files from source systems
- **Audit Trail**: Consider if processed files need to be available for future audits or investigations
### Best Practices
- **Document Decision**: When setting to `true`, document the rationale for disabling backups
- **Retention Alignment**: Ensure this setting aligns with overall data retention policies
- **Risk Assessment**: Evaluate recovery needs against data minimization requirements
- **Consistency**: Apply consistent backup settings across similar data types
### System Impact
This setting does NOT affect:
- The processing of files during integration runs
- Source files on their original servers (see `skipDelete` for that)
- Storage of processed data records in the target system
It ONLY controls whether Celigo maintains internal copies of the original files.
examples: [true, false]
decrypt:
type: string
description: |-
Decryption applied to incoming files before any other processing; decryption runs before
decompression, and a decryption failure fails the file's processing entirely. The
connection must already be configured with the private key (and passphrase, if
applicable) matching the public key used to encrypt the files. Only PGP/GPG encryption
is currently supported.
x-celigo-ai-guidance:
- |-
Specifies the decryption method to apply to incoming files before processing.
This setting enables handling of encrypted files that require decryption before
their contents can be parsed.
- |-
## SUPPORTED ENCRYPTION
### PGP/GPG Encryption (`"pgp"`)
- **File Extensions**: Typically .pgp, .gpg, or .asc
- **Encryption Standard**: OpenPGP (RFC 4880)
- **Key Requirements**: Private key must be configured on the connection
- **Common Sources**: Secure file transfers, encrypted backups, confidential data exchanges
- |-
## IMPLEMENTATION REQUIREMENTS
1. **Connection Configuration Prerequisites**:
- This field assumes the connection has already been configured with appropriate cryptographic settings
- Private key must be uploaded to the connection configuration
- Passphrase (if applicable) must be configured on the connection
- For asymmetric encryption, the corresponding public key must have been used to encrypt the files
2. **File Processing Flow**:
- Encrypted files are first retrieved from the source
- Decryption is applied using the configured connection's cryptographic settings
- After successful decryption, normal file processing continues (parsing, filtering, etc.)
- If decryption fails, the file processing will error out completely
- |-
## GUIDANCE FOR AI AGENTS
### When to Configure Decryption
1. **Security Requirements**:
- Set to "pgp" when source files are PGP/GPG encrypted
- Required for end-to-end encrypted data transfers
- Common in financial, healthcare, and other industries with sensitive data
- Essential for compliance with certain data protection regulations
2. **Technical Indicators**:
- File extensions indicate encryption (.pgp, .gpg, .asc)
- Source system documentation mentions PGP encryption
- Files cannot be opened with standard text editors
- Source system provides a public key for encryption
### Implementation Considerations
- **Key Management**: Ensure private keys are securely stored and properly configured
- **Error Handling**: Decryption failures will cause the entire file processing to fail
- **Performance Impact**: Decryption adds processing overhead before file parsing begins
- **Debugging Challenges**: Encrypted files cannot be easily examined for troubleshooting
### Security Best Practices
- **Key Rotation**: Recommend periodic key rotation according to security policies
- **Passphrase Protection**: Use strong passphrases for private keys when possible
- **Access Control**: Limit access to connections with decryption capabilities
- **Audit Logging**: Enable detailed logging for decryption operations when available
- |-
## INTEGRATION WITH OTHER SETTINGS
- If files are both encrypted AND compressed, decryption happens before decompression
- Subsequent processing (based on file type settings) occurs after decryption
- Internal backups (controlled by purgeInternalBackup) store the decrypted files unless configured otherwise
Currently, only PGP/GPG encryption is supported. For other encryption methods, custom preprocessing may be required.
enum: ["pgp"]
x-enumDescriptions:
pgp: Decrypts files encrypted with PGP/GPG (OpenPGP standard) before processing.
examples: ["pgp"]
batchSize:
type: [integer, 'null']
description: |-
Number of files retrieved per batch; if a batch fails, the whole batch is retried
(stored as `null` when not configured). Use
lower values (10-50) for large files to reduce timeout and memory pressure, and higher
values for many small files to improve throughput. Controls file retrieval only — record
paging is governed by the export's pageSize.
x-celigo-ai-guidance:
- |-
Controls the number of files processed in a single batch operation.
This setting allows fine-tuning of performance and resource utilization during file processing.
- |-
## BEHAVIOR AND PURPOSE
- **Function**: Limits the number of files processed in a single batch request
- **Maximum**: 1000 files per batch (hard system limit)
- **Impact**: Affects performance, memory usage, and error resilience, but NOT total processing capacity
- |-
## PERFORMANCE OPTIMIZATION GUIDANCE
### Large File Optimization (Set Lower Values: 10-50)
When working with large files (>10MB each), smaller batch sizes are recommended:
- **Network Benefits**: Reduces timeout risks during file transfer
- **Memory Usage**: Prevents excessive memory consumption
- **Error Isolation**: Limits the impact of processing failures
- **Example Scenarios**: Document processing, image files, complex spreadsheets
```
"batchSize": 20 // Good setting for large PDF or image files
```
### Small File Optimization (Set Higher Values: 100-1000)
When working with small files (<1MB each), larger batch sizes improve efficiency:
- **Throughput**: Processes more files with less overhead
- **API Efficiency**: Reduces the number of API calls
- **Resource Utilization**: Maximizes processing efficiency
- **Example Scenarios**: Small CSV files, transaction records, simple data files
```
"batchSize": 500 // Efficient for small data files
```
- |-
## IMPLEMENTATION GUIDANCE FOR AI AGENTS
### Recommendation Framework
1. **File Size Assessment**:
- For files averaging >10MB: Recommend 10-20
- For files averaging 1-10MB: Recommend 20-100
- For files averaging <1MB: Recommend 100-500
- For very small files (<100KB): Consider maximum (1000)
2. **Reliability Factors**:
- For critical data with no retry capability: Recommend lower values
- For unstable network connections: Recommend lower values
- For production environments: Start conservative (lower) and increase based on performance
- For development/testing: Can use higher values for efficiency
3. **System Constraints**:
- Consider available memory in the integration environment
- Evaluate network bandwidth and stability
- Account for source system rate limits or concurrent connection limits
- |-
## TECHNICAL CONSIDERATIONS
- **Error Handling**: If a batch fails, only that batch is retried (not individual files)
- **Parallelism**: Batch size affects concurrent processing but within system limits
- **Monitoring**: Larger batch sizes make monitoring individual file progress more difficult
- **Resource Scaling**: Higher batch sizes require more memory but can complete faster
- |-
## RELATIONSHIP TO OTHER SETTINGS
- This setting controls file retrieval batching, not record processing batch size
- Works in conjunction with compression and decryption settings
- Separate from and complementary to the main flow's pageSize setting
Consider starting with more conservative (lower) values and increasing based on performance monitoring.
maximum: 1000
examples: [10, 20, 50, 1000]
sortByFields:
type: array
description: |-
Sorts the records parsed from each file before they are processed, establishing a
deterministic processing order (for example, chronological or priority-based). Sorting
happens after parsing but before any filtering or grouping, and does not modify the
source files. Sorting by the same fields used in groupByFields improves grouping
performance.
x-celigo-ai-guidance:
- |-
Allows you to sort all records in a file before processing them.
This configuration enables deterministic ordering of records,
which can be critical for maintaining data consistency and enabling specific
processing patterns.
- |-
## FUNCTIONALITY OVERVIEW
- **Purpose**: Establishes a specific processing order for records within files
- **Timing**: Sorting is applied after file parsing but before any filtering or grouping
- **Scope**: Affects only the in-memory representation of records (doesn't modify source files)
- **Performance**: Has computational cost proportional to number of records × log(number of records)
- |-
## STRATEGIC USES FOR AI AGENTS
### Business Process Optimization
1. **Chronological Processing**:
- Sort by date/timestamp fields to process events in time order
- Essential for financial transactions, audit logs, event sequences
- Example: `[{"field": "transactionDate", "descending": false}]`
2. **Hierarchical Data Handling**:
- Sort by parent records before children
- Ensures referential integrity in relational data
- Example: `[{"field": "parentId", "descending": false}, {"field": "lineNumber", "descending": false}]`
3. **Priority-Based Processing**:
- Sort by importance/priority fields to handle critical items first
- Useful for SLA-driven processes, tiered operations
- Example: `[{"field": "priority", "descending": true}, {"field": "createdDate", "descending": false}]`
### Technical Optimization
1. **Grouping Efficiency**:
- Sorting by the same fields used in groupByFields improves grouping performance
- Reduces memory usage when processing large files
- Example: `[{"field": "customerId", "descending": false}]` with corresponding groupByFields
2. **Lookup Optimization**:
- Sorting by reference fields enhances performance of subsequent lookups
- Minimizes database calls by enabling batch lookups
- Example: `[{"field": "productSku", "descending": false}]`
3. **Error Reduction**:
- Sorting can ensure dependencies are processed in correct order
- Reduces failures from out-of-sequence processing
- Example: `[{"field": "sequenceNumber", "descending": false}]`
- |-
## IMPLEMENTATION GUIDANCE
### Field Selection Considerations
- **Data Type Compatibility**: Fields must contain comparable values (dates, numbers, strings)
- **Nulls Handling**: Null values are typically sorted last (after all non-null values)
- **Nested Fields**: Use dot notation for accessing nested properties (`customer.region`)
- **Performance Impact**: Each additional sort field increases computational cost
### Common Implementation Patterns
```json
// Simple single-field ascending sort (most common)
[
{"field": "orderDate", "descending": false}
]
// Multi-field sort with primary and secondary criteria
[
{"field": "region", "descending": false},
{"field": "revenue", "descending": true}
]
// Descending priority sort with tie-breaker
[
{"field": "priority", "descending": true},
{"field": "createdDate", "descending": false}
]
```
### Limitations and Constraints
- Sorting large datasets has memory implications; consider record volume
- Maximum recommended number of sort fields: 3-5 (performance considerations)
- Sorting effectiveness depends on data consistency in source files
- Complex sorting logic might be better implemented in custom scripts
items:
type: object
properties:
field:
type: string
description: |-
Record field to sort by; use dot notation for nested properties (e.g.
customer.name). Field names are case-sensitive.
x-celigo-ai-guidance:
- |-
Specifies the record field to use as a sort key.
This field name identifies which property of each record will be used for
comparison when establishing processing order.
- |-
## FIELD SELECTION GUIDELINES
### Data Type Considerations
- **Date/Time Fields**: Provide chronological sorting (`createdDate`, `timestamp`)
- **Numeric Fields**: Enable quantitative ordering (`amount`, `sequenceNumber`, `priority`)
- **String Fields**: Sort alphabetically (`name`, `status`, `category`)
- **Boolean Fields**: Group records by true/false values (`isActive`, `isProcessed`)
### Accessing Field Paths
- **Top-level Properties**: Direct field names (`orderNumber`, `date`)
- **Nested Objects**: Use dot notation (`customer.name`, `address.country`)
- **Array Elements**: Not directly supported in basic sorting; use preprocessing
### Common Field Patterns by Domain
1. **Order Processing**:
- `orderDate`, `orderNumber`, `customerId`, `lineNumber`
2. **Financial Data**:
- `transactionDate`, `accountNumber`, `amount`, `documentNumber`
3. **Customer Records**:
- `lastName`, `firstName`, `customerType`, `region`
4. **Inventory/Products**:
- `productCategory`, `itemNumber`, `stockLevel`, `reorderDate`
5. **Event Logs**:
- `timestamp`, `severity`, `eventType`, `sourceSystem`
- |-
## IMPLEMENTATION NOTES
- Field names are case-sensitive
- Fields must exist in all records (or have consistent representation when missing)
- Non-existent fields or null values are typically sorted last
- Maximum recommended field name length: 64 characters
examples: ["date", "priority", "orderNumber", "customer.name"]
descending:
type: boolean
description: |-
When true, sorts this field in descending order (newest/highest first); when false
or omitted, sorts ascending. Directions can be mixed across fields in a
multi-field sort.
x-celigo-ai-guidance:
- |-
Controls the sort direction for the specified field.
This setting determines whether records will be arranged in ascending (lowest to
highest) or descending (highest to lowest) order.
- |-
## BEHAVIOR
- **When false or omitted**: Sorts in ascending order (A→Z, 0→9, oldest→newest)
- **When true**: Sorts in descending order (Z→A, 9→0, newest→oldest)
- |-
## STRATEGIC DIRECTION SELECTION
### Ascending Order (descending: false)
Recommended for:
- Chronological event processing (earliest first)
- Sequential operations with dependencies
- Reference data that builds on previous records
- Incremental ID or sequence numbers
Example use cases:
- Processing dated transactions in chronological order
- Handling items in order of creation
- Incrementally building state that depends on previous records
### Descending Order (descending: true)
Recommended for:
- Priority-based processing (highest first)
- Recent-first temporal processing
- Most significant items first
- Limited processing where only top N items matter
Example use cases:
- Processing high-priority items before low-priority
- Handling most recent updates first
- Focusing on highest-value transactions first
- |-
## IMPLEMENTATION PATTERNS
### Single Field Direction
```json
{"field": "createdDate", "descending": false} // Oldest first
{"field": "createdDate", "descending": true} // Newest first
```
### Mixed Directions in Multi-field Sorts
```json
// Group by category (A→Z) but show highest priority first in each category
[
{"field": "category", "descending": false},
{"field": "priority", "descending": true}
]
```
- |-
## TECHNICAL CONSIDERATIONS
- Default value is `false` if omitted (ascending sort)
- For date fields, ascending means oldest first
- For numeric fields, ascending means smallest first
- For string fields, ascending means alphabetical order
examples: [true, false]
groupByFields:
$ref: './group-by.yml#/GroupBy'
groupEmptyValues:
type: boolean
description: |-
When true, records whose groupByFields values are empty are still grouped together rather
than each forming its own group; when false (the default), empty-keyed records are not
grouped. Only relevant when groupByFields is set.
examples: [true, false]
csv:
type: object
description: |-
Parsing settings for delimiter-separated text files (CSV, TSV, pipe-delimited, and
similar). Configure when type is "csv".
Required when type is csv.
x-celigo-ai-guidance:
- |-
Configuration settings for parsing CSV (Comma-Separated Values) files.
This object defines how the system interprets delimited text files,
handling variations in format, structure, and content.
- |-
## WHEN TO USE
Configure this object when the `type` field is set to "csv". This configuration is required for properly parsing:
- Standard CSV files (.csv)
- Tab-delimited files (.tsv, .tab)
- Other character-delimited files (semicolon, pipe, etc.)
- Fixed-width text files converted to delimited format
- |-
## IMPLEMENTATION STRATEGY FOR AI AGENTS
1. **Format Analysis**:
- Examine sample files to identify delimiter pattern
- Check for presence/absence of header row
- Look for whitespace or quote pattern inconsistencies
- Identify any rows that should be skipped (headers, metadata, etc.)
2. **Configuration Priority**:
- `columnDelimiter`: Most critical setting; incorrect delimiter causes parsing failures
- `hasHeaderRow`: Affects field mapping and identification
- `rowDelimiter`: Usually auto-detected but important for non-standard files
- `trimSpaces`: Important for inconsistent formatting
- `rowsToSkip`: Necessary when files contain metadata/comments before data
3. **Common File Source Patterns**:
| Source System | Typical Delimiter | Header Row | Common Issues |
|--------------|-------------------|------------|---------------|
| Excel (US) | Comma (,) | Yes | Quoted fields with embedded commas |
| Excel (EU) | Semicolon (;) | Yes | Decimal separator conflicts |
| Legacy Systems | Pipe (\|) or Tab | Varies | Inconsistent field counts |
| POS Systems | Comma or Tab | Often No | Trailing delimiters |
| ERP Exports | Varies widely | Usually Yes| Fixed field counts with padding |
- |-
## ERROR PREVENTION
- **Misaligned Columns**: Usually caused by incorrect delimiter or quotes handling
- **Truncated Data**: Can result from wrong row delimiter settings
- **Field Misinterpretation**: Often caused by incorrect header row setting
- **Character Encoding Issues**: Address with the parent `encoding` setting
- **Whitespace Problems**: Resolve with `trimSpaces` setting
- |-
## OPTIMIZATION OPPORTUNITIES
- For maximum parsing speed, set only the minimal required settings
- For problematic files with inconsistent formatting, use more restrictive settings
- Balance between permissive parsing (more data accepted) and strict validation (cleaner data)
properties:
columnDelimiter:
type: string
description: |-
Character sequence separating fields within each row; comma when omitted. Use "\t"
for tab-delimited files; European-locale exports often use semicolons. An incorrect
delimiter is the most common cause of parsing failures.
x-celigo-ai-guidance:
- |-
Specifies the character sequence that separates individual fields (columns)
within each row of the CSV file.
- |-
## BEHAVIOR
- Controls how the parser identifies individual fields in each row
- Default value: comma (,) if not specified
- Special characters may need to be escaped
- |-
## COMMON DELIMITER PATTERNS
### Standard CSV (`,`)
```
"columnDelimiter": ","
```
- Most common format in US/UK systems
- Default for most spreadsheet exports
- Used by: Microsoft Excel (US), Google Sheets, many database exports
### European CSV (`;`)
```
"columnDelimiter": ";"
```
- Common in European locales where comma is the decimal separator
- Standard format in many EU countries
- Used by: Microsoft Excel (many EU locales), European business systems
### Tab-Delimited (`\t`)
```
"columnDelimiter": "\t"
```
- Used for tab-separated values (TSV) files
- Better for data containing commas
- Used by: Database exports, scientific data, legacy systems
### Other Common Delimiters
- Pipe: `"columnDelimiter": "|"` (used in mainframes, legacy systems)
- Colon: `"columnDelimiter": ":"` (less common, specialized formats)
- Space: `"columnDelimiter": " "` (uncommon, problematic with text fields)
- |-
## DETERMINATION STRATEGY FOR AI AGENTS
1. **File Extension Check**:
- .csv → Usually comma (,)
- .tsv → Always tab (\t)
- .txt → Could be any delimiter; needs inspection
2. **Source System Analysis**:
- EU-based systems often use semicolon (;)
- Legacy/mainframe systems often use pipe (|)
- Scientific/statistical data often uses tab (\t)
3. **File Content Inspection**:
- Open file in text editor to identify separating character
- Check for character frequency patterns
- Look for consistent character between data elements
4. **System Documentation**:
- Check export settings in source system
- Review file specifications if available
- |-
## IMPLEMENTATION NOTES
- For tab delimiter, use `"\t"` (escape sequence for tab)
- If file contains the delimiter within text fields, ensure proper quoting
- Multi-character delimiters are supported but rare
- Setting the wrong delimiter is the most common parsing error
examples: [",", ";", "\t"]
rowDelimiter:
type: string
description: |-
Character sequence marking the end of each record; auto-detected from file content
when omitted. Set explicitly ("\n", "\r\n", or "\r") only when auto-detection fails,
such as for files with mixed line endings — an incorrect value merges or splits
records.
x-celigo-ai-guidance:
- Specifies the character sequence that indicates the end of each record (row) in the CSV file.
- |-
## BEHAVIOR
- Controls how the parser identifies the boundaries between records
- Default: Auto-detect (system attempts to determine from file content)
- Common values: newline (`\n`), carriage return + newline (`\r\n`)
- |-
## COMMON ROW DELIMITER PATTERNS
### Windows-Style (`\r\n`)
```
"rowDelimiter": "\r\n"
```
- CRLF (Carriage Return + Line Feed) sequence
- Standard for files created on Windows systems
- Used by: Microsoft Office, Windows-based applications
### Unix-Style (`\n`)
```
"rowDelimiter": "\n"
```
- LF (Line Feed) character only
- Standard for files created on Unix/Linux/macOS (modern) systems
- Used by: Linux applications, macOS applications, web exports
### Classic Mac-Style (`\r`)
```
"rowDelimiter": "\r"
```
- CR (Carriage Return) character only
- Legacy format used by older Mac systems (pre-OSX)
- Rare in modern files but still found in some legacy systems
- |-
## WHEN TO SPECIFY EXPLICITLY
In most cases, the auto-detection works well, but explicitly set this when:
1. **Mixed Line Endings**: Files containing inconsistent line ending styles
2. **Custom Record Separators**: Files using unconventional record delimiters
3. **Parsing Errors**: When auto-detection fails to correctly separate records
4. **Performance Optimization**: To avoid detection overhead in high-volume processing
- |-
## DETERMINATION STRATEGY FOR AI AGENTS
1. **Source System Analysis**:
- Windows systems typically use `\r\n`
- Unix/Linux/macOS typically use `\n`
- Web downloads could use either format
2. **Troubleshooting Guidance**:
- If records are merged or split incorrectly, check for proper row delimiter
- If file opens correctly in text editor but parsing fails, row delimiter may be the issue
- For files with unusual record counts, examine row delimiter setting
- |-
## IMPLEMENTATION NOTES
- Use escape sequences (`\n`, `\r\n`, `\r`) to represent control characters
- Setting incorrect row delimiter may result in merged records or split records
- When in doubt, leave unspecified to use auto-detection
- Multi-character delimiters beyond standard line endings are supported but rare
examples: ["\n", "\r\n"]
hasHeaderRow:
type: boolean
description: |-
When true (the default), the first row is read as field names rather than data, and
those names are available in mappings. When false, every row is treated as a data
record and fields are referenced by position.
x-celigo-ai-guidance:
- Indicates whether the CSV file contains a header row with field names as the first row.
- |-
## BEHAVIOR
- **When true** (default): First row is treated as field names, not data
- **When false**: All rows including the first are treated as data records
- Impacts field mapping, validation, and record counting
- |-
## IMPLEMENTATION IMPACT
### With Header Row (true)
- Field names from the header row can be referenced in mappings
- Record count excludes the header row
- First row of data is the second physical row in the file
- Provides self-documenting data structure
### Without Header Row (false)
- Fields are referenced by position/index (e.g., Column1, Column2)
- Record count includes all rows in the file
- First row of data is the first physical row in the file
- Requires external schema or position-based mapping
- |-
## DETERMINATION STRATEGY FOR AI AGENTS
1. **Visual Inspection**:
- Check if the first row contains descriptive labels rather than actual data
- Look for data type consistency (headers are typically text, while data may be mixed)
- Headers often use camelCase, PascalCase, or snake_case formatting
2. **Source System Analysis**:
- Most business systems include headers by default
- Legacy/mainframe systems may omit headers
- Data extracts intended for human use typically include headers
3. **Content Patterns**:
- Headers typically don't match the pattern of subsequent data rows
- Headers often contain special characters not found in data (spaces, symbols)
- Data rows typically have consistent patterns while headers may differ
- |-
## COMMON CONFIGURATIONS BY SOURCE
| Source Type | Typical Setting | Notes |
|-------------|-----------------|-------|
| Business Reports | true | Headers provide field context |
| Database Exports | true | Column names as headers |
| Legacy System Feeds | false | Often position-based fixed formats |
| IoT/Sensor Data | false | Often compact, headerless formats |
| Manual Data Entry | true | Helps maintain field alignment |
- |-
## BEST PRACTICES
- Always explicitly set this value rather than relying on the default
- For data without headers, consider adding them in preprocessing if possible
- When headers exist but should be ignored, use `hasHeaderRow: true` and `rowsToSkip: 1`
- Document field positions when working with headerless files
examples: [true, false]
trimSpaces:
type: boolean
description: |-
When true, removes leading and trailing whitespace from every field value during
parsing; when false (the default), whitespace is preserved exactly as in the source.
Header row values are always trimmed regardless of this setting, and spaces between
words are never affected.
x-celigo-ai-guidance:
- |-
Controls whether leading and trailing whitespace should be removed from field
values during parsing.
- |-
## IMPLEMENTATION IMPACT
### With Trimming Enabled (true)
- More consistent data for comparison and matching operations
- Prevents issues with invisible whitespace affecting equality checks
- Reduces storage space for text-heavy datasets
- Helps normalize data from inconsistent sources
### With Trimming Disabled (false)
- Preserves exact data as represented in the source file
- Required when whitespace is semantically meaningful
- Maintains original field lengths exactly
- Necessary for certain data validation scenarios
- |-
## USAGE GUIDANCE FOR AI AGENTS
### Recommend `trimSpaces: true` when:
1. **Data Consistency Issues**:
- Source systems are known to have inconsistent spacing
- Data will be used for matching or comparison operations
- Files are generated by multiple different systems
- Human-entered data is present (prone to spacing errors)
2. **Data Type Considerations**:
- Fields contain numeric values (where spaces are not meaningful)
- Fields contain codes, IDs, or reference values
- Fields will be used in lookups or joins
- Normalization is more important than exact representation
### Recommend `trimSpaces: false` when:
1. **Data Fidelity Requirements**:
- Working with fixed-width fields where spaces matter
- Dealing with formatted data where spacing is semantic
- Legal or compliance scenarios requiring exact preservation
- Scientific data where precision of representation matters
2. **Content Characteristics**:
- Working with text fields where leading/trailing spaces could be intentional
- Processing creative content, addresses, or formatted text
- Source system uses space padding for alignment purposes
- |-
## IMPLEMENTATION NOTES
- This setting affects all fields consistently (cannot be applied to select fields)
- Only affects leading and trailing spaces, not spaces between words
- Has no effect on empty fields (empty remains empty)
- For selective trimming, use transformation rules after parsing
examples: [true, false]
rowsToSkip:
type: integer
description: |-
Number of rows at the top of the file to ignore before parsing begins — useful for
report titles, timestamps, or other metadata above the data. The header row (when
hasHeaderRow is true) is expected after the skipped rows.
x-celigo-ai-guidance:
- |-
Specifies the number of rows at the beginning of the file to ignore before
starting data processing.
- |-
## BEHAVIOR
- Skips the specified number of rows from the beginning of the file
- These rows are completely ignored and not processed as data
- The header row (if present) is counted after the skipped rows
- Default value is 0 (no rows skipped)
- |-
## IMPLEMENTATION IMPACT
### Common Use Cases
1. **Metadata Headers**:
- Skip report titles, generated timestamps, system information
- Skip explanatory text at the beginning of files
- Skip company letterhead or report identification rows
2. **Multi-Header Files**:
- Skip category headers or section titles
- Skip nested headers or hierarchy information
- Skip column grouping indicators
3. **Technical Requirements**:
- Skip binary file markers or encoding identifiers
- Skip non-data content like instructions or disclaimers
- Skip inconsistent early rows before standardized data begins
- |-
## CALCULATION GUIDANCE FOR AI AGENTS
When determining the correct value for `rowsToSkip`:
1. **Count from Zero**:
- Row 1 = 0, Row 2 = 1, Row 3 = 2, etc.
2. **For Files with Headers**:
- Set rowsToSkip = (first data row position - 1) - (hasHeaderRow ? 1 : 0)
- Example: If data starts on row 5, and file has a header row:
rowsToSkip = (5 - 1) - 1 = 3
3. **For Files without Headers**:
- Set rowsToSkip = (first data row position - 1)
- Example: If data starts on row 3, and file has no header row:
rowsToSkip = (3 - 1) = 2
- |-
## DETERMINATION STRATEGY
1. **Visual Inspection**:
- Open file in text editor and count non-data rows at the top
- Identify the first row containing actual data values
- Note if a header row exists separately from skipped content
2. **Common Patterns by Source**:
- ERP Reports: Often 2-5 rows of report metadata
- Exported Spreadsheets: May have title rows, date stamps
- Database Extracts: Usually minimal (0-1) skipped rows
- Legacy Systems: May have control records or job information
- |-
## IMPLEMENTATION NOTES
- Setting too high skips valid data; setting too low includes non-data as records
- When in doubt, visually inspect the file to confirm correct skip count
- Remember that header row (if hasHeaderRow=true) is counted AFTER skipped rows
- Maximum recommended value: 100 (larger values may indicate format misunderstanding)
examples: [0, 1, 2]
disableQuoteAndStripEnclosingQuotes:
type: boolean
description: |-
When true, disables CSV quote processing: quotes are treated as literal characters,
enclosing quotes are stripped, and delimiters inside quoted text split the field.
When false (the default), standard RFC 4180 quoting applies and quoted fields
protect embedded delimiters. Enable only for files with non-standard or malformed
quoting; review this setting first when field counts vary unexpectedly between rows.
x-celigo-ai-guidance:
- |-
Controls the handling of quoted fields in CSV files,
specifically how the parser manages quotation marks around field values.
- |-
## BEHAVIOR
- **When false** (default): Standard CSV quoting rules are applied
- Quotation marks around fields protect embedded delimiters
- Parser intelligently handles escaped quotes within quoted fields
- Follows RFC 4180 CSV specifications for quote handling
- **When true**: Quote detection and processing is disabled
- All quotes are treated as literal characters, not field delimiters
- Any quotes surrounding field values are removed
- Embedded delimiters in quoted fields will cause field splitting
- |-
## IMPLEMENTATION IMPACT
### Standard Quote Handling (false)
Example input: `"Smith, John",42,"Notes with ""quotes"" inside"`
Result:
- Field 1: `Smith, John` (comma preserved inside quotes)
- Field 2: `42`
- Field 3: `Notes with "quotes" inside` (embedded quotes normalized)
### Disabled Quote Handling (true)
Example input: `"Smith, John",42,"Notes with ""quotes"" inside"`
Result:
- Field 1: `"Smith`
- Field 2: ` John"`
- Field 3: `42`
- Field 4: `"Notes with ""quotes"" inside"`
- |-
## USAGE GUIDANCE FOR AI AGENTS
### Recommend `disableQuoteAndStripEnclosingQuotes: true` when:
1. **Quote-Related Parsing Problems**:
- Files contain inconsistent or malformed quote usage
- Source system doesn't follow standard CSV quoting rules
- Quotes appear as literal data rather than field delimiters
- Quotes are present but delimiters are never embedded in fields
2. **Special Data Formats**:
- Working with custom delimited formats that don't use quotes for escaping
- Files use alternate escaping mechanisms for embedded delimiters
- Source system adds quotes to all fields regardless of content
### Recommend `disableQuoteAndStripEnclosingQuotes: false` (default) when:
1. **Standard CSV Compliance**:
- Files follow RFC 4180 or similar CSV standards
- Fields contain embedded delimiters that must be preserved
- Quotes are used properly to enclose fields with special characters
- Source is a standard database, spreadsheet, or business system export
2. **Data Content Characteristics**:
- Fields contain embedded commas, newlines, or other delimiters
- Text fields might contain quotation marks as part of the content
- Preserving the exact structure of complex text fields is important
- |-
## TROUBLESHOOTING INDICATORS
Consider changing this setting when encountering these issues:
- Field counts vary unexpectedly between rows
- Text with embedded delimiters is being split into multiple fields
- Quotes appearing at the beginning and end of every field in the result
- Extra quote characters appearing within field values
- |-
## IMPLEMENTATION NOTES
- This setting significantly changes parsing behavior - test thoroughly
- Affects all fields in the file consistently
- Incorrect setting can cause severe data misalignment
- When field count inconsistency occurs, review this setting first
examples: [true, false]
keyColumns:
type: array
description: |-
Field names whose values together identify a record's group when consecutive rows
share the same key — used to merge multi-line records (e.g. an order header repeated
across line-item rows) into a single record. Leave empty for one record per row; the
columns must exist in the parsed header.
items:
type: string
examples: [["Event Type"], ["settlement-id", "order-id", "transaction-type", "marketplace-name"]]
json:
type: object
description: |-
Parsing settings for JSON files. Configure when type is "json". resourcePath locates the
array of records when they are nested inside a container object; malformed JSON fails
the entire file's processing.
Required when type is json.
x-celigo-ai-guidance:
- |-
## WHEN TO USE
Configure this object when the `type` field is set to "json". This configuration is required for properly parsing:
- Standard JSON files (.json)
- JSON data exports from APIs or databases
- JSON Lines format (newline-delimited JSON)
- Nested or hierarchical data structures
- |-
Configuration settings for parsing JSON (JavaScript Object Notation) files.
This object defines how the system interprets and processes hierarchical data
contained in JSON-formatted files.
- |-
## JSON PARSING CHARACTERISTICS
- **Hierarchical Data**: JSON naturally supports nested objects and arrays
- **Type Preservation**: Numbers, booleans, nulls, and strings are correctly typed
- **Flexible Structure**: Can handle varying record structures
- **Tree Navigation**: Supports complex object traversal via path expressions
- |-
## IMPLEMENTATION STRATEGY FOR AI AGENTS
1. **Data Structure Analysis**:
- Examine sample files to understand the object hierarchy
- Identify where the actual records/rows are located in the structure
- Determine if records are at the root or nested within containers
- Check for array structures that contain the target records
2. **Common JSON Data Patterns**:
### Root Array Pattern
```json
[
{"id": 1, "name": "Product 1"},
{"id": 2, "name": "Product 2"}
]
```
- Records are directly at the root as an array
- No resourcePath needed (leave blank)
- Most straightforward structure for processing
### Container Object Pattern
```json
{
"data": [
{"id": 1, "name": "Product 1"},
{"id": 2, "name": "Product 2"}
],
"metadata": {
"count": 2,
"page": 1
}
}
```
- Records are in an array inside a container object
- Requires resourcePath (e.g., "data")
- Common in API responses with metadata
### Nested Container Pattern
```json
{
"response": {
"results": [
{"id": 1, "name": "Product 1"},
{"id": 2, "name": "Product 2"}
],
"pagination": {
"nextPage": 2
}
},
"status": "success"
}
```
- Records are deeply nested in the hierarchy
- Requires dot notation in resourcePath (e.g., "response.results")
- Common in complex API responses
- |-
## ERROR PREVENTION
- **Invalid Path**: Incorrectly specified resourcePath results in zero records found
- **Type Mismatch**: resourcePath must point to an array of objects for proper record processing
- **Empty Results**: If path resolves to null or non-existent field, no error is thrown but no records are processed
- **Parsing Failures**: Malformed JSON will cause the entire file processing to fail
- |-
## OPTIMIZATION OPPORTUNITIES
- For large JSON files, consider preprocessing to extract only relevant sections
- For files with complex structures, validate the resourcePath with sample data
- When processing API responses, coordinate resourcePath with the API documentation
- For very large datasets, consider using streaming JSON parsing (NDJSON format)
properties:
resourcePath:
type: string
description: |-
Dot-notation path to the array of records within the JSON structure (e.g.
"response.data.customers"); leave empty when the file's root is already the record
array. The path must resolve to an array of objects — array indexing and wildcards
are not supported. A path that resolves to nothing produces zero records without
raising an error.
x-celigo-ai-guidance:
- |-
Specifies the path to the array of records within the JSON structure.
This field helps the system locate and extract the target records when they're
nested inside a larger JSON object hierarchy.
- |-
## BEHAVIOR
- **Purpose**: Identifies where the array of records is located in the JSON structure
- **Format**: Dot notation path to navigate nested objects (e.g., "data.records")
- **When Empty**: System expects records to be at the root level as an array
- **Result**: Array found at this path is processed as individual records
- |-
## PATH NOTATION GUIDELINES
### Basic Path Patterns
- **Root Level Array**: Leave empty or null when records are a direct array at root
- **Single Level Nesting**: Use the property name (e.g., "data", "results", "items")
- **Multi-Level Nesting**: Use dot notation (e.g., "response.data.items")
### Path Construction Rules
1. **Object Navigation**:
- Use dots to traverse object properties: "parent.child.grandchild"
- Each segment must be a valid property name in the JSON
2. **Target Requirements**:
- The path MUST resolve to an array of objects
- Each object in the array will be processed as one record
- The array must be the final element in the path
3. **Limitations**:
- Array indexing is not supported in the path (e.g., "data[0]")
- Wildcard selectors are not supported
- Regular expressions are not supported
- |-
## DETERMINATION STRATEGY FOR AI AGENTS
To identify the correct resourcePath:
1. **Examine Sample Data**:
- Open a sample JSON file or API response
- Locate the array containing the actual data records
- Note the full path from root to this array
2. **Common Patterns by Source**:
| Source Type | Common Paths | Example |
|-------------|--------------|---------|
| REST APIs | "data", "results", "items" | "data" |
| Complex APIs | "response.data", "data.items" | "response.data" |
| Database Exports | "rows", "records", "exports" | "rows" |
| CRM Systems | "contacts", "accounts", "opportunities" | "contacts" |
| Analytics APIs | "data.rows", "response.data.rows" | "data.rows" |
3. **Verification Approach**:
- The path should resolve to an array (typically with square brackets in the JSON)
- Each element in this array should represent one complete record
- The array should not be a property array (like tags or categories)
- |-
## IMPLEMENTATION EXAMPLES
### Root Array (No Path Needed)
JSON Structure:
```json
[
{"id": 1, "name": "Record 1"},
{"id": 2, "name": "Record 2"}
]
```
Configuration:
```json
"resourcePath": "" // or omit entirely
```
### Single-Level Nesting
JSON Structure:
```json
{
"orders": [
{"id": "A001", "customer": "John"},
{"id": "A002", "customer": "Jane"}
],
"count": 2
}
```
Configuration:
```json
"resourcePath": "orders"
```
### Multi-Level Nesting
JSON Structure:
```json
{
"response": {
"data": {
"customers": [
{"id": 1, "name": "Acme Corp"},
{"id": 2, "name": "Globex Inc"}
]
},
"status": "success"
}
}
```
Configuration:
```json
"resourcePath": "response.data.customers"
```
- |-
## TROUBLESHOOTING INDICATORS
If you encounter these issues, review the resourcePath setting:
- Export completes successfully but processes 0 records
- "Cannot read property 'forEach' of undefined" errors
- "Expected array but got object/string/number" errors
- Records appear flattened or with unexpected structure
- |-
## BEST PRACTICES
- Always verify the path with sample data before deployment
- Use the simplest path that reaches the target array
- Document the expected JSON structure alongside the configuration
- For APIs with changing response structures, implement validation checks
examples: ["data.orders", "response.results", "items", "data.customers.records"]
xlsx:
type: object
description: |-
Parsing settings for Microsoft Excel (.xlsx) workbooks. Configure when type is "xlsx".
Calculated cell values are extracted rather than formulas; legacy .xls files are not
supported — use the modern Open XML format.
Required when type is xlsx.
x-celigo-ai-guidance:
- |-
## WHEN TO USE
Configure this object when the `type` field is set to "xlsx". This configuration is required for properly parsing:
- Modern Excel files (.xlsx) using the Open XML format
- Excel workbooks with multiple sheets
- Files exported from Microsoft Excel or compatible applications
- Spreadsheet data with formatting, formulas, or multiple worksheets
- |-
Configuration settings for parsing Microsoft Excel (XLSX) files.
This object defines how the system interprets and extracts data from Excel workbooks,
handling their unique structures and formatting.
- |-
## EXCEL PARSING CHARACTERISTICS
- **Multiple Worksheets**: Can access data from specific sheets within workbooks
- **Cell Formatting**: Handles various data types (text, numbers, dates, etc.)
- **Formula Resolution**: Retrieves calculated values rather than formulas
- **Data Extraction**: Converts tabular Excel data to structured records
- |-
## IMPLEMENTATION STRATEGY FOR AI AGENTS
1. **File Analysis**:
- Determine if the source file is an actual .xlsx format (not .xls, .csv, etc.)
- Identify which worksheet contains the target data
- Check for header rows, merged cells, or other special formatting
- Note any preprocessing required (hidden rows, filtered data, etc.)
2. **Common Excel File Patterns**:
### Standard Data Table
- Data organized in clear rows and columns
- First row contains headers
- No merged cells or complex formatting
- Most straightforward to process
### Report-Style Workbook
- Contains titles, headers, and possibly footers
- May have merged cells for headings
- Could have multiple tables on a single sheet
- May require specific sheet selection or row skipping
### Multi-Sheet Workbook
- Data distributed across multiple worksheets
- May require multiple export configurations
- Often needs sheet name specification (via pre-processing)
- Common in financial or complex business reports
- |-
## LIMITATIONS AND CONSIDERATIONS
- **Hidden Data**: Hidden rows/columns are still processed unless filtered
- **Formatting Loss**: Visual formatting and styles are ignored
- **Formula Handling**: Only calculated values are extracted, not formulas
- **Non-Tabular Data**: Pivot tables and non-tabular layouts may cause issues
- **Large Files**: Very large Excel files may require additional memory
- |-
## ERROR PREVENTION
- **Format Compatibility**: Ensure the file is modern .xlsx format, not legacy .xls
- **Data Structure**: Verify data is in a consistent tabular format
- **Special Characters**: Watch for special characters in header rows
- **Empty Sheets**: Check that target worksheets contain actual data
- |-
## OPTIMIZATION OPPORTUNITIES
- For complex workbooks, consider pre-processing to simplify structure
- For large files, extract only necessary worksheets/ranges before processing
- When possible, use files with consistent tabular layouts
- Consider converting Excel data to CSV format for simpler processing
properties:
hasHeaderRow:
type: boolean
description: |-
When true (the default), the first row is read as field names rather than data;
blank header cells are auto-named and duplicate names are made unique with suffixes.
When false, every row is treated as data and fields receive generic positional names
(Column1, Column2, ...).
x-celigo-ai-guidance:
- |-
## BEHAVIOR
- **When true** (default): First row is treated as field names, not data
- **When false**: All rows including the first are treated as data records
- Impacts field mapping, validation, and record counting
- |-
## IMPLEMENTATION IMPACT
### With Header Row (true)
- Field names from the header row can be referenced in mappings
- Record count excludes the header row
- First row of data is the second physical row in the spreadsheet
- Column names are derived from the first row text values
- Blank header cells may be auto-named (Column1, Column2, etc.)
### Without Header Row (false)
- Fields are referenced by position/Excel column letters (A, B, C, etc.)
- Record count includes all rows in the sheet
- First row of data is the first physical row in the spreadsheet
- Requires external schema or position-based mapping
- All fields are given generic names (Column1, Column2, etc.)
- Indicates whether the Excel file contains a header row with field names as the first row.
- |-
## DETERMINATION STRATEGY FOR AI AGENTS
To determine if a header row exists and should be configured:
1. **Visual Inspection**:
- Open the Excel file and examine the first row
- Look for descriptive labels rather than actual data values
- Check for formatting differences between the first row and others
- Header rows often use bold formatting or different background colors
2. **Content Analysis**:
- Headers typically contain text while data rows may contain mixed types
- Headers often use naming conventions (camelCase, Title Case, etc.)
- Headers don't follow the pattern/format of subsequent data rows
- Headers rarely contain numeric-only values (unless they're codes)
3. **Source Context**:
- Business reports almost always include headers
- Data exports from systems typically include column names
- Machine-generated data might skip headers
- Scientific or technical data sometimes omits headers
- |-
## USAGE GUIDANCE FOR AI AGENTS
### Recommend `hasHeaderRow: true` when:
1. **Standard Business Data**:
- Most business Excel files include headers
- Reports and exports from business systems
- Files intended for human readability
- When column names provide important context
2. **Integration Requirements**:
- When field names are needed for mapping
- When data needs to be self-describing
- When header names match target system fields
- For maintaining field identity across systems
### Recommend `hasHeaderRow: false` when:
1. **Special Data Types**:
- Scientific or sensor data without labels
- Machine-generated output files
- Legacy system exports with position-based fields
- When all rows contain actual data values
2. **Technical Scenarios**:
- When the first row contains required data
- When column positions are used for mapping
- When headers are inconsistent or misleading
- For maximum data extraction with minimal configuration
- |-
## IMPLEMENTATION NOTES
- This setting affects all worksheets in multi-sheet processing
- Excel column names with spaces or special characters may be normalized
- Duplicate header names will be made unique with suffixes
- Empty header cells will get automatically generated names
- Maximum recommended header length: 64 characters
- Consider pre-processing files without headers to add them for clarity
examples: [true, false]
xml:
type: object
description: |-
Parsing settings for XML documents. Configure when type is "xml". resourcePath is an
XPath expression selecting the elements treated as records; namespaces are handled
automatically.
Required when type is xml.
x-celigo-ai-guidance:
- |-
## WHEN TO USE
Configure this object when the `type` field is set to "xml". This configuration is required for properly parsing:
- Standard XML files (.xml)
- SOAP API responses and web service outputs
- Industry-specific XML formats (EDI, NIEM, UBL, etc.)
- Document-oriented data with hierarchical structure
- |-
Configuration settings for parsing XML (Extensible Markup Language) files.
This object defines how the system navigates and extracts hierarchical data from XML documents,
enabling processing of structured markup data.
- |-
## XML PARSING CHARACTERISTICS
- **Hierarchical Structure**: Processes nested elements and attributes
- **Schema Independence**: Works with or without formal XML schemas
- **Node Selection**: Uses XPath to precisely target record elements
- **Namespace Support**: Handles XML namespaces in complex documents
- |-
## IMPLEMENTATION STRATEGY FOR AI AGENTS
1. **Document Analysis**:
- Examine the XML structure to identify repeating elements (records)
- Determine the hierarchical level where target records exist
- Identify any namespaces that must be addressed
- Note attributes vs. element content patterns
2. **Common XML Data Patterns**:
### Simple Element List
```xml
<Records>
<Record id="1">
<Name>Product 1</Name>
<Price>10.99</Price>
</Record>
<Record id="2">
<Name>Product 2</Name>
<Price>20.99</Price>
</Record>
</Records>
```
- Records are identical element types with similar structure
- Direct children of a container element
- XPath: `/Records/Record`
### Namespaced XML
```xml
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Body>
<ns:GetCustomersResponse xmlns:ns="http://example.com/api">
<ns:Customer id="1">
<ns:Name>Acme Corp</ns:Name>
</ns:Customer>
<ns:Customer id="2">
<ns:Name>Globex Inc</ns:Name>
</ns:Customer>
</ns:GetCustomersResponse>
</soap:Body>
</soap:Envelope>
```
- Elements use XML namespaces
- Records are nested within service response structures
- XPath: `//ns:Customer` or `/soap:Envelope/soap:Body/ns:GetCustomersResponse/ns:Customer`
### Heterogeneous Records
```xml
<Feed>
<Entry type="product">
<ProductId>123</ProductId>
<Name>Widget</Name>
</Entry>
<Entry type="category">
<CategoryId>A5</CategoryId>
<Label>Supplies</Label>
</Entry>
</Feed>
```
- Same element type may have different internal structures
- Usually identified by an attribute or child element type
- May require multiple export configurations
- XPath: `/Feed/Entry[@type="product"]`
- |-
## XPATH QUERY FORMULATION
XPath is a powerful language for selecting nodes in XML documents. When formulating a resourcePath:
- **Absolute Paths** (starting with `/`): Select from the document root
- **Relative Paths** (no leading `/`): Select from the current context
- **Any-Level Selection** (`//`): Select matching nodes regardless of location
- **Predicates** (`[]`): Filter elements based on attributes or content
- **Attribute Selection** (`@`): Select attribute values instead of elements
- |-
## ERROR PREVENTION
- **Invalid XPath**: Test the resourcePath against sample data before deployment
- **Namespace Issues**: Ensure proper namespace handling in complex documents
- **Empty Results**: Verify that the XPath selects the intended nodes and not an empty set
- **Encoding Problems**: Use the correct encoding setting for international content
- |-
## OPTIMIZATION OPPORTUNITIES
- For large XML files, use more specific XPaths to reduce processing overhead
- For complex structures, consider preprocessing to simplify before parsing
- For SOAP responses, extract just the response body before processing
- For repeating integration, document the exact XPath with examples
properties:
resourcePath:
type: string
description: |-
XPath expression selecting the elements treated as records — each matching element
becomes one record, with its child elements as fields. Required for XML parsing;
there is no default. Use an absolute path (/Root/Order) when the structure is fixed,
//Element to match at any depth, or predicates (//Element[@type="product"]) to
filter; XPath is case-sensitive, and a non-matching path yields zero records.
x-celigo-ai-guidance:
- |-
Specifies the XPath expression used to locate record elements within the XML document.
This critical field determines which XML nodes are treated as individual records for processing.
- |-
## BEHAVIOR
- **Purpose**: Identifies which elements in the XML represent individual records
- **Format**: Uses XPath syntax to select nodes from the document structure
- **Requirement**: MANDATORY for XML processing - no default value exists
- **Result**: Each XML element matching the XPath is processed as one record
- |-
## XPATH SYNTAX GUIDANCE
### Core XPath Patterns
1. **Direct Child Selection** (`/Root/Element`):
```xml
<Root>
<Element>Record 1</Element>
<Element>Record 2</Element>
</Root>
```
- XPath: `/Root/Element`
- Selects elements that are direct children following exact path
- Most precise, requires exact hierarchy knowledge
- Recommended when structure is consistent and well-known
2. **Any-Level Selection** (`//Element`):
```xml
<Root>
<Section>
<Element>Record 1</Element>
</Section>
<Container>
<Element>Record 2</Element>
</Container>
</Root>
```
- XPath: `//Element`
- Selects all matching elements regardless of location
- More flexible, works across varying structures
- Use when element hierarchy may vary or is unknown
3. **Filtered Selection** (`//Element[@attr="value"]`):
```xml
<Root>
<Element type="product">Record 1</Element>
<Element type="category">Not a record</Element>
<Element type="product">Record 2</Element>
</Root>
```
- XPath: `//Element[@type="product"]`
- Selects only elements matching both name and attribute criteria
- Precise targeting when elements have identifying attributes
- Useful for heterogeneous XML with type indicators
### Advanced Selection Techniques
1. **Position-Based** (`/Root/Element[1]`):
- Selects first element only
- Use when only certain occurrences should be processed
2. **Content-Based** (`//Element[contains(text(),"Value")]`):
- Selects elements containing specific text
- Useful for filtering based on content
3. **Parent-Relative** (`//Parent[Child="Value"]/Element`):
- Selects elements with specific sibling or parent conditions
- Powerful for complex structural conditions
- |-
## NAMESPACE HANDLING
When working with namespaced XML:
```xml
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:ns="http://example.com/api">
<soap:Body>
<ns:Response>
<ns:Customer>Record 1</ns:Customer>
<ns:Customer>Record 2</ns:Customer>
</ns:Response>
</soap:Body>
</soap:Envelope>
```
The system automatically handles namespaces, but for clarity and precision:
1. **Namespace-Aware Path**:
- XPath: `/soap:Envelope/soap:Body/ns:Response/ns:Customer`
- Include namespace prefixes as they appear in the document
2. **Namespace-Agnostic Path**:
- XPath: `//Customer` or `//*[local-name()="Customer"]`
- Use when you want to ignore namespaces entirely
- |-
## DETERMINATION STRATEGY FOR AI AGENTS
1. **Identify Record Elements**:
- Look for repeating elements that represent individual "rows" of data
- These elements typically have the same name and similar structure
- They often contain multiple child elements representing "fields"
2. **Analyze Element Hierarchy**:
- Note the path from root to record elements
- Determine if records appear at consistent locations or vary
- Check if they need to be filtered by attributes or position
3. **Test Path Specificity**:
- More specific paths reduce processing overhead but are less flexible
- More general paths (with `//`) are robust to structure changes but less efficient
- Balance specificity with flexibility based on source stability
- |-
## COMMON XPATH PATTERNS BY SOURCE
| Source Type | Common XPath Pattern | Example |
|-------------|----------------------|---------|
| SOAP APIs | `/Envelope/Body/*/Response/*` | `/soap:Envelope/soap:Body/ns:GetOrdersResponse/ns:Order` |
| REST XML | `/Response/Results/*` | `/ApiResponse/Results/Customer` |
| Feeds | `/Feed/Entry` or `/Feed/Item` | `/rss/channel/item` |
| Documents | `//Section/Item` | `//Chapter/Paragraph` |
| EDI/Business | `/Document/Transaction/Line` | `/Invoice/LineItems/Item` |
- |-
## TROUBLESHOOTING INDICATORS
If you encounter these issues, review the resourcePath:
- Export completes successfully but processes 0 records
- Records contain unexpected or partial data
- Only first level of data is extracted (missing nested content)
- Namespace-related "element not found" errors
- |-
## IMPLEMENTATION NOTES
- XPath is case-sensitive; element and attribute names must match exactly
- Each matching element becomes a separate record for processing
- Child elements become fields in the processed record
- Attributes can be included in field data if needed
- Namespaces are handled automatically but may require explicit prefixes
- Testing with an XPath tool on sample data is highly recommended
examples: ["/Root/Orders/Order", "//Order", "//Invoice", "/Customers/Customer", "//Products/Product"]
includeParentData:
$ref: './record-extraction.yml#/IncludeParentData'
fileDefinition:
type: object
description: |-
Parsing via a predefined file definition resource, for formats the standard parsers
cannot handle — fixed-width files, EDI documents (X12, EDIFACT), and multi-record-type
or proprietary formats. Configure when type is "filedefinition".
Required when type is filedefinition.
x-celigo-ai-guidance:
- |-
## WHEN TO USE
Configure this object when the `type` field is set to "filedefinition". This approach is required for properly handling:
- Legacy or proprietary file formats with complex structures
- Fixed-width text files where field positions are defined by character positions
- Electronic Data Interchange (EDI) documents (X12, EDIFACT, etc.)
- Multi-record type files where different lines have different formats
- Files requiring complex preprocessing or custom parsing logic
- |-
Configuration settings for parsing files using a predefined file definition.
This object enables processing of complex, non-standard,
or proprietary file formats that require specialized parsing logic beyond what
the standard parsers (CSV,
JSON, XML, etc.) can handle.
- |-
## FILE DEFINITION CHARACTERISTICS
- **Custom Parsing Rules**: Applies predefined parsing logic to complex file formats
- **Reusable Configurations**: References externally defined parsing rules that can be reused
- **Complex Format Support**: Handles formats that standard parsers cannot process
- **Specialized Processing**: Often used for industry-specific or legacy formats
- |-
## IMPLEMENTATION STRATEGY FOR AI AGENTS
1. **Format Analysis**:
- Determine if the file format is standard (CSV, JSON, XML) or requires custom parsing
- Check if the format follows industry standards like EDI, SWIFT, or fixed-width
- Assess if there are multiple record types within the same file
- Identify if specialized logic is needed to interpret the file structure
2. **File Definition Selection**:
- Verify that a suitable file definition has already been created in the system
- Check if existing file definitions match the format requirements
- Confirm the file definition ID from system administrators if needed
- Ensure the file definition is compatible with the export's needs
- |-
## USE CASE SCENARIOS
### Fixed-Width Files
Files where each field has a specific starting position and length:
```
CUST00001JOHN DOE 123 MAIN ST
CUST00002JANE SMITH 456 OAK AVE
```
- Fields are positioned by character count rather than delimiters
- Requires precise position and length definitions
- Common in legacy mainframe and banking systems
### EDI Documents
Electronic Data Interchange formats for business transactions:
```
ISA*00* *00* *ZZ*SENDER *ZZ*RECEIVER *...
GS*PO*SENDER*RECEIVER*20210101*1200*1*X*004010
ST*850*0001
BEG*00*SA*123456**20210101
...
```
- Highly structured with segment identifiers and element separators
- Contains multiple record types with different structures
- Requires complex parsing rules and validation
### Multi-Record Files
Files containing different record types identified by indicators:
```
H|SHIPMENT|20210115|PRIORITY
D|ITEM001|5|WIDGET|RED
D|ITEM002|10|GADGET|BLUE
T|2|15|COMPLETE
```
- Each line starts with a record type indicator
- Different record types have different field structures
- Requires conditional processing based on record type
- |-
## ERROR PREVENTION
- **Definition Mismatch**: Ensure the file definition matches the actual file format
- **Missing Definition**: Verify the file definition exists before referencing it
- **Access Issues**: Confirm the integration has permission to use the file definition
- **Version Compatibility**: Check if file definition version matches current file format
- |-
## OPTIMIZATION OPPORTUNITIES
- Document which file definition is used and why it's appropriate for the file format
- Consider creating purpose-specific file definitions for complex formats
- Test file definitions with sample files before deploying in production
- Maintain documentation of the file structure alongside the file definition reference
properties:
_fileDefinitionId:
type: string
format: objectId
x-celigo-refModel: filedefinitions
description: |-
File definition resource containing the parsing rules for this format. Must
reference an existing, accessible file definition — never guess or fabricate the ID.
Definitions are reusable across exports, so changing one affects every export that
uses it.
x-celigo-ai-guidance:
- |-
The unique identifier of the file definition to use for parsing the file.
This ID references a preconfigured file definition resource that contains the
detailed parsing instructions for a specific file format.
- |-
## FIELD BEHAVIOR
- **Purpose**: References an existing file definition resource in the system
- **Format**: MongoDB ObjectId (24-character hexadecimal string)
- **Requirement**: MANDATORY when type="filedefinition"
- **Validation**: Must reference a valid, accessible file definition
- |-
## UNDERSTANDING FILE DEFINITIONS
A file definition is a separate resource that defines:
1. **Record Structure**:
- Field names, positions, and data types
- Record identifiers and format specifications
- Parsing rules and field extraction logic
2. **Processing Rules**:
- How to identify different record types
- How to handle headers, footers, and details
- Data validation and transformation rules
3. **Format-Specific Settings**:
- For fixed-width: Character positions and field lengths
- For EDI: Segment identifiers and element separators
- For proprietary formats: Custom parsing instructions
- |-
## OBTAINING THE CORRECT ID
To identify the appropriate file definition ID:
1. **System Administration**:
- Check with system administrators for a list of available file definitions
- Request the specific ID for the file format you need to process
- Verify the file definition's compatibility with your file format
2. **File Definition Catalog**:
- If available, consult the file definition catalog in the system
- Search for definitions matching your file format requirements
- Note the ObjectId of the appropriate definition
3. **Custom Definition Creation**:
- If no suitable definition exists, request creation of a new one
- Provide sample files and format specifications
- Obtain the new file definition's ID after creation
- |-
## IMPLEMENTATION GUIDANCE FOR AI AGENTS
### Recommendation Framework
When implementing a file definition-based export:
1. **Verify Definition Existence**:
- Confirm the file definition exists before configuration
- Do not guess or generate random IDs
- Request specific ID from system administrators
2. **Documentation Requirements**:
- Document which file definition is being used and why
- Note any specific requirements or limitations of the definition
- Record the mapping between file fields and integration needs
3. **Testing Approach**:
- Recommend testing with sample files before production use
- Verify all required fields are correctly extracted
- Validate the parsing results meet integration requirements
### Common File Definition Categories
| Category | Description | Example Formats |
|----------|-------------|----------------|
| Fixed-Width | Fields defined by character positions | Banking transactions, government reports |
| EDI | Electronic Data Interchange standards | X12, EDIFACT, TRADACOMS |
| Hierarchical | Complex parent-child structures | Specialized industry formats |
| Multi-Record | Different record types in one file | Inventory systems, financial exports |
| Proprietary | Custom or legacy system formats | Mainframe exports, specialized software |
- |-
## TECHNICAL CONSIDERATIONS
- File definitions are reusable across multiple exports
- Changes to a file definition affect all exports using it
- File definitions may have version dependencies
- Some file definitions may require specific pre-processing settings
- Performance impact varies based on definition complexity
- |-
## TROUBLESHOOTING INDICATORS
If you encounter these issues, verify the file definition ID:
- "File definition not found" errors
- Unexpected field mapping or missing fields
- Data type conversion errors
- Parsing failures with specific record types
Always document the exact file definition ID with its purpose to facilitate troubleshooting and maintenance.
examples: ["60a2c4e6f321d800129a1a3c", "5f8d43a1b9e5a80011a35f2c", "61b7d2e8c912e500118c4d9f"]
allowPartialSuccess:
type: boolean
description: |-
When true, a file whose records partially fail parsing against the file definition
still emits the records that did parse, instead of failing the entire file. Applies to
file-definition (fixed-width, EDI) parsing only.
examples: [true, false]
filter:
description: |-
Selects which files are processed: files matching the filter criteria are exported,
and non-matching files are skipped entirely before processing begins. Filterable
fields come from each file's metadata — most providers expose filename, filesize,
and lastmodified (e.g. ["endswith", ["extract", "filename"], ".csv"]).
allOf:
- $ref: '../../../common/schemas/filter.yml#/Filter'
x-celigo-ai-guidance:
- |-
## AVAILABLE FILTER FIELDS
The specific fields available for file filtering are the contained in the `fileMeta` property.
### Common Filter Fields
These are the most commonly available fields across most file providers:
1. **filename**: The name of the file (with extension)
- Example filter: Match files with specific extensions or naming patterns
- Usage: `["endswith", ["extract", "filename"], ".csv"]`
2. **filesize**: The size of the file in bytes
- Example filter: Skip files that are too large or too small
- Usage: `["lessthan", ["number", ["extract", "filesize"]], 1000000]`
3. **lastmodified**: The last modification timestamp of the file
- Example filter: Process only files created/modified within a specific date range
- Usage: `["greaterthan", ["extract", "lastmodified"], "2023-01-01T00:00:00Z"]`
backupPath:
type: string
description: |-
Path where backup copies of source files are stored before processing.
x-celigo-ai-guidance:
- |-
The file system path where backup files will be stored before processing.
This path specifies a directory location where the system will create backup
copies of files before they are processed by the export flow.
- |-
## BACKUP MECHANISM OVERVIEW
The backup mechanism creates a copy of source files in the specified location before processing begins. This provides:
- **Data Safety**: Preserves original files in case of processing errors
- **Audit Trail**: Maintains historical record of exported data
- **Recovery Option**: Enables reprocessing from original files if needed
- **Compliance Support**: Helps meet data retention requirements
- |-
## PATH CONFIGURATION GUIDELINES
The path format must follow these conventions:
- **Absolute Paths**: Must start with "/" (Unix/Linux) or include drive letter (Windows)
- **Relative Paths**: Interpreted relative to the application's working directory
- **Network Paths**: Can use UNC format (\\server\share\path) or mounted network drives
- **Access Requirements**: The path must be writable by the service account running the integration
- |-
## IMPLEMENTATION STRATEGY FOR AI AGENTS
When configuring the backup path, consider these factors:
1. **Storage Capacity Planning**:
- Estimate average file sizes and volumes
- Calculate required storage based on retention period
- Implement monitoring for storage utilization
- Plan for storage growth based on business projections
2. **Path Selection Criteria**:
- Choose locations with sufficient disk space
- Ensure appropriate read/write permissions
- Select paths with reliable access (avoid temporary or volatile storage)
- Consider network latency for remote locations
3. **Backup Naming Convention**:
- Default: Original filename with timestamp suffix
- Custom: Can be controlled through integration settings
- Avoid paths that may contain special characters that need escaping
- Consider filename length limitations of target filesystem
4. **Security Considerations**:
- Restrict access to backup location to authorized personnel only
- Avoid public-facing directories
- Consider encryption for sensitive data backups
- Implement appropriate file permissions
- |-
## BACKUP STRATEGY RECOMMENDATIONS
| Data Sensitivity | Recommended Approach | Path Considerations |
|------------------|----------------------|---------------------|
| Low | Local directory backup | Fast access, limited protection |
| Medium | Network share with permissions | Balanced access/protection |
| High | Secure storage with encryption | Highest protection, potential performance impact |
| Regulated | Compliant storage with audit trail | Must meet specific regulatory requirements |
- |-
## INTEGRATION PATTERNS
### Temporary Processing Pattern
For short-term processing needs:
```
/tmp/exports/backups
```
- Files stored temporarily during processing
- Limited retention period
- Optimized for processing speed
- May be automatically cleaned up
### Long-term Archival Pattern
For regulatory or business retention requirements:
```
/archive/exports/2023/Q4
```
- Organized by time period
- Structured for easy retrieval
- May include additional metadata
- Designed for long-term storage
### Cloud Storage Pattern
For scalable, managed storage:
```
/mnt/cloud/exports/client123
```
- Mounted cloud storage location
- Potentially unlimited capacity
- May include built-in versioning
- Often includes automatic replication
- |-
## ERROR HANDLING GUIDANCE
When configuring backup paths, anticipate these common issues:
- **Permission Denied**: Ensure service account has write access
- **Path Not Found**: Verify directory exists or create it programmatically
- **Disk Full**: Monitor storage capacity and implement alerts
- **Path Too Long**: Be aware of filesystem path length limitations
- |-
## TECHNICAL CONSIDERATIONS
- Backup operations may impact performance for large files
- Network paths may introduce latency and availability concerns
- Some filesystems have case sensitivity differences (important for path matching)
- Path separators vary by platform (/ vs \)
- Special characters in paths may require escaping in certain contexts
- Consider implementing automatic cleanup policies for backups
- |-
## SYSTEM ADMINISTRATION NOTES
- Backup paths should be included in system backup procedures
- Monitor space utilization on backup volumes
- Implement appropriate retention policies
- Document backup path locations in system configuration
- Consider periodic validation of backup file integrity
examples: ["/var/backups/exports", "C:\\integrations\\backups", "/mnt/nas/backups/clients/acme", "./backups", "~/integrations/backups"]
directory:
type: object
required:
- pathMode
x-celigo-canon:
decision: verified-exact
reason: >-
The validator rejects a directory object saved without a valid pathMode
(missing_required_field); an unrecognized pathMode value is treated as missing.
method: live-probe
verified: '2026-08-28'
description: |-
Structured source location for cloud file-provider sources (Google Drive shared drives,
Box, Dropbox), replacing the flat path used by classic FTP/S3 sources. Select the location
by folder ID or by a path relative to a configured storage root via pathMode. For these
cloud providers, directoryId mode addresses the folder by id alone — the flat
relative-path field is not consulted at runtime.
x-celigo-ai-guidance:
- >-
Save-time validation accepts any string for id without checking it against the
provider — set it only from a folder ID the user supplied or the UI folder browser
stored; never fabricate one. To address a folder by path instead, set pathMode to
relativePath and put the path in the adaptor's flat path field.
properties:
pathMode:
type: string
enum: ["relativePath", "directoryId"]
x-enumDescriptions:
relativePath: Address the location by a path relative to the storage root identified by storageRootId.
directoryId: Address the location directly by the provider's folder ID set in id.
description: How the directory is addressed — by provider folder ID, or by a path relative to a configured storage root.
examples: ["relativePath", "directoryId"]
id:
type: string
description: |-
Provider-native folder ID of the source directory (e.g. a Google Drive folder ID); set
when pathMode is directoryId. This is the file provider's own ID, not a Celigo resource ID.
examples: ["1KppvTTr4jKhJqwPAhGWN6SRwk5m45DL2"]
name:
type: string
description: |-
Display name of the folder identified by id, kept for readability in the UI; the
folder browser fills it when the location is picked visually.
examples: ["Invoices - Inbound"]
storageRootId:
type: string
description: |-
Provider-native ID of the storage root (e.g. a Google Drive shared-drive ID) the relative
path resolves against; set when pathMode is relativePath. The provider's own ID, not a Celigo ID.
examples: ["1qU1hvFarBFMb4-twqKm1rR5o_6j9vs9g"]
storageRootName:
type: string
description: Display name of the storage root identified by storageRootId, retained for reference in the UI.
examples: ["Shared drive - Finance"]
backupDirectory:
type: object
required:
- pathMode
description: |-
Structured destination for backup copies of source files on cloud file providers — the
file-provider counterpart to backupPath. Addressed the same way as directory.
properties:
pathMode:
type: string
enum: ["relativePath", "directoryId"]
x-enumDescriptions:
relativePath: Address the backup location by a path relative to the storage root identified by storageRootId.
directoryId: Address the backup location directly by the provider's folder ID set in id.
description: How the backup location is addressed — by provider folder ID, or by a path relative to a configured storage root.
examples: ["relativePath", "directoryId"]
id:
type: string
description: Provider-native folder ID of the backup directory; set when pathMode is directoryId.
examples: ["1KppvTTr4jKhJqwPAhGWN6SRwk5m45DL2"]
name:
type: string
description: |-
Display name of the backup folder identified by id, kept for readability in the UI;
the folder browser fills it when the location is picked visually.
examples: ["Processed archive"]
storageRootId:
type: string
description: Provider-native ID of the storage root the backup relative path resolves against; set when pathMode is relativePath.
examples: ["1qU1hvFarBFMb4-twqKm1rR5o_6j9vs9g"]
references/schemas/filesystem.yml
FileSystem:
type: object
description: |-
Defines which files to read from a local or mounted folder on the host running the
on-premise agent. Required when the _connectionId field references a file-system (on-premise)
connection; must not be included for other connection types.
x-celigo-ai-guidance:
- Configuration for FileSystem exports
required:
- directoryPath
properties:
directoryPath:
type: string
x-celigo-agent:
- handlebars
description: |-
Folder on the on-premise agent's host to read files from; the agent's OS account must
have read permission on it. Accepts a local OS path or a UNC network share, and supports
handlebars templates for dynamic folders.
x-celigo-ai-guidance:
- Directory path to retrieve files from (required)
examples: ["C:\\Celigo\\Inbound", "D:\\exports\\orders", "\\\\NAS\\Accounting\\AP"]
references/schemas/ftp.yml
FTP:
type: object
description: |-
Defines which files to retrieve from an FTP, FTPS, or SFTP server. Required when the
_connectionId field references an FTP/SFTP connection; must not be included for other
connection types. directoryPath selects the folder, fileNameStartsWith/fileNameEndsWith
filter files by name, and backupDirectoryPath controls where files are moved after
retrieval.
x-celigo-ai-guidance:
- Configuration object for FTP/SFTP connection settings in export integrations.
- |-
The FTP export object has the following requirements:
- Required fields: directoryPath
- Optional fields: fileNameStartsWith, fileNameEndsWith, backupDirectoryPath, _tpConnectorId
- |-
## PURPOSE
This configuration specifies:
- Which directory to retrieve files from
- How to filter files by name patterns
- Where to move files after retrieval (optional)
- Any trading partner-specific connection settings
required:
- directoryPath
properties:
_tpConnectorId:
type: string
format: objectId
x-celigo-refModel: tradingpartnerconnectors
description: |-
Trading Partner Connector that supplies partner-specific B2B settings for this export.
When set, the export inherits the connector's pre-configured settings; omit to use only
the FTP connection details.
x-celigo-ai-guidance:
- References a Trading Partner Connector for standardized B2B integrations.
- |-
## FIELD BEHAVIOR
This field links to pre-configured trading partner settings:
- OPTIONAL: If omitted, uses only the FTP connection details
- References a Celigo Trading Partner Connector by _id
- When specified, inherits partner-specific configurations
examples: ["60a2c4e6f321d800129a1a3c", "5f8e7d2c1b54e6003a9c7e12"]
directoryPath:
type: string
x-celigo-agent:
- handlebars
x-celigo-ui-override: >-
Required by the FTP export form (fieldDefinitions/resources/exports/ftp.js: ftp.directoryPath
is required:true). Encoded to mirror the form so builders produce connectable configurations.
description: |-
Directory on the server to retrieve files from, either absolute or relative to the login
directory; the FTP user must have read permission on it. Use forward slashes regardless
of server OS — paths are case-sensitive on UNIX/Linux servers. Supports handlebars
templates, e.g. archive/{{date 'YYYY-MM-DD'}}.
examples: ["users/dave", "incoming/orders", "/var/ftp/public", "EDI/850"]
fileNameStartsWith:
type: string
x-celigo-agent:
- handlebars
description: |-
Only retrieves files whose names start with this value (case-sensitive on most servers);
accepts static text or handlebars templates. When combined with fileNameEndsWith, files
must match both.
x-celigo-ai-guidance:
- Optional prefix filter for filenames.
- |-
- Filters files based on starting characters
- Case-sensitive on most FTP servers
- Can use static text or handlebars templates
- Examples:
- `"ORDER_"` - matches ORDER_123.csv but not order_123.csv
- `"INV_{{date 'YYYYMMDD'}}"` - matches current date's invoices
examples: ["ORDER_", "INV_", "EXPORT_", "PO_{{date 'YYYYMMDD'}}"]
fileNameEndsWith:
type: string
x-celigo-agent:
- handlebars
description: |-
Only retrieves files whose names end with this value, commonly a file extension
(case-sensitive on most servers). When combined with fileNameStartsWith, files must
match both.
x-celigo-ai-guidance:
- Optional suffix filter for filenames.
- |-
- Commonly used to filter by file extension
- Case-sensitive on most FTP servers
- Examples:
- `".csv"` - retrieves only CSV files
- `"_FINAL.xml"` - retrieves only XML files with _FINAL suffix
- `"_READY"` - retrieves files with status indicator
examples: [".csv", ".xml", ".json", "_READY", "_FINAL.txt"]
backupDirectoryPath:
type: string
description: |-
Directory on the same server where files are moved after successful export, giving you
an independent backup; if omitted, files are simply deleted from the source directory
after successful export (Celigo also keeps its own copy of processed files for a set
period). Supports static paths or handlebars templates.
x-celigo-ai-guidance:
- Optional directory where files are moved before deletion.
- |-
IMPORTANT:
Celigo automatically deletes files from the source directory after successful export.
The backup directory is for users who want to maintain their own independent
backup of exported files.
Celigo also maintains its own backup of processed files for a set period of time.
- |-
IMPORTANT:
Celigo automatically deletes files from the source directory after successful export.
The backup directory is for users who want to maintain their own independent
backup of exported files.
Celigo also maintains its own backup of processed files for a set period of time.
examples: ["processed", "archive", "backup/{{date 'YYYY-MM'}}", "/var/ftp/archive"]
x-celigo-agent:
- handlebars
references/schemas/group-by.yml
GroupBy:
type: array
description: |
Specifies which fields to use for grouping records in the export results. When configured, records with
the same values in these fields will be grouped together and treated as a single record by downstream
steps in your flow.
For example:
- Group sales orders by customer ID to process all orders for each customer together
- Group journal entries by accounting period to consolidate related transactions
- Group inventory items by location to process inventory by warehouse
When grouping is used, the export's page size determines the maximum number of groups per page, not individual
records. Note that effective grouping typically requires that records with the same group field values appear
together in the export data.
x-celigo-ai-guidance:
- |-
A grouped record's wire shape is a BARE ARRAY of the grouped rows
(``[{row}, {row}]``) — there is no ``rows`` key on the record
itself. ``rows`` is the NAME Handlebars and editor contexts bind
array-records to (templates reference ``{{rows.0.<field>}}`` /
iterate ``{{#each rows}}``); hook code sees the array as the
record element and must not assume object shape.
- |-
Transform 2.0 cannot reshape a grouped record at the array root —
``mode: "modify"`` passes the array through untouched and
``mode: "create"`` yields an empty object. Reshape grouped data in
a preSavePage hook (or downstream postResponseMap) instead.
items:
type: string
examples: [["customerId", "orderId"], ["accountId", "locationId"], ["journalId"], ["department", "class"]]references/schemas/http.yml
Http:
type: object
description: Configuration for HTTP exports. Required when the export's `_connectionId` references a connection of type `http`.
x-celigo-ai-guidance:
- |-
IMPORTANT:
When the _connectionId field points to a connection where the type is http,
this object MUST be populated for the export to function properly.
This is a required configuration for all HTTP based exports,
as determined by the connection associated with the export.
properties:
type:
type: string
enum:
- file
- blob
- csvstream
x-enumDescriptions:
file: Treats the HTTP response as raw file content (PDFs, images, binary data) rather than structured data records.
blob: Transfers the raw response body downstream as a blob without parsing it into records.
csvstream: Streams a large CSV response, parsing it row-by-row instead of loading the whole body into memory.
description: |-
Set to `file` only when the endpoint returns raw file content (PDFs, images, binary data) to be saved as a file rather than parsed as records; the `file` object must also be configured. Leave undefined for standard data exports that return structured records (JSON, XML, GraphQL, SOAP responses) — the response is then parsed into records for downstream steps. When set to `file` or `blob`, the export appears as a "Transfer" step in Flow Builder instead of a standard "Export" step.
x-celigo-ai-guidance:
- |-
## IMPORTANT: This field should be LEFT UNDEFINED for the vast majority of HTTP exports.
This is an OPTIONAL field that should only be set in rare, specific cases. For standard REST API exports
(Shopify, Salesforce, NetSuite, custom REST APIs, etc.), this field MUST be left undefined.
- |-
## WHEN TO LEAVE THIS FIELD UNDEFINED (MOST COMMON CASE):
Leave this field undefined for ALL standard data exports, including:
- REST API exports that return JSON records
- APIs that return XML records or structured data
- Any export that retrieves business records, entities, or data objects
- Standard CRUD operations that return record collections
- GraphQL queries that return structured data
- SOAP APIs that return structured responses
Examples of exports that should have this field undefined:
- "Export all Shopify Customers" → undefined (returns JSON customer records)
- "Retrieve orders from custom REST API" → undefined (returns JSON order records)
- |-
## WHEN TO SET THIS FIELD TO 'file' (RARE USE CASE):
Set this field to 'file' ONLY when the HTTP endpoint is specifically designed to download files:
- The endpoint returns raw binary file content (PDFs, images, ZIP files, etc.)
- The endpoint is a file download service (e.g., downloading invoices, reports, attachments)
- The response body contains file data that needs to be saved as a file, not parsed as records
- You need to download and process files from a remote server
Examples of when to set type: "file":
- "Download PDF invoices from the API" → type: "file"
- "Retrieve image files from a file server" → type: "file"
- "Download CSV files from an FTP server via HTTP" → type: "file"
- |-
## IMPLEMENTATION DETAILS
When this field is set to 'file':
- The 'file' object property MUST also be configured
- The export appears as a "Transfer" step in the Flow Builder UI
- The system applies file-specific processing to the HTTP response
- Downstream steps receive file content rather than record data
When this field is undefined (default for most exports):
- The export appears as a standard "Export" step in the Flow Builder UI
- The system parses the HTTP response as structured data (JSON, XML, etc.)
- Downstream steps receive record data that can be mapped and transformed
- |-
## DECISION FLOWCHART
1. Does the API endpoint return business records/entities (customers, orders, products, etc.)?
→ YES: Leave this field undefined
2. Does the API endpoint return structured data (JSON objects, XML records)?
→ YES: Leave this field undefined
3. Does the API endpoint return raw file content (PDFs, images, binary data)?
→ YES: Set this field to "file" (and configure the 'file' property)
Remember: When in doubt, leave this field undefined. Most HTTP exports are standard data exports.
examples:
- file
- blob
formType:
# Server-enforced enum: values outside this set are rejected with a 422
# ("not a valid enum value for path `http.formType`").
type: string
enum:
- http
- rest
- graph_ql
- assistant
- assistant_graphql
x-enumDescriptions:
http: Standard HTTP request form — the default authoring mode.
rest: Legacy REST-adaptor authoring mode, present on exports migrated from the REST adaptor.
graph_ql: GraphQL authoring mode; the query is composed in GraphQL-specific fields.
assistant: Connector-assistant authoring mode driven by the `assistant` metadata.
assistant_graphql: Connector-assistant authoring mode for GraphQL-based assistants.
description: |-
Authoring mode of the HTTP request form. Controls which editor the UI presents
(standard HTTP, legacy REST, GraphQL, or connector-assistant driven) and how the
request fields are interpreted. When omitted, the standard HTTP form is used.
examples: ["http", "graph_ql"]
method:
type: string
description: HTTP method used to request data from the target API. Consult the target API's documentation to determine the appropriate method.
enum:
- GET
- POST
- PUT
- PATCH
- DELETE
x-enumDescriptions:
GET: Retrieves data from the target API without modifying resources.
POST: Sends a request body to the target API when body criteria are needed, commonly used for RPC or SOAP/XML-based data retrieval
PUT: Sends a full replacement payload, available for APIs that support it for data retrieval.
PATCH: Sends a partial update payload, less common for exports but available for specialized APIs.
DELETE: Requests resource deletion, rarely used for exports but available for specialized use cases.
examples:
- GET
- POST
- PUT
followRedirects:
type: boolean
default: true
description: |-
When explicitly false, 3xx responses are not followed — the redirect
response itself (status code, `Location` header, body) becomes the
record. Omitted or true follows allowed redirects (the default
behavior).
examples: [false]
maxRedirects:
type: integer
minimum: 1
maximum: 10
description: |-
Caps how many consecutive 3xx redirects are followed. Only applies
when `followRedirects` is not false; to not follow at all, set
`followRedirects: false` rather than `maxRedirects: 0`. Decimal
values are rejected on save.
examples: [3]
relativeURI:
type: string
x-celigo-agent:
- handlebars
x-celigo-handlebars-context: pre-mapped
description: |-
Resource path appended to the connection's `baseURI` to form the complete request URL. Path segments, query parameters, or the entire value can be built with handlebars expressions for endpoints determined at runtime. For lookup exports (`isLookup: true`) with mappings configured, handlebars always render against the original pre-mapped input record, so fields removed or renamed by mappings remain available for URI construction.
x-celigo-ai-guidance:
- The resource path portion of the API endpoint used for this export.
- |-
Examples:
- Simple resource paths: "/products", "/orders", "/customers"
- With query parameters: "/orders?status=pending", "/products?category=electronics&limit=100"
- With path parameters: "/customers/{{record.customerId}}/orders", "/accounts/{{record.accountId}}/transactions"
- With dynamic query values: "/orders?since={{lastExportDateTime}}"
- Fully dynamic path: "{{record.dynamicPath}}"
- |-
Path parameters, query parameters,
or the entire URI can be dynamically generated using handlebars syntax.
This is particularly useful for parameterized API calls or when the endpoint
needs to be determined at runtime based on data or context.
- |-
## LOOKUP EXPORT BEHAVIOR WITH MAPPINGS
**CRITICAL**: For lookup exports (isLookup: true) that have mappings configured, the handlebars template evaluation for relativeURI always uses the **original input record** before any mapping transformations are applied.
This design ensures that:
- Mappings can transform the record structure for the request body without affecting URI construction
- Essential fields like record IDs remain accessible for building dynamic endpoints
- The request body can be optimized for the target API while preserving URI parameters
**Example Scenario:**
```
Input record: {"customerId": "12345", "name": "John Doe", "email": "john@example.com"}
Mappings: Transform to {"customer_name": "John Doe", "contact_email": "john@example.com"}
relativeURI: "/customers/{{record.customerId}}/details"
Result: "/customers/12345/details" (uses original customerId, not mapped version)
```
This prevents situations where mapping transformations would remove or rename fields needed for endpoint construction, ensuring reliable API calls regardless of how the request body is structured.
- 'The entire relativeURI can be defined using handlebars expressions to create dynamic paths:'
- |-
This value is combined with the baseURI defined in the associated connection to
form the complete API endpoint URL.
examples:
- /api/v1/contacts
- /customers/{{record.customerId}}
- /orders?since={{lastExportDateTime}}
- "{{record.dynamicEndpoint}}"
- /users/{{record.userId}}/transactions?from={{record.startDate}}
headers:
type: array
description: |-
Headers specific to this export, merged with (and able to override) headers defined on the connection. Define common headers such as authentication on the connection instead. Values support handlebars expressions; for lookup exports with mappings, values render against the pre-mapped input record.
x-celigo-ai-guidance:
- |-
Export-specific HTTP headers to include with API requests.
Note that common headers like authentication are typically defined on the
connection record rather than here.
- |-
Examples of export-specific headers:
- Accept: To request specific content format for this export only
- X-Custom-Filter: Export-specific filtering parameters
- |-
Header values can be defined using handlebars expressions if you need to
reference any dynamic data or configurations.
- |-
For lookup exports (isLookup: true) with mappings configured,
header value templates render against the **pre-mapped** record (the original
input record from the upstream flow step) — mappings do not rewrite header
evaluation.
- |-
Use this field only for headers that are specific to this particular export operation.
Headers defined here will be merged with (and can override) headers from the connection.
items:
type: object
properties:
name:
type: string
description: Name of the HTTP header to send; a header defined here overrides a same-named header from the connection.
examples:
- Accept
- X-Custom-Filter
value:
type: string
description: Value sent for the header. Supports handlebars expressions; for lookup exports with mappings, the value renders against the pre-mapped input record.
x-celigo-agent:
- handlebars
x-celigo-handlebars-context: pre-mapped
examples:
- application/json
- "{{record.customValue}}"
- "{{lastExportDateTime}}"
requestMediaType:
type: string
description: |-
Overrides the connection-level request media type for this export. Set only when this endpoint requires a different format than the connection default.
x-celigo-ai-guidance:
- |-
Override request media type.
Use this field to handle the use case where the HTTP request requires a
different media type than what is configured on the connection.
- |-
Most APIs use a consistent media type across all endpoints, which should be configured at the connection resource. Use this field only when:
- This specific endpoint requires a different format than other endpoints in the API
- You need to override the connection-level setting for this particular export only
enum:
- json
- xml
- urlencoded
- form-data
- plaintext
x-enumDescriptions:
json: "Sends the request body as JSON (Content-Type: application/json)."
xml: "Sends the request body as XML (Content-Type: application/xml)."
urlencoded: "Sends the request body as URL-encoded form data (Content-Type: application/x-www-form-urlencoded)."
form-data: Sends the request body as multipart form data, typically used for file uploads.
plaintext: "Sends the request body as plain text (Content-Type: text/plain)."
examples:
- json
- xml
- urlencoded
body:
type: string
x-celigo-agent:
- handlebars
- graphql
description: |-
Request body sent with POST, PUT, or PATCH requests, typically carrying query or filter criteria for APIs (such as GraphQL or SOAP) that expect them in the body. The content must match the format set by `requestMediaType` and supports handlebars expressions for dynamic values.
x-celigo-ai-guidance:
- |-
The HTTP request body to send with POST, PUT, or PATCH requests. This field is typically used to:
1. Send query parameters to APIs that require them in the request body (e.g., GraphQL or SOAP APIs)
2. Provide filtering criteria for data exports
- |-
The body content must match the format specified in the requestMediaType field (JSON,
XML, etc.).
- |-
You can use handlebars expressions to create dynamic content:
```
{
"query": "SELECT Id, Name FROM Account WHERE LastModifiedDate > {{lastExportDateTime}}",
"parameters": {
"customerId": "{{record.customerId}}",
"limit": 100
}
}
```
- |-
For XML or SOAP requests:
```
<request>
<filter>
<updatedSince>{{lastExportDateTime}}</updatedSince>
<type>{{record.type}}</type>
</filter>
</request>
```
examples:
- '{"query": "SELECT * FROM Contacts WHERE LastModifiedDate > {{lastExportDateTime}}"}'
- <request><filters><filter><updatedSince>{{lastExportDateTime}}</updatedSince></filter></filters></request>
- '{"ids": [{{record.id}}], "includeDetails": true}'
- '{"lastSyncDate": "{{lastExportDateTime}}", "limit": 100}'
successMediaType:
type: string
description: |-
Media type used to parse successful response bodies. Set only when the response format differs from the request format.
x-celigo-ai-guidance:
- |-
Specifies the media type (content type) expected in successful responses for this specific export. This field should only be used when:
1. The response format differs from the request format
- |-
Most APIs return responses in the same format as the request,
so this field is often unnecessary.
- |-
Common values:
- "json": For JSON responses (typically with Content-Type: application/json)
- "xml": For XML responses (typically with Content-Type: application/xml)
- "csv": For CSV data (typically with Content-Type: text/csv)
- "plaintext": For plain text responses
enum:
- json
- xml
- csv
- plaintext
x-enumDescriptions:
json: "Parses the successful response body as JSON (Content-Type: application/json)."
xml: "Parses the successful response body as XML (Content-Type: application/xml)."
csv: "Parses the successful response body as CSV (Content-Type: text/csv)."
plaintext: Treats the successful response body as plain text.
examples:
- json
- xml
errorMediaType:
type: string
description: |-
Media type used to parse error response bodies. Set only when error responses use a different format than the request.
x-celigo-ai-guidance:
- |-
Most APIs return responses in the same format as the request,
so this field is often unnecessary.
- |-
Specifies the media type (content type) expected in error responses for this specific export. This field should only be used when:
1. Error response format differs from the request format
- |-
Common values:
- "json": For JSON error responses (most common in modern APIs)
- "xml": For XML error responses (common in SOAP and older REST APIs)
- "plaintext": For plain text error messages
enum:
- json
- xml
- plaintext
x-enumDescriptions:
json: Parses the error response body as JSON, the most common error format for modern APIs
xml: Parses the error response body as XML.
plaintext: Treats the error response body as plain text.
examples:
- json
- xml
_asyncHelperId:
type: string
format: objectId
x-celigo-refModel: asynchelpers
description: |-
AsyncHelper resource that handles polling for long-running operations on APIs that process requests asynchronously (HTTP 202 responses, job tickets, feed or document IDs). Set when the export must submit a request, poll for status, and retrieve results once the external process completes — for example Amazon SP-API feeds or large report generators.
x-celigo-ai-guidance:
- |-
Reference to an AsyncHelper resource that polls an asynchronous source API on
this export's behalf.
- |-
Set this ONLY when the source API is genuinely asynchronous — it acknowledges
the request (HTTP 202,
a job ticket, a feed/document id) and processes it in the background,
so results must be polled for before they can be retrieved (e.g.
Amazon SP-API feeds, large report generators).
Most exports are synchronous and need NO async helper;
adding one to a synchronous source just adds polling overhead plus a status and
result export to maintain.
When in doubt, leave it unset.
- |-
The referenced helper bundles the polling config plus a required status export
(polled to check progress) and a result export (fetches the final payload).
An export configured with an async helper cannot carry its own transform,
output filter,
or preSavePage hook — put that processing in the result export instead.
examples:
- 60a2c4e6f321d800129a1a3c
- 5f8d43a1b9e5a80011a35f2c
x-celigo-agent:
- async_helper
once:
type: object
description: Callback configuration for once exports, used to mark records as exported in the source system after successful processing.
properties:
relativeURI:
type: string
x-celigo-agent:
- handlebars
x-celigo-handlebars-context: pre-mapped
description: |-
Relative path (starting with `/`) called on the source system to mark each record as exported after successful processing. Supports handlebars variables and renders against the pre-mapped record — mappings do not apply to the callback URI.
x-celigo-ai-guidance:
- |-
- Must be a relative path starting with "/"
- Can include Handlebars variables: "/orders/{{record.Id}}/exported"
- Common patterns: dedicated status endpoint or record-specific updates
- Renders against the **pre-mapped** record (original extracted record); mappings do not apply to the callback URI.
examples:
- /api/v1/mark-exported
- /export/flag-records
- /orders/update-status
- /v2/transactions/{{record.id}}/exported
method:
type: string
description: HTTP method used for the mark-as-exported callback request.
enum:
- GET
- PUT
- POST
- PATCH
- DELETE
x-enumDescriptions:
GET: Uses an HTTP GET request to mark records as exported.
PUT: Uses an HTTP PUT request to mark records as exported.
POST: Uses an HTTP POST request to mark records as exported.
PATCH: Uses an HTTP PATCH request to mark records as exported.
DELETE: Uses an HTTP DELETE request to mark records as exported.
examples:
- POST
- PUT
body:
type: string
x-celigo-agent:
- handlebars
description: Request body sent with the mark-as-exported callback. Supports handlebars expressions for dynamic values.
x-celigo-ai-guidance:
- |-
The HTTP request body used when calling back to mark records as exported.
Can include Handlebars expressions for dynamic values.
examples:
- '{"status": "exported", "exportId": "{{_exportId}}"}'
- '{"records": [{"id": "{{record.id}}", "exported": true}]}'
paging:
type: object
description: |-
Controls how the export requests subsequent pages when the API returns multi-page responses. The `method` field determines which companion fields are required and how each next page is requested. For the page, skip, and token methods, reference the matching pagination variable (`{{export.http.paging.page}}`, `{{export.http.paging.skip}}`, or `{{export.http.paging.token}}`) in the relative URI or request body — misconfigured pagination is a common cause of incomplete data retrieval.
x-celigo-ai-guidance:
- Configuration object for navigating through multi-page API responses.
- |-
## OVERVIEW FOR AI AGENTS
This object is critical for retrieving large datasets that cannot be returned in a single API response.
The pagination implementation determines how the system will retrieve subsequent pages of data after
the first request, enabling complete data collection regardless of volume.
- |-
## KEY DECISION POINTS
1. **Identify the API's pagination mechanism** (check API documentation)
2. **Select the corresponding method** value (most important field)
3. **Configure the required fields** based on your selected method
4. **Add pagination variables** to your request configuration
5. **Consider last page detection** options if needed
- |-
## FIELD DEPENDENCIES BY PAGINATION METHOD
1. **page**: Page number-based pagination (e.g., ?page=2)
- Required: Set `method` to "page"
- Optional: `page` - Set if first page index is not 0 (e.g., set to 1 for APIs that start at page 1)
- Optional: `maxPagePath` - Path to find total pages in response
- Optional: `maxCountPath` - Path to find total records in response
- Optional: `relativeURI` - Only if subsequent page URLs differ from first page
- Optional: `resourcePath` - Only if records location changes in follow-up responses
2. **skip**: Offset/limit pagination (e.g., ?offset=100&limit=50)
- Required: Set `method` to "skip"
- Optional: `skip` - Set if first skip index is not 0
- Optional: `maxPagePath` - Path to find total pages in response
- Optional: `maxCountPath` - Path to find total records in response
- Optional: `relativeURI` - Only if subsequent page URLs differ from first page
- Optional: `resourcePath` - Only if records location changes in follow-up responses
3. **token**: Token-based pagination (e.g., ?page_token=abc123)
- Required: Set `method` to "token"
- Required: `path` - Location of the token in the response
- Required: `pathLocation` - Whether token is in "body" or "header"
- Optional: `token` - Set to provide initial token (rare)
- Optional: `pathAfterFirstRequest` - Only if token location changes after first page
- Optional: `relativeURI` - Only if subsequent page URLs differ from first page
- Optional: `resourcePath` - Only if records location changes in follow-up responses
4. **linkheader**: Link header pagination (uses HTTP Link header with rel values)
- Required: Set `method` to "linkheader"
- Optional: `linkHeaderRelation` - Set if relation is not the default "next"
- Optional: `resourcePath` - Only if records location changes in follow-up responses
5. **nextpageurl**: Complete next URL in response
- Required: Set `method` to "nextpageurl"
- Required: `path` - Location of the next URL in the response
- Optional: `resourcePath` - Only if records location changes in follow-up responses
6. **relativeuri**: Custom relative URI pagination
- Required: Set `method` to "relativeuri"
- Required: `relativeURI` - Configure using handlebars with previous_page context
- Optional: `resourcePath` - Only if records location changes in follow-up responses
7. **body**: Custom request body pagination
- Required: Set `method` to "body"
- Required: `body` - Configure using handlebars with previous_page context
- Optional: `resourcePath` - Only if records location changes in follow-up responses
- |-
## PAGINATION VARIABLES
Based on your selected method, you MUST add one of these variables to your request configuration:
- For page-based: Add `{{export.http.paging.page}}` to the URI or body
- For offset-based: Add `{{export.http.paging.skip}}` to the URI or body
- For token-based: Add `{{export.http.paging.token}}` to the URI or body
- |-
## LAST PAGE DETECTION OPTIONS
These fields can be used with any pagination method to detect the last page:
- `lastPageStatusCode` - Detect last page by HTTP status code
- `lastPagePath` - JSON path to check for last page indicator
- `lastPageValues` - Values at lastPagePath that indicate last page
- |-
## COMMON IMPLEMENTATION PATTERNS
Most APIs require only 2-3 fields to be configured. The most common patterns are:
```json
// Page-based pagination (starting at page 1)
{
"method": "page",
"page": 1
}
// Token-based pagination
{
"method": "token",
"path": "meta.nextToken",
"pathLocation": "body"
}
// Link header pagination (simplest to configure)
{
"method": "linkheader"
}
```
IMPORTANT: Incorrect pagination configuration is one of the most common causes of incomplete data retrieval. Take time to properly identify and configure the correct pagination method for your API.
properties:
method:
type: string
description: |-
Pagination strategy used to request each subsequent page; match it to the mechanism documented by the target API. Determines which companion fields are required: `token` needs `path` and `pathLocation`, `url` needs `path`, `relativeuri` needs `relativeURI`, `body` needs `body`, while `linkheader` typically needs no extra configuration. Using the wrong method results in errors or incomplete data retrieval.
x-celigo-ai-guidance:
- Defines the pagination strategy that will be used to retrieve all data pages.
- |-
## PAGINATION METHODS AND THEIR REQUIREMENTS
### Page-Based Pagination (`"page"`)
```
"method": "page"
```
- **Implementation**: Uses increasing page numbers (e.g., ?page=1, ?page=2)
- **Required Setup**: Add `{{export.http.paging.page}}` to your URI or body
- **Common Fields**: page (if starting at 1 instead of 0)
- **API Examples**: Most REST APIs, Shopify, WordPress
- **When to Use**: APIs that accept a page number parameter
### Offset/Skip Pagination (`"skip"`)
```
"method": "skip"
```
- **Implementation**: Uses increasing offset values (e.g., ?offset=0, ?offset=100)
- **Required Setup**: Add `{{export.http.paging.skip}}` to your URI or body
- **Common Fields**: Usually none (system handles offset increments)
- **API Examples**: MongoDB, SQL-based APIs
- **When to Use**: APIs that use offset/limit or skip/limit parameters
### Token-Based Pagination (`"token"`)
```
"method": "token"
```
- **Implementation**: Passes tokens from previous responses to get next pages
- **Required Setup**:
1. Add `{{export.http.paging.token}}` to your URI or body
2. Set path to location of token in response
3. Set pathLocation to "body" or "header"
- **API Examples**: AWS, Google Cloud, modern REST APIs
- **When to Use**: APIs that provide continuation tokens/cursors
### Link Header Pagination (`"linkheader"`)
```
"method": "linkheader"
```
- **Implementation**: Follows URLs in HTTP Link headers automatically
- **Required Setup**: None (simplest to configure)
- **Common Fields**: Usually none (automatic)
- **API Examples**: GitHub, GitLab, any API following RFC 5988
- **When to Use**: APIs that return Link headers with rel="next"
### Next Page URL (`"nextpageurl"`)
```
"method": "nextpageurl"
```
- **Implementation**: Uses complete URLs returned in response body
- **Required Setup**: Set path to location of next URL in response
- **API Examples**: Some social media APIs, GraphQL implementations
- **When to Use**: APIs that include complete next page URLs in responses
### Custom Relative URI (`"relativeuri"`)
```
"method": "relativeuri"
```
- **Implementation**: Builds custom URIs based on previous responses
- **Required Setup**: Configure relativeURI with handlebars templates
- **When to Use**: Non-standard pagination requiring custom logic
### Custom Request Body (`"body"`)
```
"method": "body"
```
- **Implementation**: Creates custom request bodies for pagination
- **Required Setup**: Configure body with handlebars templates
- **API Examples**: GraphQL, SOAP, RPC APIs
- **When to Use**: APIs requiring POST requests with pagination in body
- |-
## SELECTION GUIDANCE
To determine the correct method:
1. Check the API documentation for pagination instructions
2. Look for examples of multi-page requests in API samples
3. Test with a small request to observe pagination mechanics
4. Choose the method matching the API's expected behavior
IMPORTANT: Using the wrong pagination method will result in either errors or incomplete data retrieval.
- |-
## IMPORTANCE FOR AI AGENTS
This is the MOST CRITICAL field in pagination configuration. It determines:
- Which other fields are required vs. optional
- How subsequent pages will be requested
- Which pagination variables must be used in requests
- How the system detects the last page
enum:
- linkheader
- page
- skip
- token
- url
- relativeuri
- body
x-enumDescriptions:
linkheader: Follows URLs in the HTTP Link header with rel="next" for pagination.
page: Uses incrementing page numbers for pagination (e.g., ?page=1, ?page=2).
skip: Uses incrementing offset values for pagination (e.g., ?offset=0, ?offset=100).
token: Passes a continuation token from the previous response to fetch the next page.
url: Follows a complete next-page URL returned in the response body.
relativeuri: Builds a custom relative URI for each subsequent page using handlebars templates.
body: Sends a custom request body for each subsequent page using handlebars templates.
examples:
- page
- token
- linkheader
- relativeuri
page:
type: integer
description: |-
Starting page number for `method: page`. Set to 1 for APIs whose first page is not zero-indexed; when omitted, paging starts at 0. The value is incremented automatically for each subsequent page request.
x-celigo-ai-guidance:
- Specifies the starting page number for page-based pagination.
- |-
## FIELD BEHAVIOR
- RELEVANT ONLY for method="page"
- OPTIONAL: Defaults to 0 if not provided
- COMMON VALUES: 1 (most APIs), 0 (zero-indexed APIs)
- |-
## IMPLEMENTATION GUIDANCE
This field should be set when the API's first page is not zero-indexed. Most APIs use 1 as
their first page number, in which case you should set:
```json
{
"method": "page",
"page": 1
}
```
The system will automatically increment this value for each subsequent page request.
- |-
## EXAMPLES
- Shopify uses page=1 for first page
- Some GraphQL APIs use page=0 for first page
examples:
- 1
skip:
type: integer
description: |-
Starting offset for `method: skip`. Rarely needed — most APIs start at 0, and the value is incremented by the page size automatically for each subsequent request. Set only when the API requires a non-zero starting offset.
x-celigo-ai-guidance:
- Specifies the starting offset value for offset/skip-based pagination.
- |-
## FIELD BEHAVIOR
- RELEVANT ONLY for method="skip"
- OPTIONAL: Defaults to 0 if not provided
- COMMON VALUES: 0 (vast majority of APIs)
- |-
## IMPLEMENTATION GUIDANCE
This field rarely needs to be set since most APIs use 0 as the starting offset.
The system will automatically increment this value by the pageSize for each subsequent request.
Example calculation for page transitions:
- First page: offset=0 (or your configured value)
- Second page: offset=pageSize
- Third page: offset=pageSize*2
- |-
## WHEN TO USE
Only set this if the API requires a non-zero starting offset value, which is very uncommon.
examples:
- 0
- 100
token:
type: string
description: |-
Initial token for `method: token`. Leave empty for normal pagination — the first request is sent without a token, and subsequent tokens are extracted from each response via `path`. Set only to resume from a known token or for APIs that require a token on the first request.
x-celigo-ai-guidance:
- Specifies an initial token value for token-based pagination.
- |-
## FIELD BEHAVIOR
- RELEVANT ONLY for method="token"
- OPTIONAL: Leave empty for normal pagination from the beginning
- ADVANCED USE ONLY: Most implementations should NOT set this
- |-
## IMPLEMENTATION GUIDANCE
Token-based pagination normally works by:
1. Making the first request with no token
2. Extracting a token from the response (using the path field)
3. Using that token for the next request
This field should ONLY be set in rare scenarios:
- Resuming a previous pagination sequence from a known token
- APIs that require a token value even for the first request
- Testing specific pagination scenarios
- |-
## EXAMPLE SCENARIOS
```json
// To resume pagination from a specific point:
{
"method": "token",
"path": "meta.nextToken",
"pathLocation": "body",
"token": "eyJwYWdlIjozfQ=="
}
// For APIs requiring an initial token:
{
"method": "token",
"path": "pagination.nextToken",
"pathLocation": "body",
"token": "start"
}
```
examples:
- next_token_123
path:
type: string
description: |-
Location of the pagination value in each response — the continuation token for `method: token`, or the complete next-page URL for `method: url`. When `pathLocation` is `body`, use a dot-notation JSON path (e.g. `meta.nextToken`); when `header`, use the exact case-sensitive header name. Not used by other pagination methods.
x-celigo-ai-guidance:
- |-
## IMPLEMENTATION GUIDANCE
### For token-based pagination (method="token"):
1. When pathLocation="body":
- Set to a JSON path that points to the token in the response body
- Uses dot notation to navigate JSON objects
Example response:
```json
{
"data": [...],
"meta": {
"nextToken": "abc123"
}
}
```
Correct path: "meta.nextToken"
2. When pathLocation="header":
- Set to the exact name of the HTTP header containing the token
- Case-sensitive, must match the header exactly
Example header:
```
X-Pagination-Token: abc123
```
Correct path: "X-Pagination-Token"
### For next page URL pagination (method="nextpageurl"):
- Set to a JSON path that points to the complete URL in the response
Example response:
```json
{
"data": [...],
"pagination": {
"next_url": "https://api.example.com/data?page=2"
}
}
```
Correct path: "pagination.next_url"
- Specifies the location of pagination information in API responses.
- |-
## COMMON ERROR PATTERNS
1. Missing dot notation: "meta.nextToken" not "meta/nextToken"
2. Incorrect case: "Meta.NextToken" when API returns "meta.nextToken"
3. Missing array indices when needed: "items[0].next" not "items.next"
examples:
- meta.nextPage
- pagination.next
- X-Next-Token
- pagination.nextUrl
pathLocation:
type: string
description: |-
Where the export looks for the pagination token referenced by `path`. Required for `method: token`; not used by other pagination methods.
x-celigo-canon:
decision: stricter-than-server
reason: >-
The server defaults an omitted pathLocation to body, and most stored
token-paging exports omit it — but new configs must declare where the
token is read from; a header-delivered token fails silently under the
implicit body default. Stored-document carriage is not grounds to relax.
method: full-population
verified: '2026-07-04'
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR
- REQUIRED for method="token"
- NOT USED for other pagination methods
- LIMITED to two possible values: "body" or "header"
- |-
## IMPLEMENTATION GUIDANCE
When using token-based pagination, you must:
1. Set method="token"
2. Set path to locate the token
3. Set pathLocation to indicate where the token is found
### When to use "body":
Set to "body" when the token is contained in the JSON response body.
This is the most common scenario for modern APIs.
Example configuration:
```json
{
"method": "token",
"path": "metadata.nextToken",
"pathLocation": "body"
}
```
### When to use "header":
Set to "header" when the token is returned as an HTTP header.
Example configuration:
```json
{
"method": "token",
"path": "X-Next-Page-Token",
"pathLocation": "header"
}
```
- Specifies where to find the pagination token in the API response.
- |-
## DEPENDENCY CHAIN
This field participates in a critical dependency chain:
1. Set method="token"
2. Set pathLocation="body" or "header"
3. Set path to token location based on pathLocation value
4. Add {{export.http.paging.token}} to URI or body parameters
All four elements must be properly configured for token pagination to work.
enum:
- body
- header
x-enumDescriptions:
body: The pagination token is located in the JSON response body.
header: The pagination token is located in an HTTP response header.
examples:
- body
- header
pathAfterFirstRequest:
type: string
description: |-
Alternative token location used for responses after the first page, in the same format as `path`. Set only when the API moves the token to a different location in subsequent responses — setting it unnecessarily can cause pagination to fail.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR
- RELEVANT ONLY for method="token"
- OPTIONAL: Only needed when token location changes after first page
- Uses same format as the path field (JSON path or header name)
- |-
## IMPLEMENTATION GUIDANCE
This field should only be set when the API changes its response structure between
the first page and subsequent pages. Most APIs maintain consistent structure, but
some APIs may:
1. Use different response formats for first vs. subsequent pages
2. Move the token to a different location after the initial response
3. Change the field name for the token in follow-up responses
Example scenario where this is needed:
```json
// First page response:
{
"data": [...],
"meta": {
"initialNextToken": "abc123"
}
}
// Subsequent page responses:
{
"data": [...],
"pagination": {
"nextToken": "def456"
}
}
```
In this case:
- path = "meta.initialNextToken" (for first page)
- pathAfterFirstRequest = "pagination.nextToken" (for subsequent pages)
- |-
## DEPENDENCY CHAIN
This field works in conjunction with the main path field:
1. First request: token is extracted using the path field
2. Subsequent requests: token is extracted using pathAfterFirstRequest
IMPORTANT: Only set this field if you've verified that the API actually changes
its response structure. Setting it unnecessarily can cause pagination to fail.
examples:
- pagination.nextToken
- meta.continueToken
- X-Next-Page-Token
relativeURI:
type: string
x-celigo-agent:
- handlebars
description: |-
Overrides the main relative URI for second and later page requests; leave empty when the main relative URI works for all pages. Build it with handlebars using the `previous_page` context — `previous_page.full_response` (the prior response body), `previous_page.last_record` (the last record of the prior page), and `previous_page.headers` (the prior response headers).
x-celigo-ai-guidance:
- |-
Override relative URI for subsequent page requests.
This field appears as "Override relative URI for subsequent page requests" in the UI.
- |-
This field only needs to be set if subsequent page requests require a different
relative URI than what is configured in the primary relative URI field.
Most APIs use the same endpoint for all pages and vary only the query parameters,
but some may require a completely different path for subsequent requests.
- |-
You can use handlebars expressions to reference data from the previous API response using the `previous_page` context object, which contains:
- `previous_page.full_response` - The entire JSON response body from the previous request
- `previous_page.last_record` - The last record from the previous page of results
- `previous_page.headers` - All HTTP headers from the previous response
- |-
Common patterns include:
- `{{previous_page.full_response.next_page}}` - Use a complete next page URL returned by the API
- `/customers?page={{previous_page.full_response.page_count}}` - Use a page number from the response
- `/orders?cursor={{previous_page.full_response.next_cursor}}` - Use a cursor/token from the response
- The exact structure of data available depends on your specific API's response format.
examples:
- "{{previous_page.full_response.next_page}}"
- /customers?page={{previous_page.full_response.page}}
- /orders?token={{previous_page.full_response.next_token}}
- /items?after={{previous_page.last_record.id}}&limit=100
x-celigo-handlebars-context: previous_page
body:
type: string
x-celigo-agent:
- handlebars
- graphql
description: |-
Overrides the main request body for second and later page requests, typically for GraphQL or SOAP APIs that paginate through the body; leave empty when the main body works for all pages. Build it with handlebars using the `previous_page` context — `previous_page.full_response`, `previous_page.last_record`, and `previous_page.headers`.
x-celigo-ai-guidance:
- |-
This field only needs to be set if subsequent page requests require a different
HTTP request body than what is configured in the primary HTTP request body
field.
Most APIs use query parameters for pagination,
but some (especially GraphQL or SOAP APIs) may require pagination parameters to
be sent in the request body.
- |-
You can use handlebars expressions to reference data from the previous API response using the `previous_page` context object, which contains:
- `previous_page.full_response` - The entire JSON response body from the previous request
- `previous_page.last_record` - The last record from the previous page of results
- `previous_page.headers` - All HTTP headers from the previous response
- The exact structure of data available depends on your specific API's response format.
- |-
Override HTTP request body for subsequent page requests.
This field appears as "Override HTTP request body for subsequent page requests" in the UI.
- |-
Common patterns include:
- Including the next cursor in a GraphQL query: `{"query": "...", "variables": {"cursor": "{{previous_page.full_response.pageInfo.endCursor}}"}}`
- Using the last record's ID: `{"after": "{{previous_page.last_record.id}}", "limit": 100}`
- Including a page number: `{"page": {{previous_page.full_response.meta.next_page}}, "pageSize": 50}`
- Leave this field empty if the main HTTP request body can be used for all page requests.
examples:
- '{"query": "query($cursor: String) { items(after: $cursor) { edges { node { id name } } pageInfo { endCursor } } }", "variables": {"cursor": "{{previous_page.full_response.data.items.pageInfo.endCursor}}"}}'
- '{"pageToken": "{{previous_page.full_response.nextPageToken}}"}'
- '{"startIndex": {{previous_page.full_response.nextStartIndex}}, "maxResults": 100}'
mergeBodyParamsToPagingBody:
type: boolean
description: |-
Only applies when `paging.method` is `body`. When true, the body
parameters produced by the export's mappings at runtime are merged
into the evaluated `paging.body` for page 2+ requests, so paging
requests keep the same mapped parameters as the first page.
Omitted/false keeps the original behavior — page 2+ requests send
only the evaluated `paging.body`.
x-celigo-handlebars-context: previous_page
linkHeaderRelation:
type: string
description: |-
Link header relation followed for `method: linkheader` when the API uses a value other than the default `next`. Case-sensitive and must exactly match the `rel` value in the Link header, without the `rel=` prefix.
x-celigo-ai-guidance:
- Specifies which relation in the Link header to use for pagination.
- |-
## FIELD BEHAVIOR
- RELEVANT ONLY for method="linkheader"
- OPTIONAL: Defaults to "next" if not provided
- Case-sensitive value matching the rel attribute in Link header
- |-
## IMPLEMENTATION GUIDANCE
Link header pagination follows the RFC 5988 standard where pagination links
are provided in HTTP headers. A typical Link header looks like:
```
Link: <https://api.example.com/items?page=2>; rel="next", <https://api.example.com/items?page=1>; rel="prev"
```
This field allows you to specify which relation type to follow for pagination:
```
"linkHeaderRelation": "next" // Default value
```
Some APIs use non-standard relation names, which is when you'd need to change this:
```
"linkHeaderRelation": "successor" // Custom relation name
```
- |-
## COMMON VALUES
- "next" (default): Standard for most RFC 5988 compliant APIs
- "successor": Alternative used by some APIs
- "forward": Alternative used by some APIs
- "nextpage": Non-standard but used by some implementations
IMPORTANT: This is case-sensitive and must exactly match the relation value in
the Link header. If the API includes the prefix "rel=" in the header, do NOT
include it here.
examples:
- next
- successor
- forward
- nextpage
resourcePath:
type: string
description: |-
Overrides the path to records for second and later page responses. Set only when follow-up pages place records at a different location than the first response; leave empty when all pages share the same structure.
x-celigo-ai-guidance:
- |-
Override path to records for subsequent page requests.
This field appears as "Override path to records for subsequent page requests" in the UI.
- |-
This field only needs to be set if subsequent page requests return a different
response structure,
and the records are located in a different place than the original request.
- |-
For example, if the first request returns records in a structure like {"data":
[...]} but subsequent page responses have records in {"results": [...]} instead,
you would set this field to "results" to correctly extract data from the follow-up pages.
examples:
- results
- data.items
- response.records
lastPageStatusCode:
type: integer
description: |-
HTTP status code that signals the last page, replacing the default behavior of treating 404 as the end of pagination. When this status is received, paging stops and the response is not treated as an error. Set only when the API uses a non-404 code (such as 204 or 400) to indicate no more pages.
x-celigo-ai-guidance:
- Specifies a custom HTTP status code that indicates the last page of results.
- |-
## FIELD BEHAVIOR
- OPTIONAL: Only needed for APIs with non-standard last page indicators
- Applies to all pagination methods
- Overrides the default 404 end-of-pagination detection
- |-
## IMPLEMENTATION GUIDANCE
By default, the system treats a 404 status code as an indicator that
pagination is complete. This field allows you to specify a different
status code if your API uses an alternative convention.
Common scenarios where this is needed:
1. APIs that return 204 (No Content) for empty result sets
```
"lastPageStatusCode": 204
```
2. APIs that return 400 (Bad Request) when requesting beyond available pages
```
"lastPageStatusCode": 400
```
3. APIs with custom error codes for pagination completion
```
"lastPageStatusCode": 499
```
- |-
## TECHNICAL DETAILS
When this status code is received, the system:
- Stops the pagination process
- Considers the data collection complete
- Does not treat the response as an error
- Does not attempt to process any response body
IMPORTANT: Only set this if your API explicitly uses a non-404 status code
to indicate the end of pagination. Setting this incorrectly could cause
premature termination of data collection or error handling issues.
examples:
- 204
- 400
- 500
lastPagePath:
type: string
description: |-
JSON path to a response-body field that signals the end of pagination, such as a "has more" flag or a cursor that empties on the last page. Must be used with `lastPageValues`, which lists the values at this path that stop paging. If the path does not exist in a response, the condition is not considered met.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR
- OPTIONAL: Only needed for APIs with field-based pagination completion signals
- Works with all pagination methods
- Used in conjunction with lastPageValues
- JSON path notation to a field in the response body
- |-
## IMPLEMENTATION GUIDANCE
This field is used when an API indicates the last page through a field
in the response body rather than using HTTP status codes. The system
checks this path in each response to determine if pagination is complete.
Common patterns include:
1. Boolean flag fields
```
"lastPagePath": "meta.isLastPage"
```
2. "Has more" indicators
```
"lastPagePath": "pagination.hasMore"
```
3. Cursor/token fields that are null/empty on the last page
```
"lastPagePath": "meta.nextCursor"
```
4. Error message fields
```
"lastPagePath": "error.message"
```
- |-
## DEPENDENCY CHAIN
This field must be used with lastPageValues, which specifies the value(s)
at this path that indicate pagination is complete. For example:
```json
"lastPagePath": "pagination.hasMore",
"lastPageValues": ["false", "0"]
```
IMPORTANT: The path is evaluated against each response using JSON path notation.
If the path doesn't exist in the response, the condition is not considered met.
examples:
- meta.isLastPage
- pagination.hasMore
- meta.nextCursor
- error.message
lastPageValues:
type: array
description: |-
Values at `lastPagePath` that stop pagination; a match on any entry ends paging. All entries are compared as exact, case-sensitive strings, even for boolean or numeric fields — use `"true"`, `"false"`, `"null"` for JSON null, or `""` for an empty string. Required when `lastPagePath` is set.
x-celigo-ai-guidance:
- Specifies which value(s) at the lastPagePath indicate the end of pagination.
- |-
## FIELD BEHAVIOR
- REQUIRED when lastPagePath is used
- Array of string values (even for boolean or numeric comparisons)
- Case-sensitive matching against the value at lastPagePath
- Multiple values create an OR condition (any match indicates last page)
- |-
## IMPLEMENTATION GUIDANCE
This field works in conjunction with lastPagePath to determine when
pagination is complete. The system looks for the field specified by
lastPagePath and compares its value against each entry in this array.
Common patterns include:
1. For boolean "isLastPage" flags (true means last page)
```json
"lastPagePath": "meta.isLastPage",
"lastPageValues": ["true"]
```
2. For "hasMore" flags (false means last page)
```json
"lastPagePath": "pagination.hasMore",
"lastPageValues": ["false", "0"]
```
3. For empty cursors (null/empty string means last page)
```json
"lastPagePath": "meta.nextCursor",
"lastPageValues": ["null", ""]
```
4. For specific error messages
```json
"lastPagePath": "error.message",
"lastPageValues": ["No more pages", "End of results"]
```
- |-
## TECHNICAL DETAILS
- All values must be specified as strings, even for boolean or numeric comparisons
- JSON null should be represented as the string "null"
- Empty string is represented as ""
- The comparison is exact and case-sensitive
IMPORTANT: This field is only considered when the lastPagePath exists in the
response. Both lastPagePath and lastPageValues must be configured correctly
for proper pagination termination.
items:
type: string
examples:
- - "true"
- - "false"
- "0"
- - "null"
- ""
- - No more pages
- End of results
maxPagePath:
type: string
description: |-
JSON path to the total page count in the response, used to stop paging once the last page is reached. Only applies to the `page` and `skip` methods. Point it at the total number of pages, not the current page number.
x-celigo-ai-guidance:
- Specifies a JSON path to a field containing the total number of pages available.
- |-
## FIELD BEHAVIOR
- OPTIONAL: Only relevant for "page" and "skip" pagination methods
- JSON path to a numeric field in the response
- Used to optimize pagination by detecting the last page early
- Ignored for other pagination methods
- |-
## IMPLEMENTATION GUIDANCE
This field enables pagination optimization when an API includes metadata
about the total number of pages. When configured, the system:
1. Extracts the total page count from each response
2. Compares the current page number against this total
3. Stops pagination when the maximum page is reached
Common API response patterns include:
```json
// Pattern 1: Metadata section with page counts
{
"data": [...],
"meta": {
"totalPages": 5,
"currentPage": 2
}
}
// Pattern 2: Pagination object
{
"results": [...],
"pagination": {
"pageCount": 5,
"page": 2
}
}
// Pattern 3: Root level pagination info
{
"items": [...],
"pages": 5,
"current": 2
}
```
- |-
## USAGE SCENARIOS
Most useful when:
- The API reliably includes total page counts
- You want to prevent unnecessary requests after the last page
- The 404/last page detection mechanisms aren't suitable
IMPORTANT: This field should point to the TOTAL number of pages,
not the current page number. The value must be numeric (integer).
examples:
- meta.totalPages
- pagination.pageCount
- response.paging.total
- pages
maxCountPath:
type: string
description: |-
JSON path to the total record count in the response, used to stop paging once all records have been retrieved. Only applies to the `page` and `skip` methods; when both are set, `maxPagePath` takes precedence. Point it at the total number of records, not the count in the current page.
x-celigo-ai-guidance:
- Specifies a JSON path to a field containing the total number of records available.
- |-
## FIELD BEHAVIOR
- OPTIONAL: Only relevant for "page" and "skip" pagination methods
- JSON path to a numeric field in the response
- Alternative to maxPagePath for record-based termination
- Used when APIs provide total record count instead of page count
- |-
## IMPLEMENTATION GUIDANCE
This field enables pagination optimization when an API includes metadata
about the total number of records rather than pages. When configured,
the system:
1. Extracts the total record count from each response
2. Tracks the total number of records processed so far
3. Stops pagination when all records have been processed
Common API response patterns include:
```json
// Pattern 1: Metadata section with record counts
{
"data": [...],
"meta": {
"totalCount": 42,
"page": 2,
"pageSize": 10
}
}
// Pattern 2: Pagination object
{
"results": [...],
"pagination": {
"total": 42,
"offset": 20,
"limit": 10
}
}
// Pattern 3: Root level count info
{
"items": [...],
"count": 42,
"page": 2
}
```
- |-
## RELATIONSHIP WITH maxPagePath
This field is an alternative to maxPagePath:
- Use maxPagePath when the API provides a total page count
- Use maxCountPath when the API provides a total record count
- If both are provided, maxPagePath takes precedence
IMPORTANT: This field should point to the TOTAL number of records,
not the number of records in the current page. The value must be
numeric (integer).
examples:
- meta.totalCount
- pagination.total
- response.totalResults
- count
if: { properties: { method: { const: body } }, required: [method] }
then: { required: [body] }
else:
# token paging requires path AND pathLocation. The server defaults an omitted
# pathLocation to body (and most stored docs omit it), but declaring where the
# token is read from is enforced for new configs — a token that actually arrives
# in a header fails silently under the implicit default.
if: { properties: { method: { const: token } }, required: [method] }
then: { required: [path, pathLocation] }
else:
if: { properties: { method: { const: relativeuri } }, required: [method] }
then: { required: [relativeURI] }
else:
if: { properties: { method: { const: url } }, required: [method] }
then: { required: [path] }
response:
type: object
description: |-
Controls how records are extracted from the API response and how success or failure is detected at the response level. When the API wraps records in an envelope object, set `resourcePath` to the path of the records array — without it, the entire response body is treated as a single record. Leave this object undefined when the API returns a bare JSON array.
x-celigo-ai-guidance:
- Configuration for parsing and interpreting HTTP responses returned by the source API.
- |-
## MOST IMPORTANT FIELD: resourcePath
`resourcePath` is the single most commonly needed field in this object. When an API
wraps its records inside a JSON envelope, you MUST set resourcePath to the dot-path
(or `$`-prefixed JSONPath expression) that points to the array of records. Without it,
the export treats the entire response as a single record.
Example API response:
```json
{
"status": "ok",
"data": {
"customers": [
{"id": 1, "name": "Alice"},
{"id": 2, "name": "Bob"}
]
}
}
```
→ Set `resourcePath` to `data.customers` so the export produces 2 records.
- |-
## WHEN TO LEAVE THIS OBJECT UNDEFINED
If the API returns a bare JSON array (e.g. `[{"id":1}, {"id":2}]`) with no
wrapper object, you do not need this object at all.
properties:
resourcePath:
type: string
x-celigo-canon:
decision: no-enum
reason: >-
Freeform user-defined JSON path into each API's own response envelope — low
observed cardinality is an artifact of common envelope names, not a vocabulary.
method: full-population
verified: '2026-07-03'
description: |-
Dot-separated path to the array of records inside the response body (e.g. `data.customers` for `{"data": {"customers": [...]}}`); without it, a wrapped response is treated as a single record. Values starting with `$` are evaluated as JSONPath instead of dot notation — including unions (`$['orders','invoices'][*]`) and filter expressions (`$.items[?(@.qty>3)]`); all other values keep the existing dot-notation behavior unchanged. Leave undefined when the API returns a bare JSON array. This extracts records from the API response — not to be confused with `oneToMany`/`pathToMany`, which unwrap arrays from input records, or `paging.resourcePath`, which applies only to subsequent page responses.
x-celigo-ai-guidance:
- |-
## CRITICAL FIELD FOR CORRECT DATA EXTRACTION
Most APIs wrap their data in an envelope object. This field tells the export
where to find the actual records within that envelope. Without this field,
the export treats the entire response body as a single record, which is
almost never the desired behavior when the response has a wrapper.
- |-
## HOW IT WORKS
Given an API response like:
```json
{
"meta": {"page": 1, "total": 42},
"results": [
{"id": "A", "value": 10},
{"id": "B", "value": 20}
]
}
```
Setting `resourcePath` to `results` causes the export to produce 2 records
(`{"id":"A","value":10}` and `{"id":"B","value":20}`).
For deeply nested responses:
```json
{
"slideshow": {
"slides": [{"title": "Slide 1"}, {"title": "Slide 2"}]
}
}
```
Set `resourcePath` to `slideshow.slides` to get each slide as a record.
- |-
## WHEN TO SET THIS FIELD
- The API response is a JSON object (not a bare array) and the records are
nested inside it → set this to the path
- The API response is a bare JSON array → leave undefined (records are
already at the top level)
- |-
## JSONPATH SYNTAX (values starting with `$`)
Values starting with `$` are evaluated as JSONPath instead of dot notation:
- `$.data.orders` — equivalent to dot notation `data.orders`
- `$['orders','invoices'][*]` — union: concatenates the elements of several
arrays into one record stream, in listed order (the trailing `[*]` is
required; without it each array arrives as a single record)
- `$.items[?(@.qty>3)]` — filter: emits only the elements matching the
expression (filters may only access the current node `@`)
- A path that resolves to nothing emits zero records for that response;
evaluation failures surface as record-level errors
Dot-notation values (no `$` prefix) keep the existing behavior unchanged.
- |-
## COMMON PATTERNS
| API response structure | resourcePath value |
|---|---|
| `{"data": [...]}` | `data` |
| `{"results": [...]}` | `results` |
| `{"items": [...]}` | `items` |
| `{"records": [...]}` | `records` |
| `{"response": {"data": [...]}}` | `response.data` |
| `{"slideshow": {"slides": [...]}}` | `slideshow.slides` |
| `[...]` (bare array) | leave undefined |
- |-
## IMPORTANT DISTINCTION
This field extracts records from the **API response**. Do NOT confuse it with:
- `oneToMany` + `pathToMany` — which unwrap child arrays from *input records*
in lookup/import steps (a completely different mechanism)
- `paging.resourcePath` — which overrides the record location for *subsequent*
page requests only (when follow-up pages use a different response structure)
examples:
- data
- results
- items
- records
- slideshow.slides
- response.data
- data.customers
- $.data.orders
includeParentData:
$ref: './record-extraction.yml#/IncludeParentData'
resourceIdPath:
type: string
description: |-
Path to the unique identifier within each record, used primarily when processing results of asynchronous import responses. When omitted, the system looks for standard `id` or `_id` fields automatically.
examples:
- id
- _id
- recordId
successPath:
type: string
description: |-
Path to a response field that signals whether the call succeeded, for APIs that return HTTP 200 even on failure. Must be used with `successValues` to define which values at this path count as success.
x-celigo-ai-guidance:
- |-
Use this when the API returns HTTP 200 for all requests but signals success or
failure through a field in the response body.
- |-
Example: If the API returns `{"status": "ok", "data": [...]}`,
set `successPath` to `status` and `successValues` to `["ok"]`.
examples:
- status
- success
- result.code
- meta.status
successValues:
type: array
items:
type: string
description: |-
Values at `successPath` that mark the response as successful; any other value is treated as an error. All comparisons are string-based — use `"true"` or `"false"` for boolean fields.
x-celigo-ai-guidance:
- Values at the `successPath` location that indicate the API call was successful.
- |-
When the value at `successPath` matches any entry in this array,
the response is treated as successful.
If the value does not match, the response is treated as an error.
examples:
- ["ok"]
- ["true", "success"]
- ["200", "201"]
errorPath:
type: string
description: |-
Path to the error message in the response body. The value at this path is included in error logs and error records when the API returns an error.
examples:
- error.message
- errorMessage
- errors[0].detail
failPath:
type: string
description: |-
Path to a response field that signals failure even when the HTTP status code is 200 — the inverse of `successPath`. Must be used with `failValues`.
x-celigo-ai-guidance:
- |-
Similar to `successPath` but inverted logic — checks for failure indicators.
Must be used together with `failValues`.
examples:
- error
- status
- result.error
failValues:
type: array
items:
type: string
description: |-
Values at `failPath` that mark the response as failed, even when the HTTP status code is 200.
x-celigo-ai-guidance:
- Values at the `failPath` location that indicate the API call failed.
- |-
When the value at `failPath` matches any entry in this array,
the response is treated as a failure even if the HTTP status code was 200.
examples:
- ["error", "failed"]
- ["false"]
allowArrayforSuccessPath:
type: boolean
description: |-
When true, treats the value at `successPath` as an array and counts the response as successful if any element matches `successValues`, rather than requiring a single scalar match. Set it for APIs that return per-record status arrays in a batch response.
twoDArray:
type: object
description: |-
Parsing options for endpoints that return tabular data as a two-dimensional array (rows of cells) rather than an array of objects, such as spreadsheet-style or report APIs.
properties:
hasHeader:
type: boolean
description: When true, the first row is treated as column headers and used to name the fields of each generated record.
doNotNormalize:
type: boolean
description: When true, the rows are passed through as raw arrays instead of being normalized into keyed records.
blobFormat:
type: string
description: |-
Controls how the binary response body is decoded for blob exports. Only relevant when `http.type` is `file` or the export type is `blob`.
enum:
- utf8
- ucs2
- utf-16le
- ascii
- binary
- base64
- hex
x-enumDescriptions:
utf8: Decodes the blob response using UTF-8 character encoding.
ucs2: Decodes the blob response using UCS-2 (two-byte Unicode) encoding.
utf-16le: Decodes the blob response using UTF-16 Little Endian encoding.
ascii: Decodes the blob response using 7-bit ASCII encoding.
binary: Treats the blob response as raw binary data without character decoding.
base64: Decodes the blob response from Base64-encoded text.
hex: Decodes the blob response from hexadecimal-encoded text.
examples:
- utf8
- base64
# No successPath->successValues / failPath->failValues pairing: the paths are
# meaningful alone (existence/error-extraction semantics) and the value lists are
# optional refinements.
_httpConnectorVersionId:
type: string
format: objectId
readOnly: true
description: Identifies the HTTP connector version used by this export. Set by the connector framework; client-supplied values are ignored (write-tested).
_httpConnectorResourceId:
type: string
format: objectId
readOnly: true
description: Identifies the HTTP connector resource used by this export. Set by the connector framework; client-supplied values are ignored (write-tested).
sendAuthForFileDownloads:
type: boolean
description: When true, includes authentication headers when downloading files.
_httpConnectorEndpointId:
type: string
format: objectId
readOnly: true
x-celigo-refModel: httpconnectorendpoints
description: Identifies the HTTP connector endpoint configuration used for this export's requests. Set by the connector framework; client-supplied values are ignored (write-tested).
if:
properties:
type:
const: file
required: [type]
not:
properties:
formType:
const: assistant
required: [formType]
then:
required: [file]
references/schemas/jdbc.yml
JDBC:
type: object
description: |-
Configuration object for JDBC (Java Database Connectivity) data integration exports.
This object is REQUIRED when the _connectionId field references a JDBC database connection
and must not be included for other connection types. It defines how data is extracted
from relational databases using SQL queries.
**Jdbc export capabilities**
- Execute custom SQL SELECT statements
- Support for joins, aggregations, and functions
- Flexible data retrieval from any accessible tables or views
- Compatible with all major database systems
**Critical:** WHAT BELONGS IN THIS OBJECT
- `query` - **ALWAYS REQUIRED** - The SQL SELECT statement
- `once` - **REQUIRED** when the export's Object Type is `"once"` (set _include_once: true)
- **DO NOT** put `delta` inside this object - delta is handled via the query
**Delta exports (type: "delta")**
For delta/incremental exports, do NOT populate a `delta` object inside `jdbc`.
Instead, use `{{lastExportDateTime}}` or `{{currentExportDateTime}}` directly in the query:
```json
{
"type": "delta",
"jdbc": {
"query": "SELECT * FROM customers WHERE updatedAt > {{lastExportDateTime}}"
}
}
```
**Once exports (type: "once")**
For once exports (mark records as processed), populate `jdbc.once.query`:
```json
{
"type": "once",
"jdbc": {
"query": "SELECT * FROM customers WHERE exported = false",
"once": {
"query": "UPDATE customers SET exported = true WHERE id = {{record.id}}"
}
}
}
```
**Standard exports (type: null or not specified)**
Just provide the query:
```json
{
"jdbc": {
"query": "SELECT * FROM customers WHERE status = 'ACTIVE'"
}
}
```
required:
- query
properties:
formType:
# Nullable: the UI persists an explicit null until a mode is picked;
# null means the 'sql' default.
type: [string, 'null']
enum:
- sql
- simple
- null
x-enumDescriptions:
sql: The query is authored as raw SQL in the `query` field.
simple: The query is built visually from `simple.table`, `simple.columns`, and `simple.filter` instead of raw SQL.
description: |-
Selects how the export's query is authored. Use `sql` for a hand-written SQL statement in
`query`; use `simple` to build the query from the `simple` object's table/columns/filter.
examples: ["sql", "simple"]
query:
type: string
x-celigo-agent:
- sql
x-celigo-ui-override: >-
Required by the JDBC export form (jdbc.query is required:true). Encoded to mirror the form
so builders produce connectable configurations.
description: |-
SQL SELECT statement executed to retrieve data, from simple table selections to joins and
aggregations. For delta exports, include {{lastExportDateTime}}/{{currentExportDateTime}}
in the WHERE clause rather than configuring a separate delta object.
x-celigo-ai-guidance:
- |-
This field contains the SQL SELECT statement that will be executed to fetch data
from the database.
The query can range from simple table selections to complex joins and aggregations.
- |-
Examples:
- Basic: `SELECT id, name, email FROM customers WHERE status = 'ACTIVE'`
- Join: `SELECT o.id, c.name, o.amount FROM orders o JOIN customers c ON o.customer_id = c.id`
- Aggregate: `SELECT category, COUNT(*) as count FROM orders GROUP BY category`
- Parameterized: `SELECT * FROM orders WHERE customer_id = {{record.customer_id}}`
- |-
## FOR DELTA EXPORTS (when top-level type is "delta")
Include `{{lastExportDateTime}}` or `{{currentExportDateTime}}` in the WHERE clause:
- `SELECT * FROM customers WHERE updatedAt > {{lastExportDateTime}}`
- `SELECT * FROM orders WHERE modified_date >= {{lastExportDateTime}}`
- |-
This field contains the SQL SELECT statement that will be executed to fetch data
from the database.
The query can range from simple table selections to complex joins and aggregations.
examples: ["SELECT * FROM customers WHERE updatedAt > {{lastExportDateTime}}", "SELECT * FROM orders WHERE exported = false", "SELECT * FROM customers WHERE region = 'Northeast'", "SELECT o.id, c.name, o.total FROM orders o JOIN customers c ON o.customer_id = c.id WHERE o.status = 'SHIPPED'", "SELECT product_category, SUM(quantity) as total_sold FROM order_items GROUP BY product_category", "SELECT * FROM customer_orders WHERE customer_id = {{record.id}}"]
once:
type: object
description: |
**CRITICAL: REQUIRED when the export's Object Type is "once".**
If Object Type is "once", you MUST set _include_once to true (or include this object).
This object has ONLY ONE property: "query" (a SQL UPDATE string).
DO NOT create any other properties like "update", "table", "set", "where", etc.
CORRECT format:
```json
{"query": "UPDATE customers SET exported=true WHERE id={{record.id}}"}
```
WRONG format (DO NOT DO THIS):
```json
{"update": {"table": "customers", "set": {...}}}
```
properties:
query:
type: string
x-celigo-agent:
- sql
description: |
**REQUIRED** - A SQL UPDATE statement string to mark records as processed.
This is a plain SQL UPDATE query string, NOT a structured object.
The query runs AFTER each record is successfully exported, setting a flag
to indicate the record has been processed.
Format: "UPDATE <table> SET <column>=<value> WHERE <id_column>={{record.<id_field>}}"
Example: "UPDATE customers SET exported=true WHERE id={{record.id}}"
The {{record.id}} placeholder is replaced with the actual record ID from each exported row.
examples: ["UPDATE orders SET exported=true WHERE id={{record.id}}", "UPDATE customers SET exported=true WHERE customer_id={{record.customer_id}}", "UPDATE inventory SET is_exported=1 WHERE sku={{record.sku}}"]
simple:
type: object
description: |-
Visual query-builder configuration used when `formType` is `simple`; the platform builds
the SELECT from these instead of raw SQL. Only relevant when `formType` is `simple`.
properties:
table:
type: string
description: Table or view to select rows from.
examples: ["dbo.Orders", "public.customers"]
columns:
type: array
items:
type: string
description: Columns to return; an empty list selects all columns.
examples: [["id", "name", "total"]]
filter:
# Stored as the platform's structured filter object, never a string.
x-celigo-agent:
- filter
description: |-
Filter applied as the WHERE clause for the generated query, stored as the
platform's structured filter-rules object.
allOf:
- $ref: '../../../common/schemas/filter.yml#/Filter'
references/schemas/mock-output.yml
MockOutput:
# Nullable: the UI clears mock output by writing null.
type: [object, 'null']
description: |
Sample data that simulates the output from an export for testing and configuration purposes.
Mock output allows you to configure and test flows without executing the actual export or
waiting for real-time data to arrive. This is particularly useful for:
- Initial flow configuration and testing
- Mapping development without requiring live data
- Generating metadata for downstream flow steps
- Creating realistic test scenarios
- Documenting expected data structures
**Structure**
The mock output must follow the integrator.io canonical format, which consists of a
`page_of_records` array containing record objects. Each record object has a `record`
property that contains the actual data fields.
```json
{
"page_of_records": [
{
"record": {
"field1": "value1",
"field2": "value2",
...
}
},
...
]
}
```
**Usage**
When executing a test run or configuring a flow, integrator.io will use this mock output
instead of executing the export to retrieve live data. This allows you to:
- Test mappings with representative data
- Configure downstream flow steps without waiting for real data
- Simulate various data scenarios
**Limitations**
- Maximum of 10 records
- Maximum size of 1 MB
- Must follow the canonical format shown above
Mock output can be populated automatically from preview data or entered manually.
properties:
page_of_records:
type: array
description: |
Array of record objects in the integrator.io canonical format.
Each item in this array represents one record that would be processed
by the flow during execution.
items:
type: object
properties:
record:
type: object
description: |
Container for the actual record data fields.
The structure of this object will vary depending on the specific
export configuration and the source system's data structure.
additionalProperties: true
example:
id: "12345"
name: "Sample Product"
price: 99.99
inStock: true
categories: ["Electronics", "Accessories"]
example:
page_of_records:
- record:
id: "12345"
name: "Sample Product"
price: 99.99
inStock: true
categories: ["Electronics", "Accessories"]
- record:
id: "67890"
name: "Another Product"
price: 49.99
inStock: false
categories: ["Home", "Kitchen"]references/schemas/mongodb.yml
MongoDB:
type: object
description: |-
Defines how documents are retrieved from MongoDB collections. Required when the
_connectionId field references a MongoDB connection; must not be included for other
connection types. Supports find queries with optional filter criteria and field projections.
x-celigo-ai-guidance:
- Configuration object for MongoDB data integration exports.
- |-
MongoDB exports currently support the following operational modes:
- Retrieves documents from specified collections
- Filters documents based on query criteria
- Selects specific fields with projections
- Provides NoSQL flexibility with JSON query syntax
required:
- collection
properties:
method:
type: string
enum: ["find"]
x-enumDescriptions:
find: Retrieves documents from a MongoDB collection using filter and projection criteria.
description: |-
MongoDB operation used to retrieve documents. Only "find" is currently supported,
equivalent to db.collection.find(filter, projection).
x-celigo-ai-guidance:
- Specifies the MongoDB operation to perform when retrieving data.
- |-
## FIELD BEHAVIOR
This field defines the query approach:
- REQUIRED for all MongoDB exports
- Currently only supports "find" operations
- Determines how other parameters are interpreted
- Corresponds to MongoDB's db.collection.find() method
- Future versions may support additional methods
- |-
## QUERY METHOD TYPES
### Find Method
```
"method": "find"
```
- **Behavior**: Retrieves documents from a collection based on filter criteria
- **MongoDB Equivalent**: db.collection.find(filter, projection)
- **Required Parameters**: collection
- **Optional Parameters**: filter, projection
- **Use Cases**: Standard document retrieval, filtered queries, field selection
- |-
## TECHNICAL CONSIDERATIONS
The method selection influences:
- What other fields must be provided
- How the query will be executed against MongoDB
- What indexing strategies should be applied
- Performance characteristics of the operation
IMPORTANT: While only "find" is currently supported, the schema is designed
for future expansion to include other MongoDB operations like "aggregate"
for more complex data transformations and aggregations.
examples: ["find"]
collection:
type: string
x-celigo-ui-override: >-
Required by the MongoDB export form (mongodb.collection is required:true). Encoded to
mirror the form so builders produce connectable configurations.
description: |-
Name of the MongoDB collection to query. Case-sensitive and must reference an existing
collection in the connected database.
x-celigo-ai-guidance:
- Specifies the MongoDB collection to query for documents.
- |-
## FIELD BEHAVIOR
This field identifies the data source:
- REQUIRED for all MongoDB exports
- Must reference a valid collection in the MongoDB database
- Case-sensitive according to MongoDB collection naming
- The primary container for documents to be retrieved
examples: ["customers", "orders", "product_catalog", "user_profiles"]
filter:
type: string
description: |-
MongoDB query document, as a JSON string, that selects which documents to export; omit
to return every document in the collection. Supports standard MongoDB query operators
and handlebars variables for dynamic values — for example
{"lastModified": {"$gt": "{{lastRun}}"}} for incremental processing.
x-celigo-ai-guidance:
- Defines query criteria for selecting documents from the collection.
- |-
## FIELD BEHAVIOR
This field narrows document selection:
- OPTIONAL: If omitted, all documents in the collection are returned
- Contains a MongoDB query document as a JSON string
- Supports all standard MongoDB query operators
- Provides precise control over which documents are retrieved
- |-
## QUERY PATTERNS
### Simple Equality Query
```
"filter": "{"status": "active"}"
```
- **Behavior**: Returns only documents where status equals "active"
- **MongoDB Equivalent**: db.collection.find({"status": "active"})
- **Matching Documents**: {"_id": 1, "status": "active", "name": "Example"}
- **Use Cases**: Status filtering, category selection, type filtering
### Comparison Operator Query
```
"filter": "{"createdDate": {"$gt": "2023-01-01T00:00:00Z"}}"
```
- **Behavior**: Returns documents created after January 1, 2023
- **MongoDB Equivalent**: db.collection.find({"createdDate": {"$gt": "2023-01-01T00:00:00Z"}})
- **Operators**: $eq, $gt, $gte, $lt, $lte, $ne, $in, $nin
- **Use Cases**: Date ranges, numeric thresholds, incremental processing
### Logical Operator Query
```
"filter": "{"$or": [{"status": "pending"}, {"status": "processing"}]}"
```
- **Behavior**: Returns documents with either pending or processing status
- **MongoDB Equivalent**: db.collection.find({"$or": [{"status": "pending"}, {"status": "processing"}]})
- **Operators**: $and, $or, $nor, $not
- **Use Cases**: Multiple conditions, alternative criteria, complex filtering
### Nested Document Query
```
"filter": "{"address.country": "USA"}"
```
- **Behavior**: Returns documents where the nested country field equals "USA"
- **MongoDB Equivalent**: db.collection.find({"address.country": "USA"})
- **Dot Notation**: Accesses nested document fields
- **Use Cases**: Nested data filtering, object property matching
### Handlebars Template Query
```
"filter": "{"customerId": "{{record.customer_id}}", "status": "{{record.status}}"}"
```
- **Behavior**: Dynamically filters based on record field values
- **MongoDB Equivalent**: db.collection.find({"customerId": "123", "status": "active"})
- **Template Variables**: Values replaced at runtime with actual record data
- **Use Cases**: Dynamic filtering, context-aware queries, relational lookups
### Incremental Processing Query
```
"filter": "{"lastModified": {"$gt": "{{lastRun}}"}}"
```
- **Behavior**: Returns only documents modified since last execution
- **MongoDB Equivalent**: db.collection.find({"lastModified": {"$gt": "2023-06-15T10:30:00Z"}})
- **System Variables**: {{lastRun}} replaced with timestamp of previous execution
- **Use Cases**: Change data capture, delta synchronization, incremental updates
examples: ['{"status": "active"}', '{"createdDate": {"$gt": "{{lastRun}}"}}', '{"$or": [{"status": "pending"}, {"status": "processing"}]}', '{"address.country": "USA", "age": {"$gte": 21}}', '{"customerId": "{{record.customer_id}}", "orderStatus": "{{record.status}}"}', '{"productId": "{{record.product.id}}", "category": "{{record.product.category}}"}']
projection:
type: string
description: |-
MongoDB projection document, as a JSON string, that limits which fields are returned;
omit to return all fields. Use 1 to include fields or 0 to exclude them — the two modes
cannot be mixed except for _id, which is included by default unless explicitly excluded.
Projection affects only the shape of returned documents, not which documents match.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR
This field optimizes data retrieval:
- OPTIONAL: If omitted, all fields are returned
- Contains a MongoDB projection document as a JSON string
- Can include fields (1) or exclude fields (0), but not both (except _id)
- Helps minimize data transfer by selecting only needed fields
- |-
## PROJECTION PATTERNS
### Field Inclusion Projection
```
"projection": "{"name": 1, "email": 1, "_id": 0}"
```
- **Behavior**: Returns only name and email fields, excludes _id
- **MongoDB Equivalent**: db.collection.find({}, {"name": 1, "email": 1, "_id": 0})
- **Result Format**: {"name": "Example", "email": "user@example.com"}
- **Use Cases**: Specific field selection, minimizing payload size
### Field Exclusion Projection
```
"projection": "{"password": 0, "internal_notes": 0}"
```
- **Behavior**: Returns all fields except password and internal_notes
- **MongoDB Equivalent**: db.collection.find({}, {"password": 0, "internal_notes": 0})
- **Result Impact**: Removes sensitive or unnecessary fields
- **Use Cases**: Security filtering, removing large fields, data protection
### Nested Field Projection
```
"projection": "{"profile.firstName": 1, "profile.lastName": 1, "orders": 1, "_id": 0}"
```
- **Behavior**: Returns only specific nested fields and the orders array
- **MongoDB Equivalent**: db.collection.find({}, {"profile.firstName": 1, "profile.lastName": 1, "orders": 1, "_id": 0})
- **Dot Notation**: Accesses specific nested document fields
- **Use Cases**: Partial nested document selection, specific array inclusion
- |-
## TECHNICAL CONSIDERATIONS
- Maximum size: 128KB
- Must be a valid JSON string representing a MongoDB projection
- Cannot mix inclusion and exclusion modes (except _id field)
- _id field is included by default unless explicitly excluded
- Projection does not affect which documents are returned, only their fields
IMPORTANT: When working with nested documents or arrays, be aware that including
a specific field path does not automatically include parent documents or arrays.
For example, including "addresses.zipcode" will only return that specific field,
not the entire addresses array or documents within it.
examples: ['{"name": 1, "email": 1, "_id": 0}', '{"password": 0, "internal_notes": 0}', '{"profile.firstName": 1, "profile.lastName": 1, "orders": 1, "_id": 0}', '{"metadata": 0, "system_tags": 0}']
pipeline:
type: string
description: |-
MongoDB aggregation pipeline, as a JSON-array string of stages, used instead of a plain
find when documents need to be transformed, joined, or grouped server-side (e.g. `$match`,
`$lookup`, `$group`). When set, it takes the place of `filter`/`projection`. Supports
handlebars variables for incremental processing.
examples: ['[{"$match": {"status": "active"}}, {"$project": {"name": 1, "email": 1}}]', '[{"$match": {"lastModified": {"$gt": "{{lastRun}}"}}}]']
readPreference:
type: string
enum:
- doNotOverride
- primary
- primaryPreferred
- secondary
- secondaryPreferred
- nearest
x-enumDescriptions:
doNotOverride: Uses the read preference configured on the connection (the default).
primary: Reads only from the replica-set primary.
primaryPreferred: Reads from the primary when available, otherwise a secondary.
secondary: Reads only from a replica-set secondary.
secondaryPreferred: Reads from a secondary when available, otherwise the primary.
nearest: Reads from the member with the lowest network latency.
default: doNotOverride
description: |-
Overrides which replica-set member this export reads from, trading consistency for load
distribution. Leave at `doNotOverride` to inherit the connection's setting.
pathToRecords:
$ref: './record-extraction.yml#/PathToRecords'
includeParentData:
$ref: './record-extraction.yml#/IncludeParentData'
references/schemas/netsuite.yml
NetSuite:
type: object
description: |-
NetSuite-specific export configuration. Required when `_connectionId` references a NetSuite
connection; omit for all other connection types. Supports saved-search, RESTlet, and
distributed (SuiteApp) exports, plus two file cabinet modes: blob exports transfer files
as-is (set the export's top-level `type` to `blob` and configure `netsuite.blob`), while
file exports parse file contents into records (leave the export's top-level `type` unset
and configure `netsuite.file`).
x-celigo-ai-guidance:
- Configuration object for NetSuite data integration exports.
- |-
This object is REQUIRED when the _connectionId field references a NetSuite
connection and must not be included for other connection types.
It defines how data is extracted from NetSuite, including saved searches,
RESTlets, and distributed/SuiteApp exports.
- |-
## NETSUITE EXPORT MODES
NetSuite exports support several operating modes:
1. **Saved Search Exports** - Uses NetSuite saved searches to retrieve data
2. **RESTlet Exports** - Uses custom RESTlet scripts for data retrieval
3. **Distributed Exports** - Uses SuiteApp for real-time or batch processing
4. **Blob Exports** - Retrieves files from the NetSuite file cabinet and transfers them WITHOUT parsing them into records (raw binary transfer)
5. **File Exports** - Retrieves files from the NetSuite file cabinet and PARSES them into records (CSV, XML, JSON, etc.)
- |-
## CRITICAL: Blob vs File Export Configuration
The export `type` field at the top level determines whether file content is parsed:
- **For Blob Exports (no parsing)**: Set the export's `type: "blob"` AND configure `netsuite.blob`
- **For File Exports (with parsing)**: Leave the export's `type` as null/undefined AND configure `netsuite.file`
Do NOT set `type: "blob"` when you want file content parsed into records. The "blob" type is specifically for raw file transfers without any parsing.
- |-
## IMPLEMENTATION REQUIREMENTS
- For saved search exports: Configure the `searches` or `type` properties
- For RESTlet exports: Configure the `restlet` property with script details
- For distributed exports: Configure the `distributed` property
- For blob exports (no parsing): Set export `type: "blob"` and configure `netsuite.blob`
- For file exports (with parsing): Leave export `type` null and configure `netsuite.file`
properties:
type:
type: string
enum:
- search
- basicSearch
- metadata
- selectoption
- restlet
- getList
- getServerTime
- distributed
- file
x-enumDescriptions:
search: Retrieves records using a NetSuite saved search.
basicSearch: Retrieves records using a basic NetSuite search query.
metadata: Retrieves metadata about a NetSuite record type.
selectoption: Retrieves the available select options for a specific field.
restlet: Retrieves data using a custom NetSuite RESTlet script.
getList: Retrieves a list of records by their internal IDs.
getServerTime: Retrieves the current NetSuite server time.
distributed: Uses the SuiteApp for real-time, event-driven, or batch exports
file: Exports and parses files from the NetSuite file cabinet into records.
description: |-
Controls how data is retrieved from NetSuite and which sibling object must be configured:
`search` pairs with `netsuite.searches`, `restlet` with `netsuite.restlet`, `distributed`
with `netsuite.distributed`, and `file` with `netsuite.file.folderInternalId`. For blob
exports (raw file transfer without parsing), leave this field unset, set the export's
top-level `type` to `blob`, and configure `netsuite.internalId` instead. For lookups
(`isLookup: true`), prefer `restlet`, which supports `suiteapp2.0` saved searches with
dynamic inputs; `search` is limited for dynamic lookups.
examples: ["file", "search"]
x-celigo-ai-guidance:
- |-
## CRITICAL: File exports vs Blob exports
- **File exports (with parsing)**: Set netsuite.type to "file" and configure netsuite.file.folderInternalId
- **Blob exports (raw transfer, no parsing)**: Leave netsuite.type BLANK/null, set the export's top-level type to "blob", and configure netsuite.internalId
Do NOT set netsuite.type to "file" for blob exports. For blob exports, this property should be omitted or null.
- |-
## RECOMMENDED TYPES:
- **For Lookups (isLookup: true)**:
- **PREFER "restlet"**: This allows you to use `suiteapp2.0` saved searches with dynamic inputs easily.
- **AVOID "search"**: Standard search type is often limited for dynamic lookups.
- |-
## IMPLEMENTATION GUIDANCE:
- For file exports WITH parsing: Set netsuite.type to "file" and configure netsuite.file.folderInternalId
- For blob exports (no parsing): Leave netsuite.type blank, set export type to "blob", configure netsuite.internalId
- For saved search exports: Set type to "search" and configure netsuite.searches
- For RESTlet exports: Set type to "restlet" and configure netsuite.restlet
- For distributed/real-time exports: Set type to "distributed" and configure netsuite.distributed
- |-
## EXAMPLES:
- "file" - For file cabinet exports with parsing
- "search" - For saved search exports
- null - For blob exports (raw file transfer without parsing)
searches:
type: array
description: |-
Saved search configurations that query NetSuite for records. Each entry references a
saved search by internal ID, names the record type being searched, and can add filter
criteria.
x-celigo-ai-guidance:
- |-
An array of search configurations used to query and retrieve data from NetSuite.
Each search object defines a saved search or ad-hoc query configuration.
- |-
## STRUCTURE
Each item in the array is an object with the following properties:
- savedSearchId: The internal ID of a saved search in NetSuite (string)
- recordType: The NetSuite record type being searched (string, e.g., "customer", "salesorder")
- criteria: Array of search criteria/filters (optional)
- |-
## EXAMPLES
```json
[
{
"savedSearchId": "10",
"recordType": "customer",
"criteria": []
}
]
```
- |-
## IMPLEMENTATION GUIDANCE
- Use savedSearchId to reference an existing saved search in NetSuite
- recordType should match a valid NetSuite record type
- criteria can be used to add additional filters to the search
items:
type: object
properties:
savedSearchId:
type: string
description: Internal ID of the NetSuite saved search to run.
recordType:
type: string
description: |-
NetSuite record type being searched, as its exact lowercase script ID
(e.g. "customer", "salesorder") — not the display name.
x-celigo-ai-guidance:
- |-
Must be the exact lowercase script ID as defined in NetSuite (e.g., "customer",
"salesorder", "invoice", "vendorbill").
This is NOT the display name - use the script ID which is always lowercase with no spaces.
criteria:
type: array
description: Additional filter criteria applied on top of the saved search.
x-celigo-ai-guidance:
- |-
Array of search filters to apply.
Each criterion becomes a `search.createFilter()` call inside Celigo's NetSuite
SuiteScript runtime.
- |-
## REQUIRED FIELDS PER CRITERION
The Celigo backend translates each criterion into a
`search.createFilter({ name, operator, values })` call.
`field` (script-id of the search column) and `operator`
are mandatory -- omitting either crashes the saved search
at runtime with `search.createFilter: Missing a required
argument: name`.
- |-
## EXAMPLE
```json
[
{"field": "trandate", "operator": "after", "searchValue": "2025-01-01"},
{"field": "status", "operator": "anyof", "searchValue": ["SalesOrd:B"]}
]
```
- |-
## FIELD vs SEARCH COLUMN NAMING
The `field` value MUST be a valid search-column `id` for
the chosen `recordType` (the ones listed in the
SEARCH COLUMNS metadata block) -- NOT a body-field name.
For example, Sales Order shipping address line 1 is
`shipaddress1` (search column), not `shipaddr1` (body
field).
items:
type: object
properties:
field:
type: string
description: |-
Script-id of the search column to filter on. REQUIRED.
Must come from the record type's SEARCH COLUMNS list
(NOT the body-fields list). Becomes the `name`
argument of NetSuite's `search.createFilter()`.
operator:
type: string
description: |-
Comparison operator. REQUIRED. Supported values
depend on the column's data type (e.g. `is`,
`isnot`, `after`, `before`, `anyof`,
`noneof`, `contains`, `startswith`,
`greaterthan`, `lessthan`). Must be a string
NetSuite recognises -- mismatched operators cause
runtime rejection.
join:
type: string
description: |-
Optional join name when filtering through a related
record (e.g. `customer` from a sales order). Use
only when the column lives on a related record and
NetSuite supports the join from the base record type.
searchValue:
description: |-
Value (or array of values for `anyof`-style
operators) to compare against the column. Required
for most operators; omit for the few unary operators
(e.g. `isempty`, `isnotempty`).
required:
- field
- operator
metadata:
type: object
description: Supplemental metadata associated with the NetSuite export.
selectoption:
type: object
description: |-
Configuration for `selectoption` exports, which retrieve the available select options
for a NetSuite field.
x-celigo-ai-guidance:
- |-
selectoption: > Represents a selectable option within a NetSuite field,
typically used in dropdown menus, radio buttons, or other selection controls.
Each selectoption consists of a user-friendly label and an associated value that
uniquely identifies the option internally.
This structure enables consistent data entry, filtering,
and categorization within NetSuite forms and records.
- |-
## FIELD BEHAVIOR:
- Defines a single, discrete choice available to users in selection interfaces such as dropdowns, radio buttons, or multi-select lists.
- Can be part of a collection of options presented to the user for making a selection.
- Includes both a display label (visible to users) and a corresponding value (used internally or in API interactions).
- Supports filtering, categorization, and conditional logic based on the selected option.
- May be dynamically generated or statically defined depending on the field configuration.
- |-
## IMPLEMENTATION GUIDANCE:
- Assign a unique and stable value to each selectoption to prevent ambiguity and maintain data integrity.
- Use clear, concise, and user-friendly labels that accurately describe the option's meaning.
- Validate option values against expected data types and formats to ensure compatibility with backend processing.
- Implement localization strategies for labels to support multiple languages without altering the underlying values.
- Consistently apply selectoption structures across all fields requiring predefined choices to standardize user experience.
- Consider accessibility best practices when designing labels and selection controls.
- |-
## EXAMPLES:
- { label: "Active", value: "1" }
- { label: "Inactive", value: "2" }
- { label: "Pending Approval", value: "3" }
- { label: "High Priority", value: "high" }
- { label: "Low Priority", value: "low" }
- |-
## IMPORTANT NOTES:
- The label is intended for display purposes and may be localized; the value is the definitive identifier used in data processing and API calls.
- Values should remain consistent over time to avoid breaking integrations or corrupting data.
- When supporting multiple languages, labels should be translated appropriately while keeping values unchanged.
- Changes to selectoption values or labels should be managed carefully to prevent unintended side effects.
- Selectoption entries may be influenced by the context of the parent record, user roles, or permissions.
- |-
## DEPENDENCY CHAIN:
- Utilized within field definitions that support selection inputs (e
customFieldMetadata:
type: object
description: Metadata describing the custom fields defined in the NetSuite account.
x-celigo-ai-guidance:
- |-
customFieldMetadata:
Metadata information related to custom fields defined within the NetSuite environment,
providing comprehensive details about each custom field's configuration,
behavior,
and constraints to facilitate accurate data handling and UI generation.
- |-
## FIELD BEHAVIOR:
- Contains detailed metadata about custom fields, including their definitions, types, configurations, and constraints.
- Provides contextual information necessary for understanding, validating, and manipulating custom fields programmatically.
- May include attributes such as field ID, label, data type, default values, validation rules, display settings, sourcing information, and field dependencies.
- Used to dynamically interpret or generate UI elements, data validation logic, or data structures based on custom field configurations.
- Reflects the current state of custom fields as defined in the NetSuite account, enabling synchronization between the API consumer and the NetSuite environment.
- |-
## IMPLEMENTATION GUIDANCE:
- Ensure that the metadata accurately reflects the current state of custom fields in the NetSuite account by synchronizing regularly or on configuration changes.
- Update the metadata whenever custom fields are added, modified, or removed to maintain consistency and prevent data integrity issues.
- Use this metadata to validate input data against custom field constraints (e.g., data type, required status, allowed values) before processing or submission.
- Consider caching metadata for performance optimization but implement mechanisms to refresh it periodically or on-demand to capture updates.
- Handle cases where customFieldMetadata might be null, incomplete, or partially loaded gracefully, including fallback logic or error handling.
- Respect user permissions and access controls when retrieving or exposing custom field metadata to ensure compliance with security policies.
- |-
## EXAMPLES:
- A custom field metadata object describing a custom checkbox field with ID "custfield_123", label "Approved", default value false, and display type "inline".
- Metadata for a custom list/record field specifying the list of valid options, their internal IDs, and whether multiple selections are allowed.
- Information about a custom date field including its date format, minimum and maximum allowed dates, and any validation rules applied.
- Metadata describing a custom currency field with precision settings and default currency.
- |-
## IMPORTANT NOTES:
- The structure and content of customFieldMetadata may vary depending on the NetSuite configuration, customizations, and API version.
- Access to custom field metadata may require appropriate permissions within the NetSuite environment; unauthorized access may result in incomplete or no metadata.
skipGrouping:
type: boolean
description: |-
When true, each result row is processed individually instead of being aggregated with
related rows. When false or omitted, related rows are grouped before processing.
x-celigo-ai-guidance:
- |-
skipGrouping:
Indicates whether to bypass the grouping of related records or transactions during processing,
allowing each item to be handled individually rather than aggregated into groups.
- |-
## FIELD BEHAVIOR:
- When set to true, the system processes each record or transaction independently, without combining them into groups based on shared attributes.
- When set to false or omitted, related records or transactions are aggregated according to predefined grouping criteria (e.g., by customer, date, or transaction type) before processing.
- Influences how data is structured, summarized, and reported in outputs or passed to downstream systems.
- Affects the level of detail and granularity available in the processed data.
- |-
## IMPLEMENTATION GUIDANCE:
- Utilize this flag to control processing granularity, especially when detailed, record-level analysis or reporting is required.
- Confirm that downstream systems, reports, or integrations can accommodate ungrouped data if skipGrouping is enabled.
- Assess the potential impact on system performance and data volume, as disabling grouping may significantly increase the number of processed items.
- Consider the use case carefully: grouping is generally preferred for summary reports, while skipping grouping suits detailed audits or troubleshooting.
- For `exportSelect` / `refreshableSelect` dropdowns backed by a NetSuite RESTlet, set `skipGrouping: true`: the UI's option mapper expects flat per-row results, and the platform otherwise wraps each row in its own grouping list (every option's `label` then renders as `undefined`).
- |-
## EXAMPLES:
- skipGrouping: true — Processes each transaction separately, providing detailed, unaggregated data.
- skipGrouping: false — Groups transactions by customer or date, producing summarized results.
- skipGrouping omitted — Defaults to grouping enabled, aggregating related records.
- |-
## IMPORTANT NOTES:
- Enabling skipGrouping can lead to increased processing time, higher memory usage, and larger output datasets.
- Some reports, dashboards, or integrations may require grouped data; verify compatibility before enabling this option.
- The default behavior is typically grouping enabled (skipGrouping = false) unless explicitly overridden.
- Changes to this setting may affect data consistency and comparability with previously generated reports.
- |-
## DEPENDENCY CHAIN:
- Often depends on other properties that define grouping keys or criteria (e.g., groupBy fields).
- May interact with filtering, sorting, or pagination settings within the processing pipeline.
- Could influence or be influenced by aggregation functions or summary calculations applied downstream.
- |-
## TECHNICAL DETAILS:
- Data type: Boolean.
- Default value: false (grouping enabled).
- Implemented as a conditional flag checked during the data aggregation phase.
- When true, bypasses aggregation logic and processes each record individually.
- Typically integrated into the processing workflow to toggle between grouped and ungrouped data handling.
statsOnly:
type: boolean
description: |-
When true, returns only summary statistics about matching records instead of the
detailed records themselves.
x-celigo-ai-guidance:
- |-
statsOnly indicates whether the API response should include only aggregated
statistical summary data without any detailed individual records.
This property is used to optimize response size and improve performance when
detailed data is unnecessary.
- |-
## FIELD BEHAVIOR:
- When set to true, the API returns only summary statistics such as counts, averages, sums, or other aggregate metrics.
- When set to false or omitted, the response includes both detailed data records and the associated statistical summaries.
- Helps reduce network bandwidth and processing time by excluding verbose record-level data.
- Primarily intended for use cases like dashboards, reports, or monitoring tools where only high-level metrics are required.
- |-
## IMPLEMENTATION GUIDANCE:
- Default the value to false to ensure full data retrieval unless explicitly requesting summary-only data.
- Validate that the input is a boolean to prevent unexpected API behavior.
- Use this flag selectively in scenarios where detailed records are not needed to avoid loss of critical information.
- Ensure the API endpoint supports this flag before usage, as some endpoints may not implement statsOnly functionality.
- Adjust client-side logic to handle different response structures depending on the flag's value.
- |-
## EXAMPLES:
- statsOnly: true — returns only aggregated statistics such as total counts, averages, or sums without any detailed entries.
- statsOnly: false — returns full detailed data records along with statistical summaries.
- statsOnly omitted — defaults to false, returning detailed data and statistics.
- |-
## IMPORTANT NOTES:
- Enabling statsOnly disables access to individual record details, which may limit in-depth data analysis.
- The response schema changes significantly when statsOnly is true; clients must handle these differences gracefully.
- Some API endpoints may not support this property; verify compatibility in the API documentation.
- Pagination parameters may be ignored or behave differently when statsOnly is enabled, since detailed records are excluded.
- |-
## DEPENDENCY CHAIN:
- May interact with filtering, sorting, or date range parameters that influence the statistical data returned.
- Can affect pagination logic because detailed records are omitted when statsOnly is true.
- Dependent on the API endpoint's support for summary-only responses.
- |-
## TECHNICAL DETAILS:
- Data type: Boolean.
- Default value: false.
- Typically implemented as a query parameter or part of the request payload depending on API design.
- Alters the response payload structure by excluding detailed record arrays and including only aggregated metrics.
- Helps optimize API performance and reduce response payload size in scenarios where detailed data is unnecessary.
internalId:
type: string
description: |-
Internal ID of the file in the NetSuite file cabinet to export. Required for blob
exports (the export's top-level `type` is `blob`); the file is transferred as-is
without parsing. For parsed file exports, use `netsuite.file.folderInternalId` instead.
examples: ["12345", "67890"]
x-celigo-ai-guidance:
- |-
## CRITICAL: Required for blob exports
This property is REQUIRED when the export type is "blob". For blob exports, you must specify the internalId of the file to export from the NetSuite file cabinet.
- |-
## FIELD BEHAVIOR:
- Identifies a specific file in the NetSuite file cabinet by its internal ID
- Required for blob exports (raw binary file transfers without parsing)
- The file at this internal ID will be exported as-is without parsing
- |-
## IMPLEMENTATION GUIDANCE:
- For blob exports: Set netsuite.type to "blob" (on the export, not netsuite) and provide netsuite.internalId
- Obtain the file's internalId from NetSuite's file cabinet or via API
- Validate the internalId corresponds to an existing file before export
- |-
## EXAMPLES:
- "12345" - Internal ID of a specific file
- "67890" - Another file internal ID
- |-
## IMPORTANT NOTES:
- This is different from netsuite.file.folderInternalId which specifies a folder for file exports with parsing
- For blob exports: Use netsuite.internalId (file ID)
- For file exports with parsing: Use netsuite.file.folderInternalId (folder ID)
blob:
type: object
properties:
purgeFileAfterExport:
type: boolean
description: |-
When true, permanently deletes the file from the NetSuite file cabinet after a
successful export. When false or omitted, the file is left in place.
x-celigo-ai-guidance:
- |-
purgeFileAfterExport:
Whether to delete the file from the system after it has been successfully exported.
This property controls the automatic removal of the source file post-export to
help manage storage and maintain system cleanliness.
- |-
## FIELD BEHAVIOR:
- Determines if the exported file should be removed from the source storage immediately after a successful export operation.
- When set to `true`, the system deletes the file as soon as the export completes without errors.
- When set to `false` or omitted, the file remains intact in its original location after export.
- Facilitates automated cleanup of files to prevent unnecessary storage consumption.
- Does not affect the export process itself; deletion occurs only after confirming export success.
- |-
## IMPLEMENTATION GUIDANCE:
- Confirm that the export operation has fully completed and succeeded before initiating file deletion to avoid data loss.
- Verify that the executing user or system has sufficient permissions to delete files from the source location.
- Assess downstream workflows or processes that might require access to the file after export before enabling purging.
- Implement logging or notification mechanisms to record when files are purged for audit trails and troubleshooting.
- Consider integrating with retention policies or backup systems to prevent accidental loss of important data.
- |-
## EXAMPLES:
- `true` — The file will be deleted immediately after a successful export.
- `false` — The file will remain in the source location after export.
- Omitted or `null` — Defaults to `false`, meaning the file is retained post-export.
- |-
## IMPORTANT NOTES:
- File deletion is permanent and cannot be undone; ensure that the file is no longer needed before enabling this option.
- Use caution in multi-user or multi-process environments where files may be shared or required beyond the export operation.
- Immediate purging may interfere with backup, archival, or compliance requirements if files are deleted too soon.
- Consider implementing safeguards or confirmation steps if enabling automatic purging in production environments.
- |-
## DEPENDENCY CHAIN:
- Relies on the successful completion of the export operation to trigger file deletion.
- Dependent on file system permissions and access controls to allow deletion.
- May be affected by other system settings related to file retention, archival, or cleanup policies.
- Could interact with error handling mechanisms to prevent deletion if export fails or is incomplete.
- |-
## TECHNICAL DETAILS:
- Typically represented as a boolean value (`true` or `false`).
- Default behavior is to retain files unless explicitly set to `true`.
- Deletion should be performed using secure
description: |-
Configuration for blob exports, which transfer files from the NetSuite file cabinet
as-is without parsing them into records. To export a blob, set the export's top-level
`type` to `blob`, set `netsuite.internalId` to the file's internal ID, and leave
`netsuite.type` unset. Use `netsuite.file` instead when file contents should be parsed
into records.
x-celigo-ai-guidance:
- |-
blob:
Configuration for retrieving raw binary files from NetSuite file cabinet WITHOUT
parsing them into records.
Use this for binary file transfers (images, PDFs,
executables) where the file content should be transferred as-is.
- |-
## CRITICAL: Blob export configuration
For blob exports, configure:
1. Set the export's top-level `type` to "blob"
2. Set `netsuite.internalId` to the file's internal ID
3. Leave `netsuite.type` blank/null (do NOT set it to "file")
4. Optionally configure `netsuite.blob.purgeFileAfterExport`
- |-
## When to use blob vs file
- Blob exports: Raw binary transfer WITHOUT parsing - leave netsuite.type blank
- File exports: Parse file contents into records - set netsuite.type to "file"
Do NOT use blob configuration when you want file content parsed into data records.
- |-
## FIELD BEHAVIOR:
- Stores raw binary data including files, images, audio, video, or any non-textual content.
- Supports download operations for binary content from the NetSuite file cabinet.
- File content is transferred as-is without any parsing or transformation.
- May be immutable or mutable depending on the specific NetSuite entity and operation.
- Requires careful handling to maintain data integrity during transmission and storage.
- |-
## IMPLEMENTATION GUIDANCE:
- Always encode binary data (e.g., using base64) when transmitting over text-based protocols such as JSON or XML to ensure data integrity.
- Validate the size of the blob against NetSuite API limits and storage constraints to prevent errors or truncation.
- Implement secure handling practices, including encryption in transit and at rest, to protect sensitive binary data.
- Use appropriate MIME/content-type headers when uploading or downloading blobs to correctly identify the data format.
- Consider chunked uploads/downloads or streaming for large blobs to optimize performance and resource usage.
- Ensure consistent encoding and decoding mechanisms between client and server to avoid data corruption.
- |-
## EXAMPLES:
- A base64-encoded PDF document attached to a NetSuite customer record.
- An image file (PNG or JPEG) stored as a blob for product catalog entries.
- A binary export of transaction data in a proprietary format used for integration with external systems.
- Audio or video files associated with marketing campaigns or training materials.
- Encrypted binary blobs containing sensitive configuration or credential data.
- |-
## IMPORTANT NOTES:
- Blob size may be limited by NetSuite API constraints or underlying storage capabilities; exceeding these limits can cause failures.
- Encoding and decoding must be consistent and correctly implemented to prevent data corruption or loss.
- Large blobs should be handled using chunked or streamed transfers to avoid memory issues and improve reliability.
- Security is paramount; blobs may contain sensitive information requiring encryption and strict access controls.
- Access to blob data typically requires proper authentication and authorization aligned with NetSuite's security model.
- |-
## DEPENDENCY CHAIN:
- Dependent on authentication and authorization mechanisms
restlet:
type: object
properties:
recordType:
type: string
description: |-
NetSuite record type the RESTlet operates on, as its script ID. Custom record
types use the "customrecord_" prefix.
examples: ["customer", "salesOrder", "invoice", "employee", "customrecord_myCustomRecord", "vendor", "purchaseOrder"]
x-celigo-ai-guidance:
- |-
recordType specifies the type of NetSuite record that the RESTlet will interact with.
This property determines the schema, validation rules,
and operations applicable to the record within the NetSuite environment,
directly influencing how data is processed and managed by the RESTlet.
- |-
## FIELD BEHAVIOR:
- Defines the specific NetSuite record type (e.g., customer, salesOrder, invoice) targeted by the RESTlet.
- Influences the structure and format of data payloads sent to and received from the RESTlet.
- Controls validation rules, mandatory fields, and available operations based on the selected record type.
- Affects permissions and access controls enforced during RESTlet execution, ensuring compliance with NetSuite security settings.
- Determines the applicable business logic and workflows triggered by the RESTlet for the specified record type.
- |-
## IMPLEMENTATION GUIDANCE:
- Use the exact internal ID or script ID of the NetSuite record type as recognized by the NetSuite system to ensure accurate targeting.
- Validate the recordType value against the list of supported NetSuite record types to prevent runtime errors and ensure compatibility.
- Confirm that the RESTlet script has the necessary permissions and roles assigned to access and manipulate the specified record type.
- When handling multiple record types dynamically, implement conditional logic to accommodate differences in data structure and processing requirements.
- For custom record types, always use the script ID format (e.g., "customrecord_myCustomRecord") to avoid ambiguity.
- Test RESTlet behavior thoroughly after changing the recordType to verify correct handling of data and operations.
- |-
## EXAMPLES:
- "customer"
- "salesOrder"
- "invoice"
- "employee"
- "customrecord_myCustomRecord"
- "vendor"
- "purchaseOrder"
- |-
## IMPORTANT NOTES:
- The recordType must correspond to a valid and supported NetSuite record type; invalid values will cause API calls to fail.
- Custom record types require referencing by their script IDs, which typically start with "customrecord_".
- Modifying the recordType may necessitate updates to the RESTlet's codebase to handle different data schemas and business logic.
- Permissions and role restrictions in NetSuite can limit access to certain record types, impacting RESTlet functionality.
- Consistency in recordType usage is critical for maintaining data integrity and predictable RESTlet behavior.
- |-
## DEPENDENCY CHAIN:
- Depends on the NetSuite environment's available record types and their configurations.
- Influences the RESTlet's data validation, processing logic, and response formatting.
batchSize:
type: number
description: |-
Number of records the RESTlet returns per page. Larger pages reduce round-trips but
raise NetSuite governance and memory use per call; tune to the record's size.
examples: [100, 200, 1000]
searchId:
type: [string, 'null']
description: |-
Internal ID of the NetSuite saved search the RESTlet executes. Results follow the
saved search's current definition, so editing the search in NetSuite changes the
export output without changing this ID.
examples: ["1234", "5678", "1001", "2002"]
x-celigo-ai-guidance:
- |-
searchId: The unique identifier for a saved search in NetSuite,
used to specify which saved search the RESTlet should execute.
This ID corresponds to the internal ID assigned to saved searches within the NetSuite system.
It enables the RESTlet to run predefined queries and retrieve data based on the
saved search's criteria and configuration.
- |-
## FIELD BEHAVIOR:
- Specifies the exact saved search to be executed by the RESTlet.
- Must correspond to a valid and existing saved search internal ID within the NetSuite account.
- Determines the dataset and filters applied when retrieving search results.
- Typically required when invoking the RESTlet to perform search operations.
- Influences the structure and content of the response based on the saved search definition.
- |-
## IMPLEMENTATION GUIDANCE:
- Verify that the searchId matches an existing saved search internal ID in the target NetSuite environment.
- Validate the searchId format and existence before making the RESTlet call to prevent runtime errors.
- Use the internal ID as a string or numeric value consistent with NetSuite's conventions.
- Implement error handling for scenarios where the searchId is invalid, missing, or inaccessible due to permission restrictions.
- Ensure the integration role or user has appropriate permissions to access and execute the saved search.
- Consider caching or documenting frequently used searchIds to improve maintainability.
- |-
## EXAMPLES:
- "1234" — a numeric internal ID representing a specific saved search.
- "5678" — another valid saved search internal ID.
- "1001" — an example of a saved search ID used to retrieve customer records.
- "2002" — a saved search ID configured to return transaction data.
- |-
## IMPORTANT NOTES:
- The searchId must be accessible by the user or integration role making the RESTlet call; otherwise, access will be denied.
- Providing an incorrect or non-existent searchId will result in errors or empty search results.
- Permissions and sharing settings on the saved search directly affect the data returned by the RESTlet.
- The saved search must be properly configured with the desired filters, columns, and criteria to ensure meaningful results.
- Changes to the saved search (e.g., modifying filters or columns) will impact the RESTlet output without changing the searchId.
- |-
## DEPENDENCY CHAIN:
- Depends on the existence of a saved search configured in the NetSuite account.
- Requires appropriate user or integration role permissions to access the saved search.
useSS2Restlets:
type: boolean
description: |-
When true, calls SuiteScript 2.0 RESTlets. When false or omitted, legacy
SuiteScript 1.0 RESTlets are used.
Set at step creation together with restletVersion — the same
creation-time-only constraint applies (see restletVersion).
x-celigo-ai-guidance:
- |-
useSS2Restlets:
> Specifies whether to use SuiteScript 2.0 RESTlets for API interactions instead
of SuiteScript 1.0 RESTlets.
This setting controls the version of RESTlets invoked during API communication with NetSuite,
impacting compatibility, performance, and available features.
- |-
## FIELD BEHAVIOR:
- Determines the RESTlet version used for all API interactions within the NetSuite integration.
- When set to `true`, the system exclusively uses SuiteScript 2.0 RESTlets.
- When set to `false` or omitted, SuiteScript 1.0 RESTlets are used by default.
- Influences the structure, capabilities, and response formats of API calls.
- |-
## IMPLEMENTATION GUIDANCE:
- This is an authoring-time choice: set it when creating the step (true unless the user explicitly needs a legacy version). It cannot be toggled on a saved step — moving an existing step to a different version means recreating or cloning it.
- Verify that the integrator.io SuiteApp is installed in the target NetSuite environment when authoring SuiteScript 2.0 steps.
- When migrating by recreation, verify any SuiteScript hooks are written for the matching version before cutting over.
- |-
## EXAMPLES:
- `true` — API calls will utilize SuiteScript 2.0 RESTlets, enabling modern scripting features.
- `false` — API calls will continue using legacy SuiteScript 1.0 RESTlets for backward compatibility.
- |-
## IMPORTANT NOTES:
- SuiteScript 2.0 RESTlets support modular script architecture and ES6+ JavaScript features, improving maintainability and performance.
- Legacy RESTlets written in SuiteScript 1.0 may not be compatible with SuiteScript 2.0; migration or parallel support might be required.
- This setting is fixed once the step is saved (same creation-time-only constraint as restletVersion) — moving a step between RESTlet versions means recreating or cloning it, which can change API response formats and behaviors for downstream systems.
- Plan a version migration deliberately: recreate the step with the new version, verify any hooks are written for the matching SuiteScript version, and keep the old step until the new one is verified.
- |-
## DEPENDENCY CHAIN:
- Depends on the deployment and availability of SuiteScript 2.0 RESTlets within the NetSuite account.
- Requires that the API client and integration logic support the RESTlet version selected.
- May depend on other configuration settings related to authentication and script permissions.
- |-
## TECHNICAL DETAILS:
- SuiteScript 2.0 RESTlets use the AMD module format and support
restletVersion:
type: string
enum:
- suitebundle
- suiteapp1.0
- suiteapp2.0
x-enumDescriptions:
suitebundle: Legacy SuiteBundle RESTlet version; slated for deprecation — NetSuite no longer updates SuiteScript 1.0.
suiteapp1.0: SuiteApp 1.0 RESTlet version using SuiteScript 1.0.
suiteapp2.0: SuiteApp 2.0 RESTlet version using SuiteScript 2.0, the modern default.
x-lowercase: true
description: |-
RESTlet version the export invokes. Defaults to `suiteapp2.0` when
`useSS2Restlets` is true, `suitebundle` otherwise.
The version is fixed when the step is created — the Advanced selector is
disabled on existing steps, and migrating an existing export to a
different version means recreating or cloning it.
x-celigo-ai-guidance:
- |-
The NetSuite API version cannot be changed on an existing step — the
UI disables the selector once the step is saved. When asked to migrate
an existing export off the SuiteBundle (or between versions), do not
update this field in place; the migration path is recreating or
cloning the step with the new version. (Real-time listener steps are
`type: distributed` exports — their version lives at
`netsuite.distributed.frameworkVersion`, with the same constraint.)
- |-
SuiteScript hooks are a separate setting from the step's API version:
hook code must be written for the same SuiteScript version as the step
(1.0 and 2.x are not interchangeable), and migrating a hook is its own
task — flag it whenever migrating a step that carries hooks.
- |-
Author new steps with suiteapp2.0 unless the user explicitly requires
a legacy version — NetSuite no longer updates SuiteScript 1.0 and
Celigo plans to deprecate the SuiteBundle option.
criteria:
type: array
description: |-
Filter conditions that limit which records the RESTlet returns (live: an array of
field/operator/value conditions). Each condition pairs a field with an operator and
one or two comparison values.
x-celigo-ai-guidance:
- |-
criteria:
> Defines the set of conditions or filters used to specify which records should
be retrieved or affected by the NetSuite RESTlet operation.
This property enables clients to precisely narrow down the dataset by applying
one or more criteria based on record fields,
comparison operators, and values,
supporting complex logical combinations to tailor the query results.
- |-
## FIELD BEHAVIOR:
- Accepts a structured object or an array representing one or multiple filtering conditions.
- Supports logical operators such as AND, OR, and nested groupings to combine multiple criteria flexibly.
- Each criterion typically includes a field name, an operator (e.g., equals, contains, greaterThan), and a value or set of values.
- Enables filtering on various data types including strings, numbers, dates, and booleans.
- Used to limit the scope of data returned or manipulated by the RESTlet to only those records that meet the specified conditions.
- When omitted or empty, the RESTlet may return all records or apply default filtering behavior as defined by the implementation.
- |-
## IMPLEMENTATION GUIDANCE:
- Validate the criteria structure rigorously to ensure it conforms to the expected schema before processing.
- Support nested criteria groups to allow complex and hierarchical filtering logic.
- Map criteria fields and operators accurately to corresponding NetSuite record fields and search operators, considering data types and operator compatibility.
- Handle empty or undefined criteria gracefully by returning all records or applying sensible default filters.
- Sanitize all input values to prevent injection attacks, malformed queries, or unexpected behavior.
- Provide clear error messages when criteria are invalid or unsupported.
- Optimize query performance by translating criteria into efficient NetSuite search queries.
- |-
## EXAMPLES:
- A single criterion filtering records where status equals "Open":
`{ "field": "status", "operator": "equals", "value": "Open" }`
- Multiple criteria combined with AND logic:
`[{"field": "status", "operator": "equals", "value": "Open"}, {"field": "priority", "operator": "greaterThan", "value": 2}]`
- Nested criteria combining OR and AND:
`{ "operator": "OR", "criteria": [ {"field": "status", "operator": "equals", "value": "Open"}, { "operator": "AND", "criteria": [ {"field": "priority", "operator": "greaterThan", "value":
items:
type: object
properties:
field:
type: string
description: |-
Script-id of the search column to filter on. This is the primary
field the criterion is evaluated against (e.g. "trandate", "status").
join:
type: string
description: |-
Join relationship that applies this filter to a related record's fields instead
of the base record. Must match a join name NetSuite defines for the record type.
operator:
type: string
description: |-
Comparison operator applied between the field and the search value, using
NetSuite's search operator names (e.g. equals, contains, greaterThan).
examples: ["operator"]
x-celigo-ai-guidance:
- |-
operator:
> Specifies the comparison operator used to evaluate the criteria in a NetSuite RESTlet request.
This operator determines how the field value is compared against the specified
criteria value(s) to filter or query records.
- |-
## FIELD BEHAVIOR:
- Defines the type of comparison between a field and a value (e.g., equality, inequality, greater than).
- Influences the logic of the criteria evaluation in RESTlet queries.
- Supports various operators such as equals, not equals, greater than, less than, contains, etc.
- |-
## IMPLEMENTATION GUIDANCE:
- Use valid NetSuite-supported operators to ensure correct query behavior.
- Match the operator type with the data type of the field being compared (e.g., use numeric operators for numeric fields).
- Combine multiple criteria with appropriate logical operators if needed.
- Validate operator values to prevent errors in RESTlet execution.
- |-
## EXAMPLES:
- "operator": "is" (checks if the field value is equal to the specified value)
- "operator": "isnot" (checks if the field value is not equal to the specified value)
- "operator": "greaterthan" (checks if the field value is greater than the specified value)
- "operator": "contains" (checks if the field value contains the specified substring)
- |-
## IMPORTANT NOTES:
- The operator must be compatible with the field type and the value provided.
- Incorrect operator usage can lead to unexpected query results or errors.
- Operators are case-sensitive and should match NetSuite's expected operator strings.
- |-
## DEPENDENCY CHAIN:
- Depends on the field specified in the criteria to determine valid operators.
- Works in conjunction with the criteria value(s) to form a complete condition.
- May be combined with logical operators when multiple criteria are used.
- |-
## TECHNICAL DETAILS:
- Typically represented as a string value in the RESTlet criteria JSON object.
- Supported operators align with NetSuite's SuiteScript search operators.
- Must conform to the list of operators recognized by the NetSuite RESTlet API.
searchValue:
type: string
description: Value the criteria field is compared against when filtering records.
examples: ["Acme Corporation", "2024-01-01", "Pending"]
x-celigo-ai-guidance:
- |-
searchValue:
The value used as the search criterion to filter results in the NetSuite RESTlet API.
This value is matched against the specified search field to retrieve relevant
records based on the search parameters provided.
- |-
## FIELD BEHAVIOR:
- Acts as the primary input for filtering search results.
- Supports various data types depending on the search field (e.g., string, number, date).
- Used in conjunction with other search criteria to refine query results.
- Can be a partial or full match depending on the search configuration.
- |-
## IMPLEMENTATION GUIDANCE:
- Ensure the value type matches the expected type of the search field.
- Validate the input to prevent injection attacks or malformed queries.
- Use appropriate encoding if the value contains special characters.
- Combine with logical operators or additional criteria for complex searches.
- |-
## EXAMPLES:
- "Acme Corporation" for searching customer names.
- 1001 for searching by internal record ID.
- "2024-01-01" for searching records created on or after a specific date.
- "Pending" for filtering records by status.
- |-
## IMPORTANT NOTES:
- The effectiveness of the search depends on the accuracy and format of the searchValue.
- Case sensitivity may vary based on the underlying NetSuite configuration.
- Large or complex search values may impact performance.
- Null or empty values may result in no filtering or return all records.
- |-
## DEPENDENCY CHAIN:
- Depends on the searchField property to determine which field the searchValue applies to.
- Works alongside searchOperator to define how the searchValue is compared.
- Influences the results returned by the RESTlet endpoint.
- |-
## TECHNICAL DETAILS:
- Typically passed as a string in the API request payload.
- May require serialization or formatting based on the API specification.
- Integrated into the NetSuite search query logic on the server side.
- Subject to NetSuite's search limitations and indexing capabilities.
searchValue2:
type: string
description: |-
Second comparison value for operators that take two values, such as "between"
and "not between". Omit when the operator needs only one value.
x-celigo-ai-guidance:
- |-
searchValue2 is an optional property used to specify the second value in a
search criterion within the NetSuite RESTlet API.
It is typically used in conjunction with search operators that require two values,
such as "between" or "not between," to define a range or a pair of comparison values.
- |-
## FIELD BEHAVIOR:
- Represents the second operand or value in a search condition.
- Used primarily with operators that require two values (e.g., "between", "not between").
- Optional field; may be omitted if the operator only requires a single value.
- Works alongside searchValue (the first value) to form a complete search criterion.
- |-
## IMPLEMENTATION GUIDANCE:
- Ensure that searchValue2 is provided only when the selected operator requires two values.
- Validate the data type of searchValue2 to match the expected type for the field being searched (e.g., date, number, string).
- When using range-based operators, searchValue2 should represent the upper bound or second boundary of the range.
- If the operator does not require a second value, omit this property to avoid errors.
- |-
## EXAMPLES:
- For a date range search: searchValue = "2023-01-01", searchValue2 = "2023-12-31" with operator "between".
- For a numeric range: searchValue = 100, searchValue2 = 200 with operator "between".
- For a "not between" operator: searchValue = 50, searchValue2 = 100.
- |-
## IMPORTANT NOTES:
- Providing searchValue2 without a compatible operator may result in an invalid search query.
- The data type and format of searchValue2 must be consistent with searchValue and the field being queried.
- This property is ignored if the operator only requires a single value.
- Proper validation and error handling should be implemented when processing this field.
- |-
## DEPENDENCY CHAIN:
- Dependent on the "operator" property within the same search criterion.
- Works in conjunction with "searchValue" to define the search condition.
- Part of the "criteria" array or object in the NetSuite RESTlet search request.
- |-
## TECHNICAL DETAILS:
- Data type varies depending on the field being searched (string, number, date, etc.).
- Typically serialized as a JSON property in the RESTlet request payload.
- Must conform to the expected format for the field and operator to avoid API errors.
- Used internally by NetSuite to construct the appropriate
formula:
type: string
description: |-
NetSuite formula expression used as the filter condition, for criteria that
standard field-operator-value comparisons can't express.
examples: ["CASE WHEN {status} =", "TO_DATE({createddate}) >= TO_DATE(", "NVL({amount}, 0) > 1000"]
x-celigo-ai-guidance:
- |-
formula:
> A string representing a custom formula used to define criteria for filtering
or querying data within the NetSuite RESTlet API.
This formula allows users to specify complex conditions using NetSuite's formula syntax,
enabling advanced and flexible data retrieval.
- |-
## FIELD BEHAVIOR:
- Accepts a formula expression as a string that defines custom filtering logic.
- Used to create dynamic and complex criteria beyond standard field-value comparisons.
- Evaluated by the NetSuite backend to filter records according to the specified logic.
- Can incorporate NetSuite formula functions, operators, and field references.
- |-
## IMPLEMENTATION GUIDANCE:
- Ensure the formula syntax complies with NetSuite's formula language and supported functions.
- Validate the formula string before submission to avoid runtime errors.
- Use this field when standard criteria fields are insufficient for the required filtering.
- Combine with other criteria fields as needed to build comprehensive queries.
- |-
## EXAMPLES:
- "CASE WHEN {status} = 'Open' THEN 1 ELSE 0 END = 1"
- "TO_DATE({createddate}) >= TO_DATE('2023-01-01')"
- "NVL({amount}, 0) > 1000"
- |-
## IMPORTANT NOTES:
- Incorrect or invalid formulas may cause the API request to fail or return errors.
- The formula must be compatible with the context of the query and the fields available.
- Performance may be impacted if complex formulas are used extensively.
- Formula evaluation is subject to NetSuite's formula engine capabilities and limitations.
- |-
## DEPENDENCY CHAIN:
- Depends on the availability of fields referenced within the formula.
- Works in conjunction with other criteria properties in the request.
- Requires understanding of NetSuite's formula syntax and functions.
- |-
## TECHNICAL DETAILS:
- Data type: string.
- Supports NetSuite formula syntax including SQL-like expressions and functions.
- Evaluated server-side during the processing of the RESTlet request.
- Must be URL-encoded if included in query parameters of HTTP requests.
columns:
type: array
description: |-
Columns (record fields) the RESTlet returns for each matching record (live: an array
of column descriptors). When omitted, all available columns for the record type are
returned.
x-celigo-ai-guidance:
- |-
columns:
> Specifies the set of columns (fields) to be retrieved or manipulated in the
NetSuite RESTlet operation.
This property defines which specific fields from the records should be included
in the response or used during processing,
enabling precise control over the data returned or affected.
By selecting only relevant columns,
it helps optimize performance and reduce payload size,
ensuring efficient data handling tailored to the operation's requirements.
- |-
## FIELD BEHAVIOR:
- Determines the exact fields (columns) to be included in data retrieval, update, or manipulation operations.
- Supports specifying multiple columns to customize the dataset returned or processed.
- Limits the data payload by including only the specified columns, improving performance and reducing bandwidth.
- Influences the structure, content, and size of the response from the RESTlet.
- If omitted, defaults to retrieving all available columns for the target record type, which may impact performance.
- Columns specified must be valid and accessible for the target record type to avoid errors.
- |-
## IMPLEMENTATION GUIDANCE:
- Accepts an array or list of column identifiers, which can be simple strings or objects with detailed specifications (e.g., `{ name: "fieldname" }`).
- Column identifiers should correspond exactly to valid NetSuite record field names or internal IDs.
- Validate column names against the target record schema before execution to prevent runtime errors.
- Use this property to optimize RESTlet calls by limiting data to only necessary fields, especially in large datasets.
- When specifying complex columns (e.g., joined fields or formula fields), ensure the correct syntax and structure are used.
- Consider the permissions and roles associated with the RESTlet user to ensure access to the specified columns.
- |-
## EXAMPLES:
- `["internalid", "entityid", "email"]` — retrieves basic identifying and contact fields.
- `[ { name: "internalid" }, { name: "entityid" }, { name: "email" } ]` — object notation for specifying columns.
- `["tranid", "amount", "status"]` — retrieves transaction-specific fields.
- `[ { name: "custbody_custom_field" }, { name: "createddate" } ]` — includes custom and system fields.
- `["item", "quantity", "rate"]` — fields relevant to item records or line items.
- |-
## IMPORTANT NOTES:
- Omitting the `columns` property typically
items:
type: object
properties:
_id:
type: string
description: Column field id (NetSuite field internal id).
name:
type: string
description: Column field name as referenced in the search.
x-celigo-ai-guidance:
- |-
REQUIRED. The NetSuite field ID (script ID) of the column to return -- e.g.
`"internalid"`, `"tranid"`, `"entity"`, `"trandate"`, `"status"`,
`"shipaddress1"`, `"item"`.
This is the same identifier that appears in the NetSuite UI's "Field" picker on saved searches,
NOT a label or display name (use `label` for that).
- |-
## PLATFORM ENFORCEMENT
NetSuite's `search.createColumn` rejects entries
without a `name` argument with
`[application · stage=apiCall] search.createColumn:
Missing a required argument: name`. The platform
validation runs before any column data is read, so
the build fails fast.
- |-
## NAME MUST BE A SEARCH COLUMN, NOT A BODY FIELD
NetSuite distinguishes between body fields (the
record's intrinsic fields like `shipaddr1`) and
search columns (the fields exposed for filtering /
sorting in saved searches, like `shipaddress1`).
`columns[].name` MUST be a search column ID, not
a body field ID. See the SEARCH COLUMNS section of
the metadata fetcher output for the valid search
column names for the chosen `recordType`.
- |-
## EXAMPLES
- `"internalid"` -- the record's internal id
- `"tranid"` -- transaction number
- `"entity"` -- customer / vendor reference
- `"datecreated"`, `"lastmodifieddate"`
- `"status"` -- transaction status
- `"shipaddress1"` -- shipping address line 1
- `"item"` -- line item field
join:
type: string
description: |-
Join relationship that sources this column from a related record instead of
the base record. Supports dot notation for nested joins (e.g. "employee.manager").
examples: ["customer", "item", "employee.manager", "vendor"]
summary:
type: string
description: |-
Aggregation function applied to this column in a saved search
(e.g. SUM, COUNT, AVG, MIN, MAX).
formula:
type: string
description: |-
NetSuite formula expression that computes this column's value at runtime, such
as a CASE statement or date formatting, instead of reading a stored field.
examples: ["CASE WHEN {status} =", "NVL({amount}, 0) * 0.1", "TO_CHAR({trandate},"]
x-celigo-ai-guidance:
- |-
## IMPORTANT NOTES:
- The formula must be compatible with the context in which it is used (e.g., search column, RESTlet).
- Incorrect formulas can cause runtime errors or unexpected results.
- Some functions or operators may not be supported depending on the NetSuite version or API context.
- Formula evaluation respects user permissions and data visibility.
- |-
## DEPENDENCY CHAIN:
- Depends on the availability of referenced fields within the record or search context.
- Relies on NetSuite's formula parsing and evaluation engine.
- Interacts with the RESTlet execution environment to produce output.
- |-
## TECHNICAL DETAILS:
- Data type: string containing a formula expression.
- Supports NetSuite formula syntax including SQL-like CASE statements, arithmetic operations, and built-in functions.
- Evaluated server-side during RESTlet execution or saved search processing.
- |-
formula:
> A string representing a custom formula used to calculate or derive values
dynamically within the context of the NetSuite RESTlet columns.
This formula can include field references, operators,
and functions supported by NetSuite's formula syntax to perform computations or
conditional logic on record data.
- |-
## FIELD BEHAVIOR:
- Accepts a formula expression as a string that defines how to compute the column's value.
- Can reference other fields, constants, and use NetSuite-supported functions and operators.
- Evaluated at runtime to produce dynamic results based on the current record data.
- Used primarily in saved searches, reports, or RESTlet responses to customize output.
- |-
## IMPLEMENTATION GUIDANCE:
- Ensure the formula syntax complies with NetSuite's formula language and supported functions.
- Validate the formula string to prevent errors during execution.
- Use field IDs or aliases correctly within the formula to reference data fields.
- Test formulas thoroughly in NetSuite UI before deploying via RESTlet to ensure correctness.
- Consider performance implications of complex formulas on large datasets.
- |-
## EXAMPLES:
- "CASE WHEN {status} = 'Open' THEN 1 ELSE 0 END" — returns 1 if status is Open, else 0.
- "NVL({amount}, 0) * 0.1" — calculates 10% of the amount, treating null as zero.
- "TO_CHAR({trandate}, 'YYYY-MM-DD')" — formats the transaction date as a string.
label:
type: string
description: |-
Display name for the column in search results and reports. Purely cosmetic; it
does not affect the underlying data.
examples: ["Customer Name", "Invoice Date", "Total Amount", "Status"]
x-celigo-ai-guidance:
- |-
label:
| The display name or title of the column as it appears in the user interface or reports.
- |-
## FIELD BEHAVIOR:
- Represents the human-readable name for a column in a dataset or report.
- Used to identify the column in UI elements such as tables, forms, or export files.
- Should be concise yet descriptive enough to convey the column's content.
- |-
## IMPLEMENTATION GUIDANCE:
- Ensure the label is localized if the application supports multiple languages.
- Avoid using technical jargon; prefer user-friendly terminology.
- Keep the label length reasonable to prevent UI truncation.
- Update the label consistently when the underlying data or purpose changes.
- |-
## EXAMPLES:
- "Customer Name"
- "Invoice Date"
- "Total Amount"
- "Status"
- |-
## IMPORTANT NOTES:
- The label does not affect the data or the column's functionality; it is purely for display.
- Changing the label does not impact data processing or storage.
- Labels should be unique within the same context to avoid confusion.
- |-
## DEPENDENCY CHAIN:
- Depends on the column definition within the dataset or report configuration.
- May be linked to localization resources if internationalization is supported.
- |-
## TECHNICAL DETAILS:
- Typically a string data type.
- May support Unicode characters for internationalization.
- Stored as metadata associated with the column definition in the system.
sort:
type: boolean
description: When true, query results are sorted by this column's values.
markExportedBatchSize:
type: integer
description: |-
Number of records updated per RESTlet call when marking records as exported in
NetSuite (live: an integer count, e.g. 100).
maximum: 100
minimum: 1
x-celigo-ui-override: >-
min 1 / max 100 mirror the integrator-ui NetSuite export form
(netsuite.restlet.markExportedBatchSize, fallsWithinNumericalRange min 1 / max 100).
Encoded to mirror the form so builders produce valid configurations.
hooks:
type: [object, 'null']
properties:
batchSize:
type: number
description: Number of records processed per batch when the RESTlet hook runs.
x-celigo-ai-guidance:
- |-
batchSize specifies the number of records or items to be processed in a single
batch during the execution of the NetSuite RESTlet hook.
This parameter helps control the workload size for each batch operation,
optimizing performance and resource utilization by balancing processing
efficiency and system constraints.
- |-
## FIELD BEHAVIOR:
- Determines the maximum number of records or items processed in one batch cycle.
- Directly influences the frequency and duration of batch processing operations.
- Helps manage memory consumption and processing time by limiting the batch workload.
- Affects overall throughput and latency of batch operations, impacting system responsiveness.
- Controls how data is segmented and processed in discrete units during RESTlet execution.
- |-
## IMPLEMENTATION GUIDANCE:
- Configure batchSize based on the system's processing capacity, expected data volume, and performance goals.
- Use smaller batch sizes in environments with limited resources or strict execution time limits to prevent timeouts.
- Larger batch sizes can improve throughput by reducing the number of batch cycles but may increase individual batch processing time and risk of hitting governance limits.
- Always validate that batchSize is a positive integer greater than zero to ensure proper operation.
- Take into account NetSuite API governance limits, such as usage units and execution time, when determining batchSize.
- Monitor system performance and adjust batchSize dynamically if possible to optimize processing efficiency.
- Ensure batchSize aligns with other batch-related configurations to maintain consistency and predictable behavior.
- |-
## EXAMPLES:
- batchSize: 100 — processes 100 records per batch, balancing throughput and resource use.
- batchSize: 500 — processes 500 records per batch for higher throughput in robust environments.
- batchSize: 10 — processes 10 records per batch for fine-grained control and minimal resource impact.
- batchSize: 1 — processes records individually, useful for debugging or very resource-sensitive scenarios.
- |-
## IMPORTANT NOTES:
- Excessively high batchSize values may cause processing timeouts, exceed NetSuite governance limits, or lead to memory exhaustion.
- Very low batchSize values can result in inefficient processing due to increased overhead and more frequent batch invocations.
- The optimal batchSize is context-dependent and should be determined through testing and monitoring.
- batchSize should be consistent with other batch processing parameters to avoid conflicts or unexpected behavior.
- Changes to batchSize may require adjustments in error handling and retry logic to accommodate different batch sizes.
- |-
## DEPENDENCY CHAIN:
- Depends on the batch processing logic implemented within the RESTlet hook.
preSend:
type: object
properties:
fileInternalId:
type: string
description: Internal ID of the file in the NetSuite file cabinet that the preSend hook references.
examples: ["12345", "67890", "file_98765"]
x-celigo-ai-guidance:
- |-
fileInternalId:
The unique internal identifier assigned to a file within the NetSuite system.
This identifier is used to precisely reference and manipulate a specific file
during API operations,
particularly within pre-send processing hooks in RESTlets.
It ensures accurate targeting of file resources by uniquely identifying files
stored in the NetSuite file cabinet.
- |-
## FIELD BEHAVIOR:
- Represents a unique numeric or alphanumeric identifier assigned by NetSuite to each file.
- Used to retrieve, update, or reference a file during the preSend hook execution.
- Must correspond to an existing file within the NetSuite file cabinet.
- Immutable throughout the file's lifecycle; remains constant unless the file is deleted and recreated.
- Serves as a key reference for file-related operations in automated workflows and integrations.
- |-
## IMPLEMENTATION GUIDANCE:
- Always validate that the fileInternalId exists and is accessible before performing operations.
- Use this ID to fetch file metadata, content, or perform updates within the preSend hook.
- Implement error handling to manage cases where the fileInternalId does not correspond to a valid or accessible file.
- Ensure that the executing user or integration has the necessary permissions to access the file referenced by this ID.
- Avoid hardcoding this ID; retrieve dynamically when possible to maintain flexibility and accuracy.
- |-
## EXAMPLES:
- 12345
- "67890"
- "file_98765"
- |-
## IMPORTANT NOTES:
- The fileInternalId is specific to each NetSuite account and environment; it is not globally unique across different accounts.
- Do not expose this identifier publicly, as it may reveal sensitive internal system details.
- Modifications to the file's name, location, or metadata do not affect the internal ID.
- This ID is essential for linking files reliably in automated processes, integrations, and RESTlet hooks.
- Deleting and recreating a file will result in a new fileInternalId.
- |-
## DEPENDENCY CHAIN:
- Depends on the existence of the file within the NetSuite file cabinet.
- Requires appropriate permissions to access or manipulate the file.
- Utilized within preSend hooks to reference files accurately during API operations.
- |-
## TECHNICAL DETAILS:
- Typically a numeric or alphanumeric string assigned by NetSuite upon file creation.
- Stored internally within NetSuite's database as the primary key for file records.
- Used as a parameter in RESTlet API calls to identify and operate on specific files.
- Immutable identifier that does not change unless the file is deleted and recreated.
function:
type: string
description: |-
Name of the function invoked as the preSend hook. The function can modify
the payload immediately before it is sent.
examples: ["sanitizePayload", "addAuthenticationHeaders", "transformRequestData", "logRequestDetails"]
x-celigo-ai-guidance:
- |-
function:
> Specifies the name of the custom function to be invoked during the preSend
hook phase in the NetSuite RESTlet integration.
This function enables developers to implement custom logic for processing or
modifying the request payload immediately before it is dispatched to the
NetSuite RESTlet endpoint.
It serves as a critical extension point for tailoring request data,
adding headers, sanitizing inputs,
or performing any preparatory steps necessary to meet integration requirements.
- |-
## FIELD BEHAVIOR:
- Identifies the exact function to execute during the preSend hook phase.
- Allows customization and transformation of the outgoing request payload or context.
- The specified function is called synchronously or asynchronously depending on implementation support.
- Modifications made by this function directly affect the data sent to the NetSuite RESTlet.
- Must reference a valid, accessible function within the integration's runtime environment.
- |-
## IMPLEMENTATION GUIDANCE:
- Confirm that the function name matches a defined and exported function within the integration codebase.
- The function should accept the current request payload or context as input and return the modified payload or context.
- Implement robust error handling within the function to avoid unhandled exceptions that could disrupt the request flow.
- Optimize the function for performance to minimize latency in request processing.
- If asynchronous operations are supported, ensure proper handling of promises or callbacks.
- Document the function's behavior clearly to facilitate maintenance and future updates.
- |-
## EXAMPLES:
- "sanitizePayload" — cleans and validates request data before sending.
- "addAuthenticationHeaders" — injects necessary authentication tokens or headers.
- "transformRequestData" — restructures or enriches the payload to match API expectations.
- "logRequestDetails" — captures request metadata for auditing or debugging purposes.
- |-
## IMPORTANT NOTES:
- The function must be correctly implemented and accessible; otherwise, runtime errors will occur.
- This hook executes immediately before the request is sent, so any changes here directly impact the outgoing data.
- Ensure that the function's side effects do not unintentionally alter unrelated parts of the request or integration state.
- If asynchronous processing is used, verify that the integration framework supports it to avoid unexpected behavior.
- Testing the function thoroughly is critical to ensure reliable integration behavior.
- |-
## DEPENDENCY CHAIN:
- Requires the preSend hook to be enabled and properly configured in the integration settings.
- Depends on the presence of the named
configuration:
type: object
description: Settings that control the preSend hook's behavior.
x-celigo-ai-guidance:
- '## IMPORTANT'
- |-
configuration:
> An object containing configuration settings that influence the behavior of the
preSend hook in the NetSuite RESTlet integration.
This object serves as a centralized control point for customizing how requests
are processed and modified before being sent to the NetSuite RESTlet endpoint.
It can include a variety of parameters such as authentication credentials,
logging preferences, request modification flags, timeout settings,
retry policies,
and feature toggles that tailor the preSend hook's operation to specific integration needs.
- |-
## FIELD BEHAVIOR:
- Holds key-value pairs that define how the preSend hook processes and modifies outgoing requests.
- Can include settings such as authentication parameters (e.g., tokens, API keys), request modification flags (e.g., header adjustments), logging options (e.g., enable/disable logging), timeout durations, retry counts, and feature toggles.
- Is accessed and potentially updated dynamically during the execution of the preSend hook to adapt request handling based on current context or conditions.
- Influences the flow and outcome of the preSend hook, potentially altering request payloads, headers, or other metadata before transmission.
- |-
## IMPLEMENTATION GUIDANCE:
- Define clear, descriptive, and consistent keys within the configuration object to avoid ambiguity and ensure maintainability.
- Validate all configuration values rigorously before applying them to prevent runtime errors or unexpected behavior.
- Use this object to centralize control over preSend hook behavior, enabling easier updates, debugging, and feature management.
- Document all possible configuration options, their expected data types, default values, and their specific effects on the preSend hook's operation.
- Ensure sensitive information within the configuration (e.g., authentication tokens) is handled securely, following best practices for encryption and access control.
- Consider versioning the configuration schema if multiple versions of the preSend hook or integration exist.
- |-
## EXAMPLES:
- `{ "enableLogging": true, "authToken": "abc123", "modifyHeaders": false }`
- `{ "retryCount": 3, "timeout": 5000 }`
- `{ "useSandbox": true, "customHeader": "X-Custom-Value" }`
- `{ "authenticationType": "OAuth2", "refreshToken": "xyz789", "logLevel": "verbose" }`
- `{ "enableCaching": false, "maxRetries": 5, "requestPriority": "high" }`
description: |-
Hook invoked immediately before the payload is sent, allowing custom logic to
modify or validate it.
x-celigo-ai-guidance:
- |-
preSend is a hook function that is executed immediately before a RESTlet sends a
response back to the client.
It allows for last-minute modifications or logging of the response data,
enabling customization of the output or performing additional processing steps
prior to transmission.
This hook provides a critical interception point to ensure the response adheres
to business rules,
compliance requirements,
or client-specific formatting before it leaves the server.
- |-
## FIELD BEHAVIOR:
- Invoked right before the RESTlet response is sent to the client.
- Receives the response data as input and can modify it.
- Can be used to log, audit, or transform the response payload.
- Should return the final response object to be sent.
- Supports both synchronous and asynchronous execution depending on the implementation.
- Any changes made here directly impact the final output received by the client.
- |-
## IMPLEMENTATION GUIDANCE:
- Implement as a synchronous or asynchronous function depending on the environment and use case.
- Ensure any modifications maintain the expected response format and data integrity.
- Avoid long-running or blocking operations to prevent delaying the response delivery.
- Handle errors gracefully within the hook to prevent disrupting the overall RESTlet response flow.
- Validate the modified response to ensure it complies with API schema and client expectations.
- Use this hook to enforce security measures such as masking sensitive data or adding audit trails.
- |-
## EXAMPLES:
- Adding a timestamp or metadata (e.g., request ID, processing duration) to the response object.
- Masking or removing sensitive information (e.g., personal identifiers, confidential fields) from the response.
- Logging response details for auditing or debugging purposes.
- Transforming response data structure or formatting to match client-specific requirements.
- Injecting additional headers or status information into the response payload.
- |-
## IMPORTANT NOTES:
- This hook runs after all business logic but before the response is finalized and sent.
- Modifications here directly affect what the client ultimately receives.
- Errors thrown in this hook may cause the RESTlet to fail or return an error response.
- Use this hook to enforce response-level policies, compliance, or data governance rules.
- Avoid introducing side effects that could alter the idempotency or consistency of the response.
- Testing this hook thoroughly is critical to ensure it does not unintentionally break client integrations.
- |-
## DEPENDENCY CHAIN:
- Triggered after the main RESTlet processing logic completes and the response object is prepared.
- Precedes the actual sending of the HTTP response to the client.
description: Custom hook functions executed at defined points in the RESTlet export lifecycle.
x-celigo-ai-guidance:
- |-
hooks:
> An array of hook definitions that specify custom functions to be executed at
various points during the lifecycle of the RESTlet script in NetSuite.
These hooks enable developers to inject additional logic before or after
standard processing events,
allowing for extensive customization and extension of the RESTlet's behavior to
meet specific business requirements.
- |-
## FIELD BEHAVIOR:
- Defines one or more hooks that trigger custom code execution at designated lifecycle events.
- Hooks can be configured to run at standard lifecycle events such as beforeLoad, beforeSubmit, afterSubmit, or at custom-defined events tailored to specific needs.
- Each hook entry typically includes the event name, the callback function to execute, and optional parameters or context information.
- Supports both synchronous and asynchronous execution modes depending on the hook type and implementation context.
- Hooks execute in the order they are defined, allowing for controlled sequencing of custom logic.
- Hooks can modify input data, perform validations, log information, or alter output responses as needed.
- |-
## IMPLEMENTATION GUIDANCE:
- Ensure that each hook function is properly defined, accessible, and tested within the RESTlet script context to avoid runtime failures.
- Validate hook event names against the list of supported lifecycle events to prevent misconfiguration and errors.
- Use hooks to encapsulate reusable business logic, enforce data integrity, or integrate with external systems and services.
- Implement robust error handling within hook functions to prevent exceptions from disrupting the main RESTlet processing flow.
- Document each hook's purpose, expected inputs, outputs, and side effects clearly to facilitate maintainability and future enhancements.
- Consider performance implications of hooks, especially those performing asynchronous operations or external calls, to maintain RESTlet responsiveness.
- When multiple hooks are defined for the same event, design them to avoid conflicts and ensure predictable outcomes.
- |-
## EXAMPLES:
- Defining a hook to validate and sanitize input data before processing a RESTlet request.
- Adding a hook to log detailed request and response information after the RESTlet completes execution for auditing purposes.
- Using a hook to modify or enrich the response payload dynamically before it is returned to the client application.
- Implementing a hook to trigger notifications or update related records asynchronously after data submission.
- Creating a custom hook event to perform additional security checks beyond standard validation.
- |-
## IMPORTANT NOTES:
- Improper use or misconfiguration of hooks can lead to unexpected behavior, performance degradation, or runtime errors.
cLocked:
type: object
# TODO(verify): 0 occurrences across all stored exports (restlet/distributed/top-level); semantics unverified
description: Lock state that prevents modification of the configuration when set.
description: |-
Configuration for `restlet` exports. Identifies the RESTlet script that retrieves data
from NetSuite, along with the saved search, criteria, and columns it executes with.
x-celigo-ai-guidance:
- |-
restlet:
The identifier or URL of the NetSuite Restlet script to be invoked for
performing custom server-side logic or data processing within the NetSuite
environment.
This property specifies which Restlet endpoint the integration or application
should call to execute specific business logic,
automate workflows, or retrieve and manipulate data dynamically.
It can be represented as an internal script ID, a relative URL path,
or a full external URL depending on the integration scenario and access method.
- |-
## FIELD BEHAVIOR:
- Defines the specific target Restlet script or endpoint for API calls within the NetSuite environment.
- Routes requests to custom server-side scripts developed using NetSuite's SuiteScript framework.
- Enables execution of tailored business processes, data validations, transformations, or integrations.
- Supports various HTTP methods such as GET, POST, PUT, and DELETE depending on the Restlet's implementation.
- Can be specified as a script ID, a relative URL path, or a fully qualified URL based on deployment and access context.
- Acts as the primary entry point for invoking custom logic that extends or complements standard NetSuite functionality.
- |-
## IMPLEMENTATION GUIDANCE:
- Confirm that the Restlet script is properly deployed, enabled, and accessible within the target NetSuite account.
- Use the internal script ID format (e.g., "customscript_my_restlet") when calling via SuiteScript or internal APIs.
- Use the relative URL path (e.g., "/app/site/hosting/restlet.nl?script=123&deploy=1") or full URL for external integrations or REST clients.
- Verify that the Restlet supports the required HTTP methods and handles input/output data formats correctly (JSON, XML, etc.).
- Secure the Restlet endpoint by implementing authentication mechanisms such as OAuth 2.0, token-based authentication, or NetSuite session credentials.
- Implement robust error handling and retry logic to manage scenarios where the Restlet is unavailable or returns errors.
- Test the Restlet thoroughly in a sandbox environment before deploying to production to ensure expected behavior and security compliance.
- |-
## EXAMPLES:
- "customscript_my_restlet" (internal script ID used in SuiteScript calls)
- "/app/site/hosting/restlet.nl?script=123&deploy=1" (relative URL for REST calls within NetSuite)
- "https://rest.netsuite.com/app/site/hosting/restlet.nl?script=456&deploy=2" (full external URL for third-party integrations)
- "customscript_sales_order_processor" (a Rest
distributed:
type: object
properties:
recordType:
type: string
description: |-
NetSuite record type the distributed export listens to, as its exact lowercase
script ID (e.g. "customer", "salesorder") — not the display name. Custom record
types use the "customrecord_" prefix.
examples: ["customer", "invoice", "salesorder", "itemfulfillment", "vendorbill", "employee", "purchaseorder", "creditmemo"]
x-celigo-ai-guidance:
- |-
Must be the exact lowercase script ID as defined in NetSuite (e.g., "customer",
"salesorder", "invoice", "vendorbill").
This is NOT the display name - use the script ID which is always lowercase with no spaces.
- |-
## EXAMPLES:
- "customer"
- "invoice"
- "salesorder"
- "itemfulfillment"
- "vendorbill"
- "employee"
- "purchaseorder"
- "creditmemo"
- |-
## IMPORTANT NOTES:
- Must be lowercase script ID, not the display name
- Custom record types use format "customrecord_scriptid"
executionContext:
type: array
description: |-
NetSuite execution contexts that trigger this distributed export. A record change
fires the export only when it occurs in one of the listed contexts.
x-celigo-ai-guidance:
- |-
Specifies which NetSuite execution contexts should trigger this export.
When a record change occurs in one of the specified contexts,
the export will be triggered.
- |-
## DEFAULT VALUE
If not specified, defaults to: ["userinterface", "webstore"]
- |-
## VALID VALUES
- "userinterface" - User interactions in the NetSuite UI
- "webservices" - SOAP web services calls
- "csvimport" - CSV import operations
- "offlineclient" - Offline client synchronization
- "portlet" - Portlet interactions
- "scheduled" - Scheduled script executions
- "suitelet" - Suitelet executions
- "custommassupdate" - Custom mass update operations
- "workflow" - Workflow actions
- "webstore" - Web store transactions
- "userevent" - User event script triggers
- "mapreduce" - Map/Reduce script operations
- "restlet" - RESTlet API calls
- "webapplication" - Web application interactions
- "restwebservices" - REST web services calls
- |-
## EXAMPLE
```json
["userinterface", "webstore"]
```
default:
- userinterface
- webstore
items:
type: string
enum:
- userinterface
- webservices
- csvimport
- offlineclient
- portlet
- scheduled
- suitelet
- custommassupdate
- workflow
- webstore
- userevent
- mapreduce
- restlet
- webapplication
- restwebservices
x-enumDescriptions:
userinterface: Triggered by user interactions in the NetSuite UI.
webservices: Triggered by SOAP web services API calls.
csvimport: Triggered by CSV import operations.
offlineclient: Triggered by offline client synchronization.
portlet: Triggered by portlet interactions on dashboards.
scheduled: Triggered by scheduled script executions.
suitelet: Triggered by Suitelet script executions.
custommassupdate: Triggered by custom mass update operations.
workflow: Triggered by SuiteFlow workflow actions.
webstore: Triggered by SuiteCommerce web store transactions.
userevent: Triggered by user event script executions.
mapreduce: Triggered by Map/Reduce script operations.
restlet: Triggered by RESTlet API calls.
webapplication: Triggered by web application interactions.
restwebservices: Triggered by REST web services API calls.
disabled:
type: boolean
description: |-
When true, disables the distributed export so record changes no longer trigger it.
When false or omitted, the export remains active.
x-celigo-ai-guidance:
- |-
disabled:
Indicates whether the distributed feature in NetSuite is disabled or not.
This boolean flag controls the availability and operational status of the
distributed functionalities within the NetSuite integration,
allowing administrators or systems to enable or disable these features as needed.
- |-
## FIELD BEHAVIOR:
- When set to true, the distributed feature is fully disabled, preventing any distributed operations or workflows from executing.
- When set to false or omitted, the distributed feature remains enabled and fully operational.
- Acts as a toggle switch to control the accessibility of distributed capabilities within the NetSuite environment.
- Changes to this flag directly influence the behavior of distributed-related processes and integrations.
- |-
## IMPLEMENTATION GUIDANCE:
- Use a boolean value: `true` to disable the distributed feature, `false` to enable it.
- Before disabling, verify that no critical processes depend on distributed functionality to avoid disruptions.
- Implement validation checks to confirm the current state before initiating distributed operations.
- Provide clear user notifications or system logs when the feature is disabled to aid in troubleshooting and auditing.
- Consider the impact on dependent modules and ensure coordinated updates if disabling this feature.
- |-
## EXAMPLES:
- `disabled: true` # The distributed feature is turned off, disabling all related operations.
- `disabled: false` # The distributed feature is active and available for use.
- Omitted `disabled` property defaults to `false`, enabling the feature by default.
- |-
## IMPORTANT NOTES:
- Disabling this feature may interrupt workflows or processes that rely on distributed capabilities, potentially causing failures or delays.
- Some systems may require a restart or reinitialization after changing this setting for the change to take full effect.
- Modifying this property should be restricted to users with appropriate permissions to prevent unauthorized disruptions.
- Always assess the broader impact on the NetSuite integration before toggling this flag.
- |-
## DEPENDENCY CHAIN:
- Directly affects modules and properties that rely on distributed functionality within the NetSuite integration.
- Should be checked and respected by any API calls, workflows, or processes that involve distributed features.
- May influence error handling and fallback mechanisms in distributed-related operations.
- |-
## TECHNICAL DETAILS:
- Data type: Boolean
- Default value: `false` (distributed feature enabled)
- Located under the `netsuite.distributed` namespace in the API schema
- Changing this property triggers state changes in distributed feature availability within the system
executionType:
type: array
description: |-
Record operations that trigger this distributed export. A record event fires the
export only when its operation matches one of the listed types.
x-celigo-ai-guidance:
- |-
Specifies which types of record operations should trigger the export.
When a record operation matches one of the specified types,
the export will be triggered.
- |-
## DEFAULT VALUE
If not specified, defaults to: ["create", "edit", "xedit"]
- |-
## VALID VALUES
- "create" - New record creation
- "edit" - Record editing via UI
- "delete" - Record deletion
- "xedit" - Inline editing (edit without opening the record)
- "copy" - Record copy operation
- "view" - Record view
- "cancel" - Transaction cancellation
- "approve" - Approval action
- "reject" - Rejection action
- "pack" - Pack operation (fulfillment)
- "ship" - Ship operation (fulfillment)
- "markcomplete" - Mark as complete
- "reassign" - Reassignment action
- "editforecast" - Forecast editing
- "dropship" - Drop ship operation
- "specialorder" - Special order operation
- "orderitems" - Order items action
- "paybills" - Pay bills action
- "print" - Print action
- "email" - Email action
- |-
## EXAMPLE
```json
["create", "edit", "xedit"]
```
default:
- create
- edit
- xedit
items:
type: string
enum:
- create
- edit
- delete
- xedit
- copy
- view
- cancel
- approve
- reject
- pack
- ship
- markcomplete
- reassign
- editforecast
- dropship
- specialorder
- orderitems
- paybills
- print
- email
x-enumDescriptions:
create: Triggers when a new record is created.
edit: Triggers when a record is edited via the UI.
delete: Triggers when a record is deleted.
xedit: Triggers when a record is inline-edited without opening it.
copy: Triggers when a record is duplicated.
view: Triggers when a record is viewed.
cancel: Triggers when a transaction is cancelled.
approve: Triggers when a record is approved.
reject: Triggers when a record is rejected.
pack: Triggers when a fulfillment record is packed.
ship: Triggers when a fulfillment record is shipped.
markcomplete: Triggers when a record is marked as complete.
reassign: Triggers when a record is reassigned to another user.
editforecast: Triggers when a forecast is edited.
dropship: Triggers when a drop ship operation is performed.
specialorder: Triggers when a special order operation is performed.
orderitems: Triggers when an order items action is performed.
paybills: Triggers when a pay bills action is performed.
print: Triggers when a print action is performed on a record.
email: Triggers when an email action is performed on a record.
qualifier:
# No object form: the previous 'type: object' matched no stored document.
# The array form is the platform's filter-expression format, e.g.
# [["custbody_field","empty",true],"and",["custbody_other","=","x"]]; the string
# form is a legacy expression; null means no qualifier.
type: [array, string, 'null']
description: |-
Qualification criteria that further restrict which record changes the distributed
export processes. Stored either as a filter-expression array (current format), a
legacy expression string, or null when no qualifier is set.
x-celigo-ai-guidance:
- |-
qualifier:
A string value used to specify a particular qualifier or modifier that further defines,
categorizes,
or scopes the associated data within the NetSuite distributed context.
This property enables more granular identification, filtering,
and processing of data by applying specific criteria or attributes relevant to business logic,
integration workflows, or operational requirements.
It serves as an optional but powerful tool to distinguish data subsets,
enhance data semantics,
and support conditional handling in distributed NetSuite environments.
- |-
## FIELD BEHAVIOR:
- Acts as an additional identifier or modifier to refine the meaning, scope, or classification of the associated data.
- Enables filtering, categorization, or qualification of data entries in distributed NetSuite operations based on specific business rules.
- Typically optional but may be mandatory in certain contexts or API endpoints where precise data segmentation is required.
- Accepts string values that correspond to predefined, standardized, or custom qualifiers recognized by the system or integration layer.
- Supports multiple use cases including regional segmentation, priority tagging, type classification, and channel identification.
- |-
## IMPLEMENTATION GUIDANCE:
- Ensure the qualifier value strictly aligns with the accepted set of qualifiers defined in the business domain, integration specifications, or system configuration.
- Implement validation mechanisms to verify that the qualifier string matches allowed or expected values to avoid errors, misclassification, or unintended behavior.
- Adopt consistent naming conventions and formatting standards (e.g., lowercase, hyphen-separated) for qualifiers to maintain clarity, readability, and interoperability across systems.
- Maintain comprehensive documentation of all custom and standard qualifiers used, including their intended meaning and usage scenarios, to facilitate maintenance, troubleshooting, and future integrations.
- Consider the impact of qualifiers on downstream processing, reporting, and analytics to ensure they are leveraged effectively and do not introduce ambiguity.
- |-
## EXAMPLES:
- "region-us" to specify data related to the United States region.
- "priority-high" to indicate transactions or records with high priority status.
- "type-inventory" to qualify records associated with inventory management.
- "channel-online" to denote sales or operations conducted through online channels.
- "segment-enterprise" to classify data pertaining to enterprise-level customers.
- "status-active" to filter or identify active records within a dataset.
- |-
## IMPORTANT NOTES:
- The qualifier should be meaningful, contextually relevant, and aligned with the business logic to ensure accurate data interpretation.
- Incorrect, inconsistent, or ambiguous qualifiers can lead to data misinterpretation, processing errors, or integration failures.
- The property may interact with other filtering
skipExportFieldId:
type: string
description: |-
ID of the NetSuite field used to flag records that should be skipped during export.
Only affects export output; the data remains unchanged in NetSuite.
x-celigo-ai-guidance:
- |-
skipExportFieldId is an identifier for a specific field within the NetSuite
distributed configuration that determines whether certain data should be
excluded from export processes.
It serves as a control mechanism to selectively omit data associated with
particular fields during export operations,
enabling tailored and efficient data handling.
- |-
## FIELD BEHAVIOR:
- Acts as a flag or marker to skip exporting data associated with the specified field ID.
- When set, the export routines will omit the data linked to this field from being included in the export payload.
- Helps control and customize the export behavior on a per-field basis within distributed NetSuite configurations.
- Does not affect data visibility or storage within NetSuite; it only influences export output.
- Supports multiple uses in scenarios where sensitive, redundant, or irrelevant data should be excluded from exports.
- |-
## IMPLEMENTATION GUIDANCE:
- Ensure the field ID provided corresponds to a valid and existing field within the NetSuite schema to prevent export errors.
- Use this property to optimize export operations by excluding unnecessary or sensitive data fields, improving performance and compliance.
- Validate the field ID format and existence before applying it to avoid runtime issues during export.
- Integrate with export logic to check this property before including fields in the export output, ensuring consistent behavior.
- Consider maintaining a centralized list or configuration of skipExportFieldIds for easier management and auditing.
- |-
## EXAMPLES:
- skipExportFieldId: "custbody_internal_notes" (skips exporting the internal notes custom field)
- skipExportFieldId: "item_custom_field_123" (excludes a specific item custom field from export)
- skipExportFieldId: "custentity_sensitive_data" (prevents export of sensitive customer entity data)
- |-
## IMPORTANT NOTES:
- This property only affects export operations and does not alter data storage or visibility within NetSuite.
- Misconfiguration may lead to incomplete data exports if critical fields are skipped unintentionally, potentially impacting downstream processes.
- Should be used judiciously to maintain data integrity and compliance with business rules and regulatory requirements.
- Changes to this property should be documented and reviewed to avoid unintended data omissions.
- |-
## DEPENDENCY CHAIN
- Depends on the existence and validity of the specified field ID within the NetSuite schema.
- Relies on export routines to check and respect this property during data export processes.
- May interact with other export configuration settings that control data inclusion/exclusion.
hooks:
type: object
properties:
preSend:
type: object
properties:
fileInternalId:
type: string
description: Internal ID of the file in the NetSuite file cabinet that the preSend hook references.
examples: ["12345", "987654", "1001"]
x-celigo-ai-guidance:
- |-
fileInternalId:
The unique internal identifier assigned to a file within the NetSuite system.
This identifier is essential for accurately referencing and manipulating a
specific file during various operations such as retrieval,
update, or deletion within NetSuite's environment.
It acts as a primary key that ensures precise targeting of files in automated workflows,
scripts, and API calls.
- |-
## FIELD BEHAVIOR:
- Serves as a unique and immutable key to identify a file in the NetSuite file cabinet.
- Utilized in pre-send hooks and other automation points to specify the exact file being processed or referenced.
- Must correspond to an existing file's internal ID within the NetSuite account to ensure valid operations.
- Enables consistent and reliable file operations by linking actions directly to the file's system-assigned identifier.
- |-
## IMPLEMENTATION GUIDANCE:
- Always ensure the value is a valid integer that corresponds to an existing file's internal ID in NetSuite.
- Validate the internal ID before performing any file operations to avoid runtime errors or failed transactions.
- Use this ID when invoking NetSuite APIs, SuiteScript, or other integration points to fetch, update, or delete files.
- Avoid hardcoding the internal ID; instead, dynamically retrieve it through queries or API calls to maintain adaptability and reduce maintenance overhead.
- Handle exceptions gracefully when the ID does not correspond to any file, providing meaningful error messages or fallback logic.
- |-
## EXAMPLES:
- 12345
- 987654
- 1001
- |-
## IMPORTANT NOTES:
- The internal ID is system-generated by NetSuite and guaranteed to be unique within the account.
- This ID is distinct from file names, external URLs, or folder identifiers and should not be confused with them.
- Using an incorrect or non-existent internal ID will cause operations to fail, potentially interrupting workflows.
- The internal ID remains constant for the lifetime of the file and does not change even if the file is moved or renamed.
- |-
## DEPENDENCY CHAIN:
- Depends on the file existing in the NetSuite file cabinet prior to referencing.
- Often used alongside related properties such as file name, folder ID, file type, or metadata to provide context or additional filtering.
- May be required input for downstream processes that manipulate or validate file contents.
- |-
## TECHNICAL DETAILS:
- Represented as an integer value assigned by NetSuite upon file creation.
- Immutable once assigned; cannot be altered or reassigned to a different file.
- Used internally by NetSuite APIs, SuiteScript, and integration
function:
type: string
description: |-
Name of the function invoked as the preSend hook. The function can validate
or modify the payload before it is sent.
examples: ["validateCustomerData", "sanitizePayloadBeforeSend", "logPreSendActivity", "customAuthorizationCheck", "enrichOrderDetails", "checkInventoryAvailability"]
x-celigo-ai-guidance:
- '## DEPENDENCY'
- |-
function:
> Specifies the name of the custom function to be executed as a pre-send hook
within the NetSuite distributed system.
This function is invoked immediately before sending data or requests,
allowing for custom processing, validation,
or modification of the payload to ensure data integrity and compliance with business rules.
- |-
## FIELD BEHAVIOR:
- Defines the exact function to be called prior to sending data or requests.
- Enables interception, inspection, and manipulation of data before transmission.
- Supports integration of custom business logic, validation, enrichment, or logging steps.
- Must reference a valid, accessible function within the current execution context or environment.
- The function's execution outcome can influence whether the sending process proceeds, is modified, or is aborted.
- |-
## IMPLEMENTATION GUIDANCE:
- Ensure the function name corresponds exactly to a defined function in the codebase, script environment, or registered hooks.
- The function should accept the expected input parameters (such as the payload or context) and return appropriate results or modifications.
- Implement robust error handling within the function to prevent unhandled exceptions that could disrupt the sending workflow.
- Document the function's purpose, input/output contract, and side effects clearly for maintainability and future reference.
- Validate that the function executes efficiently and completes promptly to avoid introducing latency or blocking the sending process.
- If asynchronous operations are necessary, ensure they are properly awaited or handled to guarantee completion before sending.
- Follow consistent naming conventions aligned with the overall codebase or organizational standards.
- |-
## EXAMPLES:
- "validateCustomerData"
- "sanitizePayloadBeforeSend"
- "logPreSendActivity"
- "customAuthorizationCheck"
- "enrichOrderDetails"
- "checkInventoryAvailability"
- |-
## IMPORTANT NOTES:
- The function must be synchronous or correctly handle asynchronous behavior to ensure it completes before the send operation proceeds.
- If the function throws an error or returns a failure state, it may block, modify, or abort the sending process depending on the implementation.
- Avoid performing long-running or blocking operations within the function to maintain system responsiveness.
- The function should not perform irreversible side effects unless explicitly intended, as it runs prior to data transmission.
- Ensure the function does not introduce security vulnerabilities, such as exposing sensitive data or allowing injection attacks.
- Consistent and clear error reporting within the function aids in troubleshooting and operational monitoring.
configuration:
type: object
description: Settings that control the preSend hook's behavior.
x-celigo-ai-guidance:
- |-
configuration:
> Configuration settings for the preSend hook in the NetSuite distributed system.
This property defines the parameters and options that control the behavior and
execution of the preSend hook,
allowing customization of how data is processed before being sent.
It enables fine-tuning of operational aspects such as retries, timeouts,
validation rules, logging,
and payload constraints to ensure reliable and efficient data transmission.
- |-
## FIELD BEHAVIOR:
- Specifies the customizable settings that dictate how the preSend hook operates.
- Controls data manipulation, validation, and preparation steps prior to sending.
- Can include flags, thresholds, retry policies, timeout durations, logging options, and other operational parameters.
- May be optional or mandatory depending on the specific implementation and requirements of the preSend hook.
- Supports nested configuration objects to allow detailed and structured settings.
- |-
## IMPLEMENTATION GUIDANCE:
- Define clear, well-documented configuration options that directly impact the preSend process.
- Validate all configuration values rigorously to ensure they conform to expected data types, ranges, and formats.
- Provide sensible default values for optional parameters to enhance usability and reduce configuration errors.
- Ensure backward compatibility when extending or modifying configuration options.
- Include comprehensive documentation for each configuration parameter, including its purpose, accepted values, and effect on hook behavior.
- Consider security implications when allowing configuration of headers or other sensitive parameters.
- |-
## EXAMPLES:
- `{ "retryCount": 3, "timeout": 5000, "enableLogging": true }`
- `{ "validateSchema": true, "maxPayloadSize": 1048576 }`
- `{ "customHeaders": { "X-Custom-Header": "value" } }`
- `{ "retryPolicy": { "maxAttempts": 5, "backoffStrategy": "exponential" }, "enableLogging": false }`
- `{ "payloadCompression": "gzip", "timeout": 10000 }`
- |-
## IMPORTANT NOTES:
- Incorrect or invalid configuration values can cause the preSend hook to fail or behave unpredictably.
- Thorough testing of configuration changes in development or staging environments is critical before deploying to production.
- Some configuration changes may require restarting or reinitializing the hook or related services to take effect.
- Sensitive configuration parameters should be handled securely to prevent exposure of confidential information.
- Configuration should be version-controlled and documented to facilitate maintenance
description: |-
Hook invoked immediately before the payload is sent, allowing custom
processing, modification, or validation.
x-celigo-ai-guidance:
- |-
preSend is a hook function that is invoked immediately before a request is sent
to the NetSuite API.
It allows for custom processing, modification,
or validation of the request payload and headers,
enabling dynamic adjustments or logging prior to transmission.
- |-
## FIELD BEHAVIOR:
- Executed synchronously or asynchronously just before the API request is dispatched to the NetSuite endpoint.
- Receives the full request object, including headers, body, query parameters, and other relevant metadata.
- Permits modification of any part of the request, such as altering headers, adjusting the payload, or changing query parameters.
- Supports validation logic to ensure the request meets required criteria; throwing an error will abort the request.
- Enables injection of dynamic data like authentication tokens, custom headers, or correlation IDs.
- Can be used for logging or auditing outgoing request details for debugging or monitoring purposes.
- |-
## IMPLEMENTATION GUIDANCE:
- Implement as a function or asynchronous callback that accepts the request context object.
- Ensure that any asynchronous operations within the hook are properly awaited to maintain request integrity.
- Keep processing lightweight to avoid introducing latency or blocking the request pipeline.
- Handle exceptions carefully; unhandled errors will prevent the request from being sent.
- Centralize request customization logic here to improve maintainability and reduce duplication.
- Avoid side effects that could impact other parts of the system or subsequent requests.
- Validate inputs thoroughly to prevent malformed requests from being sent to the API.
- |-
## EXAMPLES:
- Adding a Bearer token or API key to the Authorization header dynamically before sending.
- Logging the complete request payload and headers for troubleshooting network issues.
- Modifying request parameters based on user roles or feature flags at runtime.
- Validating that required fields are present and correctly formatted, throwing an error if validation fails.
- Adding a unique request ID header for tracing requests across distributed systems.
- |-
## IMPORTANT NOTES:
- This hook executes on every outgoing request, so its performance impact should be minimized.
- Any modifications made within preSend directly affect the final request sent to NetSuite.
- Throwing an error inside this hook will abort the request and propagate the error upstream.
- This hook is strictly for pre-request processing and should not be used for handling responses.
- Avoid making network calls or heavy computations inside this hook to prevent delays.
- Ensure thread safety if the hook accesses shared resources or global state.
- |-
## DEPENDENCY CHAIN:
- Invoked after request construction but before the request is dispatched.
- Precedes any network transmission or retry
description: Custom hook functions executed at defined points in the distributed export lifecycle.
x-celigo-ai-guidance:
- |-
hooks:
> A collection of user-defined functions or callbacks that are executed at
specific points during the lifecycle of the distributed process within the
NetSuite integration.
These hooks enable customization and extension of the default behavior by
injecting custom logic before,
during, or after key operations,
allowing for flexible adaptation to unique business requirements and integration scenarios.
- |-
## FIELD BEHAVIOR:
- Contains one or more functions or callback references mapped to specific lifecycle events.
- Each hook corresponds to a distinct event or stage in the distributed process, such as pre-processing, post-processing, error handling, or data transformation.
- Hooks are invoked automatically by the system at predefined points in the workflow.
- Can modify data payloads, trigger additional workflows or external API calls, perform validations, or handle errors.
- Supports both synchronous and asynchronous execution models depending on the hook's purpose and implementation.
- Execution order of hooks for the same event is deterministic and should be documented.
- Hooks should be designed to avoid side effects that could impact other parts of the process.
- |-
## IMPLEMENTATION GUIDANCE:
- Define hooks as named functions or references to executable code blocks compatible with the integration environment.
- Ensure hooks are idempotent to prevent unintended consequences from repeated or retried executions.
- Validate all inputs and outputs rigorously within hooks to maintain data integrity and system stability.
- Use hooks to integrate with external systems, perform custom validations, enrich data, or implement business-specific logic.
- Document each hook's purpose, expected inputs, outputs, and any side effects clearly for maintainability.
- Implement robust error handling within hooks to gracefully manage exceptions without disrupting the main process flow.
- Test hooks thoroughly in isolated and integrated environments to ensure reliability and performance.
- Consider security implications, such as data exposure or injection risks, when implementing hooks.
- |-
## EXAMPLES:
- A hook that validates transaction data before it is sent to NetSuite to ensure compliance with business rules.
- A hook that logs detailed transaction metadata after a successful operation for auditing purposes.
- A hook that modifies or enriches payload data during transformation stages to align with NetSuite's schema.
- A hook that triggers email or system notifications upon error occurrences to alert support teams.
- A hook that retries failed operations with exponential backoff to improve resilience.
- |-
## IMPORTANT NOTES:
- Improperly implemented hooks can cause process failures, data inconsistencies, or performance degradation.
sublists:
type: [array, 'null']
items:
type: string
description: |-
IDs of the NetSuite sublists to include with each exported record (for
example `item` for transaction line items). Records are exported with
body fields only when no sublists are selected. Legacy documents may
store null (equivalent to no sublists).
examples: [["item"]]
referencedFields:
type: object
description: Field identifiers referenced by the distributed export.
x-celigo-ai-guidance:
- |-
referencedFields:
> A list of field identifiers that are referenced within the current context,
typically used to denote dependencies or relationships between fields in a
NetSuite distributed environment.
This property helps in mapping out how different fields interact or rely on each other,
facilitating data integrity, validation,
and synchronization across distributed components or services.
- |-
## FIELD BEHAVIOR:
- Contains identifiers of fields that the current field or process depends on or interacts with.
- Used to establish explicit relationships or dependencies between multiple fields.
- Enables tracking of data flow and ensures consistency across distributed systems.
- Supports dynamic resolution of dependencies during runtime or configuration.
- |-
## IMPLEMENTATION GUIDANCE:
- Populate with valid and existing field identifiers as defined in the NetSuite schema or metadata.
- Verify that all referenced fields are accessible and correctly scoped within the current context.
- Use this property to manage dependencies critical for data validation, synchronization, or processing logic.
- Keep the list updated to reflect any schema changes to avoid broken references or inconsistencies.
- Avoid circular references by carefully managing dependencies between fields.
- |-
## EXAMPLES:
- ["customerId", "orderDate", "shippingAddress"]
- ["invoiceNumber", "paymentStatus"]
- ["productCode", "inventoryLevel", "reorderThreshold"]
- |-
## IMPORTANT NOTES:
- Referenced fields must be unique within the list to prevent redundancy and confusion.
- Modifications to referenced fields can impact dependent processes; changes should be tested thoroughly.
- This property contains only the identifiers (names or keys) of fields, not their actual data or values.
- Proper documentation of referenced fields improves maintainability and clarity of dependencies.
- |-
## DEPENDENCY CHAIN:
- Often linked with fields that require validation or data aggregation from other fields.
- May influence or be influenced by business rules, workflows, or automation scripts that depend on multiple fields.
- Changes in referenced fields can cascade to affect dependent fields or processes.
- |-
## TECHNICAL DETAILS:
- Data type: Array of strings.
- Each string represents a unique field identifier within the NetSuite distributed environment.
- The array should be serialized in a format compatible with the consuming system (e.g., JSON array).
- Maximum length and allowed characters for field identifiers should conform to NetSuite naming conventions.
relatedLists:
type: object
description: Related record lists associated with the exported NetSuite record.
x-celigo-ai-guidance:
- |-
relatedLists:
A collection of related list objects that represent associated records or
entities linked to the primary record within the NetSuite distributed data
model.
These related lists provide contextual information and enable navigation to connected data,
facilitating comprehensive data retrieval and management.
Each related list encapsulates a set of records that share a defined
relationship with the primary record,
such as transactions, contacts, or custom entities,
thereby supporting a holistic view of the data ecosystem.
- |-
## FIELD BEHAVIOR:
- Contains multiple related list entries, each representing a distinct association to the primary record.
- Enables retrieval of linked records such as transactions, custom records, subsidiary data, or other relevant entities.
- Supports hierarchical or relational data structures by referencing related entities, allowing nested or multi-level associations.
- Typically read-only in the context of distributed data retrieval but may support updates or synchronization depending on API capabilities and permissions.
- May include metadata such as record counts, last updated timestamps, or status indicators for each related list.
- Supports dynamic inclusion or exclusion based on user permissions, record type, and system configuration.
- |-
## IMPLEMENTATION GUIDANCE:
- Populate with relevant related list objects that are directly associated with the primary record, ensuring accurate representation of relationships.
- Ensure each related list entry includes unique identifiers, descriptive metadata, and navigation links or references necessary for data access and traversal.
- Maintain consistency in naming conventions, data structures, and field formats to align with NetSuite's standard data model and API specifications.
- Implement pagination, filtering, or sorting mechanisms to efficiently handle large sets of related records within each list.
- Validate all references and links to ensure data integrity, preventing broken or stale connections within the distributed data environment.
- Consider caching strategies or incremental updates to optimize performance when dealing with frequently accessed related lists.
- Respect and enforce access control and permission checks to ensure users only see related lists they are authorized to access.
- |-
## EXAMPLES:
- A customer record's relatedLists might include "Transactions" (e.g., sales orders, invoices), "Contacts" (associated individuals), and "Cases" (customer support tickets).
- An invoice record's relatedLists could contain "Payments" (payment records), "Shipments" (delivery details), and "Adjustments" (billing corrections).
- A custom record type might have relatedLists such as "Attachments" (files linked to the record) or "Notes" (user comments or annotations).
- A vendor record's relatedLists may include "Purchase Orders," "Bills," and "Vendor Contacts
forceReload:
type: boolean
description: When true, bypasses cached data and reloads directly from the source.
x-celigo-ai-guidance:
- |-
## IMPLEMENTATION GUIDANCE:
- Use this flag judiciously to balance between data freshness and system performance, avoiding unnecessary reloads that could degrade responsiveness.
- Ensure that enabling forceReload initiates a comprehensive refresh cycle, including clearing relevant caches and reinitializing configuration or data layers.
- Implement robust error handling during the reload process to manage potential failures without causing system downtime or inconsistent states.
- Monitor system resource utilization and response times when forceReload is active to identify and mitigate performance bottlenecks.
- Document scenarios and triggers for using forceReload to guide developers and operators in its appropriate application.
- |-
## EXAMPLES:
- forceReload: true
(forces the system to bypass caches and reload data/configuration from the authoritative source immediately)
- forceReload: false
(allows the system to serve data from cache if available, improving response time)
- forceReload omitted
(defaults to false behavior, relying on cached data unless otherwise specified)
- |-
## FIELD BEHAVIOR:
- When set to true, the system bypasses all caches and reloads data or configurations directly from the primary source, ensuring the latest state is retrieved.
- When set to false or omitted, the system may utilize cached or previously stored data to optimize performance and reduce load times.
- Primarily used in contexts where stale data could lead to errors, inconsistencies, or outdated processing results.
- The reload operation triggered by this flag typically involves invalidating caches and refreshing dependent components or services.
- |-
## IMPORTANT NOTES:
- Excessive or unnecessary use of forceReload can lead to increased latency, higher resource consumption, and potential service degradation.
- This flag does not validate the correctness or integrity of the source data; it only ensures the latest available data is fetched.
- Downstream systems or processes should be designed to handle the potential delays or transient states caused by forced reloads.
- Coordination with cache invalidation policies and data synchronization mechanisms is essential to maintain overall system consistency.
- |-
## DEPENDENCY CHAIN:
- Relies on underlying cache management and invalidation frameworks to effectively bypass stored data.
- Interacts with data retrieval modules, configuration loaders, and possibly distributed synchronization services.
- |-
forceReload:
Indicates whether the system should forcibly reload the data or configuration,
bypassing any cached or stored versions to ensure the most up-to-date information is used.
This flag is critical in scenarios where data accuracy and freshness are paramount,
such as after configuration changes or data updates that must be immediately reflected.
ioEnvironment:
type: string
description: integrator.io environment the distributed NetSuite bundle communicates with. Live-observed values include `production` and `staging`.
examples: ["development", "staging", "production", "custom"]
x-celigo-ai-guidance:
- |-
ioEnvironment specifies the input/output environment configuration for the
NetSuite distributed system,
defining how data is handled, processed,
and routed across different operational environments.
This property determines the context in which I/O operations occur,
influencing data flow, security protocols, performance characteristics,
and consistency guarantees within the distributed architecture.
- |-
## FIELD BEHAVIOR:
- Determines the operational context for all input/output processes within the distributed NetSuite system.
- Influences how data is read from and written to various storage systems, message queues, or communication channels.
- Affects performance tuning, security measures, and data consistency mechanisms based on the selected environment.
- Typically set during system initialization or configuration phases and remains stable during runtime to ensure predictable behavior.
- May trigger environment-specific logging, monitoring, and error-handling strategies.
- |-
## IMPLEMENTATION GUIDANCE:
- Validate the ioEnvironment value against a predefined set of supported environments such as "development," "staging," "production," and any custom configurations.
- Ensure that the selected environment is compatible with other system settings related to data handling, network communication, and security policies.
- Implement robust error handling and fallback mechanisms to manage unsupported or invalid environment values gracefully.
- Clearly document the operational implications, limitations, and recommended use cases for each environment option to guide system administrators and developers.
- Coordinate environment settings across all distributed nodes to maintain consistency and prevent configuration drift.
- |-
## EXAMPLES:
- "development" — used for local testing and debugging with relaxed security and simplified data handling.
- "staging" — a pre-production environment that closely mirrors production settings for validation and testing.
- "production" — the live environment optimized for security, performance, and data integrity.
- "custom" — user-defined environment configurations tailored for specialized I/O requirements or experimental setups.
- |-
## IMPORTANT NOTES:
- Changing the ioEnvironment typically requires restarting services or reinitializing connections to apply new configurations.
- The environment setting directly impacts data integrity, access controls, and compliance with security policies.
- Sensitive data must be handled according to the security standards appropriate for the selected environment.
- Consistency across all distributed nodes is critical; all nodes should be configured with compatible ioEnvironment values to avoid data inconsistencies or communication failures.
- Misconfiguration can lead to degraded performance, security vulnerabilities, or data loss.
- |-
## DEPENDENCY CHAIN:
- Depends on system initialization and configuration management components.
- Interacts with data storage modules, network communication layers, and security frameworks.
- Influences logging, monitoring, and error
ioDomain:
type: string
description: |-
integrator.io domain the distributed NetSuite bundle calls back to (e.g.
`integrator.io`). Live-observed on distributed exports.
examples: ["api.netsuite.com", "distributed-services.companydomain.com", "staging-netsuite.io.company.com", "eu-west-1.api.netsuite.com", "dev-networks.internal.company.com"]
x-celigo-ai-guidance:
- |-
ioDomain specifies the Internet domain name used for input/output operations
within the distributed NetSuite environment.
This domain is critical for routing data requests and responses between
distributed components and services,
ensuring seamless communication and integration across the system.
- |-
## FIELD BEHAVIOR:
- Defines the domain name utilized for network communication in distributed NetSuite environments.
- Serves as the base domain for constructing URLs for API calls, data synchronization, and service endpoints.
- Must be a valid, fully qualified domain name (FQDN) adhering to DNS standards.
- Typically remains consistent within a deployment environment but can differ across environments such as development, staging, and production.
- Influences routing, load balancing, and failover mechanisms within distributed services.
- |-
## IMPLEMENTATION GUIDANCE:
- Verify that the domain is properly configured in DNS and is resolvable by all distributed components.
- Validate the domain format against standard domain naming conventions (e.g., RFC 1035).
- Ensure the domain supports secure communication protocols (e.g., HTTPS with valid SSL/TLS certificates).
- Coordinate updates to ioDomain with network, security, and operations teams to maintain service continuity.
- When migrating or scaling services, update ioDomain accordingly and propagate changes to all dependent components.
- Monitor domain accessibility and performance to detect and resolve connectivity issues promptly.
- |-
## EXAMPLES:
- "api.netsuite.com"
- "distributed-services.companydomain.com"
- "staging-netsuite.io.company.com"
- "eu-west-1.api.netsuite.com"
- "dev-networks.internal.company.com"
- |-
## IMPORTANT NOTES:
- Incorrect or misconfigured ioDomain values can cause failed network requests, service interruptions, and data synchronization errors.
- The domain must support necessary security certificates to enable encrypted communication and protect data in transit.
- Changes to ioDomain may necessitate updates to firewall rules, proxy configurations, and network security policies.
- Consistency in ioDomain usage across distributed components is essential to avoid routing conflicts and authentication issues.
- Consider the impact on caching, CDN configurations, and DNS propagation delays when changing ioDomain.
- |-
## DEPENDENCY CHAIN:
- Dependent on underlying network infrastructure, DNS setup, and domain registration.
- Utilized by distributed service components for constructing communication endpoints.
- May affect authentication and authorization workflows that rely on domain validation or origin verification.
- Interacts with security components such as SSL/TLS certificate management and firewall configurations.
- Influences monitoring, logging, and troubleshooting processes related to network communication.
##
lastSyncedDate:
type: string
format: date-time
description: |-
Timestamp when the distributed export last completed a successful synchronization.
Not set until the first successful sync.
examples: ["2024-06-15T14:30:00Z", "2023-12-01T08:45:22Z", "2024-01-10T23:59:59Z"]
x-celigo-ai-guidance:
- |-
lastSyncedDate represents the precise date and time when the data was last
successfully synchronized between the system and NetSuite.
This timestamp is essential for monitoring the freshness, consistency,
and integrity of synchronized data,
enabling systems to determine whether updates or incremental syncs are necessary.
- |-
## FIELD BEHAVIOR:
- Captures the exact date and time of the most recent successful synchronization event.
- Automatically updates only after a sync operation completes successfully without errors.
- Serves as a reference point to assess if data is current or requires refreshing.
- Typically stored and transmitted in ISO 8601 format to maintain uniformity across different systems and platforms.
- Does not reflect the start time or duration of the synchronization process, only its successful completion.
- |-
## IMPLEMENTATION GUIDANCE:
- Record the timestamp in Coordinated Universal Time (UTC) to prevent timezone-related inconsistencies.
- Update this field exclusively after confirming a successful synchronization to avoid misleading data states.
- Validate the date and time format rigorously to comply with ISO 8601 standards (e.g., "YYYY-MM-DDTHH:mm:ssZ").
- Utilize this timestamp to drive incremental synchronization logic, data refresh triggers, or audit trails in downstream workflows.
- Handle cases where the field may be null or missing, indicating that no synchronization has occurred yet.
- |-
## EXAMPLES:
- "2024-06-15T14:30:00Z"
- "2023-12-01T08:45:22Z"
- "2024-01-10T23:59:59Z"
- |-
## IMPORTANT NOTES:
- This timestamp marks the completion of synchronization, not its initiation.
- Do not update this field if the synchronization process fails or is incomplete.
- Maintaining timezone consistency (UTC) is critical to avoid synchronization conflicts or data mismatches.
- The field may be null or omitted if synchronization has never been performed.
- Systems relying on this field should implement fallback or error handling for missing or invalid timestamps.
- |-
## DEPENDENCY CHAIN:
- Depends on successful completion of the synchronization process between the system and NetSuite.
- Influences downstream processes such as incremental sync triggers, data validation, and audit logging.
- May be referenced by monitoring or alerting systems to detect synchronization delays or failures.
- |-
## TECHNICAL DETAILS:
- Stored as a string in ISO 8601 format with UTC timezone designator (e.g., "YYYY-MM-DDTHH:mm:ssZ").
- Should be generated programmatically at the moment synchronization completes successfully
settings:
type: object
description: Additional key-value configuration settings for the distributed export.
x-celigo-ai-guidance:
- |-
settings:
> Configuration settings specific to the distributed module within the NetSuite integration,
enabling fine-grained control over distributed processing behavior and performance optimization.
- |-
## FIELD BEHAVIOR:
- Encapsulates a collection of key-value pairs representing various configuration parameters that govern the distributed NetSuite integration's operation.
- Includes toggles (boolean flags), numeric thresholds, timeouts, batch sizes, logging options, and other customizable settings relevant to distributed processing workflows.
- Typically optional for basic usage but essential for advanced customization, performance tuning, and adapting the integration to specific deployment environments.
- Changes to these settings can dynamically alter the integration's behavior, such as retry logic, concurrency limits, and error handling strategies.
- |-
## IMPLEMENTATION GUIDANCE:
- Define each setting with a clear, descriptive key name and an appropriate data type (e.g., integer, boolean, string).
- Validate input values rigorously to ensure they fall within acceptable ranges or conform to expected formats to prevent runtime errors.
- Provide sensible default values for all settings to maintain stable and predictable integration behavior when explicit configuration is absent.
- Document each setting comprehensively, including its purpose, valid values, default, and impact on the integration's operation.
- Consider versioning or schema validation to manage changes in settings structure over time.
- Ensure that sensitive information is either excluded or securely handled if included within settings.
- |-
## EXAMPLES:
- `{ "retryCount": 3, "enableLogging": true, "timeoutSeconds": 120 }` — configures retry attempts, enables detailed logging, and sets operation timeout.
- `{ "batchSize": 50, "useSandbox": false }` — sets the number of records processed per batch and specifies production environment usage.
- `{ "maxConcurrentJobs": 10, "errorThreshold": 5, "logLevel": "DEBUG" }` — limits concurrent jobs, sets error tolerance, and defines logging verbosity.
- |-
## IMPORTANT NOTES:
- Modifications to settings may require restarting or reinitializing the integration service to apply changes effectively.
- Incorrect or suboptimal configuration can cause integration failures, data inconsistencies, or degraded performance.
- Avoid storing sensitive credentials or secrets in settings unless encrypted or otherwise secured.
- Settings should be managed carefully in multi-environment deployments to prevent configuration drift.
useSS2Framework:
type: boolean
description: |-
When true, the distributed export runs on the SuiteScript 2.0 framework. When
false or omitted, the legacy SuiteScript 1.0 framework is used.
Set at step creation together with frameworkVersion — the same
creation-time-only constraint applies (see frameworkVersion).
x-celigo-ai-guidance:
- |-
useSS2Framework indicates whether to utilize the SuiteScript 2.0 framework for
the NetSuite distributed configuration,
enabling modern scripting capabilities and modular architecture within the NetSuite environment.
- |-
## FIELD BEHAVIOR:
- Determines if the SuiteScript 2.0 (SS2) framework is enabled for the NetSuite integration.
- When set to true, the system uses SS2 APIs, modular script definitions, and updated scripting conventions.
- When set to false or omitted, the system defaults to using SuiteScript 1.0 or legacy frameworks.
- Influences script loading mechanisms, module resolution, and API compatibility within NetSuite.
- |-
## IMPLEMENTATION GUIDANCE:
- This is an authoring-time choice: set it when creating the listener step (true unless the user explicitly needs a legacy version). It cannot be toggled on a saved step — moving an existing listener to a different framework version means recreating or cloning it.
- Verify that the integrator.io SuiteApp is installed in the target NetSuite environment when authoring SuiteScript 2.0 steps.
- When migrating by recreation, verify any SuiteScript hooks are written for the matching version before cutting over.
- |-
## EXAMPLES:
- `useSS2Framework: true` — Enables SuiteScript 2.0 framework usage, activating modern scripting features.
- `useSS2Framework: false` — Disables SuiteScript 2.0, falling back to legacy SuiteScript 1.0 framework.
- Property omitted — Defaults to legacy SuiteScript framework (typically 1.0), maintaining backward compatibility.
- |-
## IMPORTANT NOTES:
- Enabling the SS2 framework may require refactoring existing scripts to comply with SuiteScript 2.0 syntax, including the use of define/require for module loading.
- Some legacy APIs, global objects, and modules available in SuiteScript 1.0 may be deprecated or behave differently in SuiteScript 2.0.
- Performance improvements and new features in SS2 may not be realized if scripts are not properly adapted.
- Ensure that all scheduled scripts, workflows, and integrations are reviewed for compatibility to prevent runtime errors.
- Documentation and developer training may be necessary to fully leverage SuiteScript 2.0 capabilities.
frameworkVersion:
type: string
enum:
- suitebundle
- suiteapp1.0
- suiteapp2.0
x-enumDescriptions:
suitebundle: Legacy SuiteBundle framework; slated for deprecation — NetSuite no longer updates SuiteScript 1.0.
suiteapp1.0: SuiteApp 1.0 framework using SuiteScript 1.0.
suiteapp2.0: SuiteApp 2.0 framework using SuiteScript 2.0, the modern default.
description: |-
SuiteApp framework version used by the distributed export.
Fixed when the step is created — the Advanced selector is disabled on
existing steps, and migrating an existing listener to a different
version means recreating or cloning it.
x-celigo-ai-guidance:
- |-
The NetSuite API version cannot be changed on an existing step — the
UI disables the selector once the step is saved. When asked to
migrate an existing real-time listener off the SuiteBundle (or
between versions), do not update this field in place; the migration
path is recreating or cloning the step with the new version.
- |-
SuiteScript hooks are a separate setting from the step's API
version: hook code must be written for the same SuiteScript version
as the step (1.0 and 2.x are not interchangeable), and migrating a
hook is its own task — flag it whenever migrating a step that
carries hooks.
- |-
Author new steps with suiteapp2.0 unless the user explicitly
requires a legacy version — NetSuite no longer updates SuiteScript
1.0 and Celigo plans to deprecate the SuiteBundle option.
description: |-
Configuration for `distributed` exports, which use the NetSuite SuiteApp to fire
real-time, event-driven exports when records change. Define the record type to listen
to and the execution contexts and operations that trigger it.
getList:
# Array, not object; stored occurrences are empty — the item shape below documents
# the getList record-reference semantics but is not observed populated.
type: array
description: |-
Configuration for `getList` exports — a list of record references, each retrieving one
NetSuite record by its identifier. The export returns the fetched records in list order.
x-celigo-ai-guidance:
- |-
Send getList as a bare array of record references, exactly as GET returns it. Never
wrap it in an object (e.g. `"getList": {"type": [...]}`): the adaptor requires an
array, and an export saved with the wrapper shape fails at run time with
"requires getList prop".
- |-
Reference standard records with `type` plus `internalId` or `externalId`. Reference
custom records with `typeId` instead of `type`; a `typeId` starting with
"customtransaction" targets a custom transaction type. When both identifiers are
present, `internalId` wins and `externalId` is ignored.
items:
type: object
properties:
type:
type: string
description: |-
Standard NetSuite record type to retrieve (e.g. "customer", "salesOrder").
Used when `typeId` is absent. Case-sensitive.
typeId:
type: string
description: |-
Script ID of a custom record type ("customrecord_...") or custom transaction
type ("customtransaction_..."). Takes precedence over `type` when present.
Case-sensitive.
internalId:
type: string
description: |-
Internal ID of the NetSuite record to retrieve. Assigned by NetSuite at record
creation and unique within the record type. Takes precedence over `externalId`.
externalId:
type: string
description: |-
Identifier assigned by an external system, used to reference and synchronize the
record across systems. Distinct from the NetSuite-assigned internal ID.
searchPreferences:
type: object
properties:
bodyFieldsOnly:
type: boolean
description: |-
When true, search results include only the record's body fields and exclude fields
from joined or related records. Reduces payload size when related data isn't needed.
x-celigo-ai-guidance:
- |-
bodyFieldsOnly indicates whether the search results should include only the body
fields of the records,
excluding any joined or related record fields.
This setting controls the scope of data returned by the search operation,
allowing for more focused and efficient retrieval when only the main record's
fields are necessary.
- |-
## FIELD BEHAVIOR:
- When set to true, search results will include only the fields that belong directly to the main record (body fields), excluding any fields from joined or related records.
- When set to false or omitted, search results may include fields from both the main record and any joined or related records specified in the search.
- Directly affects the volume and detail of data returned, potentially reducing payload size and improving performance.
- Influences how the search engine processes and compiles the result set, limiting it to primary record data when enabled.
- |-
## IMPLEMENTATION GUIDANCE:
- Use this property to optimize search performance and reduce data transfer when only the main record's fields are required.
- Set to true to minimize payload size, which is beneficial for large datasets or bandwidth-sensitive environments.
- Verify that your search criteria and downstream processing do not require any joined or related record fields before enabling this option.
- If joined fields are necessary for your application logic, keep this property false or unset to ensure complete data retrieval.
- Consider this setting in conjunction with other search preferences like pageSize and returnSearchColumns for optimal results.
- |-
## EXAMPLES:
- `bodyFieldsOnly: true` — returns only the main record's body fields in search results, excluding any joined record fields.
- `bodyFieldsOnly: false` — returns both body fields and fields from joined or related records as specified in the search.
- Omitted `bodyFieldsOnly` property — defaults to false behavior, including joined fields if requested.
- |-
## IMPORTANT NOTES:
- Enabling bodyFieldsOnly may omit critical related data if your search logic depends on joined fields, potentially impacting application functionality.
- This setting is particularly useful for improving performance and reducing data size in scenarios where joined data is unnecessary.
- Not all record types or search operations may support this preference; verify compatibility with your specific use case.
- Changes to this setting can affect the structure and completeness of search results, so test thoroughly when modifying.
- |-
## DEPENDENCY CHAIN:
- This property is part of the `searchPreferences` object within the NetSuite API request.
- It influences the fields returned by the search operation, affecting both data scope and payload size.
- May
pageSize:
type: number
description: Number of search results NetSuite returns per page.
x-celigo-ai-guidance:
- |-
pageSize specifies the number of search results to be returned per page in a
paginated search response.
This property controls the size of each page of results when performing searches,
enabling efficient handling and retrieval of large datasets by dividing them
into manageable chunks.
By adjusting pageSize,
clients can balance between the volume of data received per request and the
performance implications of processing large result sets.
- |-
## FIELD BEHAVIOR:
- Determines the maximum number of records returned in a single page of search results.
- Directly influences the pagination mechanism by setting how many items appear on each page.
- Helps optimize network and client performance by limiting the amount of data transferred and processed per response.
- Affects the total number of pages available, calculated based on the total number of search results divided by pageSize.
- When pageSize is changed between requests, it may affect the consistency of pagination navigation.
- |-
## IMPLEMENTATION GUIDANCE:
- Assign pageSize a positive integer value that balances response payload size and system performance.
- Ensure the value respects any minimum and maximum limits imposed by the API or backend system.
- Maintain consistent pageSize values across paginated requests to provide predictable and stable navigation through result pages.
- Consider client device capabilities, network bandwidth, and expected user interaction patterns when selecting pageSize.
- Implement validation to prevent invalid or out-of-range values that could cause errors or degraded performance.
- When dealing with very large datasets, consider smaller pageSize values to reduce memory consumption and improve responsiveness.
- |-
## EXAMPLES:
- pageSize: 25 — returns 25 search results per page, suitable for standard list views.
- pageSize: 100 — returns 100 search results per page, useful for bulk data processing or export scenarios.
- pageSize: 10 — returns 10 search results per page, ideal for quick previews or limited bandwidth environments.
- pageSize: 50 — a moderate setting balancing data volume and performance for typical use cases.
- |-
## IMPORTANT NOTES:
- Excessively large pageSize values can increase response times, memory usage, and may lead to timeouts or throttling.
- Very small pageSize values can cause a high number of API calls, increasing overall latency and server load.
- The API may enforce maximum allowable pageSize limits; requests exceeding these limits may result in errors or automatic truncation.
- Changing pageSize mid-pagination can disrupt user experience by altering the number of pages and item offsets.
- Some APIs may have default pageSize values if none is specified; explicitly setting page
returnSearchColumns:
type: boolean
description: |-
When true, search results include the column data defined in the search. When
false, column data is omitted and results contain only minimal record information.
x-celigo-ai-guidance:
- |-
returnSearchColumns:
Specifies whether the search operation should return the columns (fields)
defined in the search results,
providing detailed data for each record matching the search criteria.
- |-
## FIELD BEHAVIOR:
- Determines if the search response includes the columns specified in the search definition, such as field values and metadata.
- When set to true, the search results will contain detailed column data for each record, enabling comprehensive data retrieval.
- When set to false, the search results will omit column data, potentially returning only record identifiers or minimal information.
- Directly influences the amount of data returned, impacting response payload size and processing time.
- Affects how client applications can utilize the search results, depending on the presence or absence of column data.
- |-
## IMPLEMENTATION GUIDANCE:
- Set to true when detailed search result data is required for processing, reporting, or display purposes.
- Set to false to optimize performance and reduce bandwidth usage when only record IDs or minimal data are needed.
- Use in conjunction with other search preference settings (e.g., `pageSize`, `returnSearchRows`) to fine-tune search responses.
- Ensure client applications are designed to handle both scenarios—presence or absence of column data—to avoid errors or incomplete processing.
- Consider the trade-off between data completeness and performance when configuring this property.
- |-
## EXAMPLES:
- `returnSearchColumns: true` — The search results will include all defined columns for each record, such as names, dates, and custom fields.
- `returnSearchColumns: false` — The search results will exclude column data, returning only basic record information like internal IDs.
- |-
## IMPORTANT NOTES:
- Enabling returnSearchColumns may significantly increase response size and processing time, especially for searches returning many records or columns.
- Some search operations or API endpoints may require columns to be returned to function correctly or to provide meaningful results.
- Disabling this option can improve performance but limits the detail available in search results, which may affect downstream processing or user interfaces.
- Changes to this setting can impact caching, pagination, and sorting behaviors depending on the search implementation.
- |-
## DEPENDENCY CHAIN:
- Related to other `searchPreferences` properties such as `pageSize` (controls number of records per page) and `returnSearchRows` (controls whether search rows are returned).
- Works in tandem with search definition settings that specify which columns are included in the search.
- May affect or be affected by API-level configurations or limitations on data retrieval and response formatting.
description: Preferences that control how NetSuite executes searches and shapes the results it returns.
x-celigo-ai-guidance:
- |-
searchPreferences:
Preferences that control the behavior and parameters of search operations within
the NetSuite environment,
enabling customization of how search queries are executed and how results are
returned to optimize relevance,
performance, and user experience.
- |-
## FIELD BEHAVIOR:
- Defines the execution parameters for search queries, including pagination, sorting, filtering, and result formatting.
- Controls the scope, depth, and granularity of data retrieved during search operations.
- Influences the performance, accuracy, and relevance of search results based on configured preferences.
- Can be adjusted dynamically to tailor search behavior to specific user roles, contexts, or application requirements.
- May include settings such as page size limits, sorting criteria, case sensitivity, and filter application.
- |-
## IMPLEMENTATION GUIDANCE:
- Utilize this property to fine-tune search operations to meet specific user or application needs, improving efficiency and relevance.
- Validate all preference values against supported NetSuite search parameters to prevent errors or unexpected behavior.
- Establish sensible default preferences to ensure consistent and predictable search results when explicit preferences are not provided.
- Allow dynamic updates to preferences to adapt to changing contexts, such as different user roles or data volumes.
- Ensure that preference configurations comply with user permissions and role-based access controls to maintain security and data integrity.
- |-
## EXAMPLES:
- Setting a page size of 50 to limit the number of records returned per search query for better performance.
- Enabling case-insensitive search filters to broaden result matching.
- Specifying sorting order by transaction date in descending order to show the most recent records first.
- Applying filters to restrict search results to a particular customer segment or date range.
- Configuring search to exclude inactive records to streamline results.
- |-
## IMPORTANT NOTES:
- Misconfiguration of searchPreferences can lead to incomplete, irrelevant, or inefficient search results, negatively impacting user experience.
- Certain preferences may be restricted or overridden based on user roles, permissions, or API version constraints.
- Changes to searchPreferences can affect system performance; excessive page sizes or complex filters may increase load times.
- Always verify compatibility of preference settings with the specific NetSuite API version and environment in use.
- Consider the impact of preferences on downstream processes that consume search results.
- |-
## DEPENDENCY CHAIN:
- Depends on the overall search operation configuration and the specific search type being performed.
- Interacts with user authentication and authorization settings to enforce access controls on search results.
- Influences and is influenced by data retrieval mechanisms and indexing strategies within NetSuite.
file:
type: object
description: |-
Configuration for file exports, which retrieve files from the NetSuite file cabinet
and parse their contents (CSV, XML, JSON) into records. Leave the export's top-level
`type` unset when using this object; for raw transfers without parsing, use
`netsuite.blob` with the export's `type` set to `blob` instead.
x-celigo-ai-guidance:
- |-
Configuration for retrieving files from NetSuite file cabinet and PARSING them into records.
Use this for structured file exports (CSV, XML,
JSON) where the file content should be parsed into data records.
- |-
## CRITICAL: When to use file vs blob
- Use `netsuite.file` WITH export `type: null/undefined` for file exports WITH parsing (CSV, XML, JSON)
- Use `netsuite.blob` WITH export `type: "blob"` for raw binary transfers WITHOUT parsing
When you want file content to be parsed into individual records, use this `file` configuration and leave the export's `type` field as null or undefined (standard export). Do NOT set `type: "blob"` when using this configuration.
properties:
folderInternalId:
type: string
description: |-
Internal ID of the NetSuite file cabinet folder to export files from. Accepts a
handlebars expression (e.g. `{{record.folderId}}`) when the folder must be
selected dynamically from record data. The ID is stable across folder renames and
moves, but may differ between non-production and production accounts.
examples: ["12345", "67890", "{{record.folderId}}"]
x-celigo-ai-guidance:
- |-
Specify the internal ID for the NetSuite File Cabinet folder from which you want
to export your files.
If the folder internal ID is required to be dynamic based on the data you are integrating,
you can specify the JSON path to the field in your data containing the folder
internal ID values instead — with DOUBLE braces (e.g. {{record.folderId}}).
Production usage is dominantly a static numeric ID; dynamic substitution is rare.
- |-
## FIELD BEHAVIOR:
- Identifies the specific folder in NetSuite's file cabinet to export files from
- Must be a valid internal ID that exists in the NetSuite environment
- Supports dynamic values using handlebars notation for data-driven folder selection
- The internal ID is distinct from folder names or paths; it's a stable numeric identifier
- |-
## IMPLEMENTATION GUIDANCE:
- Obtain the folderInternalId via NetSuite's UI (File Cabinet > folder properties) or API
- For static exports, use the numeric internal ID directly (e.g., "12345")
- For dynamic exports, use handlebars syntax to reference a field in your data
- Verify folder permissions - the integration user must have access to the folder
- The folderInternalId is distinct from folder names or folder paths; it is a stable numeric identifier used internally by NetSuite.
- Using an incorrect or non-existent folderInternalId will result in errors or unintended file placement.
- Folder permissions and user roles can impact the ability to perform operations even with a valid folderInternalId.
- Folder hierarchy changes do not affect the folderInternalId, ensuring persistent reference integrity.
- |-
## DEPENDENCY CHAIN:
- Depends on the existence of the folder within the NetSuite file cabinet.
- Requires appropriate user permissions to access or modify the folder.
- Often used in conjunction with file identifiers and other file metadata fields.
- May be linked to folder creation or folder search operations to retrieve valid IDs.
- |-
## EXAMPLES:
- "12345" - Static folder internal ID (the dominant production form)
- "67890" - Another valid folder internal ID
- "{{record.folderId}}" - Dynamic folder ID from integration data (double braces)
- |-
## IMPORTANT NOTES:
- Using an incorrect or non-existent folderInternalId will result in errors
- Folder hierarchy changes do not affect the folderInternalId
- Internal IDs may differ between sandbox and production environments
backupFolderInternalId:
type: string
description: |-
Internal ID of the NetSuite file cabinet folder where backup files are stored.
IDs are account-specific and do not transfer between non-production and production
environments.
examples: ["12345", "67890", "112233"]
x-celigo-ai-guidance:
- |-
backupFolderInternalId is the internal identifier of the backup folder within
the NetSuite file cabinet where backup files are stored.
This ID uniquely identifies the folder location used for saving backup files programmatically,
ensuring that backup operations target the correct directory within the NetSuite environment.
- |-
## FIELD BEHAVIOR:
- Represents a unique internal ID assigned by NetSuite to a specific folder in the file cabinet.
- Directs backup operations to the designated folder location for storing backup files.
- Must correspond to an existing and accessible folder within the NetSuite file cabinet.
- Typically handled as an integer value in API requests and responses.
- Immutable for a given folder; changing the folder requires updating this ID accordingly.
- |-
## IMPLEMENTATION GUIDANCE:
- Verify that the folder with this internal ID exists before initiating backup operations.
- Confirm that the folder has the necessary permissions to allow writing and managing backup files.
- Use NetSuite SuiteScript APIs or REST API calls to retrieve and validate folder internal IDs dynamically.
- Avoid hardcoding the internal ID; instead, use configuration files, environment variables, or administrative settings to maintain flexibility across environments.
- Implement error handling to manage cases where the folder ID is invalid, missing, or inaccessible.
- Consider environment-specific IDs for sandbox versus production to prevent misdirected backups.
- |-
## EXAMPLES:
- 12345
- 67890
- 112233
- |-
## IMPORTANT NOTES:
- The internal ID is unique per NetSuite account and environment; IDs do not transfer between sandbox and production.
- Deleting or renaming the folder associated with this ID will disrupt backup processes until updated.
- Proper access rights and permissions are mandatory to write backup files to the specified folder.
- Changes to folder structure or permissions should be coordinated with backup scheduling to avoid failures.
- This property is critical for ensuring backup data integrity and recoverability within NetSuite.
- |-
## DEPENDENCY CHAIN:
- Depends on the existence and accessibility of the folder in the NetSuite file cabinet.
- Interacts with backup scheduling, file naming conventions, and storage management properties.
- May be linked with authentication and authorization mechanisms controlling file cabinet access.
- Relies on NetSuite API capabilities to manage and reference file cabinet folders.
- |-
## TECHNICAL DETAILS:
- Data type: Integer (typically a 32-bit integer).
- Represents the internal NetSuite folder ID, not the folder name or path.
- Used in API payloads to specify backup destination folder.
- Must be retrieved or confirmed via NetSuite SuiteScript
fileNameStartsWith:
type: string
description: |-
Optional prefix filter. Only files whose names start with this string are
exported from the configured folder.
examples: ["ORDER_", "INV_"]
fileNameEndsWith:
type: string
description: |-
Optional suffix filter. Only files whose names end with this string (e.g. a
".csv" extension) are exported from the configured folder.
examples: [".csv", ".xml"]
required: []
references/schemas/once.yml
Once:
type: object
description: |-
Configures flag-based exports that process each record exactly once. Required when the
export's type is "once"; omit for other export types. Each run retrieves records where the
tracking boolean field is false, then sets that field to true after successful processing so
later runs skip them; if a run fails, the flags are left unchanged and the records are
retried automatically on the next run.
x-celigo-ai-guidance:
- |-
This object is REQUIRED when the export's type field is set to "once" and should
not be included for other export types.
Once exports use a boolean/checkbox field in the source system to track which
records have been processed,
creating a reliable idempotent data extraction pattern.
- |-
For optimal AI agent implementation, consider these guidelines:
1. System behavior during execution:
- First, the export retrieves all records where the specified boolean field is false
- After successfully processing these records, the system automatically sets the field to true
- On subsequent runs, previously processed records are excluded
- |-
2. Prerequisites in the source system:
- The source must have a boolean/checkbox field that can be used as a processing flag
- Your connection must have write access to update this field after export
- The field should be indexed for optimal performance
- |-
3. Common implementation scenarios:
- One-time migrations where data should not be duplicated
- Processing queues where records are marked as "processed"
- Compliance scenarios requiring audit trails of exported records
- Implementing exactly-once delivery semantics
- |-
4. Error handling behavior:
- If the export fails, the boolean fields remain unchanged
- Records will be retried on the next run
- No manual intervention is required for recovery
- |-
This object is REQUIRED when the export's type field is set to "once" and should
not be included for other export types.
Once exports use a boolean/checkbox field in the source system to track which
records have been processed,
creating a reliable idempotent data extraction pattern.
properties:
booleanField:
type: string
x-celigo-ui-override: >-
Required by the export form when the export type is "once" (once.booleanField is
required:true, visibleWhen type=once). Encoded to mirror the form so builders produce
connectable configurations.
description: |-
API field name of the boolean/checkbox in the source system that tracks processed
records: the export selects only records where this field is false, then sets it to true
in batches after each successfully processed page. The field must be writeable by the
export's connection. Ensure no other process updates the same field — use a separate
flag per export process — or records may be skipped unexpectedly.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR
This field identifies which boolean field in the source system controls the export filtering:
- REQUIRED when the export's type field is set to "once"
- Must reference a valid boolean/checkbox field in the source system
- Must be writeable by the connection's authentication credentials
- The system performs two operations with this field:
1. Filters to only include records where this field is false
2. Updates processed records by setting this field to true
- |-
## IMPLEMENTATION PATTERNS
### Using dedicated tracking fields
```
"booleanField": "isExported"
```
- Create a dedicated field specifically for integration tracking
- Provides clear separation between business and integration logic
- Most maintainable approach for long-term operations
### Using existing status fields
```
"booleanField": "isProcessed"
```
- Leverage existing status fields if they align with your integration needs
- Ensure the field's meaning is compatible with your integration logic
- Consider potential conflicts with other processes using the same field
### Targeted export tracking
```
"booleanField": "exported_to_netsuite"
```
- For systems synchronizing to multiple destinations
- Create separate tracking fields for each destination system
- Enables independent control of different export processes
- |-
## TECHNICAL CONSIDERATIONS
- Field updates happen in batches after each successful page of records is processed
- The field update uses the same connection as the export operation
- For optimal performance, the boolean field should be indexed in the source database
- Boolean values of 0/1, true/false, and yes/no are all properly interpreted
IMPORTANT: Ensure the field is not being updated by other processes, as this could
cause records to be skipped unexpectedly. If multiple processes need to track exports,
use separate boolean fields for each process.
examples: ["isExported", "hasBeenSynced", "processedFlag", "exported_to_netsuite"]
references/schemas/preview-request.yml
PreviewRequest:
type: object
description: Request body for previewing export data
allOf:
- $ref: './response.yml#/Export'
- type: object
properties:
postData:
type: object
description: Additional data for the export preview
properties:
currentExportDateTime:
type: string
description: Current export date time (timestamp)
example: "1751909419"
lastExportDateTime:
type: string
description: Last export date time (timestamp)
example: "1742641523"references/schemas/preview-response.yml
PreviewResponse:
type: object
description: Response body for export data preview
properties:
data:
type: array
description: The data exported from source app
items:
type: object
description: Individual data record from the source app
dataURIs:
type: array
description: URIs to the data in the source app
items:
type: string
description: URI to a specific data record in the source app
stages:
type: array
description: Processing stages information
items:
type: object
description: Information about a processing stage
properties:
name:
type: string
description: Name of the processing stage
errors:
type: array
description: Errors encountered during this stage
items:
type: object
description: Error information
data:
type: array
description: Data after being processed by this stage
items:
type: object
description: Individual data record after being processed by this stagereferences/schemas/rdbms.yml
RDBMS:
type: object
description: |-
Configuration object for Relational Database Management System (RDBMS) data integration exports.
This object defines how data is read from a relational database and must not be included
for other connection types. For query-type exports (standard, delta, once) it is REQUIRED
when the _connectionId field references an RDBMS database connection and holds the SQL
query. For real-time stream listeners (`type: "stream"`) it holds the CDC watch list
(`tables`) instead — and a PostgreSQL listener scoped by a publication
(`cdc.publicationName`) may omit this object entirely.
**Rdbms export capabilities**
- Execute custom SQL SELECT statements
- Support for joins, aggregations, and functions
- Flexible data retrieval from any accessible tables or views
- Compatible with all major database systems
**Critical:** WHAT BELONGS IN THIS OBJECT
- `query` - The SQL SELECT statement - required for every query-type export (standard, delta, once)
- `once` - **REQUIRED** when the export's Object Type is `"once"` (set _include_once: true)
- `tables` - the CDC listener watch list - real-time stream listeners (`type: "stream"`) only
- **DO NOT** put `delta` inside this object - delta is handled via the query
- **DO NOT** author `query` on a stream listener - real-time CDC exports have no SQL query
**Delta exports (type: "delta")**
For delta/incremental exports, do NOT populate a `delta` object inside `rdbms`.
Instead, use `{{lastExportDateTime}}` or `{{currentExportDateTime}}` directly in the query:
```json
{
"type": "delta",
"rdbms": {
"query": "SELECT * FROM customers WHERE updatedAt > {{lastExportDateTime}}"
}
}
```
**Once exports (type: "once")**
For once exports (mark records as processed), populate `rdbms.once.query`:
```json
{
"type": "once",
"rdbms": {
"query": "SELECT * FROM customers WHERE exported = false",
"once": {
"query": "UPDATE customers SET exported = true WHERE id = {{record.id}}"
}
}
}
```
**Standard exports (type: null or not specified)**
Just provide the query:
```json
{
"rdbms": {
"query": "SELECT * FROM customers WHERE status = 'ACTIVE'"
}
}
```
**Real-time CDC listener exports (type: "stream")**
SQL Server and PostgreSQL connections support change-data-capture listeners that stream row
changes continuously (the `cdc` object holds the listener settings). A stream export has NO
`query` — it watches `tables`:
```json
{
"type": "stream",
"rdbms": { "tables": "public.orders,public.customers" },
"cdc": { "properties": [{ "name": "snapshot.mode", "value": "no_data" }] }
}
```
A PostgreSQL listener can instead scope tables through a publication
(`cdc.publicationName`), in which case `tables` stays unset and the `rdbms` object may be
omitted entirely.
properties:
tables:
type: string
description: |-
Comma-separated list of tables the CDC listener watches for changes. Used only by real-time
listener exports (`type` is `stream`); standard SQL exports use `query` instead. SQL Server
listeners use fully-qualified three-part names (`database.schema.table`); PostgreSQL
listeners use two-part names (`schema.table`).
x-celigo-ai-guidance:
- |-
For a PostgreSQL listener, author EITHER this field OR `cdc.publicationName`, never both —
when a publication is set it is the sole table filter and the listener form clears this
field (the API stores both if sent, so nothing catches the conflict at save time).
examples: ["UKGODS.dbo.UKG_MetadataStaging", "dbo.WorkOrderT,dbo.OrderLineT", "public.orders,public.customers"]
query:
type: string
x-celigo-agent:
- sql
x-celigo-ui-override: >-
Required by the standard RDBMS export form (rdbms.query is required:true) for query-type
exports. Deliberately NOT encoded as a schema-level required[] entry: stream listener
exports (type: "stream") save without a query — they watch `tables` or a PostgreSQL
publication — and a publication-based PostgreSQL listener persists with no rdbms object at
all, so any required[] here would reject valid listener configs. Do not re-add it.
description: |-
SQL SELECT statement executed to retrieve data, from simple table selections to joins,
aggregations, and parameterized queries with Handlebars expressions. Query-type exports
only — real-time stream listeners (`type` is `stream`) have no SQL query. For delta exports,
reference {{lastExportDateTime}}/{{currentExportDateTime}} directly in the query rather
than configuring a separate delta object.
x-celigo-ai-guidance:
- |-
This field contains the SQL SELECT statement that will be executed to fetch data
from the database.
The query can range from simple table selections to complex joins and aggregations.
- |-
Examples:
- Basic: `SELECT id, name, email FROM customers WHERE status = 'ACTIVE'`
- Join: `SELECT o.id, c.name, o.amount FROM orders o JOIN customers c ON o.customer_id = c.id`
- Aggregate: `SELECT category, COUNT(*) as count FROM orders GROUP BY category`
- Parameterized: `SELECT * FROM orders WHERE customer_id = {{record.customer_id}}`
- |-
This field contains the SQL SELECT statement that will be executed to fetch data
from the database.
The query can range from simple table selections to complex joins and aggregations.
examples: ["SELECT * FROM customers WHERE updatedAt > {{lastExportDateTime}}", "SELECT * FROM orders WHERE exported = false", "SELECT * FROM customers WHERE region = 'Northeast'", "SELECT o.id, c.name, o.total FROM orders o JOIN customers c ON o.customer_id = c.id WHERE o.status = 'SHIPPED'", "SELECT department, AVG(salary) as avg_salary FROM employees GROUP BY department", "SELECT * FROM customer_orders WHERE customer_id = {{record.id}}"]
once:
type: object
description: |
**CRITICAL: REQUIRED when the export's Object Type is "once".**
If Object Type is "once", you MUST set _include_once to true (or include this object).
This object has ONLY ONE property: "query" (a SQL UPDATE string).
DO NOT create any other properties like "update", "table", "set", "where", etc.
CORRECT format:
```json
{"query": "UPDATE customers SET exported=true WHERE id={{record.id}}"}
```
WRONG format (DO NOT DO THIS):
```json
{"update": {"table": "customers", "set": {...}}}
```
properties:
query:
type: string
x-celigo-agent:
- sql
description: |
**REQUIRED** - A SQL UPDATE statement string to mark records as processed.
This is a plain SQL UPDATE query string, NOT a structured object.
The query runs AFTER each record is successfully exported, setting a flag
to indicate the record has been processed.
Format: "UPDATE <table> SET <column>=<value> WHERE <id_column>={{record.<id_field>}}"
Example: "UPDATE customers SET exported=true WHERE id={{record.id}}"
The {{record.id}} placeholder is replaced with the actual record ID from each exported row.
examples: ["UPDATE orders SET exported=true WHERE id={{record.id}}", "UPDATE customers SET exported=true WHERE customer_id={{record.customer_id}}", "UPDATE inventory SET is_exported=1 WHERE sku={{record.sku}}"]
references/schemas/request.yml
Request:
type: object
description: |-
Fields that can be sent when creating or updating an export. Set the adaptor-specific
configuration object matching `adaptorType`, and the mode-specific object matching `type`.
`SimpleExport` (data-loader) is the exception — it needs no adaptor config object.
required:
- name
allOf:
- $ref: './base.yml#/ExportBase'
# The config object matching `adaptorType` must be supplied. Nested if/else-if (no allOf),
# connection-request style. API-enforced: POST/PUT without the object fails with 422
# "<key> subschema not defined".
# Request-only (not in base.yml): legacy persisted records violate these pairings, so the
# response schema must not assert them.
# No SimpleExport branch — POST 201 with no config object (data-loader exports carry `file`
# at most; there is no `simple` object).
if: { properties: { adaptorType: { const: HTTPExport } }, required: [adaptorType] }
then: { required: [http] }
else:
if: { properties: { adaptorType: { const: FTPExport } }, required: [adaptorType] }
then: { required: [ftp] }
else:
# AS2 listeners require `file` (parse settings), not `as2` — POST with only `file`
# returns 201, and typical AS2 exports have no `as2` object at all.
# Exception: webhook-mode AS2 listeners (`type: webhook`) receive raw payloads and
# carry no file parse settings, so they are not required to carry `file`.
if: { properties: { adaptorType: { const: AS2Export } }, required: [adaptorType] }
then:
if:
not: { properties: { type: { const: webhook } }, required: [type] }
then: { required: [file] }
else:
if: { properties: { adaptorType: { const: S3Export } }, required: [adaptorType] }
then: { required: [s3] }
else:
if: { properties: { adaptorType: { const: NetSuiteExport } }, required: [adaptorType] }
then: { required: [netsuite] }
else:
if: { properties: { adaptorType: { const: SalesforceExport } }, required: [adaptorType] }
then: { required: [salesforce] }
else:
if: { properties: { adaptorType: { const: JDBCExport } }, required: [adaptorType] }
then: { required: [jdbc] }
else:
if: { properties: { adaptorType: { const: RDBMSExport } }, required: [adaptorType] }
then: { required: [rdbms] }
else:
if: { properties: { adaptorType: { const: MongodbExport } }, required: [adaptorType] }
then: { required: [mongodb] }
else:
if: { properties: { adaptorType: { const: DynamodbExport } }, required: [adaptorType] }
then: { required: [dynamodb] }
else:
if: { properties: { adaptorType: { const: WrapperExport } }, required: [adaptorType] }
then: { required: [wrapper] }
else:
# webhook.verify is 422-required when `type` is `webhook` (the only real
# WebhookExport shape).
if: { properties: { adaptorType: { const: WebhookExport } }, required: [adaptorType] }
then: { required: [webhook] }
else:
if: { properties: { adaptorType: { const: FileSystemExport } }, required: [adaptorType] }
then: { required: [filesystem] }
else:
# The legacy REST adaptor was folded into `http`: POST and PUT of a
# RESTExport without `http` fail 422; the deprecated `rest` object
# is not required.
if: { properties: { adaptorType: { const: RESTExport } }, required: [adaptorType] }
then: { required: [http] }
references/schemas/response.yml
Export:
type: object
required:
- _id
- name
- adaptorType
- createdAt
- lastModified
description: Export object as returned by the API.
allOf:
- $ref: './base.yml#/ExportBase'
- $ref: '../../../common/schemas/resource-response.yml#/ResourceResponse'
- $ref: '../../../common/schemas/ia-resource-response.yml#/IAResourceResponse'
- type: object
properties:
aiDescription:
$ref: '../../../common/schemas/ai-description.yml#/AIDescription'
apim:
$ref: '../../../common/schemas/apim.yml#/APIM'
apiIdentifier:
type: string
readOnly: true
description: API identifier assigned to this export.
asynchronous:
type: boolean
readOnly: true
description: Server-managed execution-mode flag set on creation; client values are ignored.
__linkedLookupCacheIds:
type: array
readOnly: true
description: Lookup caches linked to this export, managed by the platform.
items:
type: string
format: objectId
sandbox:
type: boolean
deprecated: true
readOnly: true
description: When true, this export belongs to a sandbox account.
rest:
type: object
deprecated: true
readOnly: true
additionalProperties: true
description: |-
Legacy REST adaptor configuration, still returned on exports created before
the REST-to-HTTP migration (`adaptorType: RESTExport`). Mirrors the shape of
`http` (method, relativeURI, paging and response-path settings). On write the
platform maintains the equivalent `http` configuration — configure new
exports through `http` instead.
_sourceId:
type: string
format: objectId
readOnly: true
description: Reference to the source resource this export was created from.
_templateId:
type: string
format: objectId
readOnly: true
x-celigo-refModel: templates
description: Template this export was created from.
draft:
type: boolean
readOnly: true
description: When true, this export is in draft state and has not been confirmed.
draftExpiresAt:
type: string
format: date-time
readOnly: true
description: Timestamp when the draft version of this export expires.
debugUntil:
type: string
format: date-time
readOnly: true
description: Timestamp until which debug logging is enabled for this export.
references/schemas/s3.yml
S3:
type: object
description: |-
Defines which files to retrieve from an Amazon S3 bucket. Required when the _connectionId
field references an AWS S3 connection; must not be included for other connection types.
region and bucket locate the source, keyStartsWith/keyEndsWith filter objects by key, and
backupBucket with keyPrefix controls where files are moved after retrieval.
x-celigo-ai-guidance:
- Configuration object for Amazon S3 (Simple Storage Service) data integration exports.
- |-
The S3 export object has the following requirements:
- Required fields: region, bucket
- Optional fields: keyStartsWith, keyEndsWith, backupBucket, keyPrefix
- |-
## PURPOSE
This configuration specifies:
- Which S3 bucket to retrieve files from
- How to filter files by key patterns
- Where to move files after retrieval (optional)
required:
- region
- bucket
properties:
region:
type: string
default: us-east-1
x-celigo-ui-override: >-
Required by the S3 export form (fieldDefinitions/resources/exports/s3.js: s3.region is
required:true, default us-east-1). Encoded to mirror the form so builders produce
connectable configurations.
description: |-
AWS region where the bucket is located. Case-insensitive; the value is normalized to
lowercase.
x-celigo-ai-guidance:
- |-
- REQUIRED for all S3 exports
- Must be a valid AWS region identifier (e.g., us-east-1, eu-west-1)
- Case-insensitive (will be normalized to lowercase)
examples: ["us-east-1", "eu-west-1", "ap-southeast-2", "us-west-2", "ap-northeast-1"]
bucket:
type: string
x-celigo-ui-override: >-
Required by the S3 export form (fieldDefinitions/resources/exports/s3.js: s3.bucket is
required:true). Encoded to mirror the form so builders produce connectable configurations.
description: |-
S3 bucket to retrieve files from. The connection's AWS credentials must have
s3:ListBucket and s3:GetObject permissions on it.
x-celigo-ai-guidance:
- |-
- REQUIRED for all S3 exports
- Must be a valid existing S3 bucket name
- Globally unique across all AWS accounts
- AWS credentials must have s3:ListBucket and s3:GetObject permissions
examples: ["my-company-exports", "customer-data-integration", "celigo-etl-files", "financial-data-prod"]
keyStartsWith:
type: string
description: |-
Only retrieves objects whose keys start with this value (case-sensitive), effectively
selecting a folder in S3's flat key structure. When combined with keyEndsWith, objects
must match both.
x-celigo-ai-guidance:
- Optional prefix filter for S3 object keys.
- |-
- Filters files based on the beginning of their keys
- Functions as a directory path in S3's flat storage structure
- Case-sensitive (S3 keys are case-sensitive)
- Examples:
- `"exports/"` - retrieves files in the exports "directory"
- `"customer/orders/2023/"` - retrieves files in this nested path
- `"invoice_"` - retrieves files starting with "invoice_"
- When used with keyEndsWith, files must match both criteria.
examples: ["exports/", "customer/orders/", "data/2023/", "invoice_", "logs/application/"]
keyEndsWith:
type: string
description: |-
Only retrieves objects whose keys end with this value (case-sensitive), commonly a file
extension. When combined with keyStartsWith, objects must match both.
x-celigo-ai-guidance:
- Optional suffix filter for S3 object keys.
- |-
- Commonly used to filter by file extension
- Case-sensitive (S3 keys are case-sensitive)
- Examples:
- `".csv"` - retrieves only CSV files
- `"_FINAL.xml"` - retrieves only XML files with _FINAL suffix
- `"_READY"` - retrieves files with status indicator
- When used with keyStartsWith, files must match both criteria.
examples: [".csv", ".json", ".xml", "_READY", "_FINAL.txt", "-daily.parquet"]
backupBucket:
type: string
description: |-
Bucket in the same region where files are moved after successful export, giving you an
independent backup; if omitted, files are simply deleted from the source bucket after
successful export (Celigo also keeps its own copy of processed files for a set period).
The connection's AWS credentials must have s3:PutObject permission on this bucket.
x-celigo-ai-guidance:
- Optional destination bucket where files are moved before deletion.
- |-
IMPORTANT:
Celigo automatically deletes files from the source bucket after successful export.
The backup bucket is for users who want to maintain their own independent backup
of exported files.
Celigo also maintains its own backup of processed files for a set period of time.
- |-
IMPORTANT:
Celigo automatically deletes files from the source bucket after successful export.
The backup bucket is for users who want to maintain their own independent backup
of exported files.
Celigo also maintains its own backup of processed files for a set period of time.
examples: ["my-company-processed", "archive-bucket", "integration-backup", "source-bucket-archive"]
keyPrefix:
type: string
description: |-
Prefix prepended to each file's name when it is moved to the backup bucket; accepts
static text or handlebars templates. The original directory structure is not preserved —
only the filename is appended to this prefix. Ignored unless backupBucket is set.
x-celigo-ai-guidance:
- Optional prefix to prepend to keys when moving to backup bucket.
- |-
- Used only when backupBucket is specified
- Prepended to the original filename when moved to backup
- Can contain static text or handlebars templates
- Examples:
- `"processed/"` - places files under a processed folder
- `"archive/{{date 'YYYY-MM-DD'}}/"` - organizes by date
examples: ["processed/", "archive/{{date 'YYYY-MM'}}/", "history/", "completed/{{date 'YYYY/MM/DD'}}/"]
x-celigo-agent:
- handlebars
references/schemas/salesforce.yml
Salesforce:
type: object
description: |-
Configuration for Salesforce exports. Required when the export's `_connectionId` references a Salesforce connection; omit for other connection types. The `type` field selects the extraction mode: `soql` runs batch queries and requires the `soql` object, while `distributed` listens for real-time events and requires the `distributed` object. File retrieval (when the export's `type` is `blob`) requires `sObjectType` and `id` instead.
x-celigo-ai-guidance:
- |-
This object is REQUIRED when the _connectionId field references a Salesforce
connection and must not be included for other connection types.
It defines how data is extracted from Salesforce,
either through queries or real-time events.
- |-
Always set `type` on new Salesforce exports (`soql` or `distributed`) —
the form always writes it. It is absent only from legacy real-time
exports created before the field existed, and file/byId retrievals
(`id` + `sObjectType`, no `soql` object) carry `type: soql` without a
`soql` object.
- |-
## SALESFORCE EXPORT MODES
Salesforce exports offer two fundamentally different operating modes:
1. **SOQL Query-based Exports** (type="soql")
- Scheduled or on-demand batch processing
- Uses SOQL queries to retrieve data
- Supports both REST and Bulk API
- Can be configured as lookups (isLookup=true)
- Requires the "soql" object with query configuration
2. **Real-time Event Listeners** (type="distributed")
- Responds to Salesforce events as they happen
- Uses Salesforce's streaming API and platform events
- Always appears as a "Listener" in the flow builder UI
- Requires the "distributed" object with event configuration
3. **File/Blob Exports** (when export.type="blob")
- Retrieves files stored in Salesforce
- Requires sObjectType and id fields
- Supports Attachments, ContentVersion, and Document objects
- |-
## IMPLEMENTATION REQUIREMENTS
The salesforce object has conditional requirements based on the selected type:
- For SOQL exports (type="soql"):
Required fields: type, soql.query
Optional fields: api, includeDeletedRecords, bulk (when api="bulk")
- For Distributed exports (type="distributed"):
Required fields: type, distributed configuration
Optional fields: distributed.referencedFields, distributed.qualifier
- For Blob exports (when export.type="blob"):
Required fields: sObjectType, id
properties:
type:
type: string
description: |-
Selects the extraction mode and determines which configuration object is required. Use `soql` for scheduled or on-demand batch queries and lookups — it requires the `soql` object and works with both the `rest` and `bulk` APIs and the standard, delta, test, and once export types. Use `distributed` for real-time event handling — it requires the `distributed` object, always appears as a Listener in Flow Builder, ignores the `api` field, and supports only the standard export type.
x-celigo-ai-guidance:
- Defines the fundamental data extraction method for Salesforce exports.
- |-
## FIELD BEHAVIOR
This field determines the core operating mode of the Salesforce export:
- REQUIRED for all Salesforce exports
- Controls which additional configuration objects must be provided
- Affects how the export appears and functions in the flow builder UI
- Cannot be changed after creation without significant reconfiguration
- |-
## AVAILABLE TYPES
### SOQL Query-based Export
```
"type": "soql"
```
- **Behavior**: Executes SOQL queries against Salesforce on schedule or demand
- **UI Appearance**: "Export" or "Lookup" based on isLookup value
- **Required Config**: Must provide the "soql" object with a valid query
- **Use Cases**: Batch data extraction, delta synchronization, data migration
- **Dependencies**:
- Compatible with both "rest" and "bulk" API options
- Works with standard, delta, test, and once export types
### Real-time Event Listener
```
"type": "distributed"
```
- **Behavior**: Listens for real-time Salesforce events (create/update/delete)
- **UI Appearance**: Always appears as a "Listener" in the flow builder
- **Required Config**: Must provide the "distributed" object with event configuration
- **Use Cases**: Real-time synchronization, event-driven integration
- **Dependencies**:
- Only uses REST API (api field is ignored)
- Automatically configured with trigger logic in Salesforce
- Only compatible with standard export type (ignores delta/test/once)
- |-
## IMPLEMENTATION CONSIDERATIONS
The type selection creates a fundamental difference in how data flows:
- "soql" operates on a pull model where the integration initiates data retrieval
- "distributed" operates on a push model where Salesforce events trigger the integration
IMPORTANT: Choose "soql" for batch processing and lookups; choose "distributed" for
real-time event handling. This decision affects all other configuration aspects.
enum: ["soql", "distributed"]
x-enumDescriptions:
soql: Executes SOQL queries to retrieve data from Salesforce on a schedule or on demand.
distributed: Listens for real-time Salesforce events (create, update, delete) as they occur.
examples: ["soql", "distributed"]
sObjectType:
type: string
description: |-
API name of the Salesforce object the export operates on; the object must exist in the connected org and be accessible to the integration user. Required for distributed exports and for blob exports — blob exports accept only the file storage objects (`Attachment`, `ContentVersion`, `Document`) paired with the `id` field. Optional for SOQL exports, where the object can be inferred from the query.
x-celigo-ai-guidance:
- Specifies the Salesforce object type for the export operation.
- |-
## FIELD BEHAVIOR
This field determines which Salesforce object is being exported:
- **REQUIRED** when the parent export's type is "distributed"
- **REQUIRED** when the parent export's type is "blob"
- Optional for "soql" exports (can be inferred from the SOQL query)
- Must be a valid Salesforce object API name
- |-
## USE CASES BY EXPORT TYPE
### Distributed Exports
```
"sObjectType": "Account"
"sObjectType": "Contact"
"sObjectType": "Opportunity"
"sObjectType": "Custom_Object__c"
```
- **Purpose**: Specifies the primary object type being exported from Salesforce
- **Valid Values**: Any standard or custom Salesforce object (Account, Contact, Opportunity, Lead, Case, Custom_Object__c, etc.)
- **API Access**: Uses the specified object's metadata and SOQL/REST APIs
- **Use Cases**: Real-time distributed processing of Salesforce records
- **Requirements**: Object must exist in the connected Salesforce org
### Blob/File Exports
```
"sObjectType": "Attachment"
"sObjectType": "ContentVersion"
"sObjectType": "Document"
```
- **Purpose**: Specifies which Salesforce file storage object contains the file data
- **Valid Values**: File storage objects only (Attachment, ContentVersion, Document)
- **API Access**: Uses file-specific APIs for data retrieval
- **Use Cases**: Extracting files and binary data from Salesforce
- **Requirements**: Must be used with the "id" field to specify the file record
### SOQL Exports
```
"sObjectType": "Account" // Optional - can be inferred from query
```
- **Purpose**: Optional hint about the primary object in the SOQL query
- **Valid Values**: Any Salesforce object referenced in the query
- **Use Cases**: Query optimization and metadata context
- |-
## IMPLEMENTATION NOTES
For distributed exports, this field is essential for:
- Setting up proper event listeners and triggers
- Configuring field metadata and validation
- Enabling related object processing
- Determining appropriate API endpoints
For blob exports, this field works with the "id" field to retrieve specific file records.
IMPORTANT: The object specified must exist in the target Salesforce org and be accessible
to the integration user account.
examples: ["Account", "Contact", "Opportunity", "Lead", "Case", "Custom_Object__c", "Attachment", "ContentVersion", "Document"]
id:
type: string
description: |-
Salesforce record ID of the file to retrieve; required for blob exports and not used for `soql` or `distributed` exports. Accepts a static ID or a handlebars expression such as `{{record.Attachment_Id__c}}` to resolve the file at runtime — dynamic IDs require the export to run as a lookup (`isLookup: true`). The ID must belong to the object named in `sObjectType`.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR
This field identifies the specific file record in Salesforce:
- REQUIRED when the parent export's type is "blob"
- Must be a valid Salesforce ID or a handlebars expression
- Used in conjunction with sObjectType to retrieve the file
- Not used for regular data exports (type="soql" or "distributed")
- |-
## IMPLEMENTATION PATTERNS
### Static File ID
```
"id": "00P5f00000ZQcTZEA1"
```
- References a specific, fixed file in Salesforce
- Useful for retrieving standard documents or templates
- Always retrieves the same file on each execution
- Simple to configure but lacks flexibility
### Dynamic File ID (Handlebars)
```
"id": "{{record.Attachment_Id__c}}"
```
- References a file ID from input data using handlebars
- Requires the export to be used as a lookup (isLookup=true)
- Dynamically determines which file to retrieve at runtime
- Allows for contextual file retrieval based on previous steps
- |-
## TECHNICAL DETAILS
- For ContentVersion objects, this should be the ContentVersion ID
- For Attachment objects, this should be the Attachment ID
- For Document objects, this should be the Document ID
IMPORTANT: Salesforce IDs are 15 or 18 characters, case-sensitive for 15-character
versions, and case-insensitive for 18-character versions. When using handlebars,
ensure the referenced field contains a valid Salesforce ID.
examples: ["00P5f00000ZQcTZEA1", "{{record.Attachment_ID__c}}", "{{record.ContentVersion_ID__c}}"]
x-celigo-agent:
- handlebars
includeDeletedRecords:
type: boolean
description: |-
When true, SOQL exports use Salesforce's `queryAll()` instead of `query()`, including Recycle Bin records deleted within the past 15 days; each record's `IsDeleted` field identifies deletions. Useful for synchronizing deletes to target systems or maintaining a complete audit trail. Ignored for distributed and blob exports.
x-celigo-ai-guidance:
- Controls whether the export retrieves records from the Salesforce Recycle Bin.
- |-
## FIELD BEHAVIOR
This field enables access to recently deleted records:
- OPTIONAL: Defaults to false if not specified
- Only relevant for SOQL exports (type="soql")
- Ignored for distributed exports and blob exports
- Changes the underlying API method used for queries
- |-
## IMPLEMENTATION IMPACT
When set to true:
- Salesforce's queryAll() API method is used instead of query()
- Records in the Recycle Bin (deleted within the past 15 days) are included
- Each record contains an "IsDeleted" field to identify deleted status
- API usage may be higher as queryAll() counts differently against limits
- |-
## USE CASES
This field is particularly useful for:
- Synchronizing deletion operations to target systems
- Building data recovery/rollback mechanisms
- Maintaining a complete audit trail including deleted records
- Implementing soft-delete patterns across integrated systems
- |-
## TECHNICAL CONSIDERATIONS
- Records in the Recycle Bin are only available for up to 15 days
- Hard-deleted records (emptied from Recycle Bin) are not accessible
- The IsDeleted field should be checked to identify deleted records
- May increase response size and processing time slightly
IMPORTANT: This feature only works with SOQL exports (type="soql") and is ignored
for distributed exports (type="distributed") since those operate on events rather
than queries.
default: false
examples: [true, false]
api:
type: string
description: |-
Salesforce API used to run SOQL queries; ignored for distributed and blob exports. Use `rest` (the default when omitted) for smaller datasets (under 10,000 records) and for lookup exports — the Bulk API is not compatible with `isLookup: true`. Use `bulk` for large data volumes and higher throughput, optionally tuned through the `bulk` object.
x-celigo-ai-guidance:
- Specifies which Salesforce API to use for retrieving data.
- |-
## FIELD BEHAVIOR
This field controls the underlying API technology:
- OPTIONAL: Defaults to "rest" if not specified
- Only relevant for SOQL exports (type="soql")
- Ignored for distributed exports and blob exports
- Determines performance characteristics and compatibility
- |-
## AVAILABLE APIs
### REST API
```
"api": "rest"
```
- **Performance**: Optimized for immediate response and smaller datasets
- **Concurrency**: Higher - multiple queries can run simultaneously
- **Data Volume**: Best for <10,000 records
- **Use Cases**: Lookups, real-time queries, smaller datasets
- **Special Features**: Required for lookup exports (isLookup=true)
### Bulk API 2.0
```
"api": "bulk"
```
- **Performance**: Optimized for large data volumes, higher throughput
- **Concurrency**: Lower - utilizes a job queuing system
- **Data Volume**: Best for >=10,000 records
- **Use Cases**: Large data migrations, full dataset exports, reports
- **Special Features**: Requires "bulk" object configuration for settings
- |-
## DEPENDENCIES AND CONSTRAINTS
- When isLookup=true, api must be set to "rest" (or left as default)
- When api="bulk", the bulk object can be configured for additional options
- Bulk API introduces slight processing latency but handles larger volumes
- REST API provides immediate results but may time out with very large queries
- |-
## SELECTION GUIDANCE
Choose based on your data volume and response time needs:
- For smaller datasets (<10,000 records) or lookups: use "rest"
- For larger datasets or background processing: use "bulk"
- When immediacy is critical: use "rest"
- When throughput is critical: use "bulk"
IMPORTANT: The Bulk API is not compatible with lookup exports (isLookup=true).
If your export is configured as a lookup, you must use the REST API.
enum: ["rest", "bulk"]
x-enumDescriptions:
rest: Uses the Salesforce REST API, optimized for smaller datasets and lookup exports.
bulk: Uses the Salesforce Bulk API 2.0, optimized for large data volumes with higher throughput.
examples: ["rest", "bulk"]
bulk:
type: object
description: |-
Settings for Salesforce Bulk API 2.0 jobs. Only applies when `api` is `bulk` on a SOQL export; ignored otherwise.
x-celigo-ai-guidance:
- Configuration parameters for Salesforce Bulk API 2.0 exports.
- |-
## FIELD BEHAVIOR
This object contains settings specific to Bulk API operations:
- REQUIRED when api="bulk" and type="soql"
- Ignored when api="rest" or type="distributed"
- Controls behavior of Salesforce Bulk API jobs
- Provides optimization options for large data volumes
- |-
## IMPLEMENTATION CONTEXT
The Bulk API operates differently from REST API:
- Creates asynchronous jobs in Salesforce
- Processes records in batches for higher throughput
- Optimized for transferring large datasets
- Has different governor limits and behavior
properties:
maxRecords:
type: integer
description: |-
Caps how many records a single Bulk API job retrieves, which helps prevent timeouts with complex queries or large records. When omitted, Salesforce's default applies. Lower values suit complex or custom objects; higher values improve throughput for simple records.
x-celigo-ai-guidance:
- Specifies the maximum number of records to retrieve in a single Bulk API job.
- |-
## FIELD BEHAVIOR
This field controls query result size:
- OPTIONAL: Uses Salesforce's default if not specified
- Sets the `maxRecords` parameter on Bulk API requests
- Only applicable when api="bulk" and type="soql"
- Helps prevent timeouts with complex queries or large record sizes
- |-
## TECHNICAL CONSIDERATIONS
- Different Salesforce editions have different limits
- Values too high may cause timeouts with complex records
- Values too low may require multiple API calls
- Standard objects typically support higher limits than custom objects
- |-
## OPTIMIZATION GUIDANCE
- For simple records (few fields): Higher values improve throughput
- For complex records (many fields): Lower values prevent timeouts
- For standard objects: 50,000 is usually safe
- For custom objects: 10,000-25,000 is recommended
IMPORTANT: The Salesforce Bulk API 2.0 has a hard limit of 100 million records
per job, but practical limits are typically much lower based on record complexity
and Salesforce instance capacity.
minimum: 10000
examples: [10000, 50000, 100000]
purgeJobAfterExport:
type: boolean
default: true
description: |-
When true, deletes the Bulk API job in Salesforce after all data is retrieved, keeping the Bulk Data Load Jobs list clean — at the cost of making job details unavailable for later troubleshooting. Has no effect on the data retrieved or the success of the export.
x-celigo-ai-guidance:
- Controls whether Bulk API jobs are automatically deleted after completion.
- |-
## FIELD BEHAVIOR
This field manages job cleanup:
- OPTIONAL: Defaults to true if not specified
- When true, deletes the Bulk API job after all data is retrieved
- Only applicable when api="bulk" and type="soql"
- Has no effect on the actual data retrieval or results
- |-
## IMPLEMENTATION IMPACT
When enabled (true):
- Reduces clutter in the Salesforce Bulk Data Load Jobs UI
- Prevents accumulation of completed jobs
- May help stay under job retention limits
- Makes job details unavailable for later troubleshooting
When disabled (false):
- Preserves job history for troubleshooting
- Allows reviewing job details in Salesforce
- May accumulate many jobs over time
- |-
## BEST PRACTICES
- For production environments: Set to true for cleanliness
- For testing/development: Set to false for easier debugging
- For audit-heavy environments: Set to false if job history is needed
IMPORTANT: This setting only affects job metadata cleanup in Salesforce.
It has no impact on the actual data retrieved or the success of the export.
examples: [true, false]
soql:
type: object
description: |-
SOQL query configuration; required when `type` is `soql` and not used otherwise. The query controls which objects, fields, and filter conditions the export retrieves, with either the REST or Bulk API.
Required when type is soql.
x-celigo-ai-guidance:
- Configuration for SOQL query-based Salesforce exports.
- |-
## FIELD BEHAVIOR
This object contains the SOQL query settings:
- REQUIRED when type="soql"
- Not used when type="distributed" or for blob exports
- Controls what data is retrieved from Salesforce
- Works with both REST API and Bulk API methods
- |-
## IMPLEMENTATION REQUIREMENTS
The soql object must include a valid query that follows Salesforce SOQL syntax.
The query determines:
- Which objects are accessed
- Which fields are retrieved
- What filtering conditions are applied
- How results are sorted and limited
properties:
query:
type: string
x-celigo-ui-override: >-
Required by the Salesforce export form when type is soql (soql.query is required:true).
Encoded to mirror the form so builders produce connectable configurations.
description: |-
SOQL statement passed directly to Salesforce that defines which objects, fields, and records the export retrieves. Supports handlebars — for delta exports, filter with `WHERE LastModifiedDate > {{lastExportDateTime}}` — and relationship queries that fetch parent and child records together. Select only the fields you need and use `ORDER BY` for consistent results across pages.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR
This field contains the actual SOQL statement:
- REQUIRED when type="soql"
- Must follow Salesforce Object Query Language syntax
- Passed directly to Salesforce API endpoints
- Can include dynamic values via handlebars
- |-
## QUERY STRUCTURE ELEMENTS
A complete SOQL query typically includes:
### Field Selection
```
SELECT Id, Name, Email, Phone, Account.Name
```
- List specific fields to retrieve
- Include relationship fields using dot notation
- Use * sparingly (only with specific sObjects that support it)
### Object Selection
```
FROM Contact
```
- Specifies the Salesforce object to query
- Must be a valid API name (not label)
- Case-sensitive (match Salesforce API names exactly)
### Filter Conditions
```
WHERE LastModifiedDate > {{lastExportDateTime}}
AND IsActive = true
```
- Limits which records are returned
- Can reference handlebars variables (e.g., for delta exports)
- Supports standard operators (=, !=, >, <, LIKE, IN, etc.)
### Relationship Queries
```
SELECT Account.Id, (SELECT Id, FirstName FROM Contacts)
FROM Account
```
- Retrieves parent and child records in a single query
- Helps reduce API calls for related data
- Supports both lookup and master-detail relationships
- |-
## IMPLEMENTATION BEST PRACTICES
- Select only the fields you need (improves performance)
- Use WHERE clauses to limit data volume
- For delta exports, use LastModifiedDate with {{lastExportDateTime}}
- Use ORDER BY for consistent results across multiple pages
- Avoid SOQL functions in filters when using Bulk API
- |-
## TECHNICAL LIMITS
- Maximum query length: 20,000 characters
- Maximum relationships traversed: 5 levels
- Maximum subquery levels: 1 (no nested subqueries)
- Maximum batch size varies by API (REST: 2,000, Bulk: 10,000+)
IMPORTANT: When using relationship queries, child objects count against
governor limits differently. For bulk processing of many parent-child records,
consider separate queries or the oneToMany export setting.
maxLength: 200000
examples: ["SELECT Id, Name FROM Account WHERE LastModifiedDate > {{lastExportDateTime}}", "SELECT Id, Name, Email FROM Contact WHERE IsActive = true", "SELECT Id, Name, (SELECT Id, FirstName, LastName FROM Contacts) FROM Account"]
distributed:
type: object
description: |-
Real-time event listener configuration; required when `type` is `distributed` and not used otherwise. The system installs triggers in the connected Salesforce org and delivers records to the flow as create, update, and delete events occur — no scheduling or manual execution is involved.
x-celigo-ai-guidance:
- Configuration for real-time Salesforce event-driven exports.
- |-
## FIELD BEHAVIOR
This object defines real-time event listener settings:
- REQUIRED when type="distributed"
- Not used when type="soql" or for blob exports
- Creates push-based integration triggered by Salesforce events
- Implements real-time processing of creates, updates, and deletes
- |-
## IMPLEMENTATION CONTEXT
Distributed exports work fundamentally differently from SOQL exports:
- No scheduling or manual execution required
- Triggered automatically when records change in Salesforce
- Data flows in real-time as events occur
- Uses Salesforce's platform events and streaming API
- |-
## TECHNICAL ARCHITECTURE
When configured, the system:
1. Creates custom triggers in the connected Salesforce org
2. Establishes event listeners for the specified objects
3. Processes events as they occur (create/update/delete operations)
4. Delivers the changed records to the integration flow
properties:
referencedFields:
type: [array, 'null']
description: |-
Fields from related objects to include in the exported record, written in dot notation (e.g. `Account.Name`, `Owner.Email`, `Custom_Lookup__r.Field__c`). Works with lookup and master-detail relationships, up to 10 unique referenced relationships per export. Referenced fields are retrieved via separate API calls, so include only fields the integration actually needs.
x-celigo-ai-guidance:
- Specifies additional fields to retrieve from related objects via relationships.
- |-
## FIELD BEHAVIOR
This field extends the data retrieval beyond the primary object:
- OPTIONAL: If omitted, only direct fields are retrieved
- Each entry specifies a field on a related object using dot notation
- Values are included in the exported record data
- Only works with lookup and master-detail relationships
- |-
## IMPLEMENTATION PATTERNS
### Parent Object Fields
```
["Account.Name", "Account.Industry", "Account.BillingCity"]
```
- Retrieves fields from parent objects
- Useful for including context from parent records
- Common for child objects like Contacts, Opportunities
### User/Owner Fields
```
["Owner.Email", "CreatedBy.Name", "LastModifiedBy.Username"]
```
- Retrieves fields from standard user relationship fields
- Provides attribution information
- Useful for auditing and notification scenarios
### Custom Relationship Fields
```
["Custom_Lookup__r.Field_Name__c", "Another_Relation__r.Status__c"]
```
- Works with custom relationship fields
- Uses __r suffix for the relationship name
- Can access standard or custom fields on the related object
- |-
## TECHNICAL CONSIDERATIONS
- Maximum 10 unique referenced relationships per export
- Each referenced field counts against Salesforce API limits
- Fields must be accessible to the connected user
- Performance impact increases with each additional relationship
IMPORTANT: Referenced fields are retrieved via separate API calls,
which can impact performance with large numbers of records or relationships.
Only include fields that are actually needed by your integration.
items:
type: [string, 'null']
examples:
- Account.Name
- Account.Industry
- Owner.Email
- Custom_Object__r.Status__c
disabled:
type: boolean
description: |-
When true, the listener stops processing events while its configuration and Salesforce triggers stay in place. Events that occur while disabled are ignored, not queued — they are not processed retroactively when re-enabled, so consider a delta export to catch up after extended pauses.
x-celigo-ai-guidance:
- Controls whether this real-time event listener is active.
- |-
## FIELD BEHAVIOR
This field enables/disables event processing:
- OPTIONAL: Defaults to false if not specified
- When true, prevents the export from processing any events
- Preserves configuration while temporarily stopping execution
- Can be toggled without removing the entire export
- |-
## USE CASES
This field is particularly useful for:
- Temporarily pausing real-time integration during maintenance
- Testing event configuration without processing
- Creating standby event handlers for disaster recovery
- Controlling traffic during peak business periods
- |-
## IMPLEMENTATION NOTES
When disabled (true):
- Events are NOT queued - they are completely ignored
- No data will flow through this export
- The Salesforce triggers remain in place but are inactive
- No impact on Salesforce performance or API limits
IMPORTANT: When disabled, events that occur will NOT be processed retroactively
when re-enabled. Consider using a delta export for catching up on missed changes
after extended disabled periods.
examples: [true, false]
qualifier:
type: [string, 'null']
description: |-
Salesforce formula expression that filters which events are processed; when omitted (stored as `null`), all events for the object are processed. Evaluated inside Salesforce before events are sent, which is more efficient than filtering in a later flow step. Reference only fields that exist on the monitored object; formula functions such as `ISCHANGED(Status__c)` are supported.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR
This field provides server-side filtering:
- OPTIONAL: If omitted, all events for the object are processed
- Uses Salesforce formula syntax for filtering
- Evaluated before events are sent to the integration platform
- Can reference any field on the triggering record
- |-
## IMPLEMENTATION PATTERNS
### Simple Field Comparisons
```
"Status__c = 'Approved'"
```
- Processes events only when specific field values match
- Most efficient filtering approach
- Can use =, !=, >, <, >=, <= operators
### Logical Conditions
```
"Amount > 1000 AND Status__c = 'New'"
```
- Combines multiple conditions with AND, OR operators
- Can use parentheses for complex grouping
- Allows precise control over which events trigger the integration
### Formula Functions
```
"CONTAINS(Description, 'Priority') OR ISCHANGED(Status__c)"
```
- Uses Salesforce formula functions
- ISCHANGED detects specific field modifications
- ISNEW, ISDELETED detect record lifecycle events
- |-
## PERFORMANCE IMPACT
The qualifier is evaluated in Salesforce before sending events:
- Reduces network traffic and processing
- Lowers integration platform load
- More efficient than filtering in a subsequent flow step
- No additional API calls required
IMPORTANT: The qualifier is evaluated using the Salesforce formula engine.
Use valid Salesforce formula syntax and reference only fields that exist
on the primary object being monitored.
examples: ["Amount > 1000", "Status__c = 'Approved'", "LeadSource = 'Web' AND Competitor__c = 'Oracle'", "ISCHANGED(Status__c) AND NOT(ISNEW())"]
batchSize:
type: [integer, 'null']
description: |-
Controls how many event records are grouped into each real-time processing batch. Stored as `null` when not configured. Smaller batches lower latency for time-sensitive operations; larger batches improve throughput for high-volume objects. Does not limit how many records are processed in total — only how they are grouped.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR
This field affects event processing efficiency:
- OPTIONAL: Uses system default if not specified
- Valid range: 4 to 200 records per batch
- Affects how events are grouped before processing
- Balance between latency and throughput
- |-
## PERFORMANCE CONSIDERATIONS
### Smaller Batch Sizes (4-20)
```
"batchSize": 10
```
- Lower latency - events processed more immediately
- More overhead for small numbers of records
- Better for time-sensitive operations
- More resilient for complex record processing
### Larger Batch Sizes (50-200)
```
"batchSize": 100
```
- Higher throughput - better efficiency for many records
- Slight increase in processing delay
- Better for high-volume operations
- More efficient use of API calls and resources
## IMPLEMENTATION GUIDANCE
Choose based on your volume and timing requirements:
- For high-volume objects (many changes per minute): Use larger batches
- For time-sensitive operations: Use smaller batches
- For complex processing logic: Use smaller batches
- For efficiency and throughput: Use larger batches
IMPORTANT: The batch size doesn't limit how many records can be processed
in total, only how they're grouped for processing. All events will eventually
be processed regardless of batch size.
minimum: 4
maximum: 200
examples: [10, 50, 100]
skipExportFieldId:
type: [string, 'null']
description: |-
API name of a Salesforce checkbox field used to prevent infinite loops in bidirectional syncs: when the integration updates a record, this field is set so the resulting event is ignored, then cleared automatically. Required for bidirectional sync scenarios. The field must be a checkbox dedicated to integration use and updateable by the integration user.
x-celigo-ai-guidance:
- Specifies a boolean field that prevents integration loops in bidirectional sync.
- |-
## FIELD BEHAVIOR
This field provides a loop prevention mechanism:
- OPTIONAL: If omitted, no loop prevention is applied
- Must reference a valid boolean/checkbox field on the object
- Field must be updateable via the Salesforce API
- System automatically manages the field's value
- |-
## IMPLEMENTATION MECHANISM
The loop prevention works as follows:
1. When your integration updates a record in Salesforce
2. The system temporarily sets this field to true
3. The update triggers Salesforce's normal event system
4. But events where this field is true are ignored
5. The system automatically clears the field afterward
- |-
## USE CASES
This field is critical for:
- Bidirectional synchronization scenarios
- Preventing infinite update loops
- Implementing changes that flow both ways
- Distinguishing between user changes and integration changes
- |-
## FIELD REQUIREMENTS
The field you specify must be:
- A checkbox (Boolean) field in Salesforce
- Created specifically for integration purposes
- Not used by other business processes
- Updateable by the integration user
IMPORTANT: For bidirectional sync scenarios, this field is required.
Without it, updates from your integration would trigger events that
could create infinite loops between systems.
examples: ["Skip_Export__c", "Exclude_From_Integration__c", "Integration_Update__c"]
relatedLists:
type: [array, 'null']
description: |-
Child record sets to include with the parent record when it changes, one entry per related list. When omitted (stored as `null`), only the primary record is processed. Each related list adds Salesforce API calls and increases payload size.
x-celigo-ai-guidance:
- Configuration for retrieving child records related to the primary object.
- |-
## FIELD BEHAVIOR
This field enables parent-child data synchronization:
- OPTIONAL: If omitted, only the primary record is processed
- Each array entry configures one related list/child object
- Child records are included with their parent in the payload
- Automatically retrieves child records when parent changes
- |-
## IMPLEMENTATION CONTEXT
This feature allows you to:
- Synchronize complete object hierarchies in real-time
- Include child records when a parent record changes
- Process parent-child data together in a single flow
- Maintain relationships between objects across systems
- |-
## TECHNICAL IMPACT
- Each related list requires additional Salesforce API calls
- Performance impact increases with each related list
- Data volume can increase significantly with many children
- Parent-child structures may require special handling in flows
items:
type: object
description: |-
Defines how to retrieve one type of child record related to the primary object. Configure multiple entries to retrieve different types of children.
x-celigo-ai-guidance:
- Configuration for a single related list (child object) to include.
- |-
Each object in this array defines how to retrieve one type of child records
related to the primary object.
Multiple related lists can be configured to retrieve different types of children.
properties:
referencedFields:
type: array
description: |-
API names of the child object fields to retrieve; only the listed fields are included, and an empty array retrieves only the `Id` field. Include only fields the integration needs — each field adds data volume and processing time.
x-celigo-ai-guidance:
- Specifies which fields to retrieve from the child records.
- |-
## FIELD BEHAVIOR
This field selects child record fields:
- REQUIRED for each related list configuration
- Must contain valid API field names for the child object
- Only listed fields will be retrieved from child records
- Empty array will retrieve only Id field
- |-
## IMPLEMENTATION GUIDANCE
- Include only fields needed by your integration
- Always include key identifier fields
- Consider relationship fields if needed
- Balance between completeness and performance
IMPORTANT: Each field increases data volume and processing time.
Only include fields that your integration actually needs to process.
items:
type: [string, 'null']
examples:
- FirstName
- LastName
- Email
- LineItemNumber
- Status__c
parentField:
type: string
description: |-
API name of the lookup or master-detail field on the child object that references the parent (e.g. `AccountId`, `Parent_Object__c`) — the field's API name, not the relationship name. Used to build the query that fetches children for each parent record.
x-celigo-ai-guidance:
- Specifies the field on the child object that relates back to the parent.
- |-
## RELATIONSHIP FIELD PATTERNS
### Standard Relationships
```
"parentField": "AccountId"
```
- For standard parent-child relationships
- Field name typically ends with "Id"
- References standard objects
### Custom Relationships
```
"parentField": "Parent_Object__c"
```
- For custom parent-child relationships
- Field name typically ends with "__c"
- References custom objects
- |-
## TECHNICAL DETAILS
The system uses this field to construct a query like:
```
SELECT [referencedFields] FROM [sObjectType]
WHERE [parentField] = [parent record Id]
```
IMPORTANT: This must be the exact API name of the field on the child
object that creates the relationship to the parent, not the relationship
name itself.
examples: ["AccountId", "OpportunityId", "Parent_Object__c", "Case__c"]
sObjectType:
type: string
description: |-
Case-sensitive API name of the child object to retrieve (e.g. `Contact`, `Custom_Child__c`) — use the API name, not the label. The object must have a relationship field to the parent and be accessible to the connected user.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR
This field identifies the child object type:
- REQUIRED for each related list configuration
- Must be a valid Salesforce API object name
- Case-sensitive (match Salesforce naming exactly)
- Can be standard or custom object
- |-
## OBJECT NAME PATTERNS
### Standard Objects
```
"sObjectType": "Contact"
```
- Standard Salesforce objects
- No namespace or suffix
- First letter capitalized
### Custom Objects
```
"sObjectType": "Custom_Object__c"
```
- Custom Salesforce objects
- API name with "__c" suffix
- Case-sensitive, including underscores
- |-
## RELATIONSHIP COMPATIBILITY
The sObjectType must:
- Have a relationship field to the parent object
- Be accessible to the connected user
- Support standard SOQL queries
IMPORTANT: Use the exact API name of the object, not its label.
This value is case-sensitive and must match Salesforce's naming exactly.
examples: ["Contact", "OpportunityLineItem", "Custom_Child__c", "Related_Object__c"]
filter:
type: string
description: |-
SOQL condition that limits which child records are included; when omitted, all related children are retrieved. Provide only the condition expression without the `WHERE` keyword — it is combined automatically with the parent relationship filter.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR
This field adds filtering to child record retrieval:
- OPTIONAL: If omitted, all related child records are included
- Contains only the condition expression (without "WHERE" keyword)
- Uses standard SOQL syntax for conditions
- Applied in addition to the parent relationship filter
- |-
## FILTERING PATTERNS
### Simple Condition
```
"filter": "IsActive = true"
```
- Basic field comparison
- Only active related records are included
### Multiple Conditions
```
"filter": "Status__c = 'Open' AND Priority = 'High'"
```
- Combined conditions with logical operators
- Only records matching all conditions are included
### Complex Filtering
```
"filter": "CreatedDate > LAST_N_DAYS:30 OR IsClosed = false"
```
- Can use Salesforce date literals and functions
- Can mix different types of conditions
- |-
## TECHNICAL DETAILS
The system appends this to the automatically generated relationship query:
```
SELECT [fields] FROM [sObjectType]
WHERE [parentField] = [parent ID] AND ([filter])
```
IMPORTANT: Do not include the "WHERE" keyword in this field.
Only include the condition expression itself, as it will be combined
with the parent relationship condition automatically.
examples: ["IsActive = true", "Amount > 0", "Status__c = 'Open'", "CreatedDate > LAST_N_DAYS:30"]
orderBy:
type: string
description: |-
Sort order for the retrieved child records, as field names with optional `ASC`/`DESC` directions (e.g. `CreatedDate DESC`). Provide only the fields and directions without the `ORDER BY` keywords. When omitted, Salesforce determines the order.
x-celigo-ai-guidance:
- |-
## TECHNICAL DETAILS
The system appends this to the automatically generated relationship query:
```
SELECT [fields] FROM [sObjectType]
WHERE [parentField] = [parent ID]
ORDER BY [orderBy]
```
IMPORTANT: Do not include the "ORDER BY" keywords in this field.
Only include the field names and sort directions, as they will be
added to the query with the proper syntax automatically.
- |-
## FIELD BEHAVIOR
This field controls child record ordering:
- OPTIONAL: If omitted, order is determined by Salesforce
- Contains only field and direction (without "ORDER BY" keywords)
- Uses standard SOQL syntax for sorting
- Applied to the child records query
- |-
## ORDERING PATTERNS
### Single Field Ascending (Default)
```
"orderBy": "Name"
```
- Sorts by a single field in ascending order
- ASC is implied if not specified
### Single Field Descending
```
"orderBy": "CreatedDate DESC"
```
- Sorts by a single field in descending order
- Must explicitly specify DESC
### Multiple Fields
```
"orderBy": "Priority DESC, CreatedDate ASC"
```
- Sorts by multiple fields in specified directions
- Comma-separated list of fields with optional directions
examples: ["CreatedDate DESC", "Name ASC", "LineNumber", "Priority DESC, CreatedDate ASC"]
required:
- parentField
- referencedFields
- sObjectType
# Flavor gating:
# - `type: distributed` docs carry `distributed` + `sObjectType`.
# - Many real-time exports predate `salesforce.type`: a `distributed` object with
# no `type` at all still runs, so its presence alone selects the distributed flavor.
# - Blob file exports carry `id` (+ `sObjectType`) and no `soql` even when a stray
# `type: soql` is present, so `soql` must not be demanded for them.
# - Metadata-only stubs (`{metadata: ...}` written onto non-Salesforce exports)
# carry no flavor keys; requiring `type`/`soql` on them would reject server-written
# documents, so the soql-flavor requireds are stub-gated.
if:
properties:
type:
const: distributed
required:
- type
then:
required:
- distributed
- sObjectType
else:
if:
required:
- distributed
then: {}
else:
if:
required:
- id
then:
required:
- sObjectType
else:
if:
not:
propertyNames:
enum:
- metadata
then:
required:
- soql
- type
references/schemas/test.yml
Test:
type: object
description: |-
Configures test exports that cap the number of records retrieved, for safely developing and
validating against small data samples. Required when the export's type is "test"; omit for
other export types. A test export behaves like a standard export in every other way (filters,
pagination, processing) and stores no state between runs.
properties:
limit:
type: integer
default: 1
x-celigo-ui-override: >-
Required by the export form when the export type is "test" (test.limit is required:true,
default 1, visibleWhen type=test). Encoded to mirror the form so builders produce
connectable configurations.
description: |-
Caps the total records a test run processes, counted on top-level records before
oneToMany processing; the maximum exists to prevent accidentally processing large
datasets during development. The cap applies across pages — processing stops once the
limit is reached regardless of pageSize. When transitioning to production, leave this in
place and change the export's type field; the limit then no longer applies.
minimum: 1
maximum: 100
examples: [10, 25, 50]
references/schemas/webhook.yml
Webhook:
type: object
x-celigo-canon:
decision: verified-exact
reason: >-
The per-verify-method secret requireds (key/token/password) stay despite universal
absence in the warehouse — secrets are masked at rest, never persisted in clear, so
stored-doc absence is a masking artifact, not spec drift. Do not relax.
method: full-population
verified: '2026-07-03'
description: |-
Configuration for webhook listeners that receive data through incoming HTTP requests. Required when the export's `type` is `webhook`; omit for other export types. The platform generates a unique endpoint URL, validates each incoming request using the configured `verify` method, passes the payload to subsequent flow steps, and returns a configurable HTTP response to the caller.
x-celigo-ai-guidance:
- Configuration object for real-time event listeners that receive data via incoming HTTP requests.
- |-
This object is REQUIRED when the export's type field is set to "webhook" and
should not be included for other export types.
Webhook exports create dedicated HTTP endpoints that can receive data from
external systems in real-time,
enabling event-driven integration architectures.
- 'For optimal AI agent implementation, consider these guidelines:'
- |-
## WEBHOOK SECURITY MODELS
Webhooks support multiple security verification methods, each requiring different fields:
1. **HMAC Verification** (Most secure, recommended for production)
- Required fields: verify="hmac", key, algorithm, encoding, header
- Verifies a cryptographic signature included with each request
- Ensures data integrity and authenticity
2. **Token Verification** (Simple shared secret)
- Required fields: verify="token", token, path
- Checks for a specific token value in the request
- Simpler but less secure than HMAC
3. **Basic Authentication** (HTTP standard)
- Required fields: verify="basic", username, password
- Uses HTTP Basic Authentication headers
- Compatible with most HTTP clients
4. **Secret URL** (Simplest but least secure)
- Required fields: verify="secret_url", token
- Relies solely on URL obscurity for security
- The token is embedded in the webhook URL to create a unique, hard-to-guess endpoint
- Suitable only for non-sensitive data or testing
5. **Public Key** (Advanced, for specific providers)
- Required fields: verify="publickey", key
- Uses public key cryptography for verification
- Only available for certain providers
- |-
## RESPONSE CUSTOMIZATION
You can customize how the webhook responds to callers with these field groups:
1. **Standard Success Response**
- Fields: successStatusCode, successBody, successMediaType, successResponseHeaders
- Controls how the webhook responds to valid requests
2. **Challenge Response** (For subscription verification)
- Fields: challengeSuccessBody, challengeSuccessStatusCode, challengeSuccessMediaType, challengeResponseHeaders
- Controls how the webhook responds to verification/challenge requests
- |-
## IMPLEMENTATION SCENARIOS
Webhooks are commonly used for:
1. **Real-time data synchronization**
- E-commerce platforms sending order notifications
- CRM systems delivering contact updates
- Payment processors reporting transaction events
2. **Event-driven processes**
- Triggering fulfillment when orders are placed
- Initiating approval workflows on document submissions
- Executing business logic when status changes occur
3. **System integration**
- Connecting SaaS applications without polling
- Building composite applications from microservices
- Creating fan-out architectures for event distribution
properties:
provider:
type: string
description: |-
Source application sending the webhook, used to pre-configure security settings and payload parsing for that platform. Use `custom` (the default when omitted) for unlisted providers or when you need full manual control over the security configuration. Provider-specific choices may require credentials (tokens, keys) mandated by that platform.
x-celigo-ai-guidance:
- |-
Specifies the source application sending webhook data,
enabling platform-specific optimizations.
- |-
## FIELD BEHAVIOR
This field determines how the webhook handles incoming requests:
- OPTIONAL: Defaults to "custom" if not specified
- When a specific provider is selected, the system:
1. Pre-configures appropriate security settings for that platform
2. Applies platform-specific payload parsing rules
3. May enable additional features only relevant to that provider
- |-
## IMPLEMENTATION GUIDANCE
### Provider-specific configurations
When you know the exact source system, select its specific provider:
```
"provider": "shopify"
```
- Automatically configures proper HMAC verification settings
- Optimizes payload parsing for Shopify's webhook format
- May enable additional Shopify-specific features
### Custom configuration
For generic webhooks or unlisted providers, use custom:
```
"provider": "custom"
```
- Requires manual configuration of all security settings
- Maximum flexibility for handling any webhook format
- Recommended for custom applications or newer platforms
### Selection criteria
Choose a specific provider when:
- The source system is explicitly listed in the enum values
- You want to leverage pre-configured settings
- The integration must follow platform-specific practices
Choose "custom" when:
- The source system is not listed
- You need full control over webhook configuration
- You're building a custom interface or protocol
IMPORTANT: Some providers enforce specific security methods. When selecting a
provider, ensure you have the necessary security credentials (tokens, keys, etc.)
as required by that platform.
enum:
- github
- shopify
- travis
- travis-org
- slack
- dropbox
- onfleet
- helpscout
- errorception
- box
- stripe
- aha
- jira
- pagerduty
- postmark
- mailchimp
- intercom
- activecampaign
- segment
- recurly
- shipwire
- surveymonkey
- parseur
- mailparser-io
- hubspot
- integrator-extension
- custom
- sapariba
- happyreturns
- typeform
x-enumDescriptions:
github: GitHub repository event webhooks with HMAC-SHA verification.
shopify: Shopify store event webhooks with HMAC-SHA256 verification.
travis: Travis CI build event webhooks (travis-ci.com).
travis-org: Travis CI build event webhooks (travis-ci.org).
slack: Slack workspace event webhooks with token verification.
dropbox: Dropbox file change notification webhooks.
onfleet: Onfleet delivery management event webhooks.
helpscout: Help Scout customer support event webhooks.
errorception: Errorception error tracking event webhooks.
box: Box cloud storage event webhooks.
stripe: Stripe payment event webhooks with signature verification.
aha: Aha! product management event webhooks.
jira: Jira issue tracking event webhooks.
pagerduty: PagerDuty incident management event webhooks.
postmark: Postmark email delivery event webhooks.
mailchimp: Mailchimp email marketing event webhooks.
intercom: Intercom customer messaging event webhooks.
activecampaign: ActiveCampaign marketing automation event webhooks.
segment: Segment analytics event webhooks.
recurly: Recurly subscription billing event webhooks.
shipwire: Shipwire fulfillment event webhooks.
surveymonkey: SurveyMonkey survey response event webhooks.
parseur: Parseur email parsing event webhooks.
mailparser-io: Mailparser.io email parsing event webhooks.
hubspot: HubSpot CRM event webhooks.
integrator-extension: Celigo integrator extension internal event webhooks.
custom: Generic webhook with fully manual security and payload configuration.
sapariba: SAP Ariba procurement event webhooks.
happyreturns: Happy Returns return management event webhooks.
typeform: Typeform form submission event webhooks.
examples: ["shopify", "stripe", "custom"]
verify:
type: string
x-celigo-ui-override: >-
Required by the webhook export form (verify is required:true). Encoded to mirror the form
so builders produce connectable configurations.
description: |-
Verification method applied to every incoming request before processing; required for all webhook exports. Each method needs companion fields: `hmac` requires `key`, `algorithm`, `encoding`, and `header` (except on connector-backed webhooks, where the connector definition supplies them); `token` requires `token`, with `tokenLocation` defaulting to `body` and selecting which location-specific field applies; `basic` requires `username` and `password`; `secret_url` requires `token`. Prefer `hmac` when the source system supports it — it is the most secure option; `secret_url` relies only on URL obscurity and suits non-sensitive data or testing.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR
This field is the primary control for webhook security:
- REQUIRED for all webhook exports
- Determines which additional security fields must be configured
- Controls how incoming requests are validated before processing
- |-
## VERIFICATION METHODS
### HMAC Verification
```
"verify": "hmac"
```
- Most secure method, cryptographically verifies request integrity
- REQUIRES: key, algorithm, encoding, header fields
- Validates a cryptographic signature included in the request header
- Works well with providers that support HMAC (Shopify, Stripe, GitHub, etc.)
### Token Verification
```
"verify": "token"
```
- Simple verification using a shared secret token
- REQUIRES: token, path fields
- Checks for a specific token value in the request body or query params
- Good for simple scenarios with trusted networks
### Basic Authentication
```
"verify": "basic"
```
- Standard HTTP Basic Authentication
- REQUIRES: username, password fields
- Validates credentials sent in the Authorization header
- Compatible with most HTTP clients and tools
### Public Key
```
"verify": "publickey"
```
- Advanced verification using public key cryptography
- REQUIRES: key field (containing the public key)
- Only available for certain providers that use asymmetric cryptography
- Highest security level but more complex to configure
### Secret URL
```
"verify": "secret_url"
```
- Simplest method, relies solely on the obscurity of the URL
- REQUIRES: token field (the token is embedded in the webhook URL to create a unique, hard-to-guess endpoint)
- Only suitable for non-sensitive data or testing environments
- Not recommended for production use with sensitive data
IMPORTANT: Choose the security method that matches your source system's capabilities.
If the source system supports multiple verification methods, HMAC is generally the
most secure option.
enum: ["token", "hmac", "basic", "secret_url"]
x-enumDescriptions:
token: Verifies requests by matching a shared secret token found at a specified location in the request.
hmac: Verifies requests using a cryptographic HMAC signature sent in a request header.
basic: Verifies requests using standard HTTP Basic Authentication (username and password).
secret_url: Relies on a unique, hard-to-guess token embedded in the webhook URL for security.
examples: ["token", "hmac"]
token:
type: string
description: |-
Shared secret used when `verify` is `token` or `secret_url`. For `token` verification, the value found at `path` in each request must exactly match (case- and whitespace-sensitive) or the request is rejected with a 401. For `secret_url`, the token is embedded in the webhook URL to create a hard-to-guess endpoint — generate a random, high-entropy value and treat it as a sensitive credential.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR
This field defines the expected token value:
- REQUIRED when verify="token" or verify="secret_url"
- When verify="token": must be a string that exactly matches what the sender will provide. Used with the path field to locate and validate the token in the request.
- When verify="secret_url": the token is embedded in the webhook URL to create a unique, hard-to-guess endpoint. Generate a random, high-entropy value.
- Case-sensitive and whitespace-sensitive
- |-
## IMPLEMENTATION GUIDANCE
The token verification flow works as follows:
1. The webhook receives an incoming request
2. The system looks for the token at the location specified by the path field
3. If the found value exactly matches this token value, the request is processed
4. If no match is found, the request is rejected with a 401 error
### Security best practices
For maximum security:
- Use a random, high-entropy token (32+ characters)
- Include a mix of uppercase, lowercase, numbers, and special characters
- Don't use predictable values like company names or common words
- Rotate tokens periodically for sensitive integrations
### Common implementations
```
"token": "3a7c4f8b2e9d1a5c6b3e7d9f2a1c5b8e"
```
```
"token": "whsec_8fb2e91a5c6b3e7d9f2a1c5b8e3a7c4f"
```
IMPORTANT: Never share this token in public repositories or documentation.
Treat it as a sensitive credential similar to a password.
examples: ["verification_token_abcdef", "whsec_8fb2e91a5c6b3e7d9f2a1c5b8e3a7c4f"]
algorithm:
type: string
description: |-
Hashing algorithm used to validate HMAC signatures when `verify` is `hmac`. Must match the algorithm the webhook sender uses — a mismatch causes every request to be rejected. Use `sha256` unless the provider explicitly requires another value.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR
This field determines how signatures are validated:
- REQUIRED when verify="hmac"
- Must match the algorithm used by the webhook sender
- Affects security strength and compatibility
- |-
## ALGORITHM SELECTION
### SHA-256 (Recommended)
```
"algorithm": "sha256"
```
- Modern, secure hash algorithm
- Industry standard for most new webhook implementations
- Preferred choice for all new integrations
- Used by Shopify, Stripe, and many modern platforms
### SHA-1 (Legacy)
```
"algorithm": "sha1"
```
- Older, less secure algorithm
- Still used by some legacy systems
- Only select if the provider explicitly requires it
- GitHub webhooks used this historically
### SHA-384/SHA-512 (High Security)
```
"algorithm": "sha384"
"algorithm": "sha512"
```
- Higher security variants with longer digests
- Use when specified by the provider or for sensitive data
- Less common but supported by some security-focused systems
IMPORTANT: This MUST match the algorithm used by the webhook sender.
Mismatched algorithms will cause all webhook requests to be rejected.
enum: ["sha1", "sha256", "sha384", "sha512"]
x-enumDescriptions:
sha1: Uses the SHA-1 hashing algorithm for HMAC signature verification (legacy, less secure).
sha256: Uses the SHA-256 hashing algorithm for HMAC signature verification (recommended).
sha384: Uses the SHA-384 hashing algorithm for HMAC signature verification (higher security, longer digest).
sha512: Uses the SHA-512 hashing algorithm for HMAC signature verification (highest security, longer digest).
examples: ["sha256", "sha1"]
encoding:
type: string
description: |-
Encoding of the HMAC signature value when `verify` is `hmac`. Must match the encoding the webhook sender uses — a mismatch causes requests to be rejected even when the signature is otherwise correct.
x-celigo-ai-guidance:
- Specifies the encoding format used for the HMAC signature in webhook requests.
- |-
## FIELD BEHAVIOR
This field determines how signature values are encoded:
- REQUIRED when verify="hmac"
- Must match the encoding used by the webhook sender
- Affects how binary signature values are represented as strings
- |-
## ENCODING OPTIONS
### Hexadecimal (hex)
```
"encoding": "hex"
```
- Represents the signature as a string of hexadecimal characters (0-9, a-f)
- Most common encoding for web-based systems
- Used by many platforms including Stripe and some Shopify implementations
- Example output: "8f7d56a32e1c9b47d882e3aa91341f64"
### Base64
```
"encoding": "base64"
```
- Represents the signature using base64 encoding
- More compact than hex (about 33% shorter)
- Used by platforms like Shopify (newer implementations) and some GitHub scenarios
- Example output: "j31WozbhtrHYeC46qRNB9k=="
IMPORTANT: This MUST match the encoding used by the webhook sender.
Mismatched encoding will cause all webhook requests to be rejected even if
the signature is mathematically correct.
enum: ["hex", "base64"]
x-enumDescriptions:
hex: The HMAC signature is encoded as a hexadecimal string.
base64: The HMAC signature is encoded as a Base64 string.
examples: ["hex", "base64"]
key:
type: string
description: |-
Secret used to validate signatures when `verify` is `hmac`. The system computes a signature of the request body using this key and the configured `algorithm`, then compares it with the signature sent in the request header named by `header`. Treat it as a highly sensitive credential — never expose it in repositories or logs.
x-celigo-ai-guidance:
- Specifies the secret key used to verify cryptographic signatures in incoming webhooks.
- |-
## FIELD BEHAVIOR
This field provides the shared secret for signature verification:
- REQUIRED when verify="hmac" or verify="publickey"
- Contains the secret value known to both sender and receiver
- Used with the incoming payload to validate the signature
- Highly sensitive security credential
- |-
## IMPLEMENTATION GUIDANCE
### For HMAC verification
The key is used in the following verification process:
1. The webhook receives an incoming request with a signature
2. The system computes an HMAC of the request body using this key and the specified algorithm
3. This computed signature is compared with the signature from the request header
4. If they match exactly, the request is authenticated and processed
### Security best practices
For maximum security:
- Use a random, high-entropy key (32+ characters)
- Include a mix of characters and avoid dictionary words
- Never share this key in code repositories or logs
- Rotate keys periodically for sensitive integrations
- Use environment variables or secure credential storage
### Common implementations
```
"key": "whsec_3a7c4f8b2e9d1a5c6b3e7d9f2a1c5b8e3a7c4f8b"
```
```
"key": "sk_test_51LZIr9B9Y6YIwSKx8647589JKhdjs889KJsk389"
```
IMPORTANT: This key should be treated as a highly sensitive credential,
similar to a private key or password. It should never be exposed publicly
or logged in application logs.
examples: ["whsec_3a7c4f8b2e9d1a5c6b3e7d9f2a1c5b8e3a7c4f8b", "sk_test_51LZIr9B9Y6YIwSKx8647589JKhdjs889KJsk389"]
header:
type: string
description: |-
Name of the request header that carries the HMAC signature when `verify` is `hmac`. Must match the header name the webhook sender uses (header names are case-insensitive); requests without this header are rejected with a 401. Signature prefixes in the header value (such as `sha256=`) are handled automatically.
x-celigo-ai-guidance:
- Specifies the HTTP header name that contains the signature for HMAC verification.
- |-
## FIELD BEHAVIOR
This field identifies where to find the signature in incoming requests:
- REQUIRED when verify="hmac"
- Must exactly match the header name used by the webhook sender
- Case-insensitive (HTTP headers are not case-sensitive)
- |-
## COMMON HEADER PATTERNS
### Platform-specific headers
Many platforms use standardized header names for their signatures:
```
"header": "X-Shopify-Hmac-SHA256" // For Shopify webhooks
```
```
"header": "X-Hub-Signature-256" // For GitHub webhooks
```
```
"header": "Stripe-Signature" // For Stripe webhooks
```
### Generic signature headers
For custom implementations or less common platforms:
```
"header": "X-Webhook-Signature" // Common generic format
```
```
"header": "X-Signature" // Simplified format
```
- |-
## IMPLEMENTATION NOTES
- The system will look for this exact header name in incoming requests
- If the header is not found, the request will be rejected with a 401 error
- Some platforms may include a prefix in the header value (e.g., "sha256=")
which is handled automatically by the system
IMPORTANT: This must exactly match the header name used by the webhook sender.
If you're unsure about the correct header name, consult the sender's documentation
or use a tool like cURL with verbose output to inspect an example request.
examples: ["X-Webhook-Signature", "X-Shopify-Hmac-SHA256", "Stripe-Signature", "X-Hub-Signature-256"]
tokenLocation:
type: string
enum: ["body", "header", "queryParam"]
x-enumDescriptions:
body: The verification token is read from the request body at `path`.
header: The verification token is read from the request header named by `tokenHeaderName`.
queryParam: The verification token is read from the query parameter named by `tokenQueryParamName`.
default: body
description: |-
Where the verification token is found when `verify` is `token`. Each location uses a different companion field: `body` uses `path`, `header` uses `tokenHeaderName` (and `tokenHeaderScheme`), `queryParam` uses `tokenQueryParamName`.
examples: ["body", "header", "queryParam"]
path:
type: string
description: |-
JSON path into the request body holding the verification token when `verify` is `token` and `tokenLocation` is `body` (e.g. `meta.token`). The value at this path must exactly match `token` or the request is rejected.
x-celigo-ai-guidance:
- Specifies the location of the verification token in incoming webhook requests.
- |-
## FIELD BEHAVIOR
This field determines where to find the token for verification:
- REQUIRED when verify="token"
- Defines a JSON path to locate the token in the request body
- For query parameters, use the appropriate path format (typically at root level)
- |-
## IMPLEMENTATION PATTERNS
### Token in request body
For tokens embedded in JSON payloads:
```
"path": "meta.token" // For { "meta": { "token": "xyz123" } }
```
```
"path": "verification.key" // For { "verification": { "key": "xyz123" } }
```
### Token at root level
For tokens in the top level of the request:
```
"path": "token" // For { "token": "xyz123", "data": {...} }
```
### Token in query parameters
For tokens sent as URL query parameters, use the parameter name:
```
"path": "verify_token" // For /webhook?verify_token=xyz123
```
- |-
## VERIFICATION PROCESS
1. The webhook receives an incoming request
2. The system uses this path to extract the token value
3. The extracted value is compared with the configured token
4. If they match exactly, the request is processed
IMPORTANT: The path is case-sensitive and must exactly match the structure
of incoming requests. For query parameters, the system automatically checks
both the body and query string using the provided path.
examples: ["token", "verification_token", "meta.security.token", "auth.key"]
tokenHeaderName:
type: string
description: Request header carrying the verification token when `verify` is `token` and `tokenLocation` is `header`.
examples: ["X-Webhook-Token", "Authorization"]
tokenHeaderScheme:
type: string
enum: ["bearer", "custom", "none"]
x-enumDescriptions:
bearer: The header value is prefixed with `Bearer ` before the token.
custom: The header value uses the custom prefix named in `customTokenScheme`.
none: The header value is the bare token with no scheme prefix.
default: bearer
description: Scheme prefixing the token in the header when `tokenLocation` is `header`.
examples: ["bearer", "none"]
customTokenScheme:
type: string
description: Custom header scheme prefix used when `tokenHeaderScheme` is `custom`.
examples: ["Token"]
tokenQueryParamName:
type: string
description: Query parameter carrying the verification token when `verify` is `token` and `tokenLocation` is `queryParam`.
examples: ["token", "verify_token"]
_httpConnectorId:
type: string
format: objectId
x-celigo-refModel: httpconnectors
description: HTTP connector backing this listener when the webhook is provided by an assistant/connector (e.g. Slack); set by the platform for connector-backed listeners.
requestMediaType:
type: string
enum: ["json", "xml", "csv", "urlencoded", "plaintext"]
x-enumDescriptions:
json: Parse the incoming request body as JSON.
xml: Parse the incoming request body as XML.
csv: Parse the incoming request body as CSV.
urlencoded: Parse the incoming request body as URL-encoded form data.
plaintext: Treat the incoming request body as plain text.
description: Overrides how the incoming request body is parsed when the provider sends a non-standard content type. When omitted, the body is parsed by its Content-Type header.
examples: ["json", "csv"]
pathToRecords:
type: string
description: JSON path into the incoming payload to the array of records to emit; when omitted, the whole payload is emitted as a single record.
examples: ["events", "data.items"]
includeParentData:
type: boolean
description: When true and `pathToRecords` is set, each emitted record also carries the surrounding parent fields from the payload.
username:
type: string
description: |-
Username half of the credentials when `verify` is `basic`. Incoming requests must include an `Authorization: Basic` header carrying the base64-encoded `username:password` pair. Use only over HTTPS to prevent credential interception.
x-celigo-ai-guidance:
- Specifies the username for webhook HTTP Basic Authentication security.
- |-
## FIELD BEHAVIOR
This field defines one half of the Basic Authentication credentials:
- REQUIRED when verify="basic"
- Used in conjunction with the password field
- Case-sensitive string value
- Encoded in the standard HTTP Basic Authentication format
- |-
## IMPLEMENTATION NOTES
When Basic Authentication is used, the webhook requires incoming requests to include
an Authorization header containing "Basic " followed by a base64-encoded string of
"username:password".
Example header:
```
Authorization: Basic d2ViaG9va191c2VyOndlYmhvb2tfcGFzc3dvcmQ=
```
Where "d2ViaG9va191c2VyOndlYmhvb2tfcGFzc3dvcmQ=" is the base64 encoding of
"webhook_user:webhook_password".
### Security considerations
Basic Authentication:
- Is widely supported by HTTP clients and servers
- Should ONLY be used over HTTPS to prevent credential interception
- Provides a simple authentication mechanism but without integrity verification
- Is less secure than HMAC verification for webhook scenarios
IMPORTANT: Always use strong, unique credentials rather than generic or easily
guessable values. Basic Authentication is less secure than HMAC for webhooks
but can be appropriate for simple scenarios or when working with systems that
don't support more advanced verification methods.
examples: ["webhook_user", "api_client_123", "integration_user"]
password:
type: string
description: |-
Password half of the credentials when `verify` is `basic`, validated together with `username` from the request's `Authorization` header. Use a strong, unique value and treat it as a sensitive credential.
x-celigo-ai-guidance:
- Specifies the password for webhook HTTP Basic Authentication security.
- |-
## FIELD BEHAVIOR
This field defines the second half of the Basic Authentication credentials:
- REQUIRED when verify="basic"
- Used in conjunction with the username field
- Case-sensitive string value
- Encoded in the standard HTTP Basic Authentication format
- |-
## IMPLEMENTATION NOTES
This password is combined with the username and encoded in base64 format for
the HTTP Authorization header. The webhook verifies that incoming requests contain
the correct encoded credentials before processing them.
### Security best practices
For maximum security:
- Use a strong, randomly generated password (16+ characters)
- Include a mix of uppercase, lowercase, numbers, and special characters
- Don't reuse passwords from other systems
- Avoid dictionary words or predictable patterns
- Rotate passwords periodically for sensitive integrations
IMPORTANT: This password should be treated as a sensitive credential.
Never share it in public repositories, documentation, or logs. Always use
HTTPS for webhooks using Basic Authentication to prevent credential interception.
examples: ["xC7!rTp2@bN9$mQ5", "webhook_secure_pass_123"]
successStatusCode:
type: integer
description: |-
HTTP status code returned to the caller after successful processing; must be a valid 2xx code. The default 204 returns no response body and causes `successBody` to be ignored — set 200 or 202 when the caller needs a response body or expects a specific code.
x-celigo-ai-guidance:
- Specifies the HTTP status code sent back to webhook callers after successful processing.
- |-
## FIELD BEHAVIOR
This field controls the HTTP response status code:
- OPTIONAL: Defaults to 204 (No Content) if not specified
- Affects how webhook callers interpret the success response
- Must be a valid HTTP status code in the 2xx range
- |-
## COMMON STATUS CODES
### 204 No Content (Default)
```
"successStatusCode": 204
```
- Returns no response body
- Most efficient option as it minimizes response size
- Appropriate when the caller doesn't need confirmation details
- Automatically disables successBody (even if specified)
### 200 OK
```
"successStatusCode": 200
```
- Standard success response
- Allows returning a response body with details
- Most widely used and recognized success code
- Compatible with all HTTP clients
### 202 Accepted
```
"successStatusCode": 202
```
- Indicates request was accepted for processing but may not be complete
- Appropriate for asynchronous processing scenarios
- Signals that the webhook was received but full processing is pending
- |-
## IMPLEMENTATION CONSIDERATIONS
The appropriate status code depends on your webhook caller's expectations:
- Some systems require specific status codes to consider the delivery successful
- If the caller retries on anything other than 2xx, use 200 or 202
- If the caller needs confirmation details, use 200 with a response body
- If efficiency is paramount, use 204 (default)
IMPORTANT: When using 204 No Content, any successBody configuration will be ignored
as this status code specifically indicates no response body is being returned.
default: 204
examples: [200, 201, 202, 204]
successBody:
type: string
description: |-
Response body returned to the caller after successful processing; ignored when `successStatusCode` is 204. Content type is set by `successMediaType`, and the value can be static text or structured JSON/XML, including handlebars expressions for dynamic values.
x-celigo-ai-guidance:
- Specifies the HTTP response body sent back to webhook callers after successful processing.
- |-
## FIELD BEHAVIOR
This field controls the content returned to the webhook caller:
- OPTIONAL: Defaults to empty (no body) if not specified
- Ignored when successStatusCode is 204 (No Content)
- Content type is determined by the successMediaType field
- Can contain static text or structured data (JSON, XML)
- |-
## IMPLEMENTATION PATTERNS
### Simple acknowledgment
```
"successBody": "OK"
```
- Minimal plaintext response
- Confirms receipt without details
- Most efficient for basic acknowledgment
### Structured response (JSON)
```
"successBody": "{\"success\":true,\"message\":\"Webhook received\"}"
```
- Provides structured data about the result
- Can include more detailed status information
- Compatible with programmatic processing by the caller
- Remember to escape quotes in JSON strings
### Custom confirmation
```
"successBody": "{\"status\":\"received\",\"timestamp\":\"{{currentDateTime}}\"}"
```
- Can include dynamic values using handlebars templates
- Useful for providing receipt confirmation with metadata
- |-
## RESPONSE FLOW
The response body is sent after the webhook payload has been:
1. Received and authenticated
2. Validated against any configured requirements
3. Accepted for processing by the system
IMPORTANT: The successBody will only be returned if successStatusCode is NOT 204.
If you want to return a body, make sure to set successStatusCode to 200, 201, or 202.
examples: ["{\"success\":true}", "OK", "{\"status\":\"received\",\"timestamp\":\"{{currentDateTime}}\"}"]
x-celigo-agent:
- handlebars
successMediaType:
type: string
description: |-
Sets the Content-Type header on successful webhook responses. Only takes effect when a `successBody` is returned (a status code other than 204), and must match the actual format of that body.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR
This field controls how the response body is interpreted:
- OPTIONAL: Defaults to "json" if not specified
- Only relevant when returning a successBody and not using status code 204
- Determines the Content-Type header in the HTTP response
- Must be consistent with the actual format of the successBody
- |-
## MEDIA TYPE OPTIONS
### JSON (Default)
```
"successMediaType": "json"
```
- Sets Content-Type: application/json
- Use when successBody contains valid JSON
- Most common for API responses
- Allows structured data that clients can parse programmatically
### XML
```
"successMediaType": "xml"
```
- Sets Content-Type: application/xml
- Use when successBody contains valid XML
- Necessary for systems expecting XML responses
- Less common in modern APIs but still used in some enterprise systems
### Plain Text
```
"successMediaType": "plaintext"
```
- Sets Content-Type: text/plain
- Use for simple string responses
- Most compatible option for basic acknowledgments
- Appropriate when successBody is just "OK" or similar
- |-
## IMPLEMENTATION CONSIDERATIONS
- The media type must match the actual content format in successBody
- If returning JSON in successBody, use "json" (most common)
- If returning a simple text acknowledgment, use "plaintext"
- If the caller specifically requires XML, use "xml"
IMPORTANT: When successStatusCode is 204 (No Content), this field has no effect
since no body is returned, and therefore no Content-Type is needed.
default: "json"
enum: ["json", "xml", "plaintext"]
x-enumDescriptions:
json: "Sets the success response Content-Type to application/json."
xml: "Sets the success response Content-Type to application/xml."
plaintext: "Sets the success response Content-Type to text/plain."
examples: ["json", "plaintext", "xml"]
successResponseHeaders:
type: array
description: |-
Custom headers added to successful webhook responses — for example CORS headers or correlation IDs. Headers defined here take precedence over automatically set headers such as Content-Type, and values support handlebars expressions for dynamic content.
x-celigo-ai-guidance:
- Defines custom HTTP headers to include in successful webhook responses.
- |-
## FIELD BEHAVIOR
This field allows additional HTTP headers in the response:
- OPTIONAL: If omitted, only standard headers are included
- Each entry defines a name/value pair for a single header
- Applied to all successful responses (regardless of status code)
- Can override standard headers like Content-Type
- |-
## IMPLEMENTATION PATTERNS
### Standard use cases
Custom headers are useful for:
- Providing metadata about the response
- Enabling CORS for browser-based webhook callers
- Including tracking or correlation IDs
- Adding custom security headers
### Common header examples
CORS support:
```json
[
{"name": "Access-Control-Allow-Origin", "value": "*"},
{"name": "Access-Control-Allow-Methods", "value": "POST, OPTIONS"}
]
```
Request tracking:
```json
[
{"name": "X-Request-ID", "value": "{{jobId}}"},
{"name": "X-Webhook-Received", "value": "{{currentDateTime}}"}
]
```
Custom application headers:
```json
[
{"name": "X-API-Version", "value": "1.0"},
{"name": "X-Processing-Status", "value": "accepted"}
]
```
- |-
## TECHNICAL DETAILS
- Header names are case-insensitive as per HTTP specification
- Some headers like Content-Type can be set via other fields (successMediaType)
- Headers defined here take precedence over automatically set headers
- The values can contain handlebars expressions for dynamic content
IMPORTANT: Be careful when setting security-related headers like
Access-Control-Allow-Origin, as improper values could create security vulnerabilities.
items:
type: object
properties:
name:
type: string
description: Name of the header to set on successful webhook responses; headers defined here take precedence over automatically set headers such as Content-Type.
examples: ["Content-Type", "X-Webhook-Received", "Access-Control-Allow-Origin"]
value:
type: string
description: Value sent for the header; supports handlebars expressions for dynamic content.
examples: ["application/json", "true", "*"]
x-celigo-agent:
- handlebars
challengeResponseHeaders:
type: array
description: |-
Custom headers returned for webhook subscription verification (challenge) requests, which providers send before delivering real events. Required header values vary by provider — consult the provider's documentation, since incorrect challenge headers can prevent the subscription from being verified.
x-celigo-ai-guidance:
- Defines custom HTTP headers to include in webhook challenge responses.
- |-
## FIELD BEHAVIOR
This field configures headers for subscription verification:
- OPTIONAL: If omitted, only standard headers are included
- Only used for webhook verification/challenge requests
- Each entry defines a name/value pair for a single header
- Particularly important for platforms requiring specific verification headers
- |-
## CHALLENGE VERIFICATION CONTEXT
Many webhook providers implement a verification process:
1. Before sending real events, they send a "challenge" request
2. The webhook must respond with specific headers and/or body content
3. Only after successful verification will real webhook events be sent
This field allows customizing the headers sent during this verification step.
- |-
## COMMON PATTERNS BY PLATFORM
### Facebook/Instagram
```json
[
{"name": "Content-Type", "value": "text/plain"}
]
```
### Slack
```json
[
{"name": "Content-Type", "value": "application/json"}
]
```
### Custom implementations
```json
[
{"name": "X-Challenge-Response", "value": "passed"},
{"name": "X-Verification-Status", "value": "success"}
]
```
IMPORTANT: The specific headers required vary by platform. Consult the webhook
provider's documentation for the exact verification requirements. Incorrect challenge
response headers may prevent successful webhook subscription.
items:
type: object
properties:
name:
type: string
description: Name of the header to set on challenge (subscription verification) responses.
examples: ["Content-Type", "X-Challenge-Response", "X-Verification-Status"]
value:
type: string
description: Value sent for the header; consult the provider's documentation for required challenge header values.
examples: ["application/json", "passed", "success"]
challengeSuccessBody:
type: string
description: |-
Response body returned for webhook subscription verification (challenge) requests. Many providers require echoing back a challenge value from the request, which handlebars expressions can access — for example `{{hub.challenge}}` (Facebook/Instagram) or `{"challenge":"{{challenge}}"}` (Slack). An incorrect challenge response prevents the subscription from being verified.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR
This field defines the verification response content:
- OPTIONAL: If omitted, a default empty response is sent
- Only used for webhook subscription verification requests
- Content type is determined by the challengeSuccessMediaType field
- Often needs to contain specific values expected by the webhook provider
- |-
## VERIFICATION PATTERNS BY PLATFORM
Different webhook providers implement different verification mechanisms:
### Facebook/Instagram
"challengeSuccessBody": "{{hub.challenge}}"
```
- Must echo back the challenge value sent in the request
- Uses handlebars expression to access the challenge parameter
### Slack
```
"challengeSuccessBody": "{\"challenge\":\"{{challenge}}\"}"
```
- Returns the challenge value in a JSON structure
- Required for Slack's Events API verification
### Generic challenge-response
```
"challengeSuccessBody": "{\"verified\":true,\"timestamp\":\"{{currentDateTime}}\"}"
```
- Simple confirmation response for custom implementations
- Can include additional metadata as needed
## IMPLEMENTATION CONSIDERATIONS
- The exact format is dictated by the webhook provider's requirements
- Some platforms require echoing back specific request parameters
- Others require a structured response with specific fields
- Handlebars expressions ({{variable}}) can access request data
IMPORTANT: Incorrect challenge responses will prevent webhook subscription verification.
Always consult the webhook provider's documentation for exact requirements.
examples: ["{\"challenge\":\"{{challenge}}\"}", "{{hub.challenge}}", "challenge-passed"]
challengeSuccessStatusCode:
type: integer
description: |-
HTTP status code returned for webhook subscription verification (challenge) requests. Most providers expect the default 200; change it only when the provider's verification explicitly requires a different code.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR
This field controls the verification response status:
- OPTIONAL: Defaults to 200 (OK) if not specified
- Only used for webhook subscription verification requests
- Must match what the webhook provider expects for successful verification
- Most platforms require a 200 OK response
- |-
## COMMON STATUS CODES FOR VERIFICATION
### 200 OK (Default)
```
"challengeSuccessStatusCode": 200
```
- Standard success response
- Most webhook platforms expect this status code
- Generally the safest option for verification
### 201 Created
```
"challengeSuccessStatusCode": 201
```
- Used by some systems to indicate subscription was created
- Less common for verification but used in some custom implementations
- |-
## PLATFORM-SPECIFIC REQUIREMENTS
Most major webhook providers require specific status codes:
- Facebook/Instagram: 200
- Slack: 200
- GitHub: 200
- Shopify: 200
- Stripe: 200
IMPORTANT: Using the wrong status code will cause the verification to fail.
If you're unsure, keep the default 200 status code, as it's the most widely
accepted for webhook verifications.
default: 200
examples: [200, 201, 202]
challengeSuccessMediaType:
type: string
description: |-
Sets the Content-Type header on challenge responses. Must match both the format of `challengeSuccessBody` and the provider's requirements — for example, Slack expects `json` while Facebook/Instagram verification expects `plaintext`.
x-celigo-ai-guidance:
- |-
## FIELD BEHAVIOR
This field controls the challenge response format:
- OPTIONAL: Defaults to "json" if not specified
- Only used for webhook subscription verification requests
- Determines the Content-Type header in the verification response
- Must match the format of the challengeSuccessBody content
- |-
## COMMON MEDIA TYPES FOR VERIFICATION
### JSON (Default)
```
"challengeSuccessMediaType": "json"
```
- Sets Content-Type: application/json
- Required by Slack and many modern webhook providers
- Use when returning structured verification data
### Plain Text
```
"challengeSuccessMediaType": "plaintext"
```
- Sets Content-Type: text/plain
- Required by Facebook/Instagram webhook verification
- Use when the challenge response is a simple string
### XML
```
"challengeSuccessMediaType": "xml"
```
- Sets Content-Type: application/xml
- Less common but used by some enterprise systems
- Use only when the webhook provider specifically requires XML
- |-
## PLATFORM-SPECIFIC REQUIREMENTS
- Facebook/Instagram: plaintext (when echoing hub.challenge)
- Slack: json (for Events API verification)
- Most modern APIs: json
IMPORTANT: The media type must match both the format of your challengeSuccessBody
and the requirements of the webhook provider. Mismatched content types can cause
verification to fail even if the response body is correct.
default: "json"
enum: ["json", "xml", "plaintext"]
x-enumDescriptions:
json: "Sets the challenge response Content-Type to application/json."
xml: "Sets the challenge response Content-Type to application/xml."
plaintext: "Sets the challenge response Content-Type to text/plain."
examples: ["json", "plaintext", "xml"]
# hmac companions are form-required only for form-managed webhooks; connector-backed
# webhooks (_httpConnectorId set) store the signing secret per the connector definition.
if:
properties: { verify: { const: hmac } }
required: [verify]
not:
required: [_httpConnectorId]
then: { required: [key, algorithm, encoding, header] }
else:
if: { properties: { verify: { const: token } }, required: [verify] }
then:
# tokenLocation is optional: it was already declared with `default: body` before
# the drift campaign, so requiring it would demand a defaulted field. But every
# location needs its companion — including the implicit body default, hence the
# final else requires `path` when tokenLocation is omitted.
required: [token]
if: { properties: { tokenLocation: { const: body } }, required: [tokenLocation] }
then: { required: [path] }
else:
if: { properties: { tokenLocation: { const: header } }, required: [tokenLocation] }
then: { required: [tokenHeaderName] }
else:
if: { properties: { tokenLocation: { const: queryParam } }, required: [tokenLocation] }
then: { required: [tokenQueryParamName] }
else: { required: [path] }
else:
if: { properties: { verify: { const: basic } }, required: [verify] }
then: { required: [username, password] }
else:
if: { properties: { verify: { const: secret_url } }, required: [verify] }
then: { required: [token] }
references/schemas/wrapper.yml
Wrapper:
type: object
description: |-
Configuration for Wrapper exports, which delegate data retrieval to custom connector code
(typically a stack-hosted function) rather than a built-in adaptor. Required when the
_connectionId field references a wrapper connection.
required:
- function
properties:
function:
type: string
x-celigo-ui-override: >-
Required by the wrapper export form (wrapper.function is required:true). Encoded to mirror
the form so builders produce connectable configurations.
description: |-
Name of the function the wrapper invokes to retrieve records. Must match a callable
function in the wrapper's execution context; names are case-sensitive.
examples: ["getOrdersFromWooCommerce", "processItemShipmentExport", "exportData"]
configuration:
# Empty-array stubs accepted: the server writes the empty-array form when no
# settings exist. A POPULATED array is still rejected (deliberately left failing).
oneOf:
- type: object
additionalProperties: true
- type: array
maxItems: 0
x-celigo-canon:
decision: verified-exact
reason: >-
The empty-array stub is a server-written form; a POPULATED array remains rejected
(single corrupt legacy doc, none recent). Keep the oneOf exactly as encoded.
method: full-population
verified: '2026-07-03'
description: |-
Free-form settings passed to the wrapper function at runtime (connector-specific keys such
as method, apiVersion, headers, relativePath, or body). Structure is defined by the wrapper
code, not by this schema. May be stored as an empty array when no settings exist.
SKILL.md
---
name: configuring-exports
description: Configure Celigo export resources -- the data source step that fetches records from external systems. Use when creating or editing exports, choosing the right adaptor type for a target application, setting up delta/incremental syncs, webhooks, file transfers, or lookups.
---
<!-- TIER:1 -->
# Configuring Exports
An export is the **data source** in a Celigo integration. It connects to an external system and pulls data into the pipeline. Exports serve two roles:
- **Source** -- the starting point that fetches the primary batch of records
- **Lookup** -- a mid-flow enrichment step (`isLookup: true`) that fetches additional data per-record during processing
Both roles are used across flows, APIs, and tools.
Beyond fetching data, exports also handle post-retrieval processing before records enter the pipeline:
- **Output filter** -- expression-based filtering to skip records that don't match criteria
- **Transform** -- Transformation 2.0 expression rules to reshape/flatten response data before mapping
- **preSavePage hook** -- JavaScript processing on the full page of records before they enter the pipeline
- **One-to-many** -- when used as a lookup, fan out child records from a parent. Set `oneToMany: true` and `pathToMany` to the child array path so each child triggers a separate lookup. Once fanned out, the array element itself is the record -- see [One-to-many fan-out](#one-to-many-fan-out----the-array-element-is-the-record)
- **Response mapping** -- when used as a lookup, extract fields from the lookup response back into the record. Configured on the flow's `pageProcessors[]` entry, but planned when building the lookup export. The response contains a `data` array and an `errors` array. Use `data[0].fieldName` when you expect a single result (e.g., fetching one order by ID). When multiple results are expected, map the whole array with `extract: "data"` (downstream steps then read it as a normal JSON array, or fan out over it with one-to-many), or use a `lists` entry with `data[*].fieldName` extracts to build a reshaped array. Do not put `data[*].fieldName` in a top-level `fields[].extract` -- the wildcard is silently ignored there and nothing is merged. See [writing-mappings > Response Mapping Reference](../writing-mappings/SKILL.md#response-mapping-reference-transformation-10). Response mapping uses Transformation 1.0 syntax (extract/generate pairs), not the newer expression-based transforms
- **postResponseMap hook** -- JavaScript processing after response mapping merges the lookup response back into the record. Configured on the flow's `pageProcessors[]` entry, but planned when building the lookup export
## Export Execution Pipeline
When a flow runs, each export executes this pipeline in strict order:
1. **API request / query / file read** -- fetches raw data from the external system
2. **Response parsing** -- `resourcePath` extracts the record array from the response body or file (e.g., `http.response.resourcePath` for HTTP, `file.json.resourcePath` for JSON files, XPath for XML)
3. **Transformation** (optional) -- `transform` reshapes individual records after extraction (Transformation 2.0)
4. **Output filter** (optional) -- discards records that don't match filter expression rules
5. **preSavePage hook** (optional) -- JavaScript processing on the full page of records
**Key distinction:** `resourcePath` tells the export WHERE to find records in the response. Transforms reshape WHAT each record looks like after extraction. When a user says "extract records from X" or "treat each X as a separate record", that's almost always a `resourcePath` change, not a transform. Use transforms when you need to flatten nested objects, rename fields, or restructure individual records.
## Three Categories of Export
Not all exports work the same way. Before building, understand which category you need:
### Listeners
Receive data pushed to Celigo from an external system. No polling, no scheduling -- the source system sends data when events happen.
- `WebhookExport` -- inbound HTTP listener (no connection required)
- `AS2Export` -- AS2 EDI file reception
- Distributed exports (`type: "distributed"`) -- real-time event-driven push for NetSuite (via SuiteScript) and Salesforce (via streaming API). The platform installs listeners in the source system that fire when records change.
- Change data capture (`type: "stream"`) -- MongoDB change streams that tail the oplog for real-time record changes.
**When to use:** The source system supports outbound webhooks, push notifications, or change data capture and you want real-time processing.
### File Transfers
Read files from a remote location, then either parse them into records or transfer them as blobs.
- `FTPExport` / `S3Export` / `FileSystemExport` -- fetch files from FTP/SFTP, S3, or local filesystem
- `HTTPExport` with `http.type: "file"` -- fetch files over HTTP from cloud storage APIs (Google Drive, Box, Dropbox, Azure Blob Storage). The HTTP connector handles auth; the `file{}` config handles parsing.
- `NetSuiteExport` with `netsuite.type: "file"` -- fetch and parse files (CSV, JSON, XLSX, XML, EDI) from the NetSuite file cabinet
- Parsed mode (`file.output: "records"`) -- CSV, XML, JSON, XLSX, EDI files are parsed into individual records
- Blob mode (`type: "blob"`) -- binary files transferred as-is without parsing. Supported on HTTPExport, NetSuiteExport, SalesforceExport, FTPExport, and S3Export.
**When to use:** The source system drops files (CSV, EDI, XML, etc.) into a directory, bucket, file cabinet, or cloud storage rather than exposing a record-based API.
### Record-Based Exports
Actively fetch batches of records from an API or database on a schedule.
- `HTTPExport` -- REST/GraphQL APIs
- `NetSuiteExport` -- saved searches, restlets, SuiteQL
- `SalesforceExport` -- SOQL/Bulk queries
- `RDBMSExport` -- SQL SELECT queries
- `MongodbExport`, `JDBCExport`, `DynamodbExport` -- other databases
- `WrapperExport` -- custom stack (Walmart, BigCommerce)
**When to use:** You need to poll an API or query a database for records on a schedule (full fetch or delta/incremental).
## Quick Reference
### Adaptor Decision Matrix
| Your data comes from... | Use adaptorType | Category | Read schema |
|---|---|---|---|
| REST or GraphQL API | `HTTPExport` | Record-based | [http.yml](references/schemas/http.yml) |
| Files over HTTP (Google Drive, Box, Dropbox, Azure Blob) | `HTTPExport` with `http.type: "file"` | File transfer | [http.yml](references/schemas/http.yml) + [file.yml](references/schemas/file.yml) |
| NetSuite (any method) | `NetSuiteExport` | Record-based | [netsuite.yml](references/schemas/netsuite.yml) |
| Salesforce objects | `SalesforceExport` | Record-based | [salesforce.yml](references/schemas/salesforce.yml) |
| SQL database | `RDBMSExport` | Record-based | [rdbms.yml](references/schemas/rdbms.yml) |
| MongoDB | `MongodbExport` | Record-based | [mongodb.yml](references/schemas/mongodb.yml) |
| JDBC database | `JDBCExport` | Record-based | [jdbc.yml](references/schemas/jdbc.yml) |
| DynamoDB | `DynamodbExport` | Record-based | [dynamodb.yml](references/schemas/dynamodb.yml) |
| Files on FTP/SFTP | `FTPExport` | File transfer | [ftp.yml](references/schemas/ftp.yml) + [file.yml](references/schemas/file.yml) |
| Files on S3 | `S3Export` | File transfer | [s3.yml](references/schemas/s3.yml) + [file.yml](references/schemas/file.yml) |
| Webhooks / push events | `WebhookExport` | Listener | [webhook.yml](references/schemas/webhook.yml) |
| AS2 EDI messages | `AS2Export` | Listener | [as2.yml](references/schemas/as2.yml) |
| Manual file upload | `SimpleExport` | File transfer | [simple.yml](references/schemas/simple.yml) |
| Local filesystem | `FileSystemExport` | File transfer | [filesystem.yml](references/schemas/filesystem.yml) + [file.yml](references/schemas/file.yml) |
| Pre-built stack connector | `WrapperExport` | Record-based | [wrapper.yml](references/schemas/wrapper.yml) |
**Raw HTTP is the fallback, not the default.** Pick the most specific match, in order:
1. **Native adaptor** -- if the application has its own row (NetSuite, Salesforce, databases, FTP/S3), use it. Do not build an `HTTPExport` against that app's REST API.
2. **Pre-built HTTP connector** -- for any other REST/GraphQL app, check the 550+ connector catalog before writing HTTP config (see [Check for a pre-built connector](#3-check-for-a-pre-built-connector)). The step is still an `HTTPExport`, but it runs on a connector-backed connection and takes its endpoint config from the connector.
3. **Manual HTTP** -- hand-write the config from public API docs only when no connector exists or it doesn't cover the endpoint you need.
`adaptorType` is **case-sensitive**: `HTTPExport`, not `httpExport`.
### Minimum Required Fields
Every export needs at minimum:
- `name` -- human-readable label
- `adaptorType` -- from the matrix above
- `_connectionId` -- except `WebhookExport` and `SimpleExport`
- Adaptor config block -- `http{}`, `netsuite{}`, `ftp{}`, `salesforce{}`, `rdbms{}`, etc.
### Which Schemas to Read
1. **Always:** [request.yml](references/schemas/request.yml) (base fields for all exports)
2. **Plus:** the adaptor-specific file from the matrix above (e.g., `http.yml` for HTTPExport)
3. **If file-based:** also [file.yml](references/schemas/file.yml) (CSV, XML, JSON, XLSX, EDI parsing config)
4. **If delta/incremental:** check [delta.yml](references/schemas/delta.yml) or Handlebars URI pattern (`{{{lastExportDateTime}}}`)
5. **If cloning:** [clone-request.yml](references/schemas/clone-request.yml), [clone-response.yml](references/schemas/clone-response.yml)
### Schema Index
All schemas are in [references/schemas/](references/schemas/):
- **Base fields (all exports):** [request.yml](references/schemas/request.yml)
- **Response shape:** [response.yml](references/schemas/response.yml)
- **Adaptor-specific config:**
- [http.yml](references/schemas/http.yml) -- HTTP/REST/GraphQL
- [netsuite.yml](references/schemas/netsuite.yml) -- NetSuite (restlet, saved search, SuiteQL, file cabinet)
- [salesforce.yml](references/schemas/salesforce.yml) -- Salesforce (SOQL, bulk)
- [ftp.yml](references/schemas/ftp.yml) -- FTP/SFTP
- [s3.yml](references/schemas/s3.yml) -- Amazon S3
- [rdbms.yml](references/schemas/rdbms.yml) -- SQL databases
- [mongodb.yml](references/schemas/mongodb.yml) -- MongoDB
- [jdbc.yml](references/schemas/jdbc.yml) -- JDBC databases
- [dynamodb.yml](references/schemas/dynamodb.yml) -- DynamoDB
- [as2.yml](references/schemas/as2.yml) -- AS2 EDI
- [wrapper.yml](references/schemas/wrapper.yml) -- custom stack connectors
- [filesystem.yml](references/schemas/filesystem.yml) -- local filesystem
- [simple.yml](references/schemas/simple.yml) -- data loader / manual upload
- **File parsing:** [file.yml](references/schemas/file.yml) (CSV, XML, JSON, XLSX, EDI)
- **Operational modes:** [delta.yml](references/schemas/delta.yml), [webhook.yml](references/schemas/webhook.yml), [distributed.yml](references/schemas/distributed.yml), [once.yml](references/schemas/once.yml)
- **Mock output:** [mock-output.yml](references/schemas/mock-output.yml)
- **Clone:** [clone-request.yml](references/schemas/clone-request.yml), [clone-response.yml](references/schemas/clone-response.yml)
## Related Skills
- [configuring-connections > Quick Reference](../configuring-connections/SKILL.md#quick-reference) -- connection types, auth methods, iClients
- [writing-mappings > Transformation 2.0](../writing-mappings/SKILL.md#transformation-20-workflow) -- reshape export output before mapping
- [writing-scripts > Data Pipeline Hooks](../writing-scripts/SKILL.md#data-pipeline-hooks) -- preSavePage, postResponseMap hooks
- [writing-handlebars > Quick Reference](../writing-handlebars/SKILL.md#quick-reference) -- dynamic values in URIs, filters, delta tokens
- [building-flows > How to Build a Flow](../building-flows/SKILL.md#how-to-build-a-flow) -- wiring exports into flows
- [troubleshooting-flows > Diagnostic Workflow](../troubleshooting-flows/SKILL.md#diagnostic-workflow) -- diagnosing export-related failures
<!-- TIER:2 -->
## How to Build an Export
### 1. Identify the target application
What system are you pulling data from? This determines everything -- adaptor type, connection type, and configuration shape.
### 2. Check for existing patterns
Before building from scratch, look at what already exists:
```bash
# Search across the entire account for related resources
celigo account search "<keyword>"
# Show what an existing export uses (connection) and what uses it (flows)
celigo account dependencies export <id>
# Find orphaned exports not referenced by any flow
celigo account lint
# Check if a similar export already exists in the account
celigo exports list | grep -i "<application-name>"
# Search the marketplace for pre-built integration templates
celigo templates marketplace
# Preview a template to see its export configuration
celigo templates preview <id> --model Export
celigo templates preview <id> --summary
```
The account index auto-refreshes when stale (>4 hours). Force a fresh snapshot with `celigo account snapshot`.
Existing exports in the account are the best reference -- they show proven patterns for that specific customer's setup. Marketplace templates may provide a complete pre-built integration you can install rather than building from scratch.
### 3. Check for a pre-built connector
**Always run this check before writing any HTTP config.** Celigo maintains 550+ HTTP connector definitions and 590+ trading partner connectors. These provide pre-configured auth, base URLs, and endpoint definitions for common applications. Connectors are set on the **connection**, not the export -- but they determine what the export can do. Hand-write a manual `HTTPExport` from public API docs only when this search comes up empty or the connector doesn't cover the endpoint you need.
```bash
# Search HTTP connectors (REST APIs: Shopify, Stripe, HubSpot, etc.)
celigo http-connectors list | grep -i "<application-name>"
celigo http-connectors get <id> --full # see endpoints, resources, auth config
# Drill into the endpoints the connector defines for exports
celigo http-connectors catalog <id> --resource-type export --published-only
celigo http-connectors endpoint-detail <id> --resource-type export --resource-id <rid> --endpoint-id <epid>
# Search trading partner connectors (EDI, AS2, VAN)
celigo tp-connectors list
```
If an HTTP connector exists for your target app, create the connection from it (`http._httpConnectorId` -- see [configuring-connections > Check for a pre-built connector and global iClient](../configuring-connections/SKILL.md#4-check-for-a-pre-built-connector-and-global-iclient)) and take the export's `relativeURI`, method, pagination, and response paths from the connector's endpoint metadata rather than reconstructing them from public API docs. The connector-reference fields on the export itself (`http._httpConnectorEndpointId`, `http._httpConnectorVersionId`, `http._httpConnectorResourceId`) are read-only -- the platform sets them; what you control is the connection and the endpoint config you copy from the connector.
If a trading partner connector exists (EDI/AS2), reference it on the export via `ftp._tpConnectorId` (FTP exports) or `as2._tpConnectorId` (AS2 exports). You may also need to set `_ediProfileId` on the export for EDI document validation.
### 4. Query metadata for the target system
For NetSuite, Salesforce, and RDBMS connections, you can discover available record types and fields directly from the live system:
```bash
# List available record types / sObjects / tables
# NetSuite also returns saved searches alongside record types
celigo metadata types <connectionId>
# List fields for a specific entity type
celigo metadata fields <connectionId> <entityType>
```
This tells you what data is available to export before you write any configuration.
- **NetSuite:** `metadata types` returns both record types and saved searches (with IDs you need for `netsuite.restlet.searchId`). `metadata fields` returns field IDs, names, types, and group — including sublist fields you'll need for `mapping.lists[].generate` on the import side.
- **Salesforce:** `metadata types` returns sObjects with queryable/createable flags. `metadata fields` returns fields, types, and relationship names — use these to discover child objects for `distributed.relatedLists[]` and relationship field names for cross-object queries.
- **RDBMS:** `metadata types` returns table names. `metadata fields` returns column names and types for a given table — use these when writing SQL queries or building field mappings.
### 5. Determine the category
Is this a **listener** (real-time push from the source), a **file transfer** (fetch and parse/transfer files), or a **record-based export** (poll an API or query a database)? This narrows which adaptor types and modes apply.
### 6. Choose the right adaptor type
Use the [Adaptor Decision Matrix](#adaptor-decision-matrix) in Quick Reference above to select the correct `adaptorType` for your target system.
### 7. Build the export JSON
Use the [Schema Index](#schema-index) and [Which Schemas to Read](#which-schemas-to-read) in Quick Reference above. Read `request.yml` for base fields, then the adaptor-specific schema, plus `file.yml` if file-based and `delta.yml` if incremental.
## Export Design Decisions
A few design choices recur when building exports. Each has a defensible default once the framing is clear.
### Delta vs one-time vs full sync
The export's `type` field selects the sync behavior:
- **Delta** (`type: "delta"`) -- pulls only records created or modified since the last successful run. The default for ongoing scheduled syncs when the source exposes a usable "last modified" timestamp. Non-HTTP adaptors set the timestamp field via `delta.dateField`; HTTP exports instead embed `{{{lastExportDateTime}}}` in the `relativeURI` or body. See [delta.yml](references/schemas/delta.yml).
- **One-time** (`type: "once"`) -- processes each record exactly once via a tracking flag: each run selects records where `once.booleanField` is `false`, then sets it to `true` after a page succeeds so later runs skip them. Use for backfills and migrations, or when the source has no reliable timestamp but its records can carry a processed flag. See [once.yml](references/schemas/once.yml).
- **Full** (neither `delta` nor `once` mode) -- re-pulls the entire dataset every run. Use when the source has no usable modification timestamp, the dataset is small enough that re-pulling is cheap, or business logic requires a fresh snapshot each run.
When the request is vague ("sync customers"), confirm which kind of sync is intended before building. Delta is a reasonable default when the source exposes a timestamp field; full is reasonable for small static datasets.
### Listener/webhook vs scheduled export
Both are starting steps (see [Three Categories of Export](#three-categories-of-export)); the choice is driven by what the source supports and the latency budget, not preference:
- Reach for a **listener** (`WebhookExport`, or NetSuite/Salesforce `type: "distributed"`) when the source pushes events and the flow needs to react quickly ("when X happens, do Y").
- Reach for a **scheduled export** when the source has no push mechanism, or when batch timing at off-peak hours is acceptable.
NetSuite and Salesforce support both for many record types. Mixing them on one flow is a common, good pattern -- a listener handles low-latency reactions while a scheduled export runs as a safety net for backfills, end-of-day reconciliation, and catching up after a webhook outage.
### Lookup export vs separate scheduled export
The distinguishing question is when the data is needed:
- A **lookup export** (`isLookup: true`) runs per in-flight record, mid-pipeline, keyed off the upstream record -- fetching the customer for a specific order, or inventory for a specific SKU.
- A **scheduled export** runs once per flow run as a starting point, producing the first batch of records the flow processes.
If the request is "for each X, look up Y", it's a lookup. If it's "every hour, pull all Y", it's a scheduled export.
### One-to-many fan-out -- the array element IS the record
With `oneToMany: true` and `pathToMany` set to a child array path, each element of that array triggers its own lookup. Once fanned out, **the element becomes the record**: templates reference the element's own fields as `{{record.variantId}}` -- not `{{variantId}}`, and not `{{record.lineItems.variantId}}`. The array wrapper is gone; you are inside one element.
Three consequences worth knowing before you debug the wrong thing:
- **The build-time preview warning is expected.** Previewing a fanned-out lookup in isolation reports "`<field>` not defined in the model" because no upstream record is bound yet. That is not a broken template -- don't "fix" a correct `{{record.X}}` reference because of it.
- **Response mapping merges per element automatically.** To get looked-up values back onto each element, author a normal top-level `fields` response mapping; Celigo merges each result into its corresponding fanned-out element.
- **Two anti-patterns.** Don't target the array with a `lists` entry (that nests a new array inside each element), and don't attempt the per-element merge in `postResponseMap` (it sees the page of parent records, not per-element results).
### Source-side transform vs destination-side mapping
Both reshape data, but in opposite directions:
- A **transform** on a source export reshapes records as they enter the flow -- flattening nested responses, or aligning multiple sources to a common shape (see [Export Execution Pipeline](#export-execution-pipeline)).
- A **mapping** on a downstream import reshapes records as they leave the flow toward a destination.
Don't add a transform to "match a destination" -- that's the destination import mapping's job. Transforms are for entry reshaping; mappings are for exit reshaping.
### Async APIs (submit, poll, fetch)
Most APIs return data in the same call and need none of this. Some APIs only *acknowledge* a request (an HTTP 202, a job ticket, a feed or document id) and process it in the background -- Amazon SP-API feeds, large report generators, bulk extract and file-conversion jobs. For those, attach an **async helper** to the export via `http._asyncHelperId`. The helper teaches the step the submit-poll-fetch pattern; it is part of the export, not something managed on its own, and bundles three pieces:
1. **A status export** (required) -- run on each poll to ask "is it done yet?". Configure the **status path** to read in the response, the case-sensitive **in-progress / done / done-without-data / error** value lists (taken from the API's docs), and the **initial wait** and **poll wait** intervals in minutes.
2. **A result export** (optional, usually present) -- fetches the final payload once status reports done.
3. **Initial-submission handling** -- where to find the job ticket in the first acknowledgement: "same as status" when the acknowledgement is itself shaped like a status response, otherwise a resource path (plus transform rules for non-JSON acknowledgements, e.g. Amazon's XML).
Two constraints shape the design: the status and result exports must be ordinary synchronous exports (an async helper cannot nest another), and the async-configured step **cannot carry its own transform, output filter, or preSavePage hook** -- put any reshaping or filtering on the dedicated result export instead. The same pattern applies symmetrically to imports writing to asynchronous destinations (`_asyncHelperId` on the import).
Reach for an async helper only when the API genuinely forces the fire-and-check-back shape. Adding one to a synchronous API is pure overhead -- extra polling plus a status and result export to maintain.
## CLI Commands
```bash
# CRUD
celigo exports list
celigo exports get <id>
celigo exports create < export.json
celigo exports update <id> < export.json
celigo exports set <id> key=value [key2=value2 ...]
celigo exports delete <id>
# Invoke (test-run an export, see what data comes back)
celigo exports invoke [id] [--all]
# Clone and connection management
echo '{"connectionMap":{"oldConnId":"newConnId"}}' | celigo exports clone <id>
celigo exports replace-connection <id> <newConnectionId>
# Discovery
celigo account search "<keyword>"
celigo templates marketplace
celigo templates preview <id> --model Export
celigo templates preview <id> --summary
celigo http-connectors list
celigo http-connectors catalog <id> --resource-type export --published-only
celigo http-connectors endpoint-detail <id> --resource-type export --resource-id <rid> --endpoint-id <epid>
celigo tp-connectors list
celigo metadata types <connectionId>
celigo metadata fields <connectionId> <entityType>
# Debug
celigo exports enable-debug <id> [--duration <minutes>]
celigo exports disable-debug <id>
```
<!-- TIER:3 -->
## Pre-Submit Checklist
Before creating or updating an export, verify:
- [ ] **`adaptorType` is exact** -- case-sensitive, matches the [Adaptor Decision Matrix](#adaptor-decision-matrix) (e.g., `HTTPExport`, not `httpExport` or `HttpExport`)
- [ ] **Pre-built connector was checked** -- for HTTP exports, `celigo http-connectors list` found no connector for the app (or the connector lacks the endpoint) before any hand-written `relativeURI`
- [ ] **`_connectionId` is valid** -- points to an existing, online connection of the correct type. Not needed for `WebhookExport` or `SimpleExport`
- [ ] **Adaptor config block is present** -- `http{}`, `netsuite{}`, `ftp{}`, etc. matches the `adaptorType`
- [ ] **`resourcePath` or query is correct** -- wrong path silently returns 0 records with no error
- [ ] **Pagination is configured** -- for HTTP exports, set `http.paging` if the API returns paginated results
- [ ] **Delta/incremental is configured** -- if using delta, check `delta.dateField` or Handlebars `{{{lastExportDateTime}}}` in the URI
- [ ] **File parsing matches the format** -- if file-based, `file.type` matches the actual file format (csv, json, xml, xlsx, edi)
- [ ] **`mockOutput` format is correct** -- `{ "page_of_records": [{ "record": {...} }] }`, not a plain array
- [ ] **No `rest:` block** -- `rest:` creates a legacy RESTExport. Use only `http:` for new exports
- [ ] **Output filter syntax is valid** -- if using an output filter expression, test it against sample data
- [ ] **Lookup config is complete** -- if `isLookup: true`, ensure response mapping is planned for the flow's `pageProcessors[]` entry
## Gotchas
1. **PUT erases omitted fields.** Always GET first, modify, then PUT. The `set` command handles this.
2. **Including a `rest:` block creates a legacy RESTExport.** Use only `http:` for new exports.
3. **Wrong `resourcePath` produces 0 records with no error.** First thing to check when an export succeeds but returns nothing.
4. **`mockOutput` format is `{ "page_of_records": [{ "record": {...} }] }`.** Not a plain array.
5. **HTTP delta exports use Handlebars** (`{{{lastExportDateTime}}}` in `relativeURI`), not `delta.dateField`.
6. **NetSuite saved searches need `netsuite.restlet.searchId`.** Use `celigo metadata types <connectionId>` to find the search ID.
7. **File exports require the `file{}` block.** Without it, file-based exports return raw bytes instead of parsed records.
8. **Webhook exports have no `_connectionId`.** Setting one causes validation errors.
9. **Distributed exports require `type: "distributed"` on the export AND `distributed: true` on the connection.**
10. **`type: "once"` needs a dedicated, writeable tracking flag.** `once.booleanField` must be writeable by the export's connection, and no other process may update the same field -- a shared flag causes records to be skipped.
11. **An async-helper export cannot carry its own transform, output filter, or preSavePage hook.** Build that processing into the helper's result export instead. The status and result exports must themselves be plain synchronous exports -- an async helper cannot nest another. See [Async APIs (submit, poll, fetch)](#async-apis-submit-poll-fetch).
## Common Errors
| Error | Likely Cause | Fix |
|---|---|---|
| `404 Not Found` on export invoke | Wrong `relativeURI` or `resourcePath` | Verify the endpoint path against the API docs; check for missing path parameters |
| `401 Unauthorized` | Connection credentials expired or invalid | Run `celigo connections ping <connId>`; re-authorize OAuth connections |
| `0 records exported` (no error) | Wrong `resourcePath`, empty date range, or overly restrictive filter | Check `resourcePath`, widen delta window, test without output filter |
| `Cannot read property of undefined` in preSavePage | Script assumes a field exists that is missing from some records | Add null checks: `if (record.field)` before access |
| `mockOutput is invalid` | Wrong format -- used array instead of object | Use `{ "page_of_records": [{ "record": {...} }] }` |
| `Invalid adaptorType` | Case mismatch or typo | Use exact casing from the Adaptor Decision Matrix |
| `Connection is offline` | Connection failed health check | Fix credentials, re-authorize, then `celigo connections ping <id>` |
| `Rate limit exceeded` / `429` | Too many concurrent requests to the source API | Lower `concurrencyLevel` on the connection; add retry config |
| `Timeout` on large exports | Query returns too much data or API is slow | Add pagination, narrow the date range, or increase timeout settings |
| `File parsing error` | `file.type` doesn't match actual file format, or delimiter/encoding mismatch | Verify `file.type`, check `file.csv.columnDelimiter`, ensure correct encoding |