Synchronize data

An integration pushes its data (customers, orders, products…) to ShopiMind. The kit re-exports the SDK (@shopimind/sdk-js), so you import its resources and types from the kit and call its static methods against an authenticated client the kit hands you under ctx.spm. You describe what to synchronize through sync steps (SyncStep), and the kit takes care of the when (initial + incremental sync) and the safety (cursors, pagination, concurrency).

ctx.spm — the raw SDK client

ctx.spm is a raw SpmHttpClient, already bound to the installation's API key. You don't call methods on it; you pass it as the first argument to SDK static methods:

import { SpmCustomers, SpmOrders, SpmEvents } from '@shopimind/integration-kit-js';

// pushing entities (bulk create/update) — pass ctx.spm + the items
await SpmCustomers.bulkSave(ctx.spm, items, { chunk: true });
await SpmOrders.bulkSave(ctx.spm, orders, { chunk: true });

// triggering an event
await SpmEvents.trigger(ctx.spm, 'cart_abandoned', payload);

Resource	Examples
SpmCustomers / SpmOrders / SpmProducts / SpmOrderStatuses	E-commerce / catalog entities (bulk create/update).
SpmProductsVariations / SpmCategories / SpmManufacturers / SpmCarriers / SpmVouchers	Additional e-commerce entities (some take a path param, e.g. the product id for variations).
SpmContacts / SpmLists	Read contacts, a contact's lists, lists/segments (enrichment, dedup).
SpmEvents	Events — trigger, declaration.
SpmDataSources	Data sources (see Provisioning).
SpmCustomDataDefinitions / SpmCustomDataRecords	Custom data definitions and records.

The client is generic — not only for POS/e-commerce

Beyond catalog writes, the SDK exposes read paths (contacts, lists, custom-data records) and full record CRUD, so loyalty, reviews and CRM integrations are first-class. A contact cannot be created or modified directly (there is no contact-write API): a contact is materialized by the customers sync (SpmCustomers.bulkSave, for integrations allowed to sync customers), and you enrich it with custom-data records linked to the contact (record.relation = { by: 'email' | 'id_contact', value }). A loyalty/CRM profile is therefore a custom-data record on the contact, not a fake customer.

The response envelope — { ok, statusCode, data, error }

The SDK never throws on an HTTP error. Every call resolves to an envelope:

const env = await SpmCustomers.bulkSave(ctx.spm, items, { chunk: true });
// env: { ok: boolean, statusCode: number, data?: ..., error?: ... }
if (!env.ok) {
  // env.statusCode, env.error describe the failure — no exception was thrown
}

The kit re-exports two helpers so you rarely touch the raw envelope:

Helper	Role
SpmHelpers.unwrapOrThrow(env)	Returns env.data when ok, otherwise throws an error built from statusCode/error. Use it when you want a failure to bubble up.
SpmHelpers.extractCounts(env)	Pulls the { sent, rejected, failed } counts out of a bulk response.

import { SpmCustomers, SpmHelpers } from '@shopimind/integration-kit-js';

const env = await SpmCustomers.bulkSave(ctx.spm, items, { chunk: true });
const counts = SpmHelpers.extractCounts(env);   // { sent, rejected, failed }

For pushing inside a sync step you almost never call bulkSave directly — use ctx.sendBulk (below), which wraps the envelope, the counts and the failure handling for you.

Safe push — ctx.sendBulk

ctx.sendBulk(fn, items) is the recommended way to push in a sync step or an inbound handler. It:

• sends the items chunked;
• throws on a transport failure — a global !ok or a chunk reporting failed_count > 0;
• surfaces per-item rejections (validation rejects) without throwing;
• returns { sent, rejected, rejected_items }.

run: async (ctx) => {
  let items = 0;
  for await (const page of ctx.paginate((p) => fetchCustomers(p, ctx.window))) {
    const { sent, rejected, rejected_items } = await ctx.sendBulk(
      SpmCustomers.bulkSave,
      page.map(toSpmCustomer),
    );
    items += sent;
    if (rejected) ctx.logger.warn(`${rejected} customers rejected`, { rejected_items });
  }
  return { items, errors: [], advanceCursorTo: ctx.window.until };
};

The engine holds the cursor on rejections — no silent data loss

When sendBulk reports per-item rejections inside a sync step, the engine holds the cursor: the window is replayed on the next sync, so rejected items are not silently dropped. A transport failure throws and is always replayed too. (To let the cursor advance despite validation rejects, see tolerateRejects.)

Thunk form for path-param bulks

Some bulk methods take a path parameter before the items (e.g. product id for variations). Pass a thunk instead of the function reference:

await ctx.sendBulk(
  () => SpmProductsVariations.bulkSave(ctx.spm, productId, items, { chunk: true }),
);

tolerateRejects — the poison-pill escape

A SyncStep may set tolerateRejects: true. With it, the cursor is allowed to advance even when per-item validation rejects occur — a poison-pill escape so one malformed record can't block a window forever. Transport failures still always replay; only validation rejects are tolerated.

You can also pass a ratio instead of a plain boolean: tolerateRejects: { maxRatio }. The cursor is then allowed to advance only as long as the share of rejected items over the window (rejected ÷ attempted) stays at or below maxRatio (a value between 0 and 1). Once too many items are rejected and the ratio is exceeded, the step falls back to the strict behaviour and the cursor is frozen so the window replays. true is equivalent to { maxRatio: 1 } (always tolerate); false or omitting the field keeps the strict, no-loss default.

export const reviewsStep: SyncStep<MySettings> = {
  entity: 'reviews',
  tolerateRejects: true,   // a single bad review won't freeze the window
  run: async (ctx) => { /* ... */ },
};

A sync step — SyncStep

Each entity to synchronize is a SyncStep. The kit runs it on the initial sync, then periodically, incrementally.

import type { SyncStep } from '@shopimind/integration-kit-js';
import { SpmCustomers } from '@shopimind/integration-kit-js';
import type { MySettings } from './settings.js';

export const customersStep: SyncStep<MySettings> = {
  entity: 'customers',
  cursorScope: 'global',                 // 'global' | 'per-source'
  enabled: (settings) => settings.syncCustomers,
  // sources: required only if cursorScope === 'per-source'
  run: async (ctx) => {
    let items = 0;
    for await (const page of ctx.paginate((p) => fetchCustomers(p, ctx.window))) {
      const { sent } = await ctx.sendBulk(SpmCustomers.bulkSave, page.map(toSpmCustomer));
      items += sent;
    }
    // the cursor advances ONLY if errors=[] AND advanceCursorTo is set
    return { items, errors: [], advanceCursorTo: ctx.window.until };
  },
};

The contract:

Field	Type	Role
entity	string	Unique entity name (cursor key).
cursorScope	'global' | 'per-source'	A single cursor, or one cursor per source (see Multi-source).
enabled	(settings) => boolean	Does the step run (based on the merchant's config)?
sources?	(ctx) => string[]	Required if per-source: the list of sources to iterate over.
tolerateRejects?	boolean | { maxRatio: number }	Let the cursor advance despite per-item validation rejects (see above). Transport failures still replay.
run	(ctx) => Promise<SyncStepResult>	The work. Returns { items, errors, advanceCursorTo? }.

The safe cursor (no data loss)

The kit advances the cursor only if errors is empty and advanceCursorTo is set. On a partial error, the cursor stays frozen: the next sync replays the window, nothing is lost.

// ✅ clean run → the cursor advances
return { items, errors: [], advanceCursorTo: ctx.window.until };

// ⚠️ error on a page → the cursor DOES NOT MOVE (we'll replay)
return { items, errors: ['page 3: 502 Bad Gateway'] };

ctx.sendBulk cooperates with this: a transport failure throws (the window replays), and per-item rejections hold the cursor unless tolerateRejects is set.

The step context — SyncStepContext

ctx (inside run) extends the IntegrationContext with:

Field	Role
window	{ since, until } — the time window to synchronize (backfill on the 1st run, incremental afterwards).
sourceKey	The current source ('' in global scope, the source id in per-source).
cursor	The current cursor (or null).
paginate(fetchPage, opts?)	Async generator of pages — streaming pagination (anti-OOM).
mapConcurrent(items, limit, fn)	Processes items with a bounded concurrency (anti-429).

Multi-source

An integration can feed several data sources under a single ShopiMind account (for example one physical store per point of sale, alongside the e-commerce source). A per-source step:

import { SpmOrders } from '@shopimind/integration-kit-js';

export const ordersStep: SyncStep<MySettings> = {
  entity: 'orders',
  cursorScope: 'per-source',
  enabled: (s) => s.syncOrders,
  sources: (ctx) => ctx.settings.storeIds,           // one source per store
  run: async (ctx) => {
    // ctx.sourceKey = the current store's id ; each source has its own cursor
    const orders = await fetchStoreOrders(ctx.sourceKey, ctx.window);
    const src = ctx.withSource(ctx.sourceKey);
    const { sent } = await src.send(SpmOrders.bulkSave, orders.map(toSpmOrder));
    return { items: sent, errors: [], advanceCursorTo: ctx.window.until };
  },
};

A failure on one source does not prevent the others from advancing: cursors are isolated per source.

Dedicated sources — ctx.withSource

When you push catalog (customers, products, orders…) into a source of your own — so you never overwrite the merchant's native catalog — tag each entity with its id_data_source. ctx.withSource(sourceKey) returns a SourceHandle:

Member	Role
id	The resolved id_data_source of the provisioned source.
tag(items)	Adds id_data_source to each item (when you only need the tag).
send(fn, items)	tag(items) + safe push (ctx.sendBulk). The one-liner you normally use.

import { SpmProducts } from '@shopimind/integration-kit-js';

run: async (ctx) => {
  const src = ctx.withSource(ctx.sourceKey);          // the provisioned source for this step
  const { sent } = await src.send(SpmProducts.bulkSave, products.map(toSpmProduct));
  return { items: sent, errors: [], advanceCursorTo: ctx.window.until };
};

src.send(SpmProducts.bulkSave, items) tags each item with id_data_source and pushes safely (chunked, transport-throwing, rejection-surfacing) in one call.

• sourceKey must match a source declared in provisioning.dataSources (otherwise withSource throws).
• ctx.withSource is available everywhere you have a ctx (hooks, run, etc.).

Tagging the source is not enough: also namespace your identifiers

On the ShopiMind side, id_data_source is not part of an entity's uniqueness key. If you push a product whose native identifier already exists at the merchant's, you overwrite its row — even with a different source. For real isolation, namespace your identifiers in your mappers: a prefix for text ids (hib_<id>), a numeric offset for integer ids (id + 1_000_000_000). withSource guarantees the source tag; namespacing remains your responsibility.

Product unification across sources

When you push products from a source of your own (e.g. a point of sale), ShopiMind automatically unifies them with the merchant's web catalog so the same physical article is one product, not a duplicate:

• It matches your product against the web catalog by a matching column — reference by default (other columns such as ean13 may be offered later). You set the default for your integration; the merchant can override it at install, and it is locked once the integration is activated.
• Match found → your product is attached to the existing product as a per-source overlay: your price and stock are kept per source, the shared product stays the merchant's canonical record. Recommendations, best-sellers and segmentation then count one unified product.
• No match → a standalone product is created for your source (a genuinely store-only article).

Always namespace your ids (above) so a no-match product never collides with the web catalog. Contacts are unified at the email/phone level; products at the matching column. This unification is transparent for merchants with a single (web) source — nothing about their products changes.

Custom data — ctx.customData

To push custom-data records (loyalty balances, reviews, store-specific attributes…), use ctx.customData(name). It is the custom-data counterpart of withSource: name matches a definition declared in provisioning.customData, and the kit resolves the definition id for you. It returns a CustomDataHandle:

Member	Role
id	The resolved definition id.
save(records)	Safely upserts the records into that definition — chunked, transport-safe (throws on transport failure), surfaces per-item rejections.

run: async (ctx) => {
  const loyalty = ctx.customData('loyalty_points');         // matches provisioning.customData
  const { sent } = await loyalty.save(rows.map(toLoyaltyRecord));
  return { items: sent, errors: [], advanceCursorTo: ctx.window.until };
};

This is how you push custom-data records — do not resolve definition ids by hand. Link a record to the contact through its relation field: set the field you declared as a system relationship to a resolver, e.g. contact_id: { by: 'email', value } (the server stores the canonical id_contact). The definition-level relationships themselves are declared in provisioning.customData.

Provisioning

Before pushing data, an integration provisions the receiving structure: data sources (parent/children), custom data definitions, events, order statuses. Declare a ProvisioningPlan via provisioning(ctx) — the kit replays it on activation (and on every config_updated), in an idempotent way (find-or-create).

provisioning: (ctx): ProvisioningPlan => ({
  dataSources: [
    { key: 'parent', decl: { label: 'My App', type: 'api' } },
    { key: 'store-12', parentKey: 'parent', decl: { label: 'Store 12', type: 'api' } },
  ],
  customData: [
    // declare a target definition BEFORE the one that references it
    { name: 'stores', unique_keys: ['store_ref'], fields: [{ name: 'store_ref', type: 'text' }] },
    {
      name: 'loyalty_points',                              // referenced by ctx.customData('loyalty_points')
      unique_keys: ['contact_id'],
      fields: [
        { name: 'contact_id', type: 'number', required: true },
        { name: 'balance', type: 'number' },
        { name: 'store', type: 'number' },
      ],
      relationships: [
        { sourceField: 'contact_id', targetSchemaType: 'system', targetSchema: 'contacts', targetField: 'id_contact' },
        // custom -> custom: targetSchema is the SIBLING'S NAME — the kit resolves it to its id
        { sourceField: 'store', targetSchemaType: 'custom', targetSchema: 'stores', targetField: 'store_ref' },
      ],
    },
  ],
  // events: [...],       // events
  // orderStatuses: [...] // order statuses
}),

Plan field	Role
dataSources	Sources { key, decl, parentKey? }. parentKey references another key from the plan → parent/child hierarchy. The key is the one you then pass to ctx.withSource.
customData	Custom data definitions to create + activate. The name is the one you pass to ctx.customData.
events	Events to declare (tolerates a duplicate).
orderStatuses	Order statuses to push.

Relationships between definitions

A custom data definition may declare relationships to other schemas:

Field	Role
sourceField	A field of this definition that carries the link.
targetSchemaType	'system' (a ShopiMind schema like contacts, products) or 'custom' (another definition).
targetSchema	'system' → the schema name (contacts). 'custom' → the target's id or the name of a sibling definition in the same plan.
targetField?	The field matched in the target (defaults to its id — e.g. id_contact for contacts, or a target unique key like store_ref).

custom → custom by name

You rarely know a sibling definition's numeric id when authoring. Reference it by name in targetSchema (with targetSchemaType: 'custom'); the kit substitutes the resolved id at provisioning — exactly like parentKey for data sources. Declare the target before the definition that references it.

Convergent provisioning of custom data

Provisioning of customData is convergent: on a definition that already exists, the kit extends it with the missing fields and relationships (matched by name / sourceField) — idempotently. New fields or relationships in a later integration version therefore propagate properly.

When the sync triggers

• On activation: a full sync (backfill, depth backfillDays, default 365 days) starts automatically.
• Continuously: the internal scheduler runs an incremental sync of each active installation every syncIntervalMinutes (default 15).
• On demand: POST /admin/sync/{installationId}?full=true (protected by adminToken).

To push a piece of data in real time (without waiting for the next cycle), expose an inbound route that your application calls instead.

Going further

• The integration kit — ctx, runtime, options.
• Configuration — enable/disable steps via the config.
• Inbound middleware — push data in real time from your app.
• SDK reference — all resources and methods.
• Full example (Hiboutik) — multi-store end to end.