Understanding Zoho CRM's Bulk Data Flow within Catalyst

This guide explains the flow of Zoho CRM bulk data within Catalyst for streamlined operations to process and manage large datasets.

Pre-configured Catalyst Resources

When you install the CRM Bulk Data Processor CodeLib, the following Catalyst resources are automatically set up in your Catalyst project:

1. Cloud Scale Data Store

Three tables named BulkRead, ReadQueue, and WriteQueue are automatically created to manage and track bulk data processing. Each table serves a specific purpose and contains the following columns:

I. BulkRead Table

It stores the details required for initiating a Bulk Read API call, such as the fields to process and the module. It tracks the current API request and response details, including the Bulk Read job ID, which is replaced with each new API call. Once the response is ready, the table provides a download URL for the retrieved data.

• Columns: ROWID, CREATORID, CREATEDTIME, MODIFIEDTIME, FIELDS_TO_BE_PROCESSED, CRMJOBID, REQUESTED_PAGE_NO, FETCHED_PAGE_NO, MODULE_NAME, DOWNLOAD_URL

II. ReadQueue Table

It tracks the progress of data processing to ensure accurate ingestion and prevent duplicate processing. Unlike the BulkRead table, this table saves every Bulk Read job ID, mapped to the processed records stored in the File Store.

It logs details such as the number of lines processed (with respect to the File Store), whether processing has started or completed, and the corresponding CRM job ID for each set of records.

• Columns: ROWID, CREATORID, CREATEDTIME, MODIFIEDTIME, FILEID, LINE_PROCESSED, IS_PROCESS_STARTED, CRM_JOB_ID, IS_PROCESS_COMPLETED, MODULE

III. WriteQueue Table

It manages the processed file entries and tracks their upload status to Zoho CRM via the Bulk Write API. It also stores the module details required to make the API call and saves the Job ID to monitor the bulk write process.

• Columns: ROWID, CREATORID, CREATEDTIME, MODIFIEDTIME, FILE_ID, IS_UPLOADED, CRM_JOB_ID, MODULE

2. Cloud Scale Event Listener

The auto-generated Event Listener monitors for changes in the above Data Store tables and triggers event functions accordingly.

3. Serverless Functions

• A Cron Function named BulkJobSchedule is created to update the module and fields name in the BulkRead table of the Catalyst Data Store.
• An Event Function named BulkDataProcessor is built to execute bulk read operations when data is inserted/updated in the BulkRead table.
• An Advanced I/O Function named zohocrm_bulk_callback is created to receive callbacks from the bulk read and write tasks.

You can also find these auto-generated functions in your local project folder after installing the CodeLib solution.

4. Cloud Scale File Store

A folder called csvfiles is automatically created to store Zoho CRM data temporarily during bulk read and processing tasks.

Bulk Data Flow Overview

I. Bulk Read Operation

Catalyst fetches Zoho CRM data in bulk via the Bulk Read API.

Trigger Bulk Read API

• The pre-configured Cron Function (BulkJobSchedule) inserts/updates the module name in the BulkRead table of Cloud Scale Data Store. The module name is obtained from the MODULES parameter of the Cron job configured by the user in the Catalyst Developer Console.
• The FIELDS_TO_BE_PROCESSED parameter of the same Cron job determines which fields of the records are retrieved and processed.
• The desired module and selected fields are logged in the BulkRead table for further processing.

Event Trigger

• An Event Listener detects the changes in the BulkRead table and triggers the Event Function (BulkDataProcessor).
• The BulkDataProcessor function calls the Zoho CRM Bulk Read API, fetching up to 200,000 records per call.
• The zohocrm_bulk_callback function captures the response and updates the file download URL in DOWNLOAD_URL column of the BulkRead table.
• Once the Bulk Read job has been completed, it callbacks an Advanced I/O function (zohocrm_bulk_callback).

II. Data Processing

Once data is fetched,it is temporarily stored in the File Store and processed with the logic you define to suit your required business needs.

Data Storage

The event function BulkDataProcessor saves the retrieved data as .csv files in the csvfiles folder of Cloud Scale File Store.

Define Custom Logic

Developers can implement custom data transformation logic (e.g., data cleansing, lead segmentation) by overriding the ZCRMRecordsProcessor method in the ZCRMRecordsProcessorImpl.java file within the Event Functions directory of the project.

Save Processed Data

• The processed data is saved back to the File Store as .csv files, each containing 25,000 records, in accordance with the Bulk Write API limitations.
• Entries for the processed records in a file are added to the WriteQueue table in the Data Store to prevent duplicate processing.
• Entries for the processed records in a file are added to the LINE_PROCESSED column and the file details are added to the FILEID column of the ReadQueue table to prevent duplicate processing.

III. Bulk Write Operation

The processed data is uploaded back to the Zoho CRM, completing the cycle efficiently.

Compress and Upload Data

• The Event Listener detects changes in the ReadQueue table and triggers the BulkDataProcessor event function again.
• Now, the function compresses .csv files into .zipstrong> format and uploads them to Zoho CRM using the Bulk File Upload API.

Write Data to CRM

Once the file is uploaded to Zoho CRM, a Bulk Write Job is created to update the processed records back to the CRM module.