get_data_stream()#

relationalai.api
#get_data_stream(object_fq_name STRING, model_name STRING)

Retrieves information about a data stream for the specified RAI Python model, including when the stream was created, its status, and synchronization details. Requires the cdc_admin application role.

Parameters#

NameTypeDescription
object_fq_nameSTRINGThe fully-qualified name of the data stream’s source table or view in Snowflake, e.g. '<db>.<schema>.<table_or_view>'.
model_nameSTRINGThe name of the model associated with the stream.

Returns#

A table with the following schema:

Column NameData TypeDescription
IDSTRINGThe data stream’s unique identifier.
CREATED_ATTIMESTAMPThe timestamp when the data stream was created.
CREATED_BYSTRINGThe user who created the data stream.
STATUSSTRINGThe data stream’s current status. May be one of:
  • CREATED
  • DELETING
  • CANNOT_READ_SOURCE
For details on each status, see Data Stream Statuses. To get the stream’s current synchronization status, refer to the DATA_SYNC_STATUS column.
REFERENCE_NAMESTRINGThe type of the stream’s object reference. May be one of
  • DATA_STREAM_TABLE
  • DATA_STREAM_VIEW
REFERENCE_ALIASSTRINGThe unique identifier for the stream’s object reference.
FQ_OBJECT_NAMESTRINGThe fully-qualified name of the stream’s source table or view.
RAI_DATABASESTRINGThe name of the RAI Python model for which the stream was created.
RAI_RELATIONSTRINGThe name of the stream as passed to the stream_name parameter of the api.create_data_stream() procedure.
DATA_SYNC_STATUSSTRINGThe data stream’s synchronization status. May be one of:
  • SYNCED
  • SYNCING
  • QUARINTINED
  • SUSPENDED
  • NOT_INITIALIZED
See Quarantined Streams for details on what it means to be quarantined.
PENDING_BATCHES_COUNTINTThe number of pending batches left to process for the stream.
NEXT_BATCH_STATUSSTRINGThe status of the next batch to be processed.
NEXT_BATCH_UNLOADED_TIMESTAMPTIMESTAMPThe timestamp when the next batch will be unloaded.
NEXT_BATCH_DETAILSOBJECTA JSON object containing details about the next batch to be processed. Contains the same data as the api.data_stream_batches view.
LAST_BATCH_DETAILSOBJECTA JSON object containing details about the last batch processed. Contains the same data as the api.data_stream_batches view.
LAST_BATCH_UNLOADED_TIMESTAMPTIMESTAMPThe timestamp when the last batch was unloaded.
LAST_TRANSACTION_IDSTRINGThe transaction ID of the last batch process. Use the api.transactions view to get transaction details.
ERRORSARRAYAn array of error messages, if any, encountered during stream processing.
CDC_STATUSSTRINGThe status of the CDC service.

Data Stream Statuses#

StatusDescription
CREATEDThe stream has been created. Note that this does not indicate that the stream is active and syncing. To get the stream’s current data synchronization status, call the api.get_data_streams() procedure and refer to the DATA_SYNC_STATUS column.
DELETINGThe stream has been marked for deletion and will be removed.
CANNOT_READ_SOURCEThe stream cannot read the source table or view. This happens when the stream’s reference to the source table or view in Snowflake has been removed, for example, by calling the SYSTEM$REMOVE_REFERENCE Snowflake SQL function. The data stream must be deleted and then recreated.

Example#

Use api.get_data_stream() to retrieve information about a data stream for a RAI Python model:

#-- Get information about a data stream in a model named 'MyModel'. Replace the
-- placeholders with your database, schema, and table or view name.
CALL relationalai.api.get_data_stream('<db>.<schema>.<table_or_view>', 'MyModel');
/*+-----------------------------------------+-------------------------+-----------------------+--------+-------------------+--------------------------------------+-------------------------------+--------------+-------------------------------+------------------+------------------------+-------------------+-------------------------------+---------------------------------+-------------------------------+-------------------------------+----------------------------------------+--------+------------+
  | ID                                      | CREATED_AT              | CREATED_BY            | STATUS | REFERENCE_NAME    | REFERENCE_ALIAS                      | FQ_OBJECT_NAME                | RAI_DATABASE | RAI_RELATION                  | DATA_SYNC_STATUS | PENDING_BATCHES_COUNT  | NEXT_BATCH_STATUS | NEXT_BATCH_UNLOADED_TIMESTAMP | NEXT_BATCH_DETAILS              | LAST_BATCH_DETAILS            | LAST_BATCH_UNLOADED_TIMESTAMP | LAST_TRANSACTION_ID                    | ERRORS | CDC_STATUS |
  |-----------------------------------------+-------------------------+-----------------------+--------+-------------------+--------------------------------------+-------------------------------+--------------+-------------------------------+------------------+------------------------+-------------------+-------------------------------+---------------------------------+-------------------------------+-------------------------------+----------------------------------------+--------+------------|
  | ds_abcd1234_ef56_7890_abcd_1234ef567890 | 2024-10-23 10:12:34.567 | jane.doe@example.com  | ACTIVE | DATA_STREAM_TABLE | a1bcdef2-3456-7890-1234-b567c890d123 | <db>.<schema>.<table_or_view> | MyModel      | <db>.<schema>.<table_or_view> | SYNCED           | 0                      | NULL              | NULL                          | NULL                            | {"rows": 10, "size": 512, ... } 2024-10-23 10:50:00.456       | 02a1b234-5678-1234-abcdef-0123456789ab | []     | STARTED    |
  +-----------------------------------------+-------------------------+-----------------------+--------+-------------------+--------------------------------------+-------------------------------+--------------+-------------------------------+------------------+------------------------+-------------------+-------------------------------+---------------------------------+-------------------------------+-------------------------------+----------------------------------------+--------+------------+ */

See Data Management for more information data streams.

See Also#