create_data_stream()#

relationalai.api
#create_data_stream(
    object_reference OBJECT,
    model_name STRING,
    stream_name STRING,
    force_overwrite BOOLEAN DEFAULT FALSE
)

Creates a new data stream from an object reference for a Snowflake table or view. At least one engine with a READY state must be available to create a data stream. You must have SELECT privileges and change tracking must be enabled on the source object. See Supported Column Types for a list of column types supported in a data stream’s source table or view. Requires the cdc_admin application role.

Parameters#

NameTypeDescription
object_referenceOBJECTAn object reference to a Snowflake table or view.
model_nameSTRINGThe name of the model associated with the data stream.
stream_nameSTRINGThe name of the data stream to create. Must be contain only characters A-Z, a-z, or 0-9 and may not start with a number.
force_overwriteBOOLEANIf TRUE, overwrites data from a previously existing data stream with the same name. (Default: FALSE)

Returns#

STRING

Supported Column Types#

Data streams support the following Snowflake column types:

Tables or views with unsupported column types cannot be used as data stream sources.

NOTE

If you need to stream data from a source object with unsupported column types, consider creating a view without those columns or that casts the unsupported columns to a supported type.

Example#

Data streams are created automatically by the Python API when a Snowflake source table or view is referenced by a RAI model. However, you may need to create a stream manually if the stream has been quaratined due to synchronization issues or has become stale.

Use the api.create_data_stream() procedure to create a new data stream:

#-- Replace the placeholders with your database, schema, and table/view names.
SET obj_name = '<db>.<schema>.<table_or_view>';
SET obj_type = 'TABLE';  -- Set to 'VIEW' if needed.
SET obj_ref = relationalai.api.object_reference($obj_type, $obj_name);

-- Enable change tracking on the table or view
-- (if you have a custom procedure handling this, since Snowflake lacks direct support).
ALTER TABLE IDENTIFIER($obj_name) SET CHANGE_TRACKING = TRUE;

-- Stream the table or view into the model 'MyModel' with the name 'my_stream'.
CALL relationalai.api.create_data_stream($obj_ref, 'MyModel', 'my_stream', TRUE);
/*+----------------------------------+
  | Datastream created successfully. |
  +----------------------------------+ */

Note that when you delete a data stream, models retain access to a snapshot of the most recent data processed by the stream. If you are creating a data stream with the same name as a previously deleted stream, you must set the force_overwrite parameter to TRUE to overwrite the data in the snapshot, as shown in the above example.

NOTE

A total of 2000 object references may be created, 1000 table references and 1000 view references.

See Data Management for more information data streams.

See Also#