create_data_stream()#

relationalai.api
#create_data_stream(
    object_reference OBJECT,
    model_name STRING,
    stream_name STRING,
    force_overwrite BOOLEAN DEFAULT FALSE
)

Creates a new data stream from an object reference for a Snowflake table or view. At least one engine with a READY state must be available to create a data stream. You must have SELECT privileges and change tracking must be enabled on the source object. See Supported Column Types for a list of column types supported in a data stream’s source table or view. Requires the cdc_admin application role.

Parameters#

NameTypeDescription
object_referenceOBJECTAn object reference to a Snowflake table or view.
model_nameSTRINGThe name of the model associated with the data stream.
stream_nameSTRINGThe name of the data stream to create. This is the name Python users use to reference the stream in a model. Typically, the stream name is the fully-qualified name of the source table or view, e.g. '<db>.<schema>.<table>'
force_overwriteBOOLEANIf TRUE, overwrites data from a previously existing data stream with the same name. (Default: FALSE)

Returns#

STRING

Supported Column Types#

Data streams support the following Snowflake column types:

Tables or views with unsupported column types cannot be used as data stream sources.

NOTE

If you need to stream data from a source object with unsupported column types, consider creating a view without those columns or that casts the unsupported columns to a supported type.

Example#

Use the api.create_data_stream() procedure to create a new data stream to share data from a Snowflake table or view with a RAI Python model:

#-- Replace the placeholders with your database, schema, and table/view names.
SET obj_name = '<db>.<schema>.<table_or_view>';
SET obj_type = 'TABLE';  -- Set to 'VIEW' if needed.
SET obj_ref = relationalai.api.object_reference($obj_type, $obj_name);

-- Enable change tracking on the table or view
-- (if you have a custom procedure handling this, since Snowflake lacks direct support).
ALTER TABLE IDENTIFIER($obj_name) SET CHANGE_TRACKING = TRUE;

-- Stream the table or view into the model 'MyModel', using the fully-qualified
-- name as the name of the data stream.
CALL relationalai.api.create_data_stream($obj_ref, 'MyModel', $obj_name, TRUE);
/*+----------------------------------+
  | Datastream created successfully. |
  +----------------------------------+ */

Note that when you delete a data stream, models retain access to a snapshot of the most recent data processed by the stream. If you are creating a data stream with the same name as a previously deleted stream, you must set the force_overwrite parameter to TRUE to overwrite the data in the snapshot:

#-- Stream the table or view into the model 'MyModel', overwriting the data from
-- a previously deleted stream with the same name.
CALL relationalai.api.create_data_stream(
    relationalai.api.object_reference('TABLE', $table_or_view),
    'MyModel',
    $table_or_view,
    TRUE  -- Force overwrite
);
/*+----------------------------------+
  | Datastream created successfully. |
  +----------------------------------+ */
NOTE

A total of 200 object references may be created, 100 table references and 100 view references. However, each source object may have multiple streams associated with it.

See Data Management for more information data streams.

See Also#