create_data_stream()#
#create_data_stream(
object_reference OBJECT,
model_name STRING,
stream_name STRING,
force_overwrite BOOLEAN DEFAULT FALSE
)
Creates a new data stream from an object reference for a Snowflake table or view.
At least one engine with a READY
state must be available to create a data stream.
You must have SELECT
privileges and change tracking must be enabled on the source object.
See Supported Column Types for a list of column types supported in a data stream’s source table or view.
Requires the cdc_admin
application role.
Parameters#
Name | Type | Description |
---|---|---|
object_reference | OBJECT | An object reference to a Snowflake table or view. |
model_name | STRING | The name of the model associated with the data stream. |
stream_name | STRING | The name of the data stream to create. Must be contain only characters A-Z , a-z , or 0-9 and may not start with a number. |
force_overwrite | BOOLEAN | If TRUE , overwrites data from a previously existing data stream with the same name. (Default: FALSE ) |
Returns#
STRING
Supported Column Types#
Data streams support the following Snowflake column types:
Tables or views with unsupported column types cannot be used as data stream sources.
If you need to stream data from a source object with unsupported column types, consider creating a view without those columns or that casts the unsupported columns to a supported type.
Example#
Data streams are created automatically by the Python API when a Snowflake source table or view is referenced by a RAI model. However, you may need to create a stream manually if the stream has been quaratined due to synchronization issues or has become stale.
Use the api.create_data_stream()
procedure to create a new data stream:
#-- Replace the placeholders with your database, schema, and table/view names.
SET obj_name = '<db>.<schema>.<table_or_view>';
SET obj_type = 'TABLE'; -- Set to 'VIEW' if needed.
SET obj_ref = relationalai.api.object_reference($obj_type, $obj_name);
-- Enable change tracking on the table or view
-- (if you have a custom procedure handling this, since Snowflake lacks direct support).
ALTER TABLE IDENTIFIER($obj_name) SET CHANGE_TRACKING = TRUE;
-- Stream the table or view into the model 'MyModel' with the name 'my_stream'.
CALL relationalai.api.create_data_stream($obj_ref, 'MyModel', 'my_stream', TRUE);
/*+----------------------------------+
| Datastream created successfully. |
+----------------------------------+ */
Note that when you delete a data stream, models retain access to a snapshot of the most recent data processed by the stream.
If you are creating a data stream with the same name as a previously deleted stream, you must set the force_overwrite
parameter to TRUE
to overwrite the data in the snapshot, as shown in the above example.
A total of 2000 object references may be created, 1000 table references and 1000 view references.
See Data Management for more information data streams.