Intro
The OCI Streaming service is storage solution for ingesting continuous, high-volume streams of data that you can consume and process in real time. Streaming can be used for messaging, ingesting high-volume data such as application logs, and other use cases in which data is produced and processed continually and sequentially.Design
In Streaming there is producer and consumer.- A producer publishes messages to a stream, which is an append-only log. These messages can be distributed among the partitions (nodes/machines) for scalability.
- A consumer can read messages from streams. Each message is marked with an offset value, so a consumer can pick up where it left off.
Usage
To create a Stream : Burger menu - Analytics - Streaming- Create some stream pool first, for ex. admin-stream-pool
- Create stream
ex.
name: zd-stream-log
Messages Endpoint: https://api.cell-1.<oci-region>.streaming.oci.oraclecloud.com
Stream Pool: admin-stream-pool
Number of partitions: 1
Retention: 24h
Read Throughput: 2 MB/s
Write Throughput: 1 MB/s - To test stream, click "Produce test message", type some message and expect to see "success"
- To see 50 messages in last min, click "Load Messages", and you should see your test.
To emit or put message into stream.
- We can use
command in this example, and run it in OCI console shell. - I am not sure on proper input, so I use this command to see what's JSON file syntax to use.
ZARKO@cloudshell: oci streaming stream message put --generate-param-json-input messages
- The output is :
[ { "key": "string", "value": "string" }, { "key": "string", "value": "string" } ]
- The string is base64 data, hence I'll be using this one for "city,SF,state,CA"
echo city | base64 Y2l0eQo= echo SF | base64 U0YK echo state | base64 c3RhdGUK echo CA | base64 Q0EK
- So my json file (stream-input.json) with stream input data looks:
[ { "key": "Y2l0eQo=", "value": "U0YK" }, { "key": "c3RhdGUK", "value": "Q0EK" } ]
- And my final command to emit into Stream (I know stream-id and endpoint from stream creation action):
ZARKO@cloudshell: oci streaming stream message put --stream-id ocid1.stream.oc1.iad.aaaaaa-shortened-kdxlp4dtkfimh6p6h5aa \ --messages file://stream-input.json --endpoint https://api.cell-1.<oci-region>.streaming.oci.oraclecloud.com
- Recent messages are:
First we need to create a Cursor , in order to read from Stream.