Your Search did not match any results
The Oracle Cloud Infrastructure Streaming service provides a fully managed, scalable, and durable storage option for continuous, high-volume streams of data that you can consume and process in near real-time.
It's fully managed; from the underlying infrastructure to provisioning, deployment, maintenance, security patches, replication and consumer groups, which makes application development easier.
When you create a stream inside Oracle Cloud Infrastructure Streaming, Oracle automatically creates and manages 3 streaming nodes distributed across 3 different AD(s) (or fault domains for single AD-regions), ensuring that your streams stay highly available and your data highly durable.
OSS allows you to emit data and retrieve the data in near real time. The number of use cases are nearly unlimited, from messaging to complex data streams processing.
Here are some of the many possible uses for Streaming:
Start using OSS by:
A Stream can be viewed as an append-only log file that contains your messages.
Streams are divided into a number of partitions for scalability. Partitions allow you to distribute a stream by splitting the messages across multiple nodes (or brokers) — each partition can be placed on a separate machine to allow for multiple consumers to read from a topic in parallel.
A 64-bit encoded message is what you emit into a topic.
Each message within a partition has an identifier called its offset. Consumers can read messages starting from a specific offset and are allowed to read from any offset point they choose. Consumers can also commit the latest processed offset so they can resume their work without replaying or missing a message if they stop and then restart.
A key is an identifier used to group related messages.
You can create a new stream by using our Console or our API. See API here.
Your stream is created for a particular region and tenancy and optionally for a dedicated compartment. A steam's data is replicated across the entire region allowing it to tolerate AD loss or network splits without disrupting the service and offers built-in high availability in a region.
The time to provision depends on the number of partitions. Creating a new partition takes up to 10 seconds.
The number of partitions for your stream depends on the throughput expectations of your application (expected throughput = average recond size x maximum number of records written per sectond).
The throughput of a Oracle Cloud Infrastructure stream is defined by a partition. A partition provides 1MB/sec data input and 2MB/sec data output.
You can send 1,000 requests per second to a partition.
Once a stream is created and active you can publish messages. For publishing, you can use the Write API (putMessages). The message will be published to a partition in the stream. If there is more than one partition, the partition where the message will be published is calculated using the message's key.
If the key is null, the partition will be calculated using a subset of the value. For messages with a null key, do not expect messages with same value to go on the same partition, since the partitioning scheme may change; sending a null key will effectively put the message in a random partition.
If you want to make sure that messages with the same value go to the same partition, you should use the same key for those messages.
As soon as the OSS API acknowledges your putMessage without error, this messages is durable.
OSS guarantees linearizable reads and writes to a partitioning key.
When client requests exceed the limits, OSS denies the request and send out an error exception message.
Consuming messages requires you to:
Refer to the technical documentation for step by step guide on consuming data from a stream.
OSS provides two kinds of consume API:
Consumers can be configured to consume messages as part of a group. Stream partitions are distributed among members of a group so that messages from any single partition will only be sent to a single consumer.
Partition assignments are re-balanced as consumers join or leave the group.
We recommend that consumer applications take care of duplicates.
If you want to know if your consumer is falling behind (you are producing faster than you are consuming), you can use the difference between timestamp of the message and the current time. If this number gets higher, you might want to spawn a new consumer to take over some of the partitions from your first consumer.
We recommend customers allocate partitions slightly higher than their maximum throughput. This will help them to manage their application spikes as we currently don't support changing the number of partition once a stream is created.
By default, we store data for 24 hours. You can set up the retention period up to 7 days while creating a stream. Once retention period is defined, it can't be edited.
The Oracle Cloud Infrastructure Streaming console provides both operational and performance metrics, such as throughput of data input and output. OSS also integrates with Oracle Cloud Infrastructure Telemetry so that you can collect, view, and analyze telemetry metrics for your streams.
All streams in the same tenancy have unique immutable names. Every stream has a compartment assigned. So, all the power of Oracle Cloud Infrastructure access control policies may be used to describe fine-grained rules at the tenancy, compartment, or single stream level.
Access policy is specified in a form of "Allow to in where ".
Our internet API uses the Oracle Identity service. Oracle Identity Service provides convenient way to authenticate users and authorize an access to such APIs from both browser (Username/password) and code (API Key).
OSS is secure by default - User data is encrypted both at rest and in motion. Only the account and data stream owners have access to the stream resources they create. OSS supports user authentication to control access to data. You can use Oracle Cloud Infrastructure IAM policies to selectively grant permissions to users and groups of users. You can securely put and get your data from OSS through SSL endpoints using the HTTPS protocol.
You own the data you emit; you can encrypt your data before sending it to OSS.
Ingestion (your producer - Streaming gateway): Data encrypted in motion due to SSL (HTTPS).
Inside of streaming service: On the gateway SSL gets terminated, data is encrypted upon arriving with per-stream AES-128 key, and is sent to the storage layer for persistence.
On consumption: Encrypted data is read from OSS, decrypted by the gateway node, and sent to consumer over SSL.
OSS uses AES-GCM 128 algorithm for encryption.
OSS uses simple pay-as-you-go pricing. There are no upfront costs or minimum fees, and you only pay for the resources you use.
OSS doesn't have a free tier.