Public Cloud – Flexible Engine

Data Ingestion Service – collect and process your business data in real time

Real-time streaming service which is capable of capturing and processing large amount of streaming data

The Data Ingestion Service (DIS) is a highly scalable real-time streaming service which is capable of capturing and processing large amount of streaming data for specific needs. Data send to DIS can be stored for offline processing and analytics. With Data Ingestions Service enrich their application through deeper understanding of their data. DIS can be useful to thousands of existing real world scenarios such as capturing IoT sensor data, website clickstreams, stock transactions, social feeds, mobile app gaming telemetries or sensors from autonomous vehicles enabling deeper and enriching application experience through data.

DIS is a high-throughput, distributed message Publish-Subscribe managed web service system. User could create, delete, describe stream throughput through a simple web console.

Application scenarios

IoT data Capture

Capture real-time data from thousands of embedded devices sensors and push data to DIS, making the data available to you as soon as it is produced on the sensors.

Website clickstreams analysis

User could integrate DIS to help categories and visualize the source of web user traffics in real-time.

Example, owner could collect information on geographic location of customer to reroute user to faster edge node and providing a better experience. Owner could also collect the occupancy time of customer spending on the page and interactively display better product suggestions or ads.

Real-time metrics analysis

User can use data collected into Streams for simple data analysis and reporting in real time to better let User understand the status of the system or monitor any critical metrics concerning their system or business. For example, your data-processing application can work on metrics and reporting for system and application logs as the data is streaming in, rather than wait to receive batches of data allow them to quickly make or fix any issues.

Gaming Data Feed

User can use DIS to collect data about player-game locations, interactions and feed data to an AI system to analyze and produce dynamic experiences and engagement based on players’ actions, behaviors, location and their usage pattern.

Benefits

Real-time data processing

You can continuously collect data as it is generated and promptly react to critical information about your business and operations.

Simple and easy-to-use

You can create a DIS stream within seconds

Scalable streams

User can create stream throughput of megabytes to hundreds of megabytes per second.

LowCost

There’s no upfront cost and user only pay for the resources he/she use.

Reliable

Data Ingestion Service synchronously replicates user streaming data across multiple logical racks and can preserves user data up to 7 days, preventing data loss in case of application failure.

Secure

Data encryption transmission, user data isolation storage.

Limitations

  • Data records are accessible for a default of 24 hours from the time they are added to a stream. Can be extended to 7 days.
  • Each partition can support up to 1,000 records per second for write, up to maximum total data write rate of 1 MB per second.
  • Each partition can support up to a maximum data read rate of 2 MB per second.
  • Currently, DIS store data in raw format pushed by producer. It’s recommended that data should first be encrypted on producer end decrypted on consumer side if user wanted to have their data secure.