Skip to Content
This documentation is provided with the HEAT environment and is relevant for this HEAT instance only.
Core ConceptsIngests

Ingests

In HEAT, an Ingest is a managed endpoint that triggers the creation or augmentation of a Session. By sending data (e.g., a file, JSON payload, or CSV) to an Ingest’s URL, you begin the data-processing workflow defined by a specified Session Template in a particular Project.

How Ingests Work

  1. Programmatic Endpoint
    An Ingest provides an HTTP endpoint (/ingest/...) that you can POST data to. You can define a custom URL suffix for clarity (e.g., /ingest/my-simulator), but all Ingests reside under the /ingest/ prefix.

  2. Session Creation or Augmentation
    When you post data to this endpoint, HEAT automatically:

    • Creates a Session based on the assigned Session Template if no existing session is found.
    • Inserts your submitted payload into the input node specified by the template.
    • Optionally updates an existing Session if you’ve configured the Ingest to group multiple data uploads into a single Session.

  3. Data Pipeline Kickstart
    Once the payload is placed into the designated input node, HEAT begins executing the data-processing pipeline (DAG) specified in the chosen Session Template. This includes any transformations, analytics, or validation steps defined by the workflow.

Creating and Configuring an Ingest

Creation Methods

  • Cluster Manager
    Use the Cluster Manager’s GUI to create a new Ingest. Specify a desired endpoint path, target Project, Session Template, and input node.
  • HEAT API
    Programmatically create Ingests via the HEAT API. This approach allows automated setup or dynamic provisioning when new integrations come online.

Required Settings

  • Project & Session Template
    Each Ingest must be tied to a single Project and Session Template. These define where and how incoming data is processed.
  • Input Node
    Identify which node in the Session Template will receive the uploaded data. This node is typically configured to parse CSV, JSON, or handle binary files.

Data Format Rules

  • Payload Type
    You can configure each Ingest to accept:
    • Any Data: Files of any type are accepted.
    • JSON Only: Strictly JSON payloads.
    • CSV Only: Strictly CSV files.
  • Anonymous or Keyed Access
    • Anonymous: No authentication required, allowing any client to POST data.
    • Secret Key: The request must include a designated key for security. This helps ensure only authorized clients can trigger sessions.

Session Augmentation

Ingests can optionally be configured to append data to an existing session if certain conditions are met (e.g., matching an identifier or session tag). This is useful when multiple data files combine to form a single training session, or when data arrives incrementally.

Usage Example

  1. Create an Ingest
    Through Cluster Manager or the API, define an endpoint such as /ingest/my-sim. Map it to:

    • Project: DefaultProject
    • Session Template: MySimulationTemplate
    • Input Node: RawDataNode

  2. Send Data
    Use a tool like curl to post a JSON or CSV payload:

    curl -X POST -H "Content-Type: application/json" \
  -d '{"playerID":1234, "actions":[...]}'
  https://heat.example.com/ingest/my-sim
Data SourcesNode Instances