PCIe Co-processing

The Neat PCIe host API lets an application on a host machine send tensors or images to a connected Modalix PCIe Card and receive inference results. Use it when the host machine owns application I/O and orchestration while the card runs the compiled model and its configured preprocessing or postprocessing.

This is a separate API from the Neat Library that runs directly on a DevKit. The public types are in the simaai::neat::pcie C++ namespace and the pyneatpcie Python package.

Install on the host machine

Install core/pciehost on the host machine, not inside the Neat SDK container or on the Modalix PCIe Card. Follow Install PCIe Host before using this page.

How co-processing works

One pcie::Model represents one compiled model running on one physical PCIe queue:

1. The constructor reads the local model archive and exposes its input and output contract.
2. build() uploads the archive to the card over the PCIe virtual network, starts the card-side pipeline, and waits until it is ready.
3. run() or push() sends input payloads over PCIe.
4. The card executes preprocessing, inference, and configured postprocessing.
5. run() or pull() returns the output tensors to the host.
6. close() stops the card-side pipeline and releases the queue.

The model archive is transferred during build(). Inference payloads and results use the PCIe data transport.

Configure the connection

ConnectionOptions identifies the card and the queue used by this model.

Field	Default	Purpose
card_host	empty	Explicit SSH/SCP address. When empty, card N uses 10.0.N.2.
card_id	0	Card number passed to the host PCIe plugin.
user	sima	User for card-side SSH and SCP.
queue	0	Co-processing queue, from 0 through 3.
max_inflight	10	Maximum accepted inputs waiting for results.

Use the defaults for one card at 10.0.0.2 on queue 0. Set card_host explicitly when the card uses a different management address.

#include <simaai/neat/pcie/Model.h>

namespace pcie = simaai::neat::pcie;

pcie::ConnectionOptions connection;
connection.card_host = "10.0.0.2";
connection.card_id = 0;
connection.queue = 0;
connection.max_inflight = 10;

Inspect and build a model

Construction is local and does not start the card. Inspect info() before allocating input, then call build() once to start the co-processing session.

pcie::Model model("model.tar.gz", {}, connection);

const pcie::ModelInfo info = model.info();
for (const auto& input : info.inputs) {
  std::cout << input.name << " requires " << input.size_bytes << " bytes\n";
}

model.build(/*readiness_timeout_ms=*/180000);

input_specs() and output_specs() return the same lists individually. running() becomes true after a successful build and returns to false after close(). It reports lifecycle state only; it does not probe host transport or remote pipeline health.

Run synchronous inference

Use run() for the simplest request/response flow. Build the model first and use a finite timeout so application failures do not wait indefinitely. The following example constructs an input for a model whose reported input datatype is FP32.

const auto& input_spec = info.inputs.front();
if (input_spec.dtype != "FP32") {
  throw std::runtime_error("this example requires an FP32 model input");
}

std::vector<float> values(input_spec.size_bytes / sizeof(float), 0.0f);
pcie::Tensor input = pcie::Tensor::from_vector(
    std::move(values), input_spec.shape, input_spec.name);

pcie::TensorList outputs = model.run(input, /*timeout_ms=*/30000);

model.close();

For a multi-input model, pass one Tensor per logical input in the order and with the route names reported by info().inputs.

A run() timeout stops waiting but does not cancel an input already accepted by the card. After catching a timeout, either use pull() to drain that outstanding result or call close() before starting a new request sequence.

Pipeline requests with push and pull

Use push() and pull() when input preparation should overlap inference. max_inflight bounds the accepted work that has not yet returned. Pull results promptly so producers can continue.

std::size_t pushed = 0;
std::size_t pulled = 0;
while (pulled < inputs.size()) {
  while (pushed < inputs.size() && pushed - pulled < 10) {
    model.push(inputs[pushed++]);
  }

  auto outputs = model.pull(/*timeout_ms=*/30000);
  if (!outputs) {
    throw std::runtime_error("PCIe inference timed out");
  }
  consume(*outputs);
  ++pulled;
}

push() waits when max_inflight is full, so do not submit more than the configured window without pulling results. pull() returns the next available result for this model. Drain all results submitted with push() before calling run().

Send images and configure preprocessing

Set preprocess.kind to Image when sending decoded image data. The card-side Neat pipeline can resize, convert color, normalize, and decode supported object detection outputs.

This example sends a BGR image, letterboxes it to the model input inferred from the model archive, and returns a tensor containing the decoded YOLOv8 BBOX payload.

#include <opencv2/imgcodecs.hpp>

pcie::ModelOptions options;
options.preprocess.kind = pcie::InputKind::Image;
options.preprocess.color_convert.input_format = pcie::ColorFormat::BGR;
options.preprocess.resize.enable = pcie::AutoFlag::On;
options.preprocess.resize.mode = pcie::ResizeMode::Letterbox;
options.decode_type = pcie::BoxDecodeType::YoloV8;
options.score_threshold = 0.25f;
options.nms_iou_threshold = 0.45f;
options.top_k = 100;

pcie::Model detector("yolo_v8n_mpk.tar.gz", options, connection);
detector.build();

cv::Mat image = cv::imread("image.jpg", cv::IMREAD_COLOR);
pcie::TensorList detections = detector.run(image, /*timeout_ms=*/30000);
detector.close();

Do not set input_max_width, input_max_height, or input_max_depth for a seedless model unless the application needs an explicit input limit. Neat can infer the model-side resize target from the model archive.

Run the MLA only

By default the card converts the FP32 input to the MLA's dtype, runs the MLA, and converts the results back before returning them as FP32. mla_only moves that dtype-conversion boundary to the application: the card then runs nothing but the MLA, the host submits tensors that match each entry of info().inputs, and it receives the MLA's native outputs in their logical shapes.

info() describes that contract. An INT8 tensor carries quant with one scale and one zero_point:

x = (q - zero_point) * scale
q = clamp(round(x / scale) + zero_point, -128, 127)

A BF16 tensor carries no quant: the host rounds FP32 to BF16 with round to nearest even on the way in and widens the BF16 heads on the way out. In Python, BF16 travels as uint16 bit patterns through Tensor.from_bytes(..., pcie.TensorDType.BFloat16, ...) and to_numpy().

The example below is the INT8 case:

pcie::ModelOptions options;
options.mla_only = true;
pcie::Model model("model_mlatess_int8.tar.gz", options, connection);

const auto& input_spec = model.info().inputs.front();
const pcie::QuantParams& quant = *input_spec.quant;   // INT8 ingress

std::vector<std::int8_t> codes(input_spec.size_bytes);
for (std::size_t i = 0; i < codes.size(); ++i) {
  const float x = /* value in the model's input domain, e.g. [0, 1] */;
  const float q = std::nearbyint(x / quant.scale) + quant.zero_point;
  codes[i] = static_cast<std::int8_t>(std::clamp(q, -128.0f, 127.0f));
}

model.build();
pcie::TensorList outputs = model.run(
    pcie::Tensor::from_vector(std::move(codes), input_spec.shape, input_spec.name),
    /*timeout_ms=*/30000);

const auto& head = model.info().outputs.front();
const std::int8_t* q = reinterpret_cast<const std::int8_t*>(
    static_cast<const std::uint8_t*>(outputs[0].data) + outputs[0].byte_offset);
const float x0 = (q[0] - head.quant->zero_point) * head.quant->scale;
model.close();

The route is all or nothing. Each submitted tensor must match the dtype, shape and byte size of the corresponding info().inputs entry, in that order; a payload of another dtype is rejected rather than converted on the card. mla_only cannot be combined with image preprocessing or box decode. The constructor rejects archives whose MLA stage is not compiled for direct MLA input and output, for example the Model Zoo yolo_v8s build, which tessellates on the CVU (mla_only does not support stage '...' (tess)).

Tutorial Run the MLA Only with INT8 Tensors shows the INT8 variant: it quantizes an image on the host and checks the dequantized heads against the default route.

Close reliably

Call close() when the model is no longer needed and before reusing its queue. It is safe to call more than once:

model = pcie.Model("model.tar.gz", connection=connection)
model.build()
outputs = model.run([input_tensor], timeout_ms=30000)
model.close()

Alternatively, use a context manager to close the model automatically:

with pcie.Model("model.tar.gz", connection=connection) as model:
    model.build()
    outputs = model.run([input_tensor], timeout_ms=30000)

The context manager calls close() when the block exits, including when an exception is raised. Do not add another explicit close() inside the with block.

Build a C++ host application

The development package provides the SimaPCIeHost CMake package:

CMakeLists.txt

cmake_minimum_required(VERSION 3.16)
project(pcie_model LANGUAGES CXX)

set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

find_package(SimaPCIeHost REQUIRED CONFIG)

add_executable(pcie_model main.cpp)
target_link_libraries(pcie_model PRIVATE SimaPCIeHost::sima_neat_pcie_host)

Build this application natively on the host machine.

The C++ image example also uses OpenCV. Add its headers and libraries to that application target:

find_package(OpenCV REQUIRED COMPONENTS core imgcodecs)
target_include_directories(pcie_model PRIVATE ${OpenCV_INCLUDE_DIRS})
target_link_libraries(pcie_model PRIVATE ${OpenCV_LIBS})

Current scope and limits

• One pcie::Model owns one PCIe queue. Queues range from 0 through 3.
• The Modalix EV74 supports at most four concurrent co-processing pipelines.
• Do not assign two active models to the same queue.
• The host package and the Neat Library installed on the card must be from compatible releases.
• Keep the input media type and geometry stable after the first submitted payload. A later payload larger than the active transport capacity is rejected.
• The PCIe host API supports a focused subset of Neat model preprocessing and object-decode options.
• mla_only requires an archive compiled for direct MLA input and output; each input is submitted in the dtype of its info().inputs entry and the heads return contiguous in the MLA's dtype.
• This co-processing API does not expose host-side Graph, Node, or Run composition. Use the regular Neat Library on a DevKit for native application graphs.

For installation and connectivity checks, return to Install PCIe Host.