Skip to main content

Introspection

Restate exposes information on invocations and application state via its CLI and Introspection SQL API. You can use this to gain insight into the status of invocations and the service state that is stored.

This can be useful for troubleshooting. For example, a Virtual Object might be blocked and you want to kill the invocation that is blocking it, but you don't know the invocation ID. Or you want to check what is currently stored in the state of a service.

Restate exposes the following SQL tables, queryable via the psql client:

  • sys_invocation table to inspect invocations
  • sys_inbox table to inspect queue of pending invocations
  • sys_keyed_service_status table to inspect the status of a Virtual Object
  • sys_journal table to inspect the invocations' journal
  • sys_service table to inspect the registered services
  • sys_deployment table to inspect service deployments
  • sys_idempotency table to inspect idempotency keys
  • state table to inspect application state

You can find the schema of each of the tables in the references.

Install the psql client.

Connect with psql to Restate:


psql -h localhost -p 9071

You can change the default port 9071 by supplying the -p argument.

To retrieve the schema of the any of the tables, connect the psql client to Restate and execute:


describe <TABLE_NAME>;

For example:


describe sys_invocation;

The Restate Introspection SQL API has been implemented based on DataFusion and supports standard SQL syntax.

Inspecting invocations​

Listing ongoing invocations​


select * from sys_invocation;

info

The sys_invocation table only contains entries for active invocations. Active invocations are invocations that haven't completed yet and are either invoked or suspended.

Retrieving the status of an invocation​

Retrieving the status of an invocation​


select target_service_name, target_handler_name, status from sys_invocation where id = '<INVOCATION_ID>';

The status is either:

  • pending: enqueued waiting for its turn
  • ready: ready to be processed, but not yet running
  • running: actively processing
  • backing-off: retrying due to a failure
  • suspended: waiting on some external input (e.g. request-response call, awakeable, sleep, ...)
  • completed: completed (this is shown only for idempotent invocations)

Inspecting the invocation journal​

You can retrieve the journal of an invocation by querying the sys_journal table:


select * from sys_journal where id = '<INVOCATION_ID>';

If you don't know the invocation id of your request, you can find it in the sys_invocation table as described above

Inspecting invocation retries​

To have a look at the invocations that are currently in a retry loop, you can execute:


select * from sys_invocation where retry_count > 1;

The metadata around these attempts is not committed durably to storage, but is held in memory in the invoker component. This metadata can be found in the sys_invocation table.

Listing invocations that are blocking a Virtual Object​

You can retrieve the invocation ID that is currently blocking a Virtual Object via:


select invocation_id from sys_keyed_service_status where service_name = 'test.MyServiceName' and service_key = 'myKey';

You can then use the invocation ID to kill the invocation.

Checking the last time an invocation was modified​


select modified_at from sys_invocation where id = '<INVOCATION_ID>';

This includes any modification to the row in the table (e.g. when the service last switched its status from invoked to suspended, or when the last journal entry was added).

Checking how an invocation was triggered​

To find out if an invocation was triggered via the ingress or by another service:


select invoked_by, invoked_by_service_name, invoked_by_id from sys_invocation where id = '<INVOCATION_ID>';

The invoked_by field contains either ingress or service. If the invocation was triggered by another service, then the fields invoked_by_service_name and invoked_by_id will supply more information about the invoking service.

Retrieving the trace ID of an invocation​


select trace_id from sys_invocation where id = '<INVOCATION_ID>';

Afterwards, you can use this trace ID to search for spans in Jaeger.

Listing inactive invocations​

To retrieve all invocations that have not seen any activity for more than 1 hour:


select * from sys_invocation where to_timestamp(modified_at) <= now() - interval '1' hour;

Listing zombie invocations​

Zombie invocations are invocations that are pinned to a specific deployment but that deployment was forcefully removed. You can list them by executing:

This is not possible via only psql.

If you know the ID of the deployment that got forcefully removed, you can check if there are zombie invocations pinned to it via:


select * from sys_invocation where pinned_deployment_id = '<DEPLOYMENT_ID>';

Inspecting application state​

To retrieve the state of a specific service and service key, do:

You can query the application state via the state table.


select * from state where service_name = 'test.MyServiceName' and service_key = 'myKey';

Take into account the data types of the included columns

If your state value is a regular string, then you can access its content in the column value_utf8.

To retrieve the state of a specific service name, service key and state key, do:


select * from state where service_name = 'MyServiceName' and service_key = 'myKey' and key = 'myStateKey';

The state key is the name you used to store the state with the SDK. For example, the code snippet ctx.set("count", 1) stores 1 under the key count.

Joining the tables

To join the sys_invocation and state table:


select * from sys_invocation JOIN state on sys_invocation.target_service_name = state.service_name and sys_invocation.target_service_key = state.service_key;

To edit application state​

To edit the application state, do:

restate kv edit <SERVICE_NAME> <SERVICE_KEY>

This command opens your default editor (as configured in the cli env). It sends the new state values back to the runtime to be applied.

State as binary

Use --binary if the values are not JSON-encoded UTF-8 strings. In this case, you need to decode the base64-encoded string, and encode it back to base64 after editing.

State as JSON objects

Use --plain to retrieve the state as a JSON object. This can be useful in combination with tools like jq for example:

restate kv get counter bob --plain | jq '.seen'
Editing state during ongoing invocations

If during the editing of the state with the CLI, an invocation changed the state as well, then the edit of the CLI will not take affect. If you want the CLI state edit to be applied even if the state has changed in the meantime, then use the --force flag.

An example on how to edit the K/V state of the service counter for the key bob:

restate kv edit counter bob
Output
ℹ️ About to write the following state :
―――――――――――――――――――――――――――――――――――――――


service counter
key bob
force? false
binary? false

KEY VAL
seen 8

✔ Are you sure? · yes


Enqueued successfully for processing