ProtobufSyncConfiguration

Describes a single message type.

A set of one or more of these messages is specified in `ProtobufSyncConfigurationHeader`.

FieldTypeDescription
idstring

Optional id to associate with this configuration.

This can be used in the `header_id` field of `ProtobufSyncWrapper` above to reference the configuration.

If omitted the numerical index of the configuration in `ProtobufSyncConfigurationHeader` will be used instead.

message_protostring

The actual protobuf definition text.

Copy and paste the source protobuf definition from the `.proto` file.

Only simple protobuf structures are currently supported, e.g. no imports from other packages etc.

message_namestring

The name of the message within `message_proto` above that represents the data to be sync'd, e.g. `TCPDumpPacket`.

table_namestring

The name of the table within the target database into which the message data for this type should be written.

ProtobufSyncConfigurationHeader

This message is written at the beginning of every file to be ingested using the `protoDbSync` utility.

It contains a `header` entry for each message type that may appear in the file.

If the `volt logger` command is used, it will create this header automatically based on the configuration it's given.

FieldTypeDescription
idstring

This should ideally be a persistent UUID, at minimum it must be unique within the set of types of file any given instance of `protoDbSync` is processing in a given folder.

It is used to match up orphaned or split packets that might occur when receiving data from a wire, for example, if a log file is rotated midway through a packet arriving on the wire.

This id should persist for the life time of the set of data it describes, i.e. if a wire publication is stopped and restarted at some later point, the same id should be used if possible.

configurationProtobufSyncConfiguration repeated

The set of possible configurations that can appear in any given protobuf sync data file.

A serialised instance of this message must appear at the top of each data file.

Each subsequent serialised message in the data file must be an instance of `ProtobufSyncWrapper`, and the `header_lookup` field refers to an entry in this list.

maximum_message_sizeint32

Optional maximum size of the serialised messages, this doesn't need to be exact and the default is 64K if omitted.

ProtobufSyncWrapper

Wraps arbitrary protobuf messages, with an index into the `ProtobufSyncConfigurationHeader` to indicate the specific message type this message wraps.

FieldTypeDescription
header_indexuint32

The index number of the header for this message type in the Volt logger configuration file.

header_idstring

The name of the header for this message type, will be used to lookup against the `id` field in `ProtobufSyncConfiguration`.

This will incur an overhead in terms of the packet size, but might be preferrable if volume is low or managing the header index is difficult.

payloadbytes

The message payload, in serialised protobuf binary format.

n.b. the serialisation should **not** be length-prefixed.

Scalar Value Types

.proto TypeNotesC++JavaPythonGoC#PHPRuby
doubledoubledoublefloatfloat64doublefloatFloat
floatfloatfloatfloatfloat32floatfloatFloat
int32Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead.int32intintint32intintegerBignum or Fixnum (as required)
int64Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead.int64longint/longint64longinteger/stringBignum
uint32Uses variable-length encoding.uint32intint/longuint32uintintegerBignum or Fixnum (as required)
uint64Uses variable-length encoding.uint64longint/longuint64ulonginteger/stringBignum or Fixnum (as required)
sint32Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s.int32intintint32intintegerBignum or Fixnum (as required)
sint64Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s.int64longint/longint64longinteger/stringBignum
fixed32Always four bytes. More efficient than uint32 if values are often greater than 2^28.uint32intintuint32uintintegerBignum or Fixnum (as required)
fixed64Always eight bytes. More efficient than uint64 if values are often greater than 2^56.uint64longint/longuint64ulonginteger/stringBignum
sfixed32Always four bytes.int32intintint32intintegerBignum or Fixnum (as required)
sfixed64Always eight bytes.int64longint/longint64longinteger/stringBignum
boolboolbooleanbooleanboolboolbooleanTrueClass/FalseClass
stringA string must always contain UTF-8 encoded or 7-bit ASCII text.stringStringstr/unicodestringstringstringString (UTF-8)
bytesMay contain any arbitrary sequence of bytes.stringByteStringstr[]byteByteStringstringString (ASCII-8BIT)