ProtobufSyncConfiguration
Describes a single message type.
A set of one or more of these messages is specified in `ProtobufSyncConfigurationHeader`.
Field | Type | Description |
id | string | Optional id to associate with this configuration. This can be used in the `header_id` field of `ProtobufSyncWrapper` above to reference the configuration. If omitted the numerical index of the configuration in `ProtobufSyncConfigurationHeader` will be used instead. |
message_proto | string | The actual protobuf definition text. Copy and paste the source protobuf definition from the `.proto` file. Only simple protobuf structures are currently supported, e.g. no imports from other packages etc. |
message_name | string | The name of the message within `message_proto` above that represents the data to be sync'd, e.g. `TCPDumpPacket`. |
table_name | string | The name of the table within the target database into which the message data for this type should be written. |
ProtobufSyncConfigurationHeader
This message is written at the beginning of every file to be ingested using the `protoDbSync` utility.
It contains a `header` entry for each message type that may appear in the file.
If the `volt logger` command is used, it will create this header automatically based on the configuration it's given.
Field | Type | Description |
id | string | This should ideally be a persistent UUID, at minimum it must be unique within the set of types of file any given instance of `protoDbSync` is processing in a given folder. It is used to match up orphaned or split packets that might occur when receiving data from a wire, for example, if a log file is rotated midway through a packet arriving on the wire. This id should persist for the life time of the set of data it describes, i.e. if a wire publication is stopped and restarted at some later point, the same id should be used if possible. |
configuration | ProtobufSyncConfiguration repeated | The set of possible configurations that can appear in any given protobuf sync data file. A serialised instance of this message must appear at the top of each data file. Each subsequent serialised message in the data file must be an instance of `ProtobufSyncWrapper`, and the `header_lookup` field refers to an entry in this list. |
maximum_message_size | int32 | Optional maximum size of the serialised messages, this doesn't need to be exact and the default is 64K if omitted. |
ProtobufSyncWrapper
Wraps arbitrary protobuf messages, with an index into the `ProtobufSyncConfigurationHeader` to indicate the specific message type this message wraps.
Field | Type | Description |
header_index | uint32 | The index number of the header for this message type in the Volt logger configuration file. |
header_id | string | The name of the header for this message type, will be used to lookup against the `id` field in `ProtobufSyncConfiguration`. This will incur an overhead in terms of the packet size, but might be preferrable if volume is low or managing the header index is difficult. |
payload | bytes | The message payload, in serialised protobuf binary format. n.b. the serialisation should **not** be length-prefixed. |
Scalar Value Types
.proto Type | Notes | C++ | Java | Python | Go | C# | PHP | Ruby |
double | double | double | float | float64 | double | float | Float | |
float | float | float | float | float32 | float | float | Float | |
int32 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) |
int64 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. | int64 | long | int/long | int64 | long | integer/string | Bignum |
uint32 | Uses variable-length encoding. | uint32 | int | int/long | uint32 | uint | integer | Bignum or Fixnum (as required) |
uint64 | Uses variable-length encoding. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum or Fixnum (as required) |
sint32 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) |
sint64 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. | int64 | long | int/long | int64 | long | integer/string | Bignum |
fixed32 | Always four bytes. More efficient than uint32 if values are often greater than 2^28. | uint32 | int | int | uint32 | uint | integer | Bignum or Fixnum (as required) |
fixed64 | Always eight bytes. More efficient than uint64 if values are often greater than 2^56. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum |
sfixed32 | Always four bytes. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) |
sfixed64 | Always eight bytes. | int64 | long | int/long | int64 | long | integer/string | Bignum |
bool | bool | boolean | boolean | bool | bool | boolean | TrueClass/FalseClass | |
string | A string must always contain UTF-8 encoded or 7-bit ASCII text. | string | String | str/unicode | string | string | string | String (UTF-8) |
bytes | May contain any arbitrary sequence of bytes. | string | ByteString | str | []byte | ByteString | string | String (ASCII-8BIT) |