Using TCE

Using TCE

Submitting Transactions

Decide on a binary encoding for actions you want to be seen by other players. These will be the transactions you submit to TCE. This encoding does not need to conform to any specific format; use whatever makes sense for your application.

Once you have a Platform instance, you can queue transactions for consensus using .TrySend() or .SendBlocking(). For games, .TrySend() is likely preferable, as it will return immediately if the transaction queue is full. .SendBlocking() will block the current thread of execution, so you should not call it in your game's main loop. It is safe to call .SendBlocking() from another thread.

TCE pulls transactions from the queue to create events, which are the core message it exchanges with its peers. Because events have to be disseminated through the network, there is a theoretical maximum throughput the network can support at any one time, depending on the available bandwidth for every peer. For situations where the current throughput exceeds this available bandwidth (usually a dozen megabytes per second or more), TCE is designed to apply backpressure to the application by rejecting transactions when its queue is full, rather than allowing the queue to grow without bound.

Most applications should not need to worry about filling the transaction queue if submitting less than a megabyte of data per second or so, but it is still a situation to be mindful of.

Locally submitted transactions should not be applied to the application state until they come through consensus, as detailed in the next section.

The primary exception is if you're going to implement your own prediction and rollback system, or you're just using consensus ordering to resolve conflicting player actions.

Receiving Consensus Data

Platform emits data from the consensus algorithm using the .GetEvent() method. This method returns null if there is no new data instead of blocking, so it is recommended to check it every frame as part of your game's update loop. Multiple events may be buffered at one time, so it should be polled more than one per frame to empty the buffer, though you may want to use a bounded loop to allow the main loop to continue to the next frame.

The Event struct is a union of one of two other types: ConsensusEventorConsensusSyncPoint.

It may contain one or the other, but not both.

ConsensusEvent

This is the bread and butter of the consensus algorithm. A ConsensusEvent contains a list of transactions with a decided consensus order, along with two timestamps as well as some auxiliary data. Each ConsensusEvent is created by a single peer, or "creator"; you can find out who with the .CreatorPublicKey getter.

This will include events created by the local TCE instance in their proper consensus order, containing transactions submitted by the local application. They should be applied in consensus order as if they were from any other peer.

You can check if .CreatorPublicKey is equal to your own SecretKey.PublicKey to know if your local TCE instance created a given event.

The two timestamps are:

• TimestampCreated: this is the Unix timestamp, in nanoseconds, that the event was created at, according to the system clock of the peer who created it.

  • Events are created automatically by TCE so this timestamp will be sometime after the transactions contained in the event were submitted by the creator's game client, from that creator's perspective. The time difference depends on the rate at which TCE creates events, which it tunes dynamically to find the best balance of latency and CPU/bandwidth usage.

  • While TimestampCreated should always be increasing for all events created by the same creator, it is not guaranteed to be always increasing across the whole network. For example, it's likely for multiple creator' clocks to be running slow or fast relative to one another or the local clock as consumer devices typically do not have nanosecond-precise synchronization of their clocks.

  • TimestampCreated is used to calculate TimestampFinalized, but not directly.

• TimestampFinalized: this is the timestamp assigned to the event by the consensus algorithm, which is used to place events in their final consensus order. It is calculated to be the same for a given event by all parties. It is always increasing, though it may not be steady.

  • Because TimestampFinalized is always increasing and is calculated to be the same for all parties, it can be used as a clock source for timed game events. By convention, we recommend applying the time update before applying the transactions in the event.

  • You should not assume that TimestampCreated always falls before TimestampFinalized for a given event, as it depends on the clock drift of each creator.

  • The TimestampFinalized of an event is derived from the creation timestamps of events that were created contemporary to it, so it effectively becomes the average of the clocks of all peers.

Any given ConsensusEvent bundles together one or more transactions from its creator. The list of transactions in the ConsensusEvent are in the same order they were submitted by that creator, and so should be processed in that order. For convenience, the .ForEachTransaction() method will execute a closure with each transaction in its consensus order.

For consensus-driven random number generation, you can use the .ConsensusRng getter. This returns a subclass of Random which is guaranteed to return the same output on all peers that observe the same ConsensusEvent. Each new ConsensusEvent produces a different ConsensusRng instance and seed.

When a transaction in a ConsensusEvent calls for a random number, you can use the ConsensusRng from that event to generate it.

ConsensusEvent.ConsensusRng returns a new instance with the same seed every time for a given event, so to avoid generating duplicate numbers, you should use the same ConsensusRng instance to process all transactions in a single given event, then dispose of it afterward.

The seed for the ConsensusRng is not known until the event comes to consensus and is derived from the ECDSA signatures from other peers (which are theoretically unpredictable without knowing their secret keys), so its output shouldn't be possible to manipulate in a way that gives any one peer an advantage.

ConsensusSyncPoint

This is a special type of event that is emitted periodically by TCE. The period of time between sync points is called an "epoch". All parties will see a ConsensusSyncPoint at the same point in their consensus ordering, signaling the start of a new epoch.

ConsensusSyncPoint contains information about the current state of the network, including changes to the current set of creators:

• JoiningCreators: creators that will become active (will count towards consensus) in the next epoch.

• LeavingCreators: creators that will no longer count towards consensus, starting in the next epoch.

• FallenBehindCreators: creators who did not create an event in the previous epoch and were marked as fallen behind; if they fail to catch up, they will be automatically kicked from the network after PlatformConfig.FallenBehindKickSeconds.

  • Falling behind is most likely to happen due to a disconnection, but could also happen if the peer is simply not able to keep up with the rate at which events are being created, either due to bandwidth or CPU limitations. TCE will try to throttle event creation to avoid this happening, but it also tries to avoid a situation where one slow peer can stall the whole network. This may result in a slow peer getting left behind.

  • Catching up happens automatically, but being marked as fallen behind is also a signal to that peer that they may have missed some events and may need to re-synchronize their application state with the network.

• KickedForCheating: ideally, this set is always empty. TCE does not implement any form of client-side anti-cheat. "Cheating" in this context means specifically trying to cheat the consensus algorithm by submitting conflicting events to different peers, which is detected by the consensus algorithm and an automatic vote to kick the cheating creator is initiated. During normal operation, TCE should not submit conflicting events, so this would only happen if an attacker reverse-engineered the consensus algorithm and modified TCE to do this.

• SafeToForgetCreators: this is the list of creators who have been removed from the network for at least two epochs. Knowledge of a leaving or kicked creator must be retained for a short time as events they've previously submitted may still come through consensus. If a creator is in this set, it means they are truly gone and knowledge of their existence can be safely deleted.

• JustSynced and JustSyncedAddresses: when joining an existing session with an incomplete address book, JustSynced will be true and JustSyncedAddresses will be populated with the full address book.

• SessionEnded: true if a vote to end the session passed (see below). The application should stop processing events and save its state at this point.

Sync points are also the ideal time to save a checkpoint of the application state. When a new player joins, they can load their game with the checkpointed state for a given epoch and then process transactions normally from there. More detail in the "state management" section below.

Voting

Various changes in the behavior of the consensus algorithm are governed by voting in network-wide polls.

Many of these polls are run automatically by TCE for various things, including kicking "fallen behind" creators as discussed in the previous section, but others must be initiated explicitly by the application.

Voting to add a Creator

Adding a new creator to an already running session requires the whole network to vote on it, because the whole network must agree on when the new creator may begin to participate in consensus.

To vote for a creator to join, you call Platform.VoteToAddNode() and pass the address book entry as well as a timeout for the vote. There is not currently a way to know when a vote is occurring, so use of a side-channel like Unity Lobby is recommended.

Voting to kick a Creator

Similarly, kicking a peer from the network requires a vote. This can be used to remove a player from a session for an application-defined reason. Call Platform.VoteToRemoveNode() with the creator's public key and a timeout.

Voting to End the Session

The network can vote to end a session cleanly by calling Platform.VoteToEndSession(). When the vote passes, the next ConsensusSyncPoint will be the last one, with SessionEnded set to true.

State Management

You can also have TCE manage your application state for you. TCE's state management allows application instances to verify their state against each other, and lets joining peers download application state from the network in a BitTorrent-like fashion.

This feature is optional to use. If you don't submit any state reports or requests, TCE will track authoritative states passively (if reports are submitted by other peers) but will not download them. Set PlatformConfig.EnableStateSharing to false to disable all state management functionality.

Submit State Reports

To use TCE's state management: on every ConsensusSyncPoint, serialize a snapshot of your application state to a binary format of your choice and use it to construct a BlobState. Then, submit a state report for the epoch using Platform.ReportState(), passing the epoch index from ConsensusSyncPoint and the state you just created.

TCE will submit a cryptographic proof, signed by your secret key, of the application state to the network; if more than 2/3 of peers submit an identical state report for a given epoch, it is considered the authoritative state for that epoch.

If PlatformConfig.EnableStateSharing is true, then reporting state is required, else consensus may wait indefinitely for your next state report.

TCE uses cryptographic hashes to check whether one state report matches another. This means your binary serialization should be deterministic, repeatable, and portable across platforms and architectures.

If serializing floating-point numbers, for example object positions/rotations/transformations in 2D or 3D space, keep in mind that floating-point calculations may have slightly different results depending on CPU rounding mode flags and/or compiler optimizations. These would produce entirely different hashes and thus will not be considered identical states by TCE.

We are still investigating possible solutions for this.

One option to consider might be rounding values to some given fraction during serialization. For example, if using 32-bit floats, consider rounding values to some fractional unit like 1/65536 (or another reciprocal power of 2). This should hopefully cancel out most rounding errors without significantly affecting gameplay. Note that the fraction to round to depends on the range of expected values: the fractional precision of floating points decreases as their absolute magnitude (regardless of sign) increases.

Read about the IEEE 754 floating point format (single-precision floating point is common in games) for details.

Request State

When it's necessary to synchronize application state, like when joining a session for the first time or after having fallen behind, you'll want to request the latest application state from the network so you can resume processing transactions from there.

• When joining a running session, the first message in the finalized order will be the ConsensusSyncPoint for the epoch in which you are joining with JustSynced set to true.

• When you have fallen behind the rest of the session (for example, due to a temporary internet connection failure), you will receive a ConsensusSyncPoint with JustSynced set to true.

Once a ConsensusSyncPoint with JustSynced == true has been reached, either stop processing events or store them in order in a buffer; TCE will buffer events for you if you stop calling GetEvent(), but if the state request times out (see below) you will need to resume draining and discarding events until the next sync point.

Call Platform.RequestState() with the epoch index for the most recent ConsensusSyncPoint. This will begin an asynchronous state request and return a StateRequest object which will let you track its progress (for example, to update a loading bar).

StateRequest has the UpdateProgress() method which returns the ProgressKind enum telling you which stage the request is currently in:

• Confirming: this is the initial state for the request. It means an authoritative state for the epoch has yet to be established and TCE is waiting for state reports from more peers. The StateRequest.Confirming field will tell you the current progress of this stage: the current number of confirmations and the total needed to continue.

• Confirmed: an authoritative state has been selected and TCE will begin downloading it immediately.

• Downloading: TCE has started downloading the authoritative state for the given epoch. Current progress can be found in the StateRequest.Downloading field.

• Ready: TCE has finished downloading the authoritative state. Retrieve it with StateRequest.Ready. It will need to be downcasted to the State subclass currently being used by the application. As of writing, the only subclass that exists is BlobState but more may be added in the future.

  • Terminal state: future calls to UpdateProgress() will return Canceled. The StateRequest may be disposed of.

• Converged: the application submitted a state report that matches the authoritative state for the given epoch. No additional work is needed; simply resume handling events as normal.

  • Terminal state: future calls to UpdateProgress() will return Canceled. The StateRequest may be disposed of.

• Cancelled: returned if RequestState() has been called for a new epoch (TCE only processes one state request at a time) or if a terminal state was previously returned.

  • Terminal state: future calls to UpdateProgress() will return Canceled. The StateRequest may be disposed of.

• TimedOut: TCE failed to get an authoritative state for the epoch before it became too old (multiple epochs have passed since the request was initiated). Consume events from Platform.GetEvent() until the next ConsensusSyncPoint and call Platform.RequestState() again with the new epoch index.

  • This may result either from TCE being unable to select an authoritative state due to peers submitting conflicting state reports, or because TCE was unable to download the authoritative state in time. Either way, the request should be retried.

  • Terminal state: future calls to UpdateProgress() will return Canceled. The StateRequest may be disposed of.

• Error: an error occurred during the process. As of writing there is no way to get the error details directly, but the details will be logged.

  • Terminal state: future calls to UpdateProgress() will return Canceled. The StateRequest may be disposed of.

UpdateProgress() does not block and is safe to call in your game's main loop.

If a new ConsensusSyncPoint arrives, you may call RequestState() again for the new epoch which will cancel the existing request (UpdateProgress() will return Canceled).

TCE uses content-defined chunking to break your serialized application state into manageable chunks. If you cancel a state request (or it times out) and begin a new one, chunks that the new epoch's state shares with the previous one will not need to be re-downloaded.

This chunking also allows TCE to download from multiple peers simultaneously.