Benthic Logo

Optional

This component is an optional feature in the metaverse_client crate. Projects that don't require this can compile with the feature disabled in Cargo.toml

Agent

Agents are the representation of the user, and other players in the 3d world. The agent sruct stores player locations, outfit data, and skeleton data.

Implementing agents requires using the FetchInventoryDescendents2, ViewerAsset endpoints, along with the inventory crate enabled. If you are unfamiliar with how capabilities work, visit the capabilities page on these docs.


Agent data begins the same way all object data does. Through the ObjectUpdate packet.

1.

The type of the ObjectUpdate is determined to be Avatar , sending a new HandleNewAvatar message.

2.

If the session's agent ID is equal to the incoming avatar object's ID , then the objectUpdate is requesting to render the current user's model.

3.

Check if the user's inventory has loaded yet . This is done using the inventory, which should have begun downloading folders and metadata immediately after login. If it is not loaded yet, requeue the request and try again later.

4.

Add the requested user ID to the agent_list, the session-global list of all agents. This will contain its ID, position, and the number of items in the current outfit.

6.

Determine the user's current outfit, and immediately cache it if there is no current outfit version or avatar version stored in the sqlite cache.

7.

Send a Set Outfit Size message, to tell the core when to finalize the avatar.

Fetch assets from ViewerAsset endpoint

If the avatar's gltf is already cached, load it directly. If not, for each item in the outfit, send a DownloadAgentAsset message, or an AddObjectToAvatar message directly.

Each DownloadAgentAsset message for each item in the outfit is sent individually to the mailbox for handling, which allows actix to schedule each download asyncronously, preventing bottlenecks.

The ViewerAsset endpoint expects requests in the format of
    http://[UUID OF VIEWERASSET ENDPOINT]?[OBJECT TYPE]_id=[ASSET ID]

In practice, this looks like

    http://da4b15ea-1d97-4140-afe3-2dd1ce5560710000?bodypart_id=da4b15ea-1d97-4140-afe3-2dd1ce5560710000
If the agent asset is an Object, download it from the viewer's capability endpoint. This will return SceneObject data, which will be parsed and cached.

If the object is a Link, download it and add it directly to the agent list as an Item.

Evil Benthic

LLSD Formats

The ViewerAsset endpoint does not respond with uniform encodings. Some requests will result in XML, some will result in a custom JSON-adjacent format, and some will result in a custom whitespace seperated format. These can be very confusing and difficult to debug.

Item

Notation Serialization
Items are encoded using a completely unique newline seperated notation. Parsing must be done character by character.

LLWearable version 22
New Pants

	permissions 0
	{
		base_mask	00000000
		owner_mask	00000000
		group_mask	00000000
		everyone_mask	00000000
		next_owner_mask	00000000
		creator_id	11111111-1111-0000-0000-000100bba000
		owner_id	11111111-1111-0000-0000-000100bba000
		last_owner_id	00000000-0000-0000-0000-000000000000
		group_id	00000000-0000-0000-0000-000000000000
	}
	sale_info	0
	{
		sale_type	not
		sale_price	10
	}
type 5
parameters 9
625 0
638 0
806 .8
807 .2
808 .2
814 1
815 .8
816 0
869 0
textures 1
2 5748decc-f629-461c-9a36-a35a221fe21f
      

Mesh

LL binary format.
Mesh Asset Format
The data structure starts out with a header in LL binary format.
The header is uncompressed and contains the offset positions for each of the compressed values.
Extracted from the binary format to a HashMap, it looks something like this.

Map({ skin: Map({ size: Integer(598), offset: Integer(0) }),
      physics_convex: Map({ size: Integer(204), offset: Integer(598) }),
      lowest_lod: Map({ size: Integer(1305), offset: Integer(802) }), 
      low_lod:Map({ size: Integer(2246), offset: Integer(2107) }), 
      medium_lod: Map({offset: Integer(4353), size: Integer(7672) }), 
      high_lod: Map({ size:Integer(27225), offset: Integer(12025) }), });
      
The offset it points to is the exact position in the data of the next zlib magic number for decompressing each section. Once decompressed, the data is encoded in the same binary llsd format that the header is.

SceneGroup

XML Serialization The SceneGroup returns much more standard XML.
<SceneObjectGroup>
  <RootPart>
    <SceneObjectPart xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
        xmlns:xsd="http://www.w3.org/2001/XMLSchema">
      <AllowedDrop>false</AllowedDrop>
      <CreatorID>
        <UUID>fb2e5541-66ba-4019-a5de-e8b9286b0914</UUID>
      </CreatorID>
      <FolderID>
        <UUID>ea04498f-daca-4a86-9ebb-9ce704f78e62</UUID>
      </FolderID>
      <InventorySerial>0</InventorySerial>
      <UUID>
        <UUID>ea04498f-daca-4a86-9ebb-9ce704f78e62</UUID>
      </UUID>
      <LocalId>403012829</LocalId>
      <Name>body</Name>
      <Material>3</Material>
      <PassTouches>false</PassTouches>
      <PassCollisions>false</PassCollisions>
      <RegionHandle>1099511628032000</RegionHandle>
      <ScriptAccessPin>0</ScriptAccessPin>
      <GroupPosition>
        <X>-3.7252903E-09</X>
        <Y>0.21699509</Y>
        <Z>4.656613E-10</Z>
      </GroupPosition>
      <OffsetPosition> 
...
      

Download Mesh

Each SceneGroup can contain one or several parts, which each contain their own mesh. Each mesh within a SceneGroup must be downloaded individually.

Apply Bind Shape Matrix

Download each scene's mesh and Apply the skin bind shape matrix to the vertices retreived from the server. If the bind shape matrix is not applied, the model will look distorted.
a model without the bind shape matrix applied

Bind Shape Matrix Not Applied

a model with the bind shape matrix applied

Bind Shape Matrix Applied

Evil Benthic

Sculpt Texture

There are several UUIDs throughout the SceneGroup, but only one of them can be used to request the mesh from the ViewerAsset endpoint. Unintuitively, this is found in the scene's sculpt texture. The sculpt data in the SceneGroup is an artifact from the days of SculptTextures, and has gradually evolved to include modern mesh information.

Skeleton

Skeleton handling in open metaverse projects can be tricky. The assumption from the server is that each client will have its own downloaded base skeleton, which all skinned meshes will use as a reference when managing their bone structure.
create_skeleton() begins the process of handling the skeleton for the individual SceneObject.

The default global skeleton is stored in the Default Assets repo as skeleton.gltf. This can be opened in blender by importing the file as GLTF to view the bones.
The default skeleton


This skeleton is made useable by the default asset converter's build.rs build script that is run at compile time. This generates a Skeleton object, which can be used as a static representation of the default skeleton.
Evil Benthic

IBMs

The reason the default skeleton is necessary is because the server does not store the joint rotations. The IBM field in the Mesh object's Skin is retrieved from the server as almost exclusively the identity matrix.
A model with all of the skeleton joints pointing directly up

Only Server IBMs Applied

Example:
Mat4 { x_axis: Vec4(1.0, 0.0, 0.0, 0.0),
       y_axis: Vec4(0.0, 0.9999889, 0.0, 0.0),
       z_axis: Vec4(0.0, 0.0, 0.9999904, 0.0),
       w_axis: Vec4(0.001931607, -0.04089581, 0.00318855, 1.0) }
The server's IBMs contain only the translation. The joints are in the right place, but they don't have the correct rotation.
A model with the skeleton far away from the model

Only Default IBMs Applied

Example:
  Mat4 { x_axis: Vec4(-3.5313367e-6, 1.0, -8.005451e-6, -0.0),
         y_axis: Vec4(-0.8080318, -1.2663957e-5, -0.58910865, 0.0),
         z_axis: Vec4(-0.5891205, 4.293412e-6, 0.8080454, -0.0),
         w_axis: Vec4(-1.5247067, -0.11596913, 2.2314665, 1.0) }
The default skeleton's IBMs contain both rotation and translation, but are offset from the model, and don't reflect its actual scale.
A model with the skeleton far away from the model

Compose default rotation with inverse bind translation

In order to correctly combine the server's IBMs with the default skeleton's rotations, you need to compose the rotation with the translations.

You don't want to multiply, or add the matrices, what you want to is create a mtrix that combines the two operations of rotation and translation, by translating the rotation matrix.
Example:
Mat4 { x_axis: Vec4(-3.5313367e-6, 1.0, -8.005451e-6, -0.0),
       y_axis: Vec4(-0.8080318, -1.2663957e-5, -0.58910865, 0.0),
       z_axis: Vec4(-0.5891205, 4.293412e-6, 0.8080454, -0.0),
       w_axis: Vec4(0.0, 0.0, 0.0, 1.0) }
First we take the rotation values above,
Example:
Mat4 { x_axis: Vec4(1.0, 0.0, 0.0, 0.0),
       y_axis: Vec4(0.0, 0.9999889, 0.0, 0.0),
       z_axis: Vec4(0.0, 0.0, 0.9999904, 0.0),
       w_axis: Vec4(0.001931607, -0.04089581, 0.00318855, 1.0) }
And apply this translation to those rotations.

The correct transform matrix uses this formula.
let transform_matrix = Mat4::from_rotation_translation(
            default_transform.to_scale_rotation_translation().1,
            skin.inverse_bind_matrices[i].inverse().w_axis.truncate(),
        );
T = Transform(Rdefault, tinverse bind)
The values MUST be composed. A simple swap of the default's xyz and the server's w value will result in a very confused and incorrect skeleton.
A model with the joints scattered everywhere

Default Rotations * Server Transform

Example:
  Mat4 { x_axis: Vec4(-3.5313367e-6, 1.0, -8.005451e-6, -0.0),
         y_axis: Vec4(-0.8080318, -1.2663957e-5, -0.58910865, 0.0),
         z_axis: Vec4(-0.5891205, 4.293412e-6, 0.8080454, -0.0),
        
         Replace the W axis:
         w_axis: Vec4(-1.5247067, -0.11596913, 2.2314665, 1.0) }
v
Example:
Mat4 { x_axis: Vec4(-3.5313367e-6, 1.0, -8.005451e-6, -0.0),
       y_axis: Vec4(-0.8080318, -1.2663957e-5, -0.58910865, 0.0),
       z_axis: Vec4(-0.5891205, 4.293412e-6, 0.8080454, -0.0),
       w_axis: Vec4(0.001931607, -0.04089581, 0.00318855, 1.0) }
This will not work.
The skeleton hierarchy is stored in the default skeleton. By this point. all joints know their parent and their children.
in order to determine the local transforms, a simple calculation is done on each joint.

Parent Inverse

*

Child

Save JSON

Now that we have the full SceneObject and handled its skeleton, we are almost ready for rendering. The rendering component expects the path to an AvatarObject serialized as JSON.
AvatarObjects contain a serialized Skeleton object, which represents the global skeleton of the avatar, and a list of paths to the RenderObjects which contain the vertices and indices of each component mesh. This allows previews to be generated from individual RenderObjects, or outfit objects to be updated, without having to parse or regenerate entire files.
The handled object gets written to disk, where it will be added to a full AvatarObject once the rest of the outfit has downloaded.

Agent List

The agent list is the list of all agents visible to the user, including itself. Once an object is ready to be rendered, it adds it to the list of outfit items in the agent list.
Evil Benthic

Global Skeleton

Creating a global skeleton seems easy in theory, but this gets complicated when outfit objects can stream in in any order, and the skeleton can be deformed by any object.
For example, a model might have a "body" object that has custom joint positions for every joint. The player might then have a "shirt" object equipped that has different joint positions for joints that the body has already morphed. Depending on which one loads first, the shirt or the body's transforms might be applied to the global skeleton, and the model will render looking deformed.

To handle this, each joint in the skeleton object stores its transforms as a vector. Each transform then has a rank which is increased when multiple objects load in with the same joint transform values. The Transform vector is organized in order of highest to lowest transform rank, with the highest always being the one that renders.

Each time an object is added to an agent through the agent list, the model's transforms are added to the global agent skeleton.
When the final outfit object is added to the avatar from an AddObjectToAvatar message, a FinalizeAvatar message is dispatched, which calls finalize_avatar(). This function finalizes the global skeleton by adjusting its local transforms, and creates an AvatarObject containing the finalized skeleton, the json paths to all of the object mesh adat, and the list of used joints.