Read 3D models from “primitives” files

[AstreTunes]

GitHub project page: https://github.com/SEA-group/Ship-model-extraction-Demo

Tool download: https://github.com/SEA-group/Ship-model-extraction-Demo/releases

---

[Update 2020.11.06]

 

(This is an advertisement) For those who are still interested in model modding, ShadowyBandit on NA has made a primitives import/export plugin for Blender.

 

---

[Update 2020.07.28]

 

Updated for xyznuviiiww vertex type found on RRS041's net model

 

---

[Update 2020.06.27]

 

The r in xyznuvr seems to be the "radius" used in wire anti aliasing, the value should be a positive float between 0 and 1;

Tangent and binormal need to be converted to 3 floats, like normal.

 

---

[Update 2020.06.04]

 

Updated uv format for xyznuv vertex type, thanks to ShadowyBandit (NA).

Project is now moved to GitHub, it would be easier to find out the modifications in the codes.

 

---

[Update 2020.05.20]

 

Updated uv format for xyznuvr vertex type;

Updated number conversion for uint32 in "indices" block

 

---

[Update 2020.05.09]

 

Fixed a bug with glass models

 

[DL removed due to new version release]

 

---

[Update 2020.04.06]

 

Fixed incorrect armor ID

 

[DL removed due to new version release]

 

---

[Update 2020.03.25]

 

The script is now capable to extract armor models correctly.

This version is fully functional.

 

(Tested in Matlab R2018b)

[DL removed due to new version release]

 

---

[Update 2020.03.23]

 

I've finished my Matlab code to extract models:

[DL removed due to new version release]

 

It can export almost all the geometries, including wires.

 

Known bugs:

1. Armor models are incomplete.

2. Wire's uv maps look strange

(I don't know the reason at this moment

 

Note that it exports model from primitives to obj, it can't do any change to primitive files.

 

---

[Original post 2019.07.02]

 

The main objective of this thread is to share everyone's knowledge and try to have a tool that help people to edit ship models. 

 

Introduction

 

The content SDK is a good work of Wargaming. It gives us access to nearly every single file (models, textures, materials, and a lot of parameters) of a ship, but we only mod the textures in most of the cases - and that's a waste of the potential of the content SDK for sure.  To release its full power, we need to have a model tool, which can be found nowhere in our days.

 

Weeks ago, I get told by Tirpitz_1939(@Asia) the binary file structure of primitives format. Then I tried writing some codes in Matlab to read a primitives file, and it showed that this file structure is basically correct. Meanwhile, there are still something unclear in this structure, and I’m not good in programming Windows application. So I’d like to share with you what I know until now, we can have some brainstorming together, and help capable people to create the tool of dream.

 

The file structure

 

Our main dish is here:

Spoiler

unknown:4
sectionData[
    {
        data(Vertices){
            type:64                                    # string
            count:4                                    # uint32
            BPVT:{                                    # exists only when type==BPVT
                subType:64                            # string
                count:4                                # uint32
            }
            vertices[
                {
                    xyz:12                            # float *3
                    xyznuv{                            ### alpha model, i.e. glass, plane propeller, net

                        normal:12                    # float *3
                        u:4                            # float
                        v:4                        # float
                    }
                    xyznuvtb{                        ### standard model 

                        normal:4                    # float *3 (need conversion)
                        uv:8                        # float *2
                        tangent:4                    # float
                        binormal:4                    # float
                    }
                    xyznuvr{                        ### wire model

                        normal:12                    # float *3
                        u:4                            # float
                        v:4                        # float
                        r:4                            # float
                    }
                    xyznuviiiwwtb{                    ### skinned model, i.e. main turret

                        normal:4                    # float *3 (need conversion)
                        uv:8                        # float *2
                         iiiww:5                        # hex -> string as entity name
                        tangent:4                    # float
                        binormal:4                    # float
                    }

                    xyznuviiiww{                    ### skinned alpha model

                        normal:4                    # float *3 (need conversion)
                        uv:8                        # float *2
                         iiiww:5                        # hex -> string as entity name
                        tangent:4                    # float
                        binormal:4                    # float
                    }

                }
            ]
        }:sectionSize
        data(Indices){
            type:64                                    # string
            count:4                                    # uint32
            groupCount:4                            # uint32
            indices[
                list{                                # exists only when type==list
                    i3:2                            # uint16
                    i2:2                            # uint16
                    i1:2                            # uint16
                }
                list32{                                # exists only when type==list32
                    i3:4                            # uint32
                    i2:4                            # uint32
                    i1:4                            # uint32
                }
            ]
            groups[
                {
                    startIndex:4                    # uint32
                    trianglesCount:4                # uint32
                    startVertex:4                    # uint32
                    verticesCount:4                    # uint32
                }
            ]
        }:sectionSize
        data(armor){
            unknown:28
            count:4                                    # uint32
            armor[
                id:4                                # uint32                                
                unknown:24
                count:4                                # uint32
                vertices[
                {
                    xyz:12                            # float *3
                    ??normal??:4                    # the value doesn't seem like normal, I don't know what it is.
                }
            ]
        }:sectionSize
    }
]:sum(sectionSize)
sectionNames[
    {
        sectionSize:4                                # uint32
        unknown(0):16                                
        sectionNameLength:4                            # uint32
        sectionName:sectionNameLength                # string
    }
]:sectionNameLength
sectionNameLength:4                                    # uint32

 

Some notes:

1.        The unit of data length is byte.

2.        When the length of a section is not an integer multiple of 4, there will be some empty bytes added to the end of the section, that make the length become a multiple of 4. For example, the first section starts by the 5^th byte (because the first 4 bytes are unknown), the first section has 57 bytes, then the second section should start by the 5+60=65^th byte instead of the 5+57=62^nd byte.

3.        The order of processing the file: You should start the processing from the last 4 bytes, which tell you the length of the “section name” section, then you go to the start byte of “section name” section, and so on…

4.        All the strings are standard ACSII char arrays, just convert them byte by byte.

5.        All the numerical data are in little-endian order

6.        As you may have noticed, the “normal” attribute should have 3 floats, but there’s only 4 bytes for each normal. I don’t really understand how did WG stock 3 floats in 4 bytes, but here is a python code that allows you to get these 3 floats:

Spoiler

import sys

packed = float(sys.argv[1])

pkz = (int(packed) >> 22) & 0x3FF
pky = (int(packed) >> 11) & 0x7FF
pkx = int(packed) & 0x7FF

if pkx > 0x3ff:
		x = -float((pkx&0x3ff^0x3ff)+1)/0x3ff
else:
		x = float(pkx)/0x3ff
if pky > 0x3ff:
		y = -float((pky&0x3ff^0x3ff)+1)/0x3ff
else:
		y = float(pky)/0x3ff
if pkz > 0x1ff:
		z = -float((pkz&0x1ff^0x1ff)+1)/0x1ff
else:
		z = float(pkz)/0x1ff
		
sys.stdout.write(str(x)+' '+str(y)+' '+str(z))

 

7.        The section type could be “*.vertices”, “*.indices”, “*.armor” and “*.cmodl”. This is the case of WoWS, I don’t know what would happen in WoT or WoWP. A pair of vertices and indices could form a standard ship model part; the armor is a armor model; the cmodl however, I don’t know what it is.

 

The armor models

 

Guns’ “armor” type parts can be found directly in their lod0 primitives; ships’ armors are stocked in the primitives file without segment postfix in the filename.

I make this specific paragraph about the armors, because they only have vertices, you can’t find any “indices” about them.

 

Here is a xyzn vertices table of a certain part of armor on the old Yamato:

Spoiler

There are many identical vertices, so, in fact, you just take every 3 points in order, you put them together, and you get the triangle. It’s logical.

 

As to the normals, I doubt what they are, because if you consider them as normal you will get a very very strange surface, like this:

Spoiler

 

Model file format

 

We used to have some model tools that export models in obj format, when I used them to replace models, sometimes the normal texture didn’t work as intended.

 

When I see this primitives file structure, I understand that it’s because of the lack of tangent and binormal. These are 2 necessary attributes to define a tangent space, but obj format does not support them. So this time, if we make a new tool, we should consider to use another 3D file format that supports tangent and binormal. For example, the FBX.

 

If you don’t know what a tangent space is, please read this: https://docs.cryengine.com/display/SDKDOC4/Tangent+Space+Normal+Mapping

And here are some ASCII FBX documentations that may help you generate and read fbx files:

http://download.autodesk.com/us/fbx/2010/fbx_sdk_help/index.html?url=WS1a9193826455f5ff-150b16da11960d83164-6c6f.htm,topicNumber=d0e127

https://banexdevblog.wordpress.com/2014/06/23/a-quick-tutorial-about-the-fbx-ascii-format/

 

A summary of things that are not yet clear

 

1.        The “unknown” marks in the file structure

2.        The “normal” of armor vertices

 

That’s all for now. Special thanks to Tirpitz_1939(AS) IsamuKondera(EU) and real_sytax(EU). Hope we can have some constructive discussions here and maybe finally have a model tool that we have dreamed for years

 

 

[IsamuKondera]

My guess for the first unkown:4 is a so called "magic value" or "magic number" which is used to see if it is from the same filetype as you expect.

[MatroseFuchs]

Unfortunately We are currently unable to provide source files for ship models.

[AstreTunes]

On 7/2/2019 at 2:29 PM, MatroseFuchs said:

Unfortunately We are currently unable to provide source files for ship models.

I understand. The source files should not be spread to the public, it's logical.

And what we need is the capability of modifying models, not the models themselves. But I think we're about to reach the objective 😊

[AstreTunes]

@IsamuKondera Maybe it's not necessary to go for the FBX format. I heard that tangent and binormal can be calculated based on normal and uv coordinates - I'm not expert in computer graphics, but if it's true, we may still use OBJ files and calculate them when replacing model. For me the OBJ format is still more common and concise. What do you think?

 

[IsamuKondera]

8 hours ago, Yuudachi_Kai_II said:

@IsamuKondera Maybe it's not necessary to go for the FBX format. I heard that tangent and binormal can be calculated based on normal and uv coordinates - I'm not expert in computer graphics, but if it's true, we may still use OBJ files and calculate them when replacing model. For me the OBJ format is still more common and concise. What do you think?

 

 

I guess at this point it's probably easier to write an addon for blender, because they actually have a documentation, and just adjust the obj file so it meets our requirements.

And call it specifically something like sobj which stands for Ship Object then or something like that.

 

You can then still export it via FBX to other 3D suits but blender will be the main target because it's free to use.

[IsamuKondera]

Some quick search via google:

 

Getting Tangent and bitangent data:
https://blender.stackexchange.com/questions/26116/script-access-to-tangent-and-bitangent-per-face-how

 

Extending the import and export menu:

https://blender.stackexchange.com/questions/56829/python-extend-select-similar-menu

 

Overall the blender addon would be probably the easiest part. And I would also start with this one because here you can define the file properties.

 

I also think that it is a bad idea to touch the armor stuff. So my suggestion is:
Let's start with the blender addon first in this way:

 

1. Creating Menu functions

2. Export (with file dialog)

3. Import (with file dialog)

 

When this is done we can start concentrating on the primitive file.

 

 

Regarding the posisble wiremodel attribute... Could it be the... radius? like r?

 

That would explain the nature of it. Two vertices beeing connected and the radius telling the software how to render it.

UV mapping inclusive.

 

We should test this once. Because if it is the radius we would have to think how to implement it.

[AstreTunes]

10 hours ago, IsamuKondera said:

You can then still export it via FBX to other 3D suits but blender will be the main target because it's free to use.

Ok, I will have a look to this software

[AstreTunes]

7 hours ago, IsamuKondera said:

Regarding the posisble wiremodel attribute... Could it be the... radius? like r?

 

That would explain the nature of it. Two vertices beeing connected and the radius telling the software how to render it.

UV mapping inclusive.

 

We should test this once. Because if it is the radius we would have to think how to implement it.

I have thought about this too, but, the wire still has vertices and indices at the same time, that means it's a group of triangles just like other models. The radius sounds logical but I don't know where to apply this attribute

[AstreTunes]

I drew a line in 3ds max and saved it in FBX format, this is what I get:

Spoiler

Seems that there's no specific "radius" attribute, but we can make use of the "color" property which is not useful for us, like, write the radius as the red value.

[IsamuKondera]

2 hours ago, Yuudachi_Kai_II said:

I have thought about this too, but, the wire still has vertices and indices at the same time, that means it's a group of triangles just like other models. The radius sounds logical but I don't know where to apply this attribute

You obviously need indices.

 

An indice just tells you which vertex is connected to which one. 3 vertexes create a triangle when you have an indice with 3 values.

 

So I guess we will see indices with just 2 values to connect 2 vertices with the r value. Thats just a guess though.

Will have to write a provisional extractor first to make it humanly readable for me.

 

 

Also:

xyznuvr{						# exists only when type=="xyzuvr" (most likely for wire models)
						r:4							# float
					}

 

This must be at least 12 bytes long. Otherwise it doesn#t work.

[AstreTunes]

I have just finished a functional Matlab script to extract models, I attached it to the main post.

@MatroseFuchs Maybe you can move this thread to Tutorial/Tools section?

[AstreTunes]

@real_sytax 

Hi Sytax. 

If you're still interested in primitives to obj conversion, I think the method is clear now.

I wrote my script in Matlab because I'm bad at other languages. But at least it proves that the conversion is possible and the method is correct.

If you have questions don't hesitate to ask me.

 

[real_sytax]

many thx. i will give it a try soon. at moment i am a bit busy with my retraining.

 

greetings

[cb999709]

Hello AstreTunes,

I downloaded your ShipModelExtractor. It works! Thanks to your great tool, I'm able to create realistic ship renders.

Spoiler

However, the exported UV coordinates in the obj files seem to be upside down. When I was attempting to use the textures from game client, the textures need to be flipped vertically in order to fit the uv coordinates of the ship model.

Spoiler

 

[AstreTunes]

50 minutes ago, cb999709 said:

Hello AstreTunes,

I downloaded your ShipModelExtractor. It works! Thanks to your great tool, I'm able to create realistic ship renders.

  Hide contents

However, the exported UV coordinates in the obj files seem to be upside down. When I was attempting to use the textures from game client, the textures need to be flipped vertically in order to fit the uv coordinates of the ship model.

  Hide contents

 

I think it's because of Blender, it uses bottom left corner as UV origin; while in photoshop, 3ds max and wows the origin is on the top left corner

[cb999709]

48 minutes ago, AstreTunes said:

I think it's because of Blender, it uses bottom left corner as UV origin; while in photoshop, 3ds max and wows the origin is on the top left corner

Oh, yes. Now I found how to flip the UV easily in Blender UV editor: UV>Mirror>Y Axis. Problem solved. Thanks for your advice.谢谢大佬

[1MajorKoenig]

Hey, is there a possibility to change models and get them back into the game? I.e. removing these funnel caps and the clipper bow from Prinz Eitel? 

 

Or are you looking at extracting and viewing these models only?

[AstreTunes]

20 minutes ago, 1MajorKoenig said:

Hey, is there a possibility to change models and get them back into the game? I.e. removing these funnel caps and the clipper bow from Prinz Eitel? 

 

Or are you looking at extracting and viewing these models only?

There used to be some WoT tools but none of them works properly with WoWS models.

My code can only extract, it can't do the inverse.

That's why I share all what I know here and expecting someone to write an appropriate tool to replace models.

 

But there are some work-arounds. If you only need to remove something from the model (i.e. funnel caps), you can do it through texture editing, by making that part transparent.

[1MajorKoenig]

16 minutes ago, AstreTunes said:

There used to be some WoT tools but none of them works properly with WoWS models.

My code can only extract, it can't do the inverse.

That's why I share all what I know here and expecting someone to write an appropriate tool to replace models.

 

But there are some work-arounds. If you only need to remove something from the model (i.e. funnel caps), you can do it through texture editing, by making that part transparent.

 

Thanks will try that!

[cb999709]

Hello AstreTunes,

I encountered with a problem when I was converting non-symmetrical ship models, eg. Tier 2 dd USS Sampson. In game, there is a bevel at port side of the stern superstructure. As shown below:

Spoiler

However in the extracted obj files, the bevel is at starboard side. I've used multiple .obj file viewer softwares, which gave the same result.

Spoiler

So, I'm sure this time it's not Blender's problem.

I suggest that in the exported obj files the X axis coordinates should be multiplied by -1.

[AstreTunes]

48 minutes ago, cb999709 said:

Hello AstreTunes,

I encountered with a problem when I was converting non-symmetrical ship models, eg. Tier 2 dd USS Sampson. In game, there is a bevel at port side of the stern superstructure. As shown below:

  Hide contents

However in the extracted obj files, the bevel is at starboard side. I've used multiple .obj file viewer softwares, which gave the same result.

  Hide contents

So, I'm sure this time it's not Blender's problem.

I suggest that in the exported obj files the X axis coordinates should be multiplied by -1.

Emmm it could be, but I have always been working with these x-inversed models, because the previous WoT tools extract models in the same way, I have never considered it as an error...

[cb999709]

That's fine. Since it's quite easy to mirror an object in 3D softwares.

In addition, I also noticed that in the game .primitives file, for main guns and secondary guns, the mesh has already been flipped along the X axis. I mean, the main guns and secondary guns models stored in the game is mirrored in X axis. Therefore, after exporting by using your Matlab code, the gun models are flipped back so they look correct.

The other parts, eg. ship hull, torpedo tubes, AA guns, gun directors, are not flipped in the primitives files. The exported obj files will look incorrect.

[AstreTunes]

1 hour ago, cb999709 said:

That's fine. Since it's quite easy to mirror an object in 3D softwares.

In addition, I also noticed that in the game .primitives file, for main guns and secondary guns, the mesh has already been flipped along the X axis. I mean, the main guns and secondary guns models stored in the game is mirrored in X axis. Therefore, after exporting by using your Matlab code, the gun models are flipped back so they look correct.

The other parts, eg. ship hull, torpedo tubes, AA guns, gun directors, are not flipped in the primitives files. The exported obj files will look incorrect.

Indeed. Those who use "skinned models" are rotated by 180 degrees among Y axis, they also have inversed normal.

[cb999709]

14 minutes ago, AstreTunes said:

Indeed. Those who use "skinned models" are rotated by 180 degrees among Y axis, they also have inversed normal.

Could you please tell me what does "skinned models" mean?