python modding trouble with import

pyalot · August 28, 2017

I'm having some trouble with python modding related to importing your own files.

World of warships seems to support python mods in some way, and I got as far as getting it to load/execute the modfile I have in res_mods/<game-version>/scripts/client/mods . However nobody wants to put all his code into a single flat file, that's just not terribly practical.

I've tried appending a directory to sys.path that would contain some source files and then importing from there, but that didn't work, the module wasn't executed and an empty module was returned (if the .pyc was present) or the sys module (if the .pyc was not there but just the .py file). Next I tried putting my mod files into a zip file, and adding that to sys.path, which is better, now at least upon import the python file gets executed, however the import still returns the sys module instead.

I believe this has something todo with the modifications that Lesta made to __builtin__.__import__ but no matter what I do (sys.meta_path importers, sys.path_hooks, importing via __import__, imp, importlib etc.) nothing works as expected.

So right now the only practical way to write any python mod is to put it all into one flat file, and I'm wondering if I've overlooked something obvious, or if the WoWs import code is just broken.

OlliN · August 29, 2017

Still don't get why you want to append a directory to sys.path.

What's wrong with __init__.py and from . import * to import subdirectories?

pyalot · August 29, 2017

The world of warships python mod loader just walks trough res_mods/<game-version>/scripts/client/mods, reads any file ending with .pyc and runs it trough exec.

The sys.path contains the res/scripts.zip/scripts directory so that modules provided by the game can be imported. The __builtin__.__import__ has been overwritten by a custom function supplied by the game.

In python if you want to import some files, they usually need to be findable in entry on sys.path. The exception being directories directly under os.getcwd(), but the working directory is the WoWs installation path, not the mod directory.

That means that any package you make in the mod directory can't be imported. For that reason you could, for instance, add res_mods/<game-version>/scripts/client/mods to sys.path. In theory you would then be able to import any package defined in that directory in your res_mods/<game-version>/scripts/client/mods/mymod.py.

However that didn't work, if you do import myothefile in mymod.py (meaning res_mods/<game-version>/scripts/client/mods/myotherfile.py):

If the file is not present you get an import error (expected)
If just the .py file is present, you do not get an import error, the file is neither compiled nor executed, and import myotherfile -> empty module
If you have the .pyc file present, you do not get an import error, the file is neither compiled nor executed and import myotherfile -> sys
If you put the file in a mymod.zip and add that zip to sys.path as res_mods/<game-version>/scripts/client/mods/mymod.zip, then the file you're trying to import is compiled and executed, however the returning module is sys instead of your module

So the long story short is that import does not work for anything that isn't in the res/scripts.zip file. Even if you put it into sys.path.

After all of that didn't work, I tried overriding __builtin__.__import__ myself. This works, somewhat, but it oddly enough breaks imports from directories in the res/scripts.zip that the game provides. As in:

origImport = __builtin__.__import__

def myimport(*args, **kwargs):
	return origImport(*args, **kwargs)

from wsgiref import simple_server # -> cannot import simple_server from wsgiref (wsgiref was in fact referenced as sys again)

So I can't import my own files, and I can't modify the import mechanism or the game imports break. I'm a little at wits end with this problem (other than writing my own from-scratch importer and putting it on a function supplied to modules). But that would make it akward to include sources other people wrote (because they use import, and not "require" or whatever).

Edit: I'll include an example of what didn't work a bit later.

pyalot · August 29, 2017

I've prepared some examples for the import problem. You've got to unpack those to res_mods/<game-version>/scripts/client/mods . You might also need to copy res/scripts_config.xml to res_mods/<game-version>/scripts_config.xml to enable python modding (but I haven't tried if that's required).

There's some boilerplate code in each example that redirects stdout but I won't cite it everytime, the complete code is attached in the example files.

Example #1 Trying to import a package

In this example I'm trying to import a package from the local mods directory. This fails because the mod directory is not on the sys.path and the working directory is not in the mod directory.

print '### mymod: load'
import mymod

The output of this is:

### mymod: load
Traceback (most recent call last):
  File ".\mymod-load.py", line 19, in <module>
  File "Lesta", line 542, in Wargaming.net | Lesta Studio
ImportError: No module named mymod

Example #2 adding the modfolder to sys.path

The same as before, but adding the mod folder to sys.path so that hopefully the package is found.

print '### mymod: load'
sys.path.insert(0, here)
print '### sys.path: ', sys.path
import mymod
print '### mymod: ', mymod

The output of this is:

### mymod: load
### sys.path:  ['C:\\Games\\World_of_Warships\\res_mods\\0.6.9.1.1\\scripts\\client\\mods', u'C:\\Games\\World_of_Warships\\res\\scripts.zip\\scripts']
### mymod:  <module 'sys' (built-in)>

As you can see we didn't get an exception this time. However the module that was actually imported wasn't mymod, but the sys module. And the module that was intended to be executed wasn't (there's a missing line in the log if it was).

Example #3 adding the package to a zipfile

print '### mymod: load'
sys.path.insert(0, os.path.join(here, 'mymod.zip'))
print '### sys.path: ', sys.path
import mymod
print '### mymod: ', mymod

Output:

### mymod: load
### sys.path:  ['C:\\Games\\World_of_Warships\\res_mods\\0.6.9.1.1\\scripts\\client\\mods\\mymod.zip', u'C:\\Games\\World_of_Warships\\res\\scripts.zip\\scripts']
### mymod:  <module 'sys' (built-in)>

Same effect as in example #2. Getting the sys module and the module isn't executed.

Example #4 variation of #1 with . import

Trying out the . import without modification to sys.path

print '### mymod: load'
from . import mymod
print '### mymod: ', mymod

Output:

### mymod: load
Traceback (most recent call last):
  File ".\mymod-load.py", line 19, in <module>
  File "Lesta", line 542, in Wargaming.net | Lesta Studio
ValueError: Attempted relative import in non-package

The way mymod-load is executed is obviously not a package, so relative imports fail.

Example #5 variation of #3 but without a package

Maybe a problem is that packages aren't imported correctly. In this example I pack the file directly into the zipfile (no subfolder and no __init__.py).

print '### mymod: load'
sys.path.insert(0, os.path.join(here, 'mymod.zip'))
import mymod
print '### mymod: ', mymod

Output:

### mymod: load
### mymod: file
### mymod:  <module 'sys' (built-in)>

This is better, at least now the file we attempted to import gets executed. However the imported name still refers to sys. The same applies to any other file I'd try to import from within mymod, and of course the inability to import a package would be another hindrance.

Example #6 what about overriding __import__

We know that the __builtin__.__import__ is overridden, and that it does something not right. So how about we override that? Well before we do any custom thing in it, let's see if things still basically work at all if we do it. First let's test a normal import.

print '### mymod: load'
from wsgiref import simple_server
print '### simple_server:', simple_server

Output:

### mymod: load
### simple_server: <module 'wsgiref.simple_server' from 'C:\Games\World_of_Warships\res\scripts.zip\scripts\wsgiref\simple_server.pyc'>

All fine so far. Now let's test what happens if we overwrite __builtin__.__import__ with a passtrough.

print '### mymod: load'
def customImport(*args, **kwargs):
	print '### customImport: start', args[0]
	try: 
		module = origImport(*args, **kwargs)
		print '### customImport: module', module
		return module
	except:
		print '### customImport: error'
		traceback.print_exc()
		raise
__builtin__.__import__ = customImport
from wsgiref import simple_server
print '### simple_server:', simple_server

Output:

### mymod: load
### customImport: start wsgiref
### customImport: start binascii
### customImport: module <module 'binascii' (built-in)>
### customImport: start zlib
### customImport: module <module 'zlib' (built-in)>
### customImport: start sys
### customImport: module <module 'sys' (built-in)>
### customImport: start marshal
### customImport: module <module 'marshal' (built-in)>
### customImport: start copy_reg
### customImport: module <module 'copy_reg' from 'C:\Games\World_of_Warships\res\scripts.zip\scripts\copy_reg.pyc'>
an error occurred while loading module
### customImport: start binascii
### customImport: module <module 'binascii' (built-in)>
### customImport: start zlib
### customImport: module <module 'zlib' (built-in)>
### customImport: start sys
### customImport: module <module 'sys' (built-in)>
### customImport: start marshal
### customImport: module <module 'marshal' (built-in)>
### customImport: start copy_reg
### customImport: module <module 'copy_reg' from 'C:\Games\World_of_Warships\res\scripts.zip\scripts\copy_reg.pyc'>
an error occurred while loading module
### customImport: module <module 'sys' (built-in)>
Traceback (most recent call last):
  File "mymod-load.py", line 31, in <module>
ImportError: cannot import name simple_server

We can see the custom import is called, and that it seems to work a bunch of times. But for some reason it fails at copy_reg I think. The end result is that the whole import fails. So writing a custom __import__ is no solution either because it breaks the normal import from res/scripts.zip.

Conclusion

Import seems to be completely broken, and no matter what I tried, I can't get it to work in any way. If there's anything you think I overlooked please let me know.