swbf-unmunge v1.1.1

Releasing the source files for your mod or map? Post em' here. (Applies to both SWBF1 & SWBF2)

Moderator: Moderators

SleepKiller
Corporal
Corporal
Posts: 143
Joined: Thu Mar 03, 2011 5:08 pm
xbox live or psn: N/A

swbf-unmunge v1.1.1

Post by SleepKiller »

This is a tool that isn't actually that new and a fair few of you likely already know about. Anyway now that it has already been out in the open for a while and is now "finished" I feel compelled to post it here for the sake of completeness.

Anyway this tool as you might tell from the name is designed to undo the munge process. And it does quite a good job at it most of the time (for a mostly complete listing of what it can do see here) of extracting and "unmunging" an .lvl's contents.

Now let me just say I made this in my spare time because I found it fun and interesting. I did not make with the intent of using it (or enabling other people) to get access to another modder's assets without their permission. If you want to use it on a map another modder made to learn how they did something then that is probably fine, but if you want to use it so you can use something of their's without asking then that probably isn't okay. But I digress a tool is a tool, it isn't inherently evil or good. I know I would have loved to have this a few years ago when I lost one of my maps to the abyss and only had the .lvl of it. (Also it's because of making this I was able to know how to make this, so it's not all bad.)

It's a commandline application so if you struggle with those you can find a guide here.

You can find the source code on GitHub.
https://github.com/SleepKiller/swbf-unmunge
And hey would you look at that you can always find the most recent release on GitHub as well, isn't GitHub amazing.
https://github.com/SleepKiller/swbf-unmunge/releases

Be nice, do good not evil with it and all that.
Last edited by SleepKiller on Sat Mar 10, 2018 9:52 pm, edited 2 times in total.
Marth8880
Resistance Leader
Posts: 5042
Joined: Tue Feb 09, 2010 8:43 pm
Projects :: DI2 + Psychosis
Games I'm Playing :: Silent Hill 2
xbox live or psn: Marth8880
Location: Edinburgh, UK
Contact:

Re: swbf-unmunge v1.0.1

Post by Marth8880 »

Excellent!

Due to how insanely useful it'd be (especially for rebuilding the 1.3 patch), I've submitted a requisition ticket for adding Lua script support here: https://github.com/SleepKiller/swbf-unmunge/issues/10
SleepKiller
Corporal
Corporal
Posts: 143
Joined: Thu Mar 03, 2011 5:08 pm
xbox live or psn: N/A

Re: swbf-unmunge v1.0.1

Post by SleepKiller »

I'll try to make it the next feature I implement. I can't promise perfect output but I feel I should know enough to get reasonable output. Any updates I have on it's support I'll post on the GitHub issue. (My usual disclaimer of this is obviously a hobby project of mine so I don't offer a concrete promise or time frame but I'll aim for having something within the next two weeks.)
User avatar
Anakin
Master of the Force
Master of the Force
Posts: 4817
Joined: Sat Sep 19, 2009 11:37 am
Projects :: RC Side Mod - Remastered - SWBF3 Legacy
Location: Mos Espa (germany)

Re: swbf-unmunge v1.0.1

Post by Anakin »

Marth8880 wrote:Excellent!

Due to how insanely useful it'd be (especially for rebuilding the 1.3 patch), I've submitted a requisition ticket for adding Lua script support here: https://github.com/SleepKiller/swbf-unmunge/issues/10

Lua support would be MUCH more useful then the 3d models :D


==EDIT==

I just tested your tool and the msh export does not work correctly. Seams that there is a problem with the UV
Hidden/Spoiler:
Image
==EDIT2==

i tested a little bit and it seams that the UVs are mirrored horizontal
SleepKiller
Corporal
Corporal
Posts: 143
Joined: Thu Mar 03, 2011 5:08 pm
xbox live or psn: N/A

Re: swbf-unmunge v1.0.1

Post by SleepKiller »

Yeah that's been reoccurring problem for the tool. If I remember correctly the munger flips some model's texture coordinates and not others; I never sat down and worked out when and why it does this. The tool does try to flip the texture coordinate intelligently. It appears it is getting it wrong though. A hack solution is to just flip the texture to match the incorrect UV coordinates.

When/if I like work on this again I'll try to make it one of things I fix. The for loops in the read_vbuf_span overloads in this file control which vertex buffer types do and don't have their V texture coordinate flipped. If I don't end up fixing it myself and in the unlikely event someone in the future really wants it fixed that is where to start.
User avatar
giftheck
Droid Pilot Assassin
Droid Pilot Assassin
Posts: 2218
Joined: Mon Jan 19, 2009 5:58 pm
Projects :: Star Wars Battlefront Legacy

Re: swbf-unmunge v1.0.1

Post by giftheck »

The same bug exists with some of the stock models and the solution is to simply resave the TGA file. That should fix the issue.
User avatar
Anakin
Master of the Force
Master of the Force
Posts: 4817
Joined: Sat Sep 19, 2009 11:37 am
Projects :: RC Side Mod - Remastered - SWBF3 Legacy
Location: Mos Espa (germany)

Re: swbf-unmunge v1.0.1

Post by Anakin »

i noticed that mirroring the texture does not work. You need just to open and resave the tga file and it works. No idea how that could actually could be. For some reason the unmunge saves the tga in a "broken" format, that makes the texture mirrored in the viewer, but PSE, P.net,... can open those files and on resave they do everything correct, so the viewer works again.
User avatar
cbadal
Corporal
Corporal
Posts: 155
Joined: Sun Jan 18, 2015 5:23 pm
Projects :: SWBF2 XBOX Mod Environment
Games I'm Playing :: SW Battlefront 2
xbox live or psn: No gamertag set
Contact:

Re: swbf-unmunge v1.0.1

Post by cbadal »

Nice!
Happy that this exists :)

It would be helpful if you print your help message (from your .md ) when the other commands aren't recognized. Or when the user uses one of the more common help arguments [/? /h -h --help]

Code: Select all

swbf-unmunge <options>

Options:
 -file <filepath> Specify an input file to operate on.
 -files <files> Specify a list of input files to operate, delimited by ';'.
   Example: "-files foo.lvl;bar.lvl"
 -version <version> Set the game version of the input file. Can be 'swbf_ii' or 'swbf. Default is 'swbf_ii'.
 -outversion <version> Set the game version the output files will target. Can be 'swbf_ii' or 'swbf. Default is 'swbf_ii'.
 -imgfmt <format> Set the output image format for textures. Can be 'tga', 'png' or 'dds'. Default is 'tga'.
 -platform <platform> Set the platform the input file was munged for. Can be 'pc', 'ps2' or 'xbox'. Default is 'pc'.
 -verbose Enable verbose output.
 -mode <mode> Set the mode of operation for the tool. Can be 'extract', 'explode' or 'assemble'.
   'extract' (default) - Extract and "unmunge" the contents of the file.
   'explode' - Recursively explode the file's chunks into their hierarchies.
   'assemble' - Recursively assemble a previously exploded file. Input files will be treated as directories.
User avatar
giftheck
Droid Pilot Assassin
Droid Pilot Assassin
Posts: 2218
Joined: Mon Jan 19, 2009 5:58 pm
Projects :: Star Wars Battlefront Legacy

Re: swbf-unmunge v1.0.1

Post by giftheck »

This version can now unmunge console version files. Sleepkiller tested it on the XBOX DLC and it worked well there.
User avatar
Anakin
Master of the Force
Master of the Force
Posts: 4817
Joined: Sat Sep 19, 2009 11:37 am
Projects :: RC Side Mod - Remastered - SWBF3 Legacy
Location: Mos Espa (germany)

Re: swbf-unmunge v1.0.1

Post by Anakin »

nice, but i'm still waiting for the script unmunging. Any ideas what is necessary for doing that??
User avatar
cbadal
Corporal
Corporal
Posts: 155
Joined: Sun Jan 18, 2015 5:23 pm
Projects :: SWBF2 XBOX Mod Environment
Games I'm Playing :: SW Battlefront 2
xbox live or psn: No gamertag set
Contact:

Re: swbf-unmunge v1.0.1

Post by cbadal »

ggctuk wrote:This version can now unmunge console version files. Sleepkiller tested it on the XBOX DLC and it worked well there.
That's AWESOME!!!

Could it be used to unmunge a PC lvl and re-munge it for a Console version?

EDIT:
Anakin wrote:nice, but i'm still waiting for the script unmunging. Any ideas what is necessary for doing that??
For the compiled Lua (with -s option), a lot of the operations are straight forward to decompile. I have a tool that does a decent job on most of the mission lua files in 'mission.lvl', but it has been tricky to get good results with 'and' & 'or' logic so far.
User avatar
Anakin
Master of the Force
Master of the Force
Posts: 4817
Joined: Sat Sep 19, 2009 11:37 am
Projects :: RC Side Mod - Remastered - SWBF3 Legacy
Location: Mos Espa (germany)

Re: swbf-unmunge v1.1.0

Post by Anakin »

well i don't need mission stuff, i want to unmunge the shell lvl so i can see the changes Zerted did in his 1.3 patch
User avatar
giftheck
Droid Pilot Assassin
Droid Pilot Assassin
Posts: 2218
Joined: Mon Jan 19, 2009 5:58 pm
Projects :: Star Wars Battlefront Legacy

Re: swbf-unmunge v1.0.1

Post by giftheck »

cbadal wrote:
ggctuk wrote:This version can now unmunge console version files. Sleepkiller tested it on the XBOX DLC and it worked well there.
That's AWESOME!!!

Could it be used to unmunge a PC lvl and re-munge it for a Console version?
Most likely, yes.
User avatar
Teancum
Jedi Admin
Jedi Admin
Posts: 11079
Joined: Wed Sep 07, 2005 11:42 pm
Projects :: No Mod project currently.
Games I'm Playing :: Destiny
xbox live or psn: No gamertag set
Location: Indiana

Re: swbf-unmunge v1.1.0

Post by Teancum »

Some of them anyway. I've had some work out such as Rattatak and Nal Hutta to the point where I can do the rest by hand and get it back into ZeroEdit. A few favorites (Concord Dawn being one) don't unmunge everything necessary to do the rest by hand. I've been meaning to contact SleepKiller with some examples but I've been focused on other things.
User avatar
cbadal
Corporal
Corporal
Posts: 155
Joined: Sun Jan 18, 2015 5:23 pm
Projects :: SWBF2 XBOX Mod Environment
Games I'm Playing :: SW Battlefront 2
xbox live or psn: No gamertag set
Contact:

Re: swbf-unmunge v1.1.0

Post by cbadal »

Anakin wrote:well i don't need mission stuff, i want to unmunge the shell lvl so i can see the changes Zerted did in his 1.3 patch
Looking at how much that patch seems to be used, Zerted really should release the source to the SWBF2 community.
Anyone ask him yet?
User avatar
Teancum
Jedi Admin
Jedi Admin
Posts: 11079
Joined: Wed Sep 07, 2005 11:42 pm
Projects :: No Mod project currently.
Games I'm Playing :: Destiny
xbox live or psn: No gamertag set
Location: Indiana

Re: swbf-unmunge v1.1.0

Post by Teancum »

cbadal wrote:
Anakin wrote:well i don't need mission stuff, i want to unmunge the shell lvl so i can see the changes Zerted did in his 1.3 patch
Looking at how much that patch seems to be used, Zerted really should release the source to the SWBF2 community.
Anyone ask him yet?
It's out there. I can't find a download link this second, but the source code to the luas were all released anyway. What's a bigger bummer is that the custom gametype luas he used to host on his old site are gone now.
User avatar
Anakin
Master of the Force
Master of the Force
Posts: 4817
Joined: Sat Sep 19, 2009 11:37 am
Projects :: RC Side Mod - Remastered - SWBF3 Legacy
Location: Mos Espa (germany)

Re: swbf-unmunge v1.1.0

Post by Anakin »

Teancum wrote:
cbadal wrote:
Anakin wrote:well i don't need mission stuff, i want to unmunge the shell lvl so i can see the changes Zerted did in his 1.3 patch
Looking at how much that patch seems to be used, Zerted really should release the source to the SWBF2 community.
Anyone ask him yet?
It's out there. I can't find a download link this second, but the source code to the luas were all released anyway. What's a bigger bummer is that the custom gametype luas he used to host on his old site are gone now.
yeah the additional luas are released and i have them, but the problem is, that we don't know what he did to the original luas.
I unmunged the 1.3 and the stock shell and there are multiple files edited.
User avatar
cbadal
Corporal
Corporal
Posts: 155
Joined: Sun Jan 18, 2015 5:23 pm
Projects :: SWBF2 XBOX Mod Environment
Games I'm Playing :: SW Battlefront 2
xbox live or psn: No gamertag set
Contact:

Re: swbf-unmunge v1.1.0

Post by cbadal »

Anakin wrote: yeah the additional luas are released and i have them, but the problem is, that we don't know what he did to the original luas.
I unmunged the 1.3 and the stock shell and there are multiple files edited.
If you have the additional Luas that he release, then please upload them to a Github account and post the link to the 1.3 patch thread (if not already done).

I'm not sure if the decompiler will ever to decompile all Lua perfectly.

If you really, really want to see the differences you can compare the luac -l listings of the files (I'd suggest using the 'BeyondCompare' program).
You'll need to read this document to understand the listing syntax:
https://docs.google.com/viewer?a=v&pid= ... g0ZGFlMWI4

One of the files from his 1.3 patch I looked at was 'ifs_mpgs_pclogin'.
Very little of it had changed.
The only difference was at the beginning:
Hidden/Spoiler:
[code]
1 [-] GETGLOBAL 0 0 ; ScriptCB_GetOnlineService
2 [-] CALL 0 1 2
3 [-] EQ 0 0 129 ; - "Galaxy"
4 [-] JMP 0 13 ; to 18
5 [-] GETGLOBAL 0 2 ; ScriptCB_DoFile
6 [-] LOADK 1 3 ; "ifs_mpgs_galaxylogin"
7 [-] CALL 0 2 1
8 [-] GETGLOBAL 0 2 ; ScriptCB_DoFile
9 [-] LOADK 1 4 ; "ifs_mp_leaderboard"
10 [-] CALL 0 2 1
11 [-] GETGLOBAL 0 2 ; ScriptCB_DoFile
12 [-] LOADK 1 5 ; "ifs_mp_leaderboarddetails"
13 [-] CALL 0 2 1
14 [-] GETGLOBAL 0 2 ; ScriptCB_DoFile
15 [-] LOADK 1 6 ; "ifs_mpgs_friends"
16 [-] CALL 0 2 1
17 [-] JMP 0 149 ; to 167
[/code]
Which would resolve to this:
Hidden/Spoiler:
[code]
if ScriptCB_GetOnlineService() == "Galaxy" then
ScriptCB_DoFile("ifs_mpgs_galaxylogin")
ScriptCB_DoFile("ifs_mp_leaderboard")
ScriptCB_DoFile("ifs_mp_leaderboarddetails")
ScriptCB_DoFile("ifs_mpgs_friends")
else -- the corresponding 'end' is at the very end of the file.
[/code]
So if you really, really want to. You would be able to construct the differences (but it would take some effort).
User avatar
Anakin
Master of the Force
Master of the Force
Posts: 4817
Joined: Sat Sep 19, 2009 11:37 am
Projects :: RC Side Mod - Remastered - SWBF3 Legacy
Location: Mos Espa (germany)

Re: swbf-unmunge v1.1.0

Post by Anakin »

If you are looking for the recovered source, here they are: viewtopic.php?f=25&t=27604&p=473223#p473223.

Gonna try those unmunge things when i find no other way the next days
User avatar
[RDH]Zerted
Gametoast Staff
Gametoast Staff
Posts: 2982
Joined: Sun Feb 26, 2006 7:36 am
Projects :: Bos Wars AI - a RTS game
Games I'm Playing :: SWBF2 and Bos Wars
xbox live or psn: No gamertag set
Location: USA
Contact:

Re: swbf-unmunge v1.1.0

Post by [RDH]Zerted »

Teancum wrote:What's a bigger bummer is that the custom gametype luas he used to host on his old site are gone now.
I still have all my assets. Just ask through an email/PM if you need something. I hadn't thought of it before, but I do have a paid site now, so I could probably rehost everything there...

The complete source code to v1.3 was lost in a backup failure. Prior to that point, it was publicly available online but no one had downloaded the entire thing. Some scripts were kept and those are available in the recovered source archive. v1.3 was designed with modding in mind. Use user scripts to inject code into ingame and cgc scripts to inject code into the shell. Have your code wrap functions you want to modify and make the modifications before/during/after those function calls. To inject code in the middle of a function you need to wrap the primary function then wrap whatever function is called on the line before where you want your code. Set a flag in your first wrap, then when that second function is called and the flag has been triggered, you know you're at that line in the middle of the function. It's annoying and sometimes complex, but you can modify almost anything anywhere by doing things like that.

Recursively dump the Lua's global table using uf_print() from utility_functions2.lua to see every accessible variable and function.
Post Reply