swbf-unmunge v1.1.1

Releasing the source files for your mod or map? Post em' here. (Applies to both SWBF1 & SWBF2)

Moderator: Moderators

Corporal
Posts: 143
Joined: Thu Mar 03, 2011 5:08 pm
xbox live or psn: N/A

swbf-unmunge v1.1.1

Postby SleepKiller » Sat Oct 14, 2017 1:48 am

This is a tool that isn't actually that new and a fair few of you likely already know about. Anyway now that it has already been out in the open for a while and is now "finished" I feel compelled to post it here for the sake of completeness.

Anyway this tool as you might tell from the name is designed to undo the munge process. And it does quite a good job at it most of the time (for a mostly complete listing of what it can do see here) of extracting and "unmunging" an .lvl's contents.

Now let me just say I made this in my spare time because I found it fun and interesting. I did not make with the intent of using it (or enabling other people) to get access to another modder's assets without their permission. If you want to use it on a map another modder made to learn how they did something then that is probably fine, but if you want to use it so you can use something of their's without asking then that probably isn't okay. But I digress a tool is a tool, it isn't inherently evil or good. I know I would have loved to have this a few years ago when I lost one of my maps to the abyss and only had the .lvl of it. (Also it's because of making this I was able to know how to make this, so it's not all bad.)

It's a commandline application so if you struggle with those you can find a guide here.

You can find the source code on GitHub.
https://github.com/SleepKiller/swbf-unmunge
And hey would you look at that you can always find the most recent release on GitHub as well, isn't GitHub amazing.
https://github.com/SleepKiller/swbf-unmunge/releases

Be nice, do good not evil with it and all that.
Last edited by SleepKiller on Sat Mar 10, 2018 9:52 pm, edited 2 times in total.

Resistance Leader
User avatar
Posts: 5203
Joined: Tue Feb 09, 2010 8:43 pm
Location: Edinburgh, UK
Projects :: EVERYWHERE + Zero Munge + TCW
Games I'm Playing :: The Forest
xbox live or psn: Marth8880

Re: swbf-unmunge v1.0.1

Postby Marth8880 » Sun Oct 15, 2017 5:27 pm

Excellent!

Due to how insanely useful it'd be (especially for rebuilding the 1.3 patch), I've submitted a requisition ticket for adding Lua script support here: https://github.com/SleepKiller/swbf-unmunge/issues/10

Corporal
Posts: 143
Joined: Thu Mar 03, 2011 5:08 pm
xbox live or psn: N/A

Re: swbf-unmunge v1.0.1

Postby SleepKiller » Sun Oct 15, 2017 10:09 pm

I'll try to make it the next feature I implement. I can't promise perfect output but I feel I should know enough to get reasonable output. Any updates I have on it's support I'll post on the GitHub issue. (My usual disclaimer of this is obviously a hobby project of mine so I don't offer a concrete promise or time frame but I'll aim for having something within the next two weeks.)

Master of the Force
User avatar
Posts: 4761
Joined: Sat Sep 19, 2009 11:37 am
Location: Mos Espa (germany)
Projects :: Republic Commando Side Mod

Re: swbf-unmunge v1.0.1

Postby Anakin » Sun Nov 26, 2017 6:39 am

Marth8880 wrote:Excellent!

Due to how insanely useful it'd be (especially for rebuilding the 1.3 patch), I've submitted a requisition ticket for adding Lua script support here: https://github.com/SleepKiller/swbf-unmunge/issues/10



Lua support would be MUCH more useful then the 3d models :D


==EDIT==

I just tested your tool and the msh export does not work correctly. Seams that there is a problem with the UV

Hidden/Spoiler:
Image


==EDIT2==

i tested a little bit and it seams that the UVs are mirrored horizontal

Corporal
Posts: 143
Joined: Thu Mar 03, 2011 5:08 pm
xbox live or psn: N/A

Re: swbf-unmunge v1.0.1

Postby SleepKiller » Sun Nov 26, 2017 8:27 am

Yeah that's been reoccurring problem for the tool. If I remember correctly the munger flips some model's texture coordinates and not others; I never sat down and worked out when and why it does this. The tool does try to flip the texture coordinate intelligently. It appears it is getting it wrong though. A hack solution is to just flip the texture to match the incorrect UV coordinates.

When/if I like work on this again I'll try to make it one of things I fix. The for loops in the read_vbuf_span overloads in this file control which vertex buffer types do and don't have their V texture coordinate flipped. If I don't end up fixing it myself and in the unlikely event someone in the future really wants it fixed that is where to start.

Droid Pilot Assassin
User avatar
Posts: 2152
Joined: Mon Jan 19, 2009 5:58 pm
Projects :: Star Wars Battlefront Legacy

Re: swbf-unmunge v1.0.1

Postby ggctuk » Sun Nov 26, 2017 2:07 pm

The same bug exists with some of the stock models and the solution is to simply resave the TGA file. That should fix the issue.

Master of the Force
User avatar
Posts: 4761
Joined: Sat Sep 19, 2009 11:37 am
Location: Mos Espa (germany)
Projects :: Republic Commando Side Mod

Re: swbf-unmunge v1.0.1

Postby Anakin » Fri Dec 15, 2017 12:28 pm

i noticed that mirroring the texture does not work. You need just to open and resave the tga file and it works. No idea how that could actually could be. For some reason the unmunge saves the tga in a "broken" format, that makes the texture mirrored in the viewer, but PSE, P.net,... can open those files and on resave they do everything correct, so the viewer works again.

Private First Class
User avatar
Posts: 82
Joined: Sun Jan 18, 2015 5:23 pm
Projects :: SWBF2 XBOX Mod Environment
Games I'm Playing :: SW Battlefront 2
xbox live or psn: No gamertag set

Re: swbf-unmunge v1.0.1

Postby cbadal » Wed Jan 10, 2018 5:27 pm

Nice!
Happy that this exists :)

It would be helpful if you print your help message (from your .md ) when the other commands aren't recognized. Or when the user uses one of the more common help arguments [/? /h -h --help]

Code: Select all
swbf-unmunge <options>

Options:
 -file <filepath> Specify an input file to operate on.
 -files <files> Specify a list of input files to operate, delimited by ';'.
   Example: "-files foo.lvl;bar.lvl"
 -version <version> Set the game version of the input file. Can be 'swbf_ii' or 'swbf. Default is 'swbf_ii'.
 -outversion <version> Set the game version the output files will target. Can be 'swbf_ii' or 'swbf. Default is 'swbf_ii'.
 -imgfmt <format> Set the output image format for textures. Can be 'tga', 'png' or 'dds'. Default is 'tga'.
 -platform <platform> Set the platform the input file was munged for. Can be 'pc', 'ps2' or 'xbox'. Default is 'pc'.
 -verbose Enable verbose output.
 -mode <mode> Set the mode of operation for the tool. Can be 'extract', 'explode' or 'assemble'.
   'extract' (default) - Extract and "unmunge" the contents of the file.
   'explode' - Recursively explode the file's chunks into their hierarchies.
   'assemble' - Recursively assemble a previously exploded file. Input files will be treated as directories.

Droid Pilot Assassin
User avatar
Posts: 2152
Joined: Mon Jan 19, 2009 5:58 pm
Projects :: Star Wars Battlefront Legacy

Re: swbf-unmunge v1.0.1

Postby ggctuk » Wed Feb 14, 2018 11:48 am

This version can now unmunge console version files. Sleepkiller tested it on the XBOX DLC and it worked well there.

Master of the Force
User avatar
Posts: 4761
Joined: Sat Sep 19, 2009 11:37 am
Location: Mos Espa (germany)
Projects :: Republic Commando Side Mod

Re: swbf-unmunge v1.0.1

Postby Anakin » Wed Feb 14, 2018 11:58 am

nice, but i'm still waiting for the script unmunging. Any ideas what is necessary for doing that??

Private First Class
User avatar
Posts: 82
Joined: Sun Jan 18, 2015 5:23 pm
Projects :: SWBF2 XBOX Mod Environment
Games I'm Playing :: SW Battlefront 2
xbox live or psn: No gamertag set

Re: swbf-unmunge v1.0.1

Postby cbadal » Fri Feb 16, 2018 10:26 pm

ggctuk wrote:This version can now unmunge console version files. Sleepkiller tested it on the XBOX DLC and it worked well there.

That's AWESOME!!!

Could it be used to unmunge a PC lvl and re-munge it for a Console version?

EDIT:
Anakin wrote:nice, but i'm still waiting for the script unmunging. Any ideas what is necessary for doing that??

For the compiled Lua (with -s option), a lot of the operations are straight forward to decompile. I have a tool that does a decent job on most of the mission lua files in 'mission.lvl', but it has been tricky to get good results with 'and' & 'or' logic so far.

Master of the Force
User avatar
Posts: 4761
Joined: Sat Sep 19, 2009 11:37 am
Location: Mos Espa (germany)
Projects :: Republic Commando Side Mod

Re: swbf-unmunge v1.1.0

Postby Anakin » Sat Feb 17, 2018 6:05 am

well i don't need mission stuff, i want to unmunge the shell lvl so i can see the changes Zerted did in his 1.3 patch

Droid Pilot Assassin
User avatar
Posts: 2152
Joined: Mon Jan 19, 2009 5:58 pm
Projects :: Star Wars Battlefront Legacy

Re: swbf-unmunge v1.0.1

Postby ggctuk » Sat Feb 17, 2018 6:06 am

cbadal wrote:
ggctuk wrote:This version can now unmunge console version files. Sleepkiller tested it on the XBOX DLC and it worked well there.

That's AWESOME!!!

Could it be used to unmunge a PC lvl and re-munge it for a Console version?


Most likely, yes.

Jedi Admin
User avatar
Posts: 11025
Joined: Wed Sep 07, 2005 11:42 pm
Location: Indiana
Projects :: No Mod project currently.
Games I'm Playing :: Destiny
xbox live or psn: No gamertag set

Re: swbf-unmunge v1.1.0

Postby Teancum » Sat Feb 17, 2018 9:41 am

Some of them anyway. I've had some work out such as Rattatak and Nal Hutta to the point where I can do the rest by hand and get it back into ZeroEdit. A few favorites (Concord Dawn being one) don't unmunge everything necessary to do the rest by hand. I've been meaning to contact SleepKiller with some examples but I've been focused on other things.

Private First Class
User avatar
Posts: 82
Joined: Sun Jan 18, 2015 5:23 pm
Projects :: SWBF2 XBOX Mod Environment
Games I'm Playing :: SW Battlefront 2
xbox live or psn: No gamertag set

Re: swbf-unmunge v1.1.0

Postby cbadal » Sat Feb 17, 2018 4:05 pm

Anakin wrote:well i don't need mission stuff, i want to unmunge the shell lvl so i can see the changes Zerted did in his 1.3 patch


Looking at how much that patch seems to be used, Zerted really should release the source to the SWBF2 community.
Anyone ask him yet?

Jedi Admin
User avatar
Posts: 11025
Joined: Wed Sep 07, 2005 11:42 pm
Location: Indiana
Projects :: No Mod project currently.
Games I'm Playing :: Destiny
xbox live or psn: No gamertag set

Re: swbf-unmunge v1.1.0

Postby Teancum » Sat Feb 17, 2018 4:13 pm

cbadal wrote:
Anakin wrote:well i don't need mission stuff, i want to unmunge the shell lvl so i can see the changes Zerted did in his 1.3 patch


Looking at how much that patch seems to be used, Zerted really should release the source to the SWBF2 community.
Anyone ask him yet?

It's out there. I can't find a download link this second, but the source code to the luas were all released anyway. What's a bigger bummer is that the custom gametype luas he used to host on his old site are gone now.

Master of the Force
User avatar
Posts: 4761
Joined: Sat Sep 19, 2009 11:37 am
Location: Mos Espa (germany)
Projects :: Republic Commando Side Mod

Re: swbf-unmunge v1.1.0

Postby Anakin » Sat Feb 17, 2018 5:26 pm

Teancum wrote:
cbadal wrote:
Anakin wrote:well i don't need mission stuff, i want to unmunge the shell lvl so i can see the changes Zerted did in his 1.3 patch


Looking at how much that patch seems to be used, Zerted really should release the source to the SWBF2 community.
Anyone ask him yet?

It's out there. I can't find a download link this second, but the source code to the luas were all released anyway. What's a bigger bummer is that the custom gametype luas he used to host on his old site are gone now.


yeah the additional luas are released and i have them, but the problem is, that we don't know what he did to the original luas.
I unmunged the 1.3 and the stock shell and there are multiple files edited.

Private First Class
User avatar
Posts: 82
Joined: Sun Jan 18, 2015 5:23 pm
Projects :: SWBF2 XBOX Mod Environment
Games I'm Playing :: SW Battlefront 2
xbox live or psn: No gamertag set

Re: swbf-unmunge v1.1.0

Postby cbadal » Sun Feb 18, 2018 7:47 pm

Anakin wrote:yeah the additional luas are released and i have them, but the problem is, that we don't know what he did to the original luas.
I unmunged the 1.3 and the stock shell and there are multiple files edited.

If you have the additional Luas that he release, then please upload them to a Github account and post the link to the 1.3 patch thread (if not already done).

I'm not sure if the decompiler will ever to decompile all Lua perfectly.

If you really, really want to see the differences you can compare the luac -l listings of the files (I'd suggest using the 'BeyondCompare' program).
You'll need to read this document to understand the listing syntax:
https://docs.google.com/viewer?a=v&pid= ... g0ZGFlMWI4

One of the files from his 1.3 patch I looked at was 'ifs_mpgs_pclogin'.
Very little of it had changed.
The only difference was at the beginning:
Hidden/Spoiler:
Code: Select all
   1   [-]   GETGLOBAL   0 0   ; ScriptCB_GetOnlineService
   2   [-]   CALL        0 1 2
   3   [-]   EQ          0 0 129   ; - "Galaxy"
   4   [-]   JMP         0 13   ; to 18
   5   [-]   GETGLOBAL   0 2   ; ScriptCB_DoFile
   6   [-]   LOADK       1 3   ; "ifs_mpgs_galaxylogin"
   7   [-]   CALL        0 2 1
   8   [-]   GETGLOBAL   0 2   ; ScriptCB_DoFile
   9   [-]   LOADK       1 4   ; "ifs_mp_leaderboard"
   10   [-]   CALL        0 2 1
   11   [-]   GETGLOBAL   0 2   ; ScriptCB_DoFile
   12   [-]   LOADK       1 5   ; "ifs_mp_leaderboarddetails"
   13   [-]   CALL        0 2 1
   14   [-]   GETGLOBAL   0 2   ; ScriptCB_DoFile
   15   [-]   LOADK       1 6   ; "ifs_mpgs_friends"
   16   [-]   CALL        0 2 1
   17   [-]   JMP         0 149   ; to 167

Which would resolve to this:
Hidden/Spoiler:
Code: Select all
if ScriptCB_GetOnlineService() == "Galaxy" then
   ScriptCB_DoFile("ifs_mpgs_galaxylogin")
   ScriptCB_DoFile("ifs_mp_leaderboard")
   ScriptCB_DoFile("ifs_mp_leaderboarddetails")
   ScriptCB_DoFile("ifs_mpgs_friends")
else   -- the corresponding 'end' is at the very end of the file.

So if you really, really want to. You would be able to construct the differences (but it would take some effort).

Master of the Force
User avatar
Posts: 4761
Joined: Sat Sep 19, 2009 11:37 am
Location: Mos Espa (germany)
Projects :: Republic Commando Side Mod

Re: swbf-unmunge v1.1.0

Postby Anakin » Mon Feb 19, 2018 4:22 am

If you are looking for the recovered source, here they are: viewtopic.php?f=25&t=27604&p=473223#p473223.

Gonna try those unmunge things when i find no other way the next days

Gametoast Staff
User avatar
Posts: 2973
Joined: Sun Feb 26, 2006 7:36 am
Location: USA
Projects :: Bos Wars AI - a RTS game
Games I'm Playing :: SWBF2 and Bos Wars
xbox live or psn: No gamertag set

Re: swbf-unmunge v1.1.0

Postby [RDH]Zerted » Tue Feb 20, 2018 4:19 am

Teancum wrote:What's a bigger bummer is that the custom gametype luas he used to host on his old site are gone now.
I still have all my assets. Just ask through an email/PM if you need something. I hadn't thought of it before, but I do have a paid site now, so I could probably rehost everything there...

The complete source code to v1.3 was lost in a backup failure. Prior to that point, it was publicly available online but no one had downloaded the entire thing. Some scripts were kept and those are available in the recovered source archive. v1.3 was designed with modding in mind. Use user scripts to inject code into ingame and cgc scripts to inject code into the shell. Have your code wrap functions you want to modify and make the modifications before/during/after those function calls. To inject code in the middle of a function you need to wrap the primary function then wrap whatever function is called on the line before where you want your code. Set a flag in your first wrap, then when that second function is called and the flag has been triggered, you know you're at that line in the middle of the function. It's annoying and sometimes complex, but you can modify almost anything anywhere by doing things like that.

Recursively dump the Lua's global table using uf_print() from utility_functions2.lua to see every accessible variable and function.

Next

Return to Released Assets

Who is online

Users browsing this forum: No registered users and 2 guests