Army Builder file format support

[IBBoard]

Although there will be a lot of people who use Rollcall (mainly because it is free), Army Builder from LoneWolf also seems quite popular. Personally I think the interface is hideous, I'd never pay the price tag for it, and having looked at the file formats it looks like a complete mess, but I can't deny that people seem to like it.

We've already got a Rollcall plugin in the works that'll load Rollcall files in to WarFoundry (so people don't have to create all of their data files afresh to use WarFoundry), but it would be great if we had an Army Builder app. The main hurdle is the file format.

From what I've looked at so far (not a detailed inspection, but a general perusal) the .ab files are some kind of binary container, probably with some kind of compression. The end of the file definitely appears to be a file table, though. If that's too difficult to pull apart then we might need to just work with pre-extracted files.

From the pre-extracted files, they're all .dat but they're all XML files. There seems to be a ridiculous amount of different tags in there that I've got to make sense of, and there's the possibility that data is split across multiple files for the same army. Large amounts of the data appear to be additional cruft, but if they prove useful then we can customise the WarFoundry file format and add that functionality.

The only thing I've seen that might be a bit difficult to implement are the Macros. In a lot of cases they're probably used to handle special cases (I saw one that increased a unit allowance if conscripts were taken, which is just the same as saying "conscripts don't count to unit allowance") and we may be able to handle them in requirements (once they're implemented). If there is anything more complex that is a) useful and b) not some potential security/data integrity risk then we'll have to see what we can add.

If anyone can come up with any insight in to the file formats or wants to take on the job of making a file loader then please either post it here or sign up at the WarFoundry dev site and once I've added you to the Developers group you can put it in the Wiki.

[IBBoard]

Okay, some initial prodding and manual reading so far reveals:

• .def appears to be the game system definition ("AB40k4" has one - warhammer40k.def)
• .dat appears to be the most common file extension for army data, and it is just a .xml file
• .aug are "augmentations" - not entirely sure what they are yet as all the examples I've found seem to just be some form of "group tags" or notes
• There seem to be quite a few "bad" values in there - things that seem more presentational than content based in your content data
• Weapons, skills, etc all seem to be "options" with different "category" (although weapons also seem to have "equipment=yes", which implies some duplication)

[IBBoard]

I've done a bit of prodding of the .ab files and found out a small amount. It still isn't obvious to me what compression they're using (unsurprisingly it isn't a standard compressed file), but I have sussed a little bit out about the file format and the "file table" at the end.

All values in the following appear to be big-endian numbers.

• The first four bytes of the file are a 32-bit offset for the start of the file table (value + 4)
• At the offset is a 16-bit length for the file table header, including some comments and headers
• The file header consists of the following (each item in the list is separated by a tab - 09):
  
  • A fixed "LWDExport" ("Lone Wolf Development Export"?) followed by a linefeed (0A), a colon (3A), "Army Builder" (app name of the file creator?)
  • The value of the "game" attribute on the <game> tag in the .def file (in ASCII/Latin-1)
  • The "folder" attribute of the "structure" tag
  • The "major" attribute of the "release" tag (as a string)
  • The "minor" attribute of the "release" tag (as a string)
  • The major part of the "required" attribute of the "release" tag (as a string)
  • The minor part of the "required" attribute of the "release" tag (as a string)
  • An unknown number that appears to always be 0 (Hex: 30)
  • "ABExport"
  • The additional comment added when creating the file
• File entries consist of:
  
  • A 16-bit number for the file name length followed by the file name in standard ASCII/Latin-1
  • The eight bytes after a file name seem to always be 01 00 02 00 04 01 00 00 (at least in all the files I looked at - it may be something to do with versioning)
  • This is then followed by a 32-bit file size of the uncompressed file
  • This is then followed by what appears to be a 32-bit file size of the compressed data (although I need to work out the data chunk to check!)
  • ...(some stuff to work out)...
  • Each entry ends 08 00
• The file appears to be NUL (00) terminated as there is an extra 00 after the last 08 00

[IBBoard]

I've moved the technical detail to the Wiki. There is now a page on Army Builder and a working document for the .ab file format definition.

[snowblizz]

Are you talking about using the "game rule files" or the roster files? It looks like the "game rules" to me. Is it strictly necessary to use those? I guess technically it would be awesome of someone creates datafiles for say AB 2.x WHFB (Anarchistica yahoo list) and you could just implement the rule system created, but I wonder if its worth it.
Some of those are frankly a mess.

There's 2.x .ab and 3.x .ab files I think. Or if they use something else, I don't use the new 3.x version.

[IBBoard]

The .ab files that I'm working with are container files for an entire game system (game system .def file, plus race files, plus various "augmentation" files that make things horribly messy). You can grab them from here for AB3 and they'll give you the full army lists for all of the races for a given game system.

As you say, it would awesome to just take files created for Army Builder and re-use them in WarFoundry. In your case I'm assuming you're still using AB2 because it does what you need and you don't want to shell out for AB3. If that's the case then something like WarFoundry supporting AB files would let you use the data you want, but in a more up-to-date and better supported app Having said that, it would require us to work out the AB2 file format as well

[snowblizz]

That's what I thought then. Yeah I know what the .ab are. I've actually modified the gamesystem files a bit with the "internal" editor. I was updating the Tyranid files for the new codex, but stuff got complex real fast.

I, like many others, still use AB 2 because the gamefiles are available better (and more timely, more accurate) . Partly. I don't know if your aware, but basically WHFB is used on AB2.x and WH40k on AB 3.x. Only the "official" maintainers for 40k switched to the new program. There are some WHFB 3.x maintainers but they are months behind compared to the AB 2.x guys. So many like me are indeed left a bit out to dry. Which is why I'm so interested in this. You definitely have a latent "market share" out there.
I don't particularly like the 3.x licensing scheme either, but that's beside the point. The important issue for me is that the decent file maintainers are split over 2 versions of the software. And I don't know where the other 6 or so gamesystems I might use occasionally stand either. So incentive to upgrade is not exactly strong.

So there is indeed a large population of legacy AB2.x users out there. You might want to consider that. Though I also have a sneaking suspicion that a good many AB2 users just didn't want to shell out the money and got a cracked version. Another strong "selling" point for your efforts IMHO.
You might want to look in the AB2 .ab:s as well, they shouldn't be *that* much different [crosses fingers].

What would also be REALLY useful though is being able to load up AB .rst files into the rulesystem of WarFoundry (whether that is made in WF or important).

[IBBoard]

I'll have to have a look at the AB2 .ab files. From what I could tell of the AB3 file format it does look like it might not have changed much, and since the .dat files use XML then hopefully AB3 just added extra tags on top of the AB2 stuff. Even if it is more majorly different we can probably just do a switch based on version numbers or some other feature.

In terms of .rst files, I've not seen any of them so far. I know there's the Yahoo group, but I can't remember what my Yahoo account details are at the moment to get at the files! If the .rst files do what I think they'll do then I think we're likely to ignore them and have a pure programmed "Warhammer validation plugin". That may load some values from a file, but it doesn't have to.

Now I need to find more AB2 files so that I can support them. I didn't think there would be so much of a community for the older file format given the recent AB3.1 update



[edit] Any chance of pointing me to those AB2 files? I've just created a new Yahoo account and it turns out the Yahoo group I found hasn't been used for years. Everything on the official site seems to be AB3 (unsurprisingly).

[IBBoard]

We now have file parsing! I've not quite had a chance to upload it to Subversion yet (and I literally have to shut down my machine in about one minute) but we have an ArmyBuilder file parser that has produced a data dump of several files

Now to work out what the compression is when I'm not on the computer...

[IBBoard]

Curses, it looks like it might not be a standard compression method :\ I've tried Zip, GZip and BZip from the SharpZipLib and none of them seem to be working. I think I'm doing it right, but each one says "nope, not valid for this compression type".

On the positive side, the library is now in source control for anyone who wants to work with it.


[edit] A bit of poking at the app itself rather than the files and I found the following messages:

• Error creating deflate engine. ZLIB returned %d
• Deflate Normal/Fast/Slow
• Can't locate end of central directory in ZIP file %s

It might not be right (since the files aren't standard Zips) but it does look like they might be using ZLib/Zip. If only I could work out how to get the data out without errors!

[snowblizz]

I should look in more often.
The AB2.x for WHFB currently resides here:
http://groups.yahoo.com/group/anarchistica/

I forgot to mention it, and I doubt it would have helped, but AB2 does also accept .zip instead of .ab:s.

You probably won't find .rst anywhere because they are the armylist files produced when you save a roster in AB and there's seldom any reason to put those anywhere. I you want any I can provide some of my armylists.

[edit again] I have a lot of documentation on AB, this may or may not be of value to you?
I also have the programs used to create the .ab:s and convert them back into text files. Or so I believe. They come as part of the ABConstruction kit IIRC, which you can also find on the anarchistica webpages.

[IBBoard]

I tripped over mention of .rst as rosters at some point. We should hopefully be able to support them, but it'll have lower priority than getting the game data (.ab files) loaded.

Thanks for supplying the link - I'll see what I can find. Documentation about AB2 may be useful. It is interesting if AB2 accepted .zips as well. The new .ab files certainly aren't renamed Zips, so they either changed the .ab format or decided that they only wanted to support their own custom format.

In terms of the file creator, I've got the AB3 construction kit and it has been quite useful in working out some bits of the file format as it lets me build some "almost but not quite" identical files.


[edit] I've just snagged the WHFB AB2 file and overall it looks like the file table is the same. Some of the data in the header is different (it seems to say "Army Builder" and then have the file comment rather than the variety of versions etc that AB3 has) but it should be something that we can distinguish to be able to load both with one plugin.

Now, if only I could work out what to do with this damned compression!

[snowblizz]

Found this in the CKit documentation

This is where the ABExport tool comes in. The ABExport tool is used to package up one or more Army Builder files into a single import file, much like PKZip would accomplish. The created file is compressed using a compression algorithm that is compatible with the one used in PKZip. The key differences are that you can include comments within your import file and that Army Builder is able to readily import one of these files automatically. This approach eliminates the need for the user to use any special tools on his computer. All that the user needs to do is download the compressed file, then go into Army Builder and import the file. The contents of the file are automatically decompressed and copied into the proper location on the user’s computer.

Can't vouch for its veracity. They start in the beginning of the document with a "you don't need to worry about what's inside a .ab file".

[IBBoard]

Thanks for digging that out. I suspect that relates to AB2 rather than AB3, since the AB3 .ab files won't open as Zips when renamed as .zip and their structure does look like some proprietary format (it now has various header details about minimum required versions).

I'll take a look at CKit and see if there is anything else I can find.

In related news, LoneWolf Development are slow. I emailed them back on the 13th about the file format and got an auto-responder saying they would get back to me in 2-3 business days. Still no reply and we're now on the 29th!

[snowblizz]

Thanks for digging that out. I suspect that relates to AB2 rather than AB3, since the AB3 .ab files won't open as Zips when renamed as .zip and their structure does look like some proprietary format (it now has various header details about minimum required versions).

I'll take a look at CKit and see if there is anything else I can find.

In related news, LoneWolf Development are slow. I emailed them back on the 13th about the file format and got an auto-responder saying they would get back to me in 2-3 business days. Still no reply and we're now on the 29th!

Yup, this was indeed related to AB2.x the only verison I'm familiar with (and the one I'm mostly interested in).
But 3.x should have a similar CKit and documentation. I was thinking it should have a prgram like the 2.x does to "unpack" the .ab:s if you can reverse engineer that.. or if LW tells you the structure. I guess it ain't high priority...
BTW worst response time I had once was 1,5 years. The response apologizing that they were a bit behind and working on it... it ceased to matter about 1,49 years before though. At least they replied