The groups field is populated with the group capacity, not the group
size. (Thanks Ominious.)
Update the test to cover this case, by adjusting the test index so its
size and capacity are distinct.
Signed-off-by: Graham <gpe@openrs2.org>
I think this dates back to me experimenting with deobfuscating both the
client and HD client at the same time. We don't do that any more.
Signed-off-by: Graham <gpe@openrs2.org>
This improves deobfuscation of a static field initializer in the 667
client that looks like:
a = b = new X()
This transformer translates it to two separate expressions:
b = new X(); a = b
This allows the StaticFieldUnscrambler to move both fields
independently, rather than them both being forced to remain in their
scrambled class.
Signed-off-by: Graham <gpe@openrs2.org>
667 has an initializer like `a = b = new X()`. This broke the previous
version of the unscrambler, as it treated it as two separate
initializers: `a = b = new X()` and `b = new X()`. When the fields were
moved, the overlapping instructions were removed twice from the original
<clinit> method, making its size negative.
Even if this were fixed, it is still not safe to move the initializers,
as `new X()` would end up being executed twice, not once.
This commit fixes the problem by treating any simple initializer that
overlaps with any other simple initializer as a complex initializer.
Signed-off-by: Graham <gpe@openrs2.org>
This commit makes the following improvements:
- Converts the list of instructions in the entry and exit block to a
set, which makes checking for containment more efficient.
- Removes redundant excluded field filtering, which is already handled
by the unscramble() method.
- Treats fields written with PUTSTATIC outside the entry or exit block
as complex, instead of just fields read with GETSTATIC.
- Treats fields with multiple simple initializers as complex.
- Treats fields where we fail to extract a simple initializer as
complex. This fixes a bug where we accidentally thought those fields
had no initializer.
Signed-off-by: Graham <gpe@openrs2.org>
This is a slightly nicer solution than running a combined transformer
until it reaches a fixed point: we know we'd only need two passes,
rather than an arbitrary number of passes.
The 667 client uses various exception types beyond RuntimeException in
ZKM exception obfuscation handlers, including Throwable, Exception,
EOFException and IOException. This separation also allows us to remove
the type constraint for ZKM handlers.
Signed-off-by: Graham <gpe@openrs2.org>
The build 667 client has some ZKM exception handlers inside Jagex's
exeption tracing handlers. In a single pass, the regex is only capable
of matching the former and not the latter. The ZKM exception handlers
were removed, confusingly leaving Jagex exception handlers that do match
the CATCH_MATCHER regex.
Signed-off-by: Graham <gpe@openrs2.org>
Polar posting about these recently in Discord jogged my memory that I'd
missed this flag off, previously I called this FLAG_HIGHLIGHT.
I've also added the official Jagex names to the documentation, thanks to
Polar.
I don't think 550 supports any of the flags after DEDICATEDACTIVITY.
Signed-off-by: Graham <gpe@openrs2.org>
- It's now used for OSRS as well as NXT.
- We retain buildMinor if buildMajor == server_version, so we don't have
to keep retrying minor versions.
Signed-off-by: Graham <gpe@openrs2.org>
removeAll removes all instances of an item in the right list from the
left list, but we only want to remove a single instance.
Signed-off-by: Graham <gpe@openrs2.org>
This previously worked by chance with the PE32 executables we saw, but
it didn't work with the PE32+ executables we saw.
Signed-off-by: Graham <gpe@openrs2.org>
There are still some gaps but I want to get this committed and possibly
deployed before doing further work.
Remaining items include:
- Mach-O support
- New engine loader ArtifactLink support
- Post-668 client support
- FunOrb support
Signed-off-by: Graham <gpe@openrs2.org>
It's now used for both XTEA and ISAAC keys, and there's nothing
XTEA-specific about it so there's no need to duplicate it.
Signed-off-by: Graham <gpe@openrs2.org>
This makes caching simpler to reason about (no Vary header) and more
efficient (also because there is no Vary header).
Signed-off-by: Graham <gpe@openrs2.org>
We've now found both the CIO and Netty backends to be problematic in
certain cases (CIO with lots of concurrent requests, Netty with
backpressure). Hopefully Jetty is a bit more battle-tested!
Signed-off-by: Graham <gpe@openrs2.org>
If the checksum changes in the future then the flyway_schema_history
table needs to be adjusted, which is awkward.
Signed-off-by: Graham <gpe@openrs2.org>
This commit adds initial support for negotiating the ISAAC session key,
creating accounts, and checking world suitability.
Signed-off-by: Graham <gpe@openrs2.org>
This is relatively easy as OpenNXT doesn't use the actual SQLite cache
format - it still uses JS5-compressed containers, rather than ZLIB.
Signed-off-by: Graham <gpe@openrs2.org>
I'm currently working on a CreateResponse class, which is even more
distinct from LoginResponse - so I've decided to separate it. For
consistency, separating the JS5 login responses seems sensible too.
Signed-off-by: Graham <gpe@openrs2.org>
The client's UI makes it easy to submit invalid dates not supported by
LocalDate. Rather than throwing an exception in a codec, it'd be good to
support representing these for:
* Allowing the development of a debugging proxy server that sits between
the client and game server.
* Making it easier to send the invalid date of birth response to the
creation request.
We'll still switch to LocalDate as early as possible in the packet
handler in the future.
Signed-off-by: Graham <gpe@openrs2.org>
There are now three additional abstract PacketCodec classes:
FixedPacketCodec, VariableBytePacketCodec and VariableShortPacketCodec.
The PacketLength class has been removed, as it is no longer required.
The main reason for this change is that the create suggested names
packet is a bit of an oddball: its length field measures the size of the
packet in longs, not bytes. The codec for this packet will be able to
inherit from PacketCodec directly to implement the custom length logic.
Signed-off-by: Graham <gpe@openrs2.org>
I've also removed the comment - I'm going to have to do something
similar for CREATE_DOWNSTREAM, and I think it's a bit more obvious with
the more distinctive name.
This allows us to deal with a single corner case of the same opcode
being used for two slightly different login response packets, depending
on the request.
For most packets we know (or have a very good guess at) the official
Jagex name, based on the NXT/TFU leaks.
The document will be fleshed out with the payloads and a more detailed
description of every packet in the future.
Signed-off-by: Graham <gpe@openrs2.org>
There's no UPDATE_ZONE_FULL_ENCLOSED packet, so clearing and then
populating a zone actually requires two packets -
UPDATE_ZONE_FULL_FOLLOWS followed by UPDATE_ZONE_PARTIAL_ENCLOSED.
Signed-off-by: Graham <gpe@openrs2.org>
We fsync the temp file before doing the atomic move, which I think is
necessary if the underlying filesystem re-orders the operations (e.g. if
the atomic move is performed before the writes to the temp file have
been flushed to disk).
Similarly we fsync the parent dir before returning. For a single atomic
write in isolation this probably isn't important, but probably is useful
if a sequence of atomic writes is performed in order.
Signed-off-by: Graham <gpe@openrs2.org>
This fixes a bug when deobfuscating the build 225 client, which has code
that looks like a flow obstructor initializer, but it reads an argument
rather than a static variable.
Signed-off-by: Graham <gpe@openrs2.org>
Bit of a corner case, but if we ever encounter an index with a checksum
and version of 0 that resolves then this will ensure the statistics are
consistent between the overall cache and the index row.
Signed-off-by: Graham <gpe@openrs2.org>
These replace the whirlpool, group count and total uncompressed length
columns, which were kind of useless - in particular:
* The group count is also represented with the new stats columns.
* The total uncompressed length overflows, as some indexes are now
larger than 2 gigabytes. One of the new stats columns contains the
compressed size of each archive, which isn't too different.
Signed-off-by: Graham <gpe@openrs2.org>
Everything is read-only and unauthenticated, so there is no security
risk. Hopefully this will allow some cool projects based on AJAX to be
developed.
Signed-off-by: Graham <gpe@openrs2.org>
This will allow us to import FunOrb caches without worrying about the
risk of collisions with the main set of RuneScape caches.
Signed-off-by: Graham <gpe@openrs2.org>
lo should've been masked rather than hi. I've switched the code to mask
both the low and high DWORDs for simplicity.
Signed-off-by: Graham <gpe@openrs2.org>
Useful for running directly on my server, which is headless Linux. (The
current extract tool is a GUI Windows app.)
Signed-off-by: Graham <gpe@openrs2.org>
Creating the InputStream first allows us to immediately fail if a key is
invalid, without having the chance of allocating a huge ByteBuf based on
an incorrect length.
Signed-off-by: Graham <gpe@openrs2.org>
Equivalent to the built-in GZIPOutputStream, except it produces output
that is bit-for-bit identical to Jagex's implementation.
Signed-off-by: Graham <gpe@openrs2.org>
There were two problems:
- The current intention is for the uncompressed array to be 1 byte long,
not 6 bytes - as map groups only have a single file, we don't need the
group header.
- The comment was out of the sync with the intended code (replace
uncompressed with a single element byte array containing zero).
Signed-off-by: Graham <gpe@openrs2.org>
It removes the archive/group prefix and the 0xFF markers.
Unfortunately using Js5ResponseDecoder here is tricky - perhaps it
could've been done with an EmbeddedChannel.
Signed-off-by: Graham <gpe@openrs2.org>
Not really used for anything yet, but Gradle moans because
xtea-plugin/src/test/ exists and it can't find the JUnit engine.
Signed-off-by: Graham <gpe@openrs2.org>
If the future is not successful the channel may have been closed. The
pipeline of a closed channel is empty, so attempting to remove handlers
from it would cause an exception.
Signed-off-by: Graham <gpe@openrs2.org>
The API endpoint isn't always updated immediately, so this is useful for
gathering keys semi-manually if they're needed more urgently.
Signed-off-by: Graham <gpe@openrs2.org>
Keys are now initially imported into a key_queue table, which is never
locked exclusively - allowing the API endpoint to function while the
brute forcer is running. The brute forcer moves all pending keys in the
queue to the keys table before running the actual brute forcing.
Signed-off-by: Graham <gpe@openrs2.org>
In some cases the capacity would be one byte greater than necessary.
I've added a comment explaining why the calculation works and a unit
test that ensures it is calculated correctly for all lengths between 1
and 511*3+1 bytes.
Signed-off-by: Graham <gpe@openrs2.org>
Streaming .tar.gz files requires less memory, as we don't need to
remember metadata about each file for the end of directory record.
Signed-off-by: Graham <gpe@openrs2.org>
This implementation supports:
* Encoding in column- or row-major order (based on a very rough
heuristic).
* Preserving the colours of transparent pixels.
* Cutting off the borders of a transparent frame (if there is no colour
to preserve).
Signed-off-by: Graham <gpe@openrs2.org>
There are two main reasons for this change: by default, logback uses
stdout (!) and not stderr. This caused problems in programs like
packclass/unpackclass, where the packclass or JAR files were mixed in
with Netty's debug logging.
Secondly, the debug logging from lots of third-party libraries was
annoying (as it made it difficult to pick out OpenRS2-specific log
messages), so I've disabled lots of it.
Signed-off-by: Graham <gpe@openrs2.org>
It's basically a simplified MIDI file (lacks support for SMPTE and most
meta/system messages) encoded in a smarter way.
This implementation can decode anything in the 550 cache, convert it to
MIDI format and then re-encode it bit-for-bit identically to the
original.
Signed-off-by: Graham <gpe@openrs2.org>
I'm still not particularly happy with this: if the JS5 download
finishes before HTTP, it'll time out and kill the whole process.
Similarly, because it takes so long to import the indexes and as we
can't fetch groups in parallel with that, it can often time out early
during the process.
In the long term, I think I am going to try and move most of the logic
outside of the Netty threads and communicate between threads with queues
or channels. This would also allow us to run multiple JS5 clients in
parallel.
The code also needs some tidying up, particularly constants in the
Js5ChannelHandler constructors.
Signed-off-by: Graham <gpe@openrs2.org>
The main reason I'm doing this is that I want to rename TextureMeta to
Material, which fits in with the naming of its JS5 archive.
Signed-off-by: Graham <gpe@openrs2.org>
This fixes a 1 in 2^32 chance that we'll fail to fetch the world list on
startup.
Of course, updating the player counts will still be borked if we get a
collision, but at least the client won't break entirely.
Signed-off-by: Graham <gpe@openrs2.org>
The previous implementation didn't attempt to walk the call graph, so it
wouldn't remove all unused methods.
Closes#127.
Signed-off-by: Graham <gpe@openrs2.org>
This commit adds support for:
* Version negotiation.
* HEAD requests.
* Friendly error messages.
* More robust reference counting.
Signed-off-by: Graham <gpe@openrs2.org>
It supports reading and writing a cache backed by a Store, as well as
standalone .js5 files.
I'm not quite as happy with this as I am with the lower-level API yet,
and there are still a few remaining TODOs.
Signed-off-by: Graham <gpe@openrs2.org>
I think this makes more sense, as we're loading it from the etc
directory, just like etc/config.yaml, and it isn't required by most of
the users of the crypto module.
Signed-off-by: Graham <gpe@openrs2.org>
This prevents the archive module from failing on a machine without the
cache installed (like the archive.openrs2.org server).
Signed-off-by: Graham <gpe@openrs2.org>
This ensures the server sends a master index compatible with the client
even if all the indexes in the cache use the original Js5Index format.
Signed-off-by: Graham <gpe@openrs2.org>
These will be used by the high-level cache API, where we don't want to
expose mutable versions of the group/file types as that would allow the
index/cache to get out of sync.
Signed-off-by: Graham <gpe@openrs2.org>
I still don't know what Jagex call this, but calling it pack at least
avoids confusion with the files contained within the archive.
Signed-off-by: Graham <gpe@openrs2.org>
This is closer to the client's implementation, which ignores the gzip
header/trailer entirely and just decompresses the inner DEFLATE stream.
While Jagex always produce valid gzip files, some cache editors in the
private server scene do not set the checksum/length correctly in the
trailer. I'm planning to add an option to use the lax gzip
implementation to support reading these caches.
Signed-off-by: Graham <gpe@openrs2.org>
It's more concise and I suspect Kotlin's implementation is better than
Java's, as it seems to take available() into account.
Signed-off-by: Graham <gpe@openrs2.org>
This has the convenient side effect of us no longer needing to exclude
the byteunits dependency from minimisation.
Signed-off-by: Graham <gpe@openrs2.org>
I've decided to reverse my decision in
e3e0094b43.
If you treat empty loc groups as valid, some OSRS caches do have 100%
valid keys. Being able to pick these out in the list of caches will be
useful.
Signed-off-by: Graham <gpe@openrs2.org>
The group unpacking code exposes a fastutil type. (In the long run this
might be replaced with an array, allowing us to revert this commit or
drop the fastutil dependency entirely.)
Signed-off-by: Graham <gpe@openrs2.org>
This speeds up the resolved_* views by a reasonable amount, though it
does mean we won't be able to use the smarter resolution logic (which is
far too slow anyway at the moment, so I'm not sure what I'm going to do
about that in the future...)
Signed-off-by: Graham <gpe@openrs2.org>
This doesn't fix all cases, as a master index can have multiple sources,
each with a distinct copy of the same (archive, group, checksum,
version) tuple. However, it's probably as good as we'll be able to do
automatically - and it'll work particularly well for master indexes
downloaded directly over JS5, where we won't have done multiple imports
of the same cache.
Signed-off-by: Graham <gpe@openrs2.org>
These functions reduce the amount of group resolution logic
significantly, concentrating it in a single place. In addition to the
usual code de-duplication benefits, many of the queries are now much
simpler as the complexity is hidden behind the function calls.
This change also allows us to make the group resolution logic more
complicated. The first change is that the functions are guaranteed to
only return a single row, which was not true with the old JOIN-based
approach. The row that is chosen is chosen deterministically.
The resolution logic will probably be improved in the future, so we can
make a better decision where there are multiple possible groups, due to
collisions.
Signed-off-by: Graham <gpe@openrs2.org>
There are a few collisions in the production archive. I suspect these
are due to poorly modified caches, and tracking the source(s) of each
group will make it easier to determine which cache is probably
problematic.
This change also has the benefit of removing a lot of the hacky source
name/description merging logic.
Signed-off-by: Graham <gpe@openrs2.org>
Postgres would otherwise deadlock forever. I'm not sure why this worked
when testing - perhaps we always had fewer than BATCH_SIZE rows?
Signed-off-by: Graham <gpe@openrs2.org>
We still want to merge the build and timestamp as caches can be
associated with multiple build numbers, and I always want us to use the
earliest number.
Signed-off-by: Graham <gpe@openrs2.org>
Some master indexes are used across multiple builds. It makes sense to
use the minimum build number, much like how we use the minimum
timestamp.
Signed-off-by: Graham <gpe@openrs2.org>
The user running the migration might not have superuser permissions.
Using IF NOT EXISTS allows a root user to install the extension
manually, and then the migration will succeed as an unprivileged user.
Signed-off-by: Graham <gpe@openrs2.org>
It isn't solved yet, but once we've committed to v1 of the archiving
service we won't want to edit V1__init.sql ever again to avoid changing
its checksum (Flyway complains if the checksum changes).
Signed-off-by: Graham <gpe@openrs2.org>
I'm going to try to minimise use of this (as per
https://github.com/google/guice/wiki/Avoid-Injecting-Closable-Resources).
For example, I'm going to inject a pooling DataSource rather than
Connection objects, as per the advice at the end of the page. However,
HikariCP's DataSource implementation is itself Closeable.
Signed-off-by: Graham <gpe@openrs2.org>
The previous code included too many rows as the use of LEFT JOINs meant
candidate group orws with an incorrect container_id were still included
in the results.
Using an IN clause with a subquery allows us to remove those rows,
though it's a bit hacky.
(Really I want to be able to use the JOIN on the right side of a LEFT
JOIN to restrict the rows that appear in the results of the LEFT JOIN,
but that doesn't seem to be possible.)
This is similar to the issue fixed by
a920570f04.
Signed-off-by: Graham <gpe@openrs2.org>
Some non-empty loc groups are also unreachable, so I think this was
quite deceptive - e.g. on some OSRS revisions, we'll probably never hit
100% of the keys even if we exclude empty loc groups.
We can include the empty loc flag in the list of missing keys on the
per-cache pages instead.
Signed-off-by: Graham <gpe@openrs2.org>
The previous code over-counted as the use of LEFT JOINs meant candidate
group rows with the incorrect container_id were still included in the
results. Using an IN clause with a subquery allows us to remove those
rows, though it's a bit hacky. (Really I want to be able to use a JOIN
on the right side of a LEFT JOIN to restrict the rows that appear in the
results of the LEFT JOIN, but that doesn't seem to be possible.)
Signed-off-by: Graham <gpe@openrs2.org>
While the client doesn't appear to be capable of producing a trailing
block with a non-zero next block pointer (though I may have misread the
code), there are caches out there in the wild with trailing non-zero
next block pointers.
When it is reading a group, the client ignores these.
Therefore, for compatibility with the client and existing caches, this
commit removes the StoreCorruptException thrown in this case.
Signed-off-by: Graham <gpe@openrs2.org>
This commit also adds support for populating the whirlpool column, and
ensures version is set to 0 for the ORIGINAL master index format.
Signed-off-by: Graham <gpe@openrs2.org>
Green indicates we've collected a full set. For indexes, red indicates
some are missing (as this is a critical problem - the client won't start
at all if an index is missing). Yellow indicates groups or keys are
missing, as this is less critical (the client will likely work in most
cases).
Signed-off-by: Graham <gpe@openrs2.org>
I'm not sure if the auto-detection code works: I'm assuming that the new
format was introduced at the same time as the lengths flag in Js5Index,
but I haven't confirmed this.
Signed-off-by: Graham <gpe@openrs2.org>
This commit makes the following changes:
- Uses XteaKeyDeserializer instead of reading the key array manually.
- Adds support for an object of map squares to keys, which is the format
used by the OpenOSRS XTEA endpoint.
- Throws exceptions if the input is malformed, rather than silently
ignoring problems.
Signed-off-by: Graham <gpe@openrs2.org>
This was actually only compatible with OpenOSRS's code (RuneLite uses a
completely different API). Furthermore, I don't think it's really going
to be necessary as we've arranged access to RuneLite's XTEA API.
Signed-off-by: Graham <gpe@openrs2.org>
Like the tmp_indexes table, there's no need for it - we can read the
data we need from the index_groups rows we've just inserted.
Signed-off-by: Graham <gpe@openrs2.org>
This is consistent with processMasterIndex and processIndex,
particularly now that the processGroup logic is slightly more
complicated.
Signed-off-by: Graham <gpe@openrs2.org>
This makes us behave like a standard client that only keeps a single
copy of each group in its cache. This ensures we can at least detect
(crc32, version) collisions for a particular group, rather than silently
skipping colliding cached groups.
A disadvantage is that more bandwidth usage is required, especially if
the download is interrupted.
Signed-off-by: Graham <gpe@openrs2.org>
There's no need for it - we can read the index checksums and versions
from the master_index_archives rows we've just inserted.
Signed-off-by: Graham <gpe@openrs2.org>
The CTE is now declared as NOT MATERIALIZED to ensure Postgres is able
to push the WHERE master_index_id condition inside it.
Signed-off-by: Graham <gpe@openrs2.org>
As key validation has to uncompress the data anyway to confirm the key
is valid, it seems silly to uncompress twice given everywhere we
performed key validation immediately uncompressed the container
afterwards.
Signed-off-by: Graham <gpe@openrs2.org>
This commit groups the testImplementation and testRuntimeOnly
dependencies. It also depends on junit-jupiter-engine specifically at
test runtime, rather than all of junit-jupiter. This is the
configuration shown in the JUnit documentation.
Signed-off-by: Graham <gpe@openrs2.org>
There's no real use for these yet, but they might be useful with NXT
caches.
We don't need a compressed_length column because it's easy to determine
the length of a BYTEA column within the database.
Signed-off-by: Graham <gpe@openrs2.org>
Although it isn't necessary, we might as well as it doesn't take up much
extra space and we already store all the properties for all groups and
files.
Signed-off-by: Graham <gpe@openrs2.org>
This commit also changes the way the master index format detection
works, as the previous scheme could not distinguish VERSIONED from
WHIRLPOOL.
Signed-off-by: Graham <gpe@openrs2.org>
The master index uses RSA signatures in some builds. As such, we need to
be able to encrypt with a private key and decrypt with a public key.
Signed-off-by: Graham <gpe@openrs2.org>
This will reduce the impact of checksum/version collisions, as a
collision would have to happen between two indexes of the same archive
rather than any two indexes.
Signed-off-by: Graham <gpe@openrs2.org>
Originally I used a singleton map if possible, to reduce overhead in the
common case of a group containing a single file. However, the
work-in-progress Cache class needs to mutate the map.
Eventually my plan is to replace the Int2ObjectSortedMaps in Js5Index
and Group with arrays, solving both the overhead and mutation problems.
Signed-off-by: Graham <gpe@openrs2.org>
This fixes the downloader getting stuck if there's a particularly large
file, such that we've read all the current inbound data but haven't sent
a new request yet as we still have 200 in-flight.
Signed-off-by: Graham <gpe@openrs2.org>
As we're comparing a real directory tree with Jimfs, we need to ensure
the platform matches - otherwise it gets confused, and thinks \ is part
of a UNIX file name.
Most of these names come directly from the NXT/TFU clients, though a few
are made up as we don't know the official name.
Signed-off-by: Graham <gpe@openrs2.org>
I think it's actually fine to stick the jars in the nonfree/lib folder.
When we package up the entire OpenRS2 distribution, the fat jar is
placed in the lib folder, so using nonfree/lib is consistent with that.
Signed-off-by: Graham <gpe@openrs2.org>
These effectively duplicate the master index tables, but in a less
flexible manner - as they don't support importing a master index where
some of the indexes are missing.
This commit also combines MasterIndexImporter with CacheImporter, to
make it easier to re-use code.
Signed-off-by: Graham <gpe@openrs2.org>
This is in preparation for adding a new deob module which invokes the
bytecode deobfuscator, decompiler and AST deobfuscator in turn.
Signed-off-by: Graham <gpe@openrs2.org>
This is slightly per correct than nonfree/lib, as the /lib
directory is for architecture-specific code. The original client code is
not architecture-specific.
Signed-off-by: Graham <gpe@openrs2.org>
Closes#116.
This commit also changes the output directory from
nonfree/var/cache/bundle to nonfree/var/cache/client.
Signed-off-by: Graham <gpe@openrs2.org>
Excluding the 0xFF trailers, the first block is actually 512 bytes, and
the remaining blocks are all 511 bytes. I had mistakenly written the
code and tests on the basis of all blocks being 511 bytes.
The Js5ResponseDecoder now works with OSRS.
Signed-off-by: Graham <gpe@openrs2.org>
I have a small collection of these from when I ran a service that polled
the JS5 server for the master index back in 2009. We'll probably be able
to find some in old Rune-Server threads too.
They'll be useful for verifying caches with an unclear provenance.
Signed-off-by: Graham <gpe@openrs2.org>
I have spent a while thinking about the best way to implement this,
especially after I discovered that OSRS treats the packets sent during
the handshake stage as normal packets (see share/doc/protocol/login.md
for a longer explanation).
I have settled on a design where the same Rs2Decoder and Rs2Encoder
instances, which deal with decoding packet opcodes and lengths, will
remain the same for the entire connection - unless it switches into a
mode where a completely different framing scheme is used, such as
JAGGRAB and JS5.
Rs2Decoder and Rs2Encoder both support optional opcode encryption. The
initial cipher is set to NopStreamCipher. It will be set to an
IsaacRandom instance during the login process.
As the scrambled opcodes used in-game overlap with the opcodes used
during the login process, a class called Protocol is used to represent a
set of opcodes that can be used in a particular state. The protocol can
be changed at any time. Rs2Decoder sets isSingleDecode to true initially
to help do this safely, but once in the final in-game state this can be
disabled for performance.
Unlike previous servers, like Apollo and ScapeEmulator, intermediate
frame objects are not used. Rs2Decoder and Rs2Encoder call PacketCodec
to directly translate an inbound ByteBuf into a POJO, or an outbound
POJO into a ByteBuf. This saves a memory allocation per packet.
PacketCodec implementations must provide both an encode() and a decode()
method for every packet. This will ensure the library is usable as both
a client and server, allowing the development of tools, such as a JS5
client, a game server stress-tester or even a proxy server for
inspecting the flow of packets.
PacketCodec implementations may optionally override getLength() to
return their exact length, to optimise the size of the ByteBuf passed to
the decode() method. (It is not necessary to override getLength() for
fixed packets, as the size is already known.)
The 256-element packet length array is automatically generated by the
Protocol class based on the length attribute of the PacketCodec objects
passed to it. This is less error-prone than populating the array
manually. However, it does mean stub implementations of every packet
packet will need to be provided during development - all lengths must be
known in advance to ensure the client and server remain in sync.
For the moment, it is legal to throw NotImplementError from a decode()
method. Rs2Decoder catches this exception, prints a warning and then
proceed to the next packet. Once all packets have been implemented, this
code may be removed.
Signed-off-by: Graham <gpe@openrs2.org>
Spotted by Desetude.
I think this was an accident, as the similar Js5Index.Entry class can't
be a data class (as it contains an array).
Signed-off-by: Graham <gpe@openrs2.org>
There are still a small number of items I need to flesh out. I also need
to document response codes for packets other than 16/18, as some
response codes differ from the standard set.
Signed-off-by: Graham <gpe@openrs2.org>
This should make it a bit more obvious to people finding the GitHub
mirrors where the issues/pull requests are kept.
Signed-off-by: Graham <gpe@openrs2.org>
This will allow us to replace IsaacRandom in the future Netty
encoders/decoders with an implementation suitable for use in a unit
test.
Signed-off-by: Graham <gpe@openrs2.org>
Some static members are not scrambled in the client. Maintaining a
one-to-one mapping between instanced and static classes makes
refactoring easier in these cases.
The browser control filter is removed as we now get the same
functionality for free.
Signed-off-by: Graham <gpe@openrs2.org>
This was a mistake. I'm not sure what it is - it's always hard-coded to
zero in the client script, so it's a bit of a mystery.
Signed-off-by: Graham <gpe@openrs2.org>
Implementing ReferenceCounted with delegation doesn't quite work, as the
methods returning ReferenceCounted are not overriden to return
Container. This can cause code inside Netty to unintentionally unwrap
the Container.
Signed-off-by: Graham <gpe@openrs2.dev>
I'm not 100% sure on the name of this. I'm mostly basing it on the fact
that we go byte[] -> RawModel ->Â Model, and the NXT Model class has a
ConstructFromModelRawRT7 method.
Signed-off-by: Graham <gpe@openrs2.dev>
I think the fact that ParticleSystem inherits from SceneGraphNode in the
TFU client is probably not relevant, as its SceneGraphNode is totally
different to the Node class.
Signed-off-by: Graham <gpe@openrs2.dev>
I think this fits slightly better - Entry reminds me of Map.Entry, which
is similar. There's no need for Node subclasses to be suffixed with Node
in general - we just don't have any choice with classes like StringNode
(we have to avoid a collision with Java's String type).
Signed-off-by: Graham <gpe@openrs2.dev>
This is useful in the archiving service code, as it can extract them for
insertion into the database without the overhead of creating and then
throwing away an IntArray.
Signed-off-by: Graham <gpe@openrs2.dev>
This will eventually hold the names of each JS5 archive in 550, as and
when I confirm the names from the NXT beta client still match up with
the functionality of each archive in 550.
Currently it only holds then name of index 255.
Signed-off-by: Graham <gpe@openrs2.dev>
In the future I will give these more descriptive names.
I'm not yet sure if the client class is the most appropriate location,
but it fits with the current heuristic of placing static members in the
class in which they are initialised if they are initialised in an
instance method.
Signed-off-by: Graham <gpe@openrs2.dev>
This is required for the signature of writeUnsignedIntSmart to be the
same as writeShort, which the Js5Index encoder will make use of.
Signed-off-by: Graham <gpe@openrs2.dev>
Gradle seems to re-run this every time, so Drone ended up building the
KDoc twice (once in the build stage and again in the publish stage).
Removing it from the build stage ensures Drone only builds KDoc once.
Signed-off-by: Graham <gpe@openrs2.dev>
The output is significantly nicer.
External documentation links don't seem to work correctly at the moment,
so I have commented those out for now.
Signed-off-by: Graham <gpe@openrs2.dev>
I'm not yet convinced we have all the files - I've since realised the
"complete" 550 cache floating around is probably modified and includes
some files from a later revision.
Signed-off-by: Graham <gpe@openrs2.dev>
This already caught some cases of public members that should have been
private and one case where the inferred type was too specific.
Signed-off-by: Graham <gpe@openrs2.dev>
Unit tests to follow - I've been working on these classes for a few days
now, so I wanted to make sure they get backed up in the repository.
Signed-off-by: Graham <gpe@openrs2.dev>
Will be used by the packclass implementation. Like Cp1252Charset, the
JDK isn't guaranteed to provide an implementation of this charset.
Signed-off-by: Graham <gpe@openrs2.dev>
Slightly different to the standard implementation, as like the client it
considers NUL to be an unmappable character. (Furthermore, the standard
implementation isn't in StandardCharsets.)
It also provides fast methods for encoding/decoding a single byte/char
at a time.
Signed-off-by: Graham <gpe@openrs2.dev>
This allows us to make them private (rather than internal) and the type
as a whole is immutable.
It does mean we need to convert the key to an IntArray every time we use
it now. However, I hope that the JVM will be smart enough to inline the
toIntArray() method and also smart enough to realise it doesn't escape,
allowing it to allocate it on the stack rather than the heap.
This commit also adds methods for converting XteaKeys to hex, and
converting from hex/integer arrays to an XteaKey object.
Signed-off-by: Graham <gpe@openrs2.dev>
I suspect Jagex have a really clever optimizer capable of inlining the
Js5Index class (or perhaps they did it manually?)
Signed-off-by: Graham <gpe@openrs2.dev>
While the Resource class is technically not quite immutable (as it
contains a byte array), it's close enough as the array is only ever
read.
Signed-off-by: Graham <gpe@openrs2.dev>
This obviously doesn't work if the slot is loaded before something is
stored in it, but it does allow us to remove unused arguments whose slot
is re-used for another purpose.
Signed-off-by: Graham <gpe@openrs2.dev>
There are still some more contexts we can do this in - unary/binary
expressions and invocation contexts. I think Fernflower might handle the
former, but it doesn't handle the latter. Unfortunately, JavaParser's
method resolution code is still a bit too buggy to do the latter at the
moment.
Signed-off-by: Graham <gpe@openrs2.dev>
The BinaryExprTransformer deals with this to an extent. However, it
requires multiple passes to do so. Adding support to AddSubTransformer
allows it to be done in a single pass.
Signed-off-by: Graham <gpe@openrs2.dev>
It'd be nice if we could make it run a single pass (and there are
probably some improvements to reduce the number of required passes, as
I'm not convinced I'm running the transformations in a particularly
sensible order).
However, I think making it run in a single pass would be quite difficult
as some of the transformations need to propagate through else if chains
or nested if blocks. It's much easier to just run all the
transformations until they settle than fixing each individual
transformation.
Signed-off-by: Graham <gpe@openrs2.dev>
It's fairly similar to the ComplementTransformer, and translates
expressions like:
!a == !b => a == b
!a == b => a != b
Signed-off-by: Graham <gpe@openrs2.dev>
calculateResolveType() is much more expensive than checking if an
expression is an integer literal, so swap the conditions around.
Signed-off-by: Graham <gpe@openrs2.dev>
Some character literals are treated as integers by Fernflower, as the
complement operator always evaluates to an integer, even if its operand
is a character. This is fine, but ComplementTransformer removes some
complement transformers - leaving integer literals where Fernflower
would normally insert a character literal.
This transformer converts complemented integers to char literals in some
cases where it makes sense (it is difficult to perform this
transformation across every type of expression in JavaParser).
Signed-off-by: Graham <gpe@openrs2.dev>
Unfortunately we can't use the compiler to guarantee k isn't changed,
though making it internal will help. When the JVM (and Kotlin) get value
types, we might be able to improve on this (e.g. by making it an inline
class of four integers).
Signed-off-by: Graham <gpe@openrs2.dev>
This commit improves support for simplifying a series of multiple
additions/subtractions and doing so in a single pass.
Signed-off-by: Graham <gpe@openrs2.dev>
This commit makes two changes:
* Uses the is operator instead of the isXXX methods provided by
JavaParser, allowing smart casts to be used.
* Wraps unsupported expressions with a unary minus expression, rather
than throwing an exception.
Signed-off-by: Graham <gpe@openrs2.dev>
This commit updates the tests to ensure all combinations of 0-7 bytes of
leading/trailing blocks of unencrypted bytes work correctly with all
test vectors.
Signed-off-by: Graham <gpe@openrs2.dev>
We could use LocalVariableTable in theory, but it's more complicated as
we'd need to do live variable analysis and type inference. It's much
easier to leave that for the decompiler.
Signed-off-by: Graham <gpe@openrs2.dev>
The next commit will introduce an ArgRef class which uses a MemberRef
directly, rather than a MemberRef partition.
Signed-off-by: Graham <gpe@openrs2.dev>
This allows two different classes in different modules to be refactored
to the same name - for example, Node in client and unpackclass.
Under the hood, it is implemented by prefixing each class name with the
library name and an exclamation mark (which is highly unlikely to appear
in a class name, as it is invalid Java syntax).
At first, prefixing class names with the library name feels like a bit
of a hack. However, it is much simpler than trying to track libraries
throughout the existing code. Furthermore, it allows us to avoid
forking ASM classes like Remapper.
The Fernflower driver was also changed to deobfuscate each library in
its own context, rather than trying to decompile them all in one go - by
the time classes reach Fernflower, the prefixes have already been
removed and Fernflower can't deal with duplicate class names either.
Signed-off-by: Graham <gpe@openrs2.dev>
I think it is better to be explicit here, instead of relying on ASM's
current lack of equals()/hashCode() methods in the AbstractInsnNode
class.
Signed-off-by: Graham <gpe@openrs2.dev>
I suspect it is excluded on Jagex's end so the client deals correctly
with any class loading failures caused by the absence of the
netscape.javascript.JSObject class.
Signed-off-by: Graham <gpe@openrs2.dev>
This will make naming libraries easier, as we won't have to thread
library names through every Library{Reader,Writer} implementation.
Signed-off-by: Graham <gpe@openrs2.dev>
JS5 seems to refer to many things, so I'm intending to name Jagex's
custom class file format "packclass" instead of "js5". My reasoning is
that the tool for unpacking it is called unpackclass. If you use the
same logic behind naming the Pack200 packer unpack200, then "packclass"
is a reasonable name for Jagex's class file format.
Signed-off-by: Graham <gpe@openrs2.dev>
This commit adds support for mapping:
* class names
* field names
* field owners
* method names
* method owners
It does not yet support argument names, local variable names or
re-ordering method arguments. Local variable names are blocked on
Fernflower patches and method argument re-ordering is blocked on the IR.
Signed-off-by: Graham <gpe@openrs2.dev>
At the moment, this means hard-coding the HD client but we should
reintroduce support for the SD and unsigned clients in the future.
This change provides a few benefits:
* Performance (~3x faster), as I generally only care about the HD
client.
* Allows us to continue injecting the NameMap for the HD client without
worrying about how to avoid injecting it (or support a different
NameMap) for the SD and unsigned clients.
Signed-off-by: Graham <gpe@openrs2.dev>
This fixes a bug where classes disappeared from the map if they did not
need to be re-compiled when incremental compilation is enabled.
Signed-off-by: Graham <gpe@openrs2.dev>
volatile is pointless if a field is final - even on arrays/objects, as
it only applies to the reference, not the contents of the target.
Signed-off-by: Graham <gpe@openrs2.dev>
If we don't do this then we incorrectly mark some fields as final, due
to writes present inside if blocks removed by ConstantArgTransformer.
Signed-off-by: Graham <gpe@openrs2.dev>
This fixes a bug where fields could never become static if a class had a
mixture of static and non-static fields and constructors.
Signed-off-by: Graham <gpe@openrs2.dev>
The new entry set functionality allows us to distinguish fields NEVER
written from fields we haven't explored yet. This allows us to find more
EXACTLY_ONCE fields in constructors with loops.
Signed-off-by: Graham <gpe@openrs2.dev>
Propagating them looks a bit silly, because the arguments are unused. I
think this problem mainly occurs in constructors, which the dummy
argument obfuscator doesn't touch.
Signed-off-by: Graham <gpe@openrs2.dev>
The aim of this change is to ensure that the automatically chosen class,
field and method names stay as similar as possible between runs.
Signed-off-by: Graham <gpe@openrs2.dev>
This change paves the way to feed the NameMap into the TypedRemapper,
such that refactored names (and static member movements) are preserved
across deobfuscator runs.
Signed-off-by: Graham <gpe@openrs2.dev>
We don't need to track the class version and maxStack value: the
Library::remap will already have access to them (and always needs to, as
it has to fiddle with {Class,Field,Method}Nodes to actually move the
field).
Signed-off-by: Graham <gpe@openrs2.dev>
My rough plan for combining static scrambling and remapping is to split
Library::remap into three passes:
* Remove static methods, fields and initializers from the library,
storing them in a temporary location.
* Pass all classes through ClassNodeRemapper, as we do now.
* Add static methods, fields and initializers to their new classes,
remapping as we do so.
This ensures a ClassNode is never in a state where it has a mixture of
remapped and non-remapped fields, methods or instructions. This is
important to ensure no conflicts can occur when we use the refactored
names from the NameMap, rather than the auto-generated names.
It means TypedRemapper needs the ability to provide the instructions
that make up a field's initializer, such that Library::remap can move
these instructions to a different InsnList. The new getFieldInitializer
method and FieldInitializer type support this.
Signed-off-by: Graham <gpe@openrs2.dev>
Fixes a problem where `Constructor<?> var0 =
clazz.getDeclaredConstructor(); return var0.newInstance();` was transformed to
`Constructor<?> var0 = clazz.getDeclaredConstructor(); return
var0.getDeclaredConstructor().newInstance();`
Signed-off-by: Desetude <harry@desetude.com>
This reverts commit b6bba95435.
Although this code is more complicated, it allows us to control the
destination class for fields individually (rather than an entire set at
a time). This is a requirement for name mapping.
I'm going to use it in multiple places, so I think it makes sense to
share it (at the expense of the asm module depending on the yaml
module).
Signed-off-by: Graham <gpe@openrs2.dev>
I thought it'd be nice to ensure we could represent any character in the
patterns. However, this isn't required (as the client only uses
alphabetical class, method and field names). Furthermore, the period
character is already unusable in class names due to the MemberRef string
parsing.
Signed-off-by: Graham <gpe@openrs2.dev>
These are useful for identifying the files in moparisthebest's archive,
as the file names include the CRC-32 value for cache busting.
I'm keeping the SHA-256 checksums - they provide further assurance as
it's very easy to generate a CRC-32 collision.
Signed-off-by: Graham <gpe@openrs2.dev>
While the jaggl jar isn't actually obfuscated, this change means we'll
support the @OriginalXXX annotations (which is useful for local
variables, whose names aren't retained). The unused method, visibility
and final transformers will also tidy up the code slightly.
The new class/member filtering infrastructure is used to retain the
names of every class, method and field.
Signed-off-by: Graham <gpe@openrs2.dev>
<init> methods do not override each other. This caused us to incorrectly
calculate visibility in some cases in the VisibilityTransformer.
Signed-off-by: Graham <gpe@openrs2.dev>
The new system will make it easier to port the deobfuscator to different
revisions.
There are two main changes:
- The addition of a Profile class, which contains a list of excluded
classes, methods and fields, and the maximum obfuscated name length.
It is passed to Transformers that require it with dependency
injection.
- New ClassFilter and MemberFilter infrastructure. The MemberFilter
class adds support for filtering fields and methods based on the
owner and descriptor, in addition to the name. This makes the filters
more precise than the previous system. It also supports globs, which
makes it easy to filter whole groups of classes, fields and methods
in one go.
The Profile class uses a ClassFilter and MemberFilters to
represent the list of excluded classes, methods and fields.
A separate benefit is the addition of a separate entry points filter
to the Profile class. Prior to this commit, many Transformers re-used
the excluded method filter to find entry points, which is less precise
(many of the excluded methods in 550 are not entry points).
Support for filtering methods by owner and descriptor in addition to
name allows the DEFAULT_PUBLIC_CTOR_CLASSES Set in VisibilityTransformer to
be combined with the entry points filter.
In the future it might be desirable to split the excluded method set
into three separate sets:
- One to represent methods that can't be renamed.
- One to represent methods whose signature can't be changed.
- One to represent methods that can't be removed.
Signed-off-by: Graham <gpe@openrs2.dev>
This allows users to compile and run the deobfuscator's output without
access to the nonfree repository. It will be particularly useful when
the deobfuscator can make use of the deob-map files.
Signed-off-by: Graham <gpe@openrs2.dev>
It returns a single NameMap combining all the maps in share/deob-map.
This will ultimately be fed into the RemapTransformer/TypedRemapper in
the deobfuscator.
Signed-off-by: Graham <gpe@openrs2.dev>
Our convention is one Guice module per subproject, where necessary.
Although this currently only includes YamlModule, it will include a
future DeobfuscatorMapModule.
Signed-off-by: Graham <gpe@openrs2.dev>
The Resource::compress method already holds entire files in memory at
once, as does the client-side loader. We might as well do the same on
the server-side.
Signed-off-by: Graham <gpe@openrs2.dev>
This will provide a few benefits:
- Some of the implementations can now be turned into objects, reducing
memory allocation.
- A single Resource.compressLibrary() method will be able to take a
LibraryWriter, reducing duplication.
Signed-off-by: Graham <gpe@openrs2.dev>
Some of these method names only make sense when accompanied with the
class name. I think most of these were mistakes in the Java -> Kotlin
port.
Signed-off-by: Graham <gpe@openrs2.dev>
This name is slightly more descriptive and consistent with loader.p12,
which also contains a private key.
I've also removed the public.key file. It isn't read by OpenRS2 and the
two files could get out of sync with each other, which would be
confusing.
It's easy to extract the public key with OpenSSL if necessary:
openssl pkey -in game.key -pubout -out public.key
Signed-off-by: Graham <gpe@openrs2.dev>
It uses multibindings to allow additional Jackson modules to be
registered across different Guice modules while using a single
ObjectMapper for the whole application.
Signed-off-by: Graham <gpe@openrs2.dev>
Local variable name tracking with the @Pc annotation relies on changes
not yet included in a release. I'm not intending to do another
Fernflower release yet, as I've still got more changes to make, but I
would like the @Pc tracking to work.
Signed-off-by: Graham <gpe@openrs2.dev>
This reverts commit 7415ae6ec9.
The problems that caused me to write this commit in the first place have
been fixed. Using IDEA's test runner directly is nicer - e.g. for code
coverage integration.
Java 11's Pack200 implementation complains if stack map frames are present, and
passes the classes through unmodified. I'm not sure why.
Stack map frames are only required in Java 7, so for 6 and below a reasonable
workaround is to not emit them.
A nice side effect of this change is that the jar files created by the bundler
are be smaller. The pack200 files are too, and would be even if the classes
were not passed through unmodified.
I'm ignoring the deobfuscator mapping files for now, but when the
deobfuscator is finished then I'll add them to the repository and keep
them up to date.
Signed-off-by: Graham <gpe@openrs2.dev>
The next commit will make MemberRef comparable, with the owner/name
being used for sorting before the descriptor. This makes the output of
the fields consistent with the sort order.
Signed-off-by: Graham <gpe@openrs2.dev>
Now we don't use Maven, the dev.openrs2:openrs2 Maven coordinates are
available for use. I think using them for the end-user application
packaged as a .zip file is reasonable.
I don't intend to move the all submodule to the root build.gradle.kts
file, as this would cause us to lose some of the benefits of supporting
Gradle configuration on demand.
Signed-off-by: Graham <gpe@openrs2.dev>
Fixes#99, which was caused by a method call in an unused method with a
non-constant dummy argument. This prevented the argument being removed,
but it wasn't obvious why as the method preventing the argument removal
was also removed.
I think relying on just the isBodyEmpty() check is more correct too, as
the end label of a try/catch is exclusive. Furthermore, we don't want to
delete try/catch nodes where the middle of the body is not dead (even if
the start and end are dead).
Signed-off-by: Graham <gpe@openrs2.dev>
The new transformer uses a different approach to the old one. It starts
exploring the call graph from the entry points, recursively analysing
method calls. Methods are only re-analysed if their possible argument
values change, with the Unknown value being used if we can't identify a
single integer constant at a call site. This prevents us from recursing
infinitely if the client code does.
While this first pass does simplify branches in order to ignore dummy
method calls that are never evaluated at runtime, it operates on a copy
of the method (as we initially ignore more calls while the argument
value sets are smaller, ignoring fewer calls as they build up).
A separate second pass simplifies branches on the original method and
inlines singleton constants, paving the way for the UnusedArgTransformer
to actually remove the newly unused arguments.
This new approach has several benefits:
- It is much faster than the old approach, as we only re-analyse methods
as required by argument value changes, rather than re-analysing every
method during every pass.
- It doesn't require special cases for dealing with mutually recursive
dummy calls. The old approach hard-coded special cases for mutually
recursive calls involving groups of 1 and 2 methods. The code for this
wasn't clean. Furthermore, while it was just about good enough for the
HD client, the SD client contains a mutually recursive group of 3
methods. The new approach is capable of dealing with mutually
recursive groups of any size.
Finally, the new transformer has a much cleaner implementation.
Signed-off-by: Graham <gpe@openrs2.dev>
IDEA's test runner doesn't work very well with multi-module projects,
particularly where there are different versions of jars in the
classpath.
Signed-off-by: Graham <gpe@openrs2.dev>
We only need to check for the ACC_FINAL flag once at the start of
transformCode(), given the insn.owner == clazz.name condition. This
change also removes the redundant getNode() call - we already have a
reference to the ClassNode.
Signed-off-by: Graham <gpe@openrs2.dev>
As I'm splitting it up into smaller modules (e.g. compress and crypto) I
think util is a more appropriate name for the remainder.
Signed-off-by: Graham <gpe@openrs2.dev>
This is useful for checking the CRC-32 checksum of files used by the
loader. It outputs the checksum in signed decimal format, which is the
format used in the file name suffixes in the loader (e.g.
unpackclass_-1911426584.pack).
Signed-off-by: Graham <gpe@openrs2.dev>
These are useful for manipulating the files used by the loader, some of
which are compressed with DEFLATE or headerless gzip.
Signed-off-by: Graham <gpe@openrs2.dev>
The non-HD client appears to have been compiled with javac 1.1 or 1.2,
which compiles synchronized blocks differently to modern versions of
Java. For example, it uses a single try/catch block (modern versions use
two, the latter of which is recursive), subroutines (modern versions do
not) and the range of the try block is narrower.
Fernflower doesn't understand this old style of bytecode, and produces
empty synchronized blocks that do not cover the correct range.
The MONITORENTER sequence is also slightly different: it uses
ASTORE ALOAD MONITORENTER after pushing the monitor variable, rather
than DUP ASTORE MONITORENTER. This doesn't break Fernflower, but does
make it introduce a pointless assignment to a variable only used in the
synchronized() statement, rather than inlining the expression.
This commit introduces a transformer which fixes up Java 1.1-style
synchronized blocks just enough for Fernflower to be able to decompile
them cleanly. (It doesn't add a recursive try/catch, as Fernflower
ignores them.)
Signed-off-by: Graham <gpe@openrs2.dev>
The unsigned client has a Class.forName() method with two differences:
* The NoClassDefFoundError is created with a slightly different sequence
of instructions.
* The arms of the if/else statement for initializing the synthetic field
if it is null are swapped.
This commit adds support for translating Class.forName() methods with
either or both of these differences to the modern LDC form.
Signed-off-by: Graham <gpe@openrs2.dev>
game_unpacker.dat is the name of the file when downloaded in the
.jagex_cache_32/runescape directory. However, the other files containing
code are all named after the files on the web/JAGGRAB server.
This commit uses unpackclass.pack for consistency with all the other
files.
Signed-off-by: Graham <gpe@openrs2.dev>
This ensures the bundler/deobfuscator can only see classes that are part
of the Java runtime, and not classes from the bundler/deobfuscator
themselves.
Signed-off-by: Graham <gpe@openrs2.dev>
I'm very keen on being able to use the jdk.jartool module (which is only
available in JDK11 onwards) as it allows us to avoid shelling out to
jarsigner entirely.
11 is the current LTS release and is already widespread in Linux
distributions, so I think it's reasonable to require it.
This commit removes the jsobject module. We might need to re-add it in
the future (if jdk.jsobject is removed from the JDK). However, it was
only necessary in 8 because modern versions of 8 tended to be
distributed without plugin.jar. JDK11 is distributed with the
jdk.jsobject module.
Signed-off-by: Graham <gpe@openrs2.dev>
The split is no longer required - we haven't run javah automatically
since splitting the natives out into a separate repository.
Signed-off-by: Graham <gpe@openrs2.dev>
This is a special case, so I've just hard-coded the GlEnum name and
value. Luckily its value is only used in the client in an OpenGL
context.
Signed-off-by: Graham <gpe@openrs2.dev>
It's actually hard to determine whether to use a vendor suffix or not.
For example, the client sometimes uses a bunch of *ARB functions with
_ARB constants. However, it uses the non-_ARB constant with glGetFloat.
It's very hard to fix this inconsistency while still using vendor
suffixes. It seems much easier to just use the constants without the
suffixes all the time.
Signed-off-by: Graham <gpe@openrs2.dev>
Removes GOTO instructions that point to the next instruction, typically
left over from stripping exceptions inserted by the obfuscator.
Although modern decompilers can handle such GOTOs fine, removing them
makes for a nicer IR representation.
Signed-off-by: Major <major@emulate.rs>
It's possible that future versions of ASM will introduce additional
types of virtual node. Only counting real instructions should make this
more reliable.
Signed-off-by: Graham <gpe@openrs2.dev>
While it isn't useful for retaining the name of the constructor itself,
as they don't really have names, it will be useful for tracking argument
names and positions.
Signed-off-by: Graham <gpe@openrs2.dev>
The transformer does _not_ check for use via reflection. The only cases
in the 550 and OSRS clients where methods are accessed via reflection
are either 1) JRE classes, 2) when the method name is sent from the
server.
PSVM and methods declared in TypedRemapper.EXCLUDED_METHODS are never
removed.
Signed-off-by: Major <major@emulate.rs>
Fernflower fails to decompile any exception handler with an end_pc
(`to` in Fernflower nomenclature) equal to the length of the code array,
even though this is permitted in the class file spec. This transformer
inserts a NOP at the end of any code array that has such an exception
handler.
Signed-off-by: Major <major@emulate.rs>
I think we have the correct behaviour in all cases now: fields will
never override fields. Only non-static methods may override other
methods.
Signed-off-by: Graham <gpe@openrs2.dev>
A static member should not override an equivalent member in a superclass
or superinterface. This commit makes us skip the union() calls for
static methods, such that the two disjoint sets are not incorrectly
joined together.
Signed-off-by: Graham <gpe@openrs2.dev>
This commit:
* Renames isMethodImmutable to isMethodRenamable and flips the value of
the boolean it outputs accordingly.
* Updates all uses of isMethodRenamable to account for this.
* Makes the internal implementation more similar to isClassRenamable.
Signed-off-by: Graham <gpe@openrs2.dev>
This picks up a glGetString(GL_EXTENSIONS) constant.
Don't translate gl calls within the jaggl.opengl class itself. This
causes failures as there are native methods with 0 and 1 suffixes. These
methods don't exist in the OpenGL registry, causing the transformer to
throw an exception.
Signed-off-by: Graham <gpe@openrs2.dev>
This fixes a GL_OBJECT_COMPILE_STATUS_ARB constant, but I'm doing
everything else in the same set of enums for consistency.
Signed-off-by: Graham <gpe@openrs2.dev>
GETSTATIC, PUTSTATIC and INVOKESTATIC (strangely) use the same field and
method resolution algorithms as their non-static equivalents.
While this doesn't cause problems for 550, as the *STATIC instructions
seem to always refer to the class containing the static definition,
Major reports that OSRS clients sometimes change the owner to a
subclass. This broke the old transformer.
This commit fixes it by applying the old to new owner mapping to an
entire disjoint set at a time.
Signed-off-by: Graham <gpe@openrs2.dev>
This allows us to avoid needing to worry about manipulating stack frames
in individual transformers, which adds lots of complexity. It's much
easier to just make ASM generate them for us.
Signed-off-by: Graham <gpe@openrs2.dev>
I'm intending to use this in a patched version of Fernflower to track
the original index of each local variable's STORE instruction, allowing
us to retain local variable names even if the deobfuscator is run
multiple times.
This is necessary for a future commit that will track the original
indexes of AbstractInsnNodes throughout the deobfuscator, with the aim
of using this information to track local variables in the decompiled
code.
I suspect this will also improve performance, as we don't need to
re-allocate all of the tree objects.
I've been considering this for a long time, and have decided to switch to it at
the last minute before opening the repository up publicly. My reasons include:
* It's a much simpler license. GPL's complexity adds some risk - for example, it
might be incompatible with future open-source licenses (like the well-known
GPLv2/Apache v2 incompatibility problem). The "or any later version" clause
requires placing some trust in the Free Software Foundation.
* The simplicity makes it easier for people to understand and comply with the
license.
* Dishonest users who disobey the GPL would have an advantage over honest users
who refuse to do so. The ISC license provides a much more even playing field.
* OpenRS2 will primarily be server software accessible over a network. As such,
the GPL can do little to prevent use of the code in a proprietary system, as
the code is never distributed. (While the AGPL would fix this, I have already
discounted it. Enforcement would be too difficult and dishonest users would
have an unfair advantage.)
* It's much easier to switch to a stricter license in future versions, if it
turns out that is desirable (as the ISC license allows users to sublicense
the code). However, switching from the GPL to the ISC license requires all
copyright holders to grant permission.
* Other open-source projects in the community, such as Apollo, use the ISC
license and will be able to make use of OpenRS2 code if they so desire.
I've removed the FAQ entry about the reasons for using the GPL license, as I
think the ISC license is less controversial and therefore does not require an
entry.
I've discussed this with Desetude, and he's okay with his commit being
relicensed.
Fernflower was previously placing constructors in the middle of instance
methods. They look much nicer at the top.
This commit also ensures static methods appear before instance methods.
These might be added to the user's ~/gradle.properties file, at which
point having a per-project prefix is useful.
Ideally I'd have preferred to use openrs2.repo.{username,password}.
However, Gradle properties with dots in them can't be configured with
the ORG_GRADLE_PROJECT_* environment variables, which we use in the
Jenkinsfile.
By moving all the static fields in a single class in one go, we can just
combine the <clinit> methods together. This allows us to remove all the
really complicated logic for extracting initializers, and also allows us
to move the last few array fields with complex initializers.
Depending on the class loading order, moving them can result in arrays
being empty, which breaks the client's functionality (e.g. it can't get
past the JS5 CRC check).
The client scrambles static fields and methods - moving each to a
random class where it does not logically belong.
This transformer moves all static methods to a new set of empty classes.
The intention is that the human refactoring the deobfuscated client can
move the static methods around with an IDE to classes where it makes
sense.
In theory we could use heuristics to move some methods around - e.g.
ideas I have include:
* If the method takes no arguments, move it to the same class as its
return type.
* If it's a void method that takes one argument, move it to the same
class as its single argument.
* Group related methods together based on where they are called.
However, some of the heuristics are probably fairly complicated to
implement. Furthermore, as they are heuristics, they will make mistakes.
In the naive approach, the Static* classes generated by this transformer
effectively act as a queue of methods that a human has yet to move to an
appropriate class. If the Static* classes are ever emptied after
refactoring, then we will know that all static methods have been moved
to an appropriate class.
We wouldn't be able to guarantee this if some methods were moved to
Class* classes with heuristics. The heuristics would also be fairly
complicated to implement. Therefore, the transformer does not use
heuristics.
Field scrambling support will be included in a future commit.
The obfuscator (or compiler, potentially?) converts INVOKEVIRTUAL
instructions to INVOKESPECIAL in the following circumstances:
* The owner of the method is the same as the class containing the
INVOKE instruction.
* The owner's ACC_FINAL flag is set.
When those two conditions are met, and when ACC_SUPER is set on the
owner (which is always true in the RuneScape client, and the flag is
ignored in modern JVMs), then INVOKESPECIAL and INVOKEVIRTUAL are
equivalent.
This has not caused problems until now. However, the future static
scrambling transformer will break the first condition in some cases, as
it moves methods between classes.
This transformer reverses the obfuscation, such that INVOKEVIRTUAL is
used again. INVOKEVIRTUAL instructions may be moved between classes
without complication.
In Java 1.4 and earlier, the compiler creates a synthetic method for
invoking Class.forName() and a static field for caching the Class<?>
instance.
This will not interact well with the static scrambling transformer. The
obfuscator strips some synthetic flags. Furthermore, the field and
method need to be moved together with the code that uses them for
decompilers like Fernflower to recognise them and convert them back to
class literals.
If Fernflower misses one, the decompiled Class.forName() helper does not
compile cleanly (as the bytecode is missing a CHECKCAST).
Rather than complicating the future static scrambling transformer, it's
easier to convert these to use LDC and upgrade the .class file version
to Java 5.
Calling super.write(byte[], int, int) in FilterOutputStream didn't work,
as it called super.write(byte). This is inefficient and also caused us
to fiddle with skipBytes twice, causing us to skip the incorrect number
of bytes.
Desetude noticed that there are some cases where this transformation is
unsafe - for example: `i > i++`. Skipping the transform if _either_ side
of the comparison has a side effect, rather than _both_ sides, makes it
safe.
Our patched copy of Fernflower no longer produces char literals in
inappropriate places (e.g. bit shifts). This removes the requirement for
this option, allowing non-English string literals to read nicely.
I've been thinking about this for a while. On the one hand, using the
"or later" clause means we have to place some trust in the Free Software
Foundation. On the other hand, being able to adopt newer versions of the
license without having to contact all contributors is useful for
compatibility (e.g. Apache 2 and GPLv3).
I recently noticed that Section 14 states that new versions of the
license will be "similar in spirit" to the current version. I think this
reduces the amount of trust we need to place in the FSF, as I'd hope
that if a future license did diverge too much from the original spirit
that a court might not consider it valid.
Furthermore, if the FSF went totally crazy, there are many bigger and
more important projects who would experience the same problem.
