Streaming .tar.gz files requires less memory, as we don't need to
remember metadata about each file for the end of directory record.
Signed-off-by: Graham <gpe@openrs2.org>
I'm still not particularly happy with this: if the JS5 download
finishes before HTTP, it'll time out and kill the whole process.
Similarly, because it takes so long to import the indexes and as we
can't fetch groups in parallel with that, it can often time out early
during the process.
In the long term, I think I am going to try and move most of the logic
outside of the Netty threads and communicate between threads with queues
or channels. This would also allow us to run multiple JS5 clients in
parallel.
The code also needs some tidying up, particularly constants in the
Js5ChannelHandler constructors.
Signed-off-by: Graham <gpe@openrs2.org>
It supports reading and writing a cache backed by a Store, as well as
standalone .js5 files.
I'm not quite as happy with this as I am with the lower-level API yet,
and there are still a few remaining TODOs.
Signed-off-by: Graham <gpe@openrs2.org>
This prevents the archive module from failing on a machine without the
cache installed (like the archive.openrs2.org server).
Signed-off-by: Graham <gpe@openrs2.org>
This ensures the server sends a master index compatible with the client
even if all the indexes in the cache use the original Js5Index format.
Signed-off-by: Graham <gpe@openrs2.org>
These will be used by the high-level cache API, where we don't want to
expose mutable versions of the group/file types as that would allow the
index/cache to get out of sync.
Signed-off-by: Graham <gpe@openrs2.org>
It's more concise and I suspect Kotlin's implementation is better than
Java's, as it seems to take available() into account.
Signed-off-by: Graham <gpe@openrs2.org>
The group unpacking code exposes a fastutil type. (In the long run this
might be replaced with an array, allowing us to revert this commit or
drop the fastutil dependency entirely.)
Signed-off-by: Graham <gpe@openrs2.org>
While the client doesn't appear to be capable of producing a trailing
block with a non-zero next block pointer (though I may have misread the
code), there are caches out there in the wild with trailing non-zero
next block pointers.
When it is reading a group, the client ignores these.
Therefore, for compatibility with the client and existing caches, this
commit removes the StoreCorruptException thrown in this case.
Signed-off-by: Graham <gpe@openrs2.org>
I'm not sure if the auto-detection code works: I'm assuming that the new
format was introduced at the same time as the lengths flag in Js5Index,
but I haven't confirmed this.
Signed-off-by: Graham <gpe@openrs2.org>
As key validation has to uncompress the data anyway to confirm the key
is valid, it seems silly to uncompress twice given everywhere we
performed key validation immediately uncompressed the container
afterwards.
Signed-off-by: Graham <gpe@openrs2.org>
This commit also changes the way the master index format detection
works, as the previous scheme could not distinguish VERSIONED from
WHIRLPOOL.
Signed-off-by: Graham <gpe@openrs2.org>
Originally I used a singleton map if possible, to reduce overhead in the
common case of a group containing a single file. However, the
work-in-progress Cache class needs to mutate the map.
Eventually my plan is to replace the Int2ObjectSortedMaps in Js5Index
and Group with arrays, solving both the overhead and mutation problems.
Signed-off-by: Graham <gpe@openrs2.org>
