Streaming .tar.gz files requires less memory, as we don't need to
remember metadata about each file for the end of directory record.
Signed-off-by: Graham <gpe@openrs2.org>
I'm still not particularly happy with this: if the JS5 download
finishes before HTTP, it'll time out and kill the whole process.
Similarly, because it takes so long to import the indexes and as we
can't fetch groups in parallel with that, it can often time out early
during the process.
In the long term, I think I am going to try and move most of the logic
outside of the Netty threads and communicate between threads with queues
or channels. This would also allow us to run multiple JS5 clients in
parallel.
The code also needs some tidying up, particularly constants in the
Js5ChannelHandler constructors.
Signed-off-by: Graham <gpe@openrs2.org>
This has the convenient side effect of us no longer needing to exclude
the byteunits dependency from minimisation.
Signed-off-by: Graham <gpe@openrs2.org>
I've decided to reverse my decision in
e3e0094b43.
If you treat empty loc groups as valid, some OSRS caches do have 100%
valid keys. Being able to pick these out in the list of caches will be
useful.
Signed-off-by: Graham <gpe@openrs2.org>
This speeds up the resolved_* views by a reasonable amount, though it
does mean we won't be able to use the smarter resolution logic (which is
far too slow anyway at the moment, so I'm not sure what I'm going to do
about that in the future...)
Signed-off-by: Graham <gpe@openrs2.org>
This doesn't fix all cases, as a master index can have multiple sources,
each with a distinct copy of the same (archive, group, checksum,
version) tuple. However, it's probably as good as we'll be able to do
automatically - and it'll work particularly well for master indexes
downloaded directly over JS5, where we won't have done multiple imports
of the same cache.
Signed-off-by: Graham <gpe@openrs2.org>
These functions reduce the amount of group resolution logic
significantly, concentrating it in a single place. In addition to the
usual code de-duplication benefits, many of the queries are now much
simpler as the complexity is hidden behind the function calls.
This change also allows us to make the group resolution logic more
complicated. The first change is that the functions are guaranteed to
only return a single row, which was not true with the old JOIN-based
approach. The row that is chosen is chosen deterministically.
The resolution logic will probably be improved in the future, so we can
make a better decision where there are multiple possible groups, due to
collisions.
Signed-off-by: Graham <gpe@openrs2.org>
There are a few collisions in the production archive. I suspect these
are due to poorly modified caches, and tracking the source(s) of each
group will make it easier to determine which cache is probably
problematic.
This change also has the benefit of removing a lot of the hacky source
name/description merging logic.
Signed-off-by: Graham <gpe@openrs2.org>
