Bit of a corner case, but if we ever encounter an index with a checksum
and version of 0 that resolves then this will ensure the statistics are
consistent between the overall cache and the index row.
Signed-off-by: Graham <gpe@openrs2.org>
These replace the whirlpool, group count and total uncompressed length
columns, which were kind of useless - in particular:
* The group count is also represented with the new stats columns.
* The total uncompressed length overflows, as some indexes are now
larger than 2 gigabytes. One of the new stats columns contains the
compressed size of each archive, which isn't too different.
Signed-off-by: Graham <gpe@openrs2.org>
This will allow us to import FunOrb caches without worrying about the
risk of collisions with the main set of RuneScape caches.
Signed-off-by: Graham <gpe@openrs2.org>
Keys are now initially imported into a key_queue table, which is never
locked exclusively - allowing the API endpoint to function while the
brute forcer is running. The brute forcer moves all pending keys in the
queue to the keys table before running the actual brute forcing.
Signed-off-by: Graham <gpe@openrs2.org>
Streaming .tar.gz files requires less memory, as we don't need to
remember metadata about each file for the end of directory record.
Signed-off-by: Graham <gpe@openrs2.org>
I'm still not particularly happy with this: if the JS5 download
finishes before HTTP, it'll time out and kill the whole process.
Similarly, because it takes so long to import the indexes and as we
can't fetch groups in parallel with that, it can often time out early
during the process.
In the long term, I think I am going to try and move most of the logic
outside of the Netty threads and communicate between threads with queues
or channels. This would also allow us to run multiple JS5 clients in
parallel.
The code also needs some tidying up, particularly constants in the
Js5ChannelHandler constructors.
Signed-off-by: Graham <gpe@openrs2.org>
This has the convenient side effect of us no longer needing to exclude
the byteunits dependency from minimisation.
Signed-off-by: Graham <gpe@openrs2.org>
I've decided to reverse my decision in
e3e0094b43.
If you treat empty loc groups as valid, some OSRS caches do have 100%
valid keys. Being able to pick these out in the list of caches will be
useful.
Signed-off-by: Graham <gpe@openrs2.org>
This speeds up the resolved_* views by a reasonable amount, though it
does mean we won't be able to use the smarter resolution logic (which is
far too slow anyway at the moment, so I'm not sure what I'm going to do
about that in the future...)
Signed-off-by: Graham <gpe@openrs2.org>
