There are a few collisions in the production archive. I suspect these
are due to poorly modified caches, and tracking the source(s) of each
group will make it easier to determine which cache is probably
problematic.
This change also has the benefit of removing a lot of the hacky source
name/description merging logic.
Signed-off-by: Graham <gpe@openrs2.org>
Postgres would otherwise deadlock forever. I'm not sure why this worked
when testing - perhaps we always had fewer than BATCH_SIZE rows?
Signed-off-by: Graham <gpe@openrs2.org>
We still want to merge the build and timestamp as caches can be
associated with multiple build numbers, and I always want us to use the
earliest number.
Signed-off-by: Graham <gpe@openrs2.org>
Some master indexes are used across multiple builds. It makes sense to
use the minimum build number, much like how we use the minimum
timestamp.
Signed-off-by: Graham <gpe@openrs2.org>
I'm going to try to minimise use of this (as per
https://github.com/google/guice/wiki/Avoid-Injecting-Closable-Resources).
For example, I'm going to inject a pooling DataSource rather than
Connection objects, as per the advice at the end of the page. However,
HikariCP's DataSource implementation is itself Closeable.
Signed-off-by: Graham <gpe@openrs2.org>
The previous code included too many rows as the use of LEFT JOINs meant
candidate group orws with an incorrect container_id were still included
in the results.
Using an IN clause with a subquery allows us to remove those rows,
though it's a bit hacky.
(Really I want to be able to use the JOIN on the right side of a LEFT
JOIN to restrict the rows that appear in the results of the LEFT JOIN,
but that doesn't seem to be possible.)
This is similar to the issue fixed by
a920570f04.
Signed-off-by: Graham <gpe@openrs2.org>
This commit also adds support for populating the whirlpool column, and
ensures version is set to 0 for the ORIGINAL master index format.
Signed-off-by: Graham <gpe@openrs2.org>
Green indicates we've collected a full set. For indexes, red indicates
some are missing (as this is a critical problem - the client won't start
at all if an index is missing). Yellow indicates groups or keys are
missing, as this is less critical (the client will likely work in most
cases).
Signed-off-by: Graham <gpe@openrs2.org>
This commit makes the following changes:
- Uses XteaKeyDeserializer instead of reading the key array manually.
- Adds support for an object of map squares to keys, which is the format
used by the OpenOSRS XTEA endpoint.
- Throws exceptions if the input is malformed, rather than silently
ignoring problems.
Signed-off-by: Graham <gpe@openrs2.org>
This was actually only compatible with OpenOSRS's code (RuneLite uses a
completely different API). Furthermore, I don't think it's really going
to be necessary as we've arranged access to RuneLite's XTEA API.
Signed-off-by: Graham <gpe@openrs2.org>
Like the tmp_indexes table, there's no need for it - we can read the
data we need from the index_groups rows we've just inserted.
Signed-off-by: Graham <gpe@openrs2.org>
This is consistent with processMasterIndex and processIndex,
particularly now that the processGroup logic is slightly more
complicated.
Signed-off-by: Graham <gpe@openrs2.org>
This makes us behave like a standard client that only keeps a single
copy of each group in its cache. This ensures we can at least detect
(crc32, version) collisions for a particular group, rather than silently
skipping colliding cached groups.
A disadvantage is that more bandwidth usage is required, especially if
the download is interrupted.
Signed-off-by: Graham <gpe@openrs2.org>
There's no need for it - we can read the index checksums and versions
from the master_index_archives rows we've just inserted.
Signed-off-by: Graham <gpe@openrs2.org>