The previous code over-counted as the use of LEFT JOINs meant candidate
group rows with the incorrect container_id were still included in the
results. Using an IN clause with a subquery allows us to remove those
rows, though it's a bit hacky. (Really I want to be able to use a JOIN
on the right side of a LEFT JOIN to restrict the rows that appear in the
results of the LEFT JOIN, but that doesn't seem to be possible.)
Signed-off-by: Graham <gpe@openrs2.org>
This commit also adds support for populating the whirlpool column, and
ensures version is set to 0 for the ORIGINAL master index format.
Signed-off-by: Graham <gpe@openrs2.org>
Green indicates we've collected a full set. For indexes, red indicates
some are missing (as this is a critical problem - the client won't start
at all if an index is missing). Yellow indicates groups or keys are
missing, as this is less critical (the client will likely work in most
cases).
Signed-off-by: Graham <gpe@openrs2.org>
I'm not sure if the auto-detection code works: I'm assuming that the new
format was introduced at the same time as the lengths flag in Js5Index,
but I haven't confirmed this.
Signed-off-by: Graham <gpe@openrs2.org>
This commit makes the following changes:
- Uses XteaKeyDeserializer instead of reading the key array manually.
- Adds support for an object of map squares to keys, which is the format
used by the OpenOSRS XTEA endpoint.
- Throws exceptions if the input is malformed, rather than silently
ignoring problems.
Signed-off-by: Graham <gpe@openrs2.org>
This was actually only compatible with OpenOSRS's code (RuneLite uses a
completely different API). Furthermore, I don't think it's really going
to be necessary as we've arranged access to RuneLite's XTEA API.
Signed-off-by: Graham <gpe@openrs2.org>
Like the tmp_indexes table, there's no need for it - we can read the
data we need from the index_groups rows we've just inserted.
Signed-off-by: Graham <gpe@openrs2.org>
This is consistent with processMasterIndex and processIndex,
particularly now that the processGroup logic is slightly more
complicated.
Signed-off-by: Graham <gpe@openrs2.org>
This makes us behave like a standard client that only keeps a single
copy of each group in its cache. This ensures we can at least detect
(crc32, version) collisions for a particular group, rather than silently
skipping colliding cached groups.
A disadvantage is that more bandwidth usage is required, especially if
the download is interrupted.
Signed-off-by: Graham <gpe@openrs2.org>
There's no need for it - we can read the index checksums and versions
from the master_index_archives rows we've just inserted.
Signed-off-by: Graham <gpe@openrs2.org>
The CTE is now declared as NOT MATERIALIZED to ensure Postgres is able
to push the WHERE master_index_id condition inside it.
Signed-off-by: Graham <gpe@openrs2.org>
As key validation has to uncompress the data anyway to confirm the key
is valid, it seems silly to uncompress twice given everywhere we
performed key validation immediately uncompressed the container
afterwards.
Signed-off-by: Graham <gpe@openrs2.org>
There's no real use for these yet, but they might be useful with NXT
caches.
We don't need a compressed_length column because it's easy to determine
the length of a BYTEA column within the database.
Signed-off-by: Graham <gpe@openrs2.org>
Although it isn't necessary, we might as well as it doesn't take up much
extra space and we already store all the properties for all groups and
files.
Signed-off-by: Graham <gpe@openrs2.org>
This commit also changes the way the master index format detection
works, as the previous scheme could not distinguish VERSIONED from
WHIRLPOOL.
Signed-off-by: Graham <gpe@openrs2.org>
