This is consistent with processMasterIndex and processIndex,
particularly now that the processGroup logic is slightly more
complicated.
Signed-off-by: Graham <gpe@openrs2.org>
This makes us behave like a standard client that only keeps a single
copy of each group in its cache. This ensures we can at least detect
(crc32, version) collisions for a particular group, rather than silently
skipping colliding cached groups.
A disadvantage is that more bandwidth usage is required, especially if
the download is interrupted.
Signed-off-by: Graham <gpe@openrs2.org>
There's no need for it - we can read the index checksums and versions
from the master_index_archives rows we've just inserted.
Signed-off-by: Graham <gpe@openrs2.org>
The CTE is now declared as NOT MATERIALIZED to ensure Postgres is able
to push the WHERE master_index_id condition inside it.
Signed-off-by: Graham <gpe@openrs2.org>
As key validation has to uncompress the data anyway to confirm the key
is valid, it seems silly to uncompress twice given everywhere we
performed key validation immediately uncompressed the container
afterwards.
Signed-off-by: Graham <gpe@openrs2.org>
There's no real use for these yet, but they might be useful with NXT
caches.
We don't need a compressed_length column because it's easy to determine
the length of a BYTEA column within the database.
Signed-off-by: Graham <gpe@openrs2.org>
Although it isn't necessary, we might as well as it doesn't take up much
extra space and we already store all the properties for all groups and
files.
Signed-off-by: Graham <gpe@openrs2.org>
This commit also changes the way the master index format detection
works, as the previous scheme could not distinguish VERSIONED from
WHIRLPOOL.
Signed-off-by: Graham <gpe@openrs2.org>
This will reduce the impact of checksum/version collisions, as a
collision would have to happen between two indexes of the same archive
rather than any two indexes.
Signed-off-by: Graham <gpe@openrs2.org>
This fixes the downloader getting stuck if there's a particularly large
file, such that we've read all the current inbound data but haven't sent
a new request yet as we still have 200 in-flight.
Signed-off-by: Graham <gpe@openrs2.org>
These effectively duplicate the master index tables, but in a less
flexible manner - as they don't support importing a master index where
some of the indexes are missing.
This commit also combines MasterIndexImporter with CacheImporter, to
make it easier to re-use code.
Signed-off-by: Graham <gpe@openrs2.org>
I have a small collection of these from when I ran a service that polled
the JS5 server for the master index back in 2009. We'll probably be able
to find some in old Rune-Server threads too.
They'll be useful for verifying caches with an unclear provenance.
Signed-off-by: Graham <gpe@openrs2.org>
