The OpenRS2 project will provide the code for running such a service, but the project won't run an instance of it publicly.
Features:
CLI for importing/exporting caches.
Web interface for exporting caches and XTEA keys. I don't think we'll allow anyone to import caches on the web interface, to allow a human to manually check their provenance (given how easy it is to generate colliding CRC-32 checksums). Of course, the web interface would encourage people to get in touch to submit any data they have.
Write a JS5 client for OSRS to automatically import new OSRS caches in the future.
API endpoint for automatically receiving candidate XTEA keys.
Ask RuneLite/OpenOSRS about receiving XTEA keys from their endpoints.
Service for verifying candidate XTEA keys against encrypted files whose key is unknown.
CLI for importing clients, loaders and native libraries.
Remember that some of these will be stored in caches people contribute (in .dat0, .dat1, .dat3, etc. files).
Web interface for exporting clients, loaders and native libraries.
Service for downloading the current gamepack and parameters from OSRS.
Allow people to mirror the whole database:
Daily Postgres dumps + an rsync server?
Open up port 5432 and allow anyone to stand up their own replica? Might make Postgres upgrades tricky.
Tracking name hashes in the database might be nice - could then present the known name hashes in the web interface.
RS3/NXT support?
How should we resolve collisions (if any occur)? Similarly, it'd also be nice to have an option for obtaining a missing group from the closest revision.
Static file server (HTTP/FTP/rsync) for anything that doesn't fit into the service described above (e.g. RSC/3xx clients/caches - we probably have so few of these and they're very small that I don't think it's worth automating them).
as well as old Rune-Server etc. downloads and asking people on Discord.
~~The OpenRS2 project will provide the code for running such a service, but the project won't run an instance of it publicly.~~
Features:
* CLI for importing/exporting caches.
* Web interface for exporting caches and XTEA keys. I don't think we'll allow anyone to import caches on the web interface, to allow a human to manually check their provenance (given how easy it is to generate colliding CRC-32 checksums). Of course, the web interface would encourage people to get in touch to submit any data they have.
* Write a JS5 client for OSRS to automatically import new OSRS caches in the future.
* API endpoint for automatically receiving candidate XTEA keys.
* Ask RuneLite/OpenOSRS about receiving XTEA keys from their endpoints.
* Service for verifying candidate XTEA keys against encrypted files whose key is unknown.
* CLI for importing clients, loaders and native libraries.
* Remember that some of these will be stored in caches people contribute (in `.dat0`, `.dat1`, `.dat3`, etc. files).
* Web interface for exporting clients, loaders and native libraries.
* Service for downloading the current gamepack and parameters from OSRS.
* Allow people to mirror the whole database:
* Daily Postgres dumps + an rsync server?
* Open up port 5432 and allow anyone to stand up their own replica? Might make Postgres upgrades tricky.
* Tracking name hashes in the database might be nice - could then present the known name hashes in the web interface.
* RS3/NXT support?
* How should we resolve collisions (if any occur)? Similarly, it'd also be nice to have an option for obtaining a missing group from the closest revision.
* Static file server (HTTP/FTP/rsync) for anything that doesn't fit into the service described above (e.g. RSC/3xx clients/caches - we probably have so few of these and they're very small that I don't think it's worth automating them).
Existing archives to pull old data from:
* https://www.moparisthebest.com/rs/
* https://archive.runestats.com/
* https://displee.com/archive/
* https://rs-hacking.com/resources/
* https://github.com/open-osrs/service-xtea, https://xtea.openosrs.dev/get
* https://github.com/abextm/osrs-cache
* https://github.com/RuneStar/cache-names
* https://runearchive.org/
as well as old Rune-Server etc. downloads and asking people on Discord.
The JS5 client needs adjusting to make it tolerate the TCP connection closing unexpectedly.
Add length, uncompressed_length and uncompressed_checksum to the index_groups table.
Track how complete each master index is (in terms of files and XTEA keys). Not sure of the best way to do this - triggers or a materialized view?
Think about how we can ensure this will be compatible with RS3 in the future - I think the big problem is 4 vs 2 byte version numbers in containers. (Edit: actually, a bigger problem might be that the sqlite files don't contain JS5 containers and we won't be able to make the compressed CRCs match.) I think the solution is to ignore the SQLite files entirely and only download the cache over JS5/HTTP.
* [x] The JS5 client needs adjusting to make it tolerate the TCP connection closing unexpectedly.
* [x] Add length, uncompressed_length and uncompressed_checksum to the index_groups table.
* [x] Track how complete each master index is (in terms of files and XTEA keys). Not sure of the best way to do this - triggers or a materialized view?
* [x] ~~Think about how we can ensure this will be compatible with RS3 in the future - I think the big problem is 4 vs 2 byte version numbers in containers. (Edit: actually, a bigger problem might be that the sqlite files don't contain JS5 containers and we won't be able to make the compressed CRCs match.)~~ I think the solution is to ignore the SQLite files entirely and only download the cache over JS5/HTTP.
Why not use a cloud platform to perform this function, as well as the additional features that come with it? Data backups would be a byproduct of this.
It would remove the necessity for users to do this themselves, as well as reduce concerns from your hosting side.
> Allow people to mirror the whole database:
Why not use a cloud platform to perform this function, as well as the additional features that come with it? Data backups would be a byproduct of this.
Here's what google can do for data replication: https://cloud.google.com/sql/docs/mysql/replication
It would remove the necessity for users to do this themselves, as well as reduce concerns from your hosting side.
The OpenRS2 project will provide the code for running such a service, but the project won't run an instance of it publicly.Features:
.dat0,.dat1,.dat3, etc. files).Existing archives to pull old data from:
as well as old Rune-Server etc. downloads and asking people on Discord.
Think about how we can ensure this will be compatible with RS3 in the future - I think the big problem is 4 vs 2 byte version numbers in containers. (Edit: actually, a bigger problem might be that the sqlite files don't contain JS5 containers and we won't be able to make the compressed CRCs match.)I think the solution is to ignore the SQLite files entirely and only download the cache over JS5/HTTP.Why not use a cloud platform to perform this function, as well as the additional features that come with it? Data backups would be a byproduct of this.
Here's what google can do for data replication: https://cloud.google.com/sql/docs/mysql/replication
It would remove the necessity for users to do this themselves, as well as reduce concerns from your hosting side.